He himself claims to be a bastard of House Lannister.1
Tywin is known to have sex with prostitutes, as with Shae.[2]
Additionally, Varys states that a Hand of the King ordered the construction of a secret passageway to Chataya's brothel, the same brothel that the yellow-haired and green-eyed prostitute Marei works at.[3]
He has physical and characteristic similarities to other Lannisters of Tywin's line.
He has yellow hair that he wears down in a tumble, a common trait of the Lannisters of Casterly Rock.1
He has fat lips, a trait he shares with Joffrey Baratheon.1
He makes easy japes, a trait he shares with Tyrion and Jaime Lannister.1
When asked if Tywin had any bastard offspring, George R. R. Martin merely answered that "[Tywin]'d have you flogged for the mere suggestion."[4]