Most people are actually "pedophiles" in denial, as most people are attracted to children. But you don't see these people raping babies very often.
This is a repressed sex-phobia that isn't understood very well. So a lot of bullshit folk psychology seems to get tossed around about the subject.