I don't think there's any one "answer" to what causes people to be sexually attracted to a certain category of others.
Homosexuality has been practiced in almost every culture at every point in time, and is seen in the animal kingdom (Bonobo monkeys I think are very sexual and are as related to humans as their more militant cousins, the chimp). If you've ever read Plato's "Symposium," you know that Greek men used to sleep with little boys, and it was culturally OK for this to happen (although this depended as well on the city-stated as well).
I think we all have a sexual power in us that needs to go somewhere, that needs to express itself. Freud called this the "libido". We are all sexual creatures -- we need to be, biologically speaking, so that we are motivated to reproduce and keep our genes expanding. However, in the case of humans, our consciousness allows us to express ourselves sexually even where reproduction may not happen.
I think that terms like "heterosexual", "homosexual" and even "pedophilia" are culturally defined. I think at other times and in other places these categories may or may not have existed -- for better or for worse. In the case of hetero versus homosexuality, I do tend to think that this distinction has a lot to do with power (Micheal Foucoult is a good read for this theory). As you see the rise of powerful countries and especially the rise of modern states, you tend to see divisions of sexuality appear like this, and criminalization of any sexual conduct that does not lead to the man-woman coupling with the intention of procreation. As a result, homosexuality, prostitution, and even masturbation become criminal and morally "wrong". Sexuality is a powerful force, and if you can control it and make people feel bad about their behavior, you can manipulate them in a variety of ways. It's not a coincidence that powerful religions as well insist on only certain sexual relationships, again with the intention of making people feel bad about their sexuality and having more control over them.
In the case of pedophilia, for most of human history people were marrying other people at ages that we would consider wrong today. Modern criminal sanctions against sex with children (statutory rape, in other words) is a result of at least two forces -- the extension of youth well into adulthood (even people in their twenties today are hardly considered adults); and some of the control aspects I spoke of earlier. But this is a sensitive subject, and I agree that the harms of forced child sexuality are very real and very long-lasting. I think people of either gender who are looking for a relationship with someone much younger than themselves may want to think about why that is -- and this is especially urgent if they are attracted to teenagers or children. In America, too, there is also a sexualization of young women at an early age with the mass media. I can't help but think that this plays a role in creating some urge amongst psychologically weak individuals to sleep with children.