The thing about asexuals is that, even though they lack the gender attraction, they still are subjected to gender stereotyping and social constructs. If everyone were asexual, then those stereotypes would be less needed, and thus no one would feel the pressure towards them; that's when it would be less apparent. However, at the present moment, asexuals are still pushed towards a certain criteria, and although they may be less inclined to subject themselves to that, there's probably still a certain amount that they conform.
Although I have to say, I think sex does play a key role in determining stereotypes, but also prejudices. I think there would be less friction without that attraction.