I promise that I will try to answer this for you after I finish my final exams.
To be honest, I think it's because breasts have the tendency to change so often. Periods, pregnancy, weight gain/loss...all of it has to do with the fat in the breasts. Think about it: Cells have to change often in order to have that much flexibility. So it make sense to me, that the more the cells have to radically change, the more chances there are for abnormal cell growth.
That's my story, anyway.