This was kind of going through my head, too. I'm having a hard time understanding how you can view a term of endearment as offensive. Unless it's clear that the person is trying to patronize me or be passive-aggressive, I think it's sweet. I'd never behave likewise with the person, but I think it's nice nonetheless.
And not to drag this off-topic, but I also agree with [MENTION=2727]Skathac[/MENTION] about being called "sir" but for different reasons. I don't like being called sir by anyone older than me or by people my own age, it feels awkward and unnatural. I use sir and ma'am as signs of respect, but I often feel that other people use them not so much to show respect but more as a force of habit. It cheapens what I think should be used as a sincere honorific.