Yeah, we're all guilty of occasionally using language that will hurt someone else in some way, because different things hit different people. It doesn't matter that it's unintentional, it still hurts.

My family has a rough sense of humour, and one element of that is you tease the people you love. Over the years my dad made many, many harsh comments that were water off a duck's back to me. But one of the most silly and innocent, "You look like a bloody monkey when you whistle," said to me when I was about ten, cut me to the quick. I still have an internal flinch and am incredibly self-conscious every time I decide to whistle.