No, I don't worry about being mad or insane.
What worries me, is a consequence of my INTJ-ish inability to compromise my principles. Most people are two faced, instinctual, unreliable animals with the ability to speak and plan nothing but half truths and tribal games. My worry is that I am so stubbornly entrenched in my principles, that I know I will never play their games, and will always run afoul of their lower instincts, unless I remain aloof, disconnected, and unengaged.
I don't worry about insanity, I worry about loneliness.