Ok I think there's maybe a little more to your original post than I first thought, it's not just a reaction to you being an INFJ ? I hope I'm wrong but it sounds like this might be a little bit about bullying of others/you ? Sometimes this can be very subtle, sometimes subtle as a brick. I note you said acquaintances not friends or colleagues. I think if you are able to post some more details you will be able to get better support and a more rounded view than just mine alone.
INFJ can be over sensitive to conflict, but that doesn't sound to me what's happening. Sadly, we can be the target of bullies etc, as we are highly likely to stand up to aggressive people, if we see them mistreat anyone. We just do, it's instinct. If there is to be conflict then I'd think about assertiveness awareness for you and the others. It's not "all on you" but ? Personally I don't like to run from people who misbehave. I try to look objectively at the options, and what can change the situation. Honestly I think the fact that you are aware of your feelings puts you ahead of "thinking" types who may seem outwardly confident, but can sometimes be overly aggressive, and unaware of their feelings.
Conflict is part of life at times. Some people still act in a very 'pecking order' type manner, due to their insecurities. INFJ may be introverted and sensitive, but also when well adapted, are natural leaders themselves. Typically we try to do so, in a consensual and humanitarian manner. Although we maybe emotional we are very capable of using our logical thinking side, and successfully resisting those who mistreat others. Even though we are sensitive to conflict, ironically we are usually adept at resolving it for others, and dealing with it when necessary. When I believe conflict is about to occur, I look to diffuse it when I can, but when I can't ? I look to prepare for it.
To make sure it happens, where and when and how I prefer. Not to be drawn into it unplanned by others against my wishes. To have some clear goals about what I want the result to be, and to have an 'exit' strategy. Almost always it's a battle of words and we have some skill there. INFJ often have great courage and commitment but we can vulnerable, if we feel isolated or misunderstood. I hope that's me 'over reacting' and far more than you were looking for lol. I really do think the assertive path is the best way if that's what happenng. Best Wishes.