I have thought about it some, and I realised that with friends, family, or at work. I rather enjoy mindless chitter-chatter and small talk then. The thing that drives me to like it is it lets me learn more about people, in direct and indirect ways. It just comes down to the fact that I am very nosy. It makes sense, but I find it rather odd that most infj's have an adversion to small talk.
However, and this is important, if I am not in the right mood. I will not ingage in conversation at all.
Yeah, I've seen this happen with my INFJ friend as well. I'm wanting to have a deep conversation with her and she usually avoids it, she has to be in a particular mood for her to engage in such discussion otherwise I tend to hit myself with brick walls trying to pursuit it.
Other than that, I don't have that many good conversations, and the majority of the times I'm on the background just hearing the chit chat of other people and if something sparks interest that's when I will usually jump into the discussion.