Same happens to me, but I guess I fear it, it unbalances me, I often feel regret or sorrow. Doesn't happen too often though, and sometimes I think it happens to someone as well when I look at them.
In ancient times a direct look was meant to be a thread, maybe we still react like this...