I have a few thoughts on this that I'd like to share
1. I agree with Socrates in his way of think about death. We have no actual reason to fear death, because in fearing death we assume we know what death brings. All of our ideas on death are just that; ideas. It's like opening a box that's been left on your door step. It could be a bomb, it could be a million dollars, or it could be nothing. The thing is you won't know what's inside the box until you open it. Same with death. We can think and think and think about it as much as we want, but we won't actually know anything about it until we die. We all die and there's nothing we can do about it, so why waste time worrying?
2. My personal belief is that when we die, a part of us continues to exist beyond our physical body. What that is? I don't know, nor do I pretend to know. I also believe that what ever that part of us is that exists after death doesn't see things the same way we do now. We only sense things in four dimensions (the fourth being time) but for all we know is that there are many many more.
3. And going off that thought...for all we know we don't exist the way we think we do. For all we know we're a thought or a dream. Borrowing from Descartes; I think, I am. All that we know is that we exist, what we exist as, we can't be certain. But at the same time, that really shouldn't bother us. We can't affect who we are, we can't affect what happens after we die.
I guess what I'm trying to say is try not to worry about too much. There isn't a whole lot you can do about it, but in the end it's all going to work out the way it's supposed to. You're not alone in thinking that infinity is a scary thought, but don't let it bug you out too much.