I haven't read 1984 since the first time(in 1983). I should read it again, I might understand it better.
I'm reading Po Bronson's "What Should I do with my life?" He addresses your same concerns, using various real people and their career changes. The overall theme of the book kind of points out that sometimes you need to have a career that is your passion, other times you can have a career in one thing with your true passion as a hobby. I think abandoning a true "passion" completely would probably cause many people a whole new set of "issues". I think in a few cases, it is just not "doable" to follow your passions. But if the door is open at all, I wouldn't give up on it while you can still work.