Charming to me is someone who has an endearing demeanour upon first meeting them, it's rather surface, that's not to suggest someone who is charming cannot have substance it's just that part of their personality is surface and about appearance. Too much charm is pure sleaze but the right amount is found in those you are drawn to because of their social grace and ability to put people at ease.
Get to know the person and I would use a different more precise word to describe that part of them, what that is depends on if I still consider them endearing or if they've moved into the slightly sleazy category, and that part depends on if I consider their charm surface, fake and about manipulating people or if it is representative of something deeper yet pure.