You are not alone - this is a known issue with the text object:
You'd hope at least that centering the alignment and hotspot would position the text accurately, but alas not.
The only solution for accurate positioning (as far as I'm aware) is sprite fonts but this isn't ideal in all circumstances - sprite fonts don't scale well, you can't add bold/italicised formatting without adding a whole new font set and new object, and it can be a hassle getting the kerning and line spacing correct.