Is it wrong?

Is it wrong to call someone something that may be slightly hurtful, even if the title is apt? For example, if someone is exceedingly bad at all aspects of their job except for one which they excel at more than anyone else, is it wrong to call them an idiot savant? I don't think so, but then again, I have no soul.