I think it would be useful to add the score that the user won in each tag used in the question, and this below or near the flair printed in the user's answer. .
Something like this question tagged with 'c++' and 'math':
I could browse to the user's profile and take a look in there, but that diverts attention.
But I would like to see at a glance of the background of the user.
No, this will shift the focus from the content to the user.
The tag score of a user is not relevant for someone who is reading the post. A post should be judged solely on it's own merit. If tag scores are displayed in the user card, it will be a disadvantage for low rep users who have less tag score, as well as a disadvantage for those high rep users who start answering in newer tags, apart from their regular tags.
It will clutter up the space too. As all this information is already available in a user's profile at present, this change doesn't seem to add much value.
You must read the answer not looking at someones reps and tag score for knowing if that is a correct answer or not. But if you have any doubt about that users answer and just want to be sure if its the right answer, as you said, you could go to his profile and see how much this user have experience in that field.
I think adding this feature would just cause people ignore some good answers from users with low-score on that tag.
This would actually be a good way to add a level of trust that rep doesn't completely give, though maybe it should be restricted to one or two tags, so that it doesn't clutter things up excessively.
