A little late to the part, but thought to write an answer nonetheless.
... edit the post to simply remove the tag(s) that do not have relevance to the question.
... adding precision to the tag selection is good, as it helps to attract attention by those who have expertise in what is actually asked. But it also gives my answer more visibility.
Looking at these two quotes from your question I find that they contradict. Removing tags from a question will not make it more visible on the site. Perhaps even the opposite is true.
If you consider someone that has a few watched tags (because they are an expert at a certain subject), and they do not browse the site any further, removing a tag from a question would effectively reduce visibility. Not that that is a bad thing, as you want a question to be visible to those that are capable of answering it. So in that sense I don't think there is any conflict of interest in an answerer improving the tagging of a question. The only thing you need to be aware of is the way you interpret a question could be different then it was intended. If that is the case your edit might not be helpful. But as you have already written an answer, the chances are high that the OP will give you some feedback about the usefulness of the answer, and thus about your interpretation of the question.
As to whether or not removing certain tags makes the question any better, should be judged on a case by case basis. But in general tags that are only tangentially related to the question can be left out, as they only add noise to the question. The easiest case would be language/framework tags, as the question involves them, or it doesn't. Then there are the more general (programming)concepts, which are a bit more of an grey area. Luckily the SE model is perfectly equipped to evaluate your edits, as your edits put a question back on the front page, so others can see what you have changed, and make a different change when they feel that fits better.
So in general I wouldn't worry about involvement in a question, or an answer to a question, when deciding whether or not to edit. If you feel it makes the question better, go ahead and apply changes. Be sure to check for any feedback you get, which might be your edit getting reversed, or edited again. In that way you can align yourself with the rest of the community if you find your judgement deviating from that of others.
If there is a case that you don't agree with the feedback you get on the edit, open a question at the relevant meta site, discussing what the appropriate action for the given question should be.