It's not that uncommon for me to be interested in edits people make to questions or answers, especially when I'm doing review tasks and someone I downvoted/closevoted might have fixed the problem, but I usually have no interested in tag-only edits.
I understand and agree with the philosophy that all edits, no matter how minor, should result in a "bump" of some kind so that at least someone sees what you did. But it shouldn't be too hard to make that bump say "retagged" instead of "modified" so I can avoid wasting a few clicks on a change I don't care much about.
Plus, you can already see all of a question's current tags on the front page, so even if you are interested in retags, there's often no real need to click on "modified" to see the formal edit history.
