While I can live even with the current state of things, I still think it may be something worth considering.
There is a problem though: the search can contain both "positive" and "negative" tag filters at the same time (example). This result is something like:
Notice how the terms filters part actually include a "and not containing" section. This means that the tags should ideally work the same way:
results found containing
[search terms]
and not containing
[negative search terms]
tagged with
[search tags]
and not tagged with
[negative search tags]
Basically it is the same ending result proposed by Shadow, but one should remember that positive tags search may be present too, resulting in the need to have two separate sections.
That said, I would leave to the developers the final decision to determine if such change is worth spending time on. I can't say if managing the split is going to be simple with the current implementation.