Can't we trust users with a few thousand rep or some of the realy cool badges (or multiple voting users) to take care of a "Me Too" answer?
We can, and we do. Trusted users (i.e., those with 20k+ reputation) can vote to delete answers with a score of -1 or less. This does not require any diamond moderator intervention; it just requires a consensus of 3 trusted users.
Naturally, it will require that someone downvote the answer, but that's not too much to ask. For one thing, the answer needs downvoting if it's literally just "me too" and not an answer to the question. For another, you'll get the rep you lose back after the deletion takes effect, so no whining.
That said, if you're not a 20k+ user, you should still continue to flag these "me too" non-answers as "not an answer". That's the only tool you have at your disposal (in addition to downvoting), and they do meet the criteria for "not an answer". We want them deleted, and it's worth the time it takes a moderator (and/or trusted users) to process these items. As you've seen, if the flag queue load gets too high, we'll just have to run another election and bring in some new janitors.