As Shog said, you're pretty spot on with your observations.
In fact, we had not fully realized just how much of a problem anon spam suggested edits actually were until we got the new system in place.
The good news is it's working, and it's working very well. Behold, 1800 things that didn't annoy people as inbox notifications over the last seven days alone (caution: fragile CSV - sorry!). We are now blocking more spam than we're flagging or rejecting on average, and that is very good news.
... and now continuing on into December:
... and finally, most recently, through July 2014:
... where we see signal collected from users and moderators over the last few weeks now keeping a massive onslaught of it out of the queue, almost entirely (well, a little more than 85% of it). There are very few false positives, less than 1%, but that actual number is harder to crunch as it's based on manual review. As you can see from the latest graph, the tweaking and fidgeting we've been doing has really paid off, in fact you can see a lull where it's obvious that someone shut off a bot net. Yet, the work you all needed to do remained about the same.
It's still early as far as testing and data goes, but the single most profound and immediate impact that the system had was drastically reducing the amount of spam that suggested edit reviewers see. You can see how the system ramped-up to an onslaught around the 17th, this has repeated more than several times. We're still adjusting things to tighten it down even more.
The thing is, almost half of the suggested edit spam that we see comes from several distinct networks, which we've traced back to an extremely stupid bot that tests acceptable link format (does bbcode, html, etc work? that's what it's investigating) and does basic link-plant attempts, often with broken html.
What we're doing now is keeping a history of these odd posts that spam networks like to target, and if a new spammer is caught targeting one of those, we dish out a much heavier penalty in the system that keeps only anonymous edits from that neighborhood out.
I'd like to study this more before we start examining the edit summary to a greater extent - just to be sure that this change wouldn't be effective against just this family of very broken bots before we do it. We're still considering inserting regex based checks that we can tweak on a per-site basis in strategic places that directly influence the behavior of the spam protection layer, but we're not yet certain if it would really help. Once a human realizes the bot broke, it's trivially simple to just change strings, making these very distinct 'breeds' of bots harder to study and track.
Until then ...
Remember, every time you reject this crap - you're helping every site in the network block subsequent attempts - that might make you feel a little better about doing it. The system is specifically designed to pick up on signals that the communities and moderators send and learn from it.
Putting status-review on this because it is interesting and now I'm curious to look at it more in-depth, so please hang tight :)
Rejecting these with a message like "Please describe the scope of your changes to this post, we don't require your email address for submission" would probably confuse bots but not legitimate editors that are conditioned to somehow 'sign' their contributions - but I need to look a bit deeper.
