Please don't use a spam filter based on IP address

Question

I regularly have serious problems using Stack Exchange sites such as:

• not being able to see the site at all: "We're sorry... too many requests"
• not being able to log in
• not being able to post answers that I have just spent valuable time and energy writing
• enter a value into a very difficult CAPTCHA field like twenty times before realizing that I surely got it right some times by now, and so it doesn't matter whether I answer correctly or not, the site will always reject.

My crime: I'm a Tor user. (Mind you there's even a Tor Stack Exchange site; how cynical is that?)

--- sigh...

I know programming is not easy, but the art of it is making elegant solutions, not to whack into user experience with a cleaver. Just one idea springing to my mind right now (I'm sure some Knuth-like brain can think of something a million times better):

1. I'm a member for three years
2. logged in
3. not a single letter of spam has ever been posted from my account.
4. as a human, I can only post so many things per time-unit

How could I be distinguishable from a spammer (bot or otherwise)?
What if my account gets hacked at some point in the future?

Combine 3 & 4, and making sure that a genuine user does not EVER get harassed becomes pretty trivial and low risk. (E.g. use some counters that go up on posting or on having posts flagged as spam and go down over time¹, then set up some policies depending on those values.)

All that aside, even for not-logged-in users (reading the site), it must be said that blocking users based on IP address is bad practice that should stay in the early era of the internet which we've surely outlived by now.

edit for Servy, who asked what I propose:

I don't have access to specific data about the SE signal to noise ratio, but this is where I would start.

There seem to be 2 distinctive questions at hand:

1. Preventing spam, ungenuine voting, flagging, ...
2. Preventing abusive server load

--

Logged in users Much can be said, and it's surely an interesting research question, but for logged in user:

spam:

1. take this rule to heart: As long as no post from this user has been flagged as spam, never, ever put anything in their way when posting. Remember this: until a post gets flagged, nobody has seen it or at least been annoyed enough to bother to flag it, so not much damage done.

2. when a post gets flagged as spam, put every new post from this user in a waiting queue. Don't prevent the user from posting, but don't make publicly visible until approved.

3. use git bisect to let humans with more than xxx or probably xxxx rep determine the first spam post from this account. Now hide posts from that point on. Responsiveness is an important factor here, which you will have to evaluate as you go, and this step will at least sometimes require human intervention.

4. if more than x spam posts or step 3 hides genuine posts because they alternate with spam, require human intervention (password reset, reply to an email to justify what happened, etc.) before allowing any more posting.

5. beware that false positives will happen and take some preventive measures (someone might post a link to software they wrote, and someone else considers that spam and flags it...). It would probably be better to lower the rep requirements for step 3, but require x users to to agree on it. If you feel that this is using your users as spam filter, it is, but I'd rather volunteer to be used as spam filter than as OCR software for reCaptcha. In the end it's just the chores that need doing.

6. Require human interaction for making a new account, like a captcha, but it could be something else.

extras: you could probably:

• be a bit more restrictive on new accounts. The fight basically is the human effort of creating a new account (privileged enough to be interesting for a spammer) vs the human effort to bisect the spam account's posts.

• avoid trying to do anything smart!

Other abusive behaviour:

It would really depend on the specifics. I personally haven't got much complaints, so probably it's mostly working fine, except for poor policies like the "security" measures against tactical downvoting. Tactical downvoting seems to be a niche problem (young questions only, abuser answers themselves, ...) and locking votes everywhere for everyone is both draconian and useless to prevent it...

Preventing abusive server load

Again, it's hard to tell which problem you are trying to solve without seeing numbers, but something doesn't add up. The tor network is slow. Every request taxes 4 different machines. It's surprising that someone can wear down the Stack Exchange servers by using tor, and if I wanted to crawl, tor is definitely not what I'd use. If someone is doubling as a malicious user and a tor node, you might want to get in touch on the tor node mailing list.

A perfect solution, unless in extreme circumstances (unlikely here) is to check at least the cookie information send by the browser before rejecting the request. No need to use any IP-based system for logged in users.

On a sidenote, this seems a lot rarer than the compulsive logouts, captchas, never resolving captchas etc.

ps: In any case blocking tor won't stop any determined malicious person. A vps costs 0.005 dollar cents per hour these days. I can have as many non tor IP adresses as I want, either by renting them, or by hijacking one of the poor sods that hire one for their wordpress blog...

---

Asking for software not to choke, vomit at you, slap you in the face before just breaking down and keeping you from using it completely legit is not a feature request, it's a bug report.

---

¹ Better, compare them to the current Unix timestamp; they don't need to decrement...

Answer 1

My crime: I'm a Tor user.

"Doctor, it hurts when I do this!" If using Tor is making your life miserable, then switch to a different proxy!

We don't intentionally block Tor. In fact, as a long-time user with some amount of reputation on the network, you're exempt from many of the checks we would normally impose.

The problem is simply that Tor is not designed to enable certain features we require in order to provide good support for a proxy.

Lemme tell you a story: a few years back, Amazon's EC2 service was home to a number of very poorly designed scrapers. We had to block all normal requests coming from EC2 IPs in order to prevent our network from being swamped by these bots - naturally, this hurt a good many perfectly well-behaved users, but the alternative would've been poor service to all of our users. We were able to resolve this issue finally, in part because Amazon is willing to work with folks like us to prevent abuse while allowing well-behaved clients to continue working.

This sort of cooperation is not a design goal for Tor.

If you're connecting via Tor, you are at the mercy of whatever endpoint the system decides to route you through, and if that endpoint also happens to be sending an insane number of requests to our system, it will be blocked. There's no good, reliable way for us to differentiate between you and whoever else happens to be arriving from the same gateway. This isn't a failing - this is actually a design goal for Tor - so if being rate-limited in response to the actions of other people isn't your cup of tea, then then don't use Tor!

Ditto for login problems: connections through Tor can and will appear very similar to various malicious attempts to hijack a connection. This also is by design! If you don't like having to re-try a dozen times when logging in, then... Well, you know what to do!

Tor is a fine tool, but like any tool it was designed for a specific purpose: if you think using it for every day browsing is a good idea, you either have very specific needs... Or you're using it wrong.

We'll do whatever we can to balance the needs of our users with the need to prevent spam and other forms of abuse. But we are not going to subject our entire userbase to a poor experience to cater to the needs of folks who are using the wrong tool for the job. If you're determined to do that anyway, well... Get ready to live with the consequences.

Answer 2

Unfortunately tor exit nodes are a significant source of spam and other attacks for many websites, and I don't blame them for not supporting users that use public tor nodes. IP blocks and rate limiting are much more efficient (less resource intensive) at discarding spam and other attacks than per-user blocks.

If you can provide a good reason why changing this would significantly benefit many users of stack exchange, and not just you, then the policy is more likely to change.

If you can only provide supposition, or your own personal experience, then it is unlikely to change. The network has to support millions of users, setting up corner cases for a few thousand won't be practical.

Answer 3

This is almost a good suggestion! If I may be so bold, I’d like to quote what I feel is the important part of your question. In bold.

1) I'm a member for three years

This is a really good indicator that should be taken into account as far as automated spam blocking goes; the cases in which moderators have to delete accounts for spam that are three years old are rare, to put it lightly.

Apart from that, no, I don’t think using a visitor’s IP address when making decisions is in any way obsolete.

How could I be distinguishable from a spambot?

• You don’t have to be a bot to spam
• The whole point of Tor is to make you indistiguishable from someone else, who may spam