-14

I regularly have serious problems using Stack Exchange sites such as:

  • not being able to see the site at all: "We're sorry... too many requests"
  • not being able to log in
  • not being able to post answers that I have just spent valuable time and energy writing
  • enter a value into a very difficult CAPTCHA field like twenty times before realizing that I surely got it right some times by now, and so it doesn't matter whether I answer correctly or not, the site will always reject.

My crime: I'm a Tor user. (Mind you there's even a Tor Stack Exchange site; how cynical is that?)

--- sigh...

I know programming is not easy, but the art of it is making elegant solutions, not to whack into user experience with a cleaver. Just one idea springing to my mind right now (I'm sure some Knuth-like brain can think of something a million times better):

  1. I'm a member for three years
  2. logged in
  3. not a single letter of spam has ever been posted from my account.
  4. as a human, I can only post so many things per time-unit

How could I be distinguishable from a spammer (bot or otherwise)?
What if my account gets hacked at some point in the future?

Combine 3 & 4, and making sure that a genuine user does not EVER get harassed becomes pretty trivial and low risk. (E.g. use some counters that go up on posting or on having posts flagged as spam and go down over time1, then set up some policies depending on those values.)

All that aside, even for not-logged-in users (reading the site), it must be said that blocking users based on IP address is bad practice that should stay in the early era of the internet which we've surely outlived by now.

edit for Servy, who asked what I propose:

I don't have access to specific data about the SE signal to noise ratio, but this is where I would start.

There seem to be 2 distinctive questions at hand:

  1. Preventing spam, ungenuine voting, flagging, ...
  2. Preventing abusive server load

--

Logged in users Much can be said, and it's surely an interesting research question, but for logged in user:

spam:

  1. take this rule to heart: As long as no post from this user has been flagged as spam, never, ever put anything in their way when posting. Remember this: until a post gets flagged, nobody has seen it or at least been annoyed enough to bother to flag it, so not much damage done.

  2. when a post gets flagged as spam, put every new post from this user in a waiting queue. Don't prevent the user from posting, but don't make publicly visible until approved.

  3. use git bisect to let humans with more than xxx or probably xxxx rep determine the first spam post from this account. Now hide posts from that point on. Responsiveness is an important factor here, which you will have to evaluate as you go, and this step will at least sometimes require human intervention.

  4. if more than x spam posts or step 3 hides genuine posts because they alternate with spam, require human intervention (password reset, reply to an email to justify what happened, etc.) before allowing any more posting.

  5. beware that false positives will happen and take some preventive measures (someone might post a link to software they wrote, and someone else considers that spam and flags it...). It would probably be better to lower the rep requirements for step 3, but require x users to to agree on it. If you feel that this is using your users as spam filter, it is, but I'd rather volunteer to be used as spam filter than as OCR software for reCaptcha. In the end it's just the chores that need doing.

  6. Require human interaction for making a new account, like a captcha, but it could be something else.

extras: you could probably:

  • be a bit more restrictive on new accounts. The fight basically is the human effort of creating a new account (privileged enough to be interesting for a spammer) vs the human effort to bisect the spam account's posts.

  • avoid trying to do anything smart!

Other abusive behaviour:

It would really depend on the specifics. I personally haven't got much complaints, so probably it's mostly working fine, except for poor policies like the "security" measures against tactical downvoting. Tactical downvoting seems to be a niche problem (young questions only, abuser answers themselves, ...) and locking votes everywhere for everyone is both draconian and useless to prevent it...

Preventing abusive server load

Again, it's hard to tell which problem you are trying to solve without seeing numbers, but something doesn't add up. The tor network is slow. Every request taxes 4 different machines. It's surprising that someone can wear down the Stack Exchange servers by using tor, and if I wanted to crawl, tor is definitely not what I'd use. If someone is doubling as a malicious user and a tor node, you might want to get in touch on the tor node mailing list.

A perfect solution, unless in extreme circumstances (unlikely here) is to check at least the cookie information send by the browser before rejecting the request. No need to use any IP-based system for logged in users.

On a sidenote, this seems a lot rarer than the compulsive logouts, captchas, never resolving captchas etc.

ps: In any case blocking tor won't stop any determined malicious person. A vps costs 0.005 dollar cents per hour these days. I can have as many non tor IP adresses as I want, either by renting them, or by hijacking one of the poor sods that hire one for their wordpress blog...


Asking for software not to choke, vomit at you, slap you in the face before just breaking down and keeping you from using it completely legit is not a feature request, it's a bug report.


1 Better, compare them to the current Unix timestamp; they don't need to decrement...

  • 1
    You don’t have to be a bot to spam. Anyways, I suppose Stack Exchange could whitelist Tor exit nodes. – Ry- May 19 '14 at 19:44
  • 9
    Stop using Tor with Stack Exchange. No, really. What are you worried about, anyway? A bunch of guys in black suits knocking your door down at 2 in the morning because you downloaded the latest Harry Potter movie from a Torrent? News flash: we don't care about your warez. – Robert Harvey May 19 '14 at 19:46
  • 4
    @RobertHarvey not your call to make – user148312 May 19 '14 at 19:46
  • 24
    No, but we don't have to cater to the terminally paranoid either. – Robert Harvey May 19 '14 at 19:47
  • 2
    While it may be sound to make a business decision that disenfranchises all tor users, it certainly isn't kind to react this way to one trying to use your service. – Adam Davis 'ze-zir-zem' May 19 '14 at 19:56
  • 1
    @minitech No, SE couldn't whitelist Tor exit nodes. – Servy May 19 '14 at 19:56
  • 1
    @Servy: Yes, it could. – Ry- May 19 '14 at 19:58
  • 2
    @AdamDavis SE doesn't prevent use of the site to all TOR users. It adds restrictions, rate-limits the amount of contributions that can made much more aggressively than would be typical, results in more captchas, etc., but it doesn't prohibit contributions entirely. – Servy May 19 '14 at 19:58
  • @minitech Did you read the post. Tim specifically mentioned the existence of such sites, and simply stated that the data gets stale too quickly to be useful, in addition to the fact that the verification is prohibitively expensive. – Servy May 19 '14 at 19:59
  • 2
    @AdamDavis: Well, Google does exactly that to Tor users in exactly the same way. They generally use Startpage to get around it, because it’s just a proxy for Google search, but that really wouldn’t work for Stack Exchange. – Ry- May 19 '14 at 20:16
  • 4
    Re: your edit - you seem to be missing the point. We don't treat Tor specially - that's why you're running into problems. If we could reliably distinguish your requests from those of other users, you wouldn't be having a bad time - but of course, that's exactly what Tor is designed to prevent us from doing! We don't know who you are until you've already managed to get past a whole bunch of checks - once we do, it's smooth sailing, but getting that far in the first place is where you're running into trouble. – Shog9 May 20 '14 at 2:39
  • 1
    @Shog9 I don't quite understand what you're getting at. Clearly the spam part of my proposal (the most important part) deals with posting, which obviously should be dealt with after reading the cookie information, thus distinguishing different users. The other part is not much elaborated, because I just can't figure out which kind of scenario generates such a high server load from tor exit nodes. – user148312 May 20 '14 at 5:17
  • 3
    The CAPTCHA problems are probably unrelated to Tor, unless Google also has issues with it - we rely on ReCAPTCHA for this. To answer your second question: all it takes is too many requests being pushed through a single exit point. There are dozens of logged-in users going through these, and who knows how many anonymous readers - all it takes is a few of them being a little bit too active at the same time. – Shog9 May 20 '14 at 14:58
  • 1
    As to your second remark, you prove my point. Could you please post some numbers as to how many requests/second for one ip address will cause a block. I bet you these are much to tight. These shouldn't be absolute values anyways, because say it's the wee hours of the night, and there would be only one user doing requests. Who cares if that user does 1000, 10000 or 100000 requests per second. It's all relative to your actual load. As I said before, the tor network is slow! The tor nodes would probably come to a grinding halt before SE would. – user148312 May 20 '14 at 19:44
  • 1
    @Pacerier: Even if I stipulate to Tor's "ordinary" purposes, I still don't see how SE has any responsibility for catering to their (arguably flawed) design. – Robert Harvey Jan 23 '16 at 1:57
57

My crime: I'm a Tor user.

"Doctor, it hurts when I do this!" If using Tor is making your life miserable, then switch to a different proxy!

We don't intentionally block Tor. In fact, as a long-time user with some amount of reputation on the network, you're exempt from many of the checks we would normally impose.

The problem is simply that Tor is not designed to enable certain features we require in order to provide good support for a proxy.

Lemme tell you a story: a few years back, Amazon's EC2 service was home to a number of very poorly designed scrapers. We had to block all normal requests coming from EC2 IPs in order to prevent our network from being swamped by these bots - naturally, this hurt a good many perfectly well-behaved users, but the alternative would've been poor service to all of our users. We were able to resolve this issue finally, in part because Amazon is willing to work with folks like us to prevent abuse while allowing well-behaved clients to continue working.

This sort of cooperation is not a design goal for Tor.

If you're connecting via Tor, you are at the mercy of whatever endpoint the system decides to route you through, and if that endpoint also happens to be sending an insane number of requests to our system, it will be blocked. There's no good, reliable way for us to differentiate between you and whoever else happens to be arriving from the same gateway. This isn't a failing - this is actually a design goal for Tor - so if being rate-limited in response to the actions of other people isn't your cup of tea, then then don't use Tor!

Ditto for login problems: connections through Tor can and will appear very similar to various malicious attempts to hijack a connection. This also is by design! If you don't like having to re-try a dozen times when logging in, then... Well, you know what to do!

Tor is a fine tool, but like any tool it was designed for a specific purpose: if you think using it for every day browsing is a good idea, you either have very specific needs... Or you're using it wrong.

We'll do whatever we can to balance the needs of our users with the need to prevent spam and other forms of abuse. But we are not going to subject our entire userbase to a poor experience to cater to the needs of folks who are using the wrong tool for the job. If you're determined to do that anyway, well... Get ready to live with the consequences.

12

Unfortunately tor exit nodes are a significant source of spam and other attacks for many websites, and I don't blame them for not supporting users that use public tor nodes. IP blocks and rate limiting are much more efficient (less resource intensive) at discarding spam and other attacks than per-user blocks.

If you can provide a good reason why changing this would significantly benefit many users of stack exchange, and not just you, then the policy is more likely to change.

If you can only provide supposition, or your own personal experience, then it is unlikely to change. The network has to support millions of users, setting up corner cases for a few thousand won't be practical.

9

This is almost a good suggestion! If I may be so bold, I’d like to quote what I feel is the important part of your question. In bold.

1) I'm a member for three years

This is a really good indicator that should be taken into account as far as automated spam blocking goes; the cases in which moderators have to delete accounts for spam that are three years old are rare, to put it lightly.

Apart from that, no, I don’t think using a visitor’s IP address when making decisions is in any way obsolete.

How could I be distinguishable from a spambot?

  • You don’t have to be a bot to spam
  • The whole point of Tor is to make you indistiguishable from someone else, who may spam
  • 4
    Accounts can be compromised, and if a high-rep account is compromised, it can potentially do a lot of damaged before the compromise is remedied. That's why nobody is exempt from Captchas. – Robert Harvey May 19 '14 at 19:51
  • 1
    @RobertHarvey: Sure, and I’m not recommending that per-account limits be changed. (Or if it’s entirely per-address as it is, maybe per-account limits should exist.) But for users who are logged in and have been members for a long time, perhaps address-based limits could be removed. – Ry- May 19 '14 at 19:52
  • @minitech They go away based on the user's positive contributions in the form of rep/votes, to my understanding. Having an account for a long time when that account isn't doing much (it would seem) doesn't result in the system assuming you're not a spammer. – Servy May 19 '14 at 19:53
  • @Servy: I suppose that’d be the answer, then. – Ry- May 19 '14 at 19:58
  • @Servy, current SO rep 1550, apparently some fine tuning might be appropriate. Or even better, don't try to do something "smart". Simple solutions are often best. – user148312 May 19 '14 at 20:00
  • 3
    @ufotds: 1550 is barely anything, sorry. – Ry- May 19 '14 at 20:01
  • 2
    @ufotds What simple solution are you suggesting? Don't block spam? Sorry, but that's simply not an option. The site will be overrun with spam and become unusable in a matter of days. If you can propose alternate methods of detecting spam that will have a better success rate, then by all means, propose them. So far all you've done is suggest removing filters. – Servy May 19 '14 at 20:01
  • @Servy I posted a simple proposal as an example of one possible and probably adequate if not overkill solution. – user148312 May 19 '14 at 23:59
  • 1
    @ufotds Using humans to evaluate spam simply doesn't scale. Spammers can add spam at a faster rate than humans would be able to remove it, without automated mechanisms to assist in the worst types of cases. – Servy May 20 '14 at 13:38
  • @Servy I don't understand, it's the human effort of creating a new account with enough privileges to be useful for spamming vs one human assisted git bisect effort per such account. If say the two would be roughly as much effort, then for every spammer that creates an account, you would need 1 person to do the effort to identify the posts they managed to post. On young accounts as soon as the number of spam posts is more than say 5, you can directly hide all their posts and be done with it. How much legit users willing to do some cleaning/spammers are there on SE? Who scales better? – user148312 May 20 '14 at 14:46
  • @ufotds The spammers absolutely scale better. When they can hire army's of people to just sit in front of computers filling out captchas they can easily surpass the ability of the site to manage. We don't even need to be particularly theoretical about these types of questions. Until the spam prevention was recently revamped, there were a number of spammers regularly subverting the automatic mechanisms. It was completely overwhelming. The human site filters can keep up for a little bit, but after several days of determined spamming people just get drained and stop. Spammers don't. – Servy May 20 '14 at 14:51
  • well, I doubt it, but I'm not an expert, so I'll take your word for it if you have done the math. In 3 years, I have never seen any spam on SE, sorry correction it seems that I flagged 2 posts in 3 years. I have however been at the point of throwing things at my screen in frustration with SE policies numerous times. That feels like the balance is tipped completely to one side, not very elegant. Anyways, if you are right, then as I said in my proposal, be a bit more restrictive on young account. – user148312 May 20 '14 at 15:28

You must log in to answer this question.