After the last (not so big thanks to Shog(\o/)) blast of spam a couple users and I (with a mod) decided to run some tests. We found that it's extremely easy to post a spam question. Ridiculously easy.

One thing we noticed about these spam users is that most of them are unregistered.

enter image description here

This got us thinking, and I tried to post a test question on the site:

enter image description here

At first that didn't work. I got two messages, one saying "This does not meet our quality standards" and the other saying I needed to add an email.

So far so good.

I added some code, a quote and a link and put iamatestemail@idonotexist.idot in as my email.

This time the system let it right through, posting my "question" and asking me to add a password.

enter image description here

So my question is:

How can we make it harder to spam the site without making it too much harder for new users to ask questions?

  • 42
    I hate entering captchas – Mike B Mar 22 '13 at 20:12
  • 10
    @MikeB Thanks for preemptively disqualifying that horrid option. – user200500 Mar 22 '13 at 20:13
  • You do have a good point there @Oded, but I am still interested in ways we could do this. All other sites I've seen make it a lot harder to simply create posts. – ɥʇǝS Mar 22 '13 at 20:13
  • 7
    How long did your test post actually last? A new post goes to the top of the "Active" list, and almost any user who sees that post will flag it as spam, and it will get deleted fairly quickly. A spammer would quickly figure out that their time would be better spent spamming another site where their post would stick around a bit longer. – Rachel Mar 22 '13 at 20:20
  • @Rachel A mod quickly killed so as not to disrupt the site, but from all my experience spam almost never gets killed of on it's own. – ɥʇǝS Mar 22 '13 at 20:21
  • I don't really see anything obvious that could be detected as spammy against the site in the example. Just an (objectively) short question with (subjectively) useless content, and we already nag/block extremely short posts – Ben Brocka Mar 22 '13 at 20:22
  • 2
    It's even easier to spam with anonymous edits; just do some valid-looking edits to a question (preferably a long question that requires scrolling), and post your spam at the end. Odds are, the reviewers won't scroll to the end to catch it. – LittleBobbyTables - Au Revoir Mar 22 '13 at 20:23
  • 3
    @Seth My experience has been different. I can see deleted posts, and all spam I've come across has either been deleted shortly after it was posted, or is an extremely new post and is immediately flagged for deletion by me. You could probably write a Data.SE query though for proof of one way or another. Just search post text for something like www.youtube.com and see how many are valid and how many are spam :) – Rachel Mar 22 '13 at 20:25
  • 5
    Your experiment seems somewhat flawed. It doesn't even spam anything. Try posting a question spamming Gucci Handbags and see how long it lasts. – Martin Smith Mar 22 '13 at 20:31
  • 12
    What? Free Gucci handbags for all SO users over 50k? Sounds good to me. – Cody Gray Mar 22 '13 at 20:34
  • 9
    You are trying to solve a problem that does not exist – David Heffernan Mar 22 '13 at 20:49
  • 3
    @CodyGray you could ALREADY BE A WINNER – Ben Brocka Mar 22 '13 at 20:49
  • @Rachel Yes, SO is very different. They probably have enough users hanging around to kill the spam in seconds, but on other sites this doesn't happen. Far from it. – ɥʇǝS Mar 23 '13 at 0:28
  • @MartinSmith I wasn't trying to spam anything, I was trying to see how easy it was to post a spam question. – ɥʇǝS Mar 23 '13 at 0:42
  • 3
    @Seth - But your experiment essentially is just testing how easy it is to post a question. – Martin Smith Mar 23 '13 at 10:18

How can we make it harder to spam the site without making it too much harder for new users to ask questions?

Well... You can't, not easily. At least not for the case of getting a single spam post through: whatever hoops you ask a new user to jump through, a spammer can leap as well. That said...

Things we can do easily right now:

  • We can turn on required registration (this is enabled on SO and Programmers) - this will tend to slow spammers down a bit; of course, it's also another hurdle for new users. That said, I think it's warranted at this point on AU. Update: tried this briefly on Bicycles earlier today when they were being dumped on - the spam stopped immediately. Will continue testing. Update #2: has not proved to be particularly effective for the most common/annoying forms of spam; "live streaming" spammers are perfectly able and willing to register accounts, and often make use of this to hit multiple sites simultaneously.

  • We can block posts with links from new users, as Bill suggests. This may or may not help, but I'm willing to test it. That said, there's a good chance of collateral damage when you prevent new users from referencing, say, official documentation. On some sites this could be more of a problem than on others.

  • We can increase the rate-limits for new users (minutes between questions, questions per day, etc.) This only really matters if spammers aren't IP-hopping, which they do tend to do. Update: so far, this has proved to be a useful tool for slowing down spam.

Things we can do now, with a bit more overhead:

  • URL / phrase blacklisting: right now, this is a dev-only tool, and it's a bit dangerous. Also fairly trivial to work around, though it's been somewhat effective in slowing down PPV spam. Ineffective for human-driven "live streaming" spammers who quickly munge keywords and change URLs.

  • IP banning: huge potential for collateral damage, applies network-wide. Effective when spammers aren't IP-hopping.

  • Enabling quality bans for questions. Upside is that it's very quick to kick in for spam; downside is it doesn't really work if mods destroy spam accounts, which is normally the most expedient way to deal with them.

  • Just keep deleting the posts until they give up and move on (this has actually been reasonably effective in most cases).

Things we can't do, but are discussing:

  • Faster, safer, more flexible term/URL blacklisting.

  • Automatic IP-blocks in response to spam flags. Update: a design for this is being actively worked on right now - gonna take some work to implement, but stands the best chance of success out of any of these.

| improve this answer | |
  • 1
    Easier and safer blacklisting of URLs would be the most promising action in my opinion. Especially in combination with some automatic statistics on URLs collected from spam flagged posts. At least in the spam floods I've seen, the URLs were pretty much constant. – Mad Scientist Mar 22 '13 at 22:28
  • 2
    @MadScientist We haven't seen the blacklist helping for more than a day or two on Drupal Answers. – mpdonadio Mar 22 '13 at 22:49
  • Yes I know some spammers are registered, but turning on required registration will not only make it harder for them to spam, but it will give us more information to work with. I also think we should limit users with less than 10 reputation from posting more than 1 link as well; since most of the spam I have seen had two links (weird right?). – ɥʇǝS Mar 23 '13 at 18:29
  • I noticed that the links were plain text now: i.stack.imgur.com/pGc25.png Did you change anything? I'm just curious... – ɥʇǝS Mar 23 '13 at 21:41
  • 4
    there is no reason not to enable account registration on AU at this point. – Jeff Atwood Mar 25 '13 at 18:58
  • It seems to me that requiring registration on Ask Ubuntu would be a very bad thing. Many users who come to us at Ask Ubuntu for help have very minimal technical skills (though of course others are extremely skilled--AU is not only for novices). Creating any additional technical hurdles for new users is not justified in the name of fighting spam. Some spammers will always find a way to spam the site, but technically non-proficient users, some of whom have trouble just browsing the web, will not. CAPTCHAs for unregistered users would be preferable; they discriminate less against super-novices. – Eliah Kagan Apr 12 '13 at 21:41
  • "no reason not to enable" sounds almost like "reason to enable". I'd add two more negatives though, no account registration not on AU. Maybe, not at this point – gnat Apr 12 '13 at 21:42
  • so has the IP block been implemented now? – Ashish Ahuja Aug 18 '16 at 10:37
  • Yes they are, @ashish – Shog9 Aug 18 '16 at 16:05

How draconian would it be to bar new users from posting links in questions? (You could define "new user" either by reputation, number of posts, or a combination of both.) We already bar users below 10 reputation from posting images. I can't think of a reason you'd absolutely need a link in a question on Stack Overflow, but this might not be the case on some sites. Taking away spammers' ability to post links should cut down on those "Watch new episodes of the Walking Dead" posts.

Update: If that's too restrictive, each site could also whitelist certain sites that new users can link to, for example JSFiddle and Github on Stack Overflow. Links within the Stack Exchange network should probably be allowed everywhere too.

| improve this answer | |
  • 8
    Hi I am a comment please copy w w w . s p a m . c o m to your address bar and remove all the spaces in order to see me! Honest, I swear! – That Brazilian Guy Mar 22 '13 at 20:24
  • 4
    Maybe just automatically make links into plaintext from new users? It's already annoying when new users need to post images and they don't/they need to be edited to become embedded images, preventing them from posting links at all would be very painful on UX. – Ben Brocka Mar 22 '13 at 20:25
  • 6
    I often see links from new users. Typically it's a link to a tutorial they're following to explain where their code snip came from, or sometimes to their sample code or an image (since new users can't post images). I wouldn't like it if new users couldn't post links :) – Rachel Mar 22 '13 at 20:26
  • 3
    @Rachel They really should be posting their code, not a link to a tutorial. – Bill the Lizard Mar 22 '13 at 20:27
  • 1
    @ruda.almeida That's much less visible, and still not a clickable link. It doesn't need to be bulletproof, just take away most of the incentive. – Bill the Lizard Mar 22 '13 at 20:29
  • @BenBrocka Yes, that's probably a good idea for sites where it would be too painful to block links from new users altogether. – Bill the Lizard Mar 22 '13 at 20:30
  • @BenBrocka what about links to Pastebin or Github Gist? – That Brazilian Guy Mar 22 '13 at 20:41
  • 1
    This also solves part of the link only answers. It could be part of the new user restrictions. – Toon Krijthe Mar 22 '13 at 20:47
  • 1
    It already is part of the new user restrictions, @Toon - it's set at 2 by default. On some sites (where references are required), it's higher. It could be lowered; I'm not sure that'd help. The bulk of the spam Seth's been seeing is posted by humans who've been fairly adept at working around simple restrictions... Might be worth a try though. – Shog9 Mar 22 '13 at 21:12
  • Most spammers I've seen just include a single link anyway – Ben Brocka Mar 22 '13 at 21:14
  • Yes I meant to disallow any hyperlink into a post until the new user restrictions are removed. Or maybe even set the restriction to 2 rep. Real spam accounts stay at 1 rep. – Toon Krijthe Mar 22 '13 at 21:22
  • 8
    @Shog9: You've hit the nail on the head. As long as a human is doing this, they will always find a way around any measures put into place. Our only hope is that we can convince them that their time is better spent elsewhere. They're not going to waste time on something that isn't providing them with any value. – uɐɯsO uɐɥʇɐN Mar 22 '13 at 21:24
  • Hi @ruda.almeida! I'm interested in what you have to say, but you're asking for a bit too much effort for your spam to be effective. – user200500 Mar 22 '13 at 22:15

I think before trying to figure out a solution, we should figure out if there is a problem first

I very rarely see spam on any SE site, and when I do it's either a deleted post that was deleted shortly after being posted (I have 10k rep), or it's just been posted.

Most spammers are out to get people to visit their link, and why waste so much time posting on a site where your spam gets deleted within minutes? A spammer would quickly figure out that their time would be better spent spamming another site where their post would stick around a bit longer.

That's part of the beauty of community-run sites: many more users to help with the janitorial work.

In addition, I'm fairly sure there are limits to how many posts you can make within a timeframe. Per the rate-limiting FAQ post, users with < 125 rep have to wait 20 minutes between questions, and 3 minutes between answers. I'm not positive if this is different for unregistered users, but I can't image it would be higher than that. And I know I've seen meta posts about post limits per IP address, so something like clearing your cache/cookies won't work.

To summarize, I think are existing spam controls are sufficient, and we don't need to add any additional features that would make it harder to ask questions.

(Also, I did a quick check on Data.SE for posts containing "youtube.com" this month (a longer time frame times out), and it returned 539 results out of 280,033 posts. I glanced through the first few pages, and they all look like they're non-spam)

| improve this answer | |
  • 1
    The rate limits for asking questions only apply to an individual account. The user in question here is simply creating a new account for each post. – uɐɯsO uɐɥʇɐN Mar 22 '13 at 21:25
  • There are per-IP limits as well, @George - still doesn't work if the spammer switches IPs. – Shog9 Mar 22 '13 at 21:27
  • Oh, okay. Yeah, they could use one of the numerous free proxies out there to work around any IP-based bans. – uɐɯsO uɐɥʇɐN Mar 22 '13 at 21:27
  • 6
    "I very rarely see spam on any SE site"...pop over to Drupal Answers on any given weekend :) – Clive Mar 22 '13 at 22:10
  • 4
    To elaborate on what @Clive's saying: 82 spam posts deleted on DA since Sunday. – Shog9 Mar 22 '13 at 22:23
  • @Shog9 I was wondering what site those screenshots were from :) Perhaps requiring registration for users will help? – Rachel Mar 23 '13 at 0:40
  • 1
    If OpenIDs were hard to come by, maybe. But... They're really not. Screenshots are from AU though - they've been hit hard too. – Shog9 Mar 23 '13 at 0:45
  • @Shog9 OpenIds aren't hard to come by, but they will probably slow down the post creation process enough that it won't be worth it for spammers to keep posting – Rachel Mar 23 '13 at 20:09

One candidate idea: Create an automatic whitelist of domains/sites that are believed to be good. Prevent new users from including more than 2 links to non-whitelisted domains (or, prevent unregistered users from including any links to non-whitelisted domains).

How to build the whitelist: one candidate approach might be to collect the set of all links on StackExchange, parse out just the hostname (the domain portion), and then include all domains that appear at least twice (say). (One might base this upon only the set of posts/answers that are at last one week old, on the premise that spam is unlikely to last one week.) This could be automatically updated.

There are many variations on this possible.

This is just an idea. I don't know if it would actually be effective at deterring spam, nor whether it is needed, nor whether it would have problematic side-effects, nor whether it is worth the time needed to implement it. I just thought I'd share this, in the spirit of brainstorming ways to reduce spam without too much negative impact on the user experience for legitimate users.

| improve this answer | |
  • That fails for two reasons: First, you need to consider subdomains when it comes to spamming or scamming. Second, building a whitelist of all trustworthy websites will include half the internet or so...that's a lot of domains! Though, I thought there would be more then ~110 million domains registered, but that's still quite a feat to pull off. – Time Traveling Bobby Apr 12 '13 at 7:55
  • @SulfurizedDemonbobby, Thanks for your comments! Can you explain what you mean by "consider subdomains"? (I imagine my approach could be revised to treat subdomains in whatever way seems most appropriate, if needed.) Also, I'm not sure why you say the idea will fail because the whitelist will include a lot of domains. Could you elaborate? Computers are really, really good at handling large lists, and I described how to automatically construct the list. What failure mode do you expect to see? – D.W. Apr 12 '13 at 17:32
  • The problem with subdomains is that there are many hosting services which provide free webspace and a subdomain, so you get scammy.coolcompany.com as an address. The domain is valid, and so are most subdomains, except the scammy ones. For the second part, the first big problem is with building that list. I'm also not sure if the SE peoplewant to keep such a big whitelist around...they are already not very fond of the (rather small) blacklist. – Time Traveling Bobby Apr 12 '13 at 19:45
  • @SulfurizedDemonbobby, about building the list - I see that I did a poor job of explaining my idea for how to build the whitelist, so I've edited it just now. Take a look and see what you think. I don't see anything that would be a problem: it can all be automated -- the sort of thing that computers are good at. (And the subdomain problem can be solved if you treat a.example.com as a different site from b.example.com.) – D.W. Apr 12 '13 at 21:27
  • @SulfurizedDemonbobby The whitelist actually wouldn't be that long, for Stack Overflow you would just need jsfiddle, github and similar as this is only supposed to be a restriction until 10 rep or so. It's much better than completely limiting the links, which is what is done ATM. – ɥʇǝS Apr 12 '13 at 23:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .