What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

Since day one of Stack Overflow, all content posted on Stack Exchange sites by their users (i.e. you wonderful people) has been provided to the whole universe under the CC-BY-SA license. For my fellow non-lawyers, that license basically means:

  • Anyone can use any Stack Exchange posts at any time without having to ask for permission
  • Making money off of the copied content is permitted
  • You don't even have to copy stuff from here verbatim; you can just use it as a starting point and make whatever edits you want
  • There are just two rules you have to follow:
    • You have to provide attribution. Simple links to the original post and author info are just fine.
    • You have to allow other people to use your content, as long as they follow these very same rules. How meta!

(If you ever forget any of that, and want to refresh your memory, the license info is linked to in the footer of every page.)

There are, in fact, a lot of people who republish varying amounts of our content for assorted reasons. Unfortunately, there are some Stack Content Republishers Attributing Poorly and/or Excelling at Ranking (SCRAPERs, for short).

In this context, "attributing poorly" means any use that doesn't follow our attribution rules or make any other reasonable attempt at give credit. This can get pretty egregious; I've seen SCRAPERs that not only don't link back to SE originals, but also use fake author info and post dates to make it harder to find originals. By "excelling at ranking," I'm referring to copycat sites that end up higher in Google results than the original SE sites do for the same content. That's not necessarily wrong, but in some cases, it indicates inappropriate SEO hackery.

So, the question is: what can you do if you spot a SCRAPER?

share|improve this question
6  
Shouldn't this be featured? It's a pretty big change. –  Linuxios Oct 9 '13 at 20:13
4  
@Linuxios what change? It's just a call to action, trying to fight those illegal SCRAPERS. –  Shadow Wizard Oct 9 '13 at 20:14
1  
@ShaWizDowArd: It's an official new policy on something that everyone has gotten used to doing through those meta questions. –  Linuxios Oct 9 '13 at 20:19
1  
@Linuxios true, but it's not changing anything in the site itself and not affecting any users. Those who want to report will easily find this post, no need to feature it in my opinion. –  Shadow Wizard Oct 9 '13 at 20:23
    
shouldn't this be configured at server level? LIke to only allow crawlers that are either Yahoo, Google, Bing or DuckDuckGo? Scrapers are parasites imho –  Michael Oct 10 '13 at 6:49
2  
@Michael I made up the acronym SCRAPER because people have referred to these as scrapers in the past, but they don't necessarily actually scrape our pages. They might use our freely provided data dumps, for example. Using the data isn't wrong; they just need to follow the rules. –  Pops Oct 10 '13 at 16:01
3  
And that's a great acronym you invented there! –  Spontifixus Oct 16 '13 at 7:57
    
Why is SCRAPER capitalised. What is it short for? What terrible joke am I walking into unwittingly? –  badp May 24 at 0:29
1  
@badp "Stack Content Republishers Attributing Poorly and/or Excelling at Ranking" –  Pops May 27 at 13:43
    
Apparently some “hackers” have even made a sport of scraping from StackExchange sites. Note how the challenge question mentions nothing about licenses or attribution… –  Synetech Jun 7 at 14:23
    
Ha, awesome. If people want to use "scrape Stack Exchange" as a personal challenge to improve their coding skills, I'm all for it. It's only a problem when they actually start publishing the results without attribution (or severely hammering our servers, but we should be able to account for that... I think) and only annoying when they do so with the intention of making a quick buck with animated ads and pop-unders. –  Pops Jun 7 at 14:41
1  
I suggest making it more explicit that "You have to allow other people..." specifically includes "provide a link to the license" according to the Creative Commons explanation page –  Andrew Medico Jun 16 at 21:00

2 Answers 2

up vote 103 down vote accepted

tl;dr: Send all reports of SCRAPERs to us via the form on the contact us page, linked at the bottom of every page. Please include:

  • the URL of the copied post
  • the URL of the original post on SE
  • for high-rankers, the search string you used (and the name of the search engine, if not Google)
  • any other details you noticed and want to share, because you're awesome

(This is the end of the tl;dr section.)

I used to be the de facto curator of the two SCRAPER reporting posts here on Meta: one for attribution and another for ranking. The community was pretty good about using them, but like all list questions, they eventually turned into unmanageable messes that were hard to search. I've just saved all of the information posted there and deleted them.

The community team has been working on improving its internal process for SCRAPERs behind the scenes, and now that I've joined the staff, I'm going to be in charge of it. By funneling everything through our usual support mechanisms, we can organize our own efforts better, and as a side effect, we won't be giving Google-juice to the very sites that are being called out for being naughty.

share|improve this answer
22  
Wouldn't it make sense to close them as duplicates of this (and maybe just purge all the answers)? There are a lot of links all around Meta pointing to those questions that won't get properly redirected to this. –  animuson Oct 9 '13 at 20:09
    
Just be careful, doing it client side is risky. –  Shadow Wizard Oct 9 '13 at 20:18
18  
Now that it requires 10K Meta reputation to see those lists, aren't you going to be buried in a deluge of duplicate emails reporting the same SCRAPERs? –  Bill the Lizard Oct 9 '13 at 20:20
1  
@animuson Yikes, that is a lot of links. Good suggestion, I'll undelete and close them. –  Pops Oct 9 '13 at 20:21
5  
@BilltheLizard I considered that, and was hoping nobody would ask about it. I'm going to just wait and see, on the theory that not that many people are seeing these in the first place. If it starts to overwhelm the inbox, I'll reconsider. Also, I'm sorta interested in knowing which ones are getting more eyeballs. –  Pops Oct 9 '13 at 20:23
1  
Are you sure you don't want to stand up a special email address for this? Or do you expect so few reports that y'all don't mind this being mixed in with other team mail? –  Monica Cellio Oct 9 '13 at 21:35
    
@MonicaCellio It's a combination of not expecting two dozen reports a day and this being an opportunity to try out some workflow changes on our end to the team@ address. I'm open to revisiting the topic if volume becomes a problem, but I'm going to see if there is a problem, first. –  Pops Oct 10 '13 at 0:05
2  
@Pops I made a suggestion for a reporting tool back in February: meta.stackexchange.com/q/167313/61654. –  ahsteele Oct 10 '13 at 3:57
11  
Hm, why not make this a form under /contact? Bit more disciplined than email (first two bullets -> required text fields), and easier to save and query in the back end. –  Yannis Oct 10 '13 at 12:55
1  
@Pops An email address will always have a risk of getting spammed, specially if some troll registers it for free newsletters or shopping offers. A more controlled way for reporting could reduce headache. –  Amar Oct 18 '13 at 13:07
2  
@Amar thanks for the concern, but the team@ address existed as our general support address long before this new SCRAPER push. I'm happy to report that the system is working pretty well so far. –  Pops Oct 18 '13 at 16:54
8  
I'd prefer a web form, too. No matter if on the site itself or a google spreadsheets form referenced in this meta post. –  ThiefMaster Oct 24 '13 at 20:04
3  
Much easier if contact us opened a new tab - as it is I hit the link and so lost the URL I was on, –  Mark Nov 13 '13 at 23:03
2  
Shouldn't the form have an "I'm reporting a SCRAPER" reason? –  MPD Feb 15 at 20:26
    
@MPD these reports are really pretty rare. If you feel strongly, you could make a feature request, but I don't personally think it's worth the dev time. –  Pops Feb 21 at 18:30

The instruction in Pop's answer is still the first thing to do.

But as of Feb 27, 2014, Google also provides a form to directly report scraper sites. The form can be found here.

From the original announcement tweeted by Matt Cutts, head of Google's anti-spam team (at the time of this writing):

If you see a scraper URL outranking the original source of content in Google, please tell us about it: http://bit.ly/scraperspamreport

For what it's worth, I haven't seen any of the popular SO scraper results in a while (efreedom actually seems to have deleted most of the old content from their servers).

DuckDuckGo also accepts reports via their feedback page (source).

share|improve this answer
5  
If you plan to use this form, please note that Google's definition of "scraper" for the purposes of that form is not the same as the definition of "SCRAPER" used in this question, though there is some overlap. –  Pops Mar 23 at 3:50
    
@Pops True; but these forms also help the general world outside of StackOverflow fight against heavier hitters. By the way, added a DDG link (after confirming with them that they accept such reports through this link). –  Jason C Mar 24 at 5:06
2  
Yup, I wasn't suggesting that people avoid the form entirely, I just wanted to preemptively clear up any confusion. And certainly not because the first time I saw the Google form, I misunderstood it myself, or anything like that.... –  Pops Mar 24 at 5:20
    
do you think google could add a check box to filter all results showing only the stackexchange sites ones? or may be stackexchange could have a google search embedded page that does that, so instead of we going to google, we could go there. Unfortunately such scrap sites allow comments, what implies ppl may never see such replies... –  Aquarius Power Jun 18 at 22:17
    
@AquariusPower If you want to just search SE sites, use the search box here, or add the "site:" modifier to Google searches. If you want to see scraped content for a particular question, you would probably have success if you searched Google for the exact title of the question, in quotes. You probably shouldn't reply to additional comments on scraper sites though, save for possibly adding a link to the original source material on SE. In any case it's not a big deal if replies on scrapers are missed. If it's truly needed info somebody will ultimately ask on SE instead. –  Jason C Jun 18 at 22:49
1  
@AquariusPower This functionality already exists. To only Google for apples on apple.com, site:apple.com apple will do the trick. Use site:stackexchange.com to filter by Stack Exchange. –  Rainbolt Aug 5 at 14:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .