According to the stackexchange scraper policy you shouldn't report sites that:

They follow all the attribution requirements, and don't outrank us on Google

I've been thinking several ways to use SE scrapers and show the information in different ways, but this policy bothers me.

You can't detect easily a website who is outranking you in Google's SERP, because there are so many variables that each search takes into account that every user can see different results. There is no a single way to detect this, may be a webpage does better than another one for a group of keywords but that doesn't means it outranks the whole site.

Unless this is based on some external index like Alexa or similar rankings.

So, how does Stack Exchange identifies another website outranking them on Google?

  • 2
    I don't think there is any strict rule here, more like "if the scraper site appears before the original SE site when searching for some sample content" - can't know for sure though, so asked the one who added it to take a look here. – Shadow9 Jun 13 '16 at 17:48

We use the search terms that you provide us when you report the scraper. Sure, there may be other search terms that don't produce the same results, but that's exactly why we ask for them. We do get reports from users who sift through to pages 2 and 3 to find scrapers, and we don't particularly care about them because they're way down on the list.

Essentially, if we can verify that a scraper is appearing above the original content for a particular search, that is something we can actually report directly to Google and get it fixed. If it's below, there's no action for us to take on it.

  • 1
    Ok interesting but what would "get it fixed" means? If the scraper complies with CC-BY-SA, then it does not violate Google content policy, what is the goal of reporting it and what is what you report to Google? – marcanuy Jun 13 '16 at 18:04
  • 8
    @marcanuy Google will modify their algorithms to favor the original content over copied content every day of the week. Reporting it will usually cause them to give the copy-cat site less weight so that it does not appear over the site where the content originated. But, this also rarely ever happens to begin with. – animuson Jun 13 '16 at 18:08
  • Just curious, how many times it did happen? :) – Shadow9 Jun 13 '16 at 20:19
  • 3
    I think what the question is alluding to is that Google uses so much data for every search, some derived from the user who's searching, that even with identical search terms you're still fairly likely to see different results. – ArtOfCode Jun 13 '16 at 22:02
  • This "outranks us on Google" thing seems kind of uncool. I mean if the site is obeying all the rules is it fair to "hate" on them just because they're successful? Just my 2 cents. – Hack-R Jul 9 '16 at 19:14
  • @Hack-R It's about being fair to the creators. The original creator of the content should never appear below someone who just copied the content in Google results. Has nothing to do with hating on them and the only thing we do is request Google weight their content less so it doesn't appear above ours. – animuson Jul 9 '16 at 19:21
  • @animuson The attribution makes it fair. Not to mention, it's a bonus to us as the creators because it's boosting our reach. All I am talking about is the idea that something is OK as long as it doesn't outrank us on Google. That just seems petty and wrong. – Hack-R Jul 9 '16 at 19:24
  • @Hack-R No, attribution does not magically make it fair. Most scrapers exist for the express purpose of using other people's content and injecting ads into them to make money. By outranking us, they are stealing visitors from the people who actually made the content and any ads they might display for their own gain. In no world is that a fair outcome. Not to mention, most are also poorly formatted and hard to read - sending them to the actual source of the content would be a much better user experience overall. – animuson Jul 9 '16 at 19:35
  • @animuson Yes, I'm not proposing that it works by magic. I'm saying that it's one of the rules we established that makes sense and don't just say "you can't be better or more popular than us, because the owners of SO would feel threatened". I'm specifically talking about when someone has obeyed every rule we've laid out and their only "crime" is that users and Google like them enough that, at no fault of their own, they've surpassed us on some page rank. That's an innovation-killing, despicable way to keep SO on top as opposed to making sure that it's the best and continually improving. – Hack-R Jul 9 '16 at 19:39
  • @Hack-R If you truly believe that, then so be it. I see no point in arguing with you. As I said, this rarely ever comes up because Google has already developed its algorithms to the point where original content does always win, and copied content rarely ever makes it above the original anyways. This is mostly an artifact of the past when Google's algorithms allowed it to happen more readily. It's for the most part been fixed on their end. So whether you think it's wrong or not really has no impact. In the year since I've been working on scrapers, I've never once had to report one of these. – animuson Jul 9 '16 at 19:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .