Looks like scrapers can rank higher than the original SE universe site in Google search so that there's this very uncomfortable "first tag in the page title" change to combat them.

Also looks like many scapers follow all the CC-wiki and attribution requirements, so they act completely legally.

Does this mean that the original choice of license was wrong for the SE universe?

1 Answer 1


As for the first issue, "first tag in page title" isn't uncomfortable, it's more relevant than you think. For example, if my question is

why can't I get the listbox to work with strings?

that has a different (and certainly more useful) set of meanings as

C# - why can't I get the listbox to work with strings?

This particular scraper had the right idea to put the most significant tag in the title, at least until Google offers some meaningful, blessed microformat for tags. (and note that if the tag already appears in the title organically, eg the user added it somewhere, we don't add the tag to the title.)

As for the second issue, ask yourself how many Wikipedia scrapers you regularly encounter.

Or, stated another way, what kind of business model is "copy the freely copyable Wikipedia content, and republish it with ads on it to harvest the clicks?"

Not a safe one, clearly.

Further reading that may be of interest to you:

  • So do you mean that as time passes Google will likely adjust ranking mechanisms and the scrapers go away?
    – sharptooth
    Dec 11, 2010 at 12:09
  • Apparently a workable business model - wapedia has been publishing a read-only export of Wikipedia + G.ads for years (although I admit they're reformatting for mobile devices, so there's possible added value, and you're correct in the "regular encounter" part: nobody except the mobile users ever heard of them), Dec 13, 2010 at 14:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .