What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 128 Stack Exchange communities.

Looks like scrapers can rank higher than the original SE universe site in Google search so that there's this very uncomfortable "first tag in the page title" change to combat them.

Also looks like many scapers follow all the CC-wiki and attribution requirements, so they act completely legally.

Does this mean that the original choice of license was wrong for the SE universe?

share|improve this question
add comment

1 Answer 1

up vote 7 down vote accepted

As for the first issue, "first tag in page title" isn't uncomfortable, it's more relevant than you think. For example, if my question is

why can't I get the listbox to work with strings?

that has a different (and certainly more useful) set of meanings as

C# - why can't I get the listbox to work with strings?

This particular scraper had the right idea to put the most significant tag in the title, at least until Google offers some meaningful, blessed microformat for tags. (and note that if the tag already appears in the title organically, eg the user added it somewhere, we don't add the tag to the title.)

As for the second issue, ask yourself how many Wikipedia scrapers you regularly encounter.

Or, stated another way, what kind of business model is "copy the freely copyable Wikipedia content, and republish it with ads on it to harvest the clicks?"

Not a safe one, clearly.

Further reading that may be of interest to you:
http://www.google.com/support/webmasters/bin/answer.py?answer=66361

share|improve this answer
    
So do you mean that as time passes Google will likely adjust ranking mechanisms and the scrapers go away? –  sharptooth Dec 11 '10 at 12:09
    
Apparently a workable business model - wapedia has been publishing a read-only export of Wikipedia + G.ads for years (although I admit they're reformatting for mobile devices, so there's possible added value, and you're correct in the "regular encounter" part: nobody except the mobile users ever heard of them), –  Piskvor Dec 13 '10 at 14:12
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .