Questions tagged [screen-scraping]
screen scraping is the act of "scraping" or copying content from a Stack Exchange site and republishing that content on a different site. Stack Exchange content is licensed under Creative Commons BY-SA 4.0 and can be freely distributed with the attribution requirements. This tag should be used with posts concerning sites using Stack Exchange content without proper attribution.
36 questions
449
votes
1
answer
25k
views
A site (or scraper) is copying content from Stack Exchange. What should I do?
Since day one of Stack Overflow, all content posted on Stack Exchange sites by their users (i.e. you wonderful people) has been provided to the whole universe under the CC BY-SA license. For my fellow ...
164
votes
127
answers
12k
views
Report sites that use Stack Exchange content without following attribution rules here [duplicate]
NB The following is a community-generated list of websites that republish Stack Exchange content without attributing it properly. It is no longer being maintained, because the procedure for reporting ...
98
votes
23
answers
4k
views
What would you want to see in a Stack Overflow API?
Note: I'm aware of this question, but it's slightly different, and in my view is talking about the Stack Overflow folks coming up with an API themselves. I'm talking about a community-developed one, ...
75
votes
21
answers
6k
views
Report high-Google-ranking Stack Exchange content copiers here [duplicate]
NB The following is a community-generated list of web pages that display Stack Exchange content and rank higher in search results than the original SE posts they copy. It is no longer being maintained,...
63
votes
3
answers
12k
views
Etiquette of screen-scraping Stack Overflow
Is there any reasonable guidelines for how much/often I can screen-scrape from SO?
I've been looking around heavily in related topics and have seen Jeff's enthusiastic response to the user information ...
- 14.7k
38
votes
0
answers
454
views
Copycat site causes disturbing images to turn up in image search
Today I tried out various images search sites and one of my test search terms was my own name.
What struck me was that the false positives were mostly the same regardless of the search site used. Also ...
- 2,871
26
votes
1
answer
261
views
Google now has a makeshift form to report high-ranking scrapers [closed]
Source: https://twitter.com/mattcutts/status/439122708157435904
Matt Cutts (@mattcutts) 11:39 AM 27 Feb 2014
If you see a scraper URL outranking the original source of content in Google, ...
- 561
20
votes
3
answers
2k
views
StackOverflow User Statistics [closed]
A few days ago I had a post that provided data scraped from the profile pages of over 23,000 StackOverflow users with the intention of having the community mine the database and produce interesting ...
- 37.9k
20
votes
1
answer
163
views
Create a software mechanism to report sites that use SE content without following attribution rules
At the time of this post there are 87 answers to the meta question: Report sites that use SE content without following attribution rules here. The sheer volume of answers makes it impossible to ...
- 8,851
14
votes
0
answers
186
views
"so long as they follow the Creative Commons attribution requirements" - but they don't
I reported two scrapers which were violating the license* recently, as I'd done some time last year as well. I got this email in return:
Hello,
All content on Stack Exchange is licensed under either ...
- 8,314
11
votes
1
answer
779
views
Send a follow-up email when a user reports that some Stack Exchange content is plagiarised by another website
I reported a website (Quora) copying a significant amount of Stack Exchange content via https://meta.stackexchange.com/contact and got the following response from the Stack Overflow support team via ...
- 48.2k
11
votes
1
answer
654
views
Are page requests rate-limited (throttled)?
I'm building an application that detects plagiarized answers on Stack Overflow, so I need to retrieve the content of answers programatically.
I know I can do this using the Stack Exchange API, but ...
- 817
11
votes
0
answers
350
views
What to do when you find copied SF, SU or SO content on other websites? [duplicate]
Recently, while answering a Server Fault question, I stumbled upon a copied Server Fault question and answer at what looks like a content aggregation or text scraping site.
Here's the original ...
- 1,099
9
votes
1
answer
209
views
What's the deal with Altmetric? How reliable is its scraping, and how often does it happen?
Altmetric is a service that attempts to track the online impact of scholarly articles, which it does by keeping score of mentions on Wikipedia, Facebook, Twitter, blogs, and the like and then using ...
- 19.6k
8
votes
1
answer
1k
views
How often can I poll the Stack Overflow RSS before I get IP address banned?
If you poll the main page RSS less frequently than every 2 minutes, you'll miss activity during the busy parts of the day:
https://stackoverflow.com/feeds
I haven't found the "newest questions&...
- 76.3k
7
votes
1
answer
139
views
How does Stack Exchange identifies another website outranking them on Google?
According to the stackexchange scraper policy you shouldn't report sites that:
They follow all the attribution requirements, and don't outrank us on Google
I've been thinking several ways to use ...
- 241
7
votes
0
answers
55
views
What is "Trello Answers" and why is it scraping the SE sites? [duplicate]
Recently, I answered a question on Stack Overflow that is very specific to a certain platform. I tried Googling the same question to see who else was having this issue, and if the answer was available ...
- 489
6
votes
0
answers
97
views
StackExchange content scraping with obfuscation
The topic of content scraping from StackExchange and StackOverflow is very well known. I thought the admins would like to know that the scrapers have boosted their efforts and are now obfuscating ...
- 1,489
6
votes
0
answers
79
views
Sort of copy of my question on a different site [duplicate]
Possible Duplicate:
Report sites that use SE content without following attribution rules here
I am not sure if this belongs here: Report sites that use Stack Exchange content without following ...
- 21.7k
5
votes
0
answers
713
views
Scraping the chatroom
I've been webscraping the chat room with as goal to retrieve some statistics. (have a look at this meta on security.SE (What do you guys want to know about what is said in the DMZ?)
After indexing ...
- 591
4
votes
2
answers
297
views
How, and why, did the code in my question end up in Pastebin?
I recently posted a question in Mathematica SE, Can a package append its context to $DistributedContexts?. Out of idle curiosity I then googled for some of the code in the question, and I was very ...
- 19.6k
4
votes
1
answer
78
views
Is scraping *profiles* (with attribution) allowed?
If a site scrapes SE content with attribution, and they also scrape SE profiles, is that OK?
I don't see anything about that in A site (or scraper) is copying content from Stack Exchange. What do I ...
- 2,336
4
votes
1
answer
195
views
Showing off some related [M]SOF coding
After some previous questions about polite scraping('s), etc. I wrapped up work on a three site, 50 newest questions display grid. Would it be all right to ask for some peer review of it on Meta Stack ...
- 4,044
4
votes
2
answers
358
views
Streaming updates to Stack Overflow questions
I would like to find a better way to be notified when someone replies to a comment or question I post on Stack Overflow. Right now, I have to go through a few pages to figure out where I wrote, and if ...
- 289
4
votes
0
answers
221
views
Why these scrapers have higher ranks than the original stackoverflow pages in Google search results? [duplicate]
Possible Duplicate:
Did anyone notice that tech.efreedom.com seem to be scraping/republishing SO's posts?
Sites like answerspice.com and efreedom.com are not only stealing our thunder but ...
- 411
3
votes
3
answers
697
views
Stack Overflow API: Finding the UserID of a user with their Display Name
Hi! I'm building a .NET library for the Stack Overflow API, and I would like to include a function where an input of a display name (username) is converted to the userid. This can be done by using the ...
- 5,644
3
votes
2
answers
225
views
Get SO data as JSON/XML?
I've poked around MSO a bit and seen that there's a data dump and some discussion of screen-scraping, so perhaps I sadly already have the answer to my question, but is there/might there eventually be ...
- 131
2
votes
1
answer
511
views
Scrapers enter the scene - was the choice of license right?
Looks like scrapers can rank higher than the original SE universe site in Google search so that there's this very uncomfortable "first tag in the page title" change to combat them.
Also looks like ...
- 5,562
2
votes
0
answers
38
views
Hidden scraper bet6e.com [duplicate]
I have a Google alert on my name and this turned up the link http://meta.bet6e.com/users/512728/jan-doggen:
If you go there you get redirected to a supposed Wordpress blogging site about ...
- 2,336
2
votes
0
answers
220
views
Is there an open-source web scraper for Stack Exchange sites?
I wish to gather some data from a licensed corporate intranet Stack Exchange, however it doesn't seem to expose the API as with public Stack Exchange sites. Are there any existing open-source ...
- 129
1
vote
1
answer
92
views
Does Stack Overflow scrape other question sites, forums or mailing lists?
I have seen some old posts on Stack Overflow that are repeated in forums and mailing lists. Does Stack Overflow scrape other question sites, forums or mailing lists?
- 111
1
vote
3
answers
96
views
Are there any web sites that store deleted questions? [duplicate]
Are there any websites (not associated with the owners, of course!) that store and/or keep track of deleted questions and/or answers, like Deletionpedia does to Wikipedia, and YouTomb does to YouTube?
- 21.5k
1
vote
0
answers
24
views
Is there anything specific I should do when encountering an SE scraper? [duplicate]
I've seen a lot of scraped and repackaged content in my time, but I just ran across http://readquestion.com which seems to have scraped the entire network of sites and re-posted the entire thing ...
- 23.6k
1
vote
0
answers
85
views
Hows Info. StackOverflow Rip [duplicate]
Complete rip with no attribution: http://www.hows.info/
related: Report sites that use Stack Exchange content without following attribution rules here
Here is "me": http://www.hows.info/users/...
- 21.7k
-3
votes
1
answer
124
views
What is the tangible benefit to SE from legitimate "scraper" sites?
We know that some sites are "illegal" scrapers that SE does not tolerate for obvious reasons.
But why does SE tolerate sites that re-post its content (scraped or from Data Dump) with correct ...
- 10.6k
-8
votes
1
answer
169
views
Is there any software which can scrape review queues?
Is there any software one can use to scrape Stack Exchange review queues of choice and e.g. print a summary to standard output?
A possible input document:
- stackoverflow.com
- triage
- first ...
- 79