Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [screen-scraping]

The tag has no usage guidance.

1
vote
1answer
77 views

Does Stack Overflow scrape other question sites, forums or mailing lists?

I have seen some old posts on Stack Overflow that are repeated in forums and mailing lists. Does Stack Overflow scrape other question sites, forums or mailing lists?
7
votes
1answer
145 views

What's the deal with Altmetric? How reliable is its scraping, and how often does it happen?

Altmetric is a service that attempts to track the online impact of scholarly articles, which it does by keeping score of mentions on Wikipedia, Facebook, Twitter, blogs, and the like and then using ...
2
votes
0answers
20 views

Is there anything specific I should do when encountering an SE scraper? [duplicate]

I've seen a lot of scraped and repackaged content in my time, but I just ran across http://readquestion.com which seems to have scraped the entire network of sites and re-posted the entire thing ...
6
votes
1answer
103 views

How does Stack Exchange identifies another website outranking them on Google?

According to the stackexchange scraper policy you shouldn't report sites that: They follow all the attribution requirements, and don't outrank us on Google I've been thinking several ways to use ...
3
votes
2answers
161 views

How, and why, did the code in my question end up in Pastebin?

I recently posted a question in Mathematica SE, Can a package append its context to $DistributedContexts?. Out of idle curiosity I then googled for some of the code in the question, and I was very ...
4
votes
0answers
68 views

StackExchange content scraping with obfuscation

The topic of content scraping from StackExchange and StackOverflow is very well known. I thought the admins would like to know that the scrapers have boosted their efforts and are now obfuscating ...
8
votes
0answers
48 views

What is “Trello Answers” and why is it scraping the SE sites? [duplicate]

Recently, I answered a question on Stack Overflow that is very specific to a certain platform. I tried Googling the same question to see who else was having this issue, and if the answer was available ...
5
votes
1answer
190 views

Are page requests rate-limited (throttled)?

I'm building an application that detects plagiarized answers on Stack Overflow, so I need to retrieve the content of answers programatically. I know I can do this using the Stack Exchange API, but ...
5
votes
0answers
396 views

Scraping the chatroom

I've been webscraping the chat room with as goal to retrieve some statistics. (have a look at this meta on security.SE (What do you guys want to know about what is said in the DMZ?) After indexing ...
2
votes
0answers
36 views

Hidden scraper bet6e.com [duplicate]

I have a Google alert on my name and this turned up the link http://meta.bet6e.com/users/512728/jan-doggen: If you go there you get redirected to a supposed Wordpress blogging site about ...
4
votes
1answer
62 views

Is scraping *profiles* (with attribution) allowed?

If a site scrapes SE content with attribution, and they also scrape SE profiles, is that OK? I don't see anything about that in A site (or scraper) is copying content from Stack Exchange. What do I ...
-3
votes
1answer
116 views

What is the tangible benefit to SE from legitimate “scraper” sites?

We know that some sites are "illegal" scrapers that SE does not tolerate for obvious reasons. But why does SE tolerate sites that re-post its content (scraped or from Data Dump) with correct ...
26
votes
1answer
255 views

Google now has a makeshift form to report high-ranking scrapers [closed]

Source: https://twitter.com/mattcutts/status/439122708157435904 Matt Cutts (@mattcutts) 11:39 AM 27 Feb 2014 If you see a scraper URL outranking the original source of content in Google, ...
365
votes
1answer
14k views

A site (or scraper) is copying content from Stack Exchange. What should I do?

Since day one of Stack Overflow, all content posted on Stack Exchange sites by their users (i.e. you wonderful people) has been provided to the whole universe under the CC BY-SA license. For my fellow ...
1
vote
0answers
78 views

Hows Info. StackOverflow Rip [duplicate]

Complete rip with no attribution: http://www.hows.info/ related: Report sites that use Stack Exchange content without following attribution rules here Here is "me": http://www.hows.info/users/...
2
votes
0answers
209 views

Is there an open-source web scraper for Stack Exchange sites?

I wish to gather some data from a licensed corporate intranet Stack Exchange, however it doesn't seem to expose the API as with public Stack Exchange sites. Are there any existing open-source ...
21
votes
1answer
162 views

Create a software mechanism to report sites that use SE content without following attribution rules

At the time of this post there are 87 answers to the meta question: Report sites that use SE content without following attribution rules here. The sheer volume of answers makes it impossible to ...
6
votes
0answers
76 views

Sort of copy of my question on a different site [duplicate]

Possible Duplicate: Report sites that use SE content without following attribution rules here I am not sure if this belongs here: Report sites that use Stack Exchange content without following ...
37
votes
0answers
419 views

Copycat site causes disturbing images to turn up in image search

Today I tried out various images search sites and one of my test search terms was my own name. What struck me was that the false positives were mostly the same regardless of the search site used. Also ...
2
votes
1answer
469 views

Scrapers enter the scene - was the choice of license right?

Looks like scrapers can rank higher than the original SE universe site in Google search so that there's this very uncomfortable "first tag in the page title" change to combat them. Also looks like ...
4
votes
0answers
218 views

Why these scrapers have higher ranks than the original stackoverflow pages in Google search results? [duplicate]

Possible Duplicate: Did anyone notice that tech.efreedom.com seem to be scraping/republishing SO's posts? Sites like answerspice.com and efreedom.com are not only stealing our thunder but ...
10
votes
0answers
341 views

What to do when you find copied SF, SU or SO content on other websites? [duplicate]

Recently, while answering a Server Fault question, I stumbled upon a copied Server Fault question and answer at what looks like a content aggregation or text scraping site. Here's the original ...
4
votes
1answer
190 views

Showing off some related [M]SOF coding

After some previous questions about polite scraping('s), etc. I wrapped up work on a three site, 50 newest questions display grid. Would it be all right to ask for some peer review of it on Meta Stack ...
5
votes
1answer
826 views

How often can I poll the Stackoverflow RSS before I get IP banned?

If you poll the main page RSS less frequently than every 2 minutes, you'll miss activity during the busy parts of the day: https://stackoverflow.com/feeds I haven't found the "newest questions" RSS (...
4
votes
2answers
106 views

Streaming updates to Stack Overflow questions

I would like to find a better way to be notified when someone replies to a comment or question I post on Stack Overflow. Right now, I have to go through a few pages to figure out where I wrote, and if ...
3
votes
3answers
537 views

Stack Overflow API: Finding the UserID of a user with their Display Name

Hi! I'm building a .NET library for the Stack Overflow API, and I would like to include a function where an input of a display name (username) is converted to the userid. This can be done by using the ...
1
vote
3answers
84 views

Are there any web sites that store deleted questions? [duplicate]

Are there any websites (not associated with the owners, of course!) that store and/or keep track of deleted questions and/or answers, like Deletionpedia does to Wikipedia, and YouTomb does to YouTube?
4
votes
2answers
164 views

Get SO data as JSON/XML?

I've poked around MSO a bit and seen that there's a data dump and some discussion of screen-scraping, so perhaps I sadly already have the answer to my question, but is there/might there eventually be ...
3
votes
0answers
334 views

Stack Overflow screen scraping code of conduct [duplicate]

Possible Duplicate: Etiquette of Screen-scraping Stack Overflow? Based on questions such as Once your IP is banned due to screen scraping SO how long will it remain banned for? I thought it would ...
56
votes
3answers
10k views

Etiquette of Screen-scraping Stack Overflow?

Is there any reasonable guidelines for how much/often I can screen scrape from SO? I've been looking around heavily in related topics and have seen Jeff's enthusiastic response to the user information ...
93
votes
23answers
3k views

What would you want to see in a Stack Overflow API?

Note: I'm aware of this question, but it's slightly different, and in my view is talking about the Stack Overflow folks coming up with an API themselves. I'm talking about a community-developed one, ...
19
votes
3answers
2k views

StackOverflow User Statistics [closed]

A few days ago I had a post that provided data scraped from the profile pages of over 23,000 StackOverflow users with the intention of having the community mine the database and produce interesting ...