10

I have posted questions about how to deal with scraping sites here: Mass infringement of Stack Exchange copyright.

At the time I was helpfully redirected to this question here:

The answers there give a process for how to report scraping sites. I've used this. It's slow and works after several months. This question here isn't a question about how to currently report sites that steal content without attribution. It's about how to deal with the deluge of scraping that's happening. It's also an enquiry to find out what processes SE has to protect SE users' copyright outside of the site-reporting mechanism.

Recently, I've seen more and more sites scraping content off SE EL&U and SE ELL. There are now so many that I can't keep up with them. I think we need a public ongoing page accessible on Meta where people can list websites that are nicking content without the full proper attribution, and where TPTB (the powers that be) can get back to us once the situation has been resolved.

I'm sure this problem is affecting other SE sites too.

I am now more interested in what SE is doing to ensure that it isn't neglecting its users. I'm a bit surprised that users seem to have to find this stuff and report it. I kind of feel that it's SE's responsibility to check on a regular basis who is scraping its members' (and its own) material. Shouldn't this be a routine practice on their part? If not, the agreement that they put forward to users is kind of useless. It amounts to - anyone can scrape your content without attributing it to you under the terms set out here on SE - so long as you don't report them to us. This seems unacceptable to me.

Some scraping sites seem to be worse than others. Some link to SE, but not to the page they scraped the content from. Some list the username from here on SE but don't link to the page or mention SE or the specific SE site. There is definitely a range of violations going on. But it seems to me that SE has a duty of care towards its users - at least a moral one - to try to ensure that the terms set out on the site actually apply in a meaningful way.

What is the policy for checking up on who's nicking what from users here and from SE in general?

I have been advised that the answer will probably be that users need to take their own measures to protect their copyright. Is this correct?

  • I reopened this but please remove the list of scrapers, it has nothing to do with the rest of your question and there is absolutely no need to start a second one – Pëkka Jan 6 '16 at 21:26
  • @Pëkka Sorry, I don't follow. No need to open a second what? – Araucaria Jan 6 '16 at 21:27
  • Actually, on second and third read, I'm tempted to vote to close again. This should be cut down to one specific question that isn't addressed by the existing list of scrapers – Pëkka Jan 6 '16 at 21:29
  • A second list of scrapers – Pëkka Jan 6 '16 at 21:29
  • @pekka where is the current list of scrapers where people who wrote things can see where their stuff is being published? – Araucaria Jan 6 '16 at 21:30
  • 1
    So you know how and when to report scrapers, what you can do yourself, and what SE is going to do when the content is reported (because that's all covered in the duplicate) and you don't know...what? You know how to have SE help you enforce your copywrite, you know how to resolve the issue on your own if you prefer, what's left to know? This sounds like you're just not satisfied with the state of copywrite enforcement on the internet in international law, and I mean, who is, but we can't really fix it through a meta.se post. – Servy Jan 6 '16 at 21:32
  • @Pëkka The list is necessary because people have so much faith in S.E. that they don't believe this is happening. They need to see what kind of attribution is or isn't being given. Are you trying to close down free discussion here? – Araucaria Jan 6 '16 at 21:32
  • @Servy I don't think that's reasonable. S.E. publish their agreement with users. Part of that agreement is that users' work is freely publishable by third parties, including for profit publication, so long as proper attribution is given to the original authors. S.E. also hold the copyright: "You agree that all Subscriber Content that You contribute to the Network is perpetually and irrevocably licensed to Stack Exchange under the Creative Commons Attribution Share Alike license". – Araucaria Jan 6 '16 at 21:38
  • @Araucaria I fail to see how that's relevant to any of the points that I've made, nor does it in any way clarify how your question isn't a duplicate. – Servy Jan 6 '16 at 21:40
  • @Servy ... However, currently users' copyright is being violated so frequently that this contract is perhaps of dubious value, because SE's publishing of this material where it can be freely copied without attribution may be causing a deluge of illegal scraping. Now, I'm prepared to put up with this if S.E. is doing something about it in terms of monitoring scraping. But not otherwise. People should be able to SEE exactly what's happening to their work. That is only fair. If I'm not able to list those sites here, I may have to list a thousand-long one somehwere else to promote fee speech. – Araucaria Jan 6 '16 at 21:43
  • @Servy If you're prepared to speak on behalf of S.E. on this issue and have their blessing to do so then I'll take your word for it. Otherwise, please let this post and the subsequent discussion run its course. – Araucaria Jan 6 '16 at 21:45
  • 1
    @Araucaria So you're claiming that SE is under obligation to protect your copyright for you? Because they're not. They often will, because it's also in their own best interest, but the fact that you've published the content to SE doesn't mean that SE is legally responsible for preventing and pursuing copyright violators. – Servy Jan 6 '16 at 21:47
  • 1
    For the record, there used to be a "list of scrappers", but it was abandoned in favor of the current model because it was just too much of a mess with duplicate entries, and was probably difficult to track for the SE team. – psubsee2003 Jan 6 '16 at 22:21
  • 2
    @Pëkka still exists, but is locked. – psubsee2003 Jan 6 '16 at 22:23
  • 3
    @Araucaria commented "S.E. also hold the copyright" but that is not true. SE holds a perpetual license, but copyright (including the ability to dual license) remains with the author. – Ben Voigt Jan 6 '16 at 22:48
26

Glad you asked. I've actually been personally working on a new tracking system that we're going to be using to keep track of all the reports we have coming in. Previously, we used a big ugly spreadsheet which was stored on Google Sheets and anyone who has ever handled scraper reports would say is a nightmare to even look at. My aim was to fix that. We only just recently started inserting records into this new system (only reports from about the past 3 months are recorded in it), and it's still not completely finished, but it's getting there. It looks a little something like this:

Scraper Reports

To be completely honest, we don't actually do a whole lot in about half the cases that get reported to us. The site usually disappears or otherwise stops working for some other reason without us ever having to contact them. The cases where we do have to take action take a while because of our process. When we contact sites using content inappropriately, we send them at least two emails before we jump to taking other actions (such as attempting to contact their hosting provider). This process was even more dreadfully slow due to the way we were organizing this information - which made it hard to see which ones had been contacted previously or need contacted again, or what our next steps might be.

As far as finding them, we don't actively look for scrapers. We sometimes run across them while we're searching for things on our own, and we'll record them ourselves if we find them. But there is no active campaign or anyone here that is explicitly searching for sites which don't follow our license terms. Our users are a much more reliable source for finding these sites, because they're the ones with the questions, the ones searching on Google for all these random things, and the ones running across these sites on a more regular basis.

To get to some of the deeper issues in your question, let's identify one thing: we cannot protect your copyright for you. We don't own any copyrights for your content. We control the license that you agreed to use when you posted it here. The only authority we have over other sites using it is the terms we outlined in that license, which pretty much boils down to providing proper attribution.

Scrapers in general are not disallowed, and if they follow all the attribution requirements outlined, there is absolutely nothing we can do about them hosting copies of the content on their site. As outlined in the scraper FAQ post, these are the things we care about when users report the sites:

  • The site is a proxy.
  • The site outranks us on Google.
  • The site doesn't follow attribution requirements.
  • The site uses a different license or claims ownership of the content.

These are all things we can actually take action on, and want to know about.

As far as a publicly accessible list: we had considered making this new system public, but opted to keep it private since we don't want users messing around with the records or entering things on their own. Allowing this just creates more work for those who are processing these, which is exactly what we're trying to cut down on by making this new system in the first place. Keeping users out of these areas would require a login system, which we simply weren't interested in building for a tool like this. Hence, it was kept private and only accessible to Stack Exchange staff.

  • Phew an answer from an S.E. person. Thanks very much for the information. My issue is that I care about number three " The site doesn't follow attribution requirements." much more than I do about the others. Because that's the one that directly deals with my content. I'm worried that there's going to be a massive proliferation of scraping, because sites are going to scrape from the scrapers and so on. The list that was removed from this post (9 sites) only took me twenty minutes to find. Give me a day and .... I can't do that much work to protect m own copyright, or those of other users' ... – Araucaria Jan 6 '16 at 21:55
  • ... on my own. I think having some kind of dedicated monitoring by S.E. - even a modest amount - would be beneficial. It wouldn't require action being taken on every one that was found, but if any were found to be persistent and also being scraped off themselves it would be effective. If S.E. completely loose control over content, this will be a disaster. I know that obviously S.E. cares more about its own success, but S.E. relies on posters first and foremost for its success. Taking this small measure to look after users' content would surely be in its own interests. – Araucaria Jan 6 '16 at 21:59
  • By the way, do you have any qualms about me listing the scraping sites in my post? – Araucaria Jan 6 '16 at 22:01
  • 1
    @Araucaria We already tried a list of scraper sites here on meta before. It got very messy very fast, even worse than our spreadsheet is now. We're not interested in doing that again. – animuson Jan 6 '16 at 22:03
  • If I'm unhappy about how my material is being used, is it my own responsibility to take steps to protect it? – Araucaria Jan 6 '16 at 22:06
  • I'm not happy about the current processes, but I'm very happy to get a forthright, friendly and honest answer. Thanks. – Araucaria Jan 6 '16 at 22:19
  • @Araucaria you own the content, you own the license. It is your responsibility to defend your copyright, although SE does try to help on your behalf. The "how to report" scrapper post you linked does mention that. – psubsee2003 Jan 6 '16 at 22:23
  • @animuson .. and thanks for working on the spreadsheet too. – Araucaria Jan 6 '16 at 22:32
  • 3
    @animuson How about publishing it even if we can't access it? – Araucaria Jan 6 '16 at 22:55
  • Why not make a github repo? Then we can submit things via pull request or issue tracker. – PyRulez Feb 10 '16 at 4:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .