21

Twice today I have come across questions requesting assistance scraping data from copyrighted web pages. Both questions listed the site in question which, after a quick visit, shows the site content to be protected under copyright.

In each case I added a comment cautioning respondents to consider the ethical issues of assisting in the theft of intellectual property. In one case, a previous respondent replied he hadn't considered this when answering the OP's question.

Ignoring for the moment the possible legal culpability of respondents and of SO itself...

How should questions like these be handled?

Should respondents consider ethical issues like these before answering? And if so, how can we assure they do?

Improve this question
10
  • 6
    If it is clear that the OP is being abusive, I close these kinds of questions as too localized.
    – user102937
    Commented Jun 3, 2011 at 17:33
  • 1
    Knowledge should be free... It's up to the asker to determine legality of use.
    – user140178
    Commented Jun 3, 2011 at 17:39
  • 7
    Note that the Terms of Use for the StackExchange network is pretty clear about this. It specifically prohibits posting of content that violates the rights of others. See stackexchange.com/legal under the Subscriber Content section.
    – user102937
    Commented Jun 3, 2011 at 17:48
  • 4
    Does web scraping always violate copyright? I would assume that's only an issue if it's actually being copied somewhere, whereas a tool that you write to display data for your own use is more or less a stripped-down web browser. For example, I have a small python script that does a GET on one website and tells me what the headline is. I don't think I've violated anyone's copyright, at least that I can see. Would it be OK for me to ask for help constructing something like my tool?
    – dsolimano
    Commented Jun 3, 2011 at 19:01
  • 1
    @dsolimano Good question! I suppose it's a question of degree. In the questions that triggered this discussion, the OPs stated they wanted to extract all the terms and definitions from an online dictionary or all of the records from a multi-page list of contacts. This kind of scraping isn't the same as, for example, creating an RSS feed of articles from a news site for personal use.
    – Rob Raisch
    Commented Jun 3, 2011 at 19:06
  • 2
    @dsolimano: Check the website's Terms of Use. Scraping is often prohibited. If you're in doubt, just ask the website's owner. Your particular usage scenario is probably OK. There's a vast difference between obtaining the title of individual pages for personal use, and ripping an entire website for commercial purposes.
    – user102937
    Commented Jun 3, 2011 at 22:10
  • @RobertHarvey what is the legal definition of scraping? Are you breaking any law because you visit the site - your browser must scrap the content before it can show it to you!
    – Cjxcz Odjcayrwl
    Commented Sep 4, 2013 at 11:08
  • @ŁukaszLech check with the site's Terms of Service. If the TOS allows you to scrape or crawl the site with robots, then it's OK. Most sites do not allow this, however. Get permission from the site owner.
    – user102937
    Commented Sep 4, 2013 at 14:19
  • @RobertHarvey what legal meaning have the Terms of Service? Everyone can write anything, but it doesn't mean that everything will have legal power. If someone writes you are not allowed to view the site barefoot, such statement is meaningless.
    – Cjxcz Odjcayrwl
    Commented Sep 4, 2013 at 18:30
  • 1
    @ŁukaszLech: You're just trying to get me to say that it's somehow legal, so that you can do whatever you want with the site, against the site owner's wishes. I'm not going to say that. It doesn't have to be illegal to be something that we don't want to promote on Stack Overflow, so stop asking for legal justifications.
    – user102937
    Commented Sep 4, 2013 at 18:35

2 Answers 2

Reset to default
24

I'm hard pressed to find a solution better than the one you did, which is to provide technical guidance for the process in general and add a disclaimer (I'd prefer at the beginning of the answer vs. at the end) warning the developer to consider the legal and ethical ramifications of what they're doing.

At a previous job of mine this was actually a bit of an issue. We were aggregating public criminal records and in many cases that involved screen scraping on various jurisdiction websites. Some of the websites had statements indicating that the information was their property and that scraping was against the terms of service.

In that particular case, our company had a liability attorney on staff and he assured us that the statements are of no concern. The company made the decision to go forward with it against the advice of the developers. (Makes sense, he is after all the company's attorney. And the data is public records.) In one case it caused our scraper to be IP-banned by a server, but I don't know of any other problems that came about as a result.

Long story short, we as a community (both employees of and users of Stack Exchange) are in no position to offer legal advice of any kind. We aren't experts in law, we have no knowledge of the specific case, etc. Even if the site in question explicitly states that scraping is entirely illegal and violators will be killed to the fullest extent of the law, etc. that doesn't mean that the statement actually has any legal value.

The best we can do is offer technical advice and strongly urge the reader to seek legal counsel for non-technical concerns.

Edit: In re-reading your question, you make a very interesting point at the end. The idea of ensuring that respondents make such disclaimers. For that, I see two options for Stack Exchange:

  1. Have site-wide disclaimers (surely they already do, but maybe re-think the visibility of them) indicating that this isn't legal advice, etc., etc.
  2. Encourage (or at least not frown upon) users to edit other users' answers to include such disclaimers.

The former solution seems like the best. I'd hate to see the long-term results of the latter solution. Users would get upset that their answers are being filled with legal jargon when they don't see why it should. Answer meanings could be slightly changed by the edits, which does a disservice to the original respondent. Etc.

Improve this answer
6
  • 2
    Thanks much for your considered reply. But as I understand Intellectual Property law (barely), providing assistance in this case opens the respondent to legal liability should the property owner decide to protect her rights. It's not hard to imagine that an argument could also be made that SE facilitated that assistance and is thus also legally culpable. I'm not interested in providing legal advice to anyone, but I'm concerned respondents are unaware of the possibility of legal exposure here.
    – Rob Raisch
    Commented Jun 3, 2011 at 17:40
  • 1
    @Rob Raisch: Good point. As that same company lawyer once told me, "In court 'illegal' is defined as how well they can out-debate me. It's entirely subjective." Going that route, anything Stack Exchange and its userbase provides can be liable for any wrongdoings provided that an argument can be made at all, whether it makes sense or not. (Kind of scary, really.) We live in a culture (at least I do) where potential liability seems to be a way of life. Giving an injured motorist first aid can land you in court for touching them inappropriately. (This slope sure looks slippery...)
    – David
    Commented Jun 3, 2011 at 17:48
  • 1
    Indeed, but I call out this specific example (where a poster specifically requests help to commit theft) because it is so blatant. Lots of things carry the potential to create legal liability, these questions realize that potential immediately.
    – Rob Raisch
    Commented Jun 3, 2011 at 17:56
  • 4
    @Rob Raisch: Given that, I really like Robert Harvey's comment on the question above. It's up to the voters on a case-by-case basis of course, but it's definitely a viable approach. In that case I guess it really comes down to the difference between "How do I scrape data from a website?" vs. "How do I scrape data from this specific website?" If the question is asked as the latter, then I agree with you that there's a potential problem. Close as too localized. If the question is asked as the former, then it's just a technical question with no legal context.
    – David
    Commented Jun 3, 2011 at 18:00
  • I also like Robert Harvey's comment and will point others to the SO Terms when appropriate. Thanks again.
    – Rob Raisch
    Commented Jun 3, 2011 at 18:03
  • Good answer, the owner of the site can state that the web site can be viewed only by white males wearing black socks (or opposite) and any other use is the violation of terms of use, but such statements have no legal validity.
    – Cjxcz Odjcayrwl
    Commented Sep 4, 2013 at 11:10
4

The question as stated is almost useless without specific examples. Here is why:

The [legal] loophole in copyright is fair use. Under the banner of fair use, you could legally upload a video without the copyright holder's permission. Anyone who contributes anything to the web should have the four factors of fair use commited to memory by now:

  1. the purpose of the use
  2. the nature of the copyrighted work
  3. the relative amount of the portion used
  4. the market effect of the use on the copyrighted work

These are the four factors courts use to determine if something is fair use.

No way to judge if it's fair use or not until we have specifics.

We regularly smack down anything that copies wholesale; we want a contextual quote with the most relevant bit, and a link to the rest.

Improve this answer
2
  • In the examples I originally referenced, the posters requested assistance scraping the entire contents of a contact database and all listings from a dictionary of medical terms, neither of which fall under the Doctrine of Fair Use as I understand it. But even if a use could be defended as "fair", I think responders should be made aware of the legal risk they run, especially given the potential financial disparity they might experience if drawn into litigation. (I'll provide relevant quotes and links in a bit.)
    – Rob Raisch
    Commented Jun 4, 2011 at 15:56
  • @RobRaisch is the scrapping of the whole site violating the doctrine of fair use? How it differs from clicking through the whole site? All depends, what you want to do with that data. Is there any specific law against downloading sites for offline usage? But that's exactly how browser cache works! And if there are such laws, are they in power for the whole world or only for a small part of it?
    – Cjxcz Odjcayrwl
    Commented Sep 4, 2013 at 11:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .