A Terms of Service update restricting companies that scrape your profile information without your permission

Question

Update (April 19, 2016): This change is now live. You can view the updated Terms of Service here, or read about them below.

---

tl;dr:

We’re planning to make some changes to our Terms of Service to stop companies from scraping profile information from multiple websites and spamming users. The changes only restrict a specific kind of commercial use of profile data that is scraped or copied from our site without the user’s permission. The changes don’t impact Q&A content (like posts or comments) in any way.

What’s happening:

A number of companies seem to have the following business plan:

1. Use scrapers, bots, or very cheap labor to collect developers’ profile information from sites like Stack Overflow

2. Merge that information with profiles they find on other sites (open source contributions, LinkedIn, etc.)

3. Create a huge database of spammable users

4. Sell this database as a "careers product" to slimy recruiters to use for spamming purposes (often telling those recruiters that these developers are their "members")

5. Profit!

Why it’s bad:

• We don’t want any more of our users to get annoyed and spammed by these bozos people.

• It’s directly competing with our jobs business. We’ve built a jobs site that respects developers and puts them in control of when they are contacted. We don’t appreciate competitors who take our users’ data without their permission and don’t show developers the same respect.

We believe that developers should have full control over how their personal information is used, and whether or not they want to be contacted by recruiters. Acting in the developer’s interest is one of the central tenets of Stack Overflow Jobs. That means we don’t allow spam, and we put developers fully in control of who can (and can’t) contact them. So we find it pretty infuriating to see companies whose entire business model is scraping our users’ info and spamming them.

What’s changing:

The following will be explicitly prohibited in the Terms of Service, and applies to all sites in the Stack Exchange network:

Scraping users’ profile info for commercial purposes:

Profile Content that is NOT available via the Stack Exchange API ("Personal Profile Content") cannot be used for any commercial purpose, individually or in aggregate, or be republished without the explicit consent of the author of such Personal Profile Content or the explicit consent of Stack Exchange.

We were very careful in how we worded this in order to ensure three things:

1. We did not want to restrict normal user-generated content (posts, comments, etc.) in any way, or undermine one of our earliest promises: We don’t own your content, and if we turn evil, it’s available to you or someone else to liberate. (Did you know that the entire Stack Overflow data set is available as a creative-commons licensed downloadable data dump?)

2. We didn’t want to restrict anyone using the API to access profile content innocently, for doing things like linking posts to users, etc. All content previously available through the API (usernames, rep, etc.) is still available.

3. Nothing about this transfers any rights from our users to us. Users can even give permission for these "services" to scrape their profile info from our site. We don’t know why you would want to, but it’s there for ya if you do.

Today, this restricts just a handful of fields that are on the public profile, but not in the API (like "People Reached.") In the future, it will also cover the personal information in the Developer Story, including all public information about your career history.

Claiming users you found on Stack Overflow are your service’s "members":

Additionally, no Profile Content, including API Profile Content, may be used in any way that implies a user is affiliated with, has signed up for, or is in any way associated with a third party without explicit permission from Stack Exchange or the user.

Some users asked if, "without the explicit consent of the author... or the explicit consent of Stack Exchange" might mean that this granted us the right to opt you into some other organization's membership. Fear not - it grants us no new powers to put you on the membership rolls of the National Order of TRS-80 Enthusiasts (or any other org). All it says is that if a company has not gotten permission from one of us, they may be sure that they are in violation.

Creepy stuff, and we don’t have to debate what "creepy" means:

Stack Exchange may also terminate, block, or suspend any and all Services and access to the Network immediately, without prior notice or liability, in its sole discretion, for any reason or no reason at all

We don’t have to let anyone scrape our users' private data, and if they do so to harass or misrepresent our users, or do anything else that makes users’ lives even slightly less lovely, we have the right to block them. (This doesn’t really change anything - we already have the right to block malicious IPs, and do so occasionally, but our lawyers tell us that saying so explicitly in the ToS can save time dealing with bad actors.)

A couple of notes:

• We plan to keep everything that's currently available through the API available in the future, although it’s possible that could change if we see companies annoying users in ways that seem to outweigh the benefits.

• This has no impact on how "normal" content (posts, comments, etc.) is licensed on Stack Exchange. You may remember our prior proposal to change the license for code contributions. Our initial ideas for that were controversial, so we slowed that process down to give us time to work with the community on the best solution to code licensing.

Source materials:

Here are the key new sections, all in one place. (Many of these are excerpted above)

Profile Content is information about you (a Subscriber) that is contributed by you or inferred about you by your activity. Profile Content includes, but is not limited to, display names, reputation scores, avatars, your role and company, and other user generated content found on a Subscriber's profile such as "About Me" content.

Profile Content that is available via the Stack Exchange API ("API Profile Content") is perpetually and irrevocably licensed to Stack Exchange and its Subscribers under the Creative Commons Attribution Share Alike license.

Profile Content that is NOT available via the Stack Exchange API ("Personal Profile Content") cannot be used for any commercial purpose, individually or in aggregate, or be republished without the explicit consent of the author of such Personal Profile Content or the explicit consent of Stack Exchange.

Stack Exchange reserves the right to exclude Content, including Profile Content, from the Stack Exchange API at any time without prior notice.

Additionally, no Profile Content, including API Profile Content, may be used in any way that implies a user is affiliated with, signs up for, or is in any way associated with a third party without explicit permission from Stack Exchange or the user.

Under no circumstances will Subscriber use the Network or the Service to (a) send unsolicited e-mails, bulk mail, spam or other materials to users of the Network or any other individual, (b) harass, threaten, stalk or abuse any person or party, including other users of the Network, (c) create a false identity or to impersonate another person, or (d) knowingly post any false, inaccurate or incomplete material, or (e) copy, download, or scrape any Personal Profile Content for the purpose of indexing software engineers, social recruiting, sourcing, employment-related services, compiling databases of employment solicitation targets, providing content for a hiring platform without the express permission of Stack Exchange or the User.

And here’s a link to the current version, if you want to see what’s changing, or really like to read long-winded, lawyer-y stuff.

Answer 1

A few clarifications here.

The 'diff' here is minor, but not at all unimportant.

By 'minor' - we're simply saying that the scope of the change is small (in size), but not in importance. Think of it like a "minor edit" that makes a clear difference.

It clarifies how we've been executing and operating, while closing loops for bad actors to argue. I'm not sure what else to call it, small? That doesn't quite work either.

We've been doing this for a long time, that's nothing new.

Fun fact, AWS instances are restricted to using our API - they can't load full content pages. Amazon instances were restricted to our API. This was put in place to combat the hordes of SEO scrapers that were misusing our services. Better handling of scrapers in general let us relax this restriction, but it's an example of measures we take to combat evil robots.

We also drop traffic from hosts that send us bats**t-crazy numbers of requests per second without giving it another thought. This happens daily.

If we catch machines up to naughty things, we don't think twice about blocking them at the network level - we must.

But when people have built business models around scraping your profiles and information, they fight back significantly harder when we block them.

We want them to stop. They'd prefer a protracted game of rule-lawyering that puts them in a position similar to this:

They don't deserve that much time and attention from us. We'd rather give it to you. This change lets us block them at every occurrence, while alleviating us of the need to listen to any of the horses**t that ensues.

Think of it like a burglar so delusional that they complain about your door locks interfering with their profits, and thinks it's a totally valid gripe. Yeah.

This doesn't change how we handle behavioral concerns administratively.

Even spammers identified by our anti-spam system are allowed to read the site. We really hate using lower level blocks because of the risk of blocking actual humans.

This change in no way signifies that we're abandoning our long-held model of giving discussion every possible chance before reaching a series of timed suspensions of varying length. If there's any evidence that someone finds their account valuable for normal use of the site, we'll work with them to keep it. Sometimes this just doesn't work, but that's extremely rare, and nothing new.

Spammers & blatant trolls are zapped on sight, because they fail this pretty critical test of caring about their account and the normal use of the site that it enables.

As custodians of your trust, we must adapt over time.

This kind of crap needs to stop. We can't really define what 'creepy' is, but that's a pretty darn good example of it. When the actions of third-party bad actors result in folks questioning the trust that they've so graciously put in us, we must take measures.

I hope that's the last time for the foreseeable future that we have to touch that particular document. I don't know that it will be, I can't see into the future - but we will be very open and transparent about new problems that we need to solve and why they're important.

Answer 2

I'm sure this will be unpopular, however... I don't feel this is a good policy, nor a necessary one. Some key points:

• Every job I've received, (day job or otherwise), in the last 4 years was due to someone finding me on Stack Overflow. Almost all of these have been casual, 1:1 situations where I answered a few of someone's questions and they decided to hire me.
• Disallowing commercial usage of profile data, with user permission or not, will stop all legitimate relevant usage of Stack Overflow as a recruiting tool. This is an opt-in policy with no clear way to do so. Am I to put something at the bottom of my profile that says, "you may use this profile for commercial purposes"? Who would bother using Stack Overflow for this purpose if there would be 1 out of 10,000 developers that would do this?
• Not everyone using profile data for commercial purposes is some giant disrespectful spammer. This proposed policy casts far to wide of a net. For example, if I have a small project that I'm working on and I see that a particular user has a lot of domain knowledge in what I need, I might check their profile to see if they've listed an e-mail address. If they have, I might e-mail them and ask if they are available. This isn't what most people call spam. This isn't disrespectful. This is how a community networks and helps each other out.
• We can opt out today by simply not sharing information. There is no need for anything else. If you don't want someone to have your e-mail address, don't publish your e-mail address.
• A policy won't prevent illegitimate usage. Spammers will ignore your policy and do whatever they want anyway. I understand that having more legal tools in the tool box may theoretically help, but the best way to stop someone from getting your information is to not make it available in the first place. Leave it up to the user to decide what to publish, as we can already do today.

Just in case I'm not stating my argument very well, let me clarify a bit more:

Under no circumstances will Subscriber use the Network or the Service to (a) send unsolicited e-mails…

Every e-mail to someone not suspecting it is unsolicited. According to your new terms of service, I can no longer e-mail anyone on Stack Overflow for any reason, commercial or not, unless they're expecting it in some way. What's the point of having an e-mail address field at all?

The changes only restrict a specific kind of commercial use of profile data that is scraped or copied from our site without the user’s permission.

Maybe that's your intent, but that's not how I read the new ToS change. There's very little specific about it. It applies to all commercial usage, which is a very broad net.

Why it’s bad:

…

• It’s directly competing with our jobs business.

Your Jobs business is extremely expensive in some situations and not at all useful for small projects. If myself and 4 other engineers are always posting on the same related topics, and I want to e-mail one of them in relation to a short-term commercial project I'm working on, and they have willingly published their e-mail address, why shouldn't I be able to do that? If they don't want to be contacted they can either not reply, or not publish their info in the first place. Again, I've been on both sides of this, being hired and doing the hiring, and I've found it to be very beneficial. Almost everyone I've ever e-mailed replies... because I'm not spamming them, I'm sending relevant courteous e-mails from a real person. (Maybe ~5 times a year in total, not to the same person.) There is no way that a weekend project with less than $2,000 at stake is going to end up on Stack Overflow Jobs.

We believe that developers should have full control over how their personal information is used

You have already given us that full control by allowing us to choose what to publish.

In Summary

• This new policy casts too wide of a net if you truly are going after the spammers in you outlined in your example.
• Stack Overflow has a social aspect that will be damaged by this new policy.
• A new policy is not needed as we can already control our personal information.

Answer 3

Thanks for the way this is brought to us! You really learned from previous posts with hard deadlines and lack of clarity.

A few points.

1. I really think it is good to protect our content here. I receive numerous emails, LinkedIn requests, etc., some of them that seem to be related to what I post and share here.

2. My bad

3. I quote:

   without the explicit consent of the author of such Personal Profile Content or the explicit consent of Stack Exchange

   So that means SE can grant third parties access to the profile content. Can we make explicit what this profile content is? Which fields or pages are we talking about? Is there any sensitive information that could be shared? What has changed in that regard?

4. Please remove 'minor' from the post title. It seems out of place here: either the subject is not important enough to be called 'major' or you think the change isn't that big. Both not true in my opinion.

Answer 4

I do not understand this proposed change to the Terms of Service (ToS).

Stack Exchange cannot revoke Creative Commons licensing rights that have already been granted. Therefore, they cannot "un-license" any existing user profile web page content or other user-contributed content that has already been published under CC BY-SA.

However, that appears to be exactly the intention of this proposed ToS change. Hence my confusion.

In short, the intention seems to be to use the ToS to impose terms that directly contradict the license under which the data was published.

If I have misunderstood, please correct me.

If I haven't misunderstood, then the proposed new ToS amount to little more than an expression of vicarious licensor remorse, at least in the case of all existing profiles. By being inconsistent with the perpetual terms of the existing license, the new ToS would probably be unenforceable in this regard and therefore somewhat pointless.

Answer 5

Small suggestion for the text:

Under no circumstances will Subscriber use the Network or the Service to [...] person, or (d) knowingly post any false, inaccurate or incomplete material, or (e) copy, download, or scrape any Personal Profile Content for the purpose of [...]

Emphasis mine and also left out two pieces denoted by [...].

I suggest to remove the emphasized or. Between (a), (b) and (c) there are only commas. The word or is absent there. Therefore it is inconsistent to have this between part (c) and (d), because now part (e) was added. (In the current ToS, this part is present, just without part (e))

Disclaimer: I am not a lawyer and this is not legal advice, nor has it ever been legal advice.

---

However, I think that this update is a very good idea, even though I have never had any such emails. I've seen several meta posts complaining about emails.

Answer 6

Key point here:

Additionally, no Profile Content, including API Profile Content, may be used in any way that implies a user is affiliated with, signs up for, or is in any way associated with a third party without explicit permission from Stack Exchange or the user.

In other words, Stack Exchange takes upon itself the burden of associating my account with American Society for the Promotion of Elf Welfare (ASPEW) or PETA or Boston Lockpickers' Guild without my knowledge or permission.

I'm sure I don't want that to happen.

---

Another point worth making: the linked post (Recruiter claims to have gotten my email address from Stack Overflow) and answers thereto specifically state that the e-mail address wasn't scraped from Stack Overflow Profiles. I'm baffled why it is quoted as the reason for ToS change.

Answer 7

Under no circumstances will Subscriber use the Network or the Service to [...] (d) knowingly post any false, inaccurate or incomplete material

Does this mean that it's now technically against the terms of service to post an not-fully-complete response to an answer, even if it's helpful? Obviously that's not the intent of that statement, but as written it looks like that's disallowed.

Answer 8

So what if I do not agree with the new Terms of Service? The current ToS provisions state:

Use of the Services by Subscriber following such modification constitutes Subscriber's acceptance of the terms and conditions of this Agreement as modified.

where Services is defined via:

Please read these terms of service (“Agreement”) carefully before using the Network or any services provided on the Network (collectively, “Services”).

If I am not mistaken, then the act of reading the new Terms and Conditions already constitutes use of the Network. Therefore, the Terms and Conditions require me to agree to the changes before I can read what the changes are.

Am I misinterpreting this policy? Or can this policy be clarified to allow disagreeing with Terms of Service, such as by opting out of sites for which I do not agree?

Alternatively, we can have 30 days' notice before the new ToS comes into effect.

Answer 9

SE's TOS should also explain how to report abuses of user profiles, and that could be done in the edit suggested in this question. The only use of noun "report" on that page is about reporting abuses to the "Digital Millennium Copyright Act".

As a minimal edit, I suggest that that page could link to A site (or scraper) is copying content from Stack Exchange. What should I do?, or to another relevant page.

The standard "attribution required" page also doesn't explain how to report abuses.