Timeline for June 2023 Data Dump is missing
Current License: CC BY-SA 4.0
52 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 30 at 14:40 | comment | added | Kuba hasn't forgotten Monica | Human users definitely profit individually from using this site. Say a corporation has a whole department worth of developers who got better at their jobs due to StackOverflow content. You could say that this corporation "profits" from shifting the training costs over to StackOverflow and contributors. What makes LLMs somehow special? Teaching humans, teaching machines, potatoe, potato. | |
| Jun 14, 2023 at 15:29 | history | edited | Random Person | CC BY-SA 4.0 |
Added the new link after the two posts were merged
|
| Jun 14, 2023 at 6:45 | comment | added | tripleee | @Gloweye For the record, there is a separate question about that; the consensus seems to be no. | |
| Jun 14, 2023 at 0:18 | history | edited | V2BlastStaff | CC BY-SA 4.0 |
clarified link description; merged link into body text
|
| Jun 13, 2023 at 23:55 | history | edited | Jeff Atwood | CC BY-SA 4.0 |
added link to updated policy
|
| Jun 13, 2023 at 23:19 | comment | added | willeM_ Van Onsem | To prevent building LLMs? The previous dumps already contain several million q/a's the new dump only a few 100k extra. Likely the extra data does not matter that much to build an LLM... | |
| Jun 12, 2023 at 14:56 | comment | added | M-- | I miss the days when we could have a conversation about these statements and the questions and comments were getting responses from the staff. Now, we only get one statement, one that is way too general, way too political, and pretty darn late. After that, radio silence. | |
| Jun 12, 2023 at 8:18 | comment | added | OrangeDog | @SextusEmpiricus and LLMs do reproduce whole pieces | |
| Jun 12, 2023 at 6:59 | comment | added | Gloweye | SE doesn't own the data. The users do. And we've licensed it to Stack Exchange under CC BY-SA 4.0 (or earlier versions). This answer makes me concerned Stack Exchange is breaking the license terms. | |
| Jun 12, 2023 at 3:10 | comment | added | David Roberts | @Jody can you comment on MO data as distinct from the rest of the SE network? | |
| Jun 11, 2023 at 20:05 | comment | added | Sextus Empiricus | @OrangeDog everything that I write originates indirectly from others (including every SE post that I have ever read). My brain is one big copy machine that reads math books, looks at art, and listens to music and creates copies (transformed into impressions) by storing it into a network of neurons. All that copied content indirectly ends up in whatever I express as my brain is taking continuously these trained bits of impressions when I generate output (actions) in response to my environment. As long as I don't reproduce whole essential pieces then the CC-BY license used here doesn't apply. | |
| Jun 11, 2023 at 19:58 | comment | added | OrangeDog | @SextusEmpiricus if what you're writing is taken from an SE post, then yes. You also have to license it under compatible terms. | |
| Jun 11, 2023 at 19:05 | comment | added | Sextus Empiricus | @OrangeDog I also have been training my brain on SE data. Do I need to give attribution to all my training sources for every sentence that I write? | |
| Jun 11, 2023 at 16:58 | comment | added | OrangeDog | @SextusEmpiricus yes, the AI implementations. Yes, when ChatGPT tells you something it is republishing. Just the same as an SO proxy site. It doesn't give attribution for where it copied it from, so is not compliant with the license. Nor does it licence its modifications under CC BY-SA. | |
| Jun 11, 2023 at 15:42 | comment | added | Sextus Empiricus | @OrangeDog who is they that you are referring to and how do they not comply with licensing? Who is copying outside of terms like fair usage? Is it the AI-companies? That is not very well motivated. Can you ask something to chatGPT and it answers with a copy of SE/SO Q&A's? A LLM is just copying the content to it's artificial brain, it is not re-publishing the work. It is not anything different from humans copying questions and answers into the neurons of their brains (and when humans re-use parts of the information they dont give attribution either unless they copy something substantial). | |
| Jun 11, 2023 at 15:28 | comment | added | OrangeDog | @SextusEmpiricus because they don't follow the license and neither BY nor SA. | |
| Jun 11, 2023 at 15:27 | comment | added | OrangeDog | @ACuriousMind which LLM companies give attribution for their data sources? Or open-source their results? The correct solution, and the point of having a license, is to sue them. Not to make it harder for everyone to get the data. | |
| Jun 11, 2023 at 5:43 | comment | added | BryKKan | This is really problematic on so many levels. At the very least, you needed to announce it in advance. Not just for decency, but also to meet your committed responsibilities to the community. You must assume that anyone who contributed to date did so relying on SE Inc's past firm commitments to upload data. I don't see how you avoid sharing this dump, and probably the next one - in full - without incurring some form of legal liability. Protecting against such claims means releasing dumps that include any posts made before the announcement. So there's no point holding it while you work. | |
| Jun 10, 2023 at 22:27 | comment | added | 0Valt | @SextusEmpiricus let me translate the corp-speak into common tongue speech real quick, it means: "they are not paying us money for that data". Not worth wasting keystrokes on what "misuse" means, we all know what it really refers to. | |
| Jun 10, 2023 at 21:26 | comment | added | Sextus Empiricus | "Stack Overflow data from being misused by companies building LLMs." What misuse? How is building LLMs based on human conversations a misuse of data? SE/SO is all about helping other people and teach them how to solve simple questions. If a LLN uses it to learn how people answer questions then this has nothing to do with copyright infringement or other misuse. Millions of people are reading posts on SE/SO and incorporate the methods of answering into their vocabulary such that the next time that they answer a question themselves they will be able to answer more eloquently. | |
| Jun 10, 2023 at 21:21 | comment | added | Sextus Empiricus | "preventing misuse by organizations looking to profit from the work of our community" Our community? Does SE Inc own the community? Which organization is profiting most from the work of the community? Is there a list and which organization would be on top with how much money? How does SE inc., which obviously profits from the community and would not exist without it, compare to those organizations? | |
| Jun 10, 2023 at 14:58 | comment | added | Wowfunhappy | "We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations looking to profit from the work of our community." As a StackExchange contributor, I want organizations to be able to use my work to train AI models which benefit humanity. Critically, I want all organizations to have this ability, not merely a select few who can pay for the privilege! | |
| Jun 10, 2023 at 10:23 | comment | added | Caleb | By "guard rails" apparently you mean making sure the dump is actually quality training data. Ironically disabling new dumps is probably the best way you could have accomplished the opposite of your stated goals with ML… with your new policies any new dumps are going to be inreasingly poluted with LLM generated content anyway, so that's the most valuable dump there will be going forward until you add back the one guard rail the whole community has clamoured for: no generated content in the human currated data set! | |
| Jun 10, 2023 at 9:47 | comment | added | wizzwizz4 | @Someone-OnStrike Profiting from the work of the community without attribution is misuse, though. (Sadly, that's not what the company is defending against.) | |
| Jun 10, 2023 at 5:47 | comment | added | Someone | "misuse by organizations looking to profit from the work of our community" Profiting from the work of the community is not misuse. Through the CC BY-SA license, we have authorized anyone to use our content for any purpose, including commercial purposes. | |
| Jun 9, 2023 at 22:55 | comment | added | Jon Ericson | A good time to have announced this change was back in April when the CEO addressed the concern with LLMs using Stack Overflow data. There was even a handy meta question about it. Hoping nobody would notice was never a good idea. | |
| Jun 9, 2023 at 22:26 | comment | added | Ryan M | I'm all for ensuring that the community gets credit for the contribution to LLM training of material we've written. But is there really reason to believe that the data dumps, SEDE, and the API are actually how they're getting this information, rather than web scraping? Consider, for example, that OpenAI is closely partnered with Microsoft, which operates a large search engine. Unless the restrictions you impose are very specific to LLM training, I fear this will hurt the community (and by extension, the company supported by its work) more than it helps via increased funding or attribution. | |
| Jun 9, 2023 at 22:18 | comment | added | cocomac | Friendly reminder for those commenting on the licensing: While I'm not a lawyer, I think that Stack Exchange content is dual-licensed, so only looking at the Creative Commons license is somewhat less useful in this specific case. | |
| Jun 9, 2023 at 21:21 | comment | added | Erik Reasonable Rates Darling | “No downstream restrictions. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material.” | |
| Jun 9, 2023 at 20:51 | comment | added | Bryan Krause | Is there any reason you think the data dumps would be the way that LLMs have used SE training data, versus crawling/scraping the site? I would have assumed the latter, since then they can use the same tool over many sites versus digesting the SE data format specifically. | |
| Jun 9, 2023 at 20:41 | comment | added | wizzwizz4 | @DimeCadmium Stack Overflow claims copyright over the aggregation. Which is weird, since we're the ones doing all the moderation (the actual work of curating the aggregation); surely we have the copyright, there? I was happy to not question it, and permit this polite fiction, but that was when we had the data dumps. | |
| Jun 9, 2023 at 20:27 | comment | added | DimeCadmium | Those AI and LLM trainers you pretend to be concerned about? They'll just screen scrape, that's easy for them to do, remove the HTML and maybe the scripts and styles and... you've got language left. But the rest of the community? Screen scraping is a lot tougher and more fragile for the rest of us. | |
| Jun 9, 2023 at 20:12 | comment | added | Script47 | "We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations looking to profit from the work of our community." - Do you not see the irony of this statement when for years your power users and mods have been complaining about SO doing the same garbage. | |
| Jun 9, 2023 at 19:56 | comment | added | khelwood | Of course. The last thing anyone wants is some commercial organisation seeking to profit off the work of the community. | |
| Jun 9, 2023 at 19:30 | comment | added | vbnet3d | What happened to at least giving heads-up that something like this was going to happen? That's what community used to mean. This new way of just doing stuff without any communication isn't the best community experience | |
| Jun 9, 2023 at 19:30 | comment | added | CodeCaster | Please just release it and let those AIs choke on it. Your estimates in the data post are off by orders of magnitude. AI posts are way more rampant than you publicly dare to admit. Let them feed that AI-generated nonsense back into their algorithms and watch the trend burn, please. | |
| Jun 9, 2023 at 19:18 | comment | added | zcoop98 | I really, really want to take this at full face value. I also think the Company really needs to understand how important this data dump is to the community, and how easy it is to read this as a message that the dumps are "temporarily postponed" indefinitely, and will never return. I don't think the Company gets to have the benefit of the doubt with messaging like this, this week of all weeks... I beg you bear in mind how the messaging comes across, and how easy it feels, from the outside, to read this as a way to misdirect and soften the blow of doing away with the data dumps for good. | |
| Jun 9, 2023 at 19:05 | comment | added | user1937198 | How can you gate what people who you give the dump to do when the license explicitly renders such restrictions void? law.stackexchange.com/questions/11067/… You can ask people what they will do with it and stop giving it to them. But you can't implement 'safeguards' without restricting who has access significantly. | |
| Jun 9, 2023 at 18:58 | comment | added | ꓢPArcheon | As other comments said... First, why waiting (again) for someone to "catch you red handed". Second, how this fits in with the question Is SE (....) selling our content for AI model training? And what exactly does "reinvest back into our communities" mean?. Does this imply a change of plan or should be interpreted as an action to avoid others getting for free the data you plan to sell? | |
| Jun 9, 2023 at 18:54 | comment | added | ACuriousMind | How can data licensed under the CC-BY-SA licenses that SE content is licensed under be "misused"? The license explictly allows others to do essentially anything they want with the data as long as attribution is given, in particular profit off of it. It can't be "misused*, the users can only fail to give attribution or follow the share-alike requirement. It is entirely unclear in what ways restricting access to the data would improve the situation with respect to attribution. | |
| Jun 9, 2023 at 18:49 | comment | added | Franck Dernoncourt | How can SE gate access to the Dump that will allow individuals access to the data while preventing "misuse" by for-profit organizations? | |
| Jun 9, 2023 at 18:46 | comment | added | Levente | "organizations looking to profit from the work of our community" — that's funny when you say it like that, you know :) | |
| Jun 9, 2023 at 18:44 | comment | added | kaya3 | "misuse by organizations looking to profit from the work of our community" ─ I hope this means you won't start charging for API access, because most of us would consider that a misuse by an organisation looking to profit from our work. | |
| Jun 9, 2023 at 18:41 | comment | added | Resistance Is Futile | This question has been sitting on Meta for two days. This answer would look more credible if it was the first one posted. Now it just looks like poor damage control. And even then there is still a question why has this not been communicated before? | |
| Jun 9, 2023 at 18:39 | comment | added | Thomas Owens | Also, it's not "Stack Overflow data". The content here is our data that we have chosen to license. So perhaps the creators and owners of that data should be more involved in making decisions about how their data is distributed. | |
| Jun 9, 2023 at 18:37 | comment | added | Shadow Wizard | So you could say that. You say that now only because you were caught red handed, that's what I think. | |
| Jun 9, 2023 at 18:36 | comment | added | Thomas Owens | Why was this not stated in March, when the decision was made to turn off future data dumps? Or, even better, why was this not discussed to get feedback from impacted members of the community to better assess the risks of turning off the data dump versus not turning off the data dump? | |
| Jun 9, 2023 at 18:34 | comment | added | Andreas condemns Israel | Why are you not doing this openly? This concerns the data provided by the community; why are you not letting the community have a say in how this data is protected against misuse? Why don't you use our knowledge for this purpose? Excluding us from the process only brings anger, negative speculations, and mistrust. We get so much negative by doing it this way, when we could've cooperated, and gotten so much good from it instead. What is your motivation for not including us in the process? Why is there no transparency in this? | |
| Jun 9, 2023 at 18:32 | comment | added | balpha StaffMod | Just as context for casual readers since it may not be obvious, Jody is our CTO. (I am not commenting on the matter at hand, just providing this info.) | |
| Jun 9, 2023 at 18:31 | comment | added | Restore The Data Dumps Again | In order to prevent companies building LLMs from getting our data (which they already have, and have had for years) your plan is to keep all of us from getting our data, am I reading that right? | |
| Jun 9, 2023 at 18:28 | comment | added | Mad Scientist | Are you planning to keep all current uses of the dump, API and SEDE except training of LLMs available? Or will there be more restrictions that are expected to also disallow some other existing uses of the data? | |
| Jun 9, 2023 at 18:23 | history | answered | Jody BaileyStaff | CC BY-SA 4.0 |