TL;DR, from Google Labs’s generative AI: The new policy was announced without moderator or community input, lacks data to support its claims, and goes against the wishes of the community. Moderators are no longer able to use their judgment to remove harmful content, and the policy is likely to increase the spread of misinformation.
The new network policy regarding AI generated content is problematic. It was rolled out in one of the worst possible ways, goes against the wishes of the community, and is not supported by any available data.
On May 29, 2023, the Community Management team posted an announcement on the private “Stack Moderators” team. They also pushed out a notification to all moderators directing them to read this announcement. To the best of my knowledge, this was the first time that any of the community elected moderators were made aware that anyone at the company had any concerns about how algorithmically-generated content was being moderated or that any research into detecting and handling such content was ongoing.
Simply by making such a policy announcement, the company did not follow the policy on making policies. There was no substantial preview or review period - the policy was declared to be created and moderators were only given a short “heads up” period prior to it being posted publicly. Questions and concerns raised were not adequately answered or addressed prior to the policy being made public.
In addition, the day that the announcement was posted for moderator “heads up” was a holiday in the United States, Canada, the UK, and other European countries. Given that it was announced on a holiday and was then posted a day later to the public, it is highly likely that not all moderators were able to see and react to the policy edict before it was made public. Some moderators may still be unaware.
The policy, as presented to moderators, had two main components:
- Stop using detectors when investigating reports of algorithmically-generated content.
- Stop suspending users only for posting algorithmically-generated content.
To support the policy, two claims were made:
- Detectors do not work as advertised and produce unreliable results.
- Detectors are biased against non-native English speakers.
Neither claim was supported by data made available to moderators or to the broader community when the policy was made public.
With respect to the first claim that detectors were unreliable, no information was provided about when the detectors were tested (i.e., what version of the detectors were tested), what communities the test data came from, or how many posts were sampled. In addition, there were no examples given for posts that appeared to be miscategorized.
Data to support the second claim that detectors are biased against non-native English speakers, would require some level of PII, so couldn’t be shared (even with moderators), but I would have expected aggregate data, such as the number of users from each country, the participation rate from each country, and the number of suspensions over time. It’s unclear what the baseline suspension and account deletion/destruction rates for various countries are and how suspensions related to algorithmically-generated content compare to other moderation activities.
The claims do raise questions. However, in the absence of clearly specified methodologies and making data available where possible, it is inappropriate to be dictating policies that go against the desires of the community. Many communities on the network have developed policies to address posting algorithmically-generated content, some with clarifications regarding how to attribute generated content and others with a prohibition on the use of generated content. On Software Engineering and Project Management, where I’m an elected moderator, these policies have overwhelmingly positive community support. Nevertheless, in cases where the community has zero tolerance for algorithmically-generated content, the ability to take the appropriate action that the community has decided to be appropriate has been eliminated by the new policies.
I believe that not taking strong, decisive action against algorithmically-generated content is the wrong thing to do. Visitors to the Stack Exchange network come here expecting high-quality answers that draw on human knowledge and experience, with valid citations and references where appropriate. When someone plagiarizes the output of a generator, that person is misrepresenting their own knowledge and experience. Considering that these generators do not have the concept of facts and truth, and only a subject matter expert (often not the asker) can verify the claims, it would be better to remove potentially harmful content. Since the data on Stack Exchange is public, I would expect that future algorithms are trained on the data. This compounds the harm by feeding incorrect information into search engines and future algorithms which would spread the harmful information. The harm of an erroneous suspension - which could be reduced or eliminated by a moderator - is far less serious than the potential harm of supporting the proliferation of incorrect information and training future models on algorithmic outputs.
Moderators are not prevented from acting, but we’re extremely hampered. We are no longer allowed to use our personal judgment or the only tools in the toolbox to curb content that the communities - the people who elected us and have offered their inputs into community policies - do not want to see. Although we can deal with certain classes of content, our choices are to either let some algorithmically-generated content be posted on the network or to go against this new policy.
I will also say that I have, to date, suspended 7 accounts for posting algorithmically-generated content. Of these accounts, 6 of them were newly-created accounts that appear to have been created just to post such content. None of these people reached out to challenge their suspension. I did not hear directly from the Community Management team that any of these suspensions were escalated to them or deemed inappropriate. It is unclear what may have happened if they were allowed to post convincing content that allowed them to increase their reputation and gain access to other functionality.
I do think that we should have discussions around both claims and reach a baseline policy for the network. But we should have the underlying data (to the extent possible) and have discussions on the research and validity of the claims before making policies. And then, when policies are established, they must follow the agreements between the company and the elected moderators to go through working groups, moderator review, and finally appropriate public discussion and announcement.