UPDATE
We've built and will be releasing a plagiarism flag on 22 March 2023 (on SO only). For more info, see this MSO post - Plagiarism flag and moderator tooling launching to Stack Overflow
We can see this is a high priority issue for you all. Back in September, Bella asked the moderators to escalate situations where they found users who had five or more instances of plagiarism for us to have a better understanding of the scope of the issue that mods and users face when trying to deal with possible plagiarism.
After a few days the situation was clear and she reconnected with the moderators. I think her response was, “Wow. Are y’all ok?!” No. She and I know you are not.
We got together with jkm (one of our new PMs) and the Community Enablement team she's managing, who are working on improving tooling for moderators and CMs since this concern would fall to them to solve. Together, we wanted more information about two things:
- Volume of flags
- Time to resolve
One thing that she recognizes is that there's a huge backlog of asks as tools for moderators haven't been touched much over the last few years - as such, she wants to be cautious about balancing addressing emergent pain points with long-standing needs for improvements.
Our investigation process
jkm worked with Nicolas to identify some keywords like plagiar and without attribution, to categorize which flags are likely related to plagiarism. They did this to better understand the scale of the problem. Meanwhile, Bella and I chatted with the moderators to understand what problems this influx of flags was causing for them and what would make handling the flags more efficient.
Our findings
The TL;DR is that the general volume is low, but flags related to plagiarism have been increasing and taking much more time to resolve - sometimes hours per flag, as many cases of plagiarism are not one-offs, so moderators are often investigating all of a user's answer history.
We realized there's two big issues that we need to find solutions for, eventually -
- lots of posters may not be aware of what plagiarism is and the Stack Exchange policies around it.
- there is 15 years of content that we need to review for possible instances of plagiarism.
These two aspects will require different solutions. Because the concerns y'all have brought to us are centered around your ability to easily handle the flags you're getting, we feel like starting with the second issue is more pressing.
The second issue can be split into two areas and this is where we are focusing:
- The high volume of custom plagiarism flags are making it harder for mods to find other potentially important flags in the "moderator attention" flags category.
- Mods often have to ask CMs for support in handling these flags as they don’t have the tools they need to investigate or resolve flags related to plagiarism..
Our [tentative] plan
The solutions we’re working on are:
New plagiarism flag type for questions & answers
This will require flaggers to include a link to the original source and give them a space to add an explanation. This will also allow for these flags to be bucketed into the same category on the flags dashboard, separating them from other custom flags and giving us a better idea of the volume. This will only be enabled on Stack Overflow initially but other sites will be able to request it if they have need.
Allow moderators to deny reputation for deleted plagiarized posts
Our system automatically allows the poster to keep reputation earned when a post is over 60 days old and has a score of 3 or higher. In the case of plagiarism, this means that moderators often have to request that CMs ‘disassociate’ older posts to ensure the user isn't earning reputation for content they didn't create. When posts are disassociated, they're no longer connected to the poster's account, meaning that the fact they've had posts deleted as plagiarism disappears. So we need a solution that removes reputation without disassociation. Because we want to allow users to fix their own posts and get them undeleted, this solution would only impact the user's reputation while the post is deleted.
Make it easier for moderators to see when users have a history of plagiarism
Once flags have been handled, it becomes difficult for mods to see how common it is that a user has plagiarism flags validated on their posts. In order to simplify future investigation for cases of repeated plagiarism, we want to ensure these handled flags are easy to see. We think that one of the best ways to achieve this is by improving the flagged posts for user moderator page. This page shows all posts a user has that have ever been flagged but it's not sortable and can't be filtered. We think having this page look more like the flags raised by user page will be more useful for moderators in many cases.
Create a post notice explaining why the post was deleted and guiding the author
Often, moderators leave comments linking to the original sources to notify the poster and have an indication for high-reputation users why the post was deleted. We are going to investigate adding a post notice to answers deleted as plagiarism that will notify the poster and give them helpful information about how to properly attribute their post on our network and get attention to have it undeleted if the issues are fixed.
These are the initial changes that have been planned, and we’ll be monitoring the following over the next two months:
- Time that moderators are spending on flags related to plagiarism
- % of CM escalations related to plagiarism
- Answers to this linked meta question
Feedback
If you have any thoughts or concerns about what we are planning, please let us know in the answers to this linked question. As far as the timeline goes, while we're working on design concepts for this (as you can see above), we will be picking this project up as Winter Bash wraps up.
This isn’t a perfect or holistic solution, in other words we don’t expect to fully solve the plagiarism problem with these solutions, but we do hope it makes a noticeable difference in how to know which flags are related to plagiarism, and how long it takes to investigate and resolve these types of flags. We think these changes will be iterative, meaning if we are not seeing meaningful movement in either category, there’s always room to further investigate. Otherwise, we’ll be excited to tackle some of the other areas like suspicious voting, sock puppets, etc. that we know have been asked for consistently through moderator surveys, Meta posts, and Mod working groups.
