Feedback request: Proposed new flagging and moderation workflow for plagiarism

Question

UPDATE

We've built and will be releasing a plagiarism flag on 22 March 2023 (on SO only). For more info, see this MSO post - Plagiarism flag and moderator tooling launching to Stack Overflow

---

A while ago, one of the mods from SO requested help in dealing with plagiarism on SO. While we didn’t have an answer at the time, we have been working with the moderators to better understand this issue so that we could come up with a plan to help tackle this problem.

For more context, please see the linked question and Catija's response to it.

What we want

We need your feedback on our tentative plan for handling plagiarism on the network (described below).

Note: the content below is (mostly) duplicated from the answer. We're not putting it in quote markup, as both I and Catija worked on it, and the quote would be gigantic. Yes, we understand the irony here.

---

Our [tentative] plan

The solutions we’re working on are:

New plagiarism flag type for questions & answers

This new flag type will require flaggers to include a link to the original source and give them a space to add an explanation. This will also allow for these flags to be bucketed into the same category on the flags dashboard, separating them from other custom flags and giving us a better idea of the volume. This will only be enabled on Stack Overflow initially, but other sites will be able to request it if they have need.

[Note: this new flag type would initially be available only on SO, but depending on need, we could potentially roll it out on other sites.]

Allow mods to deny reputation for deleted plagiarized posts

Our system automatically allows the poster to keep reputation earned when a post is over 60 days old and has a score of 3 or higher. In the case of plagiarism, this means that moderators often have to request that CMs ‘disassociate’ older posts to ensure the user isn't earning reputation for content they didn't create. When posts are disassociated, they're no longer connected to the poster's account, meaning that the fact they've had posts deleted as plagiarism disappears. So we need a solution that removes reputation without disassociation. Because we want to allow users to fix their own posts and get them undeleted, this solution would only impact the user's reputation while the post is deleted.

Make it easier for mods to see when users have a history of plagiarism

Once flags have been handled, it becomes difficult for mods to see how common it is that a user has plagiarism flags validated on their posts. In order to simplify future investigation for cases of repeated plagiarism, we want to ensure these handled flags are easy to see. We think that one of the best ways to achieve this is by improving the flagged posts for user moderator page. This page shows all posts a user has that have ever been flagged but it's not sortable and can't be filtered. We think having this page look more like the flags raised by user page will be more useful for moderators in many cases.

Create a post notice explaining why the post was deleted and guiding the author

Often, moderators leave comments linking to the original sources to notify the poster and have an indication for high-reputation users why the post was deleted. We are going to investigate adding a post notice to answers deleted as plagiarism that will notify the poster and give them helpful information about how to properly attribute their post on our network and get attention to have it undeleted if the issues are fixed.

---

These are the initial changes that have been planned. We’ll be monitoring the following over the next couple of months:

• Time that mods are spending on flags related to plagiarism
• % of CM escalations related to plagiarism
• Answers to this question

Feedback

If you have any thoughts on the proposals above, or any concerns about what we are planning, please let us know in the answers below. We want to ensure that we understand your line of thought, so please focus on your concerns or issues and why you are concerned, rather than on specific solutions to the problems.

Answer 1

Multiple Sources

As someone who has spent a fair amount of time looking through posts of frequent plagiarists (as well as coordinating cleanup efforts of plagiarised posts) I have frequently run into situations where answers are plagiarised from multiple sources. It's a fairly common occurrence that someone will plagiarise from the top answers (multiple) on the same question. I've also seen multiple external sources used e.g. an article and an unrelated infographic from a different source.

How does this solution scale to multiple sources?

Is there going to be a built-in way to indicate there are multiple sources, or is it just going to be the case that you add one link to the link field and include the rest in the explanation?

Validation and Character Limits

Links are frequently very long especially if you're trying to link to a specific part of the page and also provide an archive.org link because the page content has changed since the answer was posted.

Additionally, I've seen that users have a tendency to... uh... just put anything in boxes and click buttons. Genuinely, people using the wrong field for the wrong thing is a regular occurrence (on Stack Overflow at least)

I have two questions related to the input fields here.

1. What are the character limits for each field? (please consider that a single off-site link can sometimes be over a hundred characters long and there are commonly multiple links required)
2. Are we going to do any text validation to ensure that the source link field contains something that at a minimum is link-like?

Answer 2

I'm really glad to see how seriously you guys are taking this plagiarism issue and how much you have planned to lessen the weight on mods' shoulders. The backlog of plagiarism flags is large, but so is the backlog of plagiarized posts that aren't yet discovered and flagged. I haven't been flagging many recently, but there really are a lot still to be found.

Let me get just one concern out of the way first:

Make sure that the mod-deletes-as-plagiarism-and-removes-reputation tool clearly indicates somewhere that it was deleted as such

Mods having a tool to bypass the post deletion reputation protection you mentioned is obviously needed even from my POV. I'm sure you guys have processed a lot of dissociation requests as a result of the inundation of plagiarism flags. For the purposes of transparency, I think it best to ensure that there's some recording on the post for a delete-as-plagiarism event that removes reputation in an event where it wouldn't ordinarily remove reputation. I know that the mods wouldn't use this functionality without merit, but I feel like the natural response to mods gaining this type of ability could make folks feel a bit uneasy.

Noting somewhere on the post/in the timeline that rep was removed because it was plagiarism ensures that the tool really is only being used for the purpose of removing truly ill-gotten reputation.

Can we improve the workflow for mods going through and sourcing other posts on a repeat-offending plagiarist?

When the going gets tough, the tough... Say "screw it" and delete everything.

One of the biggest time-sinks for mods with regards to plagiarism is going through each of that user's contributions and seeing if they're possibly plagiarized from somewhere else. In cases where the user has a lot of contributions and over X% of them are plagiarism, mods eventually get to the point where they delete everything that's likely to be plagiarized and prompt the user to give them a list of what's not plagiarized. In even more extreme cases, they escalate it to you guys for further review to determine whether the whole profile should just be deleted.

What I'm wondering is... Can we ease that process at all? I'm sure this particular pain point has been mentioned to you guys before. Has any discussion taken place regarding how to ease this particular workflow? I can't imagine it's an easy one to tackle, but I'm curious where you've landed so far.

Answer 3

Would it be possible to have a built-in moderator tool for assessing whether a post is likely to be plagiarised or not?

I realise this is potentially a big ask, and I have no idea how easy it would be to coordinate or code, but there are such tools already in existence, such as Turnitin and Ithenticate, which are used in academia and other settings. Maybe it could be possible for the SE company to make some agreement with a company like Turnitin, to integrate some of their software with our sites? Wild idea, but I thought it's worth asking and now might be exactly the right time to ask.

My experience with these tools in an academic setting leads me to caution against using them too blindly: for example, it would be a terrible idea to run every new post through such a test and automatically flag it if the similarity percentage is too high. But it could be very helpful for cases when moderators are already investigating potential plagiarism: to get a tentative estimate of similarity to existing sources (I wouldn't give Turnitin results any more credibility than that, just a tentative estimate) in order to figure out whether a post is worth further investigation or not.

In my experience of dealing with plagiarism as an SE mod (and I definitely agree it's one of the most time-consuming mod jobs, although I don't know if my workflow for it is anything like that of the SO mods), the most time-consuming part is checking through all of a user's posts and trying to figure out which ones are plagiarised. If we had a handy button like "get similarity percentage report list for all this user's posts", it could massively reduce the time spent on such cases. More investigation would still be needed to verify the automatically generated reports, but at least we'd have a clearer idea of where to look, which would make our work more efficient.

Answer 4

Here at MSE we have a definition of plagiarism, but individual sites vary on requirements to attribute and length of included text; some simply allow rewriting in your own words, that is sufficient.

• Users are calling me a plagiarist. What do I do?
• Are there highly upvoted, accepted answers that quote just a website?

We should adopt a universal standard for all our sites, so that the rule is enforced consistently.

• One policy at meta.stackoverflow states that doing it is wrong and it should be flagged (maybe it will be deleted) while another Q&A discourages flagging, stating that the post should simply be edited to correct it.

• At MSE there is talk of suspension for plagiarism which isn't mentioned in most other FAQs. At physics.mets there's no mention of suspension, just downvote or ignore it.

• On stats.meta there's only 4 posts with limited advice, while at Operations Research's meta or Quantum Computing 's meta there's no mention of any policy.

• Buried in the Artificial Intelligence's meta is a no tolerance policy from a former staff member. You'd need to search well to locate that gem.

• I've only found one post on Information Security's meta concerning the length of what can be copied. Most other sites are open with respect to length, within reason.

Different academic institutions have slightly different rules, should we adopt one of those; since our sites where plagiarism is most disliked are those where most users have an academic background.

Here are some instances where we differ from the academic standards of some institutions:

• Sometimes an answer is converted to a comment: "Answer appears automatically converted as a comment", and users are encouraged to use comments as a source for their own answers without permission or attribution: "Why do some people answer in comments?" and "Answer or comment: what's the etiquette?" and most blatantly: "What should be done with questions that have been resolved as a comment instead of an answer?":

"We discussed this on meta.serverfault a short while ago and decided that you should Steal comments that answer the question and post them as an answer. You can always tick the Community Wiki box if you're not comfortable earning reputation for someone else's work.".

• Some moderators encourage self-plagiarism:

  "Do we have a policy about self-plagiarism?".

  Yet there's an opposite answer in another location from a user:

  "Is it acceptable to copy your own answer verbatim from an off-site resource?".

There is also this answer which @Makyen had a hand (no complaint, no foul) in:

"What should moderators (or passing editors) do about suspected plagiarized or copyrighted images in posts?":

"... Complaints of copyright violation must be sent to Stack Exchange's designated agent¹ by the copyright holder or their designated agent — third parties are not competent to judge whether an image has been used by permission.

As a user, you may want to inform posters who reuse someone else's image that they may be in violation of copyright. Moderators will not handle copyright infringement complaints. This is outside their attributions. (If moderators handled copyright infringement complaints, that could make them legally responsible for any mistake that they make.)".

As you can see from the FAQ on plagiarism, it doesn't follow what is taught at some institutions, and users follow the FAQ may run afoul of site policy:

"Paste the URL and point out who the author is.".

That won't be enough on many sites, nor does it address self-plagiarism.

Examples of the differences between plagiarism policies:

• Massachusetts institute of Technology

Avoiding Plagiarism - Paraphrasing

• "In writing papers, you will paraphrase more than you will quote.
• For a report or research paper, you may need to gather background information that is important to the paper but not worthy of direct quotation. Indeed, in technical writing direct quotation is rarely used.".

At the Massachusetts Institute of Technology plagiarism is avoided simply by citing the source and paraphrasing.

• Stanford University

What is Plagiarism?

• "For purposes of the Stanford University Honor Code, plagiarism is defined as the use, without giving reasonable and appropriate credit to or acknowledging the author or source, of another person's original work, whether such work is made up of code, formulas, ideas, language, research, strategies, writing or other form(s).
• Moreover, verbatim text from another source must always be put in (or within) quotation marks.”.

At Stanford you are allowed to copy verbatim, but need to use attribution and quotation marks.

• Berkeley

Plagiarism

• "Plagiarism is defined as use of intellectual material produced by another person without acknowledging its source, for example:
• Wholesale copying of passages from works of others into your homework, essay, term paper, or dissertation without acknowledgment.
• Use of the views, opinions, or insights of another without acknowledgment.
• Paraphrasing of another person’s characteristic or original phraseology, metaphor, or other literary device without acknowledgment.".

Berkeley only requires acknowledgement, and doesn't focus on length.

• Harvard Business School

Citing sources and Plagiarism

• Plagiarism is defined as any of the following:

• The use of words or ideas of another with no credit to the original source.
• Paraphrasing or restating the ideas of another without acknowledgment.
• Presenting data or facts that have been borrowed without full citation to the original source. Fraud—a more serious offense—involves the presentation of fabricated data or facts.
• Using a unique term or concept that one has read, without acknowledging its author or source.
• Using the ideas of other students without giving proper credit. These ideas include those obtained through discussion groups, notes, the electronic transfer of notes, and the work of students who have participated in previous class discussions. While verbal plagiarism is more difficult to detect and enforce, the same standards and principles of credit and attribution apply.
• Copying a computer program from another student or any other source, or deriving a program substantially from the work of another, without permission and acknowledgment.".

Self-Plagiarism

• "It is expected that all work submitted for any HBS course has been completed solely for that course. Self-plagiarism includes the practice of submitting identical or very similar material for credit in two separate courses. While the school encourages you to continuously integrate your learning across courses, it is not acceptable to submit the same deliverable (or a very similar deliverable) to more than one course.".

At the Harvard Business School they seem to be the most strict:

• You must give credit even when paraphrasing.
• When using a unique term or concept you must give credit.
• Work of another student (user) requires both permission and acknowledgement.
• Self-plagiarism is to be avoided, you can't keep offering the same answer to different questions, even across different websites. I see a lot of that, without reference to previous usage.

Proposed answer: Stack Exchange should write a clear policy to be enforced for all sites.

People learn in school what is considered plagiarism, but the principles differ depending on the university; a single clear definition for our use is helpful to set out what our standards are.