What limits should we place on automated voting?

Question

This has come up a few times now in moderator circles, but with the exception of Andy's Robot there hasn't been a lot of public discussion that's applicable. That feels kinda negligent to me, so... Here goes.

The setting

For the past five years, it's been trivially easy to script requests to the Stack Exchange API that trigger actual changes to these sites. Kevin Montrose laid out some ground rules for this at the time...

Stack Exchange is very protective of the quality of content on its sites, and will deal harshly with harmful behavior. While it's impossible to list all forbidden behavior, a good rule of thumb is "if a user would be flagged or banned for doing something, your app will as well".

Some examples of write abuse that will be punished:

• Spam, of any sort.
• "Auto-commenting" based on post heuristics.
• Abusively "following and pestering" another user.
• Automating flagging or closing based on dumb heuristics.

Now, I think those are pretty good rules - and they certainly make it easier on the moderators; if you see a user doing something bad, suspend them - no need to worry about whether they're doing it via the API, via a userscript, or by manually clicking buttons.

But there's a wee problem when it comes to votes... Moderators and other members of the site can't see them at all.

These sites make it pretty easy to earn voting privileges, and quite generous with their daily allowance of votes once those privileges have been earned. Most folks don't vote nearly as often as they could... But an automated script certainly can. We have some fairly well-established rules about voting when it comes to targeting other users, but when it comes to heuristic-based voting... There's been relatively little consistency or discussion on what is appropriate.

The crux

There's currently a user using some sort of script to down-vote questions based on some sort of heuristics: old, unanswered, no votes... Maybe something else? Not sure. The end result of this is often deletion - the questions are either deleted by their authors, or by the system.

Now... It's debatable whether this is doing any harm; these are effectively abandoned questions, and there's a compelling argument that we should be deleting more of these anyway.

But, there are a LOT of questions being downvoted this way. 3399 questions over just the past 7 days, across several dozen sites - and over a thousand of those questions are now deleted. As a result, we've been getting questions and complaints from moderators asking us to look into this...

The question

If we were going to start consistently restricting automated voting - or punishing it - what should the criteria be for doing so? And, why?

One requirement: restrictions must be applicable to anyone voting via any normal mechanism - on-site, via an app, via the API, etc. In other words, "no one should be able to vote more than 5 times per day via the API" doesn't work, but you could argue "no one should be able to vote more than 5 times per day period."

Answer 1

What distinguishes the well-behaved robots that help moderate the sites is that they have some kind of human oversight. And they perform actual moderation actions like flagging.

Voting is not moderation alone, it was supposed to be a person judging the quality of a post. It has moderation side-effects, but it's not a pure moderation action.

Voting should not be done by bots, only by humans. As you already said, there is no way for mods to track this at all, there can't be any oversight. Voting should be about real persons evaluating posts, I would disallow voting with automated bots entirely. It's of course rather difficult to identify the bots, but I assume SE can do this if they're very active like in this case.

It might also make sense to attack this from the other direction as well. I've always been very uncomfortable with users using downvotes only for the purpose of triggering the automatic deletion. This mechanism drastically amplifies the power of users that couldn't delete or close questions at all under regular circumstances. A single downvote is as powerful as a diamond moderator or several 20k users voting to delete in these cases. Late downvotes are also almost entirely invisible to other users (unlike downvotes on new questions), so other users can't counteract this.

The automated deletion should maybe ignore votes cast after a certain age of the question, and not work retroactively. Either the question met the specific threshold after 7, 30 or 365 days and gets deleted automatically, or it didn't meet it and never gets automatically deleted.

This kind of change will affect the number of automatically deleted questions, but that probably can be balanced independently.

Answer 2

I don't think additional limits are needed to solve this problem.

Instead, I think altering the roomba so that this kind of voting can be seen before the roomba takes action on these posts would be more effective.

• The roomba should bypass any posts that have received a vote or whose answers have received a vote (up or down) in the past 7 days.
• Add a list of questions that fit this criteria to the tools page.

This would give the community a full week to take action before deletion occurs due to a recently cast vote without interfering with the normal usage of votes by users or bots alike.

If it isn't noticed within 7 days it probably isn't a post worth keeping anyway, but it might help if there were a list on the tools page that displays posts that are in that 7 day limbo period to help with detecting abuse.

Answer 3

I'm generally wary of using downvotes to trigger moderation actions in an automated way. While I do believe that automatic moderation and heuristic based flagging and content actions are appropriate, I don't know if downvotes fall into that. They are mostly inscrutable (only CMs can see them) and due to the nature of the system can be used in a way that massively amplifies the power a single user wields.

Now, downvoting, even targeted downvoting, isn't the same as sockpuppetry, but it does sometimes have the same effect. If I were to identify all questions that I could unilaterally delete by downvoting them and then made a script that does this say, a couple of hours before the roomba runs, I've just condemned a large chunk of questions to unilateral and inscrutable deletion.

We had a similar discussion around manually downvoting answers to get their questions roomba'd. This practice was rejected for obvious reasons, but some of the reasons why still apply here:

A question's answers can sit at +7 for years, but as soon as they get downvoted to <= 0 the question immediately qualifies for auto-deletion under one or more criteria. No need for 10K privileges, no rationing your delete votes, no concerns about having your name attached to the moderation you're doing...

(source)

This is talking about downvoting answers, but it's the same concept. Using downvoting in a way that effectively uses it as delete voting, and automating that process, is skirting the transparency and ruleset that we have set out for deletions. It should be stopped or at the very least regulated and be made transparent, like other automatic moderation projects have to be.

Answer 4

If such voting is harmful, then perhaps just limiting downvotes on posts older than 90 days to five per day would be palliative enough. That's old enough to not interfere with most of what the various "close-vote review" chatrooms do, and newer posts are more likely to be rescued from shenanigans.

Another tack might be to have the Roomba factor in anonymous feedback.

---

But we need more data about the targeted questions:

1. How many views?
2. Were the owners "drive by"?
3. Was there any anonymous (positive) feedback?
4. What percent of similar questions are "salvaged" after many years/months? That is, eventually get upvotes and/or positive scored (late) answers? Any get late accepted answers?

Without some more details, we can't be certain if the individual, or behavior, is doing more harm or more good.

Personally, I'm conflicted:

1. I'm all about the "Long Tail", so content should be left up if it's not harmful (sucking in invalid search hits because of sloppy or wrong terminology, for example).
2. On the other hand, I quite often manually downvote old posts so that they will get Roomba'd and count it a job well done. I could live with a 5 per day limit on such posts, no problem.

Answer 5

We should be voting for the post - and judging it by its qualities as a post. It isn't any different from voting on a specific user, or tag for whichever reason. It's simply more spread out because it's trying to take advantage of a mechanism we have to help in curation. It's not an inherent flaw in the API as it's intended - it's an inherent flaw in reasoning that it should be used like that.

Now, in this situation - where the issue is someone trying to trip the roomba by knocking over questions that are on the edge across the network - there's nothing that a mod really can do. It's no different from when we notice voting fraud, and need a CM to help (because it is).

That it's automated, and done over the API is actually secondary. I'm assuming that official (and unofficial) apps use the API - and using it to read and vote on posts is useful.

Writing some automated thing that tries to "clean up" the site on poorly thought out criteria like "it's close to being roombaed" is not, especially as people get affected.

Logically - it feels like the problem is less with the voting system - a normal human could use up their votes on one site in a day, but rather with someone going "ooh shiny! What happens when I pour nitroglycerin into my diesel engine" and using a poorly thought of "helpful" tool across many sites.

If we're going to put a break on downvotes - there's a few things we need to consider. A user who can downvote should know better - they'd need 125 reputation (though you could squeak by with 20 rep and the association bonus).

It might be useful to "borrow" the network wide question throttling mechanism - that you can only downvote X times in Y minutes, on a site you have less than 125 rep earned. Something reasonable like 3-4 downvotes an hour would slow down the casual bot and mildly annoy a human user.

At the end of the day though, someone with the reputation and the skills should know better than to do something like that.

Answer 6

The problem I'm seeing here is that voting patterns like this can be harmful, but can also be very useful. Limiting voting to less than normal users get might help in some cases, but could also be harmful in other cases where the script is performing a legitimate action.

I do agree with what Kevin said: if a normal user would be punished for doing something, they should be punished equally if their script does it. But the problem is, the system usually isn't smart enough to know if a script is doing something bad or not. Automatic suspension is a bad idea; it's better to have other users check it manually.

So I propose to make the Community user raise automatic flags for suspicious behavior performed by a script. This might include consistently downvoting old questions to trigger roomba, leaving possibly harmful comments, etc. This would allow moderators to manually check for abuse, while at the same time not preventing users from doing things that look harmful at first, but are actually helpful to the site.

Answer 7

For me your question leaves an option open; is it a kind of abuse?

On smaller sites, not many users actually upvote questions, and as such a lot of questions remain stuck at zero. It would be easy to abuse this characteristic in order for content to be removed.

If we answer yes, the system should remove all the downvotes from the user who cast them unilaterally and roomba should undo the damage done. I would not restrict for some special case.

I'd suggest a tool/report for moderators, built on the same principle that we have for the detection of serial downvoting. For the undoing of the damage, I would leave that to the CM, a person who can actually see the votes, to see whether there is a pattern or not.

My proposal would make it possible for a moderator to tell if a user is abusing the system, similar to how revenge downvoting is able to show us on a user, but it would leave the burden to see if it was an abuse or not to the CM when a new user got caught by the detection.

For the detection part, I would leave it to your discretion for an actual threshold, but any number over a certain percentage should cast an alarm.

My idea is to punish user that abuse, but not restricting the whole community because of some(s), exactly like upvote.

Answer 8

I agree with Mad Scientist that a single downvote can effectively become a unilateral delete vote in many cases:

I've always been very uncomfortable with users using downvotes only for the purpose of triggering the automatic deletion. This mechanism drastically amplifies the power of users that couldn't delete or close questions at all under regular circumstances. A single downvote is as powerful as a diamond moderator or several 20k users voting to delete in these cases.

Also, they raise a very good point:

Late downvotes are also almost entirely invisible to other users (unlike downvotes on new questions), so other users can't counteract this.

As they suggested, it might be a good idea for the RemoveDeadQuestions check to only run if the question is less than a certain minimum age, to prevent very late downvotes from unilaterally deleting the question. However, given the current state of the deletion criterion (it doesn't run if the question is less than 30 days old, and many of the questions being downvoted by said bot are anyway eligible for automatic deletion under RemoveAbandonedQuestions once they turn 365 days old), I don't know what specific threshold between 30 and 365 days should be set as the arbitrary "ignore downvotes" deadline.

Additionally, their solution doesn't address the central problem: that in case no one votes on a question or consensus as to whether the question is useful and clear is evenly split, a certain user who on their own downvotes for any reason can effectively unilaterally decide the question's fate. This is especially amplified if the voting is very controversial, such as +50/-51. In my opinion, the fate of questions should generally be decided by consensus, not by majority vote.

For this reason, I propose the following changes to the Roomba system:

• Keep the current RemoveDeadQuestions deletion criterion as is, except make it only work on questions with a score of -2, -3, or some other lower threshold. This will increase the chance that any questions automatically deleted under this criterion will have an overall consensus for deletion, and quell the "damage" being done by any sort of bot (if, say, the threshold is -3, it will already have a score of -2 before it can even be unilaterally pushed to deletion; effectively, this means that for a zero-score question, a consensus of three users is required rather than one user with a unilateral vote).

  • Also, to ensure completeness, modify RemoveAbandonedQuestions so that the minimum deletion threshold is one more than the above required score for RemoveDeadQuestions (e.g. if that's set to -3, set this to between 0 and -2, or 1 and -2 in case the owner's account is deleted).

• Raise an automatic moderator flag (e.g. "dead question") in case the question meets the current deletion criteria for RemoveDeadQuestions. In such flags, clearly indicate the time of the last downvote. Moderators can decide if the question should be deleted, if it should be immune to deletion, or if the standard criterion of [proposed threshold, used -3 in the above example] should continue to apply (effectively skip it).

  • We can also potentially make this a community queue, and allow users with 10k+ or 20k+ rep to review potentially dead questions and vote to either delete them or immunize them.

• Raise another automatic moderator flag if a certain user causes an excessively high number of posts to have a "dead question" flag raised. Clearly enumerate the number of posts for which such a flag was generated, as well as the number of [threshold + 1]-voted posts they downvoted and caused automatic deletion of.

The first bullet will help quell the problem of a single downvote becoming a unilateral delete vote in many cases, the second will ensure questions are being adequately curated per the spirit of the deletion criteria, and the third will ensure that there are no "bots" or users who tactically downvote posts just to trigger automatic deletion.

Answer 9

Suggestions:

If it is not possible to block automatic voting, then:

• Limit the number of votes per day on borderline questions to automatic deletion scripts (i.e., all criteria for deletion are met, except for one downvote). It could be a threshold of 5, 10 or 15 votes. Currently, we have the 10 "Q-votes" and 30 "QA-votes" rule, summing up 40 daily votes per user. This would be a third limit added to the algorithm (within the 40 votes, up to 'x' on borderline questions, "BQ-votes").

Additionally:

• Run the RemoveAbandonedClosed script closer to 0 UTC time. Currently, it is running near at 3 UTC time. This means there is a 3 hour window which users can burn 40 downvotes (on questions and/or answers) on borderline questions to the mentioned script, and after such posts are deleted, votes are refunded meaning a single user can delete up to 80 questions in a single day with such approach (see here; it works).

• Tweak the RemoveDeadQuestions to make it less aggressive. Add a parameter based on the user's activity:

  • user has not logged in the past 'n' days.

  This is mainly to prevent new questions being deleted while OP is there to respond to comments and make edits.

Recent experience 'using' the automatic deletion scripts

In the past 3.5 months (2019) I have used the automatic deletion scripts to manually nuke low quality questions with downvotes; but it went either ways, have upvoted, editd, answered and voted to reopen valid questions as well. I've done this by finding borderline questions in Stack Exchange Data Explorer.

In GIS SE, the number of Qs which were deleted, more than doubled in this period (I am sure I had significant participation on this; results here). Here are my findings after this experiment:

• Most of the borderline questions to the automatic deletion scripts were read/reviewed by other users and failed to collect upvotes (positive score). When I cast a downvote I did not feel I was the only one deleting that question, but also all the people who reviewed and read it before me, and did not upvote as well.
• RemoveDeadQuestions script: upon reviewing such type of questions I found a reasonable amount (more than 20%) of false positives (normal questions hanging there with zero score). I realized when reviewing the same group of questions, except that OP was also inactive (has not logged in) by 'n' days, that the number of false positives drop drastically. Basically, OP was not there to reply to comments, improve with edits, or responding to answers (query is here).
• RemoveAbandonedClosed script: I have found most of them (both question and answers) were indeed of poor quality, either because the posts were originally organically poor (for example, link only answers) and/or because the guidelines have changed (for example, shopping list questions, opinion-based, etc.) (the respective query is here). One thing I noticed was that many of the questions which had answers were neither upvoted (zero score with no up or down votes) nor edited by those who answered it, while there were opportunities to improve the question.

I engaged studying the automatic deletion scripts mainly because of spending considerable time reading low quality (besides off-topic or outdated) posts while try searching content and curating the site.

Answer 10

This must be entirely prohibited.

I am shocked that this is even being discussed. I have had various plans for automated voting to help push some of my values for a long time, but have never implemented any of them because it seemed obvious to me that non-human voting would be an extraordinary rules violation and swiftly punished.

The automated voting we have on comments is already starting to obliterate some of the remaining shreds of humanity on this site. Expanding that to all content will harm the reputation of our communities and the legitimacy of our moderation practices even further.

If y'all decide that automated voting is permissible, you're going to immediately see several bots casting the maximum number of votes, using criteria that you might not agree with, if you can even tell what they are. Maybe we'll end up with some bots that just cast votes at random, if that's allowed. Or voting down posts whose character lengths evenly divide the author's user ID. Maybe we can have "guess what my bot's doing!" challenges on meta.

It sounds pretty dsytopian to me.