37

At Sat, 30 Nov 2013 14:51:01 GMT, there was a sudden drop in the close votes review queue size from 98932 to 98314. This is about 600 questions over the course of one minute.

It doesn't seem like a routine cleanup script. If it was, this is a very odd time for it to run. Also, the drop occurred over the span of a minute, so it's highly unlikely there was simply increased review activity. So, I ask: What was its cause?

drop the queue

(yes please don't shoot me I know my freehand circle is blue) (it's actually a dark teal look again) (the source for this script I credit to acheong87 (slightly modified), found through the CV meetup post)

Raw data:

Epoch time  CV Size
1385822821  98932
1385822881  98932
1385822941  98932
1385823001  98932
*** 1385823061  98932
*** 1385823121  98314
    [author's note: the difference between iterations is 60 seconds]
1385823181  98314
1385823241  98314
1385823301  98314
1385823361  98314
Improve this question
35
  • 10
    The graph lies; it looks like a huge drop but it's more like 6%.
    – Makoto
    Commented Dec 1, 2013 at 6:47
  • @Makoto I consider a sudden drop of 600 questions in one minute to be significant.
    – user206222
    Commented Dec 1, 2013 at 6:48
  • I don't disagree; it's just that the graph makes it seem like a lot more.
    – Makoto
    Commented Dec 1, 2013 at 6:49
  • Ah, okay, yeah. The graph is somewhat exaggerated, to be fair ;]
    – user206222
    Commented Dec 1, 2013 at 6:50
  • 14
    Not 6%. 0.6%. 6/10 of 1 percent. Expiration of close votes?
    – Ken White
    Commented Dec 1, 2013 at 7:45
  • 1
    @ken Like I say, it's an awfully strange time for a cleanup script to be running. It's possible, it's just highly unlikely.
    – user206222
    Commented Dec 1, 2013 at 7:46
  • 4
    I didn't disagree with that; I disagree with the misrepresentation caused by your graph. 0.6% is not as significant as your graphic makes it appear. The scale is vastly exaggerated. A "sudden drop" of 600 votes from ~100K is not the "falling off a cliff a long distance" your image makes it appear to be; it's more like a slight step over a crack in a sidewalk.
    – Ken White
    Commented Dec 1, 2013 at 7:49
  • 2
    Meight be there where multiple downvoted unanswered automatically deleted after one month.
    – rekire
    Commented Dec 1, 2013 at 8:21
  • 1
    @rekire once again, this is a weird time for a cleanup script
    – John Dvorak
    Commented Dec 1, 2013 at 8:22
  • 17
    It's like 600 questions suddenly criend out in terror, and were suddenly silenced.
    – Kerrek SB
    Commented Dec 1, 2013 at 9:59
  • 1
    Could this be caused by an SQL server upgrade/downgrade? This would make sense if this a partial rollback to a daily backup.
    – John Dvorak
    Commented Dec 1, 2013 at 11:45
  • 1
    @sth It would be odd for Stack Exchange, because that's usually actually what happens. They set their scripts to run at a specific time.
    – user206222
    Commented Dec 1, 2013 at 15:41
  • 1
    @Cole Happy now? :P
    – user206222
    Commented Dec 2, 2013 at 8:20
  • 2
    @Emracool it's better, but there's still no unicorn.
    – Cole Tobin
    Commented Dec 2, 2013 at 18:36
  • 1
    @Emracool - Haha, no problem. Credit wasn't necessary, but appreciated. I was only curious because I had been running that script for a week, but this being my only computer, I had holes in my data from driving/flying across the country for Thanksgiving. So I was hoping you had knowledge of some external data source. Maybe we can merge our data somehow, because I'm missing that drop!
    – Andrew Cheong
    Commented Dec 3, 2013 at 22:39

1 Answer 1

Reset to default
34

The cause was our weekly task that deletes old, abandoned questions. Unlike our daily task, which is scheduled for 3 AM UTC, the weekly task is scheduled for anytime on Saturday (actually, it's 7 days since the last run, on a Saturday, so this will cause the time to shift).

The weekly task deleted around 600 questions that were in the close queue and then the review queue's sync task (every 5 minutes) invalidated those tasks.

This is quite common, actually, as you can see:

ReviewTask DeletionDate     Num Deleted
2013-11-30 14:51:31.173     625
2013-11-23 14:45:18.667     589
2013-11-16 14:35:49.823     437
Improve this answer
10
  • 6
    What happens the Saturday after it deletes at 23:55? Leap Saturday?
    – animuson StaffMod
    Commented Dec 2, 2013 at 1:41
  • 5
    So despite the over-stuffed appearance of the CV queue, it may be loosing 1600+ questions per month automatically?
    – Bill Woodger
    Commented Dec 2, 2013 at 1:42
  • 2
    @animuson I have faith that Kevin Montrose coded it to handle that case :)
    – Jarrod Dixon
    Commented Dec 2, 2013 at 1:58
  • @BillWoodger looks like it, yes.
    – Jarrod Dixon
    Commented Dec 2, 2013 at 1:59
  • Huh, interesting. How does the server determine the time of execution? I'm curious, as those times seem strangely spread out.
    – user206222
    Commented Dec 2, 2013 at 4:41
  • 2
    @Emracool without looking at the code, I'm assuming the service that calls the weekly task will start the next call 7 days after the end of the previous call's completion. As that task is long running (a few minutes, at least), you can see where the starting time will increase each week. Add to that time the 5 minute window of the review queue sync task and you could have 10 minutes of slippage each week.
    – Jarrod Dixon
    Commented Dec 2, 2013 at 4:47
  • @Jarrod Well cool, that makes a lot of sense. Sweet, thanks!
    – user206222
    Commented Dec 2, 2013 at 4:49
  • 7
    @JarrodDixon is there a documented list of background tasks/job/processes that are running in the network, like this one? If not, do you think it's legit to try create such list from the information that already exist in Meta, and devs (not just you personally) might add to it sometimes?
    – Shadow Wizard
    Commented Dec 2, 2013 at 9:26
  • @ShaWizDowArd that'd be so legit ;)
    – Cole Tobin
    Commented Dec 2, 2013 at 18:38
  • Anyone else end up here because you noticed the queue dropped from ~105k yesterday (I think) to 14.2k today? Somehow it seems like over 90k items disappeared from the queue...
    – Scott Solmer
    Commented Feb 27, 2014 at 19:08

You must log in to answer this question.