37

At Sat, 30 Nov 2013 14:51:01 GMT, there was a sudden drop in the close votes review queue size from 98932 to 98314. This is about 600 questions over the course of one minute.

It doesn't seem like a routine cleanup script. If it was, this is a very odd time for it to run. Also, the drop occurred over the span of a minute, so it's highly unlikely there was simply increased review activity. So, I ask: What was its cause?

drop the queue

(yes please don't shoot me I know my freehand circle is blue) (it's actually a dark teal look again) (the source for this script I credit to acheong87 (slightly modified), found through the CV meetup post)

Raw data:

Epoch time  CV Size
1385822821  98932
1385822881  98932
1385822941  98932
1385823001  98932
*** 1385823061  98932
*** 1385823121  98314
    [author's note: the difference between iterations is 60 seconds]
1385823181  98314
1385823241  98314
1385823301  98314
1385823361  98314
  • 10
    The graph lies; it looks like a huge drop but it's more like 6%. – Makoto Dec 1 '13 at 6:47
  • @Makoto I consider a sudden drop of 600 questions in one minute to be significant. – Aza Dec 1 '13 at 6:48
  • I don't disagree; it's just that the graph makes it seem like a lot more. – Makoto Dec 1 '13 at 6:49
  • Ah, okay, yeah. The graph is somewhat exaggerated, to be fair ;] – Aza Dec 1 '13 at 6:50
  • 14
    Not 6%. 0.6%. 6/10 of 1 percent. Expiration of close votes? – Ken White Dec 1 '13 at 7:45
  • 1
    @ken Like I say, it's an awfully strange time for a cleanup script to be running. It's possible, it's just highly unlikely. – Aza Dec 1 '13 at 7:46
  • 4
    I didn't disagree with that; I disagree with the misrepresentation caused by your graph. 0.6% is not as significant as your graphic makes it appear. The scale is vastly exaggerated. A "sudden drop" of 600 votes from ~100K is not the "falling off a cliff a long distance" your image makes it appear to be; it's more like a slight step over a crack in a sidewalk. – Ken White Dec 1 '13 at 7:49
  • 2
    Meight be there where multiple downvoted unanswered automatically deleted after one month. – rekire Dec 1 '13 at 8:21
  • 1
    @rekire once again, this is a weird time for a cleanup script – John Dvorak Dec 1 '13 at 8:22
  • 17
    It's like 600 questions suddenly criend out in terror, and were suddenly silenced. – Kerrek SB Dec 1 '13 at 9:59
  • 1
    Could this be caused by an SQL server upgrade/downgrade? This would make sense if this a partial rollback to a daily backup. – John Dvorak Dec 1 '13 at 11:45
  • 1
    @sth It would be odd for Stack Exchange, because that's usually actually what happens. They set their scripts to run at a specific time. – Aza Dec 1 '13 at 15:41
  • 1
    @Cole Happy now? :P – Aza Dec 2 '13 at 8:20
  • 2
    @Emracool it's better, but there's still no unicorn. – Cole Johnson Dec 2 '13 at 18:36
  • 1
    @Emracool - Haha, no problem. Credit wasn't necessary, but appreciated. I was only curious because I had been running that script for a week, but this being my only computer, I had holes in my data from driving/flying across the country for Thanksgiving. So I was hoping you had knowledge of some external data source. Maybe we can merge our data somehow, because I'm missing that drop! – Andrew Cheong Dec 3 '13 at 22:39
34

The cause was our weekly task that deletes old, abandoned questions. Unlike our daily task, which is scheduled for 3 AM UTC, the weekly task is scheduled for anytime on Saturday (actually, it's 7 days since the last run, on a Saturday, so this will cause the time to shift).

The weekly task deleted around 600 questions that were in the close queue and then the review queue's sync task (every 5 minutes) invalidated those tasks.

This is quite common, actually, as you can see:

ReviewTask DeletionDate     Num Deleted
2013-11-30 14:51:31.173     625
2013-11-23 14:45:18.667     589
2013-11-16 14:35:49.823     437
  • 5
    What happens the Saturday after it deletes at 23:55? Leap Saturday? – animuson Dec 2 '13 at 1:41
  • 5
    So despite the over-stuffed appearance of the CV queue, it may be loosing 1600+ questions per month automatically? – Bill Woodger Dec 2 '13 at 1:42
  • 2
    @animuson I have faith that Kevin Montrose coded it to handle that case :) – Jarrod Dixon Dec 2 '13 at 1:58
  • @BillWoodger looks like it, yes. – Jarrod Dixon Dec 2 '13 at 1:59
  • Huh, interesting. How does the server determine the time of execution? I'm curious, as those times seem strangely spread out. – Aza Dec 2 '13 at 4:41
  • 2
    @Emracool without looking at the code, I'm assuming the service that calls the weekly task will start the next call 7 days after the end of the previous call's completion. As that task is long running (a few minutes, at least), you can see where the starting time will increase each week. Add to that time the 5 minute window of the review queue sync task and you could have 10 minutes of slippage each week. – Jarrod Dixon Dec 2 '13 at 4:47
  • @Jarrod Well cool, that makes a lot of sense. Sweet, thanks! – Aza Dec 2 '13 at 4:49
  • 6
    @JarrodDixon is there a documented list of background tasks/job/processes that are running in the network, like this one? If not, do you think it's legit to try create such list from the information that already exist in Meta, and devs (not just you personally) might add to it sometimes? – ShaWiz Dec 2 '13 at 9:26
  • @ShaWizDowArd that'd be so legit ;) – Cole Johnson Dec 2 '13 at 18:38
  • Anyone else end up here because you noticed the queue dropped from ~105k yesterday (I think) to 14.2k today? Somehow it seems like over 90k items disappeared from the queue... – Okuma.Scott Feb 27 '14 at 19:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .