45

I noticed that Page 14 of "newest questions" seems to always show only half of the questions the other pages do.

So, just out of curiosity … What's up with that, SO devs?

| improve this question | |
  • 2
    Do you have the "Hide ignored tags" option selected on your profile (the "perfs" tab)? – ChrisF♦ Oct 17 '11 at 12:00
  • 1
    @ChrisF Nope. And I have no ignored tags. – vzwick Oct 17 '11 at 12:01
  • I can reproduce this, it also happens on page 7 with 30 items per page, always after 200 questions. – Mad Scientist Oct 17 '11 at 12:16
  • 5
    Reproduced by simply clicking the link: Chrome 14.0.835.202 shows five results. – Grant Thomas Oct 17 '11 at 12:23
  • 1
    @Mr.Disappointment the link was originally - http://stackoverflow.com/questions?page=14&sort=newest - so, if it depends on the page size, wouldn't necessarily show the problem – ChrisF♦ Oct 17 '11 at 12:25
  • 3
    @ChrisF Indeed, my setting of a page size of 50 is not affected; adding pagesize=15 results in 5 results, adding pagesize=30 results in 10 results. – Grant Thomas Oct 17 '11 at 12:47
  • 3
    It must be an Easter egg. :-) – kiamlaluno Oct 17 '11 at 14:16
  • 2
    @kiamlaluno i would have assumed just that if it had been page 42 :p – vzwick Oct 17 '11 at 14:17
  • Cache issue, as after navigating to different page and coming back it's fine.. – Shadow Wizard Wearing Mask Oct 17 '11 at 14:30
  • 1
    Hmm... it behave really strange. First click - five results. Closing window, trying again - five results. Closing window, opening different page, closing and opening with the original link - 15 results as expected, and it keep being OK afterwards even in different browsers - so it's server cache issue.. – Shadow Wizard Wearing Mask Oct 17 '11 at 14:37
  • 1
    @ShadowWizard Server cache issue tied to your user account? 'Cause I bet there's dozens of users hitting that page on every given second. – vzwick Oct 17 '11 at 14:39
  • 15
    SE Dev 1 to Dev 2: Hey, they finally found that page 14 joke you coded 3 years ago! – NGLN Oct 17 '11 at 15:18
  • 4
    We're aware of this, some recent tag engine internal changes are likely in play here. Marc's taking a look soon as he gets a chance. – Nick Craver♦ Oct 17 '11 at 16:37
  • 1
    @NickCraver Would be awesome if an in-depth explanation followed up here ;) – vzwick Oct 17 '11 at 17:03
  • 1
    @vzw In depth explanation: If ( page==14 ) { ... } – Dr. belisarius Oct 18 '11 at 4:44
31

So, just out of curiosity … What's up with that, SO devs?

Well, back in the day we used to use direct queries to SQL Server to build up the various question sort pages.

This worked fine for a while, and Smurfs were happy. However, slowly, over time we discovered that for our specific use SQL is just not fast enough.

I posted a question on theoretical CS that touches on the issue: Data structure that allow efficient tag based lookups

Fast forward a few months, and we reimplemented a large amount of functionality we used to depend on SQL Server for, using in-memory structures.

These days, when you click on any tag or combination of tags or click on the top level Newest, Faq, Active, Votes or Featured tabs you hit the TAG ENGINE OF DOOM

This in-memory structure holds all of the information needed to provide the various question sorts and filters in lightning speed.

The Tag Engine contains many optimisations, one such optimisation is the ...

Block size in the query cache

When the tag engine is asked for all the questions on page 14 with page size 15, we cache the results so subsequent queries do not need to traverse the tag engine.

But we are tricky.

Instead of simple caching results numbered 195 -> 210, we attempt to cache blocks of 200.

Since we perform caching in blocks of 200, hitting page 1 of any sort with a page size of 50 wil pre-fetch the next 3 pages.

But there was a bug.

The code that decided how many results we need to pre-fetched, was only fetching results 0 through 200. Somebody (Waffles) forgot to allow for a misaligned block.

I just fixed the engine so it now will fetch 0 -> 400 in the example case.

And there was peace on earth...

or was there... fast forward a few months and we discovered that .NET is freezing our app once every few minutes for about a second, it does not like having so much stuff in memory ... the resolution of this is a topic of another post

| improve this answer | |
  • 3
    Very interesting and a lot of details. +1 ! – genesis Oct 18 '11 at 5:37
  • 1
    I'm rather surprised it didn't take you 6-8 weeks to fix it :D +1! – vzwick Oct 18 '11 at 9:10
  • @waffles - Is the resolution of that secondary problem the topic of another post you've made, or the topic of a post you'll hypothetically make later? – Kevin Vermeer Oct 18 '11 at 16:22
  • @KevinVermeer I plan to blog about it – waffles Oct 19 '11 at 7:39
  • Doesn't a block size of 400 just shift the problem to page 28 (with page size 15)? Would a block size of 300 (evenly divisible by 15, 30 and 50) be better? – freiheit Oct 20 '11 at 20:06
  • @freiheit ... it actually extends the fetch size by 200 till there is full cover – waffles Oct 20 '11 at 20:09
  • in general nobody visits page 14 ... it is possible that a block size of 100 would actually be more effective – waffles Oct 20 '11 at 20:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .