45

I noticed that Page 14 of "newest questions" seems to always show only half of the questions the other pages do.

So, just out of curiosity … What's up with that, SO devs?

Improve this question
19
  • 2
    Do you have the "Hide ignored tags" option selected on your profile (the "perfs" tab)?
    – ChrisF Mod
    Oct 17, 2011 at 12:00
  • 1
    @ChrisF Nope. And I have no ignored tags.
    – vzwick
    Oct 17, 2011 at 12:01
  • I can reproduce this, it also happens on page 7 with 30 items per page, always after 200 questions.
    – Mad Scientist
    Oct 17, 2011 at 12:16
  • 5
    Reproduced by simply clicking the link: Chrome 14.0.835.202 shows five results.
    – Grant Thomas
    Oct 17, 2011 at 12:23
  • 1
    @Mr.Disappointment the link was originally - http://stackoverflow.com/questions?page=14&sort=newest - so, if it depends on the page size, wouldn't necessarily show the problem
    – ChrisF Mod
    Oct 17, 2011 at 12:25
  • 3
    @ChrisF Indeed, my setting of a page size of 50 is not affected; adding pagesize=15 results in 5 results, adding pagesize=30 results in 10 results.
    – Grant Thomas
    Oct 17, 2011 at 12:47
  • 3
    It must be an Easter egg. :-)
    – apaderno
    Oct 17, 2011 at 14:16
  • 2
    @kiamlaluno i would have assumed just that if it had been page 42 :p
    – vzwick
    Oct 17, 2011 at 14:17
  • Cache issue, as after navigating to different page and coming back it's fine..
    – Shadow The Kid Wizard
    Oct 17, 2011 at 14:30
  • 1
    Hmm... it behave really strange. First click - five results. Closing window, trying again - five results. Closing window, opening different page, closing and opening with the original link - 15 results as expected, and it keep being OK afterwards even in different browsers - so it's server cache issue..
    – Shadow The Kid Wizard
    Oct 17, 2011 at 14:37
  • 1
    @ShadowWizard Server cache issue tied to your user account? 'Cause I bet there's dozens of users hitting that page on every given second.
    – vzwick
    Oct 17, 2011 at 14:39
  • 15
    SE Dev 1 to Dev 2: Hey, they finally found that page 14 joke you coded 3 years ago!
    – NGLN
    Oct 17, 2011 at 15:18
  • 4
    We're aware of this, some recent tag engine internal changes are likely in play here. Marc's taking a look soon as he gets a chance.
    – Nick Craver
    Oct 17, 2011 at 16:37
  • 1
    @NickCraver Would be awesome if an in-depth explanation followed up here ;)
    – vzwick
    Oct 17, 2011 at 17:03
  • 1
    @vzw In depth explanation: If ( page==14 ) { ... }
    – Dr. belisarius
    Oct 18, 2011 at 4:44

1 Answer 1

Reset to default
31

So, just out of curiosity … What's up with that, SO devs?

Well, back in the day we used to use direct queries to SQL Server to build up the various question sort pages.

This worked fine for a while, and Smurfs were happy. However, slowly, over time we discovered that for our specific use SQL is just not fast enough.

I posted a question on theoretical CS that touches on the issue: Data structure that allow efficient tag based lookups

Fast forward a few months, and we reimplemented a large amount of functionality we used to depend on SQL Server for, using in-memory structures.

These days, when you click on any tag or combination of tags or click on the top level Newest, Faq, Active, Votes or Featured tabs you hit the TAG ENGINE OF DOOM

This in-memory structure holds all of the information needed to provide the various question sorts and filters in lightning speed.

The Tag Engine contains many optimisations, one such optimisation is the ...

Block size in the query cache

When the tag engine is asked for all the questions on page 14 with page size 15, we cache the results so subsequent queries do not need to traverse the tag engine.

But we are tricky.

Instead of simple caching results numbered 195 -> 210, we attempt to cache blocks of 200.

Since we perform caching in blocks of 200, hitting page 1 of any sort with a page size of 50 wil pre-fetch the next 3 pages.

But there was a bug.

The code that decided how many results we need to pre-fetched, was only fetching results 0 through 200. Somebody (Waffles) forgot to allow for a misaligned block.

I just fixed the engine so it now will fetch 0 -> 400 in the example case.

And there was peace on earth...

or was there... fast forward a few months and we discovered that .NET is freezing our app once every few minutes for about a second, it does not like having so much stuff in memory ... the resolution of this is a topic of another post

Improve this answer
7
  • 3
    Very interesting and a lot of details. +1 !
    – genesis
    Oct 18, 2011 at 5:37
  • 1
    I'm rather surprised it didn't take you 6-8 weeks to fix it :D +1!
    – vzwick
    Oct 18, 2011 at 9:10
  • @waffles - Is the resolution of that secondary problem the topic of another post you've made, or the topic of a post you'll hypothetically make later?
    – Kevin Vermeer
    Oct 18, 2011 at 16:22
  • @KevinVermeer I plan to blog about it
    – waffles
    Oct 19, 2011 at 7:39
  • Doesn't a block size of 400 just shift the problem to page 28 (with page size 15)? Would a block size of 300 (evenly divisible by 15, 30 and 50) be better?
    – freiheit
    Oct 20, 2011 at 20:06
  • @freiheit ... it actually extends the fetch size by 200 till there is full cover
    – waffles
    Oct 20, 2011 at 20:09
  • in general nobody visits page 14 ... it is possible that a block size of 100 would actually be more effective
    – waffles
    Oct 20, 2011 at 20:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .