Dissecting the Stack Overflow views counter [duplicate]

Question

The last couple of days I've been trying to understand how the Stack Overflow page views counter works. I wish to add a views counter to a project of mine and investigating the options. Stack Overflow seems to be a good source of useful ideas.

Needless to say I've browsed through all of the related questions here on Stack Overflow and on Meta Stack Overflow. I can't say I've found an explanation but some tips (presumably false leading) were given in this answer: View counter in ASP.NET MVC

From what I can tell there is now a JavaScript section in each answer page (used to be CSS in 2008):

<script type="text/javascript" src="/posts/3590653/ivc/a947"></script>
<noscript>
    <div><img src="/posts/3590653/ivc/a947" class="dno" alt=""></div>
</noscript>

The last part of the URL - a947 - seems to be dynamically generated. It is changed every 15-20 minutes and is the same for all questions and presumably users (I tried anonymous access, changing my IP address, it stays the same during the interval).

In a comment to the answer Did anyone notice that some sites seem to be scraping/republishing SO's posts? Jeff Atwood gave a bit of a hint:

our view counts are very very strict -- more akin to visits as they are unique per IP per 15 minute interval.

Observing things with Firebug reveals that this URL always returns "204 No Content". So my first idea was that the browser is likely not to retrieve that counter URL on subsequent page requests until the random code changes, thus preventing duplicate counter hits already on the client side for that 15 minutes interval.

Watching the things with Firebug doesn't seem to confirm that theory:

As you can see the counter URL is retrieved again and again (even if I don't refresh the page with F5 but just click the page link in the title).

More to it. I tried to recreate the same scheme in a test project also returning 204, and I confirm that the counter URL is retrieved again and again. No browser caching.

More peculiar, Firefox for some reason calls the URL twice (the number is always even):

At this point the idea behind this counter URL is a mystery to me. What is the point of that last changing part of the URL if it's the same for all questions and users and does not prevent repeated requests from the client-side?

In that question a possible explanation is given:

View counter in ASP.NET MVC

I think I know the answer - they are analyzing the IIS log as Ope suggested.

Hidden image src is set to

https://stackoverflow.com/posts/3590653/ivc/[Random code]

[Random code] is needed because many people may share the same IP address (in a network, for example) and the code is used to distinguish users.

I'm not sure I understand this and I somehow feel it's off course.

Can somebody or perhaps the Stack Exchange Inc. folks explain what and how they are doing with the views counter?

I'm not trying to understand how it works in order to start gaming the system, I only wish to learn how these things are done. I hope Stack Exchange Inc. won't mind revealing some secrets.

UPDATE: I'm beginning to suspect this changing value at the end of the url is used to help aggregate the collected data by dividing time in 15-minute-slots. Still thinking in what way exactly...

UPDATE 2: So I've tried my test project with different browsers to see if 204 No Content would result in browser caching. It doesn't. The counter URL is retrieved again when the page is requested next time, but the value at the end of the counter URL is not changed. I tried with Internet Explorer 8 and the current versions of Firefox 3.6, Opera, Safari and Chrome. The behavior is identical and not surprising actually since the response for 204 doesn't contain the "Expires" header (neither does a live response from stackoverflow.com) so there is no reason for the browser to cache the response.

I kindly ask again the nice folks what is then the reason of that changing value of the counter URL? It doesn't force temporary browser caching so the reason must be elsewhere.

Answer 1

All right, I have a theory... let's see what you think about it.

This changing value is meant as a time interval identifier (let's call it tid for brevity) in the following scheme:

There is a buffer in memory which accumulates counter hit data then periodically writes it to the database where it updates the [Views] column of each question involved. Since Jeff calls the mechanism in place a "buffered writing scheme", the theory fits the description.

The buffer has some fixed size. It's flushed when either it's filled up or when a predetermined time interval elapses. That's where the tid comes into play - it helps to distinguish the records pertaining to the previous interval from those belonging to the time interval still in progress. Only the hits from the previous interval are written to the database and erased from the buffer. The records from the active interval are kept until it elapses. This effectively achieves the model of a view as one hit per time interval.

The tid is generated somewhat like this:

14:00 > ca6f
14:15 > de42
14:30 > a590
14:45 > 3ab0
15:00 > 7ff1
...

Therefore it works in 15-minute-tact most of the time.

However at the times of traffic surge the buffer is going to be filled quicker. Suppose at peak hours it is filled up each 3 minutes and has to be flushed. When it happens the next tid is generated immediately with its 15-minute lifetime. The round times from the example above would obviously shift. Perhaps this is why the ids generated for Stack Overflow pages do not tact quite evenly. If the traffic surge ceases then the system works again in 15-minute intervals.

That however necessitates that the definition of a view depends on how much traffic the site is currently receiving. Makes sense actually. The more visitors there is, the more likelihood a question has changed significantly in a shorter period of time therefore quicker deprecating what the user has already seen. So one view in 3 minutes at peak hours and one view per 15 minutes during off hours increment the counter.

Do I make any sense? Your opinions?