What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

I saw it's not just a page views number, because, if I press F5 several times, it won't increase the number of views.

They could store my IP address in a table, but wouldn't that make it slow? They would need to query a database one more time for each request.

share|improve this question
1  
+1 for a great question. @Peter thanks for starting a bounty. Hopefully someone with internal knowledge will answer; however, I'd bet not since they don't want to open up the number to gaming it. There are lots of questions on SO that show how difficult it is to get views right. You have to worry about giant IP tables, large groups of people with the same NATed IP, etc. Hope someone with knowledge of inner workings answers this. –  Scott May 5 '11 at 13:39
    
RE: "They would need to query a database one more time for each request." Actually, with an intelligent design, this is not true. You would simply send the IP to the database every time, and using a stored procedure decide whether the count should be incremented. If so, increment, then return count. See? One round trip to database. ;) –  Chiramisu Jul 17 '12 at 21:23

2 Answers 2

up vote 63 down vote accepted
+100

I've recently investigated the functioning of the Stack Overflow views counter because I wanted to implement something similar (which I did). My ramblings on the matter are here: Dissecting the Stack Overflow views counter

So, how that thing works? Quite simply, as I turned out to be.

Every question page has that counter link embedded in it:

http://stackoverflow.com/posts/3590653/ivc/[Random code]

which is hit with every page load (either cached or not).

There is some sort of a throttling mechanism in action. It saves the information about a question view per visitor like in pairs:

  • for anonymous users, it is IP + QuestionNr.

  • for authenticated users it is UserNr + QuestionNr.

This information is saved in an expiring cache entry for about 15 minutes. If a subsequent hit sees the entry is still there it discards the new hit. If it is already gone it allows for a new record.

Every time a new hit is registered, it is also added to a memory buffer in addition to the expiring cache entry. The buffer itself also expires after a few minutes or after it is filled up to a certain size, whichever happens first. When it expires, everything it has accumulated is written into the database in bulk. They call it a "buffered write scheme". I like the term. Basically the buffer entries are grouped per question and then just added to the sum of the questions views, no particular table to store every visit details (too much to store), like:

UPDATE Question
SET Views = Views + @NewViews
WHERE Nr = 36278

And the same for every question which has any views registered in the buffer. To optimize and minimize the database access you send the entire data for multiple questions to your update query in one run. You can format the data as XML, join to it inside the query and perform the update in one statement.

That's pretty much it.

I haven't been able to figure out what the [Random code] in the counter url does, but that's okay. Without that mysterious part, I have implemented this scheme under ASP.NET MVC + SQL Server about two weeks ago for a project I'm currently working on. I've got it running on my development machine since then and it's worked like a charm. Views are properly registered as they should. :)

share|improve this answer
    
Why do they need to store the hit info in both the buffer and cache? Is the buffer in the cache? When you say it expires, are you using CacheItemRemovedCallback to handle the expired cache to update sqlserver? Thanks. –  ray247 Jun 29 '11 at 1:01
    
The buffer is in the cache as well and it expires more often than the hits cache. The hits cache is used to limit hits from the same user to once per 15 minutes. From the hits you are receiving you put the new hits (detected with the help of the hits cache) into the buffer and periodically flush it into the database, using a callback method. –  user136634 Jun 29 '11 at 14:38
    
Thanks Developer Art! The cache is to prevent the same user visits too many times per 15 min period, but it is still unclear to me why an additional buffer is used if the cache already collects the hits and write to DB when it expires? In other words, if I only use a cache to collect IP records for a question for 15 minutes and discard any hits from existing IP entries in the cache during that time period and then when the cahce expires at the end of 15 minutes I update DB, would this work? Could you comment on that please. Thank you. –  ray247 Jun 29 '11 at 19:39
    
@ray247: You can get away with just one buffer for hits. In that case however your callback function will be firing for each individual hit and you will be updating database the same amount of time that you have hits. It can be a costly operation. The idea with an additional buffer is to accumulate hits, group them by questions they belong to, and then update the database in bulk, just once per potentially hundreds of hits. –  user136634 Jul 3 '11 at 13:19
    
Hi Developer Art, my idea is to have a List object in the cache, and in the List I put in all the IPs of the request to the page. So the List object acuumulates all the hits to a page, and I cache the List obj for 15 minutes. So I'm not writing to the database for every hit, when the cache expires I add up all the IPs in the List and add that number to the database. –  ray247 Jul 5 '11 at 14:27
    
@ray247: That is not quite the same. In your approach, imagine a user visits a page 1 minute before the cache expiration. The visit is registered and the cache is erased quickly after. Two minutes after the original visit the user visits the page again and since the old buffer is gone the hit will be registered again - 2 times in two minutes - which is obviously wrong. If you store each pair (user + page) in a separate expiring cache entry that will provide for a true throttling mechanism without edge cases. –  user136634 Jul 5 '11 at 16:36
    
@ray247: And since there is no shared storage for all of the hits, it makes sense to add an extra buffer. Make it expire sooner though because the users will wonder why no views were registered in the first 15 minutes since the creation of a question. 3-5 minutes is more likely to get along with their impatience. –  user136634 Jul 5 '11 at 16:36
    
Hi Developer Art, very good points. I'd like to ask for the buffer that expires sooner than the cache, what do you use to time it? Do you use a Thread and let it sleep every 5 mintues and then update DB then sleep again or what do you use? Thank you so much. –  ray247 Jul 7 '11 at 15:25
    
@ray247: I made it simple with the buffer by putting it itself into an expiring cache entry. Upon expiry or upon reaching a predefined count size (whatever happens first) its content is flushed into the database. I'm sure other techniques are conceivable as well. –  user136634 Jul 9 '11 at 17:26
    
@Developer Art, awesome response. Thank you. I've been looking for information on this and it's hard to come by. I have a couple of nitpicks: 1) When you say 'not an update statement for every question', what do you mean? I imagine the buffer storage is some type of struct or array with postId/hits value pairs that's looped over with a query to update the question. 2) Are hits cached on an individual basis, or do they go into some in-memory struct/table that stores all the hit data? –  Mohamad Jul 11 '11 at 23:10
    
@Mohamad: 1) See the updates. 2) I did put them into individual cache entries. Whether it's optimal will be clear when this goes into production. –  user136634 Jul 13 '11 at 20:23
    
These answers are very old but I am still curious, if I check for views on my own question every hour, while logged in, will I increase the number of views on my OWN question ? –  Thalia Sep 23 at 19:27

I tried to cheat writing a script that continually fetched a page to get 10k views and the gold badge, but it didn't work :)

It saves all the IP addresses that viewed the page in the last hour or so. If the IP already viewed the question then the view count isn't increased. It is kinda resource intensive but it's the only way to have an accurate view count.

After an hour or so the entries are deleted so the table doesn't get too big and slow.

share|improve this answer
65  
[citation needed] –  balpha Jan 23 '10 at 14:07
    
clearing the table so it doesn't get big is the trick here so :) Thanks. –  André Pena Jan 23 '10 at 19:24
2  
I too would really like to know the validity of this answer –  barfoon Feb 11 '11 at 17:01
    
@balpha: Isn't observation a good enough source? –  Andreas Bonini Feb 23 '11 at 13:36
18  
@Kop: If you are able to observe what gets deleted from what table in the database, we have big problems :) –  balpha Feb 23 '11 at 13:48
1  
@balpha: now that you have access, are you saying that my post is inaccurate? :P –  Andreas Bonini Feb 23 '11 at 16:53
1  
Have you tried with something like multiproxy? –  systempuntoout Apr 30 '11 at 21:34
    
@Kop, when you say table, what are you talking about? A data structure? An SQL table? I presume, whatever it is, it's something stored in the memory as opposed the HD, right? –  Mohamad Jul 11 '11 at 23:00
    
@Mohamad, just a wild guess, but more likely Redis‌​. –  Arjan Jul 26 '12 at 8:40
1  
As for "the last hour or so", Jeff commented in October 2010: "our view counts are very very strict -- more akin to visits as they are unique per IP per 15 minute interval". But things surely might have changed since then. –  Arjan Jul 26 '12 at 9:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .