237

I saw it's not just a page views number, because, if I press F5 several times, it won't increase the number of views.

They could store my IP address in a table, but wouldn't that make it slow? They would need to query a database one more time for each request.

Improve this question
2
  • 4
    +1 for a great question. @Peter thanks for starting a bounty. Hopefully someone with internal knowledge will answer; however, I'd bet not since they don't want to open up the number to gaming it. There are lots of questions on SO that show how difficult it is to get views right. You have to worry about giant IP tables, large groups of people with the same NATed IP, etc. Hope someone with knowledge of inner workings answers this.
    – Scott
    Commented May 5, 2011 at 13:39
  • RE: "They would need to query a database one more time for each request." Actually, with an intelligent design, this is not true. You would simply send the IP to the database every time, and using a stored procedure decide whether the count should be incremented. If so, increment, then return count. See? One round trip to database. ;)
    – Chiramisu
    Commented Jul 17, 2012 at 21:23

2 Answers 2

Reset to default
177
+100

I've recently investigated the functioning of the Stack Overflow views counter because I wanted to implement something similar (which I did). My ramblings on the matter are here: Dissecting the Stack Overflow views counter

So, how that thing works? Quite simply, as I turned out to be.

Every question page has that counter link embedded in it:

http://stackoverflow.com/posts/3590653/ivc/[Random code]

which is hit with every page load (either cached or not).

There is some sort of a throttling mechanism in action. It saves the information about a question view per visitor like in pairs:

  • for anonymous users, it is IP + QuestionNr.

  • for authenticated users it is UserNr + QuestionNr.

This information is saved in an expiring cache entry for about 15 minutes. If a subsequent hit sees the entry is still there it discards the new hit. If it is already gone it allows for a new record.

Every time a new hit is registered, it is also added to a memory buffer in addition to the expiring cache entry. The buffer itself also expires after a few minutes or after it is filled up to a certain size, whichever happens first. When it expires, everything it has accumulated is written into the database in bulk. They call it a "buffered write scheme". I like the term. Basically the buffer entries are grouped per question and then just added to the sum of the questions views, no particular table to store every visit details (too much to store), like:

UPDATE Question
SET Views = Views + @NewViews
WHERE Nr = 36278

And the same for every question which has any views registered in the buffer. To optimize and minimize the database access you send the entire data for multiple questions to your update query in one run. You can format the data as XML, join to it inside the query and perform the update in one statement.

That's pretty much it.

I haven't been able to figure out what the [Random code] in the counter url does, but that's okay. Without that mysterious part, I have implemented this scheme under ASP.NET MVC + SQL Server about two weeks ago for a project I'm currently working on. I've got it running on my development machine since then and it's worked like a charm. Views are properly registered as they should. :)

Improve this answer
14
  • Why do they need to store the hit info in both the buffer and cache? Is the buffer in the cache? When you say it expires, are you using CacheItemRemovedCallback to handle the expired cache to update sqlserver? Thanks.
    – Ray
    Commented Jun 29, 2011 at 1:01
  • The buffer is in the cache as well and it expires more often than the hits cache. The hits cache is used to limit hits from the same user to once per 15 minutes. From the hits you are receiving you put the new hits (detected with the help of the hits cache) into the buffer and periodically flush it into the database, using a callback method.
    – user136634
    Commented Jun 29, 2011 at 14:38
  • Thanks Developer Art! The cache is to prevent the same user visits too many times per 15 min period, but it is still unclear to me why an additional buffer is used if the cache already collects the hits and write to DB when it expires? In other words, if I only use a cache to collect IP records for a question for 15 minutes and discard any hits from existing IP entries in the cache during that time period and then when the cahce expires at the end of 15 minutes I update DB, would this work? Could you comment on that please. Thank you.
    – Ray
    Commented Jun 29, 2011 at 19:39
  • @ray247: You can get away with just one buffer for hits. In that case however your callback function will be firing for each individual hit and you will be updating database the same amount of time that you have hits. It can be a costly operation. The idea with an additional buffer is to accumulate hits, group them by questions they belong to, and then update the database in bulk, just once per potentially hundreds of hits.
    – user136634
    Commented Jul 3, 2011 at 13:19
  • Hi Developer Art, my idea is to have a List object in the cache, and in the List I put in all the IPs of the request to the page. So the List object acuumulates all the hits to a page, and I cache the List obj for 15 minutes. So I'm not writing to the database for every hit, when the cache expires I add up all the IPs in the List and add that number to the database.
    – Ray
    Commented Jul 5, 2011 at 14:27
  • @ray247: That is not quite the same. In your approach, imagine a user visits a page 1 minute before the cache expiration. The visit is registered and the cache is erased quickly after. Two minutes after the original visit the user visits the page again and since the old buffer is gone the hit will be registered again - 2 times in two minutes - which is obviously wrong. If you store each pair (user + page) in a separate expiring cache entry that will provide for a true throttling mechanism without edge cases.
    – user136634
    Commented Jul 5, 2011 at 16:36
  • @ray247: And since there is no shared storage for all of the hits, it makes sense to add an extra buffer. Make it expire sooner though because the users will wonder why no views were registered in the first 15 minutes since the creation of a question. 3-5 minutes is more likely to get along with their impatience.
    – user136634
    Commented Jul 5, 2011 at 16:36
  • Hi Developer Art, very good points. I'd like to ask for the buffer that expires sooner than the cache, what do you use to time it? Do you use a Thread and let it sleep every 5 mintues and then update DB then sleep again or what do you use? Thank you so much.
    – Ray
    Commented Jul 7, 2011 at 15:25
  • @ray247: I made it simple with the buffer by putting it itself into an expiring cache entry. Upon expiry or upon reaching a predefined count size (whatever happens first) its content is flushed into the database. I'm sure other techniques are conceivable as well.
    – user136634
    Commented Jul 9, 2011 at 17:26
  • @Developer Art, awesome response. Thank you. I've been looking for information on this and it's hard to come by. I have a couple of nitpicks: 1) When you say 'not an update statement for every question', what do you mean? I imagine the buffer storage is some type of struct or array with postId/hits value pairs that's looped over with a query to update the question. 2) Are hits cached on an individual basis, or do they go into some in-memory struct/table that stores all the hit data?
    – Mohamad
    Commented Jul 11, 2011 at 23:10
  • @Mohamad: 1) See the updates. 2) I did put them into individual cache entries. Whether it's optimal will be clear when this goes into production.
    – user136634
    Commented Jul 13, 2011 at 20:23
  • These answers are very old but I am still curious, if I check for views on my own question every hour, while logged in, will I increase the number of views on my OWN question ?
    – Thalia
    Commented Sep 23, 2014 at 19:27
  • @Thalia: yes, it should increase the count. Does viewing your own question add to its view count?
    – jfs
    Commented May 19, 2018 at 17:34
  • How do you add a buffer and cache? In PHP?
    – Marvin
    Commented May 21, 2019 at 1:36
42

I tried to cheat writing a script that continually fetched a page to get 10k views and the gold badge, but it didn't work :)

It saves all the IP addresses that viewed the page in the last hour or so. If the IP already viewed the question then the view count isn't increased. It is kinda resource intensive but it's the only way to have an accurate view count.

After an hour or so the entries are deleted so the table doesn't get too big and slow.

Improve this answer
11
  • 88
    [citation needed]
    – balpha StaffMod
    Commented Jan 23, 2010 at 14:07
  • clearing the table so it doesn't get big is the trick here so :) Thanks.
    – andrerpena
    Commented Jan 23, 2010 at 19:24
  • 2
    I too would really like to know the validity of this answer
    – barfoon
    Commented Feb 11, 2011 at 17:01
  • 2
    @balpha: Isn't observation a good enough source?
    – Andreas Bonini
    Commented Feb 23, 2011 at 13:36
  • 38
    @Kop: If you are able to observe what gets deleted from what table in the database, we have big problems :)
    – balpha StaffMod
    Commented Feb 23, 2011 at 13:48
  • 3
    @balpha: now that you have access, are you saying that my post is inaccurate? :P
    – Andreas Bonini
    Commented Feb 23, 2011 at 16:53
  • 1
    Have you tried with something like multiproxy?
    – systempuntoout
    Commented Apr 30, 2011 at 21:34
  • @Kop, when you say table, what are you talking about? A data structure? An SQL table? I presume, whatever it is, it's something stored in the memory as opposed the HD, right?
    – Mohamad
    Commented Jul 11, 2011 at 23:00
  • @Mohamad, just a wild guess, but more likely Redis.
    – Arjan
    Commented Jul 26, 2012 at 8:40
  • 3
    As for "the last hour or so", Jeff commented in October 2010: "our view counts are very very strict -- more akin to visits as they are unique per IP per 15 minute interval". But things surely might have changed since then.
    – Arjan
    Commented Jul 26, 2012 at 9:09
  • If they delete the views data after 1 hour then anyone would be able to register view count by calling script after every one hour and what about registered user ? how they handle it 27k views = 27k records
    – Vikas Kandari
    Commented Jan 31, 2020 at 9:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .