During the day, almost every day, I get a handful of
408 Request Timeout pages when doing stuff on Stack Overflow or Meta Stack Overflow.
Is this an Australia thing, or are others experiencing this intermittently?
We send our web logs to SQL so we can query them. It turns out that we have a bug that prevents 408s from being logged to SQL.
I did take several random samples of 10,000,000 requests each from our raw logs and have found ~ 30 per 10,000,000 requests that come through as 408. HAProxy records termination states, for all of them I have found they are logged as "cR--" which means:
c : the client-side timeout expired while waiting for the client to send or receive data. R : the proxy was waiting for a complete, valid REQUEST from the client (HTTP mode only). Nothing was sent to any server.
Since 99.99972% of the requests sampled don't have this problem, for now I am going to regard these as a negligible error. If you are receiving a lot more of these over a long period of time (i.e. more that 30 for every 10 million requests you make) I think there is likely a client side issue.
However, since it is possible that my samples are missing spurts of 408s I will do more extensive research when our 408s are part of queryable logs (should be a week or two, the person who maintains our logging system is on vacation at the moment). I will also look closely at the IPs of people who are reporting the issue to see if I can get any more detail -- but right now it is hard with raw data due to the size of it.
Update May 26, 2014:
It is possible this related to tcp-reconnect in modern browsers and our client timeout settings in haproxy. I'm going to have our latest SRE member Shane Madden investigate this now that I have a reasonable hypothesis.
Update May 28, 2014:
Shane has investigated and this should now be resolved for Chrome users, see answer on 408 Request Time-outs still continues. Anyone else seeing these issues, let us know.