13

http://stackexchange.com/feeds/questions hasn't been updated for 12 hours and https://stackexchange.com/feeds/questions for two days. Both protocols return different content, but neither are up to date.

However, when logged in, both versions give the same content, which looks like the correct one. The <updated> field in <feed> is just minutes ago.

Diff of response headers for https between logged out and logged in (http has similar values):

Header           Logged out                      Logged in
-------------------------------------------------------------------------------------------
Cache-Control    public, max-age=63626889542     public, no-cache="Set-Cookie", max-age=120
Content-Length   37765                           36807
Expires          Tue, 04 Apr 2017 07:59:02 GMT   Wed, 05 Apr 2017 19:15:55 GMT
Last-Modified    Tue, 04 Apr 2017 07:57:02 GMT   Wed, 05 Apr 2017 19:13:55 GMT
Age              126879                          (none)
X-Served-By      cache-yyz8335-YYZ               cache-yyz8332-YYZ
X-Cache          HIT                             MISS
X-Cache-Hits     4                               0
X-Timer          S1491419523.730773,VS0,VE1      S1491419635.717309,VS0,VE102

Edit: The plot thickens: I get the same thing on my phone on data and on two machines on my network (two different ISPs), but both a friend who lives in the same city and Cai in the comments below get an up-to-date version. I have no idea what's going on.

stackexchange.com resolves to 151.101.193.69 on both ISPs, but friend's gives 151.101.65.69. We both flushed the DNS cache and now we're both getting 151.101.1.69, but it's still giving me an outdated version.

Going through a VPN fixes all the issues, which is not that surprising, but turning my VPN off and doing a ctrl-f5 gives me back the outdated version. And like I said above, sending my session cookie always gives me the correct, up-to-date version:

$ curl -I -b "cookie string copied from browser"  https://stackexchange.com/feeds/questions
HTTP/1.1 200 OK
Cache-Control: public, no-cache="Set-Cookie", max-age=120
Content-Length: 36558
Content-Type: application/atom+xml; charset=utf-8
Expires: Thu, 06 Apr 2017 18:50:22 GMT
Last-Modified: Thu, 06 Apr 2017 18:48:22 GMT          <----- Note
X-Frame-Options: SAMEORIGIN
Set-Cookie: (redacted)
X-Request-Guid: c9482245-8d6b-4de5-a8de-1e25205e54c9
Accept-Ranges: bytes
Date: Thu, 06 Apr 2017 18:48:22 GMT
Via: 1.1 varnish
Connection: keep-alive
X-Served-By: cache-yyz8330-YYZ
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1491504502.382361,VS0,VE51
Vary: *,Fastly-SSL
X-DNS-Prefetch-Control: off

$ curl -I https://stackexchange.com/feeds/questions
HTTP/1.1 200 OK
Cache-Control: public, max-age=63626889542
Content-Type: application/atom+xml; charset=utf-8
Expires: Tue, 04 Apr 2017 07:59:02 GMT
Last-Modified: Tue, 04 Apr 2017 07:57:02 GMT          <----- Note
X-Frame-Options: SAMEORIGIN
X-Request-Guid: 2087959f-edc8-4b74-a078-8bcb874b4337
Content-Length: 37765
Accept-Ranges: bytes
Date: Thu, 06 Apr 2017 18:48:37 GMT
Via: 1.1 varnish
Age: 211874
Connection: keep-alive
X-Served-By: cache-yyz8335-YYZ
X-Cache: HIT
X-Cache-Hits: 7
X-Timer: S1491504518.899367,VS0,VE0
Vary: *,Fastly-SSL
X-DNS-Prefetch-Control: off
Set-Cookie: (redacted)
  • I've only checked the HNQ, I don't know if the other feeds have issues. – isanae Apr 5 '17 at 19:43
  • Guess it's the cache control max-age, which is set to, like... forever – Cai Apr 5 '17 at 19:51
  • @Cai It might interfere with normal browser requests, but curl, wget or just ctrl-f5 wouldn't care. They all fail right now. – isanae Apr 6 '17 at 7:38
  • well fwiw they're all updating for me (last-modified is always a within the last few minutes at least, I'm not actually checking the content...) – Cai Apr 6 '17 at 8:24
  • @Cai Are you sure? They're not for me. Can you check in a private session in your browser to make sure you're not logged in? – isanae Apr 6 '17 at 16:56
  • 1
    Both in a private session, definitely not logged in: HTTPS and HTTP, I've checked a couple times today and it seems to always been up to date – Cai Apr 6 '17 at 17:14
  • Actually, just noticed the max-age is looking a bit more reasonable on HTTPS now; I was getting the same as you on that earlier (although it was still up to date) – Cai Apr 6 '17 at 17:16
  • @Cai Here's HTTP and HTTPS for me. Same thing in Firefox. Looks like server-side caching issue, but I don't know what's going on. – isanae Apr 6 '17 at 17:26
  • Yeh no idea, very strange... hopefully a dev will have a look and see what's going on – Cai Apr 6 '17 at 17:35
  • Thanks for the heads up, we're taking a look at the CDN here. – Nick Craver Apr 7 '17 at 16:50
  • @NickCraver Oh I'm so happy! My feed has been empty for two days and I've had to actually work! Tell me if you need anything else, I'm still reproducing the problem right now. – isanae Apr 7 '17 at 16:54
  • @isanae Just to help us save some dig time here - any idea when this started? You seem to be an avid user, so is this very recent? We haven't made any known changes here...so digging into the unknown. – Nick Craver Apr 7 '17 at 17:14
  • @NickCraver I've opened a chat room. – isanae Apr 7 '17 at 17:16
21

First, for all website admins using sites with output caching:

If you use [OutputCache], DO NOT UPGRADE TO .NET 4.6.2


Details:

Unfortunately, this was far larger than your feed and we're still assessing the damage. There is a critical bug in the 4.6.2 release of the .NET Framework which causes cache poisoning in a severe fashion. Though the cause is a simple errant setter, the impact is tremendous. Here's the fix in .NET 4.7 (not available for our Windows Server edition yet): .NET 4.6.2 vs 4.7 (You're looking at HttpCachePolicy.cs, here's the link for .NET 4.7)

The manifestation of this bug is an errant setting _utcTimestampRequest to 0, and amounts to running the following to calculate the max-age:

0 - (DateTime.MinValue - DateTime.UtcNow).TotalSeconds

Which is 63,627,196,173 seconds, or about 2017 years (since it's the diff to DateTime.MinValue or 0001-01-01 00:00:00Z in .NET).

This means browsers and proxies are instructed to cache content not for the 15 to 180 seconds we instructed, but 2 millennia instead. This is a huge, huge bug that needs a hotfix for all customers ASAP. I am stressing that to Microsoft as strongly as we can.

While we are rather agile and will deploy a private .NET 4.7 build if we can arrange one, most people aren't so lucky and often have to wait many months before they can update their releases, whereas a minor 4.6.x release can happen quickly. Your only options (currently) are to roll forward, roll back, or ditch output caching. We've currently opted for #3.

We have disabled all output cache directives at Fastly and stripped the rest in code. The only exception to this is user flair (a bit heavier on bytes), which we're manually overriding in Fastly to the 180 seconds it should be.

I will continue to work with Microsoft on a resolution into the evening and update this as I am able.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .