8

Here is the ugly display in my browser when accessing blog.stackoverflow.com

screen shot


Here's the Chrome Developer Windows output

Developer Window output


And here's the screenshot where I visit the CSS URL

CSS URL shows 403


wget output

$ wget http://blog.stackoverflow.com/wp-content/themes/se-company/style.css
converted 'http://blog.stackoverflow.com/wp-content/themes/se-company/style.css' (US-ASCII) -> 'http://blog.stackoverflow.com/wp-content/themes/se-company/style.css' (UTF-8)
--2015-03-18 12:34:49--  http://blog.stackoverflow.com/wp-content/themes/se-company/style.css

Resolving blog.stackoverflow.com (blog.stackoverflow.com)... 104.16.24.251, 104.16.25.251
Connecting to blog.stackoverflow.com (blog.stackoverflow.com)|104.16.24.251|:80... connected.

HTTP request sent, awaiting response... 403 Forbidden
2015-03-18 12:34:49 ERROR 403: Forbidden.

Curl Output

curl -o /dev/null -v blog.stackoverflow.com/wp-content/themes/se-company/style.css
* Adding handle: conn: 0x80134f008
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x80134f008) send_pipe: 1, recv_pipe: 0
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to blog.stackoverflow.com port 80 (#0)
*   Trying 104.16.24.251...
* Connected to blog.stackoverflow.com (104.16.24.251) port 80 (#0)
> GET /wp-content/themes/se-company/style.css HTTP/1.1
> User-Agent: curl/7.31.0
> Host: blog.stackoverflow.com
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Date: Wed, 18 Mar 2015 06:16:56 GMT
< Content-Type: text/html; charset=iso-8859-1
< Transfer-Encoding: chunked
< Connection: keep-alive
< Set-Cookie: __cfduid=dc6b1661196b68bb9218bd8bbf831bf4a1426659415; expires=Thu, 17-Mar-16 06:16:55 GMT; path=/; domain=.stackoverflow.com; HttpOnly
< CF-Cache-Status: REVALIDATED
< Set-Cookie: prov=50f26b49-a861-4312-86d3-f5b3c3de42cf; expires=Fri, 01-Jan-2048 00:00:00 GMT; path=/; domain=.stackoverflow.com; HttpOnly
* Server cloudflare-nginx is not blacklisted
< Server: cloudflare-nginx
< CF-RAY: 1c8ecfc327c5081d-SIN
<
{ [data not shown]
100   328    0   328    0     0    334      0 --:--:-- --:--:-- --:--:--   334
* Connection #0 to host blog.stackoverflow.com left intact

I also checked the status code of this URL http://blog.stackoverflow.com/wp-content/themes/se-company/style.css from several online checkers

Yay, looks like one of URL checkers can reproduce my problem here.

Additional info: My IP Address is 167.205.22.xx. It's IP address of Squid load balancer.


Update:

After getting help from Jason Harvey aka alienth (still wondering where is his diamond name), the page was load perfectly. However, recaptcha.css URL still gives 403 Forbidden status. This online checker also confirmed my case.

  • I know the cause here, but it should've been fixed. Can you please give me the output from the following command? curl -o /dev/null -v blog.stackoverflow.com/wp-content/themes/se-company/style.css – alienth Mar 18 '15 at 6:16
  • @alienth See my edit above – masegaloeh Mar 18 '15 at 6:18
  • Thanks! I've manually fixed the issue for now, but unfortunately I need to do further hunting on the root cause (which I thought had been addressed yesterday). Approximately how long have you been seeing this consistently? Also, do you recall when it last worked? – alienth Mar 18 '15 at 6:21
  • I visited that site last week ago and it was worked :). Then today when I visit it again, I got this issue :(. Based on my squid log, today I visited that site at Wed Mar 18 12:33:16.754 +0700 – masegaloeh Mar 18 '15 at 6:32
  • yay, the good looking page of blog.stackoverflow.com is come back :)) – masegaloeh Mar 18 '15 at 6:34
  • Anyway, after you get the root cause please post it in the answer section below. Then I can accept it and this question won't float in unanswered question pool :) – masegaloeh Mar 18 '15 at 7:20
  • 1
    sidenote: @alienth Can you ask a dev to add your diamond on this site? – rene Mar 18 '15 at 8:41
  • 1
    @rene actually employees should be able to add moderator-only tags even without diamond. Seen it more than once in the past. Diamond is just for show. ;-) – Shadow The Princess Wizard Mar 18 '15 at 9:19
  • @alienth, this problem still persist in rechaptca.css URL: blog.stackoverflow.com/wp-content/plugins/wp-recaptcha/… . See update above – masegaloeh Mar 18 '15 at 9:19
  • @masegaloeh Issue should be permanently resolved now. Will include details in the answer box. – alienth Mar 18 '15 at 17:01
3

So, here's what was happening:

For an as of yet undiscovered reason, Apache would occasionally 403 static assets on our blog. When it does this it sends a Last-Modified header. CloudFlare sees this 403 and deems it cachable. When CloudFlare goes to fetch the asset again, they reach out to the origin with an If-Modified-Since request for the asset, which will always return a 304 Not Modified response, since our static assets rarely change. This resulted in any 403 sticking around permanently.

To address this improper sticky 403 behaviour I've altered our reverse proxy to send a cache-control: no-cache to our 403 responses from the blog.

  • No diamond @alienth ? – Tim Mar 18 '15 at 18:39
  • 1
    Not as of yet. I have yet to swear fealty to Bahamut.. err.. get some of my SE stuff established. All in due time. Yes, all in due time. – alienth Mar 18 '15 at 18:48
  • Cool :) Congrats on the job then! (Google dictionary: vast fish :p) – Tim Mar 18 '15 at 19:11
  • Great, thanks for explanation. To all mods please retag it with [status-completed] :) – masegaloeh Mar 18 '15 at 21:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .