8

Here is the ugly display in my browser when accessing blog.stackoverflow.com

screen shot


Here's the Chrome Developer Windows output

Developer Window output


And here's the screenshot where I visit the CSS URL

CSS URL shows 403


wget output

$ wget https://blog.stackoverflow.com/wp-content/themes/se-company/style.css
converted 'https://blog.stackoverflow.com/wp-content/themes/se-company/style.css' (US-ASCII) -> 'https://blog.stackoverflow.com/wp-content/themes/se-company/style.css' (UTF-8)
--2015-03-18 12:34:49--  https://blog.stackoverflow.com/wp-content/themes/se-company/style.css

Resolving blog.stackoverflow.com (blog.stackoverflow.com)... 104.16.24.251, 104.16.25.251
Connecting to blog.stackoverflow.com (blog.stackoverflow.com)|104.16.24.251|:80... connected.

HTTP request sent, awaiting response... 403 Forbidden
2015-03-18 12:34:49 ERROR 403: Forbidden.

Curl Output

curl -o /dev/null -v blog.stackoverflow.com/wp-content/themes/se-company/style.css
* Adding handle: conn: 0x80134f008
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x80134f008) send_pipe: 1, recv_pipe: 0
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to blog.stackoverflow.com port 80 (#0)
*   Trying 104.16.24.251...
* Connected to blog.stackoverflow.com (104.16.24.251) port 80 (#0)
> GET /wp-content/themes/se-company/style.css HTTP/1.1
> User-Agent: curl/7.31.0
> Host: blog.stackoverflow.com
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Date: Wed, 18 Mar 2015 06:16:56 GMT
< Content-Type: text/html; charset=iso-8859-1
< Transfer-Encoding: chunked
< Connection: keep-alive
< Set-Cookie: __cfduid=dc6b1661196b68bb9218bd8bbf831bf4a1426659415; expires=Thu, 17-Mar-16 06:16:55 GMT; path=/; domain=.stackoverflow.com; HttpOnly
< CF-Cache-Status: REVALIDATED
< Set-Cookie: prov=50f26b49-a861-4312-86d3-f5b3c3de42cf; expires=Fri, 01-Jan-2048 00:00:00 GMT; path=/; domain=.stackoverflow.com; HttpOnly
* Server cloudflare-nginx is not blacklisted
< Server: cloudflare-nginx
< CF-RAY: 1c8ecfc327c5081d-SIN
<
{ [data not shown]
100   328    0   328    0     0    334      0 --:--:-- --:--:-- --:--:--   334
* Connection #0 to host blog.stackoverflow.com left intact

I also checked the status code of this URL https://blog.stackoverflow.com/wp-content/themes/se-company/style.css from several online checkers

Yay, looks like one of URL checkers can reproduce my problem here.

Additional info: My IP Address is 167.205.22.xx. It's IP address of Squid load balancer.


Update:

After getting help from Jason Harvey aka alienth (still wondering where is his diamond name), the page was load perfectly. However, recaptcha.css URL still gives 403 Forbidden status. This online checker also confirmed my case.

10
  • I know the cause here, but it should've been fixed. Can you please give me the output from the following command? curl -o /dev/null -v blog.stackoverflow.com/wp-content/themes/se-company/style.css
    – alienth
    Mar 18, 2015 at 6:16
  • @alienth See my edit above
    – masegaloeh
    Mar 18, 2015 at 6:18
  • Thanks! I've manually fixed the issue for now, but unfortunately I need to do further hunting on the root cause (which I thought had been addressed yesterday). Approximately how long have you been seeing this consistently? Also, do you recall when it last worked?
    – alienth
    Mar 18, 2015 at 6:21
  • I visited that site last week ago and it was worked :). Then today when I visit it again, I got this issue :(. Based on my squid log, today I visited that site at Wed Mar 18 12:33:16.754 +0700
    – masegaloeh
    Mar 18, 2015 at 6:32
  • yay, the good looking page of blog.stackoverflow.com is come back :))
    – masegaloeh
    Mar 18, 2015 at 6:34
  • Anyway, after you get the root cause please post it in the answer section below. Then I can accept it and this question won't float in unanswered question pool :)
    – masegaloeh
    Mar 18, 2015 at 7:20
  • 1
    sidenote: @alienth Can you ask a dev to add your diamond on this site?
    – rene
    Mar 18, 2015 at 8:41
  • 1
    @rene actually employees should be able to add moderator-only tags even without diamond. Seen it more than once in the past. Diamond is just for show. ;-) Mar 18, 2015 at 9:19
  • @alienth, this problem still persist in rechaptca.css URL: blog.stackoverflow.com/wp-content/plugins/wp-recaptcha/… . See update above
    – masegaloeh
    Mar 18, 2015 at 9:19
  • @masegaloeh Issue should be permanently resolved now. Will include details in the answer box.
    – alienth
    Mar 18, 2015 at 17:01

1 Answer 1

3

So, here's what was happening:

For an as of yet undiscovered reason, Apache would occasionally 403 static assets on our blog. When it does this it sends a Last-Modified header. CloudFlare sees this 403 and deems it cachable. When CloudFlare goes to fetch the asset again, they reach out to the origin with an If-Modified-Since request for the asset, which will always return a 304 Not Modified response, since our static assets rarely change. This resulted in any 403 sticking around permanently.

To address this improper sticky 403 behaviour I've altered our reverse proxy to send a cache-control: no-cache to our 403 responses from the blog.

4
  • No diamond @alienth ?
    – Tim
    Mar 18, 2015 at 18:39
  • 1
    Not as of yet. I have yet to swear fealty to Bahamut.. err.. get some of my SE stuff established. All in due time. Yes, all in due time.
    – alienth
    Mar 18, 2015 at 18:48
  • Cool :) Congrats on the job then! (Google dictionary: vast fish :p)
    – Tim
    Mar 18, 2015 at 19:11
  • Great, thanks for explanation. To all mods please retag it with [status-completed] :)
    – masegaloeh
    Mar 18, 2015 at 21:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .