13

Since yesterday and today, I've noticed that at around this time (I'd say ~4:00 GMT and on) I would get intermittent connection problems on all the sites I am participating on.

Is it just me or are there server issues?

Prior to this, I was out of town (San Francisco) and didn't see problems then, but then again, I wasn't using the sites as much as I usually am. I remember seeing the "We are offline" message only once just before I left to go home (~20:30 GMT 2 days ago) but didn't think much of it. I just returned home (San Diego) yesterday.

By connection problems, I mean when loading a page, at some point the page will just stop loading. Sometimes, waiting for it to try to finish loading would load the page eventually, otherwise my browser (FF5) would be stuck "transferring data" indefinitely. Usually reloading the page then would then work.

I don't think I'm having computer or local connection problems. The same issue occurs on my laptop (the same I was using when I was out of town) and I am not having problems connecting to other non-Stack Exchange sites on either computer. My router is set to reset every day so my connection is always fresh and I regularly shutdown my computer overnight.

Pinging Stack Overflow shows some interesting results. I'm seeing the same on my desktop computer and my laptop.

>ping -t stackoverflow.com

Pinging stackoverflow.com [64.34.119.12] with 32 bytes of data:
Reply from 64.34.119.12: bytes=32 time=78ms TTL=55
Reply from 64.34.119.12: bytes=32 time=82ms TTL=55
Reply from 64.34.119.12: bytes=32 time=84ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=81ms TTL=55
Reply from 64.34.119.12: bytes=32 time=78ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=85ms TTL=55
Reply from 64.34.119.12: bytes=32 time=87ms TTL=55
Request timed out.
Reply from 64.34.119.12: bytes=32 time=79ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=81ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=78ms TTL=55
Reply from 64.34.119.12: bytes=32 time=82ms TTL=55
Reply from 64.34.119.12: bytes=32 time=79ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=79ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=79ms TTL=55
Reply from 64.34.119.12: bytes=32 time=79ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=81ms TTL=55
Reply from 64.34.119.12: bytes=32 time=81ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=83ms TTL=55
Request timed out.
Reply from 64.34.119.12: bytes=32 time=79ms TTL=55
Reply from 64.34.119.12: bytes=32 time=81ms TTL=55
Reply from 64.34.119.12: bytes=32 time=81ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=82ms TTL=55
Reply from 64.34.119.12: bytes=32 time=79ms TTL=55
Reply from 64.34.119.12: bytes=32 time=82ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=81ms TTL=55
Request timed out.
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=82ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=81ms TTL=55
Reply from 64.34.119.12: bytes=32 time=96ms TTL=55
Reply from 64.34.119.12: bytes=32 time=81ms TTL=55
Reply from 64.34.119.12: bytes=32 time=79ms TTL=55
Reply from 64.34.119.12: bytes=32 time=81ms TTL=55
Reply from 64.34.119.12: bytes=32 time=81ms TTL=55
Request timed out.
Reply from 64.34.119.12: bytes=32 time=79ms TTL=55
Request timed out.
Reply from 64.34.119.12: bytes=32 time=79ms TTL=55
Reply from 64.34.119.12: bytes=32 time=80ms TTL=55
Reply from 64.34.119.12: bytes=32 time=79ms TTL=55

Ping statistics for 64.34.119.12:
    Packets: Sent = 58, Received = 53, Lost = 5 (8% loss),
Approximate round trip times in milli-seconds:
    Minimum = 78ms, Maximum = 96ms, Average = 80ms
Control-C
^C

I've let this pinging run for a while on my laptop and after running for about a minute, was getting ~15% packet loss.

Running a much longer ping test (30 minutes):

Ping statistics for 64.34.119.12:
    Packets: Sent = 1818, Received = 1614, Lost = 204 (11% loss),
Approximate round trip times in milli-seconds:
    Minimum = 78ms, Maximum = 238ms, Average = 81ms

Running WinMTR, it is as I had expected:

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                                       * -    1 |  155 |  154 |    0 |    1 |   13 |    0 |
|                                       * -    0 |  161 |  161 |    6 |   10 |   28 |   11 |
|                                       * -    0 |  161 |  161 |    7 |   12 |  160 |    8 |
|                                       * -    0 |  161 |  161 |    8 |   11 |   25 |   10 |
|     10ge-ten1-1.dal-eqx-cor-2.peer1.net -    0 |  161 |  161 |   38 |   44 |  251 |   39 |
|    10ge-ten2-1.atl-telx-cor-1.peer1.net -    0 |  160 |  160 |   58 |   62 |  125 |   60 |
|  10ge-ten1-1.atl-101mar-cor-1.peer1.net -    0 |  160 |  160 |   58 |   63 |  169 |   62 |
|   10ge.xe-1-0-0.wdc-eqx-dis-1.peer1.net -    0 |  160 |  160 |   72 |   78 |  149 |   75 |
|     10ge.ten1-2.wdc-sp2-cor-1.peer1.net -    0 |  160 |  160 |   73 |   77 |  214 |   74 |
|                         216.187.120.254 -    0 |  161 |  161 |   72 |   76 |  134 |   74 |
|  10ge.xe-2-0-0.nyc-telx-dis-1.peer1.net -    0 |  161 |  161 |   79 |   83 |  130 |   81 |
|  10ge.xe-0-0-0.nyc-telx-dis-2.peer1.net -    0 |  161 |  161 |   78 |   81 |  103 |   82 |
|    oc48-po3-0.nyc-75bre-dis-1.peer1.net -    0 |  160 |  160 |   78 |   82 |  193 |   79 |
|                gwny01.stackoverflow.com -    0 |  161 |  161 |   78 |   81 |  104 |   83 |
|                       stackoverflow.com -    8 |  126 |  117 |    0 |   81 |  119 |   81 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

(I've removed my host names for the first 4 hops, I could provide them if needed)

Could anyone else verify this?

Improve this question
4
  • Yup, I can repro from Austin, TX. I get about a 25% packet loss. I'm relatively surprised that the number is so high; the site has seemed fairly normal to me. Although I've seen more than a couple "site offline" messages here on Meta over the last few hours.
    – Cody Gray
    Commented Jul 12, 2011 at 8:37
  • @cody those are just normal deploys. Meta is unshielded as it runs on one server and not a web farm by design.
    – Jeff Atwood
    Commented Jul 12, 2011 at 8:51
  • Unfortunately ping isn't so useful in diagnosing where the issue is. All these pings are telling us is that there is some sort of packet loss between you and us, not where it actually is there can be an issue anywhere in between (or also our rate limiting kicking in). Can you try using the progrom MTR (Linux - bitwizard.nl/mtr ; Windows - winmtr.net) and updating with the output from that. It'll help us track down the issue.
    – Zypher Mod
    Commented Jul 12, 2011 at 15:39
  • @Zypher: Updated. It appears that it's coming from the server.
    – Jeff Mercado
    Commented Jul 12, 2011 at 18:30

4 Answers 4

Reset to default
3

We rate limit ICMP on the whole, not just per client ip (this is not what I thought we were doing, but it turns out it is what we are doing).

I am looking into better iptables rules so ICMP tests will actually be helpful.

Limit ICMP Per Source IP with IPTables

Improve this answer
0
3

I can definitely repro, from here in El Cerrito, CA. Packet loss 20%, eek!

Pinging stackoverflow.com [64.34.119.12] with 32 bytes of da
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Reply from 64.34.119.12: bytes=32 time=91ms TTL=46
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Reply from 64.34.119.12: bytes=32 time=93ms TTL=46
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Reply from 64.34.119.12: bytes=32 time=93ms TTL=46
Reply from 64.34.119.12: bytes=32 time=95ms TTL=46
Reply from 64.34.119.12: bytes=32 time=93ms TTL=46
Request timed out.
Request timed out.
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Reply from 64.34.119.12: bytes=32 time=91ms TTL=46
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Reply from 64.34.119.12: bytes=32 time=90ms TTL=46
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Reply from 64.34.119.12: bytes=32 time=93ms TTL=46
Reply from 64.34.119.12: bytes=32 time=98ms TTL=46
Request timed out.
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Reply from 64.34.119.12: bytes=32 time=90ms TTL=46
Reply from 64.34.119.12: bytes=32 time=91ms TTL=46
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Reply from 64.34.119.12: bytes=32 time=95ms TTL=46
Reply from 64.34.119.12: bytes=32 time=90ms TTL=46
Reply from 64.34.119.12: bytes=32 time=93ms TTL=46
Reply from 64.34.119.12: bytes=32 time=94ms TTL=46
Request timed out.
Reply from 64.34.119.12: bytes=32 time=91ms TTL=46
Reply from 64.34.119.12: bytes=32 time=91ms TTL=46
Reply from 64.34.119.12: bytes=32 time=91ms TTL=46
Reply from 64.34.119.12: bytes=32 time=95ms TTL=46
Reply from 64.34.119.12: bytes=32 time=96ms TTL=46
Reply from 64.34.119.12: bytes=32 time=93ms TTL=46
Reply from 64.34.119.12: bytes=32 time=91ms TTL=46
Reply from 64.34.119.12: bytes=32 time=109ms TTL=46
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Request timed out.
Reply from 64.34.119.12: bytes=32 time=90ms TTL=46
Reply from 64.34.119.12: bytes=32 time=91ms TTL=46
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Request timed out.
Request timed out.
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Reply from 64.34.119.12: bytes=32 time=90ms TTL=46
Reply from 64.34.119.12: bytes=32 time=91ms TTL=46
Reply from 64.34.119.12: bytes=32 time=91ms TTL=46
Request timed out.
Request timed out.
Reply from 64.34.119.12: bytes=32 time=93ms TTL=46
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Reply from 64.34.119.12: bytes=32 time=108ms TTL=46
Request timed out.
Request timed out.
Reply from 64.34.119.12: bytes=32 time=90ms TTL=46
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Request timed out.
Reply from 64.34.119.12: bytes=32 time=92ms TTL=46
Request timed out.

Ping statistics for 64.34.119.12:
    Packets: Sent = 62, Received = 49, Lost = 13 (20% loss),
Approximate round trip times in milli-seconds:
    Minimum = 90ms, Maximum = 109ms, Average = 92ms
Improve this answer
2

Seems to be a global problem, I get 23% packet loss from Germany

--- stackoverflow.com ping statistics ---
200 packets transmitted, 154 received, 23% packet loss, time 199276ms
rtt min/avg/max/mdev = 106.675/107.993/136.372/3.295 ms
Improve this answer
0

I can repro, pinging from my server in St. Louis and from my office in the UK.

From St. Louis data centre:

--- stackoverflow.com ping statistics ---
192 packets transmitted, 179 received, 6% packet loss, time 191015ms
rtt min/avg/max/mdev = 26.675/27.300/44.304/1.593 ms

(On my first, briefer test it was 4%.)

From UK office (DSL):

--- stackoverflow.com ping statistics ---
151 packets transmitted, 49 received, 67% packet loss, time 150364ms
rtt min/avg/max/mdev = 109.276/111.879/164.440/7.648 ms

(On my first, briefer test it was 9%.)

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .