8

The Verizon techs have the link to this post, so please add any information you can here. all additional info is helpful at this point

Chat seems to be down (requests time out). Kyle, you broke stuff again!!

Unless my link to the tubes is malfunctioning... Either way, I'm having withdrawals and it's not pretty.

And really, I was going to leave it at that, but apparently that doesn't "meet our quality standards." What does that even mean?

This is affecting all services run out of our OR DC. If you are on Verizon FiOS on the East coast you'll have issues getting to:

  • chat
  • data.se
  • blog.so

-Zypher

I've turned this question community wiki. Anyone having issue please do a trace route to chat.stackexchange.com and add the IP/Reverse dns of the router that it dies at below.

Known bad routers: 

g10-0-8-112.hrbgpa-lcr-01.verizon-gni.net [130.81.184.148]
G0-10-1-1.PHLAPA-LCR-22.verizon-gni.net [130.81.183.18]
g11-0-5-1026.nycmny-lcr-10.verizon-gni.net [130.81.180.4]
L100.WASHDC-VFTTP-125.verizon-gni.net [173.66.170.1]
G0-13-1-5.NYCMNY-LCR-22.verizon-gni.net [130.81.182.78]
g0-2-5-5.nrflva-lcr-21.verizon-gni.net [130.81.140.228]
G0-5-3-1.PHLAPA-LCR-21.verizon-gni.net [130.81.193.192]
G14-0-1-1630.WASHDC-LCR-06.verizon-gni.net (130.81.146.120)

Traceroute output if ya got it:

  traceroute to chat.stackexchange.com (69.59.196.211), 30 hops max, 40 byte packets
1  Wireless_Broadband_Router.home (192.168.1.1)  1.735 ms  0.606 ms  0.545 ms
2  L100.WASHDC-VFTTP-130.verizon-gni.net (96.231.130.1)  7.286 ms  6.926 ms  7.064 ms
3  G14-0-1-1630.WASHDC-LCR-06.verizon-gni.net (130.81.146.120)  8.613 ms !H *  9.221 ms !H

C:\Users\gbeech>tracert chat.stackexchange.com

Tracing route to chat.stackexchange.com [69.59.196.211]
over a maximum of 30 hops:

  1     8 ms    <1 ms    <1 ms  10.10.10.1
  2     4 ms     4 ms     3 ms  l300.nycmny-vfttp-226.verizon-gni.net [173.56.120.1]
  3  g11-0-5-1026.nycmny-lcr-10.verizon-gni.net [130.81.180.4]  reports: Destination host unreachable.

Trace complete.

Tracing route to chat.stackexchange.com [69.59.196.211]
over a maximum of 30 hops:

  1     6 ms     5 ms     6 ms  Wireless_Broadband_Router.home [192.168.1.1]
  2    77 ms    53 ms    32 ms  L100.WASHDC-VFTTP-125.verizon-gni.net [173.66.17
0.1]
  3     *        *        *     Request timed out.
  4  69.59.196.211  reports: Destination net unreachable.

Trace complete.
Improve this question
20
  • I'm having issues too, but the problem seems to be somewhere in the middle. Connecting from my iPhone over 3G works fine, but no luck with my home connection. Good to know it's not just me.
    – Tim Stone
    Commented Jul 1, 2011 at 1:02
  • 3
    I guess it's working for the people downvoting this...
    – Holocryptic
    Commented Jul 1, 2011 at 1:07
  • Confirming that it is broken. We're looking into it.
    – Michael Pryor
    Commented Jul 1, 2011 at 1:15
  • If you do a trace route does it die at: g11-0-5-1026.nycmny-lcr-10.verizon-gni.net [130.81.180.4]
    – Zypher
    Commented Jul 1, 2011 at 1:17
  • Good deal guys. Keep on keepin on.
    – Holocryptic
    Commented Jul 1, 2011 at 1:19
  • 1
    @Zypher Mine dies at a Philly equivalent (G0-10-1-1.PHLAPA-LCR-22.verizon-gni.net [130.81.183.18]), but yeah.
    – Tim Stone
    Commented Jul 1, 2011 at 1:20
  • I emailed [email protected] . Didn't seem to bounce -- can't find contact info on IANA or whois.
    – Kyle Brandt
    Commented Jul 1, 2011 at 1:55
  • @Zypher my tracert seems to die on different hops each time. Not sure what's up...
    – Holocryptic
    Commented Jul 1, 2011 at 2:28
  • @holo you mind posting a couple of the endpoints it dies at?
    – Zypher
    Commented Jul 1, 2011 at 2:35
  • @Zypher I'm guessing it dies at endpoints that have ICMP replies turned off, as I have no IPs for them. The last endpoint I have is L100.WASHDC-VFTTP-125.verizon-gni.net [173.66.170.1] and 69.59.196.211 reports: Destination net unreachable. at the end
    – Holocryptic
    Commented Jul 1, 2011 at 2:38
  • @Zypher and that L100 endpoint is the next hop after my router...
    – Holocryptic
    Commented Jul 1, 2011 at 2:39
  • 3
    Yep looks like it's an outage with their routing on the East cost, anyone in Boston w/ FIOS want to join the party?
    – Zypher
    Commented Jul 1, 2011 at 3:01
  • Virginia Fios User, chat's down... I die here g0-2-5-5.nrflva-lcr-21.verizon-gni.net (130.81.140.228)
    – Jacob
    Commented Jul 1, 2011 at 5:07
  • @kyle Brandt @Zypher The NOC number for Verizon is 800-243-6994
    – Jacob
    Commented Jul 1, 2011 at 5:12
  • @thebeachException - THANKS! that's going in fu_verizon.txt
    – Zypher
    Commented Jul 1, 2011 at 5:29

3 Answers 3

Reset to default
8

Looks like Verizon is upgrading Fiber connections on the east coast and this is a side effect. They say it should be resolved in 2-4 hours, around 5AM Eastern Time (US).

Improve this answer
4
  • 2
    Ah, nice of them to inform us that they were going to break the most important part of the internet. Appreciate the heads up, and since I snuck a nap instead of idling on chat I guess it isn't so bad.
    – Tim Stone
    Commented Jul 1, 2011 at 6:35
  • @Tim You're only slightly wrong. This is a travesty
    – Holocryptic
    Commented Jul 1, 2011 at 11:36
  • 2
    @Holocryptic Indeed. It seems Verizon has a special interpretation of "2-4 hours."
    – Tim Stone
    Commented Jul 1, 2011 at 12:21
  • Nope still having issues here.
    – Jacob
    Commented Jul 1, 2011 at 16:26
3

It looks like the issue has been fixed.

Tracing route to chat.stackexchange.com [69.59.196.211]
over a maximum of 30 hops:

  1     3 ms     4 ms     3 ms  Wireless_Broadband_Router.home [192.168.1.1]
  2    20 ms    14 ms    16 ms  L100.WASHDC-VFTTP-125.verizon-gni.net [173.66.17
0.1]
  3    13 ms    25 ms    20 ms  G5-0-5-1725.WASHDC-LCR-07.verizon-gni.net [130.8
1.140.110]
  4    15 ms    13 ms    12 ms  so-12-1-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net
 [130.81.151.230]
  5    20 ms    19 ms    33 ms  0.so-2-3-0.XL3.IAD8.ALTER.NET [152.63.30.17]
  6   115 ms   105 ms   113 ms  0.ge-1-2-0.XT3.POR3.ALTER.NET [152.63.106.33]
  7   102 ms   104 ms   109 ms  GigabitEthernet6-0.GW7.POR3.ALTER.NET [152.63.10
6.54]
  8   102 ms   100 ms   101 ms  noanetoregon-por-gw.customer.alter.net [157.130.
179.94]
  9   101 ms   124 ms   100 ms  vl-12-PTLDORPBCR01.lsnetworks.net [216.110.192.2
1]
 10   105 ms   103 ms   108 ms  EUGNOR53CR01.lsnetworks.net [216.110.192.42]
 11   115 ms   111 ms   106 ms  206-192-226-18.lsnetworks.net [206.192.226.18]
 12   110 ms   125 ms   112 ms  ge-0-0-0-cvo-core2.peak.org [69.59.218.197]
 13   112 ms   131 ms   111 ms  vlan5-cvo-colo2.peak.org [69.59.218.226]
 14   112 ms   105 ms   112 ms  stackoverflow.com [69.59.196.211]

Trace complete.

Now I can carry on with the naked drunken debauchery in the privacy of my own home, instead of glancing furtively over my shoulder at work.

Improve this answer
1

I'm having the same problem:

tracert

  1     1 ms    <1 ms    <1 ms  Wireless_Broadband_Router.home [192.168.1.1]
  2     4 ms     3 ms     4 ms  L100.NYCMNY-VFTTP-169.verizon-gni.net [71.190.135.1]
  3     *     G0-13-1-5.NYCMNY-LCR-22.verizon-gni.net [130.81.182.78]  reports: Destination net unreachable.

Note: I have Verizon (obv.) and I love me some FIOS, so while this is a sad state right now Verizon does not suck (most of the time.)

Improve this answer

Not the answer you're looking for? Browse other questions tagged .