11

It appears that ns3.serverfault.com (Oregon?) is not responding to DNS queries. ns1 and ns2 seem to be fine. This is causing some users to have intermittent trouble reaching Stack Exchange sites.

The nameservers being considered are:

$ host -t ns stackexchange.com. ns1.serverfault.com.
Using domain server:
Name: ns1.serverfault.com.
Address: 198.252.206.80#53
Aliases: 

stackexchange.com name server ns2.serverfault.com.
stackexchange.com name server ns3.serverfault.com.
stackexchange.com name server ns1.serverfault.com.

$ host -t ns stackexchange.com. ns2.serverfault.com.
Using domain server:
Name: ns2.serverfault.com.
Address: 198.252.206.81#53
Aliases: 

stackexchange.com name server ns2.serverfault.com.
stackexchange.com name server ns1.serverfault.com.
stackexchange.com name server ns3.serverfault.com.

ns1 and ns2 are not having any issues that I can reproduce. But ns3...

$ host -v ham.stackexchange.com ns3.serverfault.com.
Trying "ham.stackexchange.com"
;; connection timed out; no servers could be reached

A nice traceroute from Germany for you:

$ traceroute ns3.serverfault.com.
traceroute to ns3.serverfault.com. (69.59.196.217), 30 hops max, 60 byte packets
 1  router.nodedeploy.com (62.113.196.1)  0.309 ms  0.237 ms  0.388 ms
 2  30gigabitethernet4-3.core1.fra1.he.net (80.81.192.172)  0.837 ms  0.760 ms  0.851 ms
 3  10gigabitethernet10-2.core1.par2.he.net (72.52.92.26)  70.199 ms *  68.602 ms
 4  10gigabitethernet15-1.core1.ash1.he.net (184.105.213.93)  156.328 ms  156.464 ms  155.929 ms
 5  10gigabitethernet12-7.core1.chi1.he.net (184.105.213.245)  110.817 ms  110.702 ms  114.984 ms
 6  100gigabitethernet13-1.core1.msp1.he.net (184.105.223.178)  178.957 ms  178.647 ms *
 7  100gigabitethernet9-1.core1.sea1.he.net (184.105.223.193)  218.910 ms * *
 8  * * 10gigabitethernet1-3.core1.pdx1.he.net (72.52.92.10)  216.129 ms
 9  * peak-internet-llc.gigabitethernet2-1.core1.pdx1.he.net (216.218.147.226)  215.911 ms  215.654 ms
10  69.59.218.98 (69.59.218.98)  225.001 ms  224.831 ms  224.634 ms
11  ge-0-0-0-cvo-core2.peak.org (69.59.218.197)  156.471 ms  158.257 ms  154.112 ms
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  *^C

Another nice traceroute from the US for you:

$ traceroute ns3.serverfault.com.
traceroute to ns3.serverfault.com. (69.59.196.217), 30 hops max, 60 byte packets
 1  hosted.by.leaseweb.com (198.7.57.254)  2.688 ms  3.466 ms  3.474 ms
 2  108.59.15.21 (108.59.15.21)  0.394 ms 108.59.15.23 (108.59.15.23)  0.330 ms 108.59.15.21 (108.59.15.21)  0.399 ms
 3  108.59.15.154 (108.59.15.154)  0.861 ms ae2.cr1.wdc1.leaseweb.net (108.59.15.118)  0.307 ms 108.59.15.154 (108.59.15.154)  0.861 ms
 4  38.88.128.1 (38.88.128.1)  0.525 ms 38.88.128.53 (38.88.128.53)  0.567 ms 38.88.128.17 (38.88.128.17)  0.591 ms
 5  te0-13-0-31.ccr41.iad02.atlas.cogentco.com (154.54.87.221)  2.071 ms  2.073 ms  2.056 ms
 6  be2112.mpd22.dca01.atlas.cogentco.com (154.54.5.234)  2.157 ms be2176.ccr21.dca01.atlas.cogentco.com (154.54.41.54)  3.019 ms be2112.mpd22.dca01.atlas.cogentco.com (154.54.5.234)  2.286 ms
 7  be2153.ccr22.ord01.atlas.cogentco.com (154.54.30.126)  21.347 ms be2154.mpd21.ord01.atlas.cogentco.com (154.54.30.198)  21.486 ms be2155.mpd22.ord01.atlas.cogentco.com (154.54.31.78)  21.400 ms
 8  be2157.ccr22.mci01.atlas.cogentco.com (154.54.6.117)  33.545 ms be2159.mpd22.mci01.atlas.cogentco.com (154.54.24.81)  33.413 ms be2158.mpd21.mci01.atlas.cogentco.com (154.54.7.129)  33.576 ms
 9  be2129.ccr21.den01.atlas.cogentco.com (154.54.26.113)  45.519 ms  45.574 ms  45.504 ms
10  be2126.ccr21.slc01.atlas.cogentco.com (154.54.25.66)  55.829 ms be2127.ccr21.slc01.atlas.cogentco.com (154.54.25.70)  56.034 ms be2126.ccr21.slc01.atlas.cogentco.com (154.54.25.66)  56.004 ms
11  te7-2.ccr01.boi01.atlas.cogentco.com (154.54.40.94)  62.510 ms  62.510 ms te4-2.ccr01.boi01.atlas.cogentco.com (154.54.40.90)  62.811 ms
12  te4-4.ccr01.pdx02.atlas.cogentco.com (154.54.40.97)  73.713 ms  73.637 ms  73.659 ms
13  154.54.89.57 (154.54.89.57)  73.649 ms  73.500 ms  73.573 ms
14  38.104.104.194 (38.104.104.194)  96.401 ms  96.511 ms  96.324 ms
15  EUGNOR53CR01.lsnetworks.net (216.110.192.42)  82.323 ms  82.436 ms  83.034 ms
16  206-192-226-18.lsnetworks.net (206.192.226.18)  83.959 ms  83.196 ms  83.050 ms
17  * ge-0-0-0-cvo-core2.peak.org (69.59.218.197)  86.753 ms  86.731 ms
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  *^C
12
  • It also seems to make a difference whether you're using UDP or TCP for queries. TCP works for me against ns2, but UDP does not (times out). Other DNS works fine over both TCP and UDP so it shouldn't be a firewalling issue on my end. Also confirmed: same behavior from a VPS running on a totally separate network. dig @ns2.serverfault.com. +tcp ham.stackexchange.com any +norec returns a result, change +tcp to +notcp and it times out.
    – user
    Nov 17, 2013 at 20:28
  • I had no issues with UDP toward ns1 or ns2. Nov 17, 2013 at 20:30
  • Where are you getting ns3 from? It's not responding yes...but it's also not currently in rotation (hasn't been for weeks): who.is/whois/ham.stackexchange.com Nov 17, 2013 at 20:48
  • @NickCraver Updated the question with that information. I see they aren't in the whois record but they are still being served as valid NS records. Nov 17, 2013 at 20:50
  • From Amsterdam, dig NS stackexchange.com does give me ns1, ns2 and ns3, @Nick. Not sure if that means anything... :-)
    – Arjan
    Nov 17, 2013 at 20:53
  • Yeah, a caching nameserver may pick up those NS records and begin using them. What's in whois is just glue. Nov 17, 2013 at 20:54
  • @MichaelHampton need to find out who's serving that for you...ns3 has been out of rotation for over a month, that's a pretty bad cache. Nov 17, 2013 at 20:55
  • 3
    @NickCraver Your nameservers ns1 and ns2 are serving them. Nov 17, 2013 at 20:57
  • 1
    @MichaelHampton found the issue, had to be external to see it - a DNS update is going out shortly Nov 17, 2013 at 21:05
  • You're using split horizon DNS?! :) Nov 17, 2013 at 21:14
  • @MichaelHampton can you give me an update 3+ minutes from now and see if you're reporting good on our main domains? Nov 17, 2013 at 21:17
  • @NickCraver Your TTL is only 300. I see now that ns3 is no longer being returned as a NS record, by either ns1 or ns2. Nov 17, 2013 at 21:18

3 Answers 3

12

ns3 was indeed in the zone files for quite some time after it was offline for an Oregon data center rebuild. I'm not sure why this became an issue today and not earlier, but it was indeed wrong on most of our external zones.

ns3.serverfault.com has been removed from our primary zones (the trilogy, stackexchange.com, sstatic.net and stackauth.com) as an immediate remedy. Steve is aiming to get ns3 back online tonight in Oregon (it's close to ready) then we'll revert this change and get queries flowing through it again.

1
  • It's probably been an issue all this time; I've had some weird connectivity issues I couldn't explain over the past few weeks that "cleared up" after I hit reload. I had written it off to a crappy Internet connection until someone reported this to me today. Nov 17, 2013 at 21:26
2

Ditto from Ireland.

$ traceroute --resolve-hostnames ns1.serverfault.com
traceroute to ns1.serverfault.com (198.252.206.80), 64 hops max
 1   192.168.1.254 (router) 0.732ms 0.534ms 0.581ms 
 2   159.134.155.8 (b-ras1.pgs.portlaoise.eircom.net) 23.469ms 23.622ms 23.571ms 
 3   86.43.247.133 (tenge-5-1-1.pe1.pgs.lmk-pgs.eircom.net) 24.503ms 24.158ms 24.542ms 
 4   86.43.253.41 (tenge-7-2-1.core1.dbn.core.eircom.net) 34.497ms 24.882ms 27.069ms 
 5   *  *  86.43.252.30 (lag-1.core2.dbn.core.eircom.net) 21.793ms 
 6   *  86.43.253.126 (tenge-1-2-1.pe1.the.the-thn.eircom.net) 31.701ms 35.714ms 
 7   86.43.244.190 (ge7-1-0.corea.the.london.eircom.net) 35.707ms 35.069ms 35.664ms 
 8   195.69.144.122 (er1.ams1.nl.above.net) 76.344ms 44.198ms 44.562ms 
 9   64.125.32.94 (ae3.cr1.ams5.nl.above.net) 44.388ms 43.793ms 44.227ms 
 10   64.125.27.185 (xe-0-2-0.cr2.lga5.us.above.net) 124.224ms 124.928ms 125.110ms 
 11   64.125.31.246 (ae3.er4.lga5.us.above.net) 129.858ms 130.416ms 158.377ms 
 12   208.184.110.78 (208.184.110.78.IPYX-072053-004-ZYO.above.net) 195.357ms 193.333ms 193.307ms 
 13   216.52.95.65 (border1.po2-20g-bbnet2.nym008.pnap.net) 188.711ms 192.732ms 192.369ms 
 14   74.201.252.22 (stackexchange-1.border1.nym008.pnap.net) 185.968ms 181.150ms 180.815ms 
 15   198.252.206.80 (stackoverflow.com) 190.860ms 192.239ms 194.860ms 
$ traceroute --resolve-hostnames ns2.serverfault.com
traceroute to ns2.serverfault.com (198.252.206.81), 64 hops max
 1   192.168.1.254 (router) 0.793ms 0.534ms 0.531ms 
 2   159.134.155.8 (b-ras1.pgs.portlaoise.eircom.net) 26.821ms 24.370ms 24.151ms 
 3   86.43.247.129 (tenge-4-2-1.pe1.pgs.lmk-pgs.eircom.net) 24.103ms 24.203ms 23.733ms 
 4   86.43.253.41 (tenge-7-2-1.core1.dbn.core.eircom.net) 27.073ms 25.563ms 24.574ms 
 5   *  *  86.43.252.30 (lag-1.core2.dbn.core.eircom.net) 21.097ms 
 6   *  86.43.253.126 (tenge-1-2-1.pe1.the.the-thn.eircom.net) 30.926ms 35.479ms 
 7   86.43.244.190 (ge7-1-0.corea.the.london.eircom.net) 35.378ms 35.103ms 35.633ms 
 8   195.69.144.122 (er1.ams1.nl.above.net) 43.752ms 43.844ms 44.628ms 
 9   64.125.32.94 (ae3.cr1.ams5.nl.above.net) 44.131ms 71.743ms 44.581ms 
 10   64.125.27.185 (xe-0-2-0.cr2.lga5.us.above.net) 125.234ms 124.193ms 146.593ms 
 11   64.125.31.246 (ae3.er4.lga5.us.above.net) 130.337ms 129.439ms 129.110ms 
 12   208.184.110.78 (208.184.110.78.IPYX-072053-004-ZYO.above.net) 193.785ms 196.420ms 195.081ms 
 13   216.52.95.65 (border1.po2-20g-bbnet2.nym008.pnap.net) 306.407ms 269.392ms 255.813ms 
 14   74.201.252.22 (stackexchange-1.border1.nym008.pnap.net) 193.991ms 196.166ms 197.090ms 
 15   198.252.206.81 (stackoverflow.com) 203.141ms 199.835ms 197.569ms 
$ traceroute --resolve-hostnames ns3.serverfault.com
traceroute to ns3.serverfault.com (69.59.196.217), 64 hops max
 1   192.168.1.254 (router) 0.602ms 0.655ms 0.558ms 
 2   159.134.155.8 (b-ras1.pgs.portlaoise.eircom.net) 24.585ms 25.901ms 23.844ms 
 3   86.43.247.129 (tenge-4-2-1.pe1.pgs.lmk-pgs.eircom.net) 24.103ms 23.800ms 26.047ms 
 4   86.43.252.249 (tenge-7-1-1.core1.bdt.core.eircom.net) 27.797ms 30.174ms 26.210ms 
 5   *  *  86.43.253.82 (tenge-1-2-1.pe1.thn.the-thn.eircom.net) 37.107ms 
 6   86.43.244.194 (ge7-1-0.corea.thn.london.eircom.net) 40.538ms 40.513ms 39.845ms 
 7   195.66.224.21 (40gigabitethernet1-1.core1.lon1.he.net) 48.714ms 40.431ms 40.524ms 
 8   72.52.92.241 (10gigabitethernet10-4.core1.nyc4.he.net) 109.638ms 109.324ms 117.975ms 
 9   184.105.223.161 (100gigabitethernet7-2.core1.chi1.he.net) 126.625ms 126.663ms 155.462ms 
 10   184.105.223.178 (100gigabitethernet13-1.core1.msp1.he.net) 134.729ms 141.726ms 134.223ms 
 11   184.105.223.193 (100gigabitethernet9-1.core1.sea1.he.net) 167.032ms 166.448ms 166.636ms 
 12   72.52.92.10 (10gigabitethernet1-3.core1.pdx1.he.net) 170.096ms 177.125ms 174.876ms 
 13   216.218.147.226 (peak-internet-llc.gigabitethernet2-1.core1.pdx1.he.net) 170.274ms 170.035ms 169.965ms 
 14   69.59.218.98 (69.59.218.98) 176.847ms 176.310ms 176.638ms 
 15   69.59.218.197 (ge-0-0-0-cvo-core2.peak.org) 177.803ms 180.803ms 177.287ms 
 16   *  *  * 
 17   *  *  * 
 18   *  *  * 
 19   *  *  * 
 20   *  *  * 
 21   *  *  * 
 22   *  *  * 
 23   *  *  * 
 24   *  *  * 
 25   *  *  * 
 26   *  *  * 
 27   *  *  * 
 28   *  *  * 
 29   *  *  * 
 30   *  *  * 
 31   *  *  * 
 32   *  *  * 
 33   *  *  * 
 34   *  *  * 
 35   *  *  * 
 36   *  *  * 
 37   *  *  * 
 38   *  *  * 
 39   *  *  * 
 40   *  *  * 
 41   *  *  * 
 42   *  *  * 
 43   *  *  * 
 44   *  *  * 
 45   *  *  * 
 46   *  *  * 
 47   *  *  * 
 48   *  *  * 
 49   *  *  * 
 50   *  *  * 
 51   *  *  * 
 52   *  *  * 
 53   *  *  * 
 54   *  *  * 
 55   *  *  * 
 56   *  *  * 
 57   *  *  * 
 58   *  *  * 
 59   *  *  * 
 60   *  *  * 
 61   *  *  * 
 62   *  *  * 
 63   *  *  * 
 64   *  *  * 
2

The WHOIS data only shows two nameservers, while the lookups show three. Looks like you have ns1/ns2/ns3 as NS records in your zone file..

$ host -t ns stackexchange.com
stackexchange.com name server ns2.serverfault.com.
stackexchange.com name server ns1.serverfault.com.
stackexchange.com name server ns3.serverfault.com.

$ host serverfault.com ns3.serverfault.com
;; connection timed out; no servers could be reached


$ sudo tcptraceroute -p 56 ns3.serverfault.com
traceroute to ns3.serverfault.com (69.59.196.217), 30 hops max, 60 byte packets
 1  195.0.236.241 (195.0.236.241)  0.625 ms  0.641 ms  0.591 ms
 2  c51009DEB.inet.catch.no (81.0.157.235)  0.277 ms  0.244 ms  0.266 ms
 3  c51009DE5.inet.catch.no (81.0.157.229)  0.574 ms  0.646 ms  0.537 ms
 4  te2-2-1.br2.stcy.as2116.net (195.0.242.90)  7.322 ms *  7.422 ms
 5  10gigabitethernet1-2.core1.sto1.he.net (194.68.123.187)  7.702 ms  7.690 ms  7.626 ms
 6  10gigabitethernet3-1.core1.cph1.he.net (184.105.223.206)  15.424 ms  15.422 ms  15.321 ms
 7  10gigabitethernet16-1.core1.fra1.he.net (184.105.223.201)  39.016 ms  49.758 ms  49.755 ms
 8  10gigabitethernet10-2.core1.par2.he.net (72.52.92.26)  50.293 ms  50.834 ms  49.822 ms
 9  10gigabitethernet15-1.core1.ash1.he.net (184.105.213.93)  127.529 ms  132.159 ms  128.667 ms
10  10gigabitethernet12-7.core1.chi1.he.net (184.105.213.245)  146.695 ms  147.504 ms  145.187 ms
11  100gigabitethernet13-1.core1.msp1.he.net (184.105.223.178)  153.434 ms  161.820 ms  157.768 ms
12  100gigabitethernet9-1.core1.sea1.he.net (184.105.223.193)  197.122 ms  188.143 ms  194.579 ms
13  10gigabitethernet1-3.core1.pdx1.he.net (72.52.92.10)  189.380 ms  195.660 ms  199.821 ms
14  peak-internet-llc.gigabitethernet2-1.core1.pdx1.he.net (216.218.147.226)  190.027 ms  191.502 ms  191.984 ms
15  69.59.218.98 (69.59.218.98)  198.209 ms  195.632 ms  196.191 ms
16  * * *
17  * * *
18  *^C

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .