This is what my inbox looks like:

enter image description here

When I opened it today it had a red (3) and a blue (1). The notifications list is working, but I only see one out of three inbox entries.

And this is what I see when I visit my network account (to check my full inbox):

enter image description here

This seems to be happening to a lot of users (eg this guy, chem account).

Looks like SE.com (which IIRC provides the inbox as well) is down :/

  • 1
    Jeff's page is working, but it's empty. – Manishearth Mar 17 '13 at 8:44
  • Jeff's page is not entirely empty, see stackexchange.com/users/1/jeff-atwood?tab=accounts sorry for not formatting that. Yes, it seems as though something is amiss e.g. I just noticed that there are users who show no network profile at all, 404, who are active as of today on more than one SE site. My inbox is entirely blank, although the other three tabs (including notifications, which is not generic) of the pull down menu are working. – Ellie Kesselman Mar 17 '13 at 8:52
  • I thought it might be a problem with OpenID, so I logged out, cleared all the SE login credentials stored in my browser, and logged back in again. It didn't make a difference, as I still only see my account number, and my subscription to EL&U SE newsletter. (I did notice how few people subscribe to SE newsletters as I ambled around! That was surprising.) – Ellie Kesselman Mar 17 '13 at 9:32
  • The inbox is empty for me too, but my profile page isn't totally empty as of yet. – hjpotter92 Mar 17 '13 at 9:50
  • 1
    @BackinaFlash: No fair -_- – Manishearth Mar 17 '13 at 10:00
  • Accounts appear to be fine now, but old inbox items still not showing. – Shadow Wizard is Vaccinating Mar 17 '13 at 10:54

This should be back online, we've been attacked in a variety of ways the last 12 hours, it's been fun.

This was not so much a direct attack as a very abusive crawl bot from china (not even related to a search engine). This was causing us to load profiles on SE.com like crazy, tying up the API backend that serves those default profiles and inboxes, the offenders and their entire /20 IP range have been banned from our network, dropped packet style.

Everything should be running normally now and we'll continue to keep an eye on things through the day.

  • 4
    But...but...I thought you guys had the capacity to handle this! I am dissapoint. ;-) – Manishearth Mar 17 '13 at 12:10
  • China.. why I'm not surprised? – Shadow Wizard is Vaccinating Mar 17 '13 at 12:14
  • @Manishearth - turns out we do, we just had some sanity limits set beneath the thresholds that magnitude of DDoS needs. We've adjusted accordingly and our load balancers can laugh at the next one. – Nick Craver Mar 17 '13 at 12:15
  • 1
    @NickCraver: So...you had rate limited the rate limiter? – Manishearth Mar 17 '13 at 12:18
  • 1
    Good question, @ShaWizDowArd, why? – Arjan Mar 17 '13 at 12:37
  • 2
    Even rate limiters need rate limiting or they'll just rate limit all the rate limiters. – Tim Post Mar 17 '13 at 12:39
  • 2
    @Arjan recently when I hear the word "hacker" the first word that comes to mind is China. Big, severe and organized attacks on major US companies came from China, likely from their government itself. They probably use Stack Exchange sites as "test case" that's why no surprise in my side. :) – Shadow Wizard is Vaccinating Mar 17 '13 at 12:51
  • Needs moar caching. – casperOne Mar 17 '13 at 14:02
  • One of the mods found this DIY.SE clone. It seems read-only, but is reflecting the main DIY site. And is hosted in China. Could these two be connected? Perfectly cloning a site requires some pretty heavy scraping.. – Manishearth Mar 23 '13 at 18:30
  • @Manishearth Technically it is Vietnam... networksolutions.com/whois/results.jsp?domain=diy101.net – maple_shaft Mar 23 '13 at 18:32
  • @maple_shaft diy.net is registered in Vietnam, and the IP address is located in a datacenter in TX.US. Where do you see a China connection? – Gilles 'SO- stop being evil' Mar 23 '13 at 18:34
  • My bad, the talk about China confused me.. Sorry about that.. – Manishearth Mar 23 '13 at 18:35

Maybe it is due to a vagary of system maintenance. It has happened before, with similar unintended results, specifically, blanked user inboxes.

Per My inbox items have been deleted:

I gracefully brought web server 7 out of rotation... this is needed because the internal API backends used for the inbox in places need them, and they're unchecked in HAProxy. Then I not-so-gracefully shut down web server 6 instead of 7...

I purged your inbox cache, it'll return to normal for anyone else this happened to as their cache expires.

After thought: There were prior instances of blanked inboxes. The root cause of that outage was addressed in late October 2010. It was associated with SE site-wide unavailability and down time, whereas this is not, so I don't think that problem is as likely to be the reason for our current much more minor concern.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .