6

This is what my inbox looks like:

enter image description here

When I opened it today it had a red (3) and a blue (1). The notifications list is working, but I only see one out of three inbox entries.

And this is what I see when I visit my network account (to check my full inbox):

enter image description here

This seems to be happening to a lot of users (eg this guy, chem account).

Looks like SE.com (which IIRC provides the inbox as well) is down :/

Improve this question
6
  • 1
    Jeff's page is working, but it's empty.
    – Manishearth
    Mar 17, 2013 at 8:44
  • Jeff's page is not entirely empty, see stackexchange.com/users/1/jeff-atwood?tab=accounts sorry for not formatting that. Yes, it seems as though something is amiss e.g. I just noticed that there are users who show no network profile at all, 404, who are active as of today on more than one SE site. My inbox is entirely blank, although the other three tabs (including notifications, which is not generic) of the pull down menu are working.
    – Ellie Kesselman
    Mar 17, 2013 at 8:52
  • I thought it might be a problem with OpenID, so I logged out, cleared all the SE login credentials stored in my browser, and logged back in again. It didn't make a difference, as I still only see my account number, and my subscription to EL&U SE newsletter. (I did notice how few people subscribe to SE newsletters as I ambled around! That was surprising.)
    – Ellie Kesselman
    Mar 17, 2013 at 9:32
  • The inbox is empty for me too, but my profile page isn't totally empty as of yet.
    – hjpotter92
    Mar 17, 2013 at 9:50
  • 1
    @BackinaFlash: No fair -_-
    – Manishearth
    Mar 17, 2013 at 10:00
  • Accounts appear to be fine now, but old inbox items still not showing.
    – Shadow Wizard Love Zelda
    Mar 17, 2013 at 10:54

2 Answers 2

Reset to default
14

This should be back online, we've been attacked in a variety of ways the last 12 hours, it's been fun.

This was not so much a direct attack as a very abusive crawl bot from china (not even related to a search engine). This was causing us to load profiles on SE.com like crazy, tying up the API backend that serves those default profiles and inboxes, the offenders and their entire /20 IP range have been banned from our network, dropped packet style.

Everything should be running normally now and we'll continue to keep an eye on things through the day.

Improve this answer
12
  • 4
    But...but...I thought you guys had the capacity to handle this! I am dissapoint. ;-)
    – Manishearth
    Mar 17, 2013 at 12:10
  • China.. why I'm not surprised?
    – Shadow Wizard Love Zelda
    Mar 17, 2013 at 12:14
  • @Manishearth - turns out we do, we just had some sanity limits set beneath the thresholds that magnitude of DDoS needs. We've adjusted accordingly and our load balancers can laugh at the next one.
    – Nick Craver
    Mar 17, 2013 at 12:15
  • 1
    @NickCraver: So...you had rate limited the rate limiter?
    – Manishearth
    Mar 17, 2013 at 12:18
  • 1
    Good question, @ShaWizDowArd, why?
    – Arjan
    Mar 17, 2013 at 12:37
  • 2
    Even rate limiters need rate limiting or they'll just rate limit all the rate limiters.
    – user50049
    Mar 17, 2013 at 12:39
  • 2
    @Arjan recently when I hear the word "hacker" the first word that comes to mind is China. Big, severe and organized attacks on major US companies came from China, likely from their government itself. They probably use Stack Exchange sites as "test case" that's why no surprise in my side. :)
    – Shadow Wizard Love Zelda
    Mar 17, 2013 at 12:51
  • Needs moar caching.
    – casperOne
    Mar 17, 2013 at 14:02
  • One of the mods found this DIY.SE clone. It seems read-only, but is reflecting the main DIY site. And is hosted in China. Could these two be connected? Perfectly cloning a site requires some pretty heavy scraping..
    – Manishearth
    Mar 23, 2013 at 18:30
  • @Manishearth Technically it is Vietnam... networksolutions.com/whois/results.jsp?domain=diy101.net
    – maple_shaft
    Mar 23, 2013 at 18:32
  • @maple_shaft diy.net is registered in Vietnam, and the IP address 174.120.202.125 is located in a datacenter in TX.US. Where do you see a China connection?
    – Gilles 'SO- stop being evil'
    Mar 23, 2013 at 18:34
  • My bad, the talk about China confused me.. Sorry about that..
    – Manishearth
    Mar 23, 2013 at 18:35
1

Maybe it is due to a vagary of system maintenance. It has happened before, with similar unintended results, specifically, blanked user inboxes.

Per My inbox items have been deleted:

I gracefully brought web server 7 out of rotation... this is needed because the internal API backends used for the inbox in places need them, and they're unchecked in HAProxy. Then I not-so-gracefully shut down web server 6 instead of 7...

I purged your inbox cache, it'll return to normal for anyone else this happened to as their cache expires.

After thought: There were prior instances of blanked inboxes. The root cause of that outage was addressed in late October 2010. It was associated with SE site-wide unavailability and down time, whereas this is not, so I don't think that problem is as likely to be the reason for our current much more minor concern.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .