A deeper dive into the May 2019 security incident: blog post feedback

Question

We just posted an update to the security incident that happened back in May 2019 with technical details of what happened, how it happened, and the remediations we applied to prevent an incident like it from happening again. Here's a couple of excerpts from the post - first from the introduction:

On May 12th, 2019, at around 00:00 UTC, we were alerted to an unexpected privilege escalation for a new user account by multiple members of the community. A user that nobody recognised had gained moderator and developer level access across all of the sites in the Stack Exchange Network. Our immediate response was to revoke privileges and to suspend this account and then set in motion a process to identify and audit the actions that led to the event.

After initial discovery, we found that the escalation of privilege was just the tip of the iceberg and the attack had actually resulted in the exfiltration of our source code and the inadvertent exposure of the PII (email, real name, IP addresses) of 184 users of the Stack Exchange Network (all of whom were notified). Thankfully, none of the databases—neither public (read: Stack Exchange content) nor private (Teams, Talent, or Enterprise)—were exfiltrated. Additionally, there has been no evidence of any direct access to our internal network infrastructure, and at no time did the attacker ever have access to data in Teams, Talent, or Enterprise products.

And from the final paragraph:

This incident reminded us about some fundamental security practices that everyone should follow:

1. Log all your inbound traffic. We keep logs on all in-bound connections. This enabled all of our investigations. You can’t investigate what you don’t log.
2. Use 2FA. That remaining system that still uses legacy authentication can be your biggest vulnerability.
3. Guard secrets better. TeamCity has a way to protect secrets but we found we weren't using it consistently. Educate engineers that "secrets aren't just passwords”. Protect SSH keys and database connection strings too. When in doubt, protect it. If you must store secrets in a Git repo, protect them with git-crypt or Blackbox.
4. Validate customer requests. The more unusual a request from a customer, the more important it is to verify whether or not the request is legitimate.
5. Take security reports seriously. We're grateful that our community reported suspicious activity so quickly. Thank you!

There's plenty more in the blog post - please feel free to ask any questions or comments relating to the post below and we'll do our best to answer them. We are not able to comment on any other details related to the attack beyond what is included in the blog post, due to ongoing investigations.

Answer 1

Can you make any comment about the attackers intentions?

Does it appear they were after a certain goal / certain (user) data?

Or was it perhaps more of a "curious teenager" poking with sticks seeing how far they could get?

---

P.S. thanks for the openness regarding this matter, it's really appreciated!

Answer 2

This line:

This act of looking up things (visiting questions) across the Stack Exchange Network becomes a frequent occurrence and allows us to anticipate and understand the attacker’s methodology over the coming days. (emphasis mine)

makes it sound like in real time, as the attack was happening, you could pinpoint what the attacker would do based on what they visited on Stack Overflow, instead of what they did by forensically looking at what they viewed (after the attack). Which one did you mean?

Answer 3

Several questions related mainly to the attacker:

1. What happened to the attacker?
2. Did you suspend their account?
3. Did SE contact the attacker at any point?
4. Why don't you expose the attacker's identity?
5. Has anyone else tried to use this same attack method later?

Answer 4

Was there a detectable sleep cycle at the other end of events?

Edit to clarify:

After becoming aware of the attacker, and since you followed some of their actions as they unfolded, did you notice anything resembling a biological cycle, both day-to-day and retrospectively? E.g: Eating (1-2 hour breaks), sleeping (8 hour inactivity pattern), "power naps" (90 minutes), etc...?

Answer 5

This is not really part of the incident, but a more general concern about security measures around employee accounts. There were a lot of steps in this incident, but the final one was escalating privileges of an SE account. I can imagine a lot more straightforward ways to attempt this than gaining admin access to the CI server via the dev instance to execute SQL in production, and I'm interested in what mitigations and security practices SE has implemented to defend against simpler attempts to gain access to an employee account.

You can't put the main SE sites behind the firewall obviously, so they will always be exposed. And the SE internal login method does not provide any 2FA methods, which I find somewhat concerning.

• are employee accounts 2FA protected via other means (or other login providers)?
• are there any measures to ensure that no private email addresses or login providers are attached to employee accounts that could be less secure and still be used to receive recovery mails to gain access to the account?
• is there monitoring of login attempts from new sources for employee accounts?
• are there additional protections for dangerous employee tools in case someone gains access to a running session of an employee account (e.g. require password and/or 2FA token again when accessing security-critical tools)

Something like spear phishing is probably still one of the more likely ways someone could try to gain access to an employee account.

Answer 6

At around the same time this security incident happened, a few days later, some users began noticing that Twitter oneboxing in chat wasn't working anymore. An employee subsequently confirmed in February of next year that it had indeed been disabled intentionally due to having to "close some gaps" as a result of this security incident.

Can we get a full explanation as to why Twitter oneboxing in chat had to be disabled as a result of this security incident? The blog post published at the time stated that "other potential vectors" had been closed then, and the February 2020 staff message I linked above stated that the Twitter oneboxing feature "made use of one of the gaps we closed". What was that thing, and what security risk did it create?

Finally, is there any way that this functionality can be implemented again, in a secure manner? In August 2020, a few months after the staff message above, the bug report filed at the time was marked status-bydesign by another employee. Would a feature request to change the design back (in a secure manner) be considered, or is it impossible to do so without opening up an attack vector?

Answer 7

Why was the magic link in dev viewable to CMs (presumably just in dev) a real magic link?

Answer 8

I would flag that "password" parameter types in TeamCity aren't considered all that secure:

The password value is stored in the configuration files under TeamCity Data Directory. Depending on the server Encryption Settings, the value is either scrambled or encrypted with a custom key.

The build log value is hidden with a simple search-and-replace algorithm, so if you have a trivial password of "123", all occurrences of "123" will be replaced, potentially exposing the password. Setting the parameter to the password type does not guarantee that the raw value cannot be retrieved. Any project administrator can retrieve it, and any developer who can change the build script could potentially write malicious code to get the password.

Answer 9

This is really an awesome incident report! One of the best ones I've read.

Thank you Stack for making it public and Dean for a great write!

I am just curious to know few things:

• What is the size of the incident response team?
• Were there any specific protocols followed during the investigation?
• What key factor was involved to engage external security vendor? What were the points that were considered in choosing that particular vendor?
• What lessons were learned from the external security vendor? Was their audit process different (effective/ineffective) from the one used already by the team?

The article gives good glimpse of the entire architecture of Stack and the development processes. A more detailed read would or link if there is already an article about it would be great.

Answer 10

Under "Advice to Others":

Log all your inbound traffic. We keep logs on all in-bound connections. This enabled all of our investigations. You can’t investigate what you don’t log.

How can a network as busy as Stack Exchange log the entire inbound traffic? Are these logs web server entries, or IP flows, or full TCP sessions?

I could record most entries and connection attempts on my tiny network, but I have no idea how such a large network does it.

Answer 11

Can you explain more clearly what "publicly accessible properties" means in the below quote?

we have a database containing a log of all traffic to our publicly accessible properties

Answer 12

What is so special about Stack Overflow for Teams Enterprise? Does it grant developer-level access within the team?

The attacker seemed pretty interested in it:

Additionally a person claiming to be one of our Enterprise customers submits a support request to obtain a copy of source code for auditing purposes. This request is rejected because we don’t give out source code and, additionally, the email cannot be verified as coming from one of our customers. It is flagged for further investigation by our support team.

And Stack Overflow restricted access to the help articles for Enterprise, even though they didn't restrict access to the other team articles without payment.

Access to support documentation for our Enterprise product was limited to authorised users of that product.