Skip to main content
edited title
Link

A deeper dive into the May 2019 security incident: blog post feedback

added 8 characters in body
Source Link

We just posted an update to the security incident that happened back in May 2019 with technical details of what happened, how it happened, and the remediations we applied to prevent an incident like it from happening again. Here's a couple of excerpts from the post - first from the introduction:

On May 12th, 2019, at around 00:00 UTC, we were alerted to an unexpected privilege escalation for a new user account by multiple members of the community. A user that nobody recognised had gained moderator and developer level access across all of the sites in the Stack Exchange Network. Our immediate response was to revoke privileges and to suspend this account and then set in motion a process to identify and audit the actions that led to the event.

After initial discovery, we found that the escalation of privilege was just the tip of the iceberg and the attack had actually resulted in the exfiltration of our source code and the inadvertent exposure of the PII (email, real name, IP addresses) of 184 users of the Stack Exchange Network (all of whom were notified). Thankfully, none of the databases—neither public (read: Stack Exchange content) nor private (Teams, Talent, or Enterprise)—were exfiltrated. Additionally, there has been no evidence of any direct access to our internal network infrastructure, and at no time did the attacker ever have access to data in Teams, Talent, or Enterprise products.

Meme: One does not simply break into Stack Overflow without constantly looking up how to do so on Stack Overflow

And from the final paragraph:

This incident reminded us about some fundamental security practices that everyone should follow:

  1. Log all your inbound traffic. We keep logs on all in-bound connections. This enabled all of our investigations. You can’t investigate what you don’t log.
  2. Use 2FA. That remaining system that still uses legacy authentication can be your biggest vulnerability.
  3. Guard secrets better. TeamCity has a way to protect secrets but we found we weren't using it consistently. Educate engineers that "secrets aren't just passwords”. Protect SSH keys and database connection strings too. When in doubt, protect it. If you must store secrets in a Git repo, protect them with git-crypt or BlackboxBlackbox.
  4. Validate customer requests. The more unusual a request from a customer, the more important it is to verify whether or not the request is legitimate.
  5. Take security reports seriously. We're grateful that our community reported suspicious activity so quickly. Thank you!

There's plenty more in the blog post - please feel free to ask any questions or comments relating to the post below and we'll do our best to answer them. We are not able to comment on any other details related to the attack beyond what is included in the blog post, due to ongoing investigations.

We just posted an update to the security incident that happened back in May 2019 with technical details of what happened, how it happened, and the remediations we applied to prevent an incident like it from happening again. Here's a couple of excerpts from the post - first from the introduction:

On May 12th, 2019, at around 00:00 UTC, we were alerted to an unexpected privilege escalation for a new user account by multiple members of the community. A user that nobody recognised had gained moderator and developer level access across all of the sites in the Stack Exchange Network. Our immediate response was to revoke privileges and to suspend this account and then set in motion a process to identify and audit the actions that led to the event.

After initial discovery, we found that the escalation of privilege was just the tip of the iceberg and the attack had actually resulted in the exfiltration of our source code and the inadvertent exposure of the PII (email, real name, IP addresses) of 184 users of the Stack Exchange Network (all of whom were notified). Thankfully, none of the databases—neither public (read: Stack Exchange content) nor private (Teams, Talent, or Enterprise)—were exfiltrated. Additionally, there has been no evidence of any direct access to our internal network infrastructure, and at no time did the attacker ever have access to data in Teams, Talent, or Enterprise products.

Meme: One does not simply break into Stack Overflow without constantly looking up how to do so on Stack Overflow

And from the final paragraph:

This incident reminded us about some fundamental security practices that everyone should follow:

  1. Log all your inbound traffic. We keep logs on all in-bound connections. This enabled all of our investigations. You can’t investigate what you don’t log.
  2. Use 2FA. That remaining system that still uses legacy authentication can be your biggest vulnerability.
  3. Guard secrets better. TeamCity has a way to protect secrets but we found we weren't using it consistently. Educate engineers that "secrets aren't just passwords”. Protect SSH keys and database connection strings too. When in doubt, protect it. If you must store secrets in a Git repo, protect them with git-crypt or Blackbox.
  4. Validate customer requests. The more unusual a request from a customer, the more important it is to verify whether or not the request is legitimate.
  5. Take security reports seriously. We're grateful that our community reported suspicious activity so quickly. Thank you!

There's plenty more in the blog post - please feel free to ask any questions or comments relating to the post below and we'll do our best to answer them. We are not able to comment on any other details related to the attack beyond what is included in the blog post, due to ongoing investigations.

We just posted an update to the security incident that happened back in May 2019 with technical details of what happened, how it happened, and the remediations we applied to prevent an incident like it from happening again. Here's a couple of excerpts from the post - first from the introduction:

On May 12th, 2019, at around 00:00 UTC, we were alerted to an unexpected privilege escalation for a new user account by multiple members of the community. A user that nobody recognised had gained moderator and developer level access across all of the sites in the Stack Exchange Network. Our immediate response was to revoke privileges and to suspend this account and then set in motion a process to identify and audit the actions that led to the event.

After initial discovery, we found that the escalation of privilege was just the tip of the iceberg and the attack had actually resulted in the exfiltration of our source code and the inadvertent exposure of the PII (email, real name, IP addresses) of 184 users of the Stack Exchange Network (all of whom were notified). Thankfully, none of the databases—neither public (read: Stack Exchange content) nor private (Teams, Talent, or Enterprise)—were exfiltrated. Additionally, there has been no evidence of any direct access to our internal network infrastructure, and at no time did the attacker ever have access to data in Teams, Talent, or Enterprise products.

Meme: One does not simply break into Stack Overflow without constantly looking up how to do so on Stack Overflow

And from the final paragraph:

This incident reminded us about some fundamental security practices that everyone should follow:

  1. Log all your inbound traffic. We keep logs on all in-bound connections. This enabled all of our investigations. You can’t investigate what you don’t log.
  2. Use 2FA. That remaining system that still uses legacy authentication can be your biggest vulnerability.
  3. Guard secrets better. TeamCity has a way to protect secrets but we found we weren't using it consistently. Educate engineers that "secrets aren't just passwords”. Protect SSH keys and database connection strings too. When in doubt, protect it. If you must store secrets in a Git repo, protect them with git-crypt or Blackbox.
  4. Validate customer requests. The more unusual a request from a customer, the more important it is to verify whether or not the request is legitimate.
  5. Take security reports seriously. We're grateful that our community reported suspicious activity so quickly. Thank you!

There's plenty more in the blog post - please feel free to ask any questions or comments relating to the post below and we'll do our best to answer them. We are not able to comment on any other details related to the attack beyond what is included in the blog post, due to ongoing investigations.

Source Link

A deeper dive into May 2019 security incident: blog post feedback

We just posted an update to the security incident that happened back in May 2019 with technical details of what happened, how it happened, and the remediations we applied to prevent an incident like it from happening again. Here's a couple of excerpts from the post - first from the introduction:

On May 12th, 2019, at around 00:00 UTC, we were alerted to an unexpected privilege escalation for a new user account by multiple members of the community. A user that nobody recognised had gained moderator and developer level access across all of the sites in the Stack Exchange Network. Our immediate response was to revoke privileges and to suspend this account and then set in motion a process to identify and audit the actions that led to the event.

After initial discovery, we found that the escalation of privilege was just the tip of the iceberg and the attack had actually resulted in the exfiltration of our source code and the inadvertent exposure of the PII (email, real name, IP addresses) of 184 users of the Stack Exchange Network (all of whom were notified). Thankfully, none of the databases—neither public (read: Stack Exchange content) nor private (Teams, Talent, or Enterprise)—were exfiltrated. Additionally, there has been no evidence of any direct access to our internal network infrastructure, and at no time did the attacker ever have access to data in Teams, Talent, or Enterprise products.

Meme: One does not simply break into Stack Overflow without constantly looking up how to do so on Stack Overflow

And from the final paragraph:

This incident reminded us about some fundamental security practices that everyone should follow:

  1. Log all your inbound traffic. We keep logs on all in-bound connections. This enabled all of our investigations. You can’t investigate what you don’t log.
  2. Use 2FA. That remaining system that still uses legacy authentication can be your biggest vulnerability.
  3. Guard secrets better. TeamCity has a way to protect secrets but we found we weren't using it consistently. Educate engineers that "secrets aren't just passwords”. Protect SSH keys and database connection strings too. When in doubt, protect it. If you must store secrets in a Git repo, protect them with git-crypt or Blackbox.
  4. Validate customer requests. The more unusual a request from a customer, the more important it is to verify whether or not the request is legitimate.
  5. Take security reports seriously. We're grateful that our community reported suspicious activity so quickly. Thank you!

There's plenty more in the blog post - please feel free to ask any questions or comments relating to the post below and we'll do our best to answer them. We are not able to comment on any other details related to the attack beyond what is included in the blog post, due to ongoing investigations.