14

In a recent blog post on the Trello Stack, Brett Kiefer wrote:

MongoDB fills our more traditional database needs. We knew we wanted Trello to be blisteringly fast. One of the coolest and most performance-obsessed teams we know is our next-door neighbor and sister company StackExchange. Talking to their dev lead David at lunch one day, I learned that even though they use SQL Server for data storage, they actually primarily store a lot of their data in a denormalized format for performance, and normalize only when they need to.

Could David (or anyone else from the team) give some more info on exactly how you "primarily store a lot of data in a denormalized format"? The implication (from the phrasing on the Trello blog, at least) is that you store the denormalized data in a non-MSSQL database. Is this done in Redis (the use of which on SE is already documented)? Something else (MongoDB)? How far do you go with denormalization and what gains have you seen from this?

Improve this question
9
  • I would have put in tags for MongoDB, Sql-Server and Architecture, but my rep isn't high enough to create new tags.
    – Yaakov Ellis StaffMod
    Jan 23, 2012 at 13:49
  • 5
    IMNSHO, you're reading too much into this. "they actually primarily store a lot of their data in a denormalized format for performance" != "they use MongoDB"
    – Piskvor left the building
    Jan 23, 2012 at 13:52
  • I actually think this is a pretty good question. Although, the title should be edited to read: "How is MongoDB Used on Stack Overflow/Stack Exchange"
    – Aaron
    Jan 23, 2012 at 13:54
  • @Piskvor'sSemifiniteMonkeys - good point, updating question to reflect the ambiguity.
    – Yaakov Ellis StaffMod
    Jan 23, 2012 at 14:00
  • As far as I know, MongoDB isn't used at all. However, Redis is used pretty extensively.
    – Tim Stone
    Jan 23, 2012 at 14:00
  • 4
    Trello = Fog Creek Software.... != Stack Exchange... this sounds like a question for Fog Creek?
    – Marc Gravell
    Jan 23, 2012 at 20:41
  • @MarcGravell - I am aware of the difference. I am asking about Stack Exchange because the blog quoted the dev lead for StackExchange about how SE handles data - "Sql Server for data storage, primarily store a lot of their data in denormalized format for performance". I am following up on that comment re: the tech architecture for StackExchange.
    – Yaakov Ellis StaffMod
    Jan 23, 2012 at 20:47
  • @YaakovEllis ahhh, I think I see what you mean now
    – Marc Gravell
    Jan 23, 2012 at 20:51
  • See meta.stackexchange.com/questions/110320/… for more information about this (and mentions of redis)...of which you also contributed an answer @Marc
    – davidsleeps
    Jan 23, 2012 at 21:14

1 Answer 1

Reset to default
18

Firstly to clarify; we don't currently use MongoDB; no prejudice against it - simply: we use things as we need them.

Secondly, while we do make use of redis, it isn't our primary storage, and I think discussing how we use redis would only add confusion.

What we do do, though, is have a combination of normalized and denormalized data in the core SQL Server database(s). Nothing unexpected here - just: we might duplicate a few additional fields between tables if doing so avoids the need to add a join (or worse: sub-query) during some of our high-volume queries. We don't normalize everything, but reducing the JOINs and sub-queries certainly does reduce reads and IO. It does, however, introduce the risk of disagreements between data if you aren't careful - hence the need to also store the normalized version which always "wins".

Nothing too exciting to be honest. You can measure the impact purely using SQL tools - try a key query with/without it and measure ;p I don't have specific numbers to hand. Essentially:

  • can I avoid this query (cache, custom storage, etc)
  • can I make this query cheaper (denormalization, well-written sql, good tools, etc)
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .