3

I think this might be a good question for stack overflow, but it's not a question about my code, or how to code a certain thing.

Where should I ask this question?

We currently have a customer-facing web U.I. that is basically a doorway into our AWS EMR architecture. The web U.I. lets customers build reports that let them select filters and columns to output in the report. The number of filters is quite large. We give them a large date window as well. For example, they could run a report that lets them select all of their data over the last year, and aggregate it and produce sums.

We have many GB of this data coming in per day. It's on the order of a billion records per day. It's over 0.5 TB, raw and uncompressed. After cleaning and pre-aggregating and separation into a handful of Hive tables, and after applying gzip compression, we're looking at, oh, 50 GB a day coming in.

When a web report is submitted, we turn it into a Hive query and submit it to a Hadoop cluster running Hive. This gives us turnaround times of 15 minutes at the fastest (small date range, etc), but usually slower. Sometimes the queries run for hours, even overnight, before the user gets the report.

The data is not very sparse; we don't have a very high number of columns. We're looking at about 20 columns in each of 5 tables.

I want to figure out how to make the reports very performant such that the user can see the results in a web page right away. At first I considered it unfeasible, thinking it was just too much data, "big data". Big Data has 3 V's: Volume, Velocity, Variety. I'd say we don't have "variety", our data schema is very limited in scope. So perhaps this system can be optimized. I've been looking into HBase, Cassandra, Spark, etc. Cassandra and HBase seem like they might not fit our usage due to effectively being glorified key=>value stores. But, since we would have to pull so many values for a single report, and then aggregate them, I don't think it will work. We could pre-aggregate for a few use cases, but in general I don't think we can.

I'm thinking (and I could be very wrong, I'm not a big data expert or anything, far from it) that our data has too many query-able dimensions to de-norm for every use case.

What should I research to figure this out? Are there any books that could help? What technologies, databases, etc, should I review? What should I know that I don't know, before I install HBase or Cassandra (or Mongo or AWS Redshift or X or Y or ...) and start trying to figure out how to optimize our data for the db?

Improve this question
2
  • 1
    Maybe there is no good place for this question. I'm asking for advice, not an answer to a specific question. Are stack exchange sites focused only on answering specific questions, or are there advice sites among them?
    – Eddified
    Feb 16, 2013 at 0:22
  • 1
    There is an Area 51 proposal that might fit your needs: Big Data
    – BenjiWiebe
    Feb 16, 2013 at 16:44

1 Answer 1

Reset to default
3

This is an advice/recommendation question, and probably off topic everywhere. From any FAQ:

You should only ask practical, answerable questions based on actual problems that you face. Chatty, open-ended questions diminish the usefulness of our site and push other questions off the front page.

Your questions should be reasonably scoped. If you can imagine an entire book that answers your question, you’re asking too much.

The first paragraph rules out most "advice" questions since they are open ended. It also rules out recommendations. The second paragraph rules out a question asking for an in situ answer (as opposed to a book recommendation), since your question is very broadly scoped.

However, you may have some luck asking this question in the DBA chatroom.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .