I think this might be a good question for stack overflow, but it's not a question about my code, or how to code a certain thing.
Where should I ask this question?
We currently have a customer-facing web U.I. that is basically a doorway into our AWS EMR architecture. The web U.I. lets customers build reports that let them select filters and columns to output in the report. The number of filters is quite large. We give them a large date window as well. For example, they could run a report that lets them select all of their data over the last year, and aggregate it and produce sums.
We have many GB of this data coming in per day. It's on the order of a billion records per day. It's over 0.5 TB, raw and uncompressed. After cleaning and pre-aggregating and separation into a handful of Hive tables, and after applying gzip compression, we're looking at, oh, 50 GB a day coming in.
When a web report is submitted, we turn it into a Hive query and submit it to a Hadoop cluster running Hive. This gives us turnaround times of 15 minutes at the fastest (small date range, etc), but usually slower. Sometimes the queries run for hours, even overnight, before the user gets the report.
The data is not very sparse; we don't have a very high number of columns. We're looking at about 20 columns in each of 5 tables.
I want to figure out how to make the reports very performant such that the user can see the results in a web page right away. At first I considered it unfeasible, thinking it was just too much data, "big data". Big Data has 3 V's: Volume, Velocity, Variety. I'd say we don't have "variety", our data schema is very limited in scope. So perhaps this system can be optimized. I've been looking into HBase, Cassandra, Spark, etc. Cassandra and HBase seem like they might not fit our usage due to effectively being glorified key=>value stores. But, since we would have to pull so many values for a single report, and then aggregate them, I don't think it will work. We could pre-aggregate for a few use cases, but in general I don't think we can.
I'm thinking (and I could be very wrong, I'm not a big data expert or anything, far from it) that our data has too many query-able dimensions to de-norm for every use case.
What should I research to figure this out? Are there any books that could help? What technologies, databases, etc, should I review? What should I know that I don't know, before I install HBase or Cassandra (or Mongo or AWS Redshift or X or Y or ...) and start trying to figure out how to optimize our data for the db?