What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 122 Stack Exchange communities.

I have been thinking about building yet-another data dump sandbox like StackQL or StatOverflow -- both sadly defunct.

The new data dump sandbox will provide, at minimum, an OData endpoint for all "Stack" sites (That way you can query it with LINQPad etc.)

What other kinds of features do you think it should have?

EDIT the current prototype is here

share|improve this question
    
@James: that would be Meta's datadump. –  perbert May 4 '10 at 1:46
add comment

4 Answers

up vote 7 down vote accepted
+100

I am still working on StackQL -- it's just taking a while to get it going again. Maybe another six to eight weeks (that said, I built the original prototype in a single evening, so I really just need to get off my butt and get working).

That out of the way, the more the merrier, so go for it. I learned a few things building StackQL that I think are worth mentioning:

  1. Indexing is important, especially the full text indexes on post body and title and on the user's AboutMe text. Using an index vs not makes a huge difference. Unfortunately, no one knows how to use full-text indexes. My StackQL query logs are full of fail from users bungling contains queries. And the brings me to my next point:
  2. Log every query, and share the log. StackQL got really interesting only after you could share your queries with other users, and the logs are really useful for a whole lot of different things. One thing I'm adding to the next iteration is an field to indicate how long a query took to execute. This will also be useful to indicate whether the query succeeded or generated an error, so you can easily filter out the failed queries.
  3. Be careful showing the post text in a web page. It's not sanitized, so a <script> in a post can actually run. I dug up the old code Jeff posted from way back on how he sanitizes StackOverflow and just re-used that.
  4. Lots of help on the schema. I had pull down menus with field names and table names that would insert them directly in your query, and people still couldn't get it right.

If had it to do again, I'd name it "YADDA" - Yet Another Data Dump Access site.

share|improve this answer
    
I concur on the indexing. takes a while longer to import but the query speed is exponentially higher. –  Sky Sanders May 4 '10 at 9:39
    
The most commonly bungled types of queries with examples on how to get them right might make for an interesting blog post (or even a series). –  Bill the Lizard May 4 '10 at 12:22
add comment

odata.stackoverflow.com and odata.mysite.com and api.mysite.com for the other api.

share|improve this answer
4  
+1 for "odata.stackoverflow.com" - that would make developing against the data a breeze. –  Matt Hamilton May 4 '10 at 6:04
    
hummm... no idea if odata queries would put more stress/overload on the servers, maybe the StackOverflow team does not want this... –  balexandre May 4 '10 at 6:56
3  
it'll actually be odata.stackexchange.com/sitename since this is for all sites and all FUTURE sites too –  Jeff Atwood May 4 '10 at 16:10
2  
Yes, but folks want odata.domain.com. Why not standardize the subdomain? api.twitter.com, etc. –  Scott Hanselman May 4 '10 at 22:42
add comment
  1. Make queries linkable. So I can link to them. Bonus points for allowing me to produce HTML-formatted results by intentionally failing to escape the output.
  2. Keep it up long enough for me to bring it crashing down by including gratuitous links to HTML-formatted queries in my answers on Meta. Or your answers on Meta. Muahahah. Hahah. cough
share|improve this answer
1  
You know I wrote spShog9 especially for you :) I plan to have that baked in ... I do want to introduce some sort of SLA and per user limits to avoid breaking the bank –  waffles May 4 '10 at 3:36
add comment

Well, you are in control of the data in the first place, so I wonder about the necessity for tools like this, but if I can be of assistance count me in.

share|improve this answer
1  
we still want the torrents to be available. Whether the data dump site automatically pulls and imports the torrents, we're not sure, but the torrents should be there regardless. –  Jeff Atwood May 4 '10 at 3:01
    
@jeff - so, this site will not have direct access to the exchange sites db and thus needs to import or have the data pushed into it. Am I understanding you correctly? –  Sky Sanders May 4 '10 at 3:26
    
SODDI is really cool, I plan to reuse bit of it if possible (depending on the licensing) Ideally I would like this to always be up to date (within a few minutes) using the yet to come SO api for deltas –  waffles May 4 '10 at 3:35
    
@waffles - contact me via blog linked in profile. i am sure that there is a special license waiting. –  Sky Sanders May 4 '10 at 3:39
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .