Stack Overflow is quickly reaching a point where any question that you might want to ask, may already have been asked. Wouldn't it be nice to be able to take this resource with you, even in an offline environment?
I've created a static mirror of SO which shows what I've done so far:
- rendered questions and answers (not comments just yet) into individual web pages (example)
- rendered user information into pages (example)
- giant list of over a quarter million question titles on one page (24+ MB)
My vision for this is to (in collaboration with the community!) package this up into an app of some kind for iPhone/Android/whatever, with local search (Lucene or similar?) and obviously compression. Things that might be useful:
- limit questions to only those over a certain vote threshold
- slice and dice by tag, perhaps include only certain tags or exclude others
I'm not a mobile application developer, so I can't do this all by myself.
Currently the code that I've got to generate this is a hodgepodge of Python, Java, and XSLT. I started to do the whole thing with XSLT but the sheer volume of data brought my poor little server to its knees, so I had to break things up a bit. The order of processing goes something like this:
- Python script to preprocess posts.xml and group the answers by question into one xml file per question
- Java driver (I'm using xt) to run the XSLT question renderer for each question xml file (helps my server not run out of memory)
- Various other XSLT templates to render users, the index page, etc
Who is interested in helping to build this? I'll write up some basic instructions and put the code up on Github for everybody to look and poke at
(but probably tomorrow). Code is now on Github.