26

I just finished a project to convert SE data-dumps into draft PDF eBooks, which I'm calling SE2go. SE2go eBooks are interactive, they contain all the original links within the question and answer body, original formatting (as much as possible), and are sorted by tag and question title. Also included are links back to the original question, and the meta data important to each question and answer.

Any comments or unaccepted answers are not included in the eBooks. The page size of all eBooks are formatted to be easy to read on a 7” reader or tablet, include any embedded graphics and highlighted syntax. Using SE2go on larger displays is also supported without any degradation in quality.

Stack Overflow itself has the largest number of questions and accepted answers, the Posts.xml file is >20G. To convert the entire Stack Overflow site into a single eBook was just off the hook; it made an eBook that was not user friendly, so the best working idea was to create separate tag based eBooks.

The first set of draft eBooks are built from the Stack Exchange data dumps of 3Q/2013. Current plans are to release another set of drafts after community input. Right now the only format available is PDF but depending on community input other eBook formats (epub, Nook, Kindle, etc) can be done. There's a short survey to see what the community would like.

I've posted draft PDFs here: http://se2go.ka6wke.net.

  • 6
    Thanks for putting this kind of work in. The official formatting for the names are Stack Overflow and Stack Exchange. – Lance Roberts Dec 22 '13 at 1:16
  • 9
    IMO You should consider including the top voted unaccepted answer if it is voted highly enough (either as well as or instead of any accepted one that may exist) – Martin Smith Dec 22 '13 at 1:23
  • @LanceRoberts, thanks on the official formatting of site names. I've updated the website to reflect. – MarkSchoonover Dec 22 '13 at 4:54
  • @Martin - I'll look into your idea, a high enough vote could be considered community accepted. – MarkSchoonover Dec 22 '13 at 4:55
  • This would be so much more readable if the code were typeset in a monospaced font – Eric Dec 23 '13 at 1:21
  • @Eric - source code is a challenge because the SO data dumps only contain <pre><code> blocks leaving out the language. If it can't be detected, then source code is simply converted verbatim but I break long lines as best as possible. Better code detection and formatting is next on my todo list: se2go.ka6wke.net/-todo – MarkSchoonover Dec 23 '13 at 3:08
  • Syntax highlighting is an orthogonal concern to the choice of code font – Eric Dec 23 '13 at 8:51
  • @Eric I'm working on a new solution for handling code blocks. I see all code is defaulting to the same fonts that are used in regular paragraph text. – MarkSchoonover Dec 24 '13 at 4:21
  • 2
    Really needs a table of contents or index to be even minimally useful. Even then I'm not sure what the value is going to be. Good start though, and I'm sure something interesting will come of it if you keep at it. – Adam Davis Feb 7 '14 at 16:36
3

This project has run since Dec 2013, with just over 1800 downloads for the sample eBooks, and only a few comments/suggestions so I've decided there isn't that much interest from the community on portable formats of SE sites. I've decided to no longer work on this project.

This was my first non-trivial Java project and I learned a bunch. Nothing really beats programming on a project with concrete requirements to solidify a new language. With all that said, I've released all the source code on Bitbucket under the Apache 2.0 license. It does have some interesting classes like stream processing XML files into a DB, HTML->LaTeX converter, and image fetcher. The XML parser can put the entire SO Posts.xml file into a H2 database on a machine with only 4GB RAM and a 2.2GHz Celeron.

There are additional Java libraries that are required in order to compile:

  • ANTLR
  • StringTemplate 4
  • Apache Commons IO
  • Apache Commons Lang
  • Apache Commons Logging
  • Apache Commons Exec
  • Apache Commons CLI
  • Apache Commons Configuration
  • Apache Commons Lang
  • H2 Database Engine
  • Jericho HTML Parser
  • jsoup
  • pygments 1.6

If you want to build PDFs from the LaTeX files, you'll need to use xetex. The LaTeX conversion isn't 100% perfect, but it does a solid job. For the adventurous, it wouldn't take too much effort to produce epub/nook formats.

  • I think a lot of people are still very thankful for your effort to contribute to the community! Thanks! – Patrick Hofman Apr 15 '14 at 6:37
  • Eat - thanks for the edit. I couldn't get the bullets to work with IE 6. :) – MarkSchoonover Apr 15 '14 at 18:28
1

Adam Davis, there is a table of contents of sorts, I create bookmarks for each title. The titles of all questions for a given site/tag are put in alphabetical order. The concept of SE2go books is in a cookbook style; use the bookmarks to find the problem your trying to solve.

Version 1 was a private beta of sorts simply to test the concept on a few different tablets/phones.

Version 2 is currently released and are drafts for comments from the community.

I'm currently working on v3 which made improvements in fonts, source code formatting, syntax highlighting, and any bug fixes from v2. ePub and any bug fixes are planned for v4, mobi and v4 bug fixes in v5. Each new version will include more titles as well.

I am thinking about an index but not until ePub/mobi formats are done.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .