Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 158 Stack Exchange communities.

What is meta?
Here's how it works:
  1. Any Stack Exchange user can ask a question
  2. The community provides support, votes on ideas, and reports bugs
  3. Your voice helps shape the way Stack Exchange operates

There is a little problem I face while parsing a public dump of StackOverflow data published here for MSR Mining Challenge. Thing is, that I am unable to programmatically digest the 18G postshistory.xml file.

So far, I have tried three different XML libraries in order to get through the whole thing.

  • The standard Java 6 and Java 7 XMLStreamReader parsers fail with invalid character at... exception: ParseError at [row,col]:[1050673,5451]
    Message: Character reference "&# 
  • The VTDXml library, using VTDGenHuge:

    VTDGenHuge vtdGen = new VTDGenHuge();
    vtdGen.parseFile(fileLocation, false, VTDGenHuge.MEM_MAPPED);

    this fails to finish, parseFile returns false, no exception thrown.

    If I "insist" and continue with parsing anyway, it fails later for both methods: when using AutoPilotHuge iterating over "row" elements, and when using VTDNavHuge for moving the cursor.

  • Using Perl and XML::Twig with "processing an XML document chunk by chunk" approach - this one fails throwing similar exception and dumping core:

    reference to invalid character number at line 1050673, column 5445, byte 778177792 at /usr/lib/perl5/XML/ line 187

Yes, I tried to remove this particular row from the XML, it fails on the another one, and so on... Moreover, I would like to keep these rows.

Taking in account my lack of experience with XML processing, and considerable amounts of time I am spending every time learning one of the "new way of doing it", I wonder, is there a known to work solution for this problem?

share|improve this question
I remember there being errors like the ones you're seeing in the XML files when I tried to parse them eons ago. I thought they had them fixed by now. Are you using a recent data dump? – NullUserException อ_อ Oct 16 '12 at 7:54
yes, I believe that it is The latest one, - it says "updated August 2012" – seninp Oct 16 '12 at 8:02
So, what's at line 1050673 and columns 5400-5500? – Time Traveling Bobby Oct 16 '12 at 8:25
yes, right there, it is possible to use head/tail to extract the row: head -1050673 posthistory.xml | tail -1, the question is about collection of papers – seninp Oct 16 '12 at 8:38
sorry, it took me some time to get through with cut, the substring is: e rise

<b>---end example---</b>" /> – seninp Oct 16 '12 at 8:52
 is not a valid entity according to the XML spec. That is causing the parser to choke (which is expected). You should post this as a bug because that's not valid XML; see also:… – NullUserException อ_อ Oct 16 '12 at 15:53
As far your problem goes, you will have to either remove the  entities or replace them with something else that's valid in XML. Read the section on C0 control characters here: – NullUserException อ_อ Oct 16 '12 at 17:08
Thank you all for tips and links. Parsing is still going on, but it is well over 9.5M of records right now. This is how I got through this particular problem: I used sed to filter out  and , like sed 's///g' | sed 's///g' to produce corrected input for standard Java XMLStreamReader technique. Note, that on this corrected 18G XML file Perl Twig run out of memory, and VTDXml library still breaks without any exception. I have a workstation with Ubuntu and 8G RAM. Working code is here – seninp Oct 17 '12 at 10:23
How much memory do you have in your system? If u only have 8 i suggest u getting at least 8 more... – vtd-xml-author Nov 6 '12 at 7:22
Well, it is useful suggestion, however all that I have is 8G station and 4G laptop. I think this might be an extreme use case for vtd-xml (which I assume designed for different purpose) - low memory and somewhat incorrect XML, but I would appreciate a hint about where to look for the error - this way I could get it fixed and move further. – seninp Nov 6 '12 at 10:35

You must log in to answer this question.

Browse other questions tagged .