What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 150 Stack Exchange communities.

There is a little problem I face while parsing a public dump of StackOverflow data published here for MSR Mining Challenge. Thing is, that I am unable to programmatically digest the 18G postshistory.xml file.

So far, I have tried three different XML libraries in order to get through the whole thing.

  • The standard Java 6 and Java 7 XMLStreamReader parsers fail with invalid character at... exception:

    javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1050673,5451]
    Message: Character reference "&# 
    at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:592)
    at com.softwaretrajectory.msr.stack.data.PrintFields.main(PrintFields.java:64)
  • The VTDXml library, using VTDGenHuge:

    VTDGenHuge vtdGen = new VTDGenHuge();
    vtdGen.parseFile(fileLocation, false, VTDGenHuge.MEM_MAPPED);

    this fails to finish, parseFile returns false, no exception thrown.

    If I "insist" and continue with parsing anyway, it fails later for both methods: when using AutoPilotHuge iterating over "row" elements, and when using VTDNavHuge for moving the cursor.

  • Using Perl and XML::Twig with "processing an XML document chunk by chunk" approach - this one fails throwing similar exception and dumping core:

    reference to invalid character number at line 1050673, column 5445, byte 778177792 at /usr/lib/perl5/XML/Parser.pm line 187

Yes, I tried to remove this particular row from the XML, it fails on the another one, and so on... Moreover, I would like to keep these rows.

Taking in account my lack of experience with XML processing, and considerable amounts of time I am spending every time learning one of the "new way of doing it", I wonder, is there a known to work solution for this problem?

share|improve this question
I remember there being errors like the ones you're seeing in the XML files when I tried to parse them eons ago. I thought they had them fixed by now. Are you using a recent data dump? – NullUserException อ_อ Oct 16 '12 at 7:54
yes, I believe that it is The latest one, - it says "updated August 2012" – seninp Oct 16 '12 at 8:02
So, what's at line 1050673 and columns 5400-5500? – Time Traveling Bobby Oct 16 '12 at 8:25
yes, right there, it is possible to use head/tail to extract the row: head -1050673 posthistory.xml | tail -1, the question is about collection of papers – seninp Oct 16 '12 at 8:38
sorry, it took me some time to get through with cut, the substring is: e rise

<b>---end example---</b>" /> – seninp Oct 16 '12 at 8:52
 is not a valid entity according to the XML spec. That is causing the parser to choke (which is expected). You should post this as a bug because that's not valid XML; see also: meta.stackexchange.com/questions/95505/… – NullUserException อ_อ Oct 16 '12 at 15:53
As far your problem goes, you will have to either remove the  entities or replace them with something else that's valid in XML. Read the section on C0 control characters here: cafeconleche.org/books/effectivexml/chapters/03.html – NullUserException อ_อ Oct 16 '12 at 17:08
Thank you all for tips and links. Parsing is still going on, but it is well over 9.5M of records right now. This is how I got through this particular problem: I used sed to filter out  and , like sed 's///g' | sed 's///g' to produce corrected input for standard Java XMLStreamReader technique. Note, that on this corrected 18G XML file Perl Twig run out of memory, and VTDXml library still breaks without any exception. I have a workstation with Ubuntu and 8G RAM. Working code is here – seninp Oct 17 '12 at 10:23
How much memory do you have in your system? If u only have 8 i suggest u getting at least 8 more... – vtd-xml-author Nov 6 '12 at 7:22
Well, it is useful suggestion, however all that I have is 8G station and 4G laptop. I think this might be an extreme use case for vtd-xml (which I assume designed for different purpose) - low memory and somewhat incorrect XML, but I would appreciate a hint about where to look for the error - this way I could get it fixed and move further. – seninp Nov 6 '12 at 10:35

You must log in to answer this question.

Browse other questions tagged .