There is a little problem I face while parsing a public dump of StackOverflow data published here for MSR Mining Challenge. Thing is, that I am unable to programmatically digest the 18G postshistory.xml file.
So far, I have tried three different XML libraries in order to get through the whole thing.
The standard Java 6 and Java 7 XMLStreamReader parsers fail with invalid character at... exception:
javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1050673,5451] Message: Character reference "&# at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:592) at com.softwaretrajectory.msr.stack.data.PrintFields.main(PrintFields.java:64)
The VTDXml library, using VTDGenHuge:
VTDGenHuge vtdGen = new VTDGenHuge(); vtdGen.parseFile(fileLocation, false, VTDGenHuge.MEM_MAPPED);
this fails to finish, parseFile returns false, no exception thrown.
If I "insist" and continue with parsing anyway, it fails later for both methods: when using AutoPilotHuge iterating over "row" elements, and when using VTDNavHuge for moving the cursor.
Using Perl and XML::Twig with "processing an XML document chunk by chunk" approach - this one fails throwing similar exception and dumping core:
reference to invalid character number at line 1050673, column 5445, byte 778177792 at /usr/lib/perl5/XML/Parser.pm line 187
Yes, I tried to remove this particular row from the XML, it fails on the another one, and so on... Moreover, I would like to keep these rows.
Taking in account my lack of experience with XML processing, and considerable amounts of time I am spending every time learning one of the "new way of doing it", I wonder, is there a known to work solution for this problem?