We got the April 2011 data dump from Stack Overflow and, while trying to put everything into a database, we got

SyntaxError: reference to invalid character number: line 1105549, column 5445

in the middle of the posthistory.xml file. It seems that there are sets of characters (&#xC I believe) within a few posthistory entries which break our XML parser.

Is there a solution besides manually changing all the C's?

share|improve this question
    
@Jon - One of the offenders is posthistory_id 1332647, we've been reading that these characters wouldn't be a problem for xml version 1.1, but we don't know how to get our script to run a different version –  Elaina R Jun 17 '11 at 18:58
    
Try the workaround here: oxygenxml.com/forum/topic5171.html –  Jon Seigel Jun 18 '11 at 14:49
    
We have the same problem when trying to parse posthistory.xml (April 2012 this time) using Python's xml.etree.cElementTree –  Bogdan Vasilescu Aug 1 '12 at 20:16
    
We are stripping out invalid characters from XML files, so this shouldn't be happening any longer. –  Oded Jul 22 '13 at 9:55
add comment

1 Answer

up vote 3 down vote accepted

I'm running into this now, too, though in Posts.xml. I'm still importing, but I stripped some &#x1B from post id 854536. I edited the source post so it doesn't continue to get put into the Posts.xml file, but it will continue to exist in PostHistory.xml as a revision. (Edit: here is the other post. Those two were the only ones blocking the Posts.xml import for me.)

This is a real problem, because these invalid characters don't allow the files to validate as XML 1.0. It seems most parsers (including the one I'm using) don't support XML 1.1, so I'm not sure simply changing the header to 1.1 would be very helpful.

Since the invalid characters (so far) are of the non-printing, garbage type, I think we may be able to get an assist from the dev team to strip these out directly in the database, so we don't have to go through a ridiculous search-and-replace process every time there's a new data dump. (It wouldn't be so bad if the big XML files were pre-split into smaller chunks, but still.)


Okay, I got both Posts.xml and PostHistory.xml imported. All that's needed is to remove the entities , , , and . Easier said than done, I know; those are big files. I ended up doing a file split on both of them to track down the errors, but since I've done that already, the next time, I would recommend just opening each big file as-is in an editor and doing search/replace.

That said, this should still be fixed in the dump itself.

share|improve this answer
    
Thanks for the help! We ended up using Sed to rid ourselves of the problem entities after trying to replace the header with no luck. Glad that we're not crazy, but we didn't have any problems with Posts.xml, I think we only ran into trouble with the &#xC and &#xA characters... What was the date of the dump you tried this on? (Also, editing the problem posts is much appreciated for the future) –  Elaina R Jul 5 '11 at 14:46
    
I'm working with an NDA (i.e. non-public) dump from ~June 6th 2011. This dump has more data than the public dump, but it just so happened that some of the bad posts weren't deleted. As mentioned, there will still be problems in PostHistory, but Posts should be clean now (unless some new posts introduce these characters again). –  Jon Seigel Jul 5 '11 at 21:47
    
Depending on the language, another option would be to write a wrapper around the file object that strips these characters as it reads the file, and feed that to the XML parser. –  NullUserException อ_อ Oct 16 '12 at 17:22
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .