What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 127 Stack Exchange communities.

When editing a post that contains a non-BMP unicode character (codepoint U+10000 or up), the AJAX-loaded post when you submit the edit does not include that character. I suspect this is a JSON-encoder bug on the server side.

For example, in my answer to How to convert some character into five digit unicode one in Python 3.3?, I included the codepoint U+1D15D ("MUSICAL SYMBOL WHOLE NOTE"), then was confused by the fact that not even a placeholder character was being shown.

Normally, the post looks like this:

Placeholder glyph for U+1D15D

(where the font used to render the glyph is missing that codepoint so a square box placeholder is shown instead).

When I edit the post the preview works fine, but when I then submit the edit the page is refreshed using AJAX and I see:

Placeholder glyph gone

There is no character between the quotes there; the JSON response for the edit gives me:

<pre><code>&gt;&gt;&gt; '\\U0001D15D'\n''\n&gt;&gt;&gt; '\\U0001D15D'.encode('unicode_escape')\nb'\\\\U0001d15d'\n</code></pre>\n\n

for that part of the post; again no U+1D15D character there.

Clearly, the SE software stack can handle non-BMP codepoints. The JSON standard can handle it too, almost all characters can be included literally, or you can use \uabcd escapes provided you encode these codepoints using a UTF-16 surrogate. See section 2.5 of the JSON RFC.

So, what I suspect is that non-BMP codepoints are not being handled by the JSON encoder on the server side and are instead silently dropped.

Can this be fixed?

share|improve this question
    
Just tried it in the formatting sandbox. Was shown in the preview, vanished when submitting the edit. –  Daniel Fischer Jan 31 '13 at 15:40
    
@DanielFischer: Exactly, now reload the page.. –  Martijn Pieters Jan 31 '13 at 15:44
add comment

1 Answer

up vote 3 down vote accepted

This was an issue in ServiceStack.Text that Kevin fixed a while ago; looks like we didn't update the library afterwards. We'll use the newest version after the next build; I have checked that this does indeed fix your issue.

share|improve this answer
    
Thanks for the fix; that patch indeed removed a restriction where surrogates (e.g. non-BMP codepoints encoded to UTF-16) where filtered out. –  Martijn Pieters Feb 1 '13 at 12:01
    
Confirmed fixed; the AJAX response now includes the non-BMP codepoint. –  Martijn Pieters Feb 1 '13 at 15:58
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .