Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 157 Stack Exchange communities.

What is meta?
Here's how it works:
  1. Any Stack Exchange user can ask a question
  2. The community provides support, votes on ideas, and reports bugs
  3. Your voice helps shape the way Stack Exchange operates

When editing a post that contains a non-BMP unicode character (codepoint U+10000 or up), the AJAX-loaded post when you submit the edit does not include that character. I suspect this is a JSON-encoder bug on the server side.

For example, in my answer to How to convert some character into five digit unicode one in Python 3.3?, I included the codepoint U+1D15D ("MUSICAL SYMBOL WHOLE NOTE"), then was confused by the fact that not even a placeholder character was being shown.

Normally, the post looks like this:

Placeholder glyph for U+1D15D

(where the font used to render the glyph is missing that codepoint so a square box placeholder is shown instead).

When I edit the post the preview works fine, but when I then submit the edit the page is refreshed using AJAX and I see:

Placeholder glyph gone

There is no character between the quotes there; the JSON response for the edit gives me:

<pre><code>&gt;&gt;&gt; '\\U0001D15D'\n''\n&gt;&gt;&gt; '\\U0001D15D'.encode('unicode_escape')\nb'\\\\U0001d15d'\n</code></pre>\n\n

for that part of the post; again no U+1D15D character there.

Clearly, the SE software stack can handle non-BMP codepoints. The JSON standard can handle it too, almost all characters can be included literally, or you can use \uabcd escapes provided you encode these codepoints using a UTF-16 surrogate. See section 2.5 of the JSON RFC.

So, what I suspect is that non-BMP codepoints are not being handled by the JSON encoder on the server side and are instead silently dropped.

Can this be fixed?

share|improve this question
Just tried it in the formatting sandbox. Was shown in the preview, vanished when submitting the edit. – Daniel Fischer Jan 31 '13 at 15:40
@DanielFischer: Exactly, now reload the page.. – Martijn Pieters Jan 31 '13 at 15:44
up vote 3 down vote accepted

This was an issue in ServiceStack.Text that Kevin fixed a while ago; looks like we didn't update the library afterwards. We'll use the newest version after the next build; I have checked that this does indeed fix your issue.

share|improve this answer
Thanks for the fix; that patch indeed removed a restriction where surrogates (e.g. non-BMP codepoints encoded to UTF-16) where filtered out. – Martijn Pieters Feb 1 '13 at 12:01
Confirmed fixed; the AJAX response now includes the non-BMP codepoint. – Martijn Pieters Feb 1 '13 at 15:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .