# Occasionally, the Unicode character sequence U+200C U+200B (ZWNJ ZWSP) is inserted into comments

... the number "5000" does not paste as such but instead as

5
0
U+200C
U+200B
0
0


The HTML source looks like this:

perhaps the easiest would be <code>\edef\tmp{\everypar{\widowpenalty\the\widowpenalty\relax}}\tmp\widowpenalty=-50&zwnj;&#8203;00</code> that may reset


Or (smaller snippet):

\widowpenalty=-50&zwnj;&#8203;00</code> that


So in the HTML, we have extraneous &zwnj; and &#8203; in between the two middle digits of 5000.

Upon pasting and trying out this code, this first led to a mysterious compiler error. The same issue happened within a (now deleted) comment thread below this answer on TeX.SE; there, these or similar two characters were inserted after the letter "r" within the string \ref ("\ref").

Note that not all browsers wrap the same way. The above screenshot was for Firefox 19.0.2, in which it breaks between \relax}} and \tmp, with 5000 insidiously left intact on the second line but with the two special characters remaining in the middle. So that is why this took me completely by surprise. Other browsers might indeed only break on the invisible Unicode characters, such as Chrome 25:

Or on a smaller screen with the mobile site, multiple line breaks might show, possibly still leaving the Unicode characters unused:

• This is a test: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa Mar 8, 2013 at 18:55
• Yep, U+200C U+200B (ZWNJ ZWSP) is being inserted every 80 characters within a code snippet in a comment. Mar 8, 2013 at 18:55
• It's not erroneously inserted, but I guess it's problematic in this case. Mar 8, 2013 at 18:57
• @TimStone: It's generally problematic for copy-pasting code from comments. I guess I've never run into this issue because posting 80+ characters in code without a space is unusual. Mar 8, 2013 at 19:02
• @nneonneo Yeah. In TeX spaces are more often significant than the casual user would expect (though in this example there are tons of places where spaces would be perfectly fine without causing any problems). I am guessing that one reason for omitting spaces in code within comments is the (necessary) character limit for comments. Mar 8, 2013 at 19:07
• @TimStone By the way, if a fix is made, it might make sense to apply it retroactively, because the status quo might be breaking a lot of people's code :-) Mar 8, 2013 at 19:57
• Comments are rendered on-the-fly, so any change will be retroactive by nature. Mar 8, 2013 at 20:26
• @Tim, I guess that would make a nice answer? (Maybe with a reference to Jeff's explanation for the why as well.) Mar 8, 2013 at 21:27
• @Arjan Well, it seems kind of bad that this breaks copy and pasting in a non-visible way, but I don't know if there's a good solution beyond "Copy relevant comment content into the answer". Mar 8, 2013 at 21:34
• True, @Tim, it's just that future visitors might still want to understand the cause -- which is now buried in the comments a bit? (Funny to see how much trouble this has given over time...) Mar 8, 2013 at 21:38
• @TimStone (and Arjan and others): See my edit to the question (the caveat at the bottom). Mar 9, 2013 at 1:51
• @Arjan (and all others): PDF has facilities (related to accessibility support) that can change text when it is copied+pasted. I am wondering if someone familiar with the latest HTML spec knows of a similar thing there. This would be an ideal solution. And the present thing where sometimes code breaks (because this bug/feature is not known to the majority of users here) I am not too happy with: presently I need to pay special attention and test+fix all long code snippets manually. Mar 9, 2013 at 1:57
• @nneonneo, as an aside: the Unicode magic applies to all comment text, not just code. Mar 9, 2013 at 16:16
• @TimStone In the case of splitting up graphemes, it is indeed erroneously inserted. It must only be inserted between graphemes so it does not separate the combining character from its base: s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿s̳̿e̳̿x̳̿. Apr 29, 2015 at 4:28
• @Sklivvz in my opinion breaking functionality (e.g. copying code/url) just to fix design bugs is bad idea. Is there any point to start feature request asking to just remove that auto insertion of unicode characters? (there is pure CSS solution as far as I can tell, using overflow: hidden;: jsfiddle.net/ya89bzpd) Aug 17, 2016 at 14:28

Unicode magic is inserted in comments (not just in code) whenever no whitespace was seen for about 80 characters, to enforce line wrapping. Without that the layout can really be messed up.

Jeff once explained why this magic is needed:

The word-wrap:break-word suggestion is a good one.

Works fine on posts, which are fixed width, and that is implemented.

Near as I can tell, there is no way to get this to work on comments because they are variable width. That is, the actual comment size depends on whether or not the comment vote UI controls are present (fex, on your own comment, you don't have controls, if you're not logged in there are no controls, etc), and if the comment has say 100 upvotes that's wider still.

One could argue if the above really is a good reason to not have a fixed width for comments on the full site. (The mobile theme is much more fluid, but uses word-break: break-all, so probably has no need for the Unicode markers, though they are present there too. And all that looks just fine.)

Using JavaScript to remove the markers upon copying, is not going to be easy. What about:

1. Instead of the proposed word-wrap: break-word, use word-break: break-all, just like on the mobile site. This works for me in Chrome 25, Firefox 19, Safari 6, Internet Explorer 9 (in Windows 7 on Parallels on a Mac; simulating IE8 and IE7 works too), stock browser and Chrome on Android 4.1, and Safari on iOS 6.1.2. But apparently it does/might not work in Opera? See this JS Bin example.

2. Or, instead of Unicode magic, insert <span style="display: inline-block"></span>. Tested in the same browsers, and probably also supported in Opera; see the same JS Bin example above.

3. Or, use JavaScript on hovering to get the actual width, remove the Unicode magic, and then explicitly set width and word-wrap: break-word. This might reflow the text on hover, but only for affected comments. (Alternatively, one could run the JavaScript for all comments, on page load and after each resize.) Also, this needs to restore the original text (with the Unicode magic) when resizing the window, to allow the browser to calculate the proper new widths:

$(".comment-copy").hover(function() { var text =$(this).text();
var clean = text.replace(/[\u200B,\u200C]/g, "");
if(text !== clean) {
$(this).css("width",$(this).width());
$(this).css("word-wrap", "break-word");$(this).text(clean);
if(!$(this).data("orig")) { // Store original for later resizing$(this).data("orig", text);
}
}
});

$(window).resize(function() {$(".comment-copy").each(function() {
var orig = $(this).data("orig"); if(orig) { // Reset original text, to get the proper width$(this).text(orig);
$(this).css("width", "");$(this).css("word-wrap", "");
}
});
});


(See the same JS Bin example; tested in the same desktop browsers.)

4. Or, if the above still yields problems: after posting, check if the magic was applied, and warn the commenter about it with a shiny blue info message? Like: "Beware: your comment includes a sequence of more than 80 non-space characters, which might cause issues when copying. Consider adding some whitespace."

5. And/or, bind to the copy event, check the content and warn if needed: "The comment you're copying includes invisible Unicode characters". However, this event might go unnoticed when copying more than just the comment.

6. Or, even though lines might be wrapped on another character: simply insert a true space instead? At least that is visible.

Some day, <wbr> will be the solution, but not today: though IE7 supported it, IE8 and IE9 don't.

• One additional idea: mark linebreaks where there was originally no whitespace with a curved downarrow symbol to the right of the first line (or the left of the second line). I think I've seen this in some version of emacs, but can't find a screenshot now. Mar 9, 2013 at 19:15
• @Lover, the problem with visible markers is nicely shown in the difference between the rendering in Chrome and in Firefox: the latter is smarter, and happens to break the long text at an earlier place, making the actual marker no longer needed/used. Also, the markers are not placed after every 80th character, but only when no whitespace was seen for at least 80 characters. In the Chrome screen capture things would look much better if the marker was inserted each, say, 25 characters... Mar 9, 2013 at 19:22
• You are right, but it's not a problem with the idea of markers, it's a problem with the possible lack in a browser interface for us to control that markers should be there only if the browser linebreaks at a non-whitespace character. As I understand it. Correct me if I'm wrong. Mar 9, 2013 at 19:30
• True, @Lover. If one could ensure/calculate where markers are actually needed/used, then it would even be fine to insert any visible marker. Unfortunately, it's very hard to calculate that, as not every letter needs the same width, and as the total available width is not fixed (especially not on mobile). Mar 9, 2013 at 21:36
• I just observed the insertion of ZWNJ ZWSP not in a comment, but in a question's code block (Shellcode.py). This is quite disturbing. Feb 13, 2017 at 6:48
• @Arjan <wbr> is supported in pretty much every browser except for IE. IE will eventually be completely replaced by Edge. caniuse.com/#feat=wbr-element Mar 7, 2017 at 22:21

As of about a month ago, these characters are no longer inserted, since comments are no longer laid out with a <table> (and the table layout algorithm was what prevented fixing overflow in any other way).

• Any chance to do the same with chat? Mar 13, 2018 at 8:02
• @ShadowTheCurlyBracedWizard I just stumbled over this comment -- my very last commit to Stack Overflow code did just that.
– balpha Staff
May 31, 2019 at 5:43
• Awesome! Can you please mark that other bug as completed then? Oh wait... :) May 31, 2019 at 22:26
• I did the best I could :)
– balpha Staff
Jun 1, 2019 at 8:50