Occasionally, the Unicode character sequence U+200C U+200B (ZWNJ ZWSP) is inserted into comments

Question

In this comment to a question on TeX.SE

... the number "5000" does not paste as such but instead as

5
0
U+200C
U+200B
0
0

The HTML source looks like this:

perhaps the easiest would be <code>\edef\tmp{\everypar{\widowpenalty\the\widowpenalty\relax}}\tmp\widowpenalty=-50&zwnj;&#8203;00</code> that may reset

Or (smaller snippet):

\widowpenalty=-50&zwnj;&#8203;00</code> that

So in the HTML, we have extraneous ‌ and ​ in between the two middle digits of 5000.

Upon pasting and trying out this code, this first led to a mysterious compiler error. The same issue happened within a (now deleted) comment thread below this answer on TeX.SE; there, these or similar two characters were inserted after the letter "r" within the string \ref ("\ref").

Note that not all browsers wrap the same way. The above screenshot was for Firefox 19.0.2, in which it breaks between \relax}} and \tmp, with 5000 insidiously left intact on the second line but with the two special characters remaining in the middle. So that is why this took me completely by surprise. Other browsers might indeed only break on the invisible Unicode characters, such as Chrome 25:

Or on a smaller screen with the mobile site, multiple line breaks might show, possibly still leaving the Unicode characters unused:

Answer 1

As of about a month ago, these characters are no longer inserted, since comments are no longer laid out with a <table> (and the table layout algorithm was what prevented fixing overflow in any other way).

Answer 2

Unicode magic is inserted in comments (not just in `code`) whenever no whitespace was seen for about 80 characters, to enforce line wrapping. Without that the layout can really be messed up.

Jeff once explained why this magic is needed:

The word-wrap:break-word suggestion is a good one.

Works fine on posts, which are fixed width, and that is implemented.

Near as I can tell, there is no way to get this to work on comments because they are variable width. That is, the actual comment size depends on whether or not the comment vote UI controls are present (fex, on your own comment, you don't have controls, if you're not logged in there are no controls, etc), and if the comment has say 100 upvotes that's wider still.

One could argue if the above really is a good reason to not have a fixed width for comments on the full site. (The mobile theme is much more fluid, but uses word-break: break-all, so probably has no need for the Unicode markers, though they are present there too. And all that looks just fine.)

Using JavaScript to remove the markers upon copying, is not going to be easy. What about:

1. Instead of the proposed word-wrap: break-word, use word-break: break-all, just like on the mobile site. This works for me in Chrome 25, Firefox 19, Safari 6, Internet Explorer 9 (in Windows 7 on Parallels on a Mac; simulating IE8 and IE7 works too), stock browser and Chrome on Android 4.1, and Safari on iOS 6.1.2. But apparently it does/might not work in Opera? See this JS Bin example.

2. Or, instead of Unicode magic, insert <span style="display: inline-block"></span>. Tested in the same browsers, and probably also supported in Opera; see the same JS Bin example above.

3. Or, use JavaScript on hovering to get the actual width, remove the Unicode magic, and then explicitly set width and word-wrap: break-word. This might reflow the text on hover, but only for affected comments. (Alternatively, one could run the JavaScript for all comments, on page load and after each resize.) Also, this needs to restore the original text (with the Unicode magic) when resizing the window, to allow the browser to calculate the proper new widths:

   $(".comment-copy").hover(function() {
     var text = $(this).text();
     var clean = text.replace(/[\u200B,\u200C]/g, "");
     if(text !== clean) {
       $(this).css("width", $(this).width());
       $(this).css("word-wrap", "break-word");
       $(this).text(clean);
       if(!$(this).data("orig")) {
         // Store original for later resizing
         $(this).data("orig", text);
       }
     }
   });
   
   $(window).resize(function() {  
     $(".comment-copy").each(function() {
       var orig = $(this).data("orig");
       if(orig) {
         // Reset original text, to get the proper width
         $(this).text(orig);
         $(this).css("width", "");
         $(this).css("word-wrap", "");
       }
     });
   });
   

   (See the same JS Bin example; tested in the same desktop browsers.)

4. Or, if the above still yields problems: after posting, check if the magic was applied, and warn the commenter about it with a shiny blue info message? Like: "Beware: your comment includes a sequence of more than 80 non-space characters, which might cause issues when copying. Consider adding some whitespace."

5. And/or, bind to the copy event, check the content and warn if needed: "The comment you're copying includes invisible Unicode characters". However, this event might go unnoticed when copying more than just the comment.

6. Or, even though lines might be wrapped on another character: simply insert a true space instead? At least that is visible.

Some day, <wbr> will be the solution, but not today: though IE7 supported it, IE8 and IE9 don't.