Markdown formatting bug with code blocks in lists

Question

Have a look at the revisions of https://stackoverflow.com/revisions/1230618/list, in particular the formatting-related edit I made (edit 2). The formatted text contains spurious newlines and indentation errors (not to speak of the randomly inserted HTML escape sequence). Even deleting the newlines and re-inserting them didn’t help.

EDIT

Minimal example that still exhibits the buggy behaviour:

1. Item

   something
       foobar<T>
   

Source (displays exactly right):

1. Item

        something    
            foobar<T>

---

Another example, from an old posting of mine that worked before, but is now broken: Coalescing regular expressions in PHP

Notice how the posting’s source code ends on “<strong&”. Unfortunately, I can’t reproduce this bug here since the code looks fine in preview (click “edit” to look at it there); only the normal display is wrong, suggesting that the server-side Markdown processor has an additional bug.

Answer 1

The culprit was this tiny line in the Markdown specification:

If list items are separated by blank lines, Markdown will wrap the items in <p> tags in the HTML output.

Besides the fact that this is actually ambiguous (Which item is wrapped in <p> tags? The one before the blank line? The one after it? Both? All of them, once there's a blank line anywhere?) and that different implementations actually handle this differently, here's how the original implementation does it:

The first element is wrapped in <p> if it's followed by a blank line, the last element when it's preceded by a blank line, and all other ones when there's a blank line at least on one side. All items are wrapped if they contain blank lines.

All well, there's some consistency to this. The implementation, however, tries to express this logic in a regular expression:

    (\n)?                            # leading line = $1
    (^[ \t]*)                        # leading whitespace = $2
    ($marker_any) [ \t]+             # list marker = $3
    ((?s:.+?)                        # list item text   = $4
    (\n{1,2}))
    (?= \n* (\z | \2 ($marker_any) [ \t]+))

When a Markdown list is separated into items, this preserves one or two newline characters at the end (one meaning there's no blank line, two meaning there is), and one newline character at the beginning.

The presence of the double newline at the end means this item is followed by a blank line. The presence of a leading newline on top means this item is preceded by a blank line.

Well, almost. Because now we need three newline characters between the items, two to be captured at the end, and one to be captured at the top.

Well, that's easy:

# Turn double returns into triple returns, so that we can make a
# paragraph for the last item in a list, if necessary:
$list =~ s/\n{2,}/\n\n\n/g;

– and that's why code in lists started growing newlines.

Many Markdown ports to other languages do it the same way – including Attacklab Showdown (SO's client-side Markdown renderer) and MarkdownSharp (the server-side version).

After removing this piece of logic from the regex and instead keeping a little bit of extra state* between the list item processing steps, we could remove the double-triple-conversion.

_{* appropriately called last_item_had_a_double_newline}

Answer 2

There have been ZERO changes to any of the markdown code in months. FYI. Be happy to share SVN logs with you (privately) if you don't believe me.

I think this is simply a condition we hadn't discovered before. It seems to be based on spacing, like so:

1. uno

   code<t>
   code<t>
   

2. dos

   code<t>
   code<t>
   

3. tres

   code<t>           
       idented-12-spaces<t>
   

4. cuatro

   code<t>           
   
       indented-12-spaces<t>
   

Conclusion: this is not a new bug, surprisingly -- both the client and server side markdown renderers do the same thing here! That leads me to believe it's a narrow condition around code within numbered lists.

Bottom line, if you have code (8-space-indent) inside a bulleted list, you cannot have blank lines inside the code. Bizarre, but true.

Checking in http://babelmark.bobtfish.net verifies this is a bug in multiple markdown engines. Click here to see the above sample text in multiple markdown engines. There's definitely a disturbance in the force here..

Answer 3

Also note that additional scroll bars appear if there are long lines in a code block containing empty lines:

1. Create list

2. Add code:

   function() {
   
       // here we do lot's of stuff, and because comments are evil we get strange scrollbars displayed
   
   }
   

(See also the source view of this post)