25

Have a look at the revisions of https://stackoverflow.com/revisions/1230618/list, in particular the formatting-related edit I made (edit 2). The formatted text contains spurious newlines and indentation errors (not to speak of the randomly inserted HTML escape sequence). Even deleting the newlines and re-inserting them didn’t help.

EDIT

Minimal example that still exhibits the buggy behaviour:

  1. Item

    something
        foobar<T>
    

Source (displays exactly right):

1. Item

        something    
            foobar<T>

Another example, from an old posting of mine that worked before, but is now broken: Coalescing regular expressions in PHP

Notice how the posting’s source code ends on “<strong&”. Unfortunately, I can’t reproduce this bug here since the code looks fine in preview (click “edit” to look at it there); only the normal display is wrong, suggesting that the server-side Markdown processor has an additional bug.

1

3 Answers 3

33

The culprit was this tiny line in the Markdown specification:

If list items are separated by blank lines, Markdown will wrap the items in <p> tags in the HTML output.

Besides the fact that this is actually ambiguous (Which item is wrapped in <p> tags? The one before the blank line? The one after it? Both? All of them, once there's a blank line anywhere?) and that different implementations actually handle this differently, here's how the original implementation does it:

The first element is wrapped in <p> if it's followed by a blank line, the last element when it's preceded by a blank line, and all other ones when there's a blank line at least on one side. All items are wrapped if they contain blank lines.

All well, there's some consistency to this. The implementation, however, tries to express this logic in a regular expression:

    (\n)?                            # leading line = $1
    (^[ \t]*)                        # leading whitespace = $2
    ($marker_any) [ \t]+             # list marker = $3
    ((?s:.+?)                        # list item text   = $4
    (\n{1,2}))
    (?= \n* (\z | \2 ($marker_any) [ \t]+))

When a Markdown list is separated into items, this preserves one or two newline characters at the end (one meaning there's no blank line, two meaning there is), and one newline character at the beginning.

The presence of the double newline at the end means this item is followed by a blank line. The presence of a leading newline on top means this item is preceded by a blank line.

Well, almost. Because now we need three newline characters between the items, two to be captured at the end, and one to be captured at the top.

Well, that's easy:

# Turn double returns into triple returns, so that we can make a
# paragraph for the last item in a list, if necessary:
$list =~ s/\n{2,}/\n\n\n/g;

– and that's why code in lists started growing newlines.

Many Markdown ports to other languages do it the same way – including Attacklab Showdown (SO's client-side Markdown renderer) and MarkdownSharp (the server-side version).

After removing this piece of logic from the regex and instead keeping a little bit of extra state* between the list item processing steps, we could remove the double-triple-conversion.

* appropriately called last_item_had_a_double_newline

2
  • 2
    Amazing. I know from experience how hard debugging regular expressions is. And then there’s the complexity of parsing Markdown at all … William Shields’ attempt at parsing Markdown with a formal parser (as far as I see) hasn’t been fruitful either. Jan 6, 2011 at 17:21
  • 2
    amazing work Ben! Jan 6, 2011 at 21:30
8

There have been ZERO changes to any of the markdown code in months. FYI. Be happy to share SVN logs with you (privately) if you don't believe me.

I think this is simply a condition we hadn't discovered before. It seems to be based on spacing, like so:

  1. uno

    code<t>
    code<t>
    
  2. dos

    code<t>
    code<t>
    
  3. tres

    code<t>           
        idented-12-spaces<t>
    
  4. cuatro

    code<t>           
    
        indented-12-spaces<t>
    

Conclusion: this is not a new bug, surprisingly -- both the client and server side markdown renderers do the same thing here! That leads me to believe it's a narrow condition around code within numbered lists.

Bottom line, if you have code (8-space-indent) inside a bulleted list, you cannot have blank lines inside the code. Bizarre, but true.

Checking in http://babelmark.bobtfish.net verifies this is a bug in multiple markdown engines. Click here to see the above sample text in multiple markdown engines. There's definitely a disturbance in the force here..

6
  • 6
    Whitespaceisforwimpsanyway. Sep 2, 2009 at 22:14
  • I could have sworn that I’d already used this construct – and it’s not that unusual, after all. Well, if the logs say differently, that’s fair enough. Sep 3, 2009 at 10:14
  • 1
    I was surprised too but I can't think of anything that has changed -- it's duplicated on both client and server, and the babelfish Markdown processor (some of them anyway) show the SAME error. Sep 3, 2009 at 12:26
  • 1
    This is just righteous punishment for people who use 8-space-wide indenting.
    – TM.
    Dec 23, 2009 at 19:20
  • @GMan, i take his comment to be about 8-space-indenting in some random source code file, where actual tabs or 3-or-4-space-indenting is better. Dec 23, 2009 at 21:27
  • This would be awesome if it was fixed, since ctr-k doesn't work on code in a numbered list ( you have to hand indent the extra 4 lines ). Oct 3, 2010 at 0:12
1

Also note that additional scroll bars appear if there are long lines in a code block containing empty lines:

  1. Create list
  2. Add code:

    function() {
    
        // here we do lot's of stuff, and because comments are evil we get strange scrollbars displayed
    
    }
    

(See also the source view of this post)

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .