Markdown links and images are producing incorrect HTML

Question

The problem appeared recently.

Images and links are not displayed correctly.

Answer 1

This was entirely my screw-up. Here's what happened.

The problematic change was this one. Ironically the much larger Markdown change I made yesterday did not break anything.

This is how Markdown handles links:

1. Replace all advanced Markdown-syntax links and images (like [foo](http://bar.baz) and ![quux][1] with their corresponding <a href="..."> and <img src="..."> HTML.
2. Magic.
3. Replace the simple Markdown link syntax, whereby you place URLs between angle brackets (<http://google.com>), with their corresponding <a href="..."> HTML.

The "Magic" step is the auto-linking – take anything that looks like a plain link, and put it between angle brackets, so that step 3 will turn it into real HTML links.

Before I made the change, step 2 only handled URLs that were preceded by a whitespace character, which is why the fairly common idea of putting a link in parentheses:

A company called Google (http://google.com) runs a search engine.

never worked as expected – an open parenthesis is not a whitespace character.

The change I made was to allow any non-word character before the URL instead. Which was a horribly stupid idea (and in fact, I had previously shied away from this change because of exactly this problem – but for some reason, this didn't enter my mind yesterday until all hell broke loose).

Imagine this Markdown source:

[This website](http://rolex-watches.scam) is great.

After step 1, the intermediate HTML looks like this:

<a href="http://rolex-watches.scam">This website</a> is great.

Now comes step 2. Before my change, it wouldn't touch anything here, because the http:// was not preceded by a whitespace character. After my change, it did indeed touch it, because guess what – a quote sign is a non-word character:

<a href="<http://rolex-watches.scam>">This website</a> is great.

Now comes step three (linebreak added for readability):

<a href="<a href="http://rolex-watches.scam">http://rolex-watches.scam</a>">This
    website</a> is great.

Then the HTML sanitizer removes the illegal "tag" from the first < to the first >, leaving

http://rolex-watches.scam</a>">This website</a> is great.

and finally the tag balancer removes the two closing </a>s, because they have no corresponding opening tags:

http://rolex-watches.scam">This website is great.

– and that's what you saw.

So (except for just forgetting this issue despite having previously considered it), why didn't I realize this problem?

Well, the test post I played with while testing had no Markdown-syntax links, just bare URLs. So step 1 never did anything, and nothing broke.

I also ran the test suite in MarkdownSharp, and it didn't report any issues with the change either. Why? Because unfortunately, step 2 is a configurable option in MarkdownSharp, and turned off by default. And most tests run with the default options. And those few test cases that enable this option only test the auto-linking – they don't have any links for step 1 either.

Kevin backed my change out, since he felt that unbroken Markdown would be a nice thing to have, and I tend to agree with that.

This question is thus not completed at the moment, but it will be again after the next build. I've changed step 2 to require a URL to be preceded by a non-word character and neither by =" nor by < to be eligible for auto-linking.

And just to be sure, this time I have not only run the tests with the option on; I have also tested the change against a couple of thousand real-world questions and answers to make sure that the only differences in rendering are intended ones.

Answer 2

A fix for this has been deployed. It looks like some recent changes to the Markdown rendering pipeline went sideways.

Affected posts will display incorrect HTML until they're edited (as we cache the rendered Markdown)... which means I'm going to be poking a bunch of posts for the next hour or so.

---

Poking completed, all affected posts should have been corrected. If you see anything left over, let me know.