578

I’m Ham and I’m a developer on the Teams team here at Stack Overflow. Over the past few months, I’ve been heads down working on the way we turn Markdown into HTML when writing and editing posts across the network. I’d love to share what I’ve come up with.

In a nutshell: We're planning to use CommonMark for all posts across the network moving forward. To do so, we switch to CommonMark-compliant Markdown renderers on the client and the server side. We have to make sure that all existing posts work with the new renderers so we will run a big migration across the network that will convert existing posts to use the new CommonMark format. Writing, editing and reading posts should look and feel mostly the same after the change.

As of June 20, 2020, all sites are on CommonMark now. For individual sites, see the migration schedule here.


We’re using Markdown throughout the Stack Exchange network. Markdown was one of the early technology bets when Jeff and Joel started out building Stack Overflow. If you write a question, an answer or a comment anywhere on the Stack Exchange network, you’re going to write it in Markdown.

Over the years, Markdown has become a common way of writing content in online communities. It has become a wild success and even got a formalized specification with CommonMark.

Stack Exchange’s way of handling user-created Markdown today is largely the same as it was when we started. We’re using our own, home-grown Markdown parsers and renderers on the client and server side. Both of these implementations have proven to be a solid foundation and received a lot of tweaks over the years.

However, they come with their own quirks. Being created before there was a CommonMark spec they show some non-spec-compliant behavior. They’re using regular expressions for transforming Markdown into HTML (I’ll leave it to your imagination how much sweat and tears this has cost us over the years) which is perfectly doable but makes maintaining and adding new features to our Markdown parsers extra hard.

The idea

We think it’s time to move forward. You’ve asked if we’re ever going to adopt CommonMark on the Stack Exchange network a few years back, balpha ran the numbers, and while he found out that it wasn’t impossible, it didn’t seem to be easy peasy, either. With some of the past and upcoming changes, we think that now’s a great time to tackle this challenge and to migrate all network posts over to CommonMark. This includes:

  • Changing the Markdown renderer on the client side
  • Changing the Markdown renderer on the server side
  • Automatically editing and re-rendering all posts across the network that are not CommonMark-compliant

To give you a better feeling for the changes under the hood: When you write a post on the Stack Exchange network, you write it in Markdown. On the client side, you see a preview of your post as you’re writing it. This preview is created by our client-side Markdown renderer. It takes the Markdown you write, transforms it into HTML and shows you a preview of what your post will look like.

Once you save your post, we send your Markdown over to our servers where the same Markdown-to-HTML conversion takes place, again (you can’t trust user input, so we don’t blindly accept the HTML generated on the client side).

Our plan

We will migrate sites across the network to CommonMark site by site over the next couple of weeks. We plan to start with Meta Stack Exchange and Meta Stack Overflow on Wednesday, June 3rd 2020.

I’ve prepared a feature that will swap out our current, home-grown Markdown renderers with well-tested open-source implementations that adhere to the CommonMark specification. For the curious: this means we’re replacing PageDown with markdown-it on the client side and MarkdownSharp with markdig on the server side.

Once we enable that feature, new and edited posts will automatically be rendered with those new renderers. Most likely, you won’t even notice a difference when looking at posts.

With the new renderers in place, we’re going to move all existing posts across the network over to CommonMark. For the vast majority of the posts across the network (80% and up), this means nothing will change. Most of the posts on our network have been written in a way that is completely compliant to the CommonMark specification already, yay! If we convert this Markdown to HTML using a new renderer, the results will be exactly the same.

Then there are those posts that are written in a Markdown flavor that was cool for our current renderers but isn’t what CommonMark would expect. Balpha’s analysis is giving you more details on that. We’re talking about ##headlines without spaces after the hashes and other minor oversights. For these posts, we’ve built a tool that automatically fixes these well-known issues by changing a post’s Markdown source directly and re-rendering the HTML of the post in question. When we change a post’s Markdown automatically, this will end up looking like a regular edit but we’re making sure that this won’t bump posts to the top.

So now we’ve got about 80% of our posts that are already good to go. With the auto-fixing utility we estimate that we’re going to land at over 96% of all network posts being rendered completely identical after migrating to CommonMark and using the new renderers. This leaves us with a few percents of posts that end up looking differently when rendered with the new CommonMark renderers.

What you can expect

We avoid breaking existing posts by erring on the side of safety. If a post looks different using the new renderer (and if it’s just one whitespace off) we won’t automatically re-render the post and put it up for investigation first. This way we can be sure that all changes are safe.

I’ve played around with our data to get a feeling for the posts that will be rendered slightly different after using the new renderers. I found out that the differences fall into three buckets:

  1. False positives: the HTML markup changed slightly but doesn’t change semantics or presentation of the post
  2. Improvements: things where the CommonMark specification fixes some oversights in our current Markdown flavor
  3. Actual issues: things that we didn’t anticipate and need to fix

The "actual issues" category should be a tiny fraction but I won’t naively assume that they won’t happen. There will be some changes caused by the new Markdown renderer that we need to investigate because they will cause posts to look different than before in one way or another. We can’t foresee all edge cases that this change will introduce so we will surface all posts that look different when rendered with the new Markdown renderer, review them and if necessary fix them.

Ultimately, we hope to make this transition as smooth and frictionless as possible. We don’t want to break (and manually fix!) thousands of posts. We don’t want to mess with your writing experience. At the same time, I know that we won’t get this feature perfect from the get-go, so I need to ask for your patience and understanding.

As you write new posts after we’ve made the switch to the new CommonMark renderers, you will have the exact same writing experience as before. The preview will show you what your post will look like and once you save your post, it should appear just as you saw it in the preview. If you notice any differences between preview and saved post, please let us know!

Things might get funky when you're editing a post that renders differently with the new CommonMark renderer. Again, if we detected that a post would look differently when rendered with the new CommonMark renderer during the migration, we wouldn't save a new version of this post as part of the migration. This way, all posts continue to look the same when being viewed. However, once someone comes in and edits it, it will be rendered using the new CommonMark renderer and this might cause the post to look slightly different than what we had before. This will only be a small fraction of all of our posts, and of that small fraction a smaller fraction will actually be edited moving forward. However, it's important to keep in mind that editing old posts has a slight chance that you run into differences between our old and our new markdown renderers.


Frequently Asked Questions

When is this going to happen?

The new CommonMark renderers are being merged into master within the next few days. They’re hidden behind a feature flag, so they won’t do any harm until we flip the switch.

We will migrate sites across the network site-by-site over the course of the next couple of weeks. We will start with Meta Stack Exchange and Meta Stack Overflow on Wednesday, June 3rd, 2020 (assuming everything goes well and we don't discover a major blocker until then). Since we can't exactly predict what kind of dragons we'll encounter along the way, plans might change slightly. I'll post a plan for sites and their switchover dates as an answer to this question and will keep it updated as we go.

Every site is different and we need to learn as we go. Most sites can be migrated within a few hours. For our biggest network sites, changing all posts to CommonMark will probably take up to 4 days. Keep an eye on the schedule I'll post to see how we're doing.

Why are we migrating to CommonMark?

In the past, changes to our Markdown renderers have been rather risky and high-effort. We needed to carefully evaluate if a change breaks anything for the millions of existing posts we have in place. By sticking to a well-defined specification like CommonMark, we can make sure that implementations that stick to this specification will work for us. If the specification gets extended, adopting changes will be easy and safe.

Another reason is that this reduces some of the maintenance burdens of our development teams. Instead of maintaining two distinct Markdown renderers, we can now pick something off the shelf and use that instead. With markdig and markdown-it we’ve found two reputable libraries that are beating our own implementations when it comes to performance and functionality. Both are great pieces of software that we're more than happy to use in our product.

Are there some changes to the way I can write Markdown in the future?

Yes, there will be a few changes to the set of supported Markdown on Stack Exchange. For the vast majority of your writing, you won’t see a difference at all. We’re doing our best to continue to allow most of the syntax you can use right now. We’re adopting the CommonMark standard, so everything that’s valid CommonMark will work on Stack Exchange moving forward (here’s a short cheat sheet for the curious).

At the same time, we want to take this opportunity to remove some quirks we’ve built during times where there was no such thing as a CommonMark standard. Some features in Stack Exchange’s current Markdown flavor have been built during a time where there was no standardized way of doing things. Now that we’re adopting CommonMark, we want to replace some of those homegrown features with standardized notation, a notation that you know from other places all around the web as well.

The most noticeable changes will be around lists, nested lists, headlines and blockquotes.

Lists: When creating nested lists, you’ll need to indent your nested list items or paragraphs with the right amount of spaces. While one space was enough before, you’ll need to add a few more now, depending on your type of list.

To make a paragraph part of a list item it used to be enough to add one space in front of the paragraph

* this is a list item

 that goes on here

with CommonMark, the paragraph has to line up with the text of the parent, so we need a few more spaces here:

* this is a list item

  that goes on here

Headlines: Moving forward, you have to add a space after the leading # characters.

#this was cool before
# this is what's cool now

Blockquotes: Previously, empty lines between two lines marked as blockquotes would make one big blockquote. Moving forward, you'd get two distinct blockquotes this way, unless you start the empty line with a > character, too:

> old blockquotes  

> with multiple lines
> new blockquotes
> 
> with multiple lines

Indented code block highlighting: Our biggest breaking change will be around indented code blocks and the possibility to declare the language to be used for syntax highlighting.

In a nutshell: If you want to declare the language for syntax highlighting in your code block, use the code-fence notation and not indented code blocks. You can still use indented code blocks, but declaring the preferred language explicitly for them is no longer supported moving forward.

Until now, you could do this to declare the language for an indented code block:

<!-- language: python -->

    def hello():
        print("Hello, World");

Moving forward, this style is considered deprecated. Ever since we’ve introduced code fences, you can explicitly declare the language of a code block using the code fence notation:

``` python
def hello():
    print("Hello, World");
```

This is the way the CommonMark standard is proposing and this is what other websites are doing, too. We know that you might have gotten used to using the old syntax featuring a <!-- language: lang --> comment. As we’re adopting new Markdown parsers, we want to avoid patching quirky behavior into those parsers when there’s an official, standards-compliant way of achieving the same goal that we can adopt with no extra effort. This style will continue to work for the time being, but is subject to removal in the future, at which point posts using it will no longer recognize it.

Note that setting the syntax highlighting language based on the tags you’ve associated with your post will continue to work. Here’s a full overview of the current behavior of our syntax highlighting if you need a refresh. We’re going to update that post as we move forward.

What happens to SE-specific syntax elements?

On the Stack Exchange network we support some syntax elements that are not part of the CommonMark standard. Things like spoilers, MathJax, circuit diagrams, stack snippets, etc. are used on several network sites. We're going to continue to support all of those custom syntax elements even if they're not part of the official CommonMark specification.

Will this finally enable table support?

Maybe! Support for tables has been discussed intensely in the past. There are many creative workarounds out there but never any official support for rendering tables. If other sites can pull this off, why can’t we?

One of the main reasons, the fact that our Markdown parsers and renderers didn’t support tables, is now no longer valid as we’ve switched to Markdig and Markdown-it. Both support parsing and rendering tables out of the box. Still, introducing table support is a change we don’t want to blindly stuff into this big migration.

Let’s get everything to work nicely with the official CommonMark specification first – and just to be clear, tables are not part of that spec. This change is massive; we need to see how this plays out and make sure this doesn’t introduce more than a few acceptable cosmetic issues across all our communities.

Once the dust has settled and we’re all comfortable with the new Markdown renderers under the hood, we can re-evaluate if the time is right to bring table support back to the (drumroll) table!

  • 50
    Does this also mean we're getting header IDs? – Zoe the 1337 Princess Jun 1 '20 at 11:56
  • 17
    Spoiler syntax is going to remain the same - although it's not part of the CommonMark specification. – Ham Vocke Jun 1 '20 at 11:59
  • 3
    @ZoeTheLockdownPrincess "You can still use indented code blocks but can’t declare the preferred language explicitly moving forward." – Nick Jun 1 '20 at 12:00
  • 39
    This migration won't enable header IDs. This migration is already a big thing so we don't want to conflate adding new features with running the migration itself. Both, markdown-it and markdig support header IDs via plugins so implementing this feature will now be easier than before - but it remains a different discussion. – Ham Vocke Jun 1 '20 at 12:05
  • 4
    @Laurel the apps likely don't render the HTML themselves, but rather it's done in the API level. If that's the case, apps won't need any change. But if the render is done in the app itself.... this is essentially the final straw and they'll have to shut them down. Waiting for official response. – Shadow Wizard is Vaccinating Jun 1 '20 at 12:34
  • 11
    I run the markdown through the old renderer, run the markdown through the new renderer, scrub both HTML versions with good ol' regular expressions and compare the two HTML strings. It's not sophisticated but gets the job done just fine and is fast enough to handle millions of posts in my lifetime. I hope to get a blog post out soon where I can share more insights. – Ham Vocke Jun 1 '20 at 18:17
  • 10
    What about RTL direction, currently unsupported by CommonMark? – Zev Spitz Jun 1 '20 at 18:39
  • 13
    Oh boy, here we go again... – Paulo Cereda Jun 1 '20 at 18:55
  • 9
    @Mast Huh? If the displayed text of a post would get altered by the new Markdown engines, then its Markdown will not be updated, even if the difference is a single whitespace. It will continue to be displayed via its current HTML, which was created by the old Markdown engine. So its appearance will be unaltered. However, when someone attempts to edit such a post they will have to comply with the new Markdown rules. This may cause problems. Eg, someone edits a post to fix some minor thing but then discovers that they need to make major changes so that the post renders correctly. – PM 2Ring Jun 2 '20 at 10:44
  • 76
    Tables ASAP, PLEASE! – Ian Kemp Jun 2 '20 at 11:49
  • 4
    What about placeholders of the form [...], such as tag ([tag:discussion] for discussion) or site reference ([scifi.se] for Science Fiction & Fantasy)? Are those just modelled as links whose definition is invisible at the time of writing? – O. R. Mapper Jun 2 '20 at 11:52
  • 31
    Because we can't reasonably support two different active markdown renderers without tripping eventually. There are good reasons to move forward outlined in the post - compatibility, user experience, ease of maintenance, simpler future feature development being some of them. – Ham Vocke Jun 2 '20 at 15:31
  • 4
    This is good, in my opinion. It is absolutely true that Markdown is an incomplete spec, and some solid flavor needs to be used instead. My peronal favorite happens to be kramdown, but it's not a good choice to substitute for standard Markdown in most contexts. CommonMark sounds like a good selection. – matt Jun 2 '20 at 21:05
  • 4
    @Sean You can nest blockquotes pretty much the same way you nested them before. Instead of using > you'd start each line of a nested blockquote with >> (or more characters if you want to nest even deeper). – Ham Vocke Jun 4 '20 at 6:35
  • 10
    Yay! Fantastic! I've updated the commonmark.org website to reflect this change! – Jeff Atwood Jun 28 '20 at 17:29

55 Answers 55

1
2
5

Multi-paragraph quote blocks within lists

The automatic migration on Movies & TV omitted this answer, either because it failed to convert it into an HTML that didn't change or because it didn't recognize it as a case that needs adaptation.

However, it clearly needs it since it has both multi-paragraph quote blocks and multi-paragraph bullet points and when trying to edit it, it looks like a mess needing manual fixing. The problem seems to be, though, that it has multi-paragraph quote blocks inside a list, so it doesn't seem to be able to introduce both the added indentation and the > quote marker on the same line.

  • 2
    You're right, the block quotes need an additional leading space to be properly indented as part of the parent list item. This is one of the weak spots for the autofixing mechanism. I'll try to tune it using this particular example but ultimately this might be one of the cases where we just have to deal with manual fixing since every tweak I introduce to the autofixer risks breaking other occasions that look similar but aren't quite the same. – Ham Vocke Jun 4 '20 at 15:41
  • @HamVocke That...would be more than unfortunate. I can't believe this being the only case of a multi-paragraph quote block inside a list, nor this being particularly hard to fix. Trying to solve the problem generally more than tuning it to this specific answer I brought up as just an example might be a better idea. – Christian Rau Jun 4 '20 at 15:44
  • 4
    @HamVocke If this is not at all possible, though. Please, please consider sharing the list of posts that failed the HTML comparison and should definitely contain these cases. At least people would have a hint at which posts might need manual fixing then, especially when things get tuned to specific cases and it's left unclear what other possible edge cases might exist that haven't surfaced yet. – Christian Rau Jun 4 '20 at 15:45
  • 5
    I understand that this is less than ideal. The migration script is designed in a way that it can be tweaked and re-run on a site to pick up some things that haven't been picked up before (in fact, I did exactly that this morning on this site). I'm keeping an eye out on the failure cases and so far the one you highlighted is one of the few more uncomfortable ones. I'm not trying to hide anything and am happy to share a list if I can reasonably do so. The thing that's holding me back is that the list I have consists of 95% false positives that are hard to weed out even after weeks of tweaking. – Ham Vocke Jun 4 '20 at 16:15
  • @HamVocke Okay. False positives as in different HTML code but still looking the same? – Christian Rau Jun 4 '20 at 16:19
  • 6
    lots of papercuts, really. Visually equivalent but structurally different HTML in most cases, yes. Unfortunately it's pretty hard to weed out everything. I've spent the better part of the past few weeks trying and decides that ultimately we can't eliminate them all with reasonable effort. – Ham Vocke Jun 4 '20 at 18:01
5

How is this going to affect site performance?

Right now, Stack Exchange and its Markdown rendering is one of the only remaining sites on the internet that's blazingly fast, without UX lag all over the place. Will this stay the same under the new renderer? Or will the new CommonMark implementation be built on ten layers of obscenely bloated JS frameworks that makes the browser slow to a crawl and makes the site unusable, like basically every other dynamic site has become?

  • 1
    Well, you wrote this answer using CommonMark, how did it feel? – Spevacus Jun 11 '20 at 17:33
  • 1
    @Spevacus: Excellent, exactly what I wanted to hear! – R.. GitHub STOP HELPING ICE Jun 11 '20 at 17:36
  • 1
    Given that posts are converted to HTML only upon creation/editing, I don't see how it could affect any random everyday page loads. – Christian Rau Jun 12 '20 at 9:03
  • 1
    @ChristianRau: Loads are not the problem, editing is. To actually use this site you use the editor, writing questions and answers. Lots of dynamic sites with custom editing widgets have up to 1s latency for each keypress. Patreon has it and new Twitter had it at first but has mostly fixed it. Facebook kinda has it too but it stabilizes over time (JIT kicking in?) Others are not as extreme but still have serious usability problems from JS event handlers running on everything you input, so this absolutely was/is a legitimate concern. – R.. GitHub STOP HELPING ICE Jun 12 '20 at 17:09
5

User profile descriptions haven't been re-rendered with CommonMark, neither have they been migrated

Even on sites that have been migrated over to CommonMark, user profile descriptions haven't been migrated to use the new renderer, and prior user profile descriptions are still rendering as is.

I noticed this when editing my my user profile, which has a heading that uses the now-no-longer-valid form of not having any space after the hash signs (i.e. ##heading):

Heading that uses now-invalid markup still renders

Here's a screenshot of my profile editor, to prove that that heading is the result of the now-invalid form above. (The preview is rendered with CommonMark, but this doesn't match the output in the above screenshot.)

Profile editor, which uses CommonMark renderer

I initially thought that this was because user profiles hadn't been updated to use CommonMark, but when I saved my profile without making any new changes, it re-rendered with CommonMark, and the heading didn't show up correctly (as would be expected).

Are user profile descriptions going to be migrated to CommonMark just like posts, with them all being re-rendered and edited by the migration script, or will they remain as is?

And if they are to be retained as is, can there please be warning notices on the editor page to notify the user to check to see if their profile description would render the same? Saving the profile without making any changes to the description field (e.g. to just change something else, such as the username, location, etc.) will re-render it with CommonMark, potentially making it break unexpectedly if the bio uses now-unsupported markup. (I read the comments indicating that this would be difficult on posts because of having to detect issues, but this is much easier as the warning can just be applied to all profiles that haven't been edited since the site's transition.)

  • 4
    You're right, this migration only tackles posts (i.e. questions and answers). Ultimately, we want to use CommonMark for tag wikis and user profiles too so we can get rid of the old Markdown renderer. We need to take careful steps, that's why tag wikis and user profiles are not included in this migration. We'll follow up with replacing the renderer properly for both at a later point. – Ham Vocke Jun 15 '20 at 6:43
  • 2
    @HamVocke As my answer states, on transitioned sites, user profiles are rendered with the CommonMark renderer when the user edits any part of their profile (not just the description). Is it possible that a warning can be shown on profiles that haven't been edited since the site's transition and whose about me isn't blank that they should check their description to see if it would render differently? – Sonic the Curiouser Hedgehog Jun 15 '20 at 6:49
  • 6
    Sorry, early morning confusion here. You're right, the new renderer is already being used for edits in tag wikis and user profiles, but existing profiles aren't migrated. Showing a warning sounds like a possible solution. I'll have to do some digging to see if there are any dragons to be aware of. – Ham Vocke Jun 15 '20 at 7:22
  • 1
    @HamVocke as irony would have it, the tag wiki for markdown is affected. I left it in its current state in case you want to investigate. – Glorfindel Jun 16 '20 at 12:26
  • 1
    @Glorfindel thanks! I've edited the description manually to make it commonmark compliant. It was an issue with mixing HTML and markdown. Unless we'd run a automated migration for tag wikis, there'll likely be more of those cases floating around. – Ham Vocke Jun 16 '20 at 12:47
5

Newline added between links listed in "[1]: link" format

I'm honestly just wondering what happened to one of my posts. Viewing the source before the community bot edit and the source after that edit the only difference I can see is that a newline was added between two of the links. Did this happen because I (for reasons even I don't know) used [1], [1.5], and [2] for my links? (I used them in this post and nothing seems to have gone wrong)

I don't believe this really counts as a bug of any sort, but I am wondering what actually happened in this case; why did the bot add a newline in the middle of my links?

  • 3
    Yep, this is a case where the auto-fixer incorrectly made an edit. The resulting HTML is still the same which is why this edit got saved. The auto-fixer is inserting empty lines before link references at the end of the document if there's no empty line. Because of the [1.5] formatting, it didn't recognize that this line was a link reference in itself. – Ham Vocke Jun 17 '20 at 8:22
  • 1
    @HamVocke The link references can be any unique string which is explicitly called out in the help centre: "The link definitions can appear anywhere in the document -- before or after the place where you use them. The link definition names [1] and [yahoo] can be any unique string, and are case-insensitive; [yahoo] is the same as [YAHOO]." Does this mean those [yahoo] ones will have this bug as well or is it solely a problem because of the [1.5]? – TheLethalCarrot Jun 17 '20 at 8:37
  • But then again, is this really a bug, as the resulting rendered page is not affected in any way? – Luuklag Jun 17 '20 at 8:50
  • 1
    @TheLethalCoder I know. The assumption I made in the auto-fixer was overly simplified. I don't really think this is a problem beyond a slight annoyance. Only link references that include punctuation characters are affected. – Ham Vocke Jun 17 '20 at 8:54
  • @HamVocke Oh yeah I don't think it would need correcting, just wondering what would be affected by it. Good to know. – TheLethalCarrot Jun 17 '20 at 8:56
5

I just stumbled across this old answer on Money.SE. It has two numbered points, with text in the middle. The text and the preview both have a 1. and a 2., but on the main page it renders as two 1.'s. I don't know what it looked like before.

Main: Main page Edit: Edit Preview: Preview

  • That answer wasn't migrated so it is still rendering with SE's markdown at the moment I believe. At least that was my understanding of how it works, could be wrong. – TheLethalCarrot Jun 19 '20 at 13:46
  • @TheLethalCoder - Does that mean the double #1's is how it's always been, and CommonMark would clean it up once someone made an edit? How can you tell whether an answer was migrated? – Bobson Jun 19 '20 at 13:49
  • 2
    The timeline for the post (example here) will show a Community edit with "Commonmark migration". Per my understanding, but I may be wrong, only migrated posts (and new ones) are now using CommonMark, everything else is still currently rendering with markdown. I could have completely misunderstood though and only "problem" posts aren't using CommonMark now. – TheLethalCarrot Jun 19 '20 at 13:52
5

It might be a good idea to migrate the code-sample button (in the editing bar on top of the edited post) to use the fencing option if that's going to be the reasonable default. Some indication on how to change the language of the code block might be a good idea for new users, or they may stay unaware of that option.

Currently the description of the edit buttons list the way you can manually insert HTML, rather than Markdown. That may lead to posts with a mix of Markdown and HTML rendering. It certainly seems to void the principle at least surprise: hit a button that says it inserts <ol> or <pre> and then use a Markdown list or indent / fence instead. Personally I'd just leave it out: I can see what happens when I press the button. Less is more.

  • 2
    Already mentioned above: meta.stackexchange.com/a/349575. – Emil Jeřábek Jun 23 '20 at 10:28
  • 1
    Ah, right, and I tried to scan through all the answers first. There are quite a few of them so I must have skipped it. I'll leave it here for the other remarks. – Maarten Bodewes Jun 23 '20 at 11:00
5

Again, if we detected that a post would look differently when rendered with the new CommonMark renderer during the migration, we wouldn't save a new version of this post as part of the migration.

Is there a list of these posts available per stack (or queryable from SEDE), so that we can fix these manually?

  • 1
    No. During the migration we saved the IDs of all posts that couldn't be migrated in Redis so we can get a feeling for the overall shape of the migration failures. However, this data has a bad signal to noise ratio, flagging lots of posts that aren't problematic after all. As much as I'd love to publicly share that data for people to go and investigate, I feel that this would rather be counter-productive and send everyone off on the wrong tracks. – Ham Vocke Jun 29 '20 at 10:06
5

The stackoverflow editing help page

links to the original John Gruber markdown specs in the "Need more detail?" section.

Shouldn't it point to the CommonMark specs?

4

When we change a post’s Markdown automatically, this will end up looking like a regular edit but we’re making sure that this won’t bump posts to the top.

I've noticed several posts here on Meta that seem to have been bumped by the community user's "Commonmark migration" edit to an answer:

The 3rd revision to this answer for Change "comment" to "critique or request clarification"
The 2nd revision to this answer for SE site for architecture design questions
The 2nd revision to this answer for this recent question: What did Yaakov do to this post?

It looks like the question isn't bumped (e.g. the first one was bumped by a spam answer), but the Community user is shown as the latest active user, with a timestamp belonging to another action that last bumped the post. Here are some screenshots for the second question:

enter image description here

enter image description here

enter image description here

enter image description here

  • 5
    We're using the community user to create the new revision and deliberately suppress edit bumps. Looks like for all three posts listed here, there's been activity (other edits, deleted answers) during a similar time frame and that caused the activity bump? Totally plausible that I'm missing something here and please correct me if I'm wrong but to me it looks like everything's working as intended. – Ham Vocke Jun 4 '20 at 12:56
  • 1
    That makes sense. I'll leave the answer here in case anybody else sees this behavior and comes looking. – 1201ProgramAlarm Jun 4 '20 at 16:19
  • 1
    For the first one: looks like the spammer account getting nuked, the user will be shown as Community user instead. – Meta Andrew T. Jun 5 '20 at 7:06
4

Perhaps update the PageDown and MarkdownSharp readmes with a link to this question?

4

There's a weird bug with HTML comments, where it collapses future paragraph breaks:

<!-- -->Paragraph A

Paragraph B
Paragraph A

Paragraph B

<!-- -->
Paragraph A

Paragraph B

Paragraph A

Paragraph B

  • 3
    This is one of the things where CommonMark is slightly different than our old renderer. Mixing HTML with Markdown is always a fun adventure (always has been) and in this case I'll argue that this works as designed. If you really have to write raw HTML in your CommonMark, make sure to not mix HTML and Markdown but to separate it with an appropriate amount of newlines. Otherwise, CommonMark will interpret everything as one big HTML block and ignore your Markdown. – Ham Vocke Jun 12 '20 at 6:35
  • 1
    For more details take a look at this example and check out the AST tab. You'll see that in your first example Paragraph A is considered part of the HTML block, thus no paragraph node will be generated. For the second example, the HTML block stops after your closing comment and Paragraph A is detected as a paragraph according to regular Markdown. – Ham Vocke Jun 12 '20 at 6:37
  • Interesting. Not hard to work around in this case. – curiousdannii Jun 12 '20 at 7:40
4

Are tag wikis missed in the CommonMark migration?

Several tag wikis make use of headers and blockquotes, but no edits have been made by the Community User on tag wikis to change ###This header to ### This header and to add > between consecutive blockquotes.

For example, see the tag wiki here for . (Posts on Meta SE were migrated to CommonMark on 2020-06-03 as per the given schedule.)

  • 5
    Yes, for now we're only migrating posts (i.e. questions and answers). When writing or editing a tag wiki, you'll use the new renderer but we're not migrating existing ones to the new format for now. – Ham Vocke Jun 15 '20 at 6:46
4

Would it be possible to use the occasion to throw in highlighting of specific code lines (as opposed to syntax highlighting) by providing the info in the infostring?

Something like:

```cs highlight=15
<code>
```

And then it would highlight the line 15 as if with a highlighter.

Or, for a more complex example,

```cs highlight="23-25, 31"
<code>
```

We do have a recurring problem of people trying to use ** in codeblocks to highlight specific lines. In some languages it's easy to spot and edit out, in some other languages it may even look like a valid attempt at pointer manipulation, which confuses everyone.

4

Will underscore's styling role continue?

I just finished reading through commonmark.org's help section and I'm quite pleased with the rationale behind SE's decision and CommonMark's mission.

One caveat I often find, however, is the underscore used for italics, since it is so often used in file names as well. This would imply that file names must always be placed inside inline code ticks.

Consider:

some_file_name_here.doc

is actually:

_some_file_name_here_.doc

So, I have a few questions about underscore-styled italics in the future:

1. What are the percentage stats in using _ vs * for styling italics?

2. Is it considered "wrong" to use a filename outside of backtics `` since file names may start/end with the italics styler _?

  • 1
    Interesting proposition. For me it feels wrong to format a filename as code. The alternative would then be to escap each underscore, which is tedious work. – Luuklag Jun 30 '20 at 8:57
  • @Luuklag I agree about formatting filenames as code, et cetera. I intentionally do not format my filenames as code as a matter of my own style consistency. – Jesse Steele Jun 30 '20 at 8:59
  • 2
    I did some testing in the Sandbox And this will not be as problematic as I initially thought. It will only affect filenames that start and end with an underscore. – Luuklag Jun 30 '20 at 9:19
  • I saw that, and it is good, but still... It is not "infinite". "Infinite vs usual" would be the issue. – Jesse Steele Jun 30 '20 at 9:24
4

Migrated questions show up in the Reopen queue like this and this because they "have been edited after they were closed".

  • At least you know which users to ban for voting to reopen now :) – Luuklag Jul 3 '20 at 20:58
3

Share links followed by a dot within markdown syntax like italics (*) have the dot included in the link. Before the dot was not included.

*Read mode about this in https://meta.stackexchange.com/q/348746/218578.*

Read mode about this in We're switching to CommonMark


Note that in the preview, the behaviour is still as before, the dot is not included in the link:

enter image description here

  • Duplicate of meta.stackexchange.com/a/349244. – Emil Jeřábek Jun 23 '20 at 10:30
  • I also raised a similar problem here. – Big Brother Jun 23 '20 at 11:04
  • @EmilJeřábek I've seen that before posting. But it's slightly different, and mainly the other problem seems gone. Their "used to render as" and "but now renders in the preview as" sections are now identical. – Martin Prikryl Jun 23 '20 at 11:30
3

It seems I cannot paste images anymore. When I do Ctrl+V and then submit it, it loads for a while and then goes back to the text without inserting anything, and I get this error in the devtools:

Uncaught Error: Markdown renderer is not initialized
    at Commonmark.Converter.makeHtml (wmd.en.js?v=65d6d3725fef:1)
    at a.y.addLinkDef (wmd.en.js?v=65d6d3725fef:2)
    at l (wmd.en.js?v=65d6d3725fef:2)
    at Object.<anonymous> (wmd.en.js?v=65d6d3725fef:2)
    at i (jquery.min.js:2)
    at Object.fireWith [as resolveWith] (jquery.min.js:2)
    at y (jquery.min.js:4)
    at XMLHttpRequest.c (jquery.min.js:4)

Not sure if this is really because of the new renderer or something else that happened and it sounds weird that nobody else reported it here yet, but the error does come from Commonmark.Converter so I thought maybe I should post it here anyway.

EDIT: I cannot reproduce it any longer, not sure what was going on there. I had it several times in a row but now no matter what I do, it always works correctly.

  • 1
    Good catch, this is indeed caused by us lazy-loading the new CommonMark renderer. Looks like I missed the "user pastes an image without entering text first" case :) – Ham Vocke Jun 29 '20 at 12:11
  • Actually I did enter text. I wrote one line and then pasted. – CherryDT Jun 29 '20 at 12:12
  • And now I cannot reproduce it (it works even without first entering text), yet before it happened several times in a row. That's weird. – CherryDT Jun 29 '20 at 12:14
  • Nope, no matter what I do now - answer, second answer, edit, with and without draft, starting to type before page is done loading, etc. I cannot reproduce it again :-| – CherryDT Jun 29 '20 at 12:21
  • 1
    Because you only need to load it one time, unless you clear your cache I guess. You could check in a different browser or an incognito browser screen. – Luuklag Jun 29 '20 at 12:50
  • 1
    @Luuklag it should be loaded on demand even if you haven't cached it locally. Turns out I did cover the edge case I thought I'd missed as part of this bug fix so I don't see why this wouldn't be working. Can't reproduce myself, so I'm gonna mark this as norepro for now. – Ham Vocke Jun 29 '20 at 13:23
2

This was probably already asked somewhere in the comments, but I couldn't find it. I prefer to write my answers in raw (X)HTML, and the only way to get syntax highlighting there until now were the comments. Will comments still work in such cases? If not, can I write:

<pre><code class="language-c">
</code></pre>

Like some CommonMark adopters allow users to do?

  • 6
    As your answer proves (we're rendering this one with CommonMark already) it continues to work. I'm not here to tell you what to do but if your reason for using HTML over Markdown was Markdown's quirks, things might have become more reasonable now that we've switched to CommonMark and you might want to give it a try. Mixing HTML and Markdown has been and continues to be a hot take as it can result in really hard-to-predict behavior. – Ham Vocke Jun 7 '20 at 10:53
  • @HamVocke I see, thanks. I'll think about switching back later, but for now I'll stick to XHTML. – Ainar-G Jun 7 '20 at 12:07
2

List of completely broken posts

We need a tool which detects posts which are completely broken by the change in the syntax, not just with minor formatting differences. Things like completely new/lost blocks, completely different block hieararchy etc.

With this list, we could check the most popular posts, the most frequently updated posts etc. and fix them before they're accidentally broken by a random edit.

This may also help find more incompatibilities and maybe fix them automatically.

  • 3
    I've outlined this in the comments on another post: getting a sane list of all truly affected posts for public evaluation is difficult because we're dealing with a lot of tiny nuances that are detected as false positives. I don't see a good way to filter the signal from the noise yet. On the other hand, I've yet to encounter a post that I'd consider "completely broken". All cases I've seen are trivial or fairly obvious when you look at the Markdown source. I'll think about a solution to your request but I can't promise anything yet. – Ham Vocke Jun 10 '20 at 17:49
2

The helpcenter article on 10k-tools shows:

enter image description here

Can anyone update to the correct header?

1

Possible in the migration tool.

In this answer, I quoted two portions of the OP's post. Each posted as one long line, prefixed by > , with a blank line between the two:

> ... `for digit in num_as_string[::-1]:`. In this loop, it's just \$O(n)\$ where n is just number of digits of the input.

> I am assuming the time complexity of this code is something like O(n) + 2 * O(number of digits in base10) which is linear.

This produced two separate block quotes, like:

... for digit in num_as_string[::-1]:. In this loop, it's just $O(n)$ where n is just number of digits of the input.

I am assuming the time complexity of this code is something like O(n) + 2 * O(number of digits in base10) which is linear.

The Common Mark migration tool added a > line between these:

> ... `for digit in num_as_string[::-1]:`. In this loop, it's just \$O(n)\$ where n is just number of digits of the input.
>
> I am assuming the time complexity of this code is something like O(n) + 2 * O(number of digits in base10) which is linear.

Which resulted in one block quote, instead of two:

... for digit in num_as_string[::-1]:. In this loop, it's just $O(n)$ where n is just number of digits of the input.

I am assuming the time complexity of this code is something like O(n) + 2 * O(number of digits in base10) which is linear.

The intention was to have two distinct block quotes; the rendered output differs from the original rendered output. It isn't worth an edit to revert the appearance back to the intended, bumping the post back to "active" status. This isn't an isolated case; another of my answers was edited the same way, changing two block-quotes into one. It will likely arise when additional sites are migrated as well.

  • 4
    Uh...I'm kind of missing how you actually achieved this to be two distinct blockquotes before the migration. The point of introducing that > there is that before the migration these two lines didn't render as two blockquotes since that wasn't possible before CommonMark, so the migration tool tries to reproduce that. – Christian Rau Jun 10 '20 at 20:53
  • 5
    (In case you're comparing the revisions, keep in mind that the older revisions in the revision history are now rendered with the new markdown engine, too. That is, however, not how they actually looked like before the site was switched to CommonMark.) – Christian Rau Jun 10 '20 at 20:59
  • @ChristianRau I may have been exploiting a bug, previously. IIRC adding 2 spaces at the end of a block quote line (> yada yada {space}{space}\n> yada yada) allowed you to insert newlines in block quotes. Without the 2 spaces, the block quote would join the lines together, wrapping at the width of the block quote box. I probably had two spaces at the end of the first quote, the blank lines, and then started the second quote, and observed it created two separate quotes as I intended. I'd have to test on a non-upgraded site to be sure. – AJNeufeld Jun 10 '20 at 21:54
  • Well, I'm not able to reproduce it on Stack Overflow, which hasn't been upgraded yet. Colour me confused. I'm confident when I posted the answers, I did something to make the quotes appear as separate block quotes, without non-quoted text appearing in between. Adding double spaces at the end of the block quote would be a stupid hack only a few would have accidentally discovered, so it wouldn't be many posts that would be affected (that is, if it did work and I'm not going insane). Going forward, it will be much easier to make the 2 block-quotes separate, so, yay. – AJNeufeld Jun 10 '20 at 23:30
  • 2
    I used to add HTML quotes (<!-- -->) to break up blog quotes. – curiousdannii Jun 11 '20 at 23:59
1

Help me understand the "#" issue?

Hi tech crew, thanks, you mention:

enter image description here

I have many of the top answers (and questions) in iOS and Unity, on SO.

I have never put a space after the "#" for headlines :( ... so sorry.

What should I do?

  • I should do nothing, a computer will fix it
  • I should manually edit every one of my Q and A, and add a space
  • I misunderstand the situation, this does not apply to SO
  • I should do nothing regarding the existing corpus, but, from now on I must use the extra space
  • Some other unimagined option from an undreamt of paradigm

Help!

Thx, or I should say

#thx

:O

  • 4
    The bot will automatically fix all the past occurrences for you, but Stack Overflow is scheduled to be fixed last across the network, so it hasn't happened yet. In the future, the space will be required and failing to include a space will result in it not being formatted as a header, but as regular text. – animuson Jun 17 '20 at 21:19
  • 3
    Basically, if you do nothing, headers formatted as #this is a header will be (eventually) automatically fixed to # this is a header. (The exception is if the system has trouble automatically converting something in the post, in which case it'll leave that post as-is for now; then, that post will continue to stay as-is unless it gets edited. If you or someone edits that post, then the post will be re-rendered; that's when you'd need to add the space in manually after the #. And you will have to include that space for new posts made after the switch for headers to render properly. – V2Blast Jun 18 '20 at 1:49
  • 1
    The correct answer is that you don't need to do anything for the existing corpus (that will be fixed automatically in so far as possible), but you should start using the extra space from now on. – Bart van Ingen Schenau Jun 18 '20 at 7:09
  • All three BartvanIngenSchenau animuson V2Blast perfect information - thanks, great stuff. Have a good one. – Fattie Jun 18 '20 at 12:21
1

Any chance this will lead to improved (safe) HTML support in the future?

In particular:

<details open>
    <summary>Summary</summary>
    Details
</details>

<abbr title="title">abbr</abbr>

<mark>highlight</mark>

Rendered versions below — all failing at time of posting


Summary Details

abbr

highlight

1

Links don't work inside <sub> and <sup> tags

Samples:

This is [Stack Exchange](https://stackexchange.com/). This is [Stack Exchange][se]. This is [se]. This is https://stackexchange.com/
  • 5
    You need to use HTML inside them to create links. CommonMark forbids processing further Markdown within an HTML block. – animuson Sep 25 '20 at 14:56
  • Ah, I see. Thanks for the clarification. – schtandard Sep 26 '20 at 9:29
0

Not sure if it's a , but when a code block is nested with a list item and idented with 8 spaces, all the lines of the code have some extra spacing at the beginning.

Example, this markdown:

- Some text

        some code block
        idented by 8 spaces

Is rendered as:

  • Some text

      some code block
      idented by 8 spaces
    

To help visualize the issue, that's an image of the above (with the extra space free-hand-red-circled):

enter image description here


Just to compare with a non-nested code block, note that there are less spacing at the beginning of each line:

some code block
idented by 4 spaces

BTW, I've found a workaround for this: align code fences with the list item. So this:

- Some text
  ```
  some code block
  inside nested code fences
  ```

Is rendered as:

  • Some text
    some code block
    inside nested code fences
    

But it'd be nice if 8-space idented code blocks - when nested with a list item - could be rendered the same way (it was before the migration) - I had 2 answers that weren't edited by the migration so I had to fix them manually.

  • 2
    This works as designed. You're right that the behavior changed after the migration, but I'd argue it changed for the better. In order to nest anything as part of a list item, you'll have to line things up with the first letter of your list item after bullets and spaces. For indented code blocks that means you add spaces until you've cleared the bullet character and spaces and then add 4 more spaces to denote an indented code block. Unfortunately we couldn't detect this as part of the automatic migration easily. – Ham Vocke Oct 8 '20 at 7:20
  • @HamVocke You're right, it makes much more sense now. Thanks for the reply! – hkotsubo Oct 8 '20 at 11:58
1
2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .