17
$\begingroup$

Yesterday, I announced on Meta Stack Exchange that we'll switch all sites on the Stack Exchange network to CommonMark, a standardized and well-supported Markdown flavor.

You can read up on the details in the post on Meta.SE if you're curious.

We'll maintain an evolving migration schedule to show which site is supposed to switch over to CommonMark when.

We've got to learn and reflect as we're starting out migrating the first few sites, that's why you'll see that the migration schedule is still pretty empty except for a few sites.

Physics (both, meta and the main site) are going to be among the first sites to be migrated and we've scheduled them to be migrated on Thursday, June 4th, 2020.

Why start with Physics?

We've tested the migration thoroughly on data that resembles production data of some of our communities. Still, we know that certain sites use different styles of writing and there's a chance we're going to detect some issues that we haven't found yet when running our tests.

Physics is a site that's suitable for fast feedback for two reasons: it's a site we can migrate quickly with its ~400k posts and it's a site that makes heavy use of MathJax.

Don't worry, MathJax is going to keep working after the migration. However, we want to be double-certain that we're not missing any edge cases, that's why we want to migrate a site with MathJax support first to spot edge cases early and get them out of the way.

Can this break existing posts?

We don't want to break hundreds and thousands of posts. That's why the migration will only apply updates to those posts that will look exactly the same after being updated to CommonMark. As part of the migration, we'll detect if a post changes visually after the CommonMark update. If it does, we won't update the post automatically and investigate what's going on.

What if stuff goes wrong?

If things should go horribly wrong, we've got an automated rollback in place that will undo the migration for all posts.

$\endgroup$
5
  • $\begingroup$ Potential Problem I just played a bit with CommonMark demo, and I do see a potential problem. Usually while creating headings, I write something of the sort ##heading. But doing the same in CommonMark, doesn't yield a heading. You have to compulsorily add a <kbd>space</kbd> after the ##. So to make it work in CommonMark, I have to write ## heading. So I expect all my answers which contain the former sort of markdown to break. $\endgroup$
    – user258881
    Jun 3, 2020 at 13:59
  • $\begingroup$ So is this change going to be implemented? If yes, then what should I do about my answers? $\endgroup$
    – user258881
    Jun 3, 2020 at 14:03
  • 12
    $\begingroup$ @FakeMod you don't need to do anything about your answers. As part of the migration we're applying some automated fixes to the markdown version of a post for these well-known incompatibilities. There are a few more of those outlined in the announcement on meta.SE $\endgroup$
    – Ham Vocke StaffMod
    Jun 3, 2020 at 14:09
  • $\begingroup$ Possibly related: a supported HTML tag used to work, but now it doesn't. $\endgroup$
    – rob Mod
    Oct 7, 2020 at 14:42
  • $\begingroup$ @HamVocke Can you review the question rob just linked to? Hopefully you have sufficient clarity into the different systems involved to tell whether it was a result of the CommonMark migration, or something else. $\endgroup$
    – Emilio Pisanty
    Oct 7, 2020 at 15:36

5 Answers 5

Reset to default
9
$\begingroup$

Good news, then!

Suppose a user discovers a post that renders incorrectly after the transition. Should that user fix the problem themselves, or cast a custom flag for the diamond moderators, or leave a message at this question, or do something else? (possibly including nothing at all?)

$\endgroup$
3
  • 3
    $\begingroup$ Ideally they'd go and fix it themselves. Just to be clear: incorrect rendering should only ever occur once someone edits one of those posts that couldn't be transitioned correctly with our automatic migration. $\endgroup$
    – Ham Vocke StaffMod
    Jun 2, 2020 at 13:36
  • 1
    $\begingroup$ @HamVocke I doubt there's an endless spring of classes of such problems; how likely do you think it is that you won't be able to catch them all (or, rather, all the ones that affect posts on the site)? $\endgroup$
    – wizzwizz4
    Jun 2, 2020 at 18:27
  • 5
    $\begingroup$ I'm certain we won't catch them all. The question is if that matters and if we want to go down the deep rabbit hole that is fixing them all preemptively. We're aiming to spot and fix non-trivial issues while accepting that the trivial ones might be left unfixed until someone comes in and edits old posts. There's a certain level of pragmatism that we have to apply to this migration in order to be able to pull this off, and for a fraction of posts this means editing could become a bit awkward. $\endgroup$
    – Ham Vocke StaffMod
    Jun 2, 2020 at 19:33
9
$\begingroup$

Tracking the current state to keep you updated:

  • physics.meta.stackexchange.com has been migrated successfully. CommonMark is now active
  • I've started probing physics.stackexchange.com (no changes are being persisted). CommonMark is still disabled and I'm double-checking that MathJax rendering is going to stay sane. Will continue with the real migration as soon as I'm certain that we're good here.
  • The differences we're detecting are expected but our migration script is being overly cautious. Doing a first real migration run right now. I will patch up some things to do a second run tomorrow to fix those posts that haven't been caught in the first run. That means that commonmark is enabled for physics starting now and posts are being migrated.
  • Finally managed to complete the re-run for the physics main and meta sites. Everything's migrated now - calling this done. Thanks for your patience!
$\endgroup$
9
  • 1
    $\begingroup$ Looks like everything went OK? From the community bit it looks like there were some 2.5k edits. Can you give a sense for how many are left for the second run? $\endgroup$
    – Emilio Pisanty
    Jun 4, 2020 at 20:08
  • 6
    $\begingroup$ Keeping the community updated like this is exactly the kind of interaction I like to see from SE/SO as an organization. It creates a sense that community opinion matters. Thank you. $\endgroup$
    – StephenG - Help Ukraine
    Jun 4, 2020 at 21:41
  • $\begingroup$ Hi @Ham, any updates on how this is going? I imagine the second run kicked up some trouble, given the delay. $\endgroup$
    – Emilio Pisanty
    Jun 9, 2020 at 10:41
  • 1
    $\begingroup$ @EmilioPisanty sorry for the radio silence! The first run was just fine but left a few too many posts untouched. While working on a tweak to get those untouched posts migrated, I got in another tweak for sites that are using MathJax slightly differently than Physics and that took me longer to grok than I had hoped. I'm aiming to re-run with the new tweaks within the next 4 hours, after that we should be good to call this done. $\endgroup$
    – Ham Vocke StaffMod
    Jun 9, 2020 at 10:51
  • 1
    $\begingroup$ @HamVocke Thanks for the update =). $\endgroup$
    – Emilio Pisanty
    Jun 9, 2020 at 11:13
  • $\begingroup$ Relative time designations, like "now", very quickly get old. Perhaps use absolute time instead? $\endgroup$
    – Peter Mortensen
    Jun 12, 2020 at 15:18
  • $\begingroup$ @PeterMortensen Presumably "now" refers to the timestamp of the most recent edit, 2020-06-11 09:48Z, which is displayed just below the text of the post. However, your point is totally valid, especially if there are future edits. $\endgroup$
    – rob Mod
    Jun 15, 2020 at 14:24
  • $\begingroup$ @HamVocke Just for completeness, does this mean that every post has had its markdown edited so that it conforms to CommonMark without altering the produced html? Or is there some small population of 'sleeper' posts that might change when edited? $\endgroup$
    – Emilio Pisanty
    Jun 16, 2020 at 14:48
  • $\begingroup$ @EmilioPisanty it's the latter. We've only edited a post's markdown if the resulting HTML (rendered with the new CommonMark renderer) is equivalent to the old HTML. There is going to be a small population (less than 3% of all posts approx.) that are going to look slightly different if someone was to edit them. $\endgroup$
    – Ham Vocke StaffMod
    Jun 16, 2020 at 14:57
8
$\begingroup$

How can we help?

I'm happy to do my part to help this come off smoothly, but I'm not sure how and where we can help. The Mother-Meta announcement mentions the possibility that the migration will break (in the sense that the new html will be different to the old one) for a fraction of the posts, and that when this happens the migration will be discarded and the old html will be retained.

To me, this sounds like it calls for human intervention as a slow go-through by hand of all the failed markdown migrations $-$ I would much rather have a community effort to go through these and make sure everything is OK before some inattentive editor does something else on the post, fails to notice something breaking, and pushes the re-render through.

Will there be a centralized place where we can see the posts whose markdown migration has been flagged by the system, and edit them into the new renderer? If the renderer is getting bumped when you edit, will this be visible to the editor? (Say, as a banner on the Edit page saying that the renderer bumped and that extra care must be taken to ensure nothing is broken?)

As a quick partial answer -- it seems that the list of edited posts get attributed to Community ♦ and will be shown on its Revisions listing. For Meta SE they've started to come in (here), but on Physics they haven't at the time of this writing -- presumably they will show up on this listing page once posts start being edited.

Presumably this can be queried from SEDE once the data there gets updated, but in this form it cannot be used to query for failed CommonMark migrations, as the SEDE data cannot distinguish them from posts that already complied with the CommonMark spec, if I understand correctly.

$\endgroup$
6
  • 1
    $\begingroup$ We don't know the exact amount and shape of issues we're going to run into. Judging from our test runs (based on production data) an overwhelming amount of posts will be automatically fixed or show non-visible differences only. I will take a look at migration failures and try to weed out those that are non-negotiable so that they can be fixed in an automated way (also for other sites to come). We might resort to using notifications or a sort of review queue if we can't do without manual intervention but we hope we won't even have to go there. $\endgroup$
    – Ham Vocke StaffMod
    Jun 2, 2020 at 15:45
  • 1
    $\begingroup$ Oh, if my comment is coming across the wrong way: I really appreciate our communities offering to step in and help resolving this, thanks for that. It's more that we need to find a solution that scales across all different sites on the SE network, some of which have millions and millions of posts. If we can minimize manual intervention and avoid having people apply mundane fixes to posts, I'll take that any day :) $\endgroup$
    – Ham Vocke StaffMod
    Jun 2, 2020 at 17:55
  • 1
    $\begingroup$ Sure, that makes sense. If you do need human checkers to feed data into that machine (eg by going manually over the corpus of flagged posts and taking appropriate action), then just tell us what you need. When you guys do this type of process correctly (as you've done with this migration) you have a pool of users willing to step in and donate value to the post corpus. $\endgroup$
    – Emilio Pisanty
    Jun 2, 2020 at 20:34
  • 1
    $\begingroup$ In any case, it would be good to know how it's going once it gets rolling. If there's a population of posts that could be broken by editing, then please do let us know here. $\endgroup$
    – Emilio Pisanty
    Jun 2, 2020 at 20:37
  • 1
    $\begingroup$ FWIW, a minor bug in the Markdown autofixer has been found, and fixed: meta.stackexchange.com/a/348897/334566 $\endgroup$
    – PM 2Ring
    Jun 4, 2020 at 8:29
  • $\begingroup$ @PM2Ring Thanks for the heads-up. $\endgroup$
    – Emilio Pisanty
    Jun 4, 2020 at 10:17
6
$\begingroup$

More precisely, what time and date and timezone does this go live?

You said:

on Thursday, June 4th, 2020

but of course that means different things to different people, so could we get the time and date more precisely from you that the switch happens?

Also, is there a specific time when we will be presented with a completely translated site or is this translation (of old posts) being done as they are fetched by normal access or something else?

$\endgroup$
1
  • 2
    $\begingroup$ I'm sorry about the ambiguity. I'd love to get an additional pair of eyes on this and since our engineering team is spread across multiple time zones coordination can become a little more tricky given that there are other spontaneous things that pop up for every one of us. I'm confident that I'll kick off the migration within the next 2-3 hours, it shouldn't run for much longer than 30 minutes. $\endgroup$
    – Ham Vocke StaffMod
    Jun 4, 2020 at 13:50
-1
$\begingroup$

The rendering of mathematical material has gotten way, way slower since the changeover. This makes it more of a pain to write and edit answers with mathematics in them, especially since any change to a mathematical expression causes all the math in a post to be re-rendered, which can take (I timed up) upwards of fifteen seconds. This makes even small changes very hard to verify, waiting around to see whether what you have typed was correct.

$\endgroup$
2
  • $\begingroup$ This hasn't happened for me $\endgroup$
    – BioPhysicist
    Jun 18, 2020 at 3:45
  • 2
    $\begingroup$ Unfortunately I can't reproduce what you're describing. I'm not saying this isn't happening but I also don't see why MathJax rendering would become slower after switching to CommonMark since rendering Markdown to HTML and rendering MathJax equations are two completely distinct steps. Do you have any steps that could help me reproduce this behavior? $\endgroup$
    – Ham Vocke StaffMod
    Jun 18, 2020 at 6:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .