Feature: md:myst format also works as a MyST Markdown

[itcarroll]

CC @chrisjsewell @choldgraf

I think it's possible and would be helpful for jupytext to convert between notebook and markdown while preserving MyST functionality in both formats. I have been having a hard time collaborating with notebook authors, who are only comfortable working with a traditional notebook and simple development requirements, on a jupyter book. With the upcoming shift to jupyter book 2.0, I'm trying to go full MyST, and am now stuck on the issue of md:myst format encoded by jupytext not working with myst even while the paired notebook does. Having only non-working md:myst files in the repository seems to require unnecessary complications during building.

Presently the jupyterlab-myst extension utilizes a YAML block in the first cell, but the markdown that jupytext writes in this case does not pass this frontmatter to the YAML block in markdown. You get two YAML blocks, and myst does not find its frontmatter in the second.

Notebook

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1326d1c7",
   "metadata": {},
   "source": [
    "---\n",
    "title: Example\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b365a8e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "1 + 2"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md:myst"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}

MyST Markdown

---
jupytext:
  formats: ipynb,md:myst
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.16.6
kernelspec:
  display_name: Python 3 (ipykernel)
  language: python
  name: python3
---

title: Example
1 + 2

What do you think about modifying the md:myst conversion in jupytext to keep the YAML frontmatter together in markdown? The notebook metadata could go under the jupyter key, as it currently does for the md format, for separation. Yes, there would be some complication to work through with the kernelspec field to keep both jupyter and myst working and configurable.

I will share a draft PR that minimally accomplishes how it could work (while appearing to break stuff, but I can get to that).

[mwouts]

Thank you @itcarroll for opening this issue - it's great to have you here! I plan to look into the issue and the associated PR over the weekend, but sure that sounds like a welcome improvement!