[Documentation] Replace recommonmark by myst-parser (#65664)

Recommonmark has been deprecated, then archived last year. This was
tracked by: https://github.com/llvm/llvm-iwg/issues/30

See https://github.com/readthedocs/recommonmark

This patch migrates all our doc to use myst

Additional details for bot maintainers: https://discourse.llvm.org/t/maintenance-required-on-sphinx-build-bots/73612

.github/workflows/release-tasks.yml

@@ -24,6 +24,9 @@ jobs:
           release_version=$(echo "${{ github.ref_name }}" | sed 's/llvmorg-//g')
           echo "release-version=$release_version" >> "$GITHUB_OUTPUT"
 
+      - name: Checkout LLVM
+        uses: actions/checkout@v4
+
       - name: Install Dependencies
         run: |
           sudo apt-get update
@@ -31,14 +34,9 @@ jobs:
               doxygen \
               graphviz \
               python3-github \
-              python3-recommonmark \
-              python3-sphinx \
               ninja-build \
               texlive-font-utils
-          pip3 install --user sphinx-markdown-tables
-
-      - name: Checkout LLVM
-        uses: actions/checkout@v4
+          pip3 install --user -r ./llvm/docs/requirements.txt
 
       - name: Create Release
         run: |

clang/docs/conf.py

@@ -32,26 +32,11 @@ extensions = ["sphinx.ext.todo", "sphinx.ext.mathjax"]
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
 
-# The suffix of source filenames.
-source_suffix = {
-    ".rst": "restructuredtext",
-}
 
-try:
-    import recommonmark
-except ImportError:
-    # manpages do not use any .md sources
-    if not tags.has("builder-man"):
-        raise
-else:
-    import sphinx
+import sphinx
 
-    if sphinx.version_info >= (3, 0):
-        # This requires 0.5 or later.
-        extensions.append("recommonmark")
-    else:
-        source_parsers = {".md": "recommonmark.parser.CommonMarkParser"}
-    source_suffix[".md"] = "markdown"
+if sphinx.version_info >= (3, 0):
+    extensions.append("myst_parser")
 
 # The encoding of source files.
 # source_encoding = 'utf-8-sig'

flang/docs/Aliasing.md

@@ -8,9 +8,10 @@
 
 # Aliasing in Fortran
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 ## Introduction

flang/docs/ArrayComposition.md

@@ -8,9 +8,10 @@
 
 # Array Composition
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 This note attempts to describe the motivation for and design of an

flang/docs/BijectiveInternalNameUniquing.md

@@ -8,9 +8,10 @@
 
 # Bijective Internal Name Uniquing
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 FIR has a flat namespace. No two objects may have the same name at the module

flang/docs/C++17.md

@@ -8,9 +8,10 @@
 
 # C++14/17 features used in f18
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 The C++ dialect used in this project constitutes a subset of the

flang/docs/C++style.md

@@ -8,9 +8,10 @@
 
 # Flang C++ Style Guide
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 This document captures the style guide rules that are followed in the Flang codebase.

flang/docs/Calls.md

@@ -8,9 +8,10 @@
 
 # Representation of Fortran function calls
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 ## Procedure reference implementation protocol

flang/docs/Character.md

@@ -8,9 +8,10 @@
 
 # Implementation of `CHARACTER` types in f18
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 ## Kinds and Character Sets

flang/docs/ComplexOperations.md

@@ -1,7 +1,7 @@
 # Complex Operations
 
-```eval_rst
-.. contents::
+```{eval-rst}
+.. toctree::
    :local:
 ```
 

flang/docs/ControlFlowGraph.md

@@ -8,9 +8,10 @@
 
 # Control Flow Graph
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 ## Concept

flang/docs/DesignGuideline.md

@@ -7,9 +7,10 @@
 -->
 # Design Guideline
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 ## Documenting the design
 

flang/docs/DoConcurrent.md

@@ -8,9 +8,10 @@
 
 # `DO CONCURRENT` isn't necessarily concurrent
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 A variant form of Fortran's primary looping construct was

flang/docs/Extensions.md

@@ -8,9 +8,10 @@
 
 # Fortran Extensions supported by Flang
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 As a general principle, this compiler will accept by default and

flang/docs/FIRArrayOperations.md

@@ -8,9 +8,10 @@
 
 # Design: FIR Array operations
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 ## General

flang/docs/FlangDriver.md

@@ -8,9 +8,10 @@
 
 # Flang drivers
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 There are two main drivers in Flang:

flang/docs/FortranFeatureHistory.md

@@ -8,9 +8,10 @@
 
 # A Fortran feature history cheat sheet
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 ## Original IBM 704 FORTRAN

flang/docs/FortranForCProgrammers.md

@@ -8,9 +8,10 @@
 
 # Fortran For C Programmers
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 This note is limited to essential information about Fortran so that

flang/docs/FortranIR.md

@@ -8,9 +8,10 @@
 
 # Design: Fortran IR
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 ## Introduction

flang/docs/FortranLLVMTestSuite.md

@@ -1,8 +1,9 @@
 # Fortran Tests in the LLVM Test Suite
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 The [LLVM Test Suite](https://github.com/llvm/llvm-test-suite) is a

flang/docs/GettingInvolved.md

@@ -7,9 +7,10 @@
 -->
 # Getting Involved
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 The Flang Project welcomes contributions of all kinds.

flang/docs/GettingStarted.md

@@ -8,9 +8,10 @@
 
 # Getting Started
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 ## Building flang
@@ -443,7 +444,8 @@ system to create HTML pages which would be hosted on the webpage of flang and
 updated periodically.
 
 If you would like to generate and view the HTML locally:
-- Install [Sphinx](http://sphinx-doc.org/), including the [sphinx-markdown-tables](https://pypi.org/project/sphinx-markdown-tables/) extension.
+- Install [Sphinx](http://sphinx-doc.org/), and the required extensions
+  using `pip install --user -r ~/llvm-projects/docs/requirements.txt`
 - Pass `-DLLVM_ENABLE_SPHINX=ON -DSPHINX_WARNINGS_AS_ERRORS=OFF` to the cmake command.
 
 ```bash

flang/docs/IORuntimeInternals.md

@@ -8,9 +8,10 @@
 
 # Fortran I/O Runtime Library Internal Design
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 This note is meant to be an overview of the design of the *implementation*

flang/docs/ImplementingASemanticCheck.md

@@ -7,9 +7,10 @@
 -->
 # How to implement a Sematic Check in Flang
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 I recently added a semantic check to the Flang compiler front end.  This document

flang/docs/IntrinsicTypes.md

@@ -8,9 +8,10 @@
 
 # Implementation of `Intrinsic` types in f18
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 Intrinsic types are integer, real, complex, character, and logical.

flang/docs/Intrinsics.md

@@ -8,9 +8,10 @@
 
 # A categorization of standard (2018) and extended Fortran intrinsic procedures
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 This note attempts to group the intrinsic procedures of Fortran into categories

flang/docs/LabelResolution.md

@@ -8,9 +8,10 @@
 
 # Semantics: Resolving Labels and Construct Names
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 ## Overview

flang/docs/ModFiles.md

@@ -8,9 +8,10 @@
 
 # Module Files
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 Module files hold information from a module that is necessary to compile 

flang/docs/OpenACC.md

@@ -8,9 +8,10 @@
 
 # OpenACC in Flang
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 ## Intentional deviation from the specification

flang/docs/OpenMP-semantics.md

@@ -8,9 +8,10 @@
 
 # OpenMP Semantic Analysis
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 ## OpenMP for F18

flang/docs/Overview.md

@@ -8,9 +8,10 @@
 
 # Overview of Compiler Phases
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 The Flang compiler transforms Fortran source code into an executable file. 
 This transformation proceeds in three high level phases -- analysis, lowering,

flang/docs/ParserCombinators.md

@@ -8,9 +8,10 @@
 
 # Parser Combinators
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 This document is a primer on Parser Combinators and their use in Flang.

flang/docs/Parsing.md

@@ -8,9 +8,10 @@
 
 # The F18 Parser
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 This program source code implements a parser for the Fortran programming

flang/docs/Preprocessing.md

@@ -8,9 +8,10 @@
 
 # Fortran Preprocessing
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 ## Behavior common to (nearly) all compilers:

flang/docs/RuntimeDescriptor.md

@@ -8,9 +8,10 @@
 
 # Runtime Descriptors
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 ## Concept

flang/docs/RuntimeTypeInfo.md

@@ -8,9 +8,10 @@
 
 # The derived type runtime information table
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 ## Overview

flang/docs/Semantics.md

@@ -8,9 +8,10 @@
 
 # Semantic Analysis
 
-```eval_rst
-.. contents::
-   :local:
+```{contents}
+---
+local:
+---
 ```
 
 The semantic analysis pass determines if a syntactically correct Fortran

flang/docs/conf.py

@@ -9,10 +9,7 @@
 # All configuration values have a default; values that are commented out
 # serve to show the default.
 
-import sys, os
 from datetime import date
-from recommonmark.parser import CommonMarkParser
-
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
@@ -20,65 +17,22 @@ from recommonmark.parser import CommonMarkParser
 
 # -- General configuration -----------------------------------------------------
 
-# https://github.com/readthedocs/recommonmark/issues/177
-# Method used to remove the warning message.
-class CustomCommonMarkParser(CommonMarkParser):
-    def visit_document(self, node):
-        pass
-
-
 # If your documentation needs a minimal Sphinx version, state it here.
 # needs_sphinx = '1.0'
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
-extensions = ["sphinx.ext.todo", "sphinx.ext.mathjax", "sphinx.ext.intersphinx"]
+extensions = [
+    "myst_parser",
+    "sphinx.ext.todo",
+    "sphinx.ext.mathjax",
+    "sphinx.ext.intersphinx",
+    "sphinx.ext.autodoc",
+]
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
 
-# The suffix of source filenames.
-source_suffix = {
-    ".rst": "restructuredtext",
-}
-try:
-    import recommonmark
-except ImportError:
-    # manpages do not use any .md sources
-    if not tags.has("builder-man"):
-        raise
-else:
-    import sphinx
-
-    if sphinx.version_info >= (3, 0):
-        # This requires 0.5 or later.
-        extensions.append("recommonmark")
-    else:
-        source_parsers = {".md": CustomCommonMarkParser}
-    source_suffix[".md"] = "markdown"
-    extensions.append("sphinx_markdown_tables")
-
-    # Setup AutoStructify for inline .rst toctrees in index.md
-    from recommonmark.transform import AutoStructify
-
-    # Stolen from https://github.com/readthedocs/recommonmark/issues/93
-    # Monkey patch to fix recommonmark 0.4 doc reference issues.
-    from recommonmark.states import DummyStateMachine
-
-    orig_run_role = DummyStateMachine.run_role
-
-    def run_role(self, name, options=None, content=None):
-        if name == "doc":
-            name = "any"
-            return orig_run_role(self, name, options, content)
-
-    DummyStateMachine.run_role = run_role
-
-    def setup(app):
-        # Disable inline math to avoid
-        # https://github.com/readthedocs/recommonmark/issues/120 in Extensions.md
-        app.add_config_value("recommonmark_config", {"enable_inline_math": False}, True)
-        app.add_transform(AutoStructify)
-
+import sphinx
 
 # The encoding of source files.
 # source_encoding = 'utf-8-sig'

flang/docs/index.md

@@ -9,7 +9,7 @@ is capable of generating executables for a number of examples, some
 functionality is still missing. See [GettingInvolved](GettingInvolved) for tips
 on how to get in touch with us and to learn more about the current status.
 
-```eval_rst
+```{eval-rst}
 .. toctree::
    :titlesonly:
 
@@ -18,7 +18,7 @@ on how to get in touch with us and to learn more about the current status.
 
 # Contributing to Flang
 
-```eval_rst
+```{eval-rst}
 .. toctree::
    :titlesonly:
 
@@ -33,7 +33,7 @@ on how to get in touch with us and to learn more about the current status.
 
 # Design Documents
 
-```eval_rst
+```{eval-rst}
 .. toctree::
    :titlesonly:
 
@@ -70,7 +70,7 @@ on how to get in touch with us and to learn more about the current status.
 
 # Indices and tables
 
-```eval_rst
+```{eval-rst}
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`

llvm/docs/GwpAsan.rst

@@ -257,6 +257,7 @@ anything fails. This results in the following output:
 
 .. code:: console
 
+
   $ cat my_gwp_asan_error.txt | symbolize.sh
   |
   | *** GWP-ASan detected a memory error ***

llvm/docs/MarkdownQuickstartTemplate.md

@@ -151,10 +151,10 @@ without any syntax highlighting like this:
 
 If you need to do fancier things than what has been shown in this document,
 you can mail the list or check the [Common Mark spec].  Sphinx specific
-integration documentation can be found in the [recommonmark docs].
+integration documentation can be found in the [myst-parser docs].
 
 [Common Mark spec]: http://spec.commonmark.org/0.28/
-[recommonmark docs]: http://recommonmark.readthedocs.io/en/latest/index.html
+[myst-parser docs]: https://myst-parser.readthedocs.io/en/latest/
 
 ## Generating the documentation
 

llvm/docs/SphinxQuickstartTemplate.rst

@@ -171,20 +171,12 @@ Generating the documentation
 You can generate the HTML documentation from the sources locally if you want to
 see what they would look like. In addition to the normal
 `build tools <docs/GettingStarted.html>`_
-you need to install `Sphinx`_ and the
-`recommonmark <https://recommonmark.readthedocs.io/en/latest/>`_ extension.
-
-On Debian you can install these with:
+you need to install `Sphinx`_ and the necessary extensions
+using the following command inside the ``llvm-project`` checkout:
 
 .. code-block:: console
 
-   sudo apt install -y sphinx-doc python-recommonmark-doc
-
-On Ubuntu use pip to get an up-to-date version of recommonmark:
-
-.. code-block:: console
-
-   sudo pip install sphinx recommonmark
+   pip install --user -r ./llvm/docs/requirements.txt
 
 Then run cmake to build the documentation inside the ``llvm-project`` checkout:
 

llvm/docs/conf.py

@@ -26,31 +26,14 @@ from datetime import date
 
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
-extensions = ["sphinx.ext.intersphinx", "sphinx.ext.todo"]
+extensions = ["myst_parser", "sphinx.ext.intersphinx", "sphinx.ext.todo"]
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
 
-# The suffix of source filenames.
-source_suffix = {
-    ".rst": "restructuredtext",
-}
+source_suffix = [".rst", ".md"]
 
-try:
-    import recommonmark
-except ImportError:
-    # manpages do not use any .md sources
-    if not tags.has("builder-man"):
-        raise
-else:
-    import sphinx
-
-    if sphinx.version_info >= (3, 0):
-        # This requires 0.5 or later.
-        extensions.append("recommonmark")
-    else:
-        source_parsers = {".md": "recommonmark.parser.CommonMarkParser"}
-    source_suffix[".md"] = "markdown"
+import sphinx
 
 # The encoding of source files.
 # source_encoding = 'utf-8-sig'

llvm/utils/release/build-docs.sh

@@ -11,13 +11,13 @@
 #
 # Required Packages:
 # * Fedora:
-#   * dnf install doxygen python3-sphinx texlive-epstopdf ghostscript \
+#   * dnf install doxygen texlive-epstopdf ghostscript \
 #                 ninja-build gcc-c++
-#   * pip install sphinx-markdown-tables
+#   * pip install --user -r ./llvm/docs/requirements.txt
 # * Ubuntu:
-#   * apt-get install doxygen sphinx-common python3-recommonmark \
+#   * apt-get install doxygen \
 #             ninja-build graphviz texlive-font-utils
-#   * pip install sphinx-markdown-tables
+#   * pip install --user -r ./llvm/docs/requirements.txt
 #===------------------------------------------------------------------------===#
 
 set -e