MDEx (MDEx v0.5.0)

View Source

MDEx logo

Fast and extensible Markdown for Elixir.

Hex Version Hex Docs MIT

Features

  • Support formats:
    • Markdown (CommonMark)
    • HTML
    • JSON
    • XML
  • Floki-like Document AST
  • Req-like Pipeline API
  • Compliant with the CommonMark spec
  • Additional features:
    • GitHub Flavored Markdown
    • Discord and GitLab features
    • Wiki-style links
    • Emoji shortcodes
    • Syntax highlighting for code blocks
    • HTML sanitization
    • Sigils for Markdown, HTML, JSON, and XML

Foundation

The library is built on top of:

Installation

Add :mdex dependency:

def deps do
  [
    {:mdex, "~> 0.5"}
  ]
end

Usage

Mix.install([{:mdex, "~> 0.5"}])
iex> MDEx.to_html!("# Hello")
"<h1>Hello</h1>"
iex> MDEx.to_html!("# Hello :smile:", extension: [shortcodes: true])
"<h1>Hello 😄</h1>"

Req-like Pipeline

MDEx.Pipe provides a high-level API to manipulate a Markdown document and build plugins that can be attached to a pipeline:

document = ~s|
# Super Diagram

\`\`\`mermaid
graph TD
  A[Enter Chart Definition] --> B(Preview)
  B --> C{decide}
  C --> D[Keep]
  C --> E[Edit Definition]
  E --> B
  D --> F[Save Image and Code]
  F --> B
\`\`\`
|

MDEx.new()
|> MDExMermaid.attach(version: "11")
|> MDEx.to_html(document: document)

Sigils

Convert and generate AST (MDEx.Document), Markdown (CommonMark), HTML, JSON, and XML formats.

First, import the sigils:

iex> import MDEx.Sigil
iex> import MDEx.Sigil
iex> ~M|# Hello from `~M` sigil|
%MDEx.Document{
  nodes: [
    %MDEx.Heading{
      nodes: [
        %MDEx.Text{literal: "Hello from "},
        %MDEx.Code{num_backticks: 1, literal: "~M"},
        %MDEx.Text{literal: " sigil"}
      ],
      level: 1,
      setext: false
    }
  ]
}
iex> import MDEx.Sigil
iex> ~M|`~M` also converts to HTML format|HTML
"<p><code>~M</code> also converts to HTML format</p>"
iex> import MDEx.Sigil
iex> ~M|and to XML as well|XML
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE document SYSTEM \"CommonMark.dtd\">\n<document xmlns=\"http://commonmark.org/xml/1.0\">\n  <paragraph>\n    <text xml:space=\"preserve\">and to XML as well</text>\n  </paragraph>\n</document>\n"

Use ~m to interpolate variables:

iex> import MDEx.Sigil
iex> lang = :elixir
iex> ~m|`lang = #{inspect(lang)}`|
%MDEx.Document{nodes: [%MDEx.Paragraph{nodes: [%MDEx.Code{num_backticks: 1, literal: "lang = :elixir"}]}]}

See more info at https://hexdocs.pm/mdex/MDEx.Sigil.html

Safety

For security reasons, every piece of raw HTML is omitted from the output by default:

iex> MDEx.to_html!("<h1>Hello</h1>")
"<!-- raw HTML omitted -->"

That's not very useful for most cases, but you have a few options:

Escape

The most basic is render raw HTML but escape it:

iex> MDEx.to_html!("<h1>Hello</h1>", render: [escape: true])
"&lt;h1&gt;Hello&lt;/h1&gt;"

Sanitize

But if the input is provided by external sources, it might be a good idea to sanitize it:

iex> MDEx.to_html!("<a href=https://elixir-lang.org>Elixir</a>", render: [unsafe_: true], features: [sanitize: MDEx.default_sanitize_options()])
"<p><a href=\"https://elixir-lang.org\" rel=\"noopener noreferrer\">Elixir</a></p>"

Note that you must pass the unsafe_: true option to first generate the raw HTML in order to sanitize it.

It does clean HTML with a conservative set of defaults that works for most cases, but you can overwrite those rules for further customization.

For example, let's modify the link rel attribute to add "nofollow" into the rel attribute:

iex> MDEx.to_html!("<a href=https://someexternallink.com>External</a>", render: [unsafe_: true], features: [sanitize: [link_rel: "nofollow noopener noreferrer"]])
"<p><a href=\"https://someexternallink.com\" rel=\"nofollow noopener noreferrer\">External</a></p>"

In this case the default rule set is still applied but the link_rel rule is overwritten.

Unsafe

If those rules are too strict and you really trust the input, or you really need to render raw HTML, then you can just render it directly without escaping nor sanitizing:

iex> MDEx.to_html!("<script>alert('hello')</script>", render: [unsafe_: true])
"<script>alert('hello')</script>"

Parsing

Converts Markdown to an AST data structure that can be inspected and manipulated to change the content of the document programmatically.

The data structure format is inspired on Floki (with :attributes_as_maps = true) so we can keep similar APIs and keep the same mental model when working with these documents, either Markdown or HTML, where each node is represented as a struct holding the node name as the struct name and its attributes and children, for eg:

%MDEx.Heading{
  level: 1
  nodes: [...],
}

The parent node that represents the root of the document is the MDEx.Document struct, where you can find more more information about the AST and what operations are available.

The complete list of nodes is listed in the documentation, section Document Nodes.

Formatting

Formatting is the process of converting from one format to another, for example from AST or Markdown to HTML. Formatting to XML and to Markdown is also supported.

You can use MDEx.parse_document/2 to generate an AST or any of the to_* functions to convert to Markdown (CommonMark), HTML, JSON, or XML.

Examples

GitHub Flavored Markdown with emojis

MDEx.to_html!(~S"""
# GitHub Flavored Markdown :rocket:

- [x] Task A
- [x] Task B
- [ ] Task C

| Feature | Status |
| ------- | ------ |
| Fast | :white_check_mark: |
| GFM  | :white_check_mark: |

Check out the spec at https://github.github.com/gfm/
""",
extension: [
  strikethrough: true,
  tagfilter: true,
  table: true,
  autolink: true,
  tasklist: true,
  footnotes: true,
  shortcodes: true,
],
parse: [
  smart: true,
  relaxed_tasklist_matching: true,
  relaxed_autolinks: true
],
render: [
  github_pre_lang: true,
  unsafe_: true,
]) |> IO.puts()
"""
<p>GitHub Flavored Markdown 🚀</p>
<ul>
  <li><input type="checkbox" checked="" disabled="" /> Task A</li>
  <li><input type="checkbox" checked="" disabled="" /> Task B</li>
  <li><input type="checkbox" disabled="" /> Task C</li>
</ul>
<table>
  <thead>
    <tr>
      <th>Feature</th>
      <th>Status</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Fast</td>
      <td>✅</td>
    </tr>
    <tr>
      <td>GFM</td>
      <td>✅</td>
    </tr>
  </tbody>
</table>
<p>Check out the spec at <a href="https://github.github.com/gfm/">https://github.github.com/gfm/</a></p>
"""

Code Syntax Highlighting

MDEx.to_html!(~S"""
```elixir
String.upcase("elixir")
```
""",
features: [syntax_highlight_theme: "catppuccin_latte"]
) |> IO.puts()
"""
<pre class=\"autumn highlight\" style=\"background-color: #282C34; color: #ABB2BF;\">
  <code class=\"language-elixir\" translate=\"no\">
    <span class=\"namespace\" style=\"color: #61AFEF;\">String</span><span class=\"operator\" style=\"color: #C678DD;\">.</span><span class=\"function\" style=\"color: #61AFEF;\">upcase</span><span class=\"\" style=\"color: #ABB2BF;\">(</span><span class=\"string\" style=\"color: #98C379;\">&quot;elixir&quot;</span><span class=\"\" style=\"color: #ABB2BF;\">)</span>
  </code>
</pre>
"""

Pre-compilation

Pre-compiled binaries are available for the following targets, so you don't need to have Rust installed to compile and use this library:

  • aarch64-apple-darwin
  • aarch64-unknown-linux-gnu
  • aarch64-unknown-linux-musl
  • arm-unknown-linux-gnueabihf
  • riscv64gc-unknown-linux-gnu
  • x86_64-apple-darwin
  • x86_64-pc-windows-gnu
  • x86_64-pc-windows-msvc
  • x86_64-unknown-freebsd
  • x86_64-unknown-linux-gnu
  • x86_64-unknown-linux-musl

Compile manually

But in case you need or want to compile it yourself, you can do the following:

  1. Install Rust

  2. Install a C compiler or build packages

It depends on your OS, for example in Ubuntu you can install the build-essential package.

  1. Run:
export MDEX_BUILD=1
mix deps.get
mix compile

Legacy CPUs

Modern CPU features are enabled by default but if your environment has an older CPU, you can use legacy artifacts by adding the following configuration to your config.exs:

config :mdex, use_legacy_artifacts: true

Demo and Samples

A livebook and a script are available to play with and experiment with this library.

Used By

Are you using MDEx and want to list your project here? Please send a PR!

Benchmark

A simple script is available to compare existing libs:

Name              ips        average  deviation         median         99th %
cmark         22.82 K      0.0438 ms    ±16.24%      0.0429 ms      0.0598 ms
mdex           3.57 K        0.28 ms     ±9.79%        0.28 ms        0.33 ms
md             0.34 K        2.95 ms    ±10.56%        2.90 ms        3.62 ms
earmark        0.25 K        4.04 ms     ±4.50%        4.00 ms        4.44 ms

Comparison:
cmark         22.82 K
mdex           3.57 K - 6.39x slower +0.24 ms
md             0.34 K - 67.25x slower +2.90 ms
earmark        0.25 K - 92.19x slower +4.00 ms

Motivation

MDEx was born out of the necessity of parsing CommonMark files, to parse hundreds of files quickly, and to be easily extensible by consumers of the library.

  • earmark is extensible but can't parse all kinds of documents and is slow to convert hundreds of markdowns.
  • md is very extensible but the doc says "If one needs to perfectly parse the common markdown, Md is probably not the correct choice" and CommonMark was a requirement to parse many existing files.
  • markdown is not precompiled and has not received updates in a while.
  • cmark is a fast CommonMark parser but it requires compiling the C library, is hard to extend, and was archived on Apr 2024

Note that MDEx is the only one that syntax highlights out-of-the-box which contributes to make it slower than cmark.

To finish, a friendly reminder that all libs have their own strengths and trade-offs so use the one that better suit your needs.

Looking for help with your Elixir project?

DockYard logo

At DockYard we are ready to help you build your next Elixir project. We have a unique expertise in Elixir and Phoenix development that is unmatched and we love to write about Elixir.

Have a project in mind? Get in touch!

Acknowledgements

  • comrak crate for all the heavy work on parsing Markdown and rendering HTML
  • Floki for the AST
  • Req for the pipeline API
  • Logo based on markdown-mark

Summary

Types

List of extra features.

Options to customize the parsing and rendering of Markdown documents.

Source accepted by parse_document/2.

Input source document.

Functions

Returns the default options for the :extension group.

Returns the default options for the :features group.

Returns the default options for the :parse group.

Returns the default options for the :render group.

Returns the default options for the :sanitize group.

Builds a new MDEx.Pipe instance.

Same as parse_document/2 but raises if the parsing fails.

Parse a markdown string and returns only the node that represents the fragment.

Same as parse_fragment/2 but raises if the parsing fails or returns nil.

Utility function to sanitize and escape HTML.

Convert Markdown, MDEx.Document, or MDEx.Pipe to HTML using default options.

Convert Markdown, MDEx.Document, or MDEx.Pipe to HTML using custom options.

Same as to_html/1 but raises an error if the conversion fails.

Same as to_html/2 but raises error if the conversion fails.

Convert Markdown, MDEx.Document, or MDEx.Pipe to JSON using default options.

Convert Markdown, MDEx.Document, or MDEx.Pipe to JSON using custom options.

Same as to_json/1 but raises an error if the conversion fails.

Same as to_json/2 but raises error if the conversion fails.

Convert MDEx.Document or MDEx.Pipe to Markdown using default options.

Convert MDEx.Document or MDEx.Pipe to Markdown using custom options.

Same as to_markdown/1 but raises MDEx.DecodeError if the conversion fails.

Same as to_markdown/2 but raises MDEx.DecodeError if the conversion fails.

Convert Markdown, MDEx.Document, or MDEx.Pipe to XML using default options.

Convert Markdown, MDEx.Document, or MDEx.Pipe to XML using custom options.

Same as to_xml/1 but raises an error if the conversion fails.

Same as to_xml/2 but raises error if the conversion fails.

Low-level function to traverse and update the Markdown document preserving the tree structure format.

Low-level function to traverse and update the Markdown document preserving the tree structure format and keeping an accumulator.

Types

extension_options()

@type extension_options() :: [
  strikethrough: boolean(),
  tagfilter: boolean(),
  table: boolean(),
  autolink: boolean(),
  tasklist: boolean(),
  superscript: boolean(),
  header_ids: binary() | nil,
  footnotes: boolean(),
  description_lists: boolean(),
  front_matter_delimiter: binary() | nil,
  multiline_block_quotes: boolean(),
  alerts: boolean(),
  math_dollars: boolean(),
  math_code: boolean(),
  shortcodes: boolean(),
  wikilinks_title_after_pipe: boolean(),
  wikilinks_title_before_pipe: boolean(),
  underline: boolean(),
  subscript: boolean(),
  spoiler: boolean(),
  greentext: boolean()
]

List of comrak extension options.

features_options()

@type features_options() :: [
  sanitize: sanitize_options() | nil,
  syntax_highlight_theme: binary() | nil,
  syntax_highlight_inline_style: boolean()
]

List of extra features.

options()

@type options() :: [
  document: binary() | struct() | nil,
  extension: extension_options(),
  parse: parse_options(),
  render: render_options(),
  features: features_options()
]

Options to customize the parsing and rendering of Markdown documents.

Examples

  • Enable the table extension:

      iex> MDEx.to_html!("""
      ...> | lang |
      ...> |------|
      ...> | elixir |
      ...> """,
      ...> extension: [table: true])
      "<table>\n<thead>\n<tr>\n<th>lang</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>elixir</td>\n</tr>\n</tbody>\n</table>"

Options

  • :document - Markdown document, either a string or a MDEx.Document struct. The default value is "".

  • :extension (keyword/0) - Enable extensions. See comrak's ExtensionOptions for more info and examples. The default value is [].

    • :strikethrough (boolean/0) - Enables the strikethrough extension from the GFM spec. The default value is false.

    • :tagfilter (boolean/0) - Enables the tagfilter extension from the GFM spec. The default value is false.

    • :table (boolean/0) - Enables the table extension from the GFM spec. The default value is false.

    • :autolink (boolean/0) - Enables the autolink extension from the GFM spec. The default value is false.

    • :tasklist (boolean/0) - Enables the task list extension from the GFM spec. The default value is false.

    • :superscript (boolean/0) - Enables the superscript Comrak extension. The default value is false.

    • :header_ids - Enables the header IDs Comrak extension. The default value is nil.

    • :footnotes (boolean/0) - Enables the footnotes extension per cmark-gfm The default value is false.

    • :description_lists (boolean/0) - Enables the description lists extension. The default value is false.

    • :front_matter_delimiter - Enables the front matter extension. The default value is nil.

    • :multiline_block_quotes (boolean/0) - Enables the multiline block quotes extension. The default value is false.

    • :alerts (boolean/0) - Enables GitHub style alerts. The default value is false.

    • :math_dollars (boolean/0) - Enables math using dollar syntax. The default value is false.

    • :math_code (boolean/0) - Enables the math code extension from the GFM spec. The default value is false.

    • :shortcodes (boolean/0) - Phrases wrapped inside of ':' blocks will be replaced with emojis. The default value is false.

    • :wikilinks_title_after_pipe (boolean/0) - Enables wikilinks using title after pipe syntax. The default value is false.

    • :wikilinks_title_before_pipe (boolean/0) - Enables wikilinks using title before pipe syntax. The default value is false.

    • :underline (boolean/0) - Enables underlines using double underscores. The default value is false.

    • :subscript (boolean/0) - Enables subscript text using single tildes. The default value is false.

    • :spoiler (boolean/0) - Enables spoilers using double vertical bars. The default value is false.

    • :greentext (boolean/0) - Requires at least one space after a > character to generate a blockquote, and restarts blockquote nesting across unique lines of input. The default value is false.

  • :parse (keyword/0) - Configure parsing behavior. See comrak's ParseOptions for more info and examples. The default value is [].

    • :smart (boolean/0) - Punctuation (quotes, full-stops and hyphens) are converted into 'smart' punctuation. The default value is false.

    • :default_info_string - The default info string for fenced code blocks. The default value is nil.

    • :relaxed_tasklist_matching (boolean/0) - Whether or not a simple x or X is used for tasklist or any other symbol is allowed. The default value is false.

    • :relaxed_autolinks (boolean/0) - Relax parsing of autolinks, allow links to be detected inside brackets and allow all url schemes. It is intended to allow a very specific type of autolink detection, such as [this http://and.com that] or {http://foo.com}, on a best can basis. The default value is true.

  • :render (keyword/0) - Configure rendering behavior. See comrak's RenderOptions for more info and examples. The default value is [].

    • :hardbreaks (boolean/0) - Soft line breaks in the input translate into hard line breaks in the output. The default value is false.

    • :github_pre_lang (boolean/0) - GitHub-style <pre lang="xyz"> is used for fenced code blocks with info tags. The default value is false.

    • :full_info_string (boolean/0) - Enable full info strings for code blocks. The default value is false.

    • :width (integer/0) - The wrap column when outputting CommonMark. The default value is 0.

    • :unsafe_ (boolean/0) - Allow rendering of raw HTML and potentially dangerous links. The default value is false.

    • :escape (boolean/0) - Escape raw HTML instead of clobbering it. The default value is false.

    • :list_style - Set the type of bullet list marker to use. Either one of :dash, :plus, or :star. The default value is :dash.

    • :sourcepos (boolean/0) - Include source position attributes in HTML and XML output. The default value is false.

    • :experimental_inline_sourcepos (boolean/0) - Include inline sourcepos in HTML output, which is known to have issues. The default value is false.

    • :escaped_char_spans (boolean/0) - Wrap escaped characters in a <span> to allow any post-processing to recognize them. The default value is false.

    • :ignore_setext (boolean/0) - Ignore setext headings in input. The default value is false.

    • :ignore_empty_links (boolean/0) - Ignore empty links in input. The default value is false.

    • :gfm_quirks (boolean/0) - Enables GFM quirks in HTML output which break CommonMark compatibility. The default value is false.

    • :prefer_fenced (boolean/0) - Prefer fenced code blocks when outputting CommonMark. The default value is false.

    • :figure_with_caption (boolean/0) - Render the image as a figure element with the title as its caption. The default value is false.

    • :tasklist_classes (boolean/0) - Add classes to the output of the tasklist extension. This allows tasklists to be styled. The default value is false.

    • :ol_width (integer/0) - Render ordered list with a minimum marker width. Having a width lower than 3 doesn't do anything. The default value is 1.

    • :experimental_minimize_commonmark (boolean/0) - Minimise escapes used in CommonMark output (-t commonmark) by removing each individually and seeing if the resulting document roundtrips. Brute-force and expensive, but produces nicer output. Note that the result may not in fact be minimal. The default value is false.

  • :features (keyword/0) - Enable extra features. The default value is [].

    • :sanitize - Cleans HTML using ammonia after rendering.

      Use a conservative set of options by default, but you can overwrite the default options MDEx.default_sanitize_options/0. For example to build an empty base with no allowed tags:

      empty_base = Keyword.put(MDEx.default_sanitize_options(), :tags, [])
      features: [sanitize: empty_base]

      See the Safety section for more info.

      The default value is nil.

    • :syntax_highlight_theme - syntax highlight code fences using autumn themes, you should pass the filename without special chars and without extension, for example you should pass syntax_highlight_theme: "adwaita_dark" to use the Adwaita Dark theme. The default value is "onedark".

    • :syntax_highlight_inline_style (boolean/0) - embed styles in the output for each generated token. You'll need to serve CSS themes if inline styles are disabled to properly highlight code. The default value is true.

parse_options()

@type parse_options() :: [
  smart: boolean(),
  default_info_string: binary() | nil,
  relaxed_tasklist_matching: boolean(),
  relaxed_autolinks: boolean()
]

List of comrak parse options.

parse_source()

@type parse_source() :: markdown :: String.t() | {:json, String.t()}

Source accepted by parse_document/2.

render_options()

@type render_options() :: [
  hardbreaks: boolean(),
  github_pre_lang: boolean(),
  full_info_string: boolean(),
  width: integer(),
  unsafe_: boolean(),
  escape: boolean(),
  list_style: term(),
  sourcepos: boolean(),
  experimental_inline_sourcepos: boolean(),
  escaped_char_spans: boolean(),
  ignore_setext: boolean(),
  ignore_empty_links: boolean(),
  gfm_quirks: boolean(),
  prefer_fenced: boolean(),
  figure_with_caption: boolean(),
  tasklist_classes: boolean(),
  ol_width: integer(),
  experimental_minimize_commonmark: boolean()
]

List of comrak render options.

sanitize_options()

@type sanitize_options() :: [
  tags: [binary()],
  add_tags: [binary()],
  rm_tags: [binary()],
  clean_content_tags: [binary()],
  add_clean_content_tags: [binary()],
  rm_clean_content_tags: [binary()],
  tag_attributes: %{optional(binary()) => [binary()]},
  add_tag_attributes: %{optional(binary()) => [binary()]},
  rm_tag_attributes: %{optional(binary()) => [binary()]},
  tag_attribute_values: %{
    optional(binary()) => %{optional(binary()) => [binary()]}
  },
  add_tag_attribute_values: %{
    optional(binary()) => %{optional(binary()) => [binary()]}
  },
  rm_tag_attribute_values: %{
    optional(binary()) => %{optional(binary()) => [binary()]}
  },
  set_tag_attribute_values: %{
    optional(binary()) => %{optional(binary()) => binary()}
  },
  set_tag_attribute_value: %{
    optional(binary()) => %{optional(binary()) => binary()}
  },
  rm_set_tag_attribute_value: %{optional(binary()) => binary()},
  generic_attribute_prefixes: [binary()],
  add_generic_attribute_prefixes: [binary()],
  rm_generic_attribute_prefixes: [binary()],
  generic_attributes: [binary()],
  add_generic_attributes: [binary()],
  rm_generic_attributes: [binary()],
  url_schemes: [binary()],
  add_url_schemes: [binary()],
  rm_url_schemes: [binary()],
  url_relative: term() | {atom(), binary()} | {atom(), {binary(), binary()}},
  link_rel: binary() | nil,
  allowed_classes: %{optional(binary()) => [binary()]},
  add_allowed_classes: %{optional(binary()) => [binary()]},
  rm_allowed_classes: %{optional(binary()) => [binary()]},
  strip_comments: boolean(),
  id_prefix: binary() | nil
]

List of ammonia options.

source()

@type source() :: markdown :: String.t() | MDEx.Document.t() | MDEx.Pipe.t()

Input source document.

Examples

  • From Markdown to HTML

    iex> MDEx.to_html!("# Hello")
    "<h1>Hello</h1>"
  • From Markdown to MDEx.Document

    iex> MDEx.parse_document!("Hello")
    %MDEx.Document{
      nodes: [
        %MDEx.Paragraph{nodes: [%MDEx.Text{literal: "Hello"}]}
      ]
    }
  • From MDEx.Document to HTML

    iex> MDEx.to_html!(%MDEx.Document{
    ...>   nodes: [
    ...>     %MDEx.Paragraph{nodes: [%MDEx.Text{literal: "Hello"}]}
    ...>   ]
    ...> })
    "<p>Hello</p>"

You can also leverage MDEx.Document as an intermediate data type to convert between formats:

  • From JSON to HTML:

    iex> json = ~s|{"nodes":[{"nodes":[{"literal":"Hello","node_type":"MDEx.Text"}],"level":1,"setext":false,"node_type":"MDEx.Heading"}],"node_type":"MDEx.Document"}|
    iex> {:json, json} |> MDEx.parse_document!() |> MDEx.to_html!()
    "<h1>Hello</h1>"

Functions

default_extension_options()

@spec default_extension_options() :: extension_options()

Returns the default options for the :extension group.

default_features_options()

@spec default_features_options() :: features_options()

Returns the default options for the :features group.

default_parse_options()

@spec default_parse_options() :: parse_options()

Returns the default options for the :parse group.

default_render_options()

@spec default_render_options() :: render_options()

Returns the default options for the :render group.

default_sanitize_options()

@spec default_sanitize_options() :: sanitize_options()

Returns the default options for the :sanitize group.

new(options \\ [])

@spec new(options()) :: MDEx.Pipe.t()

Builds a new MDEx.Pipe instance.

Once the pipe is complete, call either one of the following functions to format the document:

Examples

  • Build a pipe with :document:

    iex> mdex = MDEx.new(document: "# Hello")
    iex> MDEx.to_html(mdex)
    {:ok, "<h1>Hello</h1>"}
    
    iex> mdex = MDEx.new(document: "Hello ~world~", extension: [strikethrough: true])
    iex> MDEx.to_json(mdex)
    {:ok, ~s|{"nodes":[{"nodes":[{"literal":"Hello ","node_type":"MDEx.Text"},{"nodes":[{"literal":"world","node_type":"MDEx.Text"}],"node_type":"MDEx.Strikethrough"}],"node_type":"MDEx.Paragraph"}],"node_type":"MDEx.Document"}|}
  • Pass a :document when formatting:

    iex> mdex = MDEx.new(extension: [strikethrough: true])
    iex> MDEx.to_xml(mdex, document: "Hello ~world~")
    {:ok, ~s|<?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE document SYSTEM "CommonMark.dtd">
    <document xmlns="http://commonmark.org/xml/1.0">
      <paragraph>
        <text xml:space="preserve">Hello </text>
        <strikethrough>
          <text xml:space="preserve">world</text>
        </strikethrough>
      </paragraph>
    </document>|}

Notes

  1. Source :document is automatically parsed into MDEx.Document before the pipeline runs so every step receives the same data type.

  2. You can pass the document when creating the pipe:

MDEx.new(document: "# Hello") |> MDEx.to_html()

Or pass it only when formatting the document, useful to reuse the same pipe with different documents and formats.

mdex = MDEx.new()
# ... attach plugins and steps

MDEx.to_html(mdex, document: "# Hello HTML")
MDEx.to_json(mdex, document: "# Hello JSON")

parse_document(source, options \\ [])

@spec parse_document(parse_source(), options()) ::
  {:ok, MDEx.Document.t()} | {:error, any()}

Parse source and returns MDEx.Document.

Source can be either a Markdown string or a tagged JSON string.

Examples

  • Parse Markdown with default options:

    iex> MDEx.parse_document!("""
    ...> # Languages
    ...>
    ...> - Elixir
    ...> - Rust
    ...> """)
    %MDEx.Document{
      nodes: [
        %MDEx.Heading{nodes: [%MDEx.Text{literal: "Languages"}], level: 1, setext: false},
        %MDEx.List{
          nodes: [
            %MDEx.ListItem{
              nodes: [%MDEx.Paragraph{nodes: [%MDEx.Text{literal: "Elixir"}]}],
              list_type: :bullet,
              marker_offset: 0,
              padding: 2,
              start: 1,
              delimiter: :period,
              bullet_char: "-",
              tight: false
            },
            %MDEx.ListItem{
              nodes: [%MDEx.Paragraph{nodes: [%MDEx.Text{literal: "Rust"}]}],
              list_type: :bullet,
              marker_offset: 0,
              padding: 2,
              start: 1,
              delimiter: :period,
              bullet_char: "-",
              tight: false
            }
          ],
          list_type: :bullet,
          marker_offset: 0,
          padding: 2,
          start: 1,
          delimiter: :period,
          bullet_char: "-",
          tight: true
        }
      ]
    }
  • Parse Markdown with custom options:

    iex> MDEx.parse_document!("Darth Vader is ||Luke's father||", extension: [spoiler: true])
    %MDEx.Document{
      nodes: [
        %MDEx.Paragraph{
          nodes: [
            %MDEx.Text{literal: "Darth Vader is "},
            %MDEx.SpoileredText{nodes: [%MDEx.Text{literal: "Luke's father"}]}
          ]
        }
      ]
    }
  • Parse JSON:

    iex> json = ~s|{"nodes":[{"nodes":[{"literal":"Title","node_type":"MDEx.Text"}],"level":1,"setext":false,"node_type":"MDEx.Heading"}],"node_type":"MDEx.Document"}|
    iex> MDEx.parse_document!({:json, json})
    %MDEx.Document{
      nodes: [
        %MDEx.Heading{
          nodes: [%MDEx.Text{literal: "Title"} ],
          level: 1,
          setext: false
        }
      ]
    }

parse_document!(source, options \\ [])

@spec parse_document!(parse_source(), options()) :: MDEx.Document.t()

Same as parse_document/2 but raises if the parsing fails.

parse_fragment(markdown, options \\ [])

@spec parse_fragment(String.t(), options()) :: {:ok, MDEx.Document.md_node()} | nil

Parse a markdown string and returns only the node that represents the fragment.

Usually that means filtering out the parent document and paragraphs.

That's useful to generate fragment nodes and inject them into the document when you're manipulating it.

Use parse_document/2 to generate a complete document.

Experimental

Consider this function experimental and subject to change.

Examples

iex> MDEx.parse_fragment("# Elixir")
{:ok, %MDEx.Heading{nodes: [%MDEx.Text{literal: "Elixir"}], level: 1, setext: false}}

iex> MDEx.parse_fragment("<h1>Elixir</h1>")
{:ok, %MDEx.HtmlBlock{nodes: [], block_type: 6, literal: "<h1>Elixir</h1>\n"}}

parse_fragment!(markdown, options \\ [])

@spec parse_fragment!(String.t(), options()) :: MDEx.Document.md_node()

Same as parse_fragment/2 but raises if the parsing fails or returns nil.

Experimental

Consider this function experimental and subject to change.

safe_html(unsafe_html, options \\ [])

@spec safe_html(
  String.t(),
  options :: [sanitize: sanitize_options() | nil, escape: [atom()]]
) :: String.t()

Utility function to sanitize and escape HTML.

Examples

iex> MDEx.safe_html("<script>console.log('attack')</script>")
""

iex> MDEx.safe_html("<h1>{'Example:'}</h1><code>{:ok, 'MDEx'}</code>")
"&lt;h1&gt;{&#x27;Example:&#x27;}&lt;&#x2f;h1&gt;&lt;code&gt;&lbrace;:ok, &#x27;MDEx&#x27;&rbrace;&lt;&#x2f;code&gt;"

iex> MDEx.safe_html("<h1>{'Example:'}</h1><code>{:ok, 'MDEx'}</code>", escape: [content: false])
"<h1>{'Example:'}</h1><code>&lbrace;:ok, 'MDEx'&rbrace;</code>"

Options

  • :sanitize - cleans HTML after rendering. Defaults to MDEx.default_sanitize_options/0.

  • :escape - which entities should be escaped. Defaults to [:content, :curly_braces_in_code].

    • :content - escape common chars like <, >, &, and others in the HTML content;
    • :curly_braces_in_code - escape { and } only inside <code> tags, particularly useful for compiling HTML in LiveView;

to_commonmark(document)

This function is deprecated. Use `to_markdown/1` instead.

to_commonmark(document, options)

This function is deprecated. Use `to_markdown/2` instead.

to_commonmark!(document)

This function is deprecated. Use `to_markdown!/1` instead.

to_commonmark!(document, options)

This function is deprecated. Use `to_markdown!/2` instead.

to_html(source)

@spec to_html(source()) ::
  {:ok, String.t()}
  | {:error, MDEx.DecodeError.t()}
  | {:error, MDEx.InvalidInputError.t()}

Convert Markdown, MDEx.Document, or MDEx.Pipe to HTML using default options.

Use to_html/2 to pass custom options.

Examples

iex> MDEx.to_html("# MDEx")
{:ok, "<h1>MDEx</h1>"}

iex> MDEx.to_html("Implemented with:\n1. Elixir\n2. Rust")
{:ok, "<p>Implemented with:</p>\n<ol>\n<li>Elixir</li>\n<li>Rust</li>\n</ol>"}

iex> MDEx.to_html(%MDEx.Document{nodes: [%MDEx.Heading{nodes: [%MDEx.Text{literal: "MDEx"}], level: 3, setext: false}]})
{:ok, "<h3>MDEx</h3>"}

Fragments of a document are also supported:

iex> MDEx.to_html(%MDEx.Paragraph{nodes: [%MDEx.Text{literal: "MDEx"}]})
{:ok, "<p>MDEx</p>"}

to_html(source, options)

@spec to_html(source(), options()) ::
  {:ok, String.t()}
  | {:error, MDEx.DecodeError.t()}
  | {:error, MDEx.InvalidInputError.t()}

Convert Markdown, MDEx.Document, or MDEx.Pipe to HTML using custom options.

Examples

iex> MDEx.to_html("Hello ~world~ there", extension: [strikethrough: true])
{:ok, "<p>Hello <del>world</del> there</p>"}

iex> MDEx.to_html("<marquee>visit https://beaconcms.org</marquee>", extension: [autolink: true], render: [unsafe_: true])
{:ok, "<p><marquee>visit <a href=\"https://beaconcms.org\">https://beaconcms.org</a></marquee></p>"}

to_html!(source)

@spec to_html!(source()) :: String.t()

Same as to_html/1 but raises an error if the conversion fails.

to_html!(source, options)

@spec to_html!(source(), options()) :: String.t()

Same as to_html/2 but raises error if the conversion fails.

to_json(source)

@spec to_json(source()) ::
  {:ok, String.t()}
  | {:error, MDEx.DecodeError.t()}
  | {:error, MDEx.InvalidInputError.t()}

Convert Markdown, MDEx.Document, or MDEx.Pipe to JSON using default options.

Use to_json/2 to pass custom options.

Examples

iex> MDEx.to_json("# Hello")
{:ok, ~s|{"nodes":[{"nodes":[{"literal":"Hello","node_type":"MDEx.Text"}],"level":1,"setext":false,"node_type":"MDEx.Heading"}],"node_type":"MDEx.Document"}|}

iex> MDEx.to_json("1. First\n2. Second")
{:ok, ~s|{"nodes":[{"start":1,"nodes":[{"start":1,"nodes":[{"nodes":[{"literal":"First","node_type":"MDEx.Text"}],"node_type":"MDEx.Paragraph"}],"delimiter":"period","padding":3,"list_type":"ordered","marker_offset":0,"bullet_char":"","tight":false,"is_task_list":false,"node_type":"MDEx.ListItem"},{"start":2,"nodes":[{"nodes":[{"literal":"Second","node_type":"MDEx.Text"}],"node_type":"MDEx.Paragraph"}],"delimiter":"period","padding":3,"list_type":"ordered","marker_offset":0,"bullet_char":"","tight":false,"is_task_list":false,"node_type":"MDEx.ListItem"}],"delimiter":"period","padding":3,"list_type":"ordered","marker_offset":0,"bullet_char":"","tight":true,"is_task_list":false,"node_type":"MDEx.List"}],"node_type":"MDEx.Document"}|}

iex> MDEx.to_json(%MDEx.Document{nodes: [%MDEx.Heading{nodes: [%MDEx.Text{literal: "Hello"}], level: 3, setext: false}]})
{:ok, ~s|{"nodes":[{"nodes":[{"literal":"Hello","node_type":"MDEx.Text"}],"level":3,"setext":false,"node_type":"MDEx.Heading"}],"node_type":"MDEx.Document"}|}

Fragments of a document are also supported:

iex> MDEx.to_json(%MDEx.Paragraph{nodes: [%MDEx.Text{literal: "Hello"}]})
{:ok, ~s|{"nodes":[{"nodes":[{"literal":"Hello","node_type":"MDEx.Text"}],"node_type":"MDEx.Paragraph"}],"node_type":"MDEx.Document"}|}

to_json(source, options)

@spec to_json(source(), options()) ::
  {:ok, String.t()}
  | {:error, MDEx.DecodeError.t()}
  | {:error, MDEx.InvalidInputError.t()}

Convert Markdown, MDEx.Document, or MDEx.Pipe to JSON using custom options.

Examples

iex> MDEx.to_json("Hello ~world~", extension: [strikethrough: true])
{:ok, ~s|{"nodes":[{"nodes":[{"literal":"Hello ","node_type":"MDEx.Text"},{"nodes":[{"literal":"world","node_type":"MDEx.Text"}],"node_type":"MDEx.Strikethrough"}],"node_type":"MDEx.Paragraph"}],"node_type":"MDEx.Document"}|}

to_json!(source)

@spec to_json!(source()) :: String.t()

Same as to_json/1 but raises an error if the conversion fails.

to_json!(source, options)

@spec to_json!(source(), options()) :: String.t()

Same as to_json/2 but raises error if the conversion fails.

to_markdown(source)

@spec to_markdown(MDEx.Document.t() | MDEx.Pipe.t()) ::
  {:ok, String.t()} | {:error, MDEx.DecodeError.t()}

Convert MDEx.Document or MDEx.Pipe to Markdown using default options.

Use to_markdown/2 to pass custom options.

Example

iex> MDEx.to_markdown(%MDEx.Document{nodes: [%MDEx.Heading{nodes: [%MDEx.Text{literal: "Hello"}], level: 3, setext: false}]})
{:ok, "### Hello"}

to_markdown(source, options)

@spec to_markdown(MDEx.Document.t() | MDEx.Pipe.t(), options()) ::
  {:ok, String.t()} | {:error, MDEx.DecodeError.t()}

Convert MDEx.Document or MDEx.Pipe to Markdown using custom options.

to_markdown!(document)

@spec to_markdown!(MDEx.Document.t()) :: String.t()

Same as to_markdown/1 but raises MDEx.DecodeError if the conversion fails.

to_markdown!(document, options)

@spec to_markdown!(MDEx.Document.t(), options()) :: String.t()

Same as to_markdown/2 but raises MDEx.DecodeError if the conversion fails.

to_xml(source)

@spec to_xml(source()) ::
  {:ok, String.t()}
  | {:error, MDEx.DecodeError.t()}
  | {:error, MDEx.InvalidInputError.t()}

Convert Markdown, MDEx.Document, or MDEx.Pipe to XML using default options.

Use to_xml/2 to pass custom options.

Examples

iex> {:ok, xml} =  MDEx.to_xml("# MDEx")
iex> xml
"""
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE document SYSTEM "CommonMark.dtd">
<document xmlns="http://commonmark.org/xml/1.0">
  <heading level="1">
    <text xml:space="preserve">MDEx</text>
  </heading>
</document>
"""

iex> {:ok, xml} = MDEx.to_xml("Implemented with:\n1. Elixir\n2. Rust")
iex> xml
"""
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE document SYSTEM "CommonMark.dtd">
<document xmlns="http://commonmark.org/xml/1.0">
  <paragraph>
    <text xml:space="preserve">Implemented with:</text>
  </paragraph>
  <list type="ordered" start="1" delim="period" tight="true">
    <item>
      <paragraph>
        <text xml:space="preserve">Elixir</text>
      </paragraph>
    </item>
    <item>
      <paragraph>
        <text xml:space="preserve">Rust</text>
      </paragraph>
    </item>
  </list>
</document>
"""

iex> {:ok, xml} = MDEx.to_xml(%MDEx.Document{nodes: [%MDEx.Heading{nodes: [%MDEx.Text{literal: "MDEx"}], level: 3, setext: false}]})
iex> xml
"""
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE document SYSTEM "CommonMark.dtd">
<document xmlns="http://commonmark.org/xml/1.0">
  <heading level="3">
    <text xml:space="preserve">MDEx</text>
  </heading>
</document>
"""

Fragments of a document are also supported:

iex> {:ok, xml} = MDEx.to_xml(%MDEx.Paragraph{nodes: [%MDEx.Text{literal: "MDEx"}]})
iex> xml
"""
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE document SYSTEM "CommonMark.dtd">
<document xmlns="http://commonmark.org/xml/1.0">
  <paragraph>
    <text xml:space="preserve">MDEx</text>
  </paragraph>
</document>
"""

to_xml(source, options)

@spec to_xml(source(), options()) ::
  {:ok, String.t()}
  | {:error, MDEx.DecodeError.t()}
  | {:error, MDEx.InvalidInputError.t()}

Convert Markdown, MDEx.Document, or MDEx.Pipe to XML using custom options.

Examples

iex> {:ok, xml} = MDEx.to_xml("Hello ~world~ there", extension: [strikethrough: true])
iex> xml
"""
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE document SYSTEM "CommonMark.dtd">
<document xmlns="http://commonmark.org/xml/1.0">
  <paragraph>
    <text xml:space="preserve">Hello </text>
    <strikethrough>
      <text xml:space="preserve">world</text>
    </strikethrough>
    <text xml:space="preserve"> there</text>
  </paragraph>
</document>
"""

iex> {:ok, xml} = MDEx.to_xml("<marquee>visit https://beaconcms.org</marquee>", extension: [autolink: true], render: [unsafe_: true])
iex> xml
"""
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE document SYSTEM "CommonMark.dtd">
<document xmlns="http://commonmark.org/xml/1.0">
  <paragraph>
    <html_inline xml:space="preserve">&lt;marquee&gt;</html_inline>
    <text xml:space="preserve">visit </text>
    <link destination="https://beaconcms.org" title="">
      <text xml:space="preserve">https://beaconcms.org</text>
    </link>
    <html_inline xml:space="preserve">&lt;/marquee&gt;</html_inline>
  </paragraph>
</document>
"""

to_xml!(source)

@spec to_xml!(source()) :: String.t()

Same as to_xml/1 but raises an error if the conversion fails.

to_xml!(source, options)

@spec to_xml!(source(), options()) :: String.t()

Same as to_xml/2 but raises error if the conversion fails.

traverse_and_update(ast, fun)

@spec traverse_and_update(MDEx.Document.t(), (MDEx.Document.md_node() ->
                                          MDEx.Document.md_node())) ::
  MDEx.Document.t()

Low-level function to traverse and update the Markdown document preserving the tree structure format.

See MDEx.Document for more information about the tree structure and for higher-level functions using the Access and Enumerable protocols.

Examples

Traverse an entire Markdown document:

iex> import MDEx.Sigil
iex> doc = ~M"""
...> # Languages
...>
...> `elixir`
...>
...> `rust`
...> """
iex> MDEx.traverse_and_update(doc, fn
...>   %MDEx.Code{literal: "elixir"} = node -> %{node | literal: "ex"}
...>   %MDEx.Code{literal: "rust"} = node -> %{node | literal: "rs"}
...>   node -> node
...> end)
%MDEx.Document{
  nodes: [
    %MDEx.Heading{nodes: [%MDEx.Text{literal: "Languages"}], level: 1, setext: false},
    %MDEx.Paragraph{nodes: [%MDEx.Code{num_backticks: 1, literal: "ex"}]},
    %MDEx.Paragraph{nodes: [%MDEx.Code{num_backticks: 1, literal: "rs"}]}
  ]
}

Or fragments of a document:

iex> fragment = MDEx.parse_fragment!("Lang: `elixir`")
iex> MDEx.traverse_and_update(fragment, fn
...>   %MDEx.Code{literal: "elixir"} = node -> %{node | literal: "ex"}
...>   node -> node
...> end)
%MDEx.Paragraph{nodes: [%MDEx.Text{literal: "Lang: "}, %MDEx.Code{num_backticks: 1, literal: "ex"}]}

traverse_and_update(ast, acc, fun)

@spec traverse_and_update(MDEx.Document.t(), any(), (MDEx.Document.md_node() ->
                                                 MDEx.Document.md_node())) ::
  MDEx.Document.t()

Low-level function to traverse and update the Markdown document preserving the tree structure format and keeping an accumulator.

See MDEx.Document for more information about the tree structure and for higher-level functions using the Access and Enumerable protocols.

Example

iex> import MDEx.Sigil
iex> doc = ~M"""
...> # Languages
...>
...> `elixir`
...>
...> `rust`
...> """
iex> MDEx.traverse_and_update(doc, 0, fn
...>   %MDEx.Code{literal: "elixir"} = node, acc -> {%{node | literal: "ex"}, acc + 1}
...>   %MDEx.Code{literal: "rust"} = node, acc -> {%{node | literal: "rs"}, acc + 1}
...>   node, acc -> {node, acc}
...> end)
{%MDEx.Document{
  nodes: [
    %MDEx.Heading{nodes: [%MDEx.Text{literal: "Languages"}], level: 1, setext: false},
    %MDEx.Paragraph{nodes: [%MDEx.Code{num_backticks: 1, literal: "ex"}]},
    %MDEx.Paragraph{nodes: [%MDEx.Code{num_backticks: 1, literal: "rs"}]}
  ]
}, 2}

Also works with fragments.