layout: true class: typo, typo-selection --- count: false class: nord-dark, middle, center # π From Markdown to TeX ## A Pandoc paper pipeline β and the traps that break it πͺ€ ### π β βοΈ β π β π @luk036 π¨βπ» Β· 2026 π --- ### π Agenda .pull-left[ **π§ Part 1 β The Pipeline** - Why Markdown, not raw TeX? - pandoc + crossref + citeproc - Anatomy of a source file **βοΈ Part 2 β The Build** - The command, flag by flag - One entry point: `make` ] .pull-right[ **π Part 3 β War Stories** - Emoji & Unicode β οΈ - SVG figures πΌοΈ - Headings & cleveref 𧩠- Offline citations π΄ **π― Part 4 β Takeaways** - A reproducibility checklist ] --- class: nord-light, middle, center ## π§ Part 1 β The Pipeline --- ### π€ Why Write Markdown, Not Raw TeX? .pull-left[ **Markdown wins** β - Readable **plain text** π - Diffs are *semantic*, not `\begin{}` noise π - Math stays LaTeX π - One source β PDF Β· HTML Β· slides π― ] .pull-right[ **But TeX wins at** π - Precise typesetting π - Journal classes ποΈ - Cross-refs, citations, floats π’ ] > **pandoc** is the bridge: Markdown **in**, TeX **out**, then a real engine compiles it. π --- ### π The Toolchain .mermaid[ <pre> graph LR MD["π ell-review.md\nMarkdown + YAML"] --> P["pandoc"] Y["ποΈ latex.yaml / beamer.yaml"] --> P B["π *.bib\nBibTeX data"] --> CP["citeproc"] S["π¨ *.csl\ncitation style"] --> CP P --> X["pandoc-crossref\nsec / fig / eq / tbl"] CP --> T["π generated TeX"] X --> T T --> E["pdflatex / xelatex"] E --> PDF["π ell-review.pdf"] style MD fill:#e3f2fd,stroke:#1565c0,stroke-width:3px style Y fill:#e3f2fd,stroke:#1565c0,stroke-width:3px style P fill:#fff9c4,stroke:#f57f17,stroke-width:3px style B fill:#d1c4e9,stroke:#6a1b9a,stroke-width:3px style S fill:#d1c4e9,stroke:#6a1b9a,stroke-width:3px style CP fill:#d1c4e9,stroke:#6a1b9a,stroke-width:3px style X fill:#fff9c4,stroke:#f57f17,stroke-width:3px style T fill:#fce4ec,stroke:#ad1457,stroke-width:3px style E fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px style PDF fill:#d5f5e3,stroke:#2e7d32,stroke-width:4px </pre> ] --- ### 𧬠Anatomy of a Pandoc-Markdown Paper Everything is plain text β but each `@`-thing is **load-bearing**: | syntax | meaning | |:--|:--| | YAML front matter | metadata: title, author, bibliography ποΈ | | `## Heading {#sec:foo}` | a **labelled** heading π·οΈ | | `@sec:foo` | cross-reference β βΒ§ 2β π | | `[@BGT81]` | citation β β[1]β π | | math delimiters | inline & display math π | | `` | a figure πΌοΈ | > A typo in any of these becomes a **`??`**, a missing reference, or a hard build error. β οΈ --- ### ποΈ YAML Front Matter = Document Metadata ```yaml title: "Ellipsoid Method and the Amazing Oracles" bibliography: ["ellipsoid.bib", "fir-ref.bib"] csl: "applied-mathematics-letters.csl" abstract: | The ellipsoid method is a powerful optimization technique ... ``` - `title` / `author` β `\title` / `\author` π - `bibliography` β which BibTeX files to search π - `csl` β the **citation style** π¨ - Extra YAML files add **class options**: `latex.yaml`, `crossref.yaml` βοΈ > Metadata is passed as **positional inputs** *before* the `.md`: `pandoc ell-review.yaml latex.yaml crossref.yaml ell-review.md`. 𧩠--- ### π Citations: From `[@key]` to `[1]` .mermaid[ <pre> graph LR K["citation in the source"] --> CP["citeproc"] DB[("π ellipsoid.bib\nBibTeX records")] --> CP CSL[("π¨ applied-math.csl\nCSL style")] --> CP CP --> N["[1]\nnumeric citation"] CP --> REF["π References section\nauto-generated"] style K fill:#e3f2fd,stroke:#1565c0,stroke-width:3px style DB fill:#fff9c4,stroke:#f57f17,stroke-width:3px style CSL fill:#d1c4e9,stroke:#6a1b9a,stroke-width:3px style CP fill:#d1c4e9,stroke:#6a1b9a,stroke-width:3px style N fill:#d5f5e3,stroke:#2e7d32,stroke-width:3px style REF fill:#d5f5e3,stroke:#2e7d32,stroke-width:3px </pre> ] - `[@key]` parenthetical Β· `@key` narrative Β· `[@a; @b]` multiple βοΈ - Cite only keys in a **listed** `.bib` β otherwise the reference silently vanishes π³οΈ --- ### π Cross-References: From `@sec:label` to βΒ§ 2β .mermaid[ <pre> graph LR L["labelled heading\n{#sec:cutting_plane}"] --> X["pandoc-crossref"] R["reference\nsee @sec:cutting_plane"] --> X X --> OUT["see Β§ 2"] style L fill:#e3f2fd,stroke:#1565c0,stroke-width:3px style R fill:#e3f2fd,stroke:#1565c0,stroke-width:3px style X fill:#fff9c4,stroke:#f57f17,stroke-width:3px style OUT fill:#d5f5e3,stroke:#2e7d32,stroke-width:3px </pre> ] - Label prefixes: `sec` Β· `fig` Β· `eq` Β· `tbl` Β· `lst` π·οΈ - **The separator is file-specific** β οΈ β `ell-review.md` uses `_`, `ell-review2.md` uses `-` - A mismatched separator **silently breaks** the link β `??` π³οΈ --- ### π Math: LaTeX Syntax, Rendered Twice Markdown math **is** LaTeX β pandoc passes it through, and the web renders it with KaTeX: $$ \mathcal{E}(x_c, P) \;=\; \{\, x \;\mid\; (x - {\color{#5e81ac}x_c})^{\top} {\color{#bf616a}P}^{-1} (x - {\color{#5e81ac}x_c}) \;\le\; 1 \,\} $$ - In the **PDF** β real TeX, typeset by the LaTeX engine π - In the **HTML** β KaTeX, pre-rendered on the raw source π > β οΈ Keep math **inside math delimiters**. Bare Unicode (`Ο`, `xΒ²`) in running text is what breaks `pdflatex`. β οΈ --- class: nord-light, middle, center ## βοΈ Part 2 β The Build --- ### π οΈ The Command, Flag by Flag ```bash pandoc -F pandoc-crossref --citeproc -s -t latex -N --reference-links \ --shift-heading-level-by=-1 \ --csl=applied-mathematics-letters.csl \ ell-review.yaml latex.yaml crossref.yaml ell-review.md -o ell-review.pdf ``` | flag | what it does | |:--|:--| | `-F pandoc-crossref` | resolve heading / figure / equation labels π·οΈ | | `--citeproc` | process citations + build References π | | `-s -t latex` | standalone LaTeX π | | `--shift-heading-level-by=-1` | `##` β `\section` πͺ | | `--csl=β¦` | pick the citation style π¨ | | `*.yaml` before the `.md` | metadata inputs ποΈ | --- ### 𧩠`pandoc-crossref`: `cref`, or plain `ref`? .mermaid[ <pre> graph TD CFG{"crossref.yaml\ncref option"} --> TT["true β cleveref"] CFG --> FF["false β ref + Β§"] TT --> C["cleveref package"] FF --> R["plain reference"] C --> OK["plain article: fine β "] C --> BAD["siamltex class: ?? β"] R --> GOOD["always works β "] style CFG fill:#d1c4e9,stroke:#6a1b9a,stroke-width:3px style TT fill:#e3f2fd,stroke:#1565c0,stroke-width:3px style FF fill:#fff9c4,stroke:#f57f17,stroke-width:3px style C fill:#e3f2fd,stroke:#1565c0,stroke-width:3px style R fill:#fff9c4,stroke:#f57f17,stroke-width:3px style OK fill:#d5f5e3,stroke:#2e7d32,stroke-width:3px style GOOD fill:#d5f5e3,stroke:#2e7d32,stroke-width:3px style BAD fill:#fadbd8,stroke:#c0392b,stroke-width:4px </pre> ] > Our policy: **`cref: false`** β a plain reference plus a literal **Β§** prefix. No cleveref, no `??`. π― --- class: nord-light, middle, center ## π Part 3 β War Stories --- ### β οΈ War Story 1: Emoji & Unicode Break `pdflatex` - One emoji πͺ β `! Unicode character (U+1FA9C) not set up for use with LaTeX` β - But also **bare Unicode math** in running text: `xΒ²`, `β€`, `Ο`, `Ξ²β`, `β` β οΈ $$ \text{broken: } \; x^2/4 \;\; \text{good: } \; {\color{#a3be8c}\tfrac{x^2}{4} + y^2 \le 1} $$ .mermaid[ <pre> graph LR E["πͺ emoji & bare Unicode"] --> ERR["pdflatex:\nUnicode not set up"] FIX["wrap math in delimiters"] --> OK["builds β "] style E fill:#fadbd8,stroke:#c0392b,stroke-width:4px style ERR fill:#fadbd8,stroke:#c0392b,stroke-width:4px style FIX fill:#fff9c4,stroke:#f57f17,stroke-width:3px style OK fill:#d5f5e3,stroke:#2e7d32,stroke-width:3px </pre> ] > Rule: **ASCII inside math, no emoji in sources** β or switch the engine to `xelatex`. π€ --- ### π War Story 2: `\argmax` Is Not a Command - The source wrote `\argmax` β but neither LaTeX nor KaTeX defines it β - Result: `! Undefined control sequence` β the build stops π The fix is a *real* operator πͺ $$ q_{\max} \;=\; {\color{#a3be8c}\operatorname*{arg\,max}_{q \in \mathcal{Q}}}\; f_0(x_0, q) $$ - `\operatorname*{arg\,max}` needs `amsmath` β pandoc already loads it β - The **same bug** hid in `cutting_plane.md` (the slides) β fix once, fix everywhere π --- ### πͺ War Story 3: Heading Levels β `\section` A formatter demoted `#` β `##`; pandoc then emitted **no** `\section` at all. .mermaid[ <pre> graph LR H1["level-1 heading"] --> S["section\nnumbering 1, 2 β "] H2["level-2 heading"] --> SS["subsection\nnumbering 0.1, 0.2 π±"] style H1 fill:#d5f5e3,stroke:#2e7d32,stroke-width:3px style S fill:#d5f5e3,stroke:#2e7d32,stroke-width:3px style H2 fill:#fadbd8,stroke:#c0392b,stroke-width:4px style SS fill:#fadbd8,stroke:#c0392b,stroke-width:4px </pre> ] - Before: body headings at `#` β `\section` β β**1. Introduction**β β - After the demotion: every heading became `\subsection` β β**0.1 Introduction**β π± - Fix **without touching the prose**: `--shift-heading-level-by=-1` πͺ --- ### πΌοΈ War Story 4: SVG Figures Need a Converter .mermaid[ <pre> graph TD IMG["figure link to *.svg"] --> P{"pandoc β LaTeX"} P -->|rsvg-convert found| PDF["svg β pdf\nincludegraphics β "] P -->|missing| SVG["svg package + InkScape\n+ shell-escape β"] P -->|pdf twin exists| TWIN["use the *.pdf directly β "] style IMG fill:#e3f2fd,stroke:#1565c0,stroke-width:3px style P fill:#fff9c4,stroke:#f57f17,stroke-width:3px style PDF fill:#d5f5e3,stroke:#2e7d32,stroke-width:3px style TWIN fill:#d5f5e3,stroke:#2e7d32,stroke-width:3px style SVG fill:#fadbd8,stroke:#c0392b,stroke-width:4px </pre> ] - `pdflatex` cannot read `.svg` β pandoc wants **`rsvg-convert`** (librsvg) π - Missing it β fallback to the LaTeX `svg` package β needs **InkScape + `--shell-escape`** β - Our shortcut: reference the **`.pdf` twin** that already ships beside every `.svg` π― --- ### π± War Story 5: `rsvg-convert` Drops Text Without a fontconfig, `rsvg-convert` renders **no text at all** β geometry only. <p align="center"> <img src="md-to-tex-remark.files/fir_strctr.svg" style="width:78%" /> </p> - This figure lost every label (`h[0]`, `x[t-1]`, `y[t]`) in the converted PDF π·οΈ - A picture can look βpresentβ yet be **informationally empty** π³οΈ - Always **inspect the output** β or diff extracted text with `pdftotext` π --- ### 𧩠War Story 6: cleveref vs `siamltex.cls` .mermaid[ <pre> graph TD CL["cleveref hooks the label\nand counter machinery"] --> M["but siamltex.cls\nredefines label + counter"] M --> E["label type = empty\nβ every reference is ?? β"] FIX["crossref: cref = false\nΒ§ + plain reference"] --> OK["Β§ 2, Β§ 3.1 β "] style CL fill:#e3f2fd,stroke:#1565c0,stroke-width:3px style M fill:#fadbd8,stroke:#c0392b,stroke-width:4px style E fill:#fadbd8,stroke:#c0392b,stroke-width:4px style FIX fill:#fff9c4,stroke:#f57f17,stroke-width:3px style OK fill:#d5f5e3,stroke:#2e7d32,stroke-width:3px </pre> ] - Proven with a **minimal test**: `article` + cleveref β fine; `siamltex` + cleveref β `??` π¬ - **Latest pandoc** does not help β it is the **class**, not the version π - Switching to `cref: false` restores βΒ§ 2β and removes every `??` β --- ### π΄ War Story 7: Citations That Phone Home .mermaid[ <pre> graph LR A["*.csl is a dependent style"] --> B["independent-parent =\nelsevier-with-titles"] B --> NET["π fetch from zotero.org"] NET --> FAIL["ConnectionTimeout β"] INLINE["inline the parent style"] --> SELF["self-contained β offline"] style A fill:#e3f2fd,stroke:#1565c0,stroke-width:3px style B fill:#e3f2fd,stroke:#1565c0,stroke-width:3px style NET fill:#fadbd8,stroke:#c0392b,stroke-width:4px style FAIL fill:#fadbd8,stroke:#c0392b,stroke-width:4px style INLINE fill:#fff9c4,stroke:#f57f17,stroke-width:3px style SELF fill:#d5f5e3,stroke:#2e7d32,stroke-width:3px </pre> ] - That CSL file was only a **stub** β its citation / bibliography logic lived in a parent 𧩠- Every fresh machine / CI fetches it β flaky and **non-reproducible** π² - Fix: **vendor the parent** β `--citeproc` works **offline** (verified with a dead proxy) π --- class: nord-light, middle, center ## π― Part 4 β Engineering the Build --- ### βοΈ One Entry Point: the `Makefile` | target | produces | |:--|:--| | `make paper` | `ell-review.pdf` π | | `make html` | `ell-review.html` (needs `katex/`) π | | `make slides` | `cutting_plane.pdf`, `ell.pdf` (xelatex) π₯οΈ | | `make clean` | remove `temp.*` and aux files π§Ή | ```make paper: ell-review.pdf ell-review.pdf: ell-review.md pandoc -F pandoc-crossref --citeproc -s -t latex -N \ --reference-links --shift-heading-level-by=-1 \ --csl=applied-mathematics-letters.csl \ ell-review.yaml latex.yaml crossref.yaml ell-review.md -o ell-review.pdf ``` > The `Makefile` records the **flags *and* the reasons** β no more tribal knowledge. π --- ### πͺ Make on Windows: Two Gotchas .pull-left[ **1. Recipes are exec'd directly** π - A *simple* recipe runs **without a shell** - `rm` is not an `.exe` here β `CreateProcess failed` β - Force a shell with a metacharacter β ```make clean: -@rm -f temp.tex temp.pdf ; true ``` ] .pull-right[ **2. Line endings** π - LF in the repo, CRLF on disk - Keeps diffs **logical**, not whole-file π **3. Version-matched tools** βοΈ - `pandoc` and `pandoc-crossref` must agree - A mismatch warns and can misbehave silently π ] --- ### π§Ή Repo Hygiene - **Editor backups** (`*~`, `*.un~`) β untrack and `.gitignore` them ποΈ - **Stale outputs** (`ell-review.tex`, `main-diff.*`) β generated, not source π - **One self-contained CSL** β no network at build time π΄ - **One linter, one config** β two disagreeing linters equal *zero* linters π - **Refresh `AGENTS.md`** β write the *why* while it is fresh π§ .mermaid[ <pre> graph LR A["commit sources only"] --> B["artifacts generated,\nnever hand-edited"] B --> C["reproducible:\nany machine, offline β "] style A fill:#e3f2fd,stroke:#1565c0,stroke-width:3px style B fill:#fff9c4,stroke:#f57f17,stroke-width:3px style C fill:#d5f5e3,stroke:#2e7d32,stroke-width:3px </pre> ] --- ### π Verify the Artifact, Not the Exit Code ```bash make paper # exit 0 is necessary, not sufficient pdfinfo ell-review.pdf # page count π pdftotext ell-review.pdf - | grep -c '??' # unresolved references π ``` - `??` count β did the cross-references resolve? π - Extracted text β are the **figure labels** present? (catches dropped SVG text) π·οΈ - Page count β did the build actually **change** anything? π - A tiny **CI job** running `make paper` would have caught the stale PDF π¨ --- ### β Reproducibility Checklist .pull-left[ **Dependencies** π¦ - pandoc **and** pandoc-crossref, version-matched βοΈ - a LaTeX engine (`pdflatex` / `xelatex`) π - an SVG converter **or** committed PDF twins πΌοΈ ] .pull-right[ **Sources** π - a self-contained CSL (offline) π΄ - ASCII math, no emoji (or xelatex) π€ - exactly one crossref policy βοΈ ] .mermaid[ <pre> graph LR A["same inputs"] --> B["same command"] --> C["same output β "] style A fill:#e3f2fd,stroke:#1565c0,stroke-width:3px style B fill:#fff9c4,stroke:#f57f17,stroke-width:3px style C fill:#d5f5e3,stroke:#2e7d32,stroke-width:3px </pre> ] --- ### π― Key Takeaways **The pipeline** π - pandoc + crossref + citeproc turns Markdown into journal-ready TeX π - One `make` target beats a notebook of shell history π **The traps** πͺ€ - Non-ASCII (emoji, bare Unicode) kills `pdflatex` β οΈ - Old classes can break cleveref β every reference becomes `??` 𧩠- βStandard-lookingβ macros may not exist (`\argmax`) π **The habit** π§ - Inspect the **artifact**, not the exit code π - Vendor **everything** remote β a build must work offline π΄ --- ### π References & Further Reading - **pandoc** β [the user's guide & MANUAL](https://pandoc.org/MANUAL.html) π - **pandoc-crossref** β labels for sections, figures, equations π - **CSL β Citation Style Language** β [citationstyles.org](https://citationstyles.org) π¨ - **cleveref** β and why older document classes can fight it 𧩠- The paper: **βEllipsoid Method and the Amazing Oraclesβ** π > Assembled for the `ellipsoid-method` repo; the full checklist lives in **issue #4**. ποΈ --- count: false class: nord-dark, middle, center # π Q&A **From Markdown to TeX β A Pandoc Paper Pipeline** ### Questions? Discussion? π¬ --- count: false class: nord-dark, middle, center # π Thank You ## Same input, same command, same output. π― Slides built with Remark.js π | KaTeX π | Mermaid 𧩠| Nord Theme π