feat!: replace pandoc/LaTeX PDF engine with Typst (zigmark + pozeiden)#113
Merged
Conversation
zigmark is pinned to the fix/typst-mermaid-image-emit commit (sc2in/zigmark#70) until v0.7.2 is tagged; swap to ?ref=v0.7.2 before merge. pozeiden is pinned to the 23-update-to-zig-0160 branch head (zig 0.16 support is not on its main yet); the lock resolves the same content via the v0.1.3 tag tarball.
underscoresToBlocks converts redaction underscores to solid █ bars for the typst engine (CommonMark parses 3+ underscores as thematic breaks, which render as thin gray rules instead of redaction marks). executableInPath moves from pandoc.zig so it survives the pandoc teardown.
New src/typst.zig renders policies in-process: markdown → utils preprocessing (unchanged) → zigmark AST → Typst markup with an eisvogel-matching preamble (title page + colored rule + 6cm logo, 3-column header/footer, zebra tables, TOC depth 3, 5-column version history incl. Revised By) → 'typst compile'. Mermaid diagrams render via pozeiden to inline SVG — no mermaid-filter/node/Chromium, and they now work inside the nix sandbox and on macOS. Parity preserved: CLI flags, output filename patterns (issue #97), draft.png watermark resolution (site root → theme fallback), redaction bars (underscores → █ via utils.underscoresToBlocks). pozeiden is not thread-safe (module-global arena and lazily initialised grammar caches) while policies compile concurrently, so the mermaid wrapper serialises calls behind a spinlock (zigmark's hook signature has no std.Io for an Io.Mutex). Removed: src/pandoc.zig, templates/eisvogel.latex (embed machinery in build.zig), .puppeteer.json, Config.data_dir. The 'pdf' build step now runs the typst engine standalone; e2e no longer copies the LaTeX template into starter/. Tests: pdf rendering gates on typst instead of xelatex and now exercises the mermaid fixture; new typst-source tests for inline SVG, redaction bars, and the draft background; draft.png resolution tests target Typst.resolveDraftPng.
pozeiden now guarantees concurrent render safety itself (threadlocal theme state, locked grammar-cache init — sc2in/pozeiden#35), so the consumer-side serialisation is unnecessary. Repin to the fix commit; swap to a tagged release once #35 merges and a tag is cut.
flake.nix: remove eisvogel-tex input, pandoc, mermaid-filter, imagemagick (unused), the mmdc/Chromium sandbox wrapper, and all fontconfig plumbing. typst joins runtimeDeps; fonts (Source Sans 3, Source Code Pro) reach typst via TYPST_FONT_PATHS in the devshells, apps, and the test check (with TYPST_IGNORE_SYSTEM_FONTS for determinism). zig build no longer depends on templates/, so Zola template edits stop busting the build cache. action.yml: drop the eisvogel.latex copy step; pin the build shell to the action's own checkout (path: flake ref) so consumers on released binaries can't be broken by flake changes on main. CI: macOS upgrades from 'zig build check' to the full test suite (the mermaid-filter/Chromium blocker is gone); remove the apparmor userns sysctls that existed only for Chromium. Docs: README, building-pdfs, writing-policies, installation (ADO sample → v1.4.0, closure-size note), live-editing; CHANGELOG [Unreleased] entry.
This was referenced Jul 3, 2026
redact_web is documented as web-only (the Zola templates read it for the site's redaction bars), but Config.load seeded the PDF pipeline's redact default from it. An org hiding content (e.g. phone numbers) on the public website could not keep it in the PDFs without knowing to pass --no-redact. PDF redaction is now controlled solely by --redact/--no-redact and the action's redact_mode input, which always passes an explicit flag, so action consumers are unaffected. Closes #115
zig2nix's zig-hook runs the compiler inside bwrap, which requires unprivileged user namespaces on Ubuntu 24.04 runners. The step's old comment attributed it solely to Chromium/mermaid-filter, but removing it broke any from-source build of zig2nix tooling with 'bwrap: setting up uid map: Permission denied'. Restored with an accurate comment.
typst's clap rejects '1' ('invalid value for --ignore-system-fonts'),
which failed the pdf-rendering test inside checks.test. Verified with an
unpiped 'nix flake check' this time (the earlier pass was tail masking
the exit code).
The PR preview workflow (uses: ./) installed the latest RELEASE binary — v1.3.x still speaks pandoc, which no longer exists in this branch's ci shell, so previews failed with 'pandoc not found'. When action_ref is empty (local action), build policypress from the action's own flake so previews exercise the code under review; remote consumers keep the release download. The theme checkout likewise falls back to the workflow's commit instead of the default branch.
Contributor
Preview deploymentPreview environment deleted on PR close. |
This was referenced Jul 3, 2026
Merged
Replaces the interim PR-commit pins now that both upstream releases are cut. zigmark v0.7.3 carries the typst mermaid emit fix and the body-only renderTypstWithMermaid API; pozeiden v0.2.0 carries zig 0.16 support and thread-safe rendering.
Immutable releases on the zigmark repo prevented re-pointing v0.7.3, so the release was re-cut as v0.7.4 (same fixes, plus the pozeiden v0.2.0 lazy-dep bump from zigmark#71). Pinned to the exact tag commit since zig fetch resolved the bare ?ref one commit behind the tag.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR replaces PolicyPress’s Pandoc/XeLaTeX-based PDF generation pipeline with a Typst-based engine (zigmark → pozeiden → typst compile), removes the Pandoc-era template/tooling, and updates docs/CI/action wiring accordingly. It also fixes config semantics so redact_web no longer implicitly enables PDF redaction.
Changes:
- Add a new Typst PDF engine (
src/typst.zig) and switch the build/test pipeline to use it instead of Pandoc. - Remove Pandoc/XeLaTeX/mermaid-filter infrastructure (code, template, Nix deps, CI plumbing) and update the GitHub Action to use a pinned shell/toolchain.
- Decouple
redact_web(web-only) from PDF redaction defaults and add tests for the new behavior and Typst output.
Reviewed changes
Copilot reviewed 23 out of 24 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| templates/eisvogel.latex | Removed Pandoc LaTeX template (Typst now reproduces the layout). |
| src/utils.zig | Adds underscore→█ conversion helper and relocates executableInPath. |
| src/typst.zig | New Typst PDF compiler: renders markdown via zigmark/pozeiden and invokes typst compile. |
| src/test.zig | Switches PDF rendering tests to Typst and adds new assertions for SVG embedding/redaction/watermark. |
| src/server.zig | Updates help text to reflect Nix-based docs command. |
| src/pandoc.zig | Removes the legacy Pandoc/XeLaTeX PDF engine implementation. |
| src/main.zig | Wires the main build pipeline to the Typst engine and updates error messages/verbose wording. |
| src/config.zig | Stops mapping redact_web onto PDF redaction defaults (fix for #115). |
| README.md | Updates product docs to reflect Typst-based PDF generation and new toolchain details. |
| flake.nix | Removes Pandoc/Chromium/fontconfig plumbing; adds typst + Typst font path wiring for checks/shells. |
| flake.lock | Updates flake inputs consistent with new dependency set. |
| content/guides/writing-policies.md | Updates Mermaid guidance (now rendered in-process via pozeiden). |
| content/guides/live-editing.md | Updates dev command references from devbox to Nix. |
| content/guides/installation.md | Removes template-copy step; bumps referenced PolicyPress version; updates CI size note. |
| content/guides/building-pdfs.md | Updates technical details to the Typst pipeline. |
| CHANGELOG.md | Documents the Typst migration, mermaid rendering change, redact_web fix, and removals. |
| build.zig.zon2json-lock | Updates pinned dependency versions (zigmark, pozeiden). |
| build.zig.zon | Repins zigmark and adds pozeiden dependency; removes template path. |
| build.zig | Removes embedded Eisvogel injection; adds Typst engine module/exe and rewires imports/steps. |
| action.yml | Pins CI shell to the action’s own flake, fixes ref fallback for local action runs, and removes Pandoc template copy step. |
| .puppeteer.json | Removed (no longer needed after dropping Chromium-based mermaid rendering). |
| .github/workflows/preview.yml | Updates userns/sysctl step naming/comments (bubblewrap context). |
| .github/workflows/deploy-docs.yml | Updates userns/sysctl step naming/comments (bubblewrap context). |
| .github/workflows/ci.yml | Runs full tests on macOS (Typst/pozeiden pipeline) and updates bubblewrap commentary. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Never delete a pre-existing .typ path on name collision; retry with a fresh random name instead (error.TypstWorkFileConflict after 16 tries). - Build the output PDF path with std.fs.path.join instead of string concatenation. - underscoresToBlocks: the closing frontmatter delimiter only counts as a whole line (line start + newline/EOF). Previously a '---' inside a frontmatter value ended the protected region early and let redaction corrupt underscores in later keys like last_reviewed. Adds a regression test.
Member
Author
|
All three Copilot findings addressed in b22a0f0:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #58. Closes #115.
What
The PDF pipeline is now: markdown → zigmark (Typst markup, in-process) → pozeiden (mermaid → inline SVG, in-process) →
typst compile. pandoc, texlive/eisvogel-tex, and mermaid-filter (Node + Puppeteer + Chromium) are gone, along with theMERMAID_FILTER_CMD_MMDCsandbox workaround and all fontconfig plumbing. The only external tool left is thetypstbinary.Measured results (same 4-policy demo content, same machine)
#cishell closureParity preserved
Title_-_v1.0.pdf,__Redacted__/__Draft__variants — PDF links for clear and redacted versions should be distinct #97) unchanged; asserted by tests.pdf_colorrule, 6cm logo, last-reviewed), running header "Title vX.Y" + logo, footerOrg © Year | Confidential | page#, TOC depth 3, zebra tables, #F7F7F7 code blocks, #A50000 links, 5-column version history (incl. Revised By).█blocks (fixes the thematic-break artifact from the old POC branches).@/links,{{ org() }},{% mermaid() %},{% admonition() %},{% redact() %}preprocessing untouched (utils.zig).Verification done
nix flake checkgreen (sandboxed test suite incl. typst compile withTYPST_FONT_PATHS);zig build test30/30;zig build e2egreen againststarter/.pdffonts: SourceSans3 + SourceCodePro embedded (DejaVu supplies the█glyph).pdftotext | grep -c 'graph TD'→ 0 (no mermaid fallback code blocks leaked).Dependencies / merge order
bytes.fromBase64does not exist in Typst — the old emit failedtypst compileoutright) and adds the body-onlyrenderTypstWithMermaidAPI. Merge + tagv0.7.2first, then repin here (currently pinned to the PR commit).nix run .#bump -- minor→ v1.4.0 + tag (the action's pinned-shell fix makes merging safe for existing consumers, but the ADO guide already references v1.4.0, so releasing promptly is best).Infra notes
path:${{ github.action_path }}#ci— pinned to the action's own version instead of floating on main (previously, merging this would have broken every consumer running the 1.3.x binary).zig build checkto the full test suite (nix shell .#zig nixpkgs#typst --command zig build test); the apparmor userns sysctl steps stay, but with corrected comments: they turned out to be required by bubblewrap (zig2nix's zig-hook wraps the compiler in bwrap), not just Chromium — removing them broke CI withbwrap: setting up uid map: Permission deniedand they were restored.TYPST_FONT_PATHS(symlinkJoinof source-sans + source-code-pro). Windows users install typst themselves (winget install Typst.Typst); missing fonts fall back to typst's embedded set (cosmetic only).pdfinfoprints a harmless "Suspects object is wrong type" nit about typst's PDF metadata; PDFs open fine everywhere tested.Also fixed here
redact_webno longer drives PDF redaction. It is web-only as documented; an org can now hide content (e.g. phone numbers) on the website while keeping it in PDFs by default. PDF redaction comes solely from--redact/--no-redact/redact_mode(the action always passes an explicit flag, so action consumers are unaffected).Observations (pre-existing, not addressed here)
--webtex; no demo content uses math, so no regression surfaced. If math-in-PDF matters later it needs a zigmark story.Follow-up to file after merge: pre-render web-side mermaid with pozeiden and drop the vendored
static/plugins/mermaidJS.