Design: generalize Content_Parser to support multiple standard.site content formats

[kraftbj]

Summary

The current Content_Parser interface and atmosphere_content_parser filter assume a single parser produces a single content record per post. site.standard.document is a union field that accepts multiple lexicon types — long-term we likely want to support more than just at.markpub.markdown (e.g. HTML, plain text, or whatever standard.site adds next), and possibly emit more than one representation per post.

This issue tracks design-and-discuss before a second parser implementation lands and forces a breaking interface change.

Current Shape

• includes/content-parser/interface-content-parser.php — parse() returns a single ?array shaped for one lexicon type. get_type() returns one NSID.
• includes/class-atmosphere.php:50 — registers a single Markpub parser via atmosphere_content_parser filter.
• includes/transformer/class-document.php:172 — fetches one parser, calls parse(), sets content to the single result.

Constraints this imposes:

1. Only one parser can win the filter (last-wins). Two third-party parsers can't coexist.
2. Document can't pick a parser based on the post (e.g. classic content → HTML parser; block content → Markpub).
3. No path to emit multiple representations even though the union allows it.

Discussion Points

Sketching a few directions — not picking one yet:

A. Registry of parsers by NSID
Replace the single-parser filter with a registry. Parsers register against their get_type(). Document picks the preferred one per post (config or filter hook), or iterates and emits all that produce non-null output.

B. Per-NSID filter pattern
Keep the filter shape but namespace it: atmosphere_content_parser_at_markpub_markdown, atmosphere_content_parser_at_html (or similar). Document iterates across known NSIDs.

C. Parser produces canonical IR; Document projects
Parser returns a structured intermediate (e.g. block tree) and a list of supported output types. Document negotiates and projects. Most flexible, biggest refactor.

Open Questions

• Does standard.site already define more than one content format today, or is this purely future-proofing?
• Should the post author be able to override which format publishes (per-post meta), or is this a site-wide config?
• Does the union field accept an array (multiple representations) or strictly one of N?
• Where does language/locale or accessibility metadata live — per-parser or shared?

Out of Scope

• Implementing a second parser. This issue is interface-design only; the next parser PR consumes whatever shape lands here.
• Changing the Markpub parser's behavior. Markpub keeps producing at.markpub.markdown regardless of the chosen registry shape.

Context

Raised by @pfefferle in #9 (comment) ("we should maybe find a more generic way, to also transform into the other standard.site content formats").

Markpub (#9) is shipping with the current single-parser interface; this issue captures the follow-up so the design is settled before a second parser implementation forces our hand.

[pfefferle]

Concrete formats: research notes

Pulled the actual lexicons for the four candidate content types so the interface design is grounded in real shape data rather than guesses. NSIDs, key constraints, and structural commitments below.

1. at.markpub.markdown

Source: github.com/AramZS/markpub.at · status: not yet published, taking feedback (March 2026).

The lexicon wraps a markdown string with optional metadata. Required field is text (ref to at.markpub.text); optional flavor (gfm | commonmark), renderingRules, extensions, frontMatter.

The interesting field is at.markpub.text itself, which is structured to allow markdown to live inline or as a blob:

markdown:  // required string, raw markdown
textBlob:  // optional blob ref (text/*, max 1MB) — overrides markdown if present
facets:    // optional union array — open union, can include
           //   at.markpub.facets.baseFormatting (#strong, #header, #idify)
           //   pub.leaflet.richtext.facet
           //   app.bsky.richtext.facet
lenses:    // optional union array — declares which facet lexicons the producer wants honored

Markpub explicitly designs its facets to be cross-compatible with Leaflet's and Bluesky's richtext facets, and provides lenses so a producer can declare "treat my facets as equivalent to lexicon X."

Input shape: raw markdown string (post-the_content → markdown conversion, or block-tree → markdown). Output: one record per post.

---

2. pub.leaflet.content

Source: hyperlink-academy/leaflet · lexicons/pub/leaflet/content.json · standard.site member.

{
  "id": "pub.leaflet.content",
  "defs": { "main": {
    "type": "object",
    "required": ["pages"],
    "properties": { "pages": { "type": "array", "items": {
      "type": "union",
      "refs": ["pub.leaflet.pages.linearDocument", "pub.leaflet.pages.canvas"]
    }}}
  }}
}

Each page (currently linearDocument or canvas) holds an array of blocks. Block union refs:

iframe, text, blockquote, header, image, unorderedList, orderedList, website, math, code, horizontalRule, bskyPost, page (nested), poll, button

Plus per-block alignment and a separate pub.leaflet.richtext.facet system. The standalone pub.leaflet.document record adds title/description/coverImage (image blob, max 1MB)/tags/publishedAt/theme.

Input shape: WordPress block tree → leaflet block tree (1:1 mapping for paragraph/heading/list/image/quote; degraded for blocks with no leaflet equivalent). Output: one record per post; multi-page possible but not required. Blob coupling: images need PDS upload before content publishes.

---

3. blog.pckt.content

Source: live on pds.pckt.cafe at did:plc:revjuqmkvrw6fnkxppqtszpv/com.atproto.lexicon.schema/blog.pckt.content · standard.site member.

{
  "id": "blog.pckt.content",
  "defs": { "main": {
    "type": "object",
    "description": "Hybrid content storage: inline for small content (≤20KB), blob for large content (>20KB)",
    "properties": {
      "blob":       { "type": "blob", "description": "External JSON blob (extended mode, >20KB)" },
      "items":      { "type": "array", "items": { "type": "union", "closed": false, "refs": [] },
                      "description": "Inline blocks (≤20KB)" },
      "references": { "type": "array", "items": { "type": "blob" },
                      "description": "Required in extended mode — full blob refs to prevent GC" }
    }
  }}
}

Two operational modes selected by content size. items and blob are mutually exclusive in practice. references is the GC anchor: in extended mode, image/file blobs referenced inside the JSON blob need their own top-level refs in the record so the PDS doesn't garbage-collect them.

Block lexicons mined directly from the PDS: text, heading, image, codeBlock, blockquote, bulletList, orderedList, listItem, hardBreak, horizontalRule, table, tableRow, tableCell, tableHeader, taskList, taskItem, mention, blueskyEmbed, gallery, iframe, website. Plus its own blog.pckt.richtext.facet (link, didMention, atMention, code, highlight, underline, strikethrough, id, bold, italic).

Standalone records: blog.pckt.publication, blog.pckt.document, blog.pckt.gallery, blog.pckt.theme.

Input shape: WordPress blocks → pckt blocks (1:1 for most), then size-gated to inline or blob. Output: one record, but with blob-upload-coordination for both extended-mode payload and embedded images. Blob coupling: highest of the four.

---

4. org.wordpress.html

Source: docs/org.wordpress.html.md · this repo.

{
  "id": "org.wordpress.html",
  "defs": { "main": {
    "type": "object",
    "required": ["html"],
    "properties": { "html": { "type": "string", "maxGraphemes": 100000 } }
  }}
}

Single field. Output of the_content filter, with shortcodes resolved and media URLs absolute. Analogous to RSS content:encoded.

Input shape: rendered HTML. Output: one record per post. Blob coupling: none (image URLs stay external).

---

Pattern analysis

The four formats split into two complexity tiers, with implications for the parser contract:

	Tier 1: flat string	Tier 2: structured block tree
Members	at.markpub.markdown, org.wordpress.html	pub.leaflet.content, blog.pckt.content
Input transformation	post_content → markdown / HTML	parse_blocks() → block tree → block tree
Output cardinality	one object	one object
Image handling	URL-only	blob upload required
Facets	optional (markpub) / none (HTML)	own facet system per lexicon
Mode-switching	none	pckt's inline-vs-blob ≤20KB threshold
Foreign-namespace risk	medium	high (more shape surface to track)

Things that recur across all four:

• Open-union content field accepts any $type. Coexistence is the norm — three of the four are foreign-namespace lexicons whose schemas can drift independently.
• Lexicon-internal facets diverge. Markpub designed lenses exactly for this. We will inevitably re-map between facet vocabularies; that work belongs in the parser, not the Document transformer.
• Most blocks (text/heading/image/list/quote/code) translate 1:1 across tier-2 formats. A shared block-tree IR genuinely covers a lot of ground.

Things that diverge in load-bearing ways:

• Blob-upload coordination is not a parser concern in tier 1; it is the dominant concern in tier 2. A parser interface that just returns an array can't express "I need these N blobs uploaded first, here are their content hashes."
• Pckt's mode switch is producer-side: the parser must measure its own JSON output size, decide inline vs. extended, and produce the right references[] list. That is not modelable as "return an object" — it requires a small state machine.
• Cardinality might evolve. Today every format produces one record per post. Markpub's blob-overflow path and pckt's extended mode hint that "publish multiple representations" could become real (e.g. emit both at.markpub.markdown and org.wordpress.html for the same post so markdown-aware readers and HTML-aware readers each get a native experience).

Implications for the parser interface

Mapping back to options A / B / C in the issue:

• Option A (registry by NSID). Necessary baseline. Three of the four formats are tier 2 and need to coexist with the others; last-wins filter is wrong for them.
• Option B (per-NSID filter). Solves coexistence but doesn't solve the bigger tier-2 problems (blob coordination, mode-switching, facet remapping). I would skip B — it costs the same as A in API churn but offers less.
• Option C (canonical IR + projector). Has real load-bearing benefits in tier 2 — block translation across leaflet ↔ pckt is most of the work, and a shared block IR pays for itself. But: (a) tier 1 doesn't benefit, and (b) the IR has to model facets, blobs, and pckt's size-mode threshold, which is a substantial refactor.

The shape that fits the data is closer to A + a blob-upload contract:

• Parsers register against get_type() (option A).
• The interface adds a way to declare blob dependencies before record finalization, e.g. a prepare() step that returns the list of blobs needing upload, followed by finalize( $uploaded_blob_refs ) that returns the content object with refs filled in. Tier-1 parsers implement these as no-ops.
• Document iterates over registered parsers, calls prepare() → uploads blobs once, deduped → calls finalize() per parser → emits all non-null content objects (collapsed to one if content is single-valued, kept as a list if it ever becomes multi-valued).
• Facet remapping stays a parser-internal concern, with lenses-style cross-vocabulary declarations encouraged (markpub already documents this pattern).

Concretely worth deciding before the second parser PR:

1. Is site.standard.document.content strictly singular today, and likely to grow into a list? That changes whether we need to support multi-emit at the Document layer or only the parser layer.
2. Does the parser own blob uploads, or does Document? I'd argue the parser owns what needs uploading (only the parser knows which images live where in its block tree), and Document owns the upload itself (single dedup point across parsers).
3. What's the policy for sites running both an HTML parser and a tier-2 parser? Implicitly: HTML always succeeds, tier-2 may produce something cleaner; do we publish both, or just the cleanest?

Sources: markpub.at · hyperlink-academy/leaflet/lexicons · pds.pckt.cafe (live) · docs/org.wordpress.html.md

[pfefferle]

Design recommendation: registry + concrete base

Building on the lexicon research above, here's a concrete shape that picks one direction and absorbs the boilerplate every parser would otherwise reinvent.

Two layers

Atmosphere\Content\Field_Transformer    ← abstract interface (registry contract)
└── Atmosphere\Content\Parser_Base      ← abstract base with helpers (inherit this)
    ├── Wordpress_Html_Content          ← tier-1, ≈10 LOC
    ├── Markpub_Content                 ← tier-1, ≈30 LOC
    ├── Leaflet_Content                 ← tier-2, override applies_to/required_blobs
    └── Pckt_Content                    ← tier-2, override applies_to/required_blobs

The interface is the minimum contract the registry consumes. The base is what plugin authors actually extend — WordPress-coupling lives there so parsers don't each re-derive "how do I get the block tree from a post?"

1. Interface

namespace Atmosphere\Content;

abstract class Field_Transformer {
    abstract public function get_type(): string;                       // NSID
    abstract public function applies_to( \WP_Post $post ): bool;
    abstract public function required_blobs( \WP_Post $post ): array;  // attachment IDs
    abstract public function transform( \WP_Post $post, array $blob_refs ): array;
}

Strictly WP_Post-typed. Comments emit app.bsky.feed.post replies only; the reply record has no content union, so they never enter this pipeline.

2. Concrete base

abstract class Parser_Base extends Field_Transformer {
    // Sane defaults for tier-1 parsers.
    public function applies_to( \WP_Post $post ): bool { return true; }
    public function required_blobs( \WP_Post $post ): array { return array(); }

    /* Helpers, cached per post-id per request: */
    final protected function get_blocks( \WP_Post $post ): array;
    final protected function has_blocks( \WP_Post $post ): bool;
    final protected function get_rendered_html( \WP_Post $post ): string;
    final protected function get_plain_text( \WP_Post $post ): string;
    final protected function collect_attachments( \WP_Post $post, array $types = array( 'image' ) ): array;
    final protected function build_image_ref( int $attachment_id, array $blob_refs ): ?array;
    final protected function truncate_graphemes( string $text, int $limit, string $marker = '…' ): string;
    final protected function get_langs( \WP_Post $post ): array;
    final protected function iso8601( string $gmt_datetime ): string;
}

Most of these helpers exist in scattered places today (includes/functions.php, Transformer\Base, Transformer\Post); the base just consolidates them with a parser-friendly API.

3. What concrete parsers reduce to

Tier-1 (HTML):

class Wordpress_Html_Content extends Parser_Base {
    public function get_type(): string {
        return 'org.wordpress.html';
    }
    public function transform( \WP_Post $post, array $blob_refs ): array {
        return array(
            '$type' => $this->get_type(),
            'html'  => $this->truncate_graphemes( $this->get_rendered_html( $post ), 100000 ),
        );
    }
}

Tier-2 (Leaflet) overrides two methods plus the block-translation work:

class Leaflet_Content extends Parser_Base {
    public function get_type(): string { return 'pub.leaflet.content'; }
    public function applies_to( \WP_Post $post ): bool { return $this->has_blocks( $post ); }
    public function required_blobs( \WP_Post $post ): array {
        return $this->collect_attachments( $post, array( 'image' ) );
    }
    public function transform( \WP_Post $post, array $blob_refs ): array {
        return array(
            '$type' => $this->get_type(),
            'pages' => array( $this->build_linear_document_page( $this->get_blocks( $post ), $blob_refs ) ),
        );
    }
    // build_linear_document_page() is the only domain logic left.
}

Pckt's inline-vs-extended mode switch stays parser-local inside transform() — exactly where it belongs.

4. Registry + Document projection

Content\Parsers::register( new Wordpress_Html_Content() );

// Inside Transformer\Document:
$applicable = array_filter( Content\Parsers::all(), fn( $p ) => $p->applies_to( $post ) );
$blobs      = Publisher::upload_blobs( merge_dedup( array_map( fn( $p ) => $p->required_blobs( $post ), $applicable ) ) );
$contents   = array_map( fn( $p ) => $p->transform( $post, $blobs ), $applicable );
$record['content'] = self::project_content( $contents );  // collapse-or-array helper

Two passes (collect-blobs / transform-with-refs) so one upload covers blobs shared between parsers (e.g., the same WP attachment used by a Leaflet image block and a pckt image block).

Why this beats A / B / C

• vs. A (plain registry): same coexistence properties, but A doesn't carry the blob contract or the two-phase lifecycle that tier-2 parsers (Leaflet, pckt) actually need. Plain A leaves them stuck.
• vs. B (per-NSID filter): same coexistence cost as A, less ergonomic, no blob story.
• vs. C (canonical IR + projector): solves a problem we don't have. Tier-1 parsers don't benefit from a shared block IR; tier-2 might if we ever want auto-rebuild between Leaflet ↔ pckt — a registry today doesn't preclude an IR tomorrow.

What the base buys us

Concern	Without base	With base
Block tree access	Each parser parses + caches per-instance	One cached call per post-id, shared
Image enumeration	Each parser walks blocks, sometimes inconsistently	collect_attachments() is the canonical walk
Rendered HTML	Each parser remembers apply_filters('the_content', ...) with setup_postdata()	One helper, edge cases solved once
Grapheme truncation	Each parser hand-rolls UTF-8/UTF-16 handling	Single helper honoring lexicon maxGraphemes
Locale → langs	Parsers re-derive the BCP-47 list + filter chain	Single get_langs() helper
Blob ref shape	Each parser builds the {$type: blob, ref, mimeType, size} shape from upload results	build_image_ref() returns canonical shape

Migration

• One filter caller today (Markpub from Add Markpub content parser #9). Wrap-and-deprecate atmosphere_content_parser as a back-compat shim that registers a wrapping Field_Transformer, drop after one release.
• New API surface: Field_Transformer abstract + Parser_Base + Content\Parsers registry. Worth the surface area because every future parser uses all three.
• Blob dedup needs a content-hash key on Publisher::upload_blobs(). Doesn't exist yet but is inevitable once the second blob-using parser ships — this design forces it now rather than rediscovering it later.

Open questions

1. Is site.standard.document.content strictly singular today, and is it expected to grow into a list? Determines whether project_content()'s array branch ever runs.
2. Does atproto have an idiomatic blob-upload helper we should adopt, or do we own dedup/upload code?
3. Where does language/locale negotiation live — inside each parser or as a Document-level filter that decides which parsers to run? My read: parser-internal.

Out of scope (explicit)

• Comments. Reply records have no content union; never enter this pipeline.
• site.standard.publication content fields. Different transformer, different concerns.