Move from JSON as an intermediary format to something more efficient?

[cjfields]

At this stage we're prototyping, so using simple JSON output for intermediary data storage. We could move to a better more compressed storage format such as Parquet or Avro.

[cjfields]

From Opus 4.7:

Verdict: moderate refactor, low risk

Disruption profile: ~8 output sites + ~5 input sites in src/cli.rs and src/main.rs, no nested-graph data — most payloads are already tabular (derep uniques, dada ASVs, merge-pairs, sequence-table count matrix, taxonomy assignments). That maps cleanly to Parquet columns. The only awkward bits are nested optional fields: DadaOutput.aux (cluster diagnostics) and assign-taxonomy's Vec<Option<String>> ranks + variable-length bootstrap arrays. Both representable as Parquet LIST/STRUCT, just more schema work.

What I'd plan for:

• Add arrow (with parquet feature) or parquet directly — no current dep, ~one extra build minute.
• Define a schema per output type. The tag-dispatch logic in src/misc.rs (dada2_rs_command) maps naturally to a schema registry.
• Replace serde_json::to_writer sites with a writer abstraction; readers (load_derep_samples in main.rs, make-sequence-table dispatch in sequence_table.rs, assign-species loader) need parallel paths.
• Keep - stdin support: Parquet is seekable-by-design and unfriendly to stdin pipes. You'd need to buffer to a temp file or stick with Arrow IPC for piping. This is the one real friction point given your recent stdin work.

Suggested approach when you do it: add --format parquet|json and ship both for a release or two rather than swapping. Phase 1 covers the linear pipeline (derep → dada → merge-pairs → sequence-table); phase 2 handles taxonomy + aux diagnostics. tax-to-tsv becomes trivial (Parquet → Arrow → CSV is one-liner).

External cost: the examples/import_mia.R path would need arrow::read_parquet instead of jsonlite::fromJSON — trivial swap, but a breaking change for any existing users.

Rough estimate: 2–3 focused days for the main pipeline, another 1–2 for taxonomy + aux. Reversible if you keep JSON behind a flag.

[cjfields]

Measured: qual_sum:[u32] vs mean_quality:[f64] on disk (in-memory counterpart shipped in #23)

Now that derep/sample JSON stores integer per-position Phred sums (qual_sum:[u32], #23 / PR #24) instead of the float mean (mean_quality:[f64]), here's the on-disk size delta. Faithful comparison — both written by the Rust serializer, identical flags, local example data:

platform	format	old raw	new raw	Δ raw	old gz	new gz	Δ gz
MiSeq (896 uniq)	compact	1.42 MB	0.93 MB	−34.4%	178 KB	159 KB	−10.4%
MiSeq	pretty	3.47 MB	2.98 MB	−14.0%	205 KB	199 KB	−2.7%
PacBio (259 uniq)	compact	2.41 MB	1.59 MB	−34.2%	236 KB	210 KB	−10.9%
PacBio	pretty	5.86 MB	5.04 MB	−14.1%	274 KB	264 KB	−3.6%

Takeaways

• Raw compact ~34% smaller, consistent across platforms (integer sums like 6962 vs float text like 33.960975…).
• Gzipped compact ~10–11% smaller — gzip already squeezes the repetitive float text, so the field win narrows but holds.
• Pretty dilutes it (−14% raw / ~3% gz): indentation whitespace dominates raw size and compresses to nothing. The honest storage comparison is compact + gzip.
• Bonus: integer sums are exact — no float-text precision baggage; the disk form is now lossless by construction.

Next on this issue

• Make the storage path compact + gzip (the read path already handles .gz transparently via read_all_maybe_gz). Plain gzip looks right over bgzf since these files are read whole (no random-access need).
• After quals, sequences are the next on-disk lever, especially post-gzip (PacBio ~1500 bp ACGT/unique). Tracking separately as a seq-encoding follow-up.