Parallelize log-archive parsing in unifiedlog_iterator (3-4x speedup)

[cvandeplas]

Summary

Adds multi-threaded processing to the unifiedlog_iterator example tool's log-archive mode,
reducing wall-clock time from ~74s to ~20s on a 66-file/5M-entry logarchive (release build).
No new dependencies are added — the implementation uses only the Rust standard library.

Approach

Profiling with perf revealed that the original bottleneck was split between:

• Unbuffered write syscalls (~25% of time)
• build_log() parsing + string resolution (~18%)
• JSON serialization (~10%)
• Memory allocation overhead (~15%)

The changes address these in two steps:

1. BufWriter: Wraps output in std::io::BufWriter, cutting syscall overhead in half.
2. Per-file parallelism: Worker threads process tracev3 files in parallel. Each worker
   reads a file, iterates chunks, calls build_log(), serializes to JSONL bytes, and sends
   batches through a bounded channel to a dedicated writer thread.

We initially implemented this using rayon and crossbeam-channel, which achieved identical
performance. We then replaced both with standard library equivalents (std::thread::scope,
Mutex<Vec> work queue, std::sync::mpsc::sync_channel) and measured no regression —
keeping the dependency footprint unchanged.

Changes

• Wrap output in BufWriter for batched write syscalls
• New parallel parse_log_archive() using std::thread::scope with N worker threads
• Workers pull files from a shared Mutex<Vec> work queue (natural load balancing)
• Pre-serialized JSONL byte batches flow through sync_channel to a writer thread
• Oversize string resolution deferred to a sequential second pass after all workers complete
• OutputWriter gains write_raw_bytes() for the parallel path and uses Send-compatible types
• Live and SingleFile modes remain sequential and unchanged

Performance (release build, 66 tracev3 files, 5.05M log entries, 177MB)

Variant	Real	User	Sys
Before (single-threaded)	1m14s	51.6s	22.4s
After (parallel, stdlib only)	20.2s	1m35s	22.7s

Correctness

Verified against the single-threaded baseline output:

• Identical entry count: 5,050,912 lines
• Every timestamp matches (verified by multiset comparison — no missing, no extra)
• All non-message fields identical (pid, subsystem, category, time, event_type, etc.)
• Only 6 lines differ: 3 pairs with different HashMap key ordering in protobuf statedump
  messages (non-deterministic by nature, both outputs equally valid)
• Zero entries where one version resolves a message and the other shows <Missing message data>

[jrouaix]

Hello @cvandeplas,
I have an other "performance-oriented" PR in progress here : shindan-io#11.
Actually it's a pretty big kinda full rewrite.
It will allow sysdiagnose parsing about 8x faster.
I targeted only to reduce CPU cost, so it's faster by actually ... doing things faster.
But I wonder if we could then apply parallelism and multiply the gains 🚀

[cvandeplas]

@jrouaix That looks interesting! Would you be able to integrate some of the multiprocessing idea's from this PR and raise a new PR for having those nice things included here?

[jrouaix]

@jrouaix That looks interesting! Would you be able to integrate some of the multiprocessing idea's from this PR and raise a new PR for having those nice things included here?

Well right now I'm just trying to land a clean PR with "only" the ambitious no-alloc-lazy-gothrough parsing
I also try to have full bit for bit same output than the legacy version.

Then I think we (as the community of this lib users) can work some more and polish more.
Perhaps there are some more performance islands we can discover and parallelism certainly is one.

(My use case - SAAS - my goal is to consume less CPU, not "same in //" but we could use parallelism as an option for some environements for instance)