Opt-in lossy DCT compression profile (tag 4)

[MariusYvard]

Hey Yusuf, thanks a lot for the warm welcome, this was a fun one to build. As you suggested in #38, this is a separate opt-in lossy profile rather than a 5th competitor in the tag race, on tag 4 (I left 5+ free for later).

What it is. A userland lossy profile for pixel mode: YUV 4:2:0, an 8x8 integer DCT with a perceptual quantization matrix, per-block skip, luma block motion compensation, zigzag plus run-length entropy, DC prediction, deadzone quantization and a rate-distortion skip. Roughly 4 to 5x smaller than the current lossy path at matched quality (for example a 480x272 hero clip around 1.8 MB where the current codec is about 8 MB for the same look), and several times smaller than lossless. All in userland: no <video>, no native decoder, it decodes on a canvas.

See it in action.

python static_player/compiler.py input.mp4 --pixel --profile --qf 70 --cols 480

This produces a normal .ascf you can play in the Studio or the static player, since the decoder outputs a standard BGR pixel framebuffer. Higher --qf means better quality and larger files.

Additive, nothing on the current path changes.

• New per-frame tag 4. Tags 0/1/2 and the reserved 3 are untouched, so existing .ascf files and the Studio play byte-for-byte identically. I verified this: the updated codec.js gives output identical to the old one on legacy streams.
• No .ascf header change. Keyframes self-describe (QF, cols, rows), so the 14-byte header and reader.js/app.js are untouched.
• Character plane stays exact in ASCII modes (the profile is pixel only in v1).
• Deterministic integer DCT and integer YUV to BGR, so codec.py and codec.js stay bit-exact. The experiments/ test mirrors gen_vectors.py to check_vectors.js: encode in Python, decode through the shipped codec.js, compare every frame.
• Small bonus: codec.js now degrades gracefully on an unknown tag (repeats the last frame instead of throwing), so future tags do not hard-crash old players.

On the live streaming side you mentioned. The encoder lives in codec.py, the same place stream_server.py and the compiler both call, so wiring the profile into the live WebSocket path is a small, natural follow-up. Happy to send that next once you are happy with the static side, and to align the profile signaling and any details with you, as you offered.

Files. codec.py (adds ProfileEncoder, encode_frame unchanged), codec.js (makeDecoder routes tag 4), static_player/compiler.py (--profile --qf), experiments/profile_vectors.py + check_profile.js, PROFILE.md.

[YusufB5]

Hey Marius,
very impressive work!
quick test notes before I call it a night.

Tested on a 2m47s clip at --cols 250 and --cols 450:

250 cols lossless: 293 MB → 250 cols profile: 32 MB (~9x smaller)
450 cols profile: 81 MB — noticeably closer to lossless quality, still some blur but getting very close
Compiler time at 450 cols: ~15 minutes
JS decoder stutters at 450 cols in the Studio (mid-playback), likely GC pressure from new Int32Array(64) allocations inside the block loop
Since the profile is fully additive, it looks great, but before we merge this, maybe we can try to solve this JS decoder recovery/stuttering issue in the Studio. I'll test it in more detail tomorrow.

[MariusYvard]

Thanks for testing it that thoroughly. You were right down to the new Int32Array(64).

The decoder allocated per 8x8 block (RLE scratch, dequantised coefficients and two more inside the IDCT) plus a fresh set of plane buffers every frame. Fixed in 479d9aa: scratch is hoisted and reused, plane buffers ping-pong, and DC-only blocks skip the IDCT entirely (the first row of the integer basis is constant, so the transform collapses to a flat value, checked exhaustively against the full IDCT).

Allocations per decoded frame at 448x256, roughly your 450-col case: keyframe 10764 → 12, inter frame 1624 → 4. My clip is mostly static so many blocks skip; on real footage the inter number sits closer to the keyframe one. Decode time roughly halves here (~5 → ~3 ms/frame, JIT-noisy), GC events drop about 4x, and decoded frames are byte-identical to the previous version.

One caveat on my own fix: the hoisted scratch is module-level, sound only because the block loop never awaits, so two decoders cannot interleave inside it. check_profile.js now drives two decoders concurrently to guard that.

If "recovery" meant playback never catches back up after a hitch rather than the hitch itself, that lives in the Studio's playback loop rather than the codec. Tell me what you see.

Compiler time: the encoder ran transform, quantisation, reconstruction and RLE one block at a time in Python. Batched over the whole plane now, about 2x faster (84 → 41 ms/frame), bitstream byte-identical. That halves your 15 minutes rather than fixing them. Beyond that the answer is parallel GOPs (keyframes every 48 frames, so chunks are independent), but that reshapes your compiler's main loop, so your call.

On the blur: I nearly told you to raise --qf, then measured it. On a hard clip (mandelbrot) QF85 costs +59% size for +0.9 dB and QF90 +103% for +1.25 dB, so your 81 MB would become ~129 MB. Poor trade on detailed content. Your footage will do better than that worst case, but I did not want to give you a number I had not measured.

Suite green, concurrency check included.