Problem
Adding a column to ai_gateway_messages (agent_id v5, parent_thread_id, planned session_id)
bumps SCHEMA_VERSION, which makes the existing cache incompatible with the new schema. To actually
see the new column you have to recreate the cache and re-run hyp backfill — the old cache doesn't
update in place.
Why it should be avoidable
Most of the machinery already exists:
- icebird reads files missing a column as
null (read.js:382) — old data stays readable under a
newer schema.
- icebird's commit layer can apply schema evolution (
add-schema + set-current-schema).
store.js already computes mergeFieldIdsFromTable (preserving/assigning field ids) — but
currently discards it instead of passing it to the append.
The blocker is that icebird's schema-evolution path isn't reachable from here yet. See hyparam/icebird#25
Solution
Additive (nullable) column changes should evolve the cache schema in place — the new column shows up
without a recreate, and existing rows just read it as null. A full recreate stays necessary only for
genuinely breaking changes (e.g. partition-key moves, tracked separately).
Acceptance
- Add a nullable column, restart daemon → new column is queryable, old rows read it as
null, new
rows populate it, no recreate and no data loss.
- Backfill works against the evolved schema.
Problem
Adding a column to
ai_gateway_messages(agent_idv5,parent_thread_id, plannedsession_id)bumps
SCHEMA_VERSION, which makes the existing cache incompatible with the new schema. To actuallysee the new column you have to recreate the cache and re-run
hyp backfill— the old cache doesn'tupdate in place.
Why it should be avoidable
Most of the machinery already exists:
null(read.js:382) — old data stays readable under anewer schema.
add-schema+set-current-schema).store.jsalready computesmergeFieldIdsFromTable(preserving/assigning field ids) — butcurrently discards it instead of passing it to the append.
The blocker is that icebird's schema-evolution path isn't reachable from here yet. See hyparam/icebird#25
Solution
Additive (nullable) column changes should evolve the cache schema in place — the new column shows up
without a recreate, and existing rows just read it as
null. A full recreate stays necessary only forgenuinely breaking changes (e.g. partition-key moves, tracked separately).
Acceptance
null, newrows populate it, no recreate and no data loss.