Skip to content

Additive schema changes require a recreate + backfill to take effect #102

Description

@bgmcmullen

Problem

Adding a column to ai_gateway_messages (agent_id v5, parent_thread_id, planned session_id)
bumps SCHEMA_VERSION, which makes the existing cache incompatible with the new schema. To actually
see the new column you have to recreate the cache and re-run hyp backfill — the old cache doesn't
update in place.

Why it should be avoidable

Most of the machinery already exists:

  • icebird reads files missing a column as null (read.js:382) — old data stays readable under a
    newer schema.
  • icebird's commit layer can apply schema evolution (add-schema + set-current-schema).
  • store.js already computes mergeFieldIdsFromTable (preserving/assigning field ids) — but
    currently discards it instead of passing it to the append.

The blocker is that icebird's schema-evolution path isn't reachable from here yet. See hyparam/icebird#25

Solution

Additive (nullable) column changes should evolve the cache schema in place — the new column shows up
without a recreate, and existing rows just read it as null. A full recreate stays necessary only for
genuinely breaking changes (e.g. partition-key moves, tracked separately).

Acceptance

  • Add a nullable column, restart daemon → new column is queryable, old rows read it as null, new
    rows populate it, no recreate and no data loss.
  • Backfill works against the evolved schema.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions