Skip to content

Plugin refactor + classifier support + idempotency#10

Open
joefutrelle wants to merge 5 commits into
mainfrom
plugin-refactor
Open

Plugin refactor + classifier support + idempotency#10
joefutrelle wants to merge 5 commits into
mainfrom
plugin-refactor

Conversation

@joefutrelle

Copy link
Copy Markdown
Collaborator

Plugin refactor + classifier support + idempotency

Refactors the plugin system to use dependency injection so the core never imports plugins at load time. Plugins are now generic and parameterized; instrument-specific presets (IFCB features, IFCB CNN classification) move under improv.plugins.ifcb and pin a kind/index_table onto a generic plugin. Adds machine classification support end-to-end and makes provenance/index writes idempotent.

Highlights

  • Plugin DI — plugins register at runtime instead of being imported by the core; store/indexes.py removed, store/dedup.py added.
  • Classification — new classification plugin, router, and schema (class-scores schema change); taxonomy registration/lookup plus stateless and decoded read paths; 002_classifier_taxonomy migration.
  • Idempotency — append-only writes deduplicated at read time; provenance records keyed by (image_id, kind, source, data_hash) using RFC 8785 (JCS) canonical hashing (new rfc8785 dep). Backend-agnostic (VAST DB / DuckDB+Parquet).
  • Small-batch POSTPOST /images/provenance/batch lets low-volume producers write provenance over REST without depending on the columnar store (one instrument per batch).
  • ClientImprovClient gains taxonomy registration.
  • Tests — added coverage for classification, provenance, client, plugins, store dedup (~590 net test lines).
  • Docs — README updated (plugin model, classifier API, idempotency, client contract on event-time timestamp).

Review note: client contract — timestamp must be event time, not wall-clock-at-send, or retries won't dedup.

@joefutrelle joefutrelle self-assigned this Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant