fix(indexjobs): deleting a catalog spec no longer strands index jobs in permanent Degraded

[cjimti]

Closes #998

Problem

Deleting an API catalog spec while its index job was queued or running orphaned that job: LoadItems failed with catalogSource: get spec: catalog: not found, the job landed in an open failure, and the api_catalog index kind reported Degraded in the portal permanently, even at 100% vectors indexed. The failure could never self-clear (no future job can carry the deleted spec's key), retention never removes open failures, and the only exit was a manual operator dismiss. The surfaced error also named the wrong object: the catalog was present and healthy the whole time; it was the spec that was gone.

What this PR does

Deletes unwind their queue residue. DELETE .../specs/{spec} cancels the spec's pending index jobs and resolves its open failures as part of the delete; DELETE /api-catalogs/{id} does the same for every spec the cascade removes, matched by the encoded source_id prefix (starts_with, so a catalog id containing a LIKE metacharacter cannot over-match) with no pre-delete spec listing. Both cleanups are best-effort and run under context.WithoutCancel: the row delete has already committed, so a client disconnect cannot strand them.

The framework gains a generalized source-gone contract. A Source.LoadItems error wrapping the new indexjobs.ErrSourceGone sentinel tells the worker the unit was deleted on purpose, not that loading failed: the worker clears the unit's vectors, completes the job, and resolves any open failures for the key, instead of recording a terminal failure nothing can ever supersede. catalogSource maps catalog.ErrNotFound to the sentinel. This closes the same hole for any future source kind, not just api_catalog; the five existing data consumers (memory, prompts, portal assets/collections/knowledge-pages) already return empty items for missing rows and were verified unaffected.

The delete/worker race is closed. If a spec is deleted while its job's final attempt is mid-flight, the delete-side cancel finds nothing to drop (the job is running) and nothing to resolve (no failure exists yet), and the attempt's terminal failure would recreate the permanent residue. The worker now re-probes the source before pinning any exhausted-attempts failure and routes a gone unit to resolution. Earlier attempts need no probe: their retry re-enters LoadItems, which surfaces the sentinel itself.

Spec-missing errors name the spec. catalog.GetSpec (Postgres and memory stores) returns spec "A" not found in catalog "c1": ... wrapping ErrNotFound, so errors.Is callers are unchanged and the message identifies what is actually missing.

API surface changes

• indexjobs: new ErrSourceGone sentinel, new Store.CancelPending(key) (deletes a unit's pending rows; running rows are left to the worker's source-gone path), updated Source.LoadItems contract docs.
• catalogindex: Store gains Cancel(key) and CancelCatalog(catalogID); new EnqueueBestEffort / CancelBestEffort / CancelCatalogBestEffort hooks own the log-and-continue semantics the admin handler previously inlined.
• pkg/admin: the EmbedJobsStore interface is removed; it had become a verbatim duplicate of catalogindex.Store, which Deps.EmbedJobs now uses directly. The enqueue hook moved to catalogindex with the new cancel hooks, keeping pkg/admin within its package LOC budget.
• testdata/allowed_internal_imports.txt: one new edge, catalogindex -> internal/logsan, for sanitizing operator-supplied catalog/spec names in the hooks' log lines (the same sanitization the admin handler applied to these values before the move).

Tests

• Real-Postgres integration test drives the actual admin delete handlers against the real indexjobs store (TestCatalogSpecDelete_ClearsIndexResidue_RealDB): a pending job is gone after spec delete, an already-failed job's open failure is resolved by spec delete, and a catalog delete clears both residue shapes across its specs. Each subtest asserts zero pending/running rows, zero open failures, and UnresolvedFailures == 0 for the kind.
• A second integration test (TestIndexJobs_SourceGoneSelfHeals_RealDB) runs the fully assembled queue (indexqueue.New: real catalogSource, sink, worker, Postgres store) against an orphaned job whose spec was deleted without any handler-side cancel: the job completes as source-gone and its completion resolves the unit's earlier open failure.
• Unit coverage for every new function: worker source-gone resolution (including completion despite a sink error and the exhausted-attempts probe), CancelPending and CancelCatalog SQL, the AdminStore cancel translation, the best-effort hooks, and the ErrSourceGone mapping boundary (a store read error must NOT be treated as gone).
• Both integration tests use the shared internal/testdb harness, whose wait strategy now requires the mapped port to be reachable in addition to the second "ready" log line; the log line alone raced under parallel container load and could fail ConnectionString with port "5432/tcp" not found.

[codecov]

Codecov Report

❌ Patch coverage is 92.30769% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.25%. Comparing base (191c6e4) to head (8c5e83a).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/indexjobs/worker.go	81.81%	3 Missing and 1 partial ⚠️
pkg/indexjobs/store_postgres.go	84.61%	1 Missing and 1 partial ⚠️
pkg/admin/catalog_handler.go	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1017      +/-   ##
==========================================
+ Coverage   89.23%   89.25%   +0.01%     
==========================================
  Files         456      457       +1     
  Lines       50418    50487      +69     
==========================================
+ Hits        44993    45061      +68     
- Misses       3611     3612       +1     
  Partials     1814     1814              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.