Skip embedding tasks and Qdrant writes for unchanged content files

[mbertrand]

What are the relevant tickets?

Closes mitodl/hq#12454 (companion to mitodl/hq#12453 mechanics and mitodl/hq#12452 checksum).

Description (What does it do?)

When a course is re-ingested, skip re-embedding content files that are already up to date in Qdrant.

Every embedded file stores its content checksum in its Qdrant payload. Before doing any expensive work, embed_run_content_files now fetches those stored checksums in a few batched requests and compares them (plus metadata fields like title, description, and summary) against the database:

• File missing from Qdrant, or checksum differs → re-embed it.
• Checksum matches but metadata changed (edited title, new AI summary) → just rewrite its Qdrant payload, no re-embedding.
• Everything matches → skip the file. A fully unchanged course dispatches no embedding tasks — only the cheap pre-pass runs.

Because the check reads Qdrant rather than the database, it recovers from failures automatically: if an embedding task dies partway (deploy, autoscaling), the unfinished files still look stale on the next ingest and get retried, while completed files are skipped. Nothing is silently lost.

This replaces the earlier version of this branch, which detected changes by comparing database state before/after the load. As review pointed out, that couldn't recover from a failed embed: the database already said "done," so the file would never be retried.

Also in this PR:

• The per-file Qdrant lookups inside the embedding pipeline are batched (one request per batch instead of ~two per file). This covers backfills and other callers too.
• The content_files_loaded hook gains removed_unpublished, so the remove-unpublished-files task only runs when a file was actually unpublished.
• embed_run_content_files only embeds published files.

Numbers (1,770-file course, unchanged re-ingest): 31s of serialization and Qdrant writes on main → 0.09s and zero tasks on this branch. With 10% of files changed: 18 tasks instead of 177. A brand-new course does the same embedding work as main with fewer Qdrant calls. The branch never does more work than main in any scenario.

Known limits:

• Staleness is detected from each file's first chunk in Qdrant, so a file whose embedding died mid-file (first chunk written, rest lost) looks complete — same as main.
• Run-level payload fields (run_title, course_number, offered_by) refresh when a file changes or via the embedding backfill — roughly the same freshness as main, which only refreshed them when the course archive changed.
• The first ingest after deploy may rewrite payloads for older points missing newer fields (e.g. summary) — payload writes only, no re-embedding.
• Fields compared by the pre-pass must be exact serializer pass-throughs of ContentFile columns; a test fails if someone adds a transformed field (which would re-flag every file on every load).

How can this be tested?

Use run course-v1:MITxT+15.356.1x+3T2025 (300 content files, each with content and a checksum). If it's not in your local DB yet, ingest it first:

docker compose run --rm web python manage.py backpopulate_mitxonline_data
docker compose run --rm web python manage.py backpopulate_mitxonline_files --resource-ids <id of the course-v1:MITxT+15.356.1x course>

Queue the task for the run's pk and watch the pre-pass ratio in the celery logs:

from vector_search.tasks import embed_run_content_files
embed_run_content_files.delay(<RUN_PK>)

docker compose logs celery --since 5m | grep 'files need embedding'

Expected sequence (verified on a 34-file run; the pre-pass on the 300-file run above logs 210 of 300 against a partially stale local Qdrant, matching step 1):

1. First call (run not yet fully in Qdrant): N of N files need embedding — stale/missing files embed. If the run was already fully embedded, you'll see 0 of N immediately.
2. Call again with no changes: 0 of N — no embedding tasks dispatched, nothing written.
3. Metadata-only change — retitle a file, re-queue:

   from learning_resources.models import ContentFile
   from vector_search.tasks import embed_run_content_files
   ContentFile.objects.filter(run_id=<RUN_PK>, published=True).update(title='retitled')
   embed_run_content_files.delay(<RUN_PK>)

   Logs N of N (all titles changed); the files take the payload-only path — their Qdrant payload shows the new title, with no re-embedding. Re-queueing again logs 0 of N.
4. Content change — edit a file's content and call .save() (which recomputes the checksum), re-queue: logs 1 of N and re-embeds just that file.

Testing via a real course re-ingest works too, but note the embed task is only queued from the ETL hook when QDRANT_ENABLE_INDEXING_PLUGIN_HOOKS=True is set — with it off (the local default), a re-ingest logs nothing.

🤖 Generated with Claude Code

[github-actions]

OpenAPI Changes

No changes detected

View full changelog

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[Copilot]

Pull request overview

This PR implements “Phase 2” of the embedding-load reduction by making Qdrant content-file embedding proportional to ETL changes (embed only new/changed/republished files instead of re-embedding the entire run on every load), while preserving legacy behavior when change details are unknown.

Changes:

• Add change detection in load_content_files to compute a changed subset of content files and pass it through the plugin hook (with a broker-size cap fallback).
• Extend the content_files_loaded hook to optionally carry content_file_ids and removed_unpublished, and update the search/indexing plugin logic accordingly (including tri-state removal behavior).
• Update embed_run_content_files to embed only published files and optionally restrict embedding to a provided ID list; add/extend tests for the new behavior.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
vector_search/tasks.py	Allow embedding only a provided subset of published content files for a run.
vector_search/tasks_test.py	Add coverage for subset embedding, skipping unpublished, and empty-id behavior.
main/settings.py	Add CONTENT_FILE_EMBED_ID_CAP to limit broker message size for changed-id lists.
learning_resources/utils.py	Extend content_files_loaded_actions to pass content_file_ids and removed_unpublished through the plugin hook.
learning_resources/hooks.py	Update the content_files_loaded hookspec signature/documentation to include new parameters.
learning_resources/etl/loaders.py	Snapshot prior file state, compute changed IDs, log change ratio, and pass subset/cap-fallback into the hook.
learning_resources/etl/loaders_test.py	Add unit tests for change detection and cap behavior reaching the hook.
learning_resources_search/plugins.py	Update hook implementation to embed only changed files (or all when None) and apply tri-state removal behavior.
learning_resources_search/plugins_test.py	Update existing expectations and add tests for removed_unpublished tri-state behavior.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

learning_resources/etl/loaders.py:1110

• _PAYLOAD_METADATA_FIELDS only tracks ContentFile columns, but the Qdrant content-file payload includes run/resource-derived fields (e.g., run_title, platform, offered_by, course_number, etc.). If those change on an ETL run while file checksums/metadata stay the same, changed_ids can be empty and no Qdrant task runs, leaving payload fields stale for all content-file points.

# Scalar, ETL-settable ContentFile columns whose Qdrant payload isn't covered by
# `checksum` (content-only). A deliberate subset of QDRANT_CONTENT_FILE_PARAM_MAP
# (which also holds non-columns, run-level, and AI-generated fields).
_PAYLOAD_METADATA_FIELDS = (
    "title",
    "description",
    "url",
    "file_type",

[abeglova]

This checks against the database for changes and not qdrant. The database etl job does not wait for the qdrant tasks to finish. The embedding is supposed to have 3 retries but if it does not go through for whatever reason (like celery becoming overwhelmed) qdrant will never be updated

[abeglova]

I think vector_search/utils.py#should_generate_content_embeddings already checks the checksome and does not do the expensive part of the embedding if there is no change

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

[mbertrand]

@abeglova ready for another look

[abeglova]

So i think this means that if a file is unpublished but the qdrant unpublish action fails the orphaned content file in qdrant will never be removed. I think you can just leave the loader file unchanged? I don't think the delete action is costly if the point does not exist

[abeglova]

This looks good but the pr title is no longer what the pr does - please change it