Add acks_late to ETL tasks killed by worker autoscaling scale-downs

[mbertrand]

What are the relevant tickets?

Fixes #3677

Description (What does it do?)

Adds acks_late=True, reject_on_worker_lost=True to the scheduled ETL tasks that production logs show are being lost when celery worker pods are terminated by autoscaling scale-downs mid-run (details and 7-day loss counts in the issue). With early acks (celery's default) the broker message is gone the moment the task starts, so a killed run vanishes and stays STARTED forever in the monitoring app. With late acks the broker redelivers the message after the visibility timeout (1 hour) and the run completes on a surviving worker.

This matches the pattern already used by get_ocw_courses, get_youtube_data, summarize_content_files_task, etc., which are killed just as often but demonstrably recover via redelivery.

Tasks changed (all idempotent):

• learning_resources: get_mitxonline_data, get_mit_edx_data, get_xpro_data, get_content_files, get_podcast_data, get_learning_resource_views, get_mitpe_data, get_sloan_data, get_mit_climate_data, update_next_start_date_and_prices
• news_events: get_medium_mit_news, get_ol_events, get_mitpe_news, get_mitpe_events, get_website_content_news
• vector_search: sync_topics

Interaction with cooldown_task (#3363): unchanged. The cooldown check runs on the worker at execution time regardless of ack mode, and Reject(requeue=False) is terminal, so re-triggers during a cooldown window are still dropped. Cooldown locks (900s) expire well before the 1-hour visibility timeout, so a redelivered recovery run is not blocked by its own cooldown. (get_mit_edx_data's 3600s lock ≈ the visibility timeout, so its recovery run can race the lock — worst case it is rejected, which is exactly today's behavior.)

How can this be tested?

Verified locally end to end (steps reproducible with COMPOSE_PROFILES=backend):

1. docker compose up -d celery, then enqueue a run and note the task id:

   docker compose run --rm web python manage.py shell -c "from learning_resources.tasks import get_mitxonline_data; print(get_mitxonline_data.delay(_cooldown_force=True).id)"
   

2. While it runs (~3 min), confirm the message is still unacknowledged — this is the behavior this PR adds; on main the count for this task is 0 because the message is acked at start:

   docker compose exec redis redis-cli -n 4 hlen unacked
   

3. Kill the worker mid-run, as an autoscaling scale-down does after the grace period: docker kill -s KILL mit-learn-celery-1. The message survives in unacked.
4. Restart the worker (docker compose up -d celery). Redelivery normally happens after the 1-hour visibility timeout; to avoid waiting, age the pending entries so the next restore cycle picks them up:

   docker compose exec redis redis-cli -n 4 eval "local tags = redis.call('ZRANGE','unacked_index',0,-1); for i,t in ipairs(tags) do redis.call('ZADD','unacked_index',0,t) end; return #tags" 0
   

5. Within ~a minute the worker logs show the same task id received again, and it completes:

   Task learning_resources.tasks.get_mitxonline_data[0c5b14a5-...] received      <- original, killed
   Task learning_resources.tasks.get_mitxonline_data[0c5b14a5-...] received      <- redelivered
   Task learning_resources.tasks.get_mitxonline_data[0c5b14a5-...] succeeded in 158.7s: 446
   

Also verify cooldown behavior is preserved: enqueue the same task twice within 900s — the second run logs Skipping ...: cooldown active and is rejected without executing.

Sanity check that the flags are active:

docker compose run --rm web python manage.py shell -c "from learning_resources.tasks import get_mitxonline_data as t; print(t.acks_late, t.reject_on_worker_lost)"

Additional Context

After deploy, killed runs will show as redelivered-then-succeeded in https://celery-monitoring.odl.mit.edu instead of stuck STARTED. Stuck-STARTED entries from before the deploy won't self-heal; their next scheduled run covers them.

[github-actions]

OpenAPI Changes

No changes detected

View full changelog

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[Copilot]

Pull request overview

This PR updates Celery task configuration for a set of scheduled ETL jobs so that if a worker is terminated mid-run (e.g., Kubernetes autoscaling scale-down), tasks are not lost and can be redelivered by the broker and complete on another worker.

Changes:

• Add acks_late=True and reject_on_worker_lost=True to selected scheduled ETL tasks in learning_resources, news_events, and vector_search.
• Align these tasks’ acknowledgment behavior with existing late-ack + worker-lost rejection patterns already present elsewhere in the codebase (e.g., other vector_search tasks).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
learning_resources/tasks.py	Marks several scheduled ETL tasks as late-ack + requeue-on-worker-loss to prevent silent loss on worker termination.
news_events/tasks.py	Applies late-ack + requeue-on-worker-loss to key scheduled news/event ETL tasks.
vector_search/tasks.py	Applies late-ack + requeue-on-worker-loss to the scheduled sync_topics task to improve resilience to worker scale-downs.

[shanbady]

👍