Skip to content

Enable Fedora CI for stage instance in requre package#700

Merged
majamassarini merged 1 commit intopackit:mainfrom
majamassarini:enable-fedora-ci-on-requre
Mar 25, 2026
Merged

Enable Fedora CI for stage instance in requre package#700
majamassarini merged 1 commit intopackit:mainfrom
majamassarini:enable-fedora-ci-on-requre

Conversation

@majamassarini
Copy link
Copy Markdown
Member

This is needed for the validation script to test the stage instance.

@centosinfra-prod-github-app
Copy link
Copy Markdown
Contributor

This is needed for the validation script to test the stage instance.
@majamassarini majamassarini force-pushed the enable-fedora-ci-on-requre branch from c85c4a1 to 5faacad Compare March 25, 2026 09:57
@centosinfra-prod-github-app
Copy link
Copy Markdown
Contributor

@majamassarini majamassarini merged commit 08da0e5 into packit:main Mar 25, 2026
3 of 4 checks passed
@github-project-automation github-project-automation bot moved this from New to Done in Packit pull requests Mar 25, 2026
majamassarini added a commit to majamassarini/packit-service that referenced this pull request Apr 1, 2026
Problem:
Celery workers create pidbox (control) reply queues for worker management
commands (inspect, ping, stats, etc.). These queues accumulate when workers
crash or restart improperly, leading to:
- 1,693+ orphaned *.reply.celery.pidbox keys in production
- Keys with no TTL (TTL = -1) that persist indefinitely

Root cause:
Celery's Redis transport does not provide a native way to set TTL on pidbox
reply queues when they're created. These are internal implementation details
of Celery's broadcast/control mechanism, and there's no configuration option
to automatically expire them.

Solution: Heartbeat cleanup task
Since we cannot tell Celery to natively set TTL on pidbox messages, we
implement a periodic heartbeat task that:
- Runs nightly at 12:30 AM via Celery beat
- Scans for *.reply.celery.pidbox keys without TTL
- Sets 1-hour expiration on orphaned queues
- Tracks total Redis keys via Prometheus for monitoring

Related to: packit/deployment#700
Should fix: packit#2983

Assisted-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

5 participants