UN-3197 [MISC] Remove Celery file processing workers and dead code#1777
Open
muhammad-ali-e wants to merge 5 commits intomainfrom
Open
UN-3197 [MISC] Remove Celery file processing workers and dead code#1777muhammad-ali-e wants to merge 5 commits intomainfrom
muhammad-ali-e wants to merge 5 commits intomainfrom
Conversation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Contributor
Summary by CodeRabbit
WalkthroughRemoves multiple legacy Celery worker modules and queue configs, consolidates file-processing queue constants, replaces direct Celery task calls with generic send_task dispatch, deletes the large file_execution_tasks orchestration module, and adds a new Changes
Sequence Diagram(s)mermaid Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
muhammad-ali-e
requested review from
athul-rs,
chandrasekharan-zipstack,
jagadeeswaran-zipstack and
vishnuszipstack
…' of github-muhammad:Zipstack/unstract into feat/UN-3197-MISC_remove_celery_file_processing_workers
Contributor
Test ResultsSummary
Runner Tests - Full Report
SDK1 Tests - Full Report
|
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@backend/workflow_manager/workflow_v2/workflow_helper.py`:
- Around line 244-275: The failure handler _handle_execution_failure currently
updates only the WorkflowExecution and logs errors, which can leave the pipeline
run stuck if chord enqueue exceptions bypass the outer run_workflow path; import
and call PipelineUtils.update_pipeline_status (or the equivalent pipeline status
updater) from inside _handle_execution_failure using the pipeline/pipeline_run
identifier available on workflow_execution (e.g.,
workflow_execution.pipeline_run.id or workflow_execution.workflow.pipeline.id)
to mark the pipeline run as failed/errored and include the error message, or
alternatively re-raise the exception after cleanup so the outer run_workflow
path updates pipeline status; ensure you reference
PipelineUtils.update_pipeline_status and _handle_execution_failure in the
change.
Comment on lines
+258
to
+259
| # TODO: Remove related code when v1 workers are deprecated and removed | ||
| from plugins import get_plugin |
Contributor
There was a problem hiding this comment.
@muhammad-ali-e remove this comment and move the import to the top of the file
| echo -e " -b, --build-local Build docker images locally" | ||
| echo -e " -u, --update Update services version" | ||
| echo -e " -w, --workers-v2 Use v2 dedicated worker containers" | ||
| echo -e " -w, --workers-v2 Use v2 worker containers (worker-unified image)" |
Contributor
There was a problem hiding this comment.
@muhammad-ali-e now that we have removed the v1 workers, do we need this option? We should make this the default
Contributor
|
@muhammad-ali-e what about changes to the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
file_execution_tasks.py(all remaining methods were dead code after task removal)FileExecutionTasksimport and usage fromworkflow_helper.pyutils/constants.pyWhy
process_file_batch,process_batch_callback), all 18 remaining methods inFileExecutionTaskshad zero external callers — pure dead codebackend/workers/package (file_processing, file_processing_callback) is fully supersededHow
backend/backend/workers/package entirely (file_processing + file_processing_callback workers)backend/workflow_manager/workflow_v2/file_execution_tasks.py(1200+ lines of dead code)celery_task.pyandcelery_service.pyFileProcessingQueueconstants tobackend/utils/constants.pyworkflow_helper.pyto remove all references to deleted tasksrun-platform.shand CI workflow to remove old worker servicespyproject.tomlCan this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)
FileExecutionTasksorfile_execution_tasksremain in the codebaseFileProcessingQueueconstants are preserved inutils/constants.pyfor continued use byworkflow_helper.pyDatabase Migrations
Env Config
CELERY_FILE_PROCESSING_*,CELERY_FILE_PROCESSING_CALLBACK_*) are no longer needed but their presence is harmlessRelevant Docs
Related Issues or PRs
Dependencies Versions
Notes on Testing
Screenshots
N/A
Checklist
I have read and understood the Contribution Guidelines.