Skip to content

fix(async-job): propagate wrapped failure type on async job aggregation#994

Open
devin-ai-integration[bot] wants to merge 4 commits into
mainfrom
devin/1776776408-async-job-failure-type-propagation
Open

fix(async-job): propagate wrapped failure type on async job aggregation#994
devin-ai-integration[bot] wants to merge 4 commits into
mainfrom
devin/1776776408-async-job-failure-type-propagation

Conversation

@devin-ai-integration
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot commented Apr 21, 2026

Summary

  • Preserve job-creation exceptions on placeholder async jobs so exhausted creation failures keep their original FailureType.
  • Aggregate wrapped async-job failures by precedence (config_error > transient_error > system_error) and include a deterministic failure-type breakdown in the internal message.
  • Add regression coverage for job-creation 429/transient failures and the failure-type aggregation helpers.
  • Fix CDK image build generation for Python connectors whose README.md is a symlink into Airbyte's shared connector docs but whose Docker build context omits that shared target.

Review & Testing Checklist for Human

  • Re-run or review connector image checks for source-hardcoded-records and destination-motherduck; both failed before the image-build fix with entrypoint exit 127.
  • Verify a connector with job-creation HTTP 429 now emits final aggregate failure_type=transient_error with Underlying failure breakdown: transient_error=1..
  • Confirm the user-facing async-job aggregate messages remain acceptable aggregation keys.

Notes

Local validation performed:

  • poetry run pytest unit_tests/sources/declarative/async_job/test_job_orchestrator.py unit_tests/sources/declarative/async_job/test_job.py -q
  • poetry run ruff check airbyte_cdk/sources/declarative/async_job/job.py airbyte_cdk/sources/declarative/async_job/job_orchestrator.py airbyte_cdk/utils/docker.py unit_tests/sources/declarative/async_job/test_job_orchestrator.py
  • poetry run ruff format --check airbyte_cdk/sources/declarative/async_job/job.py airbyte_cdk/sources/declarative/async_job/job_orchestrator.py airbyte_cdk/utils/docker.py unit_tests/sources/declarative/async_job/test_job_orchestrator.py
  • poetry run mypy --config-file mypy.ini airbyte_cdk/sources/declarative/async_job/job.py airbyte_cdk/sources/declarative/async_job/job_orchestrator.py airbyte_cdk/utils/docker.py
  • Local image verification succeeded for source-hardcoded-records and destination-motherduck using the patched CDK image builder.

Link to Devin session: https://app.devin.ai/sessions/2d23f8f9ce9247b7b414d1d90eab9114
Requested by: Daryna Ishchenko (@darynaishchenko)

Replace hardcoded system_error in AsyncJobOrchestrator's aggregated failure with the highest-precedence FailureType among wrapped non-breaking exceptions (config_error > transient_error > system_error). The user-facing message is chosen per FailureType to stay deterministic; underlying failure-type counts and exception reprs are moved into internal_message.

Co-Authored-By: bot_apk <apk@cognition.ai>
@devin-ai-integration
Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions
Copy link
Copy Markdown

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1776776408-async-job-failure-type-propagation#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1776776408-async-job-failure-type-propagation

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /poetry-lock - Updates poetry.lock file
  • /test - Runs connector tests with the updated CDK
  • /prerelease - Triggers a prerelease publish with default arguments
  • /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
  • /poe <command> - Runs any poe command in the CDK environment
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 21, 2026

PyTest Results (Fast)

4 065 tests  +6   4 054 ✅ +6   7m 58s ⏱️ +13s
    1 suites ±0      11 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 356535f. ± Comparison against base commit d3d1346.

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 21, 2026

PyTest Results (Full)

4 068 tests  +6   4 056 ✅ +6   11m 21s ⏱️ +25s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 356535f. ± Comparison against base commit d3d1346.

♻️ This comment has been updated with latest results.

@darynaishchenko Daryna Ishchenko (darynaishchenko) marked this pull request as ready for review May 13, 2026 12:01
@devin-ai-integration devin-ai-integration Bot changed the title fix(async-job): propagate wrapped FailureType on async job aggregation fix(async-job): propagate wrapped failure type on async job aggregation May 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant