fix: improve exhausted HTTP retry errors

[devin-ai-integration]

Summary

• Preserve connector-specific user-facing error messages when HTTP retry attempts are exhausted.
• Honor HttpResponseFilter.failure_type for retry actions, not only fail actions.
• Add and update regression coverage for custom retry failure_type and retry-exhaustion message propagation.

Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/16422
Related to https://github.com/airbytehq/oncall/issues/12575

Review & Testing Checklist for Human

• Confirm preserving message=str(e) on retry exhaustion is acceptable for rate-limit and non-rate-limit backoff exceptions.
• Confirm honoring custom failure_type on non-FAIL actions does not affect an unintended declarative connector pattern.
• Review downstream connector CI results for representative declarative connectors.

Notes

Local verification after fixing CI failures:

• poetry run pytest unit_tests/sources/declarative/requesters/error_handlers/test_default_error_handler.py unit_tests/sources/declarative/requesters/error_handlers/test_http_response_filter.py unit_tests/sources/streams/http/test_http_client.py -q
• poetry run ruff check airbyte_cdk/sources/declarative/requesters/error_handlers/http_response_filter.py airbyte_cdk/sources/streams/http/http_client.py unit_tests/sources/declarative/requesters/error_handlers/test_default_error_handler.py unit_tests/sources/declarative/requesters/error_handlers/test_http_response_filter.py unit_tests/sources/streams/http/test_http_client.py
• poetry run ruff format --check airbyte_cdk/sources/declarative/requesters/error_handlers/http_response_filter.py airbyte_cdk/sources/streams/http/http_client.py unit_tests/sources/declarative/requesters/error_handlers/test_default_error_handler.py unit_tests/sources/declarative/requesters/error_handlers/test_http_response_filter.py unit_tests/sources/streams/http/test_http_client.py

Link to Devin session: https://app.devin.ai/sessions/fbed94f0038b405e96d866a04e1daca9

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1778551689-improve-retry-exhaustion-errors#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1778551689-improve-retry-exhaustion-errors

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

4 060 tests  +1   4 049 ✅ +1   7m 52s ⏱️ +7s
    1 suites ±0      11 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 3c21116. ± Comparison against base commit d3d1346.

This pull request removes 5 and adds 6 tests. Note that renamed tests count towards both.

unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_fail_with_predicate-418-test_response_filter4-ResponseAction.FAIL-FailureType.system_error-]
unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_with_http_response_status_200_fail_with_contained_error_message-418-test_response_filter3-ResponseAction.FAIL-FailureType.system_error-]
unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_with_http_response_status_402_fail_with_default_failure_type-402-test_response_filter1-ResponseAction.FAIL-FailureType.system_error-]
unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_with_http_response_status_403_fail_with_default_failure_type-403-test_response_filter2-ResponseAction.FAIL-FailureType.config_error-HTTP Status Code: 403. Error: Forbidden. You don't have permission to access this resource.]
unit_tests.sources.declarative.requesters.error_handlers.test_http_response_filter ‑ test_matches[test_http_code_matches_failure_type_config_error_action_retry_uses_default_failure_type]

unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_fail_with_predicate-418-test_response_filter5-ResponseAction.FAIL-FailureType.system_error-]
unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_with_http_response_status_200_fail_with_contained_error_message-418-test_response_filter4-ResponseAction.FAIL-FailureType.system_error-]
unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_with_http_response_status_400_retry_with_custom_failure_type-400-test_response_filter1-ResponseAction.RETRY-FailureType.transient_error-HTTP Status Code: 400. Error: Bad request. Please check your request parameters.]
unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_with_http_response_status_402_fail_with_default_failure_type-402-test_response_filter2-ResponseAction.FAIL-FailureType.system_error-]
unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_with_http_response_status_403_fail_with_default_failure_type-403-test_response_filter3-ResponseAction.FAIL-FailureType.config_error-HTTP Status Code: 403. Error: Forbidden. You don't have permission to access this resource.]
unit_tests.sources.declarative.requesters.error_handlers.test_http_response_filter ‑ test_matches[test_http_code_matches_failure_type_config_error_action_retry_uses_configured_failure_type]

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

4 063 tests  +1   4 051 ✅ +1   11m 0s ⏱️ +4s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 3c21116. ± Comparison against base commit d3d1346.

This pull request removes 5 and adds 6 tests. Note that renamed tests count towards both.

unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_fail_with_predicate-418-test_response_filter4-ResponseAction.FAIL-FailureType.system_error-]
unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_with_http_response_status_200_fail_with_contained_error_message-418-test_response_filter3-ResponseAction.FAIL-FailureType.system_error-]
unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_with_http_response_status_402_fail_with_default_failure_type-402-test_response_filter1-ResponseAction.FAIL-FailureType.system_error-]
unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_with_http_response_status_403_fail_with_default_failure_type-403-test_response_filter2-ResponseAction.FAIL-FailureType.config_error-HTTP Status Code: 403. Error: Forbidden. You don't have permission to access this resource.]
unit_tests.sources.declarative.requesters.error_handlers.test_http_response_filter ‑ test_matches[test_http_code_matches_failure_type_config_error_action_retry_uses_default_failure_type]

unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_fail_with_predicate-418-test_response_filter5-ResponseAction.FAIL-FailureType.system_error-]
unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_with_http_response_status_200_fail_with_contained_error_message-418-test_response_filter4-ResponseAction.FAIL-FailureType.system_error-]
unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_with_http_response_status_400_retry_with_custom_failure_type-400-test_response_filter1-ResponseAction.RETRY-FailureType.transient_error-HTTP Status Code: 400. Error: Bad request. Please check your request parameters.]
unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_with_http_response_status_402_fail_with_default_failure_type-402-test_response_filter2-ResponseAction.FAIL-FailureType.system_error-]
unit_tests.sources.declarative.requesters.error_handlers.test_default_error_handler ‑ test_default_error_handler_with_custom_response_filter[_with_http_response_status_403_fail_with_default_failure_type-403-test_response_filter3-ResponseAction.FAIL-FailureType.config_error-HTTP Status Code: 403. Error: Forbidden. You don't have permission to access this resource.]
unit_tests.sources.declarative.requesters.error_handlers.test_http_response_filter ‑ test_matches[test_http_code_matches_failure_type_config_error_action_retry_uses_configured_failure_type]

♻️ This comment has been updated with latest results.