Add tests for TryNode wrapping non-streaming InlinePromptNode with PROVIDER_ERROR#3738
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5e6adffce4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| assert terminal_event.name == "workflow.execution.fulfilled" | ||
|
|
||
| # AND the error output is present | ||
| assert terminal_event.outputs.error is not None |
There was a problem hiding this comment.
Assert non-streaming call in connection-reset test
This assertion is too weak to prove the intended regression path: TryNode.wrap() (without on_error_code) will fulfill the workflow with some error for any subworkflow rejection, so the test can pass even if InlinePromptNode never calls adhoc_execute_prompt (for example, if it accidentally uses the streaming path or raises a different internal error). To make this test actionable for future regressions, assert that vellum_adhoc_prompt_client.adhoc_execute_prompt was called and validate the captured error details/code, not just non-nullity.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch. I've added vellum_adhoc_prompt_client.adhoc_execute_prompt.assert_called_once() to both tests, and strengthened the connection-reset test to also assert the error code (NODE_EXECUTION) and that the message contains "Connection reset by peer".
…OVIDER_ERROR Tests cover two scenarios: 1. Non-streaming API returns REJECTED event with PROVIDER_ERROR 2. Non-streaming API raises raw ConnectionResetError Both tests use pytest.mark.timeout(5) to detect hangs. Co-Authored-By: vargas@vellum.ai <vargas@vellum.ai>
…as called Co-Authored-By: vargas@vellum.ai <vargas@vellum.ai>
5e6adff to
5bea144
Compare
Summary
Adds two tests for TryNode wrapping a non-streaming (
stream_enabled=False) InlinePromptNode that encounters a PROVIDER_ERROR. Both tests use@pytest.mark.timeout(5)to detect hangs.test_try_node__prompt_node_non_streaming_provider_error__workflow_completes— Mocksadhoc_execute_promptto return aRejectedAdHocExecutePromptEventwithPROVIDER_ERROR. Asserts the workflow fulfills with the error captured inTryNode.Outputs.error(includingraw_data={}).test_try_node__prompt_node_non_streaming_connection_error__workflow_completes— Mocksadhoc_execute_promptto raise a rawConnectionResetError. Asserts the workflow fulfills, verifiesadhoc_execute_promptwas called, and checks the error code (NODE_EXECUTION) and message content.Both tests also assert that
vellum_adhoc_prompt_client.adhoc_execute_promptwas called exactly once, confirming the non-streaming API path was exercised.on_error_code) and all pass. More details about the exact production conditions may be needed to reproduce the hang.Updates since last revision
raw_data={}assertion mismatch that caused CI failure (main'sWorkflowErrornow includesraw_dataandstacktracefields)NODE_EXECUTIONerror code (notINTERNAL_ERROR) and verify message contentassert_called_once()onadhoc_execute_promptin both tests to confirm the non-streaming path was actually invoked (per review feedback)Review & Testing Checklist for Human
adhoc_execute_prompt.return_value = RejectedAdHocExecutePromptEvent(...)) accurately reflects what happens in production when a non-streaming prompt gets a PROVIDER_ERROR. Could the real failure path involve the streaming endpoint being called despitestream_enabled=False?resultsoutput, or whether it only manifests when the workflow is invoked via Vembda rather than the SDK'sWorkflowRunner.Notes
StateMeta.__deepcopy__preservesQueuereferences without deep-copying them. If a streaming output Queue never receives theundefinedsentinel (e.g., due to an exception mid-stream), and something later resolves that Queue, it would block forever. However, in all tested scenarios the cleanup path handles this correctly.BaseInlinePromptNode._process_prompt_event_streamskips thenext(prompt_event_stream)call (INITIATED event skip) whenstream_enabled=False, but this doesn't appear to cause issues in the tested scenarios.Link to Devin session: https://app.devin.ai/sessions/ee4bcf35c06e4c85b305580388d3d35d
Requested by: @dvargas92495