Skip to content

test: stateful multi-turn tool-call cassettes with context retention #77

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
franciscojavierarceo merged 5 commits into from
Jun 25, 2026
+15,202 −22
Merged

test: stateful multi-turn tool-call cassettes with context retention #77

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d1b0bec
test: add stateful multi-turn tool-call cassettes with previous_respo…
ashwing Jun 24, 2026
4ed6bb8
test: add OpenAI cassettes, parallel calls, and tool-output-only patt…
ashwing Jun 24, 2026
cb86e14
docs: update record.sh with all 12 cassette recording commands
ashwing Jun 24, 2026
7f4d2cb
test: re-record all 12 cassettes from record.sh for determinism
ashwing Jun 24, 2026
8ca36bb
refactor: remove assert_stateful_chaining from accumulator tests
ashwing Jun 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions crates/agentic-core/src/executor/accumulator.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ impl ResponseAccumulator {
if !self.accumulated_arguments.is_empty() && fc.arguments.is_empty() {
fc.arguments = std::mem::take(&mut self.accumulated_arguments);
}
fc.status = "completed".to_string();
fc.status = MessageStatus::Completed;
self.output.push(OutputItem::FunctionCall(fc));
}
self.accumulated_arguments.clear();
Expand Down Expand Up @@ -277,7 +277,7 @@ impl ResponseAccumulator {
call_id: call_id.clone().unwrap_or_default(),
name: name.clone().unwrap_or_default(),
arguments: String::new(),
status: "in_progress".to_string(),
status: MessageStatus::InProgress,
});
}
_ => {
Expand Down Expand Up @@ -776,7 +776,7 @@ mod tests {
assert_eq!(fc.call_id, "call_abc");
assert_eq!(fc.name, "get_weather");
assert_eq!(fc.arguments, r#"{"location":"Paris"}"#);
assert_eq!(fc.status, "completed");
assert_eq!(fc.status, MessageStatus::Completed);
} else {
panic!("expected FunctionCall");
}
Expand Down Expand Up @@ -1099,7 +1099,7 @@ mod tests {
assert_eq!(acc.output.len(), 1);
if let OutputItem::FunctionCall(fc) = &acc.output[0] {
assert_eq!(fc.arguments, r#"{"x":1}"#);
assert_eq!(fc.status, "completed");
assert_eq!(fc.status, MessageStatus::Completed);
} else {
panic!("expected FunctionCall");
}
Expand Down
18 changes: 17 additions & 1 deletion crates/agentic-core/src/types/io.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
use serde::{Deserialize, Serialize};
use serde_json::Value;

use super::event::MessageStatus;

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct InputTextContent {
#[serde(rename = "type")]
Expand Down Expand Up @@ -121,7 +123,21 @@ pub struct FunctionToolCall {
pub call_id: String,
pub name: String,
pub arguments: String,
pub status: String,
#[serde(default = "default_completed_status")]
#[serde(deserialize_with = "deserialize_status_or_default")]
pub status: MessageStatus,
}

fn default_completed_status() -> MessageStatus {
MessageStatus::Completed
}

fn deserialize_status_or_default<'de, D>(deserializer: D) -> Result<MessageStatus, D::Error>
where
D: serde::Deserializer<'de>,
{
let opt: Option<MessageStatus> = Option::deserialize(deserializer)?;
Ok(opt.unwrap_or(MessageStatus::Completed))

@maralbahari maralbahari Jun 24, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

important note for gpt-oss serving if you used vLLM upstream serving you might make sure that you have passed correct tool parser. maybe the tool parser that you served the model on vLLM was not able to catch current status.

@ashwing ashwing Jun 24, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — this isn't a tool parser config issue, it's how vLLM's Responses API serializes completed function calls (status field is always null in the output). The fix is a deserialize_with handler that defaults null to Completed. All 4 cassettes have "status": null in responses and the tests verify they deserialize to MessageStatus::Completed.

@maralbahari maralbahari Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashwing this behavior only when running gpt-oss on vLLM right? if so can you check running gpt-oss on OpenAI and see the behavior is the same there just to confirm that this is not a bug from vLLM upstream.

@ashwing ashwing Jun 25, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — vLLM-specific. The OpenAI cassettes in this PR (recorded against gpt-4o) show "status": "completed" on function_call items. vLLM returns "status": null. The deserializer handles both: null defaults to Completed, string "completed" maps directly.

@maralbahari maralbahari Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashwing I mean if such behavior is only on a harmony model being served on vLLM it might be bug on vLLM upstream we need to note about to open issue. otherwise we should not fix bug from upstream by mapping into a default. ordinarily the behaviors of responses on vLLM should match OpenAI Responses API. would need to confirm if "status": null also occurs for serving other non harmony models on vLLM.
for now it should be fine to handle this way but later if there is a bug from vLLM we need to address that. would need to confirm this.

@maralbahari maralbahari Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashwing I think the tool parser you selected is wrong. https://docs.vllm.ai/projects/recipes/en/latest/OpenAI/GPT-OSS.html#tool-use based on this vllm's recipe it has to be openai

@ashwing ashwing Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested both --tool-call-parser openai (per the vLLM recipe) and hermes — same result with both. The earlier table only showed hermes but I had already validated both.

Did root-cause investigation. Confirmed bug in vLLM's Harmony codepath on current main — no existing issue or PR for it.

Test matrix

All tests use /v1/responses, store=true, non-streaming:

Model --tool-call-parser vLLM function_call.status
Gemma4 26B functiongemma v0.21.0 "completed"
Hermes-3-Llama-3.1-8B hermes v0.22.0 "completed"
gpt-oss-20b openai v0.22.0 null
gpt-oss-20b hermes v0.22.0 null
OpenAI gpt-4o (ground truth) — — "completed"
OpenAI gpt-4o-mini (ground truth) — — "completed"

Parser doesn't matter — gpt-oss returns null with both openai and hermes. Non-Harmony models (Gemma4, Hermes-3) return "completed" correctly.

Root cause in vLLM source

Non-Harmony path (responses_parser.py:101) — correctly sets status:

ResponseFunctionToolCall(
    id=f"fc_{random_uuid()}",
    call_id=f"call_{random_uuid()}",
    type="function_call",
    status="completed",  # ✓ present
    name=tool_call.function.name,
    arguments=tool_call.function.arguments,
)

Harmony path (harmony.py:315) — omits status:

ResponseFunctionToolCall(
    arguments=content.text,
    call_id=f"call_{random_id}",
    type="function_call",
    name=function_name,
    id=f"fc_{random_id}",
    # ← no status, defaults to None via Pydantic
)

Since ResponseFunctionToolCall.status is Optional[...] = None in the OpenAI SDK, it silently serializes as null. One-line fix: add status="completed" to _parse_function_call.

Current state

  • Not fixed on vLLM main — verified on HEAD (f59db63, June 23). _parse_function_call is unchanged.
  • No existing issue or PR — searched multiple queries, nothing filed.
  • Will open an upstream vLLM issue for this.

Our deserializer handles both null and "completed" correctly in the meantime.

@maralbahari maralbahari Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashwing thanks for double checking. I do understand the default deserialization is handling the case but is not an actual fix. we need to make sure vLLM Responses API is mirroring OpenAI if not in cases like this means there is a bug on vLLM. worth taking note instead of us silently resolving issues on agentic-api this way. BTW the harmony models like gpt-oss are not really part of MVP either.It's good to have them in cassettes to know where we stand but we should rely on other models to serve on vLLM for agentic-api verification.

@tjtanaa tjtanaa Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ashwing please file an issue to vLLM upstream.

CC @bbrowning @chaunceyjiang

@ashwing ashwing Jun 28, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed upstream: vllm-project/vllm#46940

}

#[derive(Debug, Clone, Serialize, Deserialize)]
Expand Down
Loading