Skip to content

fix(traces): downgrade handle_proxy body-read failure from ERROR to WARN#1045

Closed
litianningdatadog wants to merge 2 commits intomainfrom
tianning.li/v93-degradation-investigation
Closed

fix(traces): downgrade handle_proxy body-read failure from ERROR to WARN#1045
litianningdatadog wants to merge 2 commits intomainfrom
tianning.li/v93-degradation-investigation

Conversation

@litianningdatadog
Copy link
Contributor

@litianningdatadog litianningdatadog commented Feb 24, 2026

JIRA: https://datadoghq.atlassian.net/browse/SLES-2729
Issue: #1000

Overview

When AWS Lambda freezes or terminates a function instance, the OS immediately closes all TCP sockets. The extension's proxy HTTP handler (handle_proxy) was mid-read on the request body — this read fails with error reading a body from connection and was logged at ERROR level, generating noise with no actionable signal.

This is expected Lambda lifecycle behaviour: the Lambda function completed successfully and nothing in the extension is broken. The ERROR level implies an extension fault, which misleads operators.

Change: Introduce a warn_response helper (mirroring the existing error_response / success_response trio) and use it for body-read failures in handle_proxy. The log message content is unchanged — only the severity is downgraded from ERROR to WARN.

Testing

  • unit tests

…n logger

Commit 3b533a0 introduced serde_json::json!() for JSON log formatting,
which allocates a heap Map+Value on every log call and pulled in the
tracing-serde crate via the "json" feature of tracing-subscriber,
increasing binary size and Lambda init duration.

Replace with direct write!/writeln! calls to the tracing Writer plus a
small write_json_escaped helper that handles all 6 mandatory JSON escape
sequences inline. The output format is identical.

Also remove the "json" feature from tracing-subscriber, dropping
tracing-serde as a transitive dependency.

JIRA: https://datadoghq.atlassian.net/browse/SVLS-8619
@litianningdatadog litianningdatadog requested a review from a team as a code owner February 24, 2026 22:31
@litianningdatadog litianningdatadog marked this pull request as draft February 24, 2026 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant