[question] spans_malformed(invalid_http_status_code)

[mcatany]

Hi,

We can see the following error in trace-agent

2024-08-20 15:36:34 UTC | TRACE | WARN | (pkg/trace/info/stats.go:101 in LogAndResetStats) | [lang:cpp lang_version:202002 tracer_version:v0.2.2 endpoint_version:v0.4 service:istio-ingressgateway.gateway] -> spans_malformed(invalid_http_status_code:13). Enable debug logging for more details.
2024-08-21 02:50:19 UTC | TRACE | WARN | (pkg/trace/info/stats.go:101 in LogAndResetStats) | [lang:cpp lang_version:202002 tracer_version:v0.2.2 endpoint_version:v0.4 service:istio-ingressgateway.gateway] -> spans_malformed(invalid_http_status_code:6). Enable debug logging for more details.
2024-08-21 09:44:59 UTC | TRACE | WARN | (pkg/trace/info/stats.go:101 in LogAndResetStats) | [lang:cpp lang_version:202002 tracer_version:v0.2.2 endpoint_version:v0.4 service:webapp-bo.alumni-connect] -> spans_malformed(invalid_http_status_code:1). Enable debug logging for more details.
2024-08-21 09:44:59 UTC | TRACE | WARN | (pkg/trace/info/stats.go:101 in LogAndResetStats) | [lang:cpp lang_version:202002 tracer_version:v0.2.2 endpoint_version:v0.4 service:istio-ingressgateway.gateway] -> spans_malformed(invalid_http_status_code:1). Enable debug logging for more details.
2024-08-21 09:48:09 UTC | TRACE | WARN | (pkg/trace/info/stats.go:101 in LogAndResetStats) | [lang:cpp lang_version:202002 tracer_version:v0.2.2 endpoint_version:v0.4 service:webapp-bo.alumni-connect] -> spans_malformed(invalid_http_status_code:1). Enable debug logging for more details.
2024-08-21 09:48:09 UTC | TRACE | WARN | (pkg/trace/info/stats.go:101 in LogAndResetStats) | [lang:cpp lang_version:202002 tracer_version:v0.2.2 endpoint_version:v0.4 service:istio-ingressgateway.gateway] -> spans_malformed(invalid_http_status_code:2). Enable debug logging for more details.
2024-08-21 11:23:19 UTC | TRACE | WARN | (pkg/trace/info/stats.go:101 in LogAndResetStats) | [lang:cpp lang_version:202002 tracer_version:v0.2.2 endpoint_version:v0.4 service:whitelabel.whitelabel] -> spans_malformed(invalid_http_status_code:1). Enable debug logging for more details.

What could it be the origin of it? We're guessing that it could come from the istio sidecar. Do you know if there have been errors related to this previously?

BR

[dmehala]

Hi @mcatany

It looks like the issue is coming from Istio / Envoy. This setup is a bit different from others integrations, because, while we provide the tracer, the Envoy team handles the instrumentation.

I've seen similar behavior before, where Envoy tags spans with status code outside the expected HTTP [100, 599] range. Since our Agent expects a status code within this range, those spans are flagged as malformed and aren't ingested by the trace agent.

My gut feeling is that something might be off on those requests, causing these spans to be tagged with a status code. The status code is not in a HTTP code range, maybe it's not an HTTP status code then. Are you using a gRPC or other kind of filter?

In the meantime, I strongly suggest to open a support case here for visibility.

[mcatany]

Hi @dmehala, thanks for the response.

Yes, I opened a support ticket last week and the engineering team is supposed to be checking it. What do you mean by "filter". We use some grpc requests but I would say they aren't related to those traces...

[dmehala]

By filters, I was referring to Envoy filter. Unfortunately, I don't have a solution right now, but I'l give it a some thought this week, talk with colleagues and get back to you. If you can infer any pattern, that could help.

Also, just to confirm, is the support ticket for CleverTech?

[mcatany]

Thanks,

The ticket is for Hivebrite number 1810265

[verkaufer]

👋 Was there a resolution to this issue? I'm running into it as well, and I'm wondering if you two found a solution for Envoy tagging the spans with invalid HTTP status codes.

[mcatany]

Nope, we still have the same problem :S