OTLP Logs drop Resource and Scope attributes #8205
Description
Bug Report
Describe the bug
When you send OTLP logs into Fluent Bit and then send them back out, any Resource or Scope attributes are lost. This is because out_opentelemetry sending logs just initializes and empty resource and scope, thus meaning any data sent in are ignored.
To Reproduce
- Rubular link if applicable:
- Example log message if applicable:
When this OTLP goes in:
{
"resourceLogs": [
{
"resource": {
"attributes": [
{
"key": "host.name",
"value": {
"stringValue": "debian-dev-machine"
}
},
{
"key": "os.type",
"value": {
"stringValue": "linux"
}
}
]
},
"scopeLogs": [
{
"scope": {},
"logRecords": [
{
"observedTimeUnixNano": "1700497457844514864",
"body": {
"stringValue": "here"
},
"attributes": [
{
"key": "log.file.name",
"value": {
"stringValue": "a.txt"
}
}
],
"traceId": "",
"spanId": ""
},
...
]
}
],
"schemaUrl": "https://opentelemetry.io/schemas/1.6.1"
}
]
}
I passed it into this Fluent Bit config:
[SERVICE]
Log_level debug
Flush 1
[INPUT]
Name opentelemetry
Listen 127.0.0.1
Port 6969
[OUTPUT]
Name opentelemetry
Match *
Host 127.0.0.1
Port 4318
Logs_uri /v1/logs
Then sent it to another local OTel collector and dumped the result to a file.
This is the OTLP that came out:
{
"resourceLogs": [
{
"resource": {},
"scopeLogs": [
{
"scope": {},
"logRecords": [
{
"timeUnixNano": "1700497457945114315",
"body": {
"kvlistValue": {
"values": [
{
"key": "message",
"value": {
"stringValue": "here"
}
}
]
}
},
"attributes": [
{
"key": "log.file.name",
"value": {
"stringValue": "a.txt"
}
}
],
"traceId": "",
"spanId": ""
},
...
]
}
]
}
]
}
- Steps to reproduce the problem:
The config I mentioned above, and what I did to reproduce was used an otel collector to read logs from a file and export otlphttp to Fluent Bit, then Fluent Bit output to another otel collector that receives the OTLP and dumps the result to a file.
Expected behavior
The data I passed in with OTLP should, at minimum, be the same when it comes out if there is no processing done.
Screenshots
Your Environment
- Version used: built from master
- Configuration: See above
- Environment name and version (e.g. Kubernetes? What version?): normal Debian VM
- Server type and version: Debian bullseye VM
- Operating System and version: Debian 11
- Filters and plugins: See above config
Additional context
This makes Fluent Bit basically unusable for any OTLP-focused pipelines. Losing resource attributes is a particularly big problem, especially because Fluent Bit doesn't appear to provide any way to decorate Resource with attributes to begin with. (That is worth a separate issue probably). I haven't tried traces, but when I tried metrics this wasn't a problem; the data I passed in was what I got out (although all my metric names were prefixed with underscore, which is also definitely a bug, see issue #8083