Improve Logflare.Mapper to support flat structures better#3323
Open
Improve Logflare.Mapper to support flat structures better#3323
Logflare.Mapper to support flat structures better#3323Conversation
amokan
force-pushed
the
adammokan/o11y-1607-use-logeventflattened_body-attribute-to-simplify-ch-simple
branch
from
503da75 to
95dda7e
Compare
Ziinc
approved these changes
Mar 31, 2026
Contributor
Ziinc
left a comment
Ziinc
left a comment
There was a problem hiding this comment.
lacking test coverage for new logic, adding some property tests on the elixir side would be good.
| /// | ||
| /// Returns the mapped Elixir map directly (no ok/error tuple). | ||
| /// | ||
| /// When `flat_keys` is `true`, dotted JSONPaths like `$.resource.service.name` |
Contributor
There was a problem hiding this comment.
LogEvent.flattened_body stores JSONPaths as keys? or is the initial prefix ignored? should decide which to support, would get configusing down the road, especially if we also consider schema flat_map .
Contributor
There was a problem hiding this comment.
LogEvent.flatten_body doesn't store it as JSONPaths so this might be wrong...
…dy-attribute-to-simplify-ch-simple
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a new option to
Logflare.Mapper.map/3to provide a keyword offlat_keys: trueas a last argument that avoids extra work in the Rust layer now that we have theLogEvent.flattened_bodyavailable.This also changes the actual ClickHouse pipeline to leverage the new feature.
Benchmarking shows a small increase in IPS and a drop in reductions with these changes.
This change is based on #3322 and will be rebased after that is merged