As tlparse outputs get larger, I start noticing that tlparse itself is slow for large (>100MiB) structured logs.
I did a quick profile using cargo flamegraph to see what takes the most time in tlparse (assuming --plain-text).
Here's the flamegraph:
9cfe1d0
adds stack trie mini stacks to every compilation metric page. Turns out, doing this is pretty slow overall.
I think internally, downloading/uploading is still the majority of the time spent in tlparse, so this isn't super urgent. But might worth considering if we can optimize this better (for example, instead of formatting on rust side, we could inject javascript to do the formatting when the user opens the page instead)
As tlparse outputs get larger, I start noticing that tlparse itself is slow for large (>100MiB) structured logs.
I did a quick profile using
cargo flamegraphto see what takes the most time in tlparse (assuming--plain-text).Here's the flamegraph:
9cfe1d0
adds stack trie mini stacks to every compilation metric page. Turns out, doing this is pretty slow overall.
I think internally, downloading/uploading is still the majority of the time spent in tlparse, so this isn't super urgent. But might worth considering if we can optimize this better (for example, instead of formatting on rust side, we could inject javascript to do the formatting when the user opens the page instead)