Support thread names in profiles converted from JFR

[korniltsev]

We currently do not include thread names from JFR in any way.
There are 2 ways we could improve that I know of :

1. Allow adding thread names as a proof label.
2. Allow adding thread names as the root frame in flame graph.

[polarnik]

Hello!

I tried to implement this feature.

I used the feature:

• https://grafana.com/docs/pyroscope/latest/reference-server-api/#jfr-with-labels
  and split my JFR file by several files for different thread-name groups like:
• Thread-*
• G1*
• qtp*
• ForkJoinPool.commonPool.*
• ...

and sent JFR files with labels Thread, G1, qtp into the pyroscope storage.

I tried 2 years ago. And I could not filter the flame graphs by labels.

I will try again for the actual version of pyroscope

[korniltsev]

Hello, @polarnik

The Labels sent in the LabelsSnapshot proto are "dynamic labels" and is a feature of our async-profiler fork.
I will describe how it works, so that you can have better understanding why it could not work in your experiments.

In the fork each sample event with a stacktrace has a contextID field.
https://github.com/grafana/async-profiler/blob/feat/context_id/src/jfrMetadata.cpp#L95 And we have API to set this contetID https://github.com/grafana/async-profiler/blob/8e690a4d2c361d0c3b9da160b856f2043c6004e6/src/javaApi.cpp#L58

contextID is just long

Then we have these protos send along the

message Context {
    // string_id -> string_id
    map<int64, int64> labels = 1;
}
message LabelsSnapshot {
  // context_id -> Context
  map<int64, Context> contexts = 1;
  // this is just a global string table for deduplication, string_id -> string
  map<int64, string> strings = 2;
}

Then in order to annotate an event with labels, it has to have some contextID , and a corresponding context in the snapshot.

snapshot := LabelsSnapshot {
    contexts: map[int64]Context {
         239: Context {
              labels: map[int64]int64{ 
                   1:2,
                    3:4,
              }
         }
    }
    strings: []string{
        "", // 0
        "thread_name", // 1
        "ForkJoinPool.commonPool.", // 2
        "label_key2",
        "value_key2"
   }
}

In this example if a JFR event has contextID=239 then it will be annotated with labels [{"thread_name": "ForkJoinPool.commonPool."}, {"label_key2", "value_key2"}]

[korniltsev]

If you have already split the JFR into multiples groupped by thread - It is possible to just ingest them with static labels as the thread name.

cat ForkJoinPool.jfr  | curl -X POST --data-binary @- 'http://localhost:4040/ingest?name=IntellijIdeaUltimate{thread_name="ForkJoinPool.commonPool"}&format=jfr'

[korniltsev]

All of the above options are for "advanced" pyroscope users. Pyroscope should have an easier way to do this for people who don't want to split JFRs by hand. I think this should be the way for them grafana/jfr-parser#59