feat: add support for reading symbols from perf map files

[def]

This PR adds support for perf-<pid>.map files to improve symbolization for Java and Node.js applications.

For Node.js, perf map files can be enabled using the following args:

--perf-basic-prof-only-functions --interpreted-frames-native-stack --perf-basic-prof

For Java, the JVM should be started with -XX:+PreserveFramePointer. The perf map file can be dumped manually using:

jcmd <pid> Compiler.perfmap

On the agent side, periodic dumping can be triggered from the TargetFinder component.

Fixes #2766.

[CLAassistant]

All committers have signed the CLA.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[korniltsev]

Thanks for the contribution!

Note: We're in the process of migration from our own profiler implementation to otel profiler implementation. See here grafana/alloy#2920 .
It's very unlikely there will be an update in alloy with this feature included.
It's very likely the whole ebpf module will be removed.

The otel profiler has proper support for java and likely don't even need the perf map support.

Please consider give new otel profiler implementation a try and help us testing / dogfooding new profiler in your dev environments.

I will let this PR open for a while, then merge if you still think (given above) we need to merge this. It will not be released and will be removed some time soon.

[def]

Got it, thanks for the explanation.
From what we've seen so far, otel-ebpf-profiler doesn’t noticeably improve profiling quality for JVM or V8 apps. Plus, it uses about 5x more memory compared to pyroscope-ebpf.
Still, it's definitely worth taking a closer look.

[korniltsev]

Plus, it uses about 5x more memory compared to pyroscope-ebpf.

Yeah we see memory increase too.

One thing that is slightly different in our otel fork though is that it will be possible to select what's profiled and what's not. So you will get a bit more control over memory. Internally we had to disable profiling for clickhouse and oracledb because they are so huge.

[korniltsev]

From what we've seen so far, otel-ebpf-profiler doesn’t noticeably improve profiling quality for JVM

Is there any way you could share examples of flamegraphs that a bad quality or broken/missing from the otel profiler? Or maybe a small reproducer if possible? What exactly is bad quality there? It works OK on my machine profiling jetbrains IDEs. I don't have much workloads to dogfood other than kafaka.

[def]

Is there any way you could share examples of flamegraphs that a bad quality or broken/missing from the otel profiler? Or maybe a small reproducer if possible? What exactly is bad quality there? It works OK on my machine profiling jetbrains IDEs.

I tested it on the otel-demo on Kubernetes. Since the profiler isn’t integrated with our UI yet, I just hacked the source to print the stack traces to the console.

[korniltsev]

https://flamegraph.com/share/a9757406-1b57-11f0-86bc-aa320ab09ef2
https://flamegraph.com/share/b8de2a48-1b57-11f0-86bc-aa320ab09ef2

just did a quick test from oteldemo and the linked alloy PR, some services are looking OK from the first glance. What was the broken service from oteldemo?

[def]

I’m going to rerun my tests and will share more once I have it.
BTW, is there a better way for us to collaborate on this, outside of this PR?

[korniltsev]

Not sure about "better", but some alternatives could be:
Grafana labs community slack #pyroscope
CNCF slcak #otel-profiles