[Feature Request] PGO applicability

[zamazan4ik]

Feature Request

Profile-Guided Optimization (PGO) allows gaining additional performance for the software since it uses runtime profile information to perform more advanced optimization during the compilation process. Since Firecracker is a large project I guess PGO could help here to gain even more performance.

Describe the desired solution

Possible steps:

• Prepare firecracker build with PGO and bench it compared to the non-PGO firecracker.
• At least consider adding PGO to CI. Yes, it has a LOT of caveats like a huge bump in a build time, good profile preparation, profile stability between releases, and much more other stuff but in my opinion, it could be worth it.

Maybe would be a good idea to try to use BOLT too, but FMPOV "usual" PGO is much more stable.

Describe possible alternatives

Just leave things as is.

Additional context

Probably the most helpful link would be this one: vectordotdev/vector#15631

Checks

• Have you searched the Firecracker Issues database for similar requests?
• Have you read all the existing relevant Firecracker documentation?
• Have you read and understood Firecracker's core tenets?

[JonathanWoollett-Light]

Great idea. There are some concerns we have with PGO optimization for specific workloads, but feel free to post a PR on the issue.

[zamazan4ik]

There are some concerns we have with PGO optimization for specific workloads

Yeah, I understand. I think would be a good idea to start with just a testing PGO approach on some workloads with Firecracker. And then, if the results would be interesting, think about different workloads, their intersection in profiles, etc.

Also, recently I found results of applying PGO on similar projects:

• QEMU: Blog
• CrosVM: Intel blog

More PGO results on different projects could be found here.

[xmarcalx]

Hi @zamazan4ik ,

Thank you very much for your pointer and ideas.
I think it will be interesting to see a prototype of this functionality working so we can really quantify the benefits.
If you or anyway in the community is interested in take this task and showcase the data, it will be amazing and we will may have more datapoint on the best course of actions.

[ambersongliu]

Hello,

We are students from UT Austin and are looking to work on this issue for our virtualization class. If no one is working on it, we would love to. Thanks!

[zamazan4ik]

I am pretty sure that no one is working on this right now, so feel free to work on it. I would be happy to see your results on this!

[ambersongliu]

Amazing thank you! Do you mind assigning me to this issue? Thank you!

[valeriepieger]

Hi all,

Thanks so much for the details on this feature request and for outlining the concerns regarding PGO. I am in the group at UT Austin with Amber (currently assigned to this issue) and wanted to propose our current plan for this issue.

Based on the previous discussions on this issue and after reading the QEMU and CrosVM PGO workflows, we would like to start with a small, isolated prototype for PGO, without affecting the current build process or CI.

Here's what we are thinking:

1. Documentation

We will add a README-pgo.md with documentation on how to manually perform PGO builds:

• Instrumentation: Build Firecracker with RUSTFLAGS="-Cprofile-generate=/tmp/firecracker-profdata" cargo build --release to be able to generate .profraw files during execution
• Profiling: Running realistic, typical workloads (e.g., booting microVMs, basic I/O, networking)
• Optimization: merging profiling data (using llvm-profdata) and rebuilding Firecracker using RUSTFLAGS="-Cprofile-use=/tmp/firecracker.profdata -Cllvm-args=-pgo-warn-missing-function" cargo build --release

2. Prototype Developer Tooling

We will add a new script under tools/devtool called pgo-build that will allow the following subcommands:

• instrument: build Firecracker with profiling instrumentation
• profile: Run example workloads to generate profile data; will be optional
• merge: merge .profraw files into a .profdata file
• optimize: build Firecracker using the profiling data
  This should make local experimentation easier while being opt-in.

3. Optional Example Workload Runner

We may add a simple script to launch basic microVMs and exercise typical Firecracker functionality to help collect realistic profile data, but it will be purely for convenience. Users can always run their custom workloads instead.

4. Benchmarking

• If possible, we want to benchmark the normal vs optimized/PGO build to hopefully prove that PGO will be beneficial! We'll measure metrics like binary size, CPU usage, boot latency, etc.

Please let us know if this approach sounds reasonable or if you have any questions or concerns! It will be isolated so it won't change the default build behavior.

Thank you again for your time!

[xmarcalx]

Hi @valeriepieger ,

Thank you very much for the previous message.

The plan overall seems good 🙂

we would like to start with a small, isolated prototype for PGO, without affecting the current build process or CI.

Yes as you said I think this is the right first step. Let's focus on building a prototype which shows the effectiveness of a PGO approach with Firecracker. So feel free to develop a prototype in a feature branch so you can move faster than publishing PRs.

The documentation proposal is fine, let's make sure you write in your README how you perform the build and profile Firecracker such that we can enable everyone to reproduce the builds (and ourselves too, in case you will need a bit of help along the way).

Regarding benchmark to use, i think a good starting point is to actually run the performance tests that you already find into our repository and that you should be able to run locally too.
Will be good for example to see what will be the difference in those baselines between mainline Firecracker and the PGO-enabled one.