[Fleet] Improve package policy upgrade performance

[nchaulet]

Description

Package policy upgrade after a package upgrade could be really problematic when upgrading a lot of package policies or when upgrading a large package (aws is a good example to use for testing)

Note: it seems there is also an issue in the UI as we call the dry run upgrade API in the settinugs page to show the checkbox to upgrade integration policies

Possible optimizations

Work in progress

• Group bumping agent policy revision, we are bumping revision for agent policy for every package policy, if multiple package policies use the same agent policy we could avoid that (expensive) operation
• Stop fetching all ES assets and only fetch the needed one (for compiling agent input and streams we probably only need handlebars template)
• Better package info cache

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

• Group bumping agent policy revision, we are bumping revision for agent policy for every package policy, if multiple package policies use the same agent policy we could avoid that (expensive) operation
• Stop fetching all ES assets and only fetch the needed one (for compiling agent input and streams we probably only need handlebars template)

These two options seem like the most immediately actionable, so I think we should pursue these in the short term.

• Better package info cache

I think we'd need to spend some time thinking about what this means and what kind of improvements we can make. I suspect the improvements above will be sufficient to optimize the integration upgrade process, but we should measure along the way to be sure.

[cmacknz]

Do we have any kind of profiling or measurement of where the performance problem is coming from?

The suggestions sounds reasonable but it's not clear that we know they'll solve the problem or what the specific problems are yet. I would prefer to start with measurement and instrumentation so we can tell when we've improved things and that we are focusing on the correct problem.