Aggregate CPU and Memory accross all nodes instead of per node to fix AWS jobs

[ronaldngounou]

Overview

Contributes to issue #3843

Contributes to naming inconsistency issue with the AWS jobs, preventing LoadResources to be meaningful as it is not taking the average across all the nodes

2 possibilities were considered to fix:
Option 1: Arrange for the AWS nodes to have well-defined names
Option 2. Change the metrics gathering/display to just average over all the nodes without regard to their names

GCE tests don't have the same issue because they have a more consistent naming for the nodes whereas AWS logs the instance-id in the nodename. It makes it insconsisent to see the resource usage. Therefore, this PR averages CPU/Memory across all the nodes.

In this PR, option 2 was selected.

Updated parseResourceUsageData to take the average accross the nodes

Testing

> go test parser.go parser_test.go
ok      command-line-arguments  0.370s

Deployed perfdash on localhost:8080 after following https://github.com/kubernetes/perf-tests/blob/master/perfdash/README.md

make push
make run

[done] Confirming that CL2 generates the desired result files:

• LoadResources uses parseResourceUsageData and has the ResourceUsageSummary prefix, which corresponds to: https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/perf-tests/3844/pull-perf-tests-ec2-master-scale-performance-100/2028715726881165312/artifacts/ResourceUsageSummary_load_2026-03-03T06:50:12Z.json
  • Job execution: https://gcsweb.k8s.io/gcs/kubernetes-ci-logs/pr-logs/pull/perf-tests/3844/pull-perf-tests-ec2-master-scale-performance-100/2028715726881165312/artifacts/

The artifacts are not containing the new resourceUsageAggregate struct added because this change is not modifying the type resourceUsagePercentiles map[string][]resourceUsages rather, it adds type usageAggregateAtPercentiles map[string]resourceUsageAggregate which is the new struct computed. I am not modifying the struct below, which contains the resources.

perf-tests/perfdash/parser.go

Line 135 in 8d22586

usage[name][percentile] = resourceUsage{cpu, memory}

However, I after running the perfdash locally, I'm able to see the aggregation across all the nodes.

Result after

URL = http://localhost:8080/#/?jobname=gce-5000Nodes&metriccategoryname=E2E&metricname=LoadResources&Resource=CPU_Average&PodName=all-nodes after the PR

Before

• https://perf-dash.k8s.io/#/?jobname=gce-5000Nodes&metriccategoryname=E2E&metricname=LoadResources&PodName=kube-proxy-control-plane-us-east1-b%2Fkube-proxy&Resource=CPU

The all-nodes is more useful for AWS jobs as the namings differ.

[ronaldngounou]

/test pull-perf-tests-ec2-master-scale-performance-5000

[ronaldngounou]

/test pull-perf-tests-ec2-master-scale-performance-100

[ronaldngounou]

Running only ec2-master-scale-performance-100 is sufficient. We only run 5000 nodes test to debug performance issues.

[ronaldngounou]

/assign @mengqiy

[aojea]

Example on what @ronaldngounou is mentioning

GCE https://perf-dash.k8s.io/#/?jobname=gce-100Nodes-master&metriccategoryname=E2E&metricname=LoadResources&PodName=kube-proxy-nodes-us-east1-c%2Fkube-proxy&Resource=CPU

AWS https://perf-dash.k8s.io/#/?jobname=aws-100Nodes&metriccategoryname=E2E&metricname=LoadResources&PodName=i-056392ead8e6be597%2Fkubelet&Resource=CPU

the result is that we can not see in aws a trend, since each execution will create different nodes.

@ronaldngounou can we make this change so it is backward compatible and instead of replacing the existing resources, we add a new "fake" resource that we call "aggregated" or "summary" or something like that?
I can not understand from the code if we can break other reporting that depend on the node names

/assign @wojtek-t @mborsz

[ronaldngounou]

Sounds good @aojea. I will address that note and update my PR.

[ronaldngounou]

@wojtek-t @mborsz PR is ready for review please. Thank you

[ronaldngounou]

/test pull-perf-tests-ec2-master-scale-performance-100

[aojea]

/lgtm

Thanks

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aojea, ronaldngounou
Once this PR has been reviewed and has the lgtm label, please ask for approval from mborsz. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• perfdash/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ronaldngounou]

/test pull-perf-tests-ec2-master-scale-performance-100

[k8s-ci-robot]

New changes are detected. LGTM label has been removed.