Related to #5070 2.5(Propose a mechanism to track performance over time):

• have a repo inside the kokkos organization where to store all the json files: let's call this the database-repo for now
• push there (somehow) all the json files generated by the benchmark for example every time a PR is merged into main branch
• have a script inside database-repo that automatically generates plots every time files are updated

1. When the Performance Test will be run on CI we can add an additional step which will upload all generated json files to database-repo.

• There is a GitHub Action which can be used: https://github.com/marketplace/actions/push-a-file-to-another-repository
• This GA only needs a API_TOKEN_GITHUB to be set in Secrets section of your repository options.

2. After pushing benchmark results to database-repo there will be GitHub Action run in database-repo. The sole purpose of that action will be to run the analyzing software that will generate the required output data.
3. To analyze the benchmark data we can use: https://github.com/bensanmorris/benchmark_monitor

• By default this tool will generate chart with benchmark runs and HTML index file based on the Jinja2 template. This template could be modified to include benchmark context data.
• Update: By default the images will be saved using following format BenchamrkName-metric.png so for example BM_SomeFunction-real_time.png.

4. Generated files can be pushed using GitHub Actions to GitHub Pages. Github Action that can be used: https://github.com/marketplace/actions/deploy-to-github-pages

Example chart:

TODO:

• I will create a quick demo on my repositories