Fix Kineto+PTI profiling on BMG

[anmyachev]

I haven't found a workaround with the current version of pti that would work for all benchmarks. For example, setting the variable PTI_DEVICE_SYNC_DELTA=1 fixes all benchmarks except prefix_sums. If also update pti to version 0.12.2, then all benchmarks work.

Note: This problem can also be related to the version of AGAMA. The developer machine has 1099 version (CI runner has 1133 version), it is enough to update pti to version 0.12.2 and everything works.

BMG CI:

• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/15121136044
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/15121382503
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/15122129399
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/15122978565
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/15136209432
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/15139707992
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/15142554799
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/15146126315
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/15163434777
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/15164032760
• https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/15164972851

[anmyachev]

FYI @etiotto @whitneywhtsang gemm benchmark with tensor of pointer doesn't work correctly on bmg https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/15122978565/job/42509360103

[anmyachev]

cutlass on BMG currently doesn't work: https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/15121382503/job/42504328424

[sommerlukas]

Tracked through #4254.

[anmyachev]

Thanks @sommerlukas!

[pbchekin]

Note that inputs.runner_label is not set by default, most like the condition is not met on max1550 (please double check the last run). Potentially you need something like this: ${{ inputs.runner_label || 'max1550' }}.

[anmyachev]

@pbchekin do you have a suggestion how to fix it?

[anmyachev]

I decided to return it as it was, with duplication, but at least it works

[pbchekin]

Just an idea: we can add a new step in the very beginning (after installing python and intel-pti) that does not use default shell (so this code is not executed). In this step, create a file, for example, ~/.env with

PTI_LIBS_DIR=...
source /opt/intel/oneapi/setvars.sh > /dev/null
export LD_LIBRARY_PATH=$PTI_LIBS_DIR:$LD_LIBRARY_PATH;

Then the default shell can be changed to

shell: bash -noprofile --norc -eo pipefail -c "[[ -f ~/.env ]] && source ~/.env; source {0}"

[pbchekin]

If adjusting LD_LIBRARY_PATH is safer in each step, then my suggestion is to create a file and source it in each step

[pbchekin]

If adjusting LD_LIBRARY_PATH is safer in each step, then my suggestion is to create a file and source it in each step

NVM, i see in the last commit you have only one additional line per step, this looks good.

[anmyachev]

BTW, thanks for the idea!