[Test]: Refactor benchmark_geglu with standardized model configs

[noemotiovon]

Introduce benchmark_model_configs.py with canonical LLM architecture profiles (LLAMA_2_7B, LLAMA_3_8B) and two utilities:

• estimate_kernel_bytes_per_token: runtime probe that measures peak GPU memory per token for a given kernel, providing a safe upper bound.
• compute_benchmark_shape: derives batch size and sequence length from available device memory, the model profile, and the measured bytes-per-token, replacing hardcoded shapes.

Add run_speed_benchmark and run_memory_benchmark helpers to utils.py to eliminate repeated triton.testing / _test_memory blocks across scripts. Also add a --model CLI argument to parse_benchmark_script_args so callers can select a model profile at runtime.

Refactor benchmark_dyt.py, benchmark_geglu.py, and benchmark_swiglu.py to:

• Extract a setup factory function to avoid duplicating layer/tensor construction between speed and memory benchmarks.
• Delegate to run_speed_benchmark / run_memory_benchmark.
• Compute x_values, bsz, and related fields dynamically via compute_benchmark_shape instead of hardcoded constants.
• Expand memory benchmark modes from ["full"] to ["full", "forward", "backward"] for consistency with speed benchmarks.

Hardware Type: NVIDIA A100-SXM4-80GB

• run make test to ensure correctness
• run make checkstyle to ensure code style
• run make test-convergence to ensure convergence

[noemotiovon]

Hi @Tcc0403, I’ve reviewed your refactoring proposal for the benchmark. I’ve tried implementing it based on the issue — could we discuss what further optimizations might be needed?
Related issue: #1051

[noemotiovon]

Hi @Tcc0403, I’ve completed the benchmark refactor for element-wise level kernels.

At the moment, it seems that the only device-dependent factor is global memory, so I’ve temporarily removed the previously agreed device-specific checks.

Could you help review the code?

[Tcc0403]

Should we also create a guideline for adding benchmark scripts?

[Tcc0403]

missing a probe run for upperbound?

[noemotiovon]

At first I thought that since hidden_size is a model parameter, it might not need to be probed. However, I now think that for cases where we need to avoid VRAM eviction, we should consistently scale based on x_values. A new probe has already been added here.

At the same time, I also realized that not all kernels are related to seq_len. There may be other types of kernels in the future, and for different kinds of kernels we should define matching configurations. I’d also be happy to implement this part in the future.

[Tcc0403]

I believe input setup and cleanup should be done in estimate_kernel_bytes_per_token, so developers don't have to worry about them.

[noemotiovon]

Thanks for your suggestion!

[noemotiovon]

I think my previous thinking wasn’t quite right. I may need to reorganize my thoughts.

[noemotiovon]

Hi @Tcc0403, I have now completed the revisions based on the review comments above and made some adjustments to the code. First, I want to confirm that our rule is correct: we should scale based on the x_values obtained from the probe.

For the kernels I’m currently handling, each has a single x_value, but there are two situations:

1. The x_value is not a model parameter, for example seq_len. In this case, we can use the probe to compute a reasonable seq_len.

2. The x_value is a model parameter, for example hidden_size. In this case, we probably shouldn’t directly use the model parameter as the x-axis, because hidden_size is fixed within a given model. That means this kind of kernel might not really depend on the specific model. So instead, we could also infer some reasonable hidden_size values through probing. Is this the right approach?