Skip to content

02. Usage Guide

Jump to bottom
github-actions[bot] edited this page Sep 4, 2026 · 16 revisions

This guide walks you through the complete workflow of using TritonParse to analyze Triton kernel compilation processes.

📋 Overview

TritonParse workflow consists of three main steps:

  1. Generate Traces - Capture Triton compilation events
  2. Parse Traces - Process raw logs into structured format
  3. Analyze Results - Visualize and explore using the web interface

🚀 Standard Setup Pattern

All TritonParse workflows follow this pattern:

Initialize Logging

import tritonparse.structured_logging

log_path = "./logs/"
tritonparse.structured_logging.init(log_path, enable_trace_launch=True)

Parse Traces

import tritonparse.parse.utils

tritonparse.parse.utils.unified_parse(
    source=log_path,
    out="./parsed_output",
    overwrite=True
)

Alternative - Command Line:

tritonparseoss parse ./logs/ --out ./parsed_output

🚀 Step 1: Generate Triton Trace Files

Example: Complete Triton Kernel

Here's a complete example showing how to trace a Triton kernel:

import torch
import triton
import triton.language as tl
import tritonparse.structured_logging
import tritonparse.parse.utils

# Initialize logging (see Standard Setup Pattern above)
log_path = "./logs/"
tritonparse.structured_logging.init(log_path, enable_trace_launch=True)

@triton.jit
def add_kernel(
    a_ptr,
    b_ptr,
    c_ptr,
    n_elements,
    BLOCK_SIZE: tl.constexpr,
):
    pid = tl.program_id(axis=0)
    block_start = pid * BLOCK_SIZE
    offsets = block_start + tl.arange(0, BLOCK_SIZE)
    mask = offsets < n_elements

    a = tl.load(a_ptr + offsets, mask=mask)
    b = tl.load(b_ptr + offsets, mask=mask)
    c = a + b
    tl.store(c_ptr + offsets, c, mask=mask)

def tensor_add(a, b):
    n_elements = a.numel()
    c = torch.empty_like(a)
    BLOCK_SIZE = 1024
    grid = (triton.cdiv(n_elements, BLOCK_SIZE),)
    add_kernel[grid](a, b, c, n_elements, BLOCK_SIZE)
    return c

# Example usage
if __name__ == "__main__":
    device = "cuda" if torch.cuda.is_available() else "cpu"
    a = torch.randn(1024, 1024, device=device, dtype=torch.float32)
    b = torch.randn(1024, 1024, device=device, dtype=torch.float32)

    # Execute kernel (this will be traced)
    c = tensor_add(a, b)

# Parse the generated logs (see Standard Setup Pattern above)
    tritonparse.parse.utils.unified_parse(source=log_path, out="./parsed_output", overwrite=True)

💡 Tip: See tests/test_add.py in the repository for a complete runnable example.

PyTorch 2.0+ Compiled Functions

For PyTorch 2.0+ with torch.compile:

import torch
import tritonparse.structured_logging
import tritonparse.parse.utils

# Initialize logging
log_path = "./logs/"
tritonparse.structured_logging.init(log_path, enable_trace_launch=True)

def simple_add(a, b):
    return a + b

# Compile the function
compiled_add = torch.compile(simple_add)

# Execute (this will be traced)
device = "cuda"
a = torch.randn(1024, 1024, device=device, dtype=torch.float32)
b = torch.randn(1024, 1024, device=device, dtype=torch.float32)
result = compiled_add(a, b)

# Parse logs
tritonparse.parse.utils.unified_parse(source=log_path, out="./parsed_output", overwrite=True)

💡 Note: Set TORCHINDUCTOR_FX_GRAPH_CACHE=0 to ensure compilation happens every run during testing.

Environment Variables

Configure TritonParse behavior with these environment variables:

Variable Description Example
TRITON_TRACE Trace output directory "./logs/"
TRITON_TRACE_LAUNCH Enable launch tracing ("1" or "0") "1"
TORCHINDUCTOR_RUN_JIT_POST_COMPILE_HOOK Required for torch.compile kernel launch tracing "1"
TRITONPARSE_MORE_TENSOR_INFORMATION Collect tensor statistics (min/max/mean/std) "1"
TRITONPARSE_SAVE_TENSOR_BLOBS Save actual tensor data as blob files "1"
TRITONPARSE_DEBUG Enable debug logging "1"
TRITON_TRACE_COMPRESSION Compression format ("none", "gzip", "clp") "gzip"
TRITONPARSE_KERNEL_ALLOWLIST Filter specific kernels (comma-separated patterns) "my_kernel*,important_*"
TRITONPARSE_DUMP_SASS Enable NVIDIA SASS dump (slow) "1"
TORCHINDUCTOR_FX_GRAPH_CACHE Disable FX graph cache (for testing) "0"

💡 See Environment Variables Reference for complete documentation of all variables.

Usage:

export TRITON_TRACE="./logs/"
export TRITON_TRACE_LAUNCH="1"
# Required for torch.compile kernels
export TORCHINDUCTOR_RUN_JIT_POST_COMPILE_HOOK="1"
# Optional: collect tensor statistics for better reproducers
export TRITONPARSE_MORE_TENSOR_INFORMATION="1"
export TORCHINDUCTOR_FX_GRAPH_CACHE=0

python your_script.py

Running Your Code

# Run with environment variables
TORCHINDUCTOR_FX_GRAPH_CACHE=0 python your_script.py

Expected Output:

Triton kernel executed successfully
Torch compiled function executed successfully
tritonparse log file list: /tmp/tmp1gan7zky/log_file_list.json
INFO:tritonparse:Copying parsed logs from /tmp/tmp1gan7zky to /scratch/findhao/tritonparse/tests/parsed_output

================================================================================
📁 TRITONPARSE PARSING RESULTS
================================================================================
📂 Parsed files directory: /scratch/findhao/tritonparse/tests/parsed_output
📊 Total files generated: 2

📄 Generated files:
--------------------------------------------------
   1. 📝 dedicated_log_triton_trace_findhao__mapped.ndjson.gz (7.2KB)
   2. 📝 log_file_list.json (181B)
================================================================================
✅ Parsing completed successfully!
================================================================================

🔧 Step 2: Parse Trace Files

Python API

import tritonparse.parse.utils

# Basic parsing
tritonparse.parse.utils.unified_parse(
    source="./logs/",           # Input directory with raw logs
    out="./parsed_output",      # Output directory for processed files
    overwrite=True              # Overwrite existing output
)

# Advanced options
tritonparse.parse.utils.unified_parse(
    source="./logs/",
    out="./parsed_output",
    overwrite=True,
    rank=0,                     # Analyze specific rank (for multi-GPU)
    all_ranks=False,            # Or analyze all ranks
    verbose=True,               # Enable verbose logging
    kernel_allowlist="matmul*,*attention*",
)

Command Line Interface

# Basic usage
tritonparse parse ./logs/ --out ./parsed_output

# Alternative: using python -m
python -m tritonparse parse ./logs/ --out ./parsed_output

# With options
tritonparseoss parse ./logs/ --out ./parsed_output --overwrite --verbose

# Multi-GPU: parse specific rank
tritonparseoss parse ./logs/ --out ./parsed_output --rank 0

# Multi-GPU: parse all ranks
tritonparseoss parse ./logs/ --out ./parsed_output --all-ranks

# Parse only matching kernels from an existing trace
tritonparseoss parse ./logs/ --out ./parsed_output \
  --kernel-allowlist 'matmul*,*attention*'

--kernel-allowlist accepts comma-separated fnmatch patterns and keeps each matching kernel's complete compilation, launch, and autotune event group. It is a parse-time option for existing logs; it does not implicitly read the trace-time TRITONPARSE_KERNEL_ALLOWLIST environment variable. A filter that matches no kernel reports an error with a bounded sample of available names. For MAST sources, filtering reduces parse work and report size but does not reduce the raw trace download.

🌐 Step 3: Analyze with Web Interface

Option A: Online Interface (Recommended)

  1. Visit the live tool: https://meta-pytorch.org/tritonparse/

  2. Load your trace files:

    • Click "Browse Files" or drag-and-drop
    • Select .gz files from your parsed_output directory
    • Or select .ndjson files from your logs directory

  3. Explore the visualization:

    • Kernel Overview Tab: Kernel metadata, call stack, IR links
    • IR Code View Tab: Side-by-side IR viewing with line mapping

Option B: Local Development Interface

For contributors or custom deployments:

cd website
npm install
npm run dev

Access at http://localhost:5173

Supported File Formats

Format Description Source Mapping Recommended
.gz Compressed parsed traces ✅ Yes ✅ Yes
.ndjson Raw trace logs ❌ No ⚠️ Basic use only

Note: .ndjson files don't contain source code mappings between IR stages and launch diffs. Always use .gz files for full functionality.

📊 Understanding the Results

Kernel Overview

The overview page shows:

  • Kernel Information: Name, hash, grid/block sizes
  • Compilation Metadata: Device, compile time, memory usage
  • Call Stack: Python source code that triggered compilation
  • IR Navigation: Links to different IR representations
  • Launch Diff: Launch parameters that changed across different launches of the same kernel

IR Code View

The IR code view offers:

  • Side-by-side IR viewing: Compare different compilation stages
  • Synchronized highlighting: Click a line to see corresponding lines in other IRs
  • Source mapping: Trace transformations across compilation pipeline

File Diff View

Compare kernels from two different trace files side-by-side:

  • Cross-trace comparison: Validate optimizations, track kernel evolution, debug differences
  • Flexible modes: Single IR focus or all IRs simultaneously
  • Customizable diff: Ignore whitespace, word/line-level, context control
  • URL shareable: ?view=file_diff&json_url=trace1.gz&json_b_url=trace2.gz

💡 Tip: See the Web Interface Guide for detailed File Diff documentation.

IR Stages Explained

Stage Description When Generated
TTGIR Triton GPU IR - High-level GPU operations After Triton frontend
TTIR Triton IR - Language-level operations After parsing
LLIR LLVM IR - Low-level operations After LLVM conversion
PTX NVIDIA PTX Assembly For NVIDIA GPUs
AMDGCN AMD GPU Assembly For AMD GPUs

🚀 Launch Analysis

TritonParse can analyze kernel launch parameters to identify variations and commonalities across different launches of the same kernel. This is useful for understanding how dynamic shapes or other factors affect kernel execution.

How it Works

  1. Enable Launch Tracing: You must enable launch tracing during the trace generation step. This is done by passing enable_trace_launch=True to tritonparse.structured_logging.init().
  2. Parsing: During the parsing step (tritonparse.parse.utils.unified_parse), TritonParse will automatically group all launches for each kernel.
  3. Launch Diff Event: A new event of type launch_diff is generated for each kernel. This event contains:
    • total_launches: The total number of times the kernel was launched.
    • diffs: A dictionary showing which launch parameters (e.g., grid_x, grid_y) changed across launches and what their different values were.
    • sames: A dictionary showing which launch parameters remained constant across all launches.
    • launch_index_map: A mapping from the launch index to the original line number in the trace file.

Example launch_diff Event

{
  "event_type": "launch_diff",
  "hash": "...",
  "name": "triton_kernel_name",
  "total_launches": 10,
  "launch_index_map": { "0": 15, "1": 25, ... },
  "diffs": {
    "grid_x": [1024, 2048]
  },
  "sames": {
    "grid_y": 1,
    "grid_z": 1,
    "stream": 7
  }
}

This example shows that grid_x varied between 1024 and 2048 across 10 launches, while other parameters remained the same.

🔧 Reproducer - Generate Standalone Kernel Scripts

TritonParse can automatically generate standalone Python scripts that reproduce specific kernel executions. Useful for debugging, sharing test cases, and isolating performance issues.

💡 See Reproducer Guide for comprehensive documentation including advanced features, custom templates, and troubleshooting.

Quick Start

Command Line:

# Generate reproducer for first launch event
tritonparseoss reproduce ./parsed_output/trace.ndjson --line 1 --out-dir repro_output

# Using compressed files
tritonparseoss reproduce ./parsed_output/trace.ndjson.gz --line 5 --out-dir my_repro

# With custom template
tritonparseoss reproduce trace.ndjson --line 1 --template /path/to/my_template.py

Python API:

from tritonparse.reproducer.orchestrator import reproduce

result = reproduce(
    input_path="./parsed_output/trace.ndjson",
    line_index=0,                    # Which launch event (0-based index)
    out_dir="repro_output",
    template="example"               # Built-in template
)

print(f"Script: {result['repro_script']}")
print(f"Context: {result['repro_context']}")

Generated Files

repro_output/<kernel_name>/
├── repro_<timestamp>.py              # Standalone executable script
├── repro_context_<timestamp>.json    # Kernel metadata and parameters
└── <hash>.bin                        # Tensor blobs (if enabled during tracing)

Parameters

Parameter Description Default
input Path to NDJSON trace file (.ndjson or .ndjson.gz) Required
--line Line index (0-based) of launch event 0
--out-dir Output directory repro_output/<kernel>/
--template Template name or path example

Tensor Data Strategies

The reproducer reconstructs tensors using one of these strategies:

1. Blob Files (Highest Fidelity)

# Enable during tracing
tritonparse.structured_logging.init(
    "./logs/",
    enable_trace_launch=True,
    enable_tensor_blob_storage=True  # Save actual tensor data
)

2. Statistical Reconstruction (Good Approximation)

  • Uses saved mean, std, min, max to generate similar data
  • Matches shape, dtype, device of original

3. Random Data (Fallback)

  • Random generation matching only shape and dtype

Common Use Cases

Bug Isolation:

tritonparseoss reproduce trace.ndjson --line 42 --out-dir bug_repro
cd bug_repro && python repro_*.py

Performance Benchmarking:

tritonparseoss reproduce trace.ndjson --line 1 --out-dir benchmark
# Modify script to add timing

Kernel Comparison:

tritonparseoss reproduce trace_v1.ndjson --line 1 --out-dir v1
tritonparseoss reproduce trace_v2.ndjson --line 1 --out-dir v2
# Compare outputs and performance

Custom Templates

Create your own template:

# my_template.py
"""Custom reproducer template"""
import torch
# {{KERNEL_IMPORT_PLACEHOLDER}}

if __name__ == "__main__":
    # {{KERNEL_INVOCATION_PLACEHOLDER}}
    print("Custom execution complete!")

Available Placeholders:

  • {{KERNEL_IMPORT_PLACEHOLDER}} - Kernel imports
  • {{KERNEL_INVOCATION_PLACEHOLDER}} - Launch code
  • {{KERNEL_SYSPATH_PLACEHOLDER}} - System path setup
  • {{JSON_FILE_NAME_PLACEHOLDER}} - Context JSON filename

Usage:

tritonparseoss reproduce trace.ndjson --line 1 --template /path/to/my_template.py

Advanced:

Advanced: Custom Types Support

For triton_kernels projects:

from triton_kernels.tensor import Tensor, Storage, StridedLayout
# Reproducer automatically handles these if triton_kernels is installed

If not installed:

RuntimeError: Optional dependency 'triton_kernels.tensor' is not installed

Solution: pip install triton_kernels

🔄 Initialization Methods Comparison

TritonParse supports multiple initialization methods:

Method 1: Direct Initialization (Recommended)

tritonparse.structured_logging.init(
    trace_folder="./logs/",
    enable_trace_launch=True,
    enable_more_tensor_information=True,  # Collect tensor stats
)

Full Parameters:

Parameter Type Default Description
trace_folder str None Directory for trace files
enable_trace_launch bool False Enable launch event tracing for ALL launches
enable_trace_launch_within_profiling bool False Enable launch tracing only during torch.profiler RECORD phase
enable_more_tensor_information bool False Collect tensor statistics (min/max/mean/std)
enable_sass_dump bool False Enable NVIDIA SASS dump (slow)
enable_full_python_source Optional[bool] None Capture the whole Python file instead of only the kernel function. None defers to TRITON_FULL_PYTHON_SOURCE
enable_tensor_blob_storage bool False Save actual tensor data as blob files
tensor_storage_quota int 100GB Maximum storage for tensor blobs
compression str None Compression format ("none", "gzip", "clp")
tensor_save_skip_runs int None Skip blob saving for first N kernel runs
tensor_save_max_runs int None Save blobs for at most N runs after skipping

💡 See Python API Reference for complete API documentation.

Method 2: Environment Variables

⚠️ OSS Activation Required: Even when using environment variables, you must call tritonparse.structured_logging.init() in your code to activate tracing. See Environment Variables Reference for details.

import os
os.environ["TRITON_TRACE"] = "./logs/"
os.environ["TRITON_TRACE_LAUNCH"] = "1"

tritonparse.structured_logging.init()  # Required in OSS

Or from shell:

export TRITON_TRACE="./logs/"
export TRITON_TRACE_LAUNCH="1"
python my_script.py

Method 3: Context Manager (TritonParseManager)

For simplified workflow with automatic cleanup:

from tritonparse.context_manager import TritonParseManager

with TritonParseManager(
    enable_trace_launch=True,
    out="./parsed_output",
    overwrite=True,
) as manager:
    # Your kernel code here
    result = my_kernel(input_tensor)

# Logs are automatically parsed on context exit
print(f"Parsed output: {manager.output_link}")

TritonParseManager Parameters:

Parameter Type Default Description
enable_trace_launch bool False Enable launch event tracing
enable_trace_launch_within_profiling bool False Forwarded to init(); enable launch tracing only during torch.profiler RECORD phase
enable_more_tensor_information bool False Forwarded to init(); collect tensor statistics (min/max/mean/std)
split_inductor_compilations bool True Split output by compilation IDs
enable_tensor_blob_storage bool False Save tensor blob data
enable_sass_dump Optional[bool] None Forwarded to init(); None defers to TRITONPARSE_DUMP_SASS
enable_full_python_source Optional[bool] None Forwarded to init(); None defers to TRITON_FULL_PYTHON_SOURCE
tensor_storage_quota int None Storage quota for tensor blobs
compression Optional[str] None Forwarded to init(); None defers to TRITON_TRACE_COMPRESSION
tensor_save_skip_runs int None Skip blob saving for first N kernel runs
tensor_save_max_runs int None Save blobs for at most N runs after skipping
log_dir str None Directory for raw trace logs (kept after parsing if provided)
keep_logs bool False Keep temporary log directory after parsing
**parse_kwargs Additional arguments passed to unified_parse()

Workflow:

  1. On __enter__: Creates temporary directory and initializes logging
  2. Inside with block: Your code runs with tracing enabled
  3. On __exit__: Automatically parses logs and cleans up

Comparison Table

Method When to Use Pros Cons
init(path, ...) Direct control in code Explicit, all options Manual parsing required
init_with_env() Environment-based config Flexible, CI/CD friendly Requires env setup
TritonParseManager Quick experiments Auto cleanup, simple Less control

🔍 Advanced Features

Kernel Filtering

Filter while tracing to reduce the raw log size:

export TRITONPARSE_KERNEL_ALLOWLIST="my_kernel*,important_*"
python your_script.py

Filter kernels while parsing an existing raw trace:

tritonparseoss parse ./logs/ --kernel-allowlist 'my_kernel*,important_*'

Both forms use comma-separated fnmatch patterns, but they are independent. The parser never implicitly reads the trace-time environment variable. Parse filtering keeps the complete hash/autotune-session closure, so a selected autotune kernel can retain sibling configurations whose names do not directly match. Empty or comma-only values disable filtering, and shell quoting is recommended for patterns containing *, ?, or [].

The context manager accepts the parse option through **parse_kwargs:

with TritonParseManager(
    log_dir="./logs/",
    kernel_allowlist="my_kernel*,important_*",
) as manager:
    result = my_kernel(input_tensor)

This context-manager argument filters the parsed report on exit; it does not reduce trace capture inside the with block.

Trace File Naming

TritonParse writes one raw trace file per process. Filenames have the form:

dedicated_log_triton_trace_{user}_rank_{N}_pid_{PID}_host_{HOST}_.ndjson      # ranked (post dist.init)
dedicated_log_triton_trace_{user}_rank_none_pid_{PID}_host_{HOST}_.ndjson     # no-rank (pre dist.init or single-GPU)

The _rank_*_ token is always present, with * either an integer rank or the literal none. This lets MAST/lg downloads use a single rank-aware regex LOG_PREFIX.*_rank_(N|none)_ to fetch rank N and the matching pre-init no-rank file in one call. The pid_{PID}_ suffix isolates each subprocess so concurrent inductor compile workers cannot corrupt each other's output. The host_{HOST}_ suffix is needed in multi-host distributed jobs because PIDs alone are not globally unique across machines.

Older trace files without one or more of these suffixes (...{user}_rank_{N}_.ndjson, ...{user}_pid_{PID}_.ndjson, ...{user}_.ndjson) are still parseable locally — backward compatibility is preserved. Note that legacy traces lacking the _rank_*_ token are NOT pulled by MAST --rank N or --rank none downloads; use --all-ranks to include them.

By default, no-rank trace files (kernels compiled before torch.distributed.init_process_group) whose (host, pid) matches a ranked file are automatically merged into that rank's parsed output. Pass --no-pre-init-attribution to disable this and keep the buckets separate for debugging.

Multi-GPU Analysis

Parse all ranks:

tritonparse.parse.utils.unified_parse(
    source="./logs/",
    out="./parsed_output",
    all_ranks=True
)

Parse specific rank:

tritonparse.parse.utils.unified_parse(
    source="./logs/",
    out="./parsed_output",
    rank=1
)

Command line:

tritonparseoss parse ./logs/ --out ./parsed_output --all-ranks
tritonparseoss parse ./logs/ --out ./parsed_output --rank 1

# Disable pre-init kernel attribution (debug pre/post-init boundary):
tritonparseoss parse ./logs/ --out ./parsed_output --rank 0 --no-pre-init-attribution

Diff — Compare Compilation Events

Compare compilation events within a single trace or across two traces:

# List compilations in a file
tritonparseoss diff trace.ndjson --list

# Compare events 0 and 1 within one file
tritonparseoss diff trace.ndjson --events 0,1

# Compare a specific kernel across two files
tritonparseoss diff file_a.ndjson file_b.ndjson --kernel matmul_kernel --events 0,0

# Full trace comparison (all kernels)
tritonparseoss diff file_a.ndjson file_b.ndjson --trace

# Compare tensor values with tolerance
tritonparseoss diff trace.ndjson --events 0,1 --tensor-values --atol 1e-4

💡 See Python API Reference for the full list of options.

Bisect — Find Regression-Causing Commits

Bisect Triton, LLVM, or PyTorch commits to find the change that introduced a regression:

# Triton bisect
tritonparseoss bisect --triton-dir /path/to/triton --test-script test.sh --good abc123 --bad def456

# Full 4-phase workflow (Triton → LLVM)
tritonparseoss bisect --triton-dir /path/to/triton --test-script test.sh --good abc --bad def --commits-csv

# PyTorch bisect
tritonparseoss bisect --target torch --torch-dir /path/to/pytorch --test-script test.sh --good abc --bad def

💡 See Python API Reference for the full list of options.

Compat-Build — LLVM Compatibility Maps

Build a compatibility map between Triton and LLVM commits for an LLVM bump:

# Build compatibility map
tritonparseoss compat-build --triton-dir /path/to/triton --llvm-bump-commit abc123

# With AI-assisted fixes
tritonparseoss compat-build --triton-dir /path/to/triton --llvm-bump-commit abc123 --ai

💡 See Python API Reference for the full list of options.

TritonParse Wiki

Clone this wiki locally