[CUDA] Isolate unified api to only in cubin launcher #408

tqchen · 2026-01-13T01:51:00Z

This PR isolates out the unified api to be only local to cubin launcher.

Background: it is generally error-prone to mix the driver and runtime API. The particular unified api switch was mainly meant to be used in cubin launcher for a narrow set of cuda versions(around 12.8 ish to 13.0).

However, we would like the most generic macros like TVM_FFI_CHECK_CUDA_ERROR to be specific to runtime API. We should revisit if we should simply deprecate driver API usages for better maintainability.

This PR isolates out the unified api to be only local to cubin launcher. Background: it is generally hard to mix the driver and runtime API. The particular unified api switch was mainly meant to be used in cubin launcher for a narrow set of cuda versions(around 12.8 ish to 13.0). However, we would like the most generic macros like TVM_FFI_CHECK_CUDA_ERROR to be specific to runtime API. We should revisit if we should simply deprecate driver API usages for better maintainability.

gemini-code-assist · 2026-01-13T01:51:30Z

Summary of Changes

Hello @tqchen, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the CUDA error checking macros to provide a clearer distinction between unified API error handling and standard CUDA runtime API error handling. By introducing a specialized macro for the cubin_launcher component and defining a generic runtime-specific macro, the change aims to improve code maintainability and ensure that error checks are applied appropriately based on the context of CUDA API usage.

Highlights

Macro Renaming and Specialization: The existing TVM_FFI_CHECK_CUDA_ERROR macro within unified_api.h has been renamed to TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR. This new macro is now explicitly designated for error checking within the cubin launcher, which may involve a mix of CUDA driver and runtime APIs.
New Generic CUDA Runtime Error Macro: A new, purely CUDA runtime API-focused TVM_FFI_CHECK_CUDA_ERROR macro has been introduced in base.h. This macro uses standard cudaError_t and cudaSuccess checks, ensuring a consistent and predictable error handling mechanism for general CUDA runtime operations.
Cubin Launcher Codebase Update: All instances of the old TVM_FFI_CHECK_CUDA_ERROR within cubin launcher-related files (documentation, examples, and core headers like cubin_launcher.h) have been updated to use the newly specialized TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR.
Improved Maintainability and Clarity: This refactoring clearly separates the error checking logic for the unified API (specific to cubin launcher) from the generic CUDA runtime API error checking, enhancing code clarity and maintainability across the project.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request effectively isolates the unified CUDA API to the cubin launcher by renaming TVM_FFI_CHECK_CUDA_ERROR to TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR and introducing a new TVM_FFI_CHECK_CUDA_ERROR for runtime-specific API calls. This is a good refactoring that improves code clarity and maintainability. My review focuses on improving consistency across the codebase. In several places, the kernel launch and error checking can be combined into a single line, a pattern that is already present in some of the updated files. Applying this consistently will make the code more concise.

gemini-code-assist · 2026-01-13T01:52:58Z

examples/cubin_launcher/dynamic_cubin/src/lib_dynamic.cc

  // Launch kernel
  tvm::ffi::cuda_api::ResultType result = g_add_one_kernel->Launch(args, grid, block, stream);
-  TVM_FFI_CHECK_CUDA_ERROR(result);
+  TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(result);


For conciseness, you could combine the kernel launch on line 87 and this error check into a single line. This would also remove the need for the result variable. Other parts of this PR follow this more concise pattern (e.g., in docs/guides/cubin_launcher.rst).

For example:

TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(g_add_one_kernel->Launch(args, grid, block, stream));

This would replace lines 87 and 88.

gemini-code-assist · 2026-01-13T01:52:58Z

examples/cubin_launcher/dynamic_cubin/src/lib_dynamic.cc

  // Launch kernel
  tvm::ffi::cuda_api::ResultType result = g_mul_two_kernel->Launch(args, grid, block, stream);
-  TVM_FFI_CHECK_CUDA_ERROR(result);
+  TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(result);


Similar to my other comment, you can combine the kernel launch on line 127 and this error check into one line for better readability and conciseness.

TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(g_mul_two_kernel->Launch(args, grid, block, stream));

This would replace lines 127 and 128.

gemini-code-assist · 2026-01-13T01:52:58Z

examples/cubin_launcher/embedded_cubin/cpp_embed/src/lib_embedded.cc

  // Launch kernel
  tvm::ffi::cuda_api::ResultType result = kernel.Launch(args, grid, block, stream);
-  TVM_FFI_CHECK_CUDA_ERROR(result);
+  TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(result);


To improve conciseness, consider combining the kernel launch on line 75 with this error check. This pattern is used in other files in this PR, for example in examples/cubin_launcher/example_nvrtc_cubin.py.

TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(kernel.Launch(args, grid, block, stream));

gemini-code-assist · 2026-01-13T01:52:58Z

examples/cubin_launcher/embedded_cubin/cpp_embed/src/lib_embedded.cc

  // Launch kernel
  tvm::ffi::cuda_api::ResultType result = kernel.Launch(args, grid, block, stream);
-  TVM_FFI_CHECK_CUDA_ERROR(result);
+  TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(result);


Similar to my other comment, you can combine the kernel launch on line 114 and this error check into a single statement to make the code more concise.

TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(kernel.Launch(args, grid, block, stream));

gemini-code-assist · 2026-01-13T01:52:58Z

examples/cubin_launcher/embedded_cubin/embed_with_tvm_ffi/src/lib_embedded.cc

  // Launch kernel
  tvm::ffi::cuda_api::ResultType result = kernel.Launch(args, grid, block, stream);
-  TVM_FFI_CHECK_CUDA_ERROR(result);
+  TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(result);


For conciseness, you could combine the kernel launch on line 72 and this error check into a single line. This would also remove the need for the result variable. Other parts of this PR follow this pattern.

TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(kernel.Launch(args, grid, block, stream));

gemini-code-assist · 2026-01-13T01:52:58Z

examples/cubin_launcher/embedded_cubin/embed_with_tvm_ffi/src/lib_embedded.cc

  // Launch kernel
  tvm::ffi::cuda_api::ResultType result = kernel.Launch(args, grid, block, stream);
-  TVM_FFI_CHECK_CUDA_ERROR(result);
+  TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(result);


Similar to my other comment, you can combine the kernel launch on line 111 and this error check into one line for better readability and conciseness.

TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(kernel.Launch(args, grid, block, stream));

gemini-code-assist · 2026-01-13T01:52:58Z

examples/cubin_launcher/embedded_cubin/include_bin2c/src/lib_embedded.cc

  // Launch kernel
  tvm::ffi::cuda_api::ResultType result = kernel.Launch(args, grid, block, stream);
-  TVM_FFI_CHECK_CUDA_ERROR(result);
+  TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(result);


To improve conciseness, consider combining the kernel launch on line 72 with this error check. This pattern is used in other files in this PR.

TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(kernel.Launch(args, grid, block, stream));

gemini-code-assist · 2026-01-13T01:52:58Z

examples/cubin_launcher/embedded_cubin/include_bin2c/src/lib_embedded.cc

  // Launch kernel
  tvm::ffi::cuda_api::ResultType result = kernel.Launch(args, grid, block, stream);
-  TVM_FFI_CHECK_CUDA_ERROR(result);
+  TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(result);


Similar to my other comment, you can combine the kernel launch on line 111 and this error check into a single statement to make the code more concise.

TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(kernel.Launch(args, grid, block, stream));

gemini-code-assist · 2026-01-13T01:52:58Z

tests/python/test_cubin_launcher.py

+  auto result = g_kernel_add_one->Launch(args, grid, block, stream);
+  TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(result);


For conciseness, you can combine these two lines into one, as has been done in other example files in this PR. This also removes the need for the result variable.

Suggested change

auto result = g_kernel_add_one->Launch(args, grid, block, stream);

TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(result);

TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(g_kernel_add_one->Launch(args, grid, block, stream));

gemini-code-assist · 2026-01-13T01:52:59Z

tests/python/test_cubin_launcher.py

+  auto result = g_kernel_mul_two->Launch(args, grid, block, stream);
+  TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(result);


Similar to my other comment, these two lines can be combined into a single statement for better readability and conciseness.

Suggested change

auto result = g_kernel_mul_two->Launch(args, grid, block, stream);

TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(result);

TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(g_kernel_mul_two->Launch(args, grid, block, stream));

gemini-code-assist bot reviewed Jan 13, 2026

View reviewed changes

junrushao added 2 commits January 12, 2026 17:56

mute clangd

1a49c51

clang-tidy: only detect ./src and ./tests

5f880d2

junrushao approved these changes Jan 13, 2026

View reviewed changes

junrushao merged commit 10cb004 into apache:main Jan 13, 2026
8 checks passed

		auto result = g_kernel_add_one->Launch(args, grid, block, stream);
		TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(result);

		auto result = g_kernel_mul_two->Launch(args, grid, block, stream);
		TVM_FFI_CHECK_CUBIN_LAUNCHER_CUDA_ERROR(result);

[CUDA] Isolate unified api to only in cubin launcher #408

[CUDA] Isolate unified api to only in cubin launcher #408

Uh oh!

Conversation

tqchen commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot commented Jan 13, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tqchen commented Jan 13, 2026 •

edited

Loading