Skip to content

doc: abi overview #402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
junrushao merged 2 commits into from
Jan 13, 2026
+977 −876
Merged

doc: abi overview #402

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3e3bcb1
doc: abi overview
junrushao Jan 12, 2026
e38261d
save
junrushao Jan 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
462 changes: 0 additions & 462 deletions docs/concepts/abi_overview.md
View file
Open in desktop

This file was deleted.

551 changes: 551 additions & 0 deletions docs/concepts/abi_overview.rst
View file
Open in desktop

Large diffs are not rendered by default.

42 changes: 15 additions & 27 deletions docs/concepts/any.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,7 @@ values of a wide variety of types, including primitives, objects, and strings.
Unlike ``std::any``, it is designed for zero-copy inter-language exchange without RTTI,
featuring a fixed 16-byte layout with built-in reference counting and ownership semantics.

This tutorial covers everything you need to know about :cpp:class:`~tvm::ffi::Any` and :cpp:class:`~tvm::ffi::AnyView`:
common usage patterns, ownership semantics, and memory layout.

This tutorial covers common usage patterns, ownership semantics, and memory layout.

Common Usage
------------
Expand Down Expand Up @@ -169,6 +167,8 @@ Compare with ``nullptr`` to check for ``None``:
}


.. _any-ownership:

Ownership
---------

Expand All @@ -195,8 +195,8 @@ The core distinction between :cpp:class:`tvm::ffi::Any` and
- Function inputs
- Return values, storage

Code Examples
~~~~~~~~~~~~~~
Examples
~~~~~~~~

:cpp:class:`~tvm::ffi::AnyView` is a lightweight, non-owning view. Copying it simply
copies 16 bytes with no reference count updates, making it ideal for passing arguments without overhead:
Expand Down Expand Up @@ -245,25 +245,9 @@ Destruction Semantics in C

In C, which lacks RAII, you must manually destroy :cpp:class:`~tvm::ffi::Any` objects
by calling :cpp:func:`TVMFFIObjectDecRef` for heap-allocated objects.
Destroying an :cpp:class:`~tvm::ffi::AnyView` is effectively a no-op - just clear its contents.

.. code-block:: cpp

void destroy_any(TVMFFIAny* any) {
if (any->type_index >= kTVMFFIStaticObjectBegin) {
// Decrement the reference count of the heap-allocated object
TVMFFIObjectDecRef(any->v_obj);
}
*any = (TVMFFIAny){0};
}

In contrast, destroying an :cpp:class:`~tvm::ffi::AnyView` is effectively a no-op - just clear its contents.

.. code-block:: cpp

void destroy_any_view(TVMFFIAny* any_view) {
*any_view = (TVMFFIAny){0};
}

See :ref:`abi-destruct-any` for C code examples.

Layout
------
Expand Down Expand Up @@ -310,6 +294,8 @@ It is effectively a layout-stable 16-byte tagged union.
* The first 4 bytes (:cpp:member:`TVMFFIAny::type_index`) serve as a tag identifying the stored type.
* The last 8 bytes hold the actual value - either stored inline for atomic types (e.g., ``int64_t``, ``float64``, ``void*``) or as a pointer to a heap-allocated object.

.. _any-atomic-types:

Atomic Types
~~~~~~~~~~~~

Expand Down Expand Up @@ -371,6 +357,8 @@ Note that raw pointers like :c:struct:`DLTensor* <DLTensor>` and ``char*`` also
These pointers carry no ownership, so the caller must ensure the pointed-to data outlives
the :cpp:class:`~tvm::ffi::AnyView` or :cpp:class:`~tvm::ffi::Any`.

.. _any-heap-allocated-objects:

Heap-Allocated Objects
~~~~~~~~~~~~~~~~~~~~~~

Expand Down Expand Up @@ -431,7 +419,7 @@ inline using **small string optimization**, avoiding heap allocation entirely:
Further Reading
---------------

- **Object system**: :doc:`object_and_class` covers how TVM-FFI objects work, including reference counting and type checking
- **Function system**: :doc:`func_module` covers function calling conventions and the global registry
- **C examples**: :doc:`../get_started/stable_c_abi` demonstrates working with :cpp:class:`TVMFFIAny` directly in C
- **Tensor conversions**: :doc:`tensor` covers how tensors flow through :cpp:class:`~tvm::ffi::Any` and :cpp:class:`~tvm::ffi::AnyView`
- :doc:`object_and_class`: How TVM-FFI objects work, including reference counting and type checking
- :doc:`func_module`: Function calling conventions and the global registry
- :doc:`tensor`: How tensors flow through :cpp:class:`~tvm::ffi::Any` and :cpp:class:`~tvm::ffi::AnyView`
- :doc:`abi_overview`: Low-level C ABI details for working with :cpp:class:`TVMFFIAny` directly
224 changes: 59 additions & 165 deletions docs/concepts/func_module.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,20 +15,20 @@
specific language governing permissions and limitations
under the License.

Function, Exception and Module
==============================
Function and Module
===================

TVM-FFI provides a unified and ABI-stable calling convention that enables
cross-language function calls between C++, Python, Rust, and other languages.
Functions are first-class :doc:`TVM-FFI objects <object_and_class>`.

This tutorial covers everything you need to know about defining, registering,
and calling TVM-FFI functions, their exception handling, and working with modules.
This tutorial covers defining, registering, and calling TVM-FFI functions,
exception handling, and working with modules.

Glossary
--------

TVM-FFI ABI. :cpp:type:`TVMFFISafeCallType`
TVM-FFI ABI, or "Packed Function". :cpp:type:`TVMFFISafeCallType`
A stable C calling convention where every function is represented by a single signature,
which enables type-erased, cross-language function calls.
This calling convention is used across all TVM-FFI function calls at the ABI boundary.
Expand Down Expand Up @@ -190,17 +190,15 @@ to a :py:class:`tvm_ffi.Function` at the ABI boundary. The example below demonst
print(func_add(1, 2))


Exception ABI
-------------
.. _sec:function:

This section describes the exception handling contract in the TVM-FFI Stable C ABI.
Exceptions are first-class citizens in TVM-FFI, and this section specifies:
Function
--------

- How to properly throw exceptions from a TVM-FFI ABI function
- How to check for and propagate exceptions from a TVM-FFI ABI function
.. _sec:function-calling-convention:

TVM-FFI C ABI
~~~~~~~~~~~~~
Calling Convention
~~~~~~~~~~~~~~~~~~

All TVM-FFI functions ultimately conform to the :cpp:type:`TVMFFISafeCallType` signature,
which provides a stable C ABI for cross-language calls. The C calling convention is defined as:
Expand All @@ -220,147 +218,48 @@ specified by ``args`` and ``num_args``.
**Output argument**. The output argument ``result`` is an owning :cpp:type:`tvm::ffi::Any`
that the caller must zero-initialize before the call.

.. important::
The caller must zero-initialize the output argument ``result`` before the call.

**Return value**. The ABI returns an **error code** that indicates:

- ``0``: Success
- ``-1``: Error occurred, retrievable with :cpp:func:`TVMFFIErrorMoveFromRaised`
- ``-2``: Very rare frontend error
- **Error code 0**: Success
- **Error code -1**: Error occurred, retrievable with :cpp:func:`TVMFFIErrorMoveFromRaised`
- **Error code -2**: Very rare frontend error

.. hint::
See :doc:`Any <any>` for more details on the semantics of :cpp:type:`tvm::ffi::AnyView` and :cpp:type:`tvm::ffi::Any`.

Retrieve Errors in C
~~~~~~~~~~~~~~~~~~~~

When a TVM-FFI function returns a non-zero code, it indicates that an error occurred
and a :cpp:class:`tvm::ffi::ErrorObj` is stored in thread-local storage (TLS).
This section shows how to retrieve the error object and print the error message and backtrace.

.. note::

An :cpp:class:`~tvm::ffi::ErrorObj` is a :cpp:class:`~tvm::ffi::Object` with a :cpp:class:`TVMFFIErrorCell` payload
as defined below:

.. code-block:: cpp

typedef struct {
TVMFFIByteArray kind; // Error type (e.g., "ValueError")
TVMFFIByteArray message; // Error message
TVMFFIByteArray backtrace; // Stack trace (most-recent call first)
void (*update_backtrace)(...); // Hook to append/replace backtrace
} TVMFFIErrorCell;

**Print an Error**. The example code below shows how to print an error message and backtrace.

.. code-block:: cpp

#include <tvm/ffi/c_api.h>

void PrintError(TVMFFIObject* err) {
TVMFFIErrorCell* cell = (TVMFFIErrorCell*)((char*)err + sizeof(TVMFFIObject));
fprintf(stderr, "%.*s: %.*s\n", (int)cell->kind.size, cell->kind.data, (int)cell->message.size, cell->message.data);
if (cell->backtrace.size) {
fprintf(stderr, "Backtrace:\n%.*s\n", (int)cell->backtrace.size, cell->backtrace.data);
}
}

The payload of the error object is a :cpp:type:`TVMFFIErrorCell` structure
containing the error kind, message, and backtrace. It can be accessed
by skipping the :cpp:type:`TVMFFIObject` header using pointer arithmetic.

**Retrieve the error object**. When the error code is ``-1``, the error object is stored in TLS
and can be retrieved with :cpp:func:`TVMFFIErrorMoveFromRaised`.
This design is called a **packed function**, because it "packs" all arguments into a single array of type-erased :cpp:type:`tvm::ffi::AnyView`,
and further unifies calling convention across all languages without resorting to JIT compilation.

.. code-block:: cpp

void HandleReturnCode(int rc) {
TVMFFIObject* err = NULL;
if (rc == 0) {
// Success
} else if (rc == -1) {
// Move the raised error from TLS (clears TLS slot)
TVMFFIErrorMoveFromRaised(&err); // now `err` owns the error object
if (err != NULL) {
PrintError(err); // print the error
TVMFFIObjectDecRef(err); // Release the error object
}
} else if (rc == -2) {
// Frontend (e.g., Python) already has an exception set.
// Do not fetch from TLS; consult the frontend's error mechanism.
}
}
More specifically, this mechanism enables the following scenarios:

This function transfers ownership of the error object to the caller and clears the TLS slot.
You must call :cpp:func:`TVMFFIObjectDecRef` to release the object when done to avoid memory leaks.
- **Dynamic languages**. Well-optimized bindings are provided for, e.g. Python, to translate arguments into packed function format, and translate return value back to the host language.
- **Static languages**. Metaprogramming techniques, such as C++ templates, are usually available to directly instantiate packed format on stack, saving the need for dynamic examination.
- **Cross-language callbacks**. Language-agnostic :cpp:class:`tvm::ffi::Function` makes it easy to call between languages without depending on language-specific features such as GIL.

**Rare frontend errors**. Error code ``-2`` is reserved for rare frontend errors. It is returned only
when the C API :cpp:func:`TVMFFIEnvCheckSignals` returns non-zero during execution, indicating that
the Python side has a pending signal requiring attention. In this case, the caller should not fetch
the error object from TLS but instead consult the frontend's error mechanism to handle the exception.
**Performance Implications**. This approach is in practice highly efficient in machine learning workloads.

Raise Errors in C
~~~~~~~~~~~~~~~~~

As part of TVM-FFI's calling convention, returning ``-1`` indicates that an error occurred
and the error object is stored in the TLS slot. The error object can contain arbitrary
user-defined information, such as error messages, backtraces, or Python frame-local variables.

.. hint::
Compiler code generation may use similar patterns to raise errors in generated code.
- In Python/C++ calls, we can get to microsecond level overhead, which is generally similar to overhead for eager mode;
- When both sides of calls are static languages, the overhead will go down to tens of nanoseconds.

The example below sets the TLS error and returns ``-1`` using :cpp:func:`TVMFFIErrorSetRaisedFromCStr`:

.. code-block:: cpp

#include <tvm/ffi/c_api.h>

int __tvm_ffi_my_kernel(void* handle, const TVMFFIAny* args,
int32_t num_args, TVMFFIAny* result) {
// Validate inputs
if (num_args < 2) {
TVMFFIErrorSetRaisedFromCStr("ValueError", "Expected at least 2 arguments");
return -1;
}
// ... kernel implementation ...
return 0;
}

Alternatively, :cpp:func:`TVMFFIErrorSetRaisedFromCStrParts` accepts explicit string lengths,
which is useful when the error kind and message are not null-terminated.

**Propagating errors**. For chains of generated calls, simply propagate return codes—TLS carries
the error details:

.. code-block:: cpp

int outer_function(...) {
int err_code = 0;

err_code = inner_function(...);
if (err_code != 0) goto RAII; // Propagate error; TLS has the details
.. note::
Although we found it less necessary in practice, further link time optimization (LTO) is still theoretically possible
in scenarios where both sides are static languages with a known symbol and linked into a single binary.
In this case, the callee can be inlined into caller side and the stack argument memory can be passed into register passing.

RAII:
// clean up owned resources
return err_code;
}

Function
--------
.. _sec:function-layout:

Layout and ABI
~~~~~~~~~~~~~~

:cpp:class:`tvm::ffi::FunctionObj` stores two call pointers in :cpp:class:`TVMFFIFunctionCell`:

.. code-block:: cpp

typedef struct {
TVMFFISafeCallType safe_call;
void* cpp_call;
} TVMFFIFunctionCell;
- ``safe_call``: Used for cross-ABI function calls; intercepts exceptions and stores them in TLS.
- ``cpp_call``: Used within the same DSO; exceptions are thrown directly for better performance.

``safe_call`` is used for cross-ABI function calls: it intercepts exceptions and stores them in TLS.
``cpp_call`` is used within the same DSO, where exceptions are thrown directly for better performance.
See :ref:`abi-function` for the C struct definition.

.. important::

Expand Down Expand Up @@ -407,40 +306,35 @@ calling convention.
in :c:macro:`TVM_FFI_SAFE_CALL_BEGIN` / :c:macro:`TVM_FFI_SAFE_CALL_END` macros.


C Registry APIs
~~~~~~~~~~~~~~~
Compiler developers commonly need to look up global functions in generated code.
Use :cpp:func:`TVMFFIFunctionGetGlobal` to retrieve a function by name, then call it with :cpp:func:`TVMFFIFunctionCall`.
See :ref:`abi-function` for C code examples.

.. list-table::
:header-rows: 1
:widths: 40 60
.. _sec:exception:

* - C API
- Description
* - :cpp:func:`TVMFFIFunctionGetGlobal`
- Get a function by name; returns an owning handle.
* - :cpp:func:`TVMFFIFunctionSetGlobal`
- Register a function in the global registry.
* - :cpp:func:`TVMFFIFunctionCall`
- Call a function with the given arguments.
Exception
~~~~~~~~~

Compiler developers commonly need to look up global functions in generated code. Use
:cpp:func:`TVMFFIFunctionGetGlobal` to retrieve a function by name, then call it with :cpp:func:`TVMFFIFunctionCall`.
The example below demonstrates how to look up and call a global function in C:
This section describes the exception handling contract in the TVM-FFI Stable C ABI.
Exceptions are first-class citizens in TVM-FFI, and this section specifies:

.. code-block:: cpp
- How to properly throw exceptions from a TVM-FFI ABI function
- How to check for and propagate exceptions from a TVM-FFI ABI function

int LookupAndCall(const char* global_function_name, const TVMFFIAny* args, int num_args, TVMFFIAny* result) {
TVMFFIObject* func = NULL;
int err_code;
if ((err_code = TVMFFIFunctionGetGlobal(global_function_name, &func)) != 0)
goto RAII;
if ((err_code = TVMFFIFunctionCall(func, args, num_args, result)) != 0)
goto RAII;

RAII: // clean up owned resources
if (func != NULL) TVMFFIObjectDecRef(func);
return err_code;
}
When a TVM-FFI function returns a non-zero code, an error occurred.
An :cpp:class:`~tvm::ffi::ErrorObj` is stored in thread-local storage (TLS) and can be retrieved
with :cpp:func:`TVMFFIErrorMoveFromRaised`.

- **Error code -1:** Retrieve the error from TLS, print it, and release via :cpp:func:`TVMFFIObjectDecRef`.
- **Error code -2:** A rare frontend error; consult the frontend's error mechanism instead of TLS.

To raise an error, use :cpp:func:`TVMFFIErrorSetRaisedFromCStr` to set the TLS error and return ``-1``.
For chains of calls, simply propagate return codes - TLS carries the error details.

See :ref:`abi-exception` for C code examples.


.. _sec:module:

Modules
-------
Expand Down Expand Up @@ -560,5 +454,5 @@ Further Reading

- :doc:`any`: How functions are stored in :cpp:class:`~tvm::ffi::Any` containers
- :doc:`object_and_class`: The object system that backs :cpp:class:`~tvm::ffi::FunctionObj`
- :doc:`abi_overview`: Low-level C ABI details for functions and exceptions
- :doc:`../packaging/python_packaging`: Packaging functions for Python wheels
- :doc:`abi_overview`: Low-level ABI details for the function calling convention
Loading