Merge pull request #1152 from PyO3/abi3

Complete abi3 support

Author: kngwyu

.github/workflows/ci.yml

@@ -94,6 +94,10 @@ jobs:
       - if: matrix.python-version != 'pypy3'
         name: Test
         run: cargo test --features "num-bigint num-complex" --target ${{ matrix.platform.rust-target }}
+      # Run tests again, but in abi3 mode
+      - if: matrix.python-version != 'pypy3'
+        name: Test (abi3)
+        run: cargo test --no-default-features --features "abi3,macros" --target ${{ matrix.platform.rust-target }}
 
       - name: Test proc-macro code
         run: cargo test --manifest-path=pyo3-derive-backend/Cargo.toml --target ${{ matrix.platform.rust-target }}

CHANGELOG.md

@@ -12,11 +12,14 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - Drop support for Python 3.5 (as it is now end-of-life). [#1250](https://github.com/PyO3/pyo3/pull/1250)
 
 ### Added
+- Add support for building for CPython limited API. This required a few minor changes to runtime behaviour of of pyo3 `#[pyclass]` types. See the migration guide for full details. [#1152](https://github.com/PyO3/pyo3/pull/1152)
 - Add argument names to `TypeError` messages generated by pymethod wrappers. [#1212](https://github.com/PyO3/pyo3/pull/1212)
 - Add `PyEval_SetProfile` and `PyEval_SetTrace` to FFI. [#1255](https://github.com/PyO3/pyo3/pull/1255)
 - Add context.h functions (`PyContext_New`, etc) to FFI. [#1259](https://github.com/PyO3/pyo3/pull/1259)
 
 ### Changed
+- Change return type `PyType::name()` from `Cow<str>` to `PyResult<&str>`. [#1152](https://github.com/PyO3/pyo3/pull/1152)
+- `#[pyclass(subclass)]` is now required for subclassing from Rust (was previously just required for subclassing from Python). [#1152](https://github.com/PyO3/pyo3/pull/1152)
 - Change `PyIterator` to be consistent with other native types: it is now used as `&PyIterator` instead of `PyIterator<'a>`. [#1176](https://github.com/PyO3/pyo3/pull/1176)
 - Change formatting of `PyDowncastError` messages to be closer to Python's builtin error messages. [#1212](https://github.com/PyO3/pyo3/pull/1212)
 

Cargo.toml

@@ -35,6 +35,10 @@ rustversion = "1.0"
 [features]
 default = ["macros"]
 macros = ["ctor", "indoc", "inventory", "paste", "pyo3cls", "unindent"]
+# Use the Python limited API. See https://www.python.org/dev/peps/pep-0384/ for
+# more.
+abi3 = []
+
 # Optimizes PyObject to Vec conversion and so on.
 nightly = []
 
@@ -43,11 +47,6 @@ nightly = []
 # so that the module can also be used with statically linked python interpreters.
 extension-module = []
 
-# The stable cpython abi as defined in PEP 384. Currently broken with
-# many compilation errors. Pull Requests working towards fixing that
-# are welcome.
-# abi3 = []
-
 [workspace]
 members = [
     "pyo3cls",

build.rs

@@ -557,6 +557,11 @@ fn run_python_script(interpreter: &Path, script: &str) -> Result<String> {
 
 fn get_library_link_name(version: &PythonVersion, ld_version: &str) -> String {
     if cfg!(target_os = "windows") {
+        // Mirrors the behavior in CPython's `PC/pyconfig.h`.
+        if env::var_os("CARGO_FEATURE_ABI3").is_some() {
+            return "python3".to_string();
+        }
+
         let minor_or_empty_string = match version.minor {
             Some(minor) => format!("{}", minor),
             None => String::new(),

examples/rustapi_module/tests/test_datetime.py

@@ -1,6 +1,7 @@
 import datetime as pdt
 import platform
 import struct
+import re
 import sys
 
 import pytest
@@ -310,4 +311,5 @@ def test_tz_class_introspection():
     tzi = rdt.TzClass()
 
     assert tzi.__class__ == rdt.TzClass
-    assert repr(tzi).startswith("<TzClass object at")
+    # PyPy generates <importlib.bootstrap.TzClass ...> for some reason.
+    assert re.match(r"^<[\w\.]*TzClass object at", repr(tzi))

guide/src/building_and_distribution.md

@@ -36,7 +36,27 @@ On Linux/macOS you might have to change `LD_LIBRARY_PATH` to include libpython,
 
 ## Distribution
 
-There are two ways to distribute your module as a Python package: The old, [setuptools-rust](https://github.com/PyO3/setuptools-rust), and the new, [maturin](https://github.com/pyo3/maturin). setuptools-rust needs several configuration files (`setup.py`, `MANIFEST.in`, `build-wheels.sh`, etc.). maturin doesn't need any configuration files, however it does not support some functionality of setuptools such as package data ([pyo3/maturin#258](https://github.com/PyO3/maturin/issues/258)) and requires a rigid project structure, while setuptools-rust allows (and sometimes requires) configuration with python code.
+There are two ways to distribute your module as a Python package: The old, [setuptools-rust], and the new, [maturin]. setuptools-rust needs several configuration files (`setup.py`, `MANIFEST.in`, `build-wheels.sh`, etc.). maturin doesn't need any configuration files, however it does not support some functionality of setuptools such as package data ([pyo3/maturin#258](https://github.com/PyO3/maturin/issues/258)) and requires a rigid project structure, while setuptools-rust allows (and sometimes requires) configuration with python code.
+
+## `Py_LIMITED_API`/`abi3`
+
+By default, Python extension modules can only be used with the same Python version they were compiled against -- if you build an extension module with Python 3.5, you can't import it using Python 3.8. [PEP 384](https://www.python.org/dev/peps/pep-0384/) introduced the idea of the limited Python API, which would have a stable ABI enabling extension modules built with it to be used against multiple Python versions. This is also known as `abi3`.
+
+Note that [maturin] >= 0.9.0 or [setuptools-rust] >= 0.12.0 is going to support `abi3` wheels.
+See the [corresponding](https://github.com/PyO3/maturin/pull/353) [PRs](https://github.com/PyO3/setuptools-rust/pull/82) for more.
+
+There are three steps involved in making use of `abi3` when building Python packages as wheels:
+
+1. Enable the `abi3` feature in `pyo3`. This ensures `pyo3` only calls Python C-API functions which are part of the stable API, and on Windows also ensures that the project links against the correct shared object (no special behavior is required on other platforms):
+
+```toml
+[dependencies]
+pyo3 = { version = "...", features = ["abi3"]}
+```
+
+2. Ensure that the built shared objects are correctly marked as `abi3`. This is accomplished by telling your build system that you're using the limited API.
+
+3. Ensure that the `.whl` is correctly marked as `abi3`. For projects using `setuptools`, this is accomplished by passing `--py-limited-api=cp3x` (where `x` is the minimum Python version supported by the wheel, e.g. `--py-limited-api=cp35` for Python 3.5) to `setup.py bdist_wheel`.
 
 ## Cross Compiling
 
@@ -83,3 +103,8 @@ cargo build --target x86_64-pc-windows-gnu
 ## Bazel
 
 For an example of how to build python extensions using Bazel, see https://github.com/TheButlah/rules_pyo3
+
+
+[maturin]: https://github.com/PyO3/maturin
+[setuptools-rust]: https://github.com/PyO3/setuptools-rust
+

guide/src/class.md

@@ -205,7 +205,7 @@ or by `self_.into_super()` as `PyRef<Self::BaseClass>`.
 ```rust
 # use pyo3::prelude::*;
 
-#[pyclass]
+#[pyclass(subclass)]
 struct BaseClass {
     val1: usize,
 }
@@ -222,7 +222,7 @@ impl BaseClass {
     }
 }
 
-#[pyclass(extends=BaseClass)]
+#[pyclass(extends=BaseClass, subclass)]
 struct SubClass {
     val2: usize,
 }
@@ -266,12 +266,14 @@ impl SubSubClass {
 ```
 
 You can also inherit native types such as `PyDict`, if they implement
-[`PySizedLayout`](https://docs.rs/pyo3/latest/pyo3/type_object/trait.PySizedLayout.html).
+[`PySizedLayout`](https://docs.rs/pyo3/latest/pyo3/type_object/trait.PySizedLayout.html). However, this is not supported when building for the Python limited API (aka the `abi3` feature of PyO3).
 
 However, because of some technical problems, we don't currently provide safe upcasting methods for types
 that inherit native types. Even in such cases, you can unsafely get a base class by raw pointer conversion.
 
 ```rust
+# #[cfg(Py_LIMITED_API)] fn main() {}
+# #[cfg(not(Py_LIMITED_API))] fn main() {
 # use pyo3::prelude::*;
 use pyo3::types::PyDict;
 use pyo3::{AsPyPointer, PyNativeType};
@@ -300,6 +302,7 @@ impl DictWithCounter {
 # let py = gil.python();
 # let cnt = pyo3::PyCell::new(py, DictWithCounter::new()).unwrap();
 # pyo3::py_run!(py, cnt, "cnt.set('abc', 10); assert cnt['abc'] == 10")
+# }
 ```
 
 If `SubClass` does not provide a baseclass initialization, the compilation fails.
@@ -769,13 +772,23 @@ impl pyo3::class::methods::HasMethodsInventory for MyClass {
 }
 pyo3::inventory::collect!(Pyo3MethodsInventoryForMyClass);
 
-impl pyo3::class::proto_methods::HasProtoRegistry for MyClass {
-    fn registry() -> &'static pyo3::class::proto_methods::PyProtoRegistry {
-        static REGISTRY: pyo3::class::proto_methods::PyProtoRegistry
-            = pyo3::class::proto_methods::PyProtoRegistry::new();
-        &REGISTRY
+
+pub struct Pyo3ProtoInventoryForMyClass {
+    def: pyo3::class::proto_methods::PyProtoMethodDef,
+}
+impl pyo3::class::proto_methods::PyProtoInventory for Pyo3ProtoInventoryForMyClass {
+    fn new(def: pyo3::class::proto_methods::PyProtoMethodDef) -> Self {
+        Self { def }
+    }
+    fn get(&'static self) -> &'static pyo3::class::proto_methods::PyProtoMethodDef {
+        &self.def
     }
 }
+impl pyo3::class::proto_methods::HasProtoInventory for MyClass {
+    type ProtoMethods = Pyo3ProtoInventoryForMyClass;
+}
+pyo3::inventory::collect!(Pyo3ProtoInventoryForMyClass);
+
 
 impl pyo3::pyclass::PyClassSend for MyClass {
     type ThreadChecker = pyo3::pyclass::ThreadCheckerStub<MyClass>;

guide/src/migration.md

@@ -3,6 +3,18 @@
 This guide can help you upgrade code through breaking changes from one PyO3 version to the next.
 For a detailed list of all changes, see the [CHANGELOG](changelog.md).
 
+## from 0.12.* to 0.13
+
+### Runtime changes to support the CPython limited API
+
+In PyO3 `0.13` support was added for compiling against the CPython limited API. This had a number of implications for _all_ PyO3 users, described here.
+
+The largest of these is that all types created from PyO3 are what CPython calls "heap" types. The specific implications of this are:
+
+- If you wish to subclass one of these types _from Rust_ you must mark it `#[pyclass(subclass)]`, as you would if you wished to allow subclassing it from Python code.
+- Type objects are now mutable - Python code can set attributes on them.
+- `__module__` on types without `#[pyclass(module="mymodule")]` no longer returns `builtins`, it now raises `AttributeError`.
+
 ## from 0.11.* to 0.12
 
 ### `PyErr` has been reworked

guide/src/trait_bounds.md

@@ -408,8 +408,8 @@ impl Model for UserModel {
                 .call_method("get_results", (), None)
                 .unwrap();
 
-            if py_result.get_type().name() != "list" {
-                panic!("Expected a list for the get_results() method signature, got {}", py_result.get_type().name());
+            if py_result.get_type().name().unwrap() != "list" {
+                panic!("Expected a list for the get_results() method signature, got {}", py_result.get_type().name().unwrap());
             }
             py_result.extract()
         })
@@ -536,8 +536,8 @@ impl Model for UserModel {
                 .call_method("get_results", (), None)
                 .unwrap();
 
-            if py_result.get_type().name() != "list" {
-                panic!("Expected a list for the get_results() method signature, got {}", py_result.get_type().name());
+            if py_result.get_type().name().unwrap() != "list" {
+                panic!("Expected a list for the get_results() method signature, got {}", py_result.get_type().name().unwrap());
             }
             py_result.extract()
         })