Support Python 3.14

[glatterf42]

Hi there :)

This week, Python 3.14 was released. So we've been trying to check if we can support it already in the packages we maintain.
One of these attempts is iiasa/ixmp4#209, which is now running into an error with pandera/pandas. This error is also described in pandas-dev/pandas#62711 (though not originally by me and the original description doesn't mention Python 3.14). It's just that the traceback seems very similar, so it might well be related.

Either way, this is an exemplary traceback I'm seeing (full logs can be found here):

Traceback

__ ERROR at setup of TestCoreIamcReadOnly.test_mp_tabulate_big_async[sqlite] ___

request = <SubRequest 'platform_med' for <Function test_mp_tabulate_big_async[sqlite]>>

    def platform_with_td(
        request: pytest.FixtureRequest,
    ) -> Generator[Platform, Any, None]:
        type = request.param
        postgres_dsn = request.config.option.postgres_dsn
        bctx = get_backend_context(type, postgres_dsn)
    
        with bctx as backend:
            platform = Platform(_backend=backend)
>           td.load_dataset(platform)

tests/conftest.py:175: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/fixtures/__init__.py:353: in load_dataset
    cls.load_datapoints(platform)
tests/fixtures/__init__.py:346: in load_datapoints
    cls.load_dp_df(platform, cls.datapoints)
tests/fixtures/__init__.py:342: in load_dp_df
    cls.load_run_datapoints(platform, (model, scenario, version), dps)
tests/fixtures/__init__.py:319: in load_run_datapoints
    run.iamc.add(annual, type=ixmp4.DataPoint.Type.ANNUAL)
ixmp4/core/iamc/data.py:63: in add
    df = AddDataPointFrameSchema.validate(df)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.14/site-packages/pandera/api/pandas/model.py:191: in validate
    cls.to_schema().validate(
.venv/lib/python3.14/site-packages/pandera/api/pandas/container.py:118: in validate
    return self._validate(
.venv/lib/python3.14/site-packages/pandera/api/pandas/container.py:139: in _validate
    return self.get_backend(check_obj).validate(
.venv/lib/python3.14/site-packages/pandera/backends/pandas/container.py:104: in validate
    error_handler = self.run_checks_and_handle_errors(
.venv/lib/python3.14/site-packages/pandera/backends/pandas/container.py:192: in run_checks_and_handle_errors
    error_handler.collect_error(
.venv/lib/python3.14/site-packages/pandera/api/base/error_handler.py:54: in collect_error
    raise schema_error from original_exc
.venv/lib/python3.14/site-packages/pandera/backends/pandas/container.py:227: in run_schema_component_checks
    result = schema_component.validate(
.venv/lib/python3.14/site-packages/pandera/api/dataframe/components.py:149: in validate
    return self.get_backend(check_obj).validate(
.venv/lib/python3.14/site-packages/pandera/backends/pandas/components.py:142: in validate
    validated_column = validate_column(
.venv/lib/python3.14/site-packages/pandera/backends/pandas/components.py:102: in validate_column
    error_handler.collect_error(
.venv/lib/python3.14/site-packages/pandera/api/base/error_handler.py:54: in collect_error
    raise schema_error from original_exc
.venv/lib/python3.14/site-packages/pandera/backends/pandas/components.py:78: in validate_column
    validated_check_obj = super(ColumnBackend, self).validate(
.venv/lib/python3.14/site-packages/pandera/backends/pandas/array.py:74: in validate
    error_handler = self.run_checks_and_handle_errors(
.venv/lib/python3.14/site-packages/pandera/backends/pandas/array.py:138: in run_checks_and_handle_errors
    error_handler.collect_error(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pandera.api.base.error_handler.ErrorHandler object at 0x7f6d4db479a0>
error_type = <ValidationScope.DATA: 'data'>
reason_code = <SchemaErrorReason.CHECK_ERROR: 'check_error'>
schema_error = SchemaError('Error while executing check function: KeyError("<class \'pandas.core.series.Series\'>")\nTraceback (most ...nput_data_type]\n         ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^\nKeyError: <class \'pandas.core.series.Series\'>\n')
original_exc = KeyError(<class 'pandas.core.series.Series'>)

    def collect_error(
        self,
        error_type: ErrorCategory,
        reason_code: Optional[SchemaErrorReason],
        schema_error: SchemaError,
        original_exc: Union[BaseException, None] = None,
    ):
        """Collect schema error, raising exception if lazy is False.
    
        :param error_type: type of error
        :param reason_code: string representing reason for error
        :param schema_error: ``SchemaError`` object.
        """
        if not self._lazy:
>           raise schema_error from original_exc
E           pandera.errors.SchemaError: Error while executing check function: KeyError("<class 'pandas.core.series.Series'>")
E           Traceback (most recent call last):
E             File "/home/runner/work/ixmp4/ixmp4/.venv/lib/python3.14/site-packages/pandera/backends/pandas/components.py", line 242, in run_checks
E               self.run_check(
E               ~~~~~~~~~~~~~~^
E                   check_obj, schema, check, check_index, *check_args
E                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E               )
E               ^
E             File "/home/runner/work/ixmp4/ixmp4/.venv/lib/python3.14/site-packages/pandera/backends/pandas/base.py", line 115, in run_check
E               check_result: CheckResult = check(check_obj, *args)
E                                           ~~~~~^^^^^^^^^^^^^^^^^^
E             File "/home/runner/work/ixmp4/ixmp4/.venv/lib/python3.14/site-packages/pandera/api/checks.py", line 234, in __call__
E               return backend(check_obj, column)
E             File "/home/runner/work/ixmp4/ixmp4/.venv/lib/python3.14/site-packages/pandera/backends/pandas/checks.py", line 349, in __call__
E               check_output = self.apply(check_obj)
E             File "/home/runner/work/ixmp4/ixmp4/.venv/lib/python3.14/site-packages/pandera/backends/pandas/checks.py", line 148, in apply
E               return apply_fn(check_obj)
E             File "/home/runner/work/ixmp4/ixmp4/.venv/lib/python3.14/site-packages/pandera/backends/pandas/checks.py", line 156, in apply_field
E               return self.check_fn(check_obj)
E                      ~~~~~~~~~~~~~^^^^^^^^^^^
E             File "/home/runner/work/ixmp4/ixmp4/.venv/lib/python3.14/site-packages/pandera/api/function_dispatch.py", line 24, in __call__
E               fn = self._function_registry[input_data_type]
E                    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
E           KeyError: <class 'pandas.core.series.Series'>

.venv/lib/python3.14/site-packages/pandera/api/base/error_handler.py:54: SchemaError

These tests work fine with Python 3.13 (in both cases, I'm using pandas 2.3.1 and pandera 0.25.0), but they fail with Python 3.14.

Looking at the code here, I'm assuming it could be

pandera/pandera/engines/engine.py

Line 149 in 48ba659

cls._registry[cls].dispatch.register(source_dtype, _method)

or

pandera/pandera/api/extensions.py

Line 81 in 48ba659

check_dispatcher.register(fn) # type: ignore

or

pandera/pandera/api/base/checks.py

Line 106 in 48ba659

dispatcher.register(fn)

where the registration fails with Python 3.14 (the first one seems most likely to me because it mentions a dtype), but I'm not sure. It's just that the dtype in the first case is coming from typing.get_type_hints(), which wasn't officially changed in Python 3.14, but I know that e.g. pydantic had to adjust how they handle annotations in their metaclasses because of changes in 3.14. They now use the recommended new annotationlib for that, which might be necessary here, too.

I'm hoping that running the CI tests on Python 3.14, which this PR achieves, will point in the right direction and will allow you to officially support Python 3.14 :)

[glatterf42]

Some more investigation: it looks to me like https://github.com/unionai-oss/pandera/blob/main/pandera/api/function_dispatch.py#L20 is the only line adding functions to the Dispatcher._function_registry. This, in turn, relies on get_first_arg_type(), which uses inspect.signature() and typing_inspect. The former changed in Python 3.14; while this should just affect annotations from files with from __future__ import annotations, e.g. api/dataframe/container.py and dtypes.py have this line, so it could be important here.
The latter does not officially support 3.14 yet; in fact, the only thing I see about that is an open issue regarding a test failure with 3.14. So it may well be that support needs to be added there, first.

[JelleZijlstra]

The former changed in Python 3.14; while this should just affect annotations from files with from future import annotations, e.g. api/dataframe/container.py and dtypes.py have this line, so it could be important here.

The behavior under from __future__ import annotations should not have changed. (I implemented these changes in CPython.)

The latter does not officially support 3.14 yet

typing-inspect was written several years ago when the introspection capabilities of the typing module itself were much worse. I haven't checked your code but it may be easier to drop use of typing-inspect and use typing directly.

[glatterf42]

Thanks for chiming in here :)
The usage of typing_inspect in api/function_dispatch.py came indeed down to get_args() and get_origin(): two functions that by now also exist in typing. So I replaced calls to them with calls to their typing implementation.
Please note that this does not eliminate typing_inspect as a dependency entirely since the code still uses get_generic_bases(), is_optional_type(), and is_union_type(), which don't have a 1:1-replacement in typing (of course, these could still be replaced, but I don't know if you prefer using the existing logic from typing_inspect or duplicating it here to drop the dependency).

[JelleZijlstra]

Yes, replacing the rest will be a bit more work and refactoring. Not my decision whether that's worth doing, but in my opinion it's usually nice to minimize third-party dependencies.

[codecov]

Codecov Report

❌ Patch coverage is 94.44444% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.63%. Comparing base (812b2a8) to head (c0da805).
⚠️ Report is 364 commits behind head on main.

Files with missing lines	Patch %	Lines
pandera/typing/common.py	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2158      +/-   ##
==========================================
- Coverage   94.28%   93.63%   -0.65%     
==========================================
  Files          91      136      +45     
  Lines        7013    10821    +3808     
==========================================
+ Hits         6612    10132    +3520     
- Misses        401      689     +288     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[deepyaman]

Based on modelcontextprotocol/python-sdk#1451, I think the build issue on 3.14 may just be because of outdated uv.

Looks like I also need to update some tests to be compatible with Ibis 11.0.

And I also accidentally committed without signing off, so I'll fix that.

[glatterf42]

Thanks for contributing here :)
At least for the "Docs" jobs, they don't seem to be failing because of a uv pin (though I only found this PR, which doesn't explain why that pin is there), but because ray doesn't support Python 3.14 yet. They have two open issues on that:

• Support Python 3.14 ray-project/ray#56434
• Request for Python 3.14 Support ray-project/ray#57642

and estimate they'll release wheels with 3.14 support in Q4.
So if you find a way to fix that (or the other failures), thanks a lot! But we might also just have to wait.

[deepyaman]

Thanks for contributing here :) At least for the "Docs" jobs, they don't seem to be failing because of a uv pin (though I only found this PR, which doesn't explain why that pin is there), but because ray doesn't support Python 3.14 yet. They have two open issues on that:

• Support Python 3.14 ray-project/ray#56434
• Request for Python 3.14 Support ray-project/ray#57642

and estimate they'll release wheels with 3.14 support in Q4. So if you find a way to fix that (or the other failures), thanks a lot! But we might also just have to wait.

Ah, yes, I was looking at https://github.com/unionai-oss/pandera/actions/runs/18922471107/job/54022290027?pr=2158. The docs jobs are clearly failing due to Ray, but I assume it's not so bad to just exclude that from the 3.14 test.

@cosmicBboy do you know why the uv pin is there? :)

[cosmicBboy]

@cosmicBboy do you know why the uv pin is there? :)

Mainly to keep the uv dependency deterministic. We can definitely bump this to the latest uv version

[deepyaman]

@cosmicBboy do you know why the uv pin is there? :)

Mainly to keep the uv dependency deterministic. We can definitely bump this to the latest uv version

OK, let me try bumping it to see if it resolves the issue. I think can probably make the CI a bit more uv-native at some point, too (but not touching it right now).

[deepyaman]

@cosmicBboy do you know why the uv pin is there? :)

Mainly to keep the uv dependency deterministic. We can definitely bump this to the latest uv version

OK, let me try bumping it to see if it resolves the issue. I think can probably make the CI a bit more uv-native at some point, too (but not touching it right now).

My bad, the failure was due to the pinned Pydantic version not supporting 3.14; can probably revert this later (might bump it in a separate PR).

[deepyaman]

@cosmicBboy @FilipAisot Pretty much all of the remaining failures on 3.14 (on Ubuntu and OSX; not solving for Windows right now) look like they're related to an attributed that's expected to be there from __init_subclass__; would you all know anything about this? The erroring line was added in fcddf49#diff-e7c2fa1e556a53be429fa855d9f9ef073072c2c5dfd7f50a78d18660adfae845.

The only other issue I see would be resolved once Pyogrio 0.12.0 is released on PyPI: https://github.com/geopandas/pyogrio/blob/e211838ed4978792eda7764c0790beb01c0110a8/CHANGES.md?plain=1#L28