[Windows] Unit tests got Fatal python error #2942
Description
🐛 Describe the bug
last known good: https://github.com/intel/torch-xpu-ops/actions/runs/21673570195/job/62487564996, pytorch: f775242b29d455ab38d83a340df93658cbf46ce9
first crash, but another message: Windows fatal exception: access violation, https://github.com/intel/torch-xpu-ops/actions/runs/21713651295/job/62623512001, pytorch: 9a0c104b1c643c912462a4d484ca5c1389b77ea3
first crash 'fatal python error': https://github.com/intel/torch-xpu-ops/actions/runs/21994282848/job/63549803749, pytorch: 5ed4a972f278fa2e4c04c134733013645c142123
call "C:\ProgramData\miniforge3\Scripts\activate.bat"
call "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
call conda activate windows_ci
cd ../pytorch/third_party/torch-xpu-ops/test/xpu/
python run_test_win_with_skip_mtl.py
if not exist "%GITHUB_WORKSPACE%\ut_log" mkdir "%GITHUB_WORKSPACE%\ut_log"
copy test_xpu.xml %GITHUB_WORKSPACE%\ut_log /Y
shell: C:\WINDOWS\system32\cmd.EXE /D /E:ON /V:OFF /S /C "CALL "{0}""
env:
USE_XPU: 1
PYTORCH_EXTRA_INSTALL_REQUIREMENTS: intel-cmplr-lib-rt==2025.3.2 | intel-cmplr-lib-ur==2025.3.2 | intel-cmplr-lic-rt==2025.3.2 | intel-sycl-rt==2025.3.2 | onemkl-license==2025.3.1 | onemkl-sycl-blas==2025.3.1 | onemkl-sycl-dft==2025.3.1 | onemkl-sycl-lapack==2025.3.1 | onemkl-sycl-rng==2025.3.1 | onemkl-sycl-sparse==2025.3.1 | dpcpp-cpp-rt==2025.3.2 | intel-opencl-rt==2025.3.2 | mkl==2025.3.1 | intel-openmp==2025.3.2 | tbb==2022.3.1 | tcmlib==1.4.1 | umf==1.0.3 | intel-pti==0.16.0
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.14.25
** Copyright (c) 2025 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
[W219 10:54:31.000000000 unwind.cpp:11] Warning: record_context_cpp is not support on non-linux non-x86_64 platforms (function operator ())
Unhandled exception caught in c10/util/AbortHandler.h
00007FF8B33BFF1800007FF8B33B5FD0 torch_python.dll!torch::autograd::THPCppFunction_requires_grad [<unknown file> @ <unknown line number>]
00007FF92DE619D700007FF92DE619C0 ucrtbase.dll!terminate [<unknown file> @ <unknown line number>]
00007FF911E61943 <unknown symbol address> VCRUNTIME140_1.dll!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FF911E621AF <unknown symbol address> VCRUNTIME140_1.dll!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FF911E6220D <unknown symbol address> VCRUNTIME140_1.dll!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FF911E6408200007FF911E63FD0 VCRUNTIME140_1.dll!_CxxFrameHandler4 [<unknown file> @ <unknown line number>]
00007FF930A0618F00007FF930A060F0 ntdll.dll!_chkstk [<unknown file> @ <unknown line number>]
00007FF9308B232700007FF9308B1D90 ntdll.dll!RtlLocateExtendedFeature [<unknown file> @ <unknown line number>]
00007FF9308AA96100007FF9308AA740 ntdll.dll!RtlRaiseException [<unknown file> @ <unknown line number>]
00007FF92E12A80A00007FF92E12A780 KERNELBASE.dll!RaiseException [<unknown file> @ <unknown line number>]
00007FF9152252C700007FF915225230 VCRUNTIME140.dll!CxxThrowException [<unknown file> @ <unknown line number>]
00007FF8B6AD90B600007FF8B6AD78C0 sycl8.dll!sycl::_V1::queue::ext_codeplay_supports_fusion [<unknown file> @ <unknown line number>]
00007FF8B6C9031100007FF8B6C65570 sycl8.dll!_sycl_register_lib [<unknown file> @ <unknown line number>]
00007FF8FFAEC78700007FF8FFAEC6C0 c10_xpu.dll!c10::xpu::syncStreamsOnDevice [<unknown file> @ <unknown line number>]
00007FF8FFAE60E900007FF8FFAE57B0 c10_xpu.dll!c10::xpu::XPUCachingAllocator::releasePool [<unknown file> @ <unknown line number>]
00007FF8FFADF69400007FF8FFADF5F0 c10_xpu.dll!c10::xpu::XPUStream::device_type [<unknown file> @ <unknown line number>]
00007FF8FFADB34400007FF8FFADB2C0 c10_xpu.dll!c10::xpu::MemPool::~MemPool [<unknown file> @ <unknown line number>]
00007FF8B3383C9D00007FF8B30BF370 torch_python.dll!torch::PythonArgs::toSymIntOptional [<unknown file> @ <unknown line number>]
00007FF8B3383DD600007FF8B30BF370 torch_python.dll!torch::PythonArgs::toSymIntOptional [<unknown file> @ <unknown line number>]
00007FF8B2FC19CE00007FF8B2FC0740 torch_python.dll!c10::ivalue::Await::args [<unknown file> @ <unknown line number>]
00007FF8B2FD14FB00007FF8B2FD1060 torch_python.dll!c10::ivalue::Future::waitAndThrow [<unknown file> @ <unknown line number>]
00007FF8F8F4210700007FF8F8F419F0 python310.dll!PyType_GenericNew [<unknown file> @ <unknown line number>]
00007FF8F8FE664E00007FF8F8FE44D0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FF8F8FE982900007FF8F8FE44D0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FF8F8EDE75E00007FF8F8EDE720 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FF8F8EE0AB100007FF8F8EE06C0 python310.dll!PyCell_Set [<unknown file> @ <unknown line number>]
00007FF8F8EE0CA600007FF8F8EE0BB0 python310.dll!PyMethod_Self [<unknown file> @ <unknown line number>]
00007FF8F8FEB41400007FF8F8FEAE10 python310.dll!PyEval_GetFuncDesc [<unknown file> @ <unknown line number>]
00007FF8F8FE63AC00007FF8F8FE44D0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FF8F8FE982900007FF8F8FE44D0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FF8F8EDE75E00007FF8F8EDE720 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FF8F8EE0AB100007FF8F8EE06C0 python310.dll!PyCell_Set [<unknown file> @ <unknown line number>]
00007FF8F8EE0C6D00007FF8F8EE0BB0 python310.dll!PyMethod_Self [<unknown file> @ <unknown line number>]
00007FF8F8FE28F100007FF8F8FE27B0 python310.dll!PyOS_URandomNonblock [<unknown file> @ <unknown line number>]
00007FF8F8FEB20200007FF8F8FEAE10 python310.dll!PyEval_GetFuncDesc [<unknown file> @ <unknown line number>]
00007FF8F8FE700400007FF8F8FE44D0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
Fatal Python error: Aborted
Thread 0x00002c34 (most recent call first):
<no Python frame>
Thread 0x00002bc8 (most recent call first):
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\threading.py", line 324 in wait
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\threading.py", line 607 in wait
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\tqdm\_monitor.py", line 60 in run
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\threading.py", line 1016 in _bootstrap_inner
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\threading.py", line 973 in _bootstrap
Thread 0x000052b4 (most recent call first):
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\socket.py", line 293 in accept
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\pytest_rerunfailures.py", line 433 in run_server
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\threading.py", line 953 in run
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\threading.py", line 1016 in _bootstrap_inner
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\threading.py", line 973 in _bootstrap
Current thread 0x00002364 (most recent call first):
File "C:\actions-runner\_work\torch-xpu-ops\pytorch\test\test_xpu.py", line 2915 in test_mempool_id
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\torch\testing\_internal\common_utils.py", line 3364 in wrapper
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\unittest\case.py", line 549 in _callTestMethod
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\unittest\case.py", line 591 in run
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\torch\testing\_internal\common_utils.py", line 3516 in _run_custom
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\torch\testing\_internal\common_utils.py", line 3547 in run
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\unittest\case.py", line 650 in __call__
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\_pytest\unittest.py", line 333 in runtest
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\_pytest\runner.py", line 169 in pytest_runtest_call
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\pluggy\_callers.py", line 121 in _multicall
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\pluggy\_hooks.py", line 512 in __call__
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\_pytest\runner.py", line 262 in <lambda>
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\_pytest\runner.py", line 341 in from_call
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\_pytest\runner.py", line 261 in call_runtest_hook
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\_pytest\runner.py", line 222 in call_and_report
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\_pytest\runner.py", line 133 in runtestprotocol
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\_pytest\runner.py", line 114 in pytest_runtest_protocol
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\pluggy\_callers.py", line 121 in _multicall
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\pluggy\_hooks.py", line 512 in __call__
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\_pytest\main.py", line 348 in pytest_runtestloop
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\pluggy\_callers.py", line 121 in _multicall
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\pluggy\_hooks.py", line 512 in __call__
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\_pytest\main.py", line 323 in _main
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\_pytest\main.py", line 269 in wrap_session
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\_pytest\main.py", line 316 in pytest_cmdline_main
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\pluggy\_callers.py", line 121 in _multicall
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\pluggy\_hooks.py", line 512 in __call__
File "C:\ProgramData\miniforge3\envs\windows_ci\lib\site-packages\_pytest\config\__init__.py", line 166 in main
File "C:\actions-runner\_work\torch-xpu-ops\pytorch\third_party\torch-xpu-ops\test\xpu\run_test_win_with_skip_mtl.py", line 30 in <module>
Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, thriftpy2.transport.cybase, thriftpy2.transport.buffered.cybuffered, thriftpy2.transport.framed.cyframed, thriftpy2.transport.memory.cymemory, thriftpy2.transport.sasl.cysasl, thriftpy2.protocol.cybin.cybin, scipy._lib._ccallback_c, yaml._yaml, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, numba.mviewbuf, numba.types.itertools, psutil._psutil_windows, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg._cythonized_array_utils, scipy.linalg._flinalg, scipy.linalg._solve_toeplitz, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_lapack, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._isolve._iterative, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.special._ellip_harm_2, scipy.signal._sigtools, scipy.signal._max_len_seq_inner, scipy.signal._upfirdn_apply, scipy.signal._spline, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, numpy.linalg.lapack_lite, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize.__nnls, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate.dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.signal._sosfilt, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, scipy.signal._spectral, scipy.special.cython_special, scipy.stats._stats, scipy.stats.beta_ufunc, scipy.stats._boost.beta_ufunc, scipy.stats.binom_ufunc, scipy.stats._boost.binom_ufunc, scipy.stats.nbinom_ufunc, scipy.stats._boost.nbinom_ufunc, scipy.stats.hypergeom_ufunc, scipy.stats._boost.hypergeom_ufunc, scipy.stats.ncf_ufunc, scipy.stats._boost.ncf_ufunc, scipy.stats.ncx2_ufunc, scipy.stats._boost.ncx2_ufunc, scipy.stats.nct_ufunc, scipy.stats._boost.nct_ufunc, scipy.stats.skewnorm_ufunc, scipy.stats._boost.skewnorm_ufunc, scipy.stats.invgauss_ufunc, scipy.stats._boost.invgauss_ufunc, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy.stats._statlib, scipy.stats._mvn, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._rcont.rcont, scipy.signal._peak_finding_utils, markupsafe._speedups (total: 161)
00007FF8F8FE982900007FF8F8FE44D0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FF8F8EDE75E00007FF8F8EDE720 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FF8F8FE28F100007FF8F8FE27B0 python310.dll!PyOS_URandomNonblock [<unknown file> @ <unknown line number>]
00007FF8F8FEB20200007FF8F8FEAE10 python310.dll!PyEval_GetFuncDesc [<unknown file> @ <unknown line number>]
00007FF8F8FE761700007FF8F8FE44D0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FF8F8FE982900007FF8F8FE44D0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FF8F8EDE75E00007FF8F8EDE720 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FF8F8EE0AB100007FF8F8EE06C0 python310.dll!PyCell_Set [<unknown file> @ <unknown line number>]
00007FF8F8EE0C6D00007FF8F8EE0BB0 python310.dll!PyMethod_Self [<unknown file> @ <unknown line number>]
00007FF8F8FE28F100007FF8F8FE27B0 python310.dll!PyOS_URandomNonblock [<unknown file> @ <unknown line number>]
00007FF8F8FEB20200007FF8F8FEAE10 python310.dll!PyEval_GetFuncDesc [<unknown file> @ <unknown line number>]
00007FF8F8FE765C00007FF8F8FE44D0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FF8F8FE982900007FF8F8FE44D0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FF8F8EDE75E00007FF8F8EDE720 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FF8F8EE0AB100007FF8F8EE06C0 python310.dll!PyCell_Set [<unknown file> @ <unknown line number>]
00007FF8F8EE0C6D00007FF8F8EE0BB0 python310.dll!PyMethod_Self [<unknown file> @ <unknown line number>]
00007FF8F8FE28F100007FF8F8FE27B0 python310.dll!PyOS_URandomNonblock [<unknown file> @ <unknown line number>]
00007FF8F8FEB20200007FF8F8FEAE10 python310.dll!PyEval_GetFuncDesc [<unknown file> @ <unknown line number>]
00007FF8F8FE765C00007FF8F8FE44D0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FF8F8FE982900007FF8F8FE44D0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FF8F8EDE75E00007FF8F8EDE720 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
00007FF8F8EE0AB100007FF8F8EE06C0 python310.dll!PyCell_Set [<unknown file> @ <unknown line number>]
00007FF8F8EE0C6D00007FF8F8EE0BB0 python310.dll!PyMethod_Self [<unknown file> @ <unknown line number>]
00007FF8F8EDE4C100007FF8F8EDE400 python310.dll!PyVectorcall_Call [<unknown file> @ <unknown line number>]
00007FF8F8FEB41400007FF8F8FEAE10 python310.dll!PyEval_GetFuncDesc [<unknown file> @ <unknown line number>]
00007FF8F8FE63AC00007FF8F8FE44D0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FF8F8FE982900007FF8F8FE44D0 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FF8F8EDE75E00007FF8F8EDE720 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>]
Versions
https://github.com/intel/torch-xpu-ops/actions/runs/22184149573/job/64153075833#step:10:6