ci: fix missed testing with oldest

[Borda]

What does this PR do?

seems we missed it when we did the bump in #2671

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

---

📚 Documentation preview 📚: https://torchmetrics--2803.org.readthedocs.build/en/2803/

[Borda]

hitting this for the oldest version:

==================================== ERRORS ====================================
_____________________________ ERROR collecting src _____________________________
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py:513: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
conftest.py:39: in pytest_collect_file
    return DoctestModule.from_parent(parent, fspath=path)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/nodes.py:661: in from_parent
    return super().from_parent(parent=parent, fspath=fspath, path=path, **kw)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/nodes.py:275: in from_parent
    return cls._create(parent=parent, **kw)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/nodes.py:152: in _create
    return super().__call__(*k, **kw)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/nodes.py:616: in __init__
    path = _imply_path(type(self), path, fspath=fspath)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/nodes.py:108: in _imply_path
    warnings.warn(
E   pytest.PytestRemovedIn8Warning: The (fspath: py.path.local) argument to DoctestModule is deprecated. Please use the (path: pathlib.Path) argument instead.
E   See https://docs.pytest.org/en/latest/deprecations.html#fspath-argument-for-node-constructors-replaced-with-pathlib-path
=========================== short test summary info ============================
ERROR . - pytest.PytestRemovedIn8Warning: The (fspath: py.path.local) argument to DoctestModule is deprecated. Please use the (path: pathlib.Path) argument instead.
See https://docs.pytest.org/en/latest/deprecations.html#fspath-argument-for-node-constructors-replaced-with-pathlib-path

[codecov]

Codecov Report

Attention: Patch coverage is 33.33333% with 4 lines in your changes missing coverage. Please review.

Project coverage is 69%. Comparing base (9cc354c) to head (cabaeaa).
Report is 68 commits behind head on master.

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #2803   +/-   ##
======================================
- Coverage      69%     69%   -0%     
======================================
  Files         338     338           
  Lines       18330   18334    +4     
======================================
+ Hits        12584   12586    +2     
- Misses       5746    5748    +2