How to handle XRootD error: [ERROR] Operation expired

[dcervenkov]

When reading many files via XRootD, we see intermittent and seemingly non-deterministic "Operation expired" errors from XRootD. This might be an XRootD or an EOS issue, but I filed it here since we are experiencing it when using Uproot. I'd be interested to hear if you have ideas about how to troubleshoot this.

MWE to reproduce follows. Unfortunately, it requires access to LHCb files on EOS.

import uproot

samples = [
    f"root://eoslhcb.cern.ch//eos/lhcb/grid/prod/lhcb/LHCb/Collision16/PIDCALIB.ROOT/00111823/0000/00111823_{i:08}_1.pidcalib.root"
    for i in range(1, 239)
]

for file in samples:
    print(f"Reading {file}")
    tree = uproot.open(file)["DSt_PiMTuple"]

Traceback (most recent call last):
  File "mwe_xrootd.py", line 12, in <module>
    tree = uproot.open(file)["DSt_PiMTuple"]
  File "/home/dc/.local/lib/python3.6/site-packages/uproot/reading.py", line 145, in open
    **options  # NOTE: a comma after **options breaks Python 2
  File "/home/dc/.local/lib/python3.6/site-packages/uproot/reading.py", line 553, in __init__
    file_path, **self._options  # NOTE: a comma after **options breaks Python 2
  File "/home/dc/.local/lib/python3.6/site-packages/uproot/source/xrootd.py", line 434, in __init__
    self._open()
  File "/home/dc/.local/lib/python3.6/site-packages/uproot/source/xrootd.py", line 440, in _open
    for x in uproot._util.range(self._num_workers)
  File "/home/dc/.local/lib/python3.6/site-packages/uproot/source/xrootd.py", line 440, in <listcomp>
    for x in uproot._util.range(self._num_workers)
  File "/home/dc/.local/lib/python3.6/site-packages/uproot/source/xrootd.py", line 88, in __init__
    self._xrd_error(status)
  File "/home/dc/.local/lib/python3.6/site-packages/uproot/source/xrootd.py", line 108, in _xrd_error
    status.message, self._file_path
OSError: XRootD error: [ERROR] Operation expired
in file root://eoslhcb.cern.ch//eos/lhcb/grid/prod/lhcb/LHCb/Collision16/PIDCALIB.ROOT/00111823/0000/00111823_00000057_1.pidcalib.root

It is not a problem with expired credentials, etc. - just rerunning the script will cause it to fail on a different file or not at all.

The issue occurs on multiple computers with very different setups.

uproot 4.0.7
xrootd 5.1.1

[chrisburr]

Interestingly I can't reproduce it on lxplus but I can when reading the data from outside of the CERN. Now to figure out why...

[chrisburr]

This isn't really an uproot problem (ROOT also had the same issue) but I'll summarise how I debugged it in case others run in to the same issue.

1. Repeating the test many times showed that it only ever crashed on a subset of the files mentioned in this issue
2. eos fileinfo /eos/lhcb/... showed that they all contained a replica that was on the same server
3. It was reported at https://ggus.eu/?mode=ticket_info&ticket_id=151899 and indeed there was a server with a problem that prevented it from being made visible outside the firewall.

[dcervenkov]

That's good news. Thanks for taking care of it @chrisburr!

[jpivarski]

Thanks, @chrisburr! To make sure your solution remains visible, I'll convert this into a discussion.