NEXRADLevel2 Reader Error for Older Files

[TCAlert]

• xradar version: 0.8.0
• Python version: 3.12.2
• Operating System: Windows 11

Description

I was loading in some radar data from KLIX on 8/28/2005 at 1800z using the NEXRADLevel2 engine for xarray that xradar provides and ran into this error. Seems to only happen with the pre-super resolution radars -- I haven't encountered any issues with the modern sites.

What I Did

data = xr.open_dataset(r"C:\Users\deela\Downloads\radar" + f + ".gz", engine = 'nexradlevel2', group = f'sweep_{str(g).zfill(2)}')

Traceback

Traceback (most recent call last):
  File "C:\Users\deela\.anaconda\cyclobot\getRadar.py", line 155, in <module>
    print(getPastData('klix', '2005', '08', '28', '1800'))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\deela\.anaconda\cyclobot\getRadar.py", line 139, in getPastData
    data = xr.open_dataset(r"C:\Users\deela\Downloads\radar" + f + ".gz", engine = 'nexradlevel2', group = f'sweep_{str(g).zfill(2)}')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\deela\anaconda3\envs\Env2272024\Lib\site-packages\xarray\backends\api.py", line 670, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\deela\anaconda3\envs\Env2272024\Lib\site-packages\xradar\io\backends\nexrad_level2.py", line 1447, in open_dataset
    ds = store_entrypoint.open_dataset(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\deela\anaconda3\envs\Env2272024\Lib\site-packages\xarray\backends\store.py", line 44, in open_dataset
    vars, attrs = filename_or_obj.load()
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\deela\anaconda3\envs\Env2272024\Lib\site-packages\xarray\backends\common.py", line 305, in load
    (_decode_variable_name(k), v) for k, v in self.get_variables().items()
                                              ^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\deela\anaconda3\envs\Env2272024\Lib\site-packages\xradar\io\backends\nexrad_level2.py", line 1401, in get_variables
    for k, v in self.ds["sweep_data"].items()
                ^^^^^^^
  File "c:\Users\deela\anaconda3\envs\Env2272024\Lib\site-packages\xradar\io\backends\nexrad_level2.py", line 1321, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "c:\Users\deela\anaconda3\envs\Env2272024\Lib\site-packages\xradar\io\backends\nexrad_level2.py", line 1315, in _acquire
    root.get_sweep(self._group)
  File "c:\Users\deela\anaconda3\envs\Env2272024\Lib\site-packages\xradar\io\backends\nexrad_level2.py", line 490, in get_sweep
    moments = self.msg_31_data_header[sweep_number]["msg_31_data_header"].keys()
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\deela\anaconda3\envs\Env2272024\Lib\site-packages\xradar\io\backends\nexrad_level2.py", line 465, in msg_31_data_header
    ) = self.get_data_header()
        ^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\deela\anaconda3\envs\Env2272024\Lib\site-packages\xradar\io\backends\nexrad_level2.py", line 627, in get_data_header
    while self.init_next_record():
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\deela\anaconda3\envs\Env2272024\Lib\site-packages\xradar\io\backends\nexrad_level2.py", line 364, in init_next_record
    return self.init_record(self.record_number + 1)
                            ~~~~~~~~~~~~~~~~~~~^~~
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

[kmuehlbauer]

Thanks @TCAlert for filing this issue. I'm adding the example from #258 added by @ghiggi:

import os
import fsspec
import xradar as xd
import subprocess

local_tmp_dir = "tmp"
fs_args = {}
_ = fs_args.setdefault("anon", True)
fs = fsspec.filesystem("s3", **fs_args)

s3_filepath = "noaa-nexrad-level2/1996/07/01/KFSX/KFSX19960701_044028.gz"
s3_filepath = "noaa-nexrad-level2/2002/01/01/KABX/KABX20020101_001308.gz"
s3_filepath = "noaa-nexrad-level2/2005/08/28/KLIX/KLIX20050828_180149.gz"
local_filepath = os.path.join(local_tmp_dir, os.path.basename(s3_filepath))
fs.download(s3_filepath, local_tmp_dir)
subprocess.run(["gzip", "-d", local_filepath], check=True)
dt = xd.io.open_nexradlevel2_datatree(local_filepath.replace(".gz", "")) 

I've tracked down the issue. Obviously the file does conform to some older NEXRAD standard. The

https://www.roc.noaa.gov/public-documents/icds/2620010C.pdf

Old Format

Message Type	Number of Segments
15	62
13	49
18	5
3	1
5	1
2	1

https://www.roc.noaa.gov/public-documents/icds/2620010D_Final_052708.pdf

Since 2008:

Message Type	Number of Segments
15	77
13	49
18	5
3	1
5	1
2	1

So we are trying to read data from wrong locations as we extend the type 15 messages into the type 13. I'll have a PR out shortly which fixes this issue.

[kmuehlbauer]

OK, as we only have implemented message type 31 yet (and not message type 1), we would need to add this implementation here, too.

It looks like this might be straightforward...

[kmuehlbauer]

@TCAlert I've added the needed decoding steps in #267. Would you mind testing this branch with your data?

For testing in the CI we would need to add a file containing message type 1 data to open_radar_data.