Error "expected str, bytes or os.PathLike object, not list" for harmony request

[hazemmahmoudnasa]

https://harmony.uat.earthdata.nasa.gov/C1256588789-ASDC_DEV2/ogc-api-coverages/1.0.0/collections/parameter_vars/coverage/rangeset?forceAsync=true&granuleId=G1269242639-ASDC_DEV2%2CG1269374306-ASDC_DEV2&variable=ztoa_net_all_mon&skipPreview=true

I just tested net2cog on UAT and came across this failure: https://harmony.uat.earthdata.nasa.gov/logs/7b216437-8877-4e87-b0ab-a248659ed9d3/4833911
"ERROR:root:expected str, bytes or os.PathLike object, not list\r\nTraceback (most recent call last):\r\n File "/home/dockeruser/.local/lib/python3.10/site-packages/harmony/cli.py", line 244, in _invoke\r\n (out_message, stac_output) = adapter.invoke()\r\n File "/home/dockeruser/.local/lib/python3.10/site-packages/harmony/adapter.py", line 121, in invoke\r\n return (self.message, self._process_catalog_recursive(self.catalog))\r\n File "/home/dockeruser/.local/lib/python3.10/site-packages/harmony/adapter.py", line 186, in _process_catalog_recursive\r\n output_item = self.process_item(item.clone(), source)\r\n File "/worker/net2cog/netcdf_convert_harmony.py", line 109, in process_item\r\n staged_filename = os.path.basename(cog_generated)\r\n File "/usr/local/lib/python3.10/posixpath.py", line 142, in basename\r\n p = os.fspath(p)\r\nTypeError: expected str, bytes or os.PathLike object, not list\r\n",
"{"message": "timing./home/dockeruser/.local/bin/net2cog_harmony.end", "user": "hazem.mahmoud88", "requestId": "7b216437-8877-4e87-b0ab-a248659ed9d3", "durationMs": 60681, "timestamp": "2024-09-17T16:02:20.789568Z", "level": "INFO", "application": "/home/dockeruser/.local/bin/net2cog_harmony"}\r\nTraceback (most recent call last):\r\n File "/home/dockeruser/.local/bin/net2cog_harmony", line 8, in \u003Cmodule\u003E\r\n",
" sys.exit(main())\r\n File "/worker/net2cog/netcdf_convert_harmony.py", line 144, in main\r\n",

[D-Auty]

I'm pretty sure this is the result of various issues in handling different collections (see related issues) all "upstream" from the eventual exception. In these cases, the "list" object in question is None - intended to be the list of output files. When there is no output file, the exception is raised.

[owenlittlejohns]

I was taking a look at #35 in general, and get nerd sniped into trying to reproduce this error. A couple of quick things:

1. I had to tweak the example URL to express the granule IDs as separate values for the granuleId query parameter.
2. My request was then processed by HOSS, not by net2cog (good news - HOSS did a variable subset as requested 🎉).

I updated the request URL to specify a GeoTIFF output:

https://harmony.uat.earthdata.nasa.gov/C1256588789-ASDC_DEV2/ogc-api-coverages/1.0.0/collections/parameter_vars/coverage/rangeset?forceAsync=true&granuleId=G1269242639-ASDC_DEV&granuleId=G1269374306-ASDC_DEV2&variable=ztoa_net_all_mon&skipPreview=true&format=image%2Ftiff

That got me to reproduce the error as above. Looking at the logging for my request, it looks like:

• The file gets downloaded (CERES_EBAF_Edition4.2_200003-202309.nc).
• net2cog recognises that it needs to produce a COG for ztoa_net_all_mon.
• It looks like a COG is not generated (but the reason/condition why is not logged), so there's an empty list instead of a Path object.
• Then there's the error trying to do Path things on a list object.

There's a number of different exceptions being caught during this conversion, but the logging wasn't being emitted in a way Harmony was capturing. I tweaked that logging in my recent PR so I tried that version out locally and got a bit more of a clue:

raise Net2CogError(variable_name, aerr) from aerr\r\nnet2cog.netcdf_convert.Net2CogError:
Variable ztoa_net_all_mon cannot be converted to tif:
Variable 'y': Using a DataArray object to construct a variable is ambiguous, please extract the data using the .data property.

(FWIW: Looks like that's the general exception being caught here)