Index error in toposphere correction

[Akashpsas555]

Description of the problem

Some index error had been occured during the tropospheric correction in file Mintpy time series from prepared stack from the section 3.7. Correct for Tropospheric Propagation Delays (Optional). I had created CDS account and added CDS UID and API token previously. Should I add any request in the CDS website?

Full script that generated the error

if type(tropo_choice) != str:
    tropo_choice = tropo_choice.value

correct_tropo = "Perform" in tropo_choice or "Rerun" in tropo_choice or "Done" in tropo_choice

era5_path = mint_path/"ERA5"
timeseries_era5_path = mint_path/"timeseries_ERA5.h5"
inputs_era5_path = mint_path/"inputs/ERA5.h5"

if "Delete" in tropo_choice:
    for f in [timeseries_era5_path, inputs_era5_path]:
        try:
            f.unlink()
        except FileNotFoundError:
            pass
    try:
        shutil.rmtree(era5_path)
    except FileNotFoundError:
        pass

if correct_tropo and "Done" not in tropo_choice:
    set_troposhperic_correction_mintpy(config_path, "pyaps")
    !smallbaselineApp.py $config_path --work-dir {mint_path} --dostep load_data
    !smallbaselineApp.py $config_path --work-dir {mint_path} --dostep correct_troposphere
elif not correct_tropo:
    set_troposhperic_correction_mintpy(config_path, "no")
    !smallbaselineApp.py $config_path --work-dir {mint_path} --dostep load_data

Full error message

MintPy version 1.5.3, date 2023-11-23
--RUN-at-2025-04-27 19:16:03.393222--
Current directory: /home/jovyan/notebooks/SAR_Training/English/Master
Run routine processing with smallbaselineApp.py on steps: ['load_data']
Remaining steps: ['modify_network', 'reference_point', 'quick_overview', 'correct_unwrap_error', 'invert_network', 'correct_LOD', 'correct_SET', 'correct_troposphere', 'deramp', 'correct_topography', 'residual_RMS', 'reference_date', 'velocity', 'geocode', 'google_earth', 'hdfeos5']
--------------------------------------------------
Project name: akash
Go to work directory: /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy
read custom template file: /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/akash.txt
update default template based on input custom template
No new option value found, skip updating /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/smallbaselineApp.cfg
copy akash.txt            to inputs   directory for backup.
copy akash.txt            to pic      directory for backup.
read default template file: /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/smallbaselineApp.cfg


******************** step - load_data ********************

load_data.py --template /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/smallbaselineApp.cfg /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/akash.txt --project akash
processor : hyp3
SAR platform/sensor : unknown from project name "akash"
--------------------------------------------------
prepare metadata files for hyp3 products
prep_hyp3.py "/home/jovyan/notebooks/SAR_Training/English/Master/Research/*/*unw_phase_clip_clip_clip.tif"
/home/jovyan/.local/envs/osl_mintpy/lib/python3.9/site-packages/osgeo/gdal.py:312: FutureWarning: Neither gdal.UseExceptions() nor gdal.DontUseExceptions() has been explicitly called. In GDAL 4.0, exceptions will be enabled by default.
  warnings.warn(
prep_hyp3.py "/home/jovyan/notebooks/SAR_Training/English/Master/Research/*/*corr_clip_clip_clip.tif"
prep_hyp3.py "/home/jovyan/notebooks/SAR_Training/English/Master/Research/*/*dem_clip_clip_clip.tif"
prep_hyp3.py "/home/jovyan/notebooks/SAR_Training/English/Master/Research/*/*lv_theta_clip_clip_clip.tif"
prep_hyp3.py "/home/jovyan/notebooks/SAR_Training/English/Master/Research/*/*lv_phi_clip_clip_clip.tif"
prep_hyp3.py "/home/jovyan/notebooks/SAR_Training/English/Master/Research/*/*water_mask_clip_clip_clip.tif"
--------------------------------------------------
updateMode : True
compression: None
multilook x/ystep: 1/1
multilook method : nearest
--------------------------------------------------
searching geometry files info
input data files:
height          : /home/jovyan/notebooks/SAR_Training/English/Master/Research/S1AA_20220101T132327_20220218T132325_VVP048_INT80_G_ueF_EF6C/S1AA_20220101T132327_20220218T132325_VVP048_INT80_G_ueF_EF6C_dem_clip_clip_clip.tif
incidenceAngle  : /home/jovyan/notebooks/SAR_Training/English/Master/Research/S1AA_20220101T132327_20220218T132325_VVP048_INT80_G_ueF_EF6C/S1AA_20220101T132327_20220218T132325_VVP048_INT80_G_ueF_EF6C_lv_theta_clip_clip_clip.tif
azimuthAngle    : /home/jovyan/notebooks/SAR_Training/English/Master/Research/S1AA_20220101T132327_20220218T132325_VVP048_INT80_G_ueF_EF6C/S1AA_20220101T132327_20220218T132325_VVP048_INT80_G_ueF_EF6C_lv_phi_clip_clip_clip.tif
waterMask       : /home/jovyan/notebooks/SAR_Training/English/Master/Research/S1AA_20220101T132327_20220218T132325_VVP048_INT80_G_ueF_EF6C/S1AA_20220101T132327_20220218T132325_VVP048_INT80_G_ueF_EF6C_water_mask_clip_clip_clip.tif
All datasets exists in file geometryGeo.h5 with same size as required, no need to re-load.
--------------------------------------------------
searching interferogram pairs info
input data files:
unwrapPhase     : /home/jovyan/notebooks/SAR_Training/English/Master/Research/*/*unw_phase_clip_clip_clip.tif
coherence       : /home/jovyan/notebooks/SAR_Training/English/Master/Research/*/*corr_clip_clip_clip.tif
number of unwrapPhase     : 144
number of coherence       : 144
All date12   exists in file ifgramStack.h5 with same size as required, no need to re-load.
--------------------------------------------------
searching ionosphere pairs info
input data files:
WARNING: No data files found for the required dataset: ['unwrapPhase']! Skip loading for ionosphere stack.
--------------------------------------------------
searching offset pairs info
input data files:
WARNING: No data files found for the required dataset: ['rangeOffset', 'azimuthOffset']! Skip loading for offset stack.
time used: 00 mins 1.2 secs.

No lookup table (longitude or rangeCoord) found in files.
Input data seems to be geocoded. Lookup file not needed.
Loaded dataset are processed by InSAR software: hyp3
Loaded dataset are in GEO coordinates
Interferogram Stack: /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/inputs/ifgramStack.h5
Geometry      File : /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/inputs/geometryGeo.h5
Lookup Table  File : None
--------------------------------------------------
updating metadata based on custom template file akash.txt for file: ifgramStack.h5
Go back to directory: /home/jovyan/notebooks/SAR_Training/English/Master

################################################
   Normal end of smallbaselineApp processing!
################################################
Time used: 00 mins 1.2 secs

MintPy version 1.5.3, date 2023-11-23
--RUN-at-2025-04-27 19:16:05.948253--
Current directory: /home/jovyan/notebooks/SAR_Training/English/Master
Run routine processing with smallbaselineApp.py on steps: ['correct_troposphere']
Remaining steps: ['deramp', 'correct_topography', 'residual_RMS', 'reference_date', 'velocity', 'geocode', 'google_earth', 'hdfeos5']
--------------------------------------------------
Project name: akash
Go to work directory: /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy
read custom template file: /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/akash.txt
update default template based on input custom template
No new option value found, skip updating /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/smallbaselineApp.cfg
read default template file: /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/smallbaselineApp.cfg


******************** step - correct_troposphere ********************
Input data seems to be geocoded. Lookup file not needed.
Atmospheric correction using Weather Re-analysis dataset (PyAPS, Jolivet et al., 2011)
Weather Re-analysis dataset: ERA5
--------------------------------------------
Use existed tropospheric delay file: ./inputs/ERA5.h5

diff.py /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/timeseries.h5 ./inputs/ERA5.h5 -o /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/timeseries_ERA5.h5 --force
/home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/timeseries.h5 - ['./inputs/ERA5.h5'] --> /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/timeseries_ERA5.h5
the 1st input file is: timeseries
WARNING: ./inputs/ERA5.h5 does not contain all dates in /home/jovyan/notebooks/SAR_Training/English/Master/Research/MintPy/timeseries.h5
Continue and enforce the differencing for their shared dates only.
	with following dates are ignored for differencing:
['20230120', '20220314', '20230309', '20220326', '20220630', '20220817', '20231116', '20230625', '20220618', '20230929', '20220525', '20230520', '20230508', '20220419', '20230613', '20220606', '20221121', '20230707', '20230601', '20230225', '20221203', '20220724', '20220501', '20231104', '20231023', '20230824', '20220829', '20230201', '20230812', '20230731', '20220101', '20220302', '20220805', '20220922', '20220218', '20221004', '20221215', '20221227', '20230213', '20220113', '20230321', '20231011', '20231210', '20220513', '20230414', '20230917', '20220910', '20231222', '20231128', '20220125', '20220206', '20220712', '20230108', '20230402', '20230719', '20221109', '20221028', '20230426', '20220407', '20230905']
Traceback (most recent call last):
  File "/home/jovyan/.local/envs/osl_mintpy/bin/smallbaselineApp.py", line 10, in <module>
    sys.exit(main())
  File "/home/jovyan/.local/envs/osl_mintpy/lib/python3.9/site-packages/mintpy/cli/smallbaselineApp.py", line 208, in main
    run_smallbaselineApp(inps)
  File "/home/jovyan/.local/envs/osl_mintpy/lib/python3.9/site-packages/mintpy/smallbaselineApp.py", line 1117, in run_smallbaselineApp
    app.run(steps=inps.runSteps)
  File "/home/jovyan/.local/envs/osl_mintpy/lib/python3.9/site-packages/mintpy/smallbaselineApp.py", line 898, in run
    self.run_tropospheric_delay_correction(sname)
  File "/home/jovyan/.local/envs/osl_mintpy/lib/python3.9/site-packages/mintpy/smallbaselineApp.py", line 642, in run_tropospheric_delay_correction
    mintpy.cli.diff.main(iargs)
  File "/home/jovyan/.local/envs/osl_mintpy/lib/python3.9/site-packages/mintpy/cli/diff.py", line 86, in main
    diff_file(
  File "/home/jovyan/.local/envs/osl_mintpy/lib/python3.9/site-packages/mintpy/diff.py", line 390, in diff_file
    diff_timeseries(file1, file2[0], out_file, force_diff, max_num_pixel)
  File "/home/jovyan/.local/envs/osl_mintpy/lib/python3.9/site-packages/mintpy/diff.py", line 100, in diff_timeseries
    ref_val = readfile.read(file2, datasetName=date_list_shared, box=ref_box)[0] * unit_fac
  File "/home/jovyan/.local/envs/osl_mintpy/lib/python3.9/site-packages/mintpy/utils/readfile.py", line 331, in read
    dsname4atr = datasetName[0].split('-')[0]
IndexError: list index out of range

System information

• Operating system: windows 10
• Python environment:
• MintPy version: MintPy 1.5.1
• InSAR processor/product:
• Your custom / default template file (if the bug is related to a specific dataset):

[barely-sad-one]

maybe the pyaps downloaded the partial file, Trackback showing there are missing date in ERA5 data.
Try to delete all the ERA5 dataset in root and input directory and rerun the section.

otherwise it will reload the existing files

Use existed tropospheric delay file: ./inputs/ERA5.h5