OnAcid with rigid motion correction on large files causes error: IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed

[katonage]

CaImAn: 1.11.5 , windows

Goal

Trying to set up Onacid according to demo_OnACID_mesoscope.ipynb, but setting my own large file as input (>20.000 frames)
Using rigid motion correction.

Problem

This causes cnm.fit_online() throwing an error !

After debuging the problem I found the following mechanism

• in the demo: init_batch = 200.
• if params.py / check_consistency()
  -(line 925):
  num_splits = max(T//max(self.motion['num_frames_split'], 10), 1)
  -(line 927):
  self.motion['splits_rig'] = num_splits
  Since T is the total data length, being very high, this sets splits_rig to a large number, in my case larger than 1000
• online_cnmf / fit_online() calls
  -initialize_online() calls (on the init_batch)
  -motion_correct() calls
  • motion_correct_rigid() calls
    • motion_correct_batch_rigid() that tries to split the init_batch of length 200 to slits_rig parts being larger than 200, ending up

Error

in:
    File ~\Repositories\CaImAn\caiman\motion_correction.py:3186, in motion_correction_piecewise(fname, splits, strides, overlaps, add_to_movie, template, max_shifts, max_deviation_rigid, newoverlaps, newstrides, upsample_factor_grid, order, dview, save_movie, base_name, subidx, num_splits, shifts_opencv, nonneg_movie, gSig_filt, use_cuda, border_nan, var_name_hdf5, is3D, indices, shifts_interpolate)
   [3184](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/KatonaGergely/Repositories/twop/twop/twop_analysis/caiman/~/Repositories/CaImAn/caiman/motion_correction.py:3184)     logger.info('** Finished parallel motion correction **')
   [3185](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/KatonaGergely/Repositories/twop/twop/twop_analysis/caiman/~/Repositories/CaImAn/caiman/motion_correction.py:3185) else:
-> [3186](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/KatonaGergely/Repositories/twop/twop/twop_analysis/caiman/~/Repositories/CaImAn/caiman/motion_correction.py:3186)     res = list(map(tile_and_correct_wrapper, pars))
   [3188](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/KatonaGergely/Repositories/twop/twop/twop_analysis/caiman/~/Repositories/CaImAn/caiman/motion_correction.py:3188) return fname_tot, res

File ~\Repositories\CaImAn\caiman\motion_correction.py:3071, in tile_and_correct_wrapper(params)
   [3068](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/KatonaGergely/Repositories/twop/twop/twop_analysis/caiman/~/Repositories/CaImAn/caiman/motion_correction.py:3068) shift_info = []
   [3070](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/KatonaGergely/Repositories/twop/twop/twop_analysis/caiman/~/Repositories/CaImAn/caiman/motion_correction.py:3070) imgs = caiman.load(img_name, subindices=idxs, var_name_hdf5=var_name_hdf5,is3D=is3D)
-> [3071](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/KatonaGergely/Repositories/twop/twop/twop_analysis/caiman/~/Repositories/CaImAn/caiman/motion_correction.py:3071) imgs = imgs[(slice(None),) + indices]
   [3072](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/KatonaGergely/Repositories/twop/twop/twop_analysis/caiman/~/Repositories/CaImAn/caiman/motion_correction.py:3072) mc = np.zeros(imgs.shape, dtype=np.float32)
   [3073](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/KatonaGergely/Repositories/twop/twop/twop_analysis/caiman/~/Repositories/CaImAn/caiman/motion_correction.py:3073) if not imgs[0].shape == template.shape:

IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed

So the problem is that the shifts_rig parameter, that has been initialized for the whole array, is actually used to the much shorter init_batch.

Workaround:

Increase init_batch to larger values
if that is not enough,
Overdefine splits_rig after creating opts:

if opts.motion['splits_rig']>init_batch/2:
         opts.motion['splits_rig']=init_batch/2

Solution

Split parameters should be redefined during the parameter setting of the motion correction of the init batch.

Bests!

[pgunn]

We should look for this rather than stumble over it in numpy; we may prefer to throw an exception telling the user to use a larger init_batch. We'll be looking at this more to see what other options we have.