Caiman and VSCode crash while processing large video

[zyyk78]

Caiman may crash while processing large video (00:19:33, 5.3Gb, FFV1), without any reporting.
This error happened in CNMF-E (model.fit) step in about 10 seconds, and VSCode or terminal will crash and exit without any warning. I have tried different space/time shrink factor, but no effect. However smaller videos can be processed correctly.

def cnmfe_prog(cnmfe_params_dict=None):
    timetag=time.strftime("%Y%m%d_%H%M%S", time.localtime())
    cnmfe_params_dict['use_cuda']=True
    if cnmfe_params_dict is None:
        exit("Please provide CNMF parameters")
    cnmfe_params_dict["frames"]="'border_pix': bord_px"
    parameters = params.CNMFParams(params_dict=mc_dict)   
    movie_path=cnmfe_params_dict['fnames']
    folder_path=os.path.dirname(movie_path)
    
    
    if cnmfe_params_dict['do_motion_corr']:
        print("Doing Motion Corr.")
        mot_correct = MotionCorrect(movie_path, dview=cluster, **parameters.get_group('motion'))
        mot_correct.motion_correct(save_movie=True)
        fname_mc = mot_correct.fname_tot_els if cnmfe_params_dict['pw_rigid'] else mot_correct.fname_tot_rig
        if cnmfe_params_dict['pw_rigid']:
            bord_px = np.ceil(np.maximum(np.max(np.abs(mot_correct.x_shifts_els)),
                                        np.max(np.abs(mot_correct.y_shifts_els)))).astype(int)
        else:
            bord_px = np.ceil(np.max(np.abs(mot_correct.shifts_rig))).astype(int)
        bord_px = 0 if cnmfe_params_dict['border_nan'] == 'copy' else bord_px
        fname_new = cm.save_memmap(fname_mc, base_name='memmap_', order='C',
                                border_to_0=bord_px)   
        Yr, dims, T = cm.load_memmap(fname_new)
        images = Yr.T.reshape((T,) + dims, order='F')
        parameters.change_params(params_dict={'border_pix': bord_px})
        
        movie_corrected = cm.load(mot_correct.mmap_file) # load motion corrected movie
        movie_corrected.save(folder_path + r'/corrected_'+ timetag + '.avi',compress=1,q_min=0,q_max=100)
        
    else:
        print('Skip Motion Corr')
        fname_new = cm.save_memmap([movie_path], base_name='memmap_',
                               order='C', border_to_0=0, dview=cluster)
        bord_px=5
        Yr, dims, T = cm.load_memmap(fname_new)
        images = Yr.T.reshape((T,) + dims, order='F')
        parameters.change_params(params_dict={'border_pix': bord_px})
    
    print('Doing CNMF')    
    cnmfe_model = cnmf.CNMF(n_processes=n_processes, dview=cluster, params=parameters)
    cnmfe_model.fit(images)
    quality_params = {
                  'min_SNR':5,  #????????????
                  'SNR_lowest': 2.5,  #????????????
                  'rval_thr': 0.5,
                  'rval_lowest': 0.2,  #???????????
                  'use_cnn': False}
    cnmfe_model.params.change_params(params_dict=quality_params)
    cnmfe_model.estimates.evaluate_components(images, cnmfe_model.params, dview=cluster)
    cnmfe_model.estimates.threshold_spatial_components()
    cnmfe_model.estimates.remove_small_large_neurons(min_size_neuro=cnmfe_params_dict['min_cell_size'], max_size_neuro=cnmfe_params_dict['max_cell_size'],select_comp=False)
    print('*****')
    print(f"Total number of components: {len(cnmfe_model.estimates.C)}")
    print(f"Number accepted: {len(cnmfe_model.estimates.idx_components)}")
    print(f"Number rejected: {len(cnmfe_model.estimates.idx_components_bad)}")
    correlation_image, peak_to_noise_ratio = cm.summary_images.correlation_pnr(images[::max(T//1000, 1)], # subsample if needed
                                                                            gSig=cnmfe_params_dict['gSig'][0], # used for filter
                                                                            swap_dim=False) # change swap dim if output looks weird, it is a problem with tiffile
    save_path =  folder_path + r'/results_'+ timetag+'.hdf5' 
    cnmfe_model.estimates.Cn = np.max(images, axis=0) # ????max??proj?
    cnmfe_model.save(save_path)
    plt.subplot(1,2,1)
    plt.imshow(correlation_image)
    plt.title('Correlation Image')
    plt.subplot(1,2,2)
    plt.imshow(peak_to_noise_ratio)
    plt.title('Peak to Noise Ratio')
    plt.savefig(folder_path + r'/correlation_pnr_'+ timetag + '.png', dpi=600) 
    plt.close()
    print('Results saved')
    shutil.rmtree(os.environ["CAIMAN_DATA"]+r'/temp') # remove temporary memmap file
    os.mkdir(os.environ["CAIMAN_DATA"]+r'/temp')
    print('Temporary memmap file removed')  

def process_dict(dict_in):
    import ast
    dict_in['fr']=int(dict_in['fr'])
    dict_in['decay_time']=float(dict_in['decay_time'])
    dict_in['pw_rigid']=bool(int(dict_in['pw_rigid']))

    dict_in['max_shifts']=ast.literal_eval(dict_in['max_shifts'])
    dict_in['gSig_filt']=ast.literal_eval(dict_in['gSig_filt'])
    dict_in['strides']=ast.literal_eval(dict_in['strides'])
    dict_in['overlaps']=ast.literal_eval(dict_in['overlaps'])
    dict_in['max_deviation_rigid']=int(dict_in['max_deviation_rigid'])
    if dict_in['K']!='N':
        dict_in['K']=int(dict_in['K'])
    else:
        dict_in['K']=None
    dict_in['gSig']=np.asarray(ast.literal_eval(dict_in['gSig'])) 
    dict_in['gSiz']=np.asarray(ast.literal_eval(dict_in['gSiz'])) 
    dict_in['merge_thr']=float(dict_in['merge_thr'])
    dict_in['p']=int(dict_in['p'])
    dict_in['tsub']=int(dict_in['tsub'])
    dict_in['ssub']=int(dict_in['ssub'])
    if dict_in['rf']=='N':
        dict_in['rf']=None
    else:
        dict_in['rf']=int(dict_in['rf'])
    dict_in['stride']=int(dict_in['stride'])
    dict_in['only_init']=bool(int(dict_in['only_init']))
    dict_in['nb']=int(dict_in['nb'])
    dict_in['nb_patch']=int(dict_in['nb_patch'])
    if dict_in['low_rank_background']=='N':
        dict_in['low_rank_background']=None
    else:
        dict_in['low_rank_background']=int(dict_in['low_rank_background'])
    dict_in['update_background_components']=bool(int(dict_in['update_background_components']))
    dict_in['min_corr']=float(dict_in['min_corr'])
    dict_in['min_pnr']=float(dict_in['min_pnr'])
    dict_in['ssub_B']=int(dict_in['ssub_B'])
    dict_in['ring_size_factor']=float(dict_in['ring_size_factor'])
    dict_in['del_duplicates']=bool(int(dict_in['del_duplicates']))
    dict_in['do_motion_corr']=bool(int(dict_in['do_motion_corr']))
    dict_in['min_cell_size']=int(dict_in['min_cell_size'])
    dict_in['max_cell_size']=int(dict_in['max_cell_size'])
    return dict_in

conf_file=input('?????????')   # Load csv configuration file
if conf_file is None:
    exit("Please provide a configuration file")
import os
os.environ["MKL_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["VECLIB_MAXIMUM_THREADS"] = "1"
os.environ["CAIMAN_DATA"]= "/opt/CaImAn_1.11.3/caiman_data"
os.environ['PATH']=os.environ['PATH']+':/opt/anaconda3/envs/caiman/libexec/gcc/x86_64-conda-linux-gnu'
os.environ['KERAS_BACKEND']='tensorflow'

import time
import pandas as pd
import shutil
import cv2
from IPython import get_ipython
import matplotlib.pyplot as plt
import numpy as np
import psutil
import caiman as cm
from caiman.source_extraction import cnmf
from caiman.motion_correction import MotionCorrect
from caiman.source_extraction.cnmf import params as params
try:
    if __IPYTHON__:
        get_ipython().run_line_magic('load_ext', 'autoreload')
        get_ipython().run_line_magic('autoreload', '2')
except NameError:
    pass
try:
    cv2.setNumThreads(0)
except:
    pass
num_processors_to_use = 70
print(f"You have {psutil.cpu_count()} CPUs available in your current environment")
if 'cluster' in locals():  # 'locals' contains list of current local variables
    print('Closing previous cluster')
    cm.stop_server(dview=cluster)
print("Setting up new cluster")
_, cluster, n_processes = cm.cluster.setup_cluster(backend='multiprocessing', 
                                                n_processes=num_processors_to_use, 
                                                ignore_preexisting=False)
print(f"Successfully set up cluster with {n_processes} processes")
 
df_loaded = pd.read_csv(conf_file)
titles = df_loaded.columns.tolist()
titles.remove('Parameter')
titles.remove('Commitment')
for title in titles:
    print('Processing: ', title)
    tm=time.time()
    loaded_dict = dict(zip(df_loaded['Parameter'], df_loaded[title]))
    mc_dict = process_dict(loaded_dict)
    cnmfe_prog(mc_dict)
    print('Runtime:%fs'%(time.time()-tm))
cm.stop_server(dview=cluster)

here is my key paramenters:

[kushalkolar]

Does it work in jupyterlab? VScode's notebook implementation can be buggy.

BTW from your code it looks like you're trying to do param sweeps/data organization, mesmerize-core was built for this so you might want look at that to make that easier: https://github.com/nel-lab/mesmerize-core

[zyyk78]

Thank you for you reply! I have found the reason: CPU core number for using. I turnd it from 70 to 10 and it can work now (we have 104 core totally), but i don't know why.
There seems to be no relations to IDE, and it happened in jupyter notebook/lab, python in shell, python in vscode.

[zyyk78]

I found that if i use more cpu the caiman will be more instable, it may crash in any step,

[kushalkolar]

I found that if i use more cpu the caiman will be more instable, it may crash in any step,

You're probably running out of RAM, see #1468 (comment)

[zyyk78]

That make sense, thank you!

[zyyk78]

i found the ture reason: systemd-oomd killed it

[kushalkolar]

Yea that's the out of memory killer

…

On Fri, Mar 7, 2025, 12:41 PM zyyk78 ***@***.***> wrote: i found the ture reason: systemd-oomd killed it — Reply to this email directly, view it on GitHub <#1471 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACHXXRDYKXPY4T4DTRXKUKD2THK6FAVCNFSM6AAAAABXS3CXSSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBXGAZTSNZXGQ> . You are receiving this because you commented.Message ID: ***@***.***> [image: zyyk78]*zyyk78* left a comment (flatironinstitute/CaImAn#1471) <#1471 (comment)> i found the ture reason: systemd-oomd killed it — Reply to this email directly, view it on GitHub <#1471 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACHXXRDYKXPY4T4DTRXKUKD2THK6FAVCNFSM6AAAAABXS3CXSSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBXGAZTSNZXGQ> . You are receiving this because you commented.Message ID: ***@***.***>