Always delete `raw` Colabfold folder before processing

[qurat-ul-ain95]

Hi Team,

This is not an issue per se but is definitely a strong suggestion by an end user. It's possible for an openfold job to fail for N number of reasons and a lot of the times this happens before cleanup: i.e. the raw colabfold output folder and stale out.tar.gz files remain in the MSA dir. Subsequent runs skip hitting the Colabfold API because of

`if not os.path.isfile(tar_gz_file):` 

in core/data/tools/colabfold_msa_server.py

which leads to the pipeline using older colabfold raw output folder for the new query as this check only checks file existence, not whether it matches the current query. This results in nondescript downstream errors which are very difficult for the end user to investigate.

This check for tar_gz_file is redundant anyway because if the query runs properly, this tar_gz_file is deleted along with the entire raw folder. On rerun, the pipeline always calls the API again.

My suggestion is to always delete the raw colabfold folder before this line runs. What do you think? Thanks!

[jnwei]

I agree it is reasonable to check for the raw colabfold folder before the run, as it can lead to very difficult to debug issues as you raised.

As for what to do with the raw folder, I am a bit concerned if the user would want inspect the outputs themselves first and would be surprised by the deletion of the directory. Do you think this is a likely use case? In this case, it may be safer to direct the user to manually delete the raw directory, rather than having the pipeline delete it.

If we think this use case is unlikely, then I think it makes sense to log a warning and delete the directory as you suggest. Please feel free to incorporate this change in your PR #35

[qurat-ul-ain95]

I am a bit concerned if the user would want inspect the outputs themselves first and would be surprised by the deletion of the directory.

The raw folder is deleted during cleanup anyway and is never accessible to the user.

Moreover raw folder contains MSA/template information of the entire batch - the identifiers are colabfold identifiers (101, 102 etc) with no mapping provided. #35 preserves the MSAs in raw folder by rep_id which is immediately more useful for the user because the rep_id to seq mapping is also provided.