Early fusion of CNV and Histopathology to get earlier prediction

[rzuberi]

From may to august 2024 I worked on late fusion of copy number variation (CNV) and histopathology microscopy images (H) data to predict cancer in patients with Barrett's oesophagus (BE). Killcoyne et al.'s 2020 paper showed that CNV can predict cancer in BE patients up to 15 years in advance, whereas H is used by pathologists for diagnosis. We put CNV in a gradient boosting model to get the probability of BE progressing into a worst-case. We put the images through a pre-trained feature collecting model (UNI model by the Mahmood lab) and then trained a ResNet model (CLAM model by the Mahmood lab as well) and got the probability of BE progressing into a worst-case. There were two types of prediction: 3-classes (Risk Factor) which are predicting if the condition got more severe in a graded pathology fashion, and 2-classes (Progressor) which predicted if the patient became a progressor or not (it's important to note that each BE patient starts at just being BE with not cancer so thats what we were predicting from). We evaluated these models individually and then evaluated if the overall accuracy improved if we merged the probabilities of these models. We found the following results:

As can be noticed in the first column (the "Modality Type"), there is one row at the end that is called "Early Fusion". This is when we merge the CNV and the Hs together to make it a single data modality that we can then put into one model. In the research project we:

• got the features out from the images with the UNI model;
• normalised the CNV data with the mean and standard deviation of the images' features;
• append the CNV data to the image feature vectors;
• trained a model with the CLAM architecture to predict the Progressor status.
  We got good results from this method on the Progressor status, but ran out of time to do it on the Risk Factor task.

[rzuberi]

First step is to reproduce our Late Fusion Progressor Status task and see if we get the same results since I haven't looked at the code in 6 weeks, then we can do the Risk Factor.

• I found the path with the original files: /mnt/scratchc/fmlab/zuberi01/masters/saved_patches/40x_400/features2/pt_files
• I found the path with the merged files: /mnt/scratchc/fmlab/zuberi01/masters/saved_patches/40x_400/features2/pt_files_updated
  Important to remember that these are the files which I studied this summer therefore it is not the updated data its still the old one.

There's some updated data for the CNV data somewhere that I need to check for and integrate as well somehow.

Also I did this work on my Linux desktop machine before but now I am working from my Mac. This shouldn't be an issue since everything in on a cluster but good to remember when checking for issues. Furthermore, Will made some modifications which I merged and they might mean that my specifications when trying out jobs will be different.

[rzuberi]

I realised that the code I made this summer to normalise the CNV data to add it to the end of the features of the images may not be present in the repo so I'll have to redo it, shouldn't take long.

I made a script to do the progressor task with the merged data, but (see point above) the data may not be merged, and I need to remake the environment with the proper CLAM env.yaml file.

I contacted Tim to see if he knows if I can get the updated dataset.

[rzuberi]

Our CNV data is binned every 5mb. I'm exploring in a notebook which megabases is most important. I'm using a Random Forest model to classify the data into non-progressor or progressor.

We have 756 biopsies.
Non-progressors: 420
Progressors: 336

Train: 483
Val: 121
Test: 152

Model	Test Accuracy
CNV	0.8289
Only p	0.7961
Only q	0.7961
Only p and q	0.8158
CNV, p, and q	0.8487

Are there regions in p, q or CNV where we can see there are regions that are more indicative?

How do we match the rows between CNV data and Histopathology data?
How many rows are we missing?