Similarity measures

[foster999]

Hey, I have both a general question and feature request. I'm looking into similarity measures for spectra for a couple of use cases:

1. Comparing variation within a group of repeated measures of the same sample (measurement error) to the difference between samples
2. Comparing single spectra with a reference spectra, to filter out spectra that are too dissimilar to use in modelling (e.g. contaminated or partially saturated spectra)

Would you have any suggestions on measures for these two cases? I've found that Pearson coefficients are likely good enough for 2, but because most of our samples are relatively very similar it may not reflect finer differences in 1.

Then, are similarity measures something that you'd consider including in the package?

As an aside, I was really excited to find the chemtools package - I'd already implemented many similar transformers, but this will give us more options for grid searching etc. There are a few bits under Issues that I've worked on, so will create or add to pull requests where relevant

[paucablop]

hi @foster999 sorry for the late answer! your comment got lost in my inbox! First thing is to thank you for your issue, I would like to say that contributions are mooooost welcome, and I would be very very happy if you would like to contribute to the project!! :)

To your questions, it usually depends on the specific use case, whenever we measure spectra that need to be processed by a model, we do the following checks:

1. We calculate the SNR of the spectra, we have two checks on the SNR:

• https://www.researchgate.net/publication/44089652_DER_SNR_A_Simple_General_Spectroscopic_Signal-to-Noise_Measurement_Algorithm
• Vector Casting for noise reduction (Journal of Raman Spectroscopy, Volume 21, Issue 4, p. 731-743)

We make sure that the SNR is not above certain values.

2. We project the new spectra onto the latent model space, and calculate the Hotelling T2 and Q Residuals. If the values are too far from what our model was trained on, we do not accept the spectra (we consider it too different).

We try to avoid (whenever possible) to compare to a single reference spectra, because we see too much variation, and the changes in the spectra we are trying to identify are often small (low concentrations). In one case we had to check if a new spectrum belongs to product A. In this case we actually measured product A many times in different days, and made a model to assess the measurement variability. Then did a PCA model of that dataset and projected new spectra onto that space as described in point 2.

Recently, we have started using some data augmentation, where we try to simulate instrument variation that is realistic within instrument specifications (https://github.com/paucablop/chemotools/tree/main/chemotools/augmentation) and this is working quite well to increase the robustness in some of our processes.

I hope this helps, but I will be happy to further discuss it :)

And also, remember that contributions are really welcome to the project and I would be very happy if you have any feedback and inputs to the project! :) :)