Distance computation question on pearson=mdCosine

[martabe]

Hi,

I was going through the code trying to understand exactly how are the different distance metrics calculated (in particular I wanted to understand the difference between using pearson and pearson2).
I may have missed some passages, and forgive me if I did, but in line 97 of the script umap_checks.R I stumbled upon this:

available.metrics <- c(manhattan=mdManhattan, pearson2=mdCenteredPearson, pearson=mdCosine, cosine=mdCosine, euclidean=mdEuclidean)

where pearson=mdCosine.

Is there a reason for pearson and cosine to point at the same function?
Do I understand correctly that metric = pearson will use the mdCosine function to calculate distances?

Thank you for putting together this package, it has worked really well so far :)
Marta

[tkonopka]

Hi Marta. Thanks for using the package and reaching out. Yes, your understanding is correct: setting the metric to "pearson" or to "cosine" will give the same results. The reasons are partly mathematical/computational and partly accidental/historical.

From the math perspective, the formulas for pearson correlation and cosine similarity are quite similar. The formulas differ only by a shift that is applied to each vector when computing the pearson correlation. Suppose we could assume the shift is always zero, then formulas would be equivalent. Importantly, a null shift would also reduce the computational effort needed for the calculation. Since this implementation of UMAP spends quite a bit of time calculating distances, it makes sense to optimize that step. So this package asks you to pre-apply the shifts before invoking the dimensional reduction, so that the package can use cosine distance for the actual computation. To be clear, this pecularity does not limitation what you can achieve with the package, but you should be careful to use centered/shifted data when interpreting the calculations as pearson correlations.

The other reason is somewhat historical. From memory, I believe the keyword "pearson" entered the implementation in an early version of the code. Later, to retain backward compatibility, that keyword stayed. I agree that now it causes more confusion than value-add.

[martabe]

Thank you for the answer.
Then I should always center the matrix before running umap with pearson and pearson2 as metrics, right?

[tkonopka]

Yes, centering will make the distances consistent with interpretation as pearson and pearson-squared.

Btw, since you're already deep in the code, the implementation of the distances are in file src/distances.Rcpp.

Also, for satisfaction you may like to check a handful of distances manually to be sure the numerics correspond to what you have in mind. The matrices $knn$indexes and $knn$distances are useful for this

library(umap)
example = umap(iris[,1:4])

target = 1
neighbors_of_target = example$knn$indexes[target, ]
distances_to_target = example$knn$distances[target,]

This ^ will give a vector of indexes that are nearest neighbors to a target (here index=1), and a vector of corresponding distances. It should be possible to reproduce a handful of those distances via manual calculations. Hope this helps :)