Kernel PCA

Kernel PCA

Kernel Principal Component Analysis is as the name suggests an extension of PCA, that allows for nonlinear data by making use of kernels. The basic idea behind Kernel PCA is to project the data onto a higher dimensional space where there can be made new principal components as in the normal PCA.

But making a new dimension would be pretty expensive i computing cost, therefore Kernel PCA makes of something called the Kernel trick. Instead of squaring the values needed to projected onto the new dimension. We use the kernel function $\kappa(\textbf{x}_i,\textbf{x}_j)=\phi(\textbf{x}_i)^T\phi(\textbf{x}_j)$. This function can be used as the covariance matrix as in PCA, this way we can calculate eigenvalues and eigenvectors. The kernel trick is it enables to calculate the eigenvalues and eigenvectors without calculating $\phi(\textbf{x})$ explicitly.

These are the steps to calculate kernel PCA.

1. Construct the kernel Matrix $\textbf{K}$ from the set {$x_1$}

$K_{i,j} =\kappa(\textbf{x}_i,\textbf{x}_j)$

2. Compute the Gram matrix $\tilde K$

$\tilde K=K-1_N K-K1_N+1_NK1_N$ Where $1_N$ is the $N x N$ matrix with all elements equal to 1/N.

3. Use the formula to solve for vectors

$\tilde KA_k=\lambda_kNa_k$

4. Compute the kernel principal components

$y_k(x)=\phi(x)^Tv_k=\Sigma^N_{i=1}a_ki\kappa(x,x_i)$

sources

https://www.geeksforgeeks.org/ml-introduction-to-kernel-pca/

https://arxiv.org/pdf/1207.3538.pdf

https://nirpyresearch.com/pca-kernel-pca-explained/

https://ml-explained.com/blog/kernel-pca-explained

https://www.youtube.com/watch?v=efR1C6CvhmE

https://scikit-learn.org/stable/auto_examples/decomposition/plot_kernel_pca.html#sphx-glr-auto-examples-decomposition-plot-kernel-pca-py

https://towardsdatascience.com/kernel-pca-vs-pca-vs-ica-in-tensorflow-sklearn-60e17eb15a64