Dimensionality reduction is a subtle form of feature selection deserving its own section of this notebook. Unlike feature selection discussed in the previous section the techniques used in dimensionality reduction derive brand new (latent) features from the existing dataset. These features are not directly measurable but describe a general trend or direction in the data that we may project our data onto while losing a minimal amount of information. Here I will discuss 2 common methods to apply dimensionality reduction and how they work.
Principle Component Analysis
Principal Component Analysis (PCA) is a very useful unsupervised learning technique that can be used to tease out the latent features in a dataset. It achieves this by identifying and reframing the dataset by what are called its principle components (PC), the combination of attributes (directions in the feature space) that account for the most variance in the data.
It is mathematical fact that projecting data points onto the direction of maximum variance we minimise the distance from a higher dimensionality data point to its new transformed value. This translates to a minimal information loss after the transformation.
As a general process PCA is able to algorithmically create new features from a
dataset and rank the relative powers of each new feature. So if we have n
latent features that are driving most of the variation in the data (remember
this minimises the information loss) PCA will pick those out and rank them as
the 1st, 2nd, ..., nth principle components where the 1st PC has the most
effect, and the maximum number of PCs is equal to
min(num_features, num_examples).
More specifically PCA finds a new coordinate space, through translation and rotation only, moving the centre of the coordinate system to the centre of the data. PCA works on eigenvectors and eigenvalues of the covariance matrix, which is the equivalent of fitting those principle-component hyperplanes to the variance of the data. Why? Because eigenvectors trace the principal lines of force. In other words, PCA determines the lines of variance in the dataset which are so called its principal components with the first principal component having the maximum variance, second principal component (orthogonal to the first) having second maximum variance and so on.
When used for dimensionality reduction PCA will discard the principle components ranking the lowest and therefore having the least effect. This therefore reduces any noise in the dataset that is either adding little (or no) information or could be derived from other features.
Linear Discriminant Analysis
Linear Discriminant Analysis (LDA) is a supervised learning technique that in addition to identifying attributes contributing the most variance in the data (the scatter in LDA terms), as PCA does, computes the linear discriminants (LD) representing the directions in the feature space that maximise the separation between classes. This ensures good class separability in the dataset.
In general LDA algorithmically finds the boundaries around the clusters of classes, projecting the data points onto the direction of maximum distance between each class such that they are as separated as possible, with each cluster having a relative (close) distance to a centroid. Furthermore LDA identifies these cluster as satisfying two criteria:
- Maximize the distance between the centroid of each class,
- Maximize the variation within each class.
When used for dimensionality reduction LDA will discard the linear discriminants that explain the least distance between classes thus reducing any potential noise generated by features that may prosess some overlap.
Linear Discriminant Analysis is particularly suited to datasets that have more than two classes and is the preferred linear classification technique, say over logistic regression, that is limited to only two-class classification problems.