Linear discriminant analysis. LDA is based on the Bayes decision theory
and assumes that the conditional probability density functions are normally
distributed. LDA also makes the simplifying homoscedastic assumption (i.e.
that the class covariances are identical) and that the covariances have full
rank. With these assumptions, the discriminant function of an input being
in a class is purely a function of this linear combination of independent
variables.
LDA is closely related to ANOVA (analysis of variance) and linear regression
analysis, which also attempt to express one dependent variable as a
linear combination of other features or measurements. In the other two
methods, however, the dependent variable is a numerical quantity, while
for LDA it is a categorical variable (i.e. the class label). Logistic
regression and probit regression are more similar to LDA, as they also
explain a categorical variable. These other methods are preferable in
applications where it is not reasonable to assume that the independent
variables are normally distributed, which is a fundamental assumption
of the LDA method.
One complication in applying LDA (and Fisher's discriminant) to real data
occurs when the number of variables/features does not exceed
the number of samples. In this case, the covariance estimates do not have
full rank, and so cannot be inverted. This is known as small sample size
problem.