Least squares

Least squares, also known as ordinary least squares analysis, is a method for linear regression that determines the values of unknown quantities in a statistical model by minimizing the sum of the squared residuals (the difference between the predicted and observed values). This method was first described by Carl Friedrich Gauss. It can be shown that the least-squares approach to regression analysis is optimal in the sense that it satisfies the Gauss-Markov theorem.

A related method is the least mean squares (LMS) method. It occurs when the number of measured data is 1 and the gradient descent method is used to minimize the squared residual.

Many other types of optimization problems can be expressed in a least squares form, by either minimizing energy or maximizing entropy. The least squares method is particularly important in estimation of model parameters from measured data.

Problem statement

Consider the problem of adjusting a model function to best fit a data set. The chosen model function has adjustable parameters. The data set consist of n points

$(y_{i},{\mathbf {x}}_{i}),i=1,2,\dots ,n.$

The model function has the form

$y=f({\mathbf {x}};{\mathbf {a}}),$

where y is the dependent variable, x is the vector of independent variables, and a are the adjustable parameters of the model. We wish to find the values of these parameters such that the model best fits the data according to a defined error criterion. The least squares method minimizes the sum of squares of errors,

$S(a)=\sum _{i=1}^{n}(y_{i}-f({\mathbf {x}}_{i};{\mathbf {a}}))^{2},$

with respect to the adjustable parameters of the model a.

See also

• Weighted least squares
• Moving least squares
• Regression analysis
• Function approximation