To install click the Add extension button. That's it.

The source code for the WIKI 2 extension is being checked by specialists of the Mozilla Foundation, Google, and Apple. You could also do it yourself at any point in time. 4,5
Kelly Slayton
Congratulations on this excellent venture… what a great idea!
Alexander Grigorievskiy
I use WIKI 2 every day and almost forgot how the original Wikipedia looks like.
Live Statistics
English Articles
Improved in 24 Hours
Languages
Recent
Show all languages
What we do. Every page goes through several hundred of perfecting techniques; in live mode. Quite the same Wikipedia. Just better.
.
Leo
Newton
Brights
Milds # Total least squares The bivariate (Deming regression) case of total least squares. The red lines show the error in both x and y. This is different from the traditional least squares method which measures error parallel to the y axis. The case shown, with deviations measured perpendicularly, arises when x and y have equal variances.

In applied statistics, total least squares is a type of errors-in-variables regression, a least squares data modeling technique in which observational errors on both dependent and independent variables are taken into account. It is a generalization of Deming regression and also of orthogonal regression, and can be applied to both linear and non-linear models.

The total least squares approximation of the data is generically equivalent to the best, in the Frobenius norm, low-rank approximation of the data matrix.

• 1/5
Views:
112 942
495
236 616
9 910
19 859
• ✪ Least squares | MIT 18.02SC Multivariable Calculus, Fall 2010
• ✪ Lecture31 (Data2Decision) Total Regression, part 2
• ✪ Least squares approximation | Linear Algebra | Khan Academy
• ✪ Chapter 16 Problems, Singular Value Decomposition, SVD & Least Squares
• ✪ The Least Squares Formula: A Derivation

#### Transcription

CHRISTINE BREINER: Welcome back to recitation. In this video, what I'd like you to do is use least squares to fit a line to the following data, which includes three points: the point 0, 1, the point 2, 1, and the point 3, 4. And I've drawn a rough picture where these points are on a graph, and I'll be talking a little bit about that after you try this problem. So why don't you try to solve the problem based on the technique you learned in class, and then bring the video back up. I'll show you again what you're actually doing, and then I will also solve the problem and you can see, you can compare your answer to mine. OK, welcome back. So the first thing I want to do when we're talking about least squares to fit a line to these data, what I'd like to do is I'm going to draw a line on here, and we're going to talk about what the least squares actually is trying to do. And so I'm going to draw this line, say, something like this. That's my first guess on what might be the actual least squares line for these data. And if you recall, what you're actually trying to do is you're trying to minimize a certain quantity, and the quantity you're trying to minimize is the difference between the actual value you get and the expected value you get, the square of that. So this is one thing we're going to square. So even pictorially, we could say we have something that contributes some area, OK, and then this is another thing that's the difference between the expected value and y and the actual value you get. So let me try and make that squared, because we square that value. And then this last one is pretty small. The difference between the expected value and the actual value is very small over in this case from the picture I drew. And so what you're trying to do is you're trying to find the line that minimizes the area of the sum of these three, in this case, three squares. So if I, you know, if I tip the line down further here, but I kept it fixed there, the area of this square would get bigger, because this distance would get bigger, but, of course, the area of this square would get smaller. And so you're trying to figure out a way to optimize, in this case, minimize, the area of these-- sorry, the sum of the areas of these squares by figuring out where to put the line. Now, I mentioned if I could fix this here and move it up and down, then that would change the slope but fix the intercept. But I could also move the intercept up and down, and I could slide the line up and down, and that would fix the slope but vary the intercept, or I could do both. And ultimately, that's what you're doing. You're trying to find the best y-intercept and the best slope at the same time, and that's why you learned this process in multivariable calculus, because you're trying to optimize something that has two quantities that you can change. You can change slope and you can change the y-intercept, so that's why you learned it now instead of in single-variable calculus. Now, if we come over here, I wrote down ahead of time what the equations are you get when you optimize, when you're trying to optimize the least squares line. So if you write down, you know, the function as a function of a and b, and you take the derivative with respect to a and you set it equal to zero, you get the first equation. When you take the derivative with respect to b and you set it equal to zero, you get the second equation. And so I'm just going to fill in the details and then tell you what the solution is, and then I'm going to mention a little more sophisticated thing we can do, or I guess maybe the next level of least squares we can do, that's not a linear problem. So I'll do that last. So what I want to do is I put all the values I need over here on the right. So all the values we need are on the right. We're going to need the sum of all the xi's and that's 5. The sum of all the yi's is 6. The sum of the xi squareds is 13, and the sum of the xi yi product is 14. Now, where are these coming from? Remember, what is xi and what is yi? If we come back over here, this is x1, y1, x2, y2, x3, y3, OK? So that's sort of how we get-- if we switch the whole pairs around, it doesn't matter. Obviously, if we switched x- and y-values together, it does matter, and that you see actually from the equation does matter. Right here you have a product of them. So let me write this out. And again, we're solving for a and b, so in this case, a is the slope and b is the intercept, right? So let me write everything in. So we actually get that we want to solve the following system: 13a plus 5b is equal to 14. Make sure I put in everything correctly. And then the second one is 5a plus-- n in this case is 3, because there were three points-- plus 3b is equal to 6. OK, this is a system of equations. It has two equations and two unknowns, and if you solve this-- I should look at my notes, because I didn't memorize it and I'm not going to do all the algebra. If you go to solve this, what you actually get is that a is equal to 6/7 and b is equal to 4/7. So you get that the line-- let me write it here. You get a is equal to 6/7 and b is equal to 4/7, OK? And so what that tells you is if you come back over here, the line we actually want-- I'll write the solution right here-- the line we actually want is the line y equals 6/7x plus 4/7. That is our least squares line that is our solution. OK? So that is actually-- solving for a line to fit these data, you get the following solution. But I want to point out, and what we're going to do, after this video there will be a challenge problem to have you actually finish this. I want to point out that you can actually also fit a higher-degree polynomial to these data. For instance, you could try and use the technique of least squares to fit a parabola to these data. And let me point out what the function would look like in this case. So this is for the challenge problem. For the challenge problem, it now will be a function of three variables, so it will look something like this. We'll say a comma b comma c, and the point that we'll be wanting to minimize, or the function we'll be wanting to minimize, we're summing over i, again from 1 to 3-- I didn't write that down above, but it was fairly obvious that that was what we were summing over-- the following thing. We have yi minus this big quantity: axi squared plus bxi plus c and we square the whole thing. And so what this is actually giving you is, just as in the picture before, this is giving you the difference between the expected value and the actual value. So this is your actual y-value, and based on the parabola you're looking for, what's in parentheses is your expected y-value, right? So this would actually be my parabola. When I evaluate it at xi, I get a number. I get a value, and that value will be on the parabola. This will be the y-value associated to the x-value from the data. So this is the actual value. This is the expected value. We take the difference and square it. And because we want to minimize how we're going to solve this problem and this is what you'll have to do is, just like with the line situation, you're going to take the derivative of this function with respect to each of these three variables separately. So you'll take f sub a, and then you'll set that equal to zero. And then you'll take f sub b, you'll set that equal to zero. And then you'll take f sub c, and you'll set that equal to zero. Again, we set them all equal to zero, because this is really an optimization problem. We want to minimize something. We want to minimize this quantity. So you take each of those three derivatives, partial derivatives, set them equal to zero, and you have a system of three equations with three variables. And you know how to solve such a system, by using matrices, for instance. That would be one way to solve such a system. And so then you can actually come up with a formula for the sort of parabola of best fit, you could think of it as. And so I'm going to stop there, because I think from here, you guys will do it. But I just want to point out this is where you're going to be starting this next sort of challenge problem to find the quadratic of best fit for these three data points. So that's where I'll end it.

## Linear model

### Background

In the least squares method of data modeling, the objective function, S,

$S=\mathbf {r^{T}Wr} ,$ is minimized, where r is the vector of residuals and W is a weighting matrix. In linear least squares the model contains equations which are linear in the parameters appearing in the parameter vector ${\boldsymbol {\beta }}$ , so the residuals are given by

$\mathbf {r=y-X{\boldsymbol {\beta }}} .$ There are m observations in y and n parameters in β with m>n. X is a m×n matrix whose elements are either constants or functions of the independent variables, x. The weight matrix W is, ideally, the inverse of the variance-covariance matrix $\mathbf {M} _{y}$ of the observations y. The independent variables are assumed to be error-free. The parameter estimates are found by setting the gradient equations to zero, which results in the normal equations [note 1]

$\mathbf {X^{T}WX{\boldsymbol {\beta }}=X^{T}Wy} .$ ### Allowing observation errors in all variables

Now, suppose that both x and y are observed subject to error, with variance-covariance matrices $\mathbf {M} _{x}$ and $\mathbf {M} _{y}$ respectively. In this case the objective function can be written as

$S=\mathbf {r_{x}^{T}M_{x}^{-1}r_{x}+r_{y}^{T}M_{y}^{-1}r_{y}} ,$ where $\mathbf {r} _{x}$ and $\mathbf {r} _{y}$ are the residuals in x and y respectively. Clearly these residuals cannot be independent of each other, but they must be constrained by some kind of relationship. Writing the model function as $\mathbf {f(r_{x},r_{y},{\boldsymbol {\beta }})}$ , the constraints are expressed by m condition equations.

$\mathbf {F=\Delta y-{\frac {\partial f}{\partial r_{x}}}r_{x}-{\frac {\partial f}{\partial r_{y}}}r_{y}-X\Delta {\boldsymbol {\beta }}=0} .$ Thus, the problem is to minimize the objective function subject to the m constraints. It is solved by the use of Lagrange multipliers. After some algebraic manipulations, the result is obtained.

$\mathbf {X^{T}M^{-1}X\Delta {\boldsymbol {\beta }}=X^{T}M^{-1}\Delta y} ,$ or alternatively $\mathbf {X^{T}M^{-1}X{\boldsymbol {\beta }}=X^{T}M^{-1}y} ,$ where M is the variance-covariance matrix relative to both independent and dependent variables.

$\mathbf {M=K_{x}M_{x}K_{x}^{T}+K_{y}M_{y}K_{y}^{T};\ K_{x}=-{\frac {\partial f}{\partial r_{x}}},\ K_{y}=-{\frac {\partial f}{\partial r_{y}}}} .$ ### Example

When the data errors are uncorrelated, all matrices M and W are diagonal. Then, take the example of straight line fitting.

$f(x_{i},\beta )=\alpha +\beta x_{i}$ in this case

$M_{ii}=\sigma _{y,i}^{2}+\beta ^{2}\sigma _{x,i}^{2}$ showing how the variance at the ith point is determined by the variances of both independent and dependent variables and by the model being used to fit the data. The expression may be generalized by noting that the parameter $\beta$ is the slope of the line.

$M_{ii}=\sigma _{y,i}^{2}+\left({\frac {dy}{dx}}\right)_{i}^{2}\sigma _{x,i}^{2}$ An expression of this type is used in fitting pH titration data where a small error on x translates to a large error on y when the slope is large.

### Algebraic point of view

First of all it is necessary to note that the TLS problem does not have a solution in general, which was already shown in 1980. The following considers the simple case where a unique solution exists without making any particular assumptions.

The computation of the TLS using singular value decomposition is described in standard texts. We can solve the equation

$XB\approx Y$ for B where X is m-by-n and Y is m-by-k. [note 2]

That is, we seek to find B that minimizes error matrices E and F for X and Y respectively. That is,

$\mathrm {argmin} _{E,F}\|[E\;F]\|_{F},\qquad (X+E)B=Y+F$ where $[E\;F]$ is the augmented matrix with E and F side by side and $\|\cdot \|_{F}$ is the Frobenius norm, the square root of the sum of the squares of all entries in a matrix and so equivalently the square root of the sum of squares of the lengths of the rows or columns of the matrix.

This can be rewritten as

$[(X+E)\;(Y+F)]{\begin{bmatrix}B\\-I_{k}\end{bmatrix}}=0.$ where $I_{k}$ is the $k\times k$ identity matrix. The goal is then to find $[E\;F]$ that reduces the rank of $[X\;Y]$ by k. Define $[U][\Sigma ][V]^{*}$ to be the singular value decomposition of the augmented matrix $[X\;Y]$ .

$[X\;Y]=[U_{X}\;U_{Y}]{\begin{bmatrix}\Sigma _{X}&0\\0&\Sigma _{Y}\end{bmatrix}}{\begin{bmatrix}V_{XX}&V_{XY}\\V_{YX}&V_{YY}\end{bmatrix}}^{*}=[U_{X}\;U_{Y}]{\begin{bmatrix}\Sigma _{X}&0\\0&\Sigma _{Y}\end{bmatrix}}{\begin{bmatrix}V_{XX}^{*}&V_{YX}^{*}\\V_{XY}^{*}&V_{YY}^{*}\end{bmatrix}}$ where V is partitioned into blocks corresponding to the shape of X and Y.

Using the Eckart–Young theorem, the approximation minimising the norm of the error is such that matrices $U$ and $V$ are unchanged, while the $k$ -smallest singular values are replaced with zeroes. That is, we want

$[(X+E)\;(Y+F)]=[U_{X}\;U_{Y}]{\begin{bmatrix}\Sigma _{X}&0\\0&0_{k\times k}\end{bmatrix}}{\begin{bmatrix}V_{XX}&V_{XY}\\V_{YX}&V_{YY}\end{bmatrix}}^{*}$ so by linearity,

$[E\;F]=-[U_{X}\;U_{Y}]{\begin{bmatrix}0_{n\times n}&0\\0&\Sigma _{Y}\end{bmatrix}}{\begin{bmatrix}V_{XX}&V_{XY}\\V_{YX}&V_{YY}\end{bmatrix}}^{*}.$ We can then remove blocks from the U and Σ matrices, simplifying to

$[E\;F]=-U_{Y}\Sigma _{Y}{\begin{bmatrix}V_{XY}\\V_{YY}\end{bmatrix}}^{*}=-[X\;Y]{\begin{bmatrix}V_{XY}\\V_{YY}\end{bmatrix}}{\begin{bmatrix}V_{XY}\\V_{YY}\end{bmatrix}}^{*}.$ This provides E and F so that

$[(X+E)\;(Y+F)]{\begin{bmatrix}V_{XY}\\V_{YY}\end{bmatrix}}=0.$ Now if $V_{YY}$ is nonsingular, which is not always the case (note that the behavior of TLS when $V_{YY}$ is singular is not well understood yet), we can then right multiply both sides by $-V_{YY}^{-1}$ to bring the bottom block of the right matrix to the negative identity, giving

$[(X+E)\;(Y+F)]{\begin{bmatrix}-V_{XY}V_{YY}^{-1}\\-V_{YY}V_{YY}^{-1}\end{bmatrix}}=[(X+E)\;(Y+F)]{\begin{bmatrix}B\\-I_{k}\end{bmatrix}}=0,$ and so

$B=-V_{XY}V_{YY}^{-1}.$ A naive GNU Octave implementation of this is:

function B = tls(X,Y)

[m n]   = size(X);            % n is the width of X (X is m by n)
Z       = [X Y];              % Z is X augmented with Y.
[U S V] = svd(Z,0);           % find the SVD of Z.
VXY     = V(1:n,1+n:end);     % Take the block of V consisting of the first n rows and the n+1 to last column
VYY     = V(1+n:end,1+n:end); % Take the bottom-right block of V.
B       = -VXY/VYY;

end


The way described above of solving the problem, which requires that the matrix $V_{YY}$ is nonsingular, can be slightly extended by the so-called classical TLS algorithm.

### Computation

The standard implementation of classical TLS algorithm is available through Netlib, see also. All modern implementations based, for example, on solving a sequence of ordinary least squares problems, approximate the matrix $B$ (denoted $X$ in the literature), as introduced by Van Huffel and Vandewalle. It is worth noting, that this $B$ is, however, not the TLS solution in many cases.

## Non-linear model

For non-linear systems similar reasoning shows that the normal equations for an iteration cycle can be written as

$\mathbf {J^{T}M^{-1}J\Delta {\boldsymbol {\beta }}=J^{T}M^{-1}\Delta y} .$ ## Geometrical interpretation

When the independent variable is error-free a residual represents the "vertical" distance between the observed data point and the fitted curve (or surface). In total least squares a residual represents the distance between a data point and the fitted curve measured along some direction. In fact, if both variables are measured in the same units and the errors on both variables are the same, then the residual represents the shortest distance between the data point and the fitted curve, that is, the residual vector is perpendicular to the tangent of the curve. For this reason, this type of regression is sometimes called two dimensional Euclidean regression (Stein, 1983) or orthogonal regression.

## Scale invariant methods

A serious difficulty arises if the variables are not measured in the same units. First consider measuring distance between a data point and the line, as in the diagram – what are the measurement units for this distance? If we consider measuring distance based on Pythagoras' Theorem then it is clear that we shall be adding quantities measured in different units, which is meaningless. Secondly, if we rescale one of the variables e.g., measure in grams rather than kilograms, then we shall end up with different results (a different line). To avoid these problems it is sometimes suggested that we convert to dimensionless variables—this may be called normalization or standardization. However there are various ways of doing this, and these lead to fitted models which are not equivalent to each other. One approach is to normalize by known (or estimated) measurement precision thereby minimizing the Mahalanobis distance from the points to the line, providing a maximum-likelihood solution;[citation needed] the unknown precisions could be found via analysis of variance.

In short, total least squares does not have the property of units-invariance—i.e. it is not scale invariant. For a meaningful model we require this property to hold. A way forward is to realise that residuals (distances) measured in different units can be combined if multiplication is used instead of addition. Consider fitting a line: for each data point the product of the vertical and horizontal residuals equals twice the area of the triangle formed by the residual lines and the fitted line. We choose the line which minimizes the sum of these areas. Nobel laureate Paul Samuelson proved in 1942 that, in two dimensions, it is the only line expressible solely in terms of the ratios of standard deviations and the correlation coefficient which (1) fits the correct equation when the observations fall on a straight line, (2) exhibits scale invariance, and (3) exhibits invariance under interchange of variables. This solution has been rediscovered in different disciplines and is variously known as standardised major axis (Ricker 1975, Warton et al., 2006), the reduced major axis, the geometric mean functional relationship (Draper and Smith, 1998), least products regression, diagonal regression, line of organic correlation, and the least areas line (Tofallis, 2002). Tofallis (2015) has extended this approach to deal with multiple variables.