Explained variation

In statistics, explained variation measures the proportion to which a mathematical model accounts for the variation (dispersion) of a given data set. Often, variation is quantified as variance; then, the more specific term explained variance can be used.

The complementary part of the total variation is called unexplained or residual variation.

YouTube Encyclopedic

1/3
Views:
673 301
981 556
415 057

Transcription

I want to talk a little bit about direct and inverse variations. So I'll do direct variation on the left over here. And I'll do inverse variation, or two variables that vary inversely, on the right-hand side over here. So a very simple definition for two variables that vary directly would be something like this. y varies directly with x if y is equal to some constant with x. So we could rewrite this in kind of English as y varies directly with x. And if this constant seems strange to you, just remember this could be literally any constant number. So let me give you a bunch of particular examples of y varying directly with x. You could have y is equal to x. Because in this situation, the constant is 1. We didn't even write it. We could write y is equal to 1x, then k is 1. We could write y is equal to 2x. We could write y is equal to 1/2 x. We could write y is equal to negative 2x. We are still varying directly. We could have y is equal to negative 1/2 x. We could have y is equal to pi times x. We could have y is equal to negative pi times x. I don't want to beat a dead horse now. I think you get the point. Any constant times x-- we are varying directly. And to understand this maybe a little bit more tangibly, let's think about what happens. And let's pick one of these scenarios. Well, I'll take a positive version and a negative version, just because it might not be completely intuitive. So let's take the version of y is equal to 2x, and let's explore why we say they vary directly with each other. So let's pick a couple of values for x and see what the resulting y value would have to be. So if x is equal to 1, then y is 2 times 1, or is 2. If x is equal to 2, then y is 2 times 2, which is going to be equal to 4. So when we doubled x, when we went from 1 to 2-- so we doubled x-- the same thing happened to y. We doubled y. So that's what it means when something varies directly. If we scale x up by a certain amount, we're going to scale up y by the same amount. If we scale down x by some amount, we would scale down y by the same amount. And just to show you it works with all of these, let's try the situation with y is equal to negative 2x. I'll do it in magenta. y is equal to negative-- well, let me do a new example that I haven't even written here. Let's try y is equal to negative 3x. So once again, let me do my x and my y. When x is equal to 1, y is equal to negative 3 times 1, which is negative 3. When x is equal to 2, so negative 3 times 2 is negative 6. So notice, we multiplied. So if we scaled-- let me do that in that same green color. If we scale up x by 2-- it's a different green color, but it serves the purpose-- we're also scaling up y by 2. To go from 1 to 2, you multiply it by 2. To go from negative 3 to negative 6, you're also multiplying by 2. So we grew by the same scaling factor. And if you wanted to go the other way-- let's try, I don't know, let's go to x is 1/3. If x is 1/3, then y is going to be-- negative 3 times 1/3 is negative 1. So notice, to go from 1 to 1/3, we divide by 3. To go from negative 3 to negative 1, we also divide by 3. We also scale down by a factor of 3. So whatever direction you scale x in, you're going to have the same scaling direction as y. That's what it means to vary directly. Now, it's not always so clear. Sometimes it will be obfuscated. So let's take this example right over here. y is equal to negative 3x. And I'm saving this real estate for inverse variation in a second. You could write it like this, or you could algebraically manipulate it. You could maybe divide both sides of this equation by x, and then you would get y/x is equal to negative 3. Or maybe you divide both sides by x, and then you divide both sides by y. So from this, so if you divide both sides by y now, you could get 1/x is equal to negative 3 times 1/y. These three statements, these three equations, are all saying the same thing. So sometimes the direct variation isn't quite in your face. But if you do this, what I did right here with any of these, you will get the exact same result. Or you could just try to manipulate it back to this form over here. And there's other ways we could do it. We could divide both sides of this equation by negative 3. And then you would get negative 1/3 y is equal to x. And now, this is kind of an interesting case here because here, this is x varies directly with y. Or we could say x is equal to some k times y. And in general, that's true. If y varies directly with x, then we can also say that x varies directly with y. It's not going to be the same constant. It's going to be essentially the inverse of that constant, but they're still directly varying. Now with that said, so much said, about direct variation, let's explore inverse variation a little bit. Inverse variation-- the general form, if we use the same variables. And it always doesn't have to be y and x. It could be an a and a b. It could be a m and an n. If I said m varies directly with n, we would say m is equal to some constant times n. Now let's do inverse variation. So if I did it with y's and x's, this would be y is equal to some constant times 1/x. So instead of being some constant times x, it's some constant times 1/x. So let me draw you a bunch of examples. It could be y is equal to 1/x. It could be y is equal to 2 times 1/x, which is clearly the same thing as 2/x. It could be y is equal to 1/3 times 1/x, which is the same thing as 1 over 3x. it could be y is equal to negative 2 over x. And let's explore this, the inverse variation, the same way that we explored the direct variation. So let's pick-- I don't know/ let's pick y is equal to 2/x. And let me do that same table over here. So I have my table. I have my x values and my y values. If x is 1, then y is 2. If x is 2, then 2 divided by 2 is 1. So if you multiply x by 2, if you scale it up by a factor of 2, what happens to y? y gets scaled down by a factor of 2. You're dividing by 2 now. Notice the difference. Here, however we scaled x, we scaled up y by the same amount. Now, if we scale up x by a factor, when we have inverse variation, we're scaling down y by that same. So that's where the inverse is coming from. And we could go the other way. If we made x is equal to 1/2. So if we were to scale down x, we're going to see that it's going to scale up y. Because 2 divided by 1/2 is 4. So here we are scaling up y. So they're going to do the opposite things. They vary inversely. And you could try it with the negative version of it, as well. So here we're multiplying by 2. And once again, it's not always neatly written for you like this. It can be rearranged in a bunch of different ways. But it will still be inverse variation as long as they're algebraically equivalent. So you can multiply both sides of this equation right here by x. And you would get xy is equal to 2. This is also inverse variation. You would get this exact same table over here. You could divide both sides of this equation by y. And you could get x is equal to 2/y, which is also the same thing as 2 times 1/y. So notice, y varies inversely with x. And you could just manipulate this algebraically to show that x varies inversely with y. So y varies inversely with x. This is the same thing as saying-- and we just showed it over here with a particular example-- that x varies inversely with y. And there's other things. We could take this and divide both sides by 2. And you would get y/2 is equal to 1/x. There's all sorts of crazy things. And so in general, if you see an expression that relates to variables, and they say, do they vary inversely or directly or maybe neither? You could either try to do a table like this. If you scale up x by a certain amount and y gets scaled up by the same amount, then it's direct variation. If you scale up x by some-- and you might want to try a couple different times-- and you scale down y, you do the opposite with y, then it's probably inverse variation. A surefire way of knowing what you're dealing with is to actually algebraically manipulate the equation so it gets back to either this form, which would tell you that it's inverse variation, or this form, which would tell you that it is direct variation.

Definition in terms of information gain

Information gain by better modelling

Following Kent (1983),^[1] we use the Fraser information (Fraser 1965)^[2]

F(\theta )=\int {\textrm {d}}r\,g(r)\,\ln f(r;\theta )

where $g(r)$ is the probability density of a random variable $R\,$ , and $f(r;\theta )\,$ with $\theta \in \Theta _{i}$ ( $i=0,1\,$ ) are two families of parametric models. Model family 0 is the simpler one, with a restricted parameter space $\Theta _{0}\subset \Theta _{1}$ .

Parameters are determined by maximum likelihood estimation,

\theta _{i}=\operatorname {argmax} _{\theta \in \Theta _{i}}F(\theta ).

The information gain of model 1 over model 0 is written as

\Gamma (\theta _{1}:\theta _{0})=2[F(\theta _{1})-F(\theta _{0})]\,

where a factor of 2 is included for convenience. Γ is always nonnegative; it measures the extent to which the best model of family 1 is better than the best model of family 0 in explaining g(r).

Information gain by a conditional model

Assume a two-dimensional random variable $R=(X,Y)$ where X shall be considered as an explanatory variable, and Y as a dependent variable. Models of family 1 "explain" Y in terms of X,

f(y\mid x;\theta )

whereas in family 0, X and Y are assumed to be independent. We define the randomness of Y by $D(Y)=\exp[-2F(\theta _{0})]$ , and the randomness of Y, given X, by $D(Y\mid X)=\exp[-2F(\theta _{1})]$ . Then,

\rho _{C}^{2}=1-D(Y\mid X)/D(Y)

can be interpreted as proportion of the data dispersion which is "explained" by X.

Special cases and generalized usage

Linear regression

The fraction of variance unexplained is an established concept in the context of linear regression. The usual definition of the coefficient of determination is based on the fundamental concept of explained variance.

Correlation coefficient as measure of explained variance

Let X be a random vector, and Y a random variable that is modeled by a normal distribution with centre $\mu =\Psi ^{\textrm {T}}X$ . In this case, the above-derived proportion of explained variation $\rho _{C}^{2}$ equals the squared correlation coefficient $R^{2}$ .

Note the strong model assumptions: the centre of the Y distribution must be a linear function of X, and for any given x, the Y distribution must be normal. In other situations, it is generally not justified to interpret $R^{2}$ as proportion of explained variance.

In principal component analysis

Explained variance is routinely used in principal component analysis. The relation to the Fraser–Kent information gain remains to be clarified.

Criticism

As the fraction of "explained variance" equals the squared correlation coefficient $R^{2}$ , it shares all the disadvantages of the latter: it reflects not only the quality of the regression, but also the distribution of the independent (conditioning) variables.

In the words of one critic: "Thus $R^{2}$ gives the 'percentage of variance explained' by the regression, an expression that, for most social scientists, is of doubtful meaning but great rhetorical value. If this number is large, the regression gives a good fit, and there is little point in searching for additional variables. Other regression equations on different data sets are said to be less satisfactory or less powerful if their $R^{2}$ is lower. Nothing about $R^{2}$ supports these claims".^[3]^: 58 And, after constructing an example where $R^{2}$ is enhanced just by jointly considering data from two different populations: "'Explained variance' explains nothing."^[3]^{[page needed]}^[4]^: 183

References

^ Kent, J. T. (1983). "Information gain and a general measure of correlation". Biometrika. 70 (1): 163–173. doi:10.1093/biomet/70.1.163. JSTOR 2335954.
^ Fraser, D. A. S. (1965). "On Information in Statistics". Ann. Math. Statist. 36 (3): 890–896. doi:10.1214/aoms/1177700061.
^ ^a ^b Achen, C. H. (1982). Interpreting and Using Regression. Beverly Hills: Sage. pp. 58–59. ISBN 0-8039-1915-8.
^ Achen, C. H. (1990). "'What Does "Explained Variance" Explain?: Reply". Political Analysis. 2 (1): 173–184. doi:10.1093/pan/2.1.173.

External links

Explained and Unexplained Variance on a graph

This page was last edited on 28 February 2021, at 05:13

From Wikipedia, the free encyclopedia