Why we need the intercept ...

This is the first article in the series on non-hierarchical regression models. In this article we discuss the pitfalls when performing a regression without using the intercept term.

A common question is, if the intercept-term may be removed from a regression analysis in case it is not significant. Most of the time the answer to this question should be "No!" and I totally agree with that answer. But of course we would like to understand why!

We will use the following terminology:

1. A full model is a model containing the intercept and main effects: $y = b_0 + b_1\cdot x + \epsilon$
2. A no-intercept-model reduces the full model by its intercept: $y = b_1 \cdot x + \epsilon$
3. The intercept-only-model (often: baseline model) uses only the intercept to explain the data: $y = b_1  + \epsilon$

Problems of no-intercept-models

1. $R^2$ is not useful any more.
2. The slope estimators might be biased.

Explanation

The first commonly mentioned problem (see [1.]) is that the $R^2$-statistic is not useful anymore if the intercept is not included in a linear regression model. Usually $R^2$ is interpreted as the amount of variation that is explained by the model.

$$R^2 = \frac{Model SS}{Total SS} = 1 - \frac{Residual SS}{Total SS}$$

More precise: The full model is compared to a reduced model, which in this case is the intercept-only-model. So what do we do for an no-intercept-model? Well we will not compare it to the intercept-only-model. Both models are completely independent from each other so there is no sense in comparing. Instead most software packages (R, JMP, SAS, DX, SPSS, …) compare the model without intercept to a reference model that has lower order. That is a model with no intercept and no other effects.

$$\text{Noise-Model}: y = \epsilon$$

One might call it a noise-model. Of course this is no real model (it does not explain anything) and any comparison with it is not very useful. Actually there is no real reference model we could use to compare our no-intercept-model with. So there is no interpretable $R^2$ for models w/o intercept.

None the less statistics software will present an $R^2$ for no-intercept models. But as the interpretation of the $R^2$ is lost we cannot use it to evaluate the model quality. One can even show that the $R^2$ of a no-intercept model will usually be higher compared to the $R^2$ of a full model (see the mathematical details for that).

For the example presented in the graph below the $R^2s$ (calculated with R) are:

	$R^2$
Full Model	0.7846
No-Intercept-Model	0.9114

It is obvious that this is not a reasonable result. The red line is clearly not the better model!

The second problem that arises is that the least squares estimator for the slopes in a no-intercept model are biased (systematically shifted towards larger or smaller values).

Usual (teal) and regression without intercept (red)

With removing the intercept from the model we impose a restriction so that the regression line goes through the origin (x=0;y=0). The graph shows what happens to the regression line. The blue line is the common regression line the red line is the no-intercept-regression-line. It is heavily pulled down because it has to go through $(0;0)$.

Mathematical details

R^2 does not work for no-intercept-models

So what exactly happens to $R^2$? As noted above the typical $R^2$ is defined as:

$$ R^2 = \frac{\sum_i (\hat y_i - \bar y)^2 }{ \sum_i (y_i - \bar y)^2} = 1 - \frac{\sum_i (y_i - \hat y_i)^2}{\sum_i (y_i - \bar y)^2}$$

Formulate the $R^2_0$ for a no-intercept-model $y = b\cdot x$:

$$ R^2_0 = \frac{\sum_i \hat y_i^2}{\sum_i y_i^2} = 1 - \frac{\sum_i (y_i - \hat y_i)^2}{\sum_i y_i^2}$$

At the end of the last equation you see that we compare the Residual-Sum-of-Squares with the sum-of-squares of the actual observations. So this is a ratio of the variation of the data around the model (Residual sum of squares) and the magnitude of the response.

Apart from not really being interpretable $R^2_0$ is often larger than $R^2$. This often leads to the wrong (!) assumption that the model w/o intercept is better in explaining the data than the full model. So how can $R^2_0$ be greater thant $R^2$? Usually we expect $R^2$ to become greater whenever we add more parameters to the model. Now we reduce the model and $R^2$ raises?

Use $\tilde{y_i}$ for the fitted values of the no-intercept model and $\hat y_i$ for the fitted values of the full model. Then $R_0$ is greater than $R^2$ whenever:

$$ R^2_0 = 1 - \frac{\sum_i (y_i - \tilde{y}_i)^2}{\sum_i y_i^2} > R^2 = 1 - \frac{\sum_i (y_i - \hat y_i)^2}{\sum_i (y_i - \bar y)^2}$$
$$\Rightarrow  \frac{\sum_i (y_i - \tilde{y}_i)^2}{\sum_i y_i^2} < \frac{\sum_i (y_i - \hat y_i)^2}{\sum_i (y_i - \bar y)^2}$$
$$ \Rightarrow \frac{||y-\tilde y||_2^2}{||y-\hat y||_2^2} < \frac{||y||_2^2}{||y-\bar y||_2^2}$$

Now use  $||y||_2^2= ||y - \bar y + \bar y||_2^2 = ||y - \bar y||_2^2 +n\bar y^2$. Then:

$$ \frac{||y-\tilde y||_2^2}{||y-\hat y||_2^2} < \frac{||y-\bar y||_2^2 + n\bar y^2}{||y-\bar y||_2^2} $$

$$ \frac{||y-\tilde y||_2^2}{||y-\hat y||_2^2} < 1 + \frac{\bar y^2}{\frac{1}{n}||y-\bar y||_2^2}$$

The left hand side is allways greater than 1, as the fitted values of the no-intercept model $\tilde{y}$ will always be worse than the fitted values of the full model $\hat y$. The last term of the right hand side is large if the squared mean response is greater than the variance of the response. So $R_0^2$ will be larger than $R^2$ whenever the mean of the response $\bar y$ is much larger than the standard deviation of the response $\sqrt{\frac{1}{n} \sum_i (y_i - y)^2 }$ (forget about the $\frac{1}{n-1}$ for being unbiased :-)).

Proof that LS-estimator is biased in no-intercept models

Say the true data generating process is: $y_i = b_0 + b \cdot x_i + \epsilon_i$. Our estimated model is $y_i=b\cdot x_i + \epsilon_i$. Then we get the expectation of $b$:

$$ E[b] = E[(X^TX)^{-1} X^T y]$$
$$ = E[(X^TX)^{-1} X^T (b_0 + Xb + \epsilon)]$$
$$ = E[(X^TX)^{-1} X^Tb_0 + \underbrace{(X^TX)^{-1} X^TX}_{=\mathbf{1}}b + (X^TX)^{-1} X^T \epsilon]$$
If $E[\epsilon] = 0$ as assumed in linear regression:
$$ = E[(X^TX)^{-1} X^Tb_0] + b + 0$$

Obviously $b$ is biased if neither $(X^TX)^{-1} X^T$ nor $b_0$ are equal to 0. See that this applies even when $b_0$ is not significant different from 0!

When to use a no-intercept regression

Basically there is only one reason to perform a regression without using the intercept: Whenever your model is used to describe a process which is known to have a zero-intercept. Examples will be presented in the last article of this series.

So stay tuned!

Literature

[1.] $R^2$-problem on CrossValidated: Link.

[2.] William Greene: Econometrics (Link; Biasedness of the OLS-estimator for omitted variables in Sections 4.3.2)