A Primer on Regression Analysis & Evaluation

R²

What is it used for?

R² is referred to as the coefficient of determination and is a measure of how well the overall estimating equation fits the data. More specifically, this measure captures the proportion of the variation in the dependent variable that is explained by the variations in the independent variables.

How do we apply the R² to our analysis?

In the context of our example, this would tell us how much of the variation in the quantity of chuck roast is explained by the variation in the price of chuck roast. Again, we will leave the actual calculation of R² for your analytics course.

For our model, \(Q_t^{CR} = β_0 + β_1 P_t^{CR} + e_t\), the output shows that the R² is .247. This result suggests that 24.7% of the variation of the quantity demanded of chuck roast is explained by the model, which in this case is the one independent variable, the price of chuck roast.

Adjusted R²

What is the Adjusted R² used for?

Unfortunately, there is an issue with R². As we add independent variables to a given regression model, the R² cannot decrease, because the existing independent variables already explain a portion of the variation in the dependent variable. In theory, we could just simply add independent variables to the right-hand side until the entire variation is explained (i.e. as we add more variables to the right-hand side, the R² will continue to rise until the entire variation is explained as R² approaches 1). The problem is that we can add any variable, regardless of whether or not it makes sense in the context of the model.

How do we apply the adjusted R² to our analysis?

For example, suppose we expand our chuck roast model by adding a variable to the model, the average monthly precipitation in Luxembourg. Now, given the size of the chuck roast market in the U.S., it is difficult to make a case that the amount of rain and snowfall in Luxembourg would have any reasonable impact on the U.S. chuck roast market. It does not make logical sense to include this variable in our model. However, if we estimate the following model: \(Q_t^{CR} = β_0 + β_1 P_t^{CR} + β_2 R_t^{Lux} + e_t\), where \(R_t^{Lux}\), represents our new variable, what impact would this have on our R²?

The output shows the R² results from the estimated model containing the new variable for precipitation in Luxembourg. In this model the R² increased to .2593, which is more than a 1% increase in the R² from the previous model without the new variable where R² was .247. Obviously, the addition of the new variable does not make intuitive sense, as there is no reason this should better explain the variation the quantity of chuck roast in the U.S.

This is where the adjusted R² comes into play. This measure modifies the R² calculation to account for the number of coefficients/parameters in the model. The goal is to prevent you, as a researcher, from just throwing independent variables into a model to increase your R². You should only add variables if they are important to the dependent variable of interest and backed by strong intuition or economic theory. The adjusted R² helps us better explore this notion.

By adding an independent variable to a model it must have a meaningful impact on the R², enough so that it can offset the loss of degrees of freedom by adding the variable. Degrees of freedom is the number of observations minus the number of parameters or coefficients in the model. An important restriction of regression analysis as you cannot have more coefficients than you have observations. If we look at the two tables, the adjusted R² numbers are reported below the R² values. In our example with the Luxembourg precipitation, we can see that the addition of this variable caused the R² to increase (from .247 to .259), but it actually caused the adjusted R² to decrease (from .2328 to .2308). This result allows us to conclude that the addition of this variable, when controlling for its impact on the degrees of freedom, is not helpful in explaining the variation in the dependent variable.

What do you need to know?

It is better to look at the adjusted R² than the regular R², as this controls for the addition of new variables. Also, the numerical interpretation of the adjusted R² follows that of the regular R².

Now that we have a basic understanding of how to evaluate a model as a whole, let’s switch focus to understanding how we can evaluate the individual parameters of a model.

2.2.1. T-test

What it the t-test used for?

The ultimate goal of a t-test is to explore whether or not a parameter estimate is statistically significant, more specifically whether or not it is statistically different from zero. If an estimate is not statistically different zero, then we must assume that it has no actual impact on the dependent variable. If a parameter estimate is statistically significant than it is statistically different from zero and we can than interpret the coefficient accordingly.

What exactly is the t-test testing?

Specifically the t-test, explores the following hypotheses:

Null (Default) Hypothesis: \(β_1 = 0\) (the independent variable does not impact the dependent variable)

Alternative Hypothesis: \(β_1 ≠ 0\) (the independent variable does have an impact on the dependent variable)

If we fail to reject the null hypothesis, then this suggests that the independent variable of interest does not impact the dependent variable because the coefficient is 0. If we reject the null hypothesis, then we can say that the coefficient value is statistically different from 0 and we can proceed accordingly with the interpretation.

How do we determine if we reject the null hypothesis?

Similar to the F-test, we will save the calculation of the t-statistic for the analytics course, but what you should know is that the larger the t-statistic (in absolute value), the more likely you are to find statistical significance or the more likely you are to reject the null hypothesis. For us, we can just look at the p-values to ascertain the statistical significance. Typically, a p-value <.05, suggests that the coefficient is statistically significant at the 95% significance level, which implies that we can reject the null hypothesis (the β of interest is indeed statistically different from 0). In layman’s terms, a p-value = .05 suggests that we are 95% certain that this result did not happen by chance, or there is no more than a 5% probability of observing this result due to chance.

How do we apply the t-test to our analysis?

In the context of our ground chuck roast example, the following is the output for the estimated regression model, \(Q_t^{CR} = β_0 + β_1 P_t^{CR} + e_t\), with the section containing the parameter estimates, the t-statistics, and the subsequent p-values.

This is a very important component of your regression analysis, as this table presents the estimated model specification, as well as the statistical significance of the coefficients. The parameter estimates display the values for the βs in the model. Specifically, the estimated equation from this regression analysis is:

\[ Q^{CR} = 247.79 -56.74P^{CR} \]

Recall, our initial goal is to explore the relationship between the price of chuck roast and its impact on the quantity of chuck roast that is consumed. The coefficient for the price of chuck roast is \(β_1\)=-56.74, which implies that, on average, a $1 increase in price causes the quantity of chuck roast demanded to fall by 56.74 index points. This is the relationship we are looking for in regression analysis.

So what does this have to do with the t-test? The t-test ultimately tells us if we can interpret that β estimate. If we look at the p-value for the \(β_1\), the ChuckP coefficient, we see that the p-value=.0001, which suggests that this estimate (\(β_1\)=-56.74), is indeed statistically significant (since .0001<.05) using a 95% significance level threshold. This implies that this number is statistically different from 0, and we can interpret this value as we did above. However, if the p-value was greater than .05, then this coefficient would not be statistically significant (i.e. β is not statistically different from 0) and we would not be able to reject the null hypothesis.

What you need to know for this primer, is that we want to see p-values on coefficient/parameter estimates that are less than .05, which suggests statistical significance. If this is the case, then we can interpret the parameter estimate accordingly and can conclude that the independent variable has an impact on the dependent variable. If that is not the case, then we conclude that this parameter is not different from 0 and this independent variable does not impact the dependent variable.

What is the difference between statistical significance and economic significance?

One of the most common mistakes that is made when working with regression analysis is the misinterpretation of the statistical significance of a coefficient to mean economic significance. If you look back at the t-test, if we find that the p-value is indeed statistically significant, all that tells us is that we can reject the null hypothesis, which simply states that this coefficient is statistically different from 0. It says nothing about whether or not the size of the coefficient has any meaningful impact on the dependent variable when the independent variable increases.

Suppose in our previous example that the coefficient on the price of chuck roast was actually -.05 rather than -56.74, but it still had a p-value of .0001. In terms of the t-test, this coefficient is statistically significant and is statistically different from 0. However, the interpretation of this coefficient would be that a 1 dollar increase in the price of chuck roast (per pound), decreases the quantity of chuck roast consumed by .05 index points, or a 20 dollar increase in the price of chuck roast (per pound), would cause the quantity of chuck roast consumed to fall by 1 index point. A quick look at the graph from earlier and we can see that the price of chuck roast per pound ranges from about 2.00 to 3.00 dollars, and the quantity index ranges from about 75 to 200. Thus, in order to move the index by a small 1 index point, we would need the price of chuck roast to increase to $20/lb, which seems extremely unrealistic given the data. Thus, this result is indeed statistically significant, but it has little economic significance in terms of its impact. On the other hand, the actual result where this coefficient value was -56.74 has much more economic significance given its ability to impact the market.