Chapter3.MultipleRegressionAnalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Chapter 3. Multiple Regression Analysis: Estimation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In chapter 2 we learned how to use simple regression analysis to explain a dependent variable y as a function of a single independent variable x. The primary drawback in using simple regression analysis for empirical work is that it is very difficult to draw ceteris paribus conclussions about how x affects y: the key assumption, SLR.4, that all other factors affecting y are uncorrelated with x is often unrealistic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Multiple regression analysis is more amenable to ceteris paribus analysis because it allows us to explicitly control for many other factors that simultaneously affect the dependent variable. In addition, if we add more factors to our model that are useful for explaining y, then more of the variation in y can be explained. Thus multiple regression analysis can be used to build better models for predicting the dependent variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An additional advantage of multiple regression analysis is that it can incorporate fairly general functional forms of relationships"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3-1 The model with k Independent Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The general multiple linear regresion (MLR) model can be written in the population as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "y=\\beta_0+\\beta_0*x_1+\\beta_0*x_2+\\beta_0*x_2+\\dots+\\beta_k*x_k+u\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where:\n",
    "&beta;0 is the intercept. &beta;1 is the incercept associated with x1 and so on"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "u is the error term or disturbance. It contains factors other than x1,x2,..,xn that affect y. No matter how many explanatory variables we include in our model, there will always be factors we cannot include, and these are collectively contained in u."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3-2 Mechanics and Interpretation of Ordinary Least Squares"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-2a Obtaining the OLS Estimates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the general case with k independent variables, we seek estimates, $\\hat{\\beta_0},\\hat{\\beta_1},\\dots,\\hat{\\beta_k}$, in the equation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "\\tag{3.11}\n",
    "y=\\hat{\\beta_0}+\\hat{\\beta_1}*x_1+\\hat{\\beta_2}*x_2+\\dots+\\hat{\\beta_k}*x_k\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The OLS estimates, k+1 of them, are chosen to minimize the sum of squared residuals:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "\\tag{3.12}\n",
    "\\sum_{i=1}^n (y_i-\\hat{\\beta_1}x_{i1}-\\hat{\\beta_2}x_{i2}-\\dots-\\hat{\\beta_k}x_{ik})^2\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-2b Interpreting the OLS Regression Equation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More important than the details underlying the computation of the $\\hat{\\beta_j}$ is the interpretation of the estimated equation. In the case of two independent variables:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "\\tag{3.14}\n",
    "y=\\hat{\\beta_0}+\\hat{\\beta_1}*x_1+\\hat{\\beta_2}*x_2\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The intercept $\\hat{\\beta_0}$ in equation (3.14) is the predicted value of y when x1=0 and x2=0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The estimates $\\hat{\\beta_1}$ and $\\hat{\\beta_2}$ have partial effect or ceteris paribus, interpretations. From equation (3.14) we have"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "\\Delta\\hat{y}=\\hat{\\beta_1}\\Delta\\hat{x_1}+\\hat{\\beta_2}\\Delta\\hat{x_2}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In particular, when x2 is held fixed, so that $\\Delta x_2=0$ then"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "\\Delta\\hat{y}=\\hat{\\beta_1}\\Delta\\hat{x_1}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "holding x2 fixed. The key point is that by including x2 in our model, we obtain a coefficient on x1 with a ceteris paribus interpretation. This is why multiple regression analysis is so useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wooldridge. Example 3.1 Determinants of College GPA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variables in GPA1 include the college grade point average (colGPA), high school GPA (hsGPA), and achievent test score (ACT) for a sample of 141 students from a large university. Both college and high school GPAs are on a four-point scale. The following code computes OLS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "Call:\n",
       "lm(formula = colGPA ~ hsGPA + ACT, data = gpa1)\n",
       "\n",
       "Coefficients:\n",
       "(Intercept)        hsGPA          ACT  \n",
       "   1.286328     0.453456     0.009426  \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\n",
       "Call:\n",
       "lm(formula = colGPA ~ hsGPA + ACT, data = gpa1)\n",
       "\n",
       "Residuals:\n",
       "     Min       1Q   Median       3Q      Max \n",
       "-0.85442 -0.24666 -0.02614  0.28127  0.85357 \n",
       "\n",
       "Coefficients:\n",
       "            Estimate Std. Error t value Pr(>|t|)    \n",
       "(Intercept) 1.286328   0.340822   3.774 0.000238 ***\n",
       "hsGPA       0.453456   0.095813   4.733 5.42e-06 ***\n",
       "ACT         0.009426   0.010777   0.875 0.383297    \n",
       "---\n",
       "Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\n",
       "\n",
       "Residual standard error: 0.3403 on 138 degrees of freedom\n",
       "Multiple R-squared:  0.1764,\tAdjusted R-squared:  0.1645 \n",
       "F-statistic: 14.78 on 2 and 138 DF,  p-value: 1.526e-06\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "library(foreign)\n",
    "gpa1 <- read.dta(\"https://github.com/thousandoaks/Wooldridge/blob/master/gpa1.dta?raw=true\")\n",
    "\n",
    "# Just obtain parameter estimates:\n",
    "lm(colGPA ~ hsGPA+ACT, data=gpa1)\n",
    "\n",
    "# Store results under \"GPAres\" and display full table:\n",
    "GPAres <- lm(colGPA ~ hsGPA+ACT, data=gpa1)\n",
    "summary(GPAres)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do we interpret these results ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The intercept 1.29 is the predicted college GPA if hsGPA and ACT are both set as zero. Since no one who attends college has either a zero high school GPA or a zero on the achievement test, the intercept in this equation is not, by itself, meaningful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More interesting estimates are the slope of coefficients on hsGPA nad ACT. AS expected theres is a positive partial relationship between colGPA and hsGPA. Holding ACT fixed, another point on hsGPA is associated with .453 of a point on the college GPA, or almost half a point. In other words if we choose two students, A and B, and these students have the same ACT score, but the high school GPA of student A is one point higher than the high school GPA of student B, then we predict Student A to have a college GPA .453 higher than that of Student B."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sign of ACT immplies that, while holding hsGPA fixed, a change in the ACT score of 10 points (a very large change given that the maximum ACT score is 36 and its average 24) affects colGPA by less than one-tenth of a point. This is a small effect and it suggests that, once high school GPA is accounted for, the ACT score is not a strong predictor of college GPA "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Later we will show that the coefficient on ACT is not only small but also statistically insignificant."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wooldridge. Example 3.2 Hourly Wage Equation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the 526 observations on workers in WAGE1, we include educ (years of education), exper (years of labor market experience) and tenure (years with current employer) in an equation explaining log(wage). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "Call:\n",
       "lm(formula = log(wage) ~ educ + exper + tenure, data = wage1)\n",
       "\n",
       "Coefficients:\n",
       "(Intercept)         educ        exper       tenure  \n",
       "   0.284360     0.092029     0.004121     0.022067  \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\n",
       "Call:\n",
       "lm(formula = log(wage) ~ educ + exper + tenure, data = wage1)\n",
       "\n",
       "Residuals:\n",
       "     Min       1Q   Median       3Q      Max \n",
       "-2.05802 -0.29645 -0.03265  0.28788  1.42809 \n",
       "\n",
       "Coefficients:\n",
       "            Estimate Std. Error t value Pr(>|t|)    \n",
       "(Intercept) 0.284360   0.104190   2.729  0.00656 ** \n",
       "educ        0.092029   0.007330  12.555  < 2e-16 ***\n",
       "exper       0.004121   0.001723   2.391  0.01714 *  \n",
       "tenure      0.022067   0.003094   7.133 3.29e-12 ***\n",
       "---\n",
       "Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\n",
       "\n",
       "Residual standard error: 0.4409 on 522 degrees of freedom\n",
       "Multiple R-squared:  0.316,\tAdjusted R-squared:  0.3121 \n",
       "F-statistic: 80.39 on 3 and 522 DF,  p-value: < 2.2e-16\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "library(foreign)\n",
    "wage1 <- read.dta(\"https://github.com/thousandoaks/Wooldridge/blob/master/wage1.dta?raw=true\")\n",
    "\n",
    "# Just obtain parameter estimates:\n",
    "lm(log(wage) ~ educ+exper+tenure, data=wage1)\n",
    "\n",
    "# Store results under \"GPAres\" and display full table:\n",
    "WAGEres <- lm(log(wage) ~ educ+exper+tenure, data=wage1)\n",
    "summary(WAGEres)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The coefficient .092 means that, holding exper and tenure fixed, another year of education is predicted to increase log(wage) by .092 which translates into an approximate 9.2% increase in wage. Alternatively, if we take two people with the same levels of experience and job tenure, the coefficient on educ is the proportionate difference in predicted wage when their education levels differ by one year. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whether it is a good estimate of the ceteris paribus return to another year of education requires us to study the statistical properties of OLS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-2c On the Meaning of \"Holding Other Factors Fixed\" in Multiple Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In example 3.1 we observed that the coefficient on ACT measures the predicted difference in colGPA, holding hsGPA fixed. The power of multiple regression analysis is that it provides this ceteris paribus interpretation even though the data have not been collected in a ceteris paribus fashion. In giving the coefficient on ACT a partial effect interpretation, it may seem that we actually went out and sampled people with the same high school GPA but possibly with different ACT scores. This is not the case. The data are a random sample from a large university, there were no restrictions placed on the sample values og hsGPA or ACT in obtaining our sample. Rarely do we have the luxury of holding certain variables fixed in obtaining our sample. Multiple regression effectively allows us to mimic this situation without restricting the values of any independent variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The power of multiple regression analysis is that it allows us to do in nonexperimental environments what natural scientists are able to do in a controlled laboratory setting: keep other factors fixed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-2h Goodness-of-Fit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with simple regression, we can define the total sum of squares (SST), the explained sum of squares (SSE) and the residual sum of squares or sum of squared residuals (SSR) as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "\\tag{3.24}\n",
    "SST=\\sum_{i=1}^n (y_i-\\vec{y})^2\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "\\tag{3.25}\n",
    "SSE=\\sum_{i=1}^n (\\hat{y_i}-\\vec{y_i})^2\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "\\tag{3.26}\n",
    "SSR=\\sum_{i=1}^n \\hat{u_i}^2\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the same argument as in the simple regression case, it can be shown that: $SST=SSE+SSR$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just as in the simple regression case, the R-squared is defined to be"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "\\tag{3.28}\n",
    "R^2:= SSE/SST = 1-SSR/SST \n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An important fact about $R^2$ is that it never decreases, and it usually increases when another independent variable is added to a regression and the same set of observations is used for both regressions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If two regressions use different set of observation, then in general, we cannot tell how the R-squareds will compare, even if one regression uses a subset of regressors.Missing data can be an important practical issue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fact that $R^2$ never decreases when any variable is added to a regression makes it a poor tool for deciding whether one variable of several variables should be added to the model. The factor that should determine whether an explanatory variable belongs in a model is whether the explanatory variable has a nonzero partial effect on y in the population."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wooldridge. Example 3.5 Explaining Arrest Records"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CRIME1 contains data on arrests during the year 1986 and other information on 2725 men born in either 1960 or 191 in California. Each man in the sample was arrested at least once prior to 1986. the variable narr86 is the number of times the man was arrested during 1986. The variable pcnv is the proportion of arrests prior to 1986 that led to conviction, avgsen is average sentence length served for prior convictions, ptime86 is months spent in prison in 1986 and qemp86 s the number of quarters during which the man was employed in 1986."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A lineal model explaining arrests is"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "narr86=\\beta_0 + \\beta_1 * pcnv +\\beta_2 * avgsen + \\beta_3 * ptime86 +\\beta_4 * qemp86 +u\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where pcnv is a proxy for the likelihood for being convicted of a crime and avgsen is a measure of expected severity of punishment, if convicted. The variable ptime86 captures the incarcerative effects of crime. Labor market opportunities are crudely captured by qemp86"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we estimate the model without the variable avgsen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>narr86</th><th scope=col>nfarr86</th><th scope=col>nparr86</th><th scope=col>pcnv</th><th scope=col>avgsen</th><th scope=col>tottime</th><th scope=col>ptime86</th><th scope=col>qemp86</th><th scope=col>inc86</th><th scope=col>durat</th><th scope=col>black</th><th scope=col>hispan</th><th scope=col>born60</th><th scope=col>pcnvsq</th><th scope=col>pt86sq</th><th scope=col>inc86sq</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0         </td><td>0         </td><td>0         </td><td>0.38      </td><td>17.6      </td><td>35.2      </td><td>12        </td><td>0         </td><td> 0.0      </td><td> 0        </td><td>0         </td><td>0         </td><td>1         </td><td>0.1444    </td><td>144       </td><td>   0.00000</td></tr>\n",
       "\t<tr><td>2         </td><td>2         </td><td>0         </td><td>0.44      </td><td> 0.0      </td><td> 0.0      </td><td> 0        </td><td>1         </td><td> 0.8      </td><td> 0        </td><td>0         </td><td>1         </td><td>0         </td><td>0.1936    </td><td>  0       </td><td>   0.64000</td></tr>\n",
       "\t<tr><td>1         </td><td>1         </td><td>0         </td><td>0.33      </td><td>22.8      </td><td>22.8      </td><td> 0        </td><td>0         </td><td> 0.0      </td><td>11        </td><td>1         </td><td>0         </td><td>1         </td><td>0.1089    </td><td>  0       </td><td>   0.00000</td></tr>\n",
       "\t<tr><td>2         </td><td>2         </td><td>1         </td><td>0.25      </td><td> 0.0      </td><td> 0.0      </td><td> 5        </td><td>2         </td><td> 8.8      </td><td> 0        </td><td>0         </td><td>1         </td><td>1         </td><td>0.0625    </td><td> 25       </td><td>  77.44000</td></tr>\n",
       "\t<tr><td>1         </td><td>1         </td><td>0         </td><td>0.00      </td><td> 0.0      </td><td> 0.0      </td><td> 0        </td><td>2         </td><td> 8.1      </td><td> 1        </td><td>0         </td><td>0         </td><td>0         </td><td>0.0000    </td><td>  0       </td><td>  65.61001</td></tr>\n",
       "\t<tr><td>0         </td><td>0         </td><td>0         </td><td>1.00      </td><td> 0.0      </td><td> 0.0      </td><td> 0        </td><td>4         </td><td>97.6      </td><td> 0        </td><td>0         </td><td>0         </td><td>1         </td><td>1.0000    </td><td>  0       </td><td>9525.75977</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllllllllllllll}\n",
       " narr86 & nfarr86 & nparr86 & pcnv & avgsen & tottime & ptime86 & qemp86 & inc86 & durat & black & hispan & born60 & pcnvsq & pt86sq & inc86sq\\\\\n",
       "\\hline\n",
       "\t 0          & 0          & 0          & 0.38       & 17.6       & 35.2       & 12         & 0          &  0.0       &  0         & 0          & 0          & 1          & 0.1444     & 144        &    0.00000\\\\\n",
       "\t 2          & 2          & 0          & 0.44       &  0.0       &  0.0       &  0         & 1          &  0.8       &  0         & 0          & 1          & 0          & 0.1936     &   0        &    0.64000\\\\\n",
       "\t 1          & 1          & 0          & 0.33       & 22.8       & 22.8       &  0         & 0          &  0.0       & 11         & 1          & 0          & 1          & 0.1089     &   0        &    0.00000\\\\\n",
       "\t 2          & 2          & 1          & 0.25       &  0.0       &  0.0       &  5         & 2          &  8.8       &  0         & 0          & 1          & 1          & 0.0625     &  25        &   77.44000\\\\\n",
       "\t 1          & 1          & 0          & 0.00       &  0.0       &  0.0       &  0         & 2          &  8.1       &  1         & 0          & 0          & 0          & 0.0000     &   0        &   65.61001\\\\\n",
       "\t 0          & 0          & 0          & 1.00       &  0.0       &  0.0       &  0         & 4          & 97.6       &  0         & 0          & 0          & 1          & 1.0000     &   0        & 9525.75977\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "narr86 | nfarr86 | nparr86 | pcnv | avgsen | tottime | ptime86 | qemp86 | inc86 | durat | black | hispan | born60 | pcnvsq | pt86sq | inc86sq | \n",
       "|---|---|---|---|---|---|\n",
       "| 0          | 0          | 0          | 0.38       | 17.6       | 35.2       | 12         | 0          |  0.0       |  0         | 0          | 0          | 1          | 0.1444     | 144        |    0.00000 | \n",
       "| 2          | 2          | 0          | 0.44       |  0.0       |  0.0       |  0         | 1          |  0.8       |  0         | 0          | 1          | 0          | 0.1936     |   0        |    0.64000 | \n",
       "| 1          | 1          | 0          | 0.33       | 22.8       | 22.8       |  0         | 0          |  0.0       | 11         | 1          | 0          | 1          | 0.1089     |   0        |    0.00000 | \n",
       "| 2          | 2          | 1          | 0.25       |  0.0       |  0.0       |  5         | 2          |  8.8       |  0         | 0          | 1          | 1          | 0.0625     |  25        |   77.44000 | \n",
       "| 1          | 1          | 0          | 0.00       |  0.0       |  0.0       |  0         | 2          |  8.1       |  1         | 0          | 0          | 0          | 0.0000     |   0        |   65.61001 | \n",
       "| 0          | 0          | 0          | 1.00       |  0.0       |  0.0       |  0         | 4          | 97.6       |  0         | 0          | 0          | 1          | 1.0000     |   0        | 9525.75977 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  narr86 nfarr86 nparr86 pcnv avgsen tottime ptime86 qemp86 inc86 durat black\n",
       "1 0      0       0       0.38 17.6   35.2    12      0       0.0   0    0    \n",
       "2 2      2       0       0.44  0.0    0.0     0      1       0.8   0    0    \n",
       "3 1      1       0       0.33 22.8   22.8     0      0       0.0  11    1    \n",
       "4 2      2       1       0.25  0.0    0.0     5      2       8.8   0    0    \n",
       "5 1      1       0       0.00  0.0    0.0     0      2       8.1   1    0    \n",
       "6 0      0       0       1.00  0.0    0.0     0      4      97.6   0    0    \n",
       "  hispan born60 pcnvsq pt86sq inc86sq   \n",
       "1 0      1      0.1444 144       0.00000\n",
       "2 1      0      0.1936   0       0.64000\n",
       "3 0      1      0.1089   0       0.00000\n",
       "4 1      1      0.0625  25      77.44000\n",
       "5 0      0      0.0000   0      65.61001\n",
       "6 0      1      1.0000   0    9525.75977"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "library(foreign)\n",
    "crime1<-read.dta(\"https://github.com/thousandoaks/Wooldridge/blob/master/crime1.dta?raw=true\")\n",
    "\n",
    "head(crime1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "Call:\n",
       "lm(formula = narr86 ~ pcnv + ptime86 + qemp86, data = crime1)\n",
       "\n",
       "Residuals:\n",
       "    Min      1Q  Median      3Q     Max \n",
       "-0.7118 -0.4031 -0.2953  0.3452 11.4358 \n",
       "\n",
       "Coefficients:\n",
       "             Estimate Std. Error t value Pr(>|t|)    \n",
       "(Intercept)  0.711772   0.033007  21.565  < 2e-16 ***\n",
       "pcnv        -0.149927   0.040865  -3.669 0.000248 ***\n",
       "ptime86     -0.034420   0.008591  -4.007 6.33e-05 ***\n",
       "qemp86      -0.104113   0.010388 -10.023  < 2e-16 ***\n",
       "---\n",
       "Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\n",
       "\n",
       "Residual standard error: 0.8416 on 2721 degrees of freedom\n",
       "Multiple R-squared:  0.04132,\tAdjusted R-squared:  0.04027 \n",
       "F-statistic:  39.1 on 3 and 2721 DF,  p-value: < 2.2e-16\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Model without avgsen:\n",
    "\n",
    "summary( lm(narr86 ~ pcnv+ptime86+qemp86, data=crime1) )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results indicate that, as a group, the three variables pcnv, ptime86 and qemp86 explain about 4.1% of the variation in narr86."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we increase pcnv by .50 (a large increase in the probability of conviction) then, holding the other factors fixed, $\\Delta{\\hat{narr86}}=-.150(.50)=-.075$. This means that when pcnv increases by .50 the predicted fall in arrests in 100 men is 7.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly a longer prison term leads to a lower predicted number of arrests. In fact, if ptime86 increases from 0 to 12, predicted arrests for a particular man fall by .034(12)=.408"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another quarter in which legal employment is reported lowers predicted arrests by .104 (10.4 arrests among 100 men)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If avgsen is added to the model, we know that $R^2$ will increase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "Call:\n",
       "lm(formula = narr86 ~ pcnv + avgsen + ptime86 + qemp86, data = crime1)\n",
       "\n",
       "Residuals:\n",
       "    Min      1Q  Median      3Q     Max \n",
       "-0.9330 -0.4247 -0.2934  0.3506 11.4403 \n",
       "\n",
       "Coefficients:\n",
       "             Estimate Std. Error t value Pr(>|t|)    \n",
       "(Intercept)  0.706756   0.033151  21.319  < 2e-16 ***\n",
       "pcnv        -0.150832   0.040858  -3.692 0.000227 ***\n",
       "avgsen       0.007443   0.004734   1.572 0.115993    \n",
       "ptime86     -0.037391   0.008794  -4.252 2.19e-05 ***\n",
       "qemp86      -0.103341   0.010396  -9.940  < 2e-16 ***\n",
       "---\n",
       "Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1\n",
       "\n",
       "Residual standard error: 0.8414 on 2720 degrees of freedom\n",
       "Multiple R-squared:  0.04219,\tAdjusted R-squared:  0.04079 \n",
       "F-statistic: 29.96 on 4 and 2720 DF,  p-value: < 2.2e-16\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "library(foreign)\n",
    "crime1<-read.dta(\"https://github.com/thousandoaks/Wooldridge/blob/master/crime1.dta?raw=true\")\n",
    "\n",
    "# Model with avgsen:\n",
    "\n",
    "summary( lm(narr86 ~ pcnv+avgsen+ptime86+qemp86, data=crime1) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We notice that adding the average sentence variable increases $R^2$ from .0413 to .0422. The sign of the coefficient says that a longer average sentence length increases criminal activity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3-3 The expected Value of the OLS Estimators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wooldridge 2016 states and discusses four assumptions, which are direct extensions of the simple regression model assumptions, under which the OLS estimators are unbiased for the population parameters. We also explicitly obtain the bias in OLS when an important variable has been ommited from the regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assumption MLR.1 Linear in Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model in the population can be written as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "y=\\beta_0+\\beta_1*x_1+\\beta_2*x_2+\\dots+\\beta_k*x_k+u\n",
    "\\tag{3.31}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where $\\beta_0, \\beta_1, \\dots,\\beta_k$ are the unknown parameters of interest and u is an unobserved random error or disturbance term"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assumption MLR.2 Random Sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a random sample of n observations ${(x_{i1},x_{i2},\\dots,x_{ik},y_i):i=1,2,\\dots,n}$ following the population model in Assumption MLR.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Under MLR.1 and MLR.2 the OLS estimators $\\hat{\\beta}_0,\\hat{\\beta}_1,\\hat{\\beta}_2,\\dots,\\hat{\\beta}_k$ from the regression of y on $x_1,x_2,\\dots,x_k$ are now considered to be estimators of $\\beta_0,\\beta_1,\\beta_2,\\dots,\\beta_k$. Still we did not include conditions under which the OLS estimates are well defined for a given sample. The next assumption fills that gap."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assumption MLR.3 No perfect collinearity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the sample (and therefore in the population) none of the independent variables is constant, and these are not exact linear relationships among the independent variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If an independent variable in 3.31 is an exact linear combination of the other independent variables, then we say the model suffers from perfect collinearity and it cannot be estimated by OLS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to note that Assumption MLR.3 does allow independent variables to be correlated. They just cannot be perfectly correlated. Also please note that "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assumption MLR.4 Zero Conditional Mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The error u has an expected value of zero given any values of the independent variables. In other words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$E(u|x_1,x_2,\\dots, x_k)=0$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way that Assumption MLR.4 can fails is if the functional relationship between the explained and explanatory variables is misspecified in equation (3.31). Chapter 9 in wooldridge discuss ways of detecting functional form misspecification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ommiting an important factor that is correlated with any of $x_1,x_2,\\dots,x_k$ causes MLR.4 to fail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Theorem 2.1. Unbiadseness of OLS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using Assumptions MLR.1 through MLR.4,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "E(\\hat{\\beta}_j)=\\beta_j,j=0,1,\\dots,k,\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-3a Including Irrelevant Variables in a Regression Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In terms of unbiasedness of $\\hat{\\beta}_1$ and $\\hat{\\beta}_1$ adding a variable $x_3$ which is irrelevant (it has no effect on y after $x_1$ and $x_2$ have been controlled for) has no effect. However, it might affect the variances of the OLS estimators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-3b Ommited Variable Bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now suppose that, rather than including an irrelevant variable, we omit a variable that actually belongs in the true (or) population model. This is often the problem of excluding a relevant variable or underspecifying the model. It can be shown (e.g. Wooldridge 2016, page 79) that leads to OLS estimators which are biased."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3-4 The Variance of OLS Estimators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the central tendencies of the $\\hat{\\beta}_j$, we also need to obtain a measure of the variance of OLS estimators. In order to do so first we need to add a homoskedasticity assumption."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assumption MLR.5 Homoskedasticity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The error u has the same variance given any value of the explanatory variables. In other words,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "Var(u|x_1,x_2,\\dots,x_3)=\\sigma^2\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assumption MLR.5 means that the variance in the error term,u, conditional on the explanatory variables, is the same for all combinations of outcomes of the explanatory variable. If this assumption fails then the model exhibits heteroskedasticity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the equation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$wage=\\beta_0+\\beta_1*educ+\\beta_2*exper+\\beta_3*tenure+u$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "homoskedasticity requires that the variance of the unobserved error u does not depend on the levels of education, experience or tenure. That is,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$Var(u|educ, exper, tenure)=\\sigma^2$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assumptions MLR.1 through MLR.5 are collectively known as Gauss-Markov assumptions (for cross-sectional analysis). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far our assumptions are suitable only when applied to cross-sectional analysis with random sampling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Theorem 3.2 Sampling Variances of the OLS Slope Estimators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Under Assumptions MLR1 through MLR.5, conditional on the sample values of the independent variables,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "Var(\\hat{\\beta}_j)=\\frac{\\sigma^2}{SST_j(1-R_j^2)}\n",
    "\\tag{3.51}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for j=1,2,...,k where $SST_j=\\sum_{i=1}^{n}(x_{ij}-\\vec{x_j})^2$ is the total sample variation in $x_j$ and $R_i^2$ is the R-squared from regressing $x_j$ on all other independent variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-4a The Components of the OLS Variances. Multicollinearity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Equation (3.51) shows that the variance of $\\hat{\\beta_j}$ depends on three factors. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The error variance, $\\sigma^2$. From equation (3.51) a larger $\\sigma^2$ means larger sampling variances for the OLS estimators. Because the error variance is a feature of the population, it has nothing to do with the sample size. It is the one component of (3.51) that is unknown. For a given dependent variables, y, there is only one way to reduce the error variance and that is to add more explanatory variables to the equation (i.e. take some factors out of the error term)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Total Sample Variation in $x_j, SSt_j$. From equation (3.51) we see that the larger the total variation in $x_j$ the smaller is $Var(\\hat{\\beta_j})$. Thus, everyting being equal, for estimating $\\beta_j$ we prefer to have as much sample variation in $x_j$ as possible. Rarely it is possible in social sciences to choose the sample values of the independent variables, there is however a way to increase the sample variation in each of the independent variables: increase the sample size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Linear Relationships among the independent Variables $R_j^2$. It is important to note that this R-squared is distinct from the R-squared we defined in the regression of y over x's. In this case $R_1^2$ is the R-squared from the simple regression of $x_1$ on $x_2$. Thus in this case a value of $R_1^2$ close to one indicates that $x_2$ explains much of the variation in $x_1$ in the sample. This means that both variables are highly correlated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the general case,$R_j^2$ is the proportion of the total variance in $x_j$ that can be explained by the other independent variables appearing in the equation. When $R_j^2$ is \"close\" to one, we see from equation (3.51) that $Var(\\hat{\\beta_j})$ will tend to be large. High correlation between two or more variables is called multicollinearity. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just as a large value of $R_j^2$ can cause a large $Var(\\hat{\\beta_j})$, so can a small value of $SST_j$. in the social sciences, where we are usually passive colectors of data, there is no good way to reduce variances of unbiased estimators other than to collect more data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another important point is that a high degree of correlation between certain independent variables can be irrelevant as to how well we can estimate other parameters in the model. For example, consider a model with three independent variables:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "y=\\beta_0+\\beta_1*x_1+\\beta_2*x_2+\\dots+\\beta_3*x_3\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where x2 and x3 are correlated. then their variances may be large, but the amount of correlation between x2 and x3 has no direct effect on $Var(\\hat{\\beta_1})$. In fact if x1 is uncorrelated with x2 and x3 then $R_1^2=0$ and $Var(\\hat{\\beta_1})=\\sigma^2/SST_1$, regardless of how much correlation there is between x2 and x3. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-4c Estimating $\\sigma^2$: Standard Errors of the OLS Estimators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now show how to choose an unbiased estimator of $\\sigma^2$, which then allows us to obtain unbiased estimators of $Var(\\hat{\\beta_j})$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because $\\sigma^2=E(u^2)$, an unbiased estimator of $\\sigma^2$ would be the sample average of the squared errors: $n^{-1}\\sum_{i=1}^{n}u_i^2$, unfortunately this is not a true estimator because we do not observe the $u_i$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nevertheless, recall that the errors can be written as $u_i=y_i-\\beta_0-\\beta_1*x_{i1}-\\beta_2*x_{i1}-\\dots-\\beta_k*x_{ik}$ and replacing each $\\beta_j$ with its OLS estimator, we get the OLS residuals:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\hat{u}_i=y_i-\\hat{\\beta}_0-\\hat{\\beta}_1*x_{i1}-\\hat{\\beta}_2*x_{i1}-\\dots-\\hat{\\beta}_k*x_{ik}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The unbiased estimator os $\\sigma^2$ in the general multiple regression case is"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "\\hat{\\sigma}^2=\\frac{\\sum_{i=1}^{n}\\hat{u}_i^2}{n-k-1}=SSR/(n-k-1)\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Theorem 3.3 Unbiassed Estimation of $\\sigma^2$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Under the Gauss-Markov assumption MLR.1 through MLR.5, $E(\\hat{\\sigma}^2)=\\sigma^2$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For constructing confidence intervals and conducting tests we will need to estimate the standard deviation of $\\hat{\\beta_j}$, which is just the square root of the variance:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "sd(\\hat{\\beta_j})=\\sigma/[SST_j(1-R_j^2)]^{1/2}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "since $\\sigma$ is unknown, we replace it with its estimator $\\hat{\\sigma}$. This gives us the standard error of $\\hat{\\beta}_j$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "se(\\hat{\\beta_j})=\\hat{\\sigma}/[SST_j(1-R_j^2)]^{1/2}\n",
    "\\tag{3.58}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because (3.58) is obtained directly from the variance formula in (3.51) and because (3.51) relies on the homoskedasticity Assumption MLR.5. It follows that the standard error formula in (3.58) is not a valid estimator of $sd(\\hat{\\beta}_j)$ if the errors exhibit heteroskedasticity. Thus, while heteroskedasticity does not cause bias in the $\\hat{\\beta_j}$ it does lead to bias in the usual formula for $Var(\\hat{\\beta}_j)$, which then invalidates the standard errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3-5 Efficiency of OLS: The Gauss-Markov Theorem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $\\hat{\\beta}_0,\\hat{\\beta}_1,\\hat{\\beta}_2,\\dots,\\hat{\\beta}_k$ denote the OLS estimators in model (3.31) under Assumptions MLR.1 through MLR.5 The Gauss-Markov theorem says that in the class of linear unbiased estimators, OLS has the smallest variance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Theorem 3.4 Gauss-Markov Theorem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Under Assumptions MLR.1 through MLR.5 $\\hat{\\beta}_0,\\hat{\\beta}_1,\\hat{\\beta}_2,\\dots,\\hat{\\beta}_k$ are the best linear unbiased estimators (BLUES) of $\\beta_0,\\beta_1,\\beta_2,\\dots,\\beta_k$ respectifully"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "r"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.4.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}