Part 5: Regression Algebra and Fit 5-1/34 Econometrics I Professor William Greene Stern School of Business Department of Economics

Part 5: Regression Algebra and Fit 5-2/34 Gauss-Markov Theorem A theorem of Gauss and Markov: Least Squares is the minimum variance linear unbiased estimator (MVLUE) 1. Linear estimator 2. Unbiased: E[b|X] = β Theorem: Var[b*|X] – Var[b|X] is nonnegative definite for any other linear and unbiased estimator b* that is not equal to b. Definition: b is efficient in this class of estimators.

Part 5: Regression Algebra and Fit 5-3/34 Implications of Gauss-Markov Theorem: Var[b*|X] – Var[b|X] is nonnegative definite for any other linear and unbiased estimator b* that is not equal to b. Implies: b k = the kth particular element of b. Var[b k |X] = the kth diagonal element of Var[b|X] Var[b k |X] < Var[b k *|X] for each coefficient. c b = any linear combination of the elements of b. Var[c b|X] < Var[c b*|X] for any nonzero c and b* that is not equal to b.

Part 5: Regression Algebra and Fit 5-4/34 Summary: Finite Sample Properties of b Unbiased: E[b]= Variance: Var[b|X] = 2 (X X) -1 Efficiency: Gauss-Markov Theorem with all implications Distribution: Under normality, b|X ~ N[, 2 (X X) -1 (Without normality, the distribution is generally unknown.)

Part 5: Regression Algebra and Fit 5-5/34 Comparação de modelos Podemos comparar modelos sem usar testes estatsticos, mas sim medidas conhecidas como criterios de informação que traduzem a qualidade de ajustamento de um modelo. Para estes indicadores a variável principal acaba por ser uma medida do valor absoluto dos erros.

Part 5: Regression Algebra and Fit 5-6/34 Medida de Ajuste R 2 = b X M 0 Xb/y M 0 y (Very Important Result.) R 2 is bounded by zero and one only if: (a) There is a constant term in X and (b) The line is computed by linear least squares.

Part 5: Regression Algebra and Fit 5-7/34 Comparing fits of regressions Make sure the denominator in R 2 is the same - i.e., same left hand side variable. Example, linear vs. loglinear. Loglinear will almost always appear to fit better because taking logs reduces variation.

Part 5: Regression Algebra and Fit 5-8/34

Part 5: Regression Algebra and Fit 5-9/34 Adjusted R Squared Adjusted R 2 (for degrees of freedom) = 1 - [(n-1)/(n-K)](1 - R 2 ) includes a penalty for variables that dont add much fit. Can fall when a variable is added to the equation.

Part 5: Regression Algebra and Fit 5-10/34 Adjusted R 2 What is being adjusted? The penalty for using up degrees of freedom. = 1 - [e e/(n – K)]/[y M 0 y/(n-1)] uses the ratio of two unbiased estimators. Is the ratio unbiased? = 1 – [(n-1)/(n-K)(1 – R 2 )] Will rise when a variable is added to the regression? is higher with z than without z if and only if the t ratio on z is in the regression when it is added is larger than one in absolute value.

Part 5: Regression Algebra and Fit 5-11/34 Full Regression (Without PD) Ordinary least squares regression LHS=G Mean = Standard deviation = Number of observs. = 36 Model size Parameters = 9 Degrees of freedom = 27 Residuals Sum of squares = Standard error of e = Fit R-squared = <********** Adjusted R-squared = <********** Info criter. LogAmemiya Prd. Crt. = <********** Akaike Info. Criter. = <********** Model test F[ 8, 27] (prob) = 503.3(.0000) Variable| Coefficient Standard Error t-ratio P[|T|>t] Mean of X Constant| ** PG| *** Y|.02214*** PNC| PUC| PPT| PN| * PS| *** YEAR| **

Part 5: Regression Algebra and Fit 5-12/34 PD added to the model. R 2 rises, Adj. R 2 falls Ordinary least squares regression LHS=G Mean = Standard deviation = Number of observs. = 36 Model size Parameters = 10 Degrees of freedom = 26 Residuals Sum of squares = Standard error of e = Fit R-squared = Was Adjusted R-squared = Was Variable| Coefficient Standard Error t-ratio P[|T|>t] Mean of X Constant| ** PG| *** Y|.02231*** PNC| PUC| PPT| PD| (NOTE LOW t ratio) PN| PS| ** YEAR| *

Part 5: Regression Algebra and Fit 5-13/34 Outras medidas de Ajuste Para alternativas não aninhadas Inclui penalidade de grau de liberdade. Critério de Informação Schwarz (BIC): n log(e e) + k(log(n)) Akaike (AIC): n log(e e) + 2k Quando se quer decidir entre dois modelos não aninhados, o melhor é o que produz o menor valor do critério. A penalidade no BIC de incluir algo não relevante é maior que no AIC.

Part 5: Regression Algebra and Fit 5-14/34 Outras medidas de Ajuste Tanto o AIC quanto o BIC aumentam conforme SQR aumenta. Além disso, ambos critérios penalizam modelos com muitas variáveis sendo que valores menores de AIC e BIC são preferíveis. Como modelos com mais variáveis tendem a produzir menor SQR mas usam mais parâmetros, a melhor escolha é balancear o ajuste com a quantidade de variáveis. A penalidade no BIC de incluir algo não relevante é maior que no AIC.

Part 5: Regression Algebra and Fit 5-15/34 Multicolinearidade

Part 5: Regression Algebra and Fit 5-16/34 Formas funcionais

Part 5: Regression Algebra and Fit 5-17/34 Specification and Functional Form: Nonlinearity

Part 5: Regression Algebra and Fit 5-18/34 Log Income Equation Ordinary least squares regression LHS=LOGY Mean = Estimated Cov[b1,b2] Standard deviation = Number of observs. = Model size Parameters = 7 Degrees of freedom = Residuals Sum of squares = Standard error of e = Fit R-squared = Variable| Coefficient Standard Error b/St.Er. P[|Z|>z] Mean of X AGE|.06225*** AGESQ| *** D Constant| *** MARRIED|.32153*** HHKIDS| *** FEMALE| EDUC|.05542*** Average Age = Estimated Partial effect = – 2(.00074) = Estimated Variance e-6 + 4( ) 2 ( e-10) + 4( )( e-8) = e-08. Estimated standard error =

Part 5: Regression Algebra and Fit 5-19/34 Specification and Functional Form: Interaction Effect

Part 5: Regression Algebra and Fit 5-20/34 Interaction Effect Ordinary least squares regression LHS=LOGY Mean = Standard deviation = Number of observs. = Model size Parameters = 4 Degrees of freedom = Residuals Sum of squares = Standard error of e = Fit R-squared = Adjusted R-squared = Model test F[ 3, 27318] (prob) = 82.4(.0000) Variable| Coefficient Standard Error b/St.Er. P[|Z|>z] Mean of X Constant| *** AGE|.00227*** FEMALE|.21239*** AGE_FEM| *** Do women earn more than men (in this sample?) The coefficient on FEMALE would suggest so. But, the female difference is *Age. At average Age, the effect is ( ) =

Part 5: Regression Algebra and Fit 5-21/34

Part 5: Regression Algebra and Fit 5-22/34

Part 5: Regression Algebra and Fit 5-23/34 Quebra estrutural

Part 5: Regression Algebra and Fit 5-24/34 Linear Restrictions Context: How do linear restrictions affect the properties of the least squares estimator? Model: y = X + Theory (information) R - q = 0 Restricted least squares estimator: b* = b - (X X) -1 R [R(X X) -1 R ] -1 (Rb - q) Expected value: E[b*] = - (X X) -1 R [R(X X) -1 R ] -1 (Rb - q) Variance: 2 (X X) (X X) -1 R [R(X X) -1 R ] -1 R(X X) -1 = Var[b] – a nonnegative definite matrix < Var[b] Implication: (As before) nonsample information reduces the variance of the estimator.

Part 5: Regression Algebra and Fit 5-25/34 Interpretation Case 1: Theory is correct: R - q = 0 (the restrictions do hold). b* is unbiased Var[b*] is smaller than Var[b] How do we know this? Case 2: Theory is incorrect: R - q 0 (the restrictions do not hold). b* is biased – what does this mean? Var[b*] is still smaller than Var[b]

Part 5: Regression Algebra and Fit 5-26/34 Linear Least Squares Subject to Restrictions Restrictions: Theory imposes certain restrictions on parameters. Some common applications Dropping variables from the equation = certain coefficients in b forced to equal 0. (Probably the most common testing situation. Is a certain variable significant?) Adding up conditions: Sums of certain coefficients must equal fixed values. Adding up conditions in demand systems. Constant returns to scale in production functions. Equality restrictions: Certain coefficients must equal other coefficients. Using real vs. nominal variables in equations. General formulation for linear restrictions: Minimize the sum of squares, e e, subject to the linear constraint Rb = q.

Part 5: Regression Algebra and Fit 5-27/34 Restricted Least Squares

Part 5: Regression Algebra and Fit 5-28/34 Restricted Least Squares Solution General Approach: Programming Problem Minimize for L = (y - X ) (y - X ) subject to R = q Each row of R is the K coefficients in a restriction. There are J restrictions: J rows 3 = 0: R = [0,0,1,0,…] q = (0). 2 = 3 : R = [0,1,-1,0,…]q = (0) 2 = 0, 3 = 0: R = 0,1,0,0,…q = 0 0,0,1,0,… 0

Part 5: Regression Algebra and Fit 5-29/34 Solution Strategy Quadratic program: Minimize quadratic criterion subject to linear restrictions All restrictions are binding Solve using Lagrangean formulation Minimize over (, ) L* = (y - X ) (y - X ) + 2 (R -q) (The 2 is for convenience – see below.)

Part 5: Regression Algebra and Fit 5-30/34 Restricted LS Solution

Part 5: Regression Algebra and Fit 5-31/34 Restricted Least Squares

Part 5: Regression Algebra and Fit 5-32/34 Aspects of Restricted LS 1. b* = b - Cm where m = the discrepancy vector Rb - q. Note what happens if m = 0. What does m = 0 mean? 2. =[R(X X) -1 R ] -1 (Rb - q) = [R(X X) -1 R ] -1 m. When does = 0. What does this mean? 3. Combining results: b* = b - (X X) -1 R. How could b* = b?

Part 5: Regression Algebra and Fit 5-33/34