Three-variables, 20 observations

E C O N S T A T S
Regression Examples
Example :	alpha=1	alpha=0
One-Variable Regression Example From J. Johnston (1984) p19.	With Intercept	No Intercept
Two-Variable Regression Example From J. Johnston (1984) p178.	With Intercept	No Intercept
Three-Independent Variables Regression Example	With Intercept	No Intercept
Three-variables, 20 observations	With Intercept	No Intercept
Three-variables, 10 observations	With Intercept	No Intercept


         



EXAMPLE:   Three-variables, 20 observations
     |     
No Intercept





I. Data and Summary Stats 
Three-variables, 20 observations
Observations :  n=20
Independent Variables :  k=3
No Intercept


Data Table 

 obs y_i x_i,1 x_i,2 x_i,3
1. 5 4 5 4
2. 4 4 5 3
3. 9 4 9 8
4. 3 5 8 7
5. 5 5 5 9
6. 5 8 10 8
7. 8 9 7 13
8. 5 5 14 14
9. 5 14 6 12
10. 12 9 9 9
11. 4 6 12 7
12. 8 9 5 7
13. 10 11 11 19
14. 16 8 6 14
15. 2 9 12 14
16. 9 12 7 17
17. 9 12 9 5
18. 3 5 8 10
19. 11 12 10 7
20. 21 17 22 23

sum 154 168 180 210
mean 7.700 8.400 9 10.50
StD ≡ σ 4.736
3.719
4.039
5.186





Means and Standard Deviations 



 Mean 
 Var 
 StD 


 M_x= Σx_i/n   
 Var_x≡σ_x² = Σ(x-M_x)² / n-1 
 StD_x≡σ_x=Var_x^1/2

y
7.700
22.43
4.736

x₁
8.400
 13.83
 3.719

x₂
9
 16.32
 4.039

x₃
10.50
 26.89
 5.186



Covariance Matrix  -- Cov(xi,xj)=Σ[(xi-M_xi)(xj-M_xj)] / n-1 
     NOTE: be careful of MS Excel's COVAR() function,
     which divides by n instead of n-1.

  y   x₁   x₂   x₃
y 22.43 10.81 8.474 13.11
x₁ 10.81 13.83 6.684 11.42
x₂ 8.474 6.684 16.32 12.53
x₃ 13.11 11.42 12.53 26.89



Correlation Matrix  -- Corr(x_i,x_j)=Σ[(x_i-M_xi)(x_j-M_xj)] / (n-1)σ_iσ_j

  y   x₁   x₂   x₃
y  1.000  0.614  0.443  0.534
x₁  0.614  1.000  0.445  0.592
x₂  0.443  0.445  1.000  0.598
x₃  0.534  0.592  0.598  1.000



The basic input matrices are: 



 
  y =  
 (20x1)   

5
4
9
3
5
5
8
5
5
12
4
8
10
16
2
9
9
3
11
21


   


  X = 
 (20x3)

4 5 4
4 5 3
4 9 8
5 8 7
5 5 9
8 10 8
9 7 13
5 14 14
14 6 12
9 9 9
6 12 7
9 5 7
11 11 19
8 6 14
9 12 14
12 7 17
12 9 5
5 8 10
12 10 7
17 22 23



   

 

   X' = 
 (3x20)   


4 4 4 5 5 8 9 5 14 9 6 9 11 8 9 12 12 5 12 17
5 5 9 8 5 10 7 14 6 9 12 5 11 6 12 7 9 8 10 22
4 3 8 7 9 8 13 14 12 9 7 7 19 14 14 17 5 10 7 23









II. Regression Calculations

y_i =  b₁ x_i,1 +  b₂ x_i,2 +  b₃ x_i,3 +  u_i

The q.c.e. basic equation in matrix form is: 
  y = Xb + e
  where y (dependent variable) is (nx1) or  (20x1)
        X (independent vars) is (nxk) or  (20x3)
        b (betas) is (kx1) or  (3x1)
        e (errors) is (nx1) or  (20x1)
Minimizing sum or squared errors using calculus results in the OLS eqn:
  b=(X'X)^-1.X'y
To minimize the sum of squared errors of
a k dimensional line that describes the relationship 
between the k independent variables and y we
find the set of slopes (betas) that minimizes
Σ_{i=1 to n}e_i²
Re-written in linear algebra we seek to min e'e
Rearranging the regression model equation, we get e = y - Xb
So e'e = (y-Xb)'(y-Xb) = y'y - 2b'X'y + b'X'Xb   (see Judge et al (1985) p14 )
Differentiating by b we get 0 = - 2X'y + X'Xb -> 2X'Xb=2X'y
Rearranging, dividing both sides by 2 -> b = X'X^-1X'y
So to obtain the elements of the (kx1) vector b we need the elements 
of the (kxk) matrix X'X^-1 and of the (kx1) matrix X'y. 
Caclulating X'y is easy (see (1) below) but X'X^-1 requires 
first calculation of X'X then finding cofactors -- see (4) -- and 
the deteminant - see (3) - in order to invert.



(1) X'y Matrix  (3x1)


1499
1547
1866



(2) X'X Matrix  (3x3)



1674	
1639	
1981	

1639	
1930	
2128	

1981	
2128	
2716	


(3) Determinant   Det(X'X)≡|X'X|  
    i.e. the determinant of matrix of X'X 

Det(X'X) = 142959642
Det(X'X) =    1674*1930*2716 - 1674*2128*2128
        ... - 1639*1639*2716 + 1639*2128*1981
        ... + 1981*1639*2128 - 1981*1930*1981
        

(4) Cofactors(X'X) i.e. cofactor matrix of X 'X  (3x3)
713496 -235956 -335538
-235956 622223 -315413
-335538 -315413 544499


(5) Adj(X'X) i.e. adjugate matrix of X'X, this is just the 
    transpose of the cofactor matrix.  (3x3)
    For a symmetric matrix, will be same as cofactor matrix.
713496 -235956 -335538
-235956 622223 -315413
-335538 -315413 544499



(6) Inverse Matrix, inv(X'X)≡(X'X)^-1
    = adj(X'X)/|X'X| = adj(X'X)/142959642   (3x3)

0.004991 -0.001651 -0.002347
-0.001651 0.004352 -0.002206
-0.002347 -0.002206 0.003809


(7) Beta Matrix (β)
    b = [X'X^-1].[X'y] , this is  (3x1).
    Finally we can calculate b through matrix multiplication.

 Betas 

 β ₁  0.5484 
 β ₂  0.1421 
 β ₃  0.1757 

   =    
X'X^-1 

 0.004991  -0.001651  -0.002347 
 -0.001651  0.004352  -0.002206 
 -0.002347  -0.002206  0.003809 

   X   
 X'y 
 1499 
 1547 
 1866 



Yhat₁= + 0.5484x4 + 0.1421x5 + 0.1757x4 = 3.6070
Yhat₂= + 0.5484x4 + 0.1421x5 + 0.1757x3 = 3.4313
Yhat₃= + 0.5484x4 + 0.1421x9 + 0.1757x8 = 4.8784
Yhat₄= + 0.5484x5 + 0.1421x8 + 0.1757x7 = 5.1089
Yhat₅= + 0.5484x5 + 0.1421x5 + 0.1757x9 = 5.0339
Yhat₆= + 0.5484x8 + 0.1421x10 + 0.1757x8 = 7.2139
Yhat₇= + 0.5484x9 + 0.1421x7 + 0.1757x13 = 8.2144
Yhat₈= + 0.5484x5 + 0.1421x14 + 0.1757x14 = 7.1917
Yhat₉= + 0.5484x14 + 0.1421x6 + 0.1757x12 = 10.6384
Yhat₁₀= + 0.5484x9 + 0.1421x9 + 0.1757x9 = 7.7959
Yhat₁₁= + 0.5484x6 + 0.1421x12 + 0.1757x7 = 6.2258
Yhat₁₂= + 0.5484x9 + 0.1421x5 + 0.1757x7 = 6.8759
Yhat₁₃= + 0.5484x11 + 0.1421x11 + 0.1757x19 = 10.9340
Yhat₁₄= + 0.5484x8 + 0.1421x6 + 0.1757x14 = 7.6997
Yhat₁₅= + 0.5484x9 + 0.1421x12 + 0.1757x14 = 9.1008
Yhat₁₆= + 0.5484x12 + 0.1421x7 + 0.1757x17 = 10.5624
Yhat₁₇= + 0.5484x12 + 0.1421x9 + 0.1757x5 = 8.7381
Yhat₁₈= + 0.5484x5 + 0.1421x8 + 0.1757x10 = 5.6360
Yhat₁₉= + 0.5484x12 + 0.1421x10 + 0.1757x7 = 9.2316
Yhat₂₀= + 0.5484x17 + 0.1421x22 + 0.1757x23 = 16.4905


ESS
=(3.607 - 7.700)^2
=(3.431 - 7.700)^2
=(4.878 - 7.700)^2
=(5.109 - 7.700)^2
=(5.034 - 7.700)^2
=(7.214 - 7.700)^2
=(8.214 - 7.700)^2
=(7.192 - 7.700)^2
=(10.64 - 7.700)^2
=(7.796 - 7.700)^2
=(6.226 - 7.700)^2
=(6.876 - 7.700)^2
=(10.93 - 7.700)^2
=(7.700 - 7.700)^2
=(9.101 - 7.700)^2
=(10.56 - 7.700)^2
=(8.738 - 7.700)^2
=(5.636 - 7.700)^2
=(9.232 - 7.700)^2
=(16.49 - 7.700)^2
=174.584038870215

REPORT 


 obs 
 calculation of yhat_obs
yhat_obs = Σβ_ix_i,obs
 yhat_obs
a
 y_obs
 (data)
 Mean_y
 (y_obs - yhat_obs)²
 (yhat_obs - M_y)²
 (y_obs - M_y)²
a
e_obs=y_obs-yhat_obs
e_obs²

1
Yhat₁ = Σβ_ix_i,1 = 0.5484x4 + 0.1421x5 + 0.1757x4  = 3.607 5 7.700 1.941 16.75 7.290  e₁ = 5 - 3.607 = 1.393 1.941

2
Yhat₂ = Σβ_ix_i,2 = 0.5484x4 + 0.1421x5 + 0.1757x3  = 3.431 4 7.700 0.3235 18.22 13.69  e₂ = 4 - 3.431 = 0.5687 0.3235

3
Yhat₃ = Σβ_ix_i,3 = 0.5484x4 + 0.1421x9 + 0.1757x8  = 4.878 9 7.700 16.99 7.962 1.690  e₃ = 9 - 4.878 = 4.122 16.99

4
Yhat₄ = Σβ_ix_i,4 = 0.5484x5 + 0.1421x8 + 0.1757x7  = 5.109 3 7.700 4.447 6.714 22.09  e₄ = 3 - 5.109 = -2.109 4.447

5
Yhat₅ = Σβ_ix_i,5 = 0.5484x5 + 0.1421x5 + 0.1757x9  = 5.034 5 7.700 0.001148 7.108 7.290  e₅ = 5 - 5.034 = -0.03389 0.001148

6
Yhat₆ = Σβ_ix_i,6 = 0.5484x8 + 0.1421x10 + 0.1757x8  = 7.214 5 7.700 4.902 0.2363 7.290  e₆ = 5 - 7.214 = -2.214 4.902

7
Yhat₇ = Σβ_ix_i,7 = 0.5484x9 + 0.1421x7 + 0.1757x13  = 8.214 8 7.700 0.04598 0.2646 0.09000  e₇ = 8 - 8.214 = -0.2144 0.04598

8
Yhat₈ = Σβ_ix_i,8 = 0.5484x5 + 0.1421x14 + 0.1757x14  = 7.192 5 7.700 4.804 0.2584 7.290  e₈ = 5 - 7.192 = -2.192 4.804

9
Yhat₉ = Σβ_ix_i,9 = 0.5484x14 + 0.1421x6 + 0.1757x12  = 10.64 5 7.700 31.79 8.634 7.290  e₉ = 5 - 10.64 = -5.638 31.79

10
Yhat₁₀ = Σβ_ix_i,10 = 0.5484x9 + 0.1421x9 + 0.1757x9  = 7.796 12 7.700 17.67 0.009190 18.49  e₁₀ = 12 - 7.796 = 4.204 17.67

11
Yhat₁₁ = Σβ_ix_i,11 = 0.5484x6 + 0.1421x12 + 0.1757x7  = 6.226 4 7.700 4.954 2.173 13.69  e₁₁ = 4 - 6.226 = -2.226 4.954

12
Yhat₁₂ = Σβ_ix_i,12 = 0.5484x9 + 0.1421x5 + 0.1757x7  = 6.876 8 7.700 1.264 0.6792 0.09000  e₁₂ = 8 - 6.876 = 1.124 1.264

13
Yhat₁₃ = Σβ_ix_i,13 = 0.5484x11 + 0.1421x11 + 0.1757x19  = 10.93 10 7.700 0.8723 10.46 5.290  e₁₃ = 10 - 10.93 = -0.9340 0.8723

14
Yhat₁₄ = Σβ_ix_i,14 = 0.5484x8 + 0.1421x6 + 0.1757x14  = 7.700 16 7.700 68.90 0.000000 68.89  e₁₄ = 16 - 7.700 = 8.300 68.90

15
Yhat₁₅ = Σβ_ix_i,15 = 0.5484x9 + 0.1421x12 + 0.1757x14  = 9.101 2 7.700 50.42 1.962 32.49  e₁₅ = 2 - 9.101 = -7.101 50.42

16
Yhat₁₆ = Σβ_ix_i,16 = 0.5484x12 + 0.1421x7 + 0.1757x17  = 10.56 9 7.700 2.441 8.193 1.690  e₁₆ = 9 - 10.56 = -1.562 2.441

17
Yhat₁₇ = Σβ_ix_i,17 = 0.5484x12 + 0.1421x9 + 0.1757x5  = 8.738 9 7.700 0.06860 1.078 1.690  e₁₇ = 9 - 8.738 = 0.2619 0.06860

18
Yhat₁₈ = Σβ_ix_i,18 = 0.5484x5 + 0.1421x8 + 0.1757x10  = 5.636 3 7.700 6.949 4.260 22.09  e₁₈ = 3 - 5.636 = -2.636 6.949

19
Yhat₁₉ = Σβ_ix_i,19 = 0.5484x12 + 0.1421x10 + 0.1757x7  = 9.232 11 7.700 3.127 2.346 10.89  e₁₉ = 11 - 9.232 = 1.768 3.127

20
Yhat₂₀ = Σβ_ix_i,20 = 0.5484x17 + 0.1421x22 + 0.1757x23  = 16.49 21 7.700 20.34 77.27 176.9  e₂₀ = 21 - 16.49 = 4.510 20.34
RSS = 
Σ(y_obs - yhat_obs)² ESS = 
Σ(yhat_obs - M_y)² TSS = 
Σ(y_obs - M_y)² e'e=Σe_obs²
sum-> 242.2 174.6 426.2 242.2




(11) Betas and their t-Stats
     from the covar matrix of b=σ²(X'X)^-1
     the var(βi) = σ²v_ii where v_ii is the ith diag element of X'X^-1
     where σ² = e'e / n-k  (k=num of ind vars plus 1 for the intercept if present).
     and where v_ii is the ith diag element of X'X^-1
     Std(βi) = sqr root of Var(βi) 
     TStat(βi) = βi / Std(βi)
     Estimate of σ² = 14.249761617009



 Coef value 
 StD(β)
 tStat(β)

β₁ = 
   0.004991 * 1499
 + -0.001651 * 1547
 + -0.002347 * 1866
 = 0.5484
(14.25 * 0.004991)^1/2 
 = 0.2667
0.5484 / 0.2667
 = 2.056

β₂ = 
   -0.001651 * 1499
 + 0.004352 * 1547
 + -0.002206 * 1866
 = 0.1421
(14.25 * 0.004352)^1/2 
 = 0.2490
0.1421 / 0.2490
 = 0.5708

β₃ = 
   -0.002347 * 1499
 + -0.002206 * 1547
 + 0.003809 * 1866
 = 0.1757
(14.25 * 0.003809)^1/2 
 = 0.2330
0.1757 / 0.2330
 = 0.7542


(12) Table of Outputs: y_obs =  β₁  X_obs,1 +  β₂  X_obs,2 +  β₃  X_obs,3 +  e_obs
 0.5484   0.1421   0.1757  
(2.056) (0.5708) (0.7542)  <- tstats 
 r² = 0.431614    |    adj r² = 0.325042


(13)  RSS   = Sum{y     - y_hat }^2  = 242.245947489152
      TSS   = Sum{y     - y_avg }^2  = 426.2
      ESS(a)= Sum{y_hat - y_avg }^2  = 174.584038870215
           we use the ESSb (below) cuz smthn wrng w ESS when no intercept.
      ESS(b)= TSS-RSS 183.954052510848
      note:  TSS = ESS + RSS 
(14)  r² = ESS/TSS = 0.431614388810061
(15)  adjusted r² = ESS/TSS = 0.325042086711947
(16)  F-stat = [ESS/(k-1)] / [RSS/(n-k)] = 4.3030907570008
               see Johnston(1984) p186
               F measures the joint significance of 
               all explanatory variables.
      Alternatively:  F-stat = r²/(k-1) / (1-r²)/(n-k)
(17)  Durbin-Watson Statistic (DW or d) measures autocorrelation.
      DW = 2.60747380254565
 ________________________________________________________

Note, RSS, ESS and TSS stand for ... 
      Residual Sum of Squares (RSS),
      Explained Sum of Squares (ESS), and 
      Total Sum of Squares (TSS).
      However ESS is sometimes referred to as the Regression Sum of Squares.
      and     RSS is sometimes referred to as the Sum of Squares Rresidual.
Note, an alternative way of calculating TSS, ESS is...
      TSS = y'Ay   
      ESS = b_v'X_v'Ay  where b_v' X_v' are b' & X' wo intercept row col
      RSS = TSS-ESS 

Bibliography 

J. Johnston (1984) Econometric Methods, 3rd ed.
Judge et al (1985) The Theory and Practice of Econometrics 2rd ed, Wiley, New York.
Donald F. Morrison (1990) Multivariate Statistical Methods, 3rd edition, McGraw Hill, New York. 
A. H. Studenmund (1997) Using Econometrics: A Practical Guide, 3rd edition.  Addison-Wesley, Reading.