|
EXAMPLE: Three-variables, 20 observations
|
With Intercept
I. Data and Summary Stats
Three-variables, 20 observations
Observations : n=20
Independent Variables : k=3
With Intercept
Data Table
obs | yi | intcpt | xi,1 | xi,2 | xi,3
| 1. | 5 | 1 | 4 | 5 | 4
| 2. | 4 | 1 | 4 | 5 | 3
| 3. | 9 | 1 | 4 | 9 | 8
| 4. | 3 | 1 | 5 | 8 | 7
| 5. | 5 | 1 | 5 | 5 | 9
| 6. | 5 | 1 | 8 | 10 | 8
| 7. | 8 | 1 | 9 | 7 | 13
| 8. | 5 | 1 | 5 | 14 | 14
| 9. | 5 | 1 | 14 | 6 | 12
| 10. | 12 | 1 | 9 | 9 | 9
| 11. | 4 | 1 | 6 | 12 | 7
| 12. | 8 | 1 | 9 | 5 | 7
| 13. | 10 | 1 | 11 | 11 | 19
| 14. | 16 | 1 | 8 | 6 | 14
| 15. | 2 | 1 | 9 | 12 | 14
| 16. | 9 | 1 | 12 | 7 | 17
| 17. | 9 | 1 | 12 | 9 | 5
| 18. | 3 | 1 | 5 | 8 | 10
| 19. | 11 | 1 | 12 | 10 | 7
| 20. | 21 | 1 | 17 | 22 | 23
| | | | | |
| sum | 154 | 20 | 168 | 180 | 210
| mean | 7.700 | 1 | 8.400 | 9 | 10.50
| StD ≡ σ | 4.736 | 0
| 3.719
| 4.039
| 5.186
|
Means and Standard Deviations
| Mean
Var
StD
| Mx= Σxi/n
Varx≡σx2 = Σ(x-Mx)2 / n-1
StDx≡σx=Varx1/2
y
| 7.700
| 22.43
| 4.736
|
x1
| 8.400
| 13.83
| 3.719
|
x2
| 9
| 16.32
| 4.039
|
x3
| 10.50
| 26.89
| 5.186
| | | | | | |
Covariance Matrix -- Cov(xi,xj)=Σ[(xi-Mxi)(xj-Mxj)] / n-1 NOTE: be careful of MS Excel's COVAR() function, which divides by n instead of n-1.
| y | x1 | x2 | x3
| y | 22.43 | 10.81 | 8.474 | 13.11
| x1 | 10.81 | 13.83 | 6.684 | 11.42
| x2 | 8.474 | 6.684 | 16.32 | 12.53
| x3 | 13.11 | 11.42 | 12.53 | 26.89
|
Correlation Matrix -- Corr(xi,xj)=Σ[(xi-Mxi)(xj-Mxj)] / (n-1)σiσj
| y | x1 | x2 | x3
| y | 1.000 | 0.614 | 0.443 | 0.534
| x1 | 0.614 | 1.000 | 0.445 | 0.592
| x2 | 0.443 | 0.445 | 1.000 | 0.598
| x3 | 0.534 | 0.592 | 0.598 | 1.000
|
The basic input matrices are:
y = (20x1)
5
4
9
3
5
5
8
5
5
12
4
8
10
16
2
9
9
3
11
21
|
| |
|
|
X = (20x4)
1 | 4 | 5 | 4
| 1 | 4 | 5 | 3
| 1 | 4 | 9 | 8
| 1 | 5 | 8 | 7
| 1 | 5 | 5 | 9
| 1 | 8 | 10 | 8
| 1 | 9 | 7 | 13
| 1 | 5 | 14 | 14
| 1 | 14 | 6 | 12
| 1 | 9 | 9 | 9
| 1 | 6 | 12 | 7
| 1 | 9 | 5 | 7
| 1 | 11 | 11 | 19
| 1 | 8 | 6 | 14
| 1 | 9 | 12 | 14
| 1 | 12 | 7 | 17
| 1 | 12 | 9 | 5
| 1 | 5 | 8 | 10
| 1 | 12 | 10 | 7
| 1 | 17 | 22 | 23
|
| |
|
|
X' = (4x20)
1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1
| 4 | 4 | 4 | 5 | 5 | 8 | 9 | 5 | 14 | 9 | 6 | 9 | 11 | 8 | 9 | 12 | 12 | 5 | 12 | 17
| 5 | 5 | 9 | 8 | 5 | 10 | 7 | 14 | 6 | 9 | 12 | 5 | 11 | 6 | 12 | 7 | 9 | 8 | 10 | 22
| 4 | 3 | 8 | 7 | 9 | 8 | 13 | 14 | 12 | 9 | 7 | 7 | 19 | 14 | 14 | 17 | 5 | 10 | 7 | 23
|
| |
|
|
II. Regression Calculations
yi = alpha + b1 xi,1 + b2 xi,2 + b3 xi,3 + ui
The q.c.e. basic equation in matrix form is:
y = Xb + e
where y (dependent variable) is (nx1) or (20x1)
X (independent vars) is (nxk) or (20x4)
b (betas) is (kx1) or (4x1)
e (errors) is (nx1) or (20x1)
Minimizing sum or squared errors using calculus results in the OLS eqn: b=(X'X)-1.X'y
To minimize the sum of squared errors of
a k dimensional line that describes the relationship
between the k independent variables and y we
find the set of slopes (betas) that minimizes
Σi=1 to nei2
Re-written in linear algebra we seek to min e'e
Rearranging the regression model equation, we get e = y - Xb
So e'e = (y-Xb)'(y-Xb) = y'y - 2b'X'y + b'X'Xb (see Judge et al (1985) p14 )
Differentiating by b we get 0 = - 2X'y + X'Xb -> 2X'Xb=2X'y
Rearranging, dividing both sides by 2 -> b = X'X-1X'y
So to obtain the elements of the (kx1) vector b we need the elements
of the (kxk) matrix X'X-1 and of the (kx1) matrix X'y.
Caclulating X'y is easy (see (1) below) but X'X-1 requires
first calculation of X'X then finding cofactors -- see (4) -- and
the deteminant - see (3) - in order to invert.
(1) X'y Matrix (4x1)
(2) X'X Matrix (4x4)
20
| 168
| 180
| 210
|
168
| 1674
| 1639
| 1981
|
180
| 1639
| 1930
| 2128
|
210
| 1981
| 2128
| 2716
|
(3) Determinant Det(X'X)≡|X'X| i.e. the determinant of matrix of X'X
Det(X'X) = 340453596
Det(X'X) = 20*1674*1930*2716 - 20*1674*2128*2128 - 20*1639*1639*2716
... + 20*1639*2128*1981 + 20*1981*1639*2128 - 20*1981*1930*1981
... - 168*168*1930*2716 + 168*168*2128*2128 + 168*1639*180*2716
... - 168*1639*2128*210 - 168*1981*180*2128 + 168*1981*1930*210
... + 180*168*1639*2716 - 180*168*2128*1981 - 180*1674*180*2716
... + 180*1674*2128*210 + 180*1981*180*1981 - 180*1981*1639*210
... - 210*168*1639*2128 + 210*168*1930*1981 + 210*1674*180*2128
... - 210*1674*1930*210 - 210*1639*180*1981 + 210*1639*1639*210
(4) Cofactors(X'X) i.e. cofactor matrix of X 'X (4x4)
142959642-6932268-6122802-1200066
-69322682035320-265020-740880
-6122802-2650201744036-699748
-1200066-740880-6997481306780
| | | | | | | | | | | | | | | |
(5) Adj(X'X) i.e. adjugate matrix of X'X, this is just the transpose of the cofactor matrix. (4x4) For a symmetric matrix, will be same as cofactor matrix.
142959642-6932268-6122802-1200066
-69322682035320-265020-740880
-6122802-2650201744036-699748
-1200066-740880-6997481306780
| | | | | | | | | | | | | | | |
(6) Inverse Matrix, inv(X'X)≡(X'X)-1 = adj(X'X)/|X'X| = adj(X'X)/340453596 (4x4)
0.4199-0.02036-0.01798-0.003525
-0.020360.005978-0.000778-0.002176
-0.01798-0.0007780.005123-0.002055
-0.003525-0.002176-0.0020550.003838
| | | | | | | | | | | | | | | |
(7) Beta Matrix (β)
b = [X'X-1].[X'y] , this is (4x1).
Finally we can calculate b through matrix multiplication.
Betas
alpha | -0.2555
| β 1 | 0.5607
| β 2 | 0.1531
| β 3 | 0.1779
|
| =
| X'X-1
0.4199 | -0.02036 | -0.01798 | -0.003525
| -0.02036 | 0.005978 | -0.000778 | -0.002176
| -0.01798 | -0.000778 | 0.005123 | -0.002055
| -0.003525 | -0.002176 | -0.002055 | 0.003838
|
| X
| X'y
|
Yhat1= + -0.2555x1 + 0.5607x4 + 0.1531x5 + 0.1779x4 = 3.4643
Yhat2= + -0.2555x1 + 0.5607x4 + 0.1531x5 + 0.1779x3 = 3.2865
Yhat3= + -0.2555x1 + 0.5607x4 + 0.1531x9 + 0.1779x8 = 4.7881
Yhat4= + -0.2555x1 + 0.5607x5 + 0.1531x8 + 0.1779x7 = 5.0179
Yhat5= + -0.2555x1 + 0.5607x5 + 0.1531x5 + 0.1779x9 = 4.9144
Yhat6= + -0.2555x1 + 0.5607x8 + 0.1531x10 + 0.1779x8 = 7.1841
Yhat7= + -0.2555x1 + 0.5607x9 + 0.1531x7 + 0.1779x13 = 8.1749
Yhat8= + -0.2555x1 + 0.5607x5 + 0.1531x14 + 0.1779x14 = 7.1814
Yhat9= + -0.2555x1 + 0.5607x14 + 0.1531x6 + 0.1779x12 = 10.6477
Yhat10= + -0.2555x1 + 0.5607x9 + 0.1531x9 + 0.1779x9 = 7.7697
Yhat11= + -0.2555x1 + 0.5607x6 + 0.1531x12 + 0.1779x7 = 6.1910
Yhat12= + -0.2555x1 + 0.5607x9 + 0.1531x5 + 0.1779x7 = 6.8016
Yhat13= + -0.2555x1 + 0.5607x11 + 0.1531x11 + 0.1779x19 = 10.9759
Yhat14= + -0.2555x1 + 0.5607x8 + 0.1531x6 + 0.1779x14 = 7.6390
Yhat15= + -0.2555x1 + 0.5607x9 + 0.1531x12 + 0.1779x14 = 9.1182
Yhat16= + -0.2555x1 + 0.5607x12 + 0.1531x7 + 0.1779x17 = 10.5686
Yhat17= + -0.2555x1 + 0.5607x12 + 0.1531x9 + 0.1779x5 = 8.7405
Yhat18= + -0.2555x1 + 0.5607x5 + 0.1531x8 + 0.1779x10 = 5.5515
Yhat19= + -0.2555x1 + 0.5607x12 + 0.1531x10 + 0.1779x7 = 9.2493
Yhat20= + -0.2555x1 + 0.5607x17 + 0.1531x22 + 0.1779x23 = 16.7357
ESS
=(3.464 - 7.700)^2
=(3.286 - 7.700)^2
=(4.788 - 7.700)^2
=(5.018 - 7.700)^2
=(4.914 - 7.700)^2
=(7.184 - 7.700)^2
=(8.175 - 7.700)^2
=(7.181 - 7.700)^2
=(10.65 - 7.700)^2
=(7.770 - 7.700)^2
=(6.191 - 7.700)^2
=(6.802 - 7.700)^2
=(10.98 - 7.700)^2
=(7.639 - 7.700)^2
=(9.118 - 7.700)^2
=(10.57 - 7.700)^2
=(8.740 - 7.700)^2
=(5.551 - 7.700)^2
=(9.249 - 7.700)^2
=(16.74 - 7.700)^2
=184.109503872592
REPORT
obs
| calculation of yhatobs yhatobs = Σβixi,obs
| yhatobs
| a
| yobs (data)
| Meany
| (yobs - yhatobs)2
| (yhatobs - My)2
| (yobs - My)2
| a
| eobs=yobs-yhatobs
| eobs2
|
1
| Yhat1 = Σβixi,1-0.2555x1 = 0.5607x4 + 0.1531x5 + 0.1779x4 | = 3.464 | 5 | 7.700 | 2.358 | 17.94 | 7.290 | e1 = 5 - 3.464 = 1.536 | 2.358
|
2
| Yhat2 = Σβixi,2 + -0.2555x1 = 0.5607x4 + 0.1531x5 + 0.1779x3 | = 3.286 | 4 | 7.700 | 0.5091 | 19.48 | 13.69 | e2 = 4 - 3.286 = 0.7135 | 0.5091
|
3
| Yhat3 = Σβixi,3 + -0.2555x1 = 0.5607x4 + 0.1531x9 + 0.1779x8 | = 4.788 | 9 | 7.700 | 17.74 | 8.479 | 1.690 | e3 = 9 - 4.788 = 4.212 | 17.74
|
4
| Yhat4 = Σβixi,4 + -0.2555x1 = 0.5607x5 + 0.1531x8 + 0.1779x7 | = 5.018 | 3 | 7.700 | 4.072 | 7.194 | 22.09 | e4 = 3 - 5.018 = -2.018 | 4.072
|
5
| Yhat5 = Σβixi,5 + -0.2555x1 = 0.5607x5 + 0.1531x5 + 0.1779x9 | = 4.914 | 5 | 7.700 | 0.007335 | 7.760 | 7.290 | e5 = 5 - 4.914 = 0.08565 | 0.007335
|
6
| Yhat6 = Σβixi,6 + -0.2555x1 = 0.5607x8 + 0.1531x10 + 0.1779x8 | = 7.184 | 5 | 7.700 | 4.770 | 0.2661 | 7.290 | e6 = 5 - 7.184 = -2.184 | 4.770
|
7
| Yhat7 = Σβixi,7 + -0.2555x1 = 0.5607x9 + 0.1531x7 + 0.1779x13 | = 8.175 | 8 | 7.700 | 0.03060 | 0.2256 | 0.09000 | e7 = 8 - 8.175 = -0.1749 | 0.03060
|
8
| Yhat8 = Σβixi,8 + -0.2555x1 = 0.5607x5 + 0.1531x14 + 0.1779x14 | = 7.181 | 5 | 7.700 | 4.758 | 0.2690 | 7.290 | e8 = 5 - 7.181 = -2.181 | 4.758
|
9
| Yhat9 = Σβixi,9 + -0.2555x1 = 0.5607x14 + 0.1531x6 + 0.1779x12 | = 10.65 | 5 | 7.700 | 31.90 | 8.689 | 7.290 | e9 = 5 - 10.65 = -5.648 | 31.90
|
10
| Yhat10 = Σβixi,10 + -0.2555x1 = 0.5607x9 + 0.1531x9 + 0.1779x9 | = 7.770 | 12 | 7.700 | 17.90 | 0.004853 | 18.49 | e10 = 12 - 7.770 = 4.230 | 17.90
|
11
| Yhat11 = Σβixi,11 + -0.2555x1 = 0.5607x6 + 0.1531x12 + 0.1779x7 | = 6.191 | 4 | 7.700 | 4.800 | 2.277 | 13.69 | e11 = 4 - 6.191 = -2.191 | 4.800
|
12
| Yhat12 = Σβixi,12 + -0.2555x1 = 0.5607x9 + 0.1531x5 + 0.1779x7 | = 6.802 | 8 | 7.700 | 1.436 | 0.8071 | 0.09000 | e12 = 8 - 6.802 = 1.198 | 1.436
|
13
| Yhat13 = Σβixi,13 + -0.2555x1 = 0.5607x11 + 0.1531x11 + 0.1779x19 | = 10.98 | 10 | 7.700 | 0.9523 | 10.73 | 5.290 | e13 = 10 - 10.98 = -0.9759 | 0.9523
|
14
| Yhat14 = Σβixi,14 + -0.2555x1 = 0.5607x8 + 0.1531x6 + 0.1779x14 | = 7.639 | 16 | 7.700 | 69.91 | 0.003727 | 68.89 | e14 = 16 - 7.639 = 8.361 | 69.91
|
15
| Yhat15 = Σβixi,15 + -0.2555x1 = 0.5607x9 + 0.1531x12 + 0.1779x14 | = 9.118 | 2 | 7.700 | 50.67 | 2.011 | 32.49 | e15 = 2 - 9.118 = -7.118 | 50.67
|
16
| Yhat16 = Σβixi,16 + -0.2555x1 = 0.5607x12 + 0.1531x7 + 0.1779x17 | = 10.57 | 9 | 7.700 | 2.460 | 8.229 | 1.690 | e16 = 9 - 10.57 = -1.569 | 2.460
|
17
| Yhat17 = Σβixi,17 + -0.2555x1 = 0.5607x12 + 0.1531x9 + 0.1779x5 | = 8.740 | 9 | 7.700 | 0.06736 | 1.083 | 1.690 | e17 = 9 - 8.740 = 0.2595 | 0.06736
|
18
| Yhat18 = Σβixi,18 + -0.2555x1 = 0.5607x5 + 0.1531x8 + 0.1779x10 | = 5.551 | 3 | 7.700 | 6.510 | 4.616 | 22.09 | e18 = 3 - 5.551 = -2.551 | 6.510
|
19
| Yhat19 = Σβixi,19 + -0.2555x1 = 0.5607x12 + 0.1531x10 + 0.1779x7 | = 9.249 | 11 | 7.700 | 3.065 | 2.400 | 10.89 | e19 = 11 - 9.249 = 1.751 | 3.065
|
20
| Yhat20 = Σβixi,20 + -0.2555x1 = 0.5607x17 + 0.1531x22 + 0.1779x23 | = 16.74 | 21 | 7.700 | 18.18 | 81.64 | 176.9 | e20 = 21 - 16.74 = 4.264 | 18.18 | | | | | | RSS = Σ(yobs - yhatobs)2 | ESS = Σ(yhatobs - My)2 | TSS = Σ(yobs - My)2 | | e'e=Σeobs2 | sum-> | | | | | 242.1 | 184.1 | 426.2 | | 242.1
|
(11) Betas and their t-Stats
from the covar matrix of b=σ2(X'X)-1
the var(βi) = σ2vii where vii is the ith diag element of X'X-1
where σ2 = e'e / n-k (k=num of ind vars plus 1 for the intercept if present).
and where vii is the ith diag element of X'X-1
Std(βi) = sqr root of Var(βi)
TStat(βi) = βi / Std(βi)
Estimate of σ2 = 15.130656007963
| Coef value
| StD(β)
| tStat(β)
|
alpha =
| 0.4199 * 154 + -0.02036 * 1499 + -0.01798 * 1547 + -0.003525 * 1866 = -0.2555
| (15.13 * 0.4199)1/2 = 2.521
| -0.2555 / 2.521 = -0.1014
|
β1 =
| -0.02036 * 154 + 0.005978 * 1499 + -0.000778 * 1547 + -0.002176 * 1866 = 0.5607
| (15.13 * 0.005978)1/2 = 0.3008
| 0.5607 / 0.3008 = 1.864
|
β2 =
| -0.01798 * 154 + -0.000778 * 1499 + 0.005123 * 1547 + -0.002055 * 1866 = 0.1531
| (15.13 * 0.005123)1/2 = 0.2784
| 0.1531 / 0.2784 = 0.5499
|
β3 =
| -0.003525 * 154 + -0.002176 * 1499 + -0.002055 * 1547 + 0.003838 * 1866 = 0.1779
| (15.13 * 0.003838)1/2 = 0.2410
| 0.1779 / 0.2410 = 0.7380
|
(12) Table of Outputs: yobs = | alpha | + | β1 | X obs,1 + | β2 | X obs,2 + | β3 | X obs,3 + | eobs | | -0.2555 | | 0.5607 | | 0.1531 | | 0.1779 | | | | (-0.1014) | | (1.864) | | (0.5499) | | (0.7380) | | <- tstats | r2 = 0.431979 | adj r2 = 0.325475 |
(13) RSS = Sum{y - y_hat }^2 = 242.090496127408
TSS = Sum{y - y_avg }^2 = 426.2
ESS(a)= Sum{y_hat - y_avg }^2 = 184.109503872592
we use the ESSb (below) cuz smthn wrng w ESS when no intercept.
ESS(b)= TSS-RSS 184.109503872592
note: TSS = ESS + RSS
(14) r2 = ESS/TSS = 0.431979126871404
(15) adjusted r2 = ESS/TSS = 0.325475213159792
(16) F-stat = [ESS/(k-1)] / [RSS/(n-k)] = 4.05599298483157
see Johnston(1984) p186
F measures the joint significance of
all explanatory variables.
Alternatively: F-stat = r2/(k-1) / (1-r2)/(n-k)
(17) Durbin-Watson Statistic (DW or d) measures autocorrelation.
DW = 2.63094204405623
________________________________________________________
Note, RSS, ESS and TSS stand for ...
Residual Sum of Squares (RSS),
Explained Sum of Squares (ESS), and
Total Sum of Squares (TSS).
However ESS is sometimes referred to as the Regression Sum of Squares.
and RSS is sometimes referred to as the Sum of Squares Rresidual.
Note, an alternative way of calculating TSS, ESS is...
TSS = y'Ay
ESS = bv'Xv'Ay where bv' Xv' are b' & X' wo intercept row col
RSS = TSS-ESS
Bibliography
J. Johnston (1984) Econometric Methods, 3rd ed.
Judge et al (1985) The Theory and Practice of Econometrics 2rd ed, Wiley, New York.
Donald F. Morrison (1990) Multivariate Statistical Methods, 3rd edition, McGraw Hill, New York.
A. H. Studenmund (1997) Using Econometrics: A Practical Guide, 3rd edition. Addison-Wesley, Reading.
|
|