Yule-Walker equations

• $\left\{X_t\right\}$ is a casual AR$\left(p\right)$ process $$X_t={\varphi }_1{X}_{t-1}+\cdots +{\varphi }_p{X}_{t-p}+Z_t$$

• Multiplying each side by $X_t,X_{t-1},\dots ,X_{t-p}$, respectively, and taking expectation, we got the Yule-Walker equations $${\sigma }^2=\gamma \left(0\right)-{\varphi }_1\gamma \left(1\right)-\cdots {\varphi }_p\gamma \left(p\right)$$ $$\underset{{\Gamma }_p}{\underset{}{\left[\begin{array}{cccc}\gamma \left(0\right)& \gamma \left(1\right)& \cdots & \gamma (p-1)\\ \gamma \left(1\right)& \gamma \left(0\right)& \cdots & \gamma (p-2)\\ ⋮& ⋮& ⋮& ⋮\\ \gamma (p-1)& \gamma (p-2)& \cdots & \gamma \left(0\right)\end{array}\right]}}\underset{\varphi }{\underset{}{\left[\begin{array}{c}{\varphi }_1\\ {\varphi }_2\\ ⋮\\ {\varphi }_p\end{array}\right]}}=\underset{{\gamma }_p}{\underset{}{\left[\begin{array}{c}\gamma \left(1\right)\\ \gamma \left(2\right)\\ ⋮\\ \gamma \left(p\right)\end{array}\right]}}$$

• Vector representation $${\Gamma }_p\varphi ={\gamma }_p,{\sigma }^2=\gamma \left(0\right)-{\varphi }^{\prime }{\gamma }_p$$

Yule-Walker estimator and its properties

• Yule-Walker estimators $\hat{\varphi }=({\hat{\varphi }}_1,\cdots ,{\hat{\varphi }}_p)$ are obtained by solving the hatted version of the Yule-Walker equations $$\hat{\varphi }={\hat{\Gamma }}_p^{-1}{\hat{\gamma }}_p,{\hat{\sigma }}^2=\hat{\gamma }\left(0\right)-{\hat{\varphi }}^{\prime }{\hat{\gamma }}_p$$

• The fitted model is causal and ${\hat{\sigma }}^2\ge 0$ $$X_t={\hat{\varphi }}_1{X}_{t-1}+\cdots +{\hat{\varphi }}_p{X}_{t-p}+Z_t,Z_t\sim \text{WN}(0,{\hat{\sigma }}^2)$$

• Asymptotic normality $$\hat{\varphi }\stackrel{\cdot }{\sim }\text{N}(\varphi ,\frac{{\sigma }^2{\Gamma }_p^{-1}}{n})$$

Yule-Walker estimator is a moment estimator: because it is obtained by equating theoretical and sample moments

• Usually moment estimators have much higher variance than MLE

• But Yule-Walker estimators of AR$\left(p\right)$ process have the same asymptotic distribution as the MLE

• Moment estimators can fail for MA$\left(q\right)$ and general ARMA

  • For example, MA$\left(1\right)$: $X_t=Z_t+\theta Z_{t+1}$ with $\left\{Z_t\right\}\sim \text{WN}(0,{\sigma }^2)$. $$\gamma \left(0\right)=(1+{\theta }^2){\sigma }^2,\gamma \left(1\right)=\theta {\sigma }^2⟹\rho \left(1\right)=\frac{\theta }{1+{\theta }^2}$$ Moment estimator of $\theta$ is obtained by solving $$\hat{\rho }\left(1\right)=\frac{\hat{\theta }}{1+{\hat{\theta }}^2}⟹\hat{\theta }=\frac{1\pm \sqrt{1-4\hat{\rho }(1{)}^2}}{2\hat{\rho }\left(1\right)}$$ This can yield complex $\hat{\theta }$ if $|\hat{\rho }\left(1\right)|>1/2$, which can happen if $\rho \left(1\right)=1/2$, i.e., $\theta =1$

Innovations algorithm: estimate MA coefficients

• Fitted innovations MA$\left(m\right)$ model $$X_t=Z_t+{\hat{\theta }}_{m1}{Z}_{t-1}+\cdots +\cdots +{\hat{\theta }}_{mm}{Z}_{t-m},\left\{Z_t\right\}\sim \text{WN}(0,{\hat{v}}_m)$$ where ${\hat{\theta }}_m$ and ${\hat{v}}_m$ are from the innovations algorithm with ACVF replaced by the sample ACVF

• For a MA$\left(q\right)$ process, the innovations algorithm estimator ${\hat{\theta }}_q=({\hat{\theta }}_{q1},\dots ,{\hat{\theta }}_{qq}{)}^{\prime }$ is NOT consistent for $({\theta }_1,\dots ,{\theta }_q{)}^{\prime }$

• Choice of $m$: increase $m$ until the vector $({\hat{\theta }}_{m1},\dots ,{\hat{\theta }}_{mq}{)}^{\prime }$ stabilizes

Likelihood function of a Gaussian time series

• Suppose $\left\{X_t\right\}$ is a Gaussian time series with mean zero

• Assume that covariance matrix ${\Gamma }_n=E\left(X_n{X}_n^{\prime }\right)$ is nonsingular

• One-step predictors using innovations algorithm: ${\hat{X}}_1=0$ and $${\hat{X}}_{j+1}=P_j{X}_{j+1}$$ with MSE $v_j=E{(X_{j+1}-{\hat{X}}_{j+1})}^2$

  • Example: AR$\left(1\right)$ $${\hat{X}}_j=\{\begin{array}{cc}0,& j=1\\ \varphi {\hat{X}}_{j-1}& j\ge 2\end{array},v_j=\{\begin{array}{cc}\frac{{\sigma }^2}{1-{\varphi }^2},& j=0\\ {\sigma }^2& j\ge 1\end{array}$$

• Likelihood function $$\begin{array}{cc}L& \propto {\left|{\Gamma }_n\right|}^{-1/2}\exp (-\frac{1}{2}{X}_n^{\prime }{\Gamma }_n^{-1}{X}_n)\\ & ={(v_0{v}_1\cdots v_{n-1})}^{-1/2}\exp [-\frac{1}{2}\sum _{j=1}^n\frac{(X_j-{\hat{X}}_j{)}^2}{v_{j-1}}]\end{array}$$

Maximum likelihood estimation of ARMA$(p,q)$

• Innovations MSE $v_j={\sigma }^2{r}_j$, where $r_j$ depends on $\varphi$ and $\theta$

• Maximizing the likelihood is equivalent to minimizing $$-2\log L(\varphi ,\theta ,{\sigma }^2)=n\log \left({\sigma }^2\right)+\sum _{j=1}^n\log \left(r_{j-1}\right)+\frac{S(\varphi ,\theta )}{{\sigma }^2},$$ where $$S(\varphi ,\theta )=\sum _{j=1}^n\frac{(X_j-{\hat{X}}_j{)}^2}{r_{j-1}}$$

• MLE ${\hat{\sigma }}^2$ can be expressed with MLE $\hat{\varphi },\hat{\theta }$ $${\hat{\sigma }}^2=\frac{S(\hat{\varphi },\hat{\theta })}{n}$$

• MLE $\hat{\varphi },\hat{\theta }$ are obtained by minimizing $$\log \left[\frac{S(\varphi ,\theta )}{n}\right]+\frac{1}{n}\sum _{j=1}^n\log \left(r_{j-1}\right)$$ Not depend on ${\sigma }^2$!

Asymptotic normality of MLE

• When $n$ is large, for a causal and invertible ARMA$(p,q)$ process, $$\left[\begin{array}{c}\hat{\varphi }\\ \hat{\theta }\end{array}\right]\stackrel{\cdot }{\sim }{\text{N}}_{p+1}(\left[\begin{array}{c}\hat{\varphi }\\ \hat{\theta }\end{array}\right],\frac{V}{n})$$

• For an AR$\left(p\right)$ process, MLE has the same asymptotic distribution as the Yule-Walker estimator $$V={\sigma }^2{\Gamma }_p^{-1}⟹\hat{\varphi }\stackrel{\cdot }{\sim }\text{N}(\varphi ,\frac{{\sigma }^2{\Gamma }_p^{-1}}{n})$$

Examples of $V$

• AR$\left(1\right)$ $$V=1-{\varphi }_1^2$$

• AR$\left(2\right)$ $$V=\left[\begin{array}{cc}1-{\varphi }_2^2& -{\varphi }_1(1+{\varphi }_2)\\ -{\varphi }_1(1+{\varphi }_2)& 1-{\varphi }_2^2\end{array}\right]$$

• MA$\left(1\right)$ $$V=1-{\theta }_1^2$$

• MA$\left(2\right)$ $$V=\left[\begin{array}{cc}1-{\theta }_2^2& {\theta }_1(1-{\theta }_2)\\ {\theta }_1(1-{\theta }_2)& 1-{\theta }_2^2\end{array}\right]$$

• ARMA$(1,1)$ $$V=\frac{1+\varphi \theta }{(\varphi +\theta {)}^2}\left[\begin{array}{cc}(1-{\varphi }^2)(1+\varphi \theta )& -(1-{\theta }^2)(1-{\varphi }^2)\\ -(1-{\theta }^2)(1-{\varphi }^2)& (1-{\varphi }^2)(1+\varphi \theta )\end{array}\right]$$

Order selection

• Why? Harm of using too large $p,q$ to fit models:

  • Large errors arising from parameter estimation of the model
  • Large MSEs of forecasts

• FPE: only for AR$\left(p\right)$ processes $$\text{FPE}={\hat{\sigma }}^2\frac{n+p}{n-p}$$

• AIC: for ARMA$(p,q)$; approximate Kullback-Leibler discrepancy of the fitted model and the true model, a penalized likelihood method $$\text{AIC}=-2\log \left(\hat{L}\right)+2(p+q+1)$$

• AICC: for ARMA$(p,q)$; a bias-corrected version of AIC, a penalized likelihood method $$\text{AICC}=-2\log \left(\hat{L}\right)+2(p+q+1)\cdot \frac{n}{n-p-q-2}$$

Residuals and rescaled residuals

• Residuals of an ARMA$(p,q)$ process $${\hat{W}}_t=\frac{X_t-{\hat{X}}_t(\hat{\varphi },\hat{\theta })}{\sqrt{r_{t-1}(\hat{\varphi },\hat{\theta })}},t=1,\dots ,n$$
  • Residuals $\left\{{\hat{W}}_t\right\}$ should be similar to white noises $\left\{Z_t\right\}$
• Rescaled residuals $${\hat{R}}_t=\frac{{\hat{W}}_t}{\hat{\sigma }},\hat{\sigma }=\sqrt{\frac{{\sum }_{t=1}^n{\hat{W}}_t^2}{n}}$$
  • Residuals residuals should be approximately $\text{WN}(0,1)$

Residual diagnostics

1. Plot $\left\{{\hat{R}}_t\right\}$ and look for patterns

2. Compute the sample ACF of $\left\{{\hat{R}}_t\right\}$
   • It should be close to the $\text{WN}(0,1)$ sample ACF

3. Apply Chapter 1 tests for IID noises

References

• Brockwell, Peter J. and Davis, Richard A. (2016), Introduction to Time Series and Forecasting, Third Edition. New York: Springer