Stationary solution: existence and uniqueness

• A stationary solution exists and is unique if and only if $$\varphi \left(z\right)\ne 0,\text{for all complex}z\text{with}|z|=1$$

• The unit circle: the region in $z\in C$ defined by $|z|=1$

• Stationary solution: $$X_t=\theta \left(B\right)/\varphi \left(B\right)Z_t=\psi \left(B\right)Z_t=\sum _{j=-\infty }^{\infty }{\psi }_j{Z}_{t-j}$$

Causality: $\varphi \left(z\right)$ has no zeros inside the unit circle

• An ARMA$(p,q)$ process is causal: if there exist ${\psi }_0,{\psi }_1,\dots$ $$\sum _{j=0}^{\infty }|{\psi }_j|<\infty ,\text{and}$$ $$X_t=\sum _{j=0}^{\infty }{\psi }_j{Z}_{t-j},\text{for all}t$$

• Theorem (equivalent condition of causaility): $$\varphi \left(z\right)=1-{\varphi }_{1}z-\cdots -{\varphi }_p{z}^p\ne 0,\text{for all}|z|\le 1$$

• Example: ARMA$(1,1)$ $X_t-\varphi X_{t-1}=Z_t+\theta Z_{t-1}$ $$1-\varphi z=0⟹\text{only zero}z=1/\varphi$$ So $|z|=1/|\varphi |>1$, i.e., $|\varphi |<1$ is equivalent of causality

How do we get ${\psi }_j$’s?

• Letting ${\theta }_0=1$ and matching coefficients of $z^j$ based on $$1+{\theta }_{1}z+\cdots {\theta }_q{z}^q=(1-{\varphi }_{1}z-\cdots {\varphi }_p{z}^p)({\psi }_0+{\psi }_{1}z+\cdots ),$$ gives $${\theta }_j{1}_{[j\le q]}={\psi }_j-\sum _{j=1}^p{\varphi }_k{\psi }_{j-k},j=0,1,\dots$$

• Example: causal ARMA$(1,1)$
  $$\begin{array}{cc}1& ={\psi }_0\\ \theta & ={\psi }_1-\varphi {\psi }_0⟹{\psi }_1=\theta +\psi \\ 0& ={\psi }_j-\varphi {\psi }_{j-1}\text{for}j\ge 2⟹{\psi }_j=\varphi {\psi }_{j-1}\end{array}$$

Therefore, $${\psi }_0=1,{\psi }_j={\varphi }^{j-1}(\theta +\psi )\text{for}j\ge 1$$

Invertibility: $\theta \left(z\right)$ has no zeros inside the unit circle

• An ARMA$(p,q)$ process is invertible: if there exist ${\pi }_0,{\pi }_1,\dots$ $$\sum _{j=0}^{\infty }|{\pi }_j|<\infty ,\text{and}$$ $$Z_t=\sum _{j=0}^{\infty }{\pi }_j{X}_{t-j},\text{for all}t$$

• Theorem (equivalent condition of invertibility): $$\theta \left(z\right)=1+{\theta }_{1}z+\cdots +{\theta }_q{z}^q\ne 0,\text{for all}|z|\le 1$$

• Example: ARMA$(1,1)$ $X_t-\varphi X_{t-1}=Z_t+\theta Z_{t-1}$ $$1+\theta z=0⟹\text{only zero}z=-1/\theta$$ So $|z|=1/|\theta |>1$, i.e., $|\theta |<1$ is equivalent of invertibility

How do we get ${\pi }_j$’s?

• Letting ${\varphi }_0=-1$ and matching coefficients of $z^j$ based on $$1-{\varphi }_{1}z-\cdots {\varphi }_p{z}^p=(1+{\theta }_{1}z+\cdots {\theta }_q{z}^q)({\pi }_0+{\pi }_{1}z+\cdots ),$$ gives $$-{\varphi }_j{1}_{[j\le p]}={\pi }_j+\sum _{j=1}^q{\theta }_k{\pi }_{j-k},j=0,1,\dots$$

• Example: invertible ARMA$(1,1)$
  $$\begin{array}{cc}1& ={\pi }_0\\ -\varphi & ={\pi }_1+\theta {\psi }_0⟹{\pi }_1=-(\psi +\theta )\\ 0& ={\pi }_j+\theta {\pi }_{j-1}\text{for}j\ge 2⟹{\pi }_j=-\theta {\pi }_{j-1}\end{array}$$

Therefore, $${\pi }_0=1,{\pi }_j=(-1{)}^j{\theta }^{j-1}(\psi +\theta )\text{for}j\ge 1$$

Calculation of the ACVF

• Assume the ARMA$(p,q)$ process $\left\{X_t\right\}$ is causal and invertible

• Method 1: If $X_t={\sum }_{j=0}^{\infty }{\psi }_j{Z}_{t-j}$, then $$\gamma \left(h\right)=E\left(X_{t+h}{E}_t\right)={\sigma }^2\sum _{j=0}^{\infty }{\psi }_j{\psi }_{j+|h|}$$

• Method 2 (difference equation method): multiple the ARMA formula with $X_t,X_{t-1},\dots$ and take expectation

Example: ARMA$(1,1)$

• Recall that for a causal ARMA$(1,1)$, in $X_t={\sum }_{j=0}^{\infty }{\psi }_j{Z}_{t-j}$, $${\psi }_0=1,{\psi }_j={\varphi }^{j-1}(\theta +\psi )\text{for}j\ge 1$$

• Lag-0 autocorrelation $$\gamma \left(0\right)={\sigma }^2\sum _{j=0}^{\infty }{\psi }_j^2={\sigma }^2[1+(\theta +\varphi {)}^2\sum _{j=0}^{\infty }{\varphi }^{2j}]={\sigma }^2[1+\frac{(\theta +\varphi {)}^2}{1-{\varphi }^2}]$$

• Lag-1 autocorrelation $$\gamma \left(1\right)={\sigma }^2\sum _{j=0}^{\infty }{\psi }_j{\psi }_{j+1}={\sigma }^2[\theta +\varphi +\frac{(\theta +\varphi {)}^2\varphi }{1-{\varphi }^2}]$$

• Lag-$k$ autocorrelation ($k\ge 2$) $$\gamma \left(k\right)={\varphi }^{k-1}\gamma \left(1\right),k\ge 2$$

Use the difference equation method on ARMA$(1,1)$

1. Multiple $X_t-\varphi X_{t-1}=Z_t+\theta Z_{t-1}$ by $X_t$, then take expectation $$E\left(X_t^2\right)-\varphi E\left(X_t{X}_{t-1}\right)=E\left(X_t{Z}_t\right)+\theta E\left(X_t{Z}_{t-1}\right)$$ Since $E\left(X_t{Z}_k\right)=E\left[\right({\sum }_{j=0}^{\infty }{\psi }_j{Z}_{t-j}\left)Z_k\right]={\psi }_{t-k}{\sigma }^2$, we have $$\gamma \left(0\right)-\varphi \gamma \left(1\right)={\sigma }^2+\theta (\theta +\varphi ){\sigma }^2$$

2. Multiply by $X_{t-1}$ $$E\left(X_{t-1}{X}_t\right)-\varphi E\left(X_{t-1}^2\right)=E\left(X_{t-1}{Z}_t\right)+\theta E\left(X_{t-1}{Z}_{t-1}\right)$$ $$\gamma \left(1\right)-\varphi \gamma \left(0\right)=0+\theta {\sigma }^2{\psi }_0=\theta {\sigma }^2$$

Using the two equations from 1 and 2, we can solve $\gamma \left(0\right),\gamma \left(1\right)$

3. Multiply by $X_{t-k}$, for $k\ge 2$ $$E\left(X_{t-k}{X}_t\right)-\varphi E\left(X_{t-k}{X}_{t-1}\right)=E\left(X_{t-k}{Z}_t\right)+\theta E\left(X_{t-k}{Z}_{t-1}\right)$$ $$\gamma \left(k\right)-\varphi \gamma (k-1)=0⟹\gamma \left(k\right)=\varphi \gamma (k-1)$$

ACF of an MA$\left(q\right)$ process

• Suppose $\left\{X_t\right\}$ is an MA$\left(q\right)$, then $\rho \left(h\right)=0$ for all $h>q$
• By asymptotic normality $$\hat{\rho }(q+1)\stackrel{\cdot }{\sim }\text{N}(0,\frac{w_{q+1,q+1}}{n})$$ and Bartlett $$\begin{array}{cc}{w}_{q+1,q+1}& =\sum _{k=1}^{\infty }{\left[\rho \right(k+q+1)+\rho (k-q-1)-2\rho (k+1\left)\rho \right(q\left)\right]}^2\\ & =\sum _{k=1}^{\infty }\rho (k-q-1{)}^2\\ & =1+2\sum _{j=1}^q\rho (j{)}^2\end{array}$$

Test for an MA$\left(q\right)$: from the ACF

1. Hypotheses $$H_0:\left\{X_t\right\}\sim \text{MA}\left(q\right)⟷H_A:\text{Not}{H}_0$$

2. Test statistic $$Z=\frac{\hat{\rho }(q+1)-0}{\sqrt{\frac{1+2{\sum }_{j=1}^q\hat{\rho }(j{)}^2}{n}}}$$

3. Reject $H_0$ if $|Z|\ge z_{\alpha /2}$

• Note: under the null hypothesis, we use the sample ACF plot with bounds $\pm 1.96\times \sqrt{\frac{1+2{\sum }_{j=1}^q\hat{\rho }(j{)}^2}{n}}$ to check if $\hat{\rho }\left(h\right)$ for all $h\ge q+1$ are inside the bounds. But this may have some multiple testing problems.

Partial autocorrelation function (PACF)

• We define the partial autocorrelation function (PACF) of an ARMA process as the function $\alpha (\cdot )$ $$\alpha \left(0\right)=1,\alpha \left(h\right)={\varphi }_{hh},\text{for}h\ge 1$$ Here, ${\varphi }_{hh}$ is the last entry of $${\varphi }_h={\Gamma }_h^{-1}{\gamma }_h,\text{where}{\Gamma }_h=\left[\gamma \right(i-j){]}_{i,j=1}^h,{\gamma }_h=[\gamma \left(1\right),\dots ,\gamma \left(h\right){]}^{\prime }$$

• Sample PACF $\hat{\alpha }(\cdot )$: change all $\gamma (\cdot )$ above to $\hat{\gamma }(\cdot )$

• Recall: in DL algorithm ${\hat{X}}_{n+1}={\varphi }_{n1}{X}_n+\cdots +{\varphi }_{nn}{X}_1$, $${\varphi }_{nn}=\alpha \left(n\right),\text{PACF at lag}n$$

PACF property

• ${\varphi }_{nn}$ is the correlation between the prediction errors $$\alpha \left(n\right)=\text{Corr}(X_n-P(X_n|X_1,\dots ,X_{n-1}),X_0-P(X_0|X_1,\dots ,X_{n-1}\left)\right)$$

• Theorem: A stationary series is AR$\left(p\right)$ if and only if $$\alpha \left(h\right)=0\text{for all}h>p$$

• If $\left\{X_t\right\}$ is an AR$\left(p\right)$, then we have asymptotic normality $$\hat{\alpha }\left(h\right)\stackrel{\cdot }{\sim }\text{N}(0,\frac{1}{n}),\text{for all}h>p$$

Test for an AR$\left(p\right)$: from the PACF

1. Hypotheses $$H_0:\left\{X_t\right\}\sim \text{AR}\left(p\right)⟷H_A:\text{Not}{H}_0$$

2. Test statistic $$Z=\frac{\hat{\alpha }(p+1)-0}{\sqrt{\frac{1}{n}}}$$

3. Reject $H_0$ if $|Z|\ge z_{\alpha /2}$

• Note: under the null hypothesis, we use the sample PACF plot with bounds $\pm 1.96/\sqrt{n}$ to check if $\hat{\alpha }\left(h\right)$ for all $h\ge p+1$ are inside the bounds. But this may have some multiple testing problems.