Stationary solution: existence and uniqueness

• A stationary solution exists and is unique if and only if \[ \phi(z) \neq 0, \quad \text{for all complex } z \text{ with } |z| = 1 \]

• The unit circle: the region in \(z \in \mathbb{C}\) defined by \(|z|=1\)

• Stationary solution: \[ X_t = \theta(B) / \phi(B) Z_t = \psi(B) Z_t = \sum_{j=-\infty}^{\infty} \psi_j Z_{t-j} \]

Causality: \(\phi(z)\) has no zeros inside the unit circle

• An ARMA\((p, q)\) process is causal: if there exist \(\psi_0, \psi_1, \ldots\) \[ \sum_{j=0}^{\infty} |\psi_j| < \infty, \quad \text{and} \] \[ X_t = \sum_{j=0}^{\infty}\psi_j Z_{t-j}, \quad \text{for all } t \]

• Theorem (equivalent condition of causaility): \[ \phi(z) = 1 - \phi_1 z - \cdots - \phi_p z^p \neq 0, \quad \text{for all } |z| \leq 1 \]

• Example: ARMA\((1, 1)\) \(X_t - \phi X_{t-1} = Z_t + \theta Z_{t-1}\) \[ 1-\phi z = 0 \Longrightarrow \text{ only zero } z = 1/\phi \] So \(|z| = 1/|\phi| > 1\), i.e., \(|\phi| < 1\) is equivalent of causality

How do we get \(\psi_j\)’s?

• Letting \(\theta_0 = 1\) and matching coefficients of \(z^j\) based on \[ 1 + \theta_1 z + \cdots \theta_q z^q = (1 - \phi_1 z - \cdots \phi_p z^p)(\psi_0 + \psi_1 z + \cdots), \] gives \[ \theta_j \mathbf{1}_{[j \leq q]} = \psi_j - \sum_{j=1}^p \phi_k \psi_{j-k}, \quad j = 0, 1, \ldots \]

• Example: causal ARMA\((1, 1)\)
  \[\begin{align*} 1 & = \psi_0\\ \theta & = \psi_1 - \phi \psi_0 \Longrightarrow \psi_1 = \theta + \psi\\ 0 & = \psi_j - \phi \psi_{j-1} \text{ for } j \geq 2 \Longrightarrow \psi_j = \phi \psi_{j-1} \end{align*}\]

Therefore, \[ \psi_0 = 1, \quad \psi_j = \phi^{j-1}(\theta + \psi) \text{ for } j \geq 1 \]

Invertibility: \(\theta(z)\) has no zeros inside the unit circle

• An ARMA\((p, q)\) process is invertible: if there exist \(\pi_0, \pi_1, \ldots\) \[ \sum_{j=0}^{\infty} |\pi_j| < \infty, \quad \text{and} \] \[ Z_t = \sum_{j=0}^{\infty}\pi_j X_{t-j}, \quad \text{for all } t \]

• Theorem (equivalent condition of invertibility): \[ \theta(z) = 1 + \theta_1 z + \cdots + \theta_q z^q \neq 0, \quad \text{for all } |z| \leq 1 \]

• Example: ARMA\((1, 1)\) \(X_t - \phi X_{t-1} = Z_t + \theta Z_{t-1}\) \[ 1 + \theta z = 0 \Longrightarrow \text{ only zero } z = -1/\theta \] So \(|z| = 1/|\theta| > 1\), i.e., \(|\theta| < 1\) is equivalent of invertibility

How do we get \(\pi_j\)’s?

• Letting \(\phi_0 = -1\) and matching coefficients of \(z^j\) based on \[ 1 - \phi_1 z - \cdots \phi_p z^p = (1 + \theta_1 z + \cdots \theta_q z^q)(\pi_0 + \pi_1 z + \cdots), \] gives \[ -\phi_j \mathbf{1}_{[j \leq p]} = \pi_j + \sum_{j=1}^q \theta_k \pi_{j-k}, \quad j = 0, 1, \ldots \]

• Example: invertible ARMA\((1, 1)\)
  \[\begin{align*} 1 & = \pi_0\\ -\phi & = \pi_1 + \theta \psi_0 \Longrightarrow \pi_1 = -(\psi +\theta)\\ 0 & = \pi_j + \theta \pi_{j-1} \text{ for } j \geq 2 \Longrightarrow \pi_j = -\theta \pi_{j-1} \end{align*}\]

Therefore, \[ \pi_0 = 1, \quad \pi_j = (-1)^j \theta^{j-1}(\psi + \theta) \text{ for } j \geq 1 \]

Calculation of the ACVF

• Assume the ARMA\((p, q)\) process \(\{X_t\}\) is causal and invertible

• Method 1: If \(X_t = \sum_{j=0}^{\infty} \psi_j Z_{t-j}\), then \[ \gamma(h) = E(X_{t+h} E_t) = \sigma^2 \sum_{j=0}^{\infty} \psi_j \psi_{j + |h|} \]

• Method 2 (difference equation method): multiple the ARMA formula with \(X_t, X_{t-1}, \ldots\) and take expectation

Example: ARMA\((1, 1)\)

• Recall that for a causal ARMA\((1, 1)\), in \(X_t = \sum_{j=0}^{\infty} \psi_j Z_{t-j}\), \[ \psi_0 = 1, \quad \psi_j = \phi^{j-1}(\theta + \psi) \text{ for } j \geq 1 \]

• Lag-0 autocorrelation \[ \gamma(0) = \sigma^2 \sum_{j=0}^{\infty} \psi_j^2 = \sigma^2\left[ 1 + (\theta+\phi)^2 \sum_{j=0}^{\infty}\phi^{2j}\right] = \sigma^2\left[ 1 + \frac{(\theta+\phi)^2}{1-\phi^2} \right] \]

• Lag-1 autocorrelation \[ \gamma(1) = \sigma^2 \sum_{j=0}^{\infty} \psi_j \psi_{j+1} % = \sigma^2\left[ \theta+\phi + (\theta+\phi)^2\phi % \sum_{j=0}^{\infty}\phi^{2j}\right] = \sigma^2\left[ \theta+\phi + \frac{(\theta+\phi)^2 \phi}{1-\phi^2} \right] \]

• Lag-\(k\) autocorrelation (\(k \geq 2\)) \[ \gamma(k) = \phi^{k-1} \gamma(1), \quad k \geq 2 \]

Use the difference equation method on ARMA\((1, 1)\)

1. Multiple \(X_t - \phi X_{t-1} = Z_t + \theta Z_{t-1}\) by \(X_t\), then take expectation \[ E(X_t^2) - \phi E(X_t X_{t-1}) = E(X_t Z_t) + \theta E(X_t Z_{t-1}) \] Since \(E(X_t Z_k) = E[(\sum_{j=0}^{\infty} \psi_j Z_{t-j})Z_k] = \psi_{t-k} \sigma^2\), we have \[ \gamma(0) - \phi \gamma(1) = \sigma^2 + \theta (\theta + \phi) \sigma^2 \]

2. Multiply by \(X_{t-1}\) \[ E(X_{t-1} X_t) - \phi E(X_{t-1}^2) = E(X_{t-1} Z_t) + \theta E(X_{t-1} Z_{t-1}) \] \[ \gamma(1) - \phi \gamma(0) = 0 + \theta \sigma^2 \psi_0 = \theta \sigma^2 \]

Using the two equations from 1 and 2, we can solve \(\gamma(0), \gamma(1)\)

3. Multiply by \(X_{t-k}\), for \(k \geq 2\) \[ E(X_{t-k} X_t) - \phi E(X_{t-k} X_{t-1}) = E(X_{t-k} Z_t) + \theta E(X_{t-k} Z_{t-1}) \] \[ \gamma(k) - \phi \gamma(k-1) = 0 \Longrightarrow \gamma(k) = \phi \gamma(k-1) \]

ACF of an MA\((q)\) process

• Suppose \(\{X_t\}\) is an MA\((q)\), then \(\rho(h) = 0\) for all \(h > q\)
• By asymptotic normality \[ \hat{\rho}(q + 1) \stackrel{\cdot}{\sim} \textrm{N}\left(0, \frac{w_{q+1, q+1}}{n}\right) \] and Bartlett \[\begin{align*} w_{q+1, q+1} & = \sum_{k=1}^{\infty} \left[ \rho(k+q+1) + \rho(k-q-1) - 2 \rho(k+1) \rho(q) \right]^2 \\ & = \sum_{k=1}^{\infty}\rho(k-q-1)^2\\ & = 1 + 2 \sum_{j=1}^q \rho(j)^2 \end{align*}\]

Test for an MA\((q)\): from the ACF

1. Hypotheses \[ H_0: \{X_t\} \sim \textrm{MA}(q) \quad \longleftrightarrow \quad H_A: \text{Not } H_0 \]

2. Test statistic \[ Z = \frac{\hat{\rho}(q + 1) - 0}{\sqrt{\frac{1 + 2 \sum_{j=1}^q \hat{\rho}(j)^2}{n}}} \]

3. Reject \(H_0\) if \(|Z| \geq z_{\alpha/2}\)

• Note: under the null hypothesis, we use the sample ACF plot with bounds \(\pm 1.96 \times \sqrt{\frac{1 + 2 \sum_{j=1}^q \hat{\rho}(j)^2}{n}}\) to check if \(\hat{\rho}(h)\) for all \(h \geq q+1\) are inside the bounds. But this may have some multiple testing problems.

Partial autocorrelation function (PACF)

• We define the partial autocorrelation function (PACF) of an ARMA process as the function \(\alpha(\cdot)\) \[ \alpha(0) = 1, \quad \alpha(h) = \phi_{hh}, \text{ for } h \geq 1 \] Here, \(\phi_{hh}\) is the last entry of \[ \boldsymbol\phi_h = \boldsymbol\Gamma_h^{-1} \boldsymbol\gamma_h, \quad \text{where } \boldsymbol\Gamma_h = [\gamma(i-j)]_{i,j=1}^h, \ \boldsymbol\gamma_h = [\gamma(1), \ldots, \gamma(h)]' \]

• Sample PACF \(\hat{\alpha}(\cdot)\): change all \(\gamma(\cdot)\) above to \(\hat{\gamma}(\cdot)\)

• Recall: in DL algorithm \(\hat{X}_{n+1} = \phi_{n1}X_n + \cdots + \phi_{nn} X_1\), \[ \phi_{nn} = \alpha(n), \quad \text{PACF at lag }n \]

PACF property

• \(\phi_{nn}\) is the correlation between the prediction errors \[ \alpha(n) = \text{Corr} \left( X_n - P(X_n | X_1, \ldots, X_{n-1}), X_0 - P(X_0 | X_1, \ldots, X_{n-1})\right) \]

• Theorem: A stationary series is AR\((p)\) if and only if \[ \alpha(h) = 0 \text{ for all } h > p \]

• If \(\{X_t\}\) is an AR\((p)\), then we have asymptotic normality \[ \hat{\alpha}(h) \stackrel{\cdot}{\sim} \textrm{N}\left(0, \frac{1}{n}\right), \quad \text{for all } h > p \]

Test for an AR\((p)\): from the PACF

1. Hypotheses \[ H_0: \{X_t\} \sim \textrm{AR}(p) \quad \longleftrightarrow \quad H_A: \text{Not } H_0 \]

2. Test statistic \[ Z = \frac{\hat{\alpha}(p + 1) - 0}{\sqrt{\frac{1}{n}}} \]

3. Reject \(H_0\) if \(|Z| \geq z_{\alpha/2}\)

• Note: under the null hypothesis, we use the sample PACF plot with bounds \(\pm 1.96 / \sqrt{n}\) to check if \(\hat{\alpha}(h)\) for all \(h \geq p+1\) are inside the bounds. But this may have some multiple testing problems.