Stationary Models and Autocorrelation Function
- Examples of Simple Time Series Models
Estimate and Eliminate Trend and Seasonal Components
- Trend Component Only
- Also with the Seasonal Component
Test Whether Estimated Noises are IID

For the pdf slides, click here

Objective of time series models

Seasonal adjustment: recognize seasonal components and remove them to study long-term trends
Separate (or filter) noise from signals
Prediction
Test hypotheses
Predicting one series from observations of another

A general approach to time series modeling

Plot the series and check main features:
- Trend
- Seasonality
- Any sharp changes
- Outliers
Remove trend and seasonal components to get stationary residuals
- May need data transformation first
Choose a model to fit the residuals

Stationary Models and Autocorrelation Function

Definitions: stationary

Series ${X_{t}}$ has
- Mean function $μ_{X} (t) = E (X_{t})$ and
- Covariance function $γ_{X} (r, s) = Cov (X_{r}, X_{s})$
${X_{t}}$ is (weakly) stationary if
- $μ_{X} (t)$ does not depend on $t$
- $γ_{X} (t + h, t)$ does not depend on $t$ , for each $h$
- (Weakly) stationary is defined based on the first and second order properties of a series
${X_{t}}$ is strictly stationary if $(X_{1}, \dots, X_{n})$ and $(X_{1 + h}, \dots, X_{n + h})$ have the same joint distributions for all integers $h$ and $n > 0$
- If ${X_{t}}$ is strictly stationary, and $E (X_{t}^{2}) < \infty$ for all $t$ , then ${X_{t}}$ is weakly stationary
- Weakly stationary does not imply strictly stationary

Definitions: autocovariance and autorrelation

${X_{t}}$ is a stationary time series
Autocovariance function (ACVF) of at lag $h$

$γ_{X} (h) = Cov (X_{t + h}, X_{t})$

Autocorrelation function (ACF) of at lag $h$

$ρ_{X} (h) = \frac{γ_{X} (h)}{γ_{X} (0)} = Cor (X_{t + h}, X_{t})$

Note that $γ (h) = γ (- h)$ and $ρ (h) = ρ (- h)$

Definitions: sample ACVF and sample ACF

$x_{1}, \dots, x_{n}$ are observations of a time series with sample mean $\bar{x}$

Sample autocovariance function: for $- n < h < n$ , $\hat{γ} (h) = \frac{1}{n} \sum_{t = 1}^{n - | h |} (x_{t + | h |} - \bar{x}) (x_{t} - \bar{x})$
- Use $n$ in the denominator ensures the sample covariance matrix ${\hat{Γ}}_{n} = {[\hat{γ} (i - j)]}_{i, j = 1}^{n}$ is nonnegative definite
Sample autocorrelation function: for $- n < h < n$ , $\hat{ρ} (h) = \frac{\hat{γ} (h)}{\hat{γ} (0)}$
- Sample correlation matrix ${\hat{R}}_{n} = {[\hat{ρ} (i - j)]}_{i, j = 1}^{n}$ is also nonnegative definite

Examples of Simple Time Series Models

iid noise and white noise

White noise: uncorrelated, with zero mean and variance $σ^{2}$

${X_{t}} \sim WN (0, σ^{2})$

IID $(0, σ^{2})$ sequences is $WN (0, σ^{2})$ , but not conversely

Binary process and random walk

Binary process: an example of iid noise ${X_{t}, t = 1, 2, \dots}$ $P (X_{t} = 1) = p, P (X_{t} = - 1) = 1 - p$
Random walk: ${S_{t}, t = 0, 1, 2, \dots}$ , with $S_{0} = 0$ and iid noise ${X_{t}}$ $S_{t} = X_{1} + X_{2} + \dots + X_{t}, for t = 1, 2, \dots$
- ${S_{t}}$ is a simple symmetric random walk if ${X_{t}}$ is a binary process with $p = 0.5$
- Random walk is not stationary: if $Var (X_{t}) = σ^{2}$ , then $γ_{S} (t + h, t) = t σ^{2}$ depends on $t$

First-order moving average, MA $(1)$ process

Let ${Z_{t}} \sim WN (0, σ^{2})$ , and $θ \in R$ , then ${X_{t}}$ is a MA $(1)$ process: $X_{t} = Z_{t} + θ Z_{t - 1}, t = 0, \pm 1, \dots$

ACVF: does not depend on $t$ , stationary $γ_{X} (t + h, t) = {\begin{cases} (1 + θ^{2}) σ^{2}, & if h = 0, \\ θ σ^{2}, & if h = \pm 1, \\ 0, & if | h | > 1. \end{cases}$
ACF: $ρ_{X} (h) = {\begin{cases} 1, & if h = 0, \\ θ / (1 + θ^{2}), & if h = \pm 1, \\ 0, & if | h | > 1. \end{cases}$

First-order autoregression, AR $(1)$ process

Let ${Z_{t}} \sim WN (0, σ^{2})$ , and $| ϕ | < 1$ , then ${X_{t}}$ is a AR $(1)$ process: $X_{t} = ϕ X_{t - 1} + Z_{t}, t = 0, \pm 1, \dots$

ACVF: $γ_{X} (h) = \frac{σ^{2}}{1 - ϕ^{2}} \cdot ϕ^{| h |}$
ACF: $ρ_{X} (h) = ϕ^{| h |}$

Estimate and Eliminate Trend and Seasonal Components

Classcial decomposition

Observation ${X_{t}}$ can be decomposed into

a (slowly changing) trend component $m_{t}$ ,
a seasonal component $s_{t}$ with period $d$ and $\sum_{j = 1}^{d} s_{j} = 0$ ,
a zero-mean series $Y_{t}$ $X_{t} = m_{t} + s_{t} + Y_{t}$
Method 1: estimate $s_{t}$ first, then $m_{t}$ , and hope the noise component $Y_{t}$ is stationary (to model)
Method 2: differencing
Method 3: trend and seasonality can be estimated together in a regression, whose design matrix contains both polynomial and harmonic terms

Trend Component Only

Estimate trend: polynomial regression fitting

Observation ${X_{t}}$ can be decomposed into a trend component $m_{t}$ and a zero-mean series $Y_{t}$ : $X_{t} = m_{t} + Y_{t}$

Least squares polynomial regression $m_{t} = a_{0} + a_{1} t + \dots + a_{p} t^{p}$

Estimate trend: smoothing with a finite MA filter

Linear filter ${\hat{m}}_{t} = \sum_{j = - \infty}^{\infty} a_{j} X_{t - j}$
Two-sided moving average filter, with $q \in N$ $W_{t} = \frac{\sum_{j = - q}^{q} X_{t - j}}{2 q + 1}$
- $W_{t} \approx m_{t}$ for $q + 1 \leq t \leq n - q$ , if $X_{t}$ only has the trend component $m_{t}$ but not seasonality $s_{t}$ , and $m_{t}$ is approximately linear in $t$
- $W_{t}$ is a low-pass filter: remove the rapidly fluctuating (high frequency) component $Y_{t}$ , and let the slowly varying component $m_{t}$ pass

Estimate trend: exponential smoothing

For any fixed $α \in [0, 1]$ , the one-sided MA ${\hat{m}}_{t} : t = 1, \dots, n$ defined by recursions ${\hat{m}}_{t} = {\begin{cases} X_{1}, & if t = 1 \\ α X_{t} + (1 - α) {\hat{m}}_{t - 1}, & if t = 2, \dots, n \end{cases}$

Equivalently, ${\hat{m}}_{t} = \sum_{j = 0}^{t - 2} α (1 - α)^{j} X_{t - j} + (1 - α)^{t - 1} X_{1}$

Eliminate trend by differencing

Backward shift operator $B X_{t} = X_{t - 1}$
Lag-1 difference operator $\nabla X_{t} = X_{t} - X_{t - 1} = (1 - B) X_{t}$
- If $\nabla$ is applied to a linear trend function $m_{t} = c_{0} + c_{1} t$ , then $\nabla m_{t} = c_{1}$
Powers of operators $B$ and $\nabla$ : $B^{j} (X_{t}) = X_{t - j}, \nabla^{j} (X_{t}) = \nabla [\nabla^{j - 1} (X_{t})] with \nabla^{0} (X_{t}) = X_{t}$
- $\nabla^{k}$ reduces a polynomial trend of degree $k$ to a constant $\nabla^{k} (\sum_{j = 0}^{k} c_{j} t^{j}) = k! c_{k}$

Also with the Seasonal Component

Estimate seasonal component: harmonic regression

Observation ${X_{t}}$ can be decomposed into a seasonal component $s_{t}$ and a zero-mean series $Y_{t}$ : $X_{t} = s_{t} + Y_{t}$

$s_{t}$ : a periodic function of $t$ with period $d$ , i.e., $s_{t - d} = s_{t}$
Harmonic regression: a sum of harmonics (or sine waves)

$s_{t} = a_{0} + \sum_{j = 1}^{k} [a_{j} \cos (λ_{j} t) + b_{j} \sin (λ_{j} t)]$

Unknown (regression) parameters: $a_{j}, b_{j}$
Specified parameters:
- Number of harmonics: $k$
- Frequencies $λ_{j}$ , each being some integer multiple of $\frac{2 π}{d}$
- Sometimes $λ_{j}$ are instead specified through Fourier indices $f_{j} = \frac{n \cdot j}{d}$

Estimate trend and seasonal components

Estimate ${\hat{m}}_{t}$ : use a MA filter chosen to elimate the seasonality
- If $d$ is odd, let $d = 2 q$
- If $d$ is even, let $d = 2 q$ and ${\hat{m}}_{t} = (0.5 x_{t - q} + x_{t - q + 1} + \dots + x_{t + q - 1} + 0.5 x_{t + q}) / d$
Estimate ${\hat{s}}_{t}$ : for each $k = 1, \dots, d$
- Compute the average $w_{k} = {avg}_{j} (x_{k + j d} - {\hat{m}}_{k + j d})$
- To ensure $\sum_{k = 1}^{d} s_{k} = 0$ , let ${\hat{s}}_{k} = w_{k} - \bar{w}$ , where $\bar{w} = \sum_{k = 1}^{d} w_{k} / d$
Re-estimate ${\hat{m}}_{t}$ : based on the deseasonalized data $d_{t} = x_{t} - {\hat{s}}_{t}$

Eliminate trend and seasonal components: differencing

Lag- $d$ differencing $\nabla_{d} X_{t} = X_{t} - X_{t - d} = (1 - B^{d}) X_{t}$
- Note: the operators $\nabla_{d}$ and $\nabla^{d} = (1 - B)^{d}$ are different
Apply $\nabla_{d}$ to $X_{t} = m_{t} + s_{t} + Y_{t}$ $\nabla_{d} X_{t} = m_{t} - m_{t - d} + Y_{t} - Y_{t - d}$
- Then the trend $m_{t} - m_{t - d}$ can be eliminated using methods discussed before, e.g., applying a power of the operator $\nabla$

Test Whether Estimated Noises are IID

Test series ${Y_{1}, \dots, Y_{n}}$ for iid: sample ACF based

Test name	Test statistic	Distribution under $H_{0}$
Sample ACF	$\hat{ρ} (h)$ , for all $h \in N$	$N (0, 1 / n)$
Portmanteau	$Q = n \sum_{j = 1}^{h} {\hat{ρ}}^{2} (j)$	$χ^{2} (h)$

Under $H_{0}$ , about 95% of the sample ACFs should fall between $\pm 1.96 \sqrt{n}$
The Portmanteau test has some refinements
- Ljung and Box $Q_{L B} = n (n + 2) \sum_{j} {\hat{ρ}}^{2} (j) / (n - j)$
- McLeod and Li $Q_{M L} = n (n + 2) \sum_{j} {\hat{ρ}}_{W W}^{2} (j) / (n - j)$ , where ${\hat{ρ}}_{W W}^{2} (h)$ is the sample ACF of squared data

Test series ${Y_{1}, \dots, Y_{n}}$ for iid: also detect trends

Test name	Test statistic	Distribution under $H_{0}$
Turning point	$T$ : number of turning points	$N (μ_{T}, σ_{T}^{2})$
Difference-sign	$S$ : number of $i$ that $y_{i} > y_{i - 1}$	$N (μ_{S}, σ_{S}^{2})$

Time $i$ is a turning point, if $y_{i} - y_{i - 1}$ and $y_{i + 1} - y_{i}$ have flipped signs
- $μ_{T} = 2 (n - 2) / 3 \approx 2 / 3$
A large positive (or negative) value of $S - μ_{S}$ indicates increasing (or decreasing) trend
- $μ_{S} = (n - 1) / 2 \approx 1 / 2$

Test series ${Y_{1}, \dots, Y_{n}}$ for iid: other methods

Fitting an AR model
- Using Yule-Walker algorithm and choose order using AICC statistic
- If the selected order is zero, then the series is white noise
Normal qq plot: check of normality
A general strategy is to check all above mentioned tests, and proceed with caution if any of them suggests not iid

References

Brockwell, Peter J. and Davis, Richard A. (2016), Introduction to Time Series and Forecasting, Third Edition. New York: Springer

Book Notes: Introduction to Time Series and Forecasting -- Ch1 Introduction