For the pdf slides, click here
Objective of time series models
Seasonal adjustment: recognize seasonal components and remove them to study long-term trends
Separate (or filter) noise from signals
Prediction
Test hypotheses
Predicting one series from observations of another
A general approach to time series modeling
- Plot the series and check main features:
- Trend
- Seasonality
- Any sharp changes
- Outliers
- Remove trend and seasonal components to get stationary residuals
- May need data transformation first
- Choose a model to fit the residuals
Stationary Models and Autocorrelation Function
Definitions: stationary
- Series {Xt} has
- Mean function μX(t)=E(Xt) and
- Covariance function γX(r,s)=Cov(Xr,Xs)
- {Xt} is (weakly) stationary if
- μX(t) does not depend on t
- γX(t+h,t) does not depend on t, for each h
- (Weakly) stationary is defined based on the first and second order properties of a series
- {Xt} is strictly stationary if (X1,…,Xn) and
(X1+h,…,Xn+h) have the same joint distributions for all
integers h and n>0
- If {Xt} is strictly stationary, and E(X2t)<∞ for all t, then {Xt} is weakly stationary
- Weakly stationary does not imply strictly stationary
Definitions: autocovariance and autorrelation
{Xt} is a stationary time series
Autocovariance function (ACVF) of at lag h
γX(h)=Cov(Xt+h,Xt)
- Autocorrelation function (ACF) of at lag h
ρX(h)=γX(h)γX(0)=Cor(Xt+h,Xt)
- Note that γ(h)=γ(−h) and ρ(h)=ρ(−h)
Definitions: sample ACVF and sample ACF
x1,…,xn are observations of a time series with sample mean ˉx
Sample autocovariance function: for −n<h<n, ˆγ(h)=1nn−|h|∑t=1(xt+|h|−ˉx)(xt−ˉx)
- Use n in the denominator ensures the sample covariance matrix ˆΓn=[ˆγ(i−j)]ni,j=1 is nonnegative definite
- Sample autocorrelation function: for −n<h<n,
ˆρ(h)=ˆγ(h)ˆγ(0)
- Sample correlation matrix ˆRn=[ˆρ(i−j)]ni,j=1 is also nonnegative definite
Examples of Simple Time Series Models
iid noise and white noise
- White noise: uncorrelated, with zero mean and variance σ2
{Xt}∼WN(0,σ2)
- IID(0,σ2) sequences is WN(0,σ2), but not conversely
Binary process and random walk
Binary process: an example of iid noise {Xt,t=1,2,…} P(Xt=1)=p,P(Xt=−1)=1−p
Random walk: {St,t=0,1,2,…}, with S0=0 and iid noise {Xt} St=X1+X2+⋯+Xt, for t=1,2,…
{St} is a simple symmetric random walk if {Xt} is a binary process with p=0.5
Random walk is not stationary: if Var(Xt)=σ2, then γS(t+h,t)=tσ2 depends on t
First-order moving average, MA(1) process
Let {Zt}∼WN(0,σ2), and θ∈R, then {Xt} is a MA(1) process: Xt=Zt+θZt−1,t=0,±1,…
ACVF: does not depend on t, stationary γX(t+h,t)={(1+θ2)σ2, if h=0,θσ2, if h=±1,0, if |h|>1.
ACF: ρX(h)={1, if h=0,θ/(1+θ2), if h=±1,0, if |h|>1.
First-order autoregression, AR(1) process
Let {Zt}∼WN(0,σ2), and |ϕ|<1, then {Xt} is a AR(1) process: Xt=ϕXt−1+Zt,t=0,±1,…
ACVF: γX(h)=σ21−ϕ2⋅ϕ|h|
ACF: ρX(h)=ϕ|h|
Estimate and Eliminate Trend and Seasonal Components
Classcial decomposition
Observation {Xt} can be decomposed into
- a (slowly changing) trend component mt,
- a seasonal component st with period d and ∑dj=1sj=0,
a zero-mean series Yt Xt=mt+st+Yt
Method 1: estimate st first, then mt, and hope the noise component Yt is stationary (to model)
Method 2: differencing
Method 3: trend and seasonality can be estimated together in a regression, whose design matrix contains both polynomial and harmonic terms
Trend Component Only
Estimate trend: polynomial regression fitting
Observation {Xt} can be decomposed into a trend component mt and a zero-mean series Yt: Xt=mt+Yt
- Least squares polynomial regression mt=a0+a1t+⋯+aptp
Estimate trend: smoothing with a finite MA filter
Linear filter ˆmt=∞∑j=−∞ajXt−j
Two-sided moving average filter, with q∈N Wt=∑qj=−qXt−j2q+1
Wt≈mt for q+1≤t≤n−q, if Xt only has the trend component mt but not seasonality st, and mt is approximately linear in t
Wt is a low-pass filter: remove the rapidly fluctuating (high frequency) component Yt, and let the slowly varying component mt pass
Estimate trend: exponential smoothing
For any fixed α∈[0,1], the one-sided MA ˆmt:t=1,…,n defined by recursions ˆmt={X1, if t=1αXt+(1−α)ˆmt−1, if t=2,…,n
- Equivalently, ˆmt=t−2∑j=0α(1−α)jXt−j+(1−α)t−1X1
Eliminate trend by differencing
Backward shift operator BXt=Xt−1
- Lag-1 difference operator
∇Xt=Xt−Xt−1=(1−B)Xt
- If ∇ is applied to a linear trend function mt=c0+c1t, then ∇mt=c1
- Powers of operators B and ∇:
Bj(Xt)=Xt−j,∇j(Xt)=∇[∇j−1(Xt)] with ∇0(Xt)=Xt
- ∇k reduces a polynomial trend of degree k to a constant ∇k(k∑j=0cjtj)=k!ck
Also with the Seasonal Component
Estimate seasonal component: harmonic regression
Observation {Xt} can be decomposed into a seasonal component st and a zero-mean series Yt: Xt=st+Yt
st: a periodic function of t with period d, i.e., st−d=st
Harmonic regression: a sum of harmonics (or sine waves)
st=a0+k∑j=1[ajcos(λjt)+bjsin(λjt)]
Unknown (regression) parameters: aj,bj
- Specified parameters:
- Number of harmonics: k
- Frequencies λj, each being some integer multiple of 2πd
- Sometimes λj are instead specified through Fourier indices fj=n⋅jd
Estimate trend and seasonal components
Estimate ˆmt: use a MA filter chosen to elimate the seasonality
- If d is odd, let d=2q
- If d is even, let d=2q and ˆmt=(0.5xt−q+xt−q+1+⋯+xt+q−1+0.5xt+q)/d
Estimate ˆst: for each k=1,…,d
- Compute the average wk=avgj(xk+jd−ˆmk+jd)
- To ensure ∑dk=1sk=0, let ˆsk=wk−ˉw, where ˉw=∑dk=1wk/d
Re-estimate ˆmt: based on the deseasonalized data dt=xt−ˆst
Eliminate trend and seasonal components: differencing
Lag-d differencing ∇dXt=Xt−Xt−d=(1−Bd)Xt
- Note: the operators ∇d and ∇d=(1−B)d are different
Apply ∇d to Xt=mt+st+Yt ∇dXt=mt−mt−d+Yt−Yt−d
- Then the trend mt−mt−d can be eliminated using methods discussed before, e.g., applying a power of the operator ∇
Test Whether Estimated Noises are IID
Test series {Y1,…,Yn} for iid: sample ACF based
Test name | Test statistic | Distribution under H0 |
---|---|---|
Sample ACF | ˆρ(h), for all h∈N | N(0,1/n) |
Portmanteau | Q=n∑hj=1ˆρ2(j) | χ2(h) |
Under H0, about 95% of the sample ACFs should fall between ±1.96√n
- The Portmanteau test has some refinements
- Ljung and Box QLB=n(n+2)∑jˆρ2(j)/(n−j)
- McLeod and Li QML=n(n+2)∑jˆρ2WW(j)/(n−j), where ˆρ2WW(h) is the sample ACF of squared data
Test series {Y1,…,Yn} for iid: also detect trends
Test name | Test statistic | Distribution under H0 |
---|---|---|
Turning point | T: number of turning points | N(μT,σ2T) |
Difference-sign | S: number of i that yi>yi−1 | N(μS,σ2S) |
- Time i is a turning point, if yi−yi−1 and yi+1−yi have
flipped signs
- μT=2(n−2)/3≈2/3
- A large positive (or negative) value of S−μS indicates increasing (or
decreasing) trend
- μS=(n−1)/2≈1/2
Test series {Y1,…,Yn} for iid: other methods
- Fitting an AR model
- Using Yule-Walker algorithm and choose order using AICC statistic
- If the selected order is zero, then the series is white noise
Normal qq plot: check of normality
A general strategy is to check all above mentioned tests, and proceed with caution if any of them suggests not iid
References
- Brockwell, Peter J. and Davis, Richard A. (2016), Introduction to Time Series and Forecasting, Third Edition. New York: Springer