6.4 ARMA Models (Under Construction)

Autocovariance and Autocorrelation of AR($p$) Models¶

Deriving Weights for Causal Process¶

We’ve seen that the autoregressive operator $\phi(\mathbb{B})$ is extremely useful for determining stationarity. In this section, we will explore another useful application of $\phi(\mathbb{B})$, namely converting a finite AR model into a so called MA($\infty$) consisting of an infinite series of noise (or shock) terms. This in turn will help us understand the autocovariance of AR processes and derive confidence intervals for predictions.

We’ve already seen one example of converting to an infinite series of noise for AR(1) models

How might we go about finding weights, call them $\psi_j$’s with associated operator $\psi(\mathbb{B})$, for higher order AR($p$) processes? In other words, we want to find $\psi$ weights to satisfy the equation

where we have defined $\psi_0\stackrel{\triangle}{=}1$.Given that an AR model is defined as

we may combine Eqs. (2) and (3) to arrive at

Given that the left-hand side of Eq. (4) does not have any backshifted terms, we conclude that the coefficients of each backshift operator must be zero, i.e.

Fortunately, statsmodels performs the operations in Eq. (5) for us using the property impulse_response^[4]. Continuing with the example from above, add the following line to the code:

print(f"psi weights 0-9: {ar2.impulse_response()[:10]}")

Do the results agree with Eq. (5) applied in the problem above?

$\psi$ Weight Representation¶

Brockwell & Davis (1991) chapter 3.3 provides three broad methods to calculate the theoretical autocovariance and autocorrelation of AR models (and ARMA models in general). We will focus on the first here.

The first method relies on representing the model in the form of Eq. (2). $\gamma(h)$ is then calculated in the same fashion as Eq. (5)

where we have used the fact that the noise is iid resulting in zero covariance for different noise terms. Unfortunately, AR($p$) models with $p\geq2$ do not have the same convenient geometric series representation seen with an AR(1) model. Nevertheless, Eq. (6) provides an elegant method for calculating the autocovariance, and hence the autocorrelation, of any stationary AR process—or, as we shall see, any stationary ARMA process. The fact that the $\psi$ weights decay exponentially results in it being possible to truncate Eq. (6) after say, thirty terms, with relatively little loss in accuracy.

Recursion Relations¶

The second and third methods in Brockwell & Davis (1991) rely on using recursion relations relating higher $h$ values to lower ones. We will not go into great depth regarding this method, but it is instructive to see how it might be applied to an AR(2) model. Let us begin with the AR(2) process $x_t = \phi_1 x_{t-1} + \phi_2 x_{t-2} + w_t$. We can multiply through by $x_{t-h}$ and take the expectation:

Provided that our AR(2) process is causal (stationary), we can rewrite the final term in Eq. (7) as

Combining Eqs. (7) and (8) we arrive at the desired recursion relation

Without additional information, we cannot further simplify in Eq. (9). However, dividing through by $\gamma(0)$ generates a recursion relation for the autocorrelation

We know that $\rho(0)=1$, and we can evaluate $\rho(1)$ using Eq. (9):

Thus we can derive $\rho(2)$ as

and continue extending the recursion relation to higher values of $h$.