2.4 Properties of Variance and Covariance

We will make heavy use of both variance and covariance (in particular the autocovariance) throughout the book. This chapter presents a refresher on these topics laying the foundation for the forms used in time series analysis discussed in the next chapter.

Operators¶

There are several operators we will encounter in this book. Values such as mean, variance, and covariance can all be cast in the operator formalism. In later chapters we will introduce the backshift and Fourier operators. So what is an operator?

An operator $\mathbb{O}$ is defined as a rule that maps a member of a set to another member. Thus, an operator could just be a function such as 5, defined as multiplying by 5 (i.e. $f(x)=5\,x$)^[1]. However, the most common operators we will use map one function to another function. Two such operators are differentiation $\frac{d}{dx}$ and integration $\int dx$.

Linear operators are of particular interest. An operator is a linear operator if it fulfills the following two conditions:

1. $\mathbb{O}\, a F(x) = a\,\mathbb{O}\, F(x)$ for any constant $a$

2. $\mathbb{O}\, (F(x) + G(y)) = \mathbb{O}\,F(x) + \mathbb{O}\,G(y)$

We may write the two conditions more succinctly as

for any constants $a$ and $b$.

Going forward, we will assume that all operators discussed are linear unless otherwise noted.

Expectation¶

The most important operator in statistics and data science is the expectation operator $\mathbb{E}[F(x)]$, usually first encountered in the context of the arithmetic mean.

Expectation is defined as:

where $P(x)$ is either the probability function of the random variable $x$ such that

(discrete $x$) or the probability density function such that

(continuous $x$).

In general, we will not explicitly reference both the discrete and continuous cases in this book. Instead, we will use the notation $\mathbb{E}$ or one of the two methods in Eq. (3) with an understanding that a reference to either one implicitly refers to both unless specified otherwise.

Linearity of Expectation¶

An important property of expectation is that it is a linear operator. We can demonstrate this fact by proving that $\mathit{\mathbb{E}[a\,F(x) + b\,G(y)]} = \mathit{a\,\mathbb{E}[F(x)] + b\,\mathbb{E}[G(y)]}$ as follows:

where in the fourth line we have used the definition of marginal probabilities: $P(x) = \int P(x, y)\, dy$ and $P(y) = \int P(x, y)\, dx$.

Moments¶

We have not yet specified the identity of $F(x)$. By far, the most commonly used functions are powers of $x$, i.e.

The expectation value of Eq. (9) is the $n\text{th}$ moment of $P(x)$:

where we are usually interested in cases of $n=1-4$^[2]:

1. The first moment is the arithmetic mean denoted as $\mu_x$.

2. The second moment relates to the variance, or how widely spread the distribution is.

3. The third moment relates to the skew, or how symmetric the function is.

4. The fourth moment relates to the kurtosis, or the “fatness” of the tails.

Finite Moments¶

An important theorem states that if the $k\text{th}$ moment $\mathbb{E}[x^{k}]$ is finite, then all moments $j<k$ must also be finite. As a corollary, if $\mathbb{E}[x^{k}]$ is infinite, all moments $m>k$ must also be infinite.

Proof: Let $\mathbb{E}[x^{k}]$ be finite and $j<k$ ($\forall$ is read as “for all”)

where we have used the fact that $\int_{-1}^{1} 1 P(x)\, dx \leq 1$, with equality only occurring if the entire probability mass is contained in the interval $[-1,1]$.

Variance¶

The variance of a random variable $X$ is defined as

and is often denoted as $\sigma_x^2$. The variance gives us a measure of how widely the distribution is spread about the mean. In practice, we more commonly make use of the standard deviation $\sigma_x$, which is simply the square root of the variance.

As written, the variance is slightly different than our definition of the second moment.

By exploiting the linearity of expectation we can express Eq. (17) using the first and second moments exclusively:

Covariance¶

The covariance of two random variables $X, Y$ is defined as

Covariance is also written as $\sigma_{x, y}$. Note that $\text{Cov}(X, X) = \mathbb{V}(X)$.

Unlike variance, which is never negative, covariance can be negative, zero, or positive.Following the same logic used in Eq. (18), we can also express the covariance as

Covariance gives us a measure of how much random variables change in tandem with one another.

Variance of Sums of Random Variables¶

One of the most fundamental aspects of time series analysis is understanding the variance of the sum of random variables. Let us begin with the variance of a sum of two random variables, $X$ and $Y$.

Variance of Multiple Random Variables¶

The last line of Eq. (21) can be rewritten as

or, renaming the variables as $X_{0}$ and $X_{1}$

The above suggests (though does not prove) that the variance of a sum of variables is the sum of all covariance combinations. For random variables $X_{0}, X_{1}, X_{2}, \ldots,X_{n-1}$:

where the last term sums $i$ to $n-2$ and $j$ to $n-1$. Eq. (24) can be proven in the same manner as Eq. (21), though the algebra gets rather intricate. We present a more direct proof in the following problem.

Variance of Independent Variables¶

From Eq. (24) we can see that if all variables have zero covariance (most commonly due to independence), the variance of a sum of variables is the sum of the variances of each variable

While it is very tempting to simply assume that Eq. (25) holds, in real life we must justify its use, either from theoretical analysis and/or empirical evidence.

It should also be recalled that knowing that random variables have zero covariance does not inherently prove independence. As a simple counterexample, consider a random variable $X$ with zero mean and third moment and let $Y=X^2$. An example might be $X \sim N(0,1)$ and $Y = X^2 \sim \chi^2_1$. Clearly, $X$ and $Y$ and highly dependent; for example, knowing that $Y>4$ tells us $|X|>2$. Nevertheless, they still have zero covariance:

Variance-Covariance Inequality¶

Warning: The math in this section can get rather heavy. Feel free to skip this section if you’re having difficultly. While the material below does add to the overall understanding of future material, it is not absolutely necessary.

Arithmetic Inequality¶

From Eq. (21) we see that

By substituting in $-Y$, we arrive at

As variances are by definition non-negative, by combining the above two equations we arrive at the inequality $2\,|\text{Cov}(X, Y)| \leq \mathbb{V}(X) + \mathbb{V}(Y)$. We can express this in terms of the arithmetic mean of the variances:

While Eq. (28) is true, we will soon see that we can get an even tighter bound on the inequality.

Cauchy-Schwarz Inequality¶

In order to better understand the relation between variance and covariance, we must first define the Cauchy-Schwarz inequality, a valuable inequality from linear algebra. In words, it states that the square of the inner product of two vectors must always be less than or equal to the norm of the first vector squared times the second vector’s norm squared.

or, recognizing that vector norms are always non-negative

Eq.s (29) and (30) will be equalities if and only if $\mathbf{u}$ and $\mathbf{v}$ can be expressed as scalar multiples of one another (i.e. lie on the same line).

If $\mathbf{u}$ and/or $\mathbf{v}$ is the zero vector, the inequality is trivially true, let us prove the inequality when neither is:

Functions as Infinite Dimensional Vectors¶

The Cauchy-Schwarz inequality may be extended to integrals by viewing functions as infinite dimensional vectors living in Hilbert space. Let us imagine we have two continuous functions, $f(x)$ and $g(x)$ that are square-integrable on the interval $[a, b]$. Let us create $n$-dimensional vectors by sampling the value of the function at $n$ evenly spaced points along the interval, producing the vectors

and

where $f(x_0)=f(a)$ and $g(x_0)=g(a)$ are omitted since we are using a right Riemann sum.

By the Cauchy-Schwarz inequality:

As the term $[\frac{b-a}{n}]^{2}$ is a positive constant for any fixed $a$, $b$, and $n$, we may freely multiply both sides by it:

Defining $\Delta x$ as $\frac{b-a}{n}$:

Finally, by taking the limit $\lim_{n\to \infty}\Delta x$ we arrive at the Cauchy-Schwarz inequality in integral form:

Geometric Inequality¶

We can use the Cauchy-Schwarz inequality to derive a tighter upper bound on the covariance of two variables than that found in Eq. (28). We will only explicitly prove the bound for the discrete case of sample covariance, but by Eq. (34) the bound will also hold for the continuous case. Let

for random variable realizations $\mathbf{x} = (x_{1}, x_{2},\ldots,x_{n})$, and let $\mathbf{y'}$ be defined analogously. By the definitions of (sample) variance and covariance,

and

Substituting these definitions into Eq. (29), we conclude that

or

Equivalently, by taking the square roots of both sides and recognizing that covariance could be negative, we arrive at

or more concisely

We thus arrive at the tighter bound that the absolute value of the covariance must always be less than or equal to the geometric mean of the variances.

There is actually an even stricter requirement for covariance, namely that the covariance matrix by positive semidefinite. We will defer discussion of postive semidefiniteness until we encounter it in the specific time series application of covariance.

Correlation¶

Drawbacks to Covariance¶

While generally useful, the covariance does have potential drawbacks. In particular, $\text{Cov}(X, Y)$ has units of $X \times Y$, causing two major difficulties:

1. The exact value obtained will depend on the units used, for example the covariance of height and weight will be different in imperial vs. metric.

2. It can be very difficult to assess if the value of the covariance is practically significant, e.g. is a covariance of $3$ kg cm high or low?

Correlation Definition¶

To overcome these difficulties with covariance, it is often convenient to instead normalize covariance to a unitless quantity bounded by $[-1, 1]$:

This quantity is referred to as the correlation. Note that the Cauchy-Schwarz inequality as applied in Eq. (42) guarantees that Eq. (43) will always lie in $[-1, 1]$. Of course, even when using correlation determining if a value such as 0.7 should be considered a high correlation or not will be context and situation dependent.