Linear Algebra Review In order to understand how Kalman Filter theory is developed, it is useful to
review fundamental linear algebra principles.
Column Vector
A column vector is an array of n n n vector
implies a column vector.
x ( t ) = [ x 1 ( t ) ⋮ x n ( t ) ] \bm{x}(t) =
\begin{bmatrix}
x_1(t) \\
\vdots \\
x_n(t)
\end{bmatrix} x ( t ) = x 1 ( t ) ⋮ x n ( t ) Dot Product
The dot product of two column vectors is defined below.
x ⋅ y = x T y = [ x 1 , … , x n ] ⋅ [ y 1 ⋮ y n ] = ∑ i = 1 n x i y i = x 1 y 1 + … + x n y n \bm{x} \cdot \bm{y} = \bm{x}^T \bm{y} =
\begin{bmatrix} x_1, \ldots, x_n \end{bmatrix} \cdot
\begin{bmatrix} y_1 \\ \vdots \\ y_n \end{bmatrix} =
\sum_{i=1}^{n}x_i y_i =
x_1 y_1 + \ldots + x_n y_n x ⋅ y = x T y = [ x 1 , … , x n ] ⋅ y 1 ⋮ y n = i = 1 ∑ n x i y i = x 1 y 1 + … + x n y n Matrix
A matrix of dimension m × n m \times n m × n
A = [ A 1 , 1 … A 1 , n ⋮ ⋱ ⋮ A m , 1 … A m , n ] A = \begin{bmatrix}
A_{1,1} & \ldots & A_{1,n} \\
\vdots & \ddots & \vdots \\
A_{m,1} & \ldots & A_{m,n} \\
\end{bmatrix} A = A 1 , 1 ⋮ A m , 1 … ⋱ … A 1 , n ⋮ A m , n Matrix-Vector Product
A matrix of dimension m × n m \times n m × n n × 1 n \times 1 n × 1
y = A x = [ A 1 , 1 … A 1 , n ⋮ ⋱ ⋮ A m , 1 … A m , n ] [ x 1 ⋮ x n ] \bm{y} = A \bm{x} =
\begin{bmatrix}
A_{1,1} & \ldots & A_{1,n} \\
\vdots & \ddots & \vdots \\
A_{m,1} &\ldots & A_{m, n} \\
\end{bmatrix}
\begin{bmatrix}
x_1 \\
\vdots \\
x_n \\
\end{bmatrix} y = A x = A 1 , 1 ⋮ A m , 1 … ⋱ … A 1 , n ⋮ A m , n x 1 ⋮ x n Quadratic form is expressed below, with the dimensions of each element
annotated.
α ⏞ 1 × 1 = x T ⏟ 1 × n A ⏞ n × n x ⏟ n × 1 \overbrace{\alpha}^{1 \times 1} =
\underbrace{\bm{x}^T}_{1 \times n}
\overbrace{A}^{n \times n}
\underbrace{\bm{x}}_{n \times 1} α 1 × 1 = 1 × n x T A n × n n × 1 x Matrix-Matrix Product
Matrices can be multiplied with each other, as long as they satisfy the
dimensional requirements outlined below.
C ⏟ n × m = A ⏞ n × p B ⏟ p × m \underbrace{C}_{n \times m} =
\overbrace{A}^{n \times p}
\underbrace{B}_{p \times m} n × m C = A n × p p × m B And, in general, A B ≠ B A AB \neq BA A B = B A
Scalar and Vector Functions
A scalar function of a scalar has the following form, where t t t α \alpha α
α = α ( t ) \alpha = \alpha (t) α = α ( t ) A scalar function of a vector has the following form, where x 1 , … , x n x_1, \ldots , x_n x 1 , … , x n α \alpha α
α = α ( x ) = α ( x 1 , … , x n ) \alpha = \alpha (\bm{x}) = \alpha (x_1, \ldots, x_n) α = α ( x ) = α ( x 1 , … , x n ) A vector function of a vector has the following form.
y = [ y 1 ( x 1 , … , x m ) … y n ( x 1 , … , x m ) ] = y ⏞ n × 1 ( x ⏟ m × 1 ) \bm{y} =
\begin{bmatrix}
y_{1}(x_{1}, \ldots ,x_m) \\
\ldots \\
y_{n}(x_{1}, \ldots ,x_m)
\end{bmatrix} =
\overbrace{\bm{y}}^{n \times 1}(\underbrace{\bm{x}}_{m \times 1}) y = y 1 ( x 1 , … , x m ) … y n ( x 1 , … , x m ) = y n × 1 ( m × 1 x ) Scalar and Vector Derivatives
A scalar derivative has the following form.
α = α ( t ) α ˙ = d α ( t ) d t \begin{align*}
\alpha &= \alpha (t) \\
\dot{\alpha} &= \frac{d \alpha (t)}{dt}
\end{align*} α α ˙ = α ( t ) = d t d α ( t ) A vector derivative has the following form.
x = [ x 1 ( t ) ⋮ x n ( t ) ] x ˙ = d x d t = [ d x 1 d t ⋮ d x n d t ] \begin{align*}
\bm{x} &=
\begin{bmatrix}
x_1(t) \\
\vdots \\
x_n(t) \\
\end{bmatrix} \\
\dot{\bm{x}} =
\frac{d \bm{x}}{dt} &=
\begin{bmatrix}
\frac{d x_1}{dt} \\
\vdots \\
\frac{d x_n}{dt}
\end{bmatrix}
\end{align*} x x ˙ = d t d x = x 1 ( t ) ⋮ x n ( t ) = d t d x 1 ⋮ d t d x n For a scalar function of a vector, the gradient vector derivative has the
following form.
α = α ( x 1 , … , x n ) = α ( x ) d α d x = [ ∂ α ∂ x 1 … ∂ α ∂ x n ] \begin{align*}
\alpha &= \alpha (x_{1}, \ldots , x_{n}) = \alpha (\bm{x}) \\
\frac{d \alpha }{ d \bm{x} } &=
\begin{bmatrix}
\frac{\partial \alpha }{\partial x_1} & \ldots & \frac{\partial \alpha}{\partial x_n}
\end{bmatrix}
\end{align*} α d x d α = α ( x 1 , … , x n ) = α ( x ) = [ ∂ x 1 ∂ α … ∂ x n ∂ α ] For a vector function of a vector, the gradient vector derivative has the
following form.
y = y ( x ) = [ y 1 ( x 1 , … , x n ) ⋮ y m ( x 1 , … , x n ) ] d y d x = [ ∂ y 1 ∂ x 1 … ∂ y 1 ∂ x n ⋮ ⋱ ⋮ ∂ y m ∂ x 1 … ∂ y m ∂ x n ] ⏟ m × n \begin{align*}
\bm{y} = \bm{y}(\bm{x}) &=
\begin{bmatrix}
y_1(x_1, \ldots , x_n) \\
\vdots \\
y_m(x_1, \ldots , x_n)
\end{bmatrix} \\
\frac{d \bm{y}}{d \bm{x}} &=
\underbrace{
\begin{bmatrix}
\frac{\partial y_1}{\partial x_1} & \ldots & \frac{\partial y_1}{\partial x_n} \\
\vdots & \ddots & \vdots \\
\frac{\partial y_m}{\partial x_1} & \ldots & \frac{\partial y_m}{\partial x_n}
\end{bmatrix}}_{m \times n}
\end{align*} y = y ( x ) d x d y = y 1 ( x 1 , … , x n ) ⋮ y m ( x 1 , … , x n ) = m × n ∂ x 1 ∂ y 1 ⋮ ∂ x 1 ∂ y m … ⋱ … ∂ x n ∂ y 1 ⋮ ∂ x n ∂ y m Rank
Rank is the number of linearly independent rows or columns of a matrix. It
can also be thought of as the largest non-singular sub-matrix dimension,
although I don’t really understand what that means.
For example,
A = [ 1 1 1 − 1 ] det A ≠ 0 ∴ A has rank 2. \begin{align*}
A &= \begin{bmatrix}
1 & 1 \\
1 & -1
\end{bmatrix} \\
\det{A} &\neq 0 \\
&\therefore \text{A has rank 2.}
\end{align*} A det A = [ 1 1 1 − 1 ] = 0 ∴ A has rank 2. As another example,
A = [ 1 0 0 0 0 0 0 1 2 0 0 0 1 0 3 0 0 1 0 3 0 0 3 2 6 ] Observe that R 4 = R 2 + 2 R 3 ∴ A has 3 linearly independent rows, so its rank is 3. \begin{align*}
&A =
\begin{bmatrix}
1 & 0 & 0 & 0 & 0 \\
0 & 0 & 1 & 2 & 0 \\
0 & 0 & 1 & 0 & 3 \\
0 & 0 & 1 & 0 & 3 \\
0 & 0 & 3 & 2 & 6 \\
\end{bmatrix} \\
&\text{Observe that } R_4 = R_2 + 2R_3 \\
&\therefore A \text{ has 3 linearly independent rows, so its rank is 3.}
\end{align*} A = 1 0 0 0 0 0 0 0 0 0 0 1 1 1 3 0 2 0 0 2 0 0 3 3 6 Observe that R 4 = R 2 + 2 R 3 ∴ A has 3 linearly independent rows, so its rank is 3. And as a final example,
x = [ 1 2 3 ] x x T = [ 1 2 3 2 4 6 3 6 9 ] All rows and columns are multiples of R 1 . ∴ A has rank 1. \begin{align*}
\bm{x} &=
\begin{bmatrix}
1 \\ 2 \\ 3
\end{bmatrix} \\
\bm{x} \bm{x}^T &= \begin{bmatrix}
1 & 2 & 3 \\
2 & 4 & 6 \\
3 & 6 & 9
\end{bmatrix} \\
&\text{All rows and columns are multiples of } R_1 \text{.} \\
&\therefore A \text{ has rank 1.}
\end{align*} x x x T = 1 2 3 = 1 2 3 2 4 6 3 6 9 All rows and columns are multiples of R 1 . ∴ A has rank 1.