Using sparseMVN

What makes a matrix “sparse”

By “sparse matrix,” we mean a matrix for which relatively few elements are non-zero. One situation in which sparse covariance/precision matrices occur is when sampling from a normal approximation to a posterior density of a hierarchical model. For example, suppose we have \(N\) heterogeneous units, each with a parameter vector of length \(k\). Also, suppose that there are \(p\) population-level parameters (e.g., parameters on hyperpriors, homogeneous coefficients, etc). If we assume that the \(N\) parameter vectors are conditionally independent across units, then the cross-partial derivatives of elements of different vectors is zero.

If \(N=5,~k= 2\), and \(p= 2\), then there are 12 total variables, and the Hessian will have the following “block-arrow” pattern.

# 12 x 12 sparse Matrix of class "lgCMatrix"
#                              
#  [1,] | | . . . . . . . . | |
#  [2,] | | . . . . . . . . | |
#  [3,] . . | | . . . . . . | |
#  [4,] . . | | . . . . . . | |
#  [5,] . . . . | | . . . . | |
#  [6,] . . . . | | . . . . | |
#  [7,] . . . . . . | | . . | |
#  [8,] . . . . . . | | . . | |
#  [9,] . . . . . . . . | | | |
# [10,] . . . . . . . . | | | |
# [11,] | | | | | | | | | | | |
# [12,] | | | | | | | | | | | |

There are 144 elements in this symmetric matrix, but only 64 are non-zero, and only 38 values are unique. Although the reduction in RAM from using a sparse matrix structure for the Hessian may be modest, consider what would happen if \(N=1000\) instead. In that case, there are 2002 variables in the problem, and more than \(4\) million elements in the Hessian. However, only \(12004\) of those elements are non-zero. If we work with only the lower triangle of the Hessian we only need to work with only 7003 unique values.

Since a large proportion of elements in the matrix are zero, we need to store only the row and column indices, and the values, of the unique non-zero elements. The efficiency gains in sparseMVN come from storing the covariance or precision matrix in a compressed format without explicit zeros, and applying linear algebra routines that are optimized for those sparse matrix structures. The Matrix package calls sparse linear algebra routines that are implemented in the CHOLMOD library (Chen et al. 2008; Davis and Hager 1999; Davis and Hager 2009); more information about these routines is available there.

Why linear algebra matters for MVN random variables

To see why using linear algebra algorithms that are optimized for sparse matrices is useful, let’s first look at how one would, in general, sample from an MVN. Let \(X\) be a random variable, with \(q\) dimensions, that is distributed MVN with mean \(\mu\) and covariance \(\Sigma\). Let \(L\) be a lower triangular matrix root of \(\Sigma\), such that \(\Sigma=LL'\). To generate a sample from \(X\), we sample \(q\) independent standard normal random variates (call that vector \(z\)), and let \(x=\mu+Lz\). The matrix factor \(L\) could be generated via an eigenvalue, singular value, or Cholesky decomposition. The Cholesky factor of a symmetric, positive definite matrix is the unique factor \(L\) for which all of the diagonal elements are positive. For the rest of this paper, we will use that definition for \(L\).

There are three linear algebra operations involved:

1. Decomposing \(\Sigma=LL'\);
2. Multiplying the triangular matrix \(L\) by vector \(z\); and
3. Adding \(\mu\) to \(Lz\).

The rmvnorm function in the mvtnorm package (Genz et al. 2012) uses this algorithm, taking \(\Sigma\), as a base R matrix, as one of the arguments. Each call to rmvnorm factors \(\Sigma\). If \(\Sigma\) is stored as a typical, base R dense matrix, then the computation time for a Cholesky decomposition grows cubicly, and the time to multiply \(L\) and \(z\) grows quadratically, with the dimension of \(X\) (Golub and Van Loan 1996). Thus, rmvnorm can be quite resource-intensive for large \(X\), especially if the function is called repeatedly.

Computing the density of MVN draws faces similar scalability problems. The density of the MVN distribution is \[ f(x)=(2\pi)^{-\frac{k}{2}}|\Sigma|^{-\frac{1}{2}}\exp\left[-\frac{1}{2}\left(x-\mu\right)'\Sigma^{-1}\left(x-\mu\right)\right] \]

\(\Sigma^{-1}=(LL')^{-1}=L'^{-1}L^{-1}\). If we define \(y=L^{-1}(x-\mu)\), we can write the log density of \(x\) as \[ \log f(x)=-\frac{k}{2}\log(2\pi)-\log|L|-\frac{1}{2}y'y \]

As with generating MVN samples, computing the density requires a matrix factorization that, as above, grows in complexity at a rate cubic to \(q\). The determinant of \(\Sigma\) is the square of the product of the diagonal elements of \(L\). Computing \(y\) involves solving a triangular linear system \(Ly=(x-\mu)\), the complexity of which grows quadratically with \(q\). The dmvnorm function in mvtnorm computes the density this way, so, like rmvnorm, it is resource-intensive for high dimensional \(X\).

The exact amount of time and memory that are saved by saving the covariance/precision matrix in a sparse format depends on the sparsity pattern. But for the hierarchical model example from earlier in this section, the number of non-zero elements grows only linearly with \(N\). The result is that all of the steps of sampling from an MVN also grow linearly with \(N\). Section 4 of Braun and Damien (2014) explains why this is so.

rmvn.sparse(ndraws, mu, CH, prec=TRUE)
dmvn.sparse(x, mu, CH, prec=TRUE)

An example

Suppose we want to generate \(R\) samples from \(X\), where \(X\sim MVN(\mu,\Sigma)\), and the structure of \(\Sigma\) is defined by our example model. In the following code, we first construct the mean vector \(\mu\). Then, we construct a random “block arrow” covariance matrix for a given \(N\), \(p\), and \(k\). We use functions from the Matrix package to ensure that \(\Sigma\) is stored as a sparse matrix in compressed format.

require(Matrix)
set.seed(123)
N <- 5  ## number of blocks in sparse covariance matrix
p <- 2 ## size of each block
k <- 2  ##
R <- 10
    
## mean vector
mu <- seq(-3,3,length=k*N+p)

## build random block-arrow covariance/precision matrix for test
Q1 <- tril(kronecker(diag(N), Matrix(seq(0.1,1.1,length=k*k),k,k)))
Q2 <- Matrix(rnorm(N*k*p), p, N*k)
Q3 <- tril(0.2*diag(p))
Sigma <- rBind(cBind(Q1, Matrix(0, N*k, p)), cBind(Q2, Q3))
Sigma <- Matrix::tcrossprod(Sigma)
class(Sigma)
# [1] "dsCMatrix"
# attr(,"package")
# [1] "Matrix"

Next, we compute the Cholesky decomposition of \(\Sigma\) using the Cholesky function, and call rmvn.sparse, noting that chol.Sigma is the decomposition of a covariance matrix, and not a precision matrix.

chol.Sigma <- Matrix::Cholesky(Sigma)  ## creates a dCHMsimpl object
x <- rmvn.sparse(R, mu, chol.Sigma, prec=FALSE)

Each row of x is a sample, and each column is a variable.

The dmvn.sparse function returns the log density for each row. Since we have already computed the Cholesky decomposition for \(\Sigma\), we do not need to do it again.

d <- dmvn.sparse(x, mu, chol.Sigma, prec=FALSE)
str(d)
#  num [1:10] -1.707 -1.823 -0.775 0.332 -4.417 ...

Sometimes the precision matrix \(\Sigma^{-1}\) is more readily available than \(\Sigma\). For example, the negative Hessian of a log posterior density at the posterior mode is the precision of the normal approximation to that density. Let \(\Sigma^{-1}=\Lambda\Lambda'\) represent the Cholesky decomposition of \(\Sigma^{-1}\). To sample \(x\), we sample \(z\) as before, solve \(\Lambda'x=z\), and then add \(\mu\). Since \(E(z)=0\) and \(E(zz')=I_k\), we have \(E(x)=\mu\), and \(\cov(xx')=E(\Lambda'^{-1}zz'\Lambda^{-1})=\Lambda'^{-1}\Lambda^{-1}=(\Lambda\Lambda')^{-1}=\Sigma\). Then, if we let \(y=\Lambda'(x-\mu)\), the log density is \[ \log f(x)=-\frac{k}{2}\log(2\pi)+|\Lambda|-\frac{1}{2}y'y \]

By setting the argument prec=TRUE, rmvn.sparse and dmvn.sparse will recognize the Cholesky decomposition as being for a precision matrix instead of a covariance matrix. Without this option, the user would need to explicitly invert \(\Sigma\) beforehand. Even if $Sigma^{-1}* were sparse, there is no guarantee that \(\Sigma\) would be (and vice versa). rmvn.sparse and dmvn.sparse let the user work with either the covariance or precision matrix, depending on which is more convenient.

sparseHessianFD

Suppose you have a objective function that has a sparse Hessian (e.g., the log posterior density for a hierarchical model). You have an R function that computes the value of the objective, and another function that computes its gradient. You may also need the Hessian, either for a nonlinear optimization routine, or as the negative precision matrix of an MVN approximation.

It’s hard enough to get the gradient, but the derivation of the Hessian might be too tedious or complicated for it to be worthwhile. However, it should not be too hard to identify which elements of the Hessian are non-zero. If you have both the gradient, and the Hessian pattern, then you can use the sparseHessianFD package (Braun 2015)to estimate the Hessian itself.

The sparseHessianFD package estimates the Hessian numerically, but in a way that exploits the fact that the Hessian is sparse, and that the pattern is known. The package contains functions that return the Hessian as a sparse dgCMatrix. This object can be coerced into a dsCMatrix, which in turn can be used by rmvn.sparse and dmvn.sparse.

trustOptim

The trustOptim package provides a nonlinear optimization routine that takes the Hessian as a sparse dgCMatrix object. This optimizer is useful for unconstrained optimization of a high-dimensional objective function with a sparse Hessian. It uses a trust region algorithm, which may be more stable and robust than line search approaches. Also, it applies a stopping rule based on the norm of the gradient, as opposed to whether the algorithm makes “sufficient progress.” (Many optimizers, especially optim in base R, stop too early, before the gradient is truly flat).