r - Getting "node stack overflow" when cbind multiple sparse matrices -
i have 100,000 sparse matrices("dgcmatrix") store in list object. row number of every matrix same(8,000,000) , size of list approximately 25 gb. when do:
do.call(cbind, thelistofmatrices)
to combine matrices 1 big sparse matrix, got "node stack overflow". actually, can't 500 elements out of list, should output sparse matrix size of 100 mb.
my speculation cbind() function transformed sparse matrix normal dense matrix , cause stack overflow?
actually, have tried this:
tmp = do.call(cbind, thelistofmatrices[1:400])
this works fine, , tmp still sparse matrix size of 95 mb, , tried:
> tmp = do.call(cbind, thelistofmatrices[1:410]) error in stopifnot(0 <= deparse.level, deparse.level <= 2) : node stack overflow
and error occurred. however, having no trouble doing like:
cbind(tmp, tmp, tmp, tmp)
thus, believe has do.call()
reduce() seems solve problem, though still don't know reason why do.call() crushes.
in r: 2-column matrix can have 2^30-1 rows = 1073,741,823 rows. so, check row number , check ram size make sure can accommodate big matrix size.