r - Getting "node stack overflow" when cbind multiple sparse matrices -


i have 100,000 sparse matrices("dgcmatrix") store in list object. row number of every matrix same(8,000,000) , size of list approximately 25 gb. when do:

do.call(cbind, thelistofmatrices) 

to combine matrices 1 big sparse matrix, got "node stack overflow". actually, can't 500 elements out of list, should output sparse matrix size of 100 mb.

my speculation cbind() function transformed sparse matrix normal dense matrix , cause stack overflow?

actually, have tried this:

tmp = do.call(cbind, thelistofmatrices[1:400]) 

this works fine, , tmp still sparse matrix size of 95 mb, , tried:

> tmp = do.call(cbind, thelistofmatrices[1:410]) error in stopifnot(0 <= deparse.level, deparse.level <= 2) :    node stack overflow 

and error occurred. however, having no trouble doing like:

cbind(tmp, tmp, tmp, tmp) 

thus, believe has do.call()

reduce() seems solve problem, though still don't know reason why do.call() crushes.

in r: 2-column matrix can have 2^30-1 rows = 1073,741,823 rows. so, check row number , check ram size make sure can accommodate big matrix size.


Popular posts from this blog

php - How should I create my API for mobile applications (Needs Authentication) -

5 Reasons to Blog Anonymously (and 5 Reasons Not To)

Google AdWords and AdSense - A Dynamic Small Business Marketing Duo