How can I use all of my Amazon EC2 cores to carry out a parallel simulation in R? -
i run 1000 iterations of simulation in r. each iteration takes 20 seconds take ~6 hours in serial. , have several dozen simulations run, each 1000 iterations. want use parallel computing accomplish work in less time.
i new @ this. after reading material on web, using amazon’s ec2 service, running ami rstudio server , openblas helpfully provided louis aslett. details of software loaded ami here: http://www.louisaslett.com/rstudio_ami.
i experimented several ec2 instances , using c4.8xlarge ubuntu instance 36 cores. or virtual cores, i’m not entirely sure of difference.
my problem can’t seem use anywhere near available 36 cores. have used 10 cores, errors when attempt use >10 cores.
here minimal code (using cars dataset) reproduce error:
library(parallel) detectcores() #36 ptm<-proc.time() cl <- makecluster(getoption("cl.cores", 20)) #specify number of cores clustersetrngstream(cl, 123) sims <- clusterevalq(cl, { cars[sample(1:nrow(cars), 10, replace=false),] }) stopcluster(cl) that code works fine when specify 10 cores in makecluster() statement. produces list 10 elements, each of produced 1 of 10 cores. specifying >10 cores (i.e., above) results in errors such have pasted below.
as mentioned above, i’m new @ , code based on example found online meant run on single, multi-core computer (not ec2 machine). realize may have misspecified makecluster() statement purposes of running on ec2.
i appreciate or suggestions working!
many thanks,
mark b.
here error when attempting use >10 cores:
openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max /usr/lib/r/bin/r: fork: retry: resource temporarily unavailable openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max /usr/lib/r/bin/r: fork: retry: no child processes /usr/lib/r/bin/r: fork: retry: resource temporarily unavailable /usr/lib/r/bin/r: fork: retry: no child processes /usr/lib/r/bin/r: fork: retry: no child processes /usr/lib/r/bin/r: fork: retry: resource temporarily unavailable /usr/lib/r/bin/r: fork: retry: no child processes openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max openblas blas_thread_init: pthread_create: resource temporarily unavailable openblas blas_thread_init: rlimit_nproc 241473 current, 241473 max /usr/lib/r/bin/r: fork: retry: no child processes /usr/lib/r/bin/r: fork: retry: no child processes /usr/lib/r/bin/r: fork: retry: no child processes /usr/lib/r/bin/r: fork: retry: no child processes /usr/lib/r/bin/r: fork: retry: no child processes /usr/lib/r/bin/r: fork: retry: no child processes /usr/lib/r/bin/r: fork: resource temporarily unavailable error: option '-e' requires non-empty argument
we encountered problem week. did remove openblas , exiting r ec2 , re-installed r without openblas. worked. if need openblas accelerate algorithm, may need build r openblas source codes (we haven't tried approach yet).
