Greg Minshall
2013-Feb-14 14:08 UTC
[R] network connectivity is *not* symmetric -- library(parallel)
hi. i've been very happy using multicore functionality, but felt the need to get more horsepower, so have been trying to use parallel (after realizing it is the "new snow") to access an amazon EC2 cluster, but have had no success. in the internet today, if machine M (the master) can open a connection to machine S (the slave), it's somewhat of a miracle, something to be celebrated. having accomplished that miracle, expecting that machine S can *then* open a connection back to machine M is expected p^2 to be close to p. i get failures stating that the slave can't connect to the master, that port 10187 "cannot be opened". it would be nice if there were a mode whereby the *slave* would set up to listen on a socket, transmit the socket number back to the master (over the ssh connection the master has set up to the slave), and allow the *master* (with proven ability to connect to the slave) to do the connecting. would such a mode be possible? i suspect it would make a lot of people happy. (network failures are pretty mysterious.) and, while looking at the code, i note that the socket (port) number being opened is, essentially, hard coded into parallels/snow. and, it looks as if socketConnection requires a specification of the port number. it might be nice to allow the system to choose the port number [at the socket(2) call], and then report that back to the other side. this would probably involve changes to socketConnection as well as maybe the internet module, so is probably longer term. i think, though, that doing this would benefit people who now are confused when they can't start up two clusters on the same (currently, master) machine, etc. cheers, Greg Minshall
Apparently Analagous Threads
- Conflicts between 'parallel' and 'Rprof', and between two parallel R sessions
- problem with opening more than one SOCK cluster with package snow
- SUGGESTION: Proposal to mitigate problem with stray processes left behind by parallel::makeCluster()
- no snowfall on debian after lenny upgrade
- OT: snow socket clusters with VirtualBox and VMware player (Linux host, Win guest)