giuseppe1.milicia at hsbcib.com
2008-Jul-16 13:52 UTC
[Rd] Snow or alternative MPI packages on Windows
Guys, I'm running R on both Windows & Linux. I'm looking at a number of packages for parallel execution. It seems that the most used packages are "snow" and "Rmpi". snow seems more user friendly, but it doesn't run on windows. I see from searching the mailing list that I'm not the first one to try it on Windows. There was a message that kind of shed some hope on the subject, but nothing else. Rmpi works well on windows (with DeinoMPI) but it's kind of low level, so before I embark in writing code similar to snow to have some high level constructs I though I'd ask here about other peoples experiences... Is anyone around here doing parallel R on Windows? If so could you share your experience? Cheers, // Giuseppe ************************************************************ HSBC Bank plc may be solicited in the course of its placement efforts for a new issue, by investment clients of the firm for whom the Bank as a firm already provides other services. It may equally decide to allocate to its own proprietary book or with an associate of HSBC Group. This represents a potential conflict of interest. HSBC Bank plc has internal arrangements designed to ensure that the firm would give unbiased and full advice to the corporate finance client about the valuation and pricing of the offering as well as internal systems, controls and procedures to identify and manage conflicts of interest. HSBC Bank plc Registered Office: 8 Canada Square, London E14 5HQ, United Kingdom Registered in England - Number 14259 Authorised and regulated by the Financial Services Authority. ************************************************************ ----------------------------------------- SAVE PAPER - THINK BEFORE YOU PRINT! This transmission has been issued by a member of the HSBC Group "HSBC" for the information of the addressee only and should not be reproduced and/or distributed to any other person. Each page attached hereto must be read in conjunction with any disclaimer which forms part of it. Unless otherwise stated, this transmission is neither an offer nor the solicitation of an offer to sell or purchase any investment. Its contents are based on information obtained from sources believed to be reliable but HSBC makes no representation and accepts no responsibility or liability as to its completeness or accuracy.
giuseppe1.milicia at hsbcib.com writes:> Guys, > > I'm running R on both Windows & Linux. I'm looking at a number of packages > for parallel execution. It seems that the most used packages are "snow" and > "Rmpi". > > snow seems more user friendly, but it doesn't run on windows. I see from > searching the mailing list that I'm not the first one to try it on Windows. > There was a message that kind of shed some hope on the subject, but nothing > else. > > Rmpi works well on windows (with DeinoMPI) but it's kind of low level, so > before I embark in writing code similar to snow to have some high level > constructs I though I'd ask here about other peoples experiences...Rmpi has low-level facilities, but things like mpi.spawn.Rslaves, mpi.parLapply and mpi.bcast.cmd are high-level in the same way that snow is. Hopefully others will comment with windows-specific input. Martin> Is anyone around here doing parallel R on Windows? If so could you share > your experience? > > Cheers, > > // Giuseppe > > ************************************************************ > HSBC Bank plc may be solicited in the course of its placement efforts for a > new issue, by investment clients of the firm for whom the Bank as a firm > already provides other services. It may equally decide to allocate to its > own proprietary book or with an associate of HSBC Group. This represents a > potential conflict of interest. HSBC Bank plc has internal arrangements > designed to ensure that the firm would give unbiased and full advice to the > corporate finance client about the valuation and pricing of the offering as > well as internal systems, controls and procedures to identify and manage > conflicts of interest. > > HSBC Bank plc > Registered Office: 8 Canada Square, London E14 5HQ, United Kingdom > Registered in England - Number 14259 > Authorised and regulated by the Financial Services Authority. > ************************************************************ > > > ----------------------------------------- > SAVE PAPER - THINK BEFORE YOU PRINT! > > This transmission has been issued by a member of the HSBC Group > "HSBC" for the information of the addressee only and should not be > reproduced and/or distributed to any other person. Each page > attached hereto must be read in conjunction with any disclaimer > which forms part of it. Unless otherwise stated, this transmission > is neither an offer nor the solicitation of an offer to sell or > purchase any investment. Its contents are based on information > obtained from sources believed to be reliable but HSBC makes no > representation and accepts no responsibility or liability as to its > completeness or accuracy. > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel-- Martin Morgan Computational Biology / Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N. PO Box 19024 Seattle, WA 98109 Location: Arnold Building M2 B169 Phone: (206) 667-2793
Giuseppe, I've been able to use NetWorkSpaces on all platforms, and have used snow under Linux. It's possible to use snow under windows, but you'll need to build the binary yourself I think. As an aside, I think this makes it impossible to write a CRAN-friendly package that Requires: or Suggests: snow, because it will fail the windows check. It would be great if we could get around this some way, but I know this topic has been covered before (the subtle nature of Suggests: in particular, and what it really means). I would like to add SNOW support to two of my own packages, bcp and bigmemory. SNOW and NWS are similarly easy to use, at least at the superficial level which has been my experience. With NWS, you need both the package (nws, available on CRAN), and a NWS server (open-source, available on sourceforge). But REvolution has a Windows installer available, and are working to make the whole process easier on the end user/developer on all platforms. I started a rough page to help in the interim, and will update it as thing change to assist the end user/developer: http://www.stat.yale.edu/~jay/nws/ Jay << original message below >> Guys, I'm running R on both Windows & Linux. I'm looking at a number of packages for parallel execution. It seems that the most used packages are "snow" and "Rmpi". snow seems more user friendly, but it doesn't run on windows. I see from searching the mailing list that I'm not the first one to try it on Windows. There was a message that kind of shed some hope on the subject, but nothing else. Rmpi works well on windows (with DeinoMPI) but it's kind of low level, so before I embark in writing code similar to snow to have some high level constructs I though I'd ask here about other peoples experiences... Is anyone around here doing parallel R on Windows? If so could you share your experience? // Giuseppe -- John W. Emerson (Jay) Assistant Professor of Statistics Department of Statistics Yale University http://www.stat.yale.edu/~jay Statistical Advisor, REvolution Computing
giuseppe1.milicia at hsbcib.com wrote on 07/16/2008 08:52 AM:> Guys, > > I'm running R on both Windows & Linux. I'm looking at a number of packages > for parallel execution. It seems that the most used packages are "snow" and > "Rmpi". > > snow seems more user friendly, but it doesn't run on windows. I see from > searching the mailing list that I'm not the first one to try it on Windows. > There was a message that kind of shed some hope on the subject, but nothing > else. > > Rmpi works well on windows (with DeinoMPI) but it's kind of low level, so > before I embark in writing code similar to snow to have some high level > constructs I though I'd ask here about other peoples experiences... > > Is anyone around here doing parallel R on Windows? If so could you share > your experience?I remember working with Jack Dongarra's group back in the late '90's when Microsoft donated some money, hardware, and software to ensure that pvm, scalapack, and the like worked on Windows. My particular task was to port Atlas to Windows and benchmark installs with various compilers and commercial BLAS libraries. it was a total PITA, dealing with calling conventions between fortran and C/C++, and DLL hell. Anyway, the main researcher involved in the parallel Windows work realized (obviously) the hard problem was remotely connecting to Windows machines and spawning tasks, so he spent the majority of his time writing an rsh daemon, and it became kind of a stink because he went on to commercialize his work, buying the domain name myrshdaemon.com, and selling it. No one ever knew how much money he made. Point being, around ten years have passed and including Windows in a parallel setup is still a problem?!? Sheesh! Jeff -- http://biostat.mc.vanderbilt.edu/JeffreyHorner