On 05/27/2011 07:12 AM, Jon Tegner wrote:> A general question, suppose I have a parallel application, using mpi,
> where really fast access to the file system is critical.
>
> Would it be stupid to consider a ram disk based setup? Say a 36 port QDR
Ram disks won't work directly, due to lack of locking in tmpfs. You
could create a tmpfs, then create a file that fills this up, then a
loopback device pointing to that file, then build a file system atop
that, and mount it. And then mount gluster atop that.
Needless to say, all these layers significantly decrease performance and
introduce inefficiencies.
> infiniband with half of the ports connected to computational nodes and
> the other half to gluster nodes?
There may be other options, but the options are not going to be
cheap/inexpensive. How fast, and by fast do you mean bandwidth and/or
latency (e.g. streaming bandwidth or random IOPs)? What does your IO
profile look like?
You can get nodes that stream 4.6+ GB/s read, and 3.6+ GB/s writes for
single readers/writers to single files. For MPI jobs with single
readers/writers, this is good. For very large IO jobs where you need
10's of GB/s, you probably need a more specific design to your problem.
Regards,
Joe
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web : http://scalableinformatics.com
http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615