Lustre discuss - Jul 2008 - multiple OSTs accessing the same shared storage simultaneously?

Hi,

I am fairly new to lustre and clustered filesystems having only  
configured SAM-QFS shared filesystems before.  At my work, we are  
planning to deploy a lustre filesystem on RHEL5 systems running on  
top of VMware ESX 3.x.
We are hoping to serve 2 filesystems of 3 and 4 TB respectively via  
10 OSS VMs (1 per ESX host) and 2 MDS VMs (in failover configuration)  
to service about 40 lustre clients (also VMs) running parasol or  
similar.

Due to ESX limitations, each LUN had to be <=2TB, (the disk is  
exported from a SAN) and so in our case each OSS can  see 7 1T LUNS/ 
disks, of which 3 are formatted (mkfs.lustre) and associated with  
"filesystemA" and the other 4 associated with "filesystemB".
We were hoping that each OSS could "share" both complete filesystems  
(ie all 7 TB) to client VMs running locally (for the increased  
performance of the networking within the same VMware host), but that  
these client VMs could also access the same data from any other OSS  
in the VMware cluster if required.

I have looked through the manual, on google and some of the discusion  
list archives to confirm whether this could be done, and it appears  
it can''t.  Am I correct in the following observation:

* Even though you can have multiple OSS''s and OSTs with failover, at  
any one time only one OSS can be accessing a particular block device/ 
LUN.  They can failover the shared storage between them, but can''t  
simultaneously serve data from the same LUN and partition.   ??

I am sorry if I am asking a question that has already been asked on  
numerous occasions.

Thanks,

Marcus.

Systems Administrator.
University of Queensland.

My question is- why are you running the components using VMware? I predict
that you''re actually going to hurt your performance given this setup.
Lustre
will use every little bit of computing power it can get on the OSSs (and
even your cat if you let it) and if you''re sharing physical hardware
between
OSSs and clients I think you''ll experience strong resource contention.
I
would also encourage you not to run the MDS through a VM. The MDS is very
sensitive to latency (particularly disk latency) and I think the latency
introduced by VMware would hurt your performance. If you''re concerned
with
failover then I would use lustre''s builtin failover mechanisms-
they''re very
robust. I would also encourage against using the 40 virtual lustre clients.
Why not use physical clients? You''re performance will be much much
better.
Throw in an infiniband fabric and you''ll get some killer performance.

Oh and to answer your question- an OST cannot be mounted twice
simoltaneously.



Aaron

On Tue, Jul 29, 2008 at 2:58 AM, Marcus Schull <c.schull at
imb.uq.edu.au>wrote:
> Hi,
>
> I am fairly new to lustre and clustered filesystems having only
> configured SAM-QFS shared filesystems before.  At my work, we are
> planning to deploy a lustre filesystem on RHEL5 systems running on
> top of VMware ESX 3.x.
> We are hoping to serve 2 filesystems of 3 and 4 TB respectively via
> 10 OSS VMs (1 per ESX host) and 2 MDS VMs (in failover configuration)
> to service about 40 lustre clients (also VMs) running parasol or
> similar.
>
> Due to ESX limitations, each LUN had to be <=2TB, (the disk is
> exported from a SAN) and so in our case each OSS can  see 7 1T LUNS/
> disks, of which 3 are formatted (mkfs.lustre) and associated with
> "filesystemA" and the other 4 associated with
"filesystemB".
> We were hoping that each OSS could "share" both complete
filesystems
> (ie all 7 TB) to client VMs running locally (for the increased
> performance of the networking within the same VMware host), but that
> these client VMs could also access the same data from any other OSS
> in the VMware cluster if required.
>
> I have looked through the manual, on google and some of the discusion
> list archives to confirm whether this could be done, and it appears
> it can''t.  Am I correct in the following observation:
>
> * Even though you can have multiple OSS''s and OSTs with failover,
at
> any one time only one OSS can be accessing a particular block device/
> LUN.  They can failover the shared storage between them, but can''t
> simultaneously serve data from the same LUN and partition.   ??
>
> I am sorry if I am asking a question that has already been asked on
> numerous occasions.
>
> Thanks,
>
> Marcus.
>
> Systems Administrator.
> University of Queensland.
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080729/1682cb22/attachment.html

On 7/29/08 6:44 AM, "Aaron Knister" <aaron.knister at
gmail.com>did etch on
stone tablets:
> 
> Oh and to answer your question- an OST cannot be mounted twice
simoltaneously.
> 
... well, you ?can- mount it from two locations, you?re just inevitably
going to corrupt the heck out of the volume in question ... so don?t do
that. ;-)

I would also echo Aaron?s remarks about running Lustre servers on VMs ...
big waste of compute power, and you will run into significant
latency/contention issues.

cheers,
Klaus
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080729/d59602a6/attachment.html