We have an in house 27 node cluster. With no shared storage capability. I have just order 2 storage nodes with extra drives in them and was hoping I could use this with Lustre and provide a good shared storage setup. How would you suggest configuring this setup ? Each node comes with dual network adapters so I was thinking about bonding them together to provide more bandwidth ( our switch should allow us to do this.) Thanks Iain Iain Grant Linux Administrator Scottish Crop Research Institute Dundee DD2 5DA Tel: +44 (0)1382 562731 mailto:Iain.Grant at scri.ac.uk <mailto:Iain.Grant at scri.ac.uk> www.scri.ac.uk <http://www.scri.ac.uk/> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ SCRI, Invergowrie, Dundee, DD2 5DA. The Scottish Crop Research Institute is a charitable company limited by guarantee. Registered in Scotland No: SC 29367. Recognised by the Inland Revenue as a Scottish Charity No: SC 006662. DISCLAIMER: This email is from the Scottish Crop Research Institute, but the views expressed by the sender are not necessarily the views of SCRI and its subsidiaries. This email and any files transmitted with it are confidential to the intended recipient at the e-mail address to which it has been addressed. It may not be disclosed or used by any other than that addressee. If you are not the intended recipient you are requested to preserve this confidentiality and you must not use, disclose, copy, print or rely on this e-mail in any way. Please notify postmaster at scri.ac.uk quoting the name of the sender and delete the email from your system. Although SCRI has taken reasonable precautions to ensure no viruses are present in this email, neither the Institute nor the sender accepts any responsibility for any viruses, and it is your responsibility to scan the email and the attachments (if any). -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080212/36501aaa/attachment-0002.html
Hi,
You could use lacp if switch supports it (not the usual static
trunking support on lower end switches) or
adaptive load balancing without any switch support. I moved from lacp to
adaptive load balancing
as i was interested in increasing the aggregate throughout and didn''t
want to rely on a bug free
switch firmware support for 802.3ad. I have a similar setup on one of
our lustre installations with two
storage servers. So i assume that you would be having the MDS and MGS
along with a few OSTs on
one of the two servers and the other a pure OSS. I have the same setup.
If i had to do it all over again,
i would go with a dedicated MDS + MGS server and dedicated OSS servers.
Also i got only 4 GB ram on
both storage servers and had to upgrade ram to 8 GB on both servers.
Lustre installation could use quite a bit
of ram depending on the usage (in my case over night rsyncs of an almost
full volume to another big lustre volume
was the tipping point) Also you should watch for some bugs that gives
heart burns on production servers like this -
(https://bugzilla.lustre.org/show_bug.cgi?id=13438) The lustre servers
were crashing randomly almost once a day
for two months and finally the patch makes them run stable again.
Another thing with bonding is the number of interfaces
you put in the bond. The more the number of interfaces, the more the
number of interrupts they would generate. Jumbo frames
could reduce that i assume. The ring parameters needs to bumped up so
that the interfaces won''t drop frames.
Just my experiences with the situation, it may or may
not be useful in your setup. Hope it helps!
Regards
Balagopal
Iain Grant wrote:>
> We have an in house 27 node cluster. With no shared storage capability.
>
>
>
> I have just order 2 storage nodes with extra drives in them and was
> hoping I could use this with Lustre and provide a good shared storage
> setup.
>
>
>
> How would you suggest configuring this setup ?
>
>
>
> Each node comes with dual network adapters so I was thinking about
> bonding them together to provide more bandwidth ( our switch should
> allow us to do this.)
>
>
>
> Thanks
>
>
>
> Iain
>
> **Iain Grant**
> */Linux Administrator/*
> Scottish Crop Research Institute
> Dundee DD2 5DA
> Tel: +44 (0)1382 562731
> mailto:Iain.Grant at scri.ac.uk
>
> www.scri.ac.uk <http://www.scri.ac.uk/>
>
>
>
>
>
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> SCRI, Invergowrie, Dundee, DD2 5DA.
> The Scottish Crop Research Institute is a charitable company limited by
guarantee.
> Registered in Scotland No: SC 29367.
> Recognised by the Inland Revenue as a Scottish Charity No: SC 006662.
>
>
> DISCLAIMER:
>
> This email is from the Scottish Crop Research Institute, but the views
> expressed by the sender are not necessarily the views of SCRI and its
> subsidiaries. This email and any files transmitted with it are
confidential
> to the intended recipient at the e-mail address to which it has been
> addressed. It may not be disclosed or used by any other than that
addressee.
> If you are not the intended recipient you are requested to preserve this
> confidentiality and you must not use, disclose, copy, print or rely on this
> e-mail in any way. Please notify postmaster at scri.ac.uk quoting the
> name of the sender and delete the email from your system.
>
> Although SCRI has taken reasonable precautions to ensure no viruses are
> present in this email, neither the Institute nor the sender accepts any
> responsibility for any viruses, and it is your responsibility to scan the
email
> and the attachments (if any).
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
Hi Iain, One of the clusters I work with here uses jumbo-enabled LACP-aggregated NICs, and I can get about 350 MB/s total sustained reads out of two OSSes (each OSS has two NICs, so 4 GigE paths total). However, the cluster itself is smaller than yours; you might want to investigate another network fabric, as GigE performance may start to suffer with that many nodes due to the limitations of Ethernet itself. A good L3 switch (Extreme, Cisco, Foundry, etc.) will be able to provide you with optimal performance. hth, Klaus On 2/12/08 1:45 AM, "Iain Grant" <Iain.Grant at scri.ac.uk>did etch on stone tablets:> We have an in house 27 node cluster. With no shared storage capability. > > I have just order 2 storage nodes with extra drives in them and was hoping I > could use this with Lustre and provide a good shared storage setup. > > How would you suggest configuring this setup ? > > Each node comes with dual network adapters so I was thinking about bonding > them together to provide more bandwidth ( our switch should allow us to do > this.) > > Thanks > > Iain > Iain Grant > Linux Administrator > Scottish Crop Research Institute > Dundee DD2 5DA > Tel: +44 (0)1382 562731 > mailto:Iain.Grant at scri.ac.uk <mailto:Iain.Grant at scri.ac.uk> > > www.scri.ac.uk <http://www.scri.ac.uk/> > > > > > > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > SCRI, Invergowrie, Dundee, DD2 5DA. > The Scottish Crop Research Institute is a charitable company limited by > guarantee. > Registered in Scotland No: SC 29367. > Recognised by the Inland Revenue as a Scottish Charity No: SC 006662. > > > DISCLAIMER: > > This email is from the Scottish Crop Research Institute, but the views > expressed by the sender are not necessarily the views of SCRI and its > subsidiaries. This email and any files transmitted with it are confidential > to the intended recipient at the e-mail address to which it has been > addressed. It may not be disclosed or used by any other than that addressee. > If you are not the intended recipient you are requested to preserve this > confidentiality and you must not use, disclose, copy, print or rely on this > e-mail in any way. Please notify postmaster at scri.ac.uk quoting the > name of the sender and delete the email from your system. > > Although SCRI has taken reasonable precautions to ensure no viruses are > present in this email, neither the Institute nor the sender accepts any > responsibility for any viruses, and it is your responsibility to scan the > email > and the attachments (if any). > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080212/e4362048/attachment-0002.html