There was some discussion about the driver/module for the sata controlers in the thumper (x4500) in the linux kernel. My question is if we bought one of these, would the CFS kernel have everything needed to use the thumper in a safe way. Thank You. Brock Palen www.umich.edu/~brockp Center for Advanced Computing brockp at umich.edu (734)936-1985
Recently I have also been doing some linux work with the x4500 and I have been using the sata_mv driver (v0.81). The driver will properly detect all the drives and you may access them safely. However, from what I''ve seen the driver needs some further development work to actually perform well. I see only 30 MB/s write rates to a single disk using a simple streaming dd test. Much of this bad performance may simply be due to the fact that the driver can not enable the disk write-back cache forcing you to use write-thru mode. So currently the bottom line is linux will work on the x4500. But to get it working well someone is going to need to invest some development effort to improve the linux driver. Good luck, Brian> There was some discussion about the driver/module for the sata > controlers in the thumper (x4500) in the linux kernel. > > My question is if we bought one of these, would the CFS kernel have > everything needed to use the thumper in a safe way. > Thank You.-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080423/50c3191a/attachment-0002.bin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Thats disappointing, thanks for the input though. The paper at: http://wiki.lustre.org/images/7/79/Thumper-BP-6.pdf Points out how to patch it to enable that functionality, but we want to keep with the CFS stock kernel. 30MB/s might be fine for us, we only plan on bonding the 4 thumber 1Gig-e interfaces. Brock Palen www.umich.edu/~brockp Center for Advanced Computing brockp at umich.edu (734)936-1985 On Apr 23, 2008, at 1:00 PM, Brian Behlendorf wrote:> > Recently I have also been doing some linux work with the x4500 and > I have been > using the sata_mv driver (v0.81). The driver will properly detect > all the > drives and you may access them safely. However, from what I''ve > seen the > driver needs some further development work to actually perform > well. I see > only 30 MB/s write rates to a single disk using a simple streaming > dd test. > Much of this bad performance may simply be due to the fact that the > driver > can not enable the disk write-back cache forcing you to use write- > thru mode. > > So currently the bottom line is linux will work on the x4500. But > to get it > working well someone is going to need to invest some development > effort to > improve the linux driver. > > Good luck, > Brian > > >> There was some discussion about the driver/module for the sata >> controlers in the thumper (x4500) in the linux kernel. >> >> My question is if we bought one of these, would the CFS kernel have >> everything needed to use the thumper in a safe way. >> Thank You. > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (Darwin) iD8DBQFID2xmMFCQB4Bvz5QRAsDCAJ44d0zU+fUlQkEASgc3M6JBciYgdQCfaUUM r2VnAo6S0Tj3+qGYCoKrcGE=Xw7Q -----END PGP SIGNATURE-----
For the x4500, there is a marvell mv_sata driver that should be used. It should be downloadable from the x4500 downloads page, sorry I don''t have the specific link handy: http://www.sun.com/servers/x64/x4500/support.xml Regards, Mike On Apr 23, 2008, at 1:00 PM, Brian Behlendorf wrote:> > Recently I have also been doing some linux work with the x4500 and I > have been > using the sata_mv driver (v0.81). The driver will properly detect > all the > drives and you may access them safely. However, from what I''ve seen > the > driver needs some further development work to actually perform > well. I see > only 30 MB/s write rates to a single disk using a simple streaming > dd test. > Much of this bad performance may simply be due to the fact that the > driver > can not enable the disk write-back cache forcing you to use write- > thru mode. > > So currently the bottom line is linux will work on the x4500. But > to get it > working well someone is going to need to invest some development > effort to > improve the linux driver. > > Good luck, > Brian > > >> There was some discussion about the driver/module for the sata >> controlers in the thumper (x4500) in the linux kernel. >> >> My question is if we bought one of these, would the CFS kernel have >> everything needed to use the thumper in a safe way. >> Thank You. > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080423/9adb6cbe/attachment-0002.html
I''d like to offer to debunk some of this... There are plenty of thumpers (x4500) out there in the wild deployed as OSSes under lustre. During a recent deployment using the driver that was mentioned in another post in this thread, the average single disk observed performance was just over 60MB/s with sequential 256k reads. Here''s the link to the mv_sata driver, but as you mention you will need to vary from the stock RPM''s and venture into custom kernel-land. http://www.sun.com/servers/x64/x4500/downloads.jsp We see close to 1GB/s from a single thumper over IB. Your bottleneck will definitely be the bonded gigE, and you will be leaving performance on the table. -fank On Wed, 2008-04-23 at 13:05 -0400, Brock Palen wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Thats disappointing, thanks for the input though. The paper at: > http://wiki.lustre.org/images/7/79/Thumper-BP-6.pdf > > Points out how to patch it to enable that functionality, but we want > to keep with the CFS stock kernel. > 30MB/s might be fine for us, we only plan on bonding the 4 thumber > 1Gig-e interfaces. > > Brock Palen > www.umich.edu/~brockp > Center for Advanced Computing > brockp at umich.edu > (734)936-1985 > > > > On Apr 23, 2008, at 1:00 PM, Brian Behlendorf wrote: > > > > Recently I have also been doing some linux work with the x4500 and > > I have been > > using the sata_mv driver (v0.81). The driver will properly detect > > all the > > drives and you may access them safely. However, from what I''ve > > seen the > > driver needs some further development work to actually perform > > well. I see > > only 30 MB/s write rates to a single disk using a simple streaming > > dd test. > > Much of this bad performance may simply be due to the fact that the > > driver > > can not enable the disk write-back cache forcing you to use write- > > thru mode. > > > > So currently the bottom line is linux will work on the x4500. But > > to get it > > working well someone is going to need to invest some development > > effort to > > improve the linux driver. > > > > Good luck, > > Brian > > > > > >> There was some discussion about the driver/module for the sata > >> controlers in the thumper (x4500) in the linux kernel. > >> > >> My question is if we bought one of these, would the CFS kernel have > >> everything needed to use the thumper in a safe way. > >> Thank You. > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (Darwin) > > iD8DBQFID2xmMFCQB4Bvz5QRAsDCAJ44d0zU+fUlQkEASgc3M6JBciYgdQCfaUUM > r2VnAo6S0Tj3+qGYCoKrcGE> =Xw7Q > -----END PGP SIGNATURE----- > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
Hi there, Has anyone out in the Lustre community set theirs up in an environment that used Juniper switches? We''re testing one out in the lab with ours, and something about its configuration isn''t working. The same setup has been tested and operated successfully with Extreme, Cisco, and Alcatel, with no changes to the Lustre setup itself. I''m thinking it''s more than likely not the Lustre setup, but something about the switch configuration - VLAN membership, IGMP config, etc. Is there a checklist of required network protocols/behaviours we can use to verify our switch configuration? cheers, Klaus
Which kernel are you using? It looks your 0.81 is newer than original rhel4.x''s (or rhel5) one. This is not about the disk driver, but X4500(a.k.a Thumper) needs the Linux RAID patches to use as OSS. This patches are included in the lustre patched kenrel and can use on rhel4.x (CentOS 4.x) right now. Thanks, -Ihara Brian Behlendorf wrote:> Recently I have also been doing some linux work with the x4500 and I have been > using the sata_mv driver (v0.81). The driver will properly detect all the > drives and you may access them safely. However, from what I''ve seen the > driver needs some further development work to actually perform well. I see > only 30 MB/s write rates to a single disk using a simple streaming dd test. > Much of this bad performance may simply be due to the fact that the driver > can not enable the disk write-back cache forcing you to use write-thru mode. > > So currently the bottom line is linux will work on the x4500. But to get it > working well someone is going to need to invest some development effort to > improve the linux driver. > > Good luck, > Brian > > >> There was some discussion about the driver/module for the sata >> controlers in the thumper (x4500) in the linux kernel. >> >> My question is if we bought one of these, would the CFS kernel have >> everything needed to use the thumper in a safe way. >> Thank You. >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss