Hi all, Are there any pointers to Lustre-WAN info? Searches on the Lustre Wiki and Google do not turn up much (other than it exists). Scott -- Scott Atchley Myricom Inc. http://www.myri.com
Hi, Well - HP did a Lustre demo at SC 2005 and was the winner of storcloud and the bandwidth challenge with a WAN installation. During the coming months there will be betas of Lustre with Kerberos and remote UID management, the most asked for feature. We should then begin to learn what works well and what doesn''t and make appropriate changes. - Peter - > -----Original Message----- > From: lustre-discuss-bounces@clusterfs.com > [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of > Scott Atchley > Sent: Thursday, August 10, 2006 8:05 AM > To: Lustre User Discussion Mailing List > Subject: [Lustre-discuss] Pointers to Lustre-WAN? > > Hi all, > > Are there any pointers to Lustre-WAN info? Searches on the > Lustre Wiki and Google do not turn up much (other than it exists). > > Scott > > -- > Scott Atchley > Myricom Inc. > http://www.myri.com > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >
Scott, I was involved in the HP-PNNL Lustre Demo at SC05, and we did a large amount of IO from Seattle to Richland, WA over 60 Gbit/s fibre link. This link is part of the Ultra-Sciences Network, and was about 6 ms of delay in the line. We sustained the about 40GBits/s of I/O over the link. Once the network was up it pretty much worked with little changes. We were very careful to do Large file/block IO''s for our application, and that really helps. Also we have been ''playing'' with Lustre over the USN network to seattle for about a year now, and have made various loops from seattle, to ornl in our testing. Our biggest issue so far is when we could get really good throughput with iperf, and netperf using the 10 Gig cards we had in TOE mode, but when you ran lustre over it the network seemed to bork itself. We did eventually find that we could recreate this with just network tests, and have resumed testing with no TOE enabled on the card. So far we can sustain 300 MB/s over a loop to seattle.(1.5 ms delay). We are getting serious with our testing at this point, and are working on doing much larger loops 6000 miles I think is the biggest we plan, and will be publishing our results in the next few weeks. Evan> -----Original Message----- > From: lustre-discuss-bounces@clusterfs.com > [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of > Scott Atchley > Sent: Thursday, August 10, 2006 7:05 AM > To: Lustre User Discussion Mailing List > Subject: [Lustre-discuss] Pointers to Lustre-WAN? > > Hi all, > > Are there any pointers to Lustre-WAN info? Searches on the > Lustre Wiki and Google do not turn up much (other than it exists). > > Scott > > -- > Scott Atchley > Myricom Inc. > http://www.myri.com > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >
Hi Evan, Thanks for the details. Any idea how TOE was getting in the way? When it was working, we you using any hardware acceleration (TSO, checksum offload, etc.)? Scott On Aug 14, 2006, at 4:33 PM, Felix, Evan J wrote:> Scott, I was involved in the HP-PNNL Lustre Demo at SC05, and we did a > large amount of IO from Seattle to Richland, WA over 60 Gbit/s fibre > link. This link is part of the Ultra-Sciences Network, and was > about 6 > ms of delay in the line. We sustained the about 40GBits/s of I/O over > the link. Once the network was up it pretty much worked with little > changes. We were very careful to do Large file/block IO''s for our > application, and that really helps. > > Also we have been ''playing'' with Lustre over the USN network to > seattle > for about a year now, and have made various loops from seattle, to > ornl > in our testing. Our biggest issue so far is when we could get really > good throughput with iperf, and netperf using the 10 Gig cards we > had in > TOE mode, but when you ran lustre over it the network seemed to bork > itself. We did eventually find that we could recreate this with just > network tests, and have resumed testing with no TOE enabled on the > card. > So far we can sustain 300 MB/s over a loop to seattle.(1.5 ms delay). > > We are getting serious with our testing at this point, and are working > on doing much larger loops 6000 miles I think is the biggest we plan, > and will be publishing our results in the next few weeks. > > Evan > >> -----Original Message----- >> From: lustre-discuss-bounces@clusterfs.com >> [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of >> Scott Atchley >> Sent: Thursday, August 10, 2006 7:05 AM >> To: Lustre User Discussion Mailing List >> Subject: [Lustre-discuss] Pointers to Lustre-WAN? >> >> Hi all, >> >> Are there any pointers to Lustre-WAN info? Searches on the >> Lustre Wiki and Google do not turn up much (other than it exists). >> >> Scott >> >> -- >> Scott Atchley >> Myricom Inc. >> http://www.myri.com >> >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss@clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss >> >
I don''t remember the details of the TOE setup, Ryan Mooney may be able to shed some light on it. I remember things would start working, Lustre mounted, things looked good, you would start IO and it would go for a bit, then blam, the connections would bork, and things would fail.. After a bit Lustre would reconnect, replay, and work again... Very bad for I/O tests to get delays like that over and over... Ryan can you comment on the TOE configuration we were using in our USN testing? Evan> -----Original Message----- > From: Scott Atchley [mailto:atchley@myri.com] > Sent: Monday, August 14, 2006 5:21 PM > To: Felix, Evan J > Cc: Lustre User Discussion Mailing List > Subject: Re: [Lustre-discuss] Pointers to Lustre-WAN? > > Hi Evan, > > Thanks for the details. > > Any idea how TOE was getting in the way? When it was working, > we you using any hardware acceleration (TSO, checksum offload, etc.)? > > Scott > > On Aug 14, 2006, at 4:33 PM, Felix, Evan J wrote: > > > Scott, I was involved in the HP-PNNL Lustre Demo at SC05, > and we did a > > large amount of IO from Seattle to Richland, WA over 60 > Gbit/s fibre > > link. This link is part of the Ultra-Sciences Network, and > was about > > 6 ms of delay in the line. We sustained the about 40GBits/s of I/O > > over the link. Once the network was up it pretty much worked with > > little changes. We were very careful to do Large > file/block IO''s for > > our application, and that really helps. > > > > Also we have been ''playing'' with Lustre over the USN network to > > seattle for about a year now, and have made various loops from > > seattle, to ornl in our testing. Our biggest issue so far > is when we > > could get really good throughput with iperf, and netperf > using the 10 > > Gig cards we had in TOE mode, but when you ran lustre over it the > > network seemed to bork itself. We did eventually find that > we could > > recreate this with just network tests, and have resumed > testing with > > no TOE enabled on the card. > > So far we can sustain 300 MB/s over a loop to seattle.(1.5 > ms delay). > > > > We are getting serious with our testing at this point, and > are working > > on doing much larger loops 6000 miles I think is the > biggest we plan, > > and will be publishing our results in the next few weeks. > > > > Evan > > > >> -----Original Message----- > >> From: lustre-discuss-bounces@clusterfs.com > >> [mailto:lustre-discuss-bounces@clusterfs.com] On Behalf Of Scott > >> Atchley > >> Sent: Thursday, August 10, 2006 7:05 AM > >> To: Lustre User Discussion Mailing List > >> Subject: [Lustre-discuss] Pointers to Lustre-WAN? > >> > >> Hi all, > >> > >> Are there any pointers to Lustre-WAN info? Searches on the Lustre > >> Wiki and Google do not turn up much (other than it exists). > >> > >> Scott > >> > >> -- > >> Scott Atchley > >> Myricom Inc. > >> http://www.myri.com > >> > >> > >> _______________________________________________ > >> Lustre-discuss mailing list > >> Lustre-discuss@clusterfs.com > >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > >> > > >