Chris Bachmann wrote:> I''m trying to set up a simple array with 1 client, 1 mds, and 2 ost''s. What > I would like to do is to have the 2 ost volumes mirror each other. I used > the lwizard and set the failover, but when i turn off the primary ost and > use the lconf --select command line referenced in > https://wiki.clusterfs.com/lustre/LustreFailover, it doesn''t contain any of > the content of the one that I shut off (or at least, it''s not reflected in > my df -h). > > So it''s obvious that I''m doing something wrong here, I just don''t know what. > How can I make sure that the content is mirrored between the two ost volumes > so if one of them goes bye-bye, the content is still preserved?You would do that sort of mirroring below Lustre - Lustre creates a single large filesystem image from multiple ost volumes. We don''t do mirroring, or any sort of RAID. --failover is used when you have multiple systems that share physical storage, which appears to be the reverse of your case. In the failover case, the storage is only accessed by one system, --failover is used to move the storage to a second system. Typically, if you need disk replication/mirroring you would - use hardware RAID underneath Lustre (best) - use software RAID (md, etc) underneath Lustre. So, your single OST would be in reality a mirrored device. Hope this helps. cliffw> > > I''m just downloading dome iso images right now since I know that one of the > freebsd mirror sites is just 3 hops away. > destroyer:/mnt/lustre # du -h > 2.9G ./FreeBSD-iso > 2.9G . > destroyer:/mnt/lustre # df -h > Filesystem Size Used Avail Use% Mounted on > /dev/sda1 4.0G 337M 3.5G 9% / > tmpfs 504M 8.0K 504M 1% /dev/shm > /dev/sda5 18G 145M 17G 1% /home > /dev/sda2 7.9G 1.4G 6.2G 19% /usr > config 68G 402M 64G 1% /mnt/lustre >
Ah. I see. Okay. So I''m reading it all wrong. My bad. Although it is giving me some ideas on planning out the array. I''m certainly planning on using hardware raid, and I''m going to play with an idea that I have that will give a pseudo raid of sorts. If it works out, I''ll post it. It''d be nice to have it all done through lustre though. Part of the reason my boss is shoving this onto me, is that we can use off the shelf hardware and bypassing the proprietary hardware solutions that tend to have limits of the size of the array. I think he''s worried about content availability if something goes wrong with just one volume or hardware goes bad after a few years. We don''t mind spending the money on hardware, we just want to make sure it''s available. On 12/13/05, cliff white <cliffw@clusterfs.com> wrote:> > Chris Bachmann wrote: > > I''m trying to set up a simple array with 1 client, 1 mds, and 2 ost''s. > What > > I would like to do is to have the 2 ost volumes mirror each other. I > used > > the lwizard and set the failover, but when i turn off the primary ost > and > > use the lconf --select command line referenced in > > https://wiki.clusterfs.com/lustre/LustreFailover, it doesn''t contain any > of > > the content of the one that I shut off (or at least, it''s not reflected > in > > my df -h). > > > > So it''s obvious that I''m doing something wrong here, I just don''t know > what. > > How can I make sure that the content is mirrored between the two ost > volumes > > so if one of them goes bye-bye, the content is still preserved? > > You would do that sort of mirroring below Lustre - Lustre creates a > single large filesystem image from multiple ost volumes. We don''t > do mirroring, or any sort of RAID. --failover is used when you have > multiple systems that share physical storage, which appears to be the > reverse of your case. In the failover case, the storage is only accessed > by one system, --failover is used to move the storage to a second system. > > Typically, if you need disk replication/mirroring you would > - use hardware RAID underneath Lustre (best) > - use software RAID (md, etc) underneath Lustre. > > So, your single OST would be in reality a mirrored device. > > Hope this helps. > cliffw > > > > > > > I''m just downloading dome iso images right now since I know that one of > the > > freebsd mirror sites is just 3 hops away. > > destroyer:/mnt/lustre # du -h > > 2.9G ./FreeBSD-iso > > 2.9G . > > destroyer:/mnt/lustre # df -h > > Filesystem Size Used Avail Use% Mounted on > > /dev/sda1 4.0G 337M 3.5G 9% / > > tmpfs 504M 8.0K 504M 1% /dev/shm > > /dev/sda5 18G 145M 17G 1% /home > > /dev/sda2 7.9G 1.4G 6.2G 19% /usr > > config 68G 402M 64G 1% /mnt/lustre > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20051213/6eb1fb67/attachment.html
On Dec 13, 2005 18:23 -0500, Chris Bachmann wrote:> I''m certainly planning on using hardware raid, and I''m going to play with > an idea that I have that will give a pseudo raid of sorts. If it works out, > I''ll post it.Are you going to try something like (g)nbd? If yes, please post all results (success or failure) as several people have discussed this approach but no feedback so far.> It''d be nice to have it all done through lustre though.We are working on Lustre-level data redundancy, probably available in the middle of next year.> Part of the reason my boss is shoving this onto me, is that we can use > off the shelf hardware and bypassing the proprietary hardware solutions > that tend to have limits of the size of the array.You can still use MD RAID for redundancy below the Lustre filesystems if you want to avoid hardware lock-in. Most RAID cards also allow the disks to be attached as JBOD (just a bunch of disks) and the kernel MD RAID aggregates them. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
I''ll give that a shot.I might also just try some rsync style copies since the writes will be rather infrequent in my environment. I think the hardware raid5 is the better of the solutions. I''ve seen some rather disasterous software raid problems over the past 8 years.Rare, but they do happen more frequently than hardware raid. On 12/13/05, Andreas Dilger <adilger@clusterfs.com> wrote:> > On Dec 13, 2005 18:23 -0500, Chris Bachmann wrote: > > I''m certainly planning on using hardware raid, and I''m going to play > with > > an idea that I have that will give a pseudo raid of sorts. If it works > out, > > I''ll post it. > > Are you going to try something like (g)nbd? If yes, please post all > results (success or failure) as several people have discussed this > approach but no feedback so far. > > > It''d be nice to have it all done through lustre though. > > We are working on Lustre-level data redundancy, probably available in the > middle of next year. > > > Part of the reason my boss is shoving this onto me, is that we can use > > off the shelf hardware and bypassing the proprietary hardware solutions > > that tend to have limits of the size of the array. > > You can still use MD RAID for redundancy below the Lustre filesystems > if you want to avoid hardware lock-in. Most RAID cards also allow the > disks to be attached as JBOD (just a bunch of disks) and the kernel MD > RAID aggregates them. > > Cheers, Andreas > -- > Andreas Dilger > Principal Software Engineer > Cluster File Systems, Inc. > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20051213/7524fec3/attachment.html
I''m trying to set up a simple array with 1 client, 1 mds, and 2 ost''s. What I would like to do is to have the 2 ost volumes mirror each other. I used the lwizard and set the failover, but when i turn off the primary ost and use the lconf --select command line referenced in https://wiki.clusterfs.com/lustre/LustreFailover, it doesn''t contain any of the content of the one that I shut off (or at least, it''s not reflected in my df -h). So it''s obvious that I''m doing something wrong here, I just don''t know what. How can I make sure that the content is mirrored between the two ost volumes so if one of them goes bye-bye, the content is still preserved? I''m just downloading dome iso images right now since I know that one of the freebsd mirror sites is just 3 hops away. destroyer:/mnt/lustre # du -h 2.9G ./FreeBSD-iso 2.9G . destroyer:/mnt/lustre # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 4.0G 337M 3.5G 9% / tmpfs 504M 8.0K 504M 1% /dev/shm /dev/sda5 18G 145M 17G 1% /home /dev/sda2 7.9G 1.4G 6.2G 19% /usr config 68G 402M 64G 1% /mnt/lustre -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.clusterfs.com/pipermail/lustre-discuss/attachments/20051213/43d9f4a2/attachment.html