I wrote a really long e-mail but realised I could ask this question far far easier, if it doesn't make sense, the original e-mail is bellow Can I use ZFS to create a multinode storage area. Multiple HDD's in Multiple servers to create one target of, for example, //officestorage Allowing me to expand the storage space when needed and clients being able to retrieve data (like RAID0 but over devices not HDD) Here is an example I found which is where I'm getting some ideas from http://www.howtoforge.com/how-to-build-a-low-cost-san-p3 Any pointers would be helpful, Thanks --------------------------------------------------------------------- I've been searching around and I am finding myself confused and reading conflicting information. I would like to build a Storage system where by I have multiple nodes. At the minute I have a number or NAS's which work well and RAID6 works well in the situation we have, but unfortunately it's a short-term solution I inherited and once you crunch the numbers of 6 devices with 6 HDD's in RAID6 you realise how much space you have wasted then say, 1 device of RAID6 with 36 HDD's (the saving is a fair few TB) There are other issues as well, increasing the size, 3rd party NAS device features missing which other storage devices have...etc so I looked around and my grand idea was basically this; "Build a system where I can have multiple nodes which create one target (we will use //officestorage for our example) as opposed to //nas1/ which is of course 1 device. Using multiple nodes will allow us to add a new device, thus increasing the space available but the target will always be the same and to the client nothing has change (other then available space) (Think of this as RAID0). Multiple nodes will also allow for redundancy across devices (think RAID1) and give better IO as it's multiple devices and not just 1 device. I could have devices in different locations so a whole building could burn down and still not lose the data" After looking around I found this https://help.ubuntu.com/community/HighlyAvailableAoETarget Which looks quite good, it's basically RAID1 but instead of HDD's it's across servers, I've used DRBD and it worked well, but this doesn't give me better IO as only 1 device is live. I then found this http://www.howtoforge.com/how-to-build-a-low-cost-san-p3 Which looks fantastic, though I will need a master server to create the RAID0 and RAID1 across the multiple nodes and then share this out, which is ok but I would need a hot swap master server, so I'm looking at two of those. I then started thinking about ZFS, I've heard lots of good things about it in the past and thinking can ZFS do what I want. I have read some things which say it can do what that 2nd link does and others which say it can't. Everything I come across is about using just 1 device and I could build 1 device with DRBD, but that doesn't help, nor will it allow me to expand it (if your server runs out of physical space you can't add more HDD's. Anyone point me in the right direction?? Thanks
On 24/03/2010 15:47, Michal wrote:> I wrote a really long e-mail but realised I could ask this question far > far easier, if it doesn't make sense, the original e-mail is bellow > > Can I use ZFS to create a multinode storage area. Multiple HDD's in > Multiple servers to create one target of, for example, //officestorage > Allowing me to expand the storage space when needed and clients being > able to retrieve data (like RAID0 but over devices not HDD) >At least in theory you could use geom_gate and zfs I suppose, never tried it though. ggatec(8), ggated(8) are your friends for that. Vince> Here is an example I found which is where I'm getting some ideas from > http://www.howtoforge.com/how-to-build-a-low-cost-san-p3 > > Any pointers would be helpful, > Thanks > > > --------------------------------------------------------------------- > > > I've been searching around and I am finding myself confused and reading > conflicting information. I would like to build a Storage system where by > I have multiple nodes. At the minute I have a number or NAS's which work > well and RAID6 works well in the situation we have, but unfortunately > it's a short-term solution I inherited and once you crunch the numbers > of 6 devices with 6 HDD's in RAID6 you realise how much space you have > wasted then say, 1 device of RAID6 with 36 HDD's (the saving is a fair > few TB) > > There are other issues as well, increasing the size, 3rd party NAS > device features missing which other storage devices have...etc so I > looked around and my grand idea was basically this; > > "Build a system where I can have multiple nodes which create one target > (we will use //officestorage for our example) as opposed to //nas1/ > which is of course 1 device. Using multiple nodes will allow us to add a > new device, thus increasing the space available but the target will > always be the same and to the client nothing has change (other then > available space) (Think of this as RAID0). Multiple nodes will also > allow for redundancy across devices (think RAID1) and give better IO as > it's multiple devices and not just 1 device. I could have devices in > different locations so a whole building could burn down and still not > lose the data" > > > After looking around I found this > > https://help.ubuntu.com/community/HighlyAvailableAoETarget > > Which looks quite good, it's basically RAID1 but instead of HDD's it's > across servers, I've used DRBD and it worked well, but this doesn't give > me better IO as only 1 device is live. > > I then found this > > http://www.howtoforge.com/how-to-build-a-low-cost-san-p3 > > Which looks fantastic, though I will need a master server to create the > RAID0 and RAID1 across the multiple nodes and then share this out, which > is ok but I would need a hot swap master server, so I'm looking at two > of those. I then started thinking about ZFS, I've heard lots of good > things about it in the past and thinking can ZFS do what I want. I have > read some things which say it can do what that 2nd link does and others > which say it can't. Everything I come across is about using just 1 > device and I could build 1 device with DRBD, but that doesn't help, nor > will it allow me to expand it (if your server runs out of physical space > you can't add more HDD's. > > Anyone point me in the right direction?? > > Thanks > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >
On Wed, Mar 24, 2010 at 8:47 AM, Michal <michal@ionic.co.uk> wrote:> I wrote a really long e-mail but realised I could ask this question far > far easier, if it doesn't make sense, the original e-mail is bellow > > Can I use ZFS to create a multinode storage area. Multiple HDD's in > Multiple servers to create one target of, for example, //officestorage > Allowing me to expand the storage space when needed and clients being > able to retrieve data (like RAID0 but over devices not HDD) > > Here is an example I found which is where I'm getting some ideas from > http://www.howtoforge.com/how-to-build-a-low-cost-san-p3 > > Horribly, horribly, horribly complex. But, then, that's the Linux world.:) Server 1: bunch of disks exported via iSCSI Server 2: bunch of disks exported via iSCSI Server 3: bunch of disks exported via iSCSI "SAN" box: uses all those iSCSI exports to create a ZFS pool Use 1 iSCSI export from each server to create a raidz vdev. Or multiple mirror vdevs. When you need more storage, just add another server full of disks, export them via iSCSI to the "SAN" box, and expand the ZFS pool. And, if you need fail-over, on your "SAN" box, you can use HAST at the lower layers (currently only available in 9-CURRENT) to mirror the storage across two systems, and use CARP to provide a single IP for the two boxes. ---------------------------------------------------------------------> I've been searching around and I am finding myself confused and reading > conflicting information. I would like to build a Storage system where by > I have multiple nodes. At the minute I have a number or NAS's which work > well and RAID6 works well in the situation we have, but unfortunately > it's a short-term solution I inherited and once you crunch the numbers > of 6 devices with 6 HDD's in RAID6 you realise how much space you have > wasted then say, 1 device of RAID6 with 36 HDD's (the saving is a fair > few TB) > > Yes, you save space, but your throughput will be horribly horribly horriblylow. RAID arrays should be narrow (1-9 disks), not wide (30+ disks), and then combined into a larger array (multiple small RAID6 arrays joined into a RAID0 stripe). -- Freddie Cash fjwcash@gmail.com
On Wed, Mar 24, 2010 at 10:04 AM, Michael Loftis <mloftis@wgops.com> wrote:> --On Wednesday, March 24, 2010 9:20 AM -0700 Freddie Cash < > fjwcash@gmail.com> wrote: > On Wed, Mar 24, 2010 at 8:47 AM, Michal <michal@ionic.co.uk> wrote: > >> >> I wrote a really long e-mail but realised I could ask this question far >>> far easier, if it doesn't make sense, the original e-mail is bellow >>> >>> Can I use ZFS to create a multinode storage area. Multiple HDD's in >>> Multiple servers to create one target of, for example, //officestorage >>> Allowing me to expand the storage space when needed and clients being >>> able to retrieve data (like RAID0 but over devices not HDD) >>> >>> Here is an example I found which is where I'm getting some ideas from >>> http://www.howtoforge.com/how-to-build-a-low-cost-san-p3 >>> >>> Horribly, horribly, horribly complex. But, then, that's the Linux world. >>> >> :) >> >> Server 1: bunch of disks exported via iSCSI >> Server 2: bunch of disks exported via iSCSI >> Server 3: bunch of disks exported via iSCSI >> >> "SAN" box: uses all those iSCSI exports to create a ZFS pool >> >> Use 1 iSCSI export from each server to create a raidz vdev. Or multiple >> mirror vdevs. When you need more storage, just add another server full of >> disks, export them via iSCSI to the "SAN" box, and expand the ZFS pool. >> >> And, if you need fail-over, on your "SAN" box, you can use HAST at the >> lower layers (currently only available in 9-CURRENT) to mirror the >> storage across two systems, and use CARP to provide a single IP for the >> two boxes. >> > > If you were to do something like this, I'd make sure to have a fast local > ZIL (log) device on the head node. That would reduce latency for writes, > you might also do the same for reads. Then your bulk storage comes from the > iSCSI boxes. >Yes, that would be helpful (mirrored slogs, until we get slog removal support). As would an L2ARC (cache) device in the head node. As well as lots and lots and lots of RAM. And as fast of ethernet NICs as you can get between the head node and the storage nodes. And, and, and, and ... :) -- Freddie Cash fjwcash@gmail.com