zfs discuss - Mar 2009 - How to make a ZFS pool with discs of the other machines of the LAN?

Hi!

 Today I have ten computers with Xen and Linux, each with 2 discs of 500G in
raid1, each node sees only its own raid1 volume, I do not have live motion of my
virtual machines... and moving the data from one hypervisor to another is a pain
task...

 Now that I discovered this awesome file system! I want that the ZFS manages all
my discs in a network environment.

 But I don''t know the best way to make a pool using all my 20 discs in
one big pool with 10T of capacity.

 My first contact with Solaris, was with the OpenSolaris 2008.11, as a virtual
machine (paravirtual domU) on a Linux (Debian 5.0) dom0. I also have more
opensolaris on real machines to make the tests...

 I''m thinking in export all my 20 discs, through the AoE protocol, and
in my dom0 that I''m running the opensolaris domU (in HA through the
Xen), I will make the configuration file for it (zfs01.cfg) with 20 block
devices of 500G and inside the opensolaris domu, I will share the pool via iSCSI
targets and/or NFS back to the domUs of my cluster...  Is this a good idea?

 Or it''s better install one OpenSolaris per hypervisor, and then, let
to the ten opensolaris domUs, the task to export and share the discs by its own
way (not using the AoE anymore)? But I imagine that I need only one opensolaris
(in HA) with one big pool...

 Resuming, how is the best way to make a pool with twenty discs of the network?

 Thanks for any enlightenment!

Regards,
Thiago

On Sat, Mar 7, 2009 at 11:05 AM, Thiago C. M. Cordeiro | World Web
<thiago.martins at worldweb.com.br> wrote:
> Hi!
>
> ?Today I have ten computers with Xen and Linux, each with 2 discs of 500G
in raid1, each node sees only its own raid1 volume, I do not have live motion of
my virtual machines... and moving the data from one hypervisor to another is a
pain task...
>
> ?Now that I discovered this awesome file system! I want that the ZFS
manages all my discs in a network environment.
>
> ?But I don''t know the best way to make a pool using all my 20
discs in one big pool with 10T of capacity.
>
> ?My first contact with Solaris, was with the OpenSolaris 2008.11, as a
virtual machine (paravirtual domU) on a Linux (Debian 5.0) dom0. I also have
more opensolaris on real machines to make the tests...
>
> ?I''m thinking in export all my 20 discs, through the AoE protocol,
and in my dom0 that I''m running the opensolaris domU (in HA through the
Xen), I will make the configuration file for it (zfs01.cfg) with 20 block
devices of 500G and inside the opensolaris domu, I will share the pool via iSCSI
targets and/or NFS back to the domUs of my cluster... ?Is this a good idea?
I share a three disk pool over NFS for some VMWare ESXi based hosting.
There is considerably high disk I/O caused by the apps that run on
these VMs. ZFS + NFS is working fine for me.

I intend to experiment with iSCSI later when I free up some machines
for such an experiment.

-- Sriram

On Sat, Mar 7, 2009 at 11:23 AM, Sriram Narayanan <sriram at belenix.org>
wrote:
> On Sat, Mar 7, 2009 at 11:05 AM, Thiago C. M. Cordeiro | World Web
> <thiago.martins at worldweb.com.br> wrote:
>> Hi!
>>
>> ?Today I have ten computers with Xen and Linux, each with 2 discs of
500G in raid1, each node sees only its own raid1 volume, I do not have live
motion of my virtual machines... and moving the data from one hypervisor to
another is a pain task...
>>
>> ?Now that I discovered this awesome file system! I want that the ZFS
manages all my discs in a network environment.
>>
>> ?But I don''t know the best way to make a pool using all my 20
discs in one big pool with 10T of capacity.
>>
>> ?My first contact with Solaris, was with the OpenSolaris 2008.11, as a
virtual machine (paravirtual domU) on a Linux (Debian 5.0) dom0. I also have
more opensolaris on real machines to make the tests...
>>
>> ?I''m thinking in export all my 20 discs, through the AoE
protocol, and in my dom0 that I''m running the opensolaris domU (in HA
through the Xen), I will make the configuration file for it (zfs01.cfg) with 20
block devices of 500G and inside the opensolaris domu, I will share the pool via
iSCSI targets and/or NFS back to the domUs of my cluster... ?Is this a good
idea?
>
> I share a three disk pool over NFS for some VMWare ESXi based hosting.
> There is considerably high disk I/O caused by the apps that run on
> these VMs. ZFS + NFS is working fine for me.
>
> I intend to experiment with iSCSI later when I free up some machines
> for such an experiment.
My only tip for Linux based iSCSI clients would be that you should use
CentOS 5.2 based dm (device multi path) for connecting to iSCSI using
multipath. One of the dm developers at Netapp recently gave a talk at
a LUG meet we hosted at our office, and he explained how Netapp and
Redhat put in a lot of effort to ensure that dm on Redhat is stable.

After some experiments at work, I have concluded that dm on CentOS 5.2
is much more stable than on any other Linux distro.
>
> -- Sriram
>

On Sat, Mar 7, 2009 at 11:26 AM, Sriram Narayanan <sriram at belenix.org>
wrote:
> On Sat, Mar 7, 2009 at 11:23 AM, Sriram Narayanan <sriram at
belenix.org> wrote:
<snip/>
>> I intend to experiment with iSCSI later when I free up some machines
>> for such an experiment.
>
> My only tip for Linux based iSCSI clients would be that you should use
> CentOS 5.2 based dm (device multi path) for connecting to iSCSI using
> multipath.
Sorry, dm -> device mapper. multipath is one of the features that dm
provides.

-- Sriram

Hello!

 I want to know something... It is okay export the AoE discs to the virtual
OpenSolaris machine through dom0?

 For example, in dom0 I''d like to have:

from node01 via AoE to dom0 -> opensolaris01 domU
500G /dev/ether/e1.0 (c3d1 /xpvd/xdf at 1)
500G /dev/ether/e1.1 (c3d2 /xpvd/xdf at 2)

from node02 via AoE to dom0 -> opensolaris01 domU
500G /dev/ether/e2.0 (c3d3 /xpvd/xdf at 3)
500G /dev/ether/e2.1 (c3d4 /xpvd/xdf at 4)

from node03 via AoE to dom0 -> opensolaris01 domU
500G /dev/ether/e3.0 (c3d5 /xpvd/xdf at 5)
500G /dev/ether/e3.1 (c3d6 /xpvd/xdf at 6)
...
from node07 via AoE to dom0 -> opensolaris01 domU
500G /dev/ether/e7.0 (c3d5 /xpvd/xdf at 13)
500G /dev/ether/e7.1 (c3d6 /xpvd/xdf at 14)
...

 The AoE devices will appear as discs in the virtual OpenSolaris machine, then I
can make my big ZFS pool and export it via iSCSI/NFS/SMB using another dom0
ethernet interface...

 The AoE arrives to dom0 via eth1 and appear as discs in opensolaris01 domU, and
then, the pool is exported as iSCSI, etc, via opensolaris01 xnf0 interface
-bridged to- eth0 of the dom0.


-
 Thiago

----- "Sriram Narayanan" <sriram at belenix.org> escreveu:
> On Sat, Mar 7, 2009 at 11:05 AM, Thiago C. M. Cordeiro | World Web
> <thiago.martins at worldweb.com.br> wrote:
> > Hi!
> >
> > ?Today I have ten computers with Xen and Linux, each with 2 discs of
> 500G in raid1, each node sees only its own raid1 volume, I do not have
> live motion of my virtual machines... and moving the data from one
> hypervisor to another is a pain task...
> >
> > ?Now that I discovered this awesome file system! I want that the ZFS
> manages all my discs in a network environment.
> >
> > ?But I don''t know the best way to make a pool using all my 20
discs
> in one big pool with 10T of capacity.
> >
> > ?My first contact with Solaris, was with the OpenSolaris 2008.11, as
> a virtual machine (paravirtual domU) on a Linux (Debian 5.0) dom0. I
> also have more opensolaris on real machines to make the tests...
> >
> > ?I''m thinking in export all my 20 discs, through the AoE
protocol,
> and in my dom0 that I''m running the opensolaris domU (in HA
through
> the Xen), I will make the configuration file for it (zfs01.cfg) with
> 20 block devices of 500G and inside the opensolaris domu, I will share
> the pool via iSCSI targets and/or NFS back to the domUs of my
> cluster... ?Is this a good idea?
> 
> I share a three disk pool over NFS for some VMWare ESXi based
> hosting.
> There is considerably high disk I/O caused by the apps that run on
> these VMs. ZFS + NFS is working fine for me.
> 
> I intend to experiment with iSCSI later when I free up some machines
> for such an experiment.
> 
> -- Sriram

Sriram,

----- "Sriram Narayanan" <sriram at belenix.org> escreveu:
> On Sat, Mar 7, 2009 at 11:05 AM, Thiago C. M. Cordeiro | World Web
> <thiago.martins at worldweb.com.br> wrote:
> > Hi!
> >
> > ?Today I have ten computers with Xen and Linux, each with 2 discs of
> 500G in raid1, each node sees only its own raid1 volume, I do not have
> live motion of my virtual machines... and moving the data from one
> hypervisor to another is a pain task...
> >
> > ?Now that I discovered this awesome file system! I want that the ZFS
> manages all my discs in a network environment.
> >
> > ?But I don''t know the best way to make a pool using all my 20
discs
> in one big pool with 10T of capacity.
> >
> > ?My first contact with Solaris, was with the OpenSolaris 2008.11, as
> a virtual machine (paravirtual domU) on a Linux (Debian 5.0) dom0. I
> also have more opensolaris on real machines to make the tests...
> >
> > ?I''m thinking in export all my 20 discs, through the AoE
protocol,
> and in my dom0 that I''m running the opensolaris domU (in HA
through
> the Xen), I will make the configuration file for it (zfs01.cfg) with
> 20 block devices of 500G and inside the opensolaris domu, I will share
> the pool via iSCSI targets and/or NFS back to the domUs of my
> cluster... ?Is this a good idea?
> 
> I share a three disk pool over NFS for some VMWare ESXi based
> hosting.
This three discs are on the same machine or in three distinct?
I mean, your pool have three local or remote discs?
> There is considerably high disk I/O caused by the apps that run on
> these VMs. ZFS + NFS is working fine for me.
> 
> I intend to experiment with iSCSI later when I free up some machines
> for such an experiment.
> 
> -- Sriram
Thanks!
-
Thiago

I''ll drop AoE in favor of the iSCSI to export my 20 discs from Linux
Debian Lenny to OpenSolaris 2008.11. Now, I believe this setup will be more
compatible with OpenSolaris OS.

Thanks!
Thiago

----- "Thiago Martins" <thiago.martins at worldweb.com.br>
escreveu:
> Sriram,
> 
> ----- "Sriram Narayanan" <sriram at belenix.org> escreveu:
> 
> > On Sat, Mar 7, 2009 at 11:05 AM, Thiago C. M. Cordeiro | World Web
> > <thiago.martins at worldweb.com.br> wrote:
> > > Hi!
> > >
> > > ?Today I have ten computers with Xen and Linux, each with 2 discs
> of
> > 500G in raid1, each node sees only its own raid1 volume, I do not
> have
> > live motion of my virtual machines... and moving the data from one
> > hypervisor to another is a pain task...
> > >
> > > ?Now that I discovered this awesome file system! I want that the
> ZFS
> > manages all my discs in a network environment.
> > >
> > > ?But I don''t know the best way to make a pool using all
my 20
> discs
> > in one big pool with 10T of capacity.
> > >
> > > ?My first contact with Solaris, was with the OpenSolaris 2008.11,
> as
> > a virtual machine (paravirtual domU) on a Linux (Debian 5.0) dom0. I
> > also have more opensolaris on real machines to make the tests...
> > >
> > > ?I''m thinking in export all my 20 discs, through the AoE
protocol,
> > and in my dom0 that I''m running the opensolaris domU (in HA
through
> > the Xen), I will make the configuration file for it (zfs01.cfg) with
> > 20 block devices of 500G and inside the opensolaris domu, I will
> share
> > the pool via iSCSI targets and/or NFS back to the domUs of my
> > cluster... ?Is this a good idea?
> >
> > I share a three disk pool over NFS for some VMWare ESXi based
> > hosting.
> This three discs are on the same machine or in three distinct?
> I mean, your pool have three local or remote discs?
> 
> > There is considerably high disk I/O caused by the apps that run on
> > these VMs. ZFS + NFS is working fine for me.
> >
> > I intend to experiment with iSCSI later when I free up some machines
> > for such an experiment.
> >
> > -- Sriram
> 
> Thanks!
> -
> Thiago
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss