Hello list,
I''m running opensuse 11.1 dom0 + many 11.1 domU. For now PV guests
are
located on local drive with LVM. I''m in need to connect more dom0
machines to farm and for sure I need to move guests to the shared
storage. My choise of storage would be NFSv3/4, the question is - is
there
any blueprints on how to setup such infrastructure. PV guests must live
on NFS to make the live migration between dom0 hosts work.
Thanks in advance!
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Denis J. Cirulis wrote:> Hello list, > > I''m running opensuse 11.1 dom0 + many 11.1 domU. For now PV guests are > located on local drive with LVM. I''m in need to connect more dom0 > machines to farm and for sure I need to move guests to the shared > storage. My choise of storage would be NFSv3/4, the question is - is > there > any blueprints on how to setup such infrastructure. PV guests must live > on NFS to make the live migration between dom0 hosts work. > Thanks in advance!this is only practicable if you have very (very!) fast NFS servers (think NetApp middle-high models). far more efficient is to use block sharing protocols (iSCSI, AoE, gnbd, etc). -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Denis J. Cirulis wrote: >> I''m running opensuse 11.1 dom0 + many 11.1 domU. For now PV guests are >> located on local drive with LVM. I''m in need to connect more dom0 >> machines to farm and for sure I need to move guests to the shared >> storage. My choise of storage would be NFSv3/4> this is only practicable if you have very (very!) fast NFS servers (think > NetApp middle-high models). far more efficient is to use block sharing > protocols (iSCSI, AoE, gnbd, etc).That, or your guests uses disk I/O lightly :) You didn''t mention what your domU uses are. If it''s for I/O intensive database, remote storage (NFS or whatever) is probably a no-no. You might even want to stick with local storage due to its performance. However, if you use it for light-wight web servers, or DNS/DHCP servers, then NFS is probably fine. PS: since you''re looking for a NAS anyway, I suggest you try looking at Solaris 10 or opensolaris, which has zfs volumes (exportable via iscsi). This should make your job a lot easier as it supports snapshot and clones from storage server, compression, and checksum for data integrity. Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I tried AoE and iSCSI already, software iscsi target and initiator gives me high cpu load on both domU and target. AoE was less resource hungry. On Sat, Feb 28, 2009 at 11:29:22AM -0500, Javier Guerra Giraldez wrote:> Denis J. Cirulis wrote: > > Hello list, > > > > I''m running opensuse 11.1 dom0 + many 11.1 domU. For now PV guests are > > located on local drive with LVM. I''m in need to connect more dom0 > > machines to farm and for sure I need to move guests to the shared > > storage. My choise of storage would be NFSv3/4, the question is - is > > there > > any blueprints on how to setup such infrastructure. PV guests must live > > on NFS to make the live migration between dom0 hosts work. > > Thanks in advance! > > this is only practicable if you have very (very!) fast NFS servers (think > NetApp middle-high models). far more efficient is to use block sharing > protocols (iSCSI, AoE, gnbd, etc). > > -- > Javier_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
My guests aren''t very I/O intensive, mail/web/dns/dhcp/ldap servers. For a NAS I have two options, yes opensolaris or freenas appliance. On Sun, Mar 01, 2009 at 02:38:46AM +0700, Fajar A. Nugraha wrote:> > Denis J. Cirulis wrote: > >> I''m running opensuse 11.1 dom0 + many 11.1 domU. For now PV guests are > >> located on local drive with LVM. I''m in need to connect more dom0 > >> machines to farm and for sure I need to move guests to the shared > >> storage. My choise of storage would be NFSv3/4 > > > this is only practicable if you have very (very!) fast NFS servers (think > > NetApp middle-high models). far more efficient is to use block sharing > > protocols (iSCSI, AoE, gnbd, etc). > > That, or your guests uses disk I/O lightly :) > > You didn''t mention what your domU uses are. If it''s for I/O intensive > database, remote storage (NFS or whatever) is probably a no-no. You > might even want to stick with local storage due to its performance. > However, if you use it for light-wight web servers, or DNS/DHCP > servers, then NFS is probably fine. > > PS: since you''re looking for a NAS anyway, I suggest you try looking > at Solaris 10 or opensolaris, which has zfs volumes (exportable via > iscsi). This should make your job a lot easier as it supports snapshot > and clones from storage server, compression, and checksum for data > integrity. > > Regards, > > Fajar_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sun, Mar 1, 2009 at 5:00 AM, Denis J. Cirulis <denis@opensource.lv> wrote:> I tried AoE and iSCSI already, software iscsi target and initiator gives > me high cpu load on both domU and target. AoE was less resource hungry.High domU CPU? Seriously? Who''s importing the iscsi target, dom0 or domU? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Dom0. On Sun, Mar 01, 2009 at 09:19:57AM +0700, Fajar A. Nugraha wrote:> On Sun, Mar 1, 2009 at 5:00 AM, Denis J. Cirulis <denis@opensource.lv> wrote: > > I tried AoE and iSCSI already, software iscsi target and initiator gives > > me high cpu load on both domU and target. AoE was less resource hungry. > > High domU CPU? Seriously? Who''s importing the iscsi target, dom0 or domU?_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sun, Mar 1, 2009 at 2:51 PM, Denis J. Cirulis <denis@opensource.lv> wrote:> Dom0.That''s strange. I tested domU on RHEL dom0, with storage located on iscsi-exported zfs volume from opensolaris, there''s not much difference in domU CPU usage. I did notice substantial difference in domU I/O throughput due to the fact that I''m using 100Mbps switch instead of 1Gbps. What does "top" on domU say? Is the high load on "us", or is it on "sy" or "wa"? What does "iostat -x 3" on dom0 say? Is the iscsi-imported disk experiencing 100% util? If it IS I/O throughput problem, then iscsi, aoe, or nfs should be same: none of them would provide acceptable performance. If it''s something else, then you can safely use nfs.> > On Sun, Mar 01, 2009 at 09:19:57AM +0700, Fajar A. Nugraha wrote: >> On Sun, Mar 1, 2009 at 5:00 AM, Denis J. Cirulis <denis@opensource.lv> wrote: >> > I tried AoE and iSCSI already, software iscsi target and initiator gives >> > me high cpu load on both domU and target. AoE was less resource hungry. >> >> High domU CPU? Seriously? Who''s importing the iscsi target, dom0 or domU? >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I''ve tried opensuse domU on opensuse dom0 connected 20GB zfs volume off FreeNAS beta using iscsi. Yes, it works normally, no cpu overload. What is the good style of using iscsi for guests, I need to discover all targets on all dom0 hosts and then start guest on selected machines (as I know target can only be used once) ? How to make migration between dom0 hosts possible ? On Sun, Mar 01, 2009 at 05:05:23PM +0700, Fajar A. Nugraha wrote:> On Sun, Mar 1, 2009 at 2:51 PM, Denis J. Cirulis <denis@opensource.lv> wrote: > > Dom0. > > That''s strange. I tested domU on RHEL dom0, with storage located on > iscsi-exported zfs volume from opensolaris, there''s not much > difference in domU CPU usage. I did notice substantial difference in > domU I/O throughput due to the fact that I''m using 100Mbps switch > instead of 1Gbps. > > What does "top" on domU say? Is the high load on "us", or is it on "sy" or "wa"? > What does "iostat -x 3" on dom0 say? Is the iscsi-imported disk > experiencing 100% util? > > If it IS I/O throughput problem, then iscsi, aoe, or nfs should be > same: none of them would provide acceptable performance. If it''s > something else, then you can safely use nfs. > > > > > On Sun, Mar 01, 2009 at 09:19:57AM +0700, Fajar A. Nugraha wrote: > >> On Sun, Mar 1, 2009 at 5:00 AM, Denis J. Cirulis <denis@opensource.lv> wrote: > >> > I tried AoE and iSCSI already, software iscsi target and initiator gives > >> > me high cpu load on both domU and target. AoE was less resource hungry. > >> > >> High domU CPU? Seriously? Who''s importing the iscsi target, dom0 or domU? > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sun, Mar 1, 2009 at 6:09 PM, Denis J. Cirulis <denis@opensource.lv> wrote:> I''ve tried opensuse domU on opensuse dom0 connected 20GB zfs volume off > FreeNAS beta using iscsi. Yes, it works normally, no cpu overload.So I assume what you wrote about "software iscsi target and initiator gives me high cpu load on both domU and target" was from a different setup, and that FreeNAS with zfs gives acceptable performance, right?> What is the good style of using iscsi for guests, I need to discover all > targets on all dom0 hosts and then start guest on selected machines (asSomething like that. Whatever shared storage you use, it needs to be available on all dom0s. For iscsi, this means importing all iscsi shares and reference them on dom0 using a common path on all dom0s (/dev/disk/by-path or by-id should be good). As an alternative, there''s a block-iscsi helper script (try google search) that can be used easily.> I know target can only be used once) ? How to make migration between > dom0 hosts possible ?On all shared storage setup, you have to make sure that domUs only started on one dom0 at a time. You can use some tools to do this (like linux cluster''s rgmanager) or (when you only have several domUs) it might be pereferable to do it manually. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sun, Mar 01, 2009 at 08:06:24PM +0700, Fajar A. Nugraha wrote:> On Sun, Mar 1, 2009 at 6:09 PM, Denis J. Cirulis <denis@opensource.lv> wrote: > > I''ve tried opensuse domU on opensuse dom0 connected 20GB zfs volume off > > FreeNAS beta using iscsi. Yes, it works normally, no cpu overload. > > So I assume what you wrote about "software iscsi target and initiator gives > me high cpu load on both domU and target" was from a different setup, > and that FreeNAS with zfs gives acceptable performance, right?Yes, former setup was ubuntu linux with iscsi-target + opensuse with open-iscsi initiator. Now with FreeNAS on 1 GiB interface with jumbo frames enabled (as well as initiator) looks like cpu load is ok both on opensuse initiator and freenas storage server. Acceptable performace is 107 MB/s (which is very very good) when connecting one initiator, with few initiators each is getting about 45-65 MB/s what is also very very good for low I/O domUs. Fajar, You''re using zfs on opensolaris, can You backup domUs when they''re running ? I mean You are creating snapshot on the volume You want to backup and can do a dd to another storage server or backup solution ?> > What is the good style of using iscsi for guests, I need to discover all > > targets on all dom0 hosts and then start guest on selected machines (as > > Something like that. Whatever shared storage you use, it needs to be > available on all dom0s. For iscsi, this means importing all iscsi > shares and reference them on dom0 using a common path on all dom0s > (/dev/disk/by-path or by-id should be good). > > As an alternative, there''s a block-iscsi helper script (try google > search) that can be used easily. > > > I know target can only be used once) ? How to make migration between > > dom0 hosts possible ? > > On all shared storage setup, you have to make sure that domUs only > started on one dom0 at a time. You can use some tools to do this (like > linux cluster''s rgmanager) or (when you only have several domUs) it > might be pereferable to do it manually._______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sun, Mar 1, 2009 at 10:59 PM, Denis J. Cirulis <denis@opensource.lv> wrote:> can You backup domUs when > they''re running ? I mean You are creating snapshot on the volume You > want to backup and can do a dd to another storage server or backup > solution ? >What I did was do a zfs snapshot, "zfs send" + compress the resulting stream, and store the resulting file. It''s actually better than dd because "zfs send" only send used blocks, while dd sends everything. There''s a long discussion about whether or not "zfs send" should be used as a backup method, but it''s working great for me so far. Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users