Alexandr R. Ogurtzoff
2009-Jun-25 14:47 UTC
[Xen-users] Xen network tune for fileserver in DomU
Hello, gentlemen I need wise advice about how to tune network for performance usage file server under domU I use bridged network. On dom0 network settings are: auto eth0 iface eth0 inet static address 192.168.1.100 netmask 255.255.255.0 network 192.168.1.0 broadcast 192.168.1.255 up /etc/xen/scripts/network-bridge start bridge=br0 netdev=eth0 up ifconfig br0 up #up route add default gw 192.168.1.1 gateway 192.168.1.1 dns-nameserver 192.168.1.1 $ brctl show bridge name bridge id STP enabled interfaces br0 8000.000c76ba7b5e no peth0 vif1.0 vif2.0 I use two domU instances: app 192.168.1.110 file 192.168.1.140 ... and GlusterFS on file to export filesystem. then test it if bonnie I got next results: Version 1.03c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP app.cluster.xe 128M 5623 17 16612 5 1818 3 15254 46 26112 9 123.5 0 file.cluster.x 128M 9854 29 21547 4 3515 3 16907 44 30468 7 188.1 1 iscander 128M 11308 37 11344 1 555 0 10589 33 11306 1 70.1 0 file.cluster.x 128M 22408 74 23801 11 14326 4 18696 43 48676 4 545.8 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP app.cluster.xen 16 242 4 1106 5 300 4 242 4 1144 6 288 4 file.cluster.xen 16 873 8 4112 11 1009 6 792 8 4351 10 1165 8 iscander 16 565 2 2317 2 764 1 575 2 2546 2 755 1 file.cluster.xen 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ first one then run on app instance with /data mount from "file" by glusterfs second then test with /import mount from "file" by itself by glusterfs third line is then mount /data to my host in the same LAN (100Mbit) last is then test _local_ /data on "file" (ext3) localy I surprised why my external host works faster with mounted share then internals - I mean located on the same physical node VMs? How can I tune network stack for Xen and if it really Xen''s network implemenation penalty ? /etc/xen/app.cluster.xen # this is autogenerated by Ganeti, please do not edit # kernel = ''/boot/vmlinuz-2.6-xenU'' ramdisk = ''/boot/initrd-2.6-xenU'' memory = 256 vcpus = 1 name = ''app.cluster.xen'' vif = [''mac=aa:00:00:3e:98:e4, bridge=br0''] disk = [''phy:/var/run/ganeti/instance-disks/app.cluster.xen:0,sda,w''] root = ''/dev/xvda1'' on_poweroff = ''destroy'' on_reboot = ''restart'' on_crash = ''restart'' extra = ''console=hvc0'' /etc/xen/file.cluster.xen # this is autogenerated by Ganeti, please do not edit # kernel = ''/boot/vmlinuz-2.6-xenU'' ramdisk = ''/boot/initrd-2.6-xenU'' memory = 128 vcpus = 1 name = ''file.cluster.xen'' vif = [''mac=aa:00:00:d5:0f:6d, bridge=br0, ip=192.168.1.140''] disk = [''phy:/var/run/ganeti/instance-disks/file.cluster.xen:0,sda,w'',''phy:/var/run/ganeti/instance-disks/file.cluster.xen:1,sdb,w'',''phy:/var/run/ganeti/instance-disks/file.cluster.xen:2,sdc,w''] root = ''/dev/xvda1'' on_poweroff = ''destroy'' on_reboot = ''restart'' on_crash = ''restart'' extra = ''console=hvc0'' -- My best wishes. Alexandr Ogurtsov. Linux is very friendly it is just picky who its friends are _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Alexandr, Using Xen for a fileserver doesn''t make much sense. It''s an inefficient use of resources. Peter Sent from my iPhone On Jun 25, 2009, at 10:47 AM, "Alexandr R. Ogurtzoff" <iscander@gmail.com > wrote:> Hello, gentlemen > I need wise advice about how to tune network for performance usage > file server under domU > I use bridged network. > On dom0 network settings are: > auto eth0 > iface eth0 inet static > address 192.168.1.100 > netmask 255.255.255.0 > network 192.168.1.0 > broadcast 192.168.1.255 > up /etc/xen/scripts/network-bridge start bridge=br0 netdev=eth0 > up ifconfig br0 up > #up route add default gw 192.168.1.1 > gateway 192.168.1.1 > dns-nameserver 192.168.1.1 > > $ brctl show > bridge name bridge id STP enabled interfaces > br0 8000.000c76ba7b5e no peth0 > vif1.0 > vif2.0 > > I use two domU instances: > app 192.168.1.110 > file 192.168.1.140 > > ... and GlusterFS on file to export filesystem. > > then test it if bonnie I got next results: > > Version 1.03c ------Sequential Output------ --Sequential > Input- --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec > %CP /sec %CP > app.cluster.xe 128M 5623 17 16612 5 1818 3 15254 46 26112 > 9 123.5 0 > file.cluster.x 128M 9854 29 21547 4 3515 3 16907 44 30468 > 7 188.1 1 > iscander 128M 11308 37 11344 1 555 0 10589 33 11306 > 1 70.1 0 > file.cluster.x 128M 22408 74 23801 11 14326 4 18696 43 48676 > 4 545.8 1 > ------Sequential Create------ --------Random > Create-------- > -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- > files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec > %CP /sec %CP > app.cluster.xen 16 242 4 1106 5 300 4 242 4 1144 > 6 288 4 > file.cluster.xen 16 873 8 4112 11 1009 6 792 8 4351 10 1165 > 8 > iscander 16 565 2 2317 2 764 1 575 2 2546 > 2 755 1 > file.cluster.xen 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ ++ > + +++++ +++ > > > > first one then run on app instance with /data mount from "file" by > glusterfs > second then test with /import mount from "file" by itself by > glusterfs > third line is then mount /data to my host in the same LAN (100Mbit) > last is then test _local_ /data on "file" (ext3) localy > > I surprised why my external host works faster with mounted share then > internals - I mean located on the same physical node VMs? > How can I tune network stack for Xen and if it really Xen''s network > implemenation penalty ? > > /etc/xen/app.cluster.xen > # this is autogenerated by Ganeti, please do not edit > # > kernel = ''/boot/vmlinuz-2.6-xenU'' > ramdisk = ''/boot/initrd-2.6-xenU'' > memory = 256 > vcpus = 1 > name = ''app.cluster.xen'' > vif = [''mac=aa:00:00:3e:98:e4, bridge=br0''] > disk = [''phy:/var/run/ganeti/instance-disks/app.cluster.xen:0,sda,w''] > root = ''/dev/xvda1'' > on_poweroff = ''destroy'' > on_reboot = ''restart'' > on_crash = ''restart'' > extra = ''console=hvc0'' > > /etc/xen/file.cluster.xen > # this is autogenerated by Ganeti, please do not edit > # > kernel = ''/boot/vmlinuz-2.6-xenU'' > ramdisk = ''/boot/initrd-2.6-xenU'' > memory = 128 > vcpus = 1 > name = ''file.cluster.xen'' > vif = [''mac=aa:00:00:d5:0f:6d, bridge=br0, ip=192.168.1.140''] > disk = [''phy:/var/run/ganeti/instance-disks/file.cluster.xen: > 0,sda,w'',''phy:/var/run/ganeti/instance-disks/file.cluster.xen: > 1,sdb,w'',''phy:/var/run/ganeti/instance-disks/file.cluster.xen: > 2,sdc,w''] > root = ''/dev/xvda1'' > on_poweroff = ''destroy'' > on_reboot = ''restart'' > on_crash = ''restart'' > extra = ''console=hvc0'' > -- > My best wishes. > Alexandr Ogurtsov. > > Linux is very friendly it is just picky who its friends are > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Alexandr R. Ogurtzoff
2009-Jun-26 06:05 UTC
Re: [Xen-users] Xen network tune for fileserver in DomU
Peter, Could You elaborate Your opinion? Why not? What are cons? Why not to use one of DomU as shared storage for others DomUs,hosts on a same network or even on the same physical node? Facts and numbers would be greate. Does use DomO for same purpose have a sense? Are pros and cons for this solution discussable? 2009/6/25 Peter Booth <peter_booth@me.com>:> Alexandr, > > Using Xen for a fileserver doesn''t make much sense. It''s an inefficient use > of resources. > > Peter > > Sent from my iPhone > > On Jun 25, 2009, at 10:47 AM, "Alexandr R. Ogurtzoff" <iscander@gmail.com> > wrote: > >> Hello, gentlemen >> I need wise advice about how to tune network for performance usage >> file server under domU >> I use bridged network. >> On dom0 network settings are: >> auto eth0 >> iface eth0 inet static >> address 192.168.1.100 >> netmask 255.255.255.0 >> network 192.168.1.0 >> broadcast 192.168.1.255 >> up /etc/xen/scripts/network-bridge start bridge=br0 netdev=eth0 >> up ifconfig br0 up >> #up route add default gw 192.168.1.1 >> gateway 192.168.1.1 >> dns-nameserver 192.168.1.1 >> >> $ brctl show >> bridge name bridge id STP enabled interfaces >> br0 8000.000c76ba7b5e no peth0 >> vif1.0 >> vif2.0 >> >> I use two domU instances: >> app 192.168.1.110 >> file 192.168.1.140 >> >> ... and GlusterFS on file to export filesystem. >> >> then test it if bonnie I got next results: >> >> Version 1.03c ------Sequential Output------ --Sequential Input- >> --Random- >> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP >> /sec %CP >> app.cluster.xe 128M 5623 17 16612 5 1818 3 15254 46 26112 9 >> 123.5 0 >> file.cluster.x 128M 9854 29 21547 4 3515 3 16907 44 30468 7 >> 188.1 1 >> iscander 128M 11308 37 11344 1 555 0 10589 33 11306 1 >> 70.1 0 >> file.cluster.x 128M 22408 74 23801 11 14326 4 18696 43 48676 4 >> 545.8 1 >> ------Sequential Create------ --------Random >> Create-------- >> -Create-- --Read--- -Delete-- -Create-- --Read--- >> -Delete-- >> files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> /sec %CP >> app.cluster.xen 16 242 4 1106 5 300 4 242 4 1144 6 >> 288 4 >> file.cluster.xen 16 873 8 4112 11 1009 6 792 8 4351 10 >> 1165 8 >> iscander 16 565 2 2317 2 764 1 575 2 2546 2 >> 755 1 >> file.cluster.xen 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ >> +++++ +++ >> >> >> >> first one then run on app instance with /data mount from "file" by >> glusterfs >> second then test with /import mount from "file" by itself by glusterfs >> third line is then mount /data to my host in the same LAN (100Mbit) >> last is then test _local_ /data on "file" (ext3) localy >> >> I surprised why my external host works faster with mounted share then >> internals - I mean located on the same physical node VMs? >> How can I tune network stack for Xen and if it really Xen''s network >> implemenation penalty ? >> >> /etc/xen/app.cluster.xen >> # this is autogenerated by Ganeti, please do not edit >> # >> kernel = ''/boot/vmlinuz-2.6-xenU'' >> ramdisk = ''/boot/initrd-2.6-xenU'' >> memory = 256 >> vcpus = 1 >> name = ''app.cluster.xen'' >> vif = [''mac=aa:00:00:3e:98:e4, bridge=br0''] >> disk = [''phy:/var/run/ganeti/instance-disks/app.cluster.xen:0,sda,w''] >> root = ''/dev/xvda1'' >> on_poweroff = ''destroy'' >> on_reboot = ''restart'' >> on_crash = ''restart'' >> extra = ''console=hvc0'' >> >> /etc/xen/file.cluster.xen >> # this is autogenerated by Ganeti, please do not edit >> # >> kernel = ''/boot/vmlinuz-2.6-xenU'' >> ramdisk = ''/boot/initrd-2.6-xenU'' >> memory = 128 >> vcpus = 1 >> name = ''file.cluster.xen'' >> vif = [''mac=aa:00:00:d5:0f:6d, bridge=br0, ip=192.168.1.140''] >> disk >> [''phy:/var/run/ganeti/instance-disks/file.cluster.xen:0,sda,w'',''phy:/var/run/ganeti/instance-disks/file.cluster.xen:1,sdb,w'',''phy:/var/run/ganeti/instance-disks/file.cluster.xen:2,sdc,w''] >> root = ''/dev/xvda1'' >> on_poweroff = ''destroy'' >> on_reboot = ''restart'' >> on_crash = ''restart'' >> extra = ''console=hvc0'' >> -- >> My best wishes. >> Alexandr Ogurtsov. >> >> Linux is very friendly it is just picky who its friends are >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >-- My best wishes. Alexandr Ogurtsov. Linux is very friendly it is just picky who its friends are _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
<J.Witvliet@MINDEF.NL>
2009-Jun-29 09:14 UTC
RE: [Xen-users] Xen network tune for fileserver in DomU
-----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Peter Booth Sent: Thursday, June 25, 2009 7:27 PM To: Alexandr R. Ogurtzoff Cc: xen-users@lists.xensource.com Subject: Re: [Xen-users] Xen network tune for fileserver in DomU Alexandr, Using Xen for a fileserver doesn''t make much sense. It''s an inefficient use of resources. Peter .... In other threads people stated that fileservers (nfs/smb) should _not_ be done from the Dom-0. If you can''t use neither DOM-0 nor DOM-U as fileserver, isn''t that a waste of resources? Hans ______________________________________________________________________ Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is toegezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico''s verbonden aan het elektronisch verzenden van berichten. This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Stefan de Konink
2009-Jun-29 09:21 UTC
RE: [Xen-users] Xen network tune for fileserver in DomU
On Mon, 29 Jun 2009 J.Witvliet@MINDEF.NL wrote:> In other threads people stated that fileservers (nfs/smb) should _not_ > be done from the Dom-0. > If you can''t use neither DOM-0 nor DOM-U as fileserver, isn''t that a > waste of resources?What people are pointing out that running an actual /fileserver/ a thing that has harddisks at all, virtualised, performs significantly worse than a dedicated machine. Stefan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
It''s about the right tool for the job One could use a school bus to drive to the mall. but I wouldnt. Virtualization is great for business agility, its a great way to make use of unused compute resources, its a great way to affordably simulate a large network, a great way to partition systems that have different functions. Xen doesn''t help you build a (production) fileserver but it certainly hinders you. On Jun 29, 2009, at 5:14 AM, J.Witvliet@MINDEF.NL wrote:> > > -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Peter > Booth > Sent: Thursday, June 25, 2009 7:27 PM > To: Alexandr R. Ogurtzoff > Cc: xen-users@lists.xensource.com > Subject: Re: [Xen-users] Xen network tune for fileserver in DomU > > Alexandr, > > Using Xen for a fileserver doesn''t make much sense. It''s an > inefficient > use of resources. > > Peter > > .... > In other threads people stated that fileservers (nfs/smb) should _not_ > be done from the Dom-0. > If you can''t use neither DOM-0 nor DOM-U as fileserver, isn''t that a > waste of resources? > > Hans > > ______________________________________________________________________ > Dit bericht kan informatie bevatten die niet voor u is bestemd. > Indien u niet de geadresseerde bent of dit bericht abusievelijk aan > u is toegezonden, wordt u verzocht dat aan de afzender te melden en > het bericht te verwijderen. De Staat aanvaardt geen > aansprakelijkheid voor schade, van welke aard ook, die verband houdt > met risico''s verbonden aan het elektronisch verzenden van berichten. > > This message may contain information that is not intended for you. > If you are not the addressee or if this message was sent to you by > mistake, you are requested to inform the sender and delete the > message. The State accepts no liability for damage of any kind > resulting from the risks inherent in the electronic transmission of > messages. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Xen doesn''t help you build a (production) fileserver but it certainly > hinders you.We should have to look no further than the domain of your email address to call shenanigans on your comments. There is no such thing as a solution that doesn''t work well in a virtualized environment unless your particular needs out-strip the hardware or limits of the hypervisor. To the OP: Build your file server in Xen and test it thoroughly. If you find scalability issues, come to the list with specific problems (i.e., "with a xen 3.4 domU with 1GB ram and 2 cores, at 250 clients doing random read i/o on a set of 1000 files on ext3 with an lvm vbd, I run into i/o slowness") and we''ll see what knobs can be turned. John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Jun 29, 2009, at 4:23 PM, John Madden wrote:>> Xen doesn''t help you build a (production) fileserver but it certainly >> hinders you. > > We should have to look no further than the domain of your email > address > to call shenanigans on your comments. There is no such thing as aJohn, I have no idea what "shenanigans" you''re referring to - can you be explicit?> solution that doesn''t work well in a virtualized environment unless > your > particular needs out-strip the hardware or limits of the hypervisor.I didn''t say that it isn''t possible to use a (Xen) virtualized solution as a fileserver. I said that it doesn''t make sense. I''m a big fan of virtualized OSes - but I don''t see how Xen helps here. If I''m mistaken, let me know how and why - I''d rather learn something than argue a point. Han''s original question doesn''t exist in a vacuum. Our profession has an embarrassing history of hyping and overselling innovative technology, to the point that projects fail, businesses fail, and ultimately people lose jobs. You make a great point about the importance of asking specific problems. Most of my work is performance tuning and its a shame to see the amount of time and energy thats get wasted when people post vague questions, get vague replies, try out random suggestions, with predictably nebulous results. Peter> > To the OP: Build your file server in Xen and test it thoroughly. If > you > find scalability issues, come to the list with specific problems > (i.e., > "with a xen 3.4 domU with 1GB ram and 2 cores, at 250 clients doing > random read i/o on a set of 1000 files on ext3 with an lvm vbd, I run > into i/o slowness") and we''ll see what knobs can be turned. > > John > > > > > > -- > John Madden > Sr UNIX Systems Engineer > Ivy Tech Community College of Indiana > jmadden@ivytech.edu >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Luke S Crawford
2009-Jun-30 03:00 UTC
Re: [Xen-users] Xen network tune for fileserver in DomU
> I didn''t say that it isn''t possible to use a (Xen) virtualized > solution as a fileserver. > I said that it doesn''t make sense. I''m a big fan of virtualized OSes - > but I don''t > see how Xen helps here.It depends on what sort of fileserver you need. Personally, I don''t deploy hardware smaller than 8 cores/32GiB ram; which is kinda heavy for a fileserver but the cost and power draw difference between that and a 4 core/8GiB ram system isn''t huge, so I prefer to hack up a 32GiB/8 core box rather than pay for power for a bunch of slightly smaller boxes. The thing is, most fileservers don''t need much by way of CPU, but latency will kill you, so the key is to dedicate a cpu to the fileserver. As usual, dedicate a core to the Dom0, as well. If the DomU and the Dom0 both can run at the same time, you can run an ok fileserver. (of course, I''m assuming a paravirtualized guest. The overhead of emulated IO would kill you.) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Maybe Matching Threads
- How to boot Ubuntu 9.04(Jaunty) or Debian Lenny (vmlinuz-2.6.26-2-686-bigmem) as domU under Lenny''s vmlinuz-2.6.26-2-xen-686 as dom0. PVM''s initrd issue
- How to create a windows vm with ganeti2.
- debian 6 squeeze / xen 4 / ganeti troubles
- Guest to Host communication
- Introducing snf-image-creator