Felix Botner
2010-Jul-20 08:46 UTC
[Xen-users] Xen 3.4.3 - bad IO performance on drbd devices
Hi everyone,
i have two servers installed with a debian/lenny based os (64bit), a
debian/sid based kernel 2.6.32-xen-amd64 and xen 3.4.3-4. Each server has two
drbd devices (protocol c, formatted with ext4) and is primary for one of
them. Every drbd pair has a dedicated network interface (a bond mode 0
interface with two 1000 Mbps cards).
+------------------------------------------+
| server 1 |
| +----------------+ +------------------+ |
| | primary drbd0 | | secondary drbd1 | |
| | (protocol c) | | (protocol c) | |
| +----------------+ +------------------+ |
| | | |
| | | |
| +----------------+ +------------------+ |
| | +----+ +----+ | | +----+ +----+ | |
| | |eth2| |eth3| | | |eth4| |eth5| | |
| | +----+ +----+ | | +----+ +----+ | |
| | | | | |
| | bond0 (mode 0) | | bond1 (mode 0) | |
| +-------|--------+ +--------|---------+ |
+---------|--------------------|-----------+
| |
| |
+---------|--------------------|-----------+
| +-------|--------+ +--------|---------+ |
| | bond0 (mode 0) | | bond1 (mode 0) | |
| | | | | |
| | +----+ +----+ | | +----+ +----+ | |
| | |eth2| |eth3| | | |eth4| |eth5| | |
| | +----+ +----+ | | +----+ +----+ | |
| +----------------+ +------------------+ |
| | | |
| | | |
| +-----------------+ +-----------------+ |
| | (protocol c) | | (protocol c) | |
| | secondary drbd0 | | primary drbd1 | |
| +----------------+ +------------------+ |
| server 2 |
+------------------------------------------+
The io performance on the connected drbd devices is significantly worse if i
start the kernel with the xen hypervisor (with "kernel
/boot/xen-3.4.3.gz").
Without the hypervisor (but the same kernel) the systems are about 50%
faster.
bonnie (bonnie++ -f -d /mnt/drbd0(1) -s 60000 -u root) on the system (started
with xen hypervisor) gives me 115825 K/sec (Sequential Output, Block) on the
average.
server1,60000M,,,117217,23,73652,20,,,189724,24,475.0,0,16,29704,64,+++++,+++,
+++++,+++,+++++,+++,+++++,+++,+++++,+++
server2,60000M,,,120874,26,56761,16,,,359506,42,485.5,0,16,28287,64,+++++,+++,
+++++,+++,+++++,+++,+++++,+++,+++++,+++
server1,60000M,,,112371,22,73340,20,,,192337,24,478.2,0,16,31139,75,+++++,+++,
+++++,+++,+++++,+++,+++++,+++,+++++,+++
server2,60000M,,,117687,25,58340,17,,,355595,42,565.1,0,16,+++++,+++,+++++,
+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
bonnie without the hypervisor (but the same kernel and modules) gives me
212654 K/sec (Sequential Output, Block).
server1,60000M,,,236265,33,76183,9,,,278127,15,722.6,0,16,+++++,+++,+++++,+++,
+++++,+++,+++++,+++,+++++,+++,+++++,+++
server2,60000M,,,192337,31,77391,9,,,297503,16,677.5,0,16,+++++,+++,+++++,+++,
+++++,+++,+++++,+++,+++++,+++,+++++,+++
server1,60000M,,,236486,35,82206,9,,,256470,13,744.6,1,16,+++++,+++,+++++,+++,
+++++,+++,+++++,+++,+++++,+++,+++++,+++
server2,60000M,,,185530,29,79903,9,,,318024,18,748.9,1,16,+++++,+++,+++++,+++,
+++++,+++,+++++,+++,+++++,+++,+++++,+++
Why is there such a difference?
Can i optimize my xend (i already added dom0_mem=2048M dom0_max_vcpus=2
dom0_vcpus_pin as boot parameter with no effect)?
Are there any known issues using XEN and bonding/drbd?
Feel free to ask for more information about the system or the setup.
Many thanks
--
Felix Botner
Open Source Software Engineer
Univention GmbH
Linux for your business
Mary-Somerville-Str.1
28359 Bremen
Tel. : +49 421 22232-0
Fax : +49 421 22232-99
botner@univention.de>
http://www.univention.de
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Jul-20 12:35 UTC
Re: [Xen-users] Xen 3.4.3 - bad IO performance on drbd devices
On Tue, Jul 20, 2010 at 10:46:34AM +0200, Felix Botner wrote:> Hi everyone, > > i have two servers installed with a debian/lenny based os (64bit), a > debian/sid based kernel 2.6.32-xen-amd64 and xen 3.4.3-4. Each server has two > drbd devices (protocol c, formatted with ext4) and is primary for one of > them. Every drbd pair has a dedicated network interface (a bond mode 0 > interface with two 1000 Mbps cards). ><snip>> > The io performance on the connected drbd devices is significantly worse if i > start the kernel with the xen hypervisor (with "kernel /boot/xen-3.4.3.gz"). > Without the hypervisor (but the same kernel) the systems are about 50% > faster. >Are you measuring from dom0 or from a guest?><snip>> > Why is there such a difference? > Can i optimize my xend (i already added dom0_mem=2048M dom0_max_vcpus=2 > dom0_vcpus_pin as boot parameter with no effect)? > Are there any known issues using XEN and bonding/drbd? > > Feel free to ask for more information about the system or the setup. >How much memory does your server have? ie. how much ram do you have when you boot it baremetal without Xen. Remember all the free memory of the host/dom0 will be used by linux pagecache.. So if you limit dom0 to 2GB, it''ll have less cache than the baremetal case. -- Pasi> Many thanks > > -- > Felix Botner > > Open Source Software Engineer > > Univention GmbH > Linux for your business > Mary-Somerville-Str.1 > 28359 Bremen > Tel. : +49 421 22232-0 > Fax : +49 421 22232-99 > > botner@univention.de> > http://www.univention.de > > Geschäftsführer: Peter H. Ganten > HRB 20755 Amtsgericht Bremen > Steuer-Nr.: 71-597-02876 > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Jul-20 12:45 UTC
Re: [Xen-users] Xen 3.4.3 - bad IO performance on drbd devices
On Tue, Jul 20, 2010 at 03:35:30PM +0300, Pasi Kärkkäinen wrote:> > How much memory does your server have? > ie. how much ram do you have when you boot it baremetal without Xen. > > Remember all the free memory of the host/dom0 will be used by linux pagecache.. > So if you limit dom0 to 2GB, it''ll have less cache than the baremetal case. >if benchmarking from dom0, that is. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Felix Botner
2010-Jul-20 13:54 UTC
Re: [Xen-users] Xen 3.4.3 - bad IO performance on drbd devices
Am Dienstag 20 Juli 2010 14:35:30 schrieb Pasi Kärkkäinen:> On Tue, Jul 20, 2010 at 10:46:34AM +0200, Felix Botner wrote: > > Hi everyone, > > > > i have two servers installed with a debian/lenny based os (64bit), a > > debian/sid based kernel 2.6.32-xen-amd64 and xen 3.4.3-4. Each server has > > two drbd devices (protocol c, formatted with ext4) and is primary for one > > of them. Every drbd pair has a dedicated network interface (a bond mode 0 > > interface with two 1000 Mbps cards). > > <snip> > > > The io performance on the connected drbd devices is significantly worse > > if i start the kernel with the xen hypervisor (with "kernel > > /boot/xen-3.4.3.gz"). Without the hypervisor (but the same kernel) the > > systems are about 50% faster. > > Are you measuring from dom0 or from a guest?From dom0, there are no guests at the moment.> > > Why is there such a difference? > > Can i optimize my xend (i already added dom0_mem=2048M dom0_max_vcpus=2 > > dom0_vcpus_pin as boot parameter with no effect)? > > Are there any known issues using XEN and bonding/drbd? > > > > Feel free to ask for more information about the system or the setup. > > How much memory does your server have? > ie. how much ram do you have when you boot it baremetal without Xen.~20GB without Xen. Now i removed the hypervisor param dom0_mem=2048M from menu.lst (in xend-config set (dom0-min-mem 196) and (enable-dom0-ballooning yes)), rebooted and as far as i know there should be no memory restriction for dom0. "xm list" shows me the complete memory for dom0: server1-> xm list Name ID Mem VCPUs State Time(s) Domain-0 0 18323 2 r----- 1785.0 server2-> xm list Name ID Mem VCPUs State Time(s) Domain-0 0 22375 2 r----- 1754.2 But bonnie++ gives me still bad results: server1,60000M,,,113444,30,55569,13,,,231555,23,622.6,0,16,25634,70,+++++,+++, +++++,+++,+++++,+++,+++++,+++,+++++,+++ server2,60000M,,,114014,31,53864,13,,,243541,27,617.4,0,16,+++++,+++,+++++, +++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ So i don''t think caching is the issue, or? bye Felix> > Remember all the free memory of the host/dom0 will be used by linux > pagecache.. So if you limit dom0 to 2GB, it''ll have less cache than the > baremetal case. > > -- Pasi > > > Many thanks > > > > -- > > Felix Botner > > > > Open Source Software Engineer > > > > Univention GmbH > > Linux for your business > > Mary-Somerville-Str.1 > > 28359 Bremen > > Tel. : +49 421 22232-0 > > Fax : +49 421 22232-99 > > > > botner@univention.de> > > http://www.univention.de > > > > Geschäftsführer: Peter H. Ganten > > HRB 20755 Amtsgericht Bremen > > Steuer-Nr.: 71-597-02876 > > > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users-- Felix Botner Open Source Software Engineer Univention GmbH Linux for your business Mary-Somerville-Str.1 28359 Bremen Tel. : +49 421 22232-0 Fax : +49 421 22232-99 botner@univention.de> http://www.univention.de Geschäftsführer: Peter H. Ganten HRB 20755 Amtsgericht Bremen Steuer-Nr.: 71-597-02876 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Felix Botner
2010-Jul-30 10:18 UTC
Re: [Xen-users] Xen 3.4.3 - bad IO performance on drbd devices
Hi some more information about the xen drbd performance problem. Any test without memory restriction for dom0. The network connection beetween the drbd servers limits the troughput to ~230 MB for each drbd device (2 x 1000 Mbps cards, bond mode 0). bonnie on the local drive with hypevisor ~ 400 MB/sec bonnie on the connected drbd device with hypevisor ~ 170 MB/sec (bonnie on one drbd device/server) ~ 110 MB/sec (bonnie on both drbd devices/servers) bonnie on the connected drbd device without hypevisor ~ 230 MB/sec (bonnie on one drbd device/server) ~ 200 MB/sec (bonnie on both drbd devices/servers) bonnie on the disconnected drbd device with hypevisor ~ 300 MB/sec bonnie on the disconnected drbd device without hypevisor ~ 360 MB/sec What interests me is the throughput when writing to both servers. With the XEN hypevisor i get 110 MB/sec on each drbd device (220 MB/sec io throughput on each server because drbd writes "locale" and "remote"). Without the hypervisor i get 200 MB/sec on each drbd device (400 MB/sec io throughput on each server, the maximum of what the io backend allows). But even with only one drbd device the io is without hypevisor much better (230 MB/sec without and 170 MB/sec with hypervisor) The strange thing is, a drbd resync gives me with or without hypervisor ~ 230 MB/sec (one drbd device). And when i start one server without hypervisor, bonnie gives me ~ 230 MB/sec on that server (and on the other server, because also writes remote). Any hints? xm info host : samla release : 2.6.32-ucs9-xen-amd64 version : #1 SMP Thu Jul 22 04:32:22 UTC 2010 machine : x86_64 nr_cpus : 8 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 2133 hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 virt_caps : hvm total_memory : 24564 free_memory : 22230 node_to_cpu : node0:0-7 node_to_memory : node0:22230 xen_major : 3 xen_minor : 4 xen_extra : .3 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable cc_compiler : gcc version 4.3.2 (Debian 4.3.2-1.1.13.200909082302) cc_compile_by : root cc_compile_domain : [unknown] cc_compile_date : Thu Jun 17 14:54:34 UTC 2010 xend_config_format : 4 Am Dienstag 20 Juli 2010 15:54:21 schrieb Felix Botner:> Am Dienstag 20 Juli 2010 14:35:30 schrieb Pasi Kärkkäinen: > > On Tue, Jul 20, 2010 at 10:46:34AM +0200, Felix Botner wrote: > > > Hi everyone, > > > > > > i have two servers installed with a debian/lenny based os (64bit), a > > > debian/sid based kernel 2.6.32-xen-amd64 and xen 3.4.3-4. Each server > > > has two drbd devices (protocol c, formatted with ext4) and is primary > > > for one of them. Every drbd pair has a dedicated network interface (a > > > bond mode 0 interface with two 1000 Mbps cards). > > > > <snip> > > > > > The io performance on the connected drbd devices is significantly worse > > > if i start the kernel with the xen hypervisor (with "kernel > > > /boot/xen-3.4.3.gz"). Without the hypervisor (but the same kernel) the > > > systems are about 50% faster. > > > > Are you measuring from dom0 or from a guest? > > From dom0, there are no guests at the moment. > > > > Why is there such a difference? > > > Can i optimize my xend (i already added dom0_mem=2048M dom0_max_vcpus=2 > > > dom0_vcpus_pin as boot parameter with no effect)? > > > Are there any known issues using XEN and bonding/drbd? > > > > > > Feel free to ask for more information about the system or the setup. > > > > How much memory does your server have? > > ie. how much ram do you have when you boot it baremetal without Xen. > > ~20GB without Xen. Now i removed the hypervisor param dom0_mem=2048M from > menu.lst (in xend-config set (dom0-min-mem 196) and (enable-dom0-ballooning > yes)), rebooted and as far as i know there should be no memory restriction > for dom0. "xm list" shows me the complete memory for dom0: > > server1-> xm list > Name ID Mem VCPUs State > Time(s) > Domain-0 0 18323 2 r----- > 1785.0 > > server2-> xm list > Name ID Mem VCPUs State > Time(s) > Domain-0 0 22375 2 r----- > 1754.2 > > But bonnie++ gives me still bad results: > > server1,60000M,,,113444,30,55569,13,,,231555,23,622.6,0,16,25634,70,+++++,+ >++, +++++,+++,+++++,+++,+++++,+++,+++++,+++ > server2,60000M,,,114014,31,53864,13,,,243541,27,617.4,0,16,+++++,+++,+++++, > +++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ > > So i don''t think caching is the issue, or? > > bye > > Felix > > > Remember all the free memory of the host/dom0 will be used by linux > > pagecache.. So if you limit dom0 to 2GB, it''ll have less cache than the > > baremetal case. > > > > -- Pasi-- Felix Botner Open Source Software Engineer Univention GmbH Linux for your business Mary-Somerville-Str.1 28359 Bremen Tel. : +49 421 22232-0 Fax : +49 421 22232-99 botner@univention.de> http://www.univention.de Geschäftsführer: Peter H. Ganten HRB 20755 Amtsgericht Bremen Steuer-Nr.: 71-597-02876 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Felix Botner
2010-Jul-30 10:28 UTC
Re: [Xen-users] Xen 3.4.3 - bad IO performance on drbd devices
Hi some more information about the xen drbd performance problem (two servers, each server has two drbd devices (protocol c, ext4) and is primary for one of them). Any test without memory restriction for dom0. The network connection between the drbd servers limits the troughput to ~230 MB for each drbd device (2 x 1000 Mbps cards, bond mode 0). The local drive is a hardware raid10 with 12 300gb (3gbps 10k SAS) harddiscs. The kernel is the latest debian/squeeze based kernel 2.6.32-5-amd64 (with http://svn.debian.org/wsvn/kernel/?op=comp&compare[]=%2Fdists%2Fsid@15998&compare[]=%2Fdists%2Fsid@16001) bonnie on the local drive with hypevisor ~ 400 MB/sec bonnie on the connected drbd device with hypevisor ~ 170 MB/sec (bonnie on one drbd device/server) ~ 110 MB/sec (bonnie on both drbd devices/servers) bonnie on the connected drbd device without hypevisor ~ 230 MB/sec (bonnie on one drbd device/server) ~ 200 MB/sec (bonnie on both drbd devices/servers) bonnie on the disconnected drbd device with hypevisor ~ 300 MB/sec bonnie on the disconnected drbd device without hypevisor ~ 360 MB/sec What interests me is the throughput when writing to both servers. With the xen hypevisor i get 110 MB/sec on each drbd device (220 MB/sec io throughput on each server because drbd writes "locale" and "remote"). Without the hypervisor i get 200 MB/sec on each drbd device (400 MB/sec io throughput on each server, the maximum of what the io backend allows). But even with only one drbd device the io is without the hypevisor much better (230 MB/sec without and 170 MB/sec with hypervisor) The strange thing is, a drbd resync gives me with or without hypervisor ~ 230 MB/sec. And when i start one server without hypervisor, bonnie gives me ~ 230 MB/sec on that server and on the remote server with the hypervisor (because drbd also writes remote). Any hints? Thanks, Felix xm info host : samla release : 2.6.32-ucs9-xen-amd64 version : #1 SMP Thu Jul 22 04:32:22 UTC 2010 machine : x86_64 nr_cpus : 8 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 2133 hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 virt_caps : hvm total_memory : 24564 free_memory : 22230 node_to_cpu : node0:0-7 node_to_memory : node0:22230 xen_major : 3 xen_minor : 4 xen_extra : .3 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable cc_compiler : gcc version 4.3.2 (Debian 4.3.2-1.1.13.200909082302) cc_compile_by : root cc_compile_domain : [unknown] cc_compile_date : Thu Jun 17 14:54:34 UTC 2010 xend_config_format : 4 Am Dienstag 20 Juli 2010 15:54:21 schrieb Felix Botner:> Am Dienstag 20 Juli 2010 14:35:30 schrieb Pasi Kärkkäinen: > > On Tue, Jul 20, 2010 at 10:46:34AM +0200, Felix Botner wrote: > > > Hi everyone, > > > > > > i have two servers installed with a debian/lenny based os (64bit), a > > > debian/sid based kernel 2.6.32-xen-amd64 and xen 3.4.3-4. Each server > > > has two drbd devices (protocol c, formatted with ext4) and is primary > > > for one of them. Every drbd pair has a dedicated network interface (a > > > bond mode 0 interface with two 1000 Mbps cards). > > > > <snip> > > > > > The io performance on the connected drbd devices is significantly worse > > > if i start the kernel with the xen hypervisor (with "kernel > > > /boot/xen-3.4.3.gz"). Without the hypervisor (but the same kernel) the > > > systems are about 50% faster. > > > > Are you measuring from dom0 or from a guest? > > From dom0, there are no guests at the moment. > > > > Why is there such a difference? > > > Can i optimize my xend (i already added dom0_mem=2048M dom0_max_vcpus=2 > > > dom0_vcpus_pin as boot parameter with no effect)? > > > Are there any known issues using XEN and bonding/drbd? > > > > > > Feel free to ask for more information about the system or the setup. > > > > How much memory does your server have? > > ie. how much ram do you have when you boot it baremetal without Xen. > > ~20GB without Xen. Now i removed the hypervisor param dom0_mem=2048M from > menu.lst (in xend-config set (dom0-min-mem 196) and (enable-dom0-ballooning > yes)), rebooted and as far as i know there should be no memory restriction > for dom0. "xm list" shows me the complete memory for dom0: > > server1-> xm list > Name ID Mem VCPUs State > Time(s) > Domain-0 0 18323 2 r----- > 1785.0 > > server2-> xm list > Name ID Mem VCPUs State > Time(s) > Domain-0 0 22375 2 r----- > 1754.2 > > But bonnie++ gives me still bad results: > > server1,60000M,,,113444,30,55569,13,,,231555,23,622.6,0,16,25634,70,+++++,+ >++, +++++,+++,+++++,+++,+++++,+++,+++++,+++ > server2,60000M,,,114014,31,53864,13,,,243541,27,617.4,0,16,+++++,+++,+++++, > +++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ > > So i don''t think caching is the issue, or? > > bye > > Felix > > > Remember all the free memory of the host/dom0 will be used by linux > > pagecache.. So if you limit dom0 to 2GB, it''ll have less cache than the > > baremetal case. > > > > -- Pasi-- Felix Botner Open Source Software Engineer Univention GmbH Linux for your business Mary-Somerville-Str.1 28359 Bremen Tel. : +49 421 22232-0 Fax : +49 421 22232-99 botner@univention.de> http://www.univention.de Geschäftsführer: Peter H. Ganten HRB 20755 Amtsgericht Bremen Steuer-Nr.: 71-597-02876 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-Aug-15 12:22 UTC
Re: [Xen-users] Xen 3.4.3 - bad IO performance on drbd devices
On Fri, Jul 30, 2010 at 12:28:58PM +0200, Felix Botner wrote:> Hi > > some more information about the xen drbd performance problem (two servers, > each server has two drbd devices (protocol c, ext4) and is primary for one of > them). Any test without memory restriction for dom0. The network connection > between the drbd servers limits the troughput to ~230 MB for each drbd device > (2 x 1000 Mbps cards, bond mode 0). > > The local drive is a hardware raid10 with 12 300gb (3gbps 10k SAS) harddiscs. > The kernel is the latest debian/squeeze based kernel 2.6.32-5-amd64 (with > http://svn.debian.org/wsvn/kernel/?op=comp&compare[]=%2Fdists%2Fsid@15998&compare[]=%2Fdists%2Fsid@16001) > > bonnie on the local drive with hypevisor > ~ 400 MB/sec > > bonnie on the connected drbd device with hypevisor > ~ 170 MB/sec (bonnie on one drbd device/server) > ~ 110 MB/sec (bonnie on both drbd devices/servers) > > bonnie on the connected drbd device without hypevisor > ~ 230 MB/sec (bonnie on one drbd device/server) > ~ 200 MB/sec (bonnie on both drbd devices/servers) > > bonnie on the disconnected drbd device with hypevisor > ~ 300 MB/sec > > bonnie on the disconnected drbd device without hypevisor > ~ 360 MB/sec > > What interests me is the throughput when writing to both servers. With the xen > hypevisor i get 110 MB/sec on each drbd device (220 MB/sec io throughput on > each server because drbd writes "locale" and "remote"). Without the > hypervisor i get 200 MB/sec on each drbd device (400 MB/sec io throughput on > each server, the maximum of what the io backend allows). > > But even with only one drbd device the io is without the hypevisor much better > (230 MB/sec without and 170 MB/sec with hypervisor) > > The strange thing is, a drbd resync gives me with or without hypervisor ~ 230 > MB/sec. And when i start one server without hypervisor, bonnie gives me ~ 230 > MB/sec on that server and on the remote server with the hypervisor (because > drbd also writes remote). > > Any hints? >I guess DRBD performance mainly depends on the latency you have, because of the synchronization needed between the hosts. Have you tried pinning the dom0 vcpus? -- Pasi> Thanks, > Felix > > xm info > host : samla > release : 2.6.32-ucs9-xen-amd64 > version : #1 SMP Thu Jul 22 04:32:22 UTC 2010 > machine : x86_64 > nr_cpus : 8 > nr_nodes : 1 > cores_per_socket : 4 > threads_per_core : 1 > cpu_mhz : 2133 > hw_caps : > bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 > virt_caps : hvm > total_memory : 24564 > free_memory : 22230 > node_to_cpu : node0:0-7 > node_to_memory : node0:22230 > xen_major : 3 > xen_minor : 4 > xen_extra : .3 > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 > hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : unavailable > cc_compiler : gcc version 4.3.2 (Debian 4.3.2-1.1.13.200909082302) > cc_compile_by : root > cc_compile_domain : [unknown] > cc_compile_date : Thu Jun 17 14:54:34 UTC 2010 > xend_config_format : 4 > > Am Dienstag 20 Juli 2010 15:54:21 schrieb Felix Botner: > > Am Dienstag 20 Juli 2010 14:35:30 schrieb Pasi Kärkkäinen: > > > On Tue, Jul 20, 2010 at 10:46:34AM +0200, Felix Botner wrote: > > > > Hi everyone, > > > > > > > > i have two servers installed with a debian/lenny based os (64bit), a > > > > debian/sid based kernel 2.6.32-xen-amd64 and xen 3.4.3-4. Each server > > > > has two drbd devices (protocol c, formatted with ext4) and is primary > > > > for one of them. Every drbd pair has a dedicated network interface (a > > > > bond mode 0 interface with two 1000 Mbps cards). > > > > > > <snip> > > > > > > > The io performance on the connected drbd devices is significantly worse > > > > if i start the kernel with the xen hypervisor (with "kernel > > > > /boot/xen-3.4.3.gz"). Without the hypervisor (but the same kernel) the > > > > systems are about 50% faster. > > > > > > Are you measuring from dom0 or from a guest? > > > > From dom0, there are no guests at the moment. > > > > > > Why is there such a difference? > > > > Can i optimize my xend (i already added dom0_mem=2048M dom0_max_vcpus=2 > > > > dom0_vcpus_pin as boot parameter with no effect)? > > > > Are there any known issues using XEN and bonding/drbd? > > > > > > > > Feel free to ask for more information about the system or the setup. > > > > > > How much memory does your server have? > > > ie. how much ram do you have when you boot it baremetal without Xen. > > > > ~20GB without Xen. Now i removed the hypervisor param dom0_mem=2048M from > > menu.lst (in xend-config set (dom0-min-mem 196) and (enable-dom0-ballooning > > yes)), rebooted and as far as i know there should be no memory restriction > > for dom0. "xm list" shows me the complete memory for dom0: > > > > server1-> xm list > > Name ID Mem VCPUs State > > Time(s) > > Domain-0 0 18323 2 r----- > > 1785.0 > > > > server2-> xm list > > Name ID Mem VCPUs State > > Time(s) > > Domain-0 0 22375 2 r----- > > 1754.2 > > > > But bonnie++ gives me still bad results: > > > > server1,60000M,,,113444,30,55569,13,,,231555,23,622.6,0,16,25634,70,+++++,+ > >++, +++++,+++,+++++,+++,+++++,+++,+++++,+++ > > server2,60000M,,,114014,31,53864,13,,,243541,27,617.4,0,16,+++++,+++,+++++, > > +++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ > > > > So i don''t think caching is the issue, or? > > > > bye > > > > Felix > > > > > Remember all the free memory of the host/dom0 will be used by linux > > > pagecache.. So if you limit dom0 to 2GB, it''ll have less cache than the > > > baremetal case. > > > > > > -- Pasi > > > -- > Felix Botner > > Open Source Software Engineer > > Univention GmbH > Linux for your business > Mary-Somerville-Str.1 > 28359 Bremen > Tel. : +49 421 22232-0 > Fax : +49 421 22232-99 > > botner@univention.de> > http://www.univention.de > > Geschäftsführer: Peter H. Ganten > HRB 20755 Amtsgericht Bremen > Steuer-Nr.: 71-597-02876 > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users