Hello everybody
After 3 days messing around I''m at my wits'' end. I have two
Sun X4200M2
servers which I would like to setup as Xen dom0s with a drbd-mirrored
file system for paravirtualized domU images.
I setup an LVM2 on top of DRBD to create the domU volumes and to be able
to backup the domUs with LVM snapshot.
I was able to configure this setup and make it run, but when accessing
the block-frontend device in domU this is terrible slow. E.g. copying a
5G file via scp and gigabit Ethernet onto the domU leads to a lot of
stalls during transfer. The peak transfer rate is about 5MB/s every 1-2
minutes but generally it''s about a few hundred KB/s. When copying the
same file onto the DRBD volume mounted directly in dom0 it works with
the expected 30-40MB/s. Another test was to sync the portage tree but
also this takes maybe about 20 minutes or spontaneously raises some
timeout errors. Sometimes after a while the disk fully detaches from the
domU, so every call of a binary in domU leads to a "Input/output
error".
I guess the domU kernel adds this troubles, but why? There are no
statements in any log files. E.g. /var/log/xen/xend-debug.log doesn''t
notify the detached block device.
I use Gentoo Linux and tried so far with xen-3.1.2 and xen-3.2.0,
xen-sources-2.6.18-r8, xen-sources-2.6.20-r6 and xen-sources-2.6.22,
drbd-8.0.7, drbd-8.0.8 and drbd-8.0.11.
The longer the more I expect a general problem with Xen and this setup,
a kernel problem or a misconfiguration. Attached are the most important
configurations for this setup.
Are there any hints or other experiences for such a setup? All ideas are
welcome.
Thanks for reading and giving any hints...
Cheers Reto Gantenbein
################ Configurations #################################
/etc/drbd.conf
,----------------------------------------------------------------
| global {
| usage-count no;
| dialog-refresh 1;
| }
|
| common {
| startup {
| wfc-timeout 60;
| degr-wfc-timeout 60;
| }
|
| disk {
| on-io-error detach;
| # also tried ''pass-on'' with same behavior
| }
|
| syncer {
| rate 500M;
| al-extents 80;
| }
| }
|
| resource "gentoo" {
| protocol C;
| net {
| # required for xen live migration
| allow-two-primaries;
| after-sb-0pri discard-zero-changes;
| after-sb-1pri discard-secondary;
| }
|
| on vserver03 {
| device /dev/drbd1;
| disk /dev/xen-vg/gentoo;
| address 10.0.0.1:7789;
| meta-disk internal;
| }
|
| on vserver04 {
| device /dev/drbd1;
| disk /dev/xen-vg/gentoo;
| address 10.0.0.2:7789;
| meta-disk internal;
| }
| }
`----------------------------------------------------------------
/etc/xen/gentoo
,----------------------------------------------------------------
| name = "gentoo"
|
| kernel = "/cluster/xen/kernel/vmlinuz-gentoo-domU"
| extra = ""
| root = "/dev/hda1"
|
| memory = 4096
| vcpus = 3
| disk = [ ''drbd:gentoo,hda1,w'' ]
| vif = [ ''mac=00:40:a6:00:45:01,bridge=xenbr-core'',
| ''mac=00:40:a6:00:45:02,bridge=xenbr-public'' ]
`----------------------------------------------------------------
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Florian Tischler
2008-Mar-08 19:37 UTC
Re: [Xen-users] Performance problems with domU on drbd
Am Freitag, 15. Februar 2008 schrieb Reto Gantenbein:> Hello everybody > > After 3 days messing around I''m at my wits'' end. I have two Sun X4200M2 > servers which I would like to setup as Xen dom0s with a drbd-mirrored > file system for paravirtualized domU images. > > I setup an LVM2 on top of DRBD to create the domU volumes and to be able > to backup the domUs with LVM snapshot. > > I was able to configure this setup and make it run, but when accessing > the block-frontend device in domU this is terrible slow. E.g. copying a > 5G file via scp and gigabit Ethernet onto the domU leads to a lot of > stalls during transfer. The peak transfer rate is about 5MB/s every 1-2Hi, Quite a late answer, but maybe better than no answer at all... I am currently working on a Xen based 2 nodes Linux-HA cluster. System is Gentoo Hardened AMD64 (multilib) with Xen 3.2 and Kernel 2.6.20-Xen/2.6.21-Xen on a HP Proliant DL380G5. I am also using LVM2 & DRBD 8.0.11. Until now i have not discovered any problems with performance. The system is not in production use but testing is already quite far and if nothing goes wrong i will put it in production use soon. The speed of syncing portage tree or compiling programs in a domU does not feel much slower than on dom0. I have done some basic bonni benchmarks in a domU. (this domU is on a drbd disk which is on a lvm volume) Here are the results. bonnie -s 2048 -m xentest -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU xentest 2048 55503 81.0 84324 15.4 41231 1.9 42244 27.5 159844 1.4 891.5 0.0 Probably not related to your problem - in my opinion your al-extents in your drbd.conf are set quite low. this hurts performance. Default is 127, i am using 257. I don´t know if this answer is helpful for you... if you want to have more details on my setup let me know. Basically there seems to be not much difference, your configs below no not much differ from my configs. Florian> minutes but generally it''s about a few hundred KB/s. When copying the > same file onto the DRBD volume mounted directly in dom0 it works with > the expected 30-40MB/s. Another test was to sync the portage tree but > also this takes maybe about 20 minutes or spontaneously raises some > timeout errors. Sometimes after a while the disk fully detaches from the > domU, so every call of a binary in domU leads to a "Input/output error". > I guess the domU kernel adds this troubles, but why? There are no > statements in any log files. E.g. /var/log/xen/xend-debug.log doesn''t > notify the detached block device. > > I use Gentoo Linux and tried so far with xen-3.1.2 and xen-3.2.0, > xen-sources-2.6.18-r8, xen-sources-2.6.20-r6 and xen-sources-2.6.22, > drbd-8.0.7, drbd-8.0.8 and drbd-8.0.11. > > The longer the more I expect a general problem with Xen and this setup, > a kernel problem or a misconfiguration. Attached are the most important > configurations for this setup. > > Are there any hints or other experiences for such a setup? All ideas are > welcome. > > Thanks for reading and giving any hints... > Cheers Reto Gantenbein > > > ################ Configurations ################################# > > /etc/drbd.conf > ,---------------------------------------------------------------- > > | global { > | usage-count no; > | dialog-refresh 1; > | } > | > | common { > | startup { > | wfc-timeout 60; > | degr-wfc-timeout 60; > | } > | > | disk { > | on-io-error detach; > | # also tried ''pass-on'' with same behavior > | } > | > | syncer { > | rate 500M; > | al-extents 80; > | } > | } > | > | resource "gentoo" { > | protocol C; > | net { > | # required for xen live migration > | allow-two-primaries; > | after-sb-0pri discard-zero-changes; > | after-sb-1pri discard-secondary; > | } > | > | on vserver03 { > | device /dev/drbd1; > | disk /dev/xen-vg/gentoo; > | address 10.0.0.1:7789; > | meta-disk internal; > | } > | > | on vserver04 { > | device /dev/drbd1; > | disk /dev/xen-vg/gentoo; > | address 10.0.0.2:7789; > | meta-disk internal; > | } > | } > > `---------------------------------------------------------------- > > /etc/xen/gentoo > ,---------------------------------------------------------------- > > | name = "gentoo" > | > | kernel = "/cluster/xen/kernel/vmlinuz-gentoo-domU" > | extra = "" > | root = "/dev/hda1" > | > | memory = 4096 > | vcpus = 3 > | disk = [ ''drbd:gentoo,hda1,w'' ] > | vif = [ ''mac=00:40:a6:00:45:01,bridge=xenbr-core'', > | ''mac=00:40:a6:00:45:02,bridge=xenbr-public'' ] > > `---------------------------------------------------------------- > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- Florian Tischler mailto:flo_list2007@floti.at icq#11754147 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users