Maxim Doucet
2007-Aug-14 18:58 UTC
[Xen-users] [LVM2 + DRBD + Xen + DRBD 8.0] errors on dom0 and on domU
This message was originally posted in the DRBD mailing list
(http://lists.linbit.com/pipermail/drbd-user/2007-August/007261.html).
I decided to post it here too to reach a broader audience.
I''ll do my best to report any interesting to each mailing list if it
can
diserve this issue.
The issue seems to involve both Xen and DRBD ; trying to centralize some
information between them doesn''t sound too silly for me...
Here is the issue :
I''ll try with this message to link together different discussions which
are dealing with this setup : LVM2 + DRBD + Xen + DRBD 8.0
I have the following environment :
* 2 identical servers with Fedora Core 7 and it''s XEN 3.1 (kernel-xen
2.6.20-2925.13.fc7xen)
* DRBD 8.0.5 primary/primary working (without LinuxHA for now)
* LVM on top of DRBD, with LVM correctly configured to reject the disk
under the DRBD device : /dev/drbd0 with VGvmstor0 on top
* I have only one paravirtualized Virtual Machine (VM) which is using a
file (not a partition nor a LVM''s Logical Volume) as it''s disk
storage.
The VM''s storage file is on the root partition of dom0 (-> the
physical
server, the file being located in /var/lib/xen/images), so : not on LVM
and thus not on the DRBD device. The VM just runs fine.
Now, while the virtual machine is off, I change its configuration to add
a new device :
* the source storage is a LVM''s Logical Volume located on dom0. It is
located on the DRBD resource : /dev/VGvmstor0/LVvmsys1
* the newly created device which is seen by the virtual machine (the
domU) is : /dev/xdvb
I experience the following error messages when launching the virtual
machine :
*On dom0 : the physical server* (messages coming from dmesg) :
drbd0: bio would need to, but cannot, be split:
(vcnt=2,idx=0,size=2048,sector=126353855)
drbd0: bio would need to, but cannot, be split:
(vcnt=2,idx=0,size=2048,sector=126353855)
drbd0: bio would need to, but cannot, be split:
(vcnt=8,idx=0,size=4096,sector=126562682)
drbd0: bio would need to, but cannot, be split:
(vcnt=8,idx=0,size=4096,sector=126562750)
drbd0: bio would need to, but cannot, be split:
(vcnt=4,idx=0,size=4096,sector=126562495)
drbd0: bio would need to, but cannot, be split:
(vcnt=4,idx=0,size=4096,sector=126562495)
drbd0: bio would need to, but cannot, be split:
(vcnt=4,idx=0,size=4096,sector=126353855)
drbd0: bio would need to, but cannot, be split:
(vcnt=4,idx=0,size=4096,sector=126353855)
... and further
*On domU : the virtual machine* (messages appearing at boot time, on the
console) :
blkfront: xvdb: barriers enabled
xvdb: xvdb1 xvdb2
end_request: I/O error, dev xvdb, sector 208703
Buffer I/O error on device xvdb1, logical block 104320
Buffer I/O error on device xvdb1, logical block 104321
Buffer I/O error on device xvdb1, logical block 104322
Buffer I/O error on device xvdb1, logical block 104323
end_request: I/O error, dev xvdb, sector 208703
Buffer I/O error on device xvdb1, logical block 104320
Buffer I/O error on device xvdb1, logical block 104321
Buffer I/O error on device xvdb1, logical block 104322
Buffer I/O error on device xvdb1, logical block 104323
end_request: I/O error, dev xvdb, sector 10473981
Buffer I/O error on device xvdb2, logical block 10265136
Buffer I/O error on device xvdb2, logical block 10265137
... and further
Now, after searching for information about these messages through the
web, I''ve found the following threads which seems to be linked to the
problem :
1. http://forums.xensource.com/message.jspa?messageID=4666#4666 :
The beginning of the thread is not about the error "bio would need to,
but cannot, be split" but the post I link (number #4666) is reporting it
and points to :
2. ...the following patch :
http://lists.linbit.com/pipermail/drbd-user/2007-April/006677.html
3. http://lists.linbit.com/pipermail/drbd-user/2007-June/007054.html :
reports the same error "bio would need to, but cannot, be split" in
the
same configuration
Question : according to
http://lists.linbit.com/pipermail/drbd-user/2007-April/006728.html : is
this problem linked with the errors on dom0 ("bio would need to, but
cannot, be split") ? If yes, was it fixed (I''ll forward this mail
to the
xen-user mailing list).
By now, is there any workaround
(http://forums.xensource.com/message.jspa?messageID=4666#4699 talks
about a modified dm-mod.ko? or maybe the patch in
http://lists.linbit.com/pipermail/drbd-user/2007-April/006677.html) ?
Post http://forums.xensource.com/message.jspa?messageID=4666#4669
reports that everything works well with DRBD 7. Does it means that, for
now, DRBD 8.0 can''t be used on top of a dom0 Xen Kernel ?
--
Maxim Doucet - www.alamaison.fr
sys admin @ la maison
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
xen@ew.nsci.us
2007-Aug-15 23:49 UTC
Re: [Xen-users] [LVM2 + DRBD + Xen + DRBD 8.0] errors on dom0 and on domU
On Tue, 14 Aug 2007, Maxim Doucet wrote:> I experience the following error messages when launching the virtual > machine : > *On dom0 : the physical server* (messages coming from dmesg) : > drbd0: bio would need to, but cannot, be split: > (vcnt=2,idx=0,size=2048,sector=126353855) > drbd0: bio would need to, but cannot, be split: > (vcnt=2,idx=0,size=2048,sector=126353855)We are using a nearly identical configuration and experienced the same problem just today: LVM2 on DRBD under Xen 3.0.3 w/ DRBD 8.0.4 Using CentOS5 on x86_64 dom0 kernel 2.6.18-8.1.8-el5xen The virtual machine is an FC6 x86_64 PV guest and gave similar guest errors. The workaround we are using is to change disk = [ ''phy:/dev/vg-drbd/vm0,xvda,w'' ] to disk = [ ''tap:aio:/dev/vg-drbd/vm0,xvda,w'' ] This treats the underlying backing image as a file. This may have some performance loss since it is not using direct device IO, but as far as I can tell it is stable. Or at least, phy: fails miserably, where tap:aio: works fine! This seems to indicate that its not an LVM+DRBD or Xen+LVM problem, but rather a Xen+LVM+DRBD using phy: problem. I tested to see if Xen liked running LVM on a loopback device and loading a VM off it using phy: (see below). It worked fine, which makes me think this is more of a drbd issue than a Xen or LVM issue. If you are on the DRBD list, please cross-post this (as I am not) since it is probably relevant. -Eric ============== Xen+LVM+loop test: # dd if=/dev/zero bs=1G seek=32 count=1 of=/tmp/testimage # losetup /dev/loop0 /tmp/testimage # pvcreate /dev/loop0 # vgcreate vg-loop /dev/loop0 # pvscan [...] PV /dev/loop0 VG vg-loop lvm2 [11.00 GB / 6.99 GB free] [...] # lvcreate -n testvm -l 1025 vg-loop # lvascan [...] ACTIVE ''/dev/vg-loop/testvm'' [4.00 GB] inherit [...] # ls -l -rwxr-xr-x 1 root root 4294967297 Jul 23 16:22 disk0 # dd if=disk0 bs=4M of=/dev/vg-loop/testvm 1024+1 records in 1024+1 records out 4294967297 bytes (4.3 GB) copied, 396.227 seconds, 10.8 MB/s _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Maxim Doucet
2007-Aug-16 08:37 UTC
Re: [Xen-users] [LVM2 + DRBD + Xen + DRBD 8.0] errors on dom0 and on domU
xen@ew.nsci.us a écrit :> On Tue, 14 Aug 2007, Maxim Doucet wrote: > >> I experience the following error messages when launching the virtual >> machine : >> *On dom0 : the physical server* (messages coming from dmesg) : >> drbd0: bio would need to, but cannot, be split: >> (vcnt=2,idx=0,size=2048,sector=126353855) >> drbd0: bio would need to, but cannot, be split: >> (vcnt=2,idx=0,size=2048,sector=126353855) > > We are using a nearly identical configuration and experienced the same > problem just today: > > LVM2 on DRBD under Xen 3.0.3 w/ DRBD 8.0.4 Using CentOS5 on x86_64 > dom0 kernel 2.6.18-8.1.8-el5xen > > The virtual machine is an FC6 x86_64 PV guest and gave similar guest > errors. > > The workaround we are using is to change > > disk = [ ''phy:/dev/vg-drbd/vm0,xvda,w'' ] > to > disk = [ ''tap:aio:/dev/vg-drbd/vm0,xvda,w'' ] > > This treats the underlying backing image as a file. This may have > some performance loss since it is not using direct device IO, but as > far as I can tell it is stable. Or at least, phy: fails miserably, > where tap:aio: works fine! > > This seems to indicate that its not an LVM+DRBD or Xen+LVM problem, > but rather a Xen+LVM+DRBD using phy: problem. I tested to see if Xen > liked running LVM on a loopback device and loading a VM off it using > phy: (see below). It worked fine, which makes me think this is more > of a drbd issue than a Xen or LVM issue. > > If you are on the DRBD list, please cross-post this (as I am not) > since it is probably relevant. > > -Eric > > > ============== Xen+LVM+loop test: > > # dd if=/dev/zero bs=1G seek=32 count=1 of=/tmp/testimage > # losetup /dev/loop0 /tmp/testimage > # pvcreate /dev/loop0 > # vgcreate vg-loop /dev/loop0 > # pvscan > [...] > PV /dev/loop0 VG vg-loop lvm2 [11.00 GB / 6.99 GB free] > [...] > # lvcreate -n testvm -l 1025 vg-loop > > # lvascan > [...] > ACTIVE ''/dev/vg-loop/testvm'' [4.00 GB] inherit > [...] > > # ls -l > -rwxr-xr-x 1 root root 4294967297 Jul 23 16:22 disk0 > # dd if=disk0 bs=4M of=/dev/vg-loop/testvm > 1024+1 records in > 1024+1 records out > 4294967297 bytes (4.3 GB) copied, 396.227 seconds, 10.8 MB/s > >Thanks a lot for your feedback, I''ll try the workaround and report my results here. I have forwarded your message to the DRBD mailing list : http://lists.linbit.com/pipermail/drbd-user/2007-August/007267.html -- Maxim Doucet - www.alamaison.fr sys admin @ la maison _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Maxim Doucet
2007-Aug-16 18:32 UTC
Re: [Xen-users] [LVM2 + DRBD + Xen + DRBD 8.0] errors on dom0 and on domU
Maxim Doucet a écrit :> xen@ew.nsci.us a écrit : > >> On Tue, 14 Aug 2007, Maxim Doucet wrote: >> >> >>> I experience the following error messages when launching the virtual >>> machine : >>> *On dom0 : the physical server* (messages coming from dmesg) : >>> drbd0: bio would need to, but cannot, be split: >>> (vcnt=2,idx=0,size=2048,sector=126353855) >>> drbd0: bio would need to, but cannot, be split: >>> (vcnt=2,idx=0,size=2048,sector=126353855) >>> >> We are using a nearly identical configuration and experienced the same >> problem just today: >> >> LVM2 on DRBD under Xen 3.0.3 w/ DRBD 8.0.4 Using CentOS5 on x86_64 >> dom0 kernel 2.6.18-8.1.8-el5xen >> >> The virtual machine is an FC6 x86_64 PV guest and gave similar guest >> errors. >> >> The workaround we are using is to change >> >> disk = [ ''phy:/dev/vg-drbd/vm0,xvda,w'' ] >> to >> disk = [ ''tap:aio:/dev/vg-drbd/vm0,xvda,w'' ] >> >> This treats the underlying backing image as a file. This may have >> some performance loss since it is not using direct device IO, but as >> far as I can tell it is stable. Or at least, phy: fails miserably, >> where tap:aio: works fine! >> >> This seems to indicate that its not an LVM+DRBD or Xen+LVM problem, >> but rather a Xen+LVM+DRBD using phy: problem. I tested to see if Xen >> liked running LVM on a loopback device and loading a VM off it using >> phy: (see below). It worked fine, which makes me think this is more >> of a drbd issue than a Xen or LVM issue. >> >> If you are on the DRBD list, please cross-post this (as I am not) >> since it is probably relevant. >> >> -Eric >> >> >> ============== Xen+LVM+loop test: >> >> # dd if=/dev/zero bs=1G seek=32 count=1 of=/tmp/testimage >> # losetup /dev/loop0 /tmp/testimage >> # pvcreate /dev/loop0 >> # vgcreate vg-loop /dev/loop0 >> # pvscan >> [...] >> PV /dev/loop0 VG vg-loop lvm2 [11.00 GB / 6.99 GB free] >> [...] >> # lvcreate -n testvm -l 1025 vg-loop >> >> # lvascan >> [...] >> ACTIVE ''/dev/vg-loop/testvm'' [4.00 GB] inherit >> [...] >> >> # ls -l >> -rwxr-xr-x 1 root root 4294967297 Jul 23 16:22 disk0 >> # dd if=disk0 bs=4M of=/dev/vg-loop/testvm >> 1024+1 records in >> 1024+1 records out >> 4294967297 bytes (4.3 GB) copied, 396.227 seconds, 10.8 MB/s >> >> >> > Thanks a lot for your feedback, I''ll try the workaround and report my > results here. > > I have forwarded your message to the DRBD mailing list : > http://lists.linbit.com/pipermail/drbd-user/2007-August/007267.htmlGood news ! Thanks to the driver "tap:aio" workaround, I''ve been able to install a clean standard installation of Fedora Core 7 upon LVM, itself upon DRBD which happily synchronizes the data between the 2 servers. It just worked like a charm and now paves the way for further development of a redundant and high availability architecture for the services we are using. Now, for the performance cost of using the emulated (correct me if I''m wrong) driver "tap:aio" rather than the direct I/O access of the "phy" driver, Lars Ellenberg (-> http://www.linbit.com/en/company/team/lars-g-ellenberg/ : the co-author of DRBD) gave information concerning the way Xen virtual block device layer is dealing with block devices I/O. The link to his post is : http://lists.linbit.com/pipermail/drbd-user/2007-August/007269.html The post is interesting, technically informative and is followed by an answer from Ross S. Walker who gives further technical information about block layer device handler''s development here : http://lists.linbit.com/pipermail/drbd-user/2007-August/007270.html Again, thanks for your feedback and let''s hope for other developer''s reviews about the core of this problem. -- Maxim Doucet - www.alamaison.fr sys admin @ la maison _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users