Hello guys Does anyone use lvm2 backends for domU storages ? I do and I wanted to use lvm''s snapshot feature to make backups of domUs in background but I got the following problem. When I create a snapshot LV and copy data from it to backup storage it works perfect. Then I do umount and then lvremove. lvremove asks me if I really want to remove the volume and then just hangs forever. In the same time affected domU almost hangs too - it becomes very slow but still live. The only solution of this situation is to shutdown domU, deactivate domU''s LV and then run lvremove for shapshot volume. I googled about this and found that it may be a bug related to SMP (I use P-IV with hyperthreading enabled) but I can''t disable or enable SMP because it''s done by xen itself. Any suggestions/ideas ? -- Best regards Anton Khalikov _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Anton Khalikov wrote:> Hello guys > > Does anyone use lvm2 backends for domU storages ? I do and I wanted to > use lvm''s snapshot feature to make backups of domUs in background but I > got the following problem. > When I create a snapshot LV and copy data from it to backup storage it > works perfect. > Then I do umount and then lvremove. lvremove asks me if I really want to > remove the volume and then just hangs forever. In the same time affected > domU almost hangs too - it becomes very slow but still live. > The only solution of this situation is to shutdown domU, deactivate > domU''s LV and then run lvremove for shapshot volume. >What is your dom0/domU kernel version? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello Matt Matt Ayres wrote:> What is your dom0/domU kernel version?dom0: uname -a Linux andrew 2.6.16-xen0 #2 Sat May 6 15:23:55 YEKST 2006 i686 GNU/Linux domU: uname -a Linux xen1 2.6.16-xenU #2 SMP Sat May 6 15:28:43 YEKST 2006 i686 GNU/Linux xen version 3.0.2 -- Best regards Anton Khalikov _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Anton Khalikov wrote:> Hello Matt > > Matt Ayres wrote: >> What is your dom0/domU kernel version? > > dom0: > uname -a > Linux andrew 2.6.16-xen0 #2 Sat May 6 15:23:55 YEKST 2006 i686 GNU/Linux > > domU: > uname -a > Linux xen1 2.6.16-xenU #2 SMP Sat May 6 15:28:43 YEKST 2006 i686 GNU/Linux > > xen version 3.0.2 >It is a 2.6.16 issue. It was fixed in 2.6.16.12. Latest Xen is 2.6.16.13 from 3.0-testing. http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.16.12 commit f6a731290ca18b31fd447989319eb913d9c308d8 Author: Alasdair G Kergon <agk@redhat.com> Date: Mon Apr 24 13:36:06 2006 -0700 [PATCH] dm snapshot: fix kcopyd destructor Before removing a snapshot, wait for the completion of any kcopyd jobs using it. Do this by maintaining a count (nr_jobs) of how many outstanding jobs each kcopyd_client has. The snapshot destructor first unregisters the snapshot so that no new kcopyd jobs (created by writes to the origin) will reference that particular snapshot. kcopyd_client_destroy() is now run next to wait for the completion of any outstanding jobs before the snapshot exception structures (that those jobs reference) are freed. Signed-off-by: Alasdair G Kergon <agk@redhat.com> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org> Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de> _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello Matt Thanks for your help> It is a 2.6.16 issue. It was fixed in 2.6.16.12. Latest Xen is > 2.6.16.13 from 3.0-testing.Does it mean that I need to upgrade the whole thing (xen) or i can download patch to upgrade kernel to version 2.6.16.13 and just recompile dom0/domU kernels ? What option would be better ? -- Best regards Anton Khalikov _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I ran into this same problem. So... this is confirmed with the 3.0.2 distribution off xensource.com? If i use the latest from the Mercurial repo, that''ll fix it up? Also, i had to power off my system to get it un-hung from this. Now it never picks up the lvm disk with vgscan.. it doesn''t even show up in pvdisplay. The only way i can get it to work is to do pvs /dev/hdc; lvchange -a y. Did anyone else have this problem as a result of this bug and know how to fix it? Thanks, Jeff Matt Ayres wrote:> > > Anton Khalikov wrote: >> Hello Matt >> >> Matt Ayres wrote: >>> What is your dom0/domU kernel version? >> >> dom0: >> uname -a >> Linux andrew 2.6.16-xen0 #2 Sat May 6 15:23:55 YEKST 2006 i686 GNU/Linux >> >> domU: >> uname -a >> Linux xen1 2.6.16-xenU #2 SMP Sat May 6 15:28:43 YEKST 2006 i686 >> GNU/Linux >> >> xen version 3.0.2 >> > > It is a 2.6.16 issue. It was fixed in 2.6.16.12. Latest Xen is > 2.6.16.13 from 3.0-testing. > > http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.16.12 > > > > commit f6a731290ca18b31fd447989319eb913d9c308d8 > Author: Alasdair G Kergon <agk@redhat.com> > Date: Mon Apr 24 13:36:06 2006 -0700 > > [PATCH] dm snapshot: fix kcopyd destructor > > Before removing a snapshot, wait for the completion of any kcopyd > jobs using it. Do this by maintaining a count (nr_jobs) of how many > outstanding jobs each kcopyd_client has. > > The snapshot destructor first unregisters the snapshot so that no > new kcopyd jobs (created by writes to the origin) will reference that > particular snapshot. kcopyd_client_destroy() is now run next to wait > for the completion of any outstanding jobs before the snapshot > exception structures (that those jobs reference) are freed. > > Signed-off-by: Alasdair G Kergon <agk@redhat.com> > Signed-off-by: Andrew Morton <akpm@osdl.org> > Signed-off-by: Linus Torvalds <torvalds@osdl.org> > Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de> > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello Jeff Jeff Utter wrote:> I ran into this same problem. So... this is confirmed with the 3.0.2 > distribution off xensource.com? If i use the latest from the Mercurial > repo, that''ll fix it up? > > Also, i had to power off my system to get it un-hung from this. Now it > never picks up the lvm disk with vgscan.. it doesn''t even show up in > pvdisplay. The only way i can get it to work is to do pvs /dev/hdc; > lvchange -a y. Did anyone else have this problem as a result of this bug > and know how to fix it?I am about to upgrade my xen and kernels so I will post the answer if it fix the problem here in about 20-30 minutes. Actually I think you don''t need to power off your system except of domUs to run "lvchange -a n" to be able to remove backup snapshot -- Best regards Anton Khalikov _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello again I''ve upgraded my xen to the latest testing (3.0.2-2) and kernels to 2.6.16.13 but the problem is still here. Even worse - now when I do lvremove -f /dev/xenvg/backup the source domU becomes frozen and I can''t get there in any way - not via ssh and not via xm console :( But it still answeres on my pings -- Best regards Anton Khalikov _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Emiliano Gabrielli
2006-May-24 08:08 UTC
Re: [Xen-users] LVM2 snapshots and XEN = problem :(
On Wednesday 24 May 2006 05:58, Anton Khalikov wrote:> Actually I think you don''t need to power off your system except of domUs > to run "lvchange -a n" to be able to remove backup snapshotI can confirm this -- Dr. Emiliano Gabrielli - Responsabile Divisione Informatica email: emiliano.gabrielli@deArchitettura.com deArchitettura.com Via Francesco Tovaglieri, 411 - 00155 Roma tel: 0645438979 | fax: 0645438980 | url: www.deArchitettura.com _________________________________________________________________________ CONFIDENZIALE: Le informazioni contenute nella presente comunicazione ed i relativi allegati sono confidenziali e riservati. Se avete ricevuto questo messaggio per errore, vi preghiamo di distruggerlo e di informarci immediatamente all''indirizzo email info@deArchitettura.com Ai sensi del D.Lgs. 196/2003 sulla privacy e dell''art. 616 del c.p. è proibita qualsiasi forma di riproduzione o divulgazione del documento trasmesso, senza l''esplicito consenso di deArchitettura.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Emiliano Gabrielli wrote:> On Wednesday 24 May 2006 05:58, Anton Khalikov wrote: > >>Actually I think you don''t need to power off your system except of domUs >>to run "lvchange -a n" to be able to remove backup snapshot > > > I can confirm thisIt seems like it depends on the kernel version. After I installed latest xen-testing (3.0.2-2 with 2.6.16.13) I can''t power off my afrected domU anymore because after lvremove it freezes and doesn''t respond to both xm console and ssh :( -- Best regards Anton Khalikov _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 5/24/06, Anton Khalikov <anton@khalikov.ru> wrote:> > Emiliano Gabrielli wrote: > > On Wednesday 24 May 2006 05:58, Anton Khalikov wrote: > > > >>Actually I think you don''t need to power off your system except of domUs > >>to run "lvchange -a n" to be able to remove backup snapshot > > > > > > I can confirm this > > > It seems like it depends on the kernel version. After I installed latest > xen-testing (3.0.2-2 with 2.6.16.13) I can''t power off my afrected domU > anymore because after lvremove it freezes and doesn''t respond to both xm > console and ssh :( > > -- > Best regards > Anton Khalikov > > LVM Snapshots have not been 100% reliable in 2.6.x kernels for any use.IIRC Redhat has not been supporting them in their enterprise releases. They are supposed to support them in their v5 release at the end of the year, so hopefully their LVM team is working on it. One recent relevant post on the LVM list I found was: http://article.gmane.org/gmane.linux.lvm.general/7762/ Sounds like at least that fix should be in very recent 2.6.16 kernels like you tried. I would not be surprised if there are other bugs not yet addressed. Greg -- Greg Freemyer The Norcross Group Forensics for the 21st Century _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Anton Khalikov
2006-May-25 04:54 UTC
[Xen-users] unregister_netdevice: waiting for vif3.0 to become free. Usage count = 15
Hello guys I am sorry if I am annoying but I still get such messages when I try to reboot/shutdown any domU. Yesterday I upgraded my xen to the latest testing (3.0.2-2) with the latest available kernel (2.6.26.13) but the problem is still here. It happens almost always when i do shutdown and time to time when I do reboot. After this happens I can''t reboot my dom0 except of using SysRq reboot (reset). ifconfig gets frozen too. Actually nothing helps me to solve the problem without rebooting the whole server. The worst part of this problem is that I can''t boot affected domU after this because dead domU becomes Zombie process and locks all resources used by it ie memory and disk partition (lvm) so the only solution i found yet is to stop all domUs and do SysRq reset of dom0. I use bridged networking and I am sure this is something related to kernel support of bridged networks. Google says that such problem existed in old kernels (ie. 2.6.8) but after there was a patch that had to solve this problem. I checked my kernel (infact this is the stock kernel downloaded by xen from kernel.org) and my kernel contains this patch. Does anyone have any ideas about this ? -- Best regards Anton Khalikov _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Anton Khalikov
2006-May-25 05:55 UTC
Re: [Xen-users] unregister_netdevice: waiting for vif3.0 to become free. Usage count = 15
Is it possible that the problem is my NIC related ? I use onboard Marvell Technology Group Ltd. Yukon Gigabit Ethernet 10/100/1000Base-T Adapter (rev 13) May be there''s a bug in NIC''s driver ? -- Best regards Anton Khalikov _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Anton Khalikov
2006-May-26 11:13 UTC
Re: [Xen-users] unregister_netdevice: waiting for vif3.0 to become free. Usage count = 15
Hello guys I started to feel like I am the only person who has such problem. Does anyone use the same NIC card as me: Marvell Technology Group Ltd. Yukon Gigabit Ethernet 10/100/1000Base-T Adapter (rev 13) ? -- Best regards Anton Khalikov _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > > It seems like it depends on the kernel version. After I installed latest > > xen-testing (3.0.2-2 with 2.6.16.13) I can''t power off my afrected domU > > anymore because after lvremove it freezes and doesn''t respond to both xm > > console and ssh :( > >Don''t know if the problem still exists. For me it does, better it did.. i don''t think it''s a problem with the kernel (maybe as well, but i use a custom 2.6.16.19 kernel), it is a problem with the versions of lvm2 and libdevmapper. i use debian sarge stable where the versions of lvm is 2.01.04-5 and of libdevmapper 1.01.00-4.. but if you want to get it working you need the 2.02 and 1.02. So i upgraded the lvm and libdevmapper packages to these versions and the removal of the snapshots work without problems. Tobias _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Friday 08 September 2006 4:59 am, Tobias König wrote:> So i upgraded the lvm and libdevmapper packages to these versions and the > removal of the snapshots work without problems.do you mean that the problem is in userspace and not on the kernel? what about using EVMS for volume management? since it uses the same kernel facilities it might avoid the but if it''s in userspace, but not if it''s in kernelspace... i haven''t been able to reproduce the problem, but it really makes me nervous to rely on snapshots for new designs -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I have also avoided using snapshots so far for the same reasons. I''ve also been looking at EVMS (as I have a SAN, but no raid controller, and I''d like to be able to give each domU a small fraction of a mirror and still be able to individually hot-migrate domains. I believe this can be done with EVMS. see http://www.archivesat.com/Xen_technical_preview_development/thread481214.htm ) As for the LVM snapshot issues, I have avoided lvm snapshots so far out of fear of dom0 crashes. Would it be possible for me to export the master volume read-only to all the domUs, then have the domUs each create and mount their own writable snapshot in DomU-space? If it''s in domU space, stability is less of a concern.> To: xen-users@lists.xensource.com > Subject: Re: [Xen-users] LVM2 snapshots and XEN = problem :( > > On Friday 08 September 2006 4:59 am, Tobias Knig wrote: >> So i upgraded the lvm and libdevmapper packages to these versions and the >> removal of the snapshots work without problems. > > do you mean that the problem is in userspace and not on the kernel? what > about using EVMS for volume management? since it uses the same kernel > facilities it might avoid the but if it''s in userspace, but not if it''s in > kernelspace... > > i haven''t been able to reproduce the problem, but it really makes me nervous > to rely on snapshots for new designs > > -- > Javier >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Has anyone had any further success in this ?>i don''t think it''s a problem with the kernel (maybe as well, but i use acustom 2.6.16.19 kernel), >it is a problem with the versions of lvm2 and libdevmapper. i use debian sarge stable where the >versions of lvm is 2.01.04-5 and of libdevmapper 1.01.00-4.. but if you want to get it working>you need the 2.02 and 1.02. >So i upgraded the lvm and libdevmapper packages to these versions and theremoval of the >snapshots work without problems. Did you use the standard apt-get to get the packages or installed it from some other location manually ? Basically are these versions standard from the debian repo, or we need to get it manually from someplace. -- regards, Anand Gupta _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I can confirm that upgrading to lvm2 and libdevmapper solves the problem. I have built sarge packages, if anyone would like them, let me know and I will put them up on the web somewhere. Of course no gaurantees that these packages don''t eat your machine from the inside out. Greg. On 9/28/06, Anand Gupta <xen.mails@gmail.com> wrote:> > Has anyone had any further success in this ? > > >i don''t think it''s a problem with the kernel (maybe as well, but i use a > custom 2.6.16.19 kernel), >it is a problem with the versions of lvm2 and > libdevmapper. i use debian sarge stable where the >versions of lvm is > 2.01.04-5 and of libdevmapper 1.01.00-4.. but if you want to get it > working >you need the 2.02 and 1.02. > >So i upgraded the lvm and libdevmapper packages to these versions and the > removal of the >snapshots work without problems. > > Did you use the standard apt-get to get the packages or installed it from > some other location manually ? Basically are these versions standard from > the debian repo, or we need to get it manually from someplace. > > -- > regards, > > Anand Gupta > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Greg Cockburn wrote:> I can confirm that upgrading to lvm2 and libdevmapper solves the problem. > > I have built sarge packages, if anyone would like them, let me know and > I will put them up on the web somewhere. Of course no gaurantees that > these packages don''t eat your machine from the inside out.Isn''t that what backports.org is for? -- Julian Davison Note: 1) This may have come from an address @cbhs.school.nz but isn''t necessarily the (or even an) official view of Christchurch Boys'' High School 2) While replying to this address may get into my mailbox it will almost certainly be filtered into a mailing list folder. To actually reach actual me, strip off the bit after the ''-'' in the name. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users