Hello list. My server: Gentoo 2.6.21-xen dom0 xen-tools 3.2.1 lvm 2.02.28-r2 udev-115-r1 Symptoms: normally I run 4 domUs. About once a week I have a need to restart a domU. Sometimes it works just fine, but other times it does not and I''ll get a message back saying "Error: Device 0 (vif) could not be connected. Hotplug scripts not working". The domU config file has not been changed so I''m fairly certain it is configured properly. I think it comes down to the domU not being able to "talk" with udevd. I know udevd is still running and is responsive (if I run udevmonitor and mount/umount a disk I see normal output). When I attempt to start my domU I normally see udevmonitor output like this: UEVENT[1240149226.992189] add /devices/xen-backend/vbd-9-2048 (xen-backend) UDEV [1240149226.993163] add /devices/xen-backend/vbd-9-2048 (xen-backend) UEVENT[1240149227.012202] add /devices/xen-backend/vbd-9-2064 (xen-backend) UDEV [1240149227.013301] add /devices/xen-backend/vbd-9-2064 (xen-backend) UEVENT[1240149227.045410] add /devices/xen-backend/vif-9-0 (xen-backend) UDEV [1240149227.048638] add /devices/xen-backend/vif-9-0 (xen-backend) UEVENT[1240149227.051320] add /devices/virtual/net/vif9.0 (net) UEVENT[1240149227.051805] online /devices/xen-backend/vif-9-0 (xen-backend) UDEV [1240149227.054369] online /devices/xen-backend/vif-9-0 (xen-backend) UDEV [1240149227.062738] add /devices/virtual/net/vif9.0 (net) UEVENT[1240149227.108918] add /devices/xen-backend/console-9-0 (xen-backend) UDEV [1240149227.110112] add /devices/xen-backend/console-9-0 (xen-backend) But when the issue happens I see no udevmonitor output. Restarting udevd did not help. The problem does not seem specific to any particular domU -- once the condition happens, none of the domUs will restart. My only solution so far has been to restart dom0. Suggestions? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Iain J. Watson
2009-May-13 19:47 UTC
Re: [Xen-users] intermittent domU issue: related to udev?
On Sun, Apr 19, 2009 at 3:31 PM, Eric Lenio <eric@lenio.net> wrote:> Hello list. My server: > > Gentoo 2.6.21-xen dom0 > xen-tools 3.2.1 > lvm 2.02.28-r2 > udev-115-r1 > > > Symptoms: normally I run 4 domUs. About once a week I have a need to > restart a domU. Sometimes it works just fine, but other times it does > not and I''ll get a message back saying "Error: Device 0 (vif) could not > be connected. Hotplug scripts not working". The domU config file has > not been changed so I''m fairly certain it is configured properly. I > think it comes down to the domU not being able to "talk" with udevd. I > know udevd is still running and is responsive (if I run udevmonitor and > mount/umount a disk I see normal output). When I attempt to start my > domU I normally see udevmonitor output like this: > > UEVENT[1240149226.992189] add /devices/xen-backend/vbd-9-2048 (xen-backend) > UDEV [1240149226.993163] add /devices/xen-backend/vbd-9-2048 (xen-backend) > UEVENT[1240149227.012202] add /devices/xen-backend/vbd-9-2064 (xen-backend) > UDEV [1240149227.013301] add /devices/xen-backend/vbd-9-2064 (xen-backend) > UEVENT[1240149227.045410] add /devices/xen-backend/vif-9-0 (xen-backend) > UDEV [1240149227.048638] add /devices/xen-backend/vif-9-0 (xen-backend) > UEVENT[1240149227.051320] add /devices/virtual/net/vif9.0 (net) > UEVENT[1240149227.051805] online /devices/xen-backend/vif-9-0 (xen-backend) > UDEV [1240149227.054369] online /devices/xen-backend/vif-9-0 (xen-backend) > UDEV [1240149227.062738] add /devices/virtual/net/vif9.0 (net) > UEVENT[1240149227.108918] add /devices/xen-backend/console-9-0 (xen-backend) > UDEV [1240149227.110112] add /devices/xen-backend/console-9-0 (xen-backend) > > > But when the issue happens I see no udevmonitor output. Restarting > udevd did not help. The problem does not seem specific to any > particular domU -- once the condition happens, none of the domUs will > restart. My only solution so far has been to restart dom0. > Suggestions? >I am having a similar problem. I don''t know about the complete lack of udevd activity as I didn''t know about that command but I''ll be checking tomorrow. Did you have any luck chasing this problem down? Regards, Iain Watson -- "Just think how stupid the average person is, and then realize that half of them are even stupider!" --George Carlin _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Eric Lenio
2009-May-13 19:58 UTC
Re: [Xen-users] intermittent domU issue: related to udev?
On Wed, May 13, 2009 at 08:47:55PM +0100, Iain J. Watson wrote:> On Sun, Apr 19, 2009 at 3:31 PM, Eric Lenio <eric@lenio.net> wrote: > > Hello list. My server: > > > > Gentoo 2.6.21-xen dom0 > > xen-tools 3.2.1 > > lvm 2.02.28-r2 > > udev-115-r1 > > > > > > Symptoms: normally I run 4 domUs. About once a week I have a need to > > restart a domU. Sometimes it works just fine, but other times it does not > > and I''ll get a message back saying "Error: Device 0 (vif) could not be > > connected. Hotplug scripts not working". The domU config file has not been > > changed so I''m fairly certain it is configured properly. I think it comes > > down to the domU not being able to "talk" with udevd. I know udevd is > > still running and is responsive (if I run udevmonitor and mount/umount a > > disk I see normal output). When I attempt to start my domU I normally see > > udevmonitor output like this: > > > > UEVENT[1240149226.992189] add /devices/xen-backend/vbd-9-2048 (xen-backend) > > UDEV [1240149226.993163] add /devices/xen-backend/vbd-9-2048 (xen-backend) > > UEVENT[1240149227.012202] add /devices/xen-backend/vbd-9-2064 (xen-backend) > > UDEV [1240149227.013301] add /devices/xen-backend/vbd-9-2064 (xen-backend) > > UEVENT[1240149227.045410] add /devices/xen-backend/vif-9-0 (xen-backend) > > UDEV [1240149227.048638] add /devices/xen-backend/vif-9-0 (xen-backend) > > UEVENT[1240149227.051320] add /devices/virtual/net/vif9.0 (net) > > UEVENT[1240149227.051805] online /devices/xen-backend/vif-9-0 (xen-backend) > > UDEV [1240149227.054369] online /devices/xen-backend/vif-9-0 (xen-backend) > > UDEV [1240149227.062738] add /devices/virtual/net/vif9.0 (net) > > UEVENT[1240149227.108918] add /devices/xen-backend/console-9-0 (xen-backend) > > UDEV [1240149227.110112] add /devices/xen-backend/console-9-0 (xen-backend) > > > > > > But when the issue happens I see no udevmonitor output. Restarting udevd > > did not help. The problem does not seem specific to any particular domU -- > > once the condition happens, none of the domUs will restart. My only > > solution so far has been to restart dom0. Suggestions? > > > > I am having a similar problem. I don''t know about the complete lack of udevd > activity as I didn''t know about that command but I''ll be checking tomorrow. > > Did you have any luck chasing this problem down? > > Regards, > Iain WatsonNo I haven''t figured it out... the problem hasn''t happened since my last post but if history repeats itself it should happen any day now. Would be happy to work with you to try to identify a resolution (other than rebooting!). Eric. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Can you paste the contents of your domU config file? Try including the below line and let me know if the problem continues to exist. *extra = "TERM=xterm xencons=tty console=tty1" *In any case please post the contents of your config file, I would like to take a look at how different yours is compared to mine. Regards Ananth On Thu, May 14, 2009 at 1:28 AM, Eric Lenio <eric@lenio.net> wrote:> On Wed, May 13, 2009 at 08:47:55PM +0100, Iain J. Watson wrote: > > On Sun, Apr 19, 2009 at 3:31 PM, Eric Lenio <eric@lenio.net> wrote: > > > Hello list. My server: > > > > > > Gentoo 2.6.21-xen dom0 > > > xen-tools 3.2.1 > > > lvm 2.02.28-r2 > > > udev-115-r1 > > > > > > > > > Symptoms: normally I run 4 domUs. About once a week I have a need to > > > restart a domU. Sometimes it works just fine, but other times it does > not > > > and I''ll get a message back saying "Error: Device 0 (vif) could not be > > > connected. Hotplug scripts not working". The domU config file has not > been > > > changed so I''m fairly certain it is configured properly. I think it > comes > > > down to the domU not being able to "talk" with udevd. I know udevd is > > > still running and is responsive (if I run udevmonitor and mount/umount > a > > > disk I see normal output). When I attempt to start my domU I normally > see > > > udevmonitor output like this: > > > > > > UEVENT[1240149226.992189] add /devices/xen-backend/vbd-9-2048 > (xen-backend) > > > UDEV [1240149226.993163] add /devices/xen-backend/vbd-9-2048 > (xen-backend) > > > UEVENT[1240149227.012202] add /devices/xen-backend/vbd-9-2064 > (xen-backend) > > > UDEV [1240149227.013301] add /devices/xen-backend/vbd-9-2064 > (xen-backend) > > > UEVENT[1240149227.045410] add /devices/xen-backend/vif-9-0 > (xen-backend) > > > UDEV [1240149227.048638] add /devices/xen-backend/vif-9-0 > (xen-backend) > > > UEVENT[1240149227.051320] add /devices/virtual/net/vif9.0 (net) > > > UEVENT[1240149227.051805] online /devices/xen-backend/vif-9-0 > (xen-backend) > > > UDEV [1240149227.054369] online /devices/xen-backend/vif-9-0 > (xen-backend) > > > UDEV [1240149227.062738] add /devices/virtual/net/vif9.0 (net) > > > UEVENT[1240149227.108918] add /devices/xen-backend/console-9-0 > (xen-backend) > > > UDEV [1240149227.110112] add /devices/xen-backend/console-9-0 > (xen-backend) > > > > > > > > > But when the issue happens I see no udevmonitor output. Restarting > udevd > > > did not help. The problem does not seem specific to any particular > domU -- > > > once the condition happens, none of the domUs will restart. My only > > > solution so far has been to restart dom0. Suggestions? > > > > > > > I am having a similar problem. I don''t know about the complete lack of > udevd > > activity as I didn''t know about that command but I''ll be checking > tomorrow. > > > > Did you have any luck chasing this problem down? > > > > Regards, > > Iain Watson > > No I haven''t figured it out... the problem hasn''t happened since my last > post > but if history repeats itself it should happen any day now. Would be happy > to > work with you to try to identify a resolution (other than rebooting!). > > Eric. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Iain J. Watson
2009-May-14 12:23 UTC
Re: [Xen-users] intermittent domU issue: related to udev?
On Thu, May 14, 2009 at 4:23 AM, Ananth <cbananth@gmail.com> wrote:> Hi, > Can you paste the contents of your domU config file? > Try including the below line and let me know if the problem continues to > exist. > > extra = "TERM=xterm xencons=tty console=tty1" > > In any case please post the contents of your config file, I would like to > take a look at how different yours is compared to mine. > > Regards > Ananth >Do you mean to run this on a Dom0 that is already suffering from this issue, or on a Dom0 that is not affected by this issue and to see if it doesn''t come back? -- "Just think how stupid the average person is, and then realize that half of them are even stupider!" --George Carlin _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Eric Lenio
2009-May-14 14:17 UTC
Re: [Xen-users] intermittent domU issue: related to udev?
On Thu, May 14, 2009 at 08:53:39AM +0530, Ananth wrote:> Hi, > Can you paste the contents of your domU config file? > Try including the below line and let me know if the problem continues to exist. > > extra = "TERM=xterm xencons=tty console=tty1" > > In any case please post the contents of your config file, I would like to take > a look at how different yours is compared to mine. >Here is a typical config for one of my domUs: kernel = "/xen/vmlinuz-lincware-2.6.21-xenU" memory = 1024 vif = [ ''bridge=xenbr0'' ] name = "lincware3" disk = [ ''phy:vg-sda4/lincware3-xfs,sda,w'', ''phy:vg-sda4/lincware3-swap,sdb,w'', ] root = "/dev/sda rw" usb=0 (end of config) I will try adding the "extra" line you mentioned. Unfortunately since I do not know how to replicate the problem on demand it will probably take a few weeks or more to know the results. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2009-May-14 14:35 UTC
Re: [Xen-users] intermittent domU issue: related to udev?
On Thu, May 14, 2009 at 10:17:30AM -0400, Eric Lenio wrote:> On Thu, May 14, 2009 at 08:53:39AM +0530, Ananth wrote: > > Hi, > > Can you paste the contents of your domU config file? > > Try including the below line and let me know if the problem continues to exist. > > > > extra = "TERM=xterm xencons=tty console=tty1" > > > > In any case please post the contents of your config file, I would like to take > > a look at how different yours is compared to mine. > > > > Here is a typical config for one of my domUs: > > kernel = "/xen/vmlinuz-lincware-2.6.21-xenU" > memory = 1024 > vif = [ ''bridge=xenbr0'' ] > name = "lincware3" > disk = [ > ''phy:vg-sda4/lincware3-xfs,sda,w'', > ''phy:vg-sda4/lincware3-swap,sdb,w'', > ] > root = "/dev/sda rw" > usb=0 >Btw you shouldn''t be using sda and sdb, you should be using xvda and xvdb. xvd = Xen Virtual Disk. That sdX device naming thing is deprecated and should not be used anymore. Also, why do you have the kernel in dom0? I find it a lot easier to have the kernel in the domU filesystem (use the kernel-xen provided by the distro), and use pygrub/pvgrub to load the kernel from the domU filesystem. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Eric Lenio
2009-May-14 14:49 UTC
Re: [Xen-users] intermittent domU issue: related to udev?
On Thu, May 14, 2009 at 05:35:32PM +0300, Pasi Kärkkäinen wrote:> On Thu, May 14, 2009 at 10:17:30AM -0400, Eric Lenio wrote: > > Here is a typical config for one of my domUs: > > > > kernel = "/xen/vmlinuz-lincware-2.6.21-xenU" > > memory = 1024 > > vif = [ ''bridge=xenbr0'' ] > > name = "lincware3" > > disk = [ > > ''phy:vg-sda4/lincware3-xfs,sda,w'', > > ''phy:vg-sda4/lincware3-swap,sdb,w'', > > ] > > root = "/dev/sda rw" > > usb=0 > > > > Btw you shouldn''t be using sda and sdb, you should be using xvda and xvdb. > xvd = Xen Virtual Disk. > > That sdX device naming thing is deprecated and should not be used anymore. > > Also, why do you have the kernel in dom0? I find it a lot easier to have the > kernel in the domU filesystem (use the kernel-xen provided by the distro), > and use pygrub/pvgrub to load the kernel from the domU filesystem. > > -- PasiGood to know about xvd. I will switch over to that convention. Do you have reason to believe that this could be related to my original udev issue? It seems unrelated only because my domUs will work for several weeks, and at some point I attempt to restart a domU but it does not seem to properly release resources and the next domU I attempt to start (could be any of them) will always fail with the mysterious "Error: Device 0 (vif) could not be connected. Hotplug scripts not working" which you see everywhere in this list. Why is my kernel in dom0? No particular reason, it''s just how things played out as I built things. I''m using gentoo for dom0 and I basically built my dom0 and domU kernels at the same time while following gentoo''s guide. In retrospect I wish I had gone with prebuilt kernels, and I think I may eventually switch over to Debian... my preference would be Ubuntu because I use those a lot for my domUs but I have read that Ubuntu doesn''t have the greatest support for Xen. Other distro recommendations are welcome if you have them... _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2009-May-14 15:10 UTC
Re: [Xen-users] intermittent domU issue: related to udev?
On Thu, May 14, 2009 at 10:49:14AM -0400, Eric Lenio wrote:> On Thu, May 14, 2009 at 05:35:32PM +0300, Pasi Kärkkäinen wrote: > > On Thu, May 14, 2009 at 10:17:30AM -0400, Eric Lenio wrote: > > > Here is a typical config for one of my domUs: > > > > > > kernel = "/xen/vmlinuz-lincware-2.6.21-xenU" > > > memory = 1024 > > > vif = [ ''bridge=xenbr0'' ] > > > name = "lincware3" > > > disk = [ > > > ''phy:vg-sda4/lincware3-xfs,sda,w'', > > > ''phy:vg-sda4/lincware3-swap,sdb,w'', > > > ] > > > root = "/dev/sda rw" > > > usb=0 > > > > > > > Btw you shouldn''t be using sda and sdb, you should be using xvda and xvdb. > > xvd = Xen Virtual Disk. > > > > That sdX device naming thing is deprecated and should not be used anymore. > > > > Also, why do you have the kernel in dom0? I find it a lot easier to have the > > kernel in the domU filesystem (use the kernel-xen provided by the distro), > > and use pygrub/pvgrub to load the kernel from the domU filesystem. > > > > -- Pasi > > Good to know about xvd. I will switch over to that convention. Do you have > reason to believe that this could be related to my original udev issue? It > seems unrelated only because my domUs will work for several weeks, and at some > point I attempt to restart a domU but it does not seem to properly release > resources and the next domU I attempt to start (could be any of them) will > always fail with the mysterious "Error: Device 0 (vif) could not be connected. > Hotplug scripts not working" which you see everywhere in this list. > > Why is my kernel in dom0? No particular reason, it''s just how things played > out as I built things. I''m using gentoo for dom0 and I basically built my dom0 > and domU kernels at the same time while following gentoo''s guide. In retrospect > I wish I had gone with prebuilt kernels, and I think I may eventually switch > over to Debian... my preference would be Ubuntu because I use those a lot > for my domUs but I have read that Ubuntu doesn''t have the greatest support > for Xen. Other distro recommendations are welcome if you have them...RHEL5 and CentOS5 have good support for Xen. I''d recommend those as dom0. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users