Hi everyone, the issue I''m describing here has to do with para-virtual domUs, although I don''t know if it also affects full-virtualized machines, I have none to test against. My problem is that I have some domUs with VNC set up, but VNC stops working after some hours/days after the domU is "xm created". It actually varies every time, I cannot say it fails after 5 hours or 2 days every time. I''m not talking here about black screens after a while. This will always get "fixed" by pressing the Enter button and you''ll get the login screen again. The issue here is that if i create a machine and then leave it alone with no load at all and no reboots or anything done on it, VNC won''t work anymore until I reboot the machine. I always work with Debian Squeeze dom0''s and Xen 4.0. My domus are debian squeeze, ubuntu and fedora, although that does not make any difference at all, as I''ve seen the same kind of problem in the three systems. When this happens, the only thing I can do is shut down the machine (rebooting not always works) and create it again. I don''t know whether VNC depends on xend or has nothing to do at all with it, but restarting xend doesn''t fix this problem. Until I reboot the machine, VNC client will say: vncviewer 42.XXXX:5900 connected to host 42.XXXX port 5900 .......... and it will wait for ever. It doesn''t matter if you use tightvnc, vncviewer or orther software, none is capable of connecting properly. Actually, it looks as if it was working, as it doesn''t throw any connection problem or timeout or anything, but it just doesn''t do anything else, it only connects. Thanks. -- Jordi Moles Blanco Sistemes Cdmon.com ___________________________ Tlf: 902 36 41 38 Tlf: 93 567 75 77 mailto: jordi@cdmon.com http://www.cdmon.com
Hi Jordi, 2011/12/7 Jordi Moles Blanco <jordi@cdmon.com>:> Hi everyone, > > the issue I''m describing here has to do with para-virtual domUs, although I > don''t know if it also affects full-virtualized machines, I have none to test > against.Things you might want to check: - if the qemu-dm processes are still there - if your xenconsoled has died a little - qemu-dm log file in /var/log/xen - is there any stale or stale network connection to the VNC port for this VM? The VNC multi use patches are not in at least *most* Xen versions and that means if you have a VNC session still open somewhere else, then your connection attempt will simple hang forever. That sucks sooooo much. These are the ones where I could pin down some of the issues I saw, though I never had the one you see. (Wow, I skipped one!) Otherwise, run lsof on the qemu process in question, check the socket, and then try using strace on the process while attempting a connection. Btw: It would be AWESOME to know how to properly restart a qemu-dm VNC server without destroying the VM. -- the purpose of libvirt is to provide an abstraction layer hiding all xen features added since 2006 until they were finally understood and copied by the kvm devs.
+1 I hate how when the vnc connection gets hung for any reason how you lose vnc access for quite a while and possibly forever On Fri, Dec 9, 2011 at 7:20 PM, Florian Heigl <florian.heigl@gmail.com>wrote:> Hi Jordi, > > 2011/12/7 Jordi Moles Blanco <jordi@cdmon.com>: > > Hi everyone, > > > > the issue I''m describing here has to do with para-virtual domUs, > although I > > don''t know if it also affects full-virtualized machines, I have none to > test > > against. > > Things you might want to check: > > - if the qemu-dm processes are still there > - if your xenconsoled has died a little > - qemu-dm log file in /var/log/xen > - is there any stale or stale network connection to the VNC port for > this VM? The VNC multi use patches are not in at least *most* Xen > versions and that means if you have a VNC session still open somewhere > else, then your connection attempt will simple hang forever. > That sucks sooooo much. > > These are the ones where I could pin down some of the issues I saw, > though I never had the one you see. (Wow, I skipped one!) > > Otherwise, run lsof on the qemu process in question, check the socket, > and then try using strace on the process while attempting a > connection. > > Btw: It would be AWESOME to know how to properly restart a qemu-dm VNC > server without destroying the VM. > > > > -- > the purpose of libvirt is to provide an abstraction layer hiding all > xen features added since 2006 until they were finally understood and > copied by the kvm devs. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2011/12/10 chris <tknchris@gmail.com>:> +1 > > I hate how when the vnc connection gets hung for any reason how you lose vnc > access for quite a while and possibly foreverSo that makes 3 of us now for being able to restart vnc server :) would be cool if there still were a Xen wishlist. Anyway, next time when that happens do a netstat -na | grep 59 | grep _WAIT I think you''ll find a stuck connection there. Please let us know.
I recommended xen to a friend a while ago who was coming from vmware, he installs it and then plays with it and then the vnc gets hung exactly as we''re describing and hes like well what do I do now and I''m like you gotta kill it. He says are you kidding me? It always been one of those things I''ve wondered how so many people have put up with it for so long :) On Sat, Dec 10, 2011 at 1:29 PM, Florian Heigl <florian.heigl@gmail.com>wrote:> 2011/12/10 chris <tknchris@gmail.com>: > > +1 > > > > I hate how when the vnc connection gets hung for any reason how you lose > vnc > > access for quite a while and possibly forever > > So that makes 3 of us now for being able to restart vnc server :) > would be cool if there still were a Xen wishlist. > > Anyway, next time when that happens do a > netstat -na | grep 59 | grep _WAIT > > I think you''ll find a stuck connection there. Please let us know. >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi everyone, well... it''s good to know that I haven''t come across some nasty and very weird bug very difficult to reproduce. anyway... since the last time I wrote I haven''t seen this problem anymore, as we have many machines and we don''t continuously check them all. Next time I see one VNC broken I''ll try to check what you suggested. on the other hand, I understand from Florian''s words that the problem could be a socket that remains open after a normal VNC connection. Do you think that we could avoid rebooting the server by killing it? Thanks. Al 10/12/11 20:28, En/na chris ha escrit:> I recommended xen to a friend a while ago who was coming from vmware, > he installs it and then plays with it and then the vnc gets hung > exactly as we''re describing and hes like well what do I do now and I''m > like you gotta kill it. He says are you kidding me? It always been one > of those things I''ve wondered how so many people have put up with it > for so long :) > > On Sat, Dec 10, 2011 at 1:29 PM, Florian Heigl > <florian.heigl@gmail.com <mailto:florian.heigl@gmail.com>> wrote: > > 2011/12/10 chris <tknchris@gmail.com <mailto:tknchris@gmail.com>>: > > +1 > > > > I hate how when the vnc connection gets hung for any reason how > you lose vnc > > access for quite a while and possibly forever > > So that makes 3 of us now for being able to restart vnc server :) > would be cool if there still were a Xen wishlist. > > Anyway, next time when that happens do a > netstat -na | grep 59 | grep _WAIT > > I think you''ll find a stuck connection there. Please let us know. > >-- Jordi Moles Blanco Sistemes Cdmon.com ___________________________ Tlf: 902 36 41 38 Tlf: 93 567 75 77 mailto: jordi@cdmon.com http://www.cdmon.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2011/12/12 Jordi Moles Blanco <jordi@cdmon.com>:> Hi everyone, > > well... it''s good to know that I haven''t come across some nasty and very > weird bug very difficult to reproduce. > > anyway... since the last time I wrote I haven''t seen this problem anymore, > as we have many machines and we don''t continuously check them all. Next time > I see one VNC broken I''ll try to check what you suggested. > > on the other hand, I understand from Florian''s words that the problem could > be a socket that remains open after a normal VNC connection. Do you think > that we could avoid rebooting the server by killing it?How do you want to kill it? I didn''t mean to imply this is something that ends the problem. EXCEPT if you find some workmate that still has an idling VNC terminal OR if you can make a stale session time out faster. I don''t know how to wipe a stale tcp session from the kernel Killing the qemu-dm VNC server is possible (but no more VNC access after that) If you kill, then restart it, it will not have the right options (port, password, etc) But maybe the newer qemu-dm versions work better with this - I don''t know. Lets find some more debug data once this happens for one of you and then you can contact the -devel list and try to work out a fix... Florian
Al 14/12/11 16:27, En/na Florian Heigl ha escrit:> 2011/12/12 Jordi Moles Blanco<jordi@cdmon.com>: >> Hi everyone, >> >> well... it''s good to know that I haven''t come across some nasty and very >> weird bug very difficult to reproduce. >> >> anyway... since the last time I wrote I haven''t seen this problem anymore, >> as we have many machines and we don''t continuously check them all. Next time >> I see one VNC broken I''ll try to check what you suggested. >> >> on the other hand, I understand from Florian''s words that the problem could >> be a socket that remains open after a normal VNC connection. Do you think >> that we could avoid rebooting the server by killing it? > How do you want to kill it? I didn''t mean to imply this is something > that ends the problem. > EXCEPT if you find some workmate that still has an idling VNC terminal > OR if you can > make a stale session time out faster. > > I don''t know how to wipe a stale tcp session from the kernel > Killing the qemu-dm VNC server is possible (but no more VNC access after that) > If you kill, then restart it, it will not have the right options > (port, password, etc) > > But maybe the newer qemu-dm versions work better with this - I don''t know. > Lets find some more debug data once this happens for one of you and > then you can > contact the -devel list and try to work out a fix...Hi, yes, sorry, I didn''t write that properly. You don''t just kill an stale socket (as far as I know). What I meant was that we may be able to check for a process that we can kill and that will hopefully free the connection. However, I''ve looked into this and found out that no new xen or qemu processes are launched when a VNC connection is stablished, so I guess that this is not the way. When the machine starts running Xen creates this kind of process: /usr/lib64/xen/bin/qemu-dm -d 146 -serial pty -domain-name server -videoram 4 -vnc 0.0.0.0:0,password -vncunused -M xenpv but when there''s an open connection there are no more processes like that or that contain "xen" or "qemu" or "vnc".> > > Florian > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- Jordi Moles Blanco Sistemes Cdmon.com ___________________________ Tlf: 902 36 41 38 Tlf: 93 567 75 77 mailto: jordi@cdmon.com http://www.cdmon.com
Hi 2011/12/14 Jordi Moles Blanco <jordi@cdmon.com>:> Al 14/12/11 16:27, En/na Florian Heigl ha escrit:> When the machine starts running Xen creates this kind of process: > > /usr/lib64/xen/bin/qemu-dm -d 146 -serial pty -domain-name server -videoram > 4 -vnc 0.0.0.0:0,password -vncunused -M xenpv > > but when there''s an open connection there are no more processes like that or > that contain "xen" or "qemu" or "vnc".Yes, this is the only process - the VNC server itself. The others will all be on a the connecting host. Try killing it on a test box and launching it with the very same parameters as it used to have. I think it might be possible if no custom settings (i.e. vncunused) are set in the VM config at all. Oh, and in fact it would be interesting to know where the VNC server comes from anyway. If you fell like finding out, have a look at the Xen udev rules, maybe it is there.