Benjamin Weaver
2011-Oct-20 15:31 UTC
[Xen-users] domU shell prompt freezes after live migration
We are running xen 4.0 on Debian Squeeze in a two-node cluster. We are creating and running Ubuntu (lucid) vms. I have not had trouble with live migration of vms in the past, but on this installation I experience the following problem. When I create a vm I can log into it normally. That is, when I receive a prompt, I can type a command at the prompt, hit return and the command will execute, or if no command is given, the prompt is regenerated on the following line. I can migrate the vm from node to node without a problem. However, on the current installation, after the vm is migrated the first time, the shell, or prompt accessing the vm malfunctions by freezing up. When I log into the migrated vm, I do not receive a prompt. If, then I hit return, I receive the ubuntu server greeting, and then receive a prompt. But this prompt is frozen. Attempts to hit return do not produce a new prompt on a following line, and it is not possible to execute any commands. When I destroy the terminal window and repeat the process, the same thing happens. Any ideas what might be causing this problem? I include configuration data below (which we have always used successfully heretofore). And also I notice that xm top (which shows the vbd_rd and vbd_wr (virtual block device read and write request) information for the vm is 0 after migration is performed. Any help is greatly appreciated. Ben Weaver /etc/inittab [as specified in the Xen installation instructions]: 1:2345:respawn:/sbin/getty 38400 hvc0 2:23:respawn:/sbin/getty 38400 tty1 # NO getty on ttyS0! /etc/xend/xend-config.sxp: (xend-relocation-server yes) (xend-relocation-port 8002) (xend-relocation-address '''') (xend-relocation-hosts-allow '''') xm top output on vm after migration: ... vbd_rd = 0 vbd_wr = 0 (prior to migration xm top shows non-zero values for vbd_rd and vbd_wr). _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Benjamin Weaver
2011-Oct-25 13:46 UTC
[Xen-users] domU shell prompt freezes after live migration
I have narrowed down and further defined the problem using xm commands: 1. the problem does not have to do with migration from one node to another per se: an xm save and subsequent xm restore done from either box, without migration, reproduces the problem. In each case a vm is produced that I can ping and login to, but which produces the frozen prompt. I have also tried the following: 2. opened permissions on .img files under /ocfs2SAN/domains to no avail, therefore not a permissions problem 3. opened up sysctl buffers on both boxes to no avail not a sys buffers problem Might I perhaps have a file system or cluster configuration problem? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
David Della Vecchia
2011-Oct-25 14:19 UTC
Re: [Xen-users] domU shell prompt freezes after live migration
Are you on debian by chance? I had the exact same issue, it has to do with the kernel you are using for the domU. Here is the bug report describing it: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644604 I''ve had this occur with other kernel''s i''ve built, not just debian provided ones. I was able to build a 2.6.39.4 and a 3.0.1 kernel last night that restored my ability to do xm save and restore without the console lock up. Hope this helps. -David On Tue, Oct 25, 2011 at 9:46 AM, Benjamin Weaver < benjamin.weaver@phon.ox.ac.uk> wrote:> I have narrowed down and further defined the problem using xm commands: > > 1. the problem does not have to do with migration from one node to another > per se: > > an xm save and subsequent xm restore done from either box, without > migration, reproduces the problem. In each case a vm is produced that I can > ping and login to, but which produces the frozen prompt. > > I have also tried the following: > > 2. opened permissions on .img files under /ocfs2SAN/domains > to no avail, > therefore not a permissions problem > > 3. opened up sysctl buffers on both boxes > to no avail > not a sys buffers problem > > > > > Might I perhaps have a file system or cluster configuration problem? > > ______________________________**_________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/**xen-users<http://lists.xensource.com/xen-users> >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Benjamin Weaver
2011-Oct-25 14:37 UTC
Re: [Xen-users] domU shell prompt freezes after live migration
Thanks, David, This could be very useful. We are running Debian Squeeze for the dom0 (kernel: 2.6.32-5-xen-amd64), but Ubuntu (lucid) for the domUs. Ben On 25/10/11 15:19, David Della Vecchia wrote:> Are you on debian by chance? I had the exact same issue, it has to do > with the kernel you are using for the domU. > > Here is the bug report describing it: > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644604 > > I''ve had this occur with other kernel''s i''ve built, not just debian > provided ones. I was able to build a 2.6.39.4 and a 3.0.1 kernel last > night that restored my ability to do xm save and restore without the > console lock up. > > Hope this helps. > > -David > > On Tue, Oct 25, 2011 at 9:46 AM, Benjamin Weaver > <benjamin.weaver@phon.ox.ac.uk <mailto:benjamin.weaver@phon.ox.ac.uk>> > wrote: > > I have narrowed down and further defined the problem using xm > commands: > > 1. the problem does not have to do with migration from one node to > another per se: > > an xm save and subsequent xm restore done from either box, without > migration, reproduces the problem. In each case a vm is produced > that I can ping and login to, but which produces the frozen prompt. > > I have also tried the following: > > 2. opened permissions on .img files under /ocfs2SAN/domains > to no avail, > therefore not a permissions problem > > 3. opened up sysctl buffers on both boxes > to no avail > not a sys buffers problem > > > > > Might I perhaps have a file system or cluster configuration problem? > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com <mailto:Xen-users@lists.xensource.com> > http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
David Della Vecchia
2011-Oct-25 14:47 UTC
Re: [Xen-users] domU shell prompt freezes after live migration
Yep, 2.6.32-5-xen-amd64 is the one i personally confirm fails at live in this regard. I run xen4.1.1 in debian wheezy and the 2.6.39.4 kernel i got from kernel.org works flawlessly with the debian squeeze domU''s -DDV On Tue, Oct 25, 2011 at 10:37 AM, Benjamin Weaver < benjamin.weaver@phon.ox.ac.uk> wrote:> ** > Thanks, David, > > This could be very useful. We are running Debian Squeeze for the dom0 > (kernel: 2.6.32-5-xen-amd64), but Ubuntu (lucid) for the domUs. > > Ben > > > On 25/10/11 15:19, David Della Vecchia wrote: > > Are you on debian by chance? I had the exact same issue, it has to do with > the kernel you are using for the domU. > > Here is the bug report describing it: > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644604 > > I''ve had this occur with other kernel''s i''ve built, not just debian > provided ones. I was able to build a 2.6.39.4 and a 3.0.1 kernel last night > that restored my ability to do xm save and restore without the console lock > up. > > Hope this helps. > > -David > > On Tue, Oct 25, 2011 at 9:46 AM, Benjamin Weaver < > benjamin.weaver@phon.ox.ac.uk> wrote: > >> I have narrowed down and further defined the problem using xm commands: >> >> 1. the problem does not have to do with migration from one node to another >> per se: >> >> an xm save and subsequent xm restore done from either box, without >> migration, reproduces the problem. In each case a vm is produced that I can >> ping and login to, but which produces the frozen prompt. >> >> I have also tried the following: >> >> 2. opened permissions on .img files under /ocfs2SAN/domains >> to no avail, >> therefore not a permissions problem >> >> 3. opened up sysctl buffers on both boxes >> to no avail >> not a sys buffers problem >> >> >> >> >> Might I perhaps have a file system or cluster configuration problem? >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Benjamin Weaver
2011-Oct-25 17:45 UTC
Re: [Xen-users] domU shell prompt freezes after live migration
David, We arrived at what may be a related solution. Instead of lucid Ubuntu for the domUs, we dropped back to lenny (2.6.26-2-xen-amd64). The lenny domUs xm store, xm save, and migrate correctly. None of this explains why on previous installations of our cluster (ocfs2 on SAN + 2 Debian Squeeze dom0s), we did not have problems with lucid vms. But now the lenny s work. You put us onto the right track with this kernel version problem, and we may well soon be upgrading our dom0s as you suggested. Thanks very much for your help. On 25/10/11 15:47, David Della Vecchia wrote:> Yep, 2.6.32-5-xen-amd64 is the one i personally confirm fails at live > in this regard. > > I run xen4.1.1 in debian wheezy and the 2.6.39.4 kernel i got from > kernel.org <http://kernel.org> works flawlessly with the debian > squeeze domU''s > > -DDV > > On Tue, Oct 25, 2011 at 10:37 AM, Benjamin Weaver > <benjamin.weaver@phon.ox.ac.uk <mailto:benjamin.weaver@phon.ox.ac.uk>> > wrote: > > Thanks, David, > > This could be very useful. We are running Debian Squeeze for the > dom0 (kernel: 2.6.32-5-xen-amd64), but Ubuntu (lucid) for the domUs. > > Ben > > > On 25/10/11 15:19, David Della Vecchia wrote: >> Are you on debian by chance? I had the exact same issue, it has >> to do with the kernel you are using for the domU. >> >> Here is the bug report describing it: >> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644604 >> >> I''ve had this occur with other kernel''s i''ve built, not just >> debian provided ones. I was able to build a 2.6.39.4 and a 3.0.1 >> kernel last night that restored my ability to do xm save and >> restore without the console lock up. >> >> Hope this helps. >> >> -David >> >> On Tue, Oct 25, 2011 at 9:46 AM, Benjamin Weaver >> <benjamin.weaver@phon.ox.ac.uk >> <mailto:benjamin.weaver@phon.ox.ac.uk>> wrote: >> >> I have narrowed down and further defined the problem using xm >> commands: >> >> 1. the problem does not have to do with migration from one >> node to another per se: >> >> an xm save and subsequent xm restore done from either box, >> without migration, reproduces the problem. In each case a vm >> is produced that I can ping and login to, but which produces >> the frozen prompt. >> >> I have also tried the following: >> >> 2. opened permissions on .img files under /ocfs2SAN/domains >> to no avail, >> therefore not a permissions problem >> >> 3. opened up sysctl buffers on both boxes >> to no avail >> not a sys buffers problem >> >> >> >> >> Might I perhaps have a file system or cluster configuration >> problem? >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> <mailto:Xen-users@lists.xensource.com> >> http://lists.xensource.com/xen-users >> >> > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
David Della Vecchia
2011-Oct-25 17:45 UTC
Re: [Xen-users] domU shell prompt freezes after live migration
Your welcome. Looks like you fixed the problem by regressing, i tried to fix it by upgrading. I was able to fix my issue of the console freezing after the save and restore but now i am unable to get the networking to work in squeeze domu''s after restore. I was told in the #xen channel to use centos5 as my failsafe test and so far it has earned that title. Throughout all my tests the centos5 domU ALWAYS works with everything. Debian seems to be more finicky unfortunately. Perhaps i will regress as you have and see if that fixes my networking problem. Glad to help, DDV On Tue, Oct 25, 2011 at 1:45 PM, Benjamin Weaver < benjamin.weaver@phon.ox.ac.uk> wrote:> ** > David, > > We arrived at what may be a related solution. Instead of lucid Ubuntu for > the domUs, we dropped back to lenny (2.6.26-2-xen-amd64). The lenny domUs xm > store, xm save, and migrate correctly. > > None of this explains why on previous installations of our cluster (ocfs2 > on SAN + 2 Debian Squeeze dom0s), we did not have problems with lucid vms. > But now the lenny s work. > > You put us onto the right track with this kernel version problem, and we > may well soon be upgrading our dom0s as you suggested. > > Thanks very much for your help. > > On 25/10/11 15:47, David Della Vecchia wrote: > > Yep, 2.6.32-5-xen-amd64 is the one i personally confirm fails at live in > this regard. > > I run xen4.1.1 in debian wheezy and the 2.6.39.4 kernel i got from > kernel.org works flawlessly with the debian squeeze domU''s > > -DDV > > On Tue, Oct 25, 2011 at 10:37 AM, Benjamin Weaver < > benjamin.weaver@phon.ox.ac.uk> wrote: > >> Thanks, David, >> >> This could be very useful. We are running Debian Squeeze for the dom0 >> (kernel: 2.6.32-5-xen-amd64), but Ubuntu (lucid) for the domUs. >> >> Ben >> >> >> On 25/10/11 15:19, David Della Vecchia wrote: >> >> Are you on debian by chance? I had the exact same issue, it has to do with >> the kernel you are using for the domU. >> >> Here is the bug report describing it: >> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644604 >> >> I''ve had this occur with other kernel''s i''ve built, not just debian >> provided ones. I was able to build a 2.6.39.4 and a 3.0.1 kernel last night >> that restored my ability to do xm save and restore without the console lock >> up. >> >> Hope this helps. >> >> -David >> >> On Tue, Oct 25, 2011 at 9:46 AM, Benjamin Weaver < >> benjamin.weaver@phon.ox.ac.uk> wrote: >> >>> I have narrowed down and further defined the problem using xm commands: >>> >>> 1. the problem does not have to do with migration from one node to >>> another per se: >>> >>> an xm save and subsequent xm restore done from either box, without >>> migration, reproduces the problem. In each case a vm is produced that I can >>> ping and login to, but which produces the frozen prompt. >>> >>> I have also tried the following: >>> >>> 2. opened permissions on .img files under /ocfs2SAN/domains >>> to no avail, >>> therefore not a permissions problem >>> >>> 3. opened up sysctl buffers on both boxes >>> to no avail >>> not a sys buffers problem >>> >>> >>> >>> >>> Might I perhaps have a file system or cluster configuration problem? >>> >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com >>> http://lists.xensource.com/xen-users >>> >> >> >> > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
George Shuklin
2011-Oct-25 18:10 UTC
Re: [Xen-users] domU shell prompt freezes after live migration
The suse 2.6.34-xen and 2.6.37-xen works fine in my tests too. 3.1 seems be fine too but have some issues with ballooning (fixes was commited after 3.1 release) and with time (in xen 3). On 25.10.2011 21:45, David Della Vecchia wrote:> Your welcome. Looks like you fixed the problem by regressing, i tried > to fix it by upgrading. I was able to fix my issue of the console > freezing after the save and restore but now i am unable to get the > networking to work in squeeze domu''s after restore. > > I was told in the #xen channel to use centos5 as my failsafe test and > so far it has earned that title. Throughout all my tests the centos5 > domU ALWAYS works with everything. Debian seems to be more finicky > unfortunately. > > Perhaps i will regress as you have and see if that fixes my networking > problem. > > Glad to help, > DDV > > On Tue, Oct 25, 2011 at 1:45 PM, Benjamin Weaver > <benjamin.weaver@phon.ox.ac.uk <mailto:benjamin.weaver@phon.ox.ac.uk>> > wrote: > > David, > > We arrived at what may be a related solution. Instead of lucid > Ubuntu for the domUs, we dropped back to lenny > (2.6.26-2-xen-amd64). The lenny domUs xm store, xm save, and > migrate correctly. > > None of this explains why on previous installations of our cluster > (ocfs2 on SAN + 2 Debian Squeeze dom0s), we did not have problems > with lucid vms. But now the lenny s work. > > You put us onto the right track with this kernel version problem, > and we may well soon be upgrading our dom0s as you suggested. > > Thanks very much for your help. > > On 25/10/11 15:47, David Della Vecchia wrote: >> Yep, 2.6.32-5-xen-amd64 is the one i personally confirm fails at >> live in this regard. >> >> I run xen4.1.1 in debian wheezy and the 2.6.39.4 kernel i got >> from kernel.org <http://kernel.org> works flawlessly with the >> debian squeeze domU''s >> >> -DDV >> >> On Tue, Oct 25, 2011 at 10:37 AM, Benjamin Weaver >> <benjamin.weaver@phon.ox.ac.uk >> <mailto:benjamin.weaver@phon.ox.ac.uk>> wrote: >> >> Thanks, David, >> >> This could be very useful. We are running Debian Squeeze for >> the dom0 (kernel: 2.6.32-5-xen-amd64), but Ubuntu (lucid) for >> the domUs. >> >> Ben >> >> >> On 25/10/11 15:19, David Della Vecchia wrote: >>> Are you on debian by chance? I had the exact same issue, it >>> has to do with the kernel you are using for the domU. >>> >>> Here is the bug report describing it: >>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644604 >>> >>> I''ve had this occur with other kernel''s i''ve built, not just >>> debian provided ones. I was able to build a 2.6.39.4 and a >>> 3.0.1 kernel last night that restored my ability to do xm >>> save and restore without the console lock up. >>> >>> Hope this helps. >>> >>> -David >>> >>> On Tue, Oct 25, 2011 at 9:46 AM, Benjamin Weaver >>> <benjamin.weaver@phon.ox.ac.uk >>> <mailto:benjamin.weaver@phon.ox.ac.uk>> wrote: >>> >>> I have narrowed down and further defined the problem >>> using xm commands: >>> >>> 1. the problem does not have to do with migration from >>> one node to another per se: >>> >>> an xm save and subsequent xm restore done from either >>> box, without migration, reproduces the problem. In each >>> case a vm is produced that I can ping and login to, but >>> which produces the frozen prompt. >>> >>> I have also tried the following: >>> >>> 2. opened permissions on .img files under /ocfs2SAN/domains >>> to no avail, >>> therefore not a permissions problem >>> >>> 3. opened up sysctl buffers on both boxes >>> to no avail >>> not a sys buffers problem >>> >>> >>> >>> >>> Might I perhaps have a file system or cluster >>> configuration problem? >>> >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com >>> <mailto:Xen-users@lists.xensource.com> >>> http://lists.xensource.com/xen-users >>> >>> >> >> > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users