Xen users - Dec 2005 - xm create of a VMX fails - qemu-dm gets SIGSEGV

Greetings, all.

I''m having trouble getting a VMX domain started.  I''m running
on changeset
8119.  Shortly after the "xm create" finishes the qemu-dm process goes
defunct.  I wrote a little script so that strace could attach to the
qemu-dm process right after the "xm create" finishes:

xm create $1
pid=`ps ax|grep qemu-dm|grep -v grep|awk ''{print $1}''`
echo qemu-dm pid is $pid
echo strace -p $pid
strace -p $pid

Here is a sample of the strace output from one of the runs:
(See attached file: strace.qemu-dm)

It appears that the qemu-dm process is getting a SIGSEGV.

I noticed from some other runs that the qemu-dm process is getting launched
twice:

[root@dib ~]# xm create /tmp/vmx1.cfg;ps ax|grep qemu
Using config file "/tmp/vmx1.cfg".
Started domain vmx1
 9583 ?        R      0:00 /usr/lib/xen/bin/qemu-dm -d 16 -p 3 -m 768 -boot
c -localtime -serial pty -vcpus 1 -hda /dev/sdb -macaddr 00:16:3e:61:5f:87
-bridge xenbr0 -nographic
 9586 ?        R      0:00 /usr/lib/xen/bin/qemu-dm -d 16 -p 3 -m 768 -boot
c -localtime -serial pty -vcpus 1 -hda /dev/sdb -macaddr 00:16:3e:61:5f:87
-bridge xenbr0 -nographic
 9590 pts/1    S+     0:00 grep qemu

On another occasion it was the qemu-ifup that had two occurrences running:

[root@dib ~]# xm create /tmp/vmx1.cfg;ps ax|grep qemu
Using config file "/tmp/vmx1.cfg".
Started domain vmx1
10019 ?        S      0:00 /usr/lib/xen/bin/qemu-dm -d 17 -p 3 -m 768 -boot
c -localtime -serial pty -vcpus 1 -hda /dev/sdb -macaddr 00:16:3e:53:48:e4
-bridge xenbr0 -nographic
10022 ?        R      0:00 /bin/sh /etc/xen/qemu-ifup tun0 xenbr0
10023 ?        R      0:00 /bin/sh /etc/xen/qemu-ifup tun0 xenbr0
10027 pts/0    S+     0:00 grep qemu

Is this normal?  Two occurrences of the same program sounds like asking for
contention trouble, but then I am fairly clueless about the mechanics of
how VMX sessions work.

I will note that this worked on the same machine with changeset 8112.
(Well, I only once saw a VMX boot all the way once.  Subsequent attempts
using vnc, sdl, and serial output all started to boot but each ran into its
own problems during the boot.)  Now nothing appears to start since qemu-dm
dies shortly after it is launched.

Here is the output from xen-bugtool.  I should note that it is from a run
after the run done with the strace above.
(See attached file: xen-bugtool_xm-create-vmx.tar.bz2)

Here is the config file for creating the VMX domain.
(See attached file: vmx1.cfg)

Any help on solving the problem or pointers to where to look will be
appreciated.

Thanks,
Steve D.

P.S.  I notice that "xm list" lists the VMX domain in the blocked
state.
Is this a bug?  It seems to me that if qemu is dead that Xen should pick up
on that, clean up its records of the domain and take the appropriate action
for a crashed domU.  Just a thought.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

If you take "serial=''pty''" out of your config, does
the vmx domain start
for you?

I think I''m hitting the same thing you are.

Thanks,
Dan


On Wed, 2005-11-30 at 18:09 -0600, Steve Dobbelstein
wrote:
> Greetings, all.
> 
> I''m having trouble getting a VMX domain started.  I''m
running on changeset
> 8119.  Shortly after the "xm create" finishes the qemu-dm process
goes
> defunct.  I wrote a little script so that strace could attach to the
> qemu-dm process right after the "xm create" finishes:
> 
> xm create $1
> pid=`ps ax|grep qemu-dm|grep -v grep|awk ''{print $1}''`
> echo qemu-dm pid is $pid
> echo strace -p $pid
> strace -p $pid
> 
> Here is a sample of the strace output from one of the runs:
> (See attached file: strace.qemu-dm)
> 
> It appears that the qemu-dm process is getting a SIGSEGV.
> 
> I noticed from some other runs that the qemu-dm process is getting launched
> twice:
> 
> [root@dib ~]# xm create /tmp/vmx1.cfg;ps ax|grep qemu
> Using config file "/tmp/vmx1.cfg".
> Started domain vmx1
>  9583 ?        R      0:00 /usr/lib/xen/bin/qemu-dm -d 16 -p 3 -m 768 -boot
> c -localtime -serial pty -vcpus 1 -hda /dev/sdb -macaddr 00:16:3e:61:5f:87
> -bridge xenbr0 -nographic
>  9586 ?        R      0:00 /usr/lib/xen/bin/qemu-dm -d 16 -p 3 -m 768 -boot
> c -localtime -serial pty -vcpus 1 -hda /dev/sdb -macaddr 00:16:3e:61:5f:87
> -bridge xenbr0 -nographic
>  9590 pts/1    S+     0:00 grep qemu
> 
> On another occasion it was the qemu-ifup that had two occurrences running:
> 
> [root@dib ~]# xm create /tmp/vmx1.cfg;ps ax|grep qemu
> Using config file "/tmp/vmx1.cfg".
> Started domain vmx1
> 10019 ?        S      0:00 /usr/lib/xen/bin/qemu-dm -d 17 -p 3 -m 768 -boot
> c -localtime -serial pty -vcpus 1 -hda /dev/sdb -macaddr 00:16:3e:53:48:e4
> -bridge xenbr0 -nographic
> 10022 ?        R      0:00 /bin/sh /etc/xen/qemu-ifup tun0 xenbr0
> 10023 ?        R      0:00 /bin/sh /etc/xen/qemu-ifup tun0 xenbr0
> 10027 pts/0    S+     0:00 grep qemu
> 
> Is this normal?  Two occurrences of the same program sounds like asking for
> contention trouble, but then I am fairly clueless about the mechanics of
> how VMX sessions work.
> 
> I will note that this worked on the same machine with changeset 8112.
> (Well, I only once saw a VMX boot all the way once.  Subsequent attempts
> using vnc, sdl, and serial output all started to boot but each ran into its
> own problems during the boot.)  Now nothing appears to start since qemu-dm
> dies shortly after it is launched.
> 
> Here is the output from xen-bugtool.  I should note that it is from a run
> after the run done with the strace above.
> (See attached file: xen-bugtool_xm-create-vmx.tar.bz2)
> 
> Here is the config file for creating the VMX domain.
> (See attached file: vmx1.cfg)
> 
> Any help on solving the problem or pointers to where to look will be
> appreciated.
> 
> Thanks,
> Steve D.
> 
> P.S.  I notice that "xm list" lists the VMX domain in the blocked
state.
> Is this a bug?  It seems to me that if qemu is dead that Xen should pick up
> on that, clean up its records of the domain and take the appropriate action
> for a crashed domU.  Just a thought.
> _______________________________________________ Xen-users mailing list
Xen-users@lists.xensource.com http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wednesday 30 November 2005 16:09, Steve Dobbelstein
wrote:
> Greetings, all.
>
> I''m having trouble getting a VMX domain started.  I''m
running on changeset
> 8119.  Shortly after the "xm create" finishes the qemu-dm process
goes
> defunct.  I wrote a little script so that strace could attach to the
> qemu-dm process right after the "xm create" finishes:
The problem is ptsname(slave_fd) is returning NULL in the ioemu/vl.c code that 
was introduced in changeset 8118:590915af8117. 

I''ve changed ptsname to ttyname() and it works. Not sure if this is the
right
fix. 

Signed-off-by: Daniel Stekloff <dsteklof@us.ibm.com>



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Thanks Daniel. :-)
The protection mechanism is a little weak.  Here is my patch. 

diff -r 28ce48573771 tools/ioemu/vl.c
--- a/tools/ioemu/vl.c  Wed Nov 30 19:55:08 2005
+++ b/tools/ioemu/vl.c  Thu Dec  1 17:00:21 2005
@@ -1218,17 +1218,22 @@
 CharDriverState *qemu_chr_open_pty(void)
 {
     int master_fd, slave_fd;
+    char *slave_name;
     struct termios term;

     if (openpty(&master_fd, &slave_fd, NULL, NULL, NULL) < 0)
         return NULL;

-    /* Set raw attributes on the pty. */
-    cfmakeraw(&term);
-    tcsetattr(slave_fd, TCSAFLUSH, &term);
-
-    fprintf(stderr, "char device redirected to %s\n",
ptsname(slave_fd));
-    store_console_dev(domid, ptsname(slave_fd));
+    slave_name = ptsname(master_fd);
+    if (slave_name) {
+       fprintf(stderr, "char device redirected to %s\n", slave_name);
+       store_console_dev(domid, slave_name);
+       /* Set raw attributes on the pty. */
+        cfmakeraw(&term);
+        tcsetattr(slave_fd, TCSAFLUSH, &term);
+    } else {
+        fprintf(logfile, "failed to redirect char device to pty");
+    }

     return qemu_chr_open_fd(master_fd, master_fd);
 }

 
>-----Original Message-----
>From: xen-users-bounces@lists.xensource.com
>[mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Daniel Stekloff
>Sent: Thursday, December 01, 2005 3:57 PM
>To: xen-users@lists.xensource.com; Steve Dobbelstein
>Cc: xen-devel@lists.xensource.com
>Subject: Re: [Xen-users] xm create of a VMX fails - qemu-dm gets SIGSEGV
>
>On Wednesday 30 November 2005 16:09, Steve Dobbelstein wrote:
>> Greetings, all.
>>
>> I'm having trouble getting a VMX domain started.  I'm running
on changeset
>> 8119.  Shortly after the "xm create" finishes the qemu-dm
process goes
>> defunct.  I wrote a little script so that strace could attach to the
>> qemu-dm process right after the "xm create" finishes:
>
>The problem is ptsname(slave_fd) is returning NULL in the ioemu/vl.c code
that
>was introduced in changeset 8118:590915af8117.
>
>I've changed ptsname to ttyname() and it works. Not sure if this is the
right
>fix.
>
>Signed-off-by: Daniel Stekloff <dsteklof@us.ibm.com>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Add a signed-off line in attachment.

>-----Original Message-----
>From: xen-users-bounces@lists.xensource.com
>[mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Yu, Ping Y
>Sent: Thursday, December 01, 2005 5:02 PM
>To: Daniel Stekloff; xen-users@lists.xensource.com; Steve Dobbelstein
>Cc: xen-devel@lists.xensource.com
>Subject: RE: [Xen-users] xm create of a VMX fails - qemu-dm gets SIGSEGV
>
>Thanks Daniel. :-)
>The protection mechanism is a little weak.  Here is my patch.
>
>diff -r 28ce48573771 tools/ioemu/vl.c
>--- a/tools/ioemu/vl.c  Wed Nov 30 19:55:08 2005
>+++ b/tools/ioemu/vl.c  Thu Dec  1 17:00:21 2005
>@@ -1218,17 +1218,22 @@
> CharDriverState *qemu_chr_open_pty(void)
> {
>     int master_fd, slave_fd;
>+    char *slave_name;
>     struct termios term;
>
>     if (openpty(&master_fd, &slave_fd, NULL, NULL, NULL) < 0)
>         return NULL;
>
>-    /* Set raw attributes on the pty. */
>-    cfmakeraw(&term);
>-    tcsetattr(slave_fd, TCSAFLUSH, &term);
>-
>-    fprintf(stderr, "char device redirected to %s\n",
ptsname(slave_fd));
>-    store_console_dev(domid, ptsname(slave_fd));
>+    slave_name = ptsname(master_fd);
>+    if (slave_name) {
>+       fprintf(stderr, "char device redirected to %s\n",
slave_name);
>+       store_console_dev(domid, slave_name);
>+       /* Set raw attributes on the pty. */
>+        cfmakeraw(&term);
>+        tcsetattr(slave_fd, TCSAFLUSH, &term);
>+    } else {
>+        fprintf(logfile, "failed to redirect char device to
pty");
>+    }
>
>     return qemu_chr_open_fd(master_fd, master_fd);
> }
>
>
>
>>-----Original Message-----
>>From: xen-users-bounces@lists.xensource.com
>>[mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Daniel
Stekloff
>>Sent: Thursday, December 01, 2005 3:57 PM
>>To: xen-users@lists.xensource.com; Steve Dobbelstein
>>Cc: xen-devel@lists.xensource.com
>>Subject: Re: [Xen-users] xm create of a VMX fails - qemu-dm gets SIGSEGV
>>
>>On Wednesday 30 November 2005 16:09, Steve Dobbelstein wrote:
>>> Greetings, all.
>>>
>>> I'm having trouble getting a VMX domain started.  I'm
running on changeset
>>> 8119.  Shortly after the "xm create" finishes the qemu-dm
process goes
>>> defunct.  I wrote a little script so that strace could attach to
the
>>> qemu-dm process right after the "xm create" finishes:
>>
>>The problem is ptsname(slave_fd) is returning NULL in the ioemu/vl.c
code that
>>was introduced in changeset 8118:590915af8117.
>>
>>I've changed ptsname to ttyname() and it works. Not sure if this is
the right
>>fix.
>>
>>Signed-off-by: Daniel Stekloff <dsteklof@us.ibm.com>


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 1 Dec 2005, at 07:56, Daniel Stekloff wrote:
> The problem is ptsname(slave_fd) is returning NULL in the ioemu/vl.c 
> code that
> was introduced in changeset 8118:590915af8117.
>
> I''ve changed ptsname to ttyname() and it works. Not sure if this
is
> the right
> fix.
ptsname() should be run on the master_fd, not the slave_fd. I''ll check 
in the trivial fix.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

"Yu, Ping Y" <ping.y.yu@intel.com> wrote on 12/01/2005 03:01:30
AM:
> Thanks Daniel. :-)
> The protection mechanism is a little weak.  Here is my patch.
>
> diff -r 28ce48573771 tools/ioemu/vl.c
> --- a/tools/ioemu/vl.c  Wed Nov 30 19:55:08 2005
> +++ b/tools/ioemu/vl.c  Thu Dec  1 17:00:21 2005
> @@ -1218,17 +1218,22 @@
>  CharDriverState *qemu_chr_open_pty(void)
>  {
>      int master_fd, slave_fd;
> +    char *slave_name;
>      struct termios term;
>
>      if (openpty(&master_fd, &slave_fd, NULL, NULL, NULL) < 0)
>          return NULL;
>
> -    /* Set raw attributes on the pty. */
> -    cfmakeraw(&term);
> -    tcsetattr(slave_fd, TCSAFLUSH, &term);
> -
> -    fprintf(stderr, "char device redirected to %s\n",
ptsname(slave_fd));
> -    store_console_dev(domid, ptsname(slave_fd));
> +    slave_name = ptsname(master_fd);
> +    if (slave_name) {
> +       fprintf(stderr, "char device redirected to %s\n",
slave_name);
> +       store_console_dev(domid, slave_name);
> +       /* Set raw attributes on the pty. */
> +        cfmakeraw(&term);
> +        tcsetattr(slave_fd, TCSAFLUSH, &term);
> +    } else {
> +        fprintf(logfile, "failed to redirect char device to
pty");
> +    }
>
>      return qemu_chr_open_fd(master_fd, master_fd);
>  }
Thanks!  The patch gets my VMX domain to start booting.  Now I have to
figure out why it is running so s-----l-----o-----w-----l-----y.

Steve D.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Thanks!  The patch gets my VMX domain to start booting.  Now 
> I have to figure out why it is running so 
> s-----l-----o-----w-----l-----y.
Odd. I wander if one of the recent VT patches to change the interrupt
service order has broken things?

I''m sure the Intel guys will be all over this :-)

Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

"Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> wrote on 12/01/2005
02:59:48 PM:
> > Thanks!  The patch gets my VMX domain to start booting.  Now
> > I have to figure out why it is running so
> > s-----l-----o-----w-----l-----y.
>
> Odd. I wander if one of the recent VT patches to change the interrupt
> service order has broken things?
>
> I''m sure the Intel guys will be all over this :-)
>
> Ian
Found the problem.  The X server on my dom0 was hogging the CPU leaving
very little for the domU.  I had switched to runlevel 5 so that I could get
VNC to work.  I booted into run level 3 and changed the config file for the
VMX domain to use serial instead of VNC.  I am now happily running tests in
the VMX domain.

Steve D.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel