Xen devel - Aug 2006 - [PATCH] Ensure FD_CLOEXEC is set on all XenD file handles

While debugging an issue with libvirt I discovered a serious problem
with XenD''s management of file handles. Basically it does not set the
close-on-exec flag on any[1] of the file handles it has open. This means
that all XenD''s file handles propagated to programs it spawns - eg, the
network device setup scripts, qemu-dm and others those in turn spawn.

The particular case that lead me to discover the problem was when I disabled
HTTP port on XenD and wondered why port 8000 was still in LISTEN state - it
turned out qemu-dm had a handle to the server socket. Aside from this it had
access to about 10 open handles on /proc/xen/privcmd, the HTTP *client*
connection which requested the domain creation[2], and any of the other
server sockets XenD happened to have open.

I''m attaching three prototype patches - I say prototype because there
may
be other (better) places to set the FD_CLOEXEC flag - I''ve just picked
the place closest to the original socket()/accept() call.

 * xen-xend2-cloexec.patch - set the flag on the XMLRPC & HTTP server
   ports, both TCP & UNIX domain socket versions. Also ensure incoming
   clients have it set
 * xen-xc-cloexec.patch - set the flag on all handles to /proc/xen/privcmd
 * xen-xs2-cloexec.patch - set the flag on all connections to the
   xenstore daemon.

The only file handle I''ve not yet set the flag on is the one to the log
file /var/log/xend.log which (in my system) ends up on FD #9.

To test which file handles were being left open I created a dummy script:

  $ cat > /root/fake-qemu-dm.sh <<EOF
  #!/bin/sh

  lsof -P -n -p $$ >> /tmp/qemu-fds.log

  exec /usr/lib64/xen/bin/qemu-dm "$@"
  EOF
  $

And then in the domain''s config file set
''device_model'' to point to the
file ''/root/fake-qemu-dm.sh''


BTW, the patches were prepared against the latest Xen userspace code in
Fedora Core 6, test2 - this is trailing xen-unstable by a couple of weeks
but I think they should still apply. If people agree with the approach
taken in the patch I''ll re-diff against xen-unstable before posting
again.

Regards,
Dan.

[1] Actually it turns out the relocation port does have the flag set
[2] This explains an issue discovered with libvirt where it would hang 
     forever after creating an HVM domain waiting for the server to close
    its end of the socket
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 15/8/06 2:23 am, "Daniel P. Berrange" <berrange@redhat.com>
wrote:
> BTW, the patches were prepared against the latest Xen userspace code in
> Fedora Core 6, test2 - this is trailing xen-unstable by a couple of weeks
> but I think they should still apply. If people agree with the approach
> taken in the patch I''ll re-diff against xen-unstable before
posting again.
The patches look okay to me. Please re-send with a signed-off-by line.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Aug 15, 2006 at 10:53:01AM +0100, Keir Fraser
wrote:
> 
> On 15/8/06 2:23 am, "Daniel P. Berrange"
<berrange@redhat.com> wrote:
> 
> > BTW, the patches were prepared against the latest Xen userspace code
in
> > Fedora Core 6, test2 - this is trailing xen-unstable by a couple of
weeks
> > but I think they should still apply. If people agree with the approach
> > taken in the patch I''ll re-diff against xen-unstable before
posting again.
> 
> The patches look okay to me. Please re-send with a signed-off-by line.
I''m also attaching one extra patch
''xen-xend-logging-cloexec.patch'' which
sets the FD_CLOEXEC flag on the /var/log/xend.log  file. I''m not
entirely
happy with this patch though because it accesses the private
''self.stream''
field in its superclass. Unfortunately the entire python logging class
hierarchy is ''designed''  on the principle of accessing 
private class
members from superclasses, so I don''t see any immediately obvious
alternate
way to set FD_CLOEXEC on the log file.


A much more invasive patch to XenD would be to locate all places where we
call fork / exec and in between the forking & execing iterate over all
file handles explicitly setting FD_CLOEXEC, eg the equiv of this C code,
but in python

     pid = fork()
     if (pid == 0) {
        open_max = sysconf (_SC_OPEN_MAX);
        for (i = 0; i < open_max; i++)
            fcntl (i, F_SETFD, FD_CLOEXEC);

        exec(...)
     }

We''d also need to find all places where we call
''spawn'' and replace this call
with a fork/exec pair.

Attached the 3 previous patches & the new one to this mail. I''ve
tested that
they apply without trouble to latest xen-unstable.hg

  Signed-off-by: Daniel P. Berrange <berrange@redhat.com>

Regards
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel