I ran into an issue when attempting to resume an HVM guest using the libxl_domain_create_restore call. We are using this in the libxl-version of xenopsd, XenServer''s domain manager. The VM''s suspend image is stored on a disk that is mounted in dom0 for the duration of the restore operation, and unmounted afterwards. Xenopsd opens the suspend-image file, and hands the file descriptor over to libxl, as a parameter of libxl_domain_create_restore. Libxl successfully created the domain, loaded the memory image, and spawned qemu. However, the following attempt to unmount the disk that hold the suspend image failed. It turned out that qemu was holding on to the fd of the suspend-image, which blocked the umount. Our theory is that when libxl forks and executes the qemu process, qemu inherits all currentl open fds in the xenopsd process, and never gives them away anymore. We could solve this by opening the suspend-image file in xenopsd using the O_CLOEXEC flag, causing the file to be closed when executing qemu. However, we are worried that this behaviour may still cause trouble in multi-threaded programs such as xenopsd, unless all files it ever opens are opened with O_CLOEXEC, which is easily forgotten. Would it be possible for libxl to spawn qemu in such a way that it won''t inherit open files? Cheers, Rob _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Fri, 2013-10-18 at 15:26 +0000, Rob Hoes wrote:> We could solve this by opening the suspend-image file in xenopsd using > the O_CLOEXEC flag, causing the file to be closed when executing qemu. > However, we are worried that this behaviour may still cause trouble in > multi-threaded programs such as xenopsd, unless all files it ever > opens are opened with O_CLOEXEC, which is easily forgotten.I''m afraid this is exactly what you need to do. libxl itself jumps through considerable hoops to ensure it does this itself, it''s extra hard in a library... (have a look= for CLOEXEC in tools/libxl/*.h for some details) xenopsd being the consuming app should have an easier time of it.> Would it be possible for libxl to spawn qemu in such a way that it > won''t inherit open files?I don''t think POSIX has any functionality which allows for that to be reliably implemented. Ian.
> On Fri, 2013-10-18 at 15:26 +0000, Rob Hoes wrote: > > > We could solve this by opening the suspend-image file in xenopsd using > > the O_CLOEXEC flag, causing the file to be closed when executing qemu. > > However, we are worried that this behaviour may still cause trouble in > > multi-threaded programs such as xenopsd, unless all files it ever > > opens are opened with O_CLOEXEC, which is easily forgotten. > > I''m afraid this is exactly what you need to do. > > libxl itself jumps through considerable hoops to ensure it does this itself, > it''s extra hard in a library... (have a look= for CLOEXEC in tools/libxl/*.h for > some details)Alright, thanks. I''ll look into that. Cheers, Rob> xenopsd being the consuming app should have an easier time of it. > > > Would it be possible for libxl to spawn qemu in such a way that it > > won''t inherit open files? > > I don''t think POSIX has any functionality which allows for that to be reliably > implemented. > > Ian. >
On Fri, 2013-10-18 at 17:12 +0100, Rob Hoes wrote:> > On Fri, 2013-10-18 at 15:26 +0000, Rob Hoes wrote: > > > > > We could solve this by opening the suspend-image file in xenopsd using > > > the O_CLOEXEC flag, causing the file to be closed when executing qemu. > > > However, we are worried that this behaviour may still cause trouble in > > > multi-threaded programs such as xenopsd, unless all files it ever > > > opens are opened with O_CLOEXEC, which is easily forgotten. > > > > I''m afraid this is exactly what you need to do. > > > > libxl itself jumps through considerable hoops to ensure it does this itself, > > it''s extra hard in a library... (have a look= for CLOEXEC in tools/libxl/*.h for > > some details) > > Alright, thanks. I''ll look into that.BTW, you might find libxl_fd_set_cloexec helpful...