Libguestfs - Apr 2020 - Re: [PATCH virt-v2v] v2v: Allow temporary directory to be set on a global basis.

On Thu, Apr 02, 2020 at 03:21:14PM +0200, Pino Toscano
wrote:
> On Thursday, 2 April 2020 14:49:18 CEST Richard W.M. Jones wrote:
> > Previously we placed large files in g#get_cachedir () (usually
> > /var/tmp).  However the problem is this ties the libguestfs appliance
> > and the virt-v2v overlay files to the same location.
> > 
> > When virt-v2v is run in a container, or any other situation where
> > local storage is limited, it's helpful to be able to put the
overlay
> > files on an externally mounted PVC, which might be using NFS and
> > shared between containers.  But putting the libguestfs appliance on
> > NFS in a shared location is certainly not recommended.
> > 
> > This allows the two locations to be set separately:
> > 
> >   VIRT_V2V_TMPDIR - location of large temporary files, can use NFS
> >                     and may be shared
> > 
> >   LIBGUESTFS_CACHEDIR - location of libguestfs appliance
> > 
> > Another motivation for this patch is to allow more reliable cleanup of
> > temporary files by an external process, as described in the updated
> > documentation.
> > ---
> 
> I do not understand the motivation behind this, which adds yet another
> location with temporary files in addition to:
> - LIBGUESTFS_TMPDIR - $TMPDIR by default (which itself is /tmp by
>   default)
> - LIBGUESTFS_CACHEDIR - /var/tmp by default (with a .guestfs-$UID
>   subdirectory for the appliance)
> 
> Before this patch, almost all the temporary files are stored directly
> or in subdirectories of $TMPDIR, except big files such as overlays and
> OVA extracted content that are in CACHEDIR. With the proposed changes,
> _all_ the temporary files will be in CACHEDIR, so there are the
> following problems:
> - this directory will be cluttered with a lot more files than before
> - if it is shared, then other places where it is mounted will see the
>   same files
> - if it is shared, then creating temporary files will possibly mean
>   doing network I/O
> - if virt-v2v exits uncleantly, there will be a lot more files to
>   cleanup than now
> - even without being shared, /var/tmp is persistent unlike /tmp (which
>   can be tmpfs-backed on some distros/setups), meaning old temporary
>   files will linger way more
How about if we confine the change to just large files (ie. ones
which are currently placed in cachedir)?
> That said, it is not clear to me why /var/tmp should be shared among
> containers.
I also don't think /var/tmp should be shared, nor use NFS.

However the way that virt-v2v works at the moment means you cannot put
the large files (especially v2vovl*.qcow2) in a different place from
the libguestfs appliance.  This means that if you have only "some"
space in /var/tmp -- enough for the appliance, but not enough for the
potentially much larger space required by v2vovl*.qcow2 with multiple
copies of virt-v2v running -- then you cannot separate the overlays to
another directory.  This isn't just a problem for containers.

Another issue here is that with kubenetes it's not up to the tenant to
choose what backing store is used for volume mounts.  So if the
administrator of the cluster set it up with NFS, then that's what you
have to use.  (However using NFS for /var/tmp is still wrong).

That's the problem we're trying to solve.
> > diff --git a/docs/virt-v2v.pod b/docs/virt-v2v.pod
> > index ed95fdc8e..dbfd10cad 100644
> > --- a/docs/virt-v2v.pod
> > +++ b/docs/virt-v2v.pod
> > @@ -1206,8 +1206,9 @@ possible.
> >  
> >  =head3 Disk space
> >  
> > -Virt-v2v places potentially large temporary files in C<$TMPDIR>
(which
> > -is F</var/tmp> if you don't set it).  Using tmpfs is a bad
idea.
> 
> Regardless of this patch, this bit is not correct:
> - libguestfs does not places large files in $TMPDIR (but in CACHEDIR)
> - $TMPDIR is not /var/tmp by default, but /tmp
Yes this needs fixing separately.  I forgot $TMPDIR is not
related to /var/tmp.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

On Thursday, 2 April 2020 17:30:39 CEST Richard W.M. Jones
wrote:
> On Thu, Apr 02, 2020 at 03:21:14PM +0200, Pino Toscano wrote:
> > On Thursday, 2 April 2020 14:49:18 CEST Richard W.M. Jones wrote:
> > > Previously we placed large files in g#get_cachedir () (usually
> > > /var/tmp).  However the problem is this ties the libguestfs
appliance
> > > and the virt-v2v overlay files to the same location.
> > > 
> > > When virt-v2v is run in a container, or any other situation where
> > > local storage is limited, it's helpful to be able to put the
overlay
> > > files on an externally mounted PVC, which might be using NFS and
> > > shared between containers.  But putting the libguestfs appliance
on
> > > NFS in a shared location is certainly not recommended.
> > > 
> > > This allows the two locations to be set separately:
> > > 
> > >   VIRT_V2V_TMPDIR - location of large temporary files, can use
NFS
> > >                     and may be shared
> > > 
> > >   LIBGUESTFS_CACHEDIR - location of libguestfs appliance
> > > 
> > > Another motivation for this patch is to allow more reliable
cleanup of
> > > temporary files by an external process, as described in the
updated
> > > documentation.
> > > ---
> > 
> > I do not understand the motivation behind this, which adds yet another
> > location with temporary files in addition to:
> > - LIBGUESTFS_TMPDIR - $TMPDIR by default (which itself is /tmp by
> >   default)
> > - LIBGUESTFS_CACHEDIR - /var/tmp by default (with a .guestfs-$UID
> >   subdirectory for the appliance)
> > 
> > Before this patch, almost all the temporary files are stored directly
> > or in subdirectories of $TMPDIR, except big files such as overlays and
> > OVA extracted content that are in CACHEDIR. With the proposed changes,
> > _all_ the temporary files will be in CACHEDIR, so there are the
> > following problems:
> > - this directory will be cluttered with a lot more files than before
> > - if it is shared, then other places where it is mounted will see the
> >   same files
> > - if it is shared, then creating temporary files will possibly mean
> >   doing network I/O
> > - if virt-v2v exits uncleantly, there will be a lot more files to
> >   cleanup than now
> > - even without being shared, /var/tmp is persistent unlike /tmp (which
> >   can be tmpfs-backed on some distros/setups), meaning old temporary
> >   files will linger way more
> 
> How about if we confine the change to just large files (ie. ones
> which are currently placed in cachedir)?
This is already the case, isn't it?
> However the way that virt-v2v works at the moment means you cannot put
> the large files (especially v2vovl*.qcow2) in a different place from
> the libguestfs appliance.  This means that if you have only
"some"
> space in /var/tmp -- enough for the appliance, but not enough for the
> potentially much larger space required by v2vovl*.qcow2 with multiple
> copies of virt-v2v running -- then you cannot separate the overlays to
> another directory.  This isn't just a problem for containers.
/var/tmp is a temporary directory, hence it ought to have enough space
for big temporary files. This is nothing special for libguestfs or
virt-v2v.

I do not see what makes v2vovl*.qcow2 files so special in this regard,
requiring them to be handled specially than other big temporary files
already stored by libguestfs/virt-v2v. Examples:
- the appliance
- the temporary directory for qemu when using the libvirt backend
- the extracted content of an OVA, in case we cannot read it directly
- the disks when exporting to glance (-o glance)

-- 
Pino Toscano

On Thu, Apr 02, 2020 at 05:51:32PM +0200, Pino Toscano
wrote:
> On Thursday, 2 April 2020 17:30:39 CEST Richard W.M. Jones wrote:
> > On Thu, Apr 02, 2020 at 03:21:14PM +0200, Pino Toscano wrote:
> > > On Thursday, 2 April 2020 14:49:18 CEST Richard W.M. Jones wrote:
> > > > Previously we placed large files in g#get_cachedir ()
(usually
> > > > /var/tmp).  However the problem is this ties the libguestfs
appliance
> > > > and the virt-v2v overlay files to the same location.
> > > > 
> > > > When virt-v2v is run in a container, or any other situation
where
> > > > local storage is limited, it's helpful to be able to put
the overlay
> > > > files on an externally mounted PVC, which might be using NFS
and
> > > > shared between containers.  But putting the libguestfs
appliance on
> > > > NFS in a shared location is certainly not recommended.
> > > > 
> > > > This allows the two locations to be set separately:
> > > > 
> > > >   VIRT_V2V_TMPDIR - location of large temporary files, can
use NFS
> > > >                     and may be shared
> > > > 
> > > >   LIBGUESTFS_CACHEDIR - location of libguestfs appliance
> > > > 
> > > > Another motivation for this patch is to allow more reliable
cleanup of
> > > > temporary files by an external process, as described in the
updated
> > > > documentation.
> > > > ---
> > > 
> > > I do not understand the motivation behind this, which adds yet
another
> > > location with temporary files in addition to:
> > > - LIBGUESTFS_TMPDIR - $TMPDIR by default (which itself is /tmp by
> > >   default)
> > > - LIBGUESTFS_CACHEDIR - /var/tmp by default (with a .guestfs-$UID
> > >   subdirectory for the appliance)
> > > 
> > > Before this patch, almost all the temporary files are stored
directly
> > > or in subdirectories of $TMPDIR, except big files such as
overlays and
> > > OVA extracted content that are in CACHEDIR. With the proposed
changes,
> > > _all_ the temporary files will be in CACHEDIR, so there are the
> > > following problems:
> > > - this directory will be cluttered with a lot more files than
before
> > > - if it is shared, then other places where it is mounted will see
the
> > >   same files
> > > - if it is shared, then creating temporary files will possibly
mean
> > >   doing network I/O
> > > - if virt-v2v exits uncleantly, there will be a lot more files to
> > >   cleanup than now
> > > - even without being shared, /var/tmp is persistent unlike /tmp
(which
> > >   can be tmpfs-backed on some distros/setups), meaning old
temporary
> > >   files will linger way more
> > 
> > How about if we confine the change to just large files (ie. ones
> > which are currently placed in cachedir)?
> 
> This is already the case, isn't it?
> 
> > However the way that virt-v2v works at the moment means you cannot put
> > the large files (especially v2vovl*.qcow2) in a different place from
> > the libguestfs appliance.  This means that if you have only
"some"
> > space in /var/tmp -- enough for the appliance, but not enough for the
> > potentially much larger space required by v2vovl*.qcow2 with multiple
> > copies of virt-v2v running -- then you cannot separate the overlays to
> > another directory.  This isn't just a problem for containers.
> 
> /var/tmp is a temporary directory, hence it ought to have enough space
> for big temporary files. This is nothing special for libguestfs or
> virt-v2v.
> 
> I do not see what makes v2vovl*.qcow2 files so special in this regard,
> requiring them to be handled specially than other big temporary files
> already stored by libguestfs/virt-v2v. Examples:
> - the appliance
> - the temporary directory for qemu when using the libvirt backend
> - the extracted content of an OVA, in case we cannot read it directly
> - the disks when exporting to glance (-o glance)
"Because containers" is the only answer I can give to this.
Containers have weird constraints, in that /var/tmp may be
big, but not big enough for the overlay files.

I still think the principle of being able to put the
libguestfs appliance and the other big temporary files
in different places is not a bad one.

(CC-ing Dan, and Igor who is the bug reporter)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

On Thu, Apr 02, 2020 at 05:51:32PM +0200, Pino Toscano
wrote:
> On Thursday, 2 April 2020 17:30:39 CEST Richard W.M. Jones wrote:
> > On Thu, Apr 02, 2020 at 03:21:14PM +0200, Pino Toscano wrote:
> > > On Thursday, 2 April 2020 14:49:18 CEST Richard W.M. Jones wrote:
> > > > Previously we placed large files in g#get_cachedir ()
(usually
> > > > /var/tmp).  However the problem is this ties the libguestfs
appliance
> > > > and the virt-v2v overlay files to the same location.
> > > > 
> > > > When virt-v2v is run in a container, or any other situation
where
> > > > local storage is limited, it's helpful to be able to put
the overlay
> > > > files on an externally mounted PVC, which might be using NFS
and
> > > > shared between containers.  But putting the libguestfs
appliance on
> > > > NFS in a shared location is certainly not recommended.
> > > > 
> > > > This allows the two locations to be set separately:
> > > > 
> > > >   VIRT_V2V_TMPDIR - location of large temporary files, can
use NFS
> > > >                     and may be shared
> > > > 
> > > >   LIBGUESTFS_CACHEDIR - location of libguestfs appliance
> > > > 
> > > > Another motivation for this patch is to allow more reliable
cleanup of
> > > > temporary files by an external process, as described in the
updated
> > > > documentation.
> > > > ---
> > > 
> > > I do not understand the motivation behind this, which adds yet
another
> > > location with temporary files in addition to:
> > > - LIBGUESTFS_TMPDIR - $TMPDIR by default (which itself is /tmp by
> > >   default)
> > > - LIBGUESTFS_CACHEDIR - /var/tmp by default (with a .guestfs-$UID
> > >   subdirectory for the appliance)
> > > 
> > > Before this patch, almost all the temporary files are stored
directly
> > > or in subdirectories of $TMPDIR, except big files such as
overlays and
> > > OVA extracted content that are in CACHEDIR. With the proposed
changes,
> > > _all_ the temporary files will be in CACHEDIR, so there are the
> > > following problems:
> > > - this directory will be cluttered with a lot more files than
before
> > > - if it is shared, then other places where it is mounted will see
the
> > >   same files
> > > - if it is shared, then creating temporary files will possibly
mean
> > >   doing network I/O
> > > - if virt-v2v exits uncleantly, there will be a lot more files to
> > >   cleanup than now
> > > - even without being shared, /var/tmp is persistent unlike /tmp
(which
> > >   can be tmpfs-backed on some distros/setups), meaning old
temporary
> > >   files will linger way more
> > 
> > How about if we confine the change to just large files (ie. ones
> > which are currently placed in cachedir)?
> 
> This is already the case, isn't it?
> 
> > However the way that virt-v2v works at the moment means you cannot put
> > the large files (especially v2vovl*.qcow2) in a different place from
> > the libguestfs appliance.  This means that if you have only
"some"
> > space in /var/tmp -- enough for the appliance, but not enough for the
> > potentially much larger space required by v2vovl*.qcow2 with multiple
> > copies of virt-v2v running -- then you cannot separate the overlays to
> > another directory.  This isn't just a problem for containers.
> 
> /var/tmp is a temporary directory, hence it ought to have enough space
> for big temporary files. This is nothing special for libguestfs or
> virt-v2v.
It is not valid to assume that /var/tmp is arbitrarily big enough
for any files v2v might want to create. AFAIK, there's no real
rule about how large it is expected to be. Linux distros typically
don't do fine grained partitioning by default, so /var/tmp is often
the same filesystem as /, but it is not unusual for admins to have
/ relatively small, and have a separate data partition somewhere
which has the vast majority of storage space available to the host.

In the container world the / is indeed generally of a fairly
limited size, as that's an unpacked container image filesystem
usually on the local host storage.  Apps needing large storage
capacities will usually be given an explicit volume they can
use for this purpose.  NB, the provided volume may be persistent
across container boots, or it may be freshly formatted on every
boot. Whomever configures the container decides this based on
the usage reuqirements of whatever app is to be run in the
container.

> I do not see what makes v2vovl*.qcow2 files so special in this regard,
> requiring them to be handled specially than other big temporary files
> already stored by libguestfs/virt-v2v. Examples:
> - the appliance
> - the temporary directory for qemu when using the libvirt backend
> - the extracted content of an OVA, in case we cannot read it directly
> - the disks when exporting to glance (-o glance)
Out of those things, I think the appliance / QEMU temp dir are in
a different league to the disk images. They're going to be small
enough that I it wouldn't tax the storage capacity of /var/tmp.

Anything related to VM disk images is liable to be an order of
magnitude bigger, easily in the  100's of GB league.

With this POV, I would put  OVA contents, glance exports, and v2v
disk images in the same category, and would expect to be able to
explicitly control  where they are placed, independantly from anything
that goes in /var/tmp.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|