Libguestfs - Nov 2016 - Re: [ovirt-users] OVA import of FC21 VM hangs during virt-v2v conversion?

"Richard W.M. Jones" <rjones@redhat.com> writes:
> On Wed, Nov 16, 2016 at 09:06:29AM +0200, Michal Skrivanek wrote:
>> 
>> > On 7 Nov 2016, at 22:16, Derek Atkins <derek@ihtfp.com>
wrote:
>> > 
>> > Hi,
>> > 
>> > My last VM imported in 2 minutes.  This one has been sitting for
three
>> > hours.  I think this is a bug.
>> 
>> well, some time it does take a long time. Are you sure it was hung?
>> No I/O going on?
>> 
>> adding libguestfs list 
>
> The most likely reason for virt-v2v to hang (actually it's qemu which
> is hanging) is that you've run out of disk space.  Most probably in
> /var/tmp but best to check all your host filesystems.
No, the filesystem had plenty of space.

It eventually did complete, after several hours.  But I don't understand
why this particular VM took so long when others (with similar disk
sizes/usages/configurations) completed in only 1-2 minutes.  Each VM is
effectively exactly the same (it's a build slave), so it's just the
basic installation of the particular Fedora release, the build tools,
and the build slave configuration.  Beyond the underlying fedora release
version, the systems are configured exactly the same.

I migrated four similar VMs.  Three of them completed almost
immediately.  The last too much longer.  The ONLY thing different about
the one that took longer was that I decided to pack the disk first, so I
ran:

  vmware-toolbox-cmd disk shrink /

in the VM prior to packaging it up as an OVA.  All the OVA files are
relatively small and similar in size:

-rw------- 1 warlord warlord  1959553024 Nov  7 10:40 fc20-64.ova
-rw------- 1 warlord warlord  1794907648 Nov  7 12:27 fc21-64.ova
-rw------- 1 warlord warlord  2023595520 Nov  6 10:04 fc22-64.ova
-rw------- 1 warlord warlord  3381622272 Nov  5 16:18 fc23-64.ova

The FC23 and FC22 migrations took the least amount of time, even though
they were the largest.
> Rich.
-derek

-- 
       Derek Atkins                 617-623-3745
       derek@ihtfp.com             www.ihtfp.com
       Computer and Internet Security Consultant

On Wed, Nov 16, 2016 at 10:56:39AM -0500, Derek Atkins
wrote:
> "Richard W.M. Jones" <rjones@redhat.com> writes:
> 
> > On Wed, Nov 16, 2016 at 09:06:29AM +0200, Michal Skrivanek wrote:
> >> 
> >> > On 7 Nov 2016, at 22:16, Derek Atkins <derek@ihtfp.com>
wrote:
> >> > 
> >> > Hi,
> >> > 
> >> > My last VM imported in 2 minutes.  This one has been sitting
for three
> >> > hours.  I think this is a bug.
> >> 
> >> well, some time it does take a long time. Are you sure it was
hung?
> >> No I/O going on?
> >> 
> >> adding libguestfs list 
> >
> > The most likely reason for virt-v2v to hang (actually it's qemu
which
> > is hanging) is that you've run out of disk space.  Most probably
in
> > /var/tmp but best to check all your host filesystems.
> 
> No, the filesystem had plenty of space.
> 
> It eventually did complete, after several hours.  But I don't
understand
> why this particular VM took so long when others (with similar disk
> sizes/usages/configurations) completed in only 1-2 minutes.  Each VM is
> effectively exactly the same (it's a build slave), so it's just the
> basic installation of the particular Fedora release, the build tools,
> and the build slave configuration.  Beyond the underlying fedora release
> version, the systems are configured exactly the same.
> 
> I migrated four similar VMs.  Three of them completed almost
> immediately.  The last too much longer.  The ONLY thing different about
> the one that took longer was that I decided to pack the disk first, so I
> ran:
> 
>   vmware-toolbox-cmd disk shrink /
I'm not familiar with this command.  In general terms, conversions
take a time proportional to the amount of disk space in use by the
guest.

Running `virt-v2v -v -x ...' and observing the output will give you a
good idea of what precisely it was doing for those hours.

Rich.
> in the VM prior to packaging it up as an OVA.  All the OVA files are
> relatively small and similar in size:
> 
> -rw------- 1 warlord warlord  1959553024 Nov  7 10:40 fc20-64.ova
> -rw------- 1 warlord warlord  1794907648 Nov  7 12:27 fc21-64.ova
> -rw------- 1 warlord warlord  2023595520 Nov  6 10:04 fc22-64.ova
> -rw------- 1 warlord warlord  3381622272 Nov  5 16:18 fc23-64.ova
> 
> The FC23 and FC22 migrations took the least amount of time, even though
> they were the largest.
> 
> > Rich.
> 
> -derek
> 
> -- 
>        Derek Atkins                 617-623-3745
>        derek@ihtfp.com             www.ihtfp.com
>        Computer and Internet Security Consultant
-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

Hi,

On Wed, November 16, 2016 11:10 am, Richard W.M. Jones
wrote:
>
>> It eventually did complete, after several hours.  But I don't
understand
>> why this particular VM took so long when others (with similar disk
>> sizes/usages/configurations) completed in only 1-2 minutes.  Each VM is
>> effectively exactly the same (it's a build slave), so it's just
the
>> basic installation of the particular Fedora release, the build tools,
>> and the build slave configuration.  Beyond the underlying fedora
release
>> version, the systems are configured exactly the same.
>>
>> I migrated four similar VMs.  Three of them completed almost
>> immediately.  The last too much longer.  The ONLY thing different about
>> the one that took longer was that I decided to pack the disk first, so
I
>> ran:
>>
>>   vmware-toolbox-cmd disk shrink /
>
> I'm not familiar with this command.  In general terms, conversions
> take a time proportional to the amount of disk space in use by the
> guest.
This command is a disk defragment + shrink inside VMware.  Basically, the
.vmdk was about 36G even though the system was only using about 3G. 
Running this command released all that free space and then reduced the
VMDK file.

I figured the time should be proportional to disk usage, which is why I
was so surprised that it took orders of magnitude longer than other
systems with similar disk usage (and indeed, in some cases the other
systems that completed in minutes had MORE disk usage).
> Running `virt-v2v -v -x ...' and observing the output will give you a
> good idea of what precisely it was doing for those hours.
Well, as I said in my original email, it was hanging at running setfiles:

commandrvf: stdout=n stderr=y flags=0x0
commandrvf: setfiles -F -e /sysroot/dev -e /sysroot/proc -e
/sysroot/selinux -e /sysroot/sys -r /sysroot -q
/sysroot/etc/selinux/targeted/contexts/files/file_contexts /sysroot/
Can't stat exclude path "/sysroot/selinux", No such file or
directory
- ignoring.

There's no timestamp in the output or logs so it's hard to tell exactly
what happens when or how long each step takes.  For example, I wouldn't
know how long the setfiles step takes unless I sit there for hours
watching to see when it completes.  Is there some way to add timestamps to
each output line?

FWIW, here's the disk usage from within that VM:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3        44G  4.6G   37G  12% /
devtmpfs        487M     0  487M   0% /dev
tmpfs           497M     0  497M   0% /dev/shm
tmpfs           497M  348K  496M   1% /run
tmpfs           497M     0  497M   0% /sys/fs/cgroup
tmpfs           497M  4.0K  496M   1% /tmp
/dev/vda1       969M  130M  773M  15% /boot
tmpfs           100M     0  100M   0% /run/user/1000
tmpfs           100M     0  100M   0% /run/user/0

For the record, Fedora 23, which imported in 2 minutes, has the following
disk usage:

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        486M     0  486M   0% /dev
tmpfs           496M     0  496M   0% /dev/shm
tmpfs           496M  536K  496M   1% /run
tmpfs           496M     0  496M   0% /sys/fs/cgroup
/dev/vda3        45G  7.9G   37G  18% /
tmpfs           496M  4.0K  496M   1% /tmp
/dev/vda1       969M  136M  767M  15% /boot
tmpfs           100M     0  100M   0% /run/user/1000
tmpfs           100M     0  100M   0% /run/user/0

> Rich.
-derek
>
>> in the VM prior to packaging it up as an OVA.  All the OVA files are
>> relatively small and similar in size:
>>
>> -rw------- 1 warlord warlord  1959553024 Nov  7 10:40 fc20-64.ova
>> -rw------- 1 warlord warlord  1794907648 Nov  7 12:27 fc21-64.ova
>> -rw------- 1 warlord warlord  2023595520 Nov  6 10:04 fc22-64.ova
>> -rw------- 1 warlord warlord  3381622272 Nov  5 16:18 fc23-64.ova
>>
>> The FC23 and FC22 migrations took the least amount of time, even though
>> they were the largest.
>>
>> > Rich.
>>
>> -derek
>>
>> --
>>        Derek Atkins                 617-623-3745
>>        derek@ihtfp.com             www.ihtfp.com
>>        Computer and Internet Security Consultant
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-builder quickly builds VMs from scratch
> http://libguestfs.org/virt-builder.1.html
>

-- 
       Derek Atkins                 617-623-3745
       derek@ihtfp.com             www.ihtfp.com
       Computer and Internet Security Consultant