Libguestfs - Nov 2016 - Re: [ovirt-users] OVA import of FC21 VM hangs during virt-v2v conversion?

Hi Rich,

On Wed, November 16, 2016 1:34 pm, Richard W.M. Jones
wrote:
> On Wed, Nov 16, 2016 at 11:31:40AM -0500, Derek Atkins wrote:
>> > Running `virt-v2v -v -x ...' and observing the output will
give you a
>> > good idea of what precisely it was doing for those hours.
>>
>> Well, as I said in my original email, it was hanging at running
>> setfiles:
>>
>> commandrvf: stdout=n stderr=y flags=0x0
>> commandrvf: setfiles -F -e /sysroot/dev -e /sysroot/proc -e
>> /sysroot/selinux -e /sysroot/sys -r /sysroot -q
>> /sysroot/etc/selinux/targeted/contexts/files/file_contexts /sysroot/
>> Can't stat exclude path "/sysroot/selinux", No such file
or directory
>> - ignoring.
>
> OK that's interesting.  setfiles is doing SELinux labelling of the
> filesystem.  The time taken should be approximately proportional to
> the number of files in the filesystem.
>
> What it precisely does is to run the host's /usr/sbin/setfiles command
> over the whole guest filesystem, using the guest's
> /etc/selinux/targeted/contexts/files/file_contexts to control the
> labelling.  So if either of the guest filesystem or the guest's
> file_contexts is particularly large, or setfiles has a bug, then it
> might take a long time.
I suppose there could be a bug.  I just verified that it's absolutely
setfiles:

14:06:20 I: Started virt-v2v -v -x -i ova /ovirt/import/fc21-64.ova -of
raw -o null
14:06:20 O: [   0.0] Opening the source -i ova /ovirt/import/fc21-64.ova
[snip]
14:07:25 E: libguestfs: trace: v2v: aug_get = "targeted"
14:07:25 E: libguestfs: trace: v2v: aug_close
14:07:25 E: guestfsd: main_loop: new request, len 0x28
14:07:25 E: guestfsd: main_loop: proc 26 (aug_close) took 0.03 seconds
14:07:25 E: libguestfs: trace: v2v: aug_close = 0
14:07:25 E: libguestfs: trace: v2v: selinux_relabel
"/etc/selinux/targeted/contexts/files/file_contexts" "/"
"force:true"
14:07:25 E: guestfsd: main_loop: new request, len 0x6c
14:07:25 E: commandrvf: stdout=n stderr=y flags=0x0
14:07:25 E: commandrvf: setfiles -F -e /sysroot/dev -e /sysroot/proc -e
/sysroot/selinux -e /sysroot/sys -r /sysroot -q
/sysroot/etc/selinux/targeted/contexts/files/file_contexts /sysroot/
14:07:25 E: Can't stat exclude path "/sysroot/selinux", No such
file or
directory - ignoring.

[ Spacing Emphasis Mine -- there is no break (except time) here ]

16:00:42 E: libguestfs: trace: v2v: selinux_relabel = 0
16:00:42 O: virt-v2v: This guest has virtio drivers installed.
16:00:42 E: libguestfs: trace: v2v: rm_f "/.autorelabel"
16:00:42 E: guestfsd: main_loop: proc 467 (selinux_relabel) took 6797.01
seconds
16:00:42 E: guestfsd: main_loop: new request, len 0x3c
[SNIP]
16:01:13 I: Finished with exitcode 0

So it looks like it took almost two hours to run the setfiles on a 4GB
filesystem.  The rest of the import was only a minute or three.

I'm happy to help debug this however I can in order to fix any potential
bugs.  However I admit it probably wont affect me any more -- I don't
expect to migrate many more VMs.  Still, I'm happy to work towards the
common good.  :)
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-p2v converts physical machines to virtual machines.  Boot with a
> live CD or over the network (PXE) and turn machines into KVM guests.
> http://libguestfs.org/virt-v2v
-derek

-- 
       Derek Atkins                 617-623-3745
       derek@ihtfp.com             www.ihtfp.com
       Computer and Internet Security Consultant

On Wed, Nov 16, 2016 at 04:49:46PM -0500, Derek Atkins
wrote:
> I suppose there could be a bug.  I just verified that it's absolutely
> setfiles:
> 
> 14:06:20 I: Started virt-v2v -v -x -i ova /ovirt/import/fc21-64.ova -of
> raw -o null
> 14:06:20 O: [   0.0] Opening the source -i ova /ovirt/import/fc21-64.ova
> [snip]
> 14:07:25 E: libguestfs: trace: v2v: aug_get = "targeted"
> 14:07:25 E: libguestfs: trace: v2v: aug_close
> 14:07:25 E: guestfsd: main_loop: new request, len 0x28
> 14:07:25 E: guestfsd: main_loop: proc 26 (aug_close) took 0.03 seconds
> 14:07:25 E: libguestfs: trace: v2v: aug_close = 0
> 14:07:25 E: libguestfs: trace: v2v: selinux_relabel
> "/etc/selinux/targeted/contexts/files/file_contexts"
"/" "force:true"
> 14:07:25 E: guestfsd: main_loop: new request, len 0x6c
> 14:07:25 E: commandrvf: stdout=n stderr=y flags=0x0
> 14:07:25 E: commandrvf: setfiles -F -e /sysroot/dev -e /sysroot/proc -e
> /sysroot/selinux -e /sysroot/sys -r /sysroot -q
> /sysroot/etc/selinux/targeted/contexts/files/file_contexts /sysroot/
> 14:07:25 E: Can't stat exclude path "/sysroot/selinux", No
such file or
> directory - ignoring.
> 
> [ Spacing Emphasis Mine -- there is no break (except time) here ]
> 
> 16:00:42 E: libguestfs: trace: v2v: selinux_relabel = 0
Yes, this is conclusive.

The guest is Fedora 21?  Does it have an unusually large number of
files in the filesystem?  Is the host RHEL or Fedora and which
version?

I will see if I can reproduce it.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

Hi,

On Wed, November 16, 2016 5:03 pm, Richard W.M. Jones
wrote:
> On Wed, Nov 16, 2016 at 04:49:46PM -0500, Derek Atkins wrote:
>> I suppose there could be a bug.  I just verified that it's
absolutely
>> setfiles:
>>
>> 14:07:25 E: commandrvf: setfiles -F -e /sysroot/dev -e /sysroot/proc -e
>> /sysroot/selinux -e /sysroot/sys -r /sysroot -q
>> /sysroot/etc/selinux/targeted/contexts/files/file_contexts /sysroot/
>> 14:07:25 E: Can't stat exclude path "/sysroot/selinux",
No such file or
>> directory - ignoring.
>>
>> [ Spacing Emphasis Mine -- there is no break (except time) here ]
>>
>> 16:00:42 E: libguestfs: trace: v2v: selinux_relabel = 0
>
> Yes, this is conclusive.
>
> The guest is Fedora 21?  Does it have an unusually large number of
> files in the filesystem?  Is the host RHEL or Fedora and which
> version?
Guest is Fedora 21.  Host is CentOS 7.2

Guest not have an unusual number of files.  At least looking at the number
of inodes used:
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/vda3      2894304 164989 2729315    6% /

So... 165,000 files?  My F23 system which imported in 2 minutes has
174,442 inodes used, albeit significantly more free:

Filesystem       Inodes  IUsed    IFree IUse% Mounted on
/dev/vda3      23080448 174442 22906006    1% /

Another difference is ext4 (F21) vs xfs (F23).
> I will see if I can reproduce it.
Thanks.  Worst case I can probably provide you the OVA.

-derek
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-df lists disk usage of guests without needing to install any
> software inside the virtual machine.  Supports Linux and Windows.
> http://people.redhat.com/~rjones/virt-df/
>

-- 
       Derek Atkins                 617-623-3745
       derek@ihtfp.com             www.ihtfp.com
       Computer and Internet Security Consultant