thr3ads.net - Libguestfs - Re: [Libguestfs] Double fault panic in L2 upon v2v conversion [Jan 2014]

If this information is useful, please help other people find it:
Share via:

Rom Freiman

2014-Jan-15 14:35 UTC

[Libguestfs] Double fault panic in L2 upon v2v conversion

Hi everybody,

Wanted to hear your opinion and to receive a smart advice.

I'm trying to use virt-v2v in order to convert ova image (exported from
vcenter) to run on libvirt/kvm - all this inside a VM of fedora.
The converted image is also a fedora.
During the conversion process, in some point of libguestfs activity, I get
double fault panic from L2 (printed as part of libguest output) and the
conversion process fails - no errors appear neither in L0 not in L1 message
logs.

This is more detailed explanation:

Description of problem:
During v2v conversion for ova fedora image in a VM there is double
fault panic happens and L2 crashes during libguestfs conversion.

The crash happens in different stages, but usually upon memory pressure in L0.

There are no error logs in L1 and cannot find strong correlation to
patches that were added to L0-KVM to avoid L0 crash upon nested vm
with high memory pressure -
http://git.kernel.org/cgit/virt/kvm/kvm.git/patch/arch/x86/kvm/mmu.c?id=989c6b34f6a9480e397b170cc62237e89bf4fdb9.

Command within L1 to perform the conversion (fedora.ova - vmdk image
of fedora - was placed in advance on the VM):

LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 /usr/bin/virt-v2v -i ova -os
default -oc qemu:///system -of qcow2 -n default
/var/tmp/fedora-v2v.ova


Version-Release number of selected component (if applicable):
L0:
Kernel: 3.11.8-200.fc19 + nested crash patches
libvritd: 1.0.5.8
qemu: 1.6.1
libguestfs-test-tool 1.22.7fedora=19,release=1.fc19,libvirt
L1:
Kernel: 3.11.8-200..fc19.x86_64
libvirtd: 1.0.5.8
qemu: 1.6.1 + v2v patch (skip vmdk version verification)
libguestfs-test-tool 1.22.7fedora=19,release=2.fc19,libvirt
virt-v2v 0.9.0

L2:
Kernel: 3.11.10-301.fc20

How reproducible:
LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 /usr/bin/virt-v2v -i ova -os
default -oc qemu:///system -of qcow2 -n default
/var/tmp/fedora-v2v.ova

Steps to Reproduce:
1. Upload ova image to VM
2. Run v2v to perform the conversion
3. Add some memory pressure on L0 (dd if=/dev/urandom of=/tmp/bigfile count=6M)

Actual results:

libguestfs: checksum of existing appliance:
63a0a4c8acb7f789c7a725bebbb16e64a3bc0999fdca61931babda50292d83b8
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: [00337ms] begin testing qemu features
libguestfs: command: run: /usr/bin/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -help
libguestfs: command: run: /usr/bin/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -version
libguestfs: qemu version 1.6
libguestfs: command: run: /usr/bin/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -machine accel=kvm:tcg
libguestfs: command: run: \ -device ?
libguestfs: [00661ms] finished testing qemu features
[00663ms] /usr/bin/qemu-kvm \
    -global virtio-blk-pci.scsi=off \
    -nodefconfig \
    -nodefaults \
    -nographic \
    -machine accel=kvm:tcg \
    -m 500 \
    -no-reboot \
    -no-hpet \
    -kernel /var/tmp/.guestfs-0/kernel.1416 \
    -initrd /var/tmp/.guestfs-0/initrd.1416 \
    -device virtio-scsi-pci,id=scsi \
    -drive
file=/var/lib/libvirt/images/fedora-v2v-disk1.vmdk,cache=none,format=qcow2,id=hd0,if=ide
\
    -drive file=/tmp/3pEC1jAXQs,snapshot=on,format=raw,id=hd1,if=ide \
    -drive
file=/var/tmp/.guestfs-0/root.1416,snapshot=on,id=appliance,if=none,cache=unsafe
\
    -device scsi-hd,drive=appliance \
    -device virtio-serial \
    -serial stdio \
    -device sga \
    -chardev socket,path=/tmp/libguestfsKr0x6j/guestfsd.sock,id=channel0 \
    -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
    -netdev user,id=usernet,net=169.254.0.0/16 \
    -device virtio-net-pci,netdev=usernet \
    -append 'panic=1 console=ttyS0 udevtimeout=600 no_timer_check
acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdc selinux=0
guestfs_verbose=1 TERM=dumb'
\x1b[1;256r\x1b[256;256H\x1b[6n
Google, Inc.
Serial Graphics Adapter 02/18/13
SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (mockbuild@)
Mon Feb 18 08:29:02 UTC 2013
Term: 80x24
4 0
SeaBIOS (version ?-20131118_112708-)


iPXE (http://ipxe.org) 00:04.0 C100 PCI2.10 PnP PMM+1F3C7C60+1F327C60 C100



Booting from ROM...
Probing EDD (edd=off to disable)... ok
\x1b[2J[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version
3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64
(mockbuild@ellaria) (gcc version 4.8.2 20131017 (Red Hat 4.8.2-1)
(GCC) ) #1 SMP Mon Nov 18 17:54:34 IST 2013
[    0.000000] Command line: panic=1 console=ttyS0 udevtimeout=600
no_timer_check acpi=off printk.time=1 cgroup_disable=memory
root=/dev/sdc selinux=0 guestfs_verbose=1 TERM=dumb
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable

............................................
.........More logs here............
.............................................

guestfsd: main_loop: new request, len 0x3c
mount -o ro /dev/sdb /sysroot/
[   12.645305] PANIC: double fault, error_code: 0x0
[   12.645305] CPU: 0 PID: 141 Comm: mount Not tainted
3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64 #1
[   12.645305] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   12.645305] task: ffff88001cc816e0 ti: ffff88001cde6000 task.ti:
ffff88001cde6000
[   12.645305] RIP: 0033:[<00007fa602c5b99b>]  [<00007fa602c5b99b>]
0x7fa602c5b99a
[   12.645305] RSP: 002b:00007fff4f5884a0  EFLAGS: 00010216
[   12.645305] RAX: 00007fa602008ff8 RBX: 00007fa601ff0000 RCX: 00007fa601ff0000
[   12.645305] RDX: 00000000003b7068 RSI: 00007fff4f588560 RDI: 00007fa601ff3d18
[   12.645305] RBP: 00007fff4f5885d0 R08: 00007fa60200f310 R09: 0000000000000000
[   12.645305] R10: 0000000000000022 R11: 00007fa60200f310 R12: 00007fa60200e9b0
[   12.645305] R13: 0000000000000000 R14: 0000000000000000 R15: 00007fa602e6e990
[   12.645305] FS:  00007fa602e69880(0000) GS:ffff88001f000000(0000)
knlGS:0000000000000000
[   12.645305] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[   12.645305] CR2: 0000000000000000 CR3: 000000001d7fb000 CR4: 00000000000006f0
[   12.645305]
[   12.645305] Kernel panic - not syncing: Machine halted.
[   12.645305] CPU: 0 PID: 141 Comm: mount Not tainted
3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64 #1
[   12.645305] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   12.645305]  ffff88001f005f58 ffff88001f005e90 ffffffff8164024b
ffffffff819e89dc
[   12.645305]  ffff88001f005f08 ffffffff8163c272 0000000000000008
ffff88001f005f18
[   12.645305]  ffff88001f005eb8 ffffffff8163c8e5 0000000000000046
00000000000000b1
[   12.645305] Call Trace:
[   12.645305]  <#DF>  [<ffffffff8164024b>] dump_stack+0x45/0x56
[   12.645305]  [<ffffffff8163c272>] panic+0xc8/0x1d7
[   12.645305]  [<ffffffff8163c8e5>] ? printk+0x67/0x69
[   12.645305]  [<ffffffff81048ae1>] df_debug+0x31/0x40
[   12.645305]  [<ffffffff810132ed>] do_double_fault+0x5d/0x80
[   12.645305]  [<ffffffff81650b88>] double_fault+0x28/0x30
[   12.645305]  <<EOE>>
[   12.645305] Rebooting in 1 seconds..libguestfs: child_cleanup:
0x3a05f50: child process died
libguestfs: sending SIGTERM to process 1526
libguestfs: trace: mount_ro = -1 (error)
libguestfs: trace: vfs_type "/dev/sda1"
libguestfs: trace: vfs_type = NULL (error)
libguestfs: check_for_filesystem_on: /dev/sda1 (failed to get vfs type)
libguestfs: trace: internal_parse_mountable "/dev/sda1"
libguestfs: trace: internal_parse_mountable = NULL (error)
libguestfs: trace: inspect_os = NULL (error)
libguestfs: trace: close


Additional info:
The same crash also happens when L0 is 3.11.9 (with kvm patch to avoid
L0 crash in nested environment -
http://git.kernel.org/cgit/virt/kvm/kvm.git/patch/arch/x86/kvm/mmu.c?id=989c6b34f6a9480e397b170cc62237e89bf4fdb9)
but more rarely.

Thanks a lot,
Rom

Tingting Zheng

2014-Jan-16 05:43 UTC

head link

Re: [Libguestfs] Double fault panic in L2 upon v2v conversion

There is a bug which is still not fixed:Bug 956580 -[ RFE ]Support for VMware
& Citrix OVF image import into RHEV
https://bugzilla.redhat.com/show_bug.cgi?id=956580

Best regards,
Tingting Zheng（郑婷婷）

----- Original Message -----
From: "Rom Freiman" <rom@stratoscale.com>
To: libguestfs@redhat.com
Sent: Wednesday, January 15, 2014 10:35:29 PM
Subject: [Libguestfs] Double fault panic in L2 upon v2v conversion

Hi everybody, 

Wanted to hear your opinion and to receive a smart advice. 

I'm trying to use virt-v2v in order to convert ova image (exported from
vcenter) to run on libvirt/kvm - all this inside a VM of fedora.
The converted image is also a fedora. 
During the conversion process, in some point of libguestfs activity, I get
double fault panic from L2 (printed as part of libguest output) and the
conversion process fails - no errors appear neither in L0 not in L1 message
logs.

This is more detailed explanation: 

Description of problem:
During v2v conversion for ova fedora image in a VM there is double fault panic
happens and L2 crashes during libguestfs conversion.

The crash happens in different stages, but usually upon memory pressure in L0.

There are no error logs in L1 and cannot find strong correlation to patches that
were added to L0-KVM to avoid L0 crash upon nested vm with high memory pressure
-
http://git.kernel.org/cgit/virt/kvm/kvm.git/patch/arch/x86/kvm/mmu.c?id=989c6b34f6a9480e397b170cc62237e89bf4fdb9
.

Command within L1 to perform the conversion (fedora.ova - vmdk image of fedora -
was placed in advance on the VM):

LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 /usr/bin/virt-v2v -i ova -os default -oc
qemu:///system -of qcow2 -n default /var/tmp/fedora-v2v.ova


Version-Release number of selected component (if applicable):
L0: 
Kernel: 3.11.8-200.fc19 + nested crash patches
libvritd: 1.0.5.8
qemu: 1.6.1
libguestfs-test-tool 1.22.7fedora=19,release=1.fc19,libvirt
L1:
Kernel: 3.11.8-200..fc19.x86_64
libvirtd: 1.0.5.8
qemu: 1.6.1 + v2v patch (skip vmdk version verification)
libguestfs-test-tool 1.22.7fedora=19,release=2.fc19,libvirt
virt-v2v 0.9.0

L2:
Kernel: 3.11.10-301.fc20

How reproducible:
LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 /usr/bin/virt-v2v -i ova -os default -oc
qemu:///system -of qcow2 -n default /var/tmp/fedora-v2v.ova

Steps to Reproduce:
1. Upload ova image to VM
2. Run v2v to perform the conversion
3. Add some memory pressure on L0 (dd if=/dev/urandom of=/tmp/bigfile count=6M)

Actual results: 
libguestfs: checksum of existing appliance:
63a0a4c8acb7f789c7a725bebbb16e64a3bc0999fdca61931babda50292d83b8
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: [00337ms] begin testing qemu features
libguestfs: command: run: /usr/bin/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -help
libguestfs: command: run: /usr/bin/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -version
libguestfs: qemu version 1.6
libguestfs: command: run: /usr/bin/qemu-kvm
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -machine accel=kvm:tcg
libguestfs: command: run: \ -device ?
libguestfs: [00661ms] finished testing qemu features
[00663ms] /usr/bin/qemu-kvm \
    -global virtio-blk-pci.scsi=off \
    -nodefconfig \
    -nodefaults \
    -nographic \
    -machine accel=kvm:tcg \
    -m 500 \
    -no-reboot \
    -no-hpet \
    -kernel /var/tmp/.guestfs-0/kernel.1416 \
    -initrd /var/tmp/.guestfs-0/initrd.1416 \
    -device virtio-scsi-pci,id=scsi \
    -drive
file=/var/lib/libvirt/images/fedora-v2v-disk1.vmdk,cache=none,format=qcow2,id=hd0,if=ide
\
    -drive file=/tmp/3pEC1jAXQs,snapshot=on,format=raw,id=hd1,if=ide \
    -drive
file=/var/tmp/.guestfs-0/root.1416,snapshot=on,id=appliance,if=none,cache=unsafe
\
    -device scsi-hd,drive=appliance \
    -device virtio-serial \
    -serial stdio \
    -device sga \
    -chardev socket,path=/tmp/libguestfsKr0x6j/guestfsd.sock,id=channel0 \
    -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
    -netdev user,id=usernet,net= 169.254.0.0/16 \
    -device virtio-net-pci,netdev=usernet \
    -append 'panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off
printk.time=1 cgroup_disable=memory root=/dev/sdc selinux=0 guestfs_verbose=1
TERM=dumb'
\x1b[1;256r\x1b[256;256H\x1b[6n
Google, Inc.
Serial Graphics Adapter 02/18/13
SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (mockbuild@) Mon Feb 18
08:29:02 UTC 2013
Term: 80x24
4 0
SeaBIOS (version ?-20131118_112708-)


iPXE ( http://ipxe.org ) 00:04.0 C100 PCI2.10 PnP PMM+1F3C7C60+1F327C60 C100
                                                                               


Booting from ROM...
Probing EDD (edd=off to disable)... ok
\x1b[2J[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version
3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64 (mockbuild@ellaria) (gcc
version 4.8.2 20131017 (Red Hat 4.8.2-1) (GCC) ) #1 SMP Mon Nov 18 17:54:34 IST
2013
[    0.000000] Command line: panic=1 console=ttyS0 udevtimeout=600
no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdc
selinux=0 guestfs_verbose=1 TERM=dumb
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable 
............................................ 
.........More logs here............ 
............................................. 
guestfsd: main_loop: new request, len 0x3c
mount -o ro /dev/sdb /sysroot/
[   12.645305] PANIC: double fault, error_code: 0x0
[   12.645305] CPU: 0 PID: 141 Comm: mount Not tainted
3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64 #1
[   12.645305] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   12.645305] task: ffff88001cc816e0 ti: ffff88001cde6000 task.ti:
ffff88001cde6000
[   12.645305] RIP: 0033:[<00007fa602c5b99b>]  [<00007fa602c5b99b>]
0x7fa602c5b99a
[   12.645305] RSP: 002b:00007fff4f5884a0  EFLAGS: 00010216
[   12.645305] RAX: 00007fa602008ff8 RBX: 00007fa601ff0000 RCX: 00007fa601ff0000
[   12.645305] RDX: 00000000003b7068 RSI: 00007fff4f588560 RDI: 00007fa601ff3d18
[   12.645305] RBP: 00007fff4f5885d0 R08: 00007fa60200f310 R09: 0000000000000000
[   12.645305] R10: 0000000000000022 R11: 00007fa60200f310 R12: 00007fa60200e9b0
[   12.645305] R13: 0000000000000000 R14: 0000000000000000 R15: 00007fa602e6e990
[   12.645305] FS:  00007fa602e69880(0000) GS:ffff88001f000000(0000)
knlGS:0000000000000000
[   12.645305] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[   12.645305] CR2: 0000000000000000 CR3: 000000001d7fb000 CR4: 00000000000006f0
[   12.645305] 
[   12.645305] Kernel panic - not syncing: Machine halted.
[   12.645305] CPU: 0 PID: 141 Comm: mount Not tainted
3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64 #1
[   12.645305] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   12.645305]  ffff88001f005f58 ffff88001f005e90 ffffffff8164024b
ffffffff819e89dc
[   12.645305]  ffff88001f005f08 ffffffff8163c272 0000000000000008
ffff88001f005f18
[   12.645305]  ffff88001f005eb8 ffffffff8163c8e5 0000000000000046
00000000000000b1
[   12.645305] Call Trace:
[   12.645305]  <#DF>  [<ffffffff8164024b>] dump_stack+0x45/0x56
[   12.645305]  [<ffffffff8163c272>] panic+0xc8/0x1d7
[   12.645305]  [<ffffffff8163c8e5>] ? printk+0x67/0x69
[   12.645305]  [<ffffffff81048ae1>] df_debug+0x31/0x40
[   12.645305]  [<ffffffff810132ed>] do_double_fault+0x5d/0x80
[   12.645305]  [<ffffffff81650b88>] double_fault+0x28/0x30
[   12.645305]  <<EOE>> 
[   12.645305] Rebooting in 1 seconds..libguestfs: child_cleanup: 0x3a05f50:
child process died
libguestfs: sending SIGTERM to process 1526
libguestfs: trace: mount_ro = -1 (error)
libguestfs: trace: vfs_type "/dev/sda1"
libguestfs: trace: vfs_type = NULL (error)
libguestfs: check_for_filesystem_on: /dev/sda1 (failed to get vfs type)
libguestfs: trace: internal_parse_mountable "/dev/sda1"
libguestfs: trace: internal_parse_mountable = NULL (error)
libguestfs: trace: inspect_os = NULL (error)
libguestfs: trace: close


Additional info:
The same crash also happens when L0 is 3.11.9 (with kvm patch to avoid L0 crash
in nested environment -
http://git.kernel.org/cgit/virt/kvm/kvm.git/patch/arch/x86/kvm/mmu.c?id=989c6b34f6a9480e397b170cc62237e89bf4fdb9
) but more rarely.
Thanks a lot, 
Rom 


_______________________________________________
Libguestfs mailing list
Libguestfs@redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs

Rom Freiman

2014-Jan-16 10:30 UTC

head link

Re: [Libguestfs] Double fault panic in L2 upon v2v conversion

Hey,
Seems like the problem is not relevant to that bug.
The conversion works fine when I run it directly on my computer (L0) - same
OVA converted into a fully functional fedora over qemu/KVM.
The problem is the *nested environment* -   I try to do the same process
within a VM and it fails as I described.

Thanks,
Rom


On Thu, Jan 16, 2014 at 7:43 AM, Tingting Zheng <tzheng@redhat.com> wrote:
> There is a bug which is still not fixed:Bug 956580 -[ RFE ]Support for
> VMware & Citrix OVF image import into RHEV
> https://bugzilla.redhat.com/show_bug.cgi?id=956580
>
> Best regards,
> Tingting Zheng（郑婷婷）
>
> ----- Original Message -----
> From: "Rom Freiman" <rom@stratoscale.com>
> To: libguestfs@redhat.com
> Sent: Wednesday, January 15, 2014 10:35:29 PM
> Subject: [Libguestfs] Double fault panic in L2 upon v2v conversion
>
> Hi everybody,
>
> Wanted to hear your opinion and to receive a smart advice.
>
> I'm trying to use virt-v2v in order to convert ova image (exported from
> vcenter) to run on libvirt/kvm - all this inside a VM of fedora.
> The converted image is also a fedora.
> During the conversion process, in some point of libguestfs activity, I get
> double fault panic from L2 (printed as part of libguest output) and the
> conversion process fails - no errors appear neither in L0 not in L1 message
> logs.
>
> This is more detailed explanation:
>
> Description of problem:
> During v2v conversion for ova fedora image in a VM there is double fault
> panic happens and L2 crashes during libguestfs conversion.
>
> The crash happens in different stages, but usually upon memory pressure in
> L0.
>
> There are no error logs in L1 and cannot find strong correlation to
> patches that were added to L0-KVM to avoid L0 crash upon nested vm with
> high memory pressure -
>
http://git.kernel.org/cgit/virt/kvm/kvm.git/patch/arch/x86/kvm/mmu.c?id=989c6b34f6a9480e397b170cc62237e89bf4fdb9.
>
> Command within L1 to perform the conversion (fedora.ova - vmdk image of
> fedora - was placed in advance on the VM):
>
> LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 /usr/bin/virt-v2v -i ova -os default
> -oc qemu:///system -of qcow2 -n default /var/tmp/fedora-v2v.ova
>
>
> Version-Release number of selected component (if applicable):
> L0:
> Kernel: 3.11.8-200.fc19 + nested crash patches
> libvritd: 1.0.5.8
> qemu: 1.6.1
> libguestfs-test-tool 1.22.7fedora=19,release=1.fc19,libvirt
> L1:
> Kernel: 3.11.8-200..fc19.x86_64
> libvirtd: 1.0.5.8
> qemu: 1.6.1 + v2v patch (skip vmdk version verification)
> libguestfs-test-tool 1.22.7fedora=19,release=2.fc19,libvirt
> virt-v2v 0.9.0
>
> L2:
> Kernel: 3.11.10-301.fc20
>
> How reproducible:
> LIBGUESTFS_TRACE=1 LIBGUESTFS_DEBUG=1 /usr/bin/virt-v2v -i ova -os default
> -oc qemu:///system -of qcow2 -n default /var/tmp/fedora-v2v.ova
>
> Steps to Reproduce:
> 1. Upload ova image to VM
> 2. Run v2v to perform the conversion
> 3. Add some memory pressure on L0 (dd if=/dev/urandom of=/tmp/bigfile
> count=6M)
>
> Actual results:
> libguestfs: checksum of existing appliance:
> 63a0a4c8acb7f789c7a725bebbb16e64a3bc0999fdca61931babda50292d83b8
> libguestfs: trace: get_cachedir
> libguestfs: trace: get_cachedir = "/var/tmp"
> libguestfs: [00337ms] begin testing qemu features
> libguestfs: command: run: /usr/bin/qemu-kvm
> libguestfs: command: run: \ -nographic
> libguestfs: command: run: \ -help
> libguestfs: command: run: /usr/bin/qemu-kvm
> libguestfs: command: run: \ -nographic
> libguestfs: command: run: \ -version
> libguestfs: qemu version 1.6
> libguestfs: command: run: /usr/bin/qemu-kvm
> libguestfs: command: run: \ -nographic
> libguestfs: command: run: \ -machine accel=kvm:tcg
> libguestfs: command: run: \ -device ?
> libguestfs: [00661ms] finished testing qemu features
> [00663ms] /usr/bin/qemu-kvm \
>     -global virtio-blk-pci.scsi=off \
>     -nodefconfig \
>     -nodefaults \
>     -nographic \
>     -machine accel=kvm:tcg \
>     -m 500 \
>     -no-reboot \
>     -no-hpet \
>     -kernel /var/tmp/.guestfs-0/kernel.1416 \
>     -initrd /var/tmp/.guestfs-0/initrd.1416 \
>     -device virtio-scsi-pci,id=scsi \
>     -drive
>
file=/var/lib/libvirt/images/fedora-v2v-disk1.vmdk,cache=none,format=qcow2,id=hd0,if=ide
> \
>     -drive file=/tmp/3pEC1jAXQs,snapshot=on,format=raw,id=hd1,if=ide \
>     -drive
>
file=/var/tmp/.guestfs-0/root.1416,snapshot=on,id=appliance,if=none,cache=unsafe
> \
>     -device scsi-hd,drive=appliance \
>     -device virtio-serial \
>     -serial stdio \
>     -device sga \
>     -chardev socket,path=/tmp/libguestfsKr0x6j/guestfsd.sock,id=channel0 \
>     -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
>     -netdev user,id=usernet,net= 169.254.0.0/16 \
>     -device virtio-net-pci,netdev=usernet \
>     -append 'panic=1 console=ttyS0 udevtimeout=600 no_timer_check
acpi=off
> printk.time=1 cgroup_disable=memory root=/dev/sdc selinux=0
> guestfs_verbose=1 TERM=dumb'
> \x1b[1;256r\x1b[256;256H\x1b[6n
> Google, Inc.
> Serial Graphics Adapter 02/18/13
> SGABIOS $Id: sgabios.S 8 2010-04-22 00:03:40Z nlaredo $ (mockbuild@) Mon
> Feb 18 08:29:02 UTC 2013
> Term: 80x24
> 4 0
> SeaBIOS (version ?-20131118_112708-)
>
>
> iPXE ( http://ipxe.org ) 00:04.0 C100 PCI2.10 PnP PMM+1F3C7C60+1F327C60
> C100
>
>
>
> Booting from ROM...
> Probing EDD (edd=off to disable)... ok
> \x1b[2J[    0.000000] Initializing cgroup subsys cpuset
> [    0.000000] Initializing cgroup subsys cpu
> [    0.000000] Initializing cgroup subsys cpuacct
> [    0.000000] Linux version
> 3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64 (mockbuild@ellaria)
> (gcc version 4.8.2 20131017 (Red Hat 4.8.2-1) (GCC) ) #1 SMP Mon Nov 18
> 17:54:34 IST 2013
> [    0.000000] Command line: panic=1 console=ttyS0 udevtimeout=600
> no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdc
> selinux=0 guestfs_verbose=1 TERM=dumb
> [    0.000000] e820: BIOS-provided physical RAM map:
> [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff]
> usable
> ............................................
> .........More logs here............
> .............................................
> guestfsd: main_loop: new request, len 0x3c
> mount -o ro /dev/sdb /sysroot/
> [   12.645305] PANIC: double fault, error_code: 0x0
> [   12.645305] CPU: 0 PID: 141 Comm: mount Not tainted
> 3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64 #1
> [   12.645305] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
> [   12.645305] task: ffff88001cc816e0 ti: ffff88001cde6000 task.ti:
> ffff88001cde6000
> [   12.645305] RIP: 0033:[<00007fa602c5b99b>] 
[<00007fa602c5b99b>]
> 0x7fa602c5b99a
> [   12.645305] RSP: 002b:00007fff4f5884a0  EFLAGS: 00010216
> [   12.645305] RAX: 00007fa602008ff8 RBX: 00007fa601ff0000 RCX:
> 00007fa601ff0000
> [   12.645305] RDX: 00000000003b7068 RSI: 00007fff4f588560 RDI:
> 00007fa601ff3d18
> [   12.645305] RBP: 00007fff4f5885d0 R08: 00007fa60200f310 R09:
> 0000000000000000
> [   12.645305] R10: 0000000000000022 R11: 00007fa60200f310 R12:
> 00007fa60200e9b0
> [   12.645305] R13: 0000000000000000 R14: 0000000000000000 R15:
> 00007fa602e6e990
> [   12.645305] FS:  00007fa602e69880(0000) GS:ffff88001f000000(0000)
> knlGS:0000000000000000
> [   12.645305] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [   12.645305] CR2: 0000000000000000 CR3: 000000001d7fb000 CR4:
> 00000000000006f0
> [   12.645305]
> [   12.645305] Kernel panic - not syncing: Machine halted.
> [   12.645305] CPU: 0 PID: 141 Comm: mount Not tainted
> 3.11.8-200.strato0002.fc19.strato.c3850ae03e9d.x86_64 #1
> [   12.645305] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
> [   12.645305]  ffff88001f005f58 ffff88001f005e90 ffffffff8164024b
> ffffffff819e89dc
> [   12.645305]  ffff88001f005f08 ffffffff8163c272 0000000000000008
> ffff88001f005f18
> [   12.645305]  ffff88001f005eb8 ffffffff8163c8e5 0000000000000046
> 00000000000000b1
> [   12.645305] Call Trace:
> [   12.645305]  <#DF>  [<ffffffff8164024b>]
dump_stack+0x45/0x56
> [   12.645305]  [<ffffffff8163c272>] panic+0xc8/0x1d7
> [   12.645305]  [<ffffffff8163c8e5>] ? printk+0x67/0x69
> [   12.645305]  [<ffffffff81048ae1>] df_debug+0x31/0x40
> [   12.645305]  [<ffffffff810132ed>] do_double_fault+0x5d/0x80
> [   12.645305]  [<ffffffff81650b88>] double_fault+0x28/0x30
> [   12.645305]  <<EOE>>
> [   12.645305] Rebooting in 1 seconds..libguestfs: child_cleanup:
> 0x3a05f50: child process died
> libguestfs: sending SIGTERM to process 1526
> libguestfs: trace: mount_ro = -1 (error)
> libguestfs: trace: vfs_type "/dev/sda1"
> libguestfs: trace: vfs_type = NULL (error)
> libguestfs: check_for_filesystem_on: /dev/sda1 (failed to get vfs type)
> libguestfs: trace: internal_parse_mountable "/dev/sda1"
> libguestfs: trace: internal_parse_mountable = NULL (error)
> libguestfs: trace: inspect_os = NULL (error)
> libguestfs: trace: close
>
>
> Additional info:
> The same crash also happens when L0 is 3.11.9 (with kvm patch to avoid L0
> crash in nested environment -
>
http://git.kernel.org/cgit/virt/kvm/kvm.git/patch/arch/x86/kvm/mmu.c?id=989c6b34f6a9480e397b170cc62237e89bf4fdb9)
but more rarely.
> Thanks a lot,
> Rom
>
>
> _______________________________________________
> Libguestfs mailing list
> Libguestfs@redhat.com
> https://www.redhat.com/mailman/listinfo/libguestfs
>

Richard W.M. Jones

2014-Jan-16 15:23 UTC

head link

Re: [Libguestfs] Double fault panic in L2 upon v2v conversion

On Wed, Jan 15, 2014 at 04:35:29PM +0200, Rom Freiman
wrote:> Hi everybody,
> 
> Wanted to hear your opinion and to receive a smart advice.
> 
> I'm trying to use virt-v2v in order to convert ova image (exported from
> vcenter) to run on libvirt/kvm - all this inside a VM of fedora.
> The converted image is also a fedora.
> During the conversion process, in some point of libguestfs activity, I get
> double fault panic from L2 (printed as part of libguest output) and the
> conversion process fails - no errors appear neither in L0 not in L1 message
> logs.
Are you using nested KVM?

Kashyap (CC'd) has done a lot of testing on nested KVM on *Intel*,
never with satisfactory results.  It just doesn't work very well.

On AMD is a different story -- nested KVM just works.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

Rom Freiman

2014-Jan-16 21:25 UTC

head link

Re: [Libguestfs] Double fault panic in L2 upon v2v conversion

Thanks Richard for a fast reply.

Yes, indeed, im working on a nested environment. I try to run v2v inside a
VM (L1) and to create an L2 by the conversion process. And on Intel. As I
wrote, it fails once in few times, mainly when there is a memory pressure
on L0.

Kashyap, can you please share your experience? Why should it crash during
nested conversion. I'm not too familiar with libguestfs logic - maybe you
can point for me, according to the logs, at what stage of the conversion
the failure happens.

Thanks,
Rom


On Thu, Jan 16, 2014 at 5:23 PM, Richard W.M. Jones
<rjones@redhat.com>wrote:
> On Wed, Jan 15, 2014 at 04:35:29PM +0200, Rom Freiman wrote:
> > Hi everybody,
> >
> > Wanted to hear your opinion and to receive a smart advice.
> >
> > I'm trying to use virt-v2v in order to convert ova image (exported
from
> > vcenter) to run on libvirt/kvm - all this inside a VM of fedora.
> > The converted image is also a fedora.
> > During the conversion process, in some point of libguestfs activity, I
> get
> > double fault panic from L2 (printed as part of libguest output) and
the
> > conversion process fails - no errors appear neither in L0 not in L1
> message
> > logs.
>
> Are you using nested KVM?
>
> Kashyap (CC'd) has done a lot of testing on nested KVM on *Intel*,
> never with satisfactory results.  It just doesn't work very well.
>
> On AMD is a different story -- nested KVM just works.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Fedora Windows cross-compiler. Compile Windows programs, test, and
> build Windows installers. Over 100 libraries supported.
> http://fedoraproject.org/wiki/MinGW
>

Possibly Parallel Threads

Search for more apparently analagous threads

Libguestfs - Jan 2014 - Re: Double fault panic in L2 upon v2v conversion

[Libguestfs] Double fault panic in L2 upon v2v conversion

Re: [Libguestfs] Double fault panic in L2 upon v2v conversion

Re: [Libguestfs] Double fault panic in L2 upon v2v conversion

Re: [Libguestfs] Double fault panic in L2 upon v2v conversion

Re: [Libguestfs] Double fault panic in L2 upon v2v conversion

Possibly Parallel Threads