Rob Austein
2018-Nov-12 04:24 UTC
Panic (gpf) in early boot after upgrading FreeBSD 10.4 -> 11.2 on Ganeti
Belated upgrade (don't ask) of a pair of FreeBSD 10.4 VMs to 11.2.
Each VM got as far as:
freebsd-update -r 11.2-RELEASE update
freebsd-update install
reboot
Each VM got an immediate kernel panic after the reboot (log below).
The two VMs are basically identical at the system level, but run in
separate Ganeti clusters on opposite coasts, so no hardware in common.
A dozen or so other VMs run in each cluster without issues (including
at least one other FreeBSD 11.2 VM), and the VMs I'm trying to upgrade
have also been just fine until now, so the problem seems unlikely to
be hardware per se.
GENERIC, amd64, UFS2, no non-/boot/kernel modules, one CPU per VM.
The only things even slightly unusual about these VMs are:
* They're running in Ganeti clusters:
* Ganeti version 2.15.2
* KVM hypervisor
* They use virtio, so:
* Disk is vtbd0
* Net is vtnet0
Log of attempted boot with new kernel:
/boot/kernel/kernel text=0x1547d48 data=0x144138+0x4e9818
syms=[0x8+0x16aef8+0x8+0x183f99]
Booting...
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27 08:16:24 UTC 2018
root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM
6.0.0)
VT(vga): text 80x25
CPU: QEMU Virtual CPU version 2.1.2 (2666.81-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x663 Family=0x6 Model=0x6
Stepping=3
Features=0x783fbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
Features2=0x80a02001<SSE3,CX16,x2APIC,POPCNT,HV>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Hypervisor: Origin = "KVMKVMKVM"
real memory = 4294967296 (4096 MB)
avail memory = 4088406016 (3899 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <BOCHS BXPCAPIC>
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kernel trap 9 with interrupts disabled
Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer = 0x20:0xffffffff810e9ae6
stack pointer = 0x28:0xffffffff82272c20
frame pointer = 0x28:0xffffffff82272c80
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 0 (swapper)
trap number = 9
panic: general protection fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff80b3d587 at kdb_backtrace+0x67
#1 0xffffffff80af6b27 at vpanic+0x177
#2 0xffffffff80af69a3 at panic+0x43
#3 0xffffffff80f77fdf at trap_fatal+0x35f
#4 0xffffffff80f7759e at trap+0x5e
#5 0xffffffff80f57fbc at calltrap+0x8
#6 0xffffffff810ec5f3 at apic_setup_io+0x53
#7 0xffffffff80a92898 at mi_startup+0x118
#8 0xffffffff8031002c at btext+0x2c
Uptime: 1s
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,
--> or switch off the system now.
Goggling turned up a few theories about bad memory and incompatible
changes to video drivers, none of which seem likely to apply here.
Cluebat, please, somebody?
Thanks for reading.
Konstantin Belousov
2018-Nov-12 06:58 UTC
Panic (gpf) in early boot after upgrading FreeBSD 10.4 -> 11.2 on Ganeti
On Sun, Nov 11, 2018 at 11:24:48PM -0500, Rob Austein wrote:> Belated upgrade (don't ask) of a pair of FreeBSD 10.4 VMs to 11.2. > Each VM got as far as: > > freebsd-update -r 11.2-RELEASE update > freebsd-update install > reboot > > Each VM got an immediate kernel panic after the reboot (log below). > > The two VMs are basically identical at the system level, but run in > separate Ganeti clusters on opposite coasts, so no hardware in common. > A dozen or so other VMs run in each cluster without issues (including > at least one other FreeBSD 11.2 VM), and the VMs I'm trying to upgrade > have also been just fine until now, so the problem seems unlikely to > be hardware per se. > > GENERIC, amd64, UFS2, no non-/boot/kernel modules, one CPU per VM. > The only things even slightly unusual about these VMs are: > > * They're running in Ganeti clusters: > * Ganeti version 2.15.2 > * KVM hypervisor > > * They use virtio, so: > * Disk is vtbd0 > * Net is vtnet0 > > Log of attempted boot with new kernel: > > /boot/kernel/kernel text=0x1547d48 data=0x144138+0x4e9818 syms=[0x8+0x16aef8+0x8+0x183f99] > Booting... > Copyright (c) 1992-2018 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27 08:16:24 UTC 2018 > root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 > FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0) > VT(vga): text 80x25 > CPU: QEMU Virtual CPU version 2.1.2 (2666.81-MHz K8-class CPU) > Origin="GenuineIntel" Id=0x663 Family=0x6 Model=0x6 Stepping=3 > Features=0x783fbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2> > Features2=0x80a02001<SSE3,CX16,x2APIC,POPCNT,HV> > AMD Features=0x20100800<SYSCALL,NX,LM> > AMD Features2=0x1<LAHF> > Hypervisor: Origin = "KVMKVMKVM" > real memory = 4294967296 (4096 MB) > avail memory = 4088406016 (3899 MB) > Event timer "LAPIC" quality 100 > ACPI APIC Table: <BOCHS BXPCAPIC> > ioapic0 <Version 1.1> irqs 0-23 on motherboard > kernel trap 9 with interrupts disabled > > > Fatal trap 9: general protection fault while in kernel mode > cpuid = 0; apic id = 00 > instruction pointer = 0x20:0xffffffff810e9ae6 > stack pointer = 0x28:0xffffffff82272c20 > frame pointer = 0x28:0xffffffff82272c80 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = resume, IOPL = 0 > current process = 0 (swapper) > trap number = 9 > panic: general protection fault > cpuid = 0 > KDB: stack backtrace: > #0 0xffffffff80b3d587 at kdb_backtrace+0x67 > #1 0xffffffff80af6b27 at vpanic+0x177 > #2 0xffffffff80af69a3 at panic+0x43 > #3 0xffffffff80f77fdf at trap_fatal+0x35f > #4 0xffffffff80f7759e at trap+0x5e > #5 0xffffffff80f57fbc at calltrap+0x8 > #6 0xffffffff810ec5f3 at apic_setup_io+0x53 > #7 0xffffffff80a92898 at mi_startup+0x118 > #8 0xffffffff8031002c at btext+0x2c > Uptime: 1s > Automatic reboot in 15 seconds - press a key on the console to abort > --> Press a key on the console to reboot, > --> or switch off the system now. > > Goggling turned up a few theories about bad memory and incompatible > changes to video drivers, none of which seem likely to apply here. > > Cluebat, please, somebody?Try to issue the following commands at the loader prompt: set hw.x2apic_enable=0 boot