CentOS - Sep 2018 - Panic / EL6 / KVM / kernel-2.6.32-754.2.1.el6.x86_64

Since the update from kernel-2.6.32-754.2.1.el6.x86_64 
to kernel-2.6.32-754.3.5.el6.x86_64 I can not boot my 
KVM guests anymore!? The workstation panics immediately! 

I would not have expected this behavior now (last phase of OS). 
It was very robust until now (Optiplex Workstation). I see some KVM 
related lines in the changelog.diff. Before swimming upstream:

Does some one have problems related to KVM with kernel-2.6.32-754.3.5.el6.x86_64
??

--
LF

On Wed, 29 Aug 2018 at 17:41, Leon Fauster via CentOS <centos at
centos.org>
wrote:
> Since the update from kernel-2.6.32-754.2.1.el6.x86_64
> to kernel-2.6.32-754.3.5.el6.x86_64 I can not boot my
> KVM guests anymore!? The workstation panics immediately!
>
> I would not have expected this behavior now (last phase of OS).
> It was very robust until now (Optiplex Workstation). I see some KVM
> related lines in the changelog.diff. Before swimming upstream:
>
> Does some one have problems related to KVM with
> kernel-2.6.32-754.3.5.el6.x86_64 ??
>
>
Not that I know of.
* Does the problem go away if you back off to 2.1 ?
* And what type of panic does it say?
* What kind of Optiplex Workstation with memory/cpu type/cores?



> --
> LF
>
>
>
>
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>

-- 
Stephen J Smoogen.

Am 29.08.2018 um 23:46 schrieb Stephen John Smoogen <smooge at
gmail.com>:
> 
> On Wed, 29 Aug 2018 at 17:41, Leon Fauster via CentOS <centos at
centos.org> wrote:
>> Since the update from kernel-2.6.32-754.2.1.el6.x86_64 
>> to kernel-2.6.32-754.3.5.el6.x86_64 I can not boot my 
>> KVM guests anymore!? The workstation panics immediately! 
>> 
>> I would not have expected this behavior now (last phase of OS). 
>> It was very robust until now (Optiplex Workstation). I see some KVM 
>> related lines in the changelog.diff. Before swimming upstream:
>> 
>> Does some one have problems related to KVM with
kernel-2.6.32-754.3.5.el6.x86_64 ??
>> 
> 
> Not that I know of. 
> * Does the problem go away if you back off to 2.1 ? 
Yes

> * And what type of panic does it say?
I will try to grep some lines at the console tomorrow.

> * What kind of Optiplex Workstation with memory/cpu type/cores?
# virsh sysinfo
<sysinfo type='smbios'>
 <bios>
   <entry name='vendor'>Dell Inc.</entry>
   <entry name='version'>A19</entry>
   <entry name='date'>05/31/2011</entry>
   <entry name='release'>18.0</entry>
 </bios>
 <system>
   <entry name='manufacturer'>Dell Inc.</entry>
   <entry name='product'>OptiPlex 755                
</entry>
   <entry name='version'>Not Specified</entry>
   <entry name='serial'>Not Specified</entry>
   <entry name='uuid'>XXXX-3700-1058-8047-XXXX</entry>
   <entry name='sku'>Not Specified</entry>
   <entry name='family'>Not Specified</entry>
 </system>
 <processor>
   <entry name='socket_destination'>CPU</entry>
   <entry name='type'>Central Processor</entry>
   <entry name='family'>Core 2 Duo</entry>
   <entry name='manufacturer'>Intel</entry>
   <entry name='signature'>Type 0, Family 6, Model 15, Stepping
11</entry>
   <entry name='version'>Not Specified</entry>
   <entry name='external_clock'>1333 MHz</entry>
   <entry name='max_speed'>5200 MHz</entry>
   <entry name='status'>Populated, Enabled</entry>
   <entry name='serial_number'>Not Specified</entry>
   <entry name='part_number'>Not Specified</entry>
 </processor>
 <memory_device>
   <entry name='size'>2048 MB</entry>
   <entry name='form_factor'>DIMM</entry>
   <entry name='locator'>DIMM_1</entry>
   <entry name='bank_locator'>Not Specified</entry>
   <entry name='type'>DDR2</entry>
   <entry name='type_detail'>Synchronous</entry>
   <entry name='speed'>800 MHz</entry>
   <entry name='manufacturer'>CE00000000000000</entry>
   <entry name='serial_number'>DELETED</entry>
   <entry name='part_number'>M3 78T5663DZ3-CF7</entry>
 </memory_device>
 <memory_device>
   <entry name='size'>2048 MB</entry>
   <entry name='form_factor'>DIMM</entry>
   <entry name='locator'>DIMM_3</entry>
   <entry name='bank_locator'>Not Specified</entry>
   <entry name='type'>DDR2</entry>
   <entry name='type_detail'>Synchronous</entry>
   <entry name='speed'>800 MHz</entry>
   <entry name='manufacturer'>7F98000000000000</entry>
   <entry name='serial_number'>DELETED</entry>
 </memory_device>
 <memory_device>
   <entry name='size'>2048 MB</entry>
   <entry name='form_factor'>DIMM</entry>
   <entry name='locator'>DIMM_2</entry>
   <entry name='bank_locator'>Not Specified</entry>
   <entry name='type'>DDR2</entry>
   <entry name='type_detail'>Synchronous</entry>
   <entry name='speed'>800 MHz</entry>
   <entry name='manufacturer'>CE00000000000000</entry>
   <entry name='serial_number'>DELETED</entry>
   <entry name='part_number'>M3 78T5663DZ3-CF7</entry>
 </memory_device>
 <memory_device>
   <entry name='size'>2048 MB</entry>
   <entry name='form_factor'>DIMM</entry>
   <entry name='locator'>DIMM_4</entry>
   <entry name='bank_locator'>Not Specified</entry>
   <entry name='type'>DDR2</entry>
   <entry name='type_detail'>Synchronous</entry>
   <entry name='speed'>800 MHz</entry>
   <entry name='manufacturer'>7F98000000000000</entry>
   <entry name='serial_number'>DELETED</entry>
 </memory_device>
</sysinfo>

--
LF

> Since the update from kernel-2.6.32-754.2.1.el6.x86_64
> to kernel-2.6.32-754.3.5.el6.x86_64 I can not boot my
> KVM guests anymore!? The workstation panics immediately!
>
> I would not have expected this behavior now (last phase of OS).
> It was very robust until now (Optiplex Workstation). I see some KVM
> related lines in the changelog.diff. Before swimming upstream:
>
> Does some one have problems related to KVM with
> kernel-2.6.32-754.3.5.el6.x86_64 ??
Is there any chance that this is related? Could you try downgrading
qemu-img/qemu-kvm and see if it helps?

https://bugs.centos.org/view.php?id=15067

Regards,
Simon

Leon Fauster via CentOS writes:
> Since the update from kernel-2.6.32-754.2.1.el6.x86_64 
> to kernel-2.6.32-754.3.5.el6.x86_64 I can not boot my 
> KVM guests anymore!? The workstation panics immediately! 
> 
> I would not have expected this behavior now (last phase of OS). 
> It was very robust until now (Optiplex Workstation). I see some KVM 
> related lines in the changelog.diff. Before swimming upstream:
> 
> Does some one have problems related to KVM with
kernel-2.6.32-754.3.5.el6.x86_64 ??
Yes, the exact same thing happened here, and I suspect it is related to older
cpus that don't get any Spectre/Meltdown updates.

IBM x3250
Intel(R) Xeon(R) CPU           E3110  @ 3.00GHz

This is a dual-core cpu of similar vintage to yours (can we have a model #?),
pre-2010.

There goes a cheap and reliable VM dev machine :-/

Am 30.08.2018 um 10:54 schrieb isdtor <isdtor at gmail.com>:
> 
> Leon Fauster via CentOS writes:
>> Since the update from kernel-2.6.32-754.2.1.el6.x86_64 
>> to kernel-2.6.32-754.3.5.el6.x86_64 I can not boot my 
>> KVM guests anymore!? The workstation panics immediately! 
>> 
>> I would not have expected this behavior now (last phase of OS). 
>> It was very robust until now (Optiplex Workstation). I see some KVM 
>> related lines in the changelog.diff. Before swimming upstream:
>> 
>> Does some one have problems related to KVM with
kernel-2.6.32-754.3.5.el6.x86_64 ??
> 
> Yes, the exact same thing happened here, and I suspect it is related to
older cpus that don't get any Spectre/Meltdown updates.

Thanks for the feedback. I' was assuming that some kind of Spectre/Meltdown
fixes are causing this.


> IBM x3250
> Intel(R) Xeon(R) CPU           E3110  @ 3.00GHz
> 
> This is a dual-core cpu of similar vintage to yours (can we have a model
#?), pre-2010.

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Duo CPU     E6850  @ 3.00GHz
stepping	: 11
microcode	: 186
cpu MHz		: 2000.000
cache size	: 4096 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc
arch_perfmon pebs bts rep_good aperfmperf eagerfpu pni dtes64 monitor ds_cpl vmx
smx est tm2 ssse3 cx16 xtpr pdcm lahf_lm dtherm pti retpoline tpr_shadow vnmi
flexpriority
bogomips	: 5984.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:




> There goes a cheap and reliable VM dev machine :-/

No way. Should all IT departments trash a big percentage of there hardware now?

--
LF

Am 30.08.2018 um 06:37 schrieb Simon Matter <simon.matter at
invoca.ch>:
> 
>> Since the update from kernel-2.6.32-754.2.1.el6.x86_64
>> to kernel-2.6.32-754.3.5.el6.x86_64 I can not boot my
>> KVM guests anymore!? The workstation panics immediately!
>> 
>> I would not have expected this behavior now (last phase of OS).
>> It was very robust until now (Optiplex Workstation). I see some KVM
>> related lines in the changelog.diff. Before swimming upstream:
>> 
>> Does some one have problems related to KVM with
>> kernel-2.6.32-754.3.5.el6.x86_64 ??
> 
> Is there any chance that this is related? Could you try downgrading
> qemu-img/qemu-kvm and see if it helps?
> 
> https://bugs.centos.org/view.php?id=15067
> 
Thanks to pointing to that bug report. 

I downgraded from 

qemu-kvm-0.12.1.2-2.506.el6_10.1.x86_64
qemu-img-0.12.1.2-2.506.el6_10.1.x86_64 

to 

qemu-img-0.12.1.2-2.503.el6_9.6.x86_64
qemu-kvm-0.12.1.2-2.503.el6_9.6.x86_64

and booted into

kernel-2.6.32-754.3.5.el6.x86_64

unfortunately no positiv effect.

The system/host/workstation crashes/panics immediately after 
starting a guest system (EL6) with virsh start guest-name.


BTW upstream bug report:

https://bugzilla.redhat.com/show_bug.cgi?id=1623692

--
LF

Am 30.08.2018 um 20:28 schrieb Simon Matter <simon.matter at invoca.ch>:
> Am 30.08.2018 um 12:16 schrieb Leon Fauster <leonfauster at
googlemail.com>:
>> 
>> BTW upstream bug report:
>> 
>> https://bugzilla.redhat.com/show_bug.cgi?id=1623692
> 
> Well, I have an account and am logged in, still can not see the bug.

It seems that the default bugzilla classification doesn't allow bug reports
associated
with the kernel to get a read status or so ... just to summarize the status
briefly:
They fortunately can reproduce the problem and are trying to find the cause now.
So, +1!

--
LF