For myself KSM is unpredictable feature. The problem is Compute, just this node does "copy on write" , so only Compute. My concern exactly is where would it lead to worse or better Guest behavior ? I am not expecting complete fix. I would track via top/htop and dmesg via Cron on 1-2 hr period. ________________________________ From: centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> on behalf of Laurentiu Soica <laurentiu at soica.ro> Sent: Wednesday, August 17, 2016 6:38 AM To: Discussion about the virtualization on CentOS Subject: Re: [CentOS-virt] Nested KVM issue Both baremetal and compute ? Are there any other metrics do you consider useful to collect for troubleshooting purposes ? ?n mie., 17 aug. 2016 la 13:04, Boris Derzhavets <bderzhavets at hotmail.com<mailto:bderzhavets at hotmail.com>> a scris: It sounds weird, but attempt to disable KSM and see would it help or no ? ________________________________ From: centos-virt-bounces at centos.org<mailto:centos-virt-bounces at centos.org> <centos-virt-bounces at centos.org<mailto:centos-virt-bounces at centos.org>> on behalf of Laurentiu Soica <laurentiu at soica.ro<mailto:laurentiu at soica.ro>> Sent: Wednesday, August 17, 2016 4:56 AM To: Discussion about the virtualization on CentOS Subject: Re: [CentOS-virt] Nested KVM issue Enabled the logging on both compute and baremetal. Nothing strange in logs: on baremetal : Wed Aug 17 11:51:01 EEST 2016: committed 62310764 free 58501808 Wed Aug 17 11:51:01 EEST 2016: 87025667 < 123574516 and free > 24714903, stop ksm on compute: Wed Aug 17 08:52:52 UTC 2016: committed 24547132 free 76730936 Wed Aug 17 08:52:52 UTC 2016: 45139624 < 102962460 and free > 20592492, stop ksm and the compute node is again at 100% CPU utilization. ?n mar., 16 aug. 2016 la 15:26, Boris Derzhavets <bderzhavets at hotmail.com<mailto:bderzhavets at hotmail.com>> a scris: I would enable ksmtuned logging ,if it has been done verify logs ________________________________ From: centos-virt-bounces at centos.org<mailto:centos-virt-bounces at centos.org> <centos-virt-bounces at centos.org<mailto:centos-virt-bounces at centos.org>> on behalf of Laurentiu Soica <laurentiu at soica.ro<mailto:laurentiu at soica.ro>> Sent: Tuesday, August 16, 2016 7:16 AM To: Discussion about the virtualization on CentOS Subject: Re: [CentOS-virt] Nested KVM issue Yes. It is on both baremetal and compute node. ?n mar., 16 aug. 2016 la 13:37, Boris Derzhavets <bderzhavets at hotmail.com<mailto:bderzhavets at hotmail.com>> a scris: Is KSM enabled on your Compute Nodes ( presuming CentOS 7.2 on bare metal ) ? ________________________________ From: centos-virt-bounces at centos.org<mailto:centos-virt-bounces at centos.org> <centos-virt-bounces at centos.org<mailto:centos-virt-bounces at centos.org>> on behalf of Laurentiu Soica <laurentiu at soica.ro<mailto:laurentiu at soica.ro>> Sent: Tuesday, August 16, 2016 5:25 AM To: Discussion about the virtualization on CentOS Subject: Re: [CentOS-virt] Nested KVM issue Running the compute node for several days simply triggers it. ?n mar., 16 aug. 2016 la 12:12, Boris Derzhavets <bderzhavets at hotmail.com<mailto:bderzhavets at hotmail.com>> a scris: Sorry, How you trigger the problem ? B. ________________________________ From: centos-virt-bounces at centos.org<mailto:centos-virt-bounces at centos.org> <centos-virt-bounces at centos.org<mailto:centos-virt-bounces at centos.org>> on behalf of Laurentiu Soica <laurentiu at soica.ro<mailto:laurentiu at soica.ro>> Sent: Tuesday, August 16, 2016 3:28 AM To: Discussion about the virtualization on CentOS Subject: Re: [CentOS-virt] Nested KVM issue Hello, The issue reproduced again and it doesn't look like a swap problem. Some details: on the baremetal, from top: top - 08:08:52 up 5 days, 16:43, 3 users, load average: 36.19, 36.05, 36.05 Tasks: 493 total, 1 running, 492 sleeping, 0 stopped, 0 zombie %Cpu(s): 3.5 us, 87.9 sy, 0.0 ni, 8.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 12357451+total, 14296000 free, 65634428 used, 43644088 buff/cache KiB Swap: 4194300 total, 4073868 free, 120432 used. 56953888 avail Mem 19158 qemu 20 0 0.098t 0.041t 10476 S 3650 35.6 13048:24 qemu-kvm The compute node has 36 CPUs and the usage is now 100%. There are more than 50 GB of memory still available on the baremetal. The swap is barely used, 120 MB. On compute node, from top: top - 05:11:58 up 1 day, 15:08, 2 users, load average: 40.46, 40.49, 40.74 %Cpu(s): 99.1 us, 0.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.1 si, 0.1 st KiB Mem : 10296246+total, 78079936 free, 23671360 used, 1211160 buff/cache KiB Swap: 0 total, 0 free, 0 used. 78939968 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6032 qemu 20 0 10.601g 1.272g 12964 S 400.0 1.3 588:40.39 qemu-kvm 5673 qemu 20 0 10.602g 1.006g 13020 S 399.7 1.0 1161:47 qemu-kvm 5998 qemu 20 0 10.601g 1.192g 13028 S 367.9 1.2 1544:30 qemu-kvm 5951 qemu 20 0 10.601g 1.246g 13020 S 348.3 1.3 1547:38 qemu-kvm 5750 qemu 20 0 10.599g 990136 13060 S 339.1 1.0 1152:25 qemu-kvm 5752 qemu 20 0 10.598g 1.426g 13040 S 313.9 1.5 663:13.65 qemu-kvm .... There are more than 70 GB of memory available on the compute node. All VMs are using 100% their CPUs and they are not accessible anymore. Laurentiu ?n dum., 14 aug. 2016 la 21:44, Boris Derzhavets <bderzhavets at hotmail.com<mailto:bderzhavets at hotmail.com>> a scris: ________________________________ From: centos-virt-bounces at centos.org<mailto:centos-virt-bounces at centos.org> <centos-virt-bounces at centos.org<mailto:centos-virt-bounces at centos.org>> on behalf of Laurentiu Soica <laurentiu at soica.ro<mailto:laurentiu at soica.ro>> Sent: Sunday, August 14, 2016 10:17 AM To: Discussion about the virtualization on CentOS Subject: Re: [CentOS-virt] Nested KVM issue More details on the subject: I suppose it is a nested KVM issue because it raised after I enabled the nested KVM feature. Without it, anyway, the second level VMs are unusable in terms of performance. I am using CentOS 7 with: kernel: 3.10.0-327.22.2.el7.x86_64 qemu-kvm:1.5.3-105.el7_2.4 libvirt:1.2.17-13.el7_2.5 on both the baremetal and the compute VM. Please, post 1) # virsh dumpxml VM-L1 ( where on L1 level you expect nested KVM to appear) 2) Login into VM-L1 and run :- # lsmod | grep kvm 3) I need outputs from VM-L1 ( in case it is Compute Node ) # cat /etc/nova/nova.conf | grep virt_type # cat /etc/nova/nova.conf | grep cpu_mode Boris. The only workaround now is to shutdown the compute VM and start it back from baremetal with virsh start. A simple restart of the compute node doesn't help. It looks like the qemu-kvm process corresponding to the compute VM is the problem. Laurentiu ?n dum., 14 aug. 2016 la 00:19, Laurentiu Soica <laurentiu at soica.ro<mailto:laurentiu at soica.ro>> a scris: Hello, I have an OpenStack setup in virtual environment on CentOS 7. The baremetal has nested KVM enabled and 1 compute node as a VM. Inside the compute node I have multiple VMs running. After about every 3 days the VMs get inaccessible and the compute node reports high CPU usage. The qemu-kvm process for each VM inside the compute node reports full CPU usage. Please help me with some hints to debug this issue. Thanks, Laurentiu _______________________________________________ CentOS-virt mailing list CentOS-virt at centos.org<mailto:CentOS-virt at centos.org> https://lists.centos.org/mailman/listinfo/centos-virt _______________________________________________ CentOS-virt mailing list CentOS-virt at centos.org<mailto:CentOS-virt at centos.org> https://lists.centos.org/mailman/listinfo/centos-virt _______________________________________________ CentOS-virt mailing list CentOS-virt at centos.org<mailto:CentOS-virt at centos.org> https://lists.centos.org/mailman/listinfo/centos-virt _______________________________________________ CentOS-virt mailing list CentOS-virt at centos.org<mailto:CentOS-virt at centos.org> https://lists.centos.org/mailman/listinfo/centos-virt _______________________________________________ CentOS-virt mailing list CentOS-virt at centos.org<mailto:CentOS-virt at centos.org> https://lists.centos.org/mailman/listinfo/centos-virt -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-virt/attachments/20160817/c8c7cf6c/attachment-0002.html>
I've tried with KSM disabled and nothing changed. I've upgraded KVM to qemu-kvm-ev. I'm waiting to see if there are any improvements and report back. ?n mie., 17 aug. 2016 la 15:10, Boris Derzhavets <bderzhavets at hotmail.com> a scris:> For myself KSM is unpredictable feature. The problem is Compute, just this > node > > does "copy on write" , so only Compute. > > My concern exactly is where would it lead to worse or better Guest > behavior ? > > I am not expecting complete fix. I would track via top/htop and dmesg > via Cron on 1-2 hr > > period. > > > ------------------------------ > *From:* centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> > on behalf of Laurentiu Soica <laurentiu at soica.ro> > *Sent:* Wednesday, August 17, 2016 6:38 AM > > *To:* Discussion about the virtualization on CentOS > *Subject:* Re: [CentOS-virt] Nested KVM issue > Both baremetal and compute ? Are there any other metrics do you consider > useful to collect for troubleshooting purposes ? > ?n mie., 17 aug. 2016 la 13:04, Boris Derzhavets <bderzhavets at hotmail.com> > a scris: > >> It sounds weird, but attempt to disable KSM and see would it help or no ? >> >> >> ------------------------------ >> *From:* centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> >> on behalf of Laurentiu Soica <laurentiu at soica.ro> >> *Sent:* Wednesday, August 17, 2016 4:56 AM >> >> *To:* Discussion about the virtualization on CentOS >> *Subject:* Re: [CentOS-virt] Nested KVM issue >> Enabled the logging on both compute and baremetal. Nothing strange in >> logs: >> >> on baremetal : >> Wed Aug 17 11:51:01 EEST 2016: committed 62310764 free 58501808 >> Wed Aug 17 11:51:01 EEST 2016: 87025667 < 123574516 and free > 24714903, >> stop ksm >> >> on compute: >> Wed Aug 17 08:52:52 UTC 2016: committed 24547132 free 76730936 >> Wed Aug 17 08:52:52 UTC 2016: 45139624 < 102962460 and free > 20592492, >> stop ksm >> >> and the compute node is again at 100% CPU utilization. >> >> >> >> ?n mar., 16 aug. 2016 la 15:26, Boris Derzhavets <bderzhavets at hotmail.com> >> a scris: >> >>> I would enable ksmtuned logging ,if it has been done verify logs >>> >>> >>> ------------------------------ >>> *From:* centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> >>> on behalf of Laurentiu Soica <laurentiu at soica.ro> >>> *Sent:* Tuesday, August 16, 2016 7:16 AM >>> >>> *To:* Discussion about the virtualization on CentOS >>> *Subject:* Re: [CentOS-virt] Nested KVM issue >>> Yes. It is on both baremetal and compute node. >>> >>> ?n mar., 16 aug. 2016 la 13:37, Boris Derzhavets < >>> bderzhavets at hotmail.com> a scris: >>> >>>> Is KSM enabled on your Compute Nodes ( presuming CentOS 7.2 on bare >>>> metal ) ? >>>> >>>> >>>> ------------------------------ >>>> *From:* centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> >>>> on behalf of Laurentiu Soica <laurentiu at soica.ro> >>>> *Sent:* Tuesday, August 16, 2016 5:25 AM >>>> >>>> *To:* Discussion about the virtualization on CentOS >>>> *Subject:* Re: [CentOS-virt] Nested KVM issue >>>> Running the compute node for several days simply triggers it. >>>> >>>> ?n mar., 16 aug. 2016 la 12:12, Boris Derzhavets < >>>> bderzhavets at hotmail.com> a scris: >>>> >>>>> Sorry, >>>>> >>>>> How you trigger the problem ? >>>>> >>>>> B. >>>>> >>>>> >>>>> ------------------------------ >>>>> *From:* centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> >>>>> on behalf of Laurentiu Soica <laurentiu at soica.ro> >>>>> *Sent:* Tuesday, August 16, 2016 3:28 AM >>>>> >>>>> *To:* Discussion about the virtualization on CentOS >>>>> *Subject:* Re: [CentOS-virt] Nested KVM issue >>>>> Hello, >>>>> >>>>> The issue reproduced again and it doesn't look like a swap problem. >>>>> Some details: >>>>> >>>>> on the baremetal, from top: >>>>> >>>>> top - 08:08:52 up 5 days, 16:43, 3 users, load average: 36.19, >>>>> 36.05, 36.05 >>>>> Tasks: 493 total, 1 running, 492 sleeping, 0 stopped, 0 zombie >>>>> >>>>> %Cpu(s): 3.5 us, 87.9 sy, 0.0 ni, 8.6 id, 0.0 wa, 0.0 hi, 0.0 >>>>> si, 0.0 st >>>>> KiB Mem : 12357451+total, 14296000 free, 65634428 used, 43644088 >>>>> buff/cache >>>>> KiB Swap: 4194300 total, 4073868 free, 120432 used. 56953888 avail >>>>> Mem >>>>> >>>>> 19158 qemu 20 0 0.098t 0.041t 10476 S 3650 35.6 13048:24 >>>>> qemu-kvm >>>>> >>>>> The compute node has 36 CPUs and the usage is now 100%. There are more >>>>> than 50 GB of memory still available on the baremetal. The swap is barely >>>>> used, 120 MB. >>>>> >>>>> On compute node, from top: >>>>> >>>>> top - 05:11:58 up 1 day, 15:08, 2 users, load average: 40.46, 40.49, >>>>> 40.74 >>>>> >>>>> %Cpu(s): 99.1 us, 0.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.1 >>>>> si, 0.1 st >>>>> KiB Mem : 10296246+total, 78079936 free, 23671360 used, 1211160 >>>>> buff/cache >>>>> KiB Swap: 0 total, 0 free, 0 used. 78939968 avail >>>>> Mem >>>>> >>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>>> COMMAND >>>>> 6032 qemu 20 0 10.601g 1.272g 12964 S 400.0 1.3 588:40.39 >>>>> qemu-kvm >>>>> 5673 qemu 20 0 10.602g 1.006g 13020 S 399.7 1.0 1161:47 >>>>> qemu-kvm >>>>> 5998 qemu 20 0 10.601g 1.192g 13028 S 367.9 1.2 1544:30 >>>>> qemu-kvm >>>>> 5951 qemu 20 0 10.601g 1.246g 13020 S 348.3 1.3 1547:38 >>>>> qemu-kvm >>>>> 5750 qemu 20 0 10.599g 990136 13060 S 339.1 1.0 1152:25 >>>>> qemu-kvm >>>>> 5752 qemu 20 0 10.598g 1.426g 13040 S 313.9 1.5 663:13.65 >>>>> qemu-kvm >>>>> .... >>>>> >>>>> There are more than 70 GB of memory available on the compute node. All >>>>> VMs are using 100% their CPUs and they are not accessible anymore. >>>>> >>>>> Laurentiu >>>>> >>>>> ?n dum., 14 aug. 2016 la 21:44, Boris Derzhavets < >>>>> bderzhavets at hotmail.com> a scris: >>>>> >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------ >>>>>> *From:* centos-virt-bounces at centos.org < >>>>>> centos-virt-bounces at centos.org> on behalf of Laurentiu Soica < >>>>>> laurentiu at soica.ro> >>>>>> *Sent:* Sunday, August 14, 2016 10:17 AM >>>>>> *To:* Discussion about the virtualization on CentOS >>>>>> *Subject:* Re: [CentOS-virt] Nested KVM issue >>>>>> >>>>>> More details on the subject: >>>>>> >>>>>> I suppose it is a nested KVM issue because it raised after I enabled >>>>>> the nested KVM feature. Without it, anyway, the second level VMs are >>>>>> unusable in terms of performance. >>>>>> >>>>>> I am using CentOS 7 with: >>>>>> >>>>>> kernel: 3.10.0-327.22.2.el7.x86_64 >>>>>> qemu-kvm:1.5.3-105.el7_2.4 >>>>>> libvirt:1.2.17-13.el7_2.5 >>>>>> >>>>>> on both the baremetal and the compute VM. >>>>>> >>>>>> *Please, post* >>>>>> >>>>>> 1) # virsh dumpxml VM-L1 ( where on L1 level you expect nested KVM >>>>>> to appear) >>>>>> 2) Login into VM-L1 and run :- >>>>>> # lsmod | grep kvm >>>>>> 3) I need outputs from VM-L1 ( in case it is Compute Node ) >>>>>> >>>>>> # cat /etc/nova/nova.conf | grep virt_type >>>>>> # cat /etc/nova/nova.conf | grep cpu_mode >>>>>> >>>>>> Boris. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> The only workaround now is to shutdown the compute VM and start it >>>>>> back from baremetal with virsh start. >>>>>> A simple restart of the compute node doesn't help. It looks like the >>>>>> qemu-kvm process corresponding to the compute VM is the problem. >>>>>> >>>>>> Laurentiu >>>>>> >>>>>> ?n dum., 14 aug. 2016 la 00:19, Laurentiu Soica <laurentiu at soica.ro> >>>>>> a scris: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I have an OpenStack setup in virtual environment on CentOS 7. >>>>>>> >>>>>>> The baremetal has *nested KVM* enabled and 1 compute node as a VM. >>>>>>> >>>>>>> Inside the compute node I have multiple VMs running. >>>>>>> >>>>>>> After about every 3 days the VMs get inaccessible and the compute >>>>>>> node reports high CPU usage. The qemu-kvm process for each VM inside the >>>>>>> compute node reports full CPU usage. >>>>>>> >>>>>>> Please help me with some hints to debug this issue. >>>>>>> >>>>>>> Thanks, >>>>>>> Laurentiu >>>>>>> >>>>>> _______________________________________________ >>>>>> CentOS-virt mailing list >>>>>> CentOS-virt at centos.org >>>>>> https://lists.centos.org/mailman/listinfo/centos-virt >>>>>> >>>>> _______________________________________________ >>>>> CentOS-virt mailing list >>>>> CentOS-virt at centos.org >>>>> https://lists.centos.org/mailman/listinfo/centos-virt >>>>> >>>> _______________________________________________ >>>> CentOS-virt mailing list >>>> CentOS-virt at centos.org >>>> https://lists.centos.org/mailman/listinfo/centos-virt >>>> >>> _______________________________________________ >>> CentOS-virt mailing list >>> CentOS-virt at centos.org >>> https://lists.centos.org/mailman/listinfo/centos-virt >>> >> _______________________________________________ >> CentOS-virt mailing list >> CentOS-virt at centos.org >> https://lists.centos.org/mailman/listinfo/centos-virt >> > _______________________________________________ > CentOS-virt mailing list > CentOS-virt at centos.org > https://lists.centos.org/mailman/listinfo/centos-virt >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-virt/attachments/20160818/5ed574c3/attachment-0002.html>
No luck with qemu-kvm-ev, the behavior is the same. Running perf record -a -g on the baremetal shows that most of the CPU time is in _raw_spin_lock Children Self Command Shared Object Symbol - 93.62% 93.62% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock - _raw_spin_lock + 45.30% kvm_mmu_sync_roots + 28.49% kvm_mmu_load + 25.00% mmu_free_roots + 1.12% tdp_page_fault ?n joi, 18 aug. 2016 la 11:59, Laurentiu Soica <laurentiu at soica.ro> a scris:> I've tried with KSM disabled and nothing changed. > > I've upgraded KVM to qemu-kvm-ev. I'm waiting to see if there are any > improvements and report back. > > ?n mie., 17 aug. 2016 la 15:10, Boris Derzhavets <bderzhavets at hotmail.com> > a scris: > >> For myself KSM is unpredictable feature. The problem is Compute, just >> this node >> >> does "copy on write" , so only Compute. >> >> My concern exactly is where would it lead to worse or better Guest >> behavior ? >> >> I am not expecting complete fix. I would track via top/htop and dmesg >> via Cron on 1-2 hr >> >> period. >> >> >> ------------------------------ >> *From:* centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> >> on behalf of Laurentiu Soica <laurentiu at soica.ro> >> *Sent:* Wednesday, August 17, 2016 6:38 AM >> >> *To:* Discussion about the virtualization on CentOS >> *Subject:* Re: [CentOS-virt] Nested KVM issue >> Both baremetal and compute ? Are there any other metrics do you consider >> useful to collect for troubleshooting purposes ? >> ?n mie., 17 aug. 2016 la 13:04, Boris Derzhavets <bderzhavets at hotmail.com> >> a scris: >> >>> It sounds weird, but attempt to disable KSM and see would it help or no ? >>> >>> >>> ------------------------------ >>> *From:* centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> >>> on behalf of Laurentiu Soica <laurentiu at soica.ro> >>> *Sent:* Wednesday, August 17, 2016 4:56 AM >>> >>> *To:* Discussion about the virtualization on CentOS >>> *Subject:* Re: [CentOS-virt] Nested KVM issue >>> Enabled the logging on both compute and baremetal. Nothing strange in >>> logs: >>> >>> on baremetal : >>> Wed Aug 17 11:51:01 EEST 2016: committed 62310764 free 58501808 >>> Wed Aug 17 11:51:01 EEST 2016: 87025667 < 123574516 and free > 24714903, >>> stop ksm >>> >>> on compute: >>> Wed Aug 17 08:52:52 UTC 2016: committed 24547132 free 76730936 >>> Wed Aug 17 08:52:52 UTC 2016: 45139624 < 102962460 and free > 20592492, >>> stop ksm >>> >>> and the compute node is again at 100% CPU utilization. >>> >>> >>> >>> ?n mar., 16 aug. 2016 la 15:26, Boris Derzhavets < >>> bderzhavets at hotmail.com> a scris: >>> >>>> I would enable ksmtuned logging ,if it has been done verify logs >>>> >>>> >>>> ------------------------------ >>>> *From:* centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> >>>> on behalf of Laurentiu Soica <laurentiu at soica.ro> >>>> *Sent:* Tuesday, August 16, 2016 7:16 AM >>>> >>>> *To:* Discussion about the virtualization on CentOS >>>> *Subject:* Re: [CentOS-virt] Nested KVM issue >>>> Yes. It is on both baremetal and compute node. >>>> >>>> >>>> _______________________________________________ >>> CentOS-virt mailing list >>> CentOS-virt at centos.org >>> https://lists.centos.org/mailman/listinfo/centos-virt >>> >> _______________________________________________ >> CentOS-virt mailing list >> CentOS-virt at centos.org >> https://lists.centos.org/mailman/listinfo/centos-virt >> > --Laurentiu Soica -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-virt/attachments/20160822/392b8328/attachment-0002.html>