Konrad Rzeszutek Wilk
2013-Dec-04 19:57 UTC
xl vcpu-set <guest> <nr> does not work properly.
When I try this with the "SeaBIOS" QEMU it does not hotplug all of the CPUs and it complains about the existing ones: -bash-4.1# xl vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 1 -b- 87.1 any cpu Domain-0 0 1 2 -b- 14.4 any cpu Domain-0 0 2 3 -b- 11.0 any cpu Domain-0 0 3 0 r-- 13.7 any cpu latest 7 0 3 -b- 10.9 any cpu latest 7 1 0 -b- 1.7 any cpu latest 7 2 - --p 0.0 any cpu latest 7 3 - --p 0.0 any cpu -bash-4.1# xl vcpu-set 7 4 libxl: error: libxl_qmp.c:287:qmp_handle_error_response: received an error message from QMP server: Unable to add CPU: 0, it already exists libxl: error: libxl_qmp.c:287:qmp_handle_error_response: received an error message from QMP server: Unable to add CPU: 1, it already exists -bash-4.1# xl vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 -b- 87.4 any cpu Domain-0 0 1 3 -b- 14.6 any cpu Domain-0 0 2 2 -b- 11.1 any cpu Domain-0 0 3 1 r-- 13.7 any cpu latest 7 0 2 -b- 11.7 any cpu latest 7 1 1 -b- 1.7 any cpu latest 7 2 3 -b- 0.0 any cpu And sure enough - in the guest it only added one CPU: [ 85.720139] CPU2 has been hot-added [ 85.726409] installing Xen timer for CPU 2 [ 85.735674] smpboot: Booting Node 0 Processor 2 APIC 0x4 [ 85.755080] cpu 2 spinlock event irq 86 [ 85.763671] microcode: CPU2 sig=0x206a7, pf=0x2, revision=0x28 [ 85.770669] Will online and init hotplugged CPU: 2 Perhaps the SeaBIOS/new qemu needs variants of: f62079c piix4acpi, xen, hotplug: Fix race with ACPI AML code and hotplug. e28e062 piix4acpi, xen: Clarify that the qemu_set_irq calls just do an IRQ pulse. 2133c38 piix4acpi, xen, vcpu hotplug: Split the notification from the changes Thanks!
On Wed, Dec 04, 2013 at 02:57:24PM -0500, Konrad Rzeszutek Wilk wrote:> When I try this with the "SeaBIOS" QEMU it does not hotplug > all of the CPUs and it complains about the existing ones: > > -bash-4.1# xl vcpu-list > Name ID VCPU CPU State Time(s) CPU > Affinity > Domain-0 0 0 1 -b- 87.1 any > cpu > Domain-0 0 1 2 -b- 14.4 any > cpu > Domain-0 0 2 3 -b- 11.0 any > cpu > Domain-0 0 3 0 r-- 13.7 any > cpu > latest 7 0 3 -b- 10.9 any > cpu > latest 7 1 0 -b- 1.7 any > cpu > latest 7 2 - --p 0.0 any > cpu > latest 7 3 - --p 0.0 any > cpu > -bash-4.1# xl vcpu-set 7 4 > libxl: error: libxl_qmp.c:287:qmp_handle_error_response: received an > error message from QMP server: Unable to add CPU: 0, it already exists > libxl: error: libxl_qmp.c:287:qmp_handle_error_response: received an > error message from QMP server: Unable to add CPU: 1, it already existsThat because I could not know which CPU where enable in QEMU with v1.3. That should be possible know. I will look at it.> -bash-4.1# xl vcpu-list > Name ID VCPU CPU State Time(s) CPU > Affinity > Domain-0 0 0 0 -b- 87.4 any > cpu > Domain-0 0 1 3 -b- 14.6 any > cpu > Domain-0 0 2 2 -b- 11.1 any > cpu > Domain-0 0 3 1 r-- 13.7 any > cpu > latest 7 0 2 -b- 11.7 any > cpu > latest 7 1 1 -b- 1.7 any > cpu > latest 7 2 3 -b- 0.0 any > cpu > > And sure enough - in the guest it only added one CPU: > > [ 85.720139] CPU2 has been hot-added > [ 85.726409] installing Xen timer for CPU 2 > [ 85.735674] smpboot: Booting Node 0 Processor 2 APIC 0x4 > [ 85.755080] cpu 2 spinlock event irq 86 > [ 85.763671] microcode: CPU2 sig=0x206a7, pf=0x2, revision=0x28 > [ 85.770669] Will online and init hotplugged CPU: 2 > > Perhaps the SeaBIOS/new qemu needs variants of: > > f62079c piix4acpi, xen, hotplug: Fix race with ACPI AML code and hotplug. > e28e062 piix4acpi, xen: Clarify that the qemu_set_irq calls just do an IRQ pulse. > 2133c38 piix4acpi, xen, vcpu hotplug: Split the notification from the changesYes, we should fix that in QEMU. Thanks for reporting, I will work on it. -- Anthony PERARD
On Thu, Dec 05, 2013 at 12:27:21PM +0000, Anthony PERARD wrote:> On Wed, Dec 04, 2013 at 02:57:24PM -0500, Konrad Rzeszutek Wilk wrote: > > When I try this with the "SeaBIOS" QEMU it does not hotplug > > all of the CPUs and it complains about the existing ones: > >> [...]> > > -bash-4.1# xl vcpu-list > > Name ID VCPU CPU State Time(s) CPU > > Affinity > > Domain-0 0 0 0 -b- 87.4 any > > cpu > > Domain-0 0 1 3 -b- 14.6 any > > cpu > > Domain-0 0 2 2 -b- 11.1 any > > cpu > > Domain-0 0 3 1 r-- 13.7 any > > cpu > > latest 7 0 2 -b- 11.7 any > > cpu > > latest 7 1 1 -b- 1.7 any > > cpu > > latest 7 2 3 -b- 0.0 any > > cpu > > > > And sure enough - in the guest it only added one CPU: > > > > [ 85.720139] CPU2 has been hot-added > > [ 85.726409] installing Xen timer for CPU 2 > > [ 85.735674] smpboot: Booting Node 0 Processor 2 APIC 0x4 > > [ 85.755080] cpu 2 spinlock event irq 86 > > [ 85.763671] microcode: CPU2 sig=0x206a7, pf=0x2, revision=0x28 > > [ 85.770669] Will online and init hotplugged CPU: 2 > > > > Perhaps the SeaBIOS/new qemu needs variants of: > > > > f62079c piix4acpi, xen, hotplug: Fix race with ACPI AML code and hotplug. > > e28e062 piix4acpi, xen: Clarify that the qemu_set_irq calls just do an IRQ pulse. > > 2133c38 piix4acpi, xen, vcpu hotplug: Split the notification from the changes > > Yes, we should fix that in QEMU. > > Thanks for reporting, I will work on it.Instead of porting those patch, I think we can fix the issue by a small change in the ACPI table. I tried to reproduce the issue with KVM, but I could not, so I tried to compare the ACPI table and there is one detail that could be interesting, they use Edge-Triggered even, where we have Level-Triggered even. After a quick test, it looks like it works (to use Edge-Triggered even). -- Anthony PERARD
On Mon, 2013-12-09 at 17:10 +0000, Anthony PERARD wrote:> On Thu, Dec 05, 2013 at 12:27:21PM +0000, Anthony PERARD wrote: > > On Wed, Dec 04, 2013 at 02:57:24PM -0500, Konrad Rzeszutek Wilk wrote: > > > When I try this with the "SeaBIOS" QEMU it does not hotplug > > > all of the CPUs and it complains about the existing ones: > > > > > > [...] > > > > > > -bash-4.1# xl vcpu-list > > > Name ID VCPU CPU State Time(s) CPU > > > Affinity > > > Domain-0 0 0 0 -b- 87.4 any > > > cpu > > > Domain-0 0 1 3 -b- 14.6 any > > > cpu > > > Domain-0 0 2 2 -b- 11.1 any > > > cpu > > > Domain-0 0 3 1 r-- 13.7 any > > > cpu > > > latest 7 0 2 -b- 11.7 any > > > cpu > > > latest 7 1 1 -b- 1.7 any > > > cpu > > > latest 7 2 3 -b- 0.0 any > > > cpu > > > > > > And sure enough - in the guest it only added one CPU: > > > > > > [ 85.720139] CPU2 has been hot-added > > > [ 85.726409] installing Xen timer for CPU 2 > > > [ 85.735674] smpboot: Booting Node 0 Processor 2 APIC 0x4 > > > [ 85.755080] cpu 2 spinlock event irq 86 > > > [ 85.763671] microcode: CPU2 sig=0x206a7, pf=0x2, revision=0x28 > > > [ 85.770669] Will online and init hotplugged CPU: 2 > > > > > > Perhaps the SeaBIOS/new qemu needs variants of: > > > > > > f62079c piix4acpi, xen, hotplug: Fix race with ACPI AML code and hotplug. > > > e28e062 piix4acpi, xen: Clarify that the qemu_set_irq calls just do an IRQ pulse. > > > 2133c38 piix4acpi, xen, vcpu hotplug: Split the notification from the changes > > > > Yes, we should fix that in QEMU. > > > > Thanks for reporting, I will work on it. > > Instead of porting those patch, I think we can fix the issue by a small > change in the ACPI table. I tried to reproduce the issue with KVM, but I > could not, so I tried to compare the ACPI table and there is one detail > that could be interesting, they use Edge-Triggered even, where we have > Level-Triggered even. After a quick test, it looks like it works (to use > Edge-Triggered even).This setting needs to match what the hardware (or in this case emulation) expects/implements I think -- so maybe our tables are just wrong. Careful not to break qemu-trad, which may well have the opposite expectation. Ian.
Konrad Rzeszutek Wilk
2013-Dec-09 21:52 UTC
Re: xl vcpu-set <guest> <nr> does not work properly.
On Mon, Dec 09, 2013 at 05:10:28PM +0000, Anthony PERARD wrote:> On Thu, Dec 05, 2013 at 12:27:21PM +0000, Anthony PERARD wrote: > > On Wed, Dec 04, 2013 at 02:57:24PM -0500, Konrad Rzeszutek Wilk wrote: > > > When I try this with the "SeaBIOS" QEMU it does not hotplug > > > all of the CPUs and it complains about the existing ones: > > > > > > [...] > > > > > > -bash-4.1# xl vcpu-list > > > Name ID VCPU CPU State Time(s) CPU > > > Affinity > > > Domain-0 0 0 0 -b- 87.4 any > > > cpu > > > Domain-0 0 1 3 -b- 14.6 any > > > cpu > > > Domain-0 0 2 2 -b- 11.1 any > > > cpu > > > Domain-0 0 3 1 r-- 13.7 any > > > cpu > > > latest 7 0 2 -b- 11.7 any > > > cpu > > > latest 7 1 1 -b- 1.7 any > > > cpu > > > latest 7 2 3 -b- 0.0 any > > > cpu > > > > > > And sure enough - in the guest it only added one CPU: > > > > > > [ 85.720139] CPU2 has been hot-added > > > [ 85.726409] installing Xen timer for CPU 2 > > > [ 85.735674] smpboot: Booting Node 0 Processor 2 APIC 0x4 > > > [ 85.755080] cpu 2 spinlock event irq 86 > > > [ 85.763671] microcode: CPU2 sig=0x206a7, pf=0x2, revision=0x28 > > > [ 85.770669] Will online and init hotplugged CPU: 2 > > > > > > Perhaps the SeaBIOS/new qemu needs variants of: > > > > > > f62079c piix4acpi, xen, hotplug: Fix race with ACPI AML code and hotplug. > > > e28e062 piix4acpi, xen: Clarify that the qemu_set_irq calls just do an IRQ pulse. > > > 2133c38 piix4acpi, xen, vcpu hotplug: Split the notification from the changes > > > > Yes, we should fix that in QEMU. > > > > Thanks for reporting, I will work on it. > > Instead of porting those patch, I think we can fix the issue by a small > change in the ACPI table. I tried to reproduce the issue with KVM, but I > could not, so I tried to compare the ACPI table and there is one detail > that could be interesting, they use Edge-Triggered even, where we have > Level-Triggered even. After a quick test, it looks like it works (to use > Edge-Triggered even).Did you do it with a huge amount of VCPUs? Say you boot with 2 and want to hotplug 32 of them? The reason I am asking is b/c the git commits I mentioned - that is what they were fixing (And also made it work for smaller amount of VCPUs).> > -- > Anthony PERARD
On Mon, Dec 09, 2013 at 04:52:19PM -0500, Konrad Rzeszutek Wilk wrote:> On Mon, Dec 09, 2013 at 05:10:28PM +0000, Anthony PERARD wrote: > > On Thu, Dec 05, 2013 at 12:27:21PM +0000, Anthony PERARD wrote: > > > On Wed, Dec 04, 2013 at 02:57:24PM -0500, Konrad Rzeszutek Wilk wrote: > > > > When I try this with the "SeaBIOS" QEMU it does not hotplug > > > > all of the CPUs and it complains about the existing ones: > > > > > > > > > [...] > > > > > > > > > -bash-4.1# xl vcpu-list > > > > Name ID VCPU CPU State Time(s) CPU > > > > Affinity > > > > Domain-0 0 0 0 -b- 87.4 any > > > > cpu > > > > Domain-0 0 1 3 -b- 14.6 any > > > > cpu > > > > Domain-0 0 2 2 -b- 11.1 any > > > > cpu > > > > Domain-0 0 3 1 r-- 13.7 any > > > > cpu > > > > latest 7 0 2 -b- 11.7 any > > > > cpu > > > > latest 7 1 1 -b- 1.7 any > > > > cpu > > > > latest 7 2 3 -b- 0.0 any > > > > cpu > > > > > > > > And sure enough - in the guest it only added one CPU: > > > > > > > > [ 85.720139] CPU2 has been hot-added > > > > [ 85.726409] installing Xen timer for CPU 2 > > > > [ 85.735674] smpboot: Booting Node 0 Processor 2 APIC 0x4 > > > > [ 85.755080] cpu 2 spinlock event irq 86 > > > > [ 85.763671] microcode: CPU2 sig=0x206a7, pf=0x2, revision=0x28 > > > > [ 85.770669] Will online and init hotplugged CPU: 2 > > > > > > > > Perhaps the SeaBIOS/new qemu needs variants of: > > > > > > > > f62079c piix4acpi, xen, hotplug: Fix race with ACPI AML code and hotplug. > > > > e28e062 piix4acpi, xen: Clarify that the qemu_set_irq calls just do an IRQ pulse. > > > > 2133c38 piix4acpi, xen, vcpu hotplug: Split the notification from the changes > > > > > > Yes, we should fix that in QEMU. > > > > > > Thanks for reporting, I will work on it. > > > > Instead of porting those patch, I think we can fix the issue by a small > > change in the ACPI table. I tried to reproduce the issue with KVM, but I > > could not, so I tried to compare the ACPI table and there is one detail > > that could be interesting, they use Edge-Triggered even, where we have > > Level-Triggered even. After a quick test, it looks like it works (to use > > Edge-Triggered even). > > Did you do it with a huge amount of VCPUs? Say you boot with 2 and > want to hotplug 32 of them?Yes, I tried with at least 16 hotplug at the same time, and maybe more. Just in case, I''m doing another test, with 40 hotplug, just to be sure will writing the email. And the guest picked them all! I always check by adding one more hotplug vcpu and checking that thet guest get only one more.> The reason I am asking is b/c the git commits I mentioned - that is what > they were fixing (And also made it work for smaller amount of VCPUs).-- Anthony PERARD