Xen devel - Feb 2011 - [PATCH] Avoid race when moving cpu between cpupools

Moving cpus between cpupools is done under the schedule lock of the moved cpu.
When checking a cpu being member of a cpupool this must be done with the lock
of that cpu being held.
Hot-unplugging of physical cpus might encounter the same problems, but this
should happen only very rarely.

Signed-off-by: juergen.gross@ts.fujitsu.com


2 files changed, 35 insertions(+), 7 deletions(-)
xen/common/sched_credit.c |    3 ++-
xen/common/schedule.c     |   39 +++++++++++++++++++++++++++++++++------




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Juergen Gross wrote:
> Moving cpus between cpupools is done under the schedule lock of the moved
cpu.
> When checking a cpu being member of a cpupool this must be done with the
lock
> of that cpu being held.
I have reviewed and tested the patch. It fixes my problem. My script has 
been running for several hundred iterations without any Xen crash, 
whereas without the patch the hypervisor crashed mostly at the second 
iteration.

Thanks Juergen and George for the persistent work!
> Hot-unplugging of physical cpus might encounter the same problems, but this
> should happen only very rarely.
> 
> Signed-off-by: juergen.gross@ts.fujitsu.com
Acked-by: Andre Przywara <andre.przywara@amd.com>

Keir, please apply for 4.1.0.


Regards,
Andre.

-- 
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Looks good -- thanks Juergen.

Acked-by: George Dunlap <george.dunlap@eu.citrix.com>

 -George

On Thu, Feb 24, 2011 at 2:08 PM, Andre Przywara <andre.przywara@amd.com>
wrote:
> Juergen Gross wrote:
>>
>> Moving cpus between cpupools is done under the schedule lock of the
moved
>> cpu.
>> When checking a cpu being member of a cpupool this must be done with
the
>> lock
>> of that cpu being held.
>
> I have reviewed and tested the patch. It fixes my problem. My script has
> been running for several hundred iterations without any Xen crash, whereas
> without the patch the hypervisor crashed mostly at the second iteration.
>
> Thanks Juergen and George for the persistent work!
>
>> Hot-unplugging of physical cpus might encounter the same problems, but
>> this
>> should happen only very rarely.
>>
>> Signed-off-by: juergen.gross@ts.fujitsu.com
>
> Acked-by: Andre Przywara <andre.przywara@amd.com>
>
> Keir, please apply for 4.1.0.
>
>
> Regards,
> Andre.
>
> --
> Andre Przywara
> AMD-Operating System Research Center (OSRC), Dresden, Germany
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

George Dunlap wrote:
> Looks good -- thanks Juergen.
> 
> Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
> 
>  -George
> 
> On Thu, Feb 24, 2011 at 2:08 PM, Andre Przywara
<andre.przywara@amd.com> wrote:
>> Juergen Gross wrote:
>>> Moving cpus between cpupools is done under the schedule lock of the
moved
>>> cpu.
>>> When checking a cpu being member of a cpupool this must be done
with the
>>> lock
>>> of that cpu being held.
>> I have reviewed and tested the patch. It fixes my problem. My script
has
>> been running for several hundred iterations without any Xen crash,
whereas
>> without the patch the hypervisor crashed mostly at the second
iteration.
Juergen,

can you rule out that this code will be triggered on two CPUs trying to 
switch to each other? As Stephan pointed out: the code looks like as 
this could trigger a possible dead-lock condition, where:
1) CPU A grabs lock (a) while CPU B grabs lock (b)
2) CPU A tries to grab (b) and CPU B tries to grab (a)
3) both fail and loop to 1)
A possible fix would be to introduce some ordering for the locks (just 
the pointer address) and let the "bigger" pointer yield to the
"smaller"
one. I am not sure if this is really necessary, but I now see strange 
hangs after running the script for a while (30min to 1hr).
Sometimes Dom0 hangs for a while, loosing interrupts (sda or eth0) or 
getting spurious ones, on two occasions the machine totally locked up.

I am not 100% sure whether this is CPUpools related, but I put some load 
on Dom0 (without messing with CPUpools) for the whole night and it ran fine.

Sorry for this :-(
I will try to further isolate this.

Anyway, it works much better with the fix than without and I will try to 
trigger this with the "reduce number of Dom0 vCPUs" patch.

Regards,
Andre.
>>
>> Thanks Juergen and George for the persistent work!
>>
>>> Hot-unplugging of physical cpus might encounter the same problems,
but
>>> this
>>> should happen only very rarely.
>>>
>>> Signed-off-by: juergen.gross@ts.fujitsu.com
>> Acked-by: Andre Przywara <andre.przywara@amd.com>
>>
>> Keir, please apply for 4.1.0.
>>

-- 
Andre Przywara
AMD-OSRC (Dresden)
Tel: x29712


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 25/02/2011 14:25, "Andre Przywara" <andre.przywara@amd.com>
wrote:
> can you rule out that this code will be triggered on two CPUs trying to
> switch to each other? As Stephan pointed out: the code looks like as
> this could trigger a possible dead-lock condition, where:
> 1) CPU A grabs lock (a) while CPU B grabs lock (b)
> 2) CPU A tries to grab (b) and CPU B tries to grab (a)
> 3) both fail and loop to 1)
> A possible fix would be to introduce some ordering for the locks (just
> the pointer address) and let the "bigger" pointer yield to the
"smaller"
> one. I am not sure if this is really necessary, but I now see strange
> hangs after running the script for a while (30min to 1hr).
> Sometimes Dom0 hangs for a while, loosing interrupts (sda or eth0) or
> getting spurious ones, on two occasions the machine totally locked up.
In other places in Xen where we take a pair of locks with no other implicit
ordering, we enforce an ordering based lock addresses. See
common/timer.c:migrate_timer() for example. I''m sure there must be at
least
one example of this in the schedulign code already, with vcpus migrating
between cpus and needing both runqueue locks.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 02/25/11 15:25, Andre Przywara wrote:
> George Dunlap wrote:
>> Looks good -- thanks Juergen.
>>
>> Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
>>
>> -George
>>
>> On Thu, Feb 24, 2011 at 2:08 PM, Andre Przywara
>> <andre.przywara@amd.com> wrote:
>>> Juergen Gross wrote:
>>>> Moving cpus between cpupools is done under the schedule lock of
the
>>>> moved
>>>> cpu.
>>>> When checking a cpu being member of a cpupool this must be done
with
>>>> the
>>>> lock
>>>> of that cpu being held.
>>> I have reviewed and tested the patch. It fixes my problem. My
script has
>>> been running for several hundred iterations without any Xen crash,
>>> whereas
>>> without the patch the hypervisor crashed mostly at the second
iteration.
>
> Juergen,
>
> can you rule out that this code will be triggered on two CPUs trying to
> switch to each other? As Stephan pointed out: the code looks like as
> this could trigger a possible dead-lock condition, where:
> 1) CPU A grabs lock (a) while CPU B grabs lock (b)
> 2) CPU A tries to grab (b) and CPU B tries to grab (a)
> 3) both fail and loop to 1)
Good point. Not quite a dead-lock, but a possible live-lock :-)
> A possible fix would be to introduce some ordering for the locks (just
> the pointer address) and let the "bigger" pointer yield to the
"smaller"
> one.
Done this and sent a patch.
> I am not sure if this is really necessary, but I now see strange
> hangs after running the script for a while (30min to 1hr).
> Sometimes Dom0 hangs for a while, loosing interrupts (sda or eth0) or
> getting spurious ones, on two occasions the machine totally locked up.
>
> I am not 100% sure whether this is CPUpools related, but I put some load
> on Dom0 (without messing with CPUpools) for the whole night and it ran
> fine.
Did you try to do this with all Dom0-vcpus pinned to 6 physical cpus?
I had the same problems when using only few physical cpus for many vcpus.
And I''m pretty sure this was NOT the possible live-lock, as it happened
already without this change when I tried to reproduce your problem.
>
> Sorry for this :-(
> I will try to further isolate this.
>
> Anyway, it works much better with the fix than without and I will try to
> trigger this with the "reduce number of Dom0 vCPUs" patch.

Thanks, Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
TSP ES&S SWE OS6                       Telephone: +49 (0) 89 3222 2967
Fujitsu Technology Solutions              e-mail: juergen.gross@ts.fujitsu.com
Domagkstr. 28                           Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Juergen Gross wrote:
> On 02/25/11 15:25, Andre Przywara wrote:
>> George Dunlap wrote:
>>> Looks good -- thanks Juergen.
>>>
>>> Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
>>>
>>> -George
>>>
>>> On Thu, Feb 24, 2011 at 2:08 PM, Andre Przywara
>>> <andre.przywara@amd.com> wrote:
>>>> Juergen Gross wrote:
>>>>> Moving cpus between cpupools is done under the schedule
lock of the
>>>>> moved
>>>>> cpu.
>>>>> When checking a cpu being member of a cpupool this must be
done with
>>>>> the
>>>>> lock
>>>>> of that cpu being held.
>>>> I have reviewed and tested the patch. It fixes my problem. My
script has
>>>> been running for several hundred iterations without any Xen
crash,
>>>> whereas
>>>> without the patch the hypervisor crashed mostly at the second
iteration.
>> Juergen,
>>
>> can you rule out that this code will be triggered on two CPUs trying to
>> switch to each other? As Stephan pointed out: the code looks like as
>> this could trigger a possible dead-lock condition, where:
>> 1) CPU A grabs lock (a) while CPU B grabs lock (b)
>> 2) CPU A tries to grab (b) and CPU B tries to grab (a)
>> 3) both fail and loop to 1)
> 
> Good point. Not quite a dead-lock, but a possible live-lock :-)
Yeah, sorry. That was the wrong wording. Kudos go to Stephan for 
pointing this out.
> 
>> A possible fix would be to introduce some ordering for the locks (just
>> the pointer address) and let the "bigger" pointer yield to
the "smaller"
>> one.
> 
> Done this and sent a patch.
Thanks, it looks good on the first glance. Not yet tested, though.
> 
>> I am not sure if this is really necessary, but I now see strange
>> hangs after running the script for a while (30min to 1hr).
>> Sometimes Dom0 hangs for a while, loosing interrupts (sda or eth0) or
>> getting spurious ones, on two occasions the machine totally locked up.
>>
>> I am not 100% sure whether this is CPUpools related, but I put some
load
>> on Dom0 (without messing with CPUpools) for the whole night and it ran
>> fine.
> 
> Did you try to do this with all Dom0-vcpus pinned to 6 physical cpus?
> I had the same problems when using only few physical cpus for many vcpus.
> And I''m pretty sure this was NOT the possible live-lock, as it
happened
> already without this change when I tried to reproduce your problem.
That is my current theory, too. I inserted counters in the try-loop for 
the locks to detect possible lock-ups, but they didn''t went over 99, so
this is not the reason.
The high overcommit (48 vCPUs on 6 pCPUs) is probably responsible. The 
new reduction of Dom0 vCPUs should avoid this situation in the future.
> 
>> Sorry for this :-(
>> I will try to further isolate this.
>>
>> Anyway, it works much better with the fix than without and I will try
to
>> trigger this with the "reduce number of Dom0 vCPUs" patch.
Unfortunately I got a Dom0 crash with the new patch. Reverting 22934 
worked fine. I will investigate this now.

root@dosorca:/data/images# xl cpupool-numa-split
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

Regards,
Andre.

-- 
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel