search for: _qcode_val_offset

...index (4 nodes) + * Bits 2-23: CPU number + 1 (4M - 1 CPUs) + * + * A queue node code of 0 indicates that no one is waiting for the lock. + * As the value 0 cannot be used as a valid CPU number. We need to add + * 1 to it before putting it into the queue code. + */ +#define MAX_QNODES 4 +#ifndef _QCODE_VAL_OFFSET +#define _QCODE_VAL_OFFSET _QCODE_OFFSET +#endif + +/* + * Function exit status + */ +enum exitval { + NORMAL_EXIT = 0, + NOTIFY_NEXT , /* Notify the next waiting node CPU */ + RELEASE_NODE /* Release current node directly */ +}; + +/* + * The queue node structure + * + * This structure is e...

...ndex (4 nodes) + * Bits 2-15: CPU number + 1 (16K - 1 CPUs) + * + * A queue node code of 0 indicates that no one is waiting for the lock. + * As the value 0 cannot be used as a valid CPU number. We need to add + * 1 to it before putting it into the queue code. + */ +#define MAX_QNODES 4 +#ifndef _QCODE_VAL_OFFSET +#define _QCODE_VAL_OFFSET _QCODE_OFFSET +#endif + +/* + * The queue node structure + * + * This structure is essentially the same as the mcs_spinlock structure + * in mcs_spinlock.h file. This structure is retained for future extension + * where new fields may be added. + */ +struct qnode { + u32...

...ndex (4 nodes) + * Bits 2-15: CPU number + 1 (16K - 1 CPUs) + * + * A queue node code of 0 indicates that no one is waiting for the lock. + * As the value 0 cannot be used as a valid CPU number. We need to add + * 1 to it before putting it into the queue code. + */ +#define MAX_QNODES 4 +#ifndef _QCODE_VAL_OFFSET +#define _QCODE_VAL_OFFSET _QCODE_OFFSET +#endif + +/* + * The queue node structure + * + * This structure is essentially the same as the mcs_spinlock structure + * in mcs_spinlock.h file. This structure is retained for future extension + * where new fields may be added. + */ +struct qnode { + u32...

v4->v5: - Move the optimized 2-task contending code to the generic file to enable more architectures to use it without code duplication. - Address some of the style-related comments by PeterZ. - Allow the use of unfair queue spinlock in a real para-virtualized execution environment. - Add para-virtualization support to the qspinlock code by ensuring that the lock holder and queue

v4->v5: - Move the optimized 2-task contending code to the generic file to enable more architectures to use it without code duplication. - Address some of the style-related comments by PeterZ. - Allow the use of unfair queue spinlock in a real para-virtualized execution environment. - Add para-virtualization support to the qspinlock code by ensuring that the lock holder and queue

...EAD_SPIN_THRESHOLD) && qcode) { + /* + * Get the CPU number of the lock holder & kick it + * The lock may have been stealed by another CPU + * if PARAVIRT_UNFAIR_LOCKS is set, so the computed + * CPU number may not be the actual lock holder. + */ + int cpu = (qcode >> (_QCODE_VAL_OFFSET + 2)) - 1; + __queue_kick_cpu(cpu, PV_KICK_LOCK_HOLDER); + } + if (next) { + struct pv_qvars *pv = (struct pv_qvars *) + ((char *)next + offset); + + if (!pv->prehead) + pv->prehead = true; + if ((*count & CLEAR_ACTIVE_MASK) == CLEAR_ACTIVE_MASK) + pv->active = false;...

v4->v5: - Move the optimized 2-task contending code to the generic file to enable more architectures to use it without code duplication. - Address some of the style-related comments by PeterZ. - Allow the use of unfair queue spinlock in a real para-virtualized execution environment. - Add para-virtualization support to the qspinlock code by ensuring that the lock holder and queue

v4->v5: - Move the optimized 2-task contending code to the generic file to enable more architectures to use it without code duplication. - Address some of the style-related comments by PeterZ. - Allow the use of unfair queue spinlock in a real para-virtualized execution environment. - Add para-virtualization support to the qspinlock code by ensuring that the lock holder and queue

v5->v6: - Change the optimized 2-task contending code to make it fairer at the expense of a bit of performance. - Add a patch to support unfair queue spinlock for Xen. - Modify the PV qspinlock code to follow what was done in the PV ticketlock. - Add performance data for the unfair lock as well as the PV support code. v4->v5: - Move the optimized 2-task contending code to the

v5->v6: - Change the optimized 2-task contending code to make it fairer at the expense of a bit of performance. - Add a patch to support unfair queue spinlock for Xen. - Modify the PV qspinlock code to follow what was done in the PV ticketlock. - Add performance data for the unfair lock as well as the PV support code. v4->v5: - Move the optimized 2-task contending code to the

v6->v7: - Remove an atomic operation from the 2-task contending code - Shorten the names of some macros - Make the queue waiter to attempt to steal lock when unfair lock is enabled. - Remove lock holder kick from the PV code and fix a race condition - Run the unfair lock & PV code on overcommitted KVM guests to collect performance data. v5->v6: - Change the optimized

v6->v7: - Remove an atomic operation from the 2-task contending code - Shorten the names of some macros - Make the queue waiter to attempt to steal lock when unfair lock is enabled. - Remove lock holder kick from the PV code and fix a race condition - Run the unfair lock & PV code on overcommitted KVM guests to collect performance data. v5->v6: - Change the optimized

v7->v8: - Remove one unneeded atomic operation from the slowpath, thus improving performance. - Simplify some of the codes and add more comments. - Test for X86_FEATURE_HYPERVISOR CPU feature bit to enable/disable unfair lock. - Reduce unfair lock slowpath lock stealing frequency depending on its distance from the queue head. - Add performance data for IvyBridge-EX CPU.

v7->v8: - Remove one unneeded atomic operation from the slowpath, thus improving performance. - Simplify some of the codes and add more comments. - Test for X86_FEATURE_HYPERVISOR CPU feature bit to enable/disable unfair lock. - Reduce unfair lock slowpath lock stealing frequency depending on its distance from the queue head. - Add performance data for IvyBridge-EX CPU.

...arch/x86/include/asm/qspinlock.h @@ -7,12 +7,30 @@ #define _ARCH_SUPPORTS_ATOMIC_8_16_BITS_OPS +#define smp_u8_store_release(p, v) \ +do { \ + barrier(); \ + ACCESS_ONCE(*p) = (v); \ +} while (0) + +/* + * As the qcode will be accessed as a 16-bit word, no offset is needed + */ +#define _QCODE_VAL_OFFSET 0 + /* * x86-64 specific queue spinlock union structure + * Besides the slock and lock fields, the other fields are only + * valid with less than 16K CPUs. */ union arch_qspinlock { struct qspinlock slock; - u8 lock; /* Lock bit */ + struct { + u8 lock; /* Lock bit */ + u8 wait; /* Wa...

...arch/x86/include/asm/qspinlock.h @@ -7,12 +7,30 @@ #define _ARCH_SUPPORTS_ATOMIC_8_16_BITS_OPS +#define smp_u8_store_release(p, v) \ +do { \ + barrier(); \ + ACCESS_ONCE(*p) = (v); \ +} while (0) + +/* + * As the qcode will be accessed as a 16-bit word, no offset is needed + */ +#define _QCODE_VAL_OFFSET 0 + /* * x86-64 specific queue spinlock union structure + * Besides the slock and lock fields, the other fields are only + * valid with less than 16K CPUs. */ union arch_qspinlock { struct qspinlock slock; - u8 lock; /* Lock bit */ + struct { + u8 lock; /* Lock bit */ + u8 wait; /* Wa...

N.B. Sorry for the duplicate. This patch series were resent as the original one was rejected by the vger.kernel.org list server due to long header. There is no change in content. v7->v8: - Remove one unneeded atomic operation from the slowpath, thus improving performance. - Simplify some of the codes and add more comments. - Test for X86_FEATURE_HYPERVISOR CPU feature bit

N.B. Sorry for the duplicate. This patch series were resent as the original one was rejected by the vger.kernel.org list server due to long header. There is no change in content. v7->v8: - Remove one unneeded atomic operation from the slowpath, thus improving performance. - Simplify some of the codes and add more comments. - Test for X86_FEATURE_HYPERVISOR CPU feature bit

...= QSPIN_THRESHOLD) && qcode)) { + /* + * Get the CPU number of the lock holder & kick it + * The lock may have been stealed by another CPU + * if PARAVIRT_UNFAIR_LOCKS is set, so the computed + * CPU number may not be the actual lock holder. + */ + int cpu = (qcode >> (_QCODE_VAL_OFFSET + 2)) - 1; + + ACCESS_ONCE(pv->cpustate) = PV_CPU_KICKING; + __queue_kick_cpu(cpu, PV_KICK_LOCK_HOLDER); + ACCESS_ONCE(pv->cpustate) = PV_CPU_ACTIVE; + } + if (unlikely(*count >= 2*QSPIN_THRESHOLD)) { + u8 lockval; + + /* + * Set the lock byte to _QSPINLOCK_LOCKED_SLOWPATH before +...