Waiman Long
2014-Feb-27 04:32 UTC
[PATCH v5 0/8] qspinlock: a 4-byte queue spinlock with PV support
v4->v5: - Move the optimized 2-task contending code to the generic file to enable more architectures to use it without code duplication. - Address some of the style-related comments by PeterZ. - Allow the use of unfair queue spinlock in a real para-virtualized execution environment. - Add para-virtualization support to the qspinlock code by ensuring that the lock holder and queue head stay alive as much as possible. v3->v4: - Remove debugging code and fix a configuration error - Simplify the qspinlock structure and streamline the code to make it perform a bit better - Add an x86 version of asm/qspinlock.h for holding x86 specific optimization. - Add an optimized x86 code path for 2 contending tasks to improve low contention performance. v2->v3: - Simplify the code by using numerous mode only without an unfair option. - Use the latest smp_load_acquire()/smp_store_release() barriers. - Move the queue spinlock code to kernel/locking. - Make the use of queue spinlock the default for x86-64 without user configuration. - Additional performance tuning. v1->v2: - Add some more comments to document what the code does. - Add a numerous CPU mode to support >= 16K CPUs - Add a configuration option to allow lock stealing which can further improve performance in many cases. - Enable wakeup of queue head CPU at unlock time for non-numerous CPU mode. This patch set has 3 different sections: 1) Patches 1-3: Introduces a queue-based spinlock implementation that can replace the default ticket spinlock without increasing the size of the spinlock data structure. As a result, critical kernel data structures that embed spinlock won't increase in size and breaking data alignments. 2) Patches 4 and 5: Enables the use of unfair queue spinlock in a real para-virtualized execution environment. This can resolve some of the locking related performance issues due to the fact that the next CPU to get the lock may have been scheduled out for a period of time. 3) Patches 6-8: Enable qspinlock para-virtualization support by making sure that the lock holder and the queue head stay alive as long as possible. Patches 1-3 are fully tested and ready for production. Patches 4-8, on the other hands, are not fully tested. They have undergone compilation tests with various combinations of kernel config settings, boot-up tests in a bare-metal as well as simple performance test on a KVM guest. Further tests and performance characterization are still needed to be done. So comments on them are welcomed. Suggestions or recommendations on how to add PV support in the Xen environment are also needed. The queue spinlock has slightly better performance than the ticket spinlock in uncontended case. Its performance can be much better with moderate to heavy contention. This patch has the potential of improving the performance of all the workloads that have moderate to heavy spinlock contention. The queue spinlock is especially suitable for NUMA machines with at least 2 sockets, though noticeable performance benefit probably won't show up in machines with less than 4 sockets. The purpose of this patch set is not to solve any particular spinlock contention problems. Those need to be solved by refactoring the code to make more efficient use of the lock or finer granularity ones. The main purpose is to make the lock contention problems more tolerable until someone can spend the time and effort to fix them. The performance data in bare metal were discussed in the patch descriptions. For PV support, some simple performance test was performed on a 2-node 20-CPU KVM guest running 3.14-rc4 kernel in a larger 8-node machine. The disk workload of the AIM7 benchmark was run on both ext4 and xfs RAM disks at 2000 users. The JPM (jobs/minute) data of the test run were: kernel XFS FS %change ext4 FS %change ------ ------ ------- ------- ------- PV ticketlock (baseline) 2390438 - 1366743 - qspinlock 1775148 -26% 1336303 -2.2% PV qspinlock 2264151 -5.3% 1351351 -1.1% unfair qspinlock 2404810 +0.6% 1612903 +18% unfair + PV qspinlock 2419355 +1.2% 1612903 +18% The XFS test had moderate spinlock contention of 1.6% whereas the ext4 test had heavy spinlock contention of 15.4% as reported by perf. It seems like the PV qspinlock support still has room for improvement compared with the current PV ticketlock implementation. Waiman Long (8): qspinlock: Introducing a 4-byte queue spinlock implementation qspinlock, x86: Enable x86-64 to use queue spinlock qspinlock, x86: Add x86 specific optimization for 2 contending tasks pvqspinlock, x86: Allow unfair spinlock in a real PV environment pvqspinlock, x86: Enable unfair queue spinlock in a KVM guest pvqspinlock, x86: Rename paravirt_ticketlocks_enabled pvqspinlock, x86: Add qspinlock para-virtualization support pvqspinlock, x86: Enable KVM to use qspinlock's PV support arch/x86/Kconfig | 12 + arch/x86/include/asm/paravirt.h | 9 +- arch/x86/include/asm/paravirt_types.h | 12 + arch/x86/include/asm/pvqspinlock.h | 176 ++++++++++ arch/x86/include/asm/qspinlock.h | 133 +++++++ arch/x86/include/asm/spinlock.h | 9 +- arch/x86/include/asm/spinlock_types.h | 4 + arch/x86/kernel/Makefile | 1 + arch/x86/kernel/kvm.c | 73 ++++- arch/x86/kernel/paravirt-spinlocks.c | 15 +- arch/x86/xen/spinlock.c | 2 +- include/asm-generic/qspinlock.h | 122 +++++++ include/asm-generic/qspinlock_types.h | 61 ++++ kernel/Kconfig.locks | 7 + kernel/locking/Makefile | 1 + kernel/locking/qspinlock.c | 610 +++++++++++++++++++++++++++++++++ 16 files changed, 1239 insertions(+), 8 deletions(-) create mode 100644 arch/x86/include/asm/pvqspinlock.h create mode 100644 arch/x86/include/asm/qspinlock.h create mode 100644 include/asm-generic/qspinlock.h create mode 100644 include/asm-generic/qspinlock_types.h create mode 100644 kernel/locking/qspinlock.c
Waiman Long
2014-Feb-27 04:32 UTC
[PATCH v5 1/8] qspinlock: Introducing a 4-byte queue spinlock implementation
This patch introduces a new queue spinlock implementation that can serve as an alternative to the default ticket spinlock. Compared with the ticket spinlock, this queue spinlock should be almost as fair as the ticket spinlock. It has about the same speed in single-thread and it can be much faster in high contention situations. Only in light to moderate contention where the average queue depth is around 1-3 will this queue spinlock be potentially a bit slower due to the higher slowpath overhead. This queue spinlock is especially suit to NUMA machines with a large number of cores as the chance of spinlock contention is much higher in those machines. The cost of contention is also higher because of slower inter-node memory traffic. The idea behind this spinlock implementation is the fact that spinlocks are acquired with preemption disabled. In other words, the process will not be migrated to another CPU while it is trying to get a spinlock. Ignoring interrupt handling, a CPU can only be contending in one spinlock at any one time. Of course, interrupt handler can try to acquire one spinlock while the interrupted user process is in the process of getting another spinlock. By allocating a set of per-cpu queue nodes and used them to form a waiting queue, we can encode the queue node address into a much smaller 16-bit size. Together with the 1-byte lock bit, this queue spinlock implementation will only need 4 bytes to hold all the information that it needs. The current queue node address encoding of the 4-byte word is as follows: Bits 0-7 : the locked byte Bits 8-9 : queue node index in the per-cpu array (4 entries) Bits 10-31: cpu number + 1 (max cpus = 4M -1) In the extremely unlikely case that all the queue node entries are used up, the current code will fall back to busy spinning without waiting in a queue with warning message. For single-thread performance (no contention), a 256K lock/unlock loop was run on a 2.4Ghz Westmere x86-64 CPU. The following table shows the average time (in ns) for a single lock/unlock sequence (including the looping and timing overhead): Lock Type Time (ns) --------- --------- Ticket spinlock 14.1 Queue spinlock 8.8 So the queue spinlock is much faster than the ticket spinlock, even though the overhead of locking and unlocking should be pretty small when there is no contention. The performance advantage is mainly due to the fact that ticket spinlock does a read-modify-write (add) instruction in unlock whereas queue spinlock only does a simple write in unlock which can be much faster in a pipelined CPU. The AIM7 benchmark was run on a 8-socket 80-core DL980 with Westmere x86-64 CPUs with XFS filesystem on a ramdisk and HT off to evaluate the performance impact of this patch on a 3.13 kernel. +------------+----------+-----------------+---------+ | Kernel | 3.13 JPM | 3.13 with | %Change | | | | qspinlock patch | | +------------+----------+-----------------+---------+ | 10-100 users | +------------+----------+-----------------+---------+ |custom | 357459 | 363109 | +1.58% | |dbase | 496847 | 498801 | +0.39% | |disk | 2925312 | 2771387 | -5.26% | |five_sec | 166612 | 169215 | +1.56% | |fserver | 382129 | 383279 | +0.30% | |high_systime| 16356 | 16380 | +0.15% | |short | 4521978 | 4257363 | -5.85% | +------------+----------+-----------------+---------+ | 200-1000 users | +------------+----------+-----------------+---------+ |custom | 449070 | 447711 | -0.30% | |dbase | 845029 | 853362 | +0.99% | |disk | 2725249 | 4892907 | +79.54% | |five_sec | 169410 | 170638 | +0.72% | |fserver | 489662 | 491828 | +0.44% | |high_systime| 142823 | 143790 | +0.68% | |short | 7435288 | 9016171 | +21.26% | +------------+----------+-----------------+---------+ | 1100-2000 users | +------------+----------+-----------------+---------+ |custom | 432470 | 432570 | +0.02% | |dbase | 889289 | 890026 | +0.08% | |disk | 2565138 | 5008732 | +95.26% | |five_sec | 169141 | 170034 | +0.53% | |fserver | 498569 | 500701 | +0.43% | |high_systime| 229913 | 245866 | +6.94% | |short | 8496794 | 8281918 | -2.53% | +------------+----------+-----------------+---------+ The workload with the most gain was the disk workload. Without the patch, the perf profile at 1500 users looked like: 26.19% reaim [kernel.kallsyms] [k] _raw_spin_lock |--47.28%-- evict |--46.87%-- inode_sb_list_add |--1.24%-- xlog_cil_insert_items |--0.68%-- __remove_inode_hash |--0.67%-- inode_wait_for_writeback --3.26%-- [...] 22.96% swapper [kernel.kallsyms] [k] cpu_idle_loop 5.56% reaim [kernel.kallsyms] [k] mutex_spin_on_owner 4.87% reaim [kernel.kallsyms] [k] update_cfs_rq_blocked_load 2.04% reaim [kernel.kallsyms] [k] mspin_lock 1.30% reaim [kernel.kallsyms] [k] memcpy 1.08% reaim [unknown] [.] 0x0000003c52009447 There was pretty high spinlock contention on the inode_sb_list_lock and maybe the inode's i_lock. With the patch, the perf profile at 1500 users became: 26.82% swapper [kernel.kallsyms] [k] cpu_idle_loop 4.66% reaim [kernel.kallsyms] [k] mutex_spin_on_owner 3.97% reaim [kernel.kallsyms] [k] update_cfs_rq_blocked_load 2.40% reaim [kernel.kallsyms] [k] queue_spin_lock_slowpath |--88.31%-- _raw_spin_lock | |--36.02%-- inode_sb_list_add | |--35.09%-- evict | |--16.89%-- xlog_cil_insert_items | |--6.30%-- try_to_wake_up | |--2.20%-- _xfs_buf_find | |--0.75%-- __remove_inode_hash | |--0.72%-- __mutex_lock_slowpath | |--0.53%-- load_balance |--6.02%-- _raw_spin_lock_irqsave | |--74.75%-- down_trylock | |--9.69%-- rcu_check_quiescent_state | |--7.47%-- down | |--3.57%-- up | |--1.67%-- rwsem_wake | |--1.00%-- remove_wait_queue | |--0.56%-- pagevec_lru_move_fn |--5.39%-- _raw_spin_lock_irq | |--82.05%-- rwsem_down_read_failed | |--10.48%-- rwsem_down_write_failed | |--4.24%-- __down | |--2.74%-- __schedule --0.28%-- [...] 2.20% reaim [kernel.kallsyms] [k] memcpy 1.84% reaim [unknown] [.] 0x000000000041517b 1.77% reaim [kernel.kallsyms] [k] _raw_spin_lock |--21.08%-- xlog_cil_insert_items |--10.14%-- xfs_icsb_modify_counters |--7.20%-- xfs_iget_cache_hit |--6.56%-- inode_sb_list_add |--5.49%-- _xfs_buf_find |--5.25%-- evict |--5.03%-- __remove_inode_hash |--4.64%-- __mutex_lock_slowpath |--3.78%-- selinux_inode_free_security |--2.95%-- xfs_inode_is_filestream |--2.35%-- try_to_wake_up |--2.07%-- xfs_inode_set_reclaim_tag |--1.52%-- list_lru_add |--1.16%-- xfs_inode_clear_eofblocks_tag : 1.30% reaim [kernel.kallsyms] [k] effective_load 1.27% reaim [kernel.kallsyms] [k] mspin_lock 1.10% reaim [kernel.kallsyms] [k] security_compute_sid On the ext4 filesystem, the disk workload improved from 416281 JPM to 899101 JPM (+116%) with the patch. In this case, the contended spinlock is the mb_cache_spinlock. Signed-off-by: Waiman Long <Waiman.Long at hp.com> Acked-by: Rik van Riel <riel at redhat.com> --- include/asm-generic/qspinlock.h | 122 ++++++++++ include/asm-generic/qspinlock_types.h | 55 +++++ kernel/Kconfig.locks | 7 + kernel/locking/Makefile | 1 + kernel/locking/qspinlock.c | 393 +++++++++++++++++++++++++++++++++ 5 files changed, 578 insertions(+), 0 deletions(-) create mode 100644 include/asm-generic/qspinlock.h create mode 100644 include/asm-generic/qspinlock_types.h create mode 100644 kernel/locking/qspinlock.c diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinlock.h new file mode 100644 index 0000000..08da60f --- /dev/null +++ b/include/asm-generic/qspinlock.h @@ -0,0 +1,122 @@ +/* + * Queue spinlock + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * (C) Copyright 2013-2014 Hewlett-Packard Development Company, L.P. + * + * Authors: Waiman Long <waiman.long at hp.com> + */ +#ifndef __ASM_GENERIC_QSPINLOCK_H +#define __ASM_GENERIC_QSPINLOCK_H + +#include <asm-generic/qspinlock_types.h> + +/* + * External function declarations + */ +extern void queue_spin_lock_slowpath(struct qspinlock *lock, int qsval); + +/** + * queue_spin_is_locked - is the spinlock locked? + * @lock: Pointer to queue spinlock structure + * Return: 1 if it is locked, 0 otherwise + */ +static __always_inline int queue_spin_is_locked(struct qspinlock *lock) +{ + return atomic_read(&lock->qlcode) & _QSPINLOCK_LOCKED; +} + +/** + * queue_spin_value_unlocked - is the spinlock structure unlocked? + * @lock: queue spinlock structure + * Return: 1 if it is unlocked, 0 otherwise + */ +static __always_inline int queue_spin_value_unlocked(struct qspinlock lock) +{ + return !(atomic_read(&lock.qlcode) & _QSPINLOCK_LOCKED); +} + +/** + * queue_spin_is_contended - check if the lock is contended + * @lock : Pointer to queue spinlock structure + * Return: 1 if lock contended, 0 otherwise + */ +static __always_inline int queue_spin_is_contended(struct qspinlock *lock) +{ + return atomic_read(&lock->qlcode) & ~_QSPINLOCK_LOCK_MASK; +} +/** + * queue_spin_trylock - try to acquire the queue spinlock + * @lock : Pointer to queue spinlock structure + * Return: 1 if lock acquired, 0 if failed + */ +static __always_inline int queue_spin_trylock(struct qspinlock *lock) +{ + if (!atomic_read(&lock->qlcode) && + (atomic_cmpxchg(&lock->qlcode, 0, _QSPINLOCK_LOCKED) == 0)) + return 1; + return 0; +} + +/** + * queue_spin_lock - acquire a queue spinlock + * @lock: Pointer to queue spinlock structure + */ +static __always_inline void queue_spin_lock(struct qspinlock *lock) +{ + int qsval; + + /* + * To reduce memory access to only once for the cold cache case, + * a direct cmpxchg() is performed in the fastpath to optimize the + * uncontended case. The contended performance, however, may suffer + * a bit because of that. + */ + qsval = atomic_cmpxchg(&lock->qlcode, 0, _QSPINLOCK_LOCKED); + if (likely(qsval == 0)) + return; + queue_spin_lock_slowpath(lock, qsval); +} + +#ifndef queue_spin_unlock +/** + * queue_spin_unlock - release a queue spinlock + * @lock : Pointer to queue spinlock structure + */ +static __always_inline void queue_spin_unlock(struct qspinlock *lock) +{ + /* + * Use an atomic subtraction to clear the lock bit. + */ + smp_mb__before_atomic_dec(); + atomic_sub(_QSPINLOCK_LOCKED, &lock->qlcode); +} +#endif + +/* + * Initializier + */ +#define __ARCH_SPIN_LOCK_UNLOCKED { ATOMIC_INIT(0) } + +/* + * Remapping spinlock architecture specific functions to the corresponding + * queue spinlock functions. + */ +#define arch_spin_is_locked(l) queue_spin_is_locked(l) +#define arch_spin_is_contended(l) queue_spin_is_contended(l) +#define arch_spin_value_unlocked(l) queue_spin_value_unlocked(l) +#define arch_spin_lock(l) queue_spin_lock(l) +#define arch_spin_trylock(l) queue_spin_trylock(l) +#define arch_spin_unlock(l) queue_spin_unlock(l) +#define arch_spin_lock_flags(l, f) queue_spin_lock(l) + +#endif /* __ASM_GENERIC_QSPINLOCK_H */ diff --git a/include/asm-generic/qspinlock_types.h b/include/asm-generic/qspinlock_types.h new file mode 100644 index 0000000..df981d0 --- /dev/null +++ b/include/asm-generic/qspinlock_types.h @@ -0,0 +1,55 @@ +/* + * Queue spinlock + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * (C) Copyright 2013-2014 Hewlett-Packard Development Company, L.P. + * + * Authors: Waiman Long <waiman.long at hp.com> + */ +#ifndef __ASM_GENERIC_QSPINLOCK_TYPES_H +#define __ASM_GENERIC_QSPINLOCK_TYPES_H + +/* + * Including atomic.h with PARAVIRT on will cause compilation errors because + * of recursive header file incluson via paravirt_types.h. A workaround is + * to include paravirt_types.h here in this case. + */ +#ifdef CONFIG_PARAVIRT +# include <asm/paravirt_types.h> +#else +# include <linux/types.h> +# include <linux/atomic.h> +#endif + +/* + * The queue spinlock data structure - a 32-bit word + * + * For NR_CPUS >= 16K, the bits assignment are: + * Bit 0 : Set if locked + * Bits 1-7 : Not used + * Bits 8-31: Queue code + * + * For NR_CPUS < 16K, the bits assignment are: + * Bit 0 : Set if locked + * Bits 1-7 : Not used + * Bits 8-15: Reserved for architecture specific optimization + * Bits 16-31: Queue code + */ +typedef struct qspinlock { + atomic_t qlcode; /* Lock + queue code */ +} arch_spinlock_t; + +#define _QCODE_OFFSET 8 +#define _QSPINLOCK_LOCKED 1U +#define _QSPINLOCK_LOCK_MASK 0xff + +#endif /* __ASM_GENERIC_QSPINLOCK_TYPES_H */ diff --git a/kernel/Kconfig.locks b/kernel/Kconfig.locks index d2b32ac..f185584 100644 --- a/kernel/Kconfig.locks +++ b/kernel/Kconfig.locks @@ -223,3 +223,10 @@ endif config MUTEX_SPIN_ON_OWNER def_bool y depends on SMP && !DEBUG_MUTEXES + +config ARCH_USE_QUEUE_SPINLOCK + bool + +config QUEUE_SPINLOCK + def_bool y if ARCH_USE_QUEUE_SPINLOCK + depends on SMP && !PARAVIRT_SPINLOCKS diff --git a/kernel/locking/Makefile b/kernel/locking/Makefile index baab8e5..e3b3293 100644 --- a/kernel/locking/Makefile +++ b/kernel/locking/Makefile @@ -15,6 +15,7 @@ obj-$(CONFIG_LOCKDEP) += lockdep_proc.o endif obj-$(CONFIG_SMP) += spinlock.o obj-$(CONFIG_PROVE_LOCKING) += spinlock.o +obj-$(CONFIG_QUEUE_SPINLOCK) += qspinlock.o obj-$(CONFIG_RT_MUTEXES) += rtmutex.o obj-$(CONFIG_DEBUG_RT_MUTEXES) += rtmutex-debug.o obj-$(CONFIG_RT_MUTEX_TESTER) += rtmutex-tester.o diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c new file mode 100644 index 0000000..ed5efa7 --- /dev/null +++ b/kernel/locking/qspinlock.c @@ -0,0 +1,393 @@ +/* + * Queue spinlock + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * (C) Copyright 2013-2014 Hewlett-Packard Development Company, L.P. + * + * Authors: Waiman Long <waiman.long at hp.com> + */ +#include <linux/smp.h> +#include <linux/bug.h> +#include <linux/cpumask.h> +#include <linux/percpu.h> +#include <linux/hardirq.h> +#include <linux/mutex.h> +#include <linux/spinlock.h> + +/* + * The basic principle of a queue-based spinlock can best be understood + * by studying a classic queue-based spinlock implementation called the + * MCS lock. The paper below provides a good description for this kind + * of lock. + * + * cise.ufl.edu/tr/DOC/REP-1992-71.pdf + * + * This queue spinlock implementation is based on the MCS lock with twists + * to make it fit the following constraints: + * 1. A max spinlock size of 4 bytes + * 2. Good fastpath performance + * 3. No change in the locking APIs + * + * The queue spinlock fastpath is as simple as it can get, all the heavy + * lifting is done in the lock slowpath. The main idea behind this queue + * spinlock implementation is to keep the spinlock size at 4 bytes while + * at the same time implement a queue structure to queue up the waiting + * lock spinners. + * + * Since preemption is disabled before getting the lock, a given CPU will + * only need to use one queue node structure in a non-interrupt context. + * A percpu queue node structure will be allocated for this purpose and the + * cpu number will be put into the queue spinlock structure to indicate the + * tail of the queue. + * + * To handle spinlock acquisition at interrupt context (softirq or hardirq), + * the queue node structure is actually an array for supporting nested spin + * locking operations in interrupt handlers. If all the entries in the + * array are used up, a warning message will be printed (as that shouldn't + * happen in normal circumstances) and the lock spinner will fall back to + * busy spinning instead of waiting in a queue. + */ + +/* + * The 24-bit queue node code is divided into the following 2 fields: + * Bits 0-1 : queue node index (4 nodes) + * Bits 2-23: CPU number + 1 (4M - 1 CPUs) + * + * The 16-bit queue node code is divided into the following 2 fields: + * Bits 0-1 : queue node index (4 nodes) + * Bits 2-15: CPU number + 1 (16K - 1 CPUs) + * + * A queue node code of 0 indicates that no one is waiting for the lock. + * As the value 0 cannot be used as a valid CPU number. We need to add + * 1 to it before putting it into the queue code. + */ +#define MAX_QNODES 4 +#ifndef _QCODE_VAL_OFFSET +#define _QCODE_VAL_OFFSET _QCODE_OFFSET +#endif + +/* + * The queue node structure + * + * This structure is essentially the same as the mcs_spinlock structure + * in mcs_spinlock.h file. This structure is retained for future extension + * where new fields may be added. + */ +struct qnode { + u32 wait; /* Waiting flag */ + struct qnode *next; /* Next queue node addr */ +}; + +struct qnode_set { + struct qnode nodes[MAX_QNODES]; + int node_idx; /* Current node to use */ +}; + +/* + * Per-CPU queue node structures + */ +static DEFINE_PER_CPU_ALIGNED(struct qnode_set, qnset) = { {{0}}, 0 }; + +/* + ************************************************************************ + * The following optimized codes are for architectures that support: * + * 1) Atomic byte and short data write * + * 2) Byte and short data exchange and compare-exchange instructions * + * * + * For those architectures, their asm/qspinlock.h header file should * + * define the followings in order to use the optimized codes. * + * 1) The _ARCH_SUPPORTS_ATOMIC_8_16_BITS_OPS macro * + * 2) A smp_u8_store_release() macro for byte size store operation * + * 3) A "union arch_qspinlock" structure that include the individual * + * fields of the qspinlock structure, including: * + * o slock - the qspinlock structure * + * o lock - the lock byte * + * * + ************************************************************************ + */ +#ifdef _ARCH_SUPPORTS_ATOMIC_8_16_BITS_OPS +/** + * queue_spin_setlock - try to acquire the lock by setting the lock bit + * @lock: Pointer to queue spinlock structure + * Return: 1 if lock bit set successfully, 0 if failed + */ +static __always_inline int queue_spin_setlock(struct qspinlock *lock) +{ + union arch_qspinlock *qlock = (union arch_qspinlock *)lock; + + if (!ACCESS_ONCE(qlock->lock) && + (cmpxchg(&qlock->lock, 0, _QSPINLOCK_LOCKED) == 0)) + return 1; + return 0; +} +#else /* _ARCH_SUPPORTS_ATOMIC_8_16_BITS_OPS */ +/* + * Generic functions for architectures that do not support atomic + * byte or short data types. + */ +/** + *_queue_spin_setlock - try to acquire the lock by setting the lock bit + * @lock: Pointer to queue spinlock structure + * Return: 1 if lock bit set successfully, 0 if failed + */ +static __always_inline int queue_spin_setlock(struct qspinlock *lock) +{ + int qlcode = atomic_read(lock->qlcode); + + if (!(qlcode & _QSPINLOCK_LOCKED) && (atomic_cmpxchg(&lock->qlcode, + qlcode, qlcode|_QSPINLOCK_LOCKED) == qlcode)) + return 1; + return 0; +} +#endif /* _ARCH_SUPPORTS_ATOMIC_8_16_BITS_OPS */ + +/* + ************************************************************************ + * Inline functions used by the queue_spin_lock_slowpath() function * + * that may get superseded by a more optimized version. * + ************************************************************************ + */ + +#ifndef queue_get_lock_qcode +/** + * queue_get_lock_qcode - get the lock & qcode values + * @lock : Pointer to queue spinlock structure + * @qcode : Pointer to the returned qcode value + * @mycode: My qcode value (not used) + * Return : > 0 if lock is not available, = 0 if lock is free + */ +static inline int +queue_get_lock_qcode(struct qspinlock *lock, u32 *qcode, u32 mycode) +{ + int qlcode = atomic_read(&lock->qlcode); + + *qcode = qlcode; + return qlcode & _QSPINLOCK_LOCKED; +} +#endif /* queue_get_lock_qcode */ + +#ifndef queue_spin_trylock_and_clr_qcode +/** + * queue_spin_trylock_and_clr_qcode - Try to lock & clear qcode simultaneously + * @lock : Pointer to queue spinlock structure + * @qcode: The supposedly current qcode value + * Return: true if successful, false otherwise + */ +static inline int +queue_spin_trylock_and_clr_qcode(struct qspinlock *lock, u32 qcode) +{ + return atomic_cmpxchg(&lock->qlcode, qcode, _QSPINLOCK_LOCKED) == qcode; +} +#endif /* queue_spin_trylock_and_clr_qcode */ + +#ifndef queue_encode_qcode +/** + * queue_encode_qcode - Encode the CPU number & node index into a qnode code + * @cpu_nr: CPU number + * @qn_idx: Queue node index + * Return : A qnode code that can be saved into the qspinlock structure + * + * The lock bit is set in the encoded 32-bit value as the need to encode + * a qnode means that the lock should have been taken. + */ +static u32 queue_encode_qcode(u32 cpu_nr, u8 qn_idx) +{ + return ((cpu_nr + 1) << (_QCODE_VAL_OFFSET + 2)) | + (qn_idx << _QCODE_VAL_OFFSET) | _QSPINLOCK_LOCKED; +} +#endif /* queue_encode_qcode */ + +/* + ************************************************************************ + * Other inline functions needed by the queue_spin_lock_slowpath() * + * function. * + ************************************************************************ + */ + +/** + * xlate_qcode - translate the queue code into the queue node address + * @qcode: Queue code to be translated + * Return: The corresponding queue node address + */ +static inline struct qnode *xlate_qcode(u32 qcode) +{ + u32 cpu_nr = (qcode >> (_QCODE_VAL_OFFSET + 2)) - 1; + u8 qn_idx = (qcode >> _QCODE_VAL_OFFSET) & 3; + + return per_cpu_ptr(&qnset.nodes[qn_idx], cpu_nr); +} + +/** + * get_qnode - Get a queue node address + * @qn_idx: Pointer to queue node index [out] + * Return : queue node address & queue node index in qn_idx, or NULL if + * no free queue node available. + */ +static struct qnode *get_qnode(unsigned int *qn_idx) +{ + struct qnode_set *qset = this_cpu_ptr(&qnset); + int i; + + if (unlikely(qset->node_idx >= MAX_QNODES)) + return NULL; + i = qset->node_idx++; + *qn_idx = i; + return &qset->nodes[i]; +} + +/** + * put_qnode - Return a queue node to the pool + */ +static void put_qnode(void) +{ + struct qnode_set *qset = this_cpu_ptr(&qnset); + + qset->node_idx--; +} + +/** + * queue_spin_lock_slowpath - acquire the queue spinlock + * @lock : Pointer to queue spinlock structure + * @qsval: Current value of the queue spinlock 32-bit word + */ +void queue_spin_lock_slowpath(struct qspinlock *lock, int qsval) +{ + unsigned int cpu_nr, qn_idx; + struct qnode *node, *next; + u32 prev_qcode, my_qcode; + + /* + * Get the queue node + */ + cpu_nr = smp_processor_id(); + node = get_qnode(&qn_idx); + + /* + * It should never happen that all the queue nodes are being used. + */ + BUG_ON(!node); + + /* + * Set up the new cpu code to be exchanged + */ + my_qcode = queue_encode_qcode(cpu_nr, qn_idx); + + /* + * Initialize the queue node + */ + node->wait = true; + node->next = NULL; + + /* + * The lock may be available at this point, try again if no task was + * waiting in the queue. + */ + if (!(qsval >> _QCODE_OFFSET) && queue_spin_trylock(lock)) { + put_qnode(); + return; + } + + /* + * Exchange current copy of the queue node code + */ + prev_qcode = atomic_xchg(&lock->qlcode, my_qcode); + /* + * It is possible that we may accidentally steal the lock. If this is + * the case, we need to either release it if not the head of the queue + * or get the lock and be done with it. + */ + if (unlikely(!(prev_qcode & _QSPINLOCK_LOCKED))) { + if (prev_qcode == 0) { + /* + * Got the lock since it is at the head of the queue + * Now try to atomically clear the queue code. + */ + if (atomic_cmpxchg(&lock->qlcode, my_qcode, + _QSPINLOCK_LOCKED) == my_qcode) + goto release_node; + /* + * The cmpxchg fails only if one or more tasks + * are added to the queue. In this case, we need to + * notify the next one to be the head of the queue. + */ + goto notify_next; + } + /* + * Accidentally steal the lock, release the lock and + * let the queue head get it. + */ + queue_spin_unlock(lock); + } else + prev_qcode &= ~_QSPINLOCK_LOCKED; /* Clear the lock bit */ + my_qcode &= ~_QSPINLOCK_LOCKED; + + if (prev_qcode) { + /* + * Not at the queue head, get the address of the previous node + * and set up the "next" fields of the that node. + */ + struct qnode *prev = xlate_qcode(prev_qcode); + + ACCESS_ONCE(prev->next) = node; + /* + * Wait until the waiting flag is off + */ + while (smp_load_acquire(&node->wait)) + arch_mutex_cpu_relax(); + } + + /* + * At the head of the wait queue now + */ + while (true) { + u32 qcode; + int retval; + + retval = queue_get_lock_qcode(lock, &qcode, my_qcode); + if (retval > 0) + ; /* Lock not available yet */ + else if (retval < 0) + /* Lock taken, can release the node & return */ + goto release_node; + else if (qcode != my_qcode) { + /* + * Just get the lock with other spinners waiting + * in the queue. + */ + if (queue_spin_setlock(lock)) + goto notify_next; + } else { + /* + * Get the lock & clear the queue code simultaneously + */ + if (queue_spin_trylock_and_clr_qcode(lock, qcode)) + /* No need to notify the next one */ + goto release_node; + } + arch_mutex_cpu_relax(); + } + +notify_next: + /* + * Wait, if needed, until the next one in queue set up the next field + */ + while (!(next = ACCESS_ONCE(node->next))) + arch_mutex_cpu_relax(); + /* + * The next one in queue is now at the head + */ + smp_store_release(&next->wait, false); + +release_node: + put_qnode(); +} +EXPORT_SYMBOL(queue_spin_lock_slowpath); -- 1.7.1
Waiman Long
2014-Feb-27 04:32 UTC
[PATCH v5 2/8] qspinlock, x86: Enable x86-64 to use queue spinlock
This patch makes the necessary changes at the x86 architecture specific layer to enable the use of queue spinlock for x86-64. As x86-32 machines are typically not multi-socket. The benefit of queue spinlock may not be apparent. So queue spinlock is not enabled. Currently, there is some incompatibilities between the para-virtualized spinlock code (which hard-codes the use of ticket spinlock) and the queue spinlock. Therefore, the use of queue spinlock is disabled when the para-virtualized spinlock is enabled. The arch/x86/include/asm/qspinlock.h header file includes some x86 specific optimization which will make the queue spinlock code perform better than the generic implementation. Signed-off-by: Waiman Long <Waiman.Long at hp.com> Acked-by: Rik van Riel <riel at redhat.com> --- arch/x86/Kconfig | 1 + arch/x86/include/asm/qspinlock.h | 41 +++++++++++++++++++++++++++++++++ arch/x86/include/asm/spinlock.h | 5 ++++ arch/x86/include/asm/spinlock_types.h | 4 +++ 4 files changed, 51 insertions(+), 0 deletions(-) create mode 100644 arch/x86/include/asm/qspinlock.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 1b4ff87..5bf70ab 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -17,6 +17,7 @@ config X86_64 depends on 64BIT select X86_DEV_DMA_OPS select ARCH_USE_CMPXCHG_LOCKREF + select ARCH_USE_QUEUE_SPINLOCK ### Arch settings config X86 diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h new file mode 100644 index 0000000..44cefee --- /dev/null +++ b/arch/x86/include/asm/qspinlock.h @@ -0,0 +1,41 @@ +#ifndef _ASM_X86_QSPINLOCK_H +#define _ASM_X86_QSPINLOCK_H + +#include <asm-generic/qspinlock_types.h> + +#if !defined(CONFIG_X86_OOSTORE) && !defined(CONFIG_X86_PPRO_FENCE) + +#define _ARCH_SUPPORTS_ATOMIC_8_16_BITS_OPS + +/* + * x86-64 specific queue spinlock union structure + */ +union arch_qspinlock { + struct qspinlock slock; + u8 lock; /* Lock bit */ +}; + +#define queue_spin_unlock queue_spin_unlock +/** + * queue_spin_unlock - release a queue spinlock + * @lock : Pointer to queue spinlock structure + * + * No special memory barrier other than a compiler one is needed for the + * x86 architecture. A compiler barrier is added at the end to make sure + * that the clearing the lock bit is done ASAP without artificial delay + * due to compiler optimization. + */ +static inline void queue_spin_unlock(struct qspinlock *lock) +{ + union arch_qspinlock *qlock = (union arch_qspinlock *)lock; + + barrier(); + ACCESS_ONCE(qlock->lock) = 0; + barrier(); +} + +#endif /* !CONFIG_X86_OOSTORE && !CONFIG_X86_PPRO_FENCE */ + +#include <asm-generic/qspinlock.h> + +#endif /* _ASM_X86_QSPINLOCK_H */ diff --git a/arch/x86/include/asm/spinlock.h b/arch/x86/include/asm/spinlock.h index bf156de..6e6de1f 100644 --- a/arch/x86/include/asm/spinlock.h +++ b/arch/x86/include/asm/spinlock.h @@ -43,6 +43,10 @@ extern struct static_key paravirt_ticketlocks_enabled; static __always_inline bool static_key_false(struct static_key *key); +#ifdef CONFIG_QUEUE_SPINLOCK +#include <asm/qspinlock.h> +#else + #ifdef CONFIG_PARAVIRT_SPINLOCKS static inline void __ticket_enter_slowpath(arch_spinlock_t *lock) @@ -181,6 +185,7 @@ static __always_inline void arch_spin_lock_flags(arch_spinlock_t *lock, { arch_spin_lock(lock); } +#endif /* CONFIG_QUEUE_SPINLOCK */ static inline void arch_spin_unlock_wait(arch_spinlock_t *lock) { diff --git a/arch/x86/include/asm/spinlock_types.h b/arch/x86/include/asm/spinlock_types.h index 4f1bea1..7960268 100644 --- a/arch/x86/include/asm/spinlock_types.h +++ b/arch/x86/include/asm/spinlock_types.h @@ -23,6 +23,9 @@ typedef u32 __ticketpair_t; #define TICKET_SHIFT (sizeof(__ticket_t) * 8) +#ifdef CONFIG_QUEUE_SPINLOCK +#include <asm-generic/qspinlock_types.h> +#else typedef struct arch_spinlock { union { __ticketpair_t head_tail; @@ -33,6 +36,7 @@ typedef struct arch_spinlock { } arch_spinlock_t; #define __ARCH_SPIN_LOCK_UNLOCKED { { 0 } } +#endif /* CONFIG_QUEUE_SPINLOCK */ #include <asm/rwlock.h> -- 1.7.1
Waiman Long
2014-Feb-27 04:32 UTC
[PATCH v5 3/8] qspinlock, x86: Add x86 specific optimization for 2 contending tasks
A major problem with the queue spinlock patch is its performance at low contention level (2-4 contending tasks) where it is slower than the corresponding ticket spinlock code path. The following table shows the execution time (in ms) of a micro-benchmark where 5M iterations of the lock/unlock cycles were run on a 10-core Westere-EX CPU with 2 different types loads - standalone (lock and protected data in different cachelines) and embedded (lock and protected data in the same cacheline). [Standalone/Embedded] # of tasks Ticket lock Queue lock %Change ---------- ----------- ---------- ------- 1 135/111 135/102 0%/-8% 2 732/950 1315/1573 +80%/+66% 3 1827/1783 2372/2428 +30%/+36% 4 2689/2725 2934/2934 +9%/+8% 5 3736/3748 3658/3652 -2%/-3% 6 4942/4984 4434/4428 -10%/-11% 7 6304/6319 5176/5163 -18%/-18% 8 7736/7629 5955/5944 -23%/-22% It can be seen that the performance degradation is particular bad with 2 and 3 contending tasks. To reduce that performance deficit at low contention level, a special x86 specific optimized code path for 2 contending tasks was added. This special code path will only be activated with less than 16K of configured CPUs because it uses a byte in the 32-bit lock word to hold a waiting bit for the 2nd contending tasks instead of queuing the waiting task in the queue. With the change, the performance data became: [Standalone/Embedded] # of tasks Ticket lock Queue lock %Change ---------- ----------- ---------- ------- 2 732/950 523/528 -29%/-44% 3 1827/1783 2366/2384 +30%/+34% The queue spinlock code path is now a bit faster with 2 contending tasks. There is also a very slight improvement with 3 contending tasks. The performance of the optimized code path can vary depending on which of the several different code paths is taken. It is also not as fair as the ticket spinlock and some variations on the execution times of the 2 contending tasks. Testing with different pairs of cores within the same CPUs shows an execution time that varies from 400ms to 1194ms. The ticket spinlock code also shows a variation of 718-1146ms which is probably due to the CPU topology within a socket. In a multi-socketed server, the optimized code path also seems to produce a big performance improvement in cross-node contention traffic at low contention level. The table below show the performance with 1 contending task per node: [Standalone] # of nodes Ticket lock Queue lock %Change ---------- ----------- ---------- ------- 1 135 135 0% 2 4452 528 -88% 3 10767 2369 -78% 4 20835 2921 -86% The micro-benchmark was also run on a 4-core Ivy-Bridge PC. The table below shows the collected performance data: [Standalone/Embedded] # of tasks Ticket lock Queue lock %Change ---------- ----------- ---------- ------- 1 197/178 181/150 -8%/-16% 2 1109/928 435-1417/697-2125 3 1836/1702 1372-3112/1379-3138 4 2717/2429 1842-4158/1846-4170 The performance of the queue lock patch varied from run to run whereas the performance of the ticket lock was more consistent. The queue lock figures above were the range of values that were reported. This optimization can also be easily used by other architectures as long as they support 8 and 16 bits atomic operations. Signed-off-by: Waiman Long <Waiman.Long at hp.com> --- arch/x86/include/asm/qspinlock.h | 20 ++++- include/asm-generic/qspinlock_types.h | 8 ++- kernel/locking/qspinlock.c | 192 ++++++++++++++++++++++++++++++++- 3 files changed, 215 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h index 44cefee..98db42e 100644 --- a/arch/x86/include/asm/qspinlock.h +++ b/arch/x86/include/asm/qspinlock.h @@ -7,12 +7,30 @@ #define _ARCH_SUPPORTS_ATOMIC_8_16_BITS_OPS +#define smp_u8_store_release(p, v) \ +do { \ + barrier(); \ + ACCESS_ONCE(*p) = (v); \ +} while (0) + +/* + * As the qcode will be accessed as a 16-bit word, no offset is needed + */ +#define _QCODE_VAL_OFFSET 0 + /* * x86-64 specific queue spinlock union structure + * Besides the slock and lock fields, the other fields are only + * valid with less than 16K CPUs. */ union arch_qspinlock { struct qspinlock slock; - u8 lock; /* Lock bit */ + struct { + u8 lock; /* Lock bit */ + u8 wait; /* Waiting bit */ + u16 qcode; /* Queue code */ + }; + u16 lock_wait; /* Lock and wait bits */ }; #define queue_spin_unlock queue_spin_unlock diff --git a/include/asm-generic/qspinlock_types.h b/include/asm-generic/qspinlock_types.h index df981d0..3a02a9e 100644 --- a/include/asm-generic/qspinlock_types.h +++ b/include/asm-generic/qspinlock_types.h @@ -48,7 +48,13 @@ typedef struct qspinlock { atomic_t qlcode; /* Lock + queue code */ } arch_spinlock_t; -#define _QCODE_OFFSET 8 +#if CONFIG_NR_CPUS >= (1 << 14) +# define _Q_MANY_CPUS +# define _QCODE_OFFSET 8 +#else +# define _QCODE_OFFSET 16 +#endif + #define _QSPINLOCK_LOCKED 1U #define _QSPINLOCK_LOCK_MASK 0xff diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index ed5efa7..22a63fa 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -109,8 +109,11 @@ static DEFINE_PER_CPU_ALIGNED(struct qnode_set, qnset) = { {{0}}, 0 }; * 2) A smp_u8_store_release() macro for byte size store operation * * 3) A "union arch_qspinlock" structure that include the individual * * fields of the qspinlock structure, including: * - * o slock - the qspinlock structure * - * o lock - the lock byte * + * o slock - the qspinlock structure * + * o lock - the lock byte * + * o wait - the waiting byte * + * o qcode - the queue node code * + * o lock_wait - the combined lock and waiting bytes * * * ************************************************************************ */ @@ -129,6 +132,176 @@ static __always_inline int queue_spin_setlock(struct qspinlock *lock) return 1; return 0; } + +#ifndef _Q_MANY_CPUS +/* + * With less than 16K CPUs, the following optimizations are possible with + * the x86 architecture: + * 1) The 2nd byte of the 32-bit lock word can be used as a pending bit + * for waiting lock acquirer so that it won't need to go through the + * MCS style locking queuing which has a higher overhead. + * 2) The 16-bit queue code can be accessed or modified directly as a + * 16-bit short value without disturbing the first 2 bytes. + */ +#define _QSPINLOCK_WAITING 0x100U /* Waiting bit in 2nd byte */ +#define _QSPINLOCK_LWMASK 0xffff /* Mask for lock & wait bits */ + +#define queue_encode_qcode(cpu, idx) (((cpu) + 1) << 2 | (idx)) + +#define queue_spin_trylock_quick queue_spin_trylock_quick +/** + * queue_spin_trylock_quick - fast spinning on the queue spinlock + * @lock : Pointer to queue spinlock structure + * @qsval: Old queue spinlock value + * Return: 1 if lock acquired, 0 if failed + * + * This is an optimized contention path for 2 contending tasks. It + * should only be entered if no task is waiting in the queue. This + * optimized path is not as fair as the ticket spinlock, but it offers + * slightly better performance. The regular MCS locking path for 3 or + * more contending tasks, however, is fair. + * + * Depending on the exact timing, there are several different paths where + * a contending task can take. The actual contention performance depends + * on which path is taken. So it can be faster or slower than the + * corresponding ticket spinlock path. On average, it is probably on par + * with ticket spinlock. + */ +static inline int queue_spin_trylock_quick(struct qspinlock *lock, int qsval) +{ + union arch_qspinlock *qlock = (union arch_qspinlock *)lock; + u16 old; + + /* + * Fall into the quick spinning code path only if no one is waiting + * or the lock is available. + */ + if (unlikely((qsval != _QSPINLOCK_LOCKED) && + (qsval != _QSPINLOCK_WAITING))) + return 0; + + old = xchg(&qlock->lock_wait, _QSPINLOCK_WAITING|_QSPINLOCK_LOCKED); + + if (old == 0) { + /* + * Got the lock, can clear the waiting bit now + */ + smp_u8_store_release(&qlock->wait, 0); + return 1; + } else if (old == _QSPINLOCK_LOCKED) { +try_again: + /* + * Wait until the lock byte is cleared to get the lock + */ + do { + cpu_relax(); + } while (ACCESS_ONCE(qlock->lock)); + /* + * Set the lock bit & clear the waiting bit + */ + if (cmpxchg(&qlock->lock_wait, _QSPINLOCK_WAITING, + _QSPINLOCK_LOCKED) == _QSPINLOCK_WAITING) + return 1; + /* + * Someone has steal the lock, so wait again + */ + goto try_again; + } else if (old == _QSPINLOCK_WAITING) { + /* + * Another task is already waiting while it steals the lock. + * A bit of unfairness here won't change the big picture. + * So just take the lock and return. + */ + return 1; + } + /* + * Nothing need to be done if the old value is + * (_QSPINLOCK_WAITING | _QSPINLOCK_LOCKED). + */ + return 0; +} + +#define queue_code_xchg queue_code_xchg +/** + * queue_code_xchg - exchange a queue code value + * @lock : Pointer to queue spinlock structure + * @qcode: New queue code to be exchanged + * Return: The original qcode value in the queue spinlock + */ +static inline u32 queue_code_xchg(struct qspinlock *lock, u32 qcode) +{ + union arch_qspinlock *qlock = (union arch_qspinlock *)lock; + + return (u32)xchg(&qlock->qcode, (u16)qcode); +} + +#define queue_spin_trylock_and_clr_qcode queue_spin_trylock_and_clr_qcode +/** + * queue_spin_trylock_and_clr_qcode - Try to lock & clear qcode simultaneously + * @lock : Pointer to queue spinlock structure + * @qcode: The supposedly current qcode value + * Return: true if successful, false otherwise + */ +static inline int +queue_spin_trylock_and_clr_qcode(struct qspinlock *lock, u32 qcode) +{ + qcode <<= _QCODE_OFFSET; + return atomic_cmpxchg(&lock->qlcode, qcode, _QSPINLOCK_LOCKED) == qcode; +} + +#define queue_get_lock_qcode queue_get_lock_qcode +/** + * queue_get_lock_qcode - get the lock & qcode values + * @lock : Pointer to queue spinlock structure + * @qcode : Pointer to the returned qcode value + * @mycode: My qcode value + * Return : > 0 if lock is not available + * = 0 if lock is free + * < 0 if lock is taken & can return after cleanup + * + * It is considered locked when either the lock bit or the wait bit is set. + */ +static inline int +queue_get_lock_qcode(struct qspinlock *lock, u32 *qcode, u32 mycode) +{ + u32 qlcode; + + qlcode = (u32)atomic_read(&lock->qlcode); + /* + * With the special case that qlcode contains only _QSPINLOCK_LOCKED + * and mycode. It will try to transition back to the quick spinning + * code by clearing the qcode and setting the _QSPINLOCK_WAITING + * bit. + */ + if (qlcode == (_QSPINLOCK_LOCKED | (mycode << _QCODE_OFFSET))) { + u32 old = qlcode; + + qlcode = atomic_cmpxchg(&lock->qlcode, old, + _QSPINLOCK_LOCKED|_QSPINLOCK_WAITING); + if (qlcode == old) { + union arch_qspinlock *slock + (union arch_qspinlock *)lock; +try_again: + /* + * Wait until the lock byte is cleared + */ + do { + cpu_relax(); + } while (ACCESS_ONCE(slock->lock)); + /* + * Set the lock bit & clear the waiting bit + */ + if (cmpxchg(&slock->lock_wait, _QSPINLOCK_WAITING, + _QSPINLOCK_LOCKED) == _QSPINLOCK_WAITING) + return -1; /* Got the lock */ + goto try_again; + } + } + *qcode = qlcode >> _QCODE_OFFSET; + return qlcode & _QSPINLOCK_LWMASK; +} +#endif /* _Q_MANY_CPUS */ + #else /* _ARCH_SUPPORTS_ATOMIC_8_16_BITS_OPS */ /* * Generic functions for architectures that do not support atomic @@ -144,7 +317,7 @@ static __always_inline int queue_spin_setlock(struct qspinlock *lock) int qlcode = atomic_read(lock->qlcode); if (!(qlcode & _QSPINLOCK_LOCKED) && (atomic_cmpxchg(&lock->qlcode, - qlcode, qlcode|_QSPINLOCK_LOCKED) == qlcode)) + qlcode, code|_QSPINLOCK_LOCKED) == qlcode)) return 1; return 0; } @@ -156,6 +329,10 @@ static __always_inline int queue_spin_setlock(struct qspinlock *lock) * that may get superseded by a more optimized version. * ************************************************************************ */ +#ifndef queue_spin_trylock_quick +static inline int queue_spin_trylock_quick(struct qspinlock *lock, int qsval) +{ return 0; } +#endif #ifndef queue_get_lock_qcode /** @@ -266,6 +443,11 @@ void queue_spin_lock_slowpath(struct qspinlock *lock, int qsval) u32 prev_qcode, my_qcode; /* + * Try the quick spinning code path + */ + if (queue_spin_trylock_quick(lock, qsval)) + return; + /* * Get the queue node */ cpu_nr = smp_processor_id(); @@ -296,6 +478,9 @@ void queue_spin_lock_slowpath(struct qspinlock *lock, int qsval) return; } +#ifdef queue_code_xchg + prev_qcode = queue_code_xchg(lock, my_qcode); +#else /* * Exchange current copy of the queue node code */ @@ -329,6 +514,7 @@ void queue_spin_lock_slowpath(struct qspinlock *lock, int qsval) } else prev_qcode &= ~_QSPINLOCK_LOCKED; /* Clear the lock bit */ my_qcode &= ~_QSPINLOCK_LOCKED; +#endif /* queue_code_xchg */ if (prev_qcode) { /* -- 1.7.1
Waiman Long
2014-Feb-27 04:32 UTC
[PATCH RFC v5 4/8] pvqspinlock, x86: Allow unfair spinlock in a real PV environment
Locking is always an issue in a virtualized environment as the virtual CPU that is waiting on a lock may get scheduled out and hence block any progress in lock acquisition even when the lock has been freed. One solution to this problem is to allow unfair lock in a para-virtualized environment. In this case, a new lock acquirer can come and steal the lock if the next-in-line CPU to get the lock is scheduled out. Unfair lock in a native environment is generally not a good idea as there is a possibility of lock starvation for a heavily contended lock. This patch add a new configuration option for the x86 architecture to enable the use of unfair queue spinlock (PARAVIRT_UNFAIR_LOCKS) in a real para-virtualized guest. A jump label (paravirt_unfairlocks_enabled) is used to switch between a fair and an unfair version of the spinlock code. This jump label will only be enabled in a real PV guest. Enabling this configuration feature decreases the performance of an uncontended lock-unlock operation by about 1-2%. Signed-off-by: Waiman Long <Waiman.Long at hp.com> --- arch/x86/Kconfig | 11 +++++ arch/x86/include/asm/qspinlock.h | 74 ++++++++++++++++++++++++++++++++++ arch/x86/kernel/Makefile | 1 + arch/x86/kernel/paravirt-spinlocks.c | 7 +++ 4 files changed, 93 insertions(+), 0 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 5bf70ab..8d7c941 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -645,6 +645,17 @@ config PARAVIRT_SPINLOCKS If you are unsure how to answer this question, answer Y. +config PARAVIRT_UNFAIR_LOCKS + bool "Enable unfair locks in a para-virtualized guest" + depends on PARAVIRT && SMP && QUEUE_SPINLOCK + depends on !CONFIG_X86_OOSTORE && !CONFIG_X86_PPRO_FENCE + ---help--- + This changes the kernel to use unfair locks in a real + para-virtualized guest system. This will help performance + in most cases. However, there is a possibility of lock + starvation on a heavily contended lock especially in a + large guest with many virtual CPUs. + source "arch/x86/xen/Kconfig" config KVM_GUEST diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h index 98db42e..c278aed 100644 --- a/arch/x86/include/asm/qspinlock.h +++ b/arch/x86/include/asm/qspinlock.h @@ -56,4 +56,78 @@ static inline void queue_spin_unlock(struct qspinlock *lock) #include <asm-generic/qspinlock.h> +#ifdef CONFIG_PARAVIRT_UNFAIR_LOCKS +/** + * queue_spin_lock_unfair - acquire a queue spinlock unfairly + * @lock: Pointer to queue spinlock structure + */ +static __always_inline void queue_spin_lock_unfair(struct qspinlock *lock) +{ + union arch_qspinlock *qlock = (union arch_qspinlock *)lock; + + if (likely(cmpxchg(&qlock->lock, 0, _QSPINLOCK_LOCKED) == 0)) + return; + /* + * Since the lock is now unfair, there is no need to activate + * the 2-task quick spinning code path. + */ + queue_spin_lock_slowpath(lock, -1); +} + +/** + * queue_spin_trylock_unfair - try to acquire the queue spinlock unfairly + * @lock : Pointer to queue spinlock structure + * Return: 1 if lock acquired, 0 if failed + */ +static __always_inline int queue_spin_trylock_unfair(struct qspinlock *lock) +{ + union arch_qspinlock *qlock = (union arch_qspinlock *)lock; + + if (!qlock->lock && + (cmpxchg(&qlock->lock, 0, _QSPINLOCK_LOCKED) == 0)) + return 1; + return 0; +} + +/* + * Redefine arch_spin_lock and arch_spin_trylock as inline functions that will + * jump to the unfair versions if the static key paravirt_unfairlocks_enabled + * is true. + */ +#undef arch_spin_lock +#undef arch_spin_trylock +#undef arch_spin_lock_flags + +extern struct static_key paravirt_unfairlocks_enabled; + +/** + * arch_spin_lock - acquire a queue spinlock + * @lock: Pointer to queue spinlock structure + */ +static inline void arch_spin_lock(struct qspinlock *lock) +{ + if (static_key_false(¶virt_unfairlocks_enabled)) { + queue_spin_lock_unfair(lock); + return; + } + queue_spin_lock(lock); +} + +/** + * arch_spin_trylock - try to acquire the queue spinlock + * @lock : Pointer to queue spinlock structure + * Return: 1 if lock acquired, 0 if failed + */ +static inline int arch_spin_trylock(struct qspinlock *lock) +{ + if (static_key_false(¶virt_unfairlocks_enabled)) { + return queue_spin_trylock_unfair(lock); + } + return queue_spin_trylock(lock); +} + +#define arch_spin_lock_flags(l, f) arch_spin_lock(l) + +#endif /* CONFIG_PARAVIRT_UNFAIR_LOCKS */ + #endif /* _ASM_X86_QSPINLOCK_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index cb648c8..1107a20 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -88,6 +88,7 @@ obj-$(CONFIG_DEBUG_NMI_SELFTEST) += nmi_selftest.o obj-$(CONFIG_KVM_GUEST) += kvm.o kvmclock.o obj-$(CONFIG_PARAVIRT) += paravirt.o paravirt_patch_$(BITS).o obj-$(CONFIG_PARAVIRT_SPINLOCKS)+= paravirt-spinlocks.o +obj-$(CONFIG_PARAVIRT_UNFAIR_LOCKS)+= paravirt-spinlocks.o obj-$(CONFIG_PARAVIRT_CLOCK) += pvclock.o obj-$(CONFIG_PCSPKR_PLATFORM) += pcspeaker.o diff --git a/arch/x86/kernel/paravirt-spinlocks.c b/arch/x86/kernel/paravirt-spinlocks.c index bbb6c73..a50032a 100644 --- a/arch/x86/kernel/paravirt-spinlocks.c +++ b/arch/x86/kernel/paravirt-spinlocks.c @@ -8,6 +8,7 @@ #include <asm/paravirt.h> +#ifdef CONFIG_PARAVIRT_SPINLOCKS struct pv_lock_ops pv_lock_ops = { #ifdef CONFIG_SMP .lock_spinning = __PV_IS_CALLEE_SAVE(paravirt_nop), @@ -18,3 +19,9 @@ EXPORT_SYMBOL(pv_lock_ops); struct static_key paravirt_ticketlocks_enabled = STATIC_KEY_INIT_FALSE; EXPORT_SYMBOL(paravirt_ticketlocks_enabled); +#endif + +#ifdef CONFIG_PARAVIRT_UNFAIR_LOCKS +struct static_key paravirt_unfairlocks_enabled = STATIC_KEY_INIT_FALSE; +EXPORT_SYMBOL(paravirt_unfairlocks_enabled); +#endif -- 1.7.1
Waiman Long
2014-Feb-27 04:32 UTC
[PATCH RFC v5 5/8] pvqspinlock, x86: Enable unfair queue spinlock in a KVM guest
This patch adds a KVM init function to activate the unfair queue spinlock in a KVM guest when the PARAVIRT_UNFAIR_LOCKS kernel config option is selected. Signed-off-by: Waiman Long <Waiman.Long at hp.com> --- arch/x86/kernel/kvm.c | 17 +++++++++++++++++ 1 files changed, 17 insertions(+), 0 deletions(-) diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 713f1b3..a489140 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -826,3 +826,20 @@ static __init int kvm_spinlock_init_jump(void) early_initcall(kvm_spinlock_init_jump); #endif /* CONFIG_PARAVIRT_SPINLOCKS */ + +#ifdef CONFIG_PARAVIRT_UNFAIR_LOCKS +/* + * Enable unfair lock if running in a real para-virtualized environment + */ +static __init int kvm_unfair_locks_init_jump(void) +{ + if (!kvm_para_available()) + return 0; + + static_key_slow_inc(¶virt_unfairlocks_enabled); + printk(KERN_INFO "KVM setup unfair spinlock\n"); + + return 0; +} +early_initcall(kvm_unfair_locks_init_jump); +#endif -- 1.7.1
Waiman Long
2014-Feb-27 04:32 UTC
[PATCH RFC v5 6/8] pvqspinlock, x86: Rename paravirt_ticketlocks_enabled
This patch renames the paravirt_ticketlocks_enabled static key to a more generic paravirt_spinlocks_enabled name. Signed-off-by: Waiman Long <Waiman.Long at hp.com> --- arch/x86/include/asm/spinlock.h | 4 ++-- arch/x86/kernel/kvm.c | 2 +- arch/x86/kernel/paravirt-spinlocks.c | 4 ++-- arch/x86/xen/spinlock.c | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/spinlock.h b/arch/x86/include/asm/spinlock.h index 6e6de1f..283f2cf 100644 --- a/arch/x86/include/asm/spinlock.h +++ b/arch/x86/include/asm/spinlock.h @@ -40,7 +40,7 @@ /* How long a lock should spin before we consider blocking */ #define SPIN_THRESHOLD (1 << 15) -extern struct static_key paravirt_ticketlocks_enabled; +extern struct static_key paravirt_spinlocks_enabled; static __always_inline bool static_key_false(struct static_key *key); #ifdef CONFIG_QUEUE_SPINLOCK @@ -151,7 +151,7 @@ static inline void __ticket_unlock_slowpath(arch_spinlock_t *lock, static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) { if (TICKET_SLOWPATH_FLAG && - static_key_false(¶virt_ticketlocks_enabled)) { + static_key_false(¶virt_spinlocks_enabled)) { arch_spinlock_t prev; prev = *lock; diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index a489140..f318e78 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -818,7 +818,7 @@ static __init int kvm_spinlock_init_jump(void) if (!kvm_para_has_feature(KVM_FEATURE_PV_UNHALT)) return 0; - static_key_slow_inc(¶virt_ticketlocks_enabled); + static_key_slow_inc(¶virt_spinlocks_enabled); printk(KERN_INFO "KVM setup paravirtual spinlock\n"); return 0; diff --git a/arch/x86/kernel/paravirt-spinlocks.c b/arch/x86/kernel/paravirt-spinlocks.c index a50032a..8c67cbe 100644 --- a/arch/x86/kernel/paravirt-spinlocks.c +++ b/arch/x86/kernel/paravirt-spinlocks.c @@ -17,8 +17,8 @@ struct pv_lock_ops pv_lock_ops = { }; EXPORT_SYMBOL(pv_lock_ops); -struct static_key paravirt_ticketlocks_enabled = STATIC_KEY_INIT_FALSE; -EXPORT_SYMBOL(paravirt_ticketlocks_enabled); +struct static_key paravirt_spinlocks_enabled = STATIC_KEY_INIT_FALSE; +EXPORT_SYMBOL(paravirt_spinlocks_enabled); #endif #ifdef CONFIG_PARAVIRT_UNFAIR_LOCKS diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c index 581521c..06f4a64 100644 --- a/arch/x86/xen/spinlock.c +++ b/arch/x86/xen/spinlock.c @@ -290,7 +290,7 @@ static __init int xen_init_spinlocks_jump(void) if (!xen_pvspin) return 0; - static_key_slow_inc(¶virt_ticketlocks_enabled); + static_key_slow_inc(¶virt_spinlocks_enabled); return 0; } early_initcall(xen_init_spinlocks_jump); -- 1.7.1
Waiman Long
2014-Feb-27 04:32 UTC
[PATCH RFC v5 7/8] pvqspinlock, x86: Add qspinlock para-virtualization support
This patch adds para-virtualization support to the queue spinlock code by enabling the queue head to kick the lock holder CPU, if known, in when the lock isn't released for a certain amount of time. It also enables the mutual monitoring of the queue head CPU and the following node CPU in the queue to make sure that their CPUs will stay scheduled in. Signed-off-by: Waiman Long <Waiman.Long at hp.com> --- arch/x86/include/asm/paravirt.h | 9 ++- arch/x86/include/asm/paravirt_types.h | 12 +++ arch/x86/include/asm/pvqspinlock.h | 176 +++++++++++++++++++++++++++++++++ arch/x86/kernel/paravirt-spinlocks.c | 4 + kernel/locking/qspinlock.c | 41 +++++++- 5 files changed, 235 insertions(+), 7 deletions(-) create mode 100644 arch/x86/include/asm/pvqspinlock.h diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index cd6e161..06d3279 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -711,7 +711,12 @@ static inline void __set_fixmap(unsigned /* enum fixed_addresses */ idx, } #if defined(CONFIG_SMP) && defined(CONFIG_PARAVIRT_SPINLOCKS) - +#ifdef CONFIG_QUEUE_SPINLOCK +static __always_inline void __queue_kick_cpu(int cpu, enum pv_kick_type type) +{ + PVOP_VCALL2(pv_lock_ops.kick_cpu, cpu, type); +} +#else static __always_inline void __ticket_lock_spinning(struct arch_spinlock *lock, __ticket_t ticket) { @@ -723,7 +728,7 @@ static __always_inline void __ticket_unlock_kick(struct arch_spinlock *lock, { PVOP_VCALL2(pv_lock_ops.unlock_kick, lock, ticket); } - +#endif #endif #ifdef CONFIG_X86_32 diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h index 7549b8b..87f8836 100644 --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -333,9 +333,21 @@ struct arch_spinlock; typedef u16 __ticket_t; #endif +#ifdef CONFIG_QUEUE_SPINLOCK +enum pv_kick_type { + PV_KICK_LOCK_HOLDER, + PV_KICK_QUEUE_HEAD, + PV_KICK_NEXT_NODE +}; +#endif + struct pv_lock_ops { +#ifdef CONFIG_QUEUE_SPINLOCK + void (*kick_cpu)(int cpu, enum pv_kick_type); +#else struct paravirt_callee_save lock_spinning; void (*unlock_kick)(struct arch_spinlock *lock, __ticket_t ticket); +#endif }; /* This contains all the paravirt structures: we get a convenient diff --git a/arch/x86/include/asm/pvqspinlock.h b/arch/x86/include/asm/pvqspinlock.h new file mode 100644 index 0000000..45aae39 --- /dev/null +++ b/arch/x86/include/asm/pvqspinlock.h @@ -0,0 +1,176 @@ +#ifndef _ASM_X86_PVQSPINLOCK_H +#define _ASM_X86_PVQSPINLOCK_H + +/* + * Queue Spinlock Para-Virtualization Support + * + * +------+ +-----+ nxtcpu_p1 +----+ + * | Lock | |Queue|----------->|Next| + * |Holder|<-----------|Head |<-----------|Node| + * +------+ prev_qcode +-----+ prev_qcode +----+ + * + * As long as the current lock holder passes through the slowpath, the queue + * head CPU will have its CPU number stored in prev_qcode. The situation is + * the same for the node next to the queue head. + * + * The next node, while setting up the next pointer in the queue head, can + * also store its CPU number in that node. With that change, the queue head + * will have the CPU numbers of both its upstream and downstream neighbors. + * + * To make forward progress in lock acquisition and release, it is necessary + * that both the lock holder and the queue head virtual CPUs are present. + * The queue head can monitor the lock holder, but the lock holder can't + * monitor the queue head back. As a result, the next node is also brought + * into the picture to monitor the queue head. In the above diagram, all the + * 3 virtual CPUs should be present with the queue head and next node + * monitoring each other to make sure they are both present. + * + * Heartbeat counters are used to track if a neighbor is active. There are + * 3 different sets of heartbeat counter monitoring going on: + * 1) The queue head will wait until the number loop iteration exceeds a + * certain threshold (HEAD_SPIN_THRESHOLD). In that case, it will send + * a kick-cpu signal to the lock holder if it has the CPU number available. + * The kick-cpu siginal will be sent only once as the real lock holder + * may not be the same as what the queue head thinks it is. + * 2) The queue head will periodically clear the active flag of the next node. + * It will then check to see if the active flag remains cleared at the end + * of the cycle. If it is, the next node CPU may be scheduled out. So it + * send a kick-cpu signal to make sure that the next node CPU remain active. + * 3) The next node CPU will monitor its own active flag to see if it gets + * clear periodically. If it is not, the queue head CPU may be scheduled + * out. It will then send the kick-cpu signal to the queue head CPU. + */ + +/* + * Loop thresholds + */ +#define HEAD_SPIN_THRESHOLD (1<<12) /* Threshold to kick lock holder */ +#define CLEAR_ACTIVE_THRESHOLD (1<<8) /* Threahold for clearing active flag */ +#define CLEAR_ACTIVE_MASK (CLEAR_ACTIVE_THRESHOLD - 1) + +/* + * PV macros + */ +#define PV_SET_VAR(type, var, val) type var = val +#define PV_VAR(var) var +#define PV_GET_NXTCPU(node) (node)->pv.nxtcpu_p1 + +/* + * Additional fields to be added to the qnode structure + * + * Try to cram the PV fields into a 32 bits so that it won't increase the + * qnode size in x86-64. + */ +#if CONFIG_NR_CPUS >= (1 << 16) +#define _cpuid_t u32 +#else +#define _cpuid_t u16 +#endif + +struct pv_qvars { + u8 active; /* Set if CPU active */ + u8 prehead; /* Set if next to queue head */ + _cpuid_t nxtcpu_p1; /* CPU number of next node + 1 */ +}; + +/** + * pv_init_vars - initialize fields in struct pv_qvars + * @pv: pointer to struct pv_qvars + */ +static __always_inline void pv_init_vars(struct pv_qvars *pv) +{ + pv->active = false; + pv->prehead = false; + pv->nxtcpu_p1 = 0; +} + +/** + * head_spin_check - perform para-virtualization checks for queue head + * @count : loop count + * @qcode : queue code of the supposed lock holder + * @nxtcpu: CPU number of next node + 1 + * @next : pointer to the next node + * @offset: offset of the pv_qvars within the qnode + * + * 4 checks will be done: + * 1) See if it is time to kick the lock holder + * 2) Set the prehead flag of the next node + * 3) Clear the active flag of the next node periodically + * 4) If the active flag is not set after a while, assume the CPU of the + * next-in-line node is offline and kick it back up again. + */ +static __always_inline void +pv_head_spin_check(int *count, u32 qcode, int nxtcpu, void *next, int offset) +{ + if (!static_key_false(¶virt_spinlocks_enabled)) + return; + if ((++(*count) == HEAD_SPIN_THRESHOLD) && qcode) { + /* + * Get the CPU number of the lock holder & kick it + * The lock may have been stealed by another CPU + * if PARAVIRT_UNFAIR_LOCKS is set, so the computed + * CPU number may not be the actual lock holder. + */ + int cpu = (qcode >> (_QCODE_VAL_OFFSET + 2)) - 1; + __queue_kick_cpu(cpu, PV_KICK_LOCK_HOLDER); + } + if (next) { + struct pv_qvars *pv = (struct pv_qvars *) + ((char *)next + offset); + + if (!pv->prehead) + pv->prehead = true; + if ((*count & CLEAR_ACTIVE_MASK) == CLEAR_ACTIVE_MASK) + pv->active = false; + if (((*count & CLEAR_ACTIVE_MASK) == 0) && + !pv->active && nxtcpu) + /* + * The CPU of the next node doesn't seem to be + * active, need to kick it to make sure that + * it is ready to be transitioned to queue head. + */ + __queue_kick_cpu(nxtcpu - 1, PV_KICK_NEXT_NODE); + } +} + +/** + * head_spin_check - perform para-virtualization checks for queue member + * @pv : pointer to struct pv_qvars + * @count: loop count + * @qcode: queue code of the previous node (queue head if pv->prehead set) + * + * Set the active flag if it is next to the queue head + */ +static __always_inline void +pv_queue_spin_check(struct pv_qvars *pv, int *count, u32 qcode) +{ + if (!static_key_false(¶virt_spinlocks_enabled)) + return; + if (ACCESS_ONCE(pv->prehead)) { + if (pv->active == false) { + *count = 0; /* Reset counter */ + pv->active = true; + } + if ((++(*count) >= 4 * CLEAR_ACTIVE_THRESHOLD) && qcode) { + /* + * The queue head isn't clearing the active flag for + * too long. Need to kick it. + */ + int cpu = (qcode >> (_QCODE_VAL_OFFSET + 2)) - 1; + __queue_kick_cpu(cpu, PV_KICK_QUEUE_HEAD); + *count = 0; + } + } +} + +/** + * pv_set_cpu - set CPU # in the given pv_qvars structure + * @pv : pointer to struct pv_qvars to be set + * @cpu: cpu number to be set + */ +static __always_inline void pv_set_cpu(struct pv_qvars *pv, int cpu) +{ + pv->nxtcpu_p1 = cpu + 1; +} + +#endif /* _ASM_X86_PVQSPINLOCK_H */ diff --git a/arch/x86/kernel/paravirt-spinlocks.c b/arch/x86/kernel/paravirt-spinlocks.c index 8c67cbe..30d76f5 100644 --- a/arch/x86/kernel/paravirt-spinlocks.c +++ b/arch/x86/kernel/paravirt-spinlocks.c @@ -11,9 +11,13 @@ #ifdef CONFIG_PARAVIRT_SPINLOCKS struct pv_lock_ops pv_lock_ops = { #ifdef CONFIG_SMP +#ifdef CONFIG_QUEUE_SPINLOCK + .kick_cpu = paravirt_nop, +#else .lock_spinning = __PV_IS_CALLEE_SAVE(paravirt_nop), .unlock_kick = paravirt_nop, #endif +#endif }; EXPORT_SYMBOL(pv_lock_ops); diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index 22a63fa..f10446e 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -58,6 +58,26 @@ */ /* + * Para-virtualized queue spinlock support + */ +#ifdef CONFIG_PARAVIRT_SPINLOCKS +#include <asm/pvqspinlock.h> +#else + +#define PV_SET_VAR(type, var, val) +#define PV_VAR(var) 0 +#define PV_GET_NXTCPU(node) 0 + +struct pv_qvars {}; +static __always_inline void pv_init_vars(struct pv_qvars *pv) {} +static __always_inline void pv_head_spin_check(int *count, u32 qcode, + int nxtcpu, void *next, int offset) {} +static __always_inline void pv_queue_spin_check(struct pv_qvars *pv, + int *count, u32 qcode) {} +static __always_inline void pv_set_cpu(struct pv_qvars *pv, int cpu) {} +#endif + +/* * The 24-bit queue node code is divided into the following 2 fields: * Bits 0-1 : queue node index (4 nodes) * Bits 2-23: CPU number + 1 (4M - 1 CPUs) @@ -77,15 +97,13 @@ /* * The queue node structure - * - * This structure is essentially the same as the mcs_spinlock structure - * in mcs_spinlock.h file. This structure is retained for future extension - * where new fields may be added. */ struct qnode { u32 wait; /* Waiting flag */ + struct pv_qvars pv; /* Para-virtualization */ struct qnode *next; /* Next queue node addr */ }; +#define PV_OFFSET offsetof(struct qnode, pv) struct qnode_set { struct qnode nodes[MAX_QNODES]; @@ -441,6 +459,7 @@ void queue_spin_lock_slowpath(struct qspinlock *lock, int qsval) unsigned int cpu_nr, qn_idx; struct qnode *node, *next; u32 prev_qcode, my_qcode; + PV_SET_VAR(int, hcnt, 0); /* * Try the quick spinning code path @@ -468,6 +487,7 @@ void queue_spin_lock_slowpath(struct qspinlock *lock, int qsval) */ node->wait = true; node->next = NULL; + pv_init_vars(&node->pv); /* * The lock may be available at this point, try again if no task was @@ -522,13 +542,22 @@ void queue_spin_lock_slowpath(struct qspinlock *lock, int qsval) * and set up the "next" fields of the that node. */ struct qnode *prev = xlate_qcode(prev_qcode); + PV_SET_VAR(int, qcnt, 0); ACCESS_ONCE(prev->next) = node; /* + * Set current CPU number into the previous node + */ + pv_set_cpu(&prev->pv, cpu_nr); + + /* * Wait until the waiting flag is off */ - while (smp_load_acquire(&node->wait)) + while (smp_load_acquire(&node->wait)) { arch_mutex_cpu_relax(); + pv_queue_spin_check(&node->pv, PV_VAR(&qcnt), + prev_qcode); + } } /* @@ -560,6 +589,8 @@ void queue_spin_lock_slowpath(struct qspinlock *lock, int qsval) goto release_node; } arch_mutex_cpu_relax(); + pv_head_spin_check(PV_VAR(&hcnt), prev_qcode, + PV_GET_NXTCPU(node), node->next, PV_OFFSET); } notify_next: -- 1.7.1
Waiman Long
2014-Feb-27 04:32 UTC
[PATCH RFC v5 8/8] pvqspinlock, x86: Enable KVM to use qspinlock's PV support
This patch enables KVM to use the queue spinlock's PV support code when the PARAVIRT_SPINLOCKS kernel config option is set. However, PV support for Xen is not ready yet and so the queue spinlock will still have to be disabled when PARAVIRT_SPINLOCKS config option is on with Xen. Signed-off-by: Waiman Long <Waiman.Long at hp.com> --- arch/x86/kernel/kvm.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++ kernel/Kconfig.locks | 2 +- 2 files changed, 55 insertions(+), 1 deletions(-) diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index f318e78..3ddc436 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -568,6 +568,7 @@ static void kvm_kick_cpu(int cpu) kvm_hypercall2(KVM_HC_KICK_CPU, flags, apicid); } +#ifndef CONFIG_QUEUE_SPINLOCK enum kvm_contention_stat { TAKEN_SLOW, TAKEN_SLOW_PICKUP, @@ -795,6 +796,55 @@ static void kvm_unlock_kick(struct arch_spinlock *lock, __ticket_t ticket) } } } +#else /* !CONFIG_QUEUE_SPINLOCK */ + +#ifdef CONFIG_KVM_DEBUG_FS +static struct dentry *d_spin_debug; +static struct dentry *d_kvm_debug; +static u32 lh_kick_stats; /* Lock holder kick count */ +static u32 qh_kick_stats; /* Queue head kick count */ +static u32 nn_kick_stats; /* Next node kick count */ + +static int __init kvm_spinlock_debugfs(void) +{ + d_kvm_debug = debugfs_create_dir("kvm-guest", NULL); + if (!d_kvm_debug) { + printk(KERN_WARNING + "Could not create 'kvm' debugfs directory\n"); + return -ENOMEM; + } + d_spin_debug = debugfs_create_dir("spinlocks", d_kvm_debug); + + debugfs_create_u32("lh_kick_stats", 0644, d_spin_debug, &lh_kick_stats); + debugfs_create_u32("qh_kick_stats", 0644, d_spin_debug, &qh_kick_stats); + debugfs_create_u32("nn_kick_stats", 0644, d_spin_debug, &nn_kick_stats); + + return 0; +} + +static inline void inc_kick_stats(enum pv_kick_type type) +{ + if (type == PV_KICK_LOCK_HOLDER) + add_smp(&lh_kick_stats, 1); + else if (type == PV_KICK_QUEUE_HEAD) + add_smp(&qh_kick_stats, 1); + else + add_smp(&nn_kick_stats, 1); +} +fs_initcall(kvm_spinlock_debugfs); + +#else /* CONFIG_KVM_DEBUG_FS */ +static inline void inc_kick_stats(enum pv_kick_type type) +{ +} +#endif /* CONFIG_KVM_DEBUG_FS */ + +static void kvm_kick_cpu_type(int cpu, enum pv_kick_type type) +{ + kvm_kick_cpu(cpu); + inc_kick_stats(type); +} +#endif /* !CONFIG_QUEUE_SPINLOCK */ /* * Setup pv_lock_ops to exploit KVM_FEATURE_PV_UNHALT if present. @@ -807,8 +857,12 @@ void __init kvm_spinlock_init(void) if (!kvm_para_has_feature(KVM_FEATURE_PV_UNHALT)) return; +#ifdef CONFIG_QUEUE_SPINLOCK + pv_lock_ops.kick_cpu = kvm_kick_cpu_type; +#else pv_lock_ops.lock_spinning = PV_CALLEE_SAVE(kvm_lock_spinning); pv_lock_ops.unlock_kick = kvm_unlock_kick; +#endif } static __init int kvm_spinlock_init_jump(void) diff --git a/kernel/Kconfig.locks b/kernel/Kconfig.locks index f185584..a70fdeb 100644 --- a/kernel/Kconfig.locks +++ b/kernel/Kconfig.locks @@ -229,4 +229,4 @@ config ARCH_USE_QUEUE_SPINLOCK config QUEUE_SPINLOCK def_bool y if ARCH_USE_QUEUE_SPINLOCK - depends on SMP && !PARAVIRT_SPINLOCKS + depends on SMP && (!PARAVIRT_SPINLOCKS || !XEN) -- 1.7.1
Peter Zijlstra
2014-Feb-27 08:37 UTC
[PATCH v5 0/8] qspinlock: a 4-byte queue spinlock with PV support
Is this the same 8 patches you send yesterday?
Waiman Long
2014-Feb-27 18:25 UTC
[PATCH v5 0/8] qspinlock: a 4-byte queue spinlock with PV support
On 02/27/2014 03:37 AM, Peter Zijlstra wrote:> > Is this the same 8 patches you send yesterday?Sorry for the duplication. It was the same patch. It has some minor update in the cover-letter to include some KVM guest test results. I was having problem locating the patch from the LKML list and so I thought they had failed to be sent out properly. -Longman
Oleg Nesterov
2014-Mar-02 13:12 UTC
[PATCH v5 1/8] qspinlock: Introducing a 4-byte queue spinlock implementation
On 02/26, Waiman Long wrote:> > +void queue_spin_lock_slowpath(struct qspinlock *lock, int qsval) > +{ > + unsigned int cpu_nr, qn_idx; > + struct qnode *node, *next; > + u32 prev_qcode, my_qcode; > + > + /* > + * Get the queue node > + */ > + cpu_nr = smp_processor_id(); > + node = get_qnode(&qn_idx); > + > + /* > + * It should never happen that all the queue nodes are being used. > + */ > + BUG_ON(!node); > + > + /* > + * Set up the new cpu code to be exchanged > + */ > + my_qcode = queue_encode_qcode(cpu_nr, qn_idx); > + > + /* > + * Initialize the queue node > + */ > + node->wait = true; > + node->next = NULL; > + > + /* > + * The lock may be available at this point, try again if no task was > + * waiting in the queue. > + */ > + if (!(qsval >> _QCODE_OFFSET) && queue_spin_trylock(lock)) { > + put_qnode(); > + return; > + }Cosmetic, but probably "goto release_node" would be more consistent. And I am wondering how much this "qsval >> _QCODE_OFFSET" check can help. Note that this is the only usage of this arg, perhaps it would be better to simply remove it and shrink the caller's code a bit? It is also used in 3/8, but we can read the "fresh" value of ->qlcode (trylock does this anyway), and perhaps it can actually help if it is already unlocked.> + prev_qcode = atomic_xchg(&lock->qlcode, my_qcode); > + /* > + * It is possible that we may accidentally steal the lock. If this is > + * the case, we need to either release it if not the head of the queue > + * or get the lock and be done with it. > + */ > + if (unlikely(!(prev_qcode & _QSPINLOCK_LOCKED))) { > + if (prev_qcode == 0) { > + /* > + * Got the lock since it is at the head of the queue > + * Now try to atomically clear the queue code. > + */ > + if (atomic_cmpxchg(&lock->qlcode, my_qcode, > + _QSPINLOCK_LOCKED) == my_qcode) > + goto release_node; > + /* > + * The cmpxchg fails only if one or more tasks > + * are added to the queue. In this case, we need to > + * notify the next one to be the head of the queue. > + */ > + goto notify_next; > + } > + /* > + * Accidentally steal the lock, release the lock and > + * let the queue head get it. > + */ > + queue_spin_unlock(lock); > + } else > + prev_qcode &= ~_QSPINLOCK_LOCKED; /* Clear the lock bit */You know, actually I started this email because I thought that "goto notify_next" is wrong, I misread the patch as if this "goto" can happen even if prev_qcode != 0. So feel free to ignore, all my comments are cosmetic/subjective, but to me it would be more clean/clear to rewrite the code above as if (prev_qcode == 0) { if (atomic_cmpxchg(..., _QSPINLOCK_LOCKED) == my_qcode) goto release_node; goto notify_next; } if (prev_qcode & _QSPINLOCK_LOCKED) prev_qcode &= ~_QSPINLOCK_LOCKED; else queue_spin_unlock(lock);> + while (true) { > + u32 qcode; > + int retval; > + > + retval = queue_get_lock_qcode(lock, &qcode, my_qcode); > + if (retval > 0) > + ; /* Lock not available yet */ > + else if (retval < 0) > + /* Lock taken, can release the node & return */ > + goto release_node;I guess this is for 3/8which adds the optimized version of queue_get_lock_qcode(), so perhaps this "retval < 0" block can go into 3/8 as well.> + else if (qcode != my_qcode) { > + /* > + * Just get the lock with other spinners waiting > + * in the queue. > + */ > + if (queue_spin_setlock(lock)) > + goto notify_next;OTOH, at least the generic (non-optimized) version of queue_spin_setlock() could probably accept "qcode" and avoid atomic_read() + _QSPINLOCK_LOCKED check. But once again, please feel free to ignore. Oleg.
Oleg Nesterov
2014-Mar-02 13:16 UTC
[PATCH v5 3/8] qspinlock, x86: Add x86 specific optimization for 2 contending tasks
On 02/26, Waiman Long wrote:> > @@ -144,7 +317,7 @@ static __always_inline int queue_spin_setlock(struct qspinlock *lock) > int qlcode = atomic_read(lock->qlcode); > > if (!(qlcode & _QSPINLOCK_LOCKED) && (atomic_cmpxchg(&lock->qlcode, > - qlcode, qlcode|_QSPINLOCK_LOCKED) == qlcode)) > + qlcode, code|_QSPINLOCK_LOCKED) == qlcode))Hmm. didn't read the patch, but this change looks like accidental typo... Oleg.
Oleg Nesterov
2014-Mar-02 13:31 UTC
[PATCH v5 1/8] qspinlock: Introducing a 4-byte queue spinlock implementation
Forgot to ask... On 02/26, Waiman Long wrote:> > +notify_next: > + /* > + * Wait, if needed, until the next one in queue set up the next field > + */ > + while (!(next = ACCESS_ONCE(node->next))) > + arch_mutex_cpu_relax(); > + /* > + * The next one in queue is now at the head > + */ > + smp_store_release(&next->wait, false);Do we really need smp_store_release()? It seems that we can rely on the control dependency here. And afaics there is no need to serialise this store with other changes in *lock, plus they all have mb's anyway. Oleg.
Oleg Nesterov
2014-Mar-02 14:10 UTC
[PATCH v5 2/8] qspinlock, x86: Enable x86-64 to use queue spinlock
On 02/26, Waiman Long wrote:> > +#define _ARCH_SUPPORTS_ATOMIC_8_16_BITS_OPS > + > +/* > + * x86-64 specific queue spinlock union structure > + */ > +union arch_qspinlock { > + struct qspinlock slock; > + u8 lock; /* Lock bit */ > +};And this enables the optimized version of queue_spin_setlock(). But why does it check ACCESS_ONCE(qlock->lock) == 0 ? This is called right after queue_get_lock_qcode() returns 0, this locked should be likely unlocked. Oleg.
Seemingly Similar Threads
- [PATCH v5 0/8] qspinlock: a 4-byte queue spinlock with PV support
- [PATCH v5 0/8] qspinlock: a 4-byte queue spinlock with PV support
- [PATCH v5 0/8] qspinlock: a 4-byte queue spinlock with PV support
- [PATCH v6 00/11] qspinlock: a 4-byte queue spinlock with PV support
- [PATCH v6 00/11] qspinlock: a 4-byte queue spinlock with PV support