David Hildenbrand
2019-Sep-19 14:22 UTC
[PATCH RFC v3 0/9] virtio-mem: paravirtualized memory
Long time no RFC! I finally had time to get the next version of the Linux
driver side of virtio-mem into shape, incorporating ideas and feedback from
previous discussions.
This RFC is based on the series currently on the mm list:
- [PATCH 0/3] Remove __online_page_set_limits()
- [PATCH v1 0/3] mm/memory_hotplug: Export generic_online_page()
- [PATCH v4 0/8] mm/memory_hotplug: Shrink zones before removing memory
The current state (kept updated) is available on
- https://github.com/davidhildenbrand/linux/tree/virtio-mem
The basic idea of virtio-mem is to provide a flexible,
cross-architecture memory hot(un)plug solution that avoids many limitations
imposed by existing technologies, architectures, and interfaces. More
details can be found below and in linked material.
This RFC was only tested on x86-64, however, should theoretically work
on any Linux architecture that implements memory hot(un)plug - like
s390x. On x86-64, it is currently possible to add/remove memory to the
system in >= 4MB granularity. Memory hotplug works very reliable. For
memory unplug, there are no guarantees how much memory can actually get
unplugged, it depends on the setup. I have plans to improve that in the
future.
--------------------------------------------------------------------------
1. virtio-mem
--------------------------------------------------------------------------
The basic idea behind virtio-mem was presented at KVM Forum 2018. The
slides can be found at [1]. The previous RFC can be found at [2]. The
first RFC can be found at [3]. However, the concept evolved over time. The
KVM Forum slides roughly match the current design.
Patch #2 ("virtio-mem: Paravirtualized memory hotplug") contains quite
some
information, especially in "include/uapi/linux/virtio_mem.h":
Each virtio-mem device manages a dedicated region in physical address
space. Each device can belong to a single NUMA node, multiple devices
for a single NUMA node are possible. A virtio-mem device is like a
"resizable DIMM" consisting of small memory blocks that can be
plugged
or unplugged. The device driver is responsible for (un)plugging memory
blocks on demand.
Virtio-mem devices can only operate on their assigned memory region in
order to (un)plug memory. A device cannot (un)plug memory belonging to
other devices.
The "region_size" corresponds to the maximum amount of memory that
can
be provided by a device. The "size" corresponds to the amount of
memory
that is currently plugged. "requested_size" corresponds to a request
from the device to the device driver to (un)plug blocks. The
device driver should try to (un)plug blocks in order to reach the
"requested_size". It is impossible to plug more memory than
requested.
The "usable_region_size" represents the memory region that can
actually
be used to (un)plug memory. It is always at least as big as the
"requested_size" and will grow dynamically. It will only shrink when
explicitly triggered (VIRTIO_MEM_REQ_UNPLUG).
Memory in the usable region can usually be read, however, there are no
guarantees. It can happen that the device cannot process a request,
because it is busy. The device driver has to retry later.
Usually, during system resets all memory will get unplugged, so the
device driver can start with a clean state. However, in specific
scenarios (if the device is busy) it can happen that the device still
has memory plugged. The device driver can request to unplug all memory
(VIRTIO_MEM_REQ_UNPLUG) - which might take a while to succeed if the
device is busy.
--------------------------------------------------------------------------
2. Linux Implementation
--------------------------------------------------------------------------
This RFC reuses quite some existing MM infrastructure, however, has to
expose some additional functionality.
Memory blocks (e.g., 128MB) are added/removed on demand. Within these
memory blocks, subblocks (e.g., 4MB) are plugged/unplugged. The sizes
depend on the target architecture, MAX_ORDER + pageblock_order, and
the block size of a virtio-mem device.
add_memory()/try_remove_memory() is used to add/remove memory blocks.
virtio-mem will not online memory blocks itself. This has to be done by
user space, or configured into the kernel
(CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE). virtio-mem will only unplug memory
that was online to the ZONE_NORMAL. Memory is suggested to be onlined to
the ZONE_NORMAL for now.
The memory hotplug notifier is used to properly synchronize against
onlining/offlining of memory blocks and to track the states of memory
blocks (including the zone memory blocks are onlined to). Locking is
done similar to the PPC CMA implementation.
The set_online_page() callback is used to keep unplugged subblocks
of a memory block fake-offline when onlining the memory block.
generic_online_page() is used to fake-online plugged subblocks. This
handling is similar to the Hyper-V balloon driver.
PG_offline is used to mark unplugged subblocks as offline, so e.g.,
dumping tools (makedumpfile) will skip these pages. This is similar to
other balloon drivers like virtio-balloon and Hyper-V.
PG_offline + reference count of 0 [new] is now also used to mark pages as
a "skip" when offlining memory blocks. This allows to offline memory
blocks
that have partially unplugged subblocks - or are completely unplugged.
alloc_contig_range()/free_contig_range() [now exposed] is used to
unplug/plug subblocks of memory blocks the are already exposed to Linux.
offline_and_remove_memory() [new] is used to offline a fully unplugged
memory block and remove it from Linux.
A lot of additional information can be found in the separate patches and
as comments in the code itself.
--------------------------------------------------------------------------
3. Major changes since last RFC
--------------------------------------------------------------------------
A lot of things changed, especially also on the QEMU + virtio side. The
biggest changes on the Linux driver side are:
- Onlining/offlining of subblocks is now emulated on top of memory blocks.
set_online_page()+alloc_contig_range()+free_contig_range() is now used
for that. Core MM does not have to be modified and will continue to
online/offline full memory blocks.
- Onlining/offlining of memory blocks is no longer performed by virtio-mem.
- Pg_offline is upstream and can be used. It is also used to allow
offlining of partially unplugged memory blocks.
- Memory block states + subblocks are now tracked more space-efficient.
- Proper kexec(), kdump(), driver unload, driver reload, ZONE_MOVABLE, ...
handling.
--------------------------------------------------------------------------
4. Future work
--------------------------------------------------------------------------
The separate patches contain a lot of future work items. One of the next
steps is to make memory unplug more likely to succeed - currently, there
are no guarantees on how much memory can get unplugged again. I have
various ideas on how to limit fragmentation of all memory blocks that
virtio-mem added.
Memory hotplug:
- Reduce the amount of memory resources if that turnes out to be an
issue. Or try to speed up relevant code paths to deal with many
resources.
- Allocate the vmemmap from the added memory. Makes hotplug more likely
to succeed, the vmemmap is stored on the same NUMA node and that
unmovable memory will later not hinder unplug.
Memory hotunplug:
- Performance improvements:
-- Sense (lockless) if it make sense to try alloc_contig_range() at all
before directly trying to isolate and taking locks.
-- Try to unplug bigger chunks if possible first.
-- Identify free areas first, that don't have to be evacuated.
- Make unplug more likely to succeed:
-- The "issue" is that in the ZONE_NORMAL, the buddy will randomly
allocate memory. Only pageblocks somewhat limit fragmentation,
however we would want to limit fragmentation on subblock granularity
and even memory block granularity. One idea is to have a new
ZONE_PREFER_MOVABLE. Memory blocks will then be onlined to ZONE_NORMAL
/ ZONE_PREFER_MOVABLE in a certain ratio per node (e.g.,
1:4). This makes unplug of quite some memory likely to succeed in most
setups. ZONE_PREFER_MOVABLE is then a mixture of ZONE_NORMAL and
ZONE_MOVABlE. Especially, movable data can end up on that zone, but
only if really required - avoiding running out of memory on ZONE
imbalances. The zone fallback order would be
MOVABLE=>PREFER_MOVABLE=>HIGHMEM=>NORMAL=>PREFER_MOVABLE=>DMA32=>DMA
-- Allocate memmap from added memory. This way, less unmovable data can
end up on the memory blocks.
-- Call drop_slab() before trying to unplug. Eventually shrink other
caches.
- Better retry handling in case memory is busy. We certainly don't want
to try for ever in a short interval to try to get some memory back.
- OOM handling, e.g., via an OOM handler.
--------------------------------------------------------------------------
5. Example Usage
--------------------------------------------------------------------------
A very basic QEMU prototype (kept updated) is available at:
- https://github.com/davidhildenbrand/qemu/tree/virtio-mem
It lacks various features, however works to test the guest driver side:
- No support for resizable memory regions / memory backends
- No protection of unplugged memory (esp., userfaultfd-wp)
- No dump/migration/XXX optimizations to skip over unplugged memory
Start QEMU with two virtio-mem devices (one per NUMA node):
$ qemu-system-x86_64 -m 4G,maxmem=20G \
-smp sockets=2,cores=2 \
-numa node,nodeid=0,cpus=0-1 -numa node,nodeid=1,cpus=2-3 \
[...]
-object memory-backend-ram,id=mem0,size=8G \
-device virtio-mem-pci,id=vm0,memdev=mem0,node=0,requested-size=128M \
-object memory-backend-ram,id=mem1,size=8G \
-device virtio-mem-pci,id=vm1,memdev=mem1,node=1,requested-size=80M
Query the configuration:
QEMU 4.1.50 monitor - type 'help' for more information
(qemu) info memory-devices
Memory device [virtio-mem]: "vm0"
memaddr: 0x140000000
node: 0
requested-size: 134217728
size: 134217728
max-size: 8589934592
block-size: 2097152
memdev: /objects/mem0
Memory device [virtio-mem]: "vm1"
memaddr: 0x340000000
node: 1
requested-size: 83886080
size: 83886080
max-size: 8589934592
block-size: 2097152
memdev: /objects/mem1
Add some memory to node 1:
QEMU 4.1.50 monitor - type 'help' for more information
(qemu) qom-set vm1 requested-size 1G
Remove some memory from node 0:
QEMU 4.1.50 monitor - type 'help' for more information
(qemu) qom-set vm0 requested-size 64M
Query the configuration:
(qemu) info memory-devices
Memory device [virtio-mem]: "vm0"
memaddr: 0x140000000
node: 0
requested-size: 67108864
size: 67108864
max-size: 8589934592
block-size: 2097152
memdev: /objects/mem0
Memory device [virtio-mem]: "vm1"
memaddr: 0x340000000
node: 1
requested-size: 1073741824
size: 1073741824
max-size: 8589934592
block-size: 2097152
memdev: /objects/mem1
--------------------------------------------------------------------------
6. Q/A
--------------------------------------------------------------------------
Q: Why add/remove parts ("subblocks") of memory blocks/sections?
A: Flexibility (section size depends on the architecture) - e.g., some
architectures have a section size of 2GB. Also, the memory block size
is variable (e.g., on x86-64). I want to avoid any such restrictions.
Some use cases want to add/remove memory in smaller granularities to a
VM (e.g., the Hyper-V balloon also implements this) - especially smaller
VMs like used for kata containers. Also, on memory unplug, it is more
reliable to free-up and unplug multiple small chunks instead
of one big chunk. E.g., if one page of a DIMM is either unmovable or
pinned, the DIMM can't get unplugged. This approach is basically a
compromise between DIMM-based memory hot(un)plug and balloon
inflation/deflation, which works mostly on page granularity.
Q: Why care about memory blocks?
A: They are the way to tell user space about new memory. This way,
memory can get onlined/offlined by user space. Also, e.g., kdump
relies on udev events to reload kexec when memory blocks are
onlined/offlined. Memory blocks are the "real" memory hot(un)plug
granularity. Everything that's smaller has to be emulated "on
top".
Q: Won't memory unplug of subblocks fragment memory?
A: Yes and no. Unplugging e.g., >=4MB subblocks on x86-64 will not really
fragment memory like unplugging random pages like a balloon driver does.
Buddy merging will not be limited. However, any allocation that requires
bigger consecutive memory chunks (e.g., gigantic pages) might observe
the fragmentation. Possible solutions: Allocate gigantic huge pages
before unplugging memory, don't unplug memory, combine virtio-mem with
DIMM based memory or bigger initial memory. Remember, a virtio-mem
device will only unplug on the memory range it manages, not on other
DIMMs. Unplug of single memory blocks will result in similar
fragmentation in respect to gigantic huge pages.
Q: How reliable is memory unplug?
A: There are no guarantees on how much memory can get unplugged
again. However, it is more likely to find 4MB chunks to unplug than
e.g., 128MB chunks. If memory is terribly fragmented, there is nothing
we can do - for now. I consider memory hotplug the first primary use
of virtio-mem. Memory unplug might usually work, but we want to improve
the performance and the amount of memory we can actually unplug later.
Q: Why not unplug from the ZONE_MOVABLE?
A: Unplugged memory chunks are unmovable. Unmovable data must not end up
on the ZONE_MOVABLE - similar to gigantic pages - they will never be
allocated from ZONE_MOVABLE. Teaching MM to move unplugged chunks within
a device might be problematic and will require a new guest->hypervisor
command to move unplugged chunks. virtio-mem added memory can be onlined
to the ZONE_MOVABLE, but subblocks will not get unplugged from it.
Q: How big should the initial (!virtio-mem) memory of a VM be?
A: virtio-mem memory will not go to the DMA zones. So to avoid running out
of DMA memory, I suggest something like 2-3GB on x86-64. But many
VMs can most probably deal with less DMA memory - depends on the use
case.
[1]
https://events.linuxfoundation.org/wp-content/uploads/2017/12/virtio-mem-Paravirtualized-Memory-David-Hildenbrand-Red-Hat-1.pdf
[2] https://lwn.net/Articles/755423/
[3] https://lists.gnu.org/archive/html/qemu-devel/2017-06/msg03870.html
---
David Hildenbrand (9):
ACPI: NUMA: export pxm_to_node
virtio-mem: Paravirtualized memory hotplug
virtio-mem: Paravirtualized memory hotunplug part 1
mm: Export alloc_contig_range() / free_contig_range()
virtio-mem: Paravirtualized memory hotunplug part 2
mm: Allow to offline PageOffline() pages with a reference count of 0
virtio-mem: Allow to offline partially unplugged memory blocks
mm/memory_hotplug: Introduce offline_and_remove_memory()
virtio-mem: Offline and remove completely unplugged memory blocks
drivers/acpi/numa.c | 1 +
drivers/virtio/Kconfig | 18 +
drivers/virtio/Makefile | 1 +
drivers/virtio/virtio_mem.c | 1900 +++++++++++++++++++++++++++++++
include/linux/memory_hotplug.h | 1 +
include/linux/page-flags.h | 4 +
include/linux/page-isolation.h | 4 +-
include/uapi/linux/virtio_ids.h | 1 +
include/uapi/linux/virtio_mem.h | 204 ++++
mm/memory_hotplug.c | 44 +-
mm/page_alloc.c | 24 +-
mm/page_isolation.c | 18 +-
mm/swap.c | 9 +
13 files changed, 2219 insertions(+), 10 deletions(-)
create mode 100644 drivers/virtio/virtio_mem.c
create mode 100644 include/uapi/linux/virtio_mem.h
--
2.21.0
Will be needed by virtio-mem to identify the node from a pxm.
Cc: "Rafael J. Wysocki" <rjw at rjwysocki.net>
Cc: Len Brown <lenb at kernel.org>
Cc: linux-acpi at vger.kernel.org
Signed-off-by: David Hildenbrand <david at redhat.com>
---
drivers/acpi/numa.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/acpi/numa.c b/drivers/acpi/numa.c
index eadbf90e65d1..d5847fa7ac69 100644
--- a/drivers/acpi/numa.c
+++ b/drivers/acpi/numa.c
@@ -35,6 +35,7 @@ int pxm_to_node(int pxm)
return NUMA_NO_NODE;
return pxm_to_node_map[pxm];
}
+EXPORT_SYMBOL(pxm_to_node);
int node_to_pxm(int node)
{
--
2.21.0
David Hildenbrand
2019-Sep-19 14:22 UTC
[PATCH RFC v3 2/9] virtio-mem: Paravirtualized memory hotplug
Each virtio-mem device owns exactly one memory region. It is responsible
for adding/removing memory from that memory region on request.
When the device driver starts up, the requested amount of memory is
queried and then plugged to Linux. On request, further memory can be
plugged or unplugged. This patch only implements the plugging part.
On x86-64, memory can currently be plugged in 4MB ("subblock")
granularity.
When required, a new memory block will be added (e.g., usually 128MB on
x86-64) in order to plug more subblocks. Only x86-64 was tested for now.
The online_page callback is used to keep unplugged subblocks offline
when onlining memory - similar to the Hyper-V balloon driver. Unplugged
pages are marked PG_offline, to tell dump tools (e.g., makedumpfile) to
skip them.
User space is usually responsible for onlining the added memory. The
memory hotplug notifier is used to synchronize virtio-mem activity
against memory onlining/offlining.
Each virtio-mem device can belong to a NUMA node, which allows us to
easily add/remove small chunks of memory to/from a specific NUMA node by
using multiple virtio-mem devices. Something that works even when the
guest has no idea about the NUMA topology.
One way to view virtio-mem is as a "resizable DIMM" or a DIMM with
many
"sub-DIMMS".
This patch directly introduces the basic infrastructure to implement memory
unplug. Especially the memory block states and subblock bitmaps will be
heavily used there.
Notes:
- Memory blocks that are partally unplugged must not be onlined to the
MOVABLE_ZONE. They contain unmovable parts and might e.g., result
in warnings when trying to offline them.
- In the kdump kernel, we don't want to load the driver, to not mess
with memory to be dumped.
- In case memory is to be onlined by user space, we limit the amount of
offline memory blocks, to not run out of memory.
- Suspend/Hibernate is not supported due to the way virtio-mem devices
behave. Limited support might be possible in the future.
- Reloading the device driver is not supported.
- When unloading the driver, we have to remove partially plugged offline
memory blocks. Otherwise they could get fully onlined later on, when
we no longer have control over via the online_page callback.
- As the hypervisor might suddenly be busy (during reboot, in-between
requests, when adding of memory fails), we have to take care of some
corner cases - especially virtio_mem_unplug_pending_mb() and
virtio_mem_send_unplug_all_request(). The hypervisor could for example
be busy if it is currently migrating the guest.
Future work:
- Reduce the amount of memory resources if that turnes out to be an
issue. Or try to speed up relevant code paths to deal with many
resources.
- Allocate the vmemmap from the added memory. Makes hotplug more likely
to succeed, the vmemmap is stored on the same NUMA node and that
unmovable memory will later not hinder unplug.
Cc: "Michael S. Tsirkin" <mst at redhat.com>
Cc: Jason Wang <jasowang at redhat.com>
Cc: Oscar Salvador <osalvador at suse.de>
Cc: Michal Hocko <mhocko at kernel.org>
Cc: Igor Mammedov <imammedo at redhat.com>
Cc: Dave Young <dyoung at redhat.com>
Cc: Andrew Morton <akpm at linux-foundation.org>
Cc: Dan Williams <dan.j.williams at intel.com>
Cc: Pavel Tatashin <pasha.tatashin at soleen.com>
Cc: Stefan Hajnoczi <stefanha at redhat.com>
Cc: Vlastimil Babka <vbabka at suse.cz>
Signed-off-by: David Hildenbrand <david at redhat.com>
---
drivers/virtio/Kconfig | 17 +
drivers/virtio/Makefile | 1 +
drivers/virtio/virtio_mem.c | 1580 +++++++++++++++++++++++++++++++
include/uapi/linux/virtio_ids.h | 1 +
include/uapi/linux/virtio_mem.h | 204 ++++
5 files changed, 1803 insertions(+)
create mode 100644 drivers/virtio/virtio_mem.c
create mode 100644 include/uapi/linux/virtio_mem.h
diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig
index 078615cf2afc..294720d53057 100644
--- a/drivers/virtio/Kconfig
+++ b/drivers/virtio/Kconfig
@@ -64,6 +64,23 @@ config VIRTIO_BALLOON
If unsure, say M.
+config VIRTIO_MEM
+ tristate "Virtio mem driver"
+ default m
+ depends on X86_64
+ depends on VIRTIO
+ depends on MEMORY_HOTPLUG_SPARSE
+ depends on MEMORY_HOTREMOVE
+ help
+ This driver provides access to virtio-mem paravirtualized memory
+ devices, allowing to hotplug and hotunplug memory.
+
+ This driver is was only tested under x86-64, but should
+ theoretically work on all architectures that support memory
+ hotplug and hotremove.
+
+ If unsure, say M.
+
config VIRTIO_INPUT
tristate "Virtio input driver"
depends on VIRTIO
diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile
index 3a2b5c5dcf46..906d5a00ac85 100644
--- a/drivers/virtio/Makefile
+++ b/drivers/virtio/Makefile
@@ -6,3 +6,4 @@ virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o
virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o
obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o
obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o
+obj-$(CONFIG_VIRTIO_MEM) += virtio_mem.o
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
new file mode 100644
index 000000000000..597fab4933f6
--- /dev/null
+++ b/drivers/virtio/virtio_mem.c
@@ -0,0 +1,1580 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Virtio-mem device driver.
+ *
+ * Copyright Red Hat, Inc. 2019
+ *
+ * Author(s): David Hildenbrand <david at redhat.com>
+ */
+
+#include <linux/virtio.h>
+#include <linux/virtio_mem.h>
+#include <linux/workqueue.h>
+#include <linux/module.h>
+#include <linux/mm.h>
+#include <linux/memory_hotplug.h>
+#include <linux/memory.h>
+#include <linux/hrtimer.h>
+#include <linux/crash_dump.h>
+#include <linux/mutex.h>
+#include <linux/bitmap.h>
+
+#include <acpi/acpi_numa.h>
+
+enum virtio_mem_mb_state {
+ /* Unplugged, not added to Linux. Can be reused later. */
+ VIRTIO_MEM_MB_STATE_UNUSED = 0,
+ /* (Partially) plugged, not added to Linux. Error on add_memory(). */
+ VIRTIO_MEM_MB_STATE_PLUGGED,
+ /* Fully plugged, fully added to Linux, offline. */
+ VIRTIO_MEM_MB_STATE_OFFLINE,
+ /* Paritally plugged, fully added to Linux, offline. */
+ VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL,
+ /* Fully plugged, fully added to Linux, online (!ZONE_MOVABLE). */
+ VIRTIO_MEM_MB_STATE_ONLINE,
+ /* Partially plugged, fully added to Linux, online (!ZONE_MOVABLE). */
+ VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL,
+ /*
+ * Fully plugged, fully added to Linux, online (ZONE_MOVABLE).
+ * We are not allowed to allocate (unplug) parts of this block that
+ * are not movable (similar to gigantic pages). We will never allow
+ * to online OFFLINE_PARTIAL to ZONE_MOVABLE (as they would contain
+ * unmovable parts).
+ */
+ VIRTIO_MEM_MB_STATE_ONLINE_MOVABLE,
+ VIRTIO_MEM_MB_STATE_COUNT
+};
+
+struct virtio_mem {
+ struct virtio_device *vdev;
+
+ /* We might first have to unplug all memory when starting up. */
+ bool unplug_all_required;
+
+ /* Workqueue that processes the plug/unplug requests. */
+ struct work_struct wq;
+ atomic_t config_changed;
+
+ /* Virtqueue for guest->host requests. */
+ struct virtqueue *vq;
+
+ /* Wait for a host response to a guest request. */
+ wait_queue_head_t host_resp;
+
+ /* Space for one guest request and the host response. */
+ struct virtio_mem_req req;
+ struct virtio_mem_resp resp;
+
+ /* The current size of the device. */
+ uint64_t plugged_size;
+ /* The requested size of the device. */
+ uint64_t requested_size;
+
+ /* The device block size (for communicating with the device). */
+ uint32_t device_block_size;
+ /* The translated node id. NUMA_NO_NODE in case not specified. */
+ int nid;
+ /* Physical start address of the memory region. */
+ uint64_t addr;
+
+ /* The subblock size. */
+ uint32_t subblock_size;
+ /* The number of subblocks per memory block. */
+ uint32_t nb_sb_per_mb;
+
+ /* Id of the first memory block of this device. */
+ unsigned long first_mb_id;
+ /* Id of the last memory block of this device. */
+ unsigned long last_mb_id;
+ /* Id of the last usable memory block of this device. */
+ unsigned long last_usable_mb_id;
+ /* Id of the next memory bock to prepare when needed. */
+ unsigned long next_mb_id;
+
+ /* Summary of all memory block states. */
+ unsigned long nb_mb_state[VIRTIO_MEM_MB_STATE_COUNT];
+#define VIRTIO_MEM_NB_OFFLINE_THRESHOLD 10
+
+ /*
+ * One byte state per memory block.
+ *
+ * Allocated via vmalloc(). When preparing new blocks, resized
+ * (alloc+copy+free) when needed (crossing pages with the next mb).
+ * (when crossing pages).
+ *
+ * With 128MB memory blocks, we have states for 512GB of memory on one
+ * page.
+ */
+ uint8_t *mb_state;
+
+ /*
+ * $nb_sb_per_mb bit per memory block. Handled similar to mb_state.
+ *
+ * With 4MB subblocks, we manage 128GB of memory in one page.
+ */
+ unsigned long *sb_bitmap;
+
+ /*
+ * Mutex that protects the nb_mb_state, mb_state, and sb_bitmap.
+ *
+ * When this lock is held the pointers can't change, ONLINE and
+ * OFFLINE blocks can't change the state and no subblocks will get
+ * plugged.
+ */
+ struct mutex hotplug_mutex;
+ bool hotplug_active;
+
+ /* An error ocurred we cannot handle - stop processing requests. */
+ bool broken;
+
+ /* The driver is being removed. */
+ spinlock_t removal_lock;
+ bool removing;
+
+ /* Timer for retrying to plug/unplug memory. */
+ struct hrtimer retry_timer;
+ ktime_t retry_interval;
+#define VIRTIO_MEM_RETRY_MS 10000
+
+ /* Memory notifier (online/offline events). */
+ struct notifier_block memory_notifier;
+
+ /* Next device in the list of virtio-mem devices. */
+ struct list_head next;
+};
+
+/*
+ * We have to share a single online_page callback among all virtio-mem
+ * devices. We use RCU to iterate the list in the callback.
+ */
+static DEFINE_MUTEX(virtio_mem_mutex);
+static LIST_HEAD(virtio_mem_devices);
+
+static void virtio_mem_online_page_cb(struct page *page, unsigned int order);
+
+/*
+ * Register a virtio-mem device so it will be considered for the online_page
+ * callback.
+ */
+static int register_virtio_mem_device(struct virtio_mem *vm)
+{
+ int rc = 0;
+
+ /* First device registers the callback. */
+ mutex_lock(&virtio_mem_mutex);
+ if (list_empty(&virtio_mem_devices))
+ rc = set_online_page_callback(&virtio_mem_online_page_cb);
+ if (!rc)
+ list_add_rcu(&vm->next, &virtio_mem_devices);
+ mutex_unlock(&virtio_mem_mutex);
+
+ return rc;
+}
+
+/*
+ * Unregister a virtio-mem device so it will no longer be considered for the
+ * online_page callback.
+ */
+static void unregister_virtio_mem_device(struct virtio_mem *vm)
+{
+ /* Last device unregisters the callback. */
+ mutex_lock(&virtio_mem_mutex);
+ list_del_rcu(&vm->next);
+ if (list_empty(&virtio_mem_devices))
+ restore_online_page_callback(&virtio_mem_online_page_cb);
+ mutex_unlock(&virtio_mem_mutex);
+
+ synchronize_rcu();
+}
+
+/*
+ * Calculate the memory block id of a given address.
+ */
+static unsigned long virtio_mem_phys_to_mb_id(unsigned long addr)
+{
+ return addr / memory_block_size_bytes();
+}
+
+/*
+ * Calculate the physical start address of a given memory block id.
+ */
+static unsigned long virtio_mem_mb_id_to_phys(unsigned long mb_id)
+{
+ return mb_id * memory_block_size_bytes();
+}
+
+/*
+ * Calculate the subblock id of a given address.
+ */
+static unsigned long virtio_mem_phys_to_sb_id(struct virtio_mem *vm,
+ unsigned long addr)
+{
+ const unsigned long mb_id = virtio_mem_phys_to_mb_id(addr);
+ const unsigned long mb_addr = virtio_mem_mb_id_to_phys(mb_id);
+
+ return (addr - mb_addr) / vm->subblock_size;
+}
+
+/*
+ * Set the state of a memory block, taking care of the state counter.
+ */
+static void virtio_mem_mb_set_state(struct virtio_mem *vm, unsigned long mb_id,
+ enum virtio_mem_mb_state state)
+{
+ const unsigned long idx = mb_id - vm->first_mb_id;
+ enum virtio_mem_mb_state old_state;
+
+ old_state = vm->mb_state[idx];
+ vm->mb_state[idx] = state;
+
+ BUG_ON(vm->nb_mb_state[old_state] == 0);
+ vm->nb_mb_state[old_state]--;
+ vm->nb_mb_state[state]++;
+}
+
+/*
+ * Get the state of a memory block, taking care of the state counter.
+ */
+static enum virtio_mem_mb_state virtio_mem_mb_get_state(struct virtio_mem *vm,
+ unsigned long mb_id)
+{
+ const unsigned long idx = mb_id - vm->first_mb_id;
+
+ return vm->mb_state[idx];
+}
+
+/*
+ * Prepare the state array for the next memory block.
+ */
+static int virtio_mem_mb_state_prepare_next_mb(struct virtio_mem *vm)
+{
+ unsigned long old_bytes = vm->next_mb_id - vm->first_mb_id + 1;
+ unsigned long new_bytes = vm->next_mb_id - vm->first_mb_id + 2;
+ int old_pages = PFN_UP(old_bytes);
+ int new_pages = PFN_UP(new_bytes);
+ uint8_t *new_mb_state;
+
+ if (vm->mb_state && old_pages == new_pages)
+ return 0;
+
+ new_mb_state = vzalloc(new_pages * PAGE_SIZE);
+ if (!new_mb_state)
+ return -ENOMEM;
+
+ mutex_lock(&vm->hotplug_mutex);
+ if (vm->mb_state)
+ memcpy(new_mb_state, vm->mb_state, old_pages * PAGE_SIZE);
+ vfree(vm->mb_state);
+ vm->mb_state = new_mb_state;
+ mutex_unlock(&vm->hotplug_mutex);
+
+ return 0;
+}
+
+#define virtio_mem_for_each_mb_state(_vm, _mb_id, _state) \
+ for (_mb_id = _vm->first_mb_id; \
+ _mb_id < _vm->next_mb_id && _vm->nb_mb_state[_state]; \
+ _mb_id++) \
+ if (virtio_mem_mb_get_state(_vm, _mb_id) == _state)
+
+/*
+ * Mark all selected subblocks plugged.
+ *
+ * Will not modify the state of the memory block.
+ */
+static void virtio_mem_mb_set_sb_plugged(struct virtio_mem *vm,
+ unsigned long mb_id, int sb_id,
+ int count)
+{
+ const int bit = (mb_id - vm->first_mb_id) * vm->nb_sb_per_mb + sb_id;
+
+ __bitmap_set(vm->sb_bitmap, bit, count);
+}
+
+/*
+ * Mark all selected subblocks unplugged.
+ *
+ * Will not modify the state of the memory block.
+ */
+static void virtio_mem_mb_set_sb_unplugged(struct virtio_mem *vm,
+ unsigned long mb_id, int sb_id,
+ int count)
+{
+ const int bit = (mb_id - vm->first_mb_id) * vm->nb_sb_per_mb + sb_id;
+
+ __bitmap_clear(vm->sb_bitmap, bit, count);
+}
+
+/*
+ * Test if all selected subblocks are plugged.
+ */
+static bool virtio_mem_mb_test_sb_plugged(struct virtio_mem *vm,
+ unsigned long mb_id, int sb_id,
+ int count)
+{
+ const int bit = (mb_id - vm->first_mb_id) * vm->nb_sb_per_mb + sb_id;
+
+ if (count == 1)
+ return test_bit(bit, vm->sb_bitmap);
+
+ /* TODO: Helper similar to bitmap_set() */
+ return find_next_zero_bit(vm->sb_bitmap, bit + count, bit) >+ bit
+ count;
+}
+
+/*
+ * Find the first plugged subblock. Returns vm->nb_sb_per_mb in case there
is
+ * none.
+ */
+static int virtio_mem_mb_first_plugged_sb(struct virtio_mem *vm,
+ unsigned long mb_id)
+{
+ const int bit = (mb_id - vm->first_mb_id) * vm->nb_sb_per_mb;
+
+ return find_next_bit(vm->sb_bitmap, bit + vm->nb_sb_per_mb, bit) - bit;
+}
+
+/*
+ * Find the first unplugged subblock. Returns vm->nb_sb_per_mb in case there
is
+ * none.
+ */
+static int virtio_mem_mb_first_unplugged_sb(struct virtio_mem *vm,
+ unsigned long mb_id)
+{
+ const int bit = (mb_id - vm->first_mb_id) * vm->nb_sb_per_mb;
+
+ return find_next_zero_bit(vm->sb_bitmap, bit + vm->nb_sb_per_mb, bit) -
+ bit;
+}
+
+/*
+ * Prepare the subblock bitmap for the next memory block.
+ */
+static int virtio_mem_sb_bitmap_prepare_next_mb(struct virtio_mem *vm)
+{
+ const unsigned long old_nb_mb = vm->next_mb_id - vm->first_mb_id;
+ const unsigned long old_nb_bits = old_nb_mb * vm->nb_sb_per_mb;
+ const unsigned long new_nb_bits = (old_nb_mb + 1) * vm->nb_sb_per_mb;
+ int old_pages = PFN_UP(BITS_TO_LONGS(old_nb_bits) * sizeof(long));
+ int new_pages = PFN_UP(BITS_TO_LONGS(new_nb_bits) * sizeof(long));
+ unsigned long *new_sb_bitmap, *old_sb_bitmap;
+
+ if (vm->sb_bitmap && old_pages == new_pages)
+ return 0;
+
+ new_sb_bitmap = vzalloc(new_pages * PAGE_SIZE);
+ if (!new_sb_bitmap)
+ return -ENOMEM;
+
+ mutex_lock(&vm->hotplug_mutex);
+ if (new_sb_bitmap)
+ memcpy(new_sb_bitmap, vm->sb_bitmap, old_pages * PAGE_SIZE);
+
+ old_sb_bitmap = vm->sb_bitmap;
+ vm->sb_bitmap = new_sb_bitmap;
+ mutex_unlock(&vm->hotplug_mutex);
+
+ vfree(old_sb_bitmap);
+ return 0;
+}
+
+/*
+ * Try to add a memory block to Linux. This will usually only fail
+ * if out of memory.
+ *
+ * Must not be called with the vm->hotplug_mutex held (possible deadlock
with
+ * onlining code).
+ *
+ * Will not modify the state of the memory block.
+ */
+static int virtio_mem_mb_add(struct virtio_mem *vm, unsigned long mb_id)
+{
+ const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id);
+ int nid = vm->nid;
+
+ if (nid == NUMA_NO_NODE)
+ nid = memory_add_physaddr_to_nid(addr);
+
+ dev_dbg(&vm->vdev->dev, "adding memory block: %lu\n",
mb_id);
+ return add_memory(nid, addr, memory_block_size_bytes());
+}
+
+/*
+ * Try to remove a memory block from Linux. Will only fail if the memory block
+ * is not offline.
+ *
+ * Must not be called with the vm->hotplug_mutex held (possible deadlock
with
+ * onlining code).
+ *
+ * Will not modify the state of the memory block.
+ */
+static int virtio_mem_mb_remove(struct virtio_mem *vm, unsigned long mb_id)
+{
+ const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id);
+ int nid = vm->nid;
+
+ if (nid == NUMA_NO_NODE)
+ nid = memory_add_physaddr_to_nid(addr);
+
+ dev_dbg(&vm->vdev->dev, "removing memory block: %lu\n",
mb_id);
+ return remove_memory(nid, addr, memory_block_size_bytes());
+}
+
+/*
+ * Trigger the workqueue so the device can perform its magic.
+ */
+static void virtio_mem_retry(struct virtio_mem *vm)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&vm->removal_lock, flags);
+ if (!vm->removing)
+ queue_work(system_freezable_wq, &vm->wq);
+ spin_unlock_irqrestore(&vm->removal_lock, flags);
+}
+
+static int virtio_mem_translate_node_id(struct virtio_mem *vm, uint16_t
node_id)
+{
+ int node = NUMA_NO_NODE;
+
+ if (IS_ENABLED(CONFIG_NUMA) && IS_ENABLED(CONFIG_ACPI_NUMA) &&
+ virtio_has_feature(vm->vdev, VIRTIO_MEM_F_ACPI_PXM))
+ node = pxm_to_node(node_id);
+ return node;
+}
+
+/*
+ * Test if a virtio-mem device overlaps with the given range. Can be called
+ * from (notifier) callbacks lockless.
+ */
+static bool virtio_mem_overlaps_range(struct virtio_mem *vm,
+ unsigned long start, unsigned long size)
+{
+ unsigned long dev_start = virtio_mem_mb_id_to_phys(vm->first_mb_id);
+ unsigned long dev_end = virtio_mem_mb_id_to_phys(vm->last_mb_id) +
+ memory_block_size_bytes();
+ unsigned long end = start + size;
+
+ return (start >= dev_start && start < dev_end) ||
+ (end > dev_start && end <= dev_end) ||
+ (start < dev_start && end > dev_end);
+}
+
+/*
+ * Test if a virtio-mem device owns a memory block. Can be called from
+ * (notifier) callbacks lockless.
+ */
+static bool virtio_mem_owned_mb(struct virtio_mem *vm, unsigned long mb_id)
+{
+ return mb_id >= vm->first_mb_id && mb_id <=
vm->last_mb_id;
+}
+
+static int virtio_mem_notify_going_online(struct virtio_mem *vm,
+ unsigned long mb_id,
+ enum zone_type zone)
+{
+ switch (virtio_mem_mb_get_state(vm, mb_id)) {
+ case VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL:
+ /*
+ * We won't allow to online a partially plugged memory block
+ * to the MOVABLE zone - it would contain unmovable parts.
+ */
+ if (zone == ZONE_MOVABLE) {
+ dev_warn_ratelimited(&vm->vdev->dev,
+ "memory block has holes, MOVABLE not supported\n");
+ return NOTIFY_BAD;
+ }
+ return NOTIFY_OK;
+ case VIRTIO_MEM_MB_STATE_OFFLINE:
+ return NOTIFY_OK;
+ default:
+ break;
+ }
+ dev_warn_ratelimited(&vm->vdev->dev,
+ "memory block onlining denied\n");
+ return NOTIFY_BAD;
+}
+
+static void virtio_mem_notify_offline(struct virtio_mem *vm,
+ unsigned long mb_id)
+{
+ switch (virtio_mem_mb_get_state(vm, mb_id)) {
+ case VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL:
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL);
+ break;
+ case VIRTIO_MEM_MB_STATE_ONLINE:
+ case VIRTIO_MEM_MB_STATE_ONLINE_MOVABLE:
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE);
+ break;
+ default:
+ BUG();
+ break;
+ }
+}
+
+static void virtio_mem_notify_online(struct virtio_mem *vm, unsigned long
mb_id,
+ enum zone_type zone)
+{
+ unsigned long nb_offline;
+
+ switch (virtio_mem_mb_get_state(vm, mb_id)) {
+ case VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL:
+ BUG_ON(zone == ZONE_MOVABLE);
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL);
+ break;
+ case VIRTIO_MEM_MB_STATE_OFFLINE:
+ if (zone == ZONE_MOVABLE)
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE_MOVABLE);
+ else
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE);
+ break;
+ default:
+ BUG();
+ break;
+ }
+ nb_offline = vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE] +
+ vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL];
+
+ /* see if we can add new blocks now that we onlined one block */
+ if (nb_offline == VIRTIO_MEM_NB_OFFLINE_THRESHOLD - 1)
+ virtio_mem_retry(vm);
+}
+
+/*
+ * This callback will either be called synchonously from add_memory() or
+ * asynchronously (e.g., triggered via user space). We have to be careful
+ * with locking when calling add_memory().
+ */
+static int virtio_mem_memory_notifier_cb(struct notifier_block *nb,
+ unsigned long action, void *arg)
+{
+ struct virtio_mem *vm = container_of(nb, struct virtio_mem,
+ memory_notifier);
+ struct memory_notify *mhp = arg;
+ const unsigned long start = PFN_PHYS(mhp->start_pfn);
+ const unsigned long size = PFN_PHYS(mhp->nr_pages);
+ const unsigned long mb_id = virtio_mem_phys_to_mb_id(start);
+ enum zone_type zone;
+ int rc = NOTIFY_OK;
+
+ /*
+ * Memory is usually onlined/offlined in memory block granularity
+ * (via device_online()/device_offline()). This means, we
+ * cannot cross virtio-mem device boundaries and memory block
+ * boundaries.
+ *
+ * Only obscure users (powernv/memtrace.c) play dirty games
+ * manually with offline_pages() with multiple memory blocks -
+ * let's NACK these requests in case virtio-mem is involved.
+ */
+ if (size != memory_block_size_bytes() ||
+ !IS_ALIGNED(start, memory_block_size_bytes())) {
+ if (virtio_mem_overlaps_range(vm, start, size))
+ return NOTIFY_BAD;
+ return NOTIFY_DONE;
+ }
+
+ if (!virtio_mem_owned_mb(vm, mb_id))
+ return NOTIFY_DONE;
+
+ switch (action) {
+ case MEM_GOING_OFFLINE:
+ spin_lock_irq(&vm->removal_lock);
+ if (vm->removing)
+ rc = notifier_to_errno(-EBUSY);
+ spin_unlock_irq(&vm->removal_lock);
+ if (rc == NOTIFY_OK) {
+ mutex_lock(&vm->hotplug_mutex);
+ vm->hotplug_active = true;
+ }
+ break;
+ case MEM_GOING_ONLINE:
+ spin_lock_irq(&vm->removal_lock);
+ if (vm->removing)
+ rc = notifier_to_errno(-EBUSY);
+ spin_unlock_irq(&vm->removal_lock);
+ if (rc == NOTIFY_OK) {
+ mutex_lock(&vm->hotplug_mutex);
+ vm->hotplug_active = true;
+ zone = page_zonenum(pfn_to_page(mhp->start_pfn));
+ rc = virtio_mem_notify_going_online(vm, mb_id, zone);
+ }
+ break;
+ case MEM_OFFLINE:
+ virtio_mem_notify_offline(vm, mb_id);
+ vm->hotplug_active = false;
+ mutex_unlock(&vm->hotplug_mutex);
+ break;
+ case MEM_ONLINE:
+ zone = page_zonenum(pfn_to_page(mhp->start_pfn));
+ virtio_mem_notify_online(vm, mb_id, zone);
+ vm->hotplug_active = false;
+ mutex_unlock(&vm->hotplug_mutex);
+ break;
+ case MEM_CANCEL_OFFLINE:
+ case MEM_CANCEL_ONLINE:
+ /* We might not get a MEM_GOING* if somebody else canceled */
+ if (vm->hotplug_active) {
+ vm->hotplug_active = false;
+ mutex_unlock(&vm->hotplug_mutex);
+ }
+ break;
+ default:
+ break;
+ }
+
+ return rc;
+}
+
+/*
+ * Set a range of pages PG_offline.
+ */
+static void virtio_mem_set_fake_offline(unsigned long pfn,
+ unsigned int nr_pages)
+{
+ for (; nr_pages--; pfn++)
+ __SetPageOffline(pfn_to_page(pfn));
+}
+
+/*
+ * Clear PG_offline from a range of pages.
+ */
+static void virtio_mem_clear_fake_offline(unsigned long pfn,
+ unsigned int nr_pages)
+{
+ for (; nr_pages--; pfn++)
+ __ClearPageOffline(pfn_to_page(pfn));
+}
+
+/*
+ * Release a range of fake-offline pages to the buddy, effectively
+ * fake-onlining them.
+ */
+static void virtio_mem_fake_online(unsigned long pfn, unsigned int nr_pages)
+{
+ const int order = MAX_ORDER - 1;
+ int i;
+
+ /*
+ * We are always called with subblock granularity, which is at least
+ * aligned to MAX_ORDER - 1. All pages in a subblock are either
+ * reserved or not.
+ */
+ BUG_ON(!IS_ALIGNED(pfn, 1 << order));
+ BUG_ON(!IS_ALIGNED(nr_pages, 1 << order));
+
+ virtio_mem_clear_fake_offline(pfn, nr_pages);
+
+ for (i = 0; i < nr_pages; i += 1 << order) {
+ struct page *page = pfn_to_page(pfn + i);
+
+ generic_online_page(page, order);
+ }
+}
+
+static void virtio_mem_online_page_cb(struct page *page, unsigned int order)
+{
+ const unsigned long addr = page_to_phys(page);
+ const unsigned long mb_id = virtio_mem_phys_to_mb_id(addr);
+ struct virtio_mem *vm;
+ int sb_id;
+
+ /*
+ * We exploit here that subblocks have at least MAX_ORDER - 1
+ * size/alignment and that this callback is is called with such a
+ * size/alignment. So we cannot cross subblocks and therefore
+ * also not memory blocks.
+ */
+ rcu_read_lock();
+ list_for_each_entry_rcu(vm, &virtio_mem_devices, next) {
+ if (!virtio_mem_owned_mb(vm, mb_id))
+ continue;
+
+ sb_id = virtio_mem_phys_to_sb_id(vm, addr);
+ /*
+ * If plugged, online the pages, otherwise, set them fake
+ * offline (PageOffline).
+ */
+ if (virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id, 1))
+ generic_online_page(page, order);
+ else
+ virtio_mem_set_fake_offline(PFN_DOWN(addr), 1 << order);
+ rcu_read_unlock();
+ return;
+ }
+ rcu_read_unlock();
+
+ /* not virtio-mem memory, but e.g., a DIMM. online it */
+ generic_online_page(page, order);
+}
+
+static uint64_t virtio_mem_send_request(struct virtio_mem *vm,
+ const struct virtio_mem_req *req)
+{
+ struct scatterlist *sgs[2], sg_req, sg_resp;
+ unsigned int len;
+ int rc;
+
+ /* don't use the request residing on the stack (vaddr) */
+ vm->req = *req;
+
+ /* out: buffer for request */
+ sg_init_one(&sg_req, &vm->req, sizeof(vm->req));
+ sgs[0] = &sg_req;
+
+ /* in: buffer for response */
+ sg_init_one(&sg_resp, &vm->resp, sizeof(vm->resp));
+ sgs[1] = &sg_resp;
+
+ rc = virtqueue_add_sgs(vm->vq, sgs, 1, 1, vm, GFP_KERNEL);
+ if (rc < 0)
+ return rc;
+
+ virtqueue_kick(vm->vq);
+
+ /* wait for a response */
+ wait_event(vm->host_resp, virtqueue_get_buf(vm->vq, &len));
+
+ return virtio16_to_cpu(vm->vdev, vm->resp.type);
+}
+
+static int virtio_mem_send_plug_request(struct virtio_mem *vm, uint64_t addr,
+ uint64_t size)
+{
+ const uint64_t nb_vm_blocks = size / vm->device_block_size;
+ const struct virtio_mem_req req = {
+ .type = cpu_to_virtio16(vm->vdev, VIRTIO_MEM_REQ_PLUG),
+ .u.plug.addr = cpu_to_virtio64(vm->vdev, addr),
+ .u.plug.nb_blocks = cpu_to_virtio16(vm->vdev, nb_vm_blocks),
+ };
+
+ if (atomic_read(&vm->config_changed))
+ return -EAGAIN;
+
+ switch (virtio_mem_send_request(vm, &req)) {
+ case VIRTIO_MEM_RESP_ACK:
+ vm->plugged_size += size;
+ return 0;
+ case VIRTIO_MEM_RESP_NACK:
+ return -EAGAIN;
+ case VIRTIO_MEM_RESP_BUSY:
+ return -EBUSY;
+ case VIRTIO_MEM_RESP_ERROR:
+ return -EINVAL;
+ default:
+ return -ENOMEM;
+ }
+}
+
+static int virtio_mem_send_unplug_request(struct virtio_mem *vm, uint64_t addr,
+ uint64_t size)
+{
+ const uint64_t nb_vm_blocks = size / vm->device_block_size;
+ const struct virtio_mem_req req = {
+ .type = cpu_to_virtio16(vm->vdev, VIRTIO_MEM_REQ_UNPLUG),
+ .u.unplug.addr = cpu_to_virtio64(vm->vdev, addr),
+ .u.unplug.nb_blocks = cpu_to_virtio16(vm->vdev, nb_vm_blocks),
+ };
+
+ if (atomic_read(&vm->config_changed))
+ return -EAGAIN;
+
+ switch (virtio_mem_send_request(vm, &req)) {
+ case VIRTIO_MEM_RESP_ACK:
+ vm->plugged_size -= size;
+ return 0;
+ case VIRTIO_MEM_RESP_BUSY:
+ return -EBUSY;
+ case VIRTIO_MEM_RESP_ERROR:
+ return -EINVAL;
+ default:
+ return -ENOMEM;
+ }
+}
+
+static int virtio_mem_send_unplug_all_request(struct virtio_mem *vm)
+{
+ const struct virtio_mem_req req = {
+ .type = cpu_to_virtio16(vm->vdev, VIRTIO_MEM_REQ_UNPLUG_ALL),
+ };
+
+ switch (virtio_mem_send_request(vm, &req)) {
+ case VIRTIO_MEM_RESP_ACK:
+ vm->unplug_all_required = false;
+ vm->plugged_size = 0;
+ /* usable region might have shrunk */
+ atomic_set(&vm->config_changed, 1);
+ return 0;
+ case VIRTIO_MEM_RESP_BUSY:
+ return -EBUSY;
+ default:
+ return -ENOMEM;
+ }
+}
+
+/*
+ * Plug selected subblocks. Updates the plugged state, but not the state
+ * of the memory block.
+ */
+static int virtio_mem_mb_plug_sb(struct virtio_mem *vm, unsigned long mb_id,
+ int sb_id, int count)
+{
+ const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id) +
+ sb_id * vm->subblock_size;
+ const uint64_t size = count * vm->subblock_size;
+ int rc;
+
+ dev_dbg(&vm->vdev->dev, "plugging memory block: %lu : %i -
%i\n", mb_id,
+ sb_id, sb_id + count - 1);
+
+ rc = virtio_mem_send_plug_request(vm, addr, size);
+ if (!rc)
+ virtio_mem_mb_set_sb_plugged(vm, mb_id, sb_id, count);
+ return rc;
+}
+
+/*
+ * Unplug selected subblocks. Updates the plugged state, but not the state
+ * of the memory block.
+ */
+static int virtio_mem_mb_unplug_sb(struct virtio_mem *vm, unsigned long mb_id,
+ int sb_id, int count)
+{
+ const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id) +
+ sb_id * vm->subblock_size;
+ const uint64_t size = count * vm->subblock_size;
+ int rc;
+
+ dev_dbg(&vm->vdev->dev, "unplugging memory block: %lu : %i -
%i\n",
+ mb_id, sb_id, sb_id + count - 1);
+
+ rc = virtio_mem_send_unplug_request(vm, addr, size);
+ if (!rc)
+ virtio_mem_mb_set_sb_unplugged(vm, mb_id, sb_id, count);
+ return rc;
+}
+
+/*
+ * Unplug the desired number of plugged subblocks of a offline or unadded
+ * memory block. Will fail if any subblock cannot get unplugged (instead of
+ * skipping it).
+ *
+ * Will not modify the state of the memory block.
+ *
+ * Note: can fail after some subblocks were unplugged.
+ */
+static int virtio_mem_mb_unplug_any_sb(struct virtio_mem *vm,
+ unsigned long mb_id, uint64_t *nb_sb)
+{
+ int sb_id, count;
+ int rc;
+
+ while (*nb_sb) {
+ sb_id = virtio_mem_mb_first_plugged_sb(vm, mb_id);
+ if (sb_id >= vm->nb_sb_per_mb)
+ break;
+ count = 1;
+ while (count < *nb_sb &&
+ sb_id + count < vm->nb_sb_per_mb &&
+ virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id + count,
+ 1))
+ count++;
+
+ rc = virtio_mem_mb_unplug_sb(vm, mb_id, sb_id, count);
+ if (rc)
+ return rc;
+ *nb_sb -= count;
+ }
+
+ return 0;
+}
+
+/*
+ * Unplug all plugged subblocks of an offline or unadded memory block.
+ *
+ * Will not modify the state of the memory block.
+ *
+ * Note: can fail after some subblocks were unplugged.
+ */
+static int virtio_mem_mb_unplug(struct virtio_mem *vm, unsigned long mb_id)
+{
+ uint64_t nb_sb = vm->nb_sb_per_mb;
+
+ return virtio_mem_mb_unplug_any_sb(vm, mb_id, &nb_sb);
+}
+
+/*
+ * Prepare tracking data for the next memory block.
+ */
+static int virtio_mem_prepare_next_mb(struct virtio_mem *vm,
+ unsigned long *mb_id)
+{
+ int rc;
+
+ if (vm->next_mb_id > vm->last_usable_mb_id)
+ return -ENOSPC;
+
+ /* Resize the state array if required. */
+ rc = virtio_mem_mb_state_prepare_next_mb(vm);
+ if (rc)
+ return rc;
+
+ /* Resize the subblock bitmap if required. */
+ rc = virtio_mem_sb_bitmap_prepare_next_mb(vm);
+ if (rc)
+ return rc;
+
+ vm->nb_mb_state[VIRTIO_MEM_MB_STATE_UNUSED]++;
+ *mb_id = vm->next_mb_id++;
+ return 0;
+}
+
+/*
+ * Don't add too many blocks that are not onlined yet to avoid running OOM.
+ */
+static bool virtio_mem_too_many_mb_offline(struct virtio_mem *vm)
+{
+ unsigned long nb_offline;
+
+ nb_offline = vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE] +
+ vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL];
+ return nb_offline >= VIRTIO_MEM_NB_OFFLINE_THRESHOLD;
+}
+
+/*
+ * Try to plug the desired number of subblocks and add the memory block
+ * to Linux.
+ *
+ * Will modify the state of the memory block.
+ */
+static int virtio_mem_mb_plug_and_add(struct virtio_mem *vm,
+ unsigned long mb_id,
+ uint64_t *nb_sb)
+{
+ const int count = min_t(int, *nb_sb, vm->nb_sb_per_mb);
+ int rc, rc2;
+
+ if (WARN_ON_ONCE(!count))
+ return -EINVAL;
+
+ /*
+ * Plug the requested number of subblocks before adding it to linux,
+ * so that onlining will directly online all plugged subblocks.
+ */
+ rc = virtio_mem_mb_plug_sb(vm, mb_id, 0, count);
+ if (rc)
+ return rc;
+
+ /*
+ * Mark the block properly offline before adding it to Linux,
+ * so the memory notifiers will find the block in the right state.
+ */
+ if (count == vm->nb_sb_per_mb)
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE);
+ else
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL);
+
+ /* Add the memory block to linux - if that fails, try to unplug. */
+ rc = virtio_mem_mb_add(vm, mb_id);
+ if (rc) {
+ enum virtio_mem_mb_state new_state = VIRTIO_MEM_MB_STATE_UNUSED;
+
+ dev_err(&vm->vdev->dev,
+ "adding memory block %lu failed with %d\n", mb_id, rc);
+ rc2 = virtio_mem_mb_unplug_sb(vm, mb_id, 0, count);
+
+ /*
+ * TODO: Linux MM does not properly clean up yet in all cases
+ * where adding of memory failed - especially on -ENOMEM.
+ */
+ if (rc2)
+ new_state = VIRTIO_MEM_MB_STATE_PLUGGED;
+ virtio_mem_mb_set_state(vm, mb_id, new_state);
+ return rc;
+ }
+
+ *nb_sb -= count;
+ return 0;
+}
+
+/*
+ * Try to plug the desired number of subblocks of a memory block that
+ * is already added to Linux.
+ *
+ * Will modify the state of the memory block.
+ *
+ * Note: Can fail after some subblocks were successfully plugged.
+ */
+static int virtio_mem_mb_plug_any_sb(struct virtio_mem *vm, unsigned long
mb_id,
+ uint64_t *nb_sb, bool online)
+{
+ unsigned long pfn, nr_pages;
+ int sb_id, count;
+ int rc;
+
+ if (WARN_ON_ONCE(!*nb_sb))
+ return -EINVAL;
+
+ while (*nb_sb) {
+ sb_id = virtio_mem_mb_first_unplugged_sb(vm, mb_id);
+ if (sb_id >= vm->nb_sb_per_mb)
+ break;
+ count = 1;
+ while (count < *nb_sb &&
+ sb_id + count < vm->nb_sb_per_mb &&
+ !virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id + count,
+ 1))
+ count++;
+
+ rc = virtio_mem_mb_plug_sb(vm, mb_id, sb_id, count);
+ if (rc)
+ return rc;
+ *nb_sb -= count;
+ if (!online)
+ continue;
+
+ /* fake-online the pages if the memory block is online */
+ pfn = PFN_DOWN(virtio_mem_mb_id_to_phys(mb_id) +
+ sb_id * vm->subblock_size);
+ nr_pages = PFN_DOWN(count * vm->subblock_size);
+ virtio_mem_fake_online(pfn, nr_pages);
+ }
+
+ if (virtio_mem_mb_test_sb_plugged(vm, mb_id, 0, vm->nb_sb_per_mb)) {
+ if (online)
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE);
+ else
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE);
+ }
+
+ return rc;
+}
+
+/*
+ * Try to plug the requested amount of memory.
+ */
+static int virtio_mem_plug_request(struct virtio_mem *vm, uint64_t diff)
+{
+ uint64_t nb_sb = diff / vm->subblock_size;
+ unsigned long mb_id;
+ int rc;
+
+ if (!nb_sb)
+ return 0;
+
+ /* Don't race with onlining/offlining */
+ mutex_lock(&vm->hotplug_mutex);
+
+ /* Try to plug subblocks of partially plugged online blocks. */
+ virtio_mem_for_each_mb_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL) {
+ rc = virtio_mem_mb_plug_any_sb(vm, mb_id, &nb_sb, true);
+ if (rc || !nb_sb)
+ goto out_unlock;
+ cond_resched();
+ }
+
+ /* Try to plug subblocks of partially plugged offline blocks. */
+ virtio_mem_for_each_mb_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL) {
+ rc = virtio_mem_mb_plug_any_sb(vm, mb_id, &nb_sb, false);
+ if (rc || !nb_sb)
+ goto out_unlock;
+ cond_resched();
+ }
+
+ /*
+ * We won't be working on online/offline memory blocks from this point,
+ * so we can't race with memory onlining/offlining. Drop the mutex.
+ */
+ mutex_unlock(&vm->hotplug_mutex);
+
+ /* Try to plug and add unused blocks */
+ virtio_mem_for_each_mb_state(vm, mb_id, VIRTIO_MEM_MB_STATE_UNUSED) {
+ if (virtio_mem_too_many_mb_offline(vm))
+ return -ENOSPC;
+
+ rc = virtio_mem_mb_plug_and_add(vm, mb_id, &nb_sb);
+ if (rc || !nb_sb)
+ return rc;
+ cond_resched();
+ }
+
+ /* Try to prepare, plug and add new blocks */
+ while (nb_sb) {
+ if (virtio_mem_too_many_mb_offline(vm))
+ return -ENOSPC;
+
+ rc = virtio_mem_prepare_next_mb(vm, &mb_id);
+ if (rc)
+ return rc;
+ rc = virtio_mem_mb_plug_and_add(vm, mb_id, &nb_sb);
+ if (rc)
+ return rc;
+ cond_resched();
+ }
+
+ return 0;
+out_unlock:
+ mutex_unlock(&vm->hotplug_mutex);
+ return rc;
+}
+
+/*
+ * Try to unplug all blocks that couldn't be unplugged before, for example,
+ * because the hypervisor was busy.
+ */
+static int virtio_mem_unplug_pending_mb(struct virtio_mem *vm)
+{
+ unsigned long mb_id;
+ int rc;
+
+ virtio_mem_for_each_mb_state(vm, mb_id, VIRTIO_MEM_MB_STATE_PLUGGED) {
+ rc = virtio_mem_mb_unplug(vm, mb_id);
+ if (rc)
+ return rc;
+ virtio_mem_mb_set_state(vm, mb_id, VIRTIO_MEM_MB_STATE_UNUSED);
+ }
+
+ return 0;
+}
+
+/*
+ * Update all parts of the config that could have changed.
+ */
+static void virtio_mem_refresh_config(struct virtio_mem *vm)
+{
+ const uint64_t phys_limit = 1UL << MAX_PHYSMEM_BITS;
+ uint64_t new_plugged_size, usable_region_size, end_addr;
+
+ /* the plugged_size is just a reflection of what _we_ did previously */
+ virtio_cread(vm->vdev, struct virtio_mem_config, plugged_size,
+ &new_plugged_size);
+ if (WARN_ON_ONCE(new_plugged_size != vm->plugged_size))
+ vm->plugged_size = new_plugged_size;
+
+ /* calculate the last usable memory block id */
+ virtio_cread(vm->vdev, struct virtio_mem_config,
+ usable_region_size, &usable_region_size);
+ end_addr = vm->addr + usable_region_size;
+ end_addr = min(end_addr, phys_limit);
+ vm->last_usable_mb_id = virtio_mem_phys_to_mb_id(end_addr) - 1;
+
+ /* see if there is a request to change the size */
+ virtio_cread(vm->vdev, struct virtio_mem_config, requested_size,
+ &vm->requested_size);
+
+ dev_info(&vm->vdev->dev, "plugged size: 0x%llx",
vm->plugged_size);
+ dev_info(&vm->vdev->dev, "requested size: 0x%llx",
vm->requested_size);
+}
+
+/*
+ * Workqueue function for handling plug/unplug requests and config updates.
+ */
+static void virtio_mem_run_wq(struct work_struct *work)
+{
+ struct virtio_mem *vm = container_of(work, struct virtio_mem, wq);
+ uint64_t diff;
+ int rc = 0;
+
+ hrtimer_cancel(&vm->retry_timer);
+
+ if (vm->broken)
+ return;
+
+ /* Make sure we start with a clean state if there are leftovers. */
+ if (unlikely(vm->unplug_all_required))
+ rc = virtio_mem_send_unplug_all_request(vm);
+
+config_update:
+ if (atomic_read(&vm->config_changed)) {
+ atomic_set(&vm->config_changed, 0);
+ virtio_mem_refresh_config(vm);
+ }
+
+ /* Unplug any leftovers from previous runs */
+ if (!rc)
+ rc = virtio_mem_unplug_pending_mb(vm);
+
+ if (!rc && vm->requested_size != vm->plugged_size) {
+ if (vm->requested_size > vm->plugged_size) {
+ diff = vm->requested_size - vm->plugged_size;
+ rc = virtio_mem_plug_request(vm, diff);
+ }
+ /* TODO: try to unplug memory */
+ }
+
+ switch (rc) {
+ case 0:
+ break;
+ case -ENOSPC:
+ /*
+ * We cannot add any more memory (alignment, physical limit)
+ * or we have too many offline memory blocks.
+ */
+ break;
+ case -EBUSY:
+ /*
+ * The hypervisor cannot process our request right now
+ * (e.g., out of memory, migrating).
+ */
+ case -ENOMEM:
+ /* Out of memory, try again later. */
+ hrtimer_start(&vm->retry_timer, vm->retry_interval,
+ HRTIMER_MODE_REL);
+ /*
+ * TODO: more advanced retry handling. For example stop
+ * after X attempts or increase the waiting time between
+ * retries.
+ */
+ break;
+ case -EAGAIN:
+ /* Retry immediately (e.g., the config changed). */
+ goto config_update;
+ default:
+ /* Unknown error, mark as broken */
+ dev_err(&vm->vdev->dev,
+ "unknown error, marking device broken: %d\n", rc);
+ vm->broken = true;
+ }
+}
+
+static enum hrtimer_restart virtio_mem_timer_expired(struct hrtimer *timer)
+{
+ struct virtio_mem *vm = container_of(timer, struct virtio_mem,
+ retry_timer);
+
+ virtio_mem_retry(vm);
+ return HRTIMER_NORESTART;
+}
+
+static void virtio_mem_handle_response(struct virtqueue *vq)
+{
+ struct virtio_mem *vm = vq->vdev->priv;
+
+ wake_up(&vm->host_resp);
+}
+
+static int virtio_mem_init_vq(struct virtio_mem *vm)
+{
+ struct virtqueue *vq;
+
+ vq = virtio_find_single_vq(vm->vdev, virtio_mem_handle_response,
+ "guest-request");
+ if (IS_ERR(vq))
+ return PTR_ERR(vq);
+ vm->vq = vq;
+
+ return 0;
+}
+
+/*
+ * Test if any memory in the range is present in Linux.
+ */
+static bool virtio_mem_any_memory_present(unsigned long start,
+ unsigned long size)
+{
+ const unsigned long start_pfn = PFN_DOWN(start);
+ const unsigned long end_pfn = PFN_UP(start + size);
+ unsigned long pfn;
+
+ for (pfn = start_pfn; pfn != end_pfn; pfn++)
+ if (present_section_nr(pfn_to_section_nr(pfn)))
+ return true;
+
+ return false;
+}
+
+static int virtio_mem_init(struct virtio_mem *vm)
+{
+ const uint64_t phys_limit = 1UL << MAX_PHYSMEM_BITS;
+ uint64_t region_size;
+ uint16_t node_id;
+
+ if (!vm->vdev->config->get) {
+ dev_err(&vm->vdev->dev, "config access disabled\n");
+ return -EINVAL;
+ }
+
+ /*
+ * We don't want to (un)plug or reuse any memory when in kdump. The
+ * memory is still accessible (but not mapped).
+ */
+ if (is_kdump_kernel()) {
+ dev_warn(&vm->vdev->dev, "disabled in kdump kernel\n");
+ return -EBUSY;
+ }
+
+ /* Fetch all properties that can't change. */
+ virtio_cread(vm->vdev, struct virtio_mem_config, plugged_size,
+ &vm->plugged_size);
+ virtio_cread(vm->vdev, struct virtio_mem_config, block_size,
+ &vm->device_block_size);
+ virtio_cread(vm->vdev, struct virtio_mem_config, node_id,
+ &node_id);
+ vm->nid = virtio_mem_translate_node_id(vm, node_id);
+ virtio_cread(vm->vdev, struct virtio_mem_config, addr, &vm->addr);
+ virtio_cread(vm->vdev, struct virtio_mem_config, region_size,
+ ®ion_size);
+
+ /*
+ * If we still have memory plugged, we might have to unplug all
+ * memory first. However, if somebody simply unloaded the driver
+ * we would have to reinitialize the old state - something we don't
+ * support yet. Detect if we have any memory in the area present.
+ */
+ if (vm->plugged_size) {
+ uint64_t usable_region_size;
+
+ virtio_cread(vm->vdev, struct virtio_mem_config,
+ usable_region_size, &usable_region_size);
+
+ if (virtio_mem_any_memory_present(vm->addr,
+ usable_region_size)) {
+ dev_err(&vm->vdev->dev,
+ "reloading the driver is not supported\n");
+ return -EINVAL;
+ }
+ /*
+ * Note: it might happen that the device is busy and
+ * unplugging all memory might take some time.
+ */
+ dev_info(&vm->vdev->dev, "unplugging all memory
required\n");
+ vm->unplug_all_required = 1;
+ }
+
+ /*
+ * We always hotplug memory in memory block granularity. This way,
+ * we have to wait for exactly one memory block to online.
+ */
+ if (vm->device_block_size > memory_block_size_bytes()) {
+ dev_err(&vm->vdev->dev,
+ "The block size is not supported (too big).\n");
+ return -EINVAL;
+ }
+
+ /* bad device setup - warn only */
+ if (!IS_ALIGNED(vm->addr, memory_block_size_bytes()))
+ dev_warn(&vm->vdev->dev,
+ "The alignment of the physical start address can make some memory
unusable.\n");
+ if (!IS_ALIGNED(vm->addr + region_size, memory_block_size_bytes()))
+ dev_warn(&vm->vdev->dev,
+ "The alignment of the physical end address can make some memory
unusable.\n");
+ if (vm->addr + region_size > phys_limit)
+ dev_warn(&vm->vdev->dev,
+ "Some memory is not addressable. This can make some memory
unusable.\n");
+
+ /*
+ * Calculate the subblock size:
+ * - At least MAX_ORDER - 1 / pageblock_order.
+ * - At least the device block size.
+ * In the worst case, a single subblock per memory block.
+ */
+ vm->subblock_size = PAGE_SIZE * 1u << max_t(uint32_t, MAX_ORDER - 1,
+ pageblock_order);
+ vm->subblock_size = max_t(uint32_t, vm->device_block_size,
+ vm->subblock_size);
+ vm->nb_sb_per_mb = memory_block_size_bytes() / vm->subblock_size;
+
+ /* Round up to the next full memory block */
+ vm->first_mb_id = virtio_mem_phys_to_mb_id(vm->addr - 1 +
+ memory_block_size_bytes());
+ vm->next_mb_id = vm->first_mb_id;
+ vm->last_mb_id = virtio_mem_phys_to_mb_id(vm->addr + region_size) - 1;
+
+ dev_info(&vm->vdev->dev, "start address: 0x%llx",
vm->addr);
+ dev_info(&vm->vdev->dev, "region size: 0x%llx",
region_size);
+ dev_info(&vm->vdev->dev, "device block size: 0x%x",
+ vm->device_block_size);
+ dev_info(&vm->vdev->dev, "memory block size: 0x%lx",
+ memory_block_size_bytes());
+ dev_info(&vm->vdev->dev, "subblock size: 0x%x",
+ vm->subblock_size);
+
+ return 0;
+}
+
+static int virtio_mem_probe(struct virtio_device *vdev)
+{
+ struct virtio_mem *vm;
+ int rc = -EINVAL;
+
+ vdev->priv = vm = kzalloc(sizeof(*vm), GFP_KERNEL);
+ if (!vm)
+ return -ENOMEM;
+
+ init_waitqueue_head(&vm->host_resp);
+ vm->vdev = vdev;
+ INIT_WORK(&vm->wq, virtio_mem_run_wq);
+ mutex_init(&vm->hotplug_mutex);
+ /*
+ * Avoid circular locking lockdep warnings. We lock the mutex
+ * e.g., in MEM_GOING_ONLINE and unlock it in MEM_ONLINE. The
+ * blocking_notifier_call_chain() has it's own lock, which gets unlocked
+ * between both notifier calls.
+ */
+ lockdep_set_novalidate_class(&vm->hotplug_mutex);
+ INIT_LIST_HEAD(&vm->next);
+ spin_lock_init(&vm->removal_lock);
+ hrtimer_init(&vm->retry_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+ vm->retry_timer.function = virtio_mem_timer_expired;
+ vm->retry_interval = ms_to_ktime(VIRTIO_MEM_RETRY_MS);
+
+ /* register the virtqueue */
+ rc = virtio_mem_init_vq(vm);
+ if (rc)
+ goto out_free_vm;
+
+ /* initialize the device by querying the config */
+ rc = virtio_mem_init(vm);
+ if (rc)
+ goto out_del_vq;
+
+ /* register callbacks */
+ vm->memory_notifier.notifier_call = virtio_mem_memory_notifier_cb;
+ rc = register_memory_notifier(&vm->memory_notifier);
+ if (rc)
+ goto out_del_vq;
+ rc = register_virtio_mem_device(vm);
+ if (rc)
+ goto out_unreg_mem;
+
+ virtio_device_ready(vdev);
+
+ /* trigger a config update to start processing the requested_size */
+ atomic_set(&vm->config_changed, 1);
+ queue_work(system_freezable_wq, &vm->wq);
+
+ return 0;
+out_unreg_mem:
+ unregister_memory_notifier(&vm->memory_notifier);
+out_del_vq:
+ vdev->config->del_vqs(vdev);
+out_free_vm:
+ kfree(vm);
+ vdev->priv = NULL;
+
+ return rc;
+}
+
+static void virtio_mem_remove(struct virtio_device *vdev)
+{
+ struct virtio_mem *vm = vdev->priv;
+ unsigned long mb_id;
+ int rc;
+
+ /*
+ * Make sure the workqueue won't be triggered anymore and no memory
+ * blocks can be onlined/offlined until we're finished here.
+ */
+ spin_lock_irq(&vm->removal_lock);
+ vm->removing = true;
+ spin_unlock_irq(&vm->removal_lock);
+
+ /* wait until the workqueue stopped */
+ cancel_work_sync(&vm->wq);
+ hrtimer_cancel(&vm->retry_timer);
+
+ mutex_lock(&vm->hotplug_mutex);
+ /*
+ * After we unregistered our callbacks, user space can online partially
+ * plugged offline blocks. Make sure to remove them.
+ */
+ virtio_mem_for_each_mb_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL) {
+ mutex_unlock(&vm->hotplug_mutex);
+ rc = virtio_mem_mb_remove(vm, mb_id);
+ BUG_ON(rc);
+ mutex_lock(&vm->hotplug_mutex);
+ }
+ mutex_unlock(&vm->hotplug_mutex);
+
+ /* unregister callbacks */
+ unregister_virtio_mem_device(vm);
+ unregister_memory_notifier(&vm->memory_notifier);
+
+ /*
+ * There is no way we could deterministically remove all memory we have
+ * added to the system. And there is no way to stop the driver/device
+ * from going away.
+ */
+ if (vm->plugged_size)
+ dev_warn(&vdev->dev, "device still has memory plugged\n");
+
+ /* remove all tracking data - no locking needed */
+ vfree(vm->mb_state);
+ vfree(vm->sb_bitmap);
+
+ /* reset the device and cleanup the queues */
+ vdev->config->reset(vdev);
+ vdev->config->del_vqs(vdev);
+
+ kfree(vm);
+ vdev->priv = NULL;
+}
+
+static void virtio_mem_config_changed(struct virtio_device *vdev)
+{
+ struct virtio_mem *vm = vdev->priv;
+
+ atomic_set(&vm->config_changed, 1);
+ virtio_mem_retry(vm);
+}
+
+#ifdef CONFIG_PM_SLEEP
+static int virtio_mem_freeze(struct virtio_device *vdev)
+{
+ /*
+ * When restarting the VM, all memory is usually unplugged. Don't
+ * allow to suspend/hibernate.
+ */
+ dev_err(&vdev->dev, "save/restore not supported.\n");
+ return -EPERM;
+}
+
+static int virtio_mem_restore(struct virtio_device *vdev)
+{
+ return -EPERM;
+}
+#endif
+
+static unsigned int virtio_mem_features[] = {
+#if defined(CONFIG_NUMA) && defined(CONFIG_ACPI_NUMA)
+ VIRTIO_MEM_F_ACPI_PXM,
+#endif
+};
+
+static struct virtio_device_id virtio_mem_id_table[] = {
+ { VIRTIO_ID_MEM, VIRTIO_DEV_ANY_ID },
+ { 0 },
+};
+
+static struct virtio_driver virtio_mem_driver = {
+ .feature_table = virtio_mem_features,
+ .feature_table_size = ARRAY_SIZE(virtio_mem_features),
+ .driver.name = KBUILD_MODNAME,
+ .driver.owner = THIS_MODULE,
+ .id_table = virtio_mem_id_table,
+ .probe = virtio_mem_probe,
+ .remove = virtio_mem_remove,
+ .config_changed = virtio_mem_config_changed,
+#ifdef CONFIG_PM_SLEEP
+ .freeze = virtio_mem_freeze,
+ .restore = virtio_mem_restore,
+#endif
+};
+
+module_virtio_driver(virtio_mem_driver);
+MODULE_DEVICE_TABLE(virtio, virtio_mem_id_table);
+MODULE_AUTHOR("David Hildenbrand <david at redhat.com>");
+MODULE_DESCRIPTION("Virtio-mem driver");
+MODULE_LICENSE("GPL");
diff --git a/include/uapi/linux/virtio_ids.h b/include/uapi/linux/virtio_ids.h
index 585e07b27333..934e0b444454 100644
--- a/include/uapi/linux/virtio_ids.h
+++ b/include/uapi/linux/virtio_ids.h
@@ -44,6 +44,7 @@
#define VIRTIO_ID_VSOCK 19 /* virtio vsock transport */
#define VIRTIO_ID_CRYPTO 20 /* virtio crypto */
#define VIRTIO_ID_IOMMU 23 /* virtio IOMMU */
+#define VIRTIO_ID_MEM 24 /* virtio mem */
#define VIRTIO_ID_FS 26 /* virtio filesystem */
#define VIRTIO_ID_PMEM 27 /* virtio pmem */
diff --git a/include/uapi/linux/virtio_mem.h b/include/uapi/linux/virtio_mem.h
new file mode 100644
index 000000000000..bac73fca6c0e
--- /dev/null
+++ b/include/uapi/linux/virtio_mem.h
@@ -0,0 +1,204 @@
+/* SPDX-License-Identifier: BSD-3-Clause */
+/*
+ * Virtio Mem Device
+ *
+ * Copyright Red Hat, Inc. 2019
+ *
+ * Authors:
+ * David Hildenbrand <david at redhat.com>
+ *
+ * This header is BSD licensed so anyone can use the definitions
+ * to implement compatible drivers/servers:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ * 3. Neither the name of IBM nor the names of its contributors
+ * may be used to endorse or promote products derived from this software
+ * without specific prior written permission.
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL IBM OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#ifndef _LINUX_VIRTIO_MEM_H
+#define _LINUX_VIRTIO_MEM_H
+
+#include <linux/types.h>
+#include <linux/virtio_types.h>
+#include <linux/virtio_ids.h>
+#include <linux/virtio_config.h>
+
+/*
+ * Each virtio-mem device manages a dedicated region in physical address
+ * space. Each device can belong to a single NUMA node, multiple devices
+ * for a single NUMA node are possible. A virtio-mem device is like a
+ * "resizable DIMM" consisting of small memory blocks that can be
plugged
+ * or unplugged. The device driver is responsible for (un)plugging memory
+ * blocks on demand.
+ *
+ * Virtio-mem devices can only operate on their assigned memory region in
+ * order to (un)plug memory. A device cannot (un)plug memory belonging to
+ * other devices.
+ *
+ * The "region_size" corresponds to the maximum amount of memory that
can
+ * be provided by a device. The "size" corresponds to the amount of
memory
+ * that is currently plugged. "requested_size" corresponds to a
request
+ * from the device to the device driver to (un)plug blocks. The
+ * device driver should try to (un)plug blocks in order to reach the
+ * "requested_size". It is impossible to plug more memory than
requested.
+ *
+ * The "usable_region_size" represents the memory region that can
actually
+ * be used to (un)plug memory. It is always at least as big as the
+ * "requested_size" and will grow dynamically. It will only shrink
when
+ * explicitly triggered (VIRTIO_MEM_REQ_UNPLUG).
+ *
+ * Memory in the usable region can usually be read, however, there are no
+ * guarantees. It can happen that the device cannot process a request,
+ * because it is busy. The device driver has to retry later.
+ *
+ * Usually, during system resets all memory will get unplugged, so the
+ * device driver can start with a clean state. However, in specific
+ * scenarios (if the device is busy) it can happen that the device still
+ * has memory plugged. The device driver can request to unplug all memory
+ * (VIRTIO_MEM_REQ_UNPLUG) - which might take a while to succeed if the
+ * device is busy.
+ */
+
+/* --- virtio-mem: feature bits --- */
+
+/* node_id is an ACPI PXM and is valid */
+#define VIRTIO_MEM_F_ACPI_PXM 0
+
+
+/* --- virtio-mem: guest -> host requests --- */
+
+/* request to plug memory blocks */
+#define VIRTIO_MEM_REQ_PLUG 0
+/* request to unplug memory blocks */
+#define VIRTIO_MEM_REQ_UNPLUG 1
+/* request to unplug all blocks and shrink the usable size */
+#define VIRTIO_MEM_REQ_UNPLUG_ALL 2
+/* request information about the plugged state of memory blocks */
+#define VIRTIO_MEM_REQ_STATE 3
+
+struct virtio_mem_req_plug {
+ __virtio64 addr;
+ __virtio16 nb_blocks;
+};
+
+struct virtio_mem_req_unplug {
+ __virtio64 addr;
+ __virtio16 nb_blocks;
+};
+
+struct virtio_mem_req_state {
+ __virtio64 addr;
+ __virtio16 nb_blocks;
+};
+
+struct virtio_mem_req {
+ __virtio16 type;
+ __virtio16 padding[3];
+
+ union {
+ struct virtio_mem_req_plug plug;
+ struct virtio_mem_req_unplug unplug;
+ struct virtio_mem_req_state state;
+ } u;
+};
+
+
+/* --- virtio-mem: host -> guest response --- */
+
+/*
+ * Request processed successfully, applicable for
+ * - VIRTIO_MEM_REQ_PLUG
+ * - VIRTIO_MEM_REQ_UNPLUG
+ * - VIRTIO_MEM_REQ_UNPLUG_ALL
+ * - VIRTIO_MEM_REQ_STATE
+ */
+#define VIRTIO_MEM_RESP_ACK 0
+/*
+ * Request denied - e.g. trying to plug more than requested, applicable for
+ * - VIRTIO_MEM_REQ_PLUG
+ */
+#define VIRTIO_MEM_RESP_NACK 1
+/*
+ * Request cannot be processed right now, try again later, applicable for
+ * - VIRTIO_MEM_REQ_PLUG
+ * - VIRTIO_MEM_REQ_UNPLUG
+ * - VIRTIO_MEM_REQ_UNPLUG_ALL
+ */
+#define VIRTIO_MEM_RESP_BUSY 2
+/*
+ * Error in request (e.g. addresses/alignemnt), applicable for
+ * - VIRTIO_MEM_REQ_PLUG
+ * - VIRTIO_MEM_REQ_UNPLUG
+ * - VIRTIO_MEM_REQ_STATE
+ */
+#define VIRTIO_MEM_RESP_ERROR 3
+
+
+/* State of memory blocks is "plugged" */
+#define VIRTIO_MEM_STATE_PLUGGED 0
+/* State of memory blocks is "unplugged" */
+#define VIRTIO_MEM_STATE_UNPLUGGED 1
+/* State of memory blocks is "mixed" */
+#define VIRTIO_MEM_STATE_MIXED 2
+
+struct virtio_mem_resp_state {
+ __virtio16 state;
+};
+
+struct virtio_mem_resp {
+ __virtio16 type;
+ __virtio16 padding[3];
+
+ union {
+ struct virtio_mem_resp_state state;
+ } u;
+};
+
+/* --- virtio-mem: configuration --- */
+
+struct virtio_mem_config {
+ /* Block size and alignment. Cannot change. */
+ uint32_t block_size;
+ /* Valid with VIRTIO_MEM_F_ACPI_PXM. Cannot change. */
+ uint16_t node_id;
+ uint16_t padding;
+ /* Start address of the memory region. Cannot change. */
+ uint64_t addr;
+ /* Region size (maximum). Cannot change. */
+ uint64_t region_size;
+ /*
+ * Currently usable region size. Can grow up to region_size. Can
+ * shrink due to VIRTIO_MEM_REQ_UNPLUG_ALL (in which case no config
+ * update will be sent).
+ */
+ uint64_t usable_region_size;
+ /*
+ * Currently used size. Changes due to plug/unplug requests, but no
+ * config updates will be sent.
+ */
+ uint64_t plugged_size;
+ /* Requested size. New plug requests cannot exceed it. Can change. */
+ uint64_t requested_size;
+};
+
+#endif /* _LINUX_VIRTIO_MEM_H */
--
2.21.0
David Hildenbrand
2019-Sep-19 14:22 UTC
[PATCH RFC v3 3/9] virtio-mem: Paravirtualized memory hotunplug part 1
Unplugging subblocks of memory blocks that are offline is easy. All we
have to do is watch out for concurrent onlining acticity.
Cc: "Michael S. Tsirkin" <mst at redhat.com>
Cc: Jason Wang <jasowang at redhat.com>
Cc: Oscar Salvador <osalvador at suse.de>
Cc: Michal Hocko <mhocko at kernel.org>
Cc: Igor Mammedov <imammedo at redhat.com>
Cc: Dave Young <dyoung at redhat.com>
Cc: Andrew Morton <akpm at linux-foundation.org>
Cc: Dan Williams <dan.j.williams at intel.com>
Cc: Pavel Tatashin <pasha.tatashin at soleen.com>
Cc: Stefan Hajnoczi <stefanha at redhat.com>
Cc: Vlastimil Babka <vbabka at suse.cz>
Signed-off-by: David Hildenbrand <david at redhat.com>
---
drivers/virtio/virtio_mem.c | 109 +++++++++++++++++++++++++++++++++++-
1 file changed, 107 insertions(+), 2 deletions(-)
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index 597fab4933f6..6fb55d4b6f6c 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -119,7 +119,7 @@ struct virtio_mem {
*
* When this lock is held the pointers can't change, ONLINE and
* OFFLINE blocks can't change the state and no subblocks will get
- * plugged.
+ * plugged/unplugged.
*/
struct mutex hotplug_mutex;
bool hotplug_active;
@@ -322,6 +322,19 @@ static bool virtio_mem_mb_test_sb_plugged(struct virtio_mem
*vm,
bit + count;
}
+/*
+ * Test if all selected subblocks are unplugged.
+ */
+static bool virtio_mem_mb_test_sb_unplugged(struct virtio_mem *vm,
+ unsigned long mb_id, int sb_id,
+ int count)
+{
+ const int bit = (mb_id - vm->first_mb_id) * vm->nb_sb_per_mb + sb_id;
+
+ /* TODO: Helper similar to bitmap_set() */
+ return find_next_bit(vm->sb_bitmap, bit + count, bit) >= bit + count;
+}
+
/*
* Find the first plugged subblock. Returns vm->nb_sb_per_mb in case there
is
* none.
@@ -512,6 +525,9 @@ static void virtio_mem_notify_offline(struct virtio_mem *vm,
BUG();
break;
}
+
+ /* trigger the workqueue, maybe we can now unplug memory. */
+ virtio_mem_retry(vm);
}
static void virtio_mem_notify_online(struct virtio_mem *vm, unsigned long
mb_id,
@@ -1129,6 +1145,93 @@ static int virtio_mem_plug_request(struct virtio_mem *vm,
uint64_t diff)
return rc;
}
+/*
+ * Unplug the desired number of plugged subblocks of an offline memory block.
+ * Will fail if any subblock cannot get unplugged (instead of skipping it).
+ *
+ * Will modify the state of the memory block. Might temporarily drop the
+ * hotplug_mutex.
+ *
+ * Note: Can fail after some subblocks were successfully unplugged.
+ */
+static int virtio_mem_mb_unplug_any_sb_offline(struct virtio_mem *vm,
+ unsigned long mb_id,
+ uint64_t *nb_sb)
+{
+ int rc;
+
+ rc = virtio_mem_mb_unplug_any_sb(vm, mb_id, nb_sb);
+
+ /* some subblocks might have been unplugged even on failure */
+ if (!virtio_mem_mb_test_sb_plugged(vm, mb_id, 0, vm->nb_sb_per_mb))
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL);
+ if (rc)
+ return rc;
+
+ if (virtio_mem_mb_test_sb_unplugged(vm, mb_id, 0, vm->nb_sb_per_mb)) {
+ /*
+ * Remove the block from Linux - this should never fail.
+ * Hinder the block from getting onlined by marking it
+ * unplugged. Temporarily drop the mutex, so
+ * any pending GOING_ONLINE requests can be serviced/rejected.
+ */
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_UNUSED);
+
+ mutex_unlock(&vm->hotplug_mutex);
+ rc = virtio_mem_mb_remove(vm, mb_id);
+ BUG_ON(rc);
+ mutex_lock(&vm->hotplug_mutex);
+ }
+ return 0;
+}
+
+/*
+ * Try to unplug the requested amount of memory.
+ */
+static int virtio_mem_unplug_request(struct virtio_mem *vm, uint64_t diff)
+{
+ uint64_t nb_sb = diff / vm->subblock_size;
+ unsigned long mb_id;
+ int rc;
+
+ if (!nb_sb)
+ return 0;
+
+ /*
+ * We'll drop the mutex a couple of times when it is safe to do so.
+ * This might result in some blocks switching the state (online/offline)
+ * and we could miss them in this run - we will retry again later.
+ */
+ mutex_lock(&vm->hotplug_mutex);
+
+ /* Try to unplug subblocks of partially plugged offline blocks. */
+ virtio_mem_for_each_mb_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL) {
+ rc = virtio_mem_mb_unplug_any_sb_offline(vm, mb_id,
+ &nb_sb);
+ if (rc || !nb_sb)
+ goto out_unlock;
+ cond_resched();
+ }
+
+ /* Try to unplug subblocks of plugged offline blocks. */
+ virtio_mem_for_each_mb_state(vm, mb_id, VIRTIO_MEM_MB_STATE_OFFLINE) {
+ rc = virtio_mem_mb_unplug_any_sb_offline(vm, mb_id,
+ &nb_sb);
+ if (rc || !nb_sb)
+ goto out_unlock;
+ cond_resched();
+ }
+
+ mutex_unlock(&vm->hotplug_mutex);
+ return 0;
+out_unlock:
+ mutex_unlock(&vm->hotplug_mutex);
+ return rc;
+}
+
/*
* Try to unplug all blocks that couldn't be unplugged before, for example,
* because the hypervisor was busy.
@@ -1209,8 +1312,10 @@ static void virtio_mem_run_wq(struct work_struct *work)
if (vm->requested_size > vm->plugged_size) {
diff = vm->requested_size - vm->plugged_size;
rc = virtio_mem_plug_request(vm, diff);
+ } else {
+ diff = vm->plugged_size - vm->requested_size;
+ rc = virtio_mem_unplug_request(vm, diff);
}
- /* TODO: try to unplug memory */
}
switch (rc) {
--
2.21.0
David Hildenbrand
2019-Sep-19 14:22 UTC
[PATCH RFC v3 4/9] mm: Export alloc_contig_range() / free_contig_range()
A virtio-mem device wants to allocate memory from the memory region it manages in order to unplug it in the hypervisor - similar to a balloon driver. Also, it might want to plug previously unplugged (allocated) memory and give it back to Linux. alloc_contig_range() / free_contig_range() seem to be the perfect interface for this task. In contrast to existing balloon devices, a virtio-mem device operates on bigger chunks (e.g., 4MB) and only on physical memory it manages. It tracks which chunks (subblocks) are still plugged, so it can go ahead and try to alloc_contig_range()+unplug them on unplug request, or plug+free_contig_range() unplugged chunks on plug requests. A virtio-mem device will use alloc_contig_range() / free_contig_range() only on ranges that belong to the same node/zone in at least MAX(MAX_ORDER - 1, pageblock_order) order granularity - e.g., 4MB on x86-64. The virtio-mem device added that memory, so the memory exists and does not contain any holes. virtio-mem will only try to allocate on ZONE_NORMAL, never on ZONE_MOVABLE, just like when allocating gigantic pages (we don't put unmovable data into the movable zone). Cc: Andrew Morton <akpm at linux-foundation.org> Cc: Michal Hocko <mhocko at suse.com> Cc: Vlastimil Babka <vbabka at suse.cz> Cc: Oscar Salvador <osalvador at suse.de> Cc: Mel Gorman <mgorman at techsingularity.net> Cc: Mike Rapoport <rppt at linux.ibm.com> Cc: Dan Williams <dan.j.williams at intel.com> Cc: Alexander Duyck <alexander.h.duyck at linux.intel.com> Cc: Pavel Tatashin <pavel.tatashin at microsoft.com> Cc: Alexander Potapenko <glider at google.com> Signed-off-by: David Hildenbrand <david at redhat.com> --- mm/page_alloc.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3334a769eb91..d5d7944954b3 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -8469,6 +8469,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, pfn_max_align_up(end), migratetype); return ret; } +EXPORT_SYMBOL_GPL(alloc_contig_range); #endif /* CONFIG_CONTIG_ALLOC */ void free_contig_range(unsigned long pfn, unsigned int nr_pages) @@ -8483,6 +8484,7 @@ void free_contig_range(unsigned long pfn, unsigned int nr_pages) } WARN(count != 0, "%d pages are still in use!\n", count); } +EXPORT_SYMBOL_GPL(free_contig_range); #ifdef CONFIG_MEMORY_HOTPLUG /* -- 2.21.0
David Hildenbrand
2019-Sep-19 14:22 UTC
[PATCH RFC v3 5/9] virtio-mem: Paravirtualized memory hotunplug part 2
We can use alloc_contig_range() to try to unplug subblocks. Unplugged
blocks will be marked PG_offline, however, don't have the PG_reserved
flag set. This way, we can differentiate these allocated subblocks from
subblocks that were never onlined and handle them properly in
virtio_mem_fake_online(). free_contig_range() is used to hand back
subblocks to Linux.
It is worth noting that there are no guarantess on how much memory can
actually get unplugged again. All device memory might completely be
fragmented with unmovable data, such that no subblock can get unplugged.
We might want to improve the unplugging capability in the future.
We are not touching the ZONE_MOVABLE. If memory is onlined to the
ZONE_MOVABLE, it can only get unplugged after that memory was offlined
manually by user space. In normal operation, virtio-mem memory is
suggested to be onlined to ZONE_NORMAL. In the future, we will try to
make unplug more likely to succeed.
Future work:
- Offline + remove memory blocks once all subblocks were unplugged. This
might then free up unmovable data un other memory blocks.
- Performance improvements:
-- Sense (lockless) if it make sense to try alloc_contig_range() at all
before directly trying to isolate and taking locks.
-- Try to unplug bigger chunks if possible first.
-- Identify free areas first, that don't have to be evacuated.
- Make unplug more likely to succeed:
-- The "issue" is that in the ZONE_NORMAL, the buddy will randomly
allocate memory. Only pageblocks somewhat limit fragmentation,
however we would want to limit fragmentation on subblock granularity
and even memory block granularity. One idea is to have a new
ZONE_PREFER_MOVABLE. Memory blocks will then be onlined to ZONE_NORMAL
/ ZONE_PREFER_MOVABLE in a certain ratio per node (e.g.,
1:4). This makes unplug of quite some memory likely to succeed in most
setups. ZONE_PREFER_MOVABLE is then a mixture of ZONE_NORMAL and
ZONE_MOVABlE. Especially, movable data can end up on that zone, but
only if really required - avoiding running out of memory on ZONE
imbalances. The zone fallback order would be
MOVABLE=>PREFER_MOVABLE=>HIGHMEM=>NORMAL=>PREFER_MOVABLE=>DMA32=>DMA
-- Allocate memmap from added memory. This way, less unmovable data can
end up on the memory blocks.
-- Call drop_slab() before trying to unplug. Eventually shrink other
caches.
- Better retry handling in case memory is busy. We certainly don't want
to try for ever in a short interval to try to get some memory back.
- OOM handling, e.g., via an OOM handler.
Cc: "Michael S. Tsirkin" <mst at redhat.com>
Cc: Jason Wang <jasowang at redhat.com>
Cc: Oscar Salvador <osalvador at suse.de>
Cc: Michal Hocko <mhocko at kernel.org>
Cc: Igor Mammedov <imammedo at redhat.com>
Cc: Dave Young <dyoung at redhat.com>
Cc: Andrew Morton <akpm at linux-foundation.org>
Cc: Dan Williams <dan.j.williams at intel.com>
Cc: Pavel Tatashin <pasha.tatashin at soleen.com>
Cc: Stefan Hajnoczi <stefanha at redhat.com>
Cc: Vlastimil Babka <vbabka at suse.cz>
Signed-off-by: David Hildenbrand <david at redhat.com>
---
drivers/virtio/Kconfig | 1 +
drivers/virtio/virtio_mem.c | 106 +++++++++++++++++++++++++++++++++++-
2 files changed, 104 insertions(+), 3 deletions(-)
diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig
index 294720d53057..75a760f32ec7 100644
--- a/drivers/virtio/Kconfig
+++ b/drivers/virtio/Kconfig
@@ -71,6 +71,7 @@ config VIRTIO_MEM
depends on VIRTIO
depends on MEMORY_HOTPLUG_SPARSE
depends on MEMORY_HOTREMOVE
+ select CONTIG_ALLOC
help
This driver provides access to virtio-mem paravirtualized memory
devices, allowing to hotplug and hotunplug memory.
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index 6fb55d4b6f6c..91052a37d10d 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -689,7 +689,17 @@ static void virtio_mem_fake_online(unsigned long pfn,
unsigned int nr_pages)
for (i = 0; i < nr_pages; i += 1 << order) {
struct page *page = pfn_to_page(pfn + i);
- generic_online_page(page, order);
+ /*
+ * If the page is reserved, it was kept fake-offline when
+ * onlining the memory block. Otherwise, it was allocated
+ * using alloc_contig_range().
+ */
+ if (PageReserved(page))
+ generic_online_page(page, order);
+ else {
+ free_contig_range(pfn + i, 1 << order);
+ totalram_pages_add(1 << order);
+ }
}
}
@@ -1187,6 +1197,72 @@ static int virtio_mem_mb_unplug_any_sb_offline(struct
virtio_mem *vm,
return 0;
}
+/*
+ * Unplug the desired number of plugged subblocks of an online memory block.
+ * Will skip subblock that are busy.
+ *
+ * Will modify the state of the memory block.
+ *
+ * Note: Can fail after some subblocks were successfully unplugged. Can
+ * return 0 even if subblocks were busy and could not get unplugged.
+ */
+static int virtio_mem_mb_unplug_any_sb_online(struct virtio_mem *vm,
+ unsigned long mb_id,
+ uint64_t *nb_sb)
+{
+ const unsigned long nr_pages = PFN_DOWN(vm->subblock_size);
+ unsigned long start_pfn;
+ int rc, sb_id;
+
+ /*
+ * TODO: To increase the performance we want to try bigger, consecutive
+ * subblocks first before falling back to single subblocks. Also,
+ * we should sense via something like is_mem_section_removable()
+ * first if it makes sense to go ahead any try to allocate.
+ */
+ for (sb_id = 0; sb_id < vm->nb_sb_per_mb && *nb_sb; sb_id++) {
+ /* Find the next candidate subblock */
+ while (sb_id < vm->nb_sb_per_mb &&
+ !virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id, 1))
+ sb_id++;
+ if (sb_id >= vm->nb_sb_per_mb)
+ break;
+
+ start_pfn = PFN_DOWN(virtio_mem_mb_id_to_phys(mb_id) +
+ sb_id * vm->subblock_size);
+ rc = alloc_contig_range(start_pfn, start_pfn + nr_pages,
+ MIGRATE_MOVABLE, GFP_KERNEL);
+ if (rc == -ENOMEM)
+ /* whoops, out of memory */
+ return rc;
+ if (rc)
+ /* memory busy, we can't unplug this chunk */
+ continue;
+
+ /* Mark it as fake-offline before unplugging it */
+ virtio_mem_set_fake_offline(start_pfn, nr_pages);
+ totalram_pages_add(-nr_pages);
+
+ /* Try to unplug the allocated memory */
+ rc = virtio_mem_mb_unplug_sb(vm, mb_id, sb_id, 1);
+ if (rc) {
+ /* Return the memory to the buddy. */
+ virtio_mem_fake_online(start_pfn, nr_pages);
+ return rc;
+ }
+
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL);
+ *nb_sb -= 1;
+ }
+
+ /*
+ * TODO: Once all subblocks of a memory block were unplugged, we want
+ * to offline the memory block and remove it.
+ */
+ return 0;
+}
+
/*
* Try to unplug the requested amount of memory.
*/
@@ -1225,8 +1301,31 @@ static int virtio_mem_unplug_request(struct virtio_mem
*vm, uint64_t diff)
cond_resched();
}
+ /* Try to unplug subblocks of partially plugged online blocks. */
+ virtio_mem_for_each_mb_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL) {
+ rc = virtio_mem_mb_unplug_any_sb_online(vm, mb_id,
+ &nb_sb);
+ if (rc || !nb_sb)
+ goto out_unlock;
+ mutex_unlock(&vm->hotplug_mutex);
+ cond_resched();
+ mutex_lock(&vm->hotplug_mutex);
+ }
+
+ /* Try to unplug subblocks of plugged online blocks. */
+ virtio_mem_for_each_mb_state(vm, mb_id, VIRTIO_MEM_MB_STATE_ONLINE) {
+ rc = virtio_mem_mb_unplug_any_sb_online(vm, mb_id,
+ &nb_sb);
+ if (rc || !nb_sb)
+ goto out_unlock;
+ mutex_unlock(&vm->hotplug_mutex);
+ cond_resched();
+ mutex_lock(&vm->hotplug_mutex);
+ }
+
mutex_unlock(&vm->hotplug_mutex);
- return 0;
+ return nb_sb ? -EBUSY : 0;
out_unlock:
mutex_unlock(&vm->hotplug_mutex);
return rc;
@@ -1330,7 +1429,8 @@ static void virtio_mem_run_wq(struct work_struct *work)
case -EBUSY:
/*
* The hypervisor cannot process our request right now
- * (e.g., out of memory, migrating).
+ * (e.g., out of memory, migrating) or we cannot free up
+ * any memory to unplug it (all plugged memory is busy).
*/
case -ENOMEM:
/* Out of memory, try again later. */
--
2.21.0
David Hildenbrand
2019-Sep-19 14:22 UTC
[PATCH RFC v3 6/9] mm: Allow to offline PageOffline() pages with a reference count of 0
virtio-mem wants to allow to offline memory blocks of which some parts
were unplugged, especially, to later offline and remove completely
unplugged memory blocks. The important part is that PageOffline() has
to remain set until the section is offline, so these pages will never
get accessed (e.g., when dumping). The pages should not be handed
back to the buddy (which would require clearing PageOffline() and
result in issues if offlining fails and the pages are suddenly in the
buddy).
Let's use "PageOffline() + reference count = 0" as a sign to
memory offlining code that these pages can simply be skipped when
offlining, similar to free or HWPoison pages.
Pass flags to test_pages_isolated(), similar as already done for
has_unmovable_pages(). Use a new flag to indicate the
requirement of memory offlining to skip over these special pages.
In has_unmovable_pages(), make sure the pages won't be detected as
movable. This is not strictly necessary, however makes e.g.,
alloc_contig_range() stop early, trying to isolate such page blocks -
compared to failing later when testing if all pages were isolated.
Also, make sure that when a reference to a PageOffline() page is
dropped, that the page will not be returned to the buddy.
memory devices (like virtio-mem) that want to make use of this
functionality have to make sure to synchronize against memory offlining,
using the memory hotplug notifier.
Alternative: Allow to offline with a reference count of 1
and use some other sign in the struct page that offlining is permitted.
Cc: Andrew Morton <akpm at linux-foundation.org>
Cc: Juergen Gross <jgross at suse.com>
Cc: David Hildenbrand <david at redhat.com>
Cc: Pavel Tatashin <pavel.tatashin at microsoft.com>
Cc: Alexander Duyck <alexander.h.duyck at linux.intel.com>
Cc: Anthony Yznaga <anthony.yznaga at oracle.com>
Cc: Vlastimil Babka <vbabka at suse.cz>
Cc: Johannes Weiner <hannes at cmpxchg.org>
Cc: Oscar Salvador <osalvador at suse.de>
Cc: Michal Hocko <mhocko at suse.com>
Cc: Pingfan Liu <kernelfans at gmail.com>
Cc: Qian Cai <cai at lca.pw>
Cc: Dan Williams <dan.j.williams at intel.com>
Cc: Mel Gorman <mgorman at techsingularity.net>
Cc: Mike Rapoport <rppt at linux.vnet.ibm.com>
Cc: Wei Yang <richardw.yang at linux.intel.com>
Cc: Alexander Potapenko <glider at google.com>
Cc: Anshuman Khandual <anshuman.khandual at arm.com>
Cc: Jason Gunthorpe <jgg at ziepe.ca>
Cc: Stephen Rothwell <sfr at canb.auug.org.au>
Cc: Mauro Carvalho Chehab <mchehab+samsung at kernel.org>
Cc: Matthew Wilcox <willy at infradead.org>
Cc: Yu Zhao <yuzhao at google.com>
Cc: Minchan Kim <minchan at kernel.org>
Cc: Yang Shi <yang.shi at linux.alibaba.com>
Cc: Ira Weiny <ira.weiny at intel.com>
Cc: Andrey Ryabinin <aryabinin at virtuozzo.com>
Signed-off-by: David Hildenbrand <david at redhat.com>
---
include/linux/page-flags.h | 4 ++++
include/linux/page-isolation.h | 4 +++-
mm/memory_hotplug.c | 9 ++++++---
mm/page_alloc.c | 22 +++++++++++++++++++++-
mm/page_isolation.c | 18 +++++++++++++-----
mm/swap.c | 9 +++++++++
6 files changed, 56 insertions(+), 10 deletions(-)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index f91cb8898ff0..7e563eab6b4b 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -745,6 +745,10 @@ PAGE_TYPE_OPS(Buddy, buddy)
* not onlined when onlining the section).
* The content of these pages is effectively stale. Such pages should not
* be touched (read/write/dump/save) except by their owner.
+ *
+ * PageOffline() pages that have a reference count of 0 will be treated
+ * like free pages when offlining pages, allowing the containing memory
+ * block to get offlined.
*/
PAGE_TYPE_OPS(Offline, offline)
diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h
index 1099c2fee20f..024e02b60346 100644
--- a/include/linux/page-isolation.h
+++ b/include/linux/page-isolation.h
@@ -32,6 +32,8 @@ static inline bool is_migrate_isolate(int migratetype)
#define SKIP_HWPOISON 0x1
#define REPORT_FAILURE 0x2
+/* Skip PageOffline() pages with a reference count of 0. */
+#define SKIP_OFFLINE 0x4
bool has_unmovable_pages(struct zone *zone, struct page *page, int count,
int migratetype, int flags);
@@ -58,7 +60,7 @@ undo_isolate_page_range(unsigned long start_pfn, unsigned long
end_pfn,
* Test all pages in [start_pfn, end_pfn) are isolated or not.
*/
int test_pages_isolated(unsigned long start_pfn, unsigned long end_pfn,
- bool skip_hwpoisoned_pages);
+ int flags);
struct page *alloc_migrate_target(struct page *page, unsigned long private);
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index f08eb429b8f3..d23ff7c5c96b 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1127,7 +1127,8 @@ static bool is_pageblock_removable_nolock(unsigned long
pfn)
if (!zone_spans_pfn(zone, pfn))
return false;
- return !has_unmovable_pages(zone, page, 0, MIGRATE_MOVABLE, SKIP_HWPOISON);
+ return !has_unmovable_pages(zone, page, 0, MIGRATE_MOVABLE,
+ SKIP_HWPOISON | SKIP_OFFLINE);
}
/* Checks if this range of memory is likely to be hot-removable. */
@@ -1344,7 +1345,8 @@ static int
check_pages_isolated_cb(unsigned long start_pfn, unsigned long nr_pages,
void *data)
{
- return test_pages_isolated(start_pfn, start_pfn + nr_pages, true);
+ return test_pages_isolated(start_pfn, start_pfn + nr_pages,
+ SKIP_HWPOISON | SKIP_OFFLINE);
}
static int __init cmdline_parse_movable_node(char *p)
@@ -1455,7 +1457,8 @@ static int __ref __offline_pages(unsigned long start_pfn,
/* set above range as isolated */
ret = start_isolate_page_range(start_pfn, end_pfn,
MIGRATE_MOVABLE,
- SKIP_HWPOISON | REPORT_FAILURE);
+ SKIP_HWPOISON | REPORT_FAILURE |
+ SKIP_OFFLINE);
if (ret < 0) {
reason = "failure to isolate range";
goto failed_removal;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index d5d7944954b3..fef74720d8b4 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -8221,6 +8221,15 @@ bool has_unmovable_pages(struct zone *zone, struct page
*page, int count,
if (!page_ref_count(page)) {
if (PageBuddy(page))
iter += (1 << page_order(page)) - 1;
+ /*
+ * Memory devices allow to offline a page if it is
+ * marked PG_offline and has a reference count of 0.
+ * However, the pages are not movable as it would be
+ * required e.g., for alloc_contig_range().
+ */
+ if (PageOffline(page) && !(flags & SKIP_OFFLINE))
+ if (++found > count)
+ goto unmovable;
continue;
}
@@ -8444,7 +8453,7 @@ int alloc_contig_range(unsigned long start, unsigned long
end,
}
/* Make sure the range is really isolated. */
- if (test_pages_isolated(outer_start, end, false)) {
+ if (test_pages_isolated(outer_start, end, 0)) {
pr_info_ratelimited("%s: [%lx, %lx) PFNs busy\n",
__func__, outer_start, end);
ret = -EBUSY;
@@ -8563,6 +8572,17 @@ __offline_isolated_pages(unsigned long start_pfn,
unsigned long end_pfn)
offlined_pages++;
continue;
}
+ /*
+ * Memory devices allow to offline a page if it is marked
+ * PG_offline and has a reference count of 0.
+ */
+ if (PageOffline(page) && !page_count(page)) {
+ BUG_ON(PageBuddy(page));
+ pfn++;
+ SetPageReserved(page);
+ offlined_pages++;
+ continue;
+ }
BUG_ON(page_count(page));
BUG_ON(!PageBuddy(page));
diff --git a/mm/page_isolation.c b/mm/page_isolation.c
index 89c19c0feadb..0a75019d7e7c 100644
--- a/mm/page_isolation.c
+++ b/mm/page_isolation.c
@@ -171,6 +171,8 @@ __first_valid_page(unsigned long pfn, unsigned long
nr_pages)
* SKIP_HWPOISON - ignore hwpoison pages
* REPORT_FAILURE - report details about the failure to
* isolate the range
+ * SKIP_OFFLINE - ignore PageOffline() pages with a
+ * reference count of 0
*
* Making page-allocation-type to be MIGRATE_ISOLATE means free pages in
* the range will never be allocated. Any free pages and pages freed in the
@@ -257,7 +259,7 @@ void undo_isolate_page_range(unsigned long start_pfn,
unsigned long end_pfn,
*/
static unsigned long
__test_page_isolated_in_pageblock(unsigned long pfn, unsigned long end_pfn,
- bool skip_hwpoisoned_pages)
+ int flags)
{
struct page *page;
@@ -274,9 +276,16 @@ __test_page_isolated_in_pageblock(unsigned long pfn,
unsigned long end_pfn,
* simple way to verify that as VM_BUG_ON(), though.
*/
pfn += 1 << page_order(page);
- else if (skip_hwpoisoned_pages && PageHWPoison(page))
+ else if ((flags & SKIP_HWPOISON) && PageHWPoison(page))
/* A HWPoisoned page cannot be also PageBuddy */
pfn++;
+ else if ((flags & SKIP_OFFLINE) && PageOffline(page) &&
+ !page_count(page))
+ /*
+ * Memory devices allow to offline a page if it is
+ * marked PG_offline and has a reference count of 0.
+ */
+ pfn++;
else
break;
}
@@ -286,7 +295,7 @@ __test_page_isolated_in_pageblock(unsigned long pfn,
unsigned long end_pfn,
/* Caller should ensure that requested range is in a single zone */
int test_pages_isolated(unsigned long start_pfn, unsigned long end_pfn,
- bool skip_hwpoisoned_pages)
+ int isol_flags)
{
unsigned long pfn, flags;
struct page *page;
@@ -308,8 +317,7 @@ int test_pages_isolated(unsigned long start_pfn, unsigned
long end_pfn,
/* Check all pages are free or marked as ISOLATED */
zone = page_zone(page);
spin_lock_irqsave(&zone->lock, flags);
- pfn = __test_page_isolated_in_pageblock(start_pfn, end_pfn,
- skip_hwpoisoned_pages);
+ pfn = __test_page_isolated_in_pageblock(start_pfn, end_pfn, isol_flags);
spin_unlock_irqrestore(&zone->lock, flags);
trace_test_pages_isolated(start_pfn, end_pfn, pfn);
diff --git a/mm/swap.c b/mm/swap.c
index 38c3fa4308e2..f98987656ecc 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -107,6 +107,15 @@ void __put_page(struct page *page)
* not return it to page allocator.
*/
return;
+ } else if (PageOffline(page)) {
+ /*
+ * Memory devices allow to offline a page if it is
+ * marked PG_offline and has a reference count of 0. So if
+ * somebody puts a reference of such a page and the
+ * reference count drops to 0, don't return the page to the
+ * buddy.
+ */
+ return;
}
if (unlikely(PageCompound(page)))
--
2.21.0
David Hildenbrand
2019-Sep-19 14:22 UTC
[PATCH RFC v3 7/9] virtio-mem: Allow to offline partially unplugged memory blocks
Dropping the reference count of PageOffline() pages allows offlining
code to skip them. However, we also have to convert PG_reserved to
another flag - let's use PG_dirty - so has_unmovable_pages() will
properly handle them. PG_reserved pages get detected as unmovable right
away.
We need the flag to see if we are onlining pages the first time, or if
we allocated them via alloc_contig_range().
Properly take care of offlining code also modifying the stats and
special handling in case the driver gets unloaded.
Cc: "Michael S. Tsirkin" <mst at redhat.com>
Cc: Jason Wang <jasowang at redhat.com>
Cc: Oscar Salvador <osalvador at suse.de>
Cc: Michal Hocko <mhocko at kernel.org>
Cc: Igor Mammedov <imammedo at redhat.com>
Cc: Dave Young <dyoung at redhat.com>
Cc: Andrew Morton <akpm at linux-foundation.org>
Cc: Dan Williams <dan.j.williams at intel.com>
Cc: Pavel Tatashin <pasha.tatashin at soleen.com>
Cc: Stefan Hajnoczi <stefanha at redhat.com>
Cc: Vlastimil Babka <vbabka at suse.cz>
Signed-off-by: David Hildenbrand <david at redhat.com>
---
drivers/virtio/virtio_mem.c | 102 ++++++++++++++++++++++++++++++++----
1 file changed, 92 insertions(+), 10 deletions(-)
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index 91052a37d10d..9cb31459b211 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -561,6 +561,30 @@ static void virtio_mem_notify_online(struct virtio_mem *vm,
unsigned long mb_id,
virtio_mem_retry(vm);
}
+/*
+ * When we unplug subblocks, we already modify stats (e.g., subtract them
+ * from totalram_pages). Offlining code will modify the stats, too. So
+ * properly fixup the stats when GOING_OFFLINE and revert that when
+ * CANCEL_OFFLINE.
+ */
+static void virtio_mem_mb_going_offline_fixup_stats(struct virtio_mem *vm,
+ unsigned long mb_id,
+ bool cancel)
+{
+ const unsigned long nr_pages = PFN_DOWN(vm->subblock_size);
+ int sb_id;
+
+ for (sb_id = 0; sb_id < vm->nb_sb_per_mb; sb_id++) {
+ if (virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id, 1))
+ continue;
+
+ if (cancel)
+ totalram_pages_add(-nr_pages);
+ else
+ totalram_pages_add(nr_pages);
+ }
+}
+
/*
* This callback will either be called synchonously from add_memory() or
* asynchronously (e.g., triggered via user space). We have to be careful
@@ -608,6 +632,7 @@ static int virtio_mem_memory_notifier_cb(struct
notifier_block *nb,
mutex_lock(&vm->hotplug_mutex);
vm->hotplug_active = true;
}
+ virtio_mem_mb_going_offline_fixup_stats(vm, mb_id, false);
break;
case MEM_GOING_ONLINE:
spin_lock_irq(&vm->removal_lock);
@@ -633,6 +658,8 @@ static int virtio_mem_memory_notifier_cb(struct
notifier_block *nb,
mutex_unlock(&vm->hotplug_mutex);
break;
case MEM_CANCEL_OFFLINE:
+ virtio_mem_mb_going_offline_fixup_stats(vm, mb_id, true);
+ /* fall through */
case MEM_CANCEL_ONLINE:
/* We might not get a MEM_GOING* if somebody else canceled */
if (vm->hotplug_active) {
@@ -648,23 +675,55 @@ static int virtio_mem_memory_notifier_cb(struct
notifier_block *nb,
}
/*
- * Set a range of pages PG_offline.
+ * Convert PG_reserved to PG_dirty. Needed to allow isolation code to
+ * not immediately consider them as unmovable.
+ */
+static void virtio_mem_reserved_to_dirty(unsigned long pfn,
+ unsigned int nr_pages)
+{
+ for (; nr_pages--; pfn++) {
+ SetPageDirty(pfn_to_page(pfn));
+ ClearPageReserved(pfn_to_page(pfn));
+ }
+}
+
+/*
+ * Convert PG_dirty to PG_reserved. Needed so generic_online_page()
+ * works correctly.
+ */
+static void virtio_mem_dirty_to_reserved(unsigned long pfn,
+ unsigned int nr_pages)
+{
+ for (; nr_pages--; pfn++) {
+ SetPageReserved(pfn_to_page(pfn));
+ ClearPageDirty(pfn_to_page(pfn));
+ }
+}
+
+/*
+ * Set a range of pages PG_offline and drop the reference. The dropped
+ * reference (0) and the flag allows isolation code to isolate this range
+ * and offline code to offline it.
*/
static void virtio_mem_set_fake_offline(unsigned long pfn,
unsigned int nr_pages)
{
- for (; nr_pages--; pfn++)
+ for (; nr_pages--; pfn++) {
__SetPageOffline(pfn_to_page(pfn));
+ page_ref_dec(pfn_to_page(pfn));
+ }
}
/*
- * Clear PG_offline from a range of pages.
+ * Get a reference and clear PG_offline from a range of pages.
*/
static void virtio_mem_clear_fake_offline(unsigned long pfn,
unsigned int nr_pages)
{
- for (; nr_pages--; pfn++)
+ for (; nr_pages--; pfn++) {
+ page_ref_inc(pfn_to_page(pfn));
__ClearPageOffline(pfn_to_page(pfn));
+ }
}
/*
@@ -679,7 +738,7 @@ static void virtio_mem_fake_online(unsigned long pfn,
unsigned int nr_pages)
/*
* We are always called with subblock granularity, which is at least
* aligned to MAX_ORDER - 1. All pages in a subblock are either
- * reserved or not.
+ * PG_dirty (converted PG_reserved) or not.
*/
BUG_ON(!IS_ALIGNED(pfn, 1 << order));
BUG_ON(!IS_ALIGNED(nr_pages, 1 << order));
@@ -690,13 +749,14 @@ static void virtio_mem_fake_online(unsigned long pfn,
unsigned int nr_pages)
struct page *page = pfn_to_page(pfn + i);
/*
- * If the page is reserved, it was kept fake-offline when
+ * If the page is PG_dirty, it was kept fake-offline when
* onlining the memory block. Otherwise, it was allocated
* using alloc_contig_range().
*/
- if (PageReserved(page))
+ if (PageDirty(page)) {
+ virtio_mem_dirty_to_reserved(pfn + i, 1 << order);
generic_online_page(page, order);
- else {
+ } else {
free_contig_range(pfn + i, 1 << order);
totalram_pages_add(1 << order);
}
@@ -728,8 +788,10 @@ static void virtio_mem_online_page_cb(struct page *page,
unsigned int order)
*/
if (virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id, 1))
generic_online_page(page, order);
- else
+ else {
virtio_mem_set_fake_offline(PFN_DOWN(addr), 1 << order);
+ virtio_mem_reserved_to_dirty(PFN_DOWN(addr), 1 << order);
+ }
rcu_read_unlock();
return;
}
@@ -1674,7 +1736,8 @@ static int virtio_mem_probe(struct virtio_device *vdev)
static void virtio_mem_remove(struct virtio_device *vdev)
{
struct virtio_mem *vm = vdev->priv;
- unsigned long mb_id;
+ unsigned long nr_pages = PFN_DOWN(vm->subblock_size);
+ unsigned long pfn, mb_id, sb_id;
int rc;
/*
@@ -1701,6 +1764,25 @@ static void virtio_mem_remove(struct virtio_device *vdev)
BUG_ON(rc);
mutex_lock(&vm->hotplug_mutex);
}
+ /*
+ * After we unregistered our callbacks, user space can offline +
+ * re-online partially plugged online blocks. Make sure they can't
+ * get offlined by getting a reference. Also, restore PG_reserved.
+ */
+ virtio_mem_for_each_mb_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL) {
+ for (sb_id = 0; sb_id < vm->nb_sb_per_mb; sb_id++) {
+ if (virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id, 1))
+ continue;
+ pfn = PFN_DOWN(virtio_mem_mb_id_to_phys(mb_id) +
+ sb_id * vm->subblock_size);
+
+ if (PageDirty(pfn_to_page(pfn)))
+ virtio_mem_dirty_to_reserved(pfn, nr_pages);
+ for (; nr_pages--; pfn++)
+ page_ref_inc(pfn_to_page(pfn));
+ }
+ }
mutex_unlock(&vm->hotplug_mutex);
/* unregister callbacks */
--
2.21.0
David Hildenbrand
2019-Sep-19 14:22 UTC
[PATCH RFC v3 8/9] mm/memory_hotplug: Introduce offline_and_remove_memory()
virtio-mem wants to offline and remove a memory block once it unplugged
all subblocks (e.g., using alloc_contig_range()). Let's provide
an interface to do that from a driver. virtio-mem already supports to
offline partially unplugged memory blocks. Offlining a fully unplugged
memory block will not require to migrate any pages. All unplugged
subblocks are PageOffline() and have a reference count of 0 - so
offlining code will simply skip them.
All we need an interface to trigger the "offlining" and the removing
in a
single operation - to make sure the memory block cannot get onlined by
user space again before it gets removed.
To keep things simple, allow to only work on a single memory block.
Cc: Andrew Morton <akpm at linux-foundation.org>
Cc: David Hildenbrand <david at redhat.com>
Cc: Oscar Salvador <osalvador at suse.com>
Cc: Michal Hocko <mhocko at suse.com>
Cc: Pavel Tatashin <pasha.tatashin at soleen.com>
Cc: Wei Yang <richard.weiyang at gmail.com>
Cc: Dan Williams <dan.j.williams at intel.com>
Cc: Qian Cai <cai at lca.pw>
Signed-off-by: David Hildenbrand <david at redhat.com>
---
include/linux/memory_hotplug.h | 1 +
mm/memory_hotplug.c | 35 ++++++++++++++++++++++++++++++++++
2 files changed, 36 insertions(+)
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 384ffb3d69ab..a1bf868aaeba 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -313,6 +313,7 @@ extern void try_offline_node(int nid);
extern int offline_pages(unsigned long start_pfn, unsigned long nr_pages);
extern int remove_memory(int nid, u64 start, u64 size);
extern void __remove_memory(int nid, u64 start, u64 size);
+extern int offline_and_remove_memory(int nid, u64 start, u64 size);
#else
static inline bool is_mem_section_removable(unsigned long pfn,
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index d23ff7c5c96b..caf3c93f5f7c 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1742,4 +1742,39 @@ int remove_memory(int nid, u64 start, u64 size)
return rc;
}
EXPORT_SYMBOL_GPL(remove_memory);
+
+/*
+ * Try to offline and remove a memory block. Might take a long time to
+ * finish in case memory is still in use. Primarily useful for memory devices
+ * that logically unplugged all memory (so it's no longer in use) and want
to
+ * offline + remove the memory block.
+ */
+int offline_and_remove_memory(int nid, u64 start, u64 size)
+{
+ struct memory_block *mem;
+ int rc = -EINVAL;
+
+ if (!IS_ALIGNED(start, memory_block_size_bytes()) ||
+ size != memory_block_size_bytes())
+ return rc;
+
+ lock_device_hotplug();
+ mem = find_memory_block(__pfn_to_section(PFN_DOWN(start)));
+ if (mem)
+ rc = device_offline(&mem->dev);
+ /* Ignore if the device is already offline. */
+ if (rc > 0)
+ rc = 0;
+
+ /*
+ * In case we succeeded to offline the memory block, remove it.
+ * This cannot fail as it cannot get onlined in the meantime.
+ */
+ if (!rc && try_remove_memory(nid, start, size))
+ BUG();
+ unlock_device_hotplug();
+
+ return rc;
+}
+EXPORT_SYMBOL_GPL(offline_and_remove_memory);
#endif /* CONFIG_MEMORY_HOTREMOVE */
--
2.21.0
David Hildenbrand
2019-Sep-19 14:22 UTC
[PATCH RFC v3 9/9] virtio-mem: Offline and remove completely unplugged memory blocks
Let's offline+remove memory blocks once all subblocks are unplugged. We
can use the new Linux MM interface for that. As no memory is in use
anymore, this shouldn't take a long time and shouldn't fail. There might
be corner cases where the offlining could still fail (especially, if
another notifier NACKs the offlining request).
Cc: "Michael S. Tsirkin" <mst at redhat.com>
Cc: Jason Wang <jasowang at redhat.com>
Cc: Oscar Salvador <osalvador at suse.de>
Cc: Michal Hocko <mhocko at kernel.org>
Cc: Igor Mammedov <imammedo at redhat.com>
Cc: Dave Young <dyoung at redhat.com>
Cc: Andrew Morton <akpm at linux-foundation.org>
Cc: Dan Williams <dan.j.williams at intel.com>
Cc: Pavel Tatashin <pasha.tatashin at soleen.com>
Cc: Stefan Hajnoczi <stefanha at redhat.com>
Cc: Vlastimil Babka <vbabka at suse.cz>
Signed-off-by: David Hildenbrand <david at redhat.com>
---
drivers/virtio/virtio_mem.c | 39 ++++++++++++++++++++++++++++++++++---
1 file changed, 36 insertions(+), 3 deletions(-)
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index 9cb31459b211..01d5fc784d5d 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -433,6 +433,28 @@ static int virtio_mem_mb_remove(struct virtio_mem *vm,
unsigned long mb_id)
return remove_memory(nid, addr, memory_block_size_bytes());
}
+/*
+ * Try to offline and remove a memory block from Linux.
+ *
+ * Must not be called with the vm->hotplug_mutex held (possible deadlock
with
+ * onlining code).
+ *
+ * Will not modify the state of the memory block.
+ */
+static int virtio_mem_mb_offline_and_remove(struct virtio_mem *vm,
+ unsigned long mb_id)
+{
+ const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id);
+ int nid = vm->nid;
+
+ if (nid == NUMA_NO_NODE)
+ nid = memory_add_physaddr_to_nid(addr);
+
+ dev_dbg(&vm->vdev->dev, "offlining and removing memory block:
%lu\n",
+ mb_id);
+ return offline_and_remove_memory(nid, addr, memory_block_size_bytes());
+}
+
/*
* Trigger the workqueue so the device can perform its magic.
*/
@@ -1263,7 +1285,8 @@ static int virtio_mem_mb_unplug_any_sb_offline(struct
virtio_mem *vm,
* Unplug the desired number of plugged subblocks of an online memory block.
* Will skip subblock that are busy.
*
- * Will modify the state of the memory block.
+ * Will modify the state of the memory block. Might temporarily drop the
+ * hotplug_mutex.
*
* Note: Can fail after some subblocks were successfully unplugged. Can
* return 0 even if subblocks were busy and could not get unplugged.
@@ -1319,9 +1342,19 @@ static int virtio_mem_mb_unplug_any_sb_online(struct
virtio_mem *vm,
}
/*
- * TODO: Once all subblocks of a memory block were unplugged, we want
- * to offline the memory block and remove it.
+ * Once all subblocks of a memory block were unplugged, offline and
+ * remove it. This will usually not fail, as no memory is in use
+ * anymore - however some other notifiers might NACK the request.
*/
+ if (virtio_mem_mb_test_sb_unplugged(vm, mb_id, 0, vm->nb_sb_per_mb)) {
+ mutex_unlock(&vm->hotplug_mutex);
+ rc = virtio_mem_mb_offline_and_remove(vm, mb_id);
+ mutex_lock(&vm->hotplug_mutex);
+ if (!rc)
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_UNUSED);
+ }
+
return 0;
}
--
2.21.0
On 19.09.19 16:22, David Hildenbrand wrote:> Will be needed by virtio-mem to identify the node from a pxm. > > Cc: "Rafael J. Wysocki" <rjw at rjwysocki.net> > Cc: Len Brown <lenb at kernel.org> > Cc: linux-acpi at vger.kernel.org > Signed-off-by: David Hildenbrand <david at redhat.com> > --- > drivers/acpi/numa.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/drivers/acpi/numa.c b/drivers/acpi/numa.c > index eadbf90e65d1..d5847fa7ac69 100644 > --- a/drivers/acpi/numa.c > +++ b/drivers/acpi/numa.c > @@ -35,6 +35,7 @@ int pxm_to_node(int pxm) > return NUMA_NO_NODE; > return pxm_to_node_map[pxm]; > } > +EXPORT_SYMBOL(pxm_to_node);FWIW, this is a fairly old patch I dragged along and I think I'll convert this to EXPORT_SYMBOL_GPL now that I know better :)> > int node_to_pxm(int node) > { >-- Thanks, David / dhildenb
David Hildenbrand
2019-Oct-16 08:12 UTC
[PATCH RFC v3 0/9] virtio-mem: paravirtualized memory
On 19.09.19 16:22, David Hildenbrand wrote:> Long time no RFC! I finally had time to get the next version of the Linux > driver side of virtio-mem into shape, incorporating ideas and feedback from > previous discussions. > > This RFC is based on the series currently on the mm list: > - [PATCH 0/3] Remove __online_page_set_limits() > - [PATCH v1 0/3] mm/memory_hotplug: Export generic_online_page() > - [PATCH v4 0/8] mm/memory_hotplug: Shrink zones before removing memory > The current state (kept updated) is available on > - https://github.com/davidhildenbrand/linux/tree/virtio-mem > > The basic idea of virtio-mem is to provide a flexible, > cross-architecture memory hot(un)plug solution that avoids many limitations > imposed by existing technologies, architectures, and interfaces. More > details can be found below and in linked material. > > This RFC was only tested on x86-64, however, should theoretically work > on any Linux architecture that implements memory hot(un)plug - like > s390x. On x86-64, it is currently possible to add/remove memory to the > system in >= 4MB granularity. Memory hotplug works very reliable. For > memory unplug, there are no guarantees how much memory can actually get > unplugged, it depends on the setup. I have plans to improve that in the > future. > > -------------------------------------------------------------------------- > 1. virtio-mem > -------------------------------------------------------------------------- > > The basic idea behind virtio-mem was presented at KVM Forum 2018. The > slides can be found at [1]. The previous RFC can be found at [2]. The > first RFC can be found at [3]. However, the concept evolved over time. The > KVM Forum slides roughly match the current design. > > Patch #2 ("virtio-mem: Paravirtualized memory hotplug") contains quite some > information, especially in "include/uapi/linux/virtio_mem.h": > > Each virtio-mem device manages a dedicated region in physical address > space. Each device can belong to a single NUMA node, multiple devices > for a single NUMA node are possible. A virtio-mem device is like a > "resizable DIMM" consisting of small memory blocks that can be plugged > or unplugged. The device driver is responsible for (un)plugging memory > blocks on demand. > > Virtio-mem devices can only operate on their assigned memory region in > order to (un)plug memory. A device cannot (un)plug memory belonging to > other devices. > > The "region_size" corresponds to the maximum amount of memory that can > be provided by a device. The "size" corresponds to the amount of memory > that is currently plugged. "requested_size" corresponds to a request > from the device to the device driver to (un)plug blocks. The > device driver should try to (un)plug blocks in order to reach the > "requested_size". It is impossible to plug more memory than requested. > > The "usable_region_size" represents the memory region that can actually > be used to (un)plug memory. It is always at least as big as the > "requested_size" and will grow dynamically. It will only shrink when > explicitly triggered (VIRTIO_MEM_REQ_UNPLUG). > > Memory in the usable region can usually be read, however, there are no > guarantees. It can happen that the device cannot process a request, > because it is busy. The device driver has to retry later. > > Usually, during system resets all memory will get unplugged, so the > device driver can start with a clean state. However, in specific > scenarios (if the device is busy) it can happen that the device still > has memory plugged. The device driver can request to unplug all memory > (VIRTIO_MEM_REQ_UNPLUG) - which might take a while to succeed if the > device is busy. > > -------------------------------------------------------------------------- > 2. Linux Implementation > -------------------------------------------------------------------------- > > This RFC reuses quite some existing MM infrastructure, however, has to > expose some additional functionality. > > Memory blocks (e.g., 128MB) are added/removed on demand. Within these > memory blocks, subblocks (e.g., 4MB) are plugged/unplugged. The sizes > depend on the target architecture, MAX_ORDER + pageblock_order, and > the block size of a virtio-mem device. > > add_memory()/try_remove_memory() is used to add/remove memory blocks. > virtio-mem will not online memory blocks itself. This has to be done by > user space, or configured into the kernel > (CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE). virtio-mem will only unplug memory > that was online to the ZONE_NORMAL. Memory is suggested to be onlined to > the ZONE_NORMAL for now. > > The memory hotplug notifier is used to properly synchronize against > onlining/offlining of memory blocks and to track the states of memory > blocks (including the zone memory blocks are onlined to). Locking is > done similar to the PPC CMA implementation. > > The set_online_page() callback is used to keep unplugged subblocks > of a memory block fake-offline when onlining the memory block. > generic_online_page() is used to fake-online plugged subblocks. This > handling is similar to the Hyper-V balloon driver. > > PG_offline is used to mark unplugged subblocks as offline, so e.g., > dumping tools (makedumpfile) will skip these pages. This is similar to > other balloon drivers like virtio-balloon and Hyper-V. > > PG_offline + reference count of 0 [new] is now also used to mark pages as > a "skip" when offlining memory blocks. This allows to offline memory blocks > that have partially unplugged subblocks - or are completely unplugged. > > alloc_contig_range()/free_contig_range() [now exposed] is used to > unplug/plug subblocks of memory blocks the are already exposed to Linux. > > offline_and_remove_memory() [new] is used to offline a fully unplugged > memory block and remove it from Linux. > > > A lot of additional information can be found in the separate patches and > as comments in the code itself. > > -------------------------------------------------------------------------- > 3. Major changes since last RFC > -------------------------------------------------------------------------- > > A lot of things changed, especially also on the QEMU + virtio side. The > biggest changes on the Linux driver side are: > - Onlining/offlining of subblocks is now emulated on top of memory blocks. > set_online_page()+alloc_contig_range()+free_contig_range() is now used > for that. Core MM does not have to be modified and will continue to > online/offline full memory blocks. > - Onlining/offlining of memory blocks is no longer performed by virtio-mem. > - Pg_offline is upstream and can be used. It is also used to allow > offlining of partially unplugged memory blocks. > - Memory block states + subblocks are now tracked more space-efficient. > - Proper kexec(), kdump(), driver unload, driver reload, ZONE_MOVABLE, ... > handling. > > -------------------------------------------------------------------------- > 4. Future work > -------------------------------------------------------------------------- > > The separate patches contain a lot of future work items. One of the next > steps is to make memory unplug more likely to succeed - currently, there > are no guarantees on how much memory can get unplugged again. I have > various ideas on how to limit fragmentation of all memory blocks that > virtio-mem added. > > Memory hotplug: > - Reduce the amount of memory resources if that turnes out to be an > issue. Or try to speed up relevant code paths to deal with many > resources. > - Allocate the vmemmap from the added memory. Makes hotplug more likely > to succeed, the vmemmap is stored on the same NUMA node and that > unmovable memory will later not hinder unplug. > > Memory hotunplug: > - Performance improvements: > -- Sense (lockless) if it make sense to try alloc_contig_range() at all > before directly trying to isolate and taking locks. > -- Try to unplug bigger chunks if possible first. > -- Identify free areas first, that don't have to be evacuated. > - Make unplug more likely to succeed: > -- The "issue" is that in the ZONE_NORMAL, the buddy will randomly > allocate memory. Only pageblocks somewhat limit fragmentation, > however we would want to limit fragmentation on subblock granularity > and even memory block granularity. One idea is to have a new > ZONE_PREFER_MOVABLE. Memory blocks will then be onlined to ZONE_NORMAL > / ZONE_PREFER_MOVABLE in a certain ratio per node (e.g., > 1:4). This makes unplug of quite some memory likely to succeed in most > setups. ZONE_PREFER_MOVABLE is then a mixture of ZONE_NORMAL and > ZONE_MOVABlE. Especially, movable data can end up on that zone, but > only if really required - avoiding running out of memory on ZONE > imbalances. The zone fallback order would be > MOVABLE=>PREFER_MOVABLE=>HIGHMEM=>NORMAL=>PREFER_MOVABLE=>DMA32=>DMA > -- Allocate memmap from added memory. This way, less unmovable data can > end up on the memory blocks. > -- Call drop_slab() before trying to unplug. Eventually shrink other > caches. > - Better retry handling in case memory is busy. We certainly don't want > to try for ever in a short interval to try to get some memory back. > - OOM handling, e.g., via an OOM handler. > > -------------------------------------------------------------------------- > 5. Example Usage > -------------------------------------------------------------------------- > > A very basic QEMU prototype (kept updated) is available at: > - https://github.com/davidhildenbrand/qemu/tree/virtio-mem > > It lacks various features, however works to test the guest driver side: > - No support for resizable memory regions / memory backends > - No protection of unplugged memory (esp., userfaultfd-wp) > - No dump/migration/XXX optimizations to skip over unplugged memory > > Start QEMU with two virtio-mem devices (one per NUMA node): > $ qemu-system-x86_64 -m 4G,maxmem=20G \ > -smp sockets=2,cores=2 \ > -numa node,nodeid=0,cpus=0-1 -numa node,nodeid=1,cpus=2-3 \ > [...] > -object memory-backend-ram,id=mem0,size=8G \ > -device virtio-mem-pci,id=vm0,memdev=mem0,node=0,requested-size=128M \ > -object memory-backend-ram,id=mem1,size=8G \ > -device virtio-mem-pci,id=vm1,memdev=mem1,node=1,requested-size=80M > > Query the configuration: > QEMU 4.1.50 monitor - type 'help' for more information > (qemu) info memory-devices > Memory device [virtio-mem]: "vm0" > memaddr: 0x140000000 > node: 0 > requested-size: 134217728 > size: 134217728 > max-size: 8589934592 > block-size: 2097152 > memdev: /objects/mem0 > Memory device [virtio-mem]: "vm1" > memaddr: 0x340000000 > node: 1 > requested-size: 83886080 > size: 83886080 > max-size: 8589934592 > block-size: 2097152 > memdev: /objects/mem1 > > Add some memory to node 1: > QEMU 4.1.50 monitor - type 'help' for more information > (qemu) qom-set vm1 requested-size 1G > > Remove some memory from node 0: > QEMU 4.1.50 monitor - type 'help' for more information > (qemu) qom-set vm0 requested-size 64M > > Query the configuration: > (qemu) info memory-devices > Memory device [virtio-mem]: "vm0" > memaddr: 0x140000000 > node: 0 > requested-size: 67108864 > size: 67108864 > max-size: 8589934592 > block-size: 2097152 > memdev: /objects/mem0 > Memory device [virtio-mem]: "vm1" > memaddr: 0x340000000 > node: 1 > requested-size: 1073741824 > size: 1073741824 > max-size: 8589934592 > block-size: 2097152 > memdev: /objects/mem1 > > -------------------------------------------------------------------------- > 6. Q/A > -------------------------------------------------------------------------- > > Q: Why add/remove parts ("subblocks") of memory blocks/sections? > A: Flexibility (section size depends on the architecture) - e.g., some > architectures have a section size of 2GB. Also, the memory block size > is variable (e.g., on x86-64). I want to avoid any such restrictions. > Some use cases want to add/remove memory in smaller granularities to a > VM (e.g., the Hyper-V balloon also implements this) - especially smaller > VMs like used for kata containers. Also, on memory unplug, it is more > reliable to free-up and unplug multiple small chunks instead > of one big chunk. E.g., if one page of a DIMM is either unmovable or > pinned, the DIMM can't get unplugged. This approach is basically a > compromise between DIMM-based memory hot(un)plug and balloon > inflation/deflation, which works mostly on page granularity. > > Q: Why care about memory blocks? > A: They are the way to tell user space about new memory. This way, > memory can get onlined/offlined by user space. Also, e.g., kdump > relies on udev events to reload kexec when memory blocks are > onlined/offlined. Memory blocks are the "real" memory hot(un)plug > granularity. Everything that's smaller has to be emulated "on top". > > Q: Won't memory unplug of subblocks fragment memory? > A: Yes and no. Unplugging e.g., >=4MB subblocks on x86-64 will not really > fragment memory like unplugging random pages like a balloon driver does. > Buddy merging will not be limited. However, any allocation that requires > bigger consecutive memory chunks (e.g., gigantic pages) might observe > the fragmentation. Possible solutions: Allocate gigantic huge pages > before unplugging memory, don't unplug memory, combine virtio-mem with > DIMM based memory or bigger initial memory. Remember, a virtio-mem > device will only unplug on the memory range it manages, not on other > DIMMs. Unplug of single memory blocks will result in similar > fragmentation in respect to gigantic huge pages. > > Q: How reliable is memory unplug? > A: There are no guarantees on how much memory can get unplugged > again. However, it is more likely to find 4MB chunks to unplug than > e.g., 128MB chunks. If memory is terribly fragmented, there is nothing > we can do - for now. I consider memory hotplug the first primary use > of virtio-mem. Memory unplug might usually work, but we want to improve > the performance and the amount of memory we can actually unplug later. > > Q: Why not unplug from the ZONE_MOVABLE? > A: Unplugged memory chunks are unmovable. Unmovable data must not end up > on the ZONE_MOVABLE - similar to gigantic pages - they will never be > allocated from ZONE_MOVABLE. Teaching MM to move unplugged chunks within > a device might be problematic and will require a new guest->hypervisor > command to move unplugged chunks. virtio-mem added memory can be onlined > to the ZONE_MOVABLE, but subblocks will not get unplugged from it. > > Q: How big should the initial (!virtio-mem) memory of a VM be? > A: virtio-mem memory will not go to the DMA zones. So to avoid running out > of DMA memory, I suggest something like 2-3GB on x86-64. But many > VMs can most probably deal with less DMA memory - depends on the use > case. > > [1] https://events.linuxfoundation.org/wp-content/uploads/2017/12/virtio-mem-Paravirtualized-Memory-David-Hildenbrand-Red-Hat-1.pdf > [2] https://lwn.net/Articles/755423/ > [3] https://lists.gnu.org/archive/html/qemu-devel/2017-06/msg03870.html > > --- >Gentle ping. I want to continue working on this. I need feedback from MM people on ...> David Hildenbrand (9): > ACPI: NUMA: export pxm_to_node > virtio-mem: Paravirtualized memory hotplug > virtio-mem: Paravirtualized memory hotunplug part 1 > mm: Export alloc_contig_range() / free_contig_range()... this patch, if it is okay to export alloc_contig_range() / free_contig_range() (see next patch how it is used)> virtio-mem: Paravirtualized memory hotunplug part 2 > mm: Allow to offline PageOffline() pages with a reference count of 0... this patch, to see if we can use "PageOffline() + refcount == 0" as a way to allow offlining memory with unplugged pieces. (see next patch how this is used)> virtio-mem: Allow to offline partially unplugged memory blocks > mm/memory_hotplug: Introduce offline_and_remove_memory()... I assume this patch is not that debatable.> virtio-mem: Offline and remove completely unplugged memory blocksSo yeah, please feedback on "mm: Export alloc_contig_range() / free_contig_range()" and "mm: Allow to offline PageOffline() pages with a reference count of 0", because they are the essential pieces apart from what we already have (generic_online_page(), memory notifiers ...) Thanks! -- Thanks, David / dhildenb
Michal Hocko
2019-Oct-16 11:20 UTC
[PATCH RFC v3 4/9] mm: Export alloc_contig_range() / free_contig_range()
On Thu 19-09-19 16:22:23, David Hildenbrand wrote:> A virtio-mem device wants to allocate memory from the memory region it > manages in order to unplug it in the hypervisor - similar to > a balloon driver. Also, it might want to plug previously unplugged > (allocated) memory and give it back to Linux. alloc_contig_range() / > free_contig_range() seem to be the perfect interface for this task. > > In contrast to existing balloon devices, a virtio-mem device operates > on bigger chunks (e.g., 4MB) and only on physical memory it manages. It > tracks which chunks (subblocks) are still plugged, so it can go ahead > and try to alloc_contig_range()+unplug them on unplug request, or > plug+free_contig_range() unplugged chunks on plug requests. > > A virtio-mem device will use alloc_contig_range() / free_contig_range() > only on ranges that belong to the same node/zone in at least > MAX(MAX_ORDER - 1, pageblock_order) order granularity - e.g., 4MB on > x86-64. The virtio-mem device added that memory, so the memory > exists and does not contain any holes. virtio-mem will only try to allocate > on ZONE_NORMAL, never on ZONE_MOVABLE, just like when allocating > gigantic pages (we don't put unmovable data into the movable zone).Is there any real reason to export as GPL rather than generic EXPORT_SYMBOL? In other words do we need to restrict the usage this interface only to GPL modules and why if so. All other allocator APIs are EXPORT_SYMBOL so there should better be a good reason for this one to differ. I can understand that this one is slightly different by requesting a specific range of the memory but it is still under a full control of the core MM to say no.> Cc: Andrew Morton <akpm at linux-foundation.org> > Cc: Michal Hocko <mhocko at suse.com> > Cc: Vlastimil Babka <vbabka at suse.cz> > Cc: Oscar Salvador <osalvador at suse.de> > Cc: Mel Gorman <mgorman at techsingularity.net> > Cc: Mike Rapoport <rppt at linux.ibm.com> > Cc: Dan Williams <dan.j.williams at intel.com> > Cc: Alexander Duyck <alexander.h.duyck at linux.intel.com> > Cc: Pavel Tatashin <pavel.tatashin at microsoft.com> > Cc: Alexander Potapenko <glider at google.com> > Signed-off-by: David Hildenbrand <david at redhat.com>Other than that, I do not think exporting this function is harmful. It would be worse to reinvent it and do it wrong. I usually prefer to add a caller in the same patch, though, because it makes the usage explicit and clear. Acked-by: Michal Hocko <mhocko at suse.com> # to export contig range allocator API> --- > mm/page_alloc.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 3334a769eb91..d5d7944954b3 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -8469,6 +8469,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, > pfn_max_align_up(end), migratetype); > return ret; > } > +EXPORT_SYMBOL_GPL(alloc_contig_range); > #endif /* CONFIG_CONTIG_ALLOC */ > > void free_contig_range(unsigned long pfn, unsigned int nr_pages) > @@ -8483,6 +8484,7 @@ void free_contig_range(unsigned long pfn, unsigned int nr_pages) > } > WARN(count != 0, "%d pages are still in use!\n", count); > } > +EXPORT_SYMBOL_GPL(free_contig_range); > > #ifdef CONFIG_MEMORY_HOTPLUG > /* > -- > 2.21.0-- Michal Hocko SUSE Labs
Michal Hocko
2019-Oct-16 11:43 UTC
[PATCH RFC v3 6/9] mm: Allow to offline PageOffline() pages with a reference count of 0
On Thu 19-09-19 16:22:25, David Hildenbrand wrote:> virtio-mem wants to allow to offline memory blocks of which some parts > were unplugged, especially, to later offline and remove completely > unplugged memory blocks. The important part is that PageOffline() has > to remain set until the section is offline, so these pages will never > get accessed (e.g., when dumping). The pages should not be handed > back to the buddy (which would require clearing PageOffline() and > result in issues if offlining fails and the pages are suddenly in the > buddy). > > Let's use "PageOffline() + reference count = 0" as a sign to > memory offlining code that these pages can simply be skipped when > offlining, similar to free or HWPoison pages. > > Pass flags to test_pages_isolated(), similar as already done for > has_unmovable_pages(). Use a new flag to indicate the > requirement of memory offlining to skip over these special pages. > > In has_unmovable_pages(), make sure the pages won't be detected as > movable. This is not strictly necessary, however makes e.g., > alloc_contig_range() stop early, trying to isolate such page blocks - > compared to failing later when testing if all pages were isolated. > > Also, make sure that when a reference to a PageOffline() page is > dropped, that the page will not be returned to the buddy. > > memory devices (like virtio-mem) that want to make use of this > functionality have to make sure to synchronize against memory offlining, > using the memory hotplug notifier. > > Alternative: Allow to offline with a reference count of 1 > and use some other sign in the struct page that offlining is permitted.Few questions. I do not see onlining code to take care of this special case. What should happen when offline && online? Should we allow to try_remove_memory to succeed with these pages? Do we really have hook into __put_page? Why do we even care about the reference count of those pages? Wouldn't it be just more consistent to elevate the reference count (I guess this is what you suggest in the last paragraph) and the virtio driver would return that page to the buddy by regular put_page. This is also related to the above question about the physical memory remove. [...]> diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index d5d7944954b3..fef74720d8b4 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -8221,6 +8221,15 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count, > if (!page_ref_count(page)) { > if (PageBuddy(page)) > iter += (1 << page_order(page)) - 1; > + /* > + * Memory devices allow to offline a page if it is > + * marked PG_offline and has a reference count of 0. > + * However, the pages are not movable as it would be > + * required e.g., for alloc_contig_range(). > + */ > + if (PageOffline(page) && !(flags & SKIP_OFFLINE)) > + if (++found > count) > + goto unmovable; > continue; > }Do we really need to distinguish offline and hwpoison pages? They are both unmovable for allocator purposes and offlineable for the hotplug, right? Should we just hide them behind a helper and use it rather than an explicit SKIP_$FOO? -- Michal Hocko SUSE Labs
Michal Hocko
2019-Oct-16 11:47 UTC
[PATCH RFC v3 8/9] mm/memory_hotplug: Introduce offline_and_remove_memory()
On Thu 19-09-19 16:22:27, David Hildenbrand wrote:> virtio-mem wants to offline and remove a memory block once it unplugged > all subblocks (e.g., using alloc_contig_range()). Let's provide > an interface to do that from a driver. virtio-mem already supports to > offline partially unplugged memory blocks. Offlining a fully unplugged > memory block will not require to migrate any pages. All unplugged > subblocks are PageOffline() and have a reference count of 0 - so > offlining code will simply skip them. > > All we need an interface to trigger the "offlining" and the removing in a > single operation - to make sure the memory block cannot get onlined by > user space again before it gets removed. > > To keep things simple, allow to only work on a single memory block.Without a user it is not really clear why do we need this interface. I am also not really sure why do you want/need to control beyond the offlining stage. Care to explain some more? -- Michal Hocko SUSE Labs
Apparently Analagous Threads
- [PATCH RFC v3 6/9] mm: Allow to offline PageOffline() pages with a reference count of 0
- [PATCH RFC v3 6/9] mm: Allow to offline PageOffline() pages with a reference count of 0
- [PATCH RFC v3 6/9] mm: Allow to offline PageOffline() pages with a reference count of 0
- [PATCH RFC v3 6/9] mm: Allow to offline PageOffline() pages with a reference count of 0
- [PATCH RFC v3 6/9] mm: Allow to offline PageOffline() pages with a reference count of 0