Hi, I've found out that NVDIMM size and label size matter for regular (non-NV) DIMM hot plug. If the NVDIMM is not aligned correctly, the guest OS will not accept the hot plugged memory and will complain with messages such as Block size [0x8000000] unaligned hotplug range: start 0x225000000, size 0x10000000 The start address above is also reported within <memory> element of the hot plugged memory in the domain XML: <address type='dimm' slot='1' base='0x225000000'/> Apparently, in order to make memory hot plug working in the guest OS, the inserted memory must be aligned to the platform memory alignment (128 MB on x86_64). I'd like to clarify, how libvirt makes the DIMM address above. How is the NVDIMM memory range determined? According to my experiments, it seems the NVDIMM specified <size> is taken, NVDIMM <label> size is subtracted from it and the resulting value is reduced to the nearest multiple of NVDIMM <alignsize>. Is this observation correct? Is it guaranteed to be stable in future versions? I need to determine the right NVDIMM size to make the subsequent memory modules correctly aligned and then I can't change the NVDIMM size, to not damage data stored in the NVDIMM. Additionally, when adjusting maxMemory due to NVDIMM presence, should I increase it by the specified NVDIMM <size> or a different value? Thank you, Milan
On Tue, Jun 16, 2020 at 12:54:29PM +0200, Milan Zamazal wrote:> Hi, > > I've found out that NVDIMM size and label size matter for regular > (non-NV) DIMM hot plug. If the NVDIMM is not aligned correctly, the > guest OS will not accept the hot plugged memory and will complain with > messages such as > > Block size [0x8000000] unaligned hotplug range: start 0x225000000, size 0x10000000 > > The start address above is also reported within <memory> element of the > hot plugged memory in the domain XML: > > <address type='dimm' slot='1' base='0x225000000'/> > > Apparently, in order to make memory hot plug working in the guest OS, > the inserted memory must be aligned to the platform memory alignment > (128 MB on x86_64). > > I'd like to clarify, how libvirt makes the DIMM address above. How is > the NVDIMM memory range determined? According to my experiments, it > seems the NVDIMM specified <size> is taken, NVDIMM <label> size is > subtracted from it and the resulting value is reduced to the nearest > multiple of NVDIMM <alignsize>. Is this observation correct? Is it > guaranteed to be stable in future versions? I need to determine the > right NVDIMM size to make the subsequent memory modules correctly > aligned and then I can't change the NVDIMM size, to not damage data > stored in the NVDIMM.Libvirt doesn't ever assign a "base" address value itself. We just start QEMU, and then fill in the XML "base" with the value that QEMU has assigned.> Additionally, when adjusting maxMemory due to NVDIMM presence, should I > increase it by the specified NVDIMM <size> or a different value?IIRC, maxMemory has to allow for the sum of the basic RAM amount, plus RAM intended to be used for all possible future (NV)DIMMS that will be hotplugged. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Jun 16, 2020 at 12:54:29 +0200, Milan Zamazal wrote:> Hi, > > I've found out that NVDIMM size and label size matter for regular > (non-NV) DIMM hot plug. If the NVDIMM is not aligned correctly, the > guest OS will not accept the hot plugged memory and will complain with > messages such as > > Block size [0x8000000] unaligned hotplug range: start 0x225000000, size 0x10000000 > > The start address above is also reported within <memory> element of the > hot plugged memory in the domain XML: > > <address type='dimm' slot='1' base='0x225000000'/> > > Apparently, in order to make memory hot plug working in the guest OS, > the inserted memory must be aligned to the platform memory alignment > (128 MB on x86_64). > > I'd like to clarify, how libvirt makes the DIMM address above. How isIf the address isn't provided in the device XML of the attached device, libvirt attaches the device without any address at all and then refreshes the address from qemu in 'qemuDomainUpdateMemoryDeviceInfo'.> the NVDIMM memory range determined? According to my experiments, it > seems the NVDIMM specified <size> is taken, NVDIMM <label> size is > subtracted from it and the resulting value is reduced to the nearest > multiple of NVDIMM <alignsize>. Is this observation correct? Is it > guaranteed to be stable in future versions? I need to determine the > right NVDIMM size to make the subsequent memory modules correctly > aligned and then I can't change the NVDIMM size, to not damage data > stored in the NVDIMM.Unfortunatelly I didn't implement NVDIMM support so I don't know the intricacies. I've cc'd Martin Kletzander who did that part.> > Additionally, when adjusting maxMemory due to NVDIMM presence, should I > increase it by the specified NVDIMM <size> or a different value? > > Thank you, > Milan >
Daniel P. Berrangé <berrange@redhat.com> writes:> On Tue, Jun 16, 2020 at 12:54:29PM +0200, Milan Zamazal wrote: >> Hi, >> > >> I've found out that NVDIMM size and label size matter for regular >> (non-NV) DIMM hot plug. If the NVDIMM is not aligned correctly, the >> guest OS will not accept the hot plugged memory and will complain with >> messages such as >> >> Block size [0x8000000] unaligned hotplug range: start 0x225000000, size 0x10000000 >> >> The start address above is also reported within <memory> element of the >> hot plugged memory in the domain XML: >> >> <address type='dimm' slot='1' base='0x225000000'/> >> >> Apparently, in order to make memory hot plug working in the guest OS, >> the inserted memory must be aligned to the platform memory alignment >> (128 MB on x86_64). >> >> I'd like to clarify, how libvirt makes the DIMM address above. How is >> the NVDIMM memory range determined? According to my experiments, it >> seems the NVDIMM specified <size> is taken, NVDIMM <label> size is >> subtracted from it and the resulting value is reduced to the nearest >> multiple of NVDIMM <alignsize>. Is this observation correct? Is it >> guaranteed to be stable in future versions? I need to determine the >> right NVDIMM size to make the subsequent memory modules correctly >> aligned and then I can't change the NVDIMM size, to not damage data >> stored in the NVDIMM. > > Libvirt doesn't ever assign a "base" address value itself. We just > start QEMU, and then fill in the XML "base" with the value that QEMU > has assigned.I see, then I'll ask about it on the QEMU list.>> Additionally, when adjusting maxMemory due to NVDIMM presence, should I >> increase it by the specified NVDIMM <size> or a different value? > > IIRC, maxMemory has to allow for the sum of the basic RAM amount, plus > RAM intended to be used for all possible future (NV)DIMMS that will be > hotplugged.OK. Thanks, Milan
Peter Krempa <pkrempa@redhat.com> writes:> On Tue, Jun 16, 2020 at 12:54:29 +0200, Milan Zamazal wrote: >> Hi, >> > >> I've found out that NVDIMM size and label size matter for regular >> (non-NV) DIMM hot plug. If the NVDIMM is not aligned correctly, the >> guest OS will not accept the hot plugged memory and will complain with >> messages such as >> >> Block size [0x8000000] unaligned hotplug range: start 0x225000000, size 0x10000000 >> >> The start address above is also reported within <memory> element of the >> hot plugged memory in the domain XML: >> >> <address type='dimm' slot='1' base='0x225000000'/> >> >> Apparently, in order to make memory hot plug working in the guest OS, >> the inserted memory must be aligned to the platform memory alignment >> (128 MB on x86_64). >> >> I'd like to clarify, how libvirt makes the DIMM address above. How is > > If the address isn't provided in the device XML of the attached device, > libvirt attaches the device without any address at all and then > refreshes the address from qemu in 'qemuDomainUpdateMemoryDeviceInfo'.OK, I can look into the QEMU source code, but I'd still like to have some official confirmation, especially regarding possible pitfalls or future changes. We can't risk data loss.>> the NVDIMM memory range determined? According to my experiments, it >> seems the NVDIMM specified <size> is taken, NVDIMM <label> size is >> subtracted from it and the resulting value is reduced to the nearest >> multiple of NVDIMM <alignsize>. Is this observation correct? Is it >> guaranteed to be stable in future versions? I need to determine the >> right NVDIMM size to make the subsequent memory modules correctly >> aligned and then I can't change the NVDIMM size, to not damage data >> stored in the NVDIMM. > > > Unfortunatelly I didn't implement NVDIMM support so I don't know the > intricacies. I've cc'd Martin Kletzander who did that part.Martin, do you know how the QEMU part is supposed to work? I haven't received any response on the QEMU list, do you know who could I ask directly?>> Additionally, when adjusting maxMemory due to NVDIMM presence, should I >> increase it by the specified NVDIMM <size> or a different value? >> >> Thank you, >> Milan >>