Hi, I went over ( and listened to ) the presentations on PVH ( Linaro Connect, Xen Summit,... ), read up on the ARM v7a virtualization extensions, Xen,... but I still have a few questions. I was hoping someone could answer some of them. * The ARM PVH mode doesn''t make use of pvops anymore. It does make use of PV interfaces for IO. These interfaces, are they the blkback driver and such? So the kernel doesn''t user hypercalls anymore except for those part, it''s all trap & emulate ( using the syndrome registers ), correct? Does the Xen ARM PVH port still support all the hypercalls ( if they aren''t used )? * Since there is no ARM port of miniOS yet, what would be the best way to experiment with hypercalls, privcmd? Is the privcmd interface the only way to do this from userspace? * Using the vgic to inject from the hypervisor, event channels are still used for ''interrupts'' between domains right? * The port doesn''t support the new System MMU yet I read. How does the DMA work at the moment? Wouldn''t there be a need to translate the device addresses? Thanks Sander
On Mon, 2013-04-15 at 10:32 +0100, Sander Bogaert wrote:> Hi, > > I went over ( and listened to ) the presentations on PVH ( Linaro > Connect, Xen Summit,... ), read up on the ARM v7a virtualization > extensions, Xen,... but I still have a few questions. I was hoping > someone could answer some of them. > > * The ARM PVH mode doesn''t make use of pvops anymore. It does make use > of PV interfaces for IO. These interfaces, are they the blkback driver > and such? So the kernel doesn''t user hypercalls anymore except for > those part, it''s all trap & emulate ( using the syndrome registers ), > correct? Does the Xen ARM PVH port still support all the hypercalls ( > if they aren''t used )?pvops is an x86 Linux term which refers to the infrastructure on that architecture for selecting platform functionality (e.g. hypervisor vs baremetal) at boot time (strictly it''s a loosely related set of infrastructure and interfaces more than a single monolithic thing). PVH is also an x86 specific term (an operating mode which uses PV interfaces + hardware acceleration) although it is roughly analogous to the Xen on ARM way of doing things (which doesn''t have a specific term, its just the only type of guest). With that bit of pedantry out of the way: Xen on ARM does still use hypercalls for some things, e.g. event channels and grant table operations which as you have noted are used to provide PV I/O interfaces (blkif, netif etc). Hypercalls are made using the explicit hvc instruction. The hypervisor only supports the hypercalls which are actually used (e.g. we don''t and will never support the PV MMU related calls). We may add in support for other hypercalls as the need arises. You can see which ones are currently wired up in xen/arch/arm/traps.c:arm_hypercall_table. Since Xen on ARM is a mixture of hardware features and software paravirtualisation we do also trap some other stuff and emulate it, mostly CP register accesses etc.> * Since there is no ARM port of miniOS yet, what would be the best way > to experiment with hypercalls, privcmd? Is the privcmd interface the > only way to do this from userspace?privcmd or patching the kernel is your best bet. Yes privcmd is the only way to do this from userspace. You should probably use the libxenctrl library where possible instead of going at the ioctl directly.> * Using the vgic to inject from the hypervisor, event channels are > still used for ''interrupts'' between domains right?Yes. We use a hybrid model where physical hardware interrupts are injected using the GICs virtualisation extensions and event channels use the existing event channel software interrupt model. The difference with x86 PV event channels is that the initial upcall is done via a virtualised GIC interrupt on ARM instead of a software upcall as on x86.> * The port doesn''t support the new System MMU yet I read. How does the > DMA work at the moment? Wouldn''t there be a need to translate the > device addresses?Yes, there is/will be. Platforms like the vexpress and the models don''t have devices which do DMA (or at least not ones anyone has tried). On platforms such as the Arndale which do I believe people are working around the current lack of MMU support by mapping dom0 1-1. Obviously we need some sort of proper IOMMU support ASAP, it''s on the TODO[1] FWIW... [1] http://wiki.xen.org/wiki/Xen_ARM_TODO#IOMMU_support_in_Xen Ian.
On 15 April 2013 12:58, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Mon, 2013-04-15 at 10:32 +0100, Sander Bogaert wrote: >> Hi, >> >> I went over ( and listened to ) the presentations on PVH ( Linaro >> Connect, Xen Summit,... ), read up on the ARM v7a virtualization >> extensions, Xen,... but I still have a few questions. I was hoping >> someone could answer some of them. >> >> * The ARM PVH mode doesn''t make use of pvops anymore. It does make use >> of PV interfaces for IO. These interfaces, are they the blkback driver >> and such? So the kernel doesn''t user hypercalls anymore except for >> those part, it''s all trap & emulate ( using the syndrome registers ), >> correct? Does the Xen ARM PVH port still support all the hypercalls ( >> if they aren''t used )? > > pvops is an x86 Linux term which refers to the infrastructure on that > architecture for selecting platform functionality (e.g. hypervisor vs > baremetal) at boot time (strictly it''s a loosely related set of > infrastructure and interfaces more than a single monolithic thing). > > PVH is also an x86 specific term (an operating mode which uses PV > interfaces + hardware acceleration) although it is roughly analogous to > the Xen on ARM way of doing things (which doesn''t have a specific term, > its just the only type of guest). >I did not realize this before. I thought PVH was a general term. Thanks> With that bit of pedantry out of the way: > > Xen on ARM does still use hypercalls for some things, e.g. event > channels and grant table operations which as you have noted are used to > provide PV I/O interfaces (blkif, netif etc). Hypercalls are made using > the explicit hvc instruction. The hypervisor only supports the > hypercalls which are actually used (e.g. we don''t and will never support > the PV MMU related calls). We may add in support for other hypercalls as > the need arises. You can see which ones are currently wired up in > xen/arch/arm/traps.c:arm_hypercall_table. > > Since Xen on ARM is a mixture of hardware features and software > paravirtualisation we do also trap some other stuff and emulate it, > mostly CP register accesses etc. >Are those operations actively marked as "to-trap" in the hypervisor, is the CPU ''configured'' somewhere to do this? If so where in the Xen codebase can I find this configuration? I assumed with the v7a virtualization extension everything just trapped if it wasn''t done at the correct privilege level/mode.>> * Since there is no ARM port of miniOS yet, what would be the best way >> to experiment with hypercalls, privcmd? Is the privcmd interface the >> only way to do this from userspace? > > privcmd or patching the kernel is your best bet. Yes privcmd is the only > way to do this from userspace. You should probably use the libxenctrl > library where possible instead of going at the ioctl directly. >Thanks for the libxenctrl pointer. I think privcmd would suite me better because ( from what I have seen in the code ) it really just passes the hypercalls without additional checks. I want to write a fuzzer and some other stuff with the least amount of ''sanity checks'' preventing bad stuff from happening. I''m assuming privcmd is the way to go then ( a miniOS would be better ).>> * Using the vgic to inject from the hypervisor, event channels are >> still used for ''interrupts'' between domains right? > > Yes. We use a hybrid model where physical hardware interrupts are > injected using the GICs virtualisation extensions and event channels use > the existing event channel software interrupt model. > > The difference with x86 PV event channels is that the initial upcall is > done via a virtualised GIC interrupt on ARM instead of a software upcall > as on x86. >I thought event channels were really a kind of software interrupt mechanism; the guest kernel would set a callback and if xen schedules the kernel it checks if there are pending events. After the ''interrupt/event'' the connected end would then for example grab a grant reference from the xenstore and use the shared memory for further communication. From your answer this appears to be wrong, what part remains the same in this? Or does ''event channel'' refer to the whole path, including the handler and such ( which would be the part remaining the same ).>> * The port doesn''t support the new System MMU yet I read. How does the >> DMA work at the moment? Wouldn''t there be a need to translate the >> device addresses? > > Yes, there is/will be. Platforms like the vexpress and the models don''t > have devices which do DMA (or at least not ones anyone has tried). > > On platforms such as the Arndale which do I believe people are working > around the current lack of MMU support by mapping dom0 1-1. Obviously we > need some sort of proper IOMMU support ASAP, it''s on the TODO[1] FWIW... > > [1] http://wiki.xen.org/wiki/Xen_ARM_TODO#IOMMU_support_in_Xen > > Ian. >Thanks a lot for these answers! They are really very helpful. Sander
On Mon, 2013-04-15 at 15:24 +0100, Sander Bogaert wrote:> On 15 April 2013 12:58, Ian Campbell <Ian.Campbell@citrix.com> wrote: > > On Mon, 2013-04-15 at 10:32 +0100, Sander Bogaert wrote: > > > > Xen on ARM does still use hypercalls for some things, e.g. event > > channels and grant table operations which as you have noted are used to > > provide PV I/O interfaces (blkif, netif etc). Hypercalls are made using > > the explicit hvc instruction. The hypervisor only supports the > > hypercalls which are actually used (e.g. we don''t and will never support > > the PV MMU related calls). We may add in support for other hypercalls as > > the need arises. You can see which ones are currently wired up in > > xen/arch/arm/traps.c:arm_hypercall_table. > > > > Since Xen on ARM is a mixture of hardware features and software > > paravirtualisation we do also trap some other stuff and emulate it, > > mostly CP register accesses etc. > > > > Are those operations actively marked as "to-trap" in the hypervisor, > is the CPU ''configured'' somewhere to do this? If so where in the Xen > codebase can I find this configuration? I assumed with the v7a > virtualization extension everything just trapped if it wasn''t done at > the correct privilege level/mode.There are bits in the HCR register which choose whether various things trap or not, including accesses to various subsets of CP registers and the like. Other stuff is trapped by deliberately not populating the second stage table and taking a data abort trap. On the Xen side in either case the trap handler is do_trap_hypervisor() in xen/arch/arm/traps.c.> > >> * Since there is no ARM port of miniOS yet, what would be the best way > >> to experiment with hypercalls, privcmd? Is the privcmd interface the > >> only way to do this from userspace? > > > > privcmd or patching the kernel is your best bet. Yes privcmd is the only > > way to do this from userspace. You should probably use the libxenctrl > > library where possible instead of going at the ioctl directly. > > > > Thanks for the libxenctrl pointer. I think privcmd would suite me > better because ( from what I have seen in the code ) it really just > passes the hypercalls without additional checks. I want to write a > fuzzer and some other stuff with the least amount of ''sanity checks'' > preventing bad stuff from happening. I''m assuming privcmd is the way > to go then ( a miniOS would be better ).Yes, privcmd is a good option but minios will eventually be better...> >> * Using the vgic to inject from the hypervisor, event channels are > >> still used for ''interrupts'' between domains right? > > > > Yes. We use a hybrid model where physical hardware interrupts are > > injected using the GICs virtualisation extensions and event channels use > > the existing event channel software interrupt model. > > > > The difference with x86 PV event channels is that the initial upcall is > > done via a virtualised GIC interrupt on ARM instead of a software upcall > > as on x86. > > > > I thought event channels were really a kind of software interrupt > mechanism; the guest kernel would set a callback and if xen schedules > the kernel it checks if there are pending events. After the > ''interrupt/event'' the connected end would then for example grab a > grant reference from the xenstore and use the shared memory for > further communication. From your answer this appears to be wrong, what > part remains the same in this? Or does ''event channel'' refer to the > whole path, including the handler and such ( which would be the part > remaining the same ).Event channels are indeed jut a kind of software interrupt mechanism. The "upcall" I mentioned is the process by which the guest kernel is interrupted and directed into the event channel handler when an event channel is pending. On X86 PV this is done by registering the handler address with the hypervisor. When there is an event channel to be injected Xen will manually frob the guests stack + state to jump to that address. This looks a lot, but not exactly, like the way the processor would take an interrupt or exception on native. On ARM we designate a specific interrupt (a PPI) to be the upcall interrupt and so injection of the event channel happens by using the gic to inject that PPI. Once the event channel is injected the event channel handling code in the kernel makes it look like a regular IRQ to the rest of the kernel and sends it on its way, which eventually calls the relevant driver''s interrupt handler in the usual way. Ian.