Virtualization - Nov 2011 - [ANNOUNCE] Xen port to Cortex-A15 / ARMv7 with virt extensions

Hi all,
a few weeks ago I (and a few others) started hacking on a
proof-of-concept hypervisor port to Cortex-A15 which uses and requires
ARMv7 virtualization extensions. The intention of this work was to find
out how to best support ARM v7+ on Xen. See
http://old-list-archives.xen.org/archives/html/xen-arm/2011-09/msg00013.html
for more details. 

I am pleased to announce that significant progress has been made, and
that we now have a nascent Xen port for Cortex-A15. The port is based on
xen-unstable (HG CS 8d6edc3d26d2) and written from scratch exploiting
the latest virtualization, LPAE, GIC and generic timer support in
hardware.

We started the work less than three months ago, but the port is already
capable of booting a Linux 3.0 based virtual machine (dom0) up to a
shell prompt on an ARM Architecture Envelope Model, configured to emulate
an A15-based Versatile Express. In this context, we wanted to thank ARM
for making the model available to us.
Now we are looking forward to porting the tools and running multiple
guests.

The code requires virtualization, LPAE and GIC support and therefore it
won't be able to run on anything older than a Cortex-A15.
On the other hand, thanks to this, it is very small and easy to read,
write and understand.
The implementation does not distinguish between PV and HVM guests: there
is just one type of guests that would be comparable to Linux PV on HVM
in the Xen X86 world, but with no need for Qemu emulation.
The code only requires minimal changes to the Linux kernel: just enough
to support PV drivers.

Even though we are currently targeting Versatile Express and Cortex-A15
we do intend to support other machines and other ARMv7 with
virtualization extensions CPUs.
We are also looking forward to ARMv8 and 64 bits support.

Given that porting Xen to Cortex-A15 could be done with so little code,
we believe that the best course of action is to merge it into
xen-unstable as quickly as possible. There are still few rough edges to
sort out but we should be able to produce a clean and digestible patch
series for submission to xen-devel within the next couple of months. I
hope to see the first patches going to the list as soon as possible.
We would very welcome any contributions, in the form of testing, code
reviews and, of course, patches! 


A git tree is available here:

git://xenbits.xen.org/people/sstabellini/xen-unstable.git arm

the gitweb url is the following:

http://xenbits.xen.org/gitweb/?p=people/sstabellini/xen-unstable.git/.git;a=shortlog;h=refs/heads/arm

And here is the full diff:

http://xenbits.xen.org/people/sstabellini/diff


We want to point out that this effort is in addition to Samsung's
ongoing efforts to upstream Xen ARM to xen-unstable. Samsung's XenARM
port allows virtualization of Xen on ARM CPUs prior to virtualization
extensions and supports traditional PV guests.

I would like to thank Tim Deegan and Ian Campbell: if you spend some
time reading the history, you'll see that this project wouldn't have
been possible in such a short time without great contributions from
them.

Cheers,
Stefano

On Tuesday 29 November 2011, Stefano Stabellini wrote:
> Hi all,
> a few weeks ago I (and a few others) started hacking on a
> proof-of-concept hypervisor port to Cortex-A15 which uses and requires
> ARMv7 virtualization extensions. The intention of this work was to find
> out how to best support ARM v7+ on Xen. See
>
http://old-list-archives.xen.org/archives/html/xen-arm/2011-09/msg00013.html
> for more details. 
> 
> I am pleased to announce that significant progress has been made, and
> that we now have a nascent Xen port for Cortex-A15. The port is based on
> xen-unstable (HG CS 8d6edc3d26d2) and written from scratch exploiting
> the latest virtualization, LPAE, GIC and generic timer support in
> hardware.
Very nice!

Do you have a pointer to the kernel sources for the Linux guest?
Since Xen and KVM are both in an early working state right now,
it would be very nice if we could agree on the guest model to make
sure that it's always possible to run the same kernel in both
(and potentially other future) hypervisors without modifications.

	Arnd

On Fri, Dec 16, 2011 at 03:43:57PM +0000, David Vrabel
wrote:
> On 30/11/11 14:11, Catalin Marinas wrote:
> > On 30 November 2011 11:39, Stefano Stabellini
> > <stefano.stabellini@eu.citrix.com> wrote:
> >> A git branch is available here (not ready for submission):
> >>
> >> git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git arm
> >>
> >> the branch above is based on git://linux-arm.org/linux-2.6.git
arm-lpae,
> >> even though guests don''t really need lpae support to run
on Xen.
> > 
> > Indeed, you don''t really need LPAE. What you may need though
is
> > generic timers support for A15, it would allow less Hypervisor traps.
> > For up-to-date architecture patches (well, development tree, not
> > guaranteed to be stable), I would recommend this (they get into
> > mainline at some point):
> > 
> >
http://git.kernel.org/?p=linux/kernel/git/cmarinas/linux-arm-arch.git;a=summary
> > 
> > Either use master or just cherry-pick the branches that you are
interested in.
> 
> Which branches are required for the Versatile Express with the
> Cortex-A15? I merged linux-arm-arch/arm-arch/vexpress in
That''s the branch if you only need VE and A15 support
> but I get a
> instruction fault immediately after branching to __mmap_switched.
> 
> Is it not setting up the MMU correctly?
Do you run this on a software model? What config options do you use? You
would need to enable VEXPRESS_EXTENDED_MEMORY_MAP and
ARCH_VEXPRESS_CA15X4.

-- 
Catalin

On 16/12/11 16:54, Catalin Marinas wrote:
> On Fri, Dec 16, 2011 at 03:43:57PM +0000, David Vrabel wrote:
>> On 30/11/11 14:11, Catalin Marinas wrote:
>>> On 30 November 2011 11:39, Stefano Stabellini
>>> <stefano.stabellini@eu.citrix.com> wrote:
>>>> A git branch is available here (not ready for submission):
>>>>
>>>> git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git arm
>>>>
>>>> the branch above is based on git://linux-arm.org/linux-2.6.git
arm-lpae,
>>>> even though guests don''t really need lpae support to
run on Xen.
>>>
>>> Indeed, you don''t really need LPAE. What you may need
though is
>>> generic timers support for A15, it would allow less Hypervisor
traps.
>>> For up-to-date architecture patches (well, development tree, not
>>> guaranteed to be stable), I would recommend this (they get into
>>> mainline at some point):
>>>
>>>
http://git.kernel.org/?p=linux/kernel/git/cmarinas/linux-arm-arch.git;a=summary
>>>
>>> Either use master or just cherry-pick the branches that you are
interested in.
>>
>> Which branches are required for the Versatile Express with the
>> Cortex-A15? I merged linux-arm-arch/arm-arch/vexpress in
> 
> That''s the branch if you only need VE and A15 support
> 
>> but I get a
>> instruction fault immediately after branching to __mmap_switched.
>>
>> Is it not setting up the MMU correctly?
> 
> Do you run this on a software model? What config options do you use? You
> would need to enable VEXPRESS_EXTENDED_MEMORY_MAP and
> ARCH_VEXPRESS_CA15X4.
The envelope model, yes.  Both those options are enabled.  I''ve also
attached the complete config and the model configuration.

I took a closer look at the diffs between what Stefano had in his tree
(which included a bunch of LPAE support which I don''t have enabled) and
the kernel boots the addition of some isb''s when the MMU is switched
on.

These were added by: "ARM: LPAE: add ISBs around MMU enabling code"
(commit 1c553c2 in your tree) which is think is only present in the LPAE
branch.  Is this patch not actually specific to LPAE?  Are there other
similar patches?

diff --git a/arch/arm/kernel/head.S b/arch/arm/kernel/head.S
index 566c54c..b4e84e2 100644
--- a/arch/arm/kernel/head.S
+++ b/arch/arm/kernel/head.S
@@ -400,8 +400,10 @@ ENDPROC(__enable_mmu)
        .align  5
 __turn_mmu_on:
        mov     r0, r0
+        isb
        mcr     p15, 0, r0, c1, c0, 0           @ write control reg
        mrc     p15, 0, r3, c0, c0, 0           @ read id reg
+        isb
        mov     r3, r3
        mov     r3, r13
        mov     pc, r3

David



_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

On Fri, Dec 16, 2011 at 05:38:52PM +0000, David Vrabel
wrote:
> On 16/12/11 16:54, Catalin Marinas wrote:
> > On Fri, Dec 16, 2011 at 03:43:57PM +0000, David Vrabel wrote:
> >> On 30/11/11 14:11, Catalin Marinas wrote:
> >>> On 30 November 2011 11:39, Stefano Stabellini
> >>> <stefano.stabellini@eu.citrix.com> wrote:
> >>>> A git branch is available here (not ready for submission):
> >>>>
> >>>> git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git
arm
> >>>>
> >>>> the branch above is based on
git://linux-arm.org/linux-2.6.git arm-lpae,
> >>>> even though guests don''t really need lpae support
to run on Xen.
> >>>
> >>> Indeed, you don''t really need LPAE. What you may need
though is
> >>> generic timers support for A15, it would allow less Hypervisor
traps.
> >>> For up-to-date architecture patches (well, development tree,
not
> >>> guaranteed to be stable), I would recommend this (they get
into
> >>> mainline at some point):
> >>>
> >>>
http://git.kernel.org/?p=linux/kernel/git/cmarinas/linux-arm-arch.git;a=summary
> >>>
> >>> Either use master or just cherry-pick the branches that you
are interested in.
> >>
> >> Which branches are required for the Versatile Express with the
> >> Cortex-A15? I merged linux-arm-arch/arm-arch/vexpress in
> > 
> > That''s the branch if you only need VE and A15 support
> > 
> >> but I get a
> >> instruction fault immediately after branching to __mmap_switched.
> >>
> >> Is it not setting up the MMU correctly?
> > 
> > Do you run this on a software model? What config options do you use?
You
> > would need to enable VEXPRESS_EXTENDED_MEMORY_MAP and
> > ARCH_VEXPRESS_CA15X4.
> 
> The envelope model, yes.  Both those options are enabled.  I''ve
also
> attached the complete config and the model configuration.
> 
> I took a closer look at the diffs between what Stefano had in his tree
> (which included a bunch of LPAE support which I don''t have
enabled) and
> the kernel boots the addition of some isb''s when the MMU is
switched on.
> 
> These were added by: "ARM: LPAE: add ISBs around MMU enabling
code"
> (commit 1c553c2 in your tree) which is think is only present in the LPAE
> branch.  Is this patch not actually specific to LPAE?  Are there other
> similar patches?
This patch is not specific to LPAE, it''s an architecture requirement
and
it became visible with A15. The complete patch is here:

http://git.kernel.org/?p=linux/kernel/git/cmarinas/linux-arm-arch.git;a=commitdiff;h=1c553c2afdd7a57cf874a38925bc58381b28150b

though I only kept it in my LPAE branch (which is on it''s way to
mainline for 3.3-rc1).

-- 
Catalin