Hello Stefano and Ian, I''m trying to run DomU and Dom0 on ARM under Xen and have some problems (may be question of configuration). I''m using: - unstable Xen mercurial repository with your "grant table" patches and few patches from Ian Campbell (xcbuild, xen_remap_domain_mfn_range, XENMAPSPACE_gmfn_foreign, ARM support to xc_dom). - your (Stefano''s) linux kernel git repository git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git with head 3.5-rc7-arm-1. I hope all patches to Linux kernel from Stefano letters are there. - Fast Models with few models created as described in wiki page http://wiki.xen.org/wiki/Xen_ARMv7_with_Virtualization_Extensions/FastModels - device trees dts files (vexpress-v2p-ca15-tc1.dts and vexpress-virt.dts) from Stefano letter on 26 July. v2p-ca15-tc1 is attached to Xen using CONFIG_DTB_FILE and vexpress-virt.dtb is attached to DomU zImage. Well, kernel hangs after message (Calibrating delay loop...) when running on models RTSM_VE/Build_Cortex-A15x4 and RTSM_VE/Build_Cortex-A15x2. I attached logs (Dom0-A15x2 and A15x4). Logs also shows problems with device trees (HBI and arch timer). I can boot Dom0 on Cortex-A15x1 model (log file Dom0-A15x1 with warning/problems about DT and HBI) and when i''m tryng to boot zImage using xcbuild utility then it also hangs with message from Xen "Guest data abort: Translation fault at level 3". Log file is also attached. Could you please take a look and help? May be i miss important config option in Linux kernel or in Xen. Is it okay that vexpress-virt descibes V2P-AEMv7A platform and not V2P-CA15? It looks that vexpress-v2p-ca15-tc1.dts includes vexpress-v2m-rs1-rtsm.dtsi. Could you please also share this file if it has specific options? And what can be reason of errors about HBI/arch_timers when running Xen+Linux kernel+vexpress-v2p-ca15-tc1.dts on Cortex-A15x2 model? I can provide/send other info if you want. Thanks in advance. Best regards, Alexey Klimov. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Mon, 2012-07-30 at 16:19 +0100, Alexey Klimov wrote:> Well, kernel hangs after message (Calibrating delay loop...)That''s the state of my local dev tree too. I''m not fully up to date with all Stefano''s patches so I presume we are both missing something or other. Ian.
On Mon, 30 Jul 2012, Alexey Klimov wrote:> Hello Stefano and Ian, > > I''m trying to run DomU and Dom0 on ARM under Xen and have some > problems (may be question of configuration).It is great to see interest in our project!> I''m using: > - unstable Xen mercurial repository with your "grant table" patches > and few patches from Ian Campbell (xcbuild, > xen_remap_domain_mfn_range, XENMAPSPACE_gmfn_foreign, ARM support to > xc_dom).You also need "libxc/arm: allocate xenstore and console pages". Unfortunately with the 4.2 tree frozen we still have few missing pieces here and there in the Xen hypervisor and tools. I think that Ian intended to setup a Xen tree to be used for development with all the currently unapplied patches that are actually needed on top of xen-unstable. Also the xcbuild patch posted by Ian is quite limited, I am attaching the xcbuild.c that I am currently using for my tests with PV disk and network support.> - your (Stefano''s) linux kernel git repository > git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git with head > 3.5-rc7-arm-1. I hope all patches to Linux kernel from Stefano letters > are there.You might also need: "xen/events: fix unmask_evtchn for PV on HVM guests" this is the last version that I posted: http://marc.info/?l=linux-kernel&m=134263575132006&w=2> - Fast Models with few models created as described in wiki page > http://wiki.xen.org/wiki/Xen_ARMv7_with_Virtualization_Extensions/FastModels > - device trees dts files (vexpress-v2p-ca15-tc1.dts and > vexpress-virt.dts) from Stefano letter on 26 July. v2p-ca15-tc1 is > attached to Xen using CONFIG_DTB_FILE and vexpress-virt.dtb is > attached to DomU zImage.That''s correct.> Well, kernel hangs after message (Calibrating delay loop...) when > running on models RTSM_VE/Build_Cortex-A15x4 and > RTSM_VE/Build_Cortex-A15x2. I attached logs (Dom0-A15x2 and A15x4). > Logs also shows problems with device trees (HBI and arch timer). > > I can boot Dom0 on Cortex-A15x1 model (log file Dom0-A15x1 with > warning/problems about DT and HBI) and when i''m tryng to boot zImage > using xcbuild utility then it also hangs with message from Xen "Guest > data abort: Translation fault at level 3". Log file is also attached. > > Could you please take a look and help?I have been testing on the Cortex-A15x1 model exclusively so far, so I am not surprised if there are any errors on the other models. Also I know that there are still few warnings on boot, but I haven''t got around to fixing them yet.> May be i miss important config option in Linux kernel or in Xen. > > Is it okay that vexpress-virt descibes V2P-AEMv7A platform and not > V2P-CA15?That''s should be OK.> It looks that vexpress-v2p-ca15-tc1.dts includes > vexpress-v2m-rs1-rtsm.dtsi. Could you please also share this file if > it has specific options?I am attaching it. I think you might be missing an important change there.> And what can be reason of errors about > HBI/arch_timers when running Xen+Linux > kernel+vexpress-v2p-ca15-tc1.dts on Cortex-A15x2 model?I am not sure yet, but I''ll take a look. I''ll try to fix them in one of the following version of my series.> I can provide/send other info if you want. Thanks in advance.Let me know if the missing patches and the new vexpress-v2m-rs1-rtsm.dtsi fix the issue! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Tue, 2012-07-31 at 12:59 +0100, Stefano Stabellini wrote:> > And what can be reason of errors about > > HBI/arch_timers when running Xen+Linux > > kernel+vexpress-v2p-ca15-tc1.dts on Cortex-A15x2 model? > > I am not sure yet, but I''ll take a look. I''ll try to fix them in one of > the following version of my series.Do you normally use an SMP model? Mine is currently configured for UP, I''ve been meaning to build a new one. Ian.
Hello Stefano and Ian, 2012/7/31 Stefano Stabellini <stefano.stabellini@eu.citrix.com>:> On Mon, 30 Jul 2012, Alexey Klimov wrote: >> I''m trying to run DomU and Dom0 on ARM under Xen and have some >> problems (may be question of configuration). > > It is great to see interest in our project! > > >> I''m using: >> - unstable Xen mercurial repository with your "grant table" patches >> and few patches from Ian Campbell (xcbuild, >> xen_remap_domain_mfn_range, XENMAPSPACE_gmfn_foreign, ARM support to >> xc_dom). > > You also need "libxc/arm: allocate xenstore and console pages". > > Unfortunately with the 4.2 tree frozen we still have few missing pieces > here and there in the Xen hypervisor and tools. > I think that Ian intended to setup a Xen tree to be used for development > with all the currently unapplied patches that are actually needed on top > of xen-unstable.I found patch and applied.> Also the xcbuild patch posted by Ian is quite limited, I am attaching > the xcbuild.c that I am currently using for my tests with PV disk and > network support.Thank you very much, i renamed Ian early version of xcbuild to xcbuild-old, added your file and fixed Makefile to have two xcbuilds. I found your patch on June 22 and looks like i missed it, my bad. http://lists.xen.org/archives/html/xen-devel/2012-06/msg01338.html>> - your (Stefano''s) linux kernel git repository >> git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git with head >> 3.5-rc7-arm-1. I hope all patches to Linux kernel from Stefano letters >> are there. > > You might also need: > > "xen/events: fix unmask_evtchn for PV on HVM guests" > > this is the last version that I posted: > > http://marc.info/?l=linux-kernel&m=134263575132006&w=2Also applied on top of Stefano''s kernel tree cloned on my machine.>> - Fast Models with few models created as described in wiki page >> http://wiki.xen.org/wiki/Xen_ARMv7_with_Virtualization_Extensions/FastModels >> - device trees dts files (vexpress-v2p-ca15-tc1.dts and >> vexpress-virt.dts) from Stefano letter on 26 July. v2p-ca15-tc1 is >> attached to Xen using CONFIG_DTB_FILE and vexpress-virt.dtb is >> attached to DomU zImage. > > That''s correct. > > >> Well, kernel hangs after message (Calibrating delay loop...) when >> running on models RTSM_VE/Build_Cortex-A15x4 and >> RTSM_VE/Build_Cortex-A15x2. I attached logs (Dom0-A15x2 and A15x4). >> Logs also shows problems with device trees (HBI and arch timer). >> >> I can boot Dom0 on Cortex-A15x1 model (log file Dom0-A15x1 with >> warning/problems about DT and HBI) and when i''m tryng to boot zImage >> using xcbuild utility then it also hangs with message from Xen "Guest >> data abort: Translation fault at level 3". Log file is also attached. >> >> Could you please take a look and help? > > I have been testing on the Cortex-A15x1 model exclusively so far, so I > am not surprised if there are any errors on the other models. > Also I know that there are still few warnings on boot, but I haven''t got > around to fixing them yet. > > >> May be i miss important config option in Linux kernel or in Xen. >> >> Is it okay that vexpress-virt descibes V2P-AEMv7A platform and not >> V2P-CA15? > > That''s should be OK. > > >> It looks that vexpress-v2p-ca15-tc1.dts includes >> vexpress-v2m-rs1-rtsm.dtsi. Could you please also share this file if >> it has specific options? > > I am attaching it. I think you might be missing an important change there.Thanks, i checked it and used to build final dtb-file but it looks no changes were there with file that i used.>> And what can be reason of errors about >> HBI/arch_timers when running Xen+Linux >> kernel+vexpress-v2p-ca15-tc1.dts on Cortex-A15x2 model? > > I am not sure yet, but I''ll take a look. I''ll try to fix them in one of > the following version of my series. > > >> I can provide/send other info if you want. Thanks in advance. > > Let me know if the missing patches and the new > vexpress-v2m-rs1-rtsm.dtsi fix the issue!Thank you very much for help. Well, i still have problems after two additional patches (for Xen and kernel). Logs are attached: add_to_physmap failed with -22 in both xcbuild versions and bad p2m lookups and translation fault at level 2 in Xen.. As i understand from email it''s better to use Cortex-A15x1 model, so i will use this model for tests. And i saw that Ian set up git repository for xen with latest patches for ARM. So i''ll try to use this repository. Best regards, Alexey Klimov. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
2012/8/1 Alexey Klimov <trashsee@gmail.com>:> And i saw that Ian set up git repository for xen with latest patches > for ARM. So i''ll try to use this repository.Hello Stefano and Ian, I used new Ian xen-unstable git repository (git://xenbits.xen.org/people/ianc/xen-unstable.git arm-for-4.) and Stefano linux kernel git repository ( git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git 3.5-rc7-arm-2) with additional patches: - for linux kernel "xen/events: fix unmask_evtchn for PV on HVM guests", - ARM hypercall ABI: 64 bit ready patch series for xen and attached few versions of xcbuild (early version of Ian and latest version). After applying 64-bit ready patches i observed such errors when building xen and tools: 1) for i in public/callback.h public/dom0_ops.h public/elfnote.h public/event_channel.h public/features.h public/grant_table.h public/kexec.h public/mem_event.h public/memory.h public/nmi.h public/physdev.h public/platform.h public/sched.h public/tmem.h public/trace.h public/vcpu.h public/version.h public/xen-compat.h public/xen.h public/xencomm.h public/xenoprof.h public/hvm/e820.h public/hvm/hvm_info_table.h public/hvm/hvm_op.h public/hvm/ioreq.h public/hvm/params.h public/io/blkif.h public/io/console.h public/io/fbif.h public/io/fsif.h public/io/kbdif.h public/io/libxenvchan.h public/io/netif.h public/io/pciif.h public/io/protocols.h public/io/ring.h public/io/tpmif.h public/io/usbif.h public/io/vscsiif.h public/io/xenbus.h public/io/xs_wire.h; do gcc -ansi -include stdint.h -Wall -W -Werror -S -o /dev/null -xc $i || exit 1; echo $i; done >headers.chk.new public/version.h:61:5: error: unknown type name ''xen_ulong_t'' make[3]: *** [headers.chk] Error 1 make[3]: Leaving directory `/src/xen/xen/include'' Fixed by inserting #include "arch-arm.h" in xen/include/public/version.h 2) building ''xc'' extension gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -O1 -fno-omit-frame-pointer -marm -g -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wdeclaration-after-statement -Wno-unused-but-set-variable -D__XEN_TOOLS__ -MMD -MF .build.d -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -fno-optimize-sibling-calls -fPIC -I../../tools/include -I../../tools/libxc -Ixen/lowlevel/xc -I/usr/include/python2.7 -c xen/lowlevel/xc/xc.c -o build/temp.linux-armv7l-2.7/xen/lowlevel/xc/xc.o -fno-strict-aliasing -Werror xen/lowlevel/xc/xc.c: In function ''pyxc_xeninfo'': xen/lowlevel/xc/xc.c:1442:5: error: format ''%lx'' expects argument of type ''long unsigned int'', but argument 4 has type ''xen_ulong_t'' [-Werror=format] xen/lowlevel/xc/xc.c:1442:5: error: format ''%lx'' expects argument of type ''long unsigned int'', but argument 4 has type ''xen_ulong_t'' [-Werror=format] cc1: all warnings being treated as errors Just commented snprintf(str, sizeof(str), "virt_start=0x%lx", p_parms.virt_start); in xc.c Then it compiled and i tried to run DomU. It looks like allocation console_pfn and xenstore_pfn in alloc_magic_pages() in xc_dom_arm.c creates real pain for me. With this allocation/patch xen prints "bad p2m lookup" messages before booting DomU (XEN) bad p2m lookup (XEN) dom1 IPA 0x0000000090000000 (XEN) P2M @ 02ffcac0 mfn:0xffe56 (XEN) 1ST[0x2] = 0x00000000f3f686ff (XEN) 2ND[0x80] = 0x0000000000000000 (XEN) bad p2m lookup (XEN) dom1 IPA 0x0000000090001000 (XEN) P2M @ 02ffcac0 mfn:0xffe56 (XEN) 1ST[0x2] = 0x00000000f3f686ff (XEN) 2ND[0x80] = 0x0000000000000000 (XEN) bad p2m lookup (XEN) dom1 IPA 0x0000000090001000 (XEN) P2M @ 02ffcac0 mfn:0xffe56 (XEN) 1ST[0x2] = 0x00000000f3f686ff (XEN) 2ND[0x80] = 0x0000000000000000 and then everything hangs with translation fault: (XEN) DOM1: Grant tables using version 1 layout. (XEN) DOM1: Grant table initialized (XEN) DOM1: NET: Registered protocol family 16 (XEN) Guest data abort: Translation fault at level 2 (XEN) gva=88808804 (XEN) gpa=0000000090001804 (XEN) size=2 sign=0 write=0 reg=2 (XEN) eat=0 cm=0 s1ptw=0 dfsc=6 (XEN) dom1 IPA 0x0000000090001804 (XEN) P2M @ 02ffcac0 mfn:0xffe56 (XEN) 1ST[0x2] = 0x00000000f3f686ff (XEN) 2ND[0x80] = 0x0000000000000000 Detailed log is attached. Ok, i moved allocation for console and xenstore pages back in arch_setup_meminit() like in http://lists.xen.org/archives/html/xen-devel/2012-06/msg01340.html and then added kernel parameter keep_bootcon in DomU device tree file and everything booted up to "unable to open an initial console" and unable to mount rootfs. I still didn''t learn how to deal with xenstore, hvc0, xvda and how to boot from initramfs on ARM using xcbuild but i''ll try to understand and learn this :) So may be it''s good thing to investigate or take deep look why add_to_physmap failed in xcbuild and why there is bad p2m lookup in xen. Log is attached. Do you have any difference between Dom0 .config and DomU .config? Did you just attach initrd using xc_dom_ramdisk_file() call in xcbuild? Any special configuration of xen console/xen store? Well, i dont mean that i''m doing everything correctly but i tried to run it fixing/commenting as much as i can. Could you please help if you have time? I can test new changes, sent other useful info/logs. -- Best regards, Alexey. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Wed, 8 Aug 2012, Alexey Klimov wrote:> 2012/8/1 Alexey Klimov <trashsee@gmail.com>: > > And i saw that Ian set up git repository for xen with latest patches > > for ARM. So i''ll try to use this repository. > > Hello Stefano and Ian, > > I used new Ian xen-unstable git repository > (git://xenbits.xen.org/people/ianc/xen-unstable.git arm-for-4.) and > Stefano linux kernel git repository ( > git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git > 3.5-rc7-arm-2) with additional patches: > > - for linux kernel "xen/events: fix unmask_evtchn for PV on HVM guests", > - ARM hypercall ABI: 64 bit ready patch series for xen and attached > few versions of xcbuild (early version of Ian and latest version). > After applying 64-bit ready patches i observed such errors when > building xen and tools: > > 1) > for i in public/callback.h public/dom0_ops.h public/elfnote.h > public/event_channel.h public/features.h public/grant_table.h > public/kexec.h public/mem_event.h public/memory.h public/nmi.h > public/physdev.h public/platform.h public/sched.h public/tmem.h > public/trace.h public/vcpu.h public/version.h public/xen-compat.h > public/xen.h public/xencomm.h public/xenoprof.h public/hvm/e820.h > public/hvm/hvm_info_table.h public/hvm/hvm_op.h public/hvm/ioreq.h > public/hvm/params.h public/io/blkif.h public/io/console.h > public/io/fbif.h public/io/fsif.h public/io/kbdif.h > public/io/libxenvchan.h public/io/netif.h public/io/pciif.h > public/io/protocols.h public/io/ring.h public/io/tpmif.h > public/io/usbif.h public/io/vscsiif.h public/io/xenbus.h > public/io/xs_wire.h; do gcc -ansi -include stdint.h -Wall -W -Werror > -S -o /dev/null -xc $i || exit 1; echo $i; done >headers.chk.new > public/version.h:61:5: error: unknown type name ''xen_ulong_t'' > make[3]: *** [headers.chk] Error 1 > make[3]: Leaving directory `/src/xen/xen/include'' > > Fixed by inserting #include "arch-arm.h" in xen/include/public/version.hI think that this is a legitimate error, I wasn''t seeing it because I am cross-compiling.> 2) > building ''xc'' extension > gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall > -Wstrict-prototypes -O1 -fno-omit-frame-pointer -marm -g > -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes > -Wdeclaration-after-statement -Wno-unused-but-set-variable > -D__XEN_TOOLS__ -MMD -MF .build.d -D_LARGEFILE_SOURCE > -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE > -fno-optimize-sibling-calls -fPIC -I../../tools/include > -I../../tools/libxc -Ixen/lowlevel/xc -I/usr/include/python2.7 -c > xen/lowlevel/xc/xc.c -o > build/temp.linux-armv7l-2.7/xen/lowlevel/xc/xc.o -fno-strict-aliasing > -Werror > xen/lowlevel/xc/xc.c: In function ''pyxc_xeninfo'': > xen/lowlevel/xc/xc.c:1442:5: error: format ''%lx'' expects argument of > type ''long unsigned int'', but argument 4 has type ''xen_ulong_t'' > [-Werror=format] > xen/lowlevel/xc/xc.c:1442:5: error: format ''%lx'' expects argument of > type ''long unsigned int'', but argument 4 has type ''xen_ulong_t'' > [-Werror=format] > cc1: all warnings being treated as errors > > Just commented snprintf(str, sizeof(str), "virt_start=0x%lx", > p_parms.virt_start); in xc.cThat is also another legitimate error, I''ll fix it in the next version of the patch series. Thanks for testing!> Then it compiled and i tried to run DomU. It looks like allocation > console_pfn and xenstore_pfn in alloc_magic_pages() in xc_dom_arm.c > creates real pain for me. With this allocation/patch xen prints "bad > p2m lookup" messages before booting DomU > (XEN) bad p2m lookup > (XEN) dom1 IPA 0x0000000090000000 > (XEN) P2M @ 02ffcac0 mfn:0xffe56 > (XEN) 1ST[0x2] = 0x00000000f3f686ff > (XEN) 2ND[0x80] = 0x0000000000000000 > (XEN) bad p2m lookup > (XEN) dom1 IPA 0x0000000090001000 > (XEN) P2M @ 02ffcac0 mfn:0xffe56 > (XEN) 1ST[0x2] = 0x00000000f3f686ff > (XEN) 2ND[0x80] = 0x0000000000000000 > (XEN) bad p2m lookup > (XEN) dom1 IPA 0x0000000090001000 > (XEN) P2M @ 02ffcac0 mfn:0xffe56 > (XEN) 1ST[0x2] = 0x00000000f3f686ff > (XEN) 2ND[0x80] = 0x0000000000000000 > > and then everything hangs with translation fault: > > (XEN) DOM1: Grant tables using version 1 layout. > (XEN) DOM1: Grant table initialized > (XEN) DOM1: NET: Registered protocol family 16 > (XEN) Guest data abort: Translation fault at level 2 > (XEN) gva=88808804 > (XEN) gpa=0000000090001804 > (XEN) size=2 sign=0 write=0 reg=2 > (XEN) eat=0 cm=0 s1ptw=0 dfsc=6 > (XEN) dom1 IPA 0x0000000090001804 > (XEN) P2M @ 02ffcac0 mfn:0xffe56 > (XEN) 1ST[0x2] = 0x00000000f3f686ff > (XEN) 2ND[0x80] = 0x0000000000000000 > > Detailed log is attached. > Ok, i moved allocation for console and xenstore pages back in > arch_setup_meminit() like in > http://lists.xen.org/archives/html/xen-devel/2012-06/msg01340.html and > then added kernel parameter keep_bootcon in DomU device tree file and > everything booted up to "unable to open an initial console" and unable > to mount rootfs.You are probably missing Ian''s fix to alloc_magic_pages: http://marc.info/?l=xen-devel&m=134398933530124> I still didn''t learn how to deal with xenstore, hvc0, > xvda and how to boot from initramfs on ARM using xcbuild but i''ll try > to understand and learn this :) So may be it''s good thing to > investigate or take deep look why add_to_physmap failed in xcbuild and > why there is bad p2m lookup in xen. Log is attached. > > Do you have any difference between Dom0 .config and DomU .config? Did > you just attach initrd using xc_dom_ramdisk_file() call in xcbuild? > Any special configuration of xen console/xen store?I am just using one config, attached.> Well, i dont mean that i''m doing everything correctly but i tried to > run it fixing/commenting as much as i can. Could you please help if > you have time? I can test new changes, sent other useful info/logs.Looking at the guest data abort that you are getting, I think that you didn''t update the dts and dtsi to the latest version. They are attached to the 00/23 email "Introduce Xen support on ARM" for the linux kernel. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel