Hello, all. I''m writing pvgrub2 ( for those interested http://git.savannah.gnu.org/cgit/grub.git/log/?h=phcoder/newports/xen ) Now I''m able to load pvgrub2 from pvgrub2. Now I''m trying to load NetBSD. Apparently NetBSD assumes that console interface is mapping in start-of-day layout and I paid special care not to map any special pages in start-of-day. Which special pages have to be mapped and where? Accoring to xen.h: * 1. The domain is started within contiguous virtual-memory region. * 2. The contiguous region ends on an aligned 4MB boundary. * 3. This the order of bootstrap elements in the initial virtual region: * a. relocated kernel image * b. initial ram disk [mod_start, mod_len] * c. list of allocated page frames [mfn_list, nr_pages] * (unless relocated due to XEN_ELFNOTE_INIT_P2M) * d. start_info_t structure [register ESI (x86)] * e. bootstrap page tables [pt_base and CR3 (x86)] * f. bootstrap stack [register ESP (x86)] * 4. Bootstrap elements are packed together, but each is 4kB-aligned. * 5. The initial ram disk may be omitted. * 6. The list of page frames forms a contiguous ''pseudo-physical'' memory * layout for the domain. In particular, the bootstrap virtual-memory * region is a 1:1 mapping to the first section of the pseudo-physical map. * 7. All bootstrap elements are mapped read-writable for the guest OS. The * only exception is the bootstrap page table, which is mapped read-only. * 8. There is guaranteed to be at least 512kB padding after the final * bootstrap element. If necessary, the bootstrap virtual region is * extended by an extra 4MB to ensure this. I see no mention of mapped console interface at all. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Wed, Oct 30, 2013 at 01:32:30PM +0100, Vladimir 'φ-coder/phcoder' Serbinenko wrote:> Hello, all. I'm writing pvgrub2 ( for those interested > http://git.savannah.gnu.org/cgit/grub.git/log/?h=phcoder/newports/xen ) > Now I'm able to load pvgrub2 from pvgrub2. Now I'm trying to load > NetBSD. Apparently NetBSD assumes that console interface is mapping in > start-of-day layout and I paid special care not to map any special pages > in start-of-day. Which special pages have to be mapped and where? > Accoring to xen.h: > * 1. The domain is started within contiguous virtual-memory region. > * 2. The contiguous region ends on an aligned 4MB boundary. > * 3. This the order of bootstrap elements in the initial virtual region: > * a. relocated kernel image > * b. initial ram disk [mod_start, mod_len] > * c. list of allocated page frames [mfn_list, nr_pages] > * (unless relocated due to XEN_ELFNOTE_INIT_P2M) > * d. start_info_t structure [register ESI (x86)]That structure has the information: #ifdef XEN_HAVE_PV_GUEST_ENTRY struct start_info { /* THE FOLLOWING ARE FILLED IN BOTH ON INITIAL BOOT AND ON RESUME. */ char magic[32]; /* "xen-<version>-<platform>". */ unsigned long nr_pages; /* Total pages allocated to this domain. */ unsigned long shared_info; /* MACHINE address of shared info struct. */ uint32_t flags; /* SIF_xxx flags. */ xen_pfn_t store_mfn; /* MACHINE page number of shared page. */ uint32_t store_evtchn; /* Event channel for store communication. */ union { struct { xen_pfn_t mfn; /* MACHINE page number of console page. */ <=== uint32_t evtchn; /* Event channel for console page. */ } domU; struct { uint32_t info_off; /* Offset of console_info struct. */ uint32_t info_size; /* Size of console_info struct from start.*/ } dom0; } console; /* THE FOLLOWING ARE ONLY FILLED IN ON INITIAL BOOT (NOT RESUME). */ unsigned long pt_base; /* VIRTUAL address of page directory. */ unsigned long nr_pt_frames; /* Number of bootstrap p.t. frames. */ unsigned long mfn_list; /* VIRTUAL address of page-frame list. */ unsigned long mod_start; /* VIRTUAL address of pre-loaded module */ /* (PFN of pre-loaded module if */ /* SIF_MOD_START_PFN set in flags). */ unsigned long mod_len; /* Size (bytes) of pre-loaded module. */ #define MAX_GUEST_CMDLINE 1024 int8_t cmd_line[MAX_GUEST_CMDLINE]; /* The pfn range here covers both page table and p->m table frames. */ unsigned long first_p2m_pfn;/* 1st pfn forming initial P->M table. */ unsigned long nr_p2m_frames;/* # of pfns forming initial P->M table. */ }; You need to consult console.domU.mfn.> * e. bootstrap page tables [pt_base and CR3 (x86)] > * f. bootstrap stack [register ESP (x86)] > * 4. Bootstrap elements are packed together, but each is 4kB-aligned. > * 5. The initial ram disk may be omitted. > * 6. The list of page frames forms a contiguous 'pseudo-physical' memory > * layout for the domain. In particular, the bootstrap virtual-memory > * region is a 1:1 mapping to the first section of the > pseudo-physical map. > * 7. All bootstrap elements are mapped read-writable for the guest OS. The > * only exception is the bootstrap page table, which is mapped > read-only. > * 8. There is guaranteed to be at least 512kB padding after the final > * bootstrap element. If necessary, the bootstrap virtual region is > * extended by an extra 4MB to ensure this. > > I see no mention of mapped console interface at all. >> _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Vladimir 'φ-coder/phcoder' Serbinenko
2013-Oct-30 13:11 UTC
Re: start of day, special pages
On 30.10.2013 14:03, Konrad Rzeszutek Wilk wrote:> On Wed, Oct 30, 2013 at 01:32:30PM +0100, Vladimir ''φ-coder/phcoder'' Serbinenko wrote: >> Hello, all. I''m writing pvgrub2 ( for those interested >> http://git.savannah.gnu.org/cgit/grub.git/log/?h=phcoder/newports/xen ) >> Now I''m able to load pvgrub2 from pvgrub2. Now I''m trying to load >> NetBSD. Apparently NetBSD assumes that console interface is mapping in >> start-of-day layout and I paid special care not to map any special pages >> in start-of-day. Which special pages have to be mapped and where? >> Accoring to xen.h: >> * 1. The domain is started within contiguous virtual-memory region. >> * 2. The contiguous region ends on an aligned 4MB boundary. >> * 3. This the order of bootstrap elements in the initial virtual region: >> * a. relocated kernel image >> * b. initial ram disk [mod_start, mod_len] >> * c. list of allocated page frames [mfn_list, nr_pages] >> * (unless relocated due to XEN_ELFNOTE_INIT_P2M) >> * d. start_info_t structure [register ESI (x86)] > > That structure has the information: > > > #ifdef XEN_HAVE_PV_GUEST_ENTRY > struct start_info { > /* THE FOLLOWING ARE FILLED IN BOTH ON INITIAL BOOT AND ON RESUME. */ > char magic[32]; /* "xen-<version>-<platform>". */ > unsigned long nr_pages; /* Total pages allocated to this domain. */ > unsigned long shared_info; /* MACHINE address of shared info struct. */ > uint32_t flags; /* SIF_xxx flags. */ > xen_pfn_t store_mfn; /* MACHINE page number of shared page. */ > uint32_t store_evtchn; /* Event channel for store communication. */ > union { > struct { > xen_pfn_t mfn; /* MACHINE page number of console page. */ <===> uint32_t evtchn; /* Event channel for console page. */ > } domU; > struct { > uint32_t info_off; /* Offset of console_info struct. */ > uint32_t info_size; /* Size of console_info struct from start.*/ > } dom0; > } console; > /* THE FOLLOWING ARE ONLY FILLED IN ON INITIAL BOOT (NOT RESUME). */ > unsigned long pt_base; /* VIRTUAL address of page directory. */ > unsigned long nr_pt_frames; /* Number of bootstrap p.t. frames. */ > unsigned long mfn_list; /* VIRTUAL address of page-frame list. */ > unsigned long mod_start; /* VIRTUAL address of pre-loaded module */ > /* (PFN of pre-loaded module if */ > /* SIF_MOD_START_PFN set in flags). */ > unsigned long mod_len; /* Size (bytes) of pre-loaded module. */ > #define MAX_GUEST_CMDLINE 1024 > int8_t cmd_line[MAX_GUEST_CMDLINE]; > /* The pfn range here covers both page table and p->m table frames. */ > unsigned long first_p2m_pfn;/* 1st pfn forming initial P->M table. */ > unsigned long nr_p2m_frames;/* # of pfns forming initial P->M table. */ > }; > > You need to consult console.domU.mfn. >Yes, I understand that but it''s not my question. My question is where I should map it to when handing off control to another kernel.> >> * e. bootstrap page tables [pt_base and CR3 (x86)] >> * f. bootstrap stack [register ESP (x86)] >> * 4. Bootstrap elements are packed together, but each is 4kB-aligned. >> * 5. The initial ram disk may be omitted. >> * 6. The list of page frames forms a contiguous ''pseudo-physical'' memory >> * layout for the domain. In particular, the bootstrap virtual-memory >> * region is a 1:1 mapping to the first section of the >> pseudo-physical map. >> * 7. All bootstrap elements are mapped read-writable for the guest OS. The >> * only exception is the bootstrap page table, which is mapped >> read-only. >> * 8. There is guaranteed to be at least 512kB padding after the final >> * bootstrap element. If necessary, the bootstrap virtual region is >> * extended by an extra 4MB to ensure this. >> >> I see no mention of mapped console interface at all. >> > > > >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Wed, Oct 30, 2013 at 02:11:51PM +0100, Vladimir 'φ-coder/phcoder' Serbinenko wrote:> On 30.10.2013 14:03, Konrad Rzeszutek Wilk wrote: > > On Wed, Oct 30, 2013 at 01:32:30PM +0100, Vladimir 'φ-coder/phcoder' Serbinenko wrote: > >> Hello, all. I'm writing pvgrub2 ( for those interested > >> http://git.savannah.gnu.org/cgit/grub.git/log/?h=phcoder/newports/xen ) > >> Now I'm able to load pvgrub2 from pvgrub2. Now I'm trying to load > >> NetBSD. Apparently NetBSD assumes that console interface is mapping in > >> start-of-day layout and I paid special care not to map any special pages > >> in start-of-day. Which special pages have to be mapped and where? > >> Accoring to xen.h: > >> * 1. The domain is started within contiguous virtual-memory region. > >> * 2. The contiguous region ends on an aligned 4MB boundary. > >> * 3. This the order of bootstrap elements in the initial virtual region: > >> * a. relocated kernel image > >> * b. initial ram disk [mod_start, mod_len] > >> * c. list of allocated page frames [mfn_list, nr_pages] > >> * (unless relocated due to XEN_ELFNOTE_INIT_P2M) > >> * d. start_info_t structure [register ESI (x86)] > > > > That structure has the information: > > > > > > #ifdef XEN_HAVE_PV_GUEST_ENTRY > > struct start_info { > > /* THE FOLLOWING ARE FILLED IN BOTH ON INITIAL BOOT AND ON RESUME. */ > > char magic[32]; /* "xen-<version>-<platform>". */ > > unsigned long nr_pages; /* Total pages allocated to this domain. */ > > unsigned long shared_info; /* MACHINE address of shared info struct. */ > > uint32_t flags; /* SIF_xxx flags. */ > > xen_pfn_t store_mfn; /* MACHINE page number of shared page. */ > > uint32_t store_evtchn; /* Event channel for store communication. */ > > union { > > struct { > > xen_pfn_t mfn; /* MACHINE page number of console page. */ <===> > uint32_t evtchn; /* Event channel for console page. */ > > } domU; > > struct { > > uint32_t info_off; /* Offset of console_info struct. */ > > uint32_t info_size; /* Size of console_info struct from start.*/ > > } dom0; > > } console; > > /* THE FOLLOWING ARE ONLY FILLED IN ON INITIAL BOOT (NOT RESUME). */ > > unsigned long pt_base; /* VIRTUAL address of page directory. */ > > unsigned long nr_pt_frames; /* Number of bootstrap p.t. frames. */ > > unsigned long mfn_list; /* VIRTUAL address of page-frame list. */ > > unsigned long mod_start; /* VIRTUAL address of pre-loaded module */ > > /* (PFN of pre-loaded module if */ > > /* SIF_MOD_START_PFN set in flags). */ > > unsigned long mod_len; /* Size (bytes) of pre-loaded module. */ > > #define MAX_GUEST_CMDLINE 1024 > > int8_t cmd_line[MAX_GUEST_CMDLINE]; > > /* The pfn range here covers both page table and p->m table frames. */ > > unsigned long first_p2m_pfn;/* 1st pfn forming initial P->M table. */ > > unsigned long nr_p2m_frames;/* # of pfns forming initial P->M table. */ > > }; > > > > You need to consult console.domU.mfn. > > > Yes, I understand that but it's not my question. My question is where I > should map it to when handing off control to another kernel.Oh, I think you just leave it alone. The NetBSD kernel should figure it out. Oh, but you are saying: "> >> Now I'm able to load pvgrub2 from pvgrub2. Now I'm trying to load> >> NetBSD. Apparently NetBSD assumes that console interface is mapping in > >> start-of-day layout and I paid special care not to map any special pages > >> in start-of-day. Which special pages have to be mapped and where?" which means that NetBSD expects something else besides this structure? For that we would need to look in the source of NetBSD. See sys/arch/i386/i386/locore.S It eventually ends up calling xen_bootstrap_tables which does this: if ((xpmap_ptom_masked(page - KERNBASE) >> PAGE_SHIFT) == xen_start_info.console.domU.mfn) { xencons_interface = (void *)page; pte[pl1_pi(page)] = xen_start_info.console_mfn; pte[pl1_pi(page)] <<= PAGE_SHIFT; __PRINTK(("xencons_interface " "va %#lx pte %#" PRIxPADDR "\n", xencons_interface, pte[pl1_pi(page)])); } which is to say to make xencons_interface be a pointer to this MFN. Based on that I think it does the mapping as it boots up. The initial printks are done using the hypercall_io. Could it be that the hypercall page is not setup properly? As PVGRUB2 would have to put that in (I think?). Is it doing any of the ELF parsing to figure out where to start the OS and such? Hm, or am I confused? Is the toolstack suppose to parse all of the payloads and let it do the setup?> > > >> * e. bootstrap page tables [pt_base and CR3 (x86)] > >> * f. bootstrap stack [register ESP (x86)] > >> * 4. Bootstrap elements are packed together, but each is 4kB-aligned. > >> * 5. The initial ram disk may be omitted. > >> * 6. The list of page frames forms a contiguous 'pseudo-physical' memory > >> * layout for the domain. In particular, the bootstrap virtual-memory > >> * region is a 1:1 mapping to the first section of the > >> pseudo-physical map. > >> * 7. All bootstrap elements are mapped read-writable for the guest OS. The > >> * only exception is the bootstrap page table, which is mapped > >> read-only. > >> * 8. There is guaranteed to be at least 512kB padding after the final > >> * bootstrap element. If necessary, the bootstrap virtual region is > >> * extended by an extra 4MB to ensure this. > >> > >> I see no mention of mapped console interface at all. > >> > > > > > > > >> _______________________________________________ > >> Xen-devel mailing list > >> Xen-devel@lists.xen.org > >> http://lists.xen.org/xen-devel > > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Vladimir 'φ-coder/phcoder' Serbinenko
2013-Oct-30 20:17 UTC
Re: start of day, special pages
>> Yes, I understand that but it''s not my question. My question is where I >> should map it to when handing off control to another kernel. > > Oh, I think you just leave it alone.But it seems that xen does something different and in my own bootstrap code I have to take special care to move them out of the way and I put them in the end of mfn list.> Oh, but you are saying: "> >> Now I''m able to load pvgrub2 from pvgrub2. Now I''m trying to load >>>> NetBSD. Apparently NetBSD assumes that console interface is mapping in >>>> start-of-day layout and I paid special care not to map any special pages >>>> in start-of-day. Which special pages have to be mapped and where? > " which means that NetBSD expects something else besides > this structure? >Yes, looks like it expects, at least that the console page is in mfn list, no farther than some point, possibly even mapped in default. It''s not even clear to me why you would assume console page to be in mfn list in the first place.> It eventually ends up calling xen_bootstrap_tables which does this: > > if ((xpmap_ptom_masked(page - KERNBASE) >> PAGE_SHIFT) > == xen_start_info.console.domU.mfn) { > xencons_interface = (void *)page; > pte[pl1_pi(page)] = xen_start_info.console_mfn; > pte[pl1_pi(page)] <<= PAGE_SHIFT; > __PRINTK(("xencons_interface " > "va %#lx pte %#" PRIxPADDR "\n", > xencons_interface, pte[pl1_pi(page)])); > } > > which is to say to make xencons_interface be a pointer to this MFN. >Yes and this if is never triggered.> Based on that I think it does the mapping as it boots up. >Possibly but looks like console page is not there in the list where netbsd expects it.> The initial printks are done using the hypercall_io. >They don''t work for me.> Could it be that the hypercall page is not setup properly?I set up the hypercall page. I suppose that it''s correct but I''m not 100% sure.> As PVGRUB2 > Is it doing any of the ELF > parsing to figure out where to start the OS and such? >Yes, it does ELF parsing. And it parses __xen_guest/Note phdr as well. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Wed, Oct 30, 2013 at 02:11:51PM +0100, Vladimir ''??-coder/phcoder'' Serbinenko wrote:> On 30.10.2013 14:03, Konrad Rzeszutek Wilk wrote: > > On Wed, Oct 30, 2013 at 01:32:30PM +0100, Vladimir ''??-coder/phcoder'' Serbinenko wrote: > >> Hello, all. I''m writing pvgrub2 ( for those interested > >> http://git.savannah.gnu.org/cgit/grub.git/log/?h=phcoder/newports/xen ) > >> Now I''m able to load pvgrub2 from pvgrub2. Now I''m trying to load > >> NetBSD. Apparently NetBSD assumes that console interface is mapping in > >> start-of-day layout and I paid special care not to map any special pages > >> in start-of-day. Which special pages have to be mapped and where? > >> Accoring to xen.h: > >> * 1. The domain is started within contiguous virtual-memory region. > >> * 2. The contiguous region ends on an aligned 4MB boundary. > >> * 3. This the order of bootstrap elements in the initial virtual region: > >> * a. relocated kernel image > >> * b. initial ram disk [mod_start, mod_len] > >> * c. list of allocated page frames [mfn_list, nr_pages] > >> * (unless relocated due to XEN_ELFNOTE_INIT_P2M) > >> * d. start_info_t structure [register ESI (x86)] > > > > That structure has the information: > > > > > > #ifdef XEN_HAVE_PV_GUEST_ENTRY > > struct start_info { > > /* THE FOLLOWING ARE FILLED IN BOTH ON INITIAL BOOT AND ON RESUME. */ > > char magic[32]; /* "xen-<version>-<platform>". */ > > unsigned long nr_pages; /* Total pages allocated to this domain. */ > > unsigned long shared_info; /* MACHINE address of shared info struct. */ > > uint32_t flags; /* SIF_xxx flags. */ > > xen_pfn_t store_mfn; /* MACHINE page number of shared page. */ > > uint32_t store_evtchn; /* Event channel for store communication. */ > > union { > > struct { > > xen_pfn_t mfn; /* MACHINE page number of console page. */ <===> > uint32_t evtchn; /* Event channel for console page. */ > > } domU; > > struct { > > uint32_t info_off; /* Offset of console_info struct. */ > > uint32_t info_size; /* Size of console_info struct from start.*/ > > } dom0; > > } console; > > /* THE FOLLOWING ARE ONLY FILLED IN ON INITIAL BOOT (NOT RESUME). */ > > unsigned long pt_base; /* VIRTUAL address of page directory. */ > > unsigned long nr_pt_frames; /* Number of bootstrap p.t. frames. */ > > unsigned long mfn_list; /* VIRTUAL address of page-frame list. */ > > unsigned long mod_start; /* VIRTUAL address of pre-loaded module */ > > /* (PFN of pre-loaded module if */ > > /* SIF_MOD_START_PFN set in flags). */ > > unsigned long mod_len; /* Size (bytes) of pre-loaded module. */ > > #define MAX_GUEST_CMDLINE 1024 > > int8_t cmd_line[MAX_GUEST_CMDLINE]; > > /* The pfn range here covers both page table and p->m table frames. */ > > unsigned long first_p2m_pfn;/* 1st pfn forming initial P->M table. */ > > unsigned long nr_p2m_frames;/* # of pfns forming initial P->M table. */ > > }; > > > > You need to consult console.domU.mfn. > > > Yes, I understand that but it''s not my question. My question is where I > should map it to when handing off control to another kernel.I have not seen any real docs about that but IIRC pvgrub maps console and xenstore just immediately after start info. Please look for: ... /* Move current console, xenstore and boot MFNs to the allocated * place */ do_exchange(dom, dom->console_pfn, start_info.console.domU.mfn); do_exchange(dom, dom->xenstore_pfn, start_info.store_mfn); ... in xen/stubdom/grub/kexec.c file. In general this file shows how to build properly all boot stuff. You could also check my PV kexec implementation: http://lists.xenproject.org/archives/html/xen-devel/2013-07/msg02613.html Especially check kexec-tools-2.0.3_20120522.patch patch. Most interesting stuff for you is in kexec-tools-2.0.3/kexec/arch/i386/kexec-xen-pv.c file. It is xen-pv loader. xen_pv_load() will show you general overview. Do not worry too much about xenstore and console pages order. It is my arbitrary choice but I needed that in my PV kexec implementation. You will find also there some info how to use early console (check kexec-tools-2.0.3/purgatory/arch/i386/console-x86.c file for more details). If you need more info drop me a line. Daniel
Vladimir 'φ-coder/phcoder' Serbinenko
2013-Oct-31 00:39 UTC
Re: start of day, special pages
> I have not seen any real docs about that but IIRC pvgrub maps console > and xenstore just immediately after start info. Please look for: >Thanks, exactly the info I''ve been looking for. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel