Is this PAE or non-PAE?
Please can you try forceing emulation mode by toggling the "#if 0" in
arch/x86/mm.c ptwr_do_page_fault
The other thing to try is modifying set_pte_pfn_ma to call xen_l1_update
rather than set_pte. You could try set_pte_at too.
This will help narrow down the issue.
Thanks,
Ian
> Keir, Ian,
> With PCI mmconfig option on, and with the PCI express
> enabled BIOS, the dom0 kernel reads the PCI config from
> fix-mapped PCI mmconfig space.
> The PCI mmconfig space is of 256MB size, and it''s access
> is implemented differently on i386 & x86_64. On x86_64 the
> whole 256MB is mapped in the Kernel virtual address space. On
> i386 it will consume too much of the kernels virtual address
> space, hence it is implemented using a single fix-mapped
> page. This page is mapped to the desired physical address for
> every PCI mmconfig access, as seen in the following code from
> mmconfig.c .
>
> static inline void pci_exp_set_dev_base(int bus, int devfn) {
> u32 dev_base = pci_mmcfg_base_addr | (bus << 20) | (devfn
<< 12);
> if (dev_base != mmcfg_last_accessed_device) {
> mmcfg_last_accessed_device = dev_base;
> set_fixmap_nocache(FIX_PCIE_MCFG, dev_base);
> }
> }
>
> static int pci_mmcfg_read(unsigned int seg, unsigned int bus,
> unsigned int devfn, int reg, int len, u32 *value) {
> unsigned long flags;
>
> if (!value || (bus > 255) || (devfn > 255) || (reg > 4095))
> return -EINVAL;
>
> spin_lock_irqsave(&pci_config_lock, flags);
>
> pci_exp_set_dev_base(bus, devfn);
>
> switch (len) {
>
> At the time of boot the PCI mmconfig space is accessed
> thousands times, one after another; that causes fixed map &
> unmap continuously very fast for a long time. Currently the
> fix-mapped virtual address for Shared_info_page for dom0 &
> the PCI mmconfig page are adjacent in the fixed_addresses in
> the fixedmap.h.
>
> #ifdef CONFIG_PCI_MMCONFIG
> FIX_PCIE_MCFG,
> #endif
> FIX_SHARED_INFO,
> FIX_GNTTAB_BEGIN,
>
> I am suspecting that this is causing a race condition
> because of writable page tables. While accessing the PCI
> mmconfig on i386 the dom0 kernel (cpu 0) is continuously
> rewriting into the pte for FIX_PCIE_MCFG at a very fast rate.
> With writable page tables the updates to ptes are deferred.
> In the SMP case other CPUs are getting the interrupts (timer)
> at the same time, interrupts handlers access the shared_info
> page to notify the dom0 of the events such as timer event.
> The problem possibly is that because of the writable page
> tables, the L1 page is getting evicted during the mmconfig
> access, and the shared_page translation needed for event
> notification is also in the same L1 page. All the cpus are
> using the same page tables at this time. While writing the pte, the
> L2 page is getting cut off from the page table. This is
> somehow causing corruption in the dom0 page tables, and we
> see the errors.
> I belive this issue is not on x86_64 because each
> mmconfig access does not map/unmap fixmap, and the racing
> condition accessing the l2 page is not there.
> The current work around working for me is to disable
> PCI_MMCONFIG for
> i386 in the xen0 kernel config. Today or later other people
> will also notice this corruption on SMP boxes with SNMP dom0.
> I can see it once in a while on a 4 way box.
>
> Can we disable PCI_MMCONFIG for i386 in the xen0 config till
> we solve the race condition issue? Attached is the patch for
> the config.
> As I have a workaround and I am seeing issues with VMX
> guests, I am trying to fix those issues now.
>
> Thanks & Regards,
> Nitin
> --------------------------------------------------------------
> ----------
> -----------
> Sr Software Engineer
> Open Source Technology Center, Intel Corp -----Original Message-----
> From: Kamble, Nitin A
> Sent: Tuesday, August 30, 2005 10:06 AM
> To: Keir Fraser
> Cc: xen-devel
> Subject: RE: [Xen-devel] Re: SMP dom0 with 8 cpus of i386
>
> > Default but with smp enabled.
> Same here. I am seeing the issue inconsistently on a 4 way
> box. 8 way system does not have any issue with maxcpus=1.
> with 8 cpus it is consistent. More no of cpus are causing
> some corruption. It is always happening at the time of
> reading/writing the pci mmconfig space.
> I am debugging here.
>
> Thanks & Regards,
> Nitin
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
