Cinco, Dante
2009-Oct-13 22:51 UTC
[Xen-devel] pt_msix_init: Error: Can''t map physical MSI-X table: Invalid argument
I get the above error message when I boot the domU (Linux 2.6.30.1).
I''m using Xen 3.5-unstable (changeset 20303) with pv_ops kernel
2.6.31.1. I use pci-stub to hide the device so that I can assign/pass it through
to domU. The error is coming from ioemu-xxx/hw/pt-msi.c. I think it is
complaining about the table_off that is part of the mmap() call:
dev->msix->phys_iomem_base = mmap(0, total_entries * 16,
PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED,
fd, dev->msix->table_base + table_off);
When I printed table_off, its value is 0x4100 (not page-aligned) which is
consistent with the BAR4 offset of the PCI device (see below). I think mmap()
requires the last argument to be a multiple of the page size and results is a
map error if it is not. Here''s the partial output of "lspci -vv -s
0:07:0.1" from dom0:
07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 05)
Subsystem: Atto Technology Device 003c
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 11
Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1
Enable+
Address: 00000000fee11000 Data: 405e
Capabilities: [70] Express (v2) Endpoint, MSI 01
DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <4us,
L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9
Vector table: BAR=4 offset=00004100
PBA: BAR=4 offset=00004000
I''m able to workaround this problem by modifying pt-msi.c: force offset
to zero in mmap() and on the next line, adjust the pointer by table_off:
table_off_adjust = table_off & 0x0fff;
dev->msix->phys_iomem_base = mmap(0, total_entries * 16 +
table_off_adjust,
PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED,
fd, dev->msix->table_base + table_off -
table_off_adjust);
dev->msix->phys_iomem_base = (void *)((char
*)dev->msix->phys_iomem_base + table_off_adjust);
Ideally, I should only be applying this to my device but since I''m not
passing any other device that use MSI, I don''t see any negative effect.
In addition, my device only uses MSI and not MSI-X.
Obviously, this is a custom hack but I was wondering if there is a cleaner
solution for this. Is a non-page aligned BAR offset unusual? When I boot Linux
2.6.30.1 on bare metal (no Xen), I don''t see this error.
Dante Cinco
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Stefano Stabellini
2009-Oct-14 12:47 UTC
Re: [Xen-devel] pt_msix_init: Error: Can''t map physical MSI-X table: Invalid argument
On Tue, 13 Oct 2009, Cinco, Dante wrote:> I get the above error message when I boot the domU (Linux 2.6.30.1). I''m using Xen 3.5-unstable (changeset 20303) with > pv_ops kernel 2.6.31.1. I use pci-stub to hide the device so that I can assign/pass it through to domU. The error is > coming from ioemu-xxx/hw/pt-msi.c. I think it is complaining about the table_off that is part of the mmap() call: > > dev->msix->phys_iomem_base = mmap(0, total_entries * 16, > PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, > fd, dev->msix->table_base + table_off); > > When I printed table_off, its value is 0x4100 (not page-aligned) which is consistent with the BAR4 offset of the PCI > device (see below). I think mmap() requires the last argument to be a multiple of the page size and results is a map > error if it is not. Here''s the partial output of "lspci -vv -s 0:07:0.1" from dom0: > > 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 05) > Subsystem: Atto Technology Device 003c > Latency: 0, Cache Line Size: 64 bytes > Interrupt: pin A routed to IRQ 11 > Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1 Enable+ > Address: 00000000fee11000 Data: 405e > Capabilities: [70] Express (v2) Endpoint, MSI 01 > DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited > ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- > Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9 > Vector table: BAR=4 offset=00004100 > PBA: BAR=4 offset=00004000 > > I''m able to workaround this problem by modifying pt-msi.c: force offset to zero in mmap() and on the next line, adjust > the pointer by table_off: > > table_off_adjust = table_off & 0x0fff; > dev->msix->phys_iomem_base = mmap(0, total_entries * 16 + table_off_adjust, > PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, > fd, dev->msix->table_base + table_off - table_off_adjust); > dev->msix->phys_iomem_base = (void *)((char *)dev->msix->phys_iomem_base + table_off_adjust); > > Ideally, I should only be applying this to my device but since I''m not passing any other device that use MSI, I don''t see > any negative effect. In addition, my device only uses MSI and not MSI-X. > > Obviously, this is a custom hack but I was wondering if there is a cleaner solution for this. Is a non-page aligned BAR > offset unusual? When I boot Linux 2.6.30.1 on bare metal (no Xen), I don''t see this error.I think this workaround is not so bad, you just need to keep table_off_adjust around because it has to be used in munmap too. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Bruce Edge
2009-Oct-14 13:45 UTC
Re: [Xen-devel] pt_msix_init: Error: Can''t map physical MSI-X table: Invalid argument
On Wed, Oct 14, 2009 at 5:47 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> On Tue, 13 Oct 2009, Cinco, Dante wrote: >> I get the above error message when I boot the domU (Linux 2.6.30.1). I''m using Xen 3.5-unstable (changeset 20303) with >> pv_ops kernel 2.6.31.1. I use pci-stub to hide the device so that I can assign/pass it through to domU. The error is >> coming from ioemu-xxx/hw/pt-msi.c. I think it is complaining about the table_off that is part of the mmap() call: >> >> dev->msix->phys_iomem_base = mmap(0, total_entries * 16, >> PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, >> fd, dev->msix->table_base + table_off); >> >> When I printed table_off, its value is 0x4100 (not page-aligned) which is consistent with the BAR4 offset of the PCI >> device (see below). I think mmap() requires the last argument to be a multiple of the page size and results is a map >> error if it is not. Here''s the partial output of "lspci -vv -s 0:07:0.1" from dom0: >> >> 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 05) >> Subsystem: Atto Technology Device 003c >> Latency: 0, Cache Line Size: 64 bytes >> Interrupt: pin A routed to IRQ 11 >> Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1 Enable+ >> Address: 00000000fee11000 Data: 405e >> Capabilities: [70] Express (v2) Endpoint, MSI 01 >> DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited >> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- >> Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9 >> Vector table: BAR=4 offset=00004100 >> PBA: BAR=4 offset=00004000 >> >> I''m able to workaround this problem by modifying pt-msi.c: force offset to zero in mmap() and on the next line, adjust >> the pointer by table_off: >> >> table_off_adjust = table_off & 0x0fff; >> dev->msix->phys_iomem_base = mmap(0, total_entries * 16 + table_off_adjust, >> PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, >> fd, dev->msix->table_base + table_off - table_off_adjust); >> dev->msix->phys_iomem_base = (void *)((char *)dev->msix->phys_iomem_base + table_off_adjust); >> >> Ideally, I should only be applying this to my device but since I''m not passing any other device that use MSI, I don''t see >> any negative effect. In addition, my device only uses MSI and not MSI-X. >> >> Obviously, this is a custom hack but I was wondering if there is a cleaner solution for this. Is a non-page aligned BAR >> offset unusual? When I boot Linux 2.6.30.1 on bare metal (no Xen), I don''t see this error. > > I think this workaround is not so bad, you just need to keep > table_off_adjust around because it has to be used in munmap too.Where is the respective munmap call ? -Bruce _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2009-Oct-14 13:52 UTC
Re: [Xen-devel] pt_msix_init: Error: Can''t map physical MSI-X table: Invalid argument
On Wed, 14 Oct 2009, Bruce Edge wrote:> On Wed, Oct 14, 2009 at 5:47 AM, Stefano Stabellini > <stefano.stabellini@eu.citrix.com> wrote: > > On Tue, 13 Oct 2009, Cinco, Dante wrote: > >> I get the above error message when I boot the domU (Linux 2.6.30.1). I''m using Xen 3.5-unstable (changeset 20303) with > >> pv_ops kernel 2.6.31.1. I use pci-stub to hide the device so that I can assign/pass it through to domU. The error is > >> coming from ioemu-xxx/hw/pt-msi.c. I think it is complaining about the table_off that is part of the mmap() call: > >> > >> dev->msix->phys_iomem_base = mmap(0, total_entries * 16, > >> PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, > >> fd, dev->msix->table_base + table_off); > >> > >> When I printed table_off, its value is 0x4100 (not page-aligned) which is consistent with the BAR4 offset of the PCI > >> device (see below). I think mmap() requires the last argument to be a multiple of the page size and results is a map > >> error if it is not. Here''s the partial output of "lspci -vv -s 0:07:0.1" from dom0: > >> > >> 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 05) > >> Subsystem: Atto Technology Device 003c > >> Latency: 0, Cache Line Size: 64 bytes > >> Interrupt: pin A routed to IRQ 11 > >> Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1 Enable+ > >> Address: 00000000fee11000 Data: 405e > >> Capabilities: [70] Express (v2) Endpoint, MSI 01 > >> DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited > >> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- > >> Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9 > >> Vector table: BAR=4 offset=00004100 > >> PBA: BAR=4 offset=00004000 > >> > >> I''m able to workaround this problem by modifying pt-msi.c: force offset to zero in mmap() and on the next line, adjust > >> the pointer by table_off: > >> > >> table_off_adjust = table_off & 0x0fff; > >> dev->msix->phys_iomem_base = mmap(0, total_entries * 16 + table_off_adjust, > >> PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, > >> fd, dev->msix->table_base + table_off - table_off_adjust); > >> dev->msix->phys_iomem_base = (void *)((char *)dev->msix->phys_iomem_base + table_off_adjust); > >> > >> Ideally, I should only be applying this to my device but since I''m not passing any other device that use MSI, I don''t see > >> any negative effect. In addition, my device only uses MSI and not MSI-X. > >> > >> Obviously, this is a custom hack but I was wondering if there is a cleaner solution for this. Is a non-page aligned BAR > >> offset unusual? When I boot Linux 2.6.30.1 on bare metal (no Xen), I don''t see this error. > > > > I think this workaround is not so bad, you just need to keep > > table_off_adjust around because it has to be used in munmap too. > > Where is the respective munmap call ? >in the same file: pt-msi.c. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Bruce Edge
2009-Oct-14 22:41 UTC
Re: [Xen-devel] pt_msix_init: Error: Can''t map physical MSI-X table: Invalid argument
On Wed, Oct 14, 2009 at 5:47 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> On Tue, 13 Oct 2009, Cinco, Dante wrote: >> I get the above error message when I boot the domU (Linux 2.6.30.1). I''m using Xen 3.5-unstable (changeset 20303) with >> pv_ops kernel 2.6.31.1. I use pci-stub to hide the device so that I can assign/pass it through to domU. The error is >> coming from ioemu-xxx/hw/pt-msi.c. I think it is complaining about the table_off that is part of the mmap() call: >> >> dev->msix->phys_iomem_base = mmap(0, total_entries * 16, >> PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, >> fd, dev->msix->table_base + table_off); >> >> When I printed table_off, its value is 0x4100 (not page-aligned) which is consistent with the BAR4 offset of the PCI >> device (see below). I think mmap() requires the last argument to be a multiple of the page size and results is a map >> error if it is not. Here''s the partial output of "lspci -vv -s 0:07:0.1" from dom0: >> >> 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 05) >> Subsystem: Atto Technology Device 003c >> Latency: 0, Cache Line Size: 64 bytes >> Interrupt: pin A routed to IRQ 11 >> Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1 Enable+ >> Address: 00000000fee11000 Data: 405e >> Capabilities: [70] Express (v2) Endpoint, MSI 01 >> DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited >> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- >> Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9 >> Vector table: BAR=4 offset=00004100 >> PBA: BAR=4 offset=00004000 >> >> I''m able to workaround this problem by modifying pt-msi.c: force offset to zero in mmap() and on the next line, adjust >> the pointer by table_off: >> >> table_off_adjust = table_off & 0x0fff; >> dev->msix->phys_iomem_base = mmap(0, total_entries * 16 + table_off_adjust, >> PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, >> fd, dev->msix->table_base + table_off - table_off_adjust); >> dev->msix->phys_iomem_base = (void *)((char *)dev->msix->phys_iomem_base + table_off_adjust); >> >> Ideally, I should only be applying this to my device but since I''m not passing any other device that use MSI, I don''t see >> any negative effect. In addition, my device only uses MSI and not MSI-X. >> >> Obviously, this is a custom hack but I was wondering if there is a cleaner solution for this. Is a non-page aligned BAR >> offset unusual? When I boot Linux 2.6.30.1 on bare metal (no Xen), I don''t see this error. > > I think this workaround is not so bad, you just need to keep > table_off_adjust around because it has to be used in munmap too. >Here''s a patch that propagates the offset to the munmap as well. Any chance this could be applied to the hg repository? -Bruce diff -Naur ./tools/ioemu-remote/hw/pt-msi.c.orig ./tools/ioemu-remote/hw/pt-msi.c --- ./tools/ioemu-remote/hw/pt-msi.c.orig 2009-10-13 11:54:11.000000000 -0700 +++ ./tools/ioemu-remote/hw/pt-msi.c 2009-10-14 15:16:31.000000000 -0700 @@ -535,6 +535,10 @@ DPCI_REMOVE_MAPPING); } +/* Calculated offset to page-align MSI-X mmap + * Needed for munmap as well */ +static int table_off_adjust = 0; + int pt_msix_init(struct pt_dev *dev, int pos) { uint8_t id; @@ -542,6 +546,7 @@ int i, total_entries, table_off, bar_index; struct pci_dev *pd = dev->pci_dev; int fd; + int err; id = pci_read_byte(pd, pos + PCI_CAP_LIST_ID); @@ -584,9 +589,14 @@ PT_LOG("Error: Can''t open /dev/mem: %s\n", strerror(errno)); goto error_out; } - dev->msix->phys_iomem_base = mmap(0, total_entries * 16, + PT_LOG("table_off = %llx, total_entries = %d\n",table_off,total_entries); + table_off_adjust = table_off & 0x0fff; + dev->msix->phys_iomem_base = mmap(0, total_entries * 16 + table_off_adjust, PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, - fd, dev->msix->table_base + table_off); + fd, dev->msix->table_base + table_off - table_off_adjust); + dev->msix->phys_iomem_base = (void *)((char *)dev->msix->phys_iomem_base + table_off_adjust); + err = errno; + PT_LOG("errno = %d\n",err); if ( dev->msix->phys_iomem_base == MAP_FAILED ) { PT_LOG("Error: Can''t map physical MSI-X table: %s\n", strerror(errno)); @@ -612,7 +622,7 @@ { PT_LOG("unmapping physical MSI-X table from %lx\n", (unsigned long)dev->msix->phys_iomem_base); - munmap(dev->msix->phys_iomem_base, dev->msix->total_entries * 16); + munmap(dev->msix->phys_iomem_base, dev->msix->total_entries * 16 + table_off_adjust); } free(dev->msix); _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2009-Oct-15 10:08 UTC
Re: [Xen-devel] pt_msix_init: Error: Can''t map physical MSI-X table: Invalid argument
On Wed, 14 Oct 2009, Bruce Edge wrote:> On Wed, Oct 14, 2009 at 5:47 AM, Stefano Stabellini > <stefano.stabellini@eu.citrix.com> wrote: > > On Tue, 13 Oct 2009, Cinco, Dante wrote: > >> I get the above error message when I boot the domU (Linux 2.6.30.1). I''m using Xen 3.5-unstable (changeset 20303) with > >> pv_ops kernel 2.6.31.1. I use pci-stub to hide the device so that I can assign/pass it through to domU. The error is > >> coming from ioemu-xxx/hw/pt-msi.c. I think it is complaining about the table_off that is part of the mmap() call: > >> > >> dev->msix->phys_iomem_base = mmap(0, total_entries * 16, > >> PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, > >> fd, dev->msix->table_base + table_off); > >> > >> When I printed table_off, its value is 0x4100 (not page-aligned) which is consistent with the BAR4 offset of the PCI > >> device (see below). I think mmap() requires the last argument to be a multiple of the page size and results is a map > >> error if it is not. Here''s the partial output of "lspci -vv -s 0:07:0.1" from dom0: > >> > >> 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 05) > >> Subsystem: Atto Technology Device 003c > >> Latency: 0, Cache Line Size: 64 bytes > >> Interrupt: pin A routed to IRQ 11 > >> Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1 Enable+ > >> Address: 00000000fee11000 Data: 405e > >> Capabilities: [70] Express (v2) Endpoint, MSI 01 > >> DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited > >> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- > >> Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9 > >> Vector table: BAR=4 offset=00004100 > >> PBA: BAR=4 offset=00004000 > >> > >> I''m able to workaround this problem by modifying pt-msi.c: force offset to zero in mmap() and on the next line, adjust > >> the pointer by table_off: > >> > >> table_off_adjust = table_off & 0x0fff; > >> dev->msix->phys_iomem_base = mmap(0, total_entries * 16 + table_off_adjust, > >> PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, > >> fd, dev->msix->table_base + table_off - table_off_adjust); > >> dev->msix->phys_iomem_base = (void *)((char *)dev->msix->phys_iomem_base + table_off_adjust); > >> > >> Ideally, I should only be applying this to my device but since I''m not passing any other device that use MSI, I don''t see > >> any negative effect. In addition, my device only uses MSI and not MSI-X. > >> > >> Obviously, this is a custom hack but I was wondering if there is a cleaner solution for this. Is a non-page aligned BAR > >> offset unusual? When I boot Linux 2.6.30.1 on bare metal (no Xen), I don''t see this error. > > > > I think this workaround is not so bad, you just need to keep > > table_off_adjust around because it has to be used in munmap too. > > > > > Here''s a patch that propagates the offset to the munmap as well. Any > chance this could be applied to the hg repository? > > -Bruce > > diff -Naur ./tools/ioemu-remote/hw/pt-msi.c.orig > ./tools/ioemu-remote/hw/pt-msi.c > > --- ./tools/ioemu-remote/hw/pt-msi.c.orig 2009-10-13 > 11:54:11.000000000 -0700 > +++ ./tools/ioemu-remote/hw/pt-msi.c 2009-10-14 15:16:31.000000000 -0700 > @@ -535,6 +535,10 @@ > DPCI_REMOVE_MAPPING); > } > > +/* Calculated offset to page-align MSI-X mmap > + * Needed for munmap as well */ > +static int table_off_adjust = 0; > + > int pt_msix_init(struct pt_dev *dev, int pos) > { > uint8_t id; > @@ -542,6 +546,7 @@ > int i, total_entries, table_off, bar_index; > struct pci_dev *pd = dev->pci_dev; > int fd; > + int err; > > id = pci_read_byte(pd, pos + PCI_CAP_LIST_ID); > > @@ -584,9 +589,14 @@ > PT_LOG("Error: Can''t open /dev/mem: %s\n", strerror(errno)); > goto error_out; > } > - dev->msix->phys_iomem_base = mmap(0, total_entries * 16, > + PT_LOG("table_off = %llx, total_entries = %d\n",table_off,total_entries); > + table_off_adjust = table_off & 0x0fff; > + dev->msix->phys_iomem_base = mmap(0, total_entries * 16 + table_off_adjust, > PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, > - fd, dev->msix->table_base + table_off); > + fd, dev->msix->table_base + table_off - > table_off_adjust); > + dev->msix->phys_iomem_base = (void *)((char > *)dev->msix->phys_iomem_base + table_off_adjust); > + err = errno; > + PT_LOG("errno = %d\n",err); > if ( dev->msix->phys_iomem_base == MAP_FAILED ) > { > PT_LOG("Error: Can''t map physical MSI-X table: %s\n", strerror(errno)); > @@ -612,7 +622,7 @@ > { > PT_LOG("unmapping physical MSI-X table from %lx\n", > (unsigned long)dev->msix->phys_iomem_base); > - munmap(dev->msix->phys_iomem_base, dev->msix->total_entries * 16); > + munmap(dev->msix->phys_iomem_base, dev->msix->total_entries * > 16 + table_off_adjust); > } > > free(dev->msix); >I think this is fine apart from the fact that table_off_adjust should be stored as another field in dev->msix, that is "struct pt_msix_info" in hw/pass-through.h. Any other comments, Keir, Ian? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Bruce Edge
2009-Oct-15 18:13 UTC
Re: [Xen-devel] pt_msix_init: Error: Can''t map physical MSI-X table: Invalid argument
On Thu, Oct 15, 2009 at 3:08 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> On Wed, 14 Oct 2009, Bruce Edge wrote: >> On Wed, Oct 14, 2009 at 5:47 AM, Stefano Stabellini >> <stefano.stabellini@eu.citrix.com> wrote: >> > On Tue, 13 Oct 2009, Cinco, Dante wrote: >> >> I get the above error message when I boot the domU (Linux 2.6.30.1). I''m using Xen 3.5-unstable (changeset 20303) with >> >> pv_ops kernel 2.6.31.1. I use pci-stub to hide the device so that I can assign/pass it through to domU. The error is >> >> coming from ioemu-xxx/hw/pt-msi.c. I think it is complaining about the table_off that is part of the mmap() call: >> >> >> >> dev->msix->phys_iomem_base = mmap(0, total_entries * 16, >> >> PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, >> >> fd, dev->msix->table_base + table_off); >> >> >> >> When I printed table_off, its value is 0x4100 (not page-aligned) which is consistent with the BAR4 offset of the PCI >> >> device (see below). I think mmap() requires the last argument to be a multiple of the page size and results is a map >> >> error if it is not. Here''s the partial output of "lspci -vv -s 0:07:0.1" from dom0: >> >> >> >> 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 05) >> >> Subsystem: Atto Technology Device 003c >> >> Latency: 0, Cache Line Size: 64 bytes >> >> Interrupt: pin A routed to IRQ 11 >> >> Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1 Enable+ >> >> Address: 00000000fee11000 Data: 405e >> >> Capabilities: [70] Express (v2) Endpoint, MSI 01 >> >> DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited >> >> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- >> >> Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9 >> >> Vector table: BAR=4 offset=00004100 >> >> PBA: BAR=4 offset=00004000 >> >> >> >> I''m able to workaround this problem by modifying pt-msi.c: force offset to zero in mmap() and on the next line, adjust >> >> the pointer by table_off: >> >> >> >> table_off_adjust = table_off & 0x0fff; >> >> dev->msix->phys_iomem_base = mmap(0, total_entries * 16 + table_off_adjust, >> >> PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, >> >> fd, dev->msix->table_base + table_off - table_off_adjust); >> >> dev->msix->phys_iomem_base = (void *)((char *)dev->msix->phys_iomem_base + table_off_adjust); >> >> >> >> Ideally, I should only be applying this to my device but since I''m not passing any other device that use MSI, I don''t see >> >> any negative effect. In addition, my device only uses MSI and not MSI-X. >> >> >> >> Obviously, this is a custom hack but I was wondering if there is a cleaner solution for this. Is a non-page aligned BAR >> >> offset unusual? When I boot Linux 2.6.30.1 on bare metal (no Xen), I don''t see this error. >> > >> > I think this workaround is not so bad, you just need to keep >> > table_off_adjust around because it has to be used in munmap too. >> > >> >> >> Here''s a patch that propagates the offset to the munmap as well. Any >> chance this could be applied to the hg repository? >> >> -Bruce >> >> diff -Naur ./tools/ioemu-remote/hw/pt-msi.c.orig >> ./tools/ioemu-remote/hw/pt-msi.c >> >> --- ./tools/ioemu-remote/hw/pt-msi.c.orig 2009-10-13 >> 11:54:11.000000000 -0700 >> +++ ./tools/ioemu-remote/hw/pt-msi.c 2009-10-14 15:16:31.000000000 -0700 >> @@ -535,6 +535,10 @@ >> DPCI_REMOVE_MAPPING); >> } >> >> +/* Calculated offset to page-align MSI-X mmap >> + * Needed for munmap as well */ >> +static int table_off_adjust = 0; >> + >> int pt_msix_init(struct pt_dev *dev, int pos) >> { >> uint8_t id; >> @@ -542,6 +546,7 @@ >> int i, total_entries, table_off, bar_index; >> struct pci_dev *pd = dev->pci_dev; >> int fd; >> + int err; >> >> id = pci_read_byte(pd, pos + PCI_CAP_LIST_ID); >> >> @@ -584,9 +589,14 @@ >> PT_LOG("Error: Can''t open /dev/mem: %s\n", strerror(errno)); >> goto error_out; >> } >> - dev->msix->phys_iomem_base = mmap(0, total_entries * 16, >> + PT_LOG("table_off = %llx, total_entries = %d\n",table_off,total_entries); >> + table_off_adjust = table_off & 0x0fff; >> + dev->msix->phys_iomem_base = mmap(0, total_entries * 16 + table_off_adjust, >> PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, >> - fd, dev->msix->table_base + table_off); >> + fd, dev->msix->table_base + table_off - >> table_off_adjust); >> + dev->msix->phys_iomem_base = (void *)((char >> *)dev->msix->phys_iomem_base + table_off_adjust); >> + err = errno; >> + PT_LOG("errno = %d\n",err); >> if ( dev->msix->phys_iomem_base == MAP_FAILED ) >> { >> PT_LOG("Error: Can''t map physical MSI-X table: %s\n", strerror(errno)); >> @@ -612,7 +622,7 @@ >> { >> PT_LOG("unmapping physical MSI-X table from %lx\n", >> (unsigned long)dev->msix->phys_iomem_base); >> - munmap(dev->msix->phys_iomem_base, dev->msix->total_entries * 16); >> + munmap(dev->msix->phys_iomem_base, dev->msix->total_entries * >> 16 + table_off_adjust); >> } >> >> free(dev->msix); >> > > I think this is fine apart from the fact that table_off_adjust should be > stored as another field in dev->msix, that is "struct pt_msix_info" in > hw/pass-through.h. > > Any other comments, Keir, Ian? >Here it is again with the suggested changes. -Bruce --- ./tools/ioemu-remote/hw/pass-through.h.orig 2009-10-15 10:22:17.000000000 -0700 +++ ./tools/ioemu-remote/hw/pass-through.h 2009-10-15 11:08:58.000000000 -0700 @@ -193,6 +193,7 @@ int mmio_index; void *phys_iomem_base; struct msix_entry_info msix_entry[0]; + uint32_t table_offset_adjust; /* page align mmap */ }; struct pt_pm_info { --- ./tools/ioemu-remote/hw/pt-msi.c.orig 2009-10-13 11:54:11.000000000 -0700 +++ ./tools/ioemu-remote/hw/pt-msi.c 2009-10-15 10:29:50.000000000 -0700 @@ -542,6 +542,7 @@ int i, total_entries, table_off, bar_index; struct pci_dev *pd = dev->pci_dev; int fd; + int err; id = pci_read_byte(pd, pos + PCI_CAP_LIST_ID); @@ -584,9 +585,15 @@ PT_LOG("Error: Can''t open /dev/mem: %s\n", strerror(errno)); goto error_out; } - dev->msix->phys_iomem_base = mmap(0, total_entries * 16, + PT_LOG("table_off = %llx, total_entries = %d\n",table_off,total_entries); + dev->msix->table_offset_adjust = table_off & 0x0fff; + dev->msix->phys_iomem_base = mmap(0, total_entries * 16 + dev->msix->table_offset_adjust, PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED, - fd, dev->msix->table_base + table_off); + fd, dev->msix->table_base + table_off - dev->msix->table_offset_adjust); + dev->msix->phys_iomem_base = (void *)((char *)dev->msix->phys_iomem_base + + dev->msix->table_offset_adjust); + err = errno; + PT_LOG("errno = %d\n",err); if ( dev->msix->phys_iomem_base == MAP_FAILED ) { PT_LOG("Error: Can''t map physical MSI-X table: %s\n", strerror(errno)); @@ -612,7 +619,8 @@ { PT_LOG("unmapping physical MSI-X table from %lx\n", (unsigned long)dev->msix->phys_iomem_base); - munmap(dev->msix->phys_iomem_base, dev->msix->total_entries * 16); + munmap(dev->msix->phys_iomem_base, dev->msix->total_entries * 16 + + dev->msix->table_offset_adjust); } free(dev->msix); _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2009-Oct-16 12:18 UTC
Re: [Xen-devel] pt_msix_init: Error: Can''t map physical MSI-X table: Invalid argument
On Thu, 15 Oct 2009, Bruce Edge wrote:> > Here it is again with the suggested changes. >I think is fine, you may want to resent it with [PATCH] on the subject line and a Signed-off-by line to get Ian''s attention. Thanks, Stefano _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Bruce Edge
2009-Oct-16 15:03 UTC
[Xen-devel] [PATCH] pt_msix_init: Error: Can''t map physical MSI-X table: Invalid argument
Force msi-x init mmap to a page boundary or mmap fails with.
pt_msix_init: Error: Can''t map physical MSI-X table: Invalid
argument
Signed-off-by: Bruce Edge <Bruce.edge@lsi.com>
--- ./tools/ioemu-remote/hw/pass-through.h.orig 2009-10-15
10:22:17.000000000 -0700
+++ ./tools/ioemu-remote/hw/pass-through.h 2009-10-15
11:08:58.000000000 -0700
@@ -193,6 +193,7 @@
int mmio_index;
void *phys_iomem_base;
struct msix_entry_info msix_entry[0];
+ uint32_t table_offset_adjust; /* page align mmap */
};
struct pt_pm_info {
--- ./tools/ioemu-remote/hw/pt-msi.c.orig 2009-10-13
11:54:11.000000000 -0700
+++ ./tools/ioemu-remote/hw/pt-msi.c 2009-10-15 10:29:50.000000000 -0700
@@ -542,6 +542,7 @@
int i, total_entries, table_off, bar_index;
struct pci_dev *pd = dev->pci_dev;
int fd;
+ int err;
id = pci_read_byte(pd, pos + PCI_CAP_LIST_ID);
@@ -584,9 +585,15 @@
PT_LOG("Error: Can''t open /dev/mem: %s\n",
strerror(errno));
goto error_out;
}
- dev->msix->phys_iomem_base = mmap(0, total_entries * 16,
+ PT_LOG("table_off = %llx, total_entries =
%d\n",table_off,total_entries);
+ dev->msix->table_offset_adjust = table_off & 0x0fff;
+ dev->msix->phys_iomem_base = mmap(0, total_entries * 16 +
dev->msix->table_offset_adjust,
PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED,
- fd, dev->msix->table_base + table_off);
+ fd, dev->msix->table_base + table_off -
dev->msix->table_offset_adjust);
+ dev->msix->phys_iomem_base = (void *)((char
*)dev->msix->phys_iomem_base +
+ dev->msix->table_offset_adjust);
+ err = errno;
+ PT_LOG("errno = %d\n",err);
if ( dev->msix->phys_iomem_base == MAP_FAILED )
{
PT_LOG("Error: Can''t map physical MSI-X table: %s\n",
strerror(errno));
@@ -612,7 +619,8 @@
{
PT_LOG("unmapping physical MSI-X table from %lx\n",
(unsigned long)dev->msix->phys_iomem_base);
- munmap(dev->msix->phys_iomem_base, dev->msix->total_entries
* 16);
+ munmap(dev->msix->phys_iomem_base, dev->msix->total_entries
* 16 +
+ dev->msix->table_offset_adjust);
}
free(dev->msix);
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel