Dear all,
I have a PCIe device driver that I have been using on various Linux
distributions and Kernel versions (2.6.x - 3.x.y) successfully all along.
I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1.
When I boot to Linux Mint, my driver still load (via insmod manually)
successfully at Dom0 without any issue. I can do reads and write to the
hardware device. But once booted to Xen, the driver failed to complete the
driver load (via insmod manually) at Dom 0 and the console just hangs.
From my debug messages, it appears it hangs because the driver doesn''t
receive any interrupt after a command is sent to the hardware device by writing
a parameter to the mapped register. Once that register is written, the device
is expected to DMA the command from the buffer allocated by the driver.
The things that I can only think of that might have caused the problem are 1)
IRQ mapping issue, or 2) DMA mapping issue, which I am not sure.
What the driver does:
Set up a command buffer:
Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL);
unsigned long buf_addr = __pa(buf);
unsigned int buf_addr_low = (unsigned int)buf_addr;
Tell device about the buffer:
iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW);
Set up IRQ:
if (pci_find_capability(dev, PCI_CAP_ID_MSI) &&
(!pci_enable_msi(dev)))
{
if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED,
DRIVER_NAME, my_dev))
{
return -ENODEV;
}
my_dev->intr_mode = INTERRUPT_MSI;
}
Ask device to fetch command from buffer (Expect interrupt after this after
device fetched the command from buf. But interrupt did not happen.):
iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG);
From dmesg, it looks like IRQ initialization is complete.
[ 241.743769] My_driver initialization
[ 241.743787] xen: registering gsi 16 triggering 0 polarity 1
[ 241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16
[ 241.743795] xen: --> pirq=16 -> irq=16 (gsi=16)
[ 241.743801] Already setup the GSI :16
[ 241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) ->
IRQ 16
[ 241.743815] my-driver 0000:02:00.0: setting latency timer to 64
/proc/interrupts:
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
CPU6 CPU7
......
......
339: 0 0 0 0 0 0
0 0 xen-pirq-msi my-driver
......
......
Any idea what might cause the problem?
Is there anything we have to be enable/disable, use different functions, or do
differently in drivers written for Xen Dom0 environment regarding the following?
1) Allocating a DMA buffer in driver to allow the device to DMA stuffs.
2) Requesting MSI irq.
Please advise!
Thanks a lot in advance!!
Kenneth
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote:> Dear all, > > I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along. > > I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1. When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue. I can do reads and write to the hardware device. But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs. > > >From my debug messages, it appears it hangs because the driver doesn''t receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register. Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver. > > The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure.The 2).> > > What the driver does:You do need to use the PCI API (or the DMA API).> > Set up a command buffer: > Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL); > unsigned long buf_addr = __pa(buf); > unsigned int buf_addr_low = (unsigned int)buf_addr; > > Tell device about the buffer: > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW);> > Set up IRQ: > if (pci_find_capability(dev, PCI_CAP_ID_MSI) && > (!pci_enable_msi(dev))) > { > if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev)) > { > return -ENODEV; > } > my_dev->intr_mode = INTERRUPT_MSI; > } > > Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf. But interrupt did not happen.): > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG); > > > >From dmesg, it looks like IRQ initialization is complete. > [ 241.743769] My_driver initialization > [ 241.743787] xen: registering gsi 16 triggering 0 polarity 1 > [ 241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16 > [ 241.743795] xen: --> pirq=16 -> irq=16 (gsi=16) > [ 241.743801] Already setup the GSI :16 > [ 241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 > [ 241.743815] my-driver 0000:02:00.0: setting latency timer to 64 > > /proc/interrupts: > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 > ...... > ...... > 339: 0 0 0 0 0 0 0 0 xen-pirq-msi my-driver > ...... > ...... > > Any idea what might cause the problem? > > Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following? > 1) Allocating a DMA buffer in driver to allow the device to DMA stuffs. > 2) Requesting MSI irq. > > Please advise! > > Thanks a lot in advance!! > > Kenneth> _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Hi Konrad and others, Oh, there are Xen/dom0 specific APIs for PCI and DMA? May I ask the names and where can I can more info on the APIs? Many thanks! Kenneth ________________________________________ From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com] Sent: Wednesday, May 23, 2012 5:17 PM To: Kenneth Wong Cc: xen-devel@lists.xen.org Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote:> Dear all, > > I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along. > > I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1. When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue. I can do reads and write to the hardware device. But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs. > > >From my debug messages, it appears it hangs because the driver doesn''t receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register. Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver. > > The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure.The 2).> > > What the driver does:You do need to use the PCI API (or the DMA API).> > Set up a command buffer: > Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL); > unsigned long buf_addr = __pa(buf); > unsigned int buf_addr_low = (unsigned int)buf_addr; > > Tell device about the buffer: > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW);> > Set up IRQ: > if (pci_find_capability(dev, PCI_CAP_ID_MSI) && > (!pci_enable_msi(dev))) > { > if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev)) > { > return -ENODEV; > } > my_dev->intr_mode = INTERRUPT_MSI; > } > > Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf. But interrupt did not happen.): > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG); > > > >From dmesg, it looks like IRQ initialization is complete. > [ 241.743769] My_driver initialization > [ 241.743787] xen: registering gsi 16 triggering 0 polarity 1 > [ 241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16 > [ 241.743795] xen: --> pirq=16 -> irq=16 (gsi=16) > [ 241.743801] Already setup the GSI :16 > [ 241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 > [ 241.743815] my-driver 0000:02:00.0: setting latency timer to 64 > > /proc/interrupts: > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 > ...... > ...... > 339: 0 0 0 0 0 0 0 0 xen-pirq-msi my-driver > ...... > ...... > > Any idea what might cause the problem? > > Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following? > 1) Allocating a DMA buffer in driver to allow the device to DMA stuffs. > 2) Requesting MSI irq. > > Please advise! > > Thanks a lot in advance!! > > Kenneth> _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
On Wed, May 23, 2012 at 07:41:34PM -0700, Kenneth Wong wrote:> Hi Konrad and others, > > Oh, there are Xen/dom0 specific APIs for PCI and DMA? >PCI/DMA APIs are the Linux kernel APIs, not Xen specific.> May I ask the names and where can I can more info on the APIs? >Quick googling reveals: http://www.mjmwired.net/kernel/Documentation/DMA-API.txt http://www.mjmwired.net/kernel/Documentation/DMA-API-HOWTO.txt -- Pasi> Many thanks! > > Kenneth > > ________________________________________ > From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com] > Sent: Wednesday, May 23, 2012 5:17 PM > To: Kenneth Wong > Cc: xen-devel@lists.xen.org > Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 > > On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote: > > Dear all, > > > > I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along. > > > > I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1. When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue. I can do reads and write to the hardware device. But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs. > > > > >From my debug messages, it appears it hangs because the driver doesn''t receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register. Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver. > > > > The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure. > > The 2). > > > > > > What the driver does: > > You do need to use the PCI API (or the DMA API). > > > > > Set up a command buffer: > > Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL); > > unsigned long buf_addr = __pa(buf); > > unsigned int buf_addr_low = (unsigned int)buf_addr; > > > > Tell device about the buffer: > > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW); > > > > > Set up IRQ: > > if (pci_find_capability(dev, PCI_CAP_ID_MSI) && > > (!pci_enable_msi(dev))) > > { > > if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev)) > > { > > return -ENODEV; > > } > > my_dev->intr_mode = INTERRUPT_MSI; > > } > > > > Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf. But interrupt did not happen.): > > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG); > > > > > > >From dmesg, it looks like IRQ initialization is complete. > > [ 241.743769] My_driver initialization > > [ 241.743787] xen: registering gsi 16 triggering 0 polarity 1 > > [ 241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16 > > [ 241.743795] xen: --> pirq=16 -> irq=16 (gsi=16) > > [ 241.743801] Already setup the GSI :16 > > [ 241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 > > [ 241.743815] my-driver 0000:02:00.0: setting latency timer to 64 > > > > /proc/interrupts: > > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 > > ...... > > ...... > > 339: 0 0 0 0 0 0 0 0 xen-pirq-msi my-driver > > ...... > > ...... > > > > Any idea what might cause the problem? > > > > Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following? > > 1) Allocating a DMA buffer in driver to allow the device to DMA stuffs. > > 2) Requesting MSI irq. > > > > Please advise! > > > > Thanks a lot in advance!! > > > > Kenneth > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Hi Pasi, Thanks a lot for the info! I will try it. Kenneth ________________________________________ From: Pasi Kärkkäinen [pasik@iki.fi] Sent: Wednesday, May 23, 2012 11:18 PM To: Kenneth Wong Cc: xen-devel@lists.xen.org Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 On Wed, May 23, 2012 at 07:41:34PM -0700, Kenneth Wong wrote:> Hi Konrad and others, > > Oh, there are Xen/dom0 specific APIs for PCI and DMA? >PCI/DMA APIs are the Linux kernel APIs, not Xen specific.> May I ask the names and where can I can more info on the APIs? >Quick googling reveals: http://www.mjmwired.net/kernel/Documentation/DMA-API.txt http://www.mjmwired.net/kernel/Documentation/DMA-API-HOWTO.txt -- Pasi> Many thanks! > > Kenneth > > ________________________________________ > From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com] > Sent: Wednesday, May 23, 2012 5:17 PM > To: Kenneth Wong > Cc: xen-devel@lists.xen.org > Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 > > On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote: > > Dear all, > > > > I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along. > > > > I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1. When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue. I can do reads and write to the hardware device. But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs. > > > > >From my debug messages, it appears it hangs because the driver doesn''t receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register. Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver. > > > > The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure. > > The 2). > > > > > > What the driver does: > > You do need to use the PCI API (or the DMA API). > > > > > Set up a command buffer: > > Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL); > > unsigned long buf_addr = __pa(buf); > > unsigned int buf_addr_low = (unsigned int)buf_addr; > > > > Tell device about the buffer: > > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW); > > > > > Set up IRQ: > > if (pci_find_capability(dev, PCI_CAP_ID_MSI) && > > (!pci_enable_msi(dev))) > > { > > if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev)) > > { > > return -ENODEV; > > } > > my_dev->intr_mode = INTERRUPT_MSI; > > } > > > > Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf. But interrupt did not happen.): > > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG); > > > > > > >From dmesg, it looks like IRQ initialization is complete. > > [ 241.743769] My_driver initialization > > [ 241.743787] xen: registering gsi 16 triggering 0 polarity 1 > > [ 241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16 > > [ 241.743795] xen: --> pirq=16 -> irq=16 (gsi=16) > > [ 241.743801] Already setup the GSI :16 > > [ 241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 > > [ 241.743815] my-driver 0000:02:00.0: setting latency timer to 64 > > > > /proc/interrupts: > > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 > > ...... > > ...... > > 339: 0 0 0 0 0 0 0 0 xen-pirq-msi my-driver > > ...... > > ...... > > > > Any idea what might cause the problem? > > > > Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following? > > 1) Allocating a DMA buffer in driver to allow the device to DMA stuffs. > > 2) Requesting MSI irq. > > > > Please advise! > > > > Thanks a lot in advance!! > > > > Kenneth > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Hi all, I am now using dma_alloc_coherent(). However, the first parameter, pdev->dev required, which is "struct device", does not seem to have initialized. When and who is suppose to initialize it? In Linux, I can pass "NULL" and it just works. In Xen, it crashes. Sometimes insmod pass, sometimes hangs the system. Please advise! Kenneth ________________________________________ From: Pasi Kärkkäinen [pasik@iki.fi] Sent: Wednesday, May 23, 2012 11:18 PM To: Kenneth Wong Cc: xen-devel@lists.xen.org Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 On Wed, May 23, 2012 at 07:41:34PM -0700, Kenneth Wong wrote:> Hi Konrad and others, > > Oh, there are Xen/dom0 specific APIs for PCI and DMA? >PCI/DMA APIs are the Linux kernel APIs, not Xen specific.> May I ask the names and where can I can more info on the APIs? >Quick googling reveals: http://www.mjmwired.net/kernel/Documentation/DMA-API.txt http://www.mjmwired.net/kernel/Documentation/DMA-API-HOWTO.txt -- Pasi> Many thanks! > > Kenneth > > ________________________________________ > From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com] > Sent: Wednesday, May 23, 2012 5:17 PM > To: Kenneth Wong > Cc: xen-devel@lists.xen.org > Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 > > On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote: > > Dear all, > > > > I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along. > > > > I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1. When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue. I can do reads and write to the hardware device. But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs. > > > > >From my debug messages, it appears it hangs because the driver doesn''t receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register. Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver. > > > > The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure. > > The 2). > > > > > > What the driver does: > > You do need to use the PCI API (or the DMA API). > > > > > Set up a command buffer: > > Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL); > > unsigned long buf_addr = __pa(buf); > > unsigned int buf_addr_low = (unsigned int)buf_addr; > > > > Tell device about the buffer: > > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW); > > > > > Set up IRQ: > > if (pci_find_capability(dev, PCI_CAP_ID_MSI) && > > (!pci_enable_msi(dev))) > > { > > if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev)) > > { > > return -ENODEV; > > } > > my_dev->intr_mode = INTERRUPT_MSI; > > } > > > > Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf. But interrupt did not happen.): > > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG); > > > > > > >From dmesg, it looks like IRQ initialization is complete. > > [ 241.743769] My_driver initialization > > [ 241.743787] xen: registering gsi 16 triggering 0 polarity 1 > > [ 241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16 > > [ 241.743795] xen: --> pirq=16 -> irq=16 (gsi=16) > > [ 241.743801] Already setup the GSI :16 > > [ 241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 > > [ 241.743815] my-driver 0000:02:00.0: setting latency timer to 64 > > > > /proc/interrupts: > > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 > > ...... > > ...... > > 339: 0 0 0 0 0 0 0 0 xen-pirq-msi my-driver > > ...... > > ...... > > > > Any idea what might cause the problem? > > > > Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following? > > 1) Allocating a DMA buffer in driver to allow the device to DMA stuffs. > > 2) Requesting MSI irq. > > > > Please advise! > > > > Thanks a lot in advance!! > > > > Kenneth > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Dear all, The driver now loaded. But got the following exception after a few commands: May 24 17:05:27 marvell-desktop kernel: [ 84.804412] Xorg[1234]: segfault at 34 ip 00000000005067b1 sp 00007fff37a82f40 error 4 in Xorg[400000+1d4000] May 24 17:05:29 marvell-desktop kernel: [ 86.721658] rtkit-daemon[1708]: segfault at ffffffffffffff80 ip 00007fe23abee61a sp 00007fff9c249410 error 4 in libdbus-1.so.3.5.7[7fe23abc3000+42000] What IRQF flags I should use when setting up MSI IRQ, etc? What did I do wrong?? Please help! Kenneth ________________________________________ From: Kenneth Wong Sent: Thursday, May 24, 2012 4:21 PM To: xen-devel@lists.xen.org Subject: RE: [Xen-devel] Loading PCIe Device Driver at Dom0 Hi all, I am now using dma_alloc_coherent(). However, the first parameter, pdev->dev required, which is "struct device", does not seem to have initialized. When and who is suppose to initialize it? In Linux, I can pass "NULL" and it just works. In Xen, it crashes. Sometimes insmod pass, sometimes hangs the system. Please advise! Kenneth ________________________________________ From: Pasi Kärkkäinen [pasik@iki.fi] Sent: Wednesday, May 23, 2012 11:18 PM To: Kenneth Wong Cc: xen-devel@lists.xen.org Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 On Wed, May 23, 2012 at 07:41:34PM -0700, Kenneth Wong wrote:> Hi Konrad and others, > > Oh, there are Xen/dom0 specific APIs for PCI and DMA? >PCI/DMA APIs are the Linux kernel APIs, not Xen specific.> May I ask the names and where can I can more info on the APIs? >Quick googling reveals: http://www.mjmwired.net/kernel/Documentation/DMA-API.txt http://www.mjmwired.net/kernel/Documentation/DMA-API-HOWTO.txt -- Pasi> Many thanks! > > Kenneth > > ________________________________________ > From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com] > Sent: Wednesday, May 23, 2012 5:17 PM > To: Kenneth Wong > Cc: xen-devel@lists.xen.org > Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 > > On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote: > > Dear all, > > > > I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along. > > > > I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1. When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue. I can do reads and write to the hardware device. But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs. > > > > >From my debug messages, it appears it hangs because the driver doesn''t receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register. Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver. > > > > The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure. > > The 2). > > > > > > What the driver does: > > You do need to use the PCI API (or the DMA API). > > > > > Set up a command buffer: > > Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL); > > unsigned long buf_addr = __pa(buf); > > unsigned int buf_addr_low = (unsigned int)buf_addr; > > > > Tell device about the buffer: > > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW); > > > > > Set up IRQ: > > if (pci_find_capability(dev, PCI_CAP_ID_MSI) && > > (!pci_enable_msi(dev))) > > { > > if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev)) > > { > > return -ENODEV; > > } > > my_dev->intr_mode = INTERRUPT_MSI; > > } > > > > Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf. But interrupt did not happen.): > > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG); > > > > > > >From dmesg, it looks like IRQ initialization is complete. > > [ 241.743769] My_driver initialization > > [ 241.743787] xen: registering gsi 16 triggering 0 polarity 1 > > [ 241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16 > > [ 241.743795] xen: --> pirq=16 -> irq=16 (gsi=16) > > [ 241.743801] Already setup the GSI :16 > > [ 241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 > > [ 241.743815] my-driver 0000:02:00.0: setting latency timer to 64 > > > > /proc/interrupts: > > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 > > ...... > > ...... > > 339: 0 0 0 0 0 0 0 0 xen-pirq-msi my-driver > > ...... > > ...... > > > > Any idea what might cause the problem? > > > > Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following? > > 1) Allocating a DMA buffer in driver to allow the device to DMA stuffs. > > 2) Requesting MSI irq. > > > > Please advise! > > > > Thanks a lot in advance!! > > > > Kenneth > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
On Thu, May 24, 2012 at 04:21:16PM -0700, Kenneth Wong wrote:> Hi all, > > I am now using dma_alloc_coherent(). > > However, the first parameter, pdev->dev required, which is "struct device", does not seem to have initialized. > > When and who is suppose to initialize it?Um, does your driver have a PCI vendor and model? It would do it from the struct pci_driver->probe function.> > In Linux, I can pass "NULL" and it just works. In Xen, it crashes.NULL in that case is incorrect.> > Sometimes insmod pass, sometimes hangs the system. > > Please advise!Look at how other drivers do it. You might also want to pick up an Linux Device Drivers book and read the chapter about PCI devices.> > Kenneth > ________________________________________ > From: Pasi Kärkkäinen [pasik@iki.fi] > Sent: Wednesday, May 23, 2012 11:18 PM > To: Kenneth Wong > Cc: xen-devel@lists.xen.org > Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 > > On Wed, May 23, 2012 at 07:41:34PM -0700, Kenneth Wong wrote: > > Hi Konrad and others, > > > > Oh, there are Xen/dom0 specific APIs for PCI and DMA? > > > > PCI/DMA APIs are the Linux kernel APIs, not Xen specific. > > > > May I ask the names and where can I can more info on the APIs? > > > > Quick googling reveals: > > http://www.mjmwired.net/kernel/Documentation/DMA-API.txt > http://www.mjmwired.net/kernel/Documentation/DMA-API-HOWTO.txt > > > -- Pasi > > > Many thanks! > > > > Kenneth > > > > ________________________________________ > > From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com] > > Sent: Wednesday, May 23, 2012 5:17 PM > > To: Kenneth Wong > > Cc: xen-devel@lists.xen.org > > Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 > > > > On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote: > > > Dear all, > > > > > > I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along. > > > > > > I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1. When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue. I can do reads and write to the hardware device. But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs. > > > > > > >From my debug messages, it appears it hangs because the driver doesn''t receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register. Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver. > > > > > > The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure. > > > > The 2). > > > > > > > > > What the driver does: > > > > You do need to use the PCI API (or the DMA API). > > > > > > > > Set up a command buffer: > > > Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL); > > > unsigned long buf_addr = __pa(buf); > > > unsigned int buf_addr_low = (unsigned int)buf_addr; > > > > > > Tell device about the buffer: > > > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW); > > > > > > > > Set up IRQ: > > > if (pci_find_capability(dev, PCI_CAP_ID_MSI) && > > > (!pci_enable_msi(dev))) > > > { > > > if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev)) > > > { > > > return -ENODEV; > > > } > > > my_dev->intr_mode = INTERRUPT_MSI; > > > } > > > > > > Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf. But interrupt did not happen.): > > > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG); > > > > > > > > > >From dmesg, it looks like IRQ initialization is complete. > > > [ 241.743769] My_driver initialization > > > [ 241.743787] xen: registering gsi 16 triggering 0 polarity 1 > > > [ 241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16 > > > [ 241.743795] xen: --> pirq=16 -> irq=16 (gsi=16) > > > [ 241.743801] Already setup the GSI :16 > > > [ 241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 > > > [ 241.743815] my-driver 0000:02:00.0: setting latency timer to 64 > > > > > > /proc/interrupts: > > > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 > > > ...... > > > ...... > > > 339: 0 0 0 0 0 0 0 0 xen-pirq-msi my-driver > > > ...... > > > ...... > > > > > > Any idea what might cause the problem? > > > > > > Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following? > > > 1) Allocating a DMA buffer in driver to allow the device to DMA stuffs. > > > 2) Requesting MSI irq. > > > > > > Please advise! > > > > > > Thanks a lot in advance!! > > > > > > Kenneth > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xen.org > > > http://lists.xen.org/xen-devel > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Hi Konrad,> Um, does your driver have a PCI vendor and model? It would > do it from the struct pci_driver->probe function.Yeah, it has all that. It has been running fine on regular Linux, just that problems start coming up when porting to Xen env. I think because it is now less forgiving due to the virtualization layer. Any idea on the following messages? Xorg[1234]: segfault at 34 ip 00000000005067b1 sp 00007fff37a82f40 error 4 in Xorg[400000+1d4000] rtkit-daemon[1708]: segfault at ffffffffffffff80 ip 00007fe23abee61a sp 00007fff9c249410 error 4 in libdbus-1.so.3.5.7[7fe23abc3000+42000] I think this might be cause by some other things in the driver. Thanks, Kenneth ________________________________________ From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com] Sent: Thursday, May 24, 2012 6:34 PM To: Kenneth Wong Cc: xen-devel@lists.xen.org Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 On Thu, May 24, 2012 at 04:21:16PM -0700, Kenneth Wong wrote:> Hi all, > > I am now using dma_alloc_coherent(). > > However, the first parameter, pdev->dev required, which is "struct device", does not seem to have initialized. > > When and who is suppose to initialize it?Um, does your driver have a PCI vendor and model? It would do it from the struct pci_driver->probe function.> > In Linux, I can pass "NULL" and it just works. In Xen, it crashes.NULL in that case is incorrect.> > Sometimes insmod pass, sometimes hangs the system. > > Please advise!Look at how other drivers do it. You might also want to pick up an Linux Device Drivers book and read the chapter about PCI devices.> > Kenneth > ________________________________________ > From: Pasi Kärkkäinen [pasik@iki.fi] > Sent: Wednesday, May 23, 2012 11:18 PM > To: Kenneth Wong > Cc: xen-devel@lists.xen.org > Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 > > On Wed, May 23, 2012 at 07:41:34PM -0700, Kenneth Wong wrote: > > Hi Konrad and others, > > > > Oh, there are Xen/dom0 specific APIs for PCI and DMA? > > > > PCI/DMA APIs are the Linux kernel APIs, not Xen specific. > > > > May I ask the names and where can I can more info on the APIs? > > > > Quick googling reveals: > > http://www.mjmwired.net/kernel/Documentation/DMA-API.txt > http://www.mjmwired.net/kernel/Documentation/DMA-API-HOWTO.txt > > > -- Pasi > > > Many thanks! > > > > Kenneth > > > > ________________________________________ > > From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com] > > Sent: Wednesday, May 23, 2012 5:17 PM > > To: Kenneth Wong > > Cc: xen-devel@lists.xen.org > > Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 > > > > On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote: > > > Dear all, > > > > > > I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along. > > > > > > I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1. When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue. I can do reads and write to the hardware device. But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs. > > > > > > >From my debug messages, it appears it hangs because the driver doesn''t receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register. Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver. > > > > > > The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure. > > > > The 2). > > > > > > > > > What the driver does: > > > > You do need to use the PCI API (or the DMA API). > > > > > > > > Set up a command buffer: > > > Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL); > > > unsigned long buf_addr = __pa(buf); > > > unsigned int buf_addr_low = (unsigned int)buf_addr; > > > > > > Tell device about the buffer: > > > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW); > > > > > > > > Set up IRQ: > > > if (pci_find_capability(dev, PCI_CAP_ID_MSI) && > > > (!pci_enable_msi(dev))) > > > { > > > if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev)) > > > { > > > return -ENODEV; > > > } > > > my_dev->intr_mode = INTERRUPT_MSI; > > > } > > > > > > Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf. But interrupt did not happen.): > > > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG); > > > > > > > > > >From dmesg, it looks like IRQ initialization is complete. > > > [ 241.743769] My_driver initialization > > > [ 241.743787] xen: registering gsi 16 triggering 0 polarity 1 > > > [ 241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16 > > > [ 241.743795] xen: --> pirq=16 -> irq=16 (gsi=16) > > > [ 241.743801] Already setup the GSI :16 > > > [ 241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 > > > [ 241.743815] my-driver 0000:02:00.0: setting latency timer to 64 > > > > > > /proc/interrupts: > > > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 > > > ...... > > > ...... > > > 339: 0 0 0 0 0 0 0 0 xen-pirq-msi my-driver > > > ...... > > > ...... > > > > > > Any idea what might cause the problem? > > > > > > Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following? > > > 1) Allocating a DMA buffer in driver to allow the device to DMA stuffs. > > > 2) Requesting MSI irq. > > > > > > Please advise! > > > > > > Thanks a lot in advance!! > > > > > > Kenneth > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xen.org > > > http://lists.xen.org/xen-devel > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
On Thu, May 24, 2012 at 07:37:44PM -0700, Kenneth Wong wrote:> Hi Konrad, > > > Um, does your driver have a PCI vendor and model? It would > > do it from the struct pci_driver->probe function. > > Yeah, it has all that. It has been running fine on regular Linux, just that problems start coming up when porting to Xen env. I think because it is now less forgiving due to the virtualization layer.Sure. It also means that your driver would not work with IOMMU''s properly.> > Any idea on the following messages?Some, but without any details (like machine type, userspace version, Xorg version, kernel version, Xorg.0.log, etc) I''ve no clue.> > Xorg[1234]: segfault at 34 ip 00000000005067b1 sp 00007fff37a82f40 error 4 in Xorg[400000+1d4000] > rtkit-daemon[1708]: segfault at ffffffffffffff80 ip 00007fe23abee61a sp 00007fff9c249410 error 4 in libdbus-1.so.3.5.7[7fe23abc3000+42000] > > I think this might be cause by some other things in the driver.So you see this only after you load your driver?
On Fri, May 25, 2012 at 02:12:48PM -0700, Kenneth Wong wrote:> Hi Konrad,Please do not top-post.> > It''s Xen 4.1.1. > > It did not occur immediately after loading. It occurs after a few I/Os... snip.. Please provide the details I''ve asked for.> > Any idea on the following messages? > > Some, but without any details (like machine type, userspace version, Xorg version, > kernel version, Xorg.0.log, etc) I''ve no clue. > > > > > Xorg[1234]: segfault at 34 ip 00000000005067b1 sp 00007fff37a82f40 error 4 in Xorg[400000+1d4000] > > rtkit-daemon[1708]: segfault at ffffffffffffff80 ip 00007fe23abee61a sp 00007fff9c249410 error 4 in libdbus-1.so.3.5.7[7fe23abc3000+42000] > > > > I think this might be cause by some other things in the driver. > > So you see this only after you load your driver? > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Hi Konrad, It''s Xen 4.1.1. It did not occur immediately after loading. It occurs after a few I/Os. Best Regards, Kenneth -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: Friday, May 25, 2012 1:31 PM To: Kenneth Wong Cc: xen-devel@lists.xen.org Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 On Thu, May 24, 2012 at 07:37:44PM -0700, Kenneth Wong wrote:> Hi Konrad, > > > Um, does your driver have a PCI vendor and model? It would > > do it from the struct pci_driver->probe function. > > Yeah, it has all that. It has been running fine on regular Linux, just that problems start coming up when porting to Xen env. I think because it is now less forgiving due to the virtualization layer.Sure. It also means that your driver would not work with IOMMU''s properly.> > Any idea on the following messages?Some, but without any details (like machine type, userspace version, Xorg version, kernel version, Xorg.0.log, etc) I''ve no clue.> > Xorg[1234]: segfault at 34 ip 00000000005067b1 sp 00007fff37a82f40 error 4 in Xorg[400000+1d4000] > rtkit-daemon[1708]: segfault at ffffffffffffff80 ip 00007fe23abee61a sp 00007fff9c249410 error 4 in libdbus-1.so.3.5.7[7fe23abc3000+42000] > > I think this might be cause by some other things in the driver.So you see this only after you load your driver?