I asked this question on the users list a few days ago, but got no
response, perhaps it''s a bit too involved for that list, I hope
it''s not
*too* unwelcome here ...
I''m trying to use PCI passthrough of a DVB-T tuner from a fedora8 dom0,
to a centOS5.2 domU
On the dom0 I have the following kernel/xen installed
kernel-xen.x86_64 2.6.21.7-3.fc8
xen.x86_64 3.1.2-2.fc8
xen-libs.x86_64 3.1.2-2.fc8
On the domU I have the following kernel installed
kernel-xen.x86_64 2.6.18.92.1.6.el5.centos.plus
Additionally I had to manually compile the saa7134_dvb.ko module for the
digital section of the card as the centosplus kernel only had the
saa7134.ko module for the analogue section of the card.
I tried adding a pciback.hide entry to grub.conf for the dom0, but this
wasn''t recognised, presumably because pciback is compiled as a module
on
fedora8 dom0 kernel?
I realise I can add entries to modprobe.conf to ensure no dom0 drivers
bind to my device, but for now I''m using the following script to get
dom0 to relinquish the card and make it available for the domU.
rmmod saa7134
modprobe pciback
SLOT=0000:08:01.0
echo -n $SLOT > /sys/bus/pci/drivers/pciback/new_slot
echo -n $SLOT > /sys/bus/pci/drivers/pciback/bind
I then create my domU with this config file
memory = 1024
name = "mythbe"
vif = [ ''mac=00:16:3E:76:E8:92, bridge=eth0'' ]
disk = [ ''phy:/dev/vgr1/lvmythroot,xvda,w'',
''phy:/dev/vgr5/lvmythdata,xvdb,w'' ]
pci = [''08:01.0'']
bootloader = "/usr/bin/pygrub"
vcpus = 1
on_reboot = ''restart''
on_crash = ''restart''
I see a few PCI related messages in the domU dmesg
PCI: Fatal: No PCI config space access function found
PCI: setting up Xen PCI frontend stub
PCI: System does not support PCI
PCI: System does not support PCI
pcifront pci-0: Installing PCI frontend
pcifront pci-0: Creating PCI Frontend Bus 0000:00
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
ohci_hcd: 2005 April 22 USB 1.1 ''Open'' Host Controller (OHCI)
Driver (PCI)
PCI: Enabling device 0000:00:00.0 (0000 -> 0002)
The DVB-T card is visble in the domU
# lspci -v
00:00.0 Multimedia controller: Philips Semiconductors SAA7130 Video
Broadcast Decoder (rev 01)
Subsystem: Compro Technology, Inc. Videomate DVB-T200
Flags: bus master, medium devsel, latency 64, IRQ 17
Memory at febffc00 (32-bit, non-prefetchable) [size=1K]
Capabilities: [40] Power Management version 1
I assume it''s normal that the PCI bus/device/function numbers have been
changed between dom0 and domU?
The driver modules are being loaded by the domU
# lsmod | grep saa
saa7134_dvb 48004 0
dvb_pll 48965 1 saa7134_dvb
mt352 40133 1 saa7134_dvb
video_buf_dvb 40133 1 saa7134_dvb
nxt200x 46661 1 saa7134_dvb
tda1004x 48581 1 saa7134_dvb
saa7134 159017 1 saa7134_dvb
video_buf 59717 3 saa7134_dvb,video_buf_dvb,saa7134
compat_ioctl32 41793 1 saa7134
ir_kbd_i2c 42961 1 saa7134
i2c_core 56129 7
saa7134_dvb,dvb_pll,mt352,nxt200x,tda1004x,saa7134,ir_kbd_i2c
ir_common 63173 2 saa7134,ir_kbd_i2c
videodev 58049 1 saa7134
v4l1_compat 44613 2 saa7134,videodev
v4l2_common 57153 3 saa7134,compat_ioctl32,videodev
I see an error in dmesg on the domU
# dmesg | grep -i saa
saa7130/34: v4l2 driver version 0.2.14 loaded
saa7130[0]: found at 0000:00:00.0, rev: 1, irq: 17, latency: 64, mmio:
0xfebffc00
saa7130[0]: subsystem: 185b:c901, board: Compro Videomate DVB-T200
[card=71,autodetected]
saa7130[0]: can''t ioremap() MMIO memory
saa7134: probe of 0000:00:00.0 failed with error -5
The calling code in domU is here
http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.18.y.git;a=blob;f=drivers/media/video/saa7134/saa7134-core.c;h=be3a81fc90a2498769cb95ca2b4f306c4d71cfc8;hb=HEAD#l911
I also receive a corresponding message in dom0
# xm dmesg
(XEN) mm.c:625:d6 Non-privileged (6) attempt to map I/O space 000fec00
The called code in dom0 is here
http://lxr.xensource.com/lxr/source/xen/arch/x86/mm.c#L674
Am I right in thinking the pciback.permissive parameter is now removed?
Certainly modinfo is unaware of it.
Any suggestions for how to stop the ioremap() call from failing and thus
allow the driver to loading properly?
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jul-01 08:30 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 1/7/08 09:17, "Andy Burns" <lists.xensource.com@adslpipe.co.uk> wrote:> # xm dmesg > > (XEN) mm.c:625:d6 Non-privileged (6) attempt to map I/O space 000fec00What does lspci say the resource areas are for this device? It looks like fec00000 is past the end of the device''s mmio region.> The called code in dom0 is here > > http://lxr.xensource.com/lxr/source/xen/arch/x86/mm.c#L674 > > Am I right in thinking the pciback.permissive parameter is now removed? > Certainly modinfo is unaware of it.It got turned into a per-device attribute. I think you echo the PCI slot value to /sys/bus/pci/drivers/pciback/permissive -- Keir> Any suggestions for how to stop the ioremap() call from failing and thus > allow the driver to loading properly?_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andy Burns
2008-Jul-01 08:58 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 01/07/2008 09:30, Keir Fraser wrote:> What does lspci say the resource areas are for this device?Are you asking for anything more than "lspci -vvv" output? When the card is loaded in dom0 (before I unload the drivers and assign the slot to pciback) I see this 08:01.0 Multimedia controller: Philips Semiconductors SAA7130 Video Broadcast Decoder (rev 01) Subsystem: Compro Technology, Inc. Videomate DVB-T200 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 (21000ns min, 8000ns max) Interrupt: pin A routed to IRQ 17 Region 0: Memory at febffc00 (32-bit, non-prefetchable) [size=1K] Capabilities: [40] Power Management version 1 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 PME-Enable- DSel=0 DScale=1 PME- Kernel modules: saa7134 When assigned to pciback/pcifront, from within the domU I see this 00:00.0 Multimedia controller: Philips Semiconductors SAA7130 Video Broadcast Decoder (rev 01) Subsystem: Compro Technology, Inc. Videomate DVB-T200 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- Latency: 64 (21000ns min, 8000ns max) Interrupt: pin A routed to IRQ 17 Region 0: Memory at febffc00 (32-bit, non-prefetchable) [size=1K] Capabilities: [40] Power Management version 1 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 PME-Enable- DSel=0 DScale=1 PME-> It looks like fec00000 is past the end of the device''s mmio region.Well 0xFEBFFC00 + 1K = 0xFEC00000, is this some sort of off-by-one error, where it should only extend to 0xFEBFFFFF? Is it worth me attempting a change in the driver code from dev->lmmio = ioremap(pci_resource_start(pci_dev,0), 0x1000); to dev->lmmio = ioremap(pci_resource_start(pci_dev,0), 0xFFF); _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andy Burns
2008-Jul-01 09:09 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 01/07/2008 09:30, Keir Fraser wrote:> What does lspci say the resource areas are for this device?Within the dom0, after I''ve unloaded the driver and re-assigned the slot, I see this 08:01.0 Multimedia controller: Philips Semiconductors SAA7130 Video Broadcast Decoder (rev 01) Subsystem: Compro Technology, Inc. Videomate DVB-T200 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 17 Region 0: Memory at febffc00 (32-bit, non-prefetchable) [disabled] [size=1K] Capabilities: [40] Power Management version 1 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 PME-Enable- DSel=0 DScale=1 PME- Kernel driver in use: pciback Kernel modules: saa7134 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jul-01 09:45 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 1/7/08 09:58, "Andy Burns" <lists.xensource.com@adslpipe.co.uk> wrote:>> It looks like fec00000 is past the end of the device''s mmio region. > > > Well 0xFEBFFC00 + 1K = 0xFEC00000, is this some sort of off-by-one > error, where it should only extend to 0xFEBFFFFF? > > Is it worth me attempting a change in the driver code from > > dev->lmmio = ioremap(pci_resource_start(pci_dev,0), 0x1000); > > to > > dev->lmmio = ioremap(pci_resource_start(pci_dev,0), 0xFFF);It shouldn''t be necessary, since the range specified to ioremap is by base address and length and covers the inclusive range ''base'' to ''base+length-1''. It''s rather odd that the kernel tries to map FEC00000 at all. If you''re happy changing kernel code, you might try adding printk() instrumentation to __ioremap() and __direct_remap_pfn_range() in arch/i386/mm/ioremap-xen.c. The functions look like they should map the correct range of pages (in this case only the page covering FEBFF000-FEBFFFFF) but perhaps there is an off-by-one error lurking. Or perhaps the mapping attempt of FEC00000 comes from somewhere else in the kernel. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andy Burns
2008-Jul-01 13:16 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 01/07/2008 10:45, Keir Fraser wrote:> If you''re > happy changing kernel code, you might try adding printk() instrumentation to > __ioremap()printk(KERN_DEBUG "REMAP: phys=0x%lx, len=%lu\n", phys_addr, size); and __direct_remap_pfn_range() in arch/i386/mm/ioremap-xen.c. printk(KERN_DEBUG "REMAPPFN: addr=0x%lx, mfn=0x%lx, size=%lu\n", address, mfn, size);> The functions look like they should map the correct range of pages (in this > case only the page covering FEBFF000-FEBFFFFF)Built and installed kernel/modules/initrd, rebooted, when it loads the saa7134 driver I see this REMAP: phys=0xfebffc00, len=4096 REMAPPFN: addr=0xffffc20000038000, mfn=0xfebff, size=8192 So the 1K mapping has been rounded up to 4K (x86_64 page size?) before passing to __ioremap() and then rounded up again to 8K by the time it gets passed to __direct_remap_pfn_range() is that right? I can manage to build a kernel, but I''m out of my depth in terms of knowing what would be good or bad with physical addresses and page mappings ... _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jul-01 13:31 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 1/7/08 14:16, "Andy Burns" <lists.xensource.com@adslpipe.co.uk> wrote:>> The functions look like they should map the correct range of pages (in this >> case only the page covering FEBFF000-FEBFFFFF) > > Built and installed kernel/modules/initrd, rebooted, when it loads the > saa7134 driver I see this > > REMAP: phys=0xfebffc00, len=4096 > REMAPPFN: addr=0xffffc20000038000, mfn=0xfebff, size=8192 > > So the 1K mapping has been rounded up to 4K (x86_64 page size?) before > passing to __ioremap() and then rounded up again to 8K by the time it > gets passed to __direct_remap_pfn_range() is that right?Well, your analysis is correct, and the size argument to __ioremap() is bogus. It shouldn''t have been rounded up to 4096 without also rounding down the base address. I don''t think this would happen with out linux-2.6.18-xen tree. In there, ioremap() is defined in include/asm-x86_64/mach-xen/io.h as a thin wrapper around __ioremap() which does not modify the size parameter. So, could be a bug specific to the FC8 kernel. I don''t have its sources to hand to pinpoint where the size is getting changed from 0x400 to 0x1000. You should be able to dig up that detail pretty easily though. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andy Burns
2008-Jul-01 15:44 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 01/07/2008 14:31, Keir Fraser wrote:> Well, your analysis is correct, and the size argument to __ioremap() is > bogus. It shouldn''t have been rounded up to 4096 without also rounding down > the base address.Just been having a look and a think, the size isn''t being rounded up anywhere, the size is actually passed in as 4K from the driver dev->lmmio = ioremap(pci_resource_start(pci_dev,0), 0x1000); Perhaps the rounding up to 8K is happening because the address isn''t aligned on a page boundary, so is split over two pages? So I assume it is the driver at fault for mapping a larger resource area than the card actually supports? I''ll try changing the size argument to 0x400 and rebuilding the saa7134 module.> So, could be a bug specific to the FC8 kernel.Actually centos5.2 is the domU, FC8 is the dom0, since it''s x86_64, I assume it''s the Fedora/Redhat forward-port effort, rather than the paravirt_ops effort. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andy Burns
2008-Jul-01 16:42 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 01/07/2008 16:44, Andy Burns wrote:> I''ll try changing the size argument to 0x400 and rebuilding the saa7134 > module.OK, changing the ioremap() in the saa7134 driver to only request 1K allowed the driver to load :-) saa7130/34: v4l2 driver version 0.2.14 loaded PCI: Enabling device 0000:00:00.0 (0000 -> 0002) saa7130[0]: found at 0000:00:00.0, rev: 1, irq: 17, latency: 64, mmio: 0xfebffc00 saa7130[0]: subsystem: 185b:c901, board: Compro Videomate DVB-T200 [card=71,auto detected] saa7130[0]: board init: gpio is 843f00 input: saa7134 IR (Compro Videomate DV as /class/input/input1 saa7130[0]: i2c eeprom 00: 5b 18 01 c9 54 20 1c 00 43 43 a9 1c 55 d2 b2 92 saa7130[0]: i2c eeprom 10: 00 ff 86 0f ff 20 ff ff ff ff ff ff ff ff ff ff saa7130[0]: i2c eeprom 20: 01 40 01 03 03 ff 03 01 08 ff 00 88 ff ff ff ff saa7130[0]: i2c eeprom 30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff saa7130[0]: i2c eeprom 40: ff d0 00 c2 86 10 ff ff ff ff ff ff ff ff ff ff saa7130[0]: i2c eeprom 50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff cb saa7130[0]: i2c eeprom 60: 30 26 ff ff ff ff ff ff ff ff ff ff ff ff ff ff saa7130[0]: i2c eeprom 70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff saa7130[0]: registered device video0 [v4l2] saa7130[0]: registered device vbi0 From my extra printk() debugging the driver now requests 1K, and within __direct_remap_pfn_range() the size has been rounded up to 4K, presumably this *is* correct as the mfn seems to has been rounded down? REMAP: phys=0xfebffc00, len=1024 REMAPPFN: addr=0xffffc2000002e000, mfn=0xfebff, size=4096 I checked in 2.6.26 git to see if the saa7134 driver had been altered so the size had been reduced in more recent kernels, but it has not. I''ll check with the V4L developers whether it should be hardcoded to 1K instead of 4K, or whether it needs to be sized dynamically for different cards. However about 40 seconds after loading the module, I get the following error :-( irq 17: nobody cared (try booting with the "irqpoll" option) Call Trace: <IRQ> [<ffffffff802af539>] __report_bad_irq+0x30/0x7d [<ffffffff802af770>] note_interrupt+0x1ea/0x22b [<ffffffff802aec76>] __do_IRQ+0xbd/0x103 [<ffffffff8028d06f>] _local_bh_enable+0x61/0xc5 [<ffffffff8026db3a>] do_IRQ+0xe7/0xf5 [<ffffffff8039f151>] evtchn_do_upcall+0x86/0xe0 [<ffffffff802608ce>] do_hypervisor_callback+0x1e/0x2c <EOI> [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000 [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000 [<ffffffff8026f12b>] raw_safe_halt+0x84/0xa8 [<ffffffff8026c675>] xen_idle+0x38/0x4a [<ffffffff8024a9bb>] cpu_idle+0x97/0xba [<ffffffff805f6b09>] start_kernel+0x21f/0x224 [<ffffffff805f61e5>] _sinittext+0x1e5/0x1eb handlers: [<ffffffff88142ff6>] (saa7134_irq+0x0/0x284 [saa7134]) Disabling IRQ #17 mythbe kernel: Disabling IRQ #17 I tried adding irqpoll to the relevent stanza in /etc/grub.conf but the error still occurs. Is this likely that the interrupt handler for the card isn''t being wired up by pcifront, or perhaps a stray interrupt generated by the card? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jul-01 17:10 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 1/7/08 16:44, "Andy Burns" <lists.xensource.com@adslpipe.co.uk> wrote:> Perhaps the rounding up to 8K is happening because the address isn''t > aligned on a page boundary, so is split over two pages?Yes, since the argument to ioremap() is 0x1000, the underlying ioremap implementation is doing the right thing.> So I assume it is the driver at fault for mapping a larger resource area > than the card actually supports? > > I''ll try changing the size argument to 0x400 and rebuilding the saa7134 > module.Yes, it''s the driver''s fault. 0x400 would be correct.>> So, could be a bug specific to the FC8 kernel. > > Actually centos5.2 is the domU, FC8 is the dom0, since it''s x86_64, I > assume it''s the Fedora/Redhat forward-port effort, rather than the > paravirt_ops effort.Yes, it will be the FC/RH forward-port. Anyhow, the bug isn''t in the generic ioremap code after all. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jul-01 17:15 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 1/7/08 17:42, "Andy Burns" <lists.xensource.com@adslpipe.co.uk> wrote:> From my extra printk() debugging the driver now requests 1K, and within > __direct_remap_pfn_range() the size has been rounded up to 4K, > presumably this *is* correct as the mfn seems to has been rounded down?Yes, you can only map memory at page granularity, so this is the smallest mapping that contains the required resource. This is valid.> I checked in 2.6.26 git to see if the saa7134 driver had been altered so > the size had been reduced in more recent kernels, but it has not. I''ll > check with the V4L developers whether it should be hardcoded to 1K > instead of 4K, or whether it needs to be sized dynamically for different > cards.Good idea. This kind of thing can be an honest mistake which would normally never cause a problem running natively.> However about 40 seconds after loading the module, I get the following > error :-(Does the card work at all up to this point (first 40s)? Any sign of life after the message prints? This would indicate either that Xen/pciback has got irq routing wrong for this line (so the wrong interrupts are being delivered to the driver) or that for some other reason the driver has decided that there was no work to be done in response to the interrupt. Could the irq line for this device be a shared line? You could try disabling the spurious irq logic in kernel/irq/spurious.c, and then see if the device works and also what the irq rate looks like in /proc/interrupts. -- Keir> irq 17: nobody cared (try booting with the "irqpoll" option) > > Call Trace: > <IRQ> [<ffffffff802af539>] __report_bad_irq+0x30/0x7d > [<ffffffff802af770>] note_interrupt+0x1ea/0x22b > [<ffffffff802aec76>] __do_IRQ+0xbd/0x103 > [<ffffffff8028d06f>] _local_bh_enable+0x61/0xc5 > [<ffffffff8026db3a>] do_IRQ+0xe7/0xf5 > [<ffffffff8039f151>] evtchn_do_upcall+0x86/0xe0 > [<ffffffff802608ce>] do_hypervisor_callback+0x1e/0x2c > <EOI> [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000 > [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000 > [<ffffffff8026f12b>] raw_safe_halt+0x84/0xa8 > [<ffffffff8026c675>] xen_idle+0x38/0x4a > [<ffffffff8024a9bb>] cpu_idle+0x97/0xba > [<ffffffff805f6b09>] start_kernel+0x21f/0x224 > [<ffffffff805f61e5>] _sinittext+0x1e5/0x1eb > > handlers: > [<ffffffff88142ff6>] (saa7134_irq+0x0/0x284 [saa7134]) > Disabling IRQ #17 > > mythbe kernel: Disabling IRQ #17 > > I tried adding irqpoll to the relevent stanza in /etc/grub.conf but the > error still occurs. > > Is this likely that the interrupt handler for the card isn''t being wired > up by pcifront, or perhaps a stray interrupt generated by the card?_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andy Burns
2008-Jul-01 18:50 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 01/07/2008 18:15, Keir Fraser wrote:> Good idea. This kind of thing can be an honest mistake which would normally > never cause a problem running natively.I''ve asked the linuxtv folk if they are interested in a patch.> Does the card work at all up to this point (first 40s)?Well, if I modprobe the saa7134.ko and saa7134_dvb.ko I get the /dev/video0 and /dev/dvb/adapter0 devices created But if I try to scan for TV channels with # dvbscan /usr/share/dvb/dvb-t/uk-Waltham > ~/.tzap/channels.conf I get "Unable to query frontend status", so the device doesn''t seem to be functioning.> Any sign of life after the message prints?The machine doesn''t die when the stray interrupt fires, in fact I can rmmod the module and modprobe it again, the error repeats after another 40 seconds.> This would indicate either that Xen/pciback has got irq routing wrong for > this line (so the wrong interrupts are being delivered to the driver) or > that for some other reason the driver has decided that there was no work to > be done in response to the interrupt. Could the irq line for this device be > a shared line?I did wonder how (or if) shared interrupts would work. Given the relatively high number of devices on the motherboard and PCI-X/PCI/PCIe cards I suspect it will have to be shared. Here is a brief lspci 00:00.0 Host bridge: Intel Corporation 82X38 Express DRAM Controller (rev 01) 00:01.0 PCI bridge: Intel Corporation 82X38 Express Host-Primary PCI Express Bridge (rev 01) 00:1a.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 02) 00:1a.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 02) 00:1a.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 02) 00:1a.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 02) 00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 02) 00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02) 00:1c.2 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 3 (rev 02) 00:1c.3 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 4 (rev 02) 00:1c.4 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 5 (rev 02) 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02) 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02) 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02) 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) 00:1f.0 ISA bridge: Intel Corporation 82801IR (ICH9R) LPC Interface Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA AHCI Controller (rev 02) 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02) 01:00.0 VGA compatible controller: ATI Technologies Inc RV370 [Sapphire X550 Silent] 01:00.1 Display controller: ATI Technologies Inc RV370 secondary [Sapphire X550 Silent] 02:00.0 IDE interface: Marvell Technology Group Ltd. 88SE6145 SATA II PCI-E controller (rev a1) 03:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 12) 04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 12) 05:00.0 PCI bridge: NEC Corporation uPD720400 PCI Express - PCI/PCI-X Bridge (rev 06) 05:00.1 PCI bridge: NEC Corporation uPD720400 PCI Express - PCI/PCI-X Bridge (rev 06) 07:01.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09) 08:01.0 Multimedia controller: Philips Semiconductors SAA7130 Video Broadcast Decoder (rev 01) 08:03.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host Controller (rev c0) Looks like the PCI slot with the tuner shares IRQ17 with the PCI-X slot used by my 8xSATA card. 07:01.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09) Subsystem: Marvell Technology Group Ltd. Unknown device 11ab Interrupt: pin A routed to IRQ 17 08:01.0 Multimedia controller: Philips Semiconductors SAA7130 Video Broadcast Decoder (rev 01) Subsystem: Compro Technology, Inc. Videomate DVB-T200 Interrupt: pin A routed to IRQ 17 I can try to move the tuner card to the other PCI slot, but I suspect it will just share with something else instead, besides I planned to populate both PCI slots with tuners for passthrough, once I got the first one working.> You could try disabling the spurious irq logic in > kernel/irq/spurious.c, and then see if the device works and also what the > irq rate looks like in /proc/interrupts.I''ll have a go at that. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2008-Jul-01 18:52 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
Keir Fraser wrote:> On 1/7/08 14:16, "Andy Burns" <lists.xensource.com@adslpipe.co.uk> wrote: > > >>> The functions look like they should map the correct range of pages (in this >>> case only the page covering FEBFF000-FEBFFFFF) >>> >> Built and installed kernel/modules/initrd, rebooted, when it loads the >> saa7134 driver I see this >> >> REMAP: phys=0xfebffc00, len=4096 >> REMAPPFN: addr=0xffffc20000038000, mfn=0xfebff, size=8192 >> >> So the 1K mapping has been rounded up to 4K (x86_64 page size?) before >> passing to __ioremap() and then rounded up again to 8K by the time it >> gets passed to __direct_remap_pfn_range() is that right? >> > > Well, your analysis is correct, and the size argument to __ioremap() is > bogus. It shouldn''t have been rounded up to 4096 without also rounding down > the base address. I don''t think this would happen with out linux-2.6.18-xen > tree. In there, ioremap() is defined in include/asm-x86_64/mach-xen/io.h as > a thin wrapper around __ioremap() which does not modify the size parameter. > > So, could be a bug specific to the FC8 kernel. I don''t have its sources to > hand to pinpoint where the size is getting changed from 0x400 to 0x1000. You > should be able to dig up that detail pretty easily though.There was a kernel bug in which ioremap would add a 1 page redzone to the end of the mapping, and then pass that extra page to iounmap. It has been fixed for a while, but since F8. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andy Burns
2008-Jul-01 19:24 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 01/07/2008 19:50, Andy Burns wrote:> I can try to move the tuner card to the other PCI slot, but I suspect it > will just share with something else insteadIn the other PCI slot, the tuner shares with a USB controller and a SATA controller on the motherboard (instead of the PCI-X SATA card) # lspci -vv| grep IRQ Interrupt: pin A routed to IRQ 16 Interrupt: pin B routed to IRQ 21 Interrupt: pin C routed to IRQ 18 Interrupt: pin C routed to IRQ 18 Interrupt: pin A routed to IRQ 22 Interrupt: pin A routed to IRQ 23 Interrupt: pin B routed to IRQ 19 Interrupt: pin C routed to IRQ 18 Interrupt: pin A routed to IRQ 23 Interrupt: pin B routed to IRQ 22 Interrupt: pin C routed to IRQ 18 Interrupt: pin A routed to IRQ 11 Interrupt: pin A routed to IRQ 16 Interrupt: pin A routed to IRQ 19 Interrupt: pin A routed to IRQ 18 Interrupt: pin A routed to IRQ 17 Interrupt: pin A routed to IRQ 16 Interrupt: pin A routed to IRQ 15 # dmesg | grep -i apic ACPI: APIC CFF80390, 0078 (r1 A_M_I_ OEMAPIC 4000828 MSFT 97) ACPI: Local APIC address 0xfee00000 ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled) ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0]) IOAPIC[0]: apic_id 4, address 0xfec00000, GSI 0-23 ACPI: IOAPIC (id[0x05] address[0xfec10000] gsi_base[24]) IOAPIC[1]: apic_id 5, address 0xfec10000, GSI 24-279 Setting APIC routing to xen ACPI: Using IOAPIC for interrupt routing Is there any way to increase the number of "logical" interrupts to avoid (or give the impression of avoiding) sharing? When watching vmware servers boot, I remember they split devices into a huge number of interrupts numbering 100 upwards. Can Xen make use of IRQ24 to 279? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jul-01 19:57 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 1/7/08 20:24, "Andy Burns" <lists.xensource.com@adslpipe.co.uk> wrote:> # dmesg | grep -i apic > ACPI: APIC CFF80390, 0078 (r1 A_M_I_ OEMAPIC 4000828 MSFT 97) > ACPI: Local APIC address 0xfee00000 > ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) > ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) > ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled) > ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled) > ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0]) > IOAPIC[0]: apic_id 4, address 0xfec00000, GSI 0-23 > ACPI: IOAPIC (id[0x05] address[0xfec10000] gsi_base[24]) > IOAPIC[1]: apic_id 5, address 0xfec10000, GSI 24-279 > Setting APIC routing to xen > ACPI: Using IOAPIC for interrupt routing > > Is there any way to increase the number of "logical" interrupts to avoid > (or give the impression of avoiding) sharing? When watching vmware > servers boot, I remember they split devices into a huge number of > interrupts numbering 100 upwards.It would be a lie. Interrupt sharing generally depends on which lines are wired together on your mainboard. When Xen receives an interrupt on a shared line it does not have enough information to demultiplex that interrupt to the appropriate receiver -- it has to broadcast to all potential recipients. Some of these may regard it as a spurious interrupt because their device did not raise the interrupt. In our linux-2.6.18-xen tree we specifically handle this case by disabling spurious irq detection for interrupts which Xen tells us are shared.> Can Xen make use of IRQ24 to 279?I have no idea what IOAPIC[1] above really represents. Usually the GSI range for an IOAPIC corresponds to its array of interrupt input pins. I''ve never heard of a 256-pin IOAPIC! Anyhow, the assignment of PCI devices to GSIs is determined by mainboard wiring and firmware configuration. The OS (or hypervisor) does not determine that relationship. The best you could hope for would be to try out the MSI/MSI-X support in the xen-unstable tree (soon to be Xen 3.3). These are message-based interrupts which completely sidesteps the issue of interrupt lines and their wiring together. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andy Burns
2008-Jul-01 22:27 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 01/07/2008 20:57, Keir Fraser wrote:> Interrupt sharing generally depends on which lines are > wired together on your mainboard. When Xen receives an interrupt on a shared > line it does not have enough information to demultiplex that interrupt to > the appropriate receiver -- it has to broadcast to all potential recipients.OK. Sucking data at several Mb/s from a tv tuner, then squirting it to disk, where the tuner and disk are sharing the same interrupt doesn''t sound ideal then ;-) The SATA controller is presently performing an online mdraid reshape from 6 to 8 disks, averaging 6000 interrupts per second, so that''s probably what is triggering the 99,900 "ignored" interrupts in 40 seconds!> I have no idea what IOAPIC[1] above really represents. Usually the GSI range > for an IOAPIC corresponds to its array of interrupt input pins. I''ve never > heard of a 256-pin IOAPIC!Could it relate to MSI?> The best you could hope for would be to try out the MSI/MSI-X support in the > xen-unstable tree (soon to be Xen 3.3). These are message-based interrupts > which completely sidesteps the issue of interrupt lines and their wiring > together.Sounds good, but I gather it''s only for PCIe or PCI2.3 devices, so my present tuners won''t directly benefit, I suppose they might benefit indirectly by other devices using MSI and freeing up hardwired interrupts rather than being shared. Longer term I''ll probably get a PCIe dual tuner. Thanks for you help today, it''ll likely be the weekend before I can look into this any further. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andy Burns
2008-Jul-02 09:35 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 01/07/2008 17:42, Andy Burns wrote:> I tried adding irqpoll to the relevent stanza in /etc/grub.conf but the > error still occurs.DOH! I had added irqpoll to the domU when it should have been dom0. Now I have irqpoll in the dom0, tuner interrupts are being sent to the domU, and more importantly no longer getting the "nobody cared" message. # cat /proc/interrupts CPU0 16: 440 Phys-irq saa7130[0] 256: 6187 Dynamic-irq timer0 257: 0 Dynamic-irq resched0 258: 0 Dynamic-irq callfunc0 259: 689 Dynamic-irq xenbus 260: 296 Dynamic-irq xencons 261: 6624 Dynamic-irq blkif 262: 32 Dynamic-irq blkif 263: 24 Dynamic-irq blkif 264: 713 Dynamic-irq eth0 Thinking about the IRQ sharing, the SATA controller was giving about 6000 interrupts/second yesterday during expansion of a raid array, that is obviously quite exceptional, but sending so many unwanted interrupts via pciback/front is not a good thing, I''ll have to do some tests later to see if it''s a disastrously bad thing, for now I''ve moved the PCI card into the slot with less shared interrupts. dvbscan still won''t talk to the device but femon does, my second tuner is currently in a different machine, so I can compare and contrast. # femon -h FE: Philips TDA10046H DVB-T (DVBT) status | signal 0% | snr 100% | ber 0001fffe | unc 00000000 | I connected the aerial to it, and despite what dvbscan/femon said, I thought I''d try a channel scan with mythtv anyway and it found one mux and has started populating the EPG, it should find several muxes (it''s fed from the same amplifier as my other machine/tv) but I''ll class that as "initially working" :-) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andy Burns
2008-Jul-02 12:54 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 02/07/2008 10:35, Andy Burns wrote:> I connected the aerial to it, it found one mux > and has started populating the EPG, it should find several muxes (it''s > fed from the same amplifier as my other machine/tv)Turns out the aerial outlet in the study had been disconnected and it was only picking up stray signals, a quick trip into the attic and a bit of re-wiring and myth is firing on all cylinders under Xen. All muxes tuned OK, and recording 3 concurrent programs from the single tuner (which results in about 2Mbytes of data being read from the tuner and written to disk) eats less than 10% CPU in domU and 0.5% in dom0, about 130 interrupts/sec from the tuner. Fantastic! Thanks again for your help, looking forward to Xen3.3 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andy Burns
2008-Jul-02 14:09 UTC
Re: [Xen-devel] MMIO ioremap() error with PCI passthrough
On 02/07/2008 13:54, Andy Burns wrote:> All muxes tuned OK, and recording 3 concurrent programs from the single > tuner (which results in about 2Mbytes of data being read from the tuner > and written to disk) eats less than 10% CPU in domU and 0.5% in dom0, > about 130 interrupts/sec from the tuner.After about an hour I got a kernel panic on the domU Fatal DMA error! Please use ''swiotlb=force'' ----------- [cut here ] --------- [please bite here ] --------- Kernel BUG at arch/x86_64/kernel/../../i386/kernel/pci-dma-xen.c:165 invalid opcode: 0000 [1] SMP last sysfs file: /block/dm-1/range CPU 0 Modules linked in: xfs(U) saa7134_dvb(U) dvb_pll(U) mt352(U) video_buf_dvb(U) dvb_core(U) nxt200x(U) tda1004x(U) autofs4(U) sunrpc(U) xennet(U) ip6t_REJECT(U) xt_tcpudp(U) ip6table_filter(U) ip6_tables(U) x_tables(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) dm_multipath(U) parport_pc(U) lp(U) parport(U) saa7134(U) video_buf(U) compat_ioctl32(U) ir_kbd_i2c(U) i2c_core(U) ir_common(U) videodev(U) pcspkr(U) v4l1_compat(U) v4l2_common(U) dm_snapshot(U) dm_zero(U) dm_mirror(U) dm_mod(U) xenblk(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) Pid: 10678, comm: saa7130[0] dvb Tainted: G 2.6.18-prep #6 RIP: e030:[<ffffffff802720a2>] [<ffffffff802720a2>] dma_map_sg+0x13f/0x1ae RSP: e02b:ffff88002ea6be00 EFLAGS: 00010282 RAX: 000000000000002f RBX: ffff8800305c7df0 RCX: ffffffff804da728 RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000 RBP: 0000000000000003 R08: ffffffff804da728 R09: 0000000000001f5f R10: 000000000000002c R11: ffff88002ea6c000 R12: 0000000000000006 R13: ffff88003fded070 R14: ffff88003df7ede8 R15: 0000000000000003 FS: 00002b787a63a560(0000) GS:ffffffff805ac000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 Process saa7130[0] dvb (pid: 10678, threadinfo ffff88002ea6a000, task ffff880035e32860) Stack: ffff880035e6a0f0 ffff880035e6a0f0 ffff88003df7ede8 ffff88003fded000 0000000000000080 ffffffff88133937 ffff880035e6a0c0 0000000000005e00 ffff88003df7e000 ffffffff88144cf0 Call Trace: [<ffffffff88133937>] :video_buf:videobuf_dma_map+0x115/0x159 [<ffffffff88144cf0>] :saa7134:buffer_prepare+0xbb/0x19b [<ffffffff80298a84>] keventd_create_kthread+0x0/0xc4 [<ffffffff88132d43>] :video_buf:videobuf_read_start+0xa8/0x139 [<ffffffff882f034b>] :video_buf_dvb:videobuf_dvb_thread+0x2a/0x127 [<ffffffff882f0321>] :video_buf_dvb:videobuf_dvb_thread+0x0/0x127 [<ffffffff802339c8>] kthread+0xfe/0x132 [<ffffffff80260b24>] child_rip+0xa/0x12 [<ffffffff80298a84>] keventd_create_kthread+0x0/0xc4 [<ffffffff802338ca>] kthread+0x0/0x132 [<ffffffff80260b1a>] child_rip+0x0/0x12 Code: 0f 0b 68 a5 df 47 80 c2 a5 00 48 8b 7b f0 48 2b 3d 21 f1 40 RIP [<ffffffff802720a2>] dma_map_sg+0x13f/0x1ae RSP <ffff88002ea6be00> <0>Kernel panic - not syncing: Fatal exception I presume this kernel option *is* a domU option rather than dom0? when I added it to grub, the domU crashes on boot ... ----------- [cut here ] --------- [please bite here ] --------- Kernel BUG at lib/../arch/i386/kernel/swiotlb.c:160 invalid opcode: 0000 [1] SMP last sysfs file: CPU 0 Modules linked in: Pid: 0, comm: swapper Not tainted 2.6.18-prep #6 RIP: e030:[<ffffffff8033ca50>] [<ffffffff8033ca50>] swiotlb_init_with_default_size+0xa0/0x19e RSP: e02b:ffffffff805edf40 EFLAGS: 00010282 RAX: 00000000fffffff4 RBX: 0000000000000900 RCX: 000000000021d106 RDX: ffffffffff578000 RSI: 0000000000000000 RDI: 0000000000000000 RBP: 0000000004000000 R08: 0000000000001000 R09: ffffffff8064d9e0 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffffffff805ac000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 Process swapper (pid: 0, threadinfo ffffffff805ec000, task ffffffff804d5a80) Stack: 00000000000002c3 0000000002040800 0000000000000000 ffffffff80276935 0000000000000000 ffffffff8060051b 0000000000000000 0000000002040800 0000000000000000 0000000000000000 Call Trace: [<ffffffff80276935>] pci_swiotlb_init+0x9/0x2d [<ffffffff8060051b>] mem_init+0x5e/0x233 [<ffffffff805f6a73>] start_kernel+0x189/0x224 [<ffffffff805f61e5>] _sinittext+0x1e5/0x1eb Code: 0f 0b 68 26 d3 48 80 c2 a0 00 48 83 eb 80 48 8b 05 23 cf 34 RIP [<ffffffff8033ca50>] swiotlb_init_with_default_size+0xa0/0x19e RSP <ffffffff805edf40> <0>Kernel panic - not syncing: Fatal exception _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel