thr3ads.net - Xen devel - [Xen-devel] [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000. [Mar 2010]

If this information is useful, please help other people find it:
Share via:

Sander Eikelenboom

2010-Mar-21 21:19 UTC

[Xen-devel] [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Hi Han/Konrad,

In my setup i''m trying to passthrough an USB 3.0 pci-e controller  to a
PV domU.
- xen: 4.0.0-rc6
- dom0: kernel xen/next
- domU: kernel 2.6.33 from
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
        ( to have pci-front together with most recent usb3.0 xhci drivers.


- USB 3.0 xhci drivers work fine on the baremetal with the 2.6.33 kernel.

This is on a intel Q45 chipset with IOMMU.

This is my boot config:
title           xen-4.0.0-rc6.gz / Debian GNU/Linux,  kernel 2.6.32
root            (hd0,0)
kernel          /boot/xen-4.0.0-rc6.gz dom0_mem=768M loglvl=all loglvl_guest=all
iommu=pv iommu_inclusive_mapping=1
module          /boot/vmlinuz-2.6.32 root=/dev/sda1 ro earlyprintk=xen
max_loop=255 xen-pciback.hide=(03:00.0)
module          /boot/initrd.img-2.6.32

When booting the domU xm dmesg gets filled with the following when the usb
controller tries to initialize/:

(XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from
00000a07:00000000 to 00000000:00000000.
(XEN) [VT-D]iommu.c:821: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:796: DMAR:[DMA Read] Request device [03:00.0] fault addr
1ff94000, iommu reg = ffff82c3fff54000
(XEN) DMAR:[fault reason 06h] PTE Read access is not set
(XEN) print_vtd_entries: iommu = ffff83007c866970 bdf = 3:0.0 gmfn = 1ff94
(XEN)     root_entry = ffff83007c872000
(XEN)     root_entry[3] = 78f56001
(XEN)     context = ffff830078f56000
(XEN)     context[0] = 101_2f0e1001
(XEN)     l3 = ffff83002f0e1000
(XEN)     l3_index = 0
(XEN)     l3[0] = 2f0e0003
(XEN)     l2 = ffff83002f0e0000
(XEN)     l2_index = ff
(XEN)     l2[ff] = 0
(XEN)     l2[ff] not present



Anyone any tips on what i could try ?, is this something caused by xen, or
something by the usb driver not adhering to kernel DMA-api ?

Attached:

- xm-info.txt
- xm-dmesg.txt
- xend.log

- dom0-dmesg.txt
- dom0-lspci-tree.txt
- dom0-lspci.txt

- domU-lspci.txt
- domU-dmesg.txt













_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Han, Weidong

2010-Mar-22 08:04 UTC

head link

[Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

The faults were caused by that the DMA address was not mapped in VT-d page
table.

Could you have following two tries:
	1) assign it to pv domU without VT-d
	2) assign it to a hvm guest

Regards,
Weidong

-----Original Message-----
From: Sander Eikelenboom [mailto:linux@eikelenboom.it] 
Sent: Monday, March 22, 2010 5:20 AM
To: Konrad Rzeszutek Wilk
Cc: Han, Weidong; xen-devel@lists.xensource.com
Subject: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to
PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from
00000a07:00000000 to 00000000:00000000.

Hi Han/Konrad,

In my setup i''m trying to passthrough an USB 3.0 pci-e controller  to a
PV domU.
- xen: 4.0.0-rc6
- dom0: kernel xen/next
- domU: kernel 2.6.33 from
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
        ( to have pci-front together with most recent usb3.0 xhci drivers.


- USB 3.0 xhci drivers work fine on the baremetal with the 2.6.33 kernel.

This is on a intel Q45 chipset with IOMMU.

This is my boot config:
title           xen-4.0.0-rc6.gz / Debian GNU/Linux,  kernel 2.6.32
root            (hd0,0)
kernel          /boot/xen-4.0.0-rc6.gz dom0_mem=768M loglvl=all loglvl_guest=all
iommu=pv iommu_inclusive_mapping=1
module          /boot/vmlinuz-2.6.32 root=/dev/sda1 ro earlyprintk=xen
max_loop=255 xen-pciback.hide=(03:00.0)
module          /boot/initrd.img-2.6.32

When booting the domU xm dmesg gets filled with the following when the usb
controller tries to initialize/:

(XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from
00000a07:00000000 to 00000000:00000000.
(XEN) [VT-D]iommu.c:821: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:796: DMAR:[DMA Read] Request device [03:00.0] fault addr
1ff94000, iommu reg = ffff82c3fff54000
(XEN) DMAR:[fault reason 06h] PTE Read access is not set
(XEN) print_vtd_entries: iommu = ffff83007c866970 bdf = 3:0.0 gmfn = 1ff94
(XEN)     root_entry = ffff83007c872000
(XEN)     root_entry[3] = 78f56001
(XEN)     context = ffff830078f56000
(XEN)     context[0] = 101_2f0e1001
(XEN)     l3 = ffff83002f0e1000
(XEN)     l3_index = 0
(XEN)     l3[0] = 2f0e0003
(XEN)     l2 = ffff83002f0e0000
(XEN)     l2_index = ff
(XEN)     l2[ff] = 0
(XEN)     l2[ff] not present



Anyone any tips on what i could try ?, is this something caused by xen, or
something by the usb driver not adhering to kernel DMA-api ?

Attached:

- xm-info.txt
- xm-dmesg.txt
- xend.log

- dom0-dmesg.txt
- dom0-lspci-tree.txt
- dom0-lspci.txt

- domU-lspci.txt
- domU-dmesg.txt





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pasi Kärkkäinen

2010-Mar-22 09:04 UTC

head link

Re: [Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

On Mon, Mar 22, 2010 at 04:04:09PM +0800, Han, Weidong
wrote:> The faults were caused by that the DMA address was not mapped in VT-d page
table.
> 
> Could you have following two tries:
> 	1) assign it to pv domU without VT-d
Btw are there other methods of disabling VT-d PCI passhtru than having iommu=off
for xen.gz in grub.conf?
ie. can you somehow select "use normal PV passthru for this guest" and
"but still use VT-d passthru for this guest" ?

ps. I''m just writing a xen wiki page for PCI passthru and adding these
things and the common failure scenarios there.

-- Pasi
> 	2) assign it to a hvm guest
> 
> Regards,
> Weidong
> 
> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it] 
> Sent: Monday, March 22, 2010 5:20 AM
> To: Konrad Rzeszutek Wilk
> Cc: Han, Weidong; xen-devel@lists.xensource.com
> Subject: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb
controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
> 
> Hi Han/Konrad,
> 
> In my setup i''m trying to passthrough an USB 3.0 pci-e controller 
to a PV domU.
> - xen: 4.0.0-rc6
> - dom0: kernel xen/next
> - domU: kernel 2.6.33 from
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
>         ( to have pci-front together with most recent usb3.0 xhci drivers.
> 
> 
> - USB 3.0 xhci drivers work fine on the baremetal with the 2.6.33 kernel.
> 
> This is on a intel Q45 chipset with IOMMU.
> 
> This is my boot config:
> title           xen-4.0.0-rc6.gz / Debian GNU/Linux,  kernel 2.6.32
> root            (hd0,0)
> kernel          /boot/xen-4.0.0-rc6.gz dom0_mem=768M loglvl=all
loglvl_guest=all  iommu=pv iommu_inclusive_mapping=1
> module          /boot/vmlinuz-2.6.32 root=/dev/sda1 ro earlyprintk=xen
max_loop=255 xen-pciback.hide=(03:00.0)
> module          /boot/initrd.img-2.6.32
> 
> When booting the domU xm dmesg gets filled with the following when the usb
controller tries to initialize/:
> 
> (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from
00000a07:00000000 to 00000000:00000000.
> (XEN) [VT-D]iommu.c:821: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:796: DMAR:[DMA Read] Request device [03:00.0] fault
addr 1ff94000, iommu reg = ffff82c3fff54000
> (XEN) DMAR:[fault reason 06h] PTE Read access is not set
> (XEN) print_vtd_entries: iommu = ffff83007c866970 bdf = 3:0.0 gmfn = 1ff94
> (XEN)     root_entry = ffff83007c872000
> (XEN)     root_entry[3] = 78f56001
> (XEN)     context = ffff830078f56000
> (XEN)     context[0] = 101_2f0e1001
> (XEN)     l3 = ffff83002f0e1000
> (XEN)     l3_index = 0
> (XEN)     l3[0] = 2f0e0003
> (XEN)     l2 = ffff83002f0e0000
> (XEN)     l2_index = ff
> (XEN)     l2[ff] = 0
> (XEN)     l2[ff] not present
> 
> 
> 
> Anyone any tips on what i could try ?, is this something caused by xen, or
something by the usb driver not adhering to kernel DMA-api ?
> 
> Attached:
> 
> - xm-info.txt
> - xm-dmesg.txt
> - xend.log
> 
> - dom0-dmesg.txt
> - dom0-lspci-tree.txt
> - dom0-lspci.txt
> 
> - domU-lspci.txt
> - domU-dmesg.txt
> 
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Weidong Han

2010-Mar-22 09:13 UTC

head link

Re: [Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Pasi Kärkkäinen wrote:> On Mon, Mar 22, 2010 at 04:04:09PM +0800, Han, Weidong wrote:
>   
>> The faults were caused by that the DMA address was not mapped in VT-d
page table.
>>
>> Could you have following two tries:
>> 	1) assign it to pv domU without VT-d
>>     
>
> Btw are there other methods of disabling VT-d PCI passhtru than having
iommu=off for xen.gz in grub.conf?
> ie. can you somehow select "use normal PV passthru for this
guest" and "but still use VT-d passthru for this guest" ?
>
>   I''m not quite understand your point. do you mean use normal PV passthru
for pv guest, but still can passthru device to hvm guest? if so, current 
xen VT-d already does like this.

Regards,
Weidong> ps. I''m just writing a xen wiki page for PCI passthru and adding
these things and the common failure scenarios there.
>
> -- Pasi
>
>   
>> 	2) assign it to a hvm guest
>>
>> Regards,
>> Weidong
>>
>> -----Original Message-----
>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it] 
>> Sent: Monday, March 22, 2010 5:20 AM
>> To: Konrad Rzeszutek Wilk
>> Cc: Han, Weidong; xen-devel@lists.xensource.com
>> Subject: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb
controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
>>
>> Hi Han/Konrad,
>>
>> In my setup i''m trying to passthrough an USB 3.0 pci-e
controller  to a PV domU.
>> - xen: 4.0.0-rc6
>> - dom0: kernel xen/next
>> - domU: kernel 2.6.33 from
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
>>         ( to have pci-front together with most recent usb3.0 xhci
drivers.
>>
>>
>> - USB 3.0 xhci drivers work fine on the baremetal with the 2.6.33
kernel.
>>
>> This is on a intel Q45 chipset with IOMMU.
>>
>> This is my boot config:
>> title           xen-4.0.0-rc6.gz / Debian GNU/Linux,  kernel 2.6.32
>> root            (hd0,0)
>> kernel          /boot/xen-4.0.0-rc6.gz dom0_mem=768M loglvl=all
loglvl_guest=all  iommu=pv iommu_inclusive_mapping=1
>> module          /boot/vmlinuz-2.6.32 root=/dev/sda1 ro earlyprintk=xen
max_loop=255 xen-pciback.hide=(03:00.0)
>> module          /boot/initrd.img-2.6.32
>>
>> When booting the domU xm dmesg gets filled with the following when the
usb controller tries to initialize/:
>>
>> (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from
00000a07:00000000 to 00000000:00000000.
>> (XEN) [VT-D]iommu.c:821: iommu_fault_status: Primary Pending Fault
>> (XEN) [VT-D]iommu.c:796: DMAR:[DMA Read] Request device [03:00.0] fault
addr 1ff94000, iommu reg = ffff82c3fff54000
>> (XEN) DMAR:[fault reason 06h] PTE Read access is not set
>> (XEN) print_vtd_entries: iommu = ffff83007c866970 bdf = 3:0.0 gmfn =
1ff94
>> (XEN)     root_entry = ffff83007c872000
>> (XEN)     root_entry[3] = 78f56001
>> (XEN)     context = ffff830078f56000
>> (XEN)     context[0] = 101_2f0e1001
>> (XEN)     l3 = ffff83002f0e1000
>> (XEN)     l3_index = 0
>> (XEN)     l3[0] = 2f0e0003
>> (XEN)     l2 = ffff83002f0e0000
>> (XEN)     l2_index = ff
>> (XEN)     l2[ff] = 0
>> (XEN)     l2[ff] not present
>>
>>
>>
>> Anyone any tips on what i could try ?, is this something caused by xen,
or something by the usb driver not adhering to kernel DMA-api ?
>>
>> Attached:
>>
>> - xm-info.txt
>> - xm-dmesg.txt
>> - xend.log
>>
>> - dom0-dmesg.txt
>> - dom0-lspci-tree.txt
>> - dom0-lspci.txt
>>
>> - domU-lspci.txt
>> - domU-dmesg.txt
>>
>>
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>     

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pasi Kärkkäinen

2010-Mar-22 09:19 UTC

head link

Re: [Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

On Mon, Mar 22, 2010 at 05:13:33PM +0800, Weidong Han
wrote:> Pasi Kärkkäinen wrote:
>> On Mon, Mar 22, 2010 at 04:04:09PM +0800, Han, Weidong wrote:
>>   
>>> The faults were caused by that the DMA address was not mapped in
VT-d page table.
>>>
>>> Could you have following two tries:
>>> 	1) assign it to pv domU without VT-d
>>>     
>>
>> Btw are there other methods of disabling VT-d PCI passhtru than having 
>> iommu=off for xen.gz in grub.conf? ie. can you somehow select "use
>> normal PV passthru for this guest" and "but still use VT-d
passthru for
>> this guest" ?
>>
>>   
> I''m not quite understand your point. do you mean use normal PV
passthru
> for pv guest, but still can passthru device to hvm guest? if so, current  
> xen VT-d already does like this.
>
Yeah, that''s what I meant. Ok. 

I thought it was also possible to use VT-d for a PV guest? no? 

-- Pasi
> Regards,
> Weidong
>> ps. I''m just writing a xen wiki page for PCI passthru and
adding these things and the common failure scenarios there.
>>
>> -- Pasi
>>
>>   
>>> 	2) assign it to a hvm guest
>>>
>>> Regards,
>>> Weidong
>>>
>>> -----Original Message-----
>>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it] Sent:
Monday,
>>> March 22, 2010 5:20 AM
>>> To: Konrad Rzeszutek Wilk
>>> Cc: Han, Weidong; xen-devel@lists.xensource.com
>>> Subject: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb
controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
>>>
>>> Hi Han/Konrad,
>>>
>>> In my setup i''m trying to passthrough an USB 3.0 pci-e
controller  to a PV domU.
>>> - xen: 4.0.0-rc6
>>> - dom0: kernel xen/next
>>> - domU: kernel 2.6.33 from
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
>>>         ( to have pci-front together with most recent usb3.0 xhci
drivers.
>>>
>>>
>>> - USB 3.0 xhci drivers work fine on the baremetal with the 2.6.33
kernel.
>>>
>>> This is on a intel Q45 chipset with IOMMU.
>>>
>>> This is my boot config:
>>> title           xen-4.0.0-rc6.gz / Debian GNU/Linux,  kernel 2.6.32
>>> root            (hd0,0)
>>> kernel          /boot/xen-4.0.0-rc6.gz dom0_mem=768M loglvl=all
loglvl_guest=all  iommu=pv iommu_inclusive_mapping=1
>>> module          /boot/vmlinuz-2.6.32 root=/dev/sda1 ro
earlyprintk=xen max_loop=255 xen-pciback.hide=(03:00.0)
>>> module          /boot/initrd.img-2.6.32
>>>
>>> When booting the domU xm dmesg gets filled with the following when
the usb controller tries to initialize/:
>>>
>>> (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from
00000a07:00000000 to 00000000:00000000.
>>> (XEN) [VT-D]iommu.c:821: iommu_fault_status: Primary Pending Fault
>>> (XEN) [VT-D]iommu.c:796: DMAR:[DMA Read] Request device [03:00.0]
fault addr 1ff94000, iommu reg = ffff82c3fff54000
>>> (XEN) DMAR:[fault reason 06h] PTE Read access is not set
>>> (XEN) print_vtd_entries: iommu = ffff83007c866970 bdf = 3:0.0 gmfn
= 1ff94
>>> (XEN)     root_entry = ffff83007c872000
>>> (XEN)     root_entry[3] = 78f56001
>>> (XEN)     context = ffff830078f56000
>>> (XEN)     context[0] = 101_2f0e1001
>>> (XEN)     l3 = ffff83002f0e1000
>>> (XEN)     l3_index = 0
>>> (XEN)     l3[0] = 2f0e0003
>>> (XEN)     l2 = ffff83002f0e0000
>>> (XEN)     l2_index = ff
>>> (XEN)     l2[ff] = 0
>>> (XEN)     l2[ff] not present
>>>
>>>
>>>
>>> Anyone any tips on what i could try ?, is this something caused by
xen, or something by the usb driver not adhering to kernel DMA-api ?
>>>
>>> Attached:
>>>
>>> - xm-info.txt
>>> - xm-dmesg.txt
>>> - xend.log
>>>
>>> - dom0-dmesg.txt
>>> - dom0-lspci-tree.txt
>>> - dom0-lspci.txt
>>>
>>> - domU-lspci.txt
>>> - domU-dmesg.txt
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>     
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Christian Tramnitz

2010-Mar-22 09:22 UTC

head link

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

At 22.03.2010 10:19, Pasi Kärkkäinen wrote:>
> Yeah, that''s what I meant. Ok.
>
> I thought it was also possible to use VT-d for a PV guest? no?
>
> -- Pasi
It is, but this requires iommu=pv as hypervisor boot flag.
With iommu=1 you can see in the hypervisor log that VTd will be disabled 
for PV.


Best regards,
    Christian


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Weidong Han

2010-Mar-22 09:25 UTC

head link

Re: [Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Pasi Kärkkäinen wrote:> On Mon, Mar 22, 2010 at 05:13:33PM +0800, Weidong Han wrote:
>   
>> Pasi Kärkkäinen wrote:
>>     
>>> On Mon, Mar 22, 2010 at 04:04:09PM +0800, Han, Weidong wrote:
>>>   
>>>       
>>>> The faults were caused by that the DMA address was not mapped
in VT-d page table.
>>>>
>>>> Could you have following two tries:
>>>> 	1) assign it to pv domU without VT-d
>>>>     
>>>>         
>>> Btw are there other methods of disabling VT-d PCI passhtru than
having
>>> iommu=off for xen.gz in grub.conf? ie. can you somehow select
"use
>>> normal PV passthru for this guest" and "but still use
VT-d passthru for
>>> this guest" ?
>>>
>>>   
>>>       
>> I''m not quite understand your point. do you mean use normal PV
passthru
>> for pv guest, but still can passthru device to hvm guest? if so,
current
>> xen VT-d already does like this.
>>
>>     
>
> Yeah, that''s what I meant. Ok. 
>
> I thought it was also possible to use VT-d for a PV guest? no? 
>   yes. you can use VT-d to assign device to pv guest. Sander was doing 
that and encountered below issue.

Regards,
Weidong> -- Pasi
>
>   
>> Regards,
>> Weidong
>>     
>>> ps. I''m just writing a xen wiki page for PCI passthru and
adding these things and the common failure scenarios there.
>>>
>>> -- Pasi
>>>
>>>   
>>>       
>>>> 	2) assign it to a hvm guest
>>>>
>>>> Regards,
>>>> Weidong
>>>>
>>>> -----Original Message-----
>>>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it] Sent:
Monday,
>>>> March 22, 2010 5:20 AM
>>>> To: Konrad Rzeszutek Wilk
>>>> Cc: Han, Weidong; xen-devel@lists.xensource.com
>>>> Subject: [pvops xen/next ][iommu] attenpt to passthrough PCI-e
usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
>>>>
>>>> Hi Han/Konrad,
>>>>
>>>> In my setup i''m trying to passthrough an USB 3.0 pci-e
controller  to a PV domU.
>>>> - xen: 4.0.0-rc6
>>>> - dom0: kernel xen/next
>>>> - domU: kernel 2.6.33 from
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
>>>>         ( to have pci-front together with most recent usb3.0
xhci drivers.
>>>>
>>>>
>>>> - USB 3.0 xhci drivers work fine on the baremetal with the
2.6.33 kernel.
>>>>
>>>> This is on a intel Q45 chipset with IOMMU.
>>>>
>>>> This is my boot config:
>>>> title           xen-4.0.0-rc6.gz / Debian GNU/Linux,  kernel
2.6.32
>>>> root            (hd0,0)
>>>> kernel          /boot/xen-4.0.0-rc6.gz dom0_mem=768M loglvl=all
loglvl_guest=all  iommu=pv iommu_inclusive_mapping=1
>>>> module          /boot/vmlinuz-2.6.32 root=/dev/sda1 ro
earlyprintk=xen max_loop=255 xen-pciback.hide=(03:00.0)
>>>> module          /boot/initrd.img-2.6.32
>>>>
>>>> When booting the domU xm dmesg gets filled with the following
when the usb controller tries to initialize/:
>>>>
>>>> (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b
from 00000a07:00000000 to 00000000:00000000.
>>>> (XEN) [VT-D]iommu.c:821: iommu_fault_status: Primary Pending
Fault
>>>> (XEN) [VT-D]iommu.c:796: DMAR:[DMA Read] Request device
[03:00.0] fault addr 1ff94000, iommu reg = ffff82c3fff54000
>>>> (XEN) DMAR:[fault reason 06h] PTE Read access is not set
>>>> (XEN) print_vtd_entries: iommu = ffff83007c866970 bdf = 3:0.0
gmfn = 1ff94
>>>> (XEN)     root_entry = ffff83007c872000
>>>> (XEN)     root_entry[3] = 78f56001
>>>> (XEN)     context = ffff830078f56000
>>>> (XEN)     context[0] = 101_2f0e1001
>>>> (XEN)     l3 = ffff83002f0e1000
>>>> (XEN)     l3_index = 0
>>>> (XEN)     l3[0] = 2f0e0003
>>>> (XEN)     l2 = ffff83002f0e0000
>>>> (XEN)     l2_index = ff
>>>> (XEN)     l2[ff] = 0
>>>> (XEN)     l2[ff] not present
>>>>
>>>>
>>>>
>>>> Anyone any tips on what i could try ?, is this something caused
by xen, or something by the usb driver not adhering to kernel DMA-api ?
>>>>
>>>> Attached:
>>>>
>>>> - xm-info.txt
>>>> - xm-dmesg.txt
>>>> - xend.log
>>>>
>>>> - dom0-dmesg.txt
>>>> - dom0-lspci-tree.txt
>>>> - dom0-lspci.txt
>>>>
>>>> - domU-lspci.txt
>>>> - domU-dmesg.txt
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>>>     
>>>>         

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pasi Kärkkäinen

2010-Mar-22 09:29 UTC

head link

Re: [Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

On Mon, Mar 22, 2010 at 05:25:55PM +0800, Weidong Han
wrote:> Pasi Kärkkäinen wrote:
>> On Mon, Mar 22, 2010 at 05:13:33PM +0800, Weidong Han wrote:
>>   
>>> Pasi Kärkkäinen wrote:
>>>     
>>>> On Mon, Mar 22, 2010 at 04:04:09PM +0800, Han, Weidong wrote:
>>>>         
>>>>> The faults were caused by that the DMA address was not
mapped in VT-d page table.
>>>>>
>>>>> Could you have following two tries:
>>>>> 	1) assign it to pv domU without VT-d
>>>>>             
>>>> Btw are there other methods of disabling VT-d PCI passhtru than
>>>> having iommu=off for xen.gz in grub.conf? ie. can you somehow 
>>>> select "use normal PV passthru for this guest" and
"but still use
>>>> VT-d passthru for this guest" ?
>>>>
>>>>         
>>> I''m not quite understand your point. do you mean use
normal PV
>>> passthru  for pv guest, but still can passthru device to hvm guest?
>>> if so, current  xen VT-d already does like this.
>>>
>>>     
>>
>> Yeah, that''s what I meant. Ok. 
>>
>> I thought it was also possible to use VT-d for a PV guest? no?   
> yes. you can use VT-d to assign device to pv guest. Sander was doing  
> that and encountered below issue.
>
Ok, so iommu=pv is the flag that enabled VT-d passthru for PV guests. 
As a default Xen uses normal PCI passhthru for PV guests.

Thanks.

-- Pasi
> Regards,
> Weidong
>> -- Pasi
>>
>>   
>>> Regards,
>>> Weidong
>>>     
>>>> ps. I''m just writing a xen wiki page for PCI passthru
and adding these things and the common failure scenarios there.
>>>>
>>>> -- Pasi
>>>>
>>>>         
>>>>> 	2) assign it to a hvm guest
>>>>>
>>>>> Regards,
>>>>> Weidong
>>>>>
>>>>> -----Original Message-----
>>>>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
Sent:
>>>>> Monday, March 22, 2010 5:20 AM
>>>>> To: Konrad Rzeszutek Wilk
>>>>> Cc: Han, Weidong; xen-devel@lists.xensource.com
>>>>> Subject: [pvops xen/next ][iommu] attenpt to passthrough
PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
>>>>>
>>>>> Hi Han/Konrad,
>>>>>
>>>>> In my setup i''m trying to passthrough an USB 3.0
pci-e controller  to a PV domU.
>>>>> - xen: 4.0.0-rc6
>>>>> - dom0: kernel xen/next
>>>>> - domU: kernel 2.6.33 from
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
>>>>>         ( to have pci-front together with most recent
usb3.0 xhci drivers.
>>>>>
>>>>>
>>>>> - USB 3.0 xhci drivers work fine on the baremetal with the
2.6.33 kernel.
>>>>>
>>>>> This is on a intel Q45 chipset with IOMMU.
>>>>>
>>>>> This is my boot config:
>>>>> title           xen-4.0.0-rc6.gz / Debian GNU/Linux, 
kernel 2.6.32
>>>>> root            (hd0,0)
>>>>> kernel          /boot/xen-4.0.0-rc6.gz dom0_mem=768M
loglvl=all loglvl_guest=all  iommu=pv iommu_inclusive_mapping=1
>>>>> module          /boot/vmlinuz-2.6.32 root=/dev/sda1 ro
earlyprintk=xen max_loop=255 xen-pciback.hide=(03:00.0)
>>>>> module          /boot/initrd.img-2.6.32
>>>>>
>>>>> When booting the domU xm dmesg gets filled with the
following when the usb controller tries to initialize/:
>>>>>
>>>>> (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
>>>>> (XEN) [VT-D]iommu.c:821: iommu_fault_status: Primary
Pending Fault
>>>>> (XEN) [VT-D]iommu.c:796: DMAR:[DMA Read] Request device
[03:00.0] fault addr 1ff94000, iommu reg = ffff82c3fff54000
>>>>> (XEN) DMAR:[fault reason 06h] PTE Read access is not set
>>>>> (XEN) print_vtd_entries: iommu = ffff83007c866970 bdf =
3:0.0 gmfn = 1ff94
>>>>> (XEN)     root_entry = ffff83007c872000
>>>>> (XEN)     root_entry[3] = 78f56001
>>>>> (XEN)     context = ffff830078f56000
>>>>> (XEN)     context[0] = 101_2f0e1001
>>>>> (XEN)     l3 = ffff83002f0e1000
>>>>> (XEN)     l3_index = 0
>>>>> (XEN)     l3[0] = 2f0e0003
>>>>> (XEN)     l2 = ffff83002f0e0000
>>>>> (XEN)     l2_index = ff
>>>>> (XEN)     l2[ff] = 0
>>>>> (XEN)     l2[ff] not present
>>>>>
>>>>>
>>>>>
>>>>> Anyone any tips on what i could try ?, is this something
caused by xen, or something by the usb driver not adhering to kernel DMA-api ?
>>>>>
>>>>> Attached:
>>>>>
>>>>> - xm-info.txt
>>>>> - xm-dmesg.txt
>>>>> - xend.log
>>>>>
>>>>> - dom0-dmesg.txt
>>>>> - dom0-lspci-tree.txt
>>>>> - dom0-lspci.txt
>>>>>
>>>>> - domU-lspci.txt
>>>>> - domU-dmesg.txt
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Xen-devel mailing list
>>>>> Xen-devel@lists.xensource.com
>>>>> http://lists.xensource.com/xen-devel
>>>>>             
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Sander Eikelenboom

2010-Mar-22 09:30 UTC

head link

Re: [Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Hi Pasi/Weidong

I was trying that with the "iommu=pv" option, without that vt-d is not
used for passthrough to pv guests.

I will try HVM guest and without VT-D later today.

--
Sander


Monday, March 22, 2010, 10:25:55 AM, you wrote:
> Pasi Kärkkäinen wrote:
>> On Mon, Mar 22, 2010 at 05:13:33PM +0800, Weidong Han wrote:
>>   
>>> Pasi Kärkkäinen wrote:
>>>     
>>>> On Mon, Mar 22, 2010 at 04:04:09PM +0800, Han, Weidong wrote:
>>>>   
>>>>       
>>>>> The faults were caused by that the DMA address was not
mapped in VT-d page table.
>>>>>
>>>>> Could you have following two tries:
>>>>>    1) assign it to pv domU without VT-d
>>>>>     
>>>>>         
>>>> Btw are there other methods of disabling VT-d PCI passhtru than
having
>>>> iommu=off for xen.gz in grub.conf? ie. can you somehow select
"use
>>>> normal PV passthru for this guest" and "but still use
VT-d passthru for
>>>> this guest" ?
>>>>
>>>>   
>>>>       
>>> I''m not quite understand your point. do you mean use
normal PV passthru
>>> for pv guest, but still can passthru device to hvm guest? if so,
current
>>> xen VT-d already does like this.
>>>
>>>     
>>
>> Yeah, that''s what I meant. Ok. 
>>
>> I thought it was also possible to use VT-d for a PV guest? no? 
>>   
> yes. you can use VT-d to assign device to pv guest. Sander was doing 
> that and encountered below issue.
> Regards,
> Weidong
>> -- Pasi
>>
>>   
>>> Regards,
>>> Weidong
>>>     
>>>> ps. I''m just writing a xen wiki page for PCI passthru
and adding these things and the common failure scenarios there.
>>>>
>>>> -- Pasi
>>>>
>>>>   
>>>>       
>>>>>    2) assign it to a hvm guest
>>>>>
>>>>> Regards,
>>>>> Weidong
>>>>>
>>>>> -----Original Message-----
>>>>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
Sent: Monday,
>>>>> March 22, 2010 5:20 AM
>>>>> To: Konrad Rzeszutek Wilk
>>>>> Cc: Han, Weidong; xen-devel@lists.xensource.com
>>>>> Subject: [pvops xen/next ][iommu] attenpt to passthrough
PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
>>>>>
>>>>> Hi Han/Konrad,
>>>>>
>>>>> In my setup i''m trying to passthrough an USB 3.0
pci-e controller  to a PV domU.
>>>>> - xen: 4.0.0-rc6
>>>>> - dom0: kernel xen/next
>>>>> - domU: kernel 2.6.33 from
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
>>>>>         ( to have pci-front together with most recent
usb3.0 xhci drivers.
>>>>>
>>>>>
>>>>> - USB 3.0 xhci drivers work fine on the baremetal with the
2.6.33 kernel.
>>>>>
>>>>> This is on a intel Q45 chipset with IOMMU.
>>>>>
>>>>> This is my boot config:
>>>>> title           xen-4.0.0-rc6.gz / Debian GNU/Linux, 
kernel 2.6.32
>>>>> root            (hd0,0)
>>>>> kernel          /boot/xen-4.0.0-rc6.gz dom0_mem=768M
loglvl=all loglvl_guest=all  iommu=pv iommu_inclusive_mapping=1
>>>>> module          /boot/vmlinuz-2.6.32 root=/dev/sda1 ro
earlyprintk=xen max_loop=255 xen-pciback.hide=(03:00.0)
>>>>> module          /boot/initrd.img-2.6.32
>>>>>
>>>>> When booting the domU xm dmesg gets filled with the
following when the usb controller tries to initialize/:
>>>>>
>>>>> (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
>>>>> (XEN) [VT-D]iommu.c:821: iommu_fault_status: Primary
Pending Fault
>>>>> (XEN) [VT-D]iommu.c:796: DMAR:[DMA Read] Request device
[03:00.0] fault addr 1ff94000, iommu reg = ffff82c3fff54000
>>>>> (XEN) DMAR:[fault reason 06h] PTE Read access is not set
>>>>> (XEN) print_vtd_entries: iommu = ffff83007c866970 bdf =
3:0.0 gmfn = 1ff94
>>>>> (XEN)     root_entry = ffff83007c872000
>>>>> (XEN)     root_entry[3] = 78f56001
>>>>> (XEN)     context = ffff830078f56000
>>>>> (XEN)     context[0] = 101_2f0e1001
>>>>> (XEN)     l3 = ffff83002f0e1000
>>>>> (XEN)     l3_index = 0
>>>>> (XEN)     l3[0] = 2f0e0003
>>>>> (XEN)     l2 = ffff83002f0e0000
>>>>> (XEN)     l2_index = ff
>>>>> (XEN)     l2[ff] = 0
>>>>> (XEN)     l2[ff] not present
>>>>>
>>>>>
>>>>>
>>>>> Anyone any tips on what i could try ?, is this something
caused by xen, or something by the usb driver not adhering to kernel DMA-api ?
>>>>>
>>>>> Attached:
>>>>>
>>>>> - xm-info.txt
>>>>> - xm-dmesg.txt
>>>>> - xend.log
>>>>>
>>>>> - dom0-dmesg.txt
>>>>> - dom0-lspci-tree.txt
>>>>> - dom0-lspci.txt
>>>>>
>>>>> - domU-lspci.txt
>>>>> - domU-dmesg.txt
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Xen-devel mailing list
>>>>> Xen-devel@lists.xensource.com
>>>>> http://lists.xensource.com/xen-devel
>>>>>     
>>>>>         



-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Weidong Han

2010-Mar-22 09:32 UTC

head link

Re: [Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Pasi Kärkkäinen wrote:> On Mon, Mar 22, 2010 at 05:25:55PM +0800, Weidong Han wrote:
>   
>> Pasi Kärkkäinen wrote:
>>     
>>> On Mon, Mar 22, 2010 at 05:13:33PM +0800, Weidong Han wrote:
>>>   
>>>       
>>>> Pasi Kärkkäinen wrote:
>>>>     
>>>>         
>>>>> On Mon, Mar 22, 2010 at 04:04:09PM +0800, Han, Weidong
wrote:
>>>>>         
>>>>>           
>>>>>> The faults were caused by that the DMA address was not
mapped in VT-d page table.
>>>>>>
>>>>>> Could you have following two tries:
>>>>>> 	1) assign it to pv domU without VT-d
>>>>>>             
>>>>>>             
>>>>> Btw are there other methods of disabling VT-d PCI passhtru
than
>>>>> having iommu=off for xen.gz in grub.conf? ie. can you
somehow
>>>>> select "use normal PV passthru for this guest"
and "but still use
>>>>> VT-d passthru for this guest" ?
>>>>>
>>>>>         
>>>>>           
>>>> I''m not quite understand your point. do you mean use
normal PV
>>>> passthru  for pv guest, but still can passthru device to hvm
guest?
>>>> if so, current  xen VT-d already does like this.
>>>>
>>>>     
>>>>         
>>> Yeah, that''s what I meant. Ok. 
>>>
>>> I thought it was also possible to use VT-d for a PV guest? no?   
>>>       
>> yes. you can use VT-d to assign device to pv guest. Sander was doing  
>> that and encountered below issue.
>>
>>     
>
> Ok, so iommu=pv is the flag that enabled VT-d passthru for PV guests. 
> As a default Xen uses normal PCI passhthru for PV guests.
>
> Thanks.
>
> -- Pasi
>
>   exactly.

Regards,
Weidong>> Regards,
>> Weidong
>>     
>>> -- Pasi
>>>
>>>   
>>>       
>>>> Regards,
>>>> Weidong
>>>>     
>>>>         
>>>>> ps. I''m just writing a xen wiki page for PCI
passthru and adding these things and the common failure scenarios there.
>>>>>
>>>>> -- Pasi
>>>>>
>>>>>         
>>>>>           
>>>>>> 	2) assign it to a hvm guest
>>>>>>
>>>>>> Regards,
>>>>>> Weidong
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
Sent:
>>>>>> Monday, March 22, 2010 5:20 AM
>>>>>> To: Konrad Rzeszutek Wilk
>>>>>> Cc: Han, Weidong; xen-devel@lists.xensource.com
>>>>>> Subject: [pvops xen/next ][iommu] attenpt to
passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain
attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.
>>>>>>
>>>>>> Hi Han/Konrad,
>>>>>>
>>>>>> In my setup i''m trying to passthrough an USB
3.0 pci-e controller  to a PV domU.
>>>>>> - xen: 4.0.0-rc6
>>>>>> - dom0: kernel xen/next
>>>>>> - domU: kernel 2.6.33 from
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
>>>>>>         ( to have pci-front together with most recent
usb3.0 xhci drivers.
>>>>>>
>>>>>>
>>>>>> - USB 3.0 xhci drivers work fine on the baremetal with
the 2.6.33 kernel.
>>>>>>
>>>>>> This is on a intel Q45 chipset with IOMMU.
>>>>>>
>>>>>> This is my boot config:
>>>>>> title           xen-4.0.0-rc6.gz / Debian GNU/Linux, 
kernel 2.6.32
>>>>>> root            (hd0,0)
>>>>>> kernel          /boot/xen-4.0.0-rc6.gz dom0_mem=768M
loglvl=all loglvl_guest=all  iommu=pv iommu_inclusive_mapping=1
>>>>>> module          /boot/vmlinuz-2.6.32 root=/dev/sda1 ro
earlyprintk=xen max_loop=255 xen-pciback.hide=(03:00.0)
>>>>>> module          /boot/initrd.img-2.6.32
>>>>>>
>>>>>> When booting the domU xm dmesg gets filled with the
following when the usb controller tries to initialize/:
>>>>>>
>>>>>> (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
>>>>>> (XEN) [VT-D]iommu.c:821: iommu_fault_status: Primary
Pending Fault
>>>>>> (XEN) [VT-D]iommu.c:796: DMAR:[DMA Read] Request device
[03:00.0] fault addr 1ff94000, iommu reg = ffff82c3fff54000
>>>>>> (XEN) DMAR:[fault reason 06h] PTE Read access is not
set
>>>>>> (XEN) print_vtd_entries: iommu = ffff83007c866970 bdf =
3:0.0 gmfn = 1ff94
>>>>>> (XEN)     root_entry = ffff83007c872000
>>>>>> (XEN)     root_entry[3] = 78f56001
>>>>>> (XEN)     context = ffff830078f56000
>>>>>> (XEN)     context[0] = 101_2f0e1001
>>>>>> (XEN)     l3 = ffff83002f0e1000
>>>>>> (XEN)     l3_index = 0
>>>>>> (XEN)     l3[0] = 2f0e0003
>>>>>> (XEN)     l2 = ffff83002f0e0000
>>>>>> (XEN)     l2_index = ff
>>>>>> (XEN)     l2[ff] = 0
>>>>>> (XEN)     l2[ff] not present
>>>>>>
>>>>>>
>>>>>>
>>>>>> Anyone any tips on what i could try ?, is this
something caused by xen, or something by the usb driver not adhering to kernel
DMA-api ?
>>>>>>
>>>>>> Attached:
>>>>>>
>>>>>> - xm-info.txt
>>>>>> - xm-dmesg.txt
>>>>>> - xend.log
>>>>>>
>>>>>> - dom0-dmesg.txt
>>>>>> - dom0-lspci-tree.txt
>>>>>> - dom0-lspci.txt
>>>>>>
>>>>>> - domU-lspci.txt
>>>>>> - domU-dmesg.txt
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Xen-devel mailing list
>>>>>> Xen-devel@lists.xensource.com
>>>>>> http://lists.xensource.com/xen-devel
>>>>>>             
>>>>>>             

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pasi Kärkkäinen

2010-Mar-22 09:55 UTC

head link

Re: [Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

On Mon, Mar 22, 2010 at 10:22:16AM +0100, Christian Tramnitz
wrote:> At 22.03.2010 10:19, Pasi Kärkkäinen wrote:
>>
>> Yeah, that''s what I meant. Ok.
>>
>> I thought it was also possible to use VT-d for a PV guest? no?
>>
>> -- Pasi
>
> It is, but this requires iommu=pv as hypervisor boot flag.
> With iommu=1 you can see in the hypervisor log that VTd will be disabled  
> for PV.
>
Yep, got it. Thanks.

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Sander Eikelenboom

2010-Mar-22 10:15 UTC

head link

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Hello Weidong/Konrad,

1) With iommu=0, the DMAR fault is gone (of course), but
        > (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from
00000a07:00000000 to 00000000:00000000.
        Stays in xm-dmesg, the pv guests is booted, and lspci shows the pci
device. But the device doesn''t function properly.


2) With iommu enabled and iommu_inclusive_mapping=1 and HVM:
        I can''t start the HVM, it complains:
        Error: pci: to avoid potential security issue, 0000:03:00.0 is not
allowed to be assigned to guest since it is behind PCIe switch that does not
support or enable ACS.

        This USB controller does have multiple pci-e bridges of its own, as can
be seen in the pci tree
        -[0000:00]-+-00.0
           +-01.0-[0000:01-04]----00.0-[0000:02-04]--+-01.0-[0000:03]----00.0   
<-- The usb controller i try to passthrough
           |                                         \-02.0-[0000:04]----00.0
           +-02.0
           +-03.0
           +-19.0
           +-1a.0
           +-1a.1
           +-1a.2
           +-1a.7
           +-1b.0
           +-1c.0-[0000:06]--
           +-1c.4-[0000:05]----00.0
           +-1d.0
           +-1d.1
           +-1d.2
           +-1d.7
           +-1e.0-[0000:07]--+-00.0
           |                 \-03.0
           +-1f.0
           +-1f.2
           +-1f.3
           \-1f.5


         When i use pciback to hide those pci-e bridges, and try to pass them
through as well,
         although lspci -k and dmesg show they are actually owned by pciback,
starting the HVM fails because the pci-e bridges are supposedly not owned by
pciback or stub.


Attached the xm-dmesg of boot without iommu

--
Sander






Monday, March 22, 2010, 9:04:09 AM, you wrote:
> The faults were caused by that the DMA address was not mapped in VT-d page
table.
> Could you have following two tries:
>         1) assign it to pv domU without VT-d
>         2) assign it to a hvm guest
> Regards,
> Weidong
> -----Original Message-----
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it] 
> Sent: Monday, March 22, 2010 5:20 AM
> To: Konrad Rzeszutek Wilk
> Cc: Han, Weidong; xen-devel@lists.xensource.com
> Subject: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb
controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
> Hi Han/Konrad,
> In my setup i''m trying to passthrough an USB 3.0 pci-e controller 
to a PV domU.
> - xen: 4.0.0-rc6
> - dom0: kernel xen/next
> - domU: kernel 2.6.33 from
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
>         ( to have pci-front together with most recent usb3.0 xhci drivers.
> - USB 3.0 xhci drivers work fine on the baremetal with the 2.6.33 kernel.
> This is on a intel Q45 chipset with IOMMU.
> This is my boot config:
> title           xen-4.0.0-rc6.gz / Debian GNU/Linux,  kernel 2.6.32
> root            (hd0,0)
> kernel          /boot/xen-4.0.0-rc6.gz dom0_mem=768M loglvl=all
loglvl_guest=all  iommu=pv iommu_inclusive_mapping=1
> module          /boot/vmlinuz-2.6.32 root=/dev/sda1 ro earlyprintk=xen
max_loop=255 xen-pciback.hide=(03:00.0)
> module          /boot/initrd.img-2.6.32
> When booting the domU xm dmesg gets filled with the following when the usb
controller tries to initialize/:
> (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from
00000a07:00000000 to 00000000:00000000.
> (XEN) [VT-D]iommu.c:821: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:796: DMAR:[DMA Read] Request device [03:00.0] fault
addr 1ff94000, iommu reg = ffff82c3fff54000
> (XEN) DMAR:[fault reason 06h] PTE Read access is not set
> (XEN) print_vtd_entries: iommu = ffff83007c866970 bdf = 3:0.0 gmfn = 1ff94
> (XEN)     root_entry = ffff83007c872000
> (XEN)     root_entry[3] = 78f56001
> (XEN)     context = ffff830078f56000
> (XEN)     context[0] = 101_2f0e1001
> (XEN)     l3 = ffff83002f0e1000
> (XEN)     l3_index = 0
> (XEN)     l3[0] = 2f0e0003
> (XEN)     l2 = ffff83002f0e0000
> (XEN)     l2_index = ff
> (XEN)     l2[ff] = 0
> (XEN)     l2[ff] not present

> Anyone any tips on what i could try ?, is this something caused by xen, or
something by the usb driver not adhering to kernel DMA-api ?
> Attached:
> - xm-info.txt
> - xm-dmesg.txt
> - xend.log
> - dom0-dmesg.txt
> - dom0-lspci-tree.txt
> - dom0-lspci.txt
> - domU-lspci.txt
> - domU-dmesg.txt






-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jan Beulich

2010-Mar-22 10:58 UTC

head link

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

>>> Sander Eikelenboom <linux@eikelenboom.it> 22.03.10 11:15
>>>
>Hello Weidong/Konrad,
>
>1) With iommu=0, the DMAR fault is gone (of course), but
>        > (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b
from 00000a07:00000000 to 00000000:00000000.
>        Stays in xm-dmesg, the pv guests is booted, and lspci shows the pci
device. But the device doesn''t function properly.
This is certainly unrelated to $subject problem - MSR 0x8b is
MSR_AMD64_PATCH_LEVEL, and is being written by the microcode patch loader.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Sander Eikelenboom

2010-Mar-22 13:14 UTC

head link

Re: [Xen-devel] Re: [pvops xen/next ] attenpt to passthrough PCI-e usb controllor to PV domU

Hello Jan,

Thx, that was indeed unrelated, not compiling in the microcode patchloader made
it disappear.
Any tips on how to find out where the problem seems to be ? not using the
DMA-api correctly could perhaps be something ?
Don''t seem to have any errors warning indicating anything left.

I suppose the usb xhci driver has some assumption that is not valid in the xen
context, but what ..

On xen + 2.6.32 dom0, the controller shows the USB device with lsusb (no
passthrough just hypervisor + dom0, also no iommu).
On xen + 2.6.32 dom0 + 2.6.32 domU passthrough, the device doesn''t show
up with lsusb.

When i compare the output of lspci -vvvknn from the first and last i
don''t see any shocking things.

--- dom0-lspci.txt      2010-03-22 13:55:57.000000000 +0100
+++ domu-lspci.txt      2010-03-22 05:42:49.000000000 +0100
@@ -1,31 +1,32 @@
 03:00.0 USB Controller [0c03]: NEC Corporation Device [1033:0194] (rev 03)
(prog-if 30)
        Subsystem: ASUSTeK Computer Inc. Device [1043:8413]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 17
        Region 0: Memory at fe8fe000 (64-bit, non-prefetchable) [size=8K]
        Capabilities: [50] Power Management version 3
-               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold-)
+               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [70] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
Enable-
                Address: 0000000000000000  Data: 0000
        Capabilities: [90] MSI-X: Enable- Mask- TabSize=8
                Vector table: BAR=0 offset=00001000
                PBA: BAR=0 offset=00001080
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s
unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
-               DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr-
TransPend-
+               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0
<4us, L1 unlimited
                        ClockPM+ Suprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
                        ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive-
BWMgmt- ABWMgmt-
        Capabilities: [100] Advanced Error Reporting <?>
        Capabilities: [140] Device Serial Number ff-ff-ff-ff-ff-ff-ff-ff
        Capabilities: [150] #18
        Kernel driver in use: xhci_hcd





When i compare the output of lsusb -v of the first and last situation, i do see
some differences, but have no clue what caused them.


--- dom0-lsusb.txt      2010-03-22 13:55:09.000000000 +0100
+++ domu-lsusb.txt      2010-03-22 05:42:58.000000000 +0100
@@ -1,176 +1,68 @@
-Bus 007 Device 002: ID 2040:2400 Hauppauge WinTV PVR USB2 (Model 24019)
-Device Descriptor:
-  bLength                18
-  bDescriptorType         1
-  bcdUSB               2.00
-  bDeviceClass            0 (Defined at Interface level)
-  bDeviceSubClass         0 
-  bDeviceProtocol         0 
-  bMaxPacketSize0        64
-  idVendor           0x2040 Hauppauge
-  idProduct          0x2400 WinTV PVR USB2 (Model 24019)
-  bcdDevice            8.00
-  iManufacturer           1 Hauppauge
-  iProduct                2 WinTV
-  iSerial                 3 2401-00-008433DC
-  bNumConfigurations      1
-  Configuration Descriptor:
-    bLength                 9
-    bDescriptorType         2
-    wTotalLength           60
-    bNumInterfaces          1
-    bConfigurationValue     1
-    iConfiguration          0 
-    bmAttributes         0xc0
-      Self Powered
-    MaxPower                0mA
-    Interface Descriptor:
-      bLength                 9
-      bDescriptorType         4
-      bInterfaceNumber        0
-      bAlternateSetting       0
-      bNumEndpoints           6
-      bInterfaceClass       255 Vendor Specific Class
-      bInterfaceSubClass    255 Vendor Specific Subclass
-      bInterfaceProtocol    255 Vendor Specific Protocol
-      iInterface              0 
-      Endpoint Descriptor:
-        bLength                 7
-        bDescriptorType         5
-        bEndpointAddress     0x81  EP 1 IN
-        bmAttributes            2
-          Transfer Type            Bulk
-          Synch Type               None
-          Usage Type               Data
-        wMaxPacketSize     0x0200  1x 512 bytes
-        bInterval               0
-      Endpoint Descriptor:
-        bLength                 7
-        bDescriptorType         5
-        bEndpointAddress     0x84  EP 4 IN
-        bmAttributes            2
-          Transfer Type            Bulk
-          Synch Type               None
-          Usage Type               Data
-        wMaxPacketSize     0x0200  1x 512 bytes
-        bInterval               0
-      Endpoint Descriptor:
-        bLength                 7
-        bDescriptorType         5
-        bEndpointAddress     0x88  EP 8 IN
-        bmAttributes            2
-          Transfer Type            Bulk
-          Synch Type               None
-          Usage Type               Data
-        wMaxPacketSize     0x0200  1x 512 bytes
-        bInterval               0
-      Endpoint Descriptor:
-        bLength                 7
-        bDescriptorType         5
-        bEndpointAddress     0x01  EP 1 OUT
-        bmAttributes            2
-          Transfer Type            Bulk
-          Synch Type               None
-          Usage Type               Data
-        wMaxPacketSize     0x0200  1x 512 bytes
-        bInterval               0
-      Endpoint Descriptor:
-        bLength                 7
-        bDescriptorType         5
-        bEndpointAddress     0x02  EP 2 OUT
-        bmAttributes            2
-          Transfer Type            Bulk
-          Synch Type               None
-          Usage Type               Data
-        wMaxPacketSize     0x0200  1x 512 bytes
-        bInterval               0
-      Endpoint Descriptor:
-        bLength                 7
-        bDescriptorType         5
-        bEndpointAddress     0x86  EP 6 IN
-        bmAttributes            2
-          Transfer Type            Bulk
-          Synch Type               None
-          Usage Type               Data
-        wMaxPacketSize     0x0200  1x 512 bytes
-        bInterval               0
-Device Qualifier (for other device speed):
-  bLength                10
-  bDescriptorType         6
-  bcdUSB               2.00
-  bDeviceClass            0 (Defined at Interface level)
-  bDeviceSubClass         0 
-  bDeviceProtocol         0 
-  bMaxPacketSize0        64
-  bNumConfigurations      1
-Device Status:     0x0003
-  Self Powered
-  Remote Wakeup Enabled
 
-Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
+Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Device Descriptor:
   bLength                18
   bDescriptorType         1
   bcdUSB               3.00
   bDeviceClass            9 Hub
   bDeviceSubClass         0 Unused
   bDeviceProtocol         3 
   bMaxPacketSize0         9
   idVendor           0x1d6b Linux Foundation
   idProduct          0x0002 2.0 root hub
   bcdDevice            2.06
   iManufacturer           3 Linux 2.6.32 xhci_hcd
   iProduct                2 xHCI Host Controller
   iSerial                 1 0000:03:00.0
   bNumConfigurations      1
   Configuration Descriptor:
     bLength                 9
     bDescriptorType         2
     wTotalLength           25
     bNumInterfaces          1
     bConfigurationValue     1
     iConfiguration          0 
     bmAttributes         0xe0
       Self Powered
       Remote Wakeup
     MaxPower                0mA
     Interface Descriptor:
       bLength                 9
       bDescriptorType         4
       bInterfaceNumber        0
       bAlternateSetting       0
       bNumEndpoints           1
       bInterfaceClass         9 Hub
       bInterfaceSubClass      0 Unused
       bInterfaceProtocol      0 Full speed (or root) hub
       iInterface              0 
       Endpoint Descriptor:
         bLength                 7
         bDescriptorType         5
         bEndpointAddress     0x81  EP 1 IN
         bmAttributes            3
           Transfer Type            Interrupt
           Synch Type               None
           Usage Type               Data
         wMaxPacketSize     0x0004  1x 4 bytes
         bInterval              12
 Hub Descriptor:
   bLength               9
   bDescriptorType      41
   nNbrPorts             4
   wHubCharacteristic 0x0009
     Per-port power switching
     Per-port overcurrent protection
     TT think time 8 FS bits
   bPwrOn2PwrGood       10 * 2 milli seconds
   bHubContrCurrent      0 milli Ampere
   DeviceRemovable    0x00
   PortPwrCtrlMask    0xff
  Hub Port Status:
    Port 1: 0000.0100 power
    Port 2: 0000.0100 power
    Port 3: 0000.0100 power
-   Port 4: 0000.0503 highspeed power enable connect
+   Port 4: 0000.0101 power connect
 Device Status:     0x0003
   Self Powered
   Remote Wakeup Enabled





--
Sander


Monday, March 22, 2010, 11:58:09 AM, you wrote:
>>>> Sander Eikelenboom <linux@eikelenboom.it> 22.03.10 11:15
>>>
>>Hello Weidong/Konrad,
>>
>>1) With iommu=0, the DMAR fault is gone (of course), but
>>        > (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
>>        Stays in xm-dmesg, the pv guests is booted, and lspci shows the
pci device. But the device doesn''t function properly.
> This is certainly unrelated to $subject problem - MSR 0x8b is
MSR_AMD64_PATCH_LEVEL, and is being written by the microcode patch loader.
> Jan



-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Mar-22 19:12 UTC

head link

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

On Mon, Mar 22, 2010 at 11:15:24AM +0100, Sander Eikelenboom
wrote:> Hello Weidong/Konrad,
> 
> 1) With iommu=0, the DMAR fault is gone (of course), but
>         > (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b
from 00000a07:00000000 to 00000000:00000000.
>         Stays in xm-dmesg, the pv guests is booted, and lspci shows the pci
device. But the device doesn''t function properly.
How does it not function properly? Was this related to the IOMMU error ?
Did you point it out to me previously and I missed it? If so, can you
resend it to me please.

If the problem is with the message about needing
''swiotlb=force'' to be
passed, ''iommu=soft swiotlb=force'' should take care of that. 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Mar-22 20:30 UTC

head link

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

On Mon, Mar 22, 2010 at 09:35:50PM +0100, Sander Eikelenboom
wrote:> Hi Konrad,
> 
> Thx again and again and again :-)
Oh nice. Glad to hear.
> Works like a charm by adding these to the domU kernel options.
> I saw a lot of your hard work on the swiotlb on LKML as well, would be nice
if it would be accepted together with pci-front,
> that would make mainline kernels as domU work with pci-passthrough as well.
Yes!  That is the idea - to have the xen-pcifront upstream. Thought as
you experienced first hand, first the SWIOTLB need to be in the kernel
to make the PCI front patches work.> 
> I now got it running with the linux 2.6.33 tree from your git tree, which
is of course also very recent :-)
Oh boy. You are adventurous :-) .. > 
> Now running on:
> hypervisor: xen-4.0.0-rc6
> dom0: xen-next
> domU: 2.6.33 tree with pcifront and swiotlb from Konrad''s git
tree, and with some additional patches to het isochronous URB''s working
on the USB 3.0 xHCI driver from linux-usb mailing lists.
> 
> 
> Pasi, perhaps a good thing to point out in the passthrough wiki pages (at
least when using pvops kernels),
Well, in regards to those trees of mine I keep on reworking them so that they
will be
ready for upstream. They don''t have all of the nice bells and whistles
that xen/next has - just the basics to get xen-pcifront working.

> BTW how do you handle tables in the wiki''s ?
> Because i tried with the xen-hypervisor-boot-options page, but it was a
real pain in the *ss to set it up.
> 
> 
> --
> Sander
> 
> 
> 
> Monday, March 22, 2010, 8:12:37 PM, you wrote:
> 
> > On Mon, Mar 22, 2010 at 11:15:24AM +0100, Sander Eikelenboom wrote:
> >> Hello Weidong/Konrad,
> >> 
> >> 1) With iommu=0, the DMAR fault is gone (of course), but
> >>         > (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
> >>         Stays in xm-dmesg, the pv guests is booted, and lspci
shows the pci device. But the device doesn''t function properly.
> 
> > How does it not function properly? Was this related to the IOMMU error
?
> > Did you point it out to me previously and I missed it? If so, can you
> > resend it to me please.
> 
> > If the problem is with the message about needing
''swiotlb=force'' to be
> > passed, ''iommu=soft swiotlb=force'' should take care
of that.
> 
> 
> 
> 
> -- 
> Best regards,
>  Sander                            mailto:linux@eikelenboom.it
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Sander Eikelenboom

2010-Mar-22 20:35 UTC

head link

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

Hi Konrad,

Thx again and again and again :-)
Works like a charm by adding these to the domU kernel options.
I saw a lot of your hard work on the swiotlb on LKML as well, would be nice if
it would be accepted together with pci-front,
that would make mainline kernels as domU work with pci-passthrough as well.

I now got it running with the linux 2.6.33 tree from your git tree, which is of
course also very recent :-)

Now running on:
hypervisor: xen-4.0.0-rc6
dom0: xen-next
domU: 2.6.33 tree with pcifront and swiotlb from Konrad''s git tree, and
with some additional patches to het isochronous URB''s working on the
USB 3.0 xHCI driver from linux-usb mailing lists.

Pasi, perhaps a good thing to point out in the passthrough wiki pages (at least
when using pvops kernels),
BTW how do you handle tables in the wiki''s ?
Because i tried with the xen-hypervisor-boot-options page, but it was a real
pain in the *ss to set it up.

--
Sander

Monday, March 22, 2010, 8:12:37 PM, you wrote:
> On Mon, Mar 22, 2010 at 11:15:24AM +0100, Sander Eikelenboom wrote:
>> Hello Weidong/Konrad,
>> 
>> 1) With iommu=0, the DMAR fault is gone (of course), but
>>         > (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
>>         Stays in xm-dmesg, the pv guests is booted, and lspci shows the
pci device. But the device doesn''t function properly.
> How does it not function properly? Was this related to the IOMMU error ?
> Did you point it out to me previously and I missed it? If so, can you
> resend it to me please.
> If the problem is with the message about needing
''swiotlb=force'' to be
> passed, ''iommu=soft swiotlb=force'' should take care of
that.

-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pasi Kärkkäinen

2010-Mar-22 20:49 UTC

head link

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

On Mon, Mar 22, 2010 at 09:35:50PM +0100, Sander Eikelenboom
wrote:> Hi Konrad,
> 
> Thx again and again and again :-)
> Works like a charm by adding these to the domU kernel options.
> I saw a lot of your hard work on the swiotlb on LKML as well, would be nice
if it would be accepted together with pci-front,
> that would make mainline kernels as domU work with pci-passthrough as well.
> 
> I now got it running with the linux 2.6.33 tree from your git tree, which
is of course also very recent :-)
> 
> Now running on:
> hypervisor: xen-4.0.0-rc6
> dom0: xen-next
> domU: 2.6.33 tree with pcifront and swiotlb from Konrad''s git
tree, and with some additional patches to het isochronous URB''s working
on the USB 3.0 xHCI driver from linux-usb mailing lists.
> 
> 
> Pasi, perhaps a good thing to point out in the passthrough wiki pages (at
least when using pvops kernels),
>
Yeah.. hmm.. so what should I add? You can also add it yourself ;)

The current version is here:
http://wiki.xensource.com/xenwiki/XenPCIpassthrough

still a work-in-progress.. 
> BTW how do you handle tables in the wiki''s ?
> Because i tried with the xen-hypervisor-boot-options page, but it was a
real pain in the *ss to set it up.
> 
iirc in the GUI editing mode you can just right-click on the table and
add/delete rows/columns etc?

-- Pasi
> 
> --
> Sander
> 
> 
> 
> Monday, March 22, 2010, 8:12:37 PM, you wrote:
> 
> > On Mon, Mar 22, 2010 at 11:15:24AM +0100, Sander Eikelenboom wrote:
> >> Hello Weidong/Konrad,
> >> 
> >> 1) With iommu=0, the DMAR fault is gone (of course), but
> >>         > (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
> >>         Stays in xm-dmesg, the pv guests is booted, and lspci
shows the pci device. But the device doesn''t function properly.
> 
> > How does it not function properly? Was this related to the IOMMU error
?
> > Did you point it out to me previously and I missed it? If so, can you
> > resend it to me please.
> 
> > If the problem is with the message about needing
''swiotlb=force'' to be
> > passed, ''iommu=soft swiotlb=force'' should take care
of that.
> 
> 
> 
> 
> -- 
> Best regards,
>  Sander                            mailto:linux@eikelenboom.it
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Mar-22 21:12 UTC

head link

Re: [Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

On Mon, Mar 22, 2010 at 11:26:52PM +0200, Pasi Kärkkäinen
wrote:> On Mon, Mar 22, 2010 at 10:23:00PM +0100, Sander Eikelenboom wrote:
> > Hello Pasi,
> > 
> > Ok added a few lines about it to that page.
> > 
> 
> Ok. Are those options only needed for pvops domU kernel,
> or also for linux-2.6.18-xen guests? 
The ''swiotlb=force iommu=soft'' are only needed when you do PCI
passthrough to PV guests and for all kernels (SLES10, RHEL5,
SLES11) and so on.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Sander Eikelenboom

2010-Mar-22 21:23 UTC

head link

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

Hello Pasi,

Ok added a few lines about it to that page.

--

Sander

Monday, March 22, 2010, 9:49:16 PM, you wrote:
> On Mon, Mar 22, 2010 at 09:35:50PM +0100, Sander Eikelenboom wrote:
>> Hi Konrad,
>> 
>> Thx again and again and again :-)
>> Works like a charm by adding these to the domU kernel options.
>> I saw a lot of your hard work on the swiotlb on LKML as well, would be
nice if it would be accepted together with pci-front,
>> that would make mainline kernels as domU work with pci-passthrough as
well.
>> 
>> I now got it running with the linux 2.6.33 tree from your git tree,
which is of course also very recent :-)
>> 
>> Now running on:
>> hypervisor: xen-4.0.0-rc6
>> dom0: xen-next
>> domU: 2.6.33 tree with pcifront and swiotlb from Konrad''s git
tree, and with some additional patches to het isochronous URB''s working
on the USB 3.0 xHCI driver from linux-usb mailing lists.
>> 
>> 
>> Pasi, perhaps a good thing to point out in the passthrough wiki pages
(at least when using pvops kernels),
>>
> Yeah.. hmm.. so what should I add? You can also add it yourself ;)
> The current version is here:
> http://wiki.xensource.com/xenwiki/XenPCIpassthrough
> still a work-in-progress.. 
>> BTW how do you handle tables in the wiki''s ?
>> Because i tried with the xen-hypervisor-boot-options page, but it was a
real pain in the *ss to set it up.
>> 
> iirc in the GUI editing mode you can just right-click on the table and
add/delete rows/columns etc?
> -- Pasi
>> 
>> --
>> Sander
>> 
>> 
>> 
>> Monday, March 22, 2010, 8:12:37 PM, you wrote:
>> 
>> > On Mon, Mar 22, 2010 at 11:15:24AM +0100, Sander Eikelenboom
wrote:
>> >> Hello Weidong/Konrad,
>> >> 
>> >> 1) With iommu=0, the DMAR fault is gone (of course), but
>> >>         > (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
>> >>         Stays in xm-dmesg, the pv guests is booted, and lspci
shows the pci device. But the device doesn''t function properly.
>> 
>> > How does it not function properly? Was this related to the IOMMU
error ?
>> > Did you point it out to me previously and I missed it? If so, can
you
>> > resend it to me please.
>> 
>> > If the problem is with the message about needing
''swiotlb=force'' to be
>> > passed, ''iommu=soft swiotlb=force'' should take
care of that.
>> 
>> 
>> 
>> 
>> -- 
>> Best regards,
>>  Sander                            mailto:linux@eikelenboom.it
>> 


-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pasi Kärkkäinen

2010-Mar-22 21:26 UTC

head link

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

On Mon, Mar 22, 2010 at 10:23:00PM +0100, Sander Eikelenboom
wrote:> Hello Pasi,
> 
> Ok added a few lines about it to that page.
> 
Ok. Are those options only needed for pvops domU kernel,
or also for linux-2.6.18-xen guests? 

-- Pasi
> 
> Sander
> 
> Monday, March 22, 2010, 9:49:16 PM, you wrote:
> 
> > On Mon, Mar 22, 2010 at 09:35:50PM +0100, Sander Eikelenboom wrote:
> >> Hi Konrad,
> >> 
> >> Thx again and again and again :-)
> >> Works like a charm by adding these to the domU kernel options.
> >> I saw a lot of your hard work on the swiotlb on LKML as well,
would be nice if it would be accepted together with pci-front,
> >> that would make mainline kernels as domU work with pci-passthrough
as well.
> >> 
> >> I now got it running with the linux 2.6.33 tree from your git
tree, which is of course also very recent :-)
> >> 
> >> Now running on:
> >> hypervisor: xen-4.0.0-rc6
> >> dom0: xen-next
> >> domU: 2.6.33 tree with pcifront and swiotlb from Konrad''s
git tree, and with some additional patches to het isochronous URB''s
working on the USB 3.0 xHCI driver from linux-usb mailing lists.
> >> 
> >> 
> >> Pasi, perhaps a good thing to point out in the passthrough wiki
pages (at least when using pvops kernels),
> >>
> 
> > Yeah.. hmm.. so what should I add? You can also add it yourself ;)
> 
> > The current version is here:
> > http://wiki.xensource.com/xenwiki/XenPCIpassthrough
> 
> > still a work-in-progress.. 
> 
> >> BTW how do you handle tables in the wiki''s ?
> >> Because i tried with the xen-hypervisor-boot-options page, but it
was a real pain in the *ss to set it up.
> >> 
> 
> > iirc in the GUI editing mode you can just right-click on the table and
add/delete rows/columns etc?
> 
> > -- Pasi
> 
> >> 
> >> --
> >> Sander
> >> 
> >> 
> >> 
> >> Monday, March 22, 2010, 8:12:37 PM, you wrote:
> >> 
> >> > On Mon, Mar 22, 2010 at 11:15:24AM +0100, Sander Eikelenboom
wrote:
> >> >> Hello Weidong/Konrad,
> >> >> 
> >> >> 1) With iommu=0, the DMAR fault is gone (of course), but
> >> >>         > (XEN) traps.c:2309:d1 Domain attempted WRMSR
000000000000008b from 00000a07:00000000 to 00000000:00000000.
> >> >>         Stays in xm-dmesg, the pv guests is booted, and
lspci shows the pci device. But the device doesn''t function properly.
> >> 
> >> > How does it not function properly? Was this related to the
IOMMU error ?
> >> > Did you point it out to me previously and I missed it? If so,
can you
> >> > resend it to me please.
> >> 
> >> > If the problem is with the message about needing
''swiotlb=force'' to be
> >> > passed, ''iommu=soft swiotlb=force'' should
take care of that.
> >> 
> >> 
> >> 
> >> 
> >> -- 
> >> Best regards,
> >>  Sander                            mailto:linux@eikelenboom.it
> >> 
> 
> 
> 
> -- 
> Best regards,
>  Sander                            mailto:linux@eikelenboom.it
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pasi Kärkkäinen

2010-Mar-23 07:05 UTC

head link

Re: [Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

On Mon, Mar 22, 2010 at 05:12:09PM -0400, Konrad Rzeszutek Wilk
wrote:> On Mon, Mar 22, 2010 at 11:26:52PM +0200, Pasi Kärkkäinen wrote:
> > On Mon, Mar 22, 2010 at 10:23:00PM +0100, Sander Eikelenboom wrote:
> > > Hello Pasi,
> > > 
> > > Ok added a few lines about it to that page.
> > > 
> > 
> > Ok. Are those options only needed for pvops domU kernel,
> > or also for linux-2.6.18-xen guests? 
> 
> The ''swiotlb=force iommu=soft'' are only needed when you
do PCI
> passthrough to PV guests and for all kernels (SLES10, RHEL5,
> SLES11) and so on.
>
Ok, thanks. Wiki updated.

-- Pasi



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jan Beulich

2010-Mar-23 10:16 UTC

head link

Re: [Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

>>> Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> 22.03.10 22:12
>>>
>On Mon, Mar 22, 2010 at 11:26:52PM +0200, Pasi Kärkkäinen wrote:
>> On Mon, Mar 22, 2010 at 10:23:00PM +0100, Sander Eikelenboom wrote:
>> > Hello Pasi,
>> > 
>> > Ok added a few lines about it to that page.
>> > 
>> 
>> Ok. Are those options only needed for pvops domU kernel,
>> or also for linux-2.6.18-xen guests? 
>
>The ''swiotlb=force iommu=soft'' are only needed when you do
PCI
>passthrough to PV guests and for all kernels (SLES10, RHEL5,
>SLES11) and so on.
I don''t think that''s true for SLE10/SLE11.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Sander Eikelenboom

2010-Mar-23 10:23 UTC

head link

Re: [Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

Hello Jan,

How do these handle DMA related to the passthrough device in a PV-guest then ?
From what i remember from previous (debian) xen kernels was that at least
swiotlb=force was needed as kernel options to the domU kernel to passthrough
usb-controllers.

--
Sander

Tuesday, March 23, 2010, 11:16:37 AM, you wrote:
>>>> Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> 22.03.10
22:12 >>>
>>On Mon, Mar 22, 2010 at 11:26:52PM +0200, Pasi Kärkkäinen wrote:
>>> On Mon, Mar 22, 2010 at 10:23:00PM +0100, Sander Eikelenboom wrote:
>>> > Hello Pasi,
>>> > 
>>> > Ok added a few lines about it to that page.
>>> > 
>>> 
>>> Ok. Are those options only needed for pvops domU kernel,
>>> or also for linux-2.6.18-xen guests? 
>>
>>The ''swiotlb=force iommu=soft'' are only needed when
you do PCI
>>passthrough to PV guests and for all kernels (SLES10, RHEL5,
>>SLES11) and so on.
> I don''t think that''s true for SLE10/SLE11.
> Jan


-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jan Beulich

2010-Mar-23 11:23 UTC

head link

Re: [Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

>>> Sander Eikelenboom <linux@eikelenboom.it> 23.03.10 11:23
>>>
>How do these handle DMA related to the passthrough device in a PV-guest then
?
>From what i remember from previous (debian) xen kernels was that at least
swiotlb=force was needed as kernel options to the domU kernel to passthrough
usb-controllers.
Yes, exactly. But not "iommu=soft".

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pasi Kärkkäinen

2010-Mar-23 11:24 UTC

head link

Re: [Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

On Tue, Mar 23, 2010 at 11:23:36AM +0000, Jan Beulich
wrote:> >>> Sander Eikelenboom <linux@eikelenboom.it> 23.03.10 11:23
>>>
> >How do these handle DMA related to the passthrough device in a PV-guest
then ?
> >From what i remember from previous (debian) xen kernels was that at
least swiotlb=force was needed as kernel options to the domU kernel to
passthrough usb-controllers.
> 
> Yes, exactly. But not "iommu=soft".
> 
Ok, I''ll update the wiki.

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Xen devel - Mar 2010 - [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

[Xen-devel] [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

[Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Re: [Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Re: [Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Re: [Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Re: [Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Re: [Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Re: [Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Re: [Xen-devel] RE: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Re: [Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

Re: [Xen-devel] Re: [pvops xen/next ] attenpt to passthrough PCI-e usb controllor to PV domU

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU: (XEN) traps.c:2309:d1 Domain attempted WRMSR 000000000000008b from 00000a07:00000000 to 00000000:00000000.

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

Re: [Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

[Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

Re: [Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

Re: [Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

Re: [Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

Re: [Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)

Re: [Xen-devel] Re: [pvops xen/next ][iommu] attenpt to passthrough PCI-e usb controllor to PV domU SUCCESS :-)