Xen devel - Sep 2009 - Announcing xen/master: pvops git trees rearranged

Well, my current pvops/dom0 tree is finally (reasonably) stable.  There
was a fairly nasty bug which ended up corrupting dom0 memory when doing
IO on behalf of domains, but that is finally fixed.

In honour of this, I''ve renamed the rebase/* branches to xen/* (moving
the old remaining xen/* branches to xen-old/*); xen/master is now the
preferred branch for fetching current dom0 work.

The kernel tree is fairly featureful:

    * basic dom0 support
    * blkback
    * netback
    * ACPI power management
    * S3 suspend/resume (at least for some people)
    * microcode update
    * MSI support

In other words, it has as much as it ever has.

There are a few notable missing features:

    * blktap2
    * netchannel2
    * pci front/back
    * upstream Linux support
    * your pet feature


Full coordinates are:

    git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git xen/master

See http://wiki.xensource.com/xenwiki/XenParavirtOps for general
directions on configuration, compilation and use.

This is definitely a work-in-progress kernel.  I''d appreciate all bug
*and* success reports so I can get some idea of how many people are
using this thing, and how often there are problems.  Patches gratefully
accepted.

Thanks,
    J


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hello Jeremy,

Jeremy Fitzhardinge schrieb:
> Well, my current pvops/dom0 tree is finally (reasonably) stable.  There
> was a fairly nasty bug which ended up corrupting dom0 memory when doing
> IO on behalf of domains, but that is finally fixed.
> 
> In honour of this, I''ve renamed the rebase/* branches to xen/*
(moving
> the old remaining xen/* branches to xen-old/*); xen/master is now the
> preferred branch for fetching current dom0 work.
> 
> The kernel tree is fairly featureful:
> 
>     * basic dom0 support
>     * blkback
>     * netback
>     * ACPI power management
>     * S3 suspend/resume (at least for some people)
>     * microcode update
>     * MSI support
> 
> In other words, it has as much as it ever has.
waited for MSI, will give it a try :o)
> There are a few notable missing features:
> 
>     * blktap2
>     * netchannel2
>     * pci front/back
>     * upstream Linux support
>     * your pet feature
> 
> 
> Full coordinates are:
> 
>     git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git xen/master
Is HEAD already pointing to xen/master or do i have to change branches?
> See http://wiki.xensource.com/xenwiki/XenParavirtOps for general
> directions on configuration, compilation and use.
I think it would be a good idea to add some backtrace and debug related 
options to the CONFIG_ options listed on the wiki page so that one gets 
all the details needed from the testers when some bug shows up (CCd 
Pasi). The NETXEN_NIC option looks unrelated to xen...
> This is definitely a work-in-progress kernel.  I''d appreciate all
bug
> *and* success reports so I can get some idea of how many people are
> using this thing, and how often there are problems.  Patches gratefully
> accepted.
Will report something back soon,


Marc

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

I just tested this branch and found that the system gets stuck on
initializing the AHCI controller (AMD SB600).
Please find the error messages attached as png (unfortunately netconsole
doesn''t work either)

When booting the same kernel natively without xen, all works fine
(including netconsole).


Patrick



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Marc - A. Dahlhaus schrieb:
> Hello Jeremy,
> 
> Jeremy Fitzhardinge schrieb:
>> Well, my current pvops/dom0 tree is finally (reasonably) stable.  There
>> was a fairly nasty bug which ended up corrupting dom0 memory when doing
>> IO on behalf of domains, but that is finally fixed.
>>
>> In honour of this, I''ve renamed the rebase/* branches to xen/*
(moving
>> the old remaining xen/* branches to xen-old/*); xen/master is now the
>> preferred branch for fetching current dom0 work.
>>
--8<--
>> See http://wiki.xensource.com/xenwiki/XenParavirtOps for general
>> directions on configuration, compilation and use.
> 
> I think it would be a good idea to add some backtrace and debug related 
> options to the CONFIG_ options listed on the wiki page so that one gets 
> all the details needed from the testers when some bug shows up (CCd 
> Pasi). The NETXEN_NIC option looks unrelated to xen...
Somehow my mail gateway got problems with umlauts and rewrote 
mailadress, so this is another try to CC pasi...
>> This is definitely a work-in-progress kernel.  I''d appreciate
all bug
>> *and* success reports so I can get some idea of how many people are
>> using this thing, and how often there are problems.  Patches gratefully
>> accepted.
> 
> Will report something back soon,
Had no luck. It crashed during irq initialization. Somehow the serial 
logs aren''t fully transmitted. All i could catch is attached. Also is 
attached the .config used to build the dom0 kernel. Will give xen 3.4.1 
and unstable another try tomorrow.

Marc



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I forgot to mention that the version (2.6.30-rc3) from xen-tip/master
branch, which is checked out per default from the xen-unstable scripts,
does *not* have this issue and boots fine on that hardware.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sun, Sep 20, 2009 at 09:29:46PM +0200, Marc - A. Dahlhaus
wrote:
> Marc - A. Dahlhaus schrieb:
> >Hello Jeremy,
> >
> >Jeremy Fitzhardinge schrieb:
> >>Well, my current pvops/dom0 tree is finally (reasonably) stable. 
There
> >>was a fairly nasty bug which ended up corrupting dom0 memory when
doing
> >>IO on behalf of domains, but that is finally fixed.
> >>
> >>In honour of this, I''ve renamed the rebase/* branches to
xen/* (moving
> >>the old remaining xen/* branches to xen-old/*); xen/master is now
the
> >>preferred branch for fetching current dom0 work.
> >>
> --8<--
> >>See http://wiki.xensource.com/xenwiki/XenParavirtOps for general
> >>directions on configuration, compilation and use.
> >
> >I think it would be a good idea to add some backtrace and debug related
> >options to the CONFIG_ options listed on the wiki page so that one gets
> >all the details needed from the testers when some bug shows up (CCd 
> >Pasi). The NETXEN_NIC option looks unrelated to xen...
> 
> Somehow my mail gateway got problems with umlauts and rewrote 
> mailadress, so this is another try to CC pasi...
>
Heh. It''s kinda funny to have all these charset/umlaut problems still
in 2009 :)
> >>This is definitely a work-in-progress kernel.  I''d
appreciate all bug
> >>*and* success reports so I can get some idea of how many people are
> >>using this thing, and how often there are problems.  Patches
gratefully
> >>accepted.
> >
> >Will report something back soon,
> 
> Had no luck. It crashed during irq initialization. Somehow the serial 
> logs aren''t fully transmitted. All i could catch is attached. Also
is
> attached the .config used to build the dom0 kernel. Will give xen 3.4.1 
> and unstable another try tomorrow.
> 
Please paste your grub.conf. I''m going to try xen/master tree today..

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Am Montag, den 21.09.2009, 09:22 +0300 schrieb Pasi
Kärkkäinen:
> On Sun, Sep 20, 2009 at 09:29:46PM +0200, Marc - A. Dahlhaus wrote:
> > Marc - A. Dahlhaus schrieb:
> > >Hello Jeremy,
> > >
> > >Jeremy Fitzhardinge schrieb:
> > >>Well, my current pvops/dom0 tree is finally (reasonably)
stable.  There
> > >>was a fairly nasty bug which ended up corrupting dom0 memory
when doing
> > >>IO on behalf of domains, but that is finally fixed.
> > >>
> > >>In honour of this, I''ve renamed the rebase/* branches
to xen/* (moving
> > >>the old remaining xen/* branches to xen-old/*); xen/master is
now the
> > >>preferred branch for fetching current dom0 work.
> > >>
> > --8<--
> > >>See http://wiki.xensource.com/xenwiki/XenParavirtOps for
general
> > >>directions on configuration, compilation and use.
> > >
> > >I think it would be a good idea to add some backtrace and debug
related
> > >options to the CONFIG_ options listed on the wiki page so that one
gets
> > >all the details needed from the testers when some bug shows up
(CCd
> > >Pasi). The NETXEN_NIC option looks unrelated to xen...
> > 
> > Somehow my mail gateway got problems with umlauts and rewrote 
> > mailadress, so this is another try to CC pasi...
> >
> 
> Heh. It''s kinda funny to have all these charset/umlaut problems
still in 2009 :)
Stange problem as we don''t had such problems with other mail addresses
that used umlauts in headers, we''re working on reproducing it. Looks
like a Perl-Module related problem in the spam filter as far as we got
until now...
> > >>This is definitely a work-in-progress kernel.  I''d
appreciate all bug
> > >>*and* success reports so I can get some idea of how many
people are
> > >>using this thing, and how often there are problems.  Patches
gratefully
> > >>accepted.
> > >
> > >Will report something back soon,
> > 
> > Had no luck. It crashed during irq initialization. Somehow the serial 
> > logs aren''t fully transmitted. All i could catch is attached.
Also is
> > attached the .config used to build the dom0 kernel. Will give xen
3.4.1
> > and unstable another try tomorrow.
> > 
> 
> Please paste your grub.conf. I''m going to try xen/master tree
today..
> 
> -- Pasi
title XEN pv_ops dom0
root (hd0,1)
kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT earlyprintk=xen 4
module /boot/initramfs-2.6.31-xen

With this one i get no output from the kernel itself at all, i think i
have some things build as modules in my .config that needs to be buildin
for a working vga console (might be that i need to change the
earlyprintk param to vga) but didn''t had the time to hunt the problem
down.


title XEN pv_ops dom0 with serial console
root (hd0,1)
kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
com1=9600,8n1 console=com1
module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT console=ttyS0,9600n8
earlyprintk=xen 4
module /boot/initramfs-2.6.31-xen

That''s the one i used to grab the log.

This is a i945 based box with an Core 2 Duo E8400 and 4GB Ram installed
but only 2912584k usable because of an PCI-Card that supports only 32bit
addressing (appropriated-ram-crap). Could this be a problem? I try to
test again with 2GB Ram installed this evening and report if it gets any
further...


Marc


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Sep 21, 2009 at 10:49:03AM +0200, Marc - A. Dahlhaus [ Administration |
Westermann GmbH ] wrote:
> > > >>This is definitely a work-in-progress kernel. 
I''d appreciate all bug
> > > >>*and* success reports so I can get some idea of how many
people are
> > > >>using this thing, and how often there are problems. 
Patches gratefully
> > > >>accepted.
> > > >
> > > >Will report something back soon,
> > > 
> > > Had no luck. It crashed during irq initialization. Somehow the
serial
> > > logs aren''t fully transmitted. All i could catch is
attached. Also is
> > > attached the .config used to build the dom0 kernel. Will give xen
3.4.1
> > > and unstable another try tomorrow.
> > > 
> > 
> > Please paste your grub.conf. I''m going to try xen/master tree
today..
> > 
> > -- Pasi
> 
> title XEN pv_ops dom0
> root (hd0,1)
> kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT earlyprintk=xen 4
> module /boot/initramfs-2.6.31-xen
> 
> With this one i get no output from the kernel itself at all, i think i
> have some things build as modules in my .config that needs to be buildin
> for a working vga console (might be that i need to change the
> earlyprintk param to vga) but didn''t had the time to hunt the
problem
> down.
> 
> 
> title XEN pv_ops dom0 with serial console
> root (hd0,1)
> kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> com1=9600,8n1 console=com1
> module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT console=ttyS0,9600n8
> earlyprintk=xen 4
> module /boot/initramfs-2.6.31-xen
> 
> That''s the one i used to grab the log.
> 
> This is a i945 based box with an Core 2 Duo E8400 and 4GB Ram installed
> but only 2912584k usable because of an PCI-Card that supports only 32bit
> addressing (appropriated-ram-crap). Could this be a problem? I try to
> test again with 2GB Ram installed this evening and report if it gets any
> further...
> 
Here are my working grub.conf entries for pv_ops dom0:

VGA text console:

title Fedora Xen pv_ops dom0-test (2.6.31-rc5)
        root (hd0,0)
        kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all
        module /vmlinuz-2.6.31-rc5 ro root=/dev/mapper/vg_dom0test-lv01
        module /initrd-2.6.31-rc5.img


Serial console:

title Fedora Xen pv_ops dom0-test (2.6.31-rc5) / serial console
        root (hd0,0)
        kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all com1=19200,8n1
console=com1
        module /vmlinuz-2.6.31-rc5 ro root=/dev/mapper/vg_dom0test-lv01
console=hvc0 earlyprintk=xen
        module /initrd-2.6.31-rc5.img

When using serial console with Xen you don''t need to specify serial
console stuff for the dom0 kernel..
Also note the ''console=hvc0'' for dom0 kernel..

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Am Montag, den 21.09.2009, 12:03 +0300 schrieb Pasi
Kärkkäinen:
> On Mon, Sep 21, 2009 at 10:49:03AM +0200, Marc - A. Dahlhaus [
Administration | Westermann GmbH ] wrote:
--8<--
> > title XEN pv_ops dom0
> > root (hd0,1)
> > kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> > module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT earlyprintk=xen 4
> > module /boot/initramfs-2.6.31-xen
> > 
> > With this one i get no output from the kernel itself at all, i think i
> > have some things build as modules in my .config that needs to be
buildin
> > for a working vga console (might be that i need to change the
> > earlyprintk param to vga) but didn''t had the time to hunt the
problem
> > down.
> > 
> > 
> > title XEN pv_ops dom0 with serial console
> > root (hd0,1)
> > kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> > com1=9600,8n1 console=com1
> > module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT
console=ttyS0,9600n8
> > earlyprintk=xen 4
> > module /boot/initramfs-2.6.31-xen
> > 
> > That''s the one i used to grab the log.
> > 
> > This is a i945 based box with an Core 2 Duo E8400 and 4GB Ram
installed
> > but only 2912584k usable because of an PCI-Card that supports only
32bit
> > addressing (appropriated-ram-crap). Could this be a problem? I try to
> > test again with 2GB Ram installed this evening and report if it gets
any
> > further...
> > 
> 
> Here are my working grub.conf entries for pv_ops dom0:
> 
> VGA text console:
> 
> title Fedora Xen pv_ops dom0-test (2.6.31-rc5)
>         root (hd0,0)
>         kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all
>         module /vmlinuz-2.6.31-rc5 ro root=/dev/mapper/vg_dom0test-lv01
>         module /initrd-2.6.31-rc5.img
> 
> 
> Serial console:
> 
> title Fedora Xen pv_ops dom0-test (2.6.31-rc5) / serial console
>         root (hd0,0)
>         kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all
com1=19200,8n1 console=com1
>         module /vmlinuz-2.6.31-rc5 ro root=/dev/mapper/vg_dom0test-lv01
console=hvc0 earlyprintk=xen
>         module /initrd-2.6.31-rc5.img
> 
> When using serial console with Xen you don''t need to specify
serial console stuff for the dom0 kernel..
> Also note the ''console=hvc0'' for dom0 kernel..
Isn''t the selection of hvc0 as main console supposed to work out
automatically? Looks like a bug if it isn''t. Anyway thanks for the
hint.
Will give it a try with a grub config adapted to yours later today.

Are there still problems related to specific CONFIG_ options which i
should deactivate in my build?

The xen related options i used (only frontends build as modules):

[mad@darkstar ~]$ grep XEN /boot/config-2.6.31-xen
CONFIG_XEN=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=8
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_DOM0_PCI=y
CONFIG_MICROCODE_XEN=y
CONFIG_PCI_XEN=y
CONFIG_XEN_BLKDEV_FRONTEND=m
CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_XEN_KBDDEV_FRONTEND=m
CONFIG_HVC_XEN=y
CONFIG_XEN_FBDEV_FRONTEND=m
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=m
CONFIG_XEN_S3=y
CONFIG_ACPI_PROCESSOR_XEN=y


Marc


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > > title XEN pv_ops dom0 with serial console
> > > root (hd0,1)
> > > kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> > > com1=9600,8n1 console=com1
> > > module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT
console=ttyS0,9600n8
> > > earlyprintk=xen 4
Do also add ''debug'' in the line. That will give some more
verbose output.
> > > module /boot/initramfs-2.6.31-xen
> > > 
> > > That''s the one i used to grab the log.
> > > 
> > > This is a i945 based box with an Core 2 Duo E8400 and 4GB Ram
installed
> > > but only 2912584k usable because of an PCI-Card that supports
only 32bit
> > > addressing (appropriated-ram-crap). Could this be a problem? I
try to
> > > test again with 2GB Ram installed this evening and report if it
gets any
> > > further...
If you specify ''mem=2GB'' it will limit how much memory Xen and
Linux see.
But you do use ''dom0_mem'' which limits the amount of memory
that
Dom0 would use for PCI drivers. So the 32-bit support shouldn''t be an
issue.

Looking at the serial line it also looks that your Linux is 32-bit? Did you
try the 64-bit?
> > > 
> > 
> > Here are my working grub.conf entries for pv_ops dom0:
> > 
> > VGA text console:
> > 
> > title Fedora Xen pv_ops dom0-test (2.6.31-rc5)
> >         root (hd0,0)
> >         kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all
> >         module /vmlinuz-2.6.31-rc5 ro
root=/dev/mapper/vg_dom0test-lv01
> >         module /initrd-2.6.31-rc5.img
> > 
> > 
> > Serial console:
> > 
> > title Fedora Xen pv_ops dom0-test (2.6.31-rc5) / serial console
> >         root (hd0,0)
> >         kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all
com1=19200,8n1 console=com1
> >         module /vmlinuz-2.6.31-rc5 ro
root=/dev/mapper/vg_dom0test-lv01 console=hvc0 earlyprintk=xen
> >         module /initrd-2.6.31-rc5.img
> > 
> > When using serial console with Xen you don''t need to specify
serial console stuff for the dom0 kernel..
> > Also note the ''console=hvc0'' for dom0 kernel..
> 
> Isn''t the selection of hvc0 as main console supposed to work out
> automatically? Looks like a bug if it isn''t. Anyway thanks for the
hint.
Yes. As long as CONFIG_HVC_XEN is compiled in (=y).
> Will give it a try with a grub config adapted to yours later today.
> 
> Are there still problems related to specific CONFIG_ options which i
> should deactivate in my build?
It should be all clear. But we haven''t tested all of them, so you might
be hitting
one that hasn''t been tested yet.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

. snip ..
> This is definitely a work-in-progress kernel.  I''d appreciate all
bug
> *and* success reports so I can get some idea of how many people are
> using this thing, and how often there are problems.  Patches gratefully
> accepted.
If you have an AMD machine with AGP and 4GB of memory a test of this
tree would be much appreciated. The test matrix is lacking those sorely
and if you see any weird issues like USB not working or ping not working
anymore, please do report it. Or as Jeremy said, also success reports.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sat, Sep 19, 2009 at 04:46:16PM +0200, Patrick Scharrenberg
wrote:
> Hi,
> 
> I just tested this branch and found that the system gets stuck on
> initializing the AHCI controller (AMD SB600).
Ooooh. Can you give some more details on the hardware flavor? CPU? Memory?
What happens if you use dom0_mem=max:512MB or pass in
''irqpoll'' to the
Linux kernel?
> Please find the error messages attached as png (unfortunately netconsole
> doesn''t work either)
What are the errors for netconsole? Is it not initializing the right ethernet
device?
Is the ethernet device compiled in? Can you paste what you have in your
grub config?

For example, this is what I have:

 kernel /xen.gz  com1=115200,8n1,0xcf00,0  console=com1,vga guest_loglvl=all 
dom0_mem=max:512MB
 module /vmlinuz debug
netconsole=6665@192.168.100.20/eth1,6666@192.168.100.16/00:25:11:18:A1:2E
console=tty console=xvc0 irqpoll loglevel=10

That particular machine didn''t come with a serial port. But the $14 PCI
serial
card works without fault.
> 
> When booting the same kernel natively without xen, all works fine
> (including netconsole).
OK. I need the kernel boot log to figure out what might be a potential problem.
Let''s try to get your netconsole working or your serial console? Does
this
machine even have a serial port? Maybe you can use that to capture the kernel
output?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Sep 21, 2009 at 11:18:27AM +0200, Marc - A. Dahlhaus [ Administration |
Westermann GmbH ] wrote:
> Am Montag, den 21.09.2009, 12:03 +0300 schrieb Pasi Kärkkäinen:
> > On Mon, Sep 21, 2009 at 10:49:03AM +0200, Marc - A. Dahlhaus [
Administration | Westermann GmbH ] wrote:
> --8<--
> > > title XEN pv_ops dom0
> > > root (hd0,1)
> > > kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> > > module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT
earlyprintk=xen 4
> > > module /boot/initramfs-2.6.31-xen
> > > 
> > > With this one i get no output from the kernel itself at all, i
think i
> > > have some things build as modules in my .config that needs to be
buildin
> > > for a working vga console (might be that i need to change the
> > > earlyprintk param to vga) but didn''t had the time to
hunt the problem
> > > down.
> > > 
> > > 
> > > title XEN pv_ops dom0 with serial console
> > > root (hd0,1)
> > > kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> > > com1=9600,8n1 console=com1
> > > module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT
console=ttyS0,9600n8
> > > earlyprintk=xen 4
> > > module /boot/initramfs-2.6.31-xen
> > > 
> > > That''s the one i used to grab the log.
> > > 
> > > This is a i945 based box with an Core 2 Duo E8400 and 4GB Ram
installed
> > > but only 2912584k usable because of an PCI-Card that supports
only 32bit
> > > addressing (appropriated-ram-crap). Could this be a problem? I
try to
> > > test again with 2GB Ram installed this evening and report if it
gets any
> > > further...
> > > 
> > 
> > Here are my working grub.conf entries for pv_ops dom0:
> > 
> > VGA text console:
> > 
> > title Fedora Xen pv_ops dom0-test (2.6.31-rc5)
> >         root (hd0,0)
> >         kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all
> >         module /vmlinuz-2.6.31-rc5 ro
root=/dev/mapper/vg_dom0test-lv01
> >         module /initrd-2.6.31-rc5.img
> > 
> > 
> > Serial console:
> > 
> > title Fedora Xen pv_ops dom0-test (2.6.31-rc5) / serial console
> >         root (hd0,0)
> >         kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all
com1=19200,8n1 console=com1
> >         module /vmlinuz-2.6.31-rc5 ro
root=/dev/mapper/vg_dom0test-lv01 console=hvc0 earlyprintk=xen
> >         module /initrd-2.6.31-rc5.img
> > 
> > When using serial console with Xen you don''t need to specify
serial console stuff for the dom0 kernel..
> > Also note the ''console=hvc0'' for dom0 kernel..
> 
> Isn''t the selection of hvc0 as main console supposed to work out
> automatically? Looks like a bug if it isn''t. Anyway thanks for the
hint.
> Will give it a try with a grub config adapted to yours later today.
> 
> Are there still problems related to specific CONFIG_ options which i
> should deactivate in my build?
> 
CONFIG_HIGHPTE=y makes dom0 kernel crash for me.. there''s a known race
and a hacky patch available, but no clean solution..

CONFIG_HIGHPTE=n is stable for me.

This is on 32bit PAE. 

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Sep 18, 2009 at 06:19:41PM -0700, Jeremy Fitzhardinge
wrote:
> 
> This is definitely a work-in-progress kernel.  I''d appreciate all
bug
> *and* success reports so I can get some idea of how many people are
> using this thing, and how often there are problems. 
>
I just compiled latest xen/master tree and it seems to work for me. 
I was able to run CentOS 5.3, Fedora 11 and Fedora 12 PV domUs without
problems.

My dom0 is currently Fedora 12 (rawhide) running Xen 3.4.1-4 rpms from
the distro.

dom0 kernel is 32bit PAE, CONFIG_HIGHPTE=n.

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/21/09 12:25, Pasi Kärkkäinen wrote:
> I just compiled latest xen/master tree and it seems to work for me. 
> I was able to run CentOS 5.3, Fedora 11 and Fedora 12 PV domUs without
> problems.
>
> My dom0 is currently Fedora 12 (rawhide) running Xen 3.4.1-4 rpms from
> the distro.
>
> dom0 kernel is 32bit PAE, CONFIG_HIGHPTE=n.
>   
Thanks for the feedback.  What hardware do you have?

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Sep 21, 2009 at 12:30:10PM -0700, Jeremy Fitzhardinge
wrote:
> On 09/21/09 12:25, Pasi Kärkkäinen wrote:
> > I just compiled latest xen/master tree and it seems to work for me. 
> > I was able to run CentOS 5.3, Fedora 11 and Fedora 12 PV domUs without
> > problems.
> >
> > My dom0 is currently Fedora 12 (rawhide) running Xen 3.4.1-4 rpms from
> > the distro.
> >
> > dom0 kernel is 32bit PAE, CONFIG_HIGHPTE=n.
> >   
> 
> Thanks for the feedback.  What hardware do you have?
>
This is the same old evil P4 with hyperthreading.. :)

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/21/09 12:50, Pasi Kärkkäinen wrote:
> This is the same old evil P4 with hyperthreading.. :)
>   
What kind of disk and net devices again?

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Sep 21, 2009 at 01:21:05PM -0700, Jeremy Fitzhardinge
wrote:
> On 09/21/09 12:50, Pasi Kärkkäinen wrote:
> > This is the same old evil P4 with hyperthreading.. :)
> >   
> 
> What kind of disk and net devices again?
> 
Intel ICH6 controller (ata_piix), IDE disk, and tg3 gigabit NICs: 
Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 11)

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/21/09 13:26, Pasi Kärkkäinen wrote:
> Intel ICH6 controller (ata_piix), IDE disk, and tg3 gigabit NICs: 
> Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev
11)
>   
So pre-AHCI?

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Sep 21, 2009 at 01:29:55PM -0700, Jeremy Fitzhardinge
wrote:
> On 09/21/09 13:26, Pasi Kärkkäinen wrote:
> > Intel ICH6 controller (ata_piix), IDE disk, and tg3 gigabit NICs: 
> > Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express
(rev 11)
> >   
> 
> So pre-AHCI?
> 
It has also AHCI controller onboard, but I don''t have currently any
SATA
disks plugged into the controller.. I think ahci module is loaded.

I might be able to test some AHCI/SATA setup later this week..

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk wrote:
> Ooooh. Can you give some more details on the hardware flavor? CPU? Memory?
Its an AMD system on an ASUS M2A-VM mainboard with an Athlon 64 X2 CPU
and 8GB of memory.
> What happens if you use dom0_mem=max:512MB or pass in
''irqpoll'' to the
> Linux kernel?
irqpoll does not change anything but the memory limit does the trick.
The system then boots fine, I attached the dmesg log.
Since the netconsole now also works, I assume some kind of IRQ problem
which is strengthened by the fact, that the keyboard didn''t work either
(for scrolling and searching for problems) without the memory limit option.
> OK. I need the kernel boot log to figure out what might be a potential
problem.
> Let''s try to get your netconsole working or your serial console?
Does this
> machine even have a serial port? Maybe you can use that to capture the
kernel
> output?
I attached the bootlog with the 512 memory-limit enabled and the xen dmesg.

Unfortunately I have to drive 200km to install a serial cable to the
system. So if there is an IRQ problem would the serial console really
help or would it suffer from the same irq problem?




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Sep 22, 2009 at 11:00:23AM +0200, Patrick Scharrenberg
wrote:
> Konrad Rzeszutek Wilk wrote:
> > Ooooh. Can you give some more details on the hardware flavor? CPU?
Memory?
> Its an AMD system on an ASUS M2A-VM mainboard with an Athlon 64 X2 CPU
> and 8GB of memory.
> 
> > What happens if you use dom0_mem=max:512MB or pass in
''irqpoll'' to the
> > Linux kernel?
> irqpoll does not change anything but the memory limit does the trick.
> The system then boots fine, I attached the dmesg log.
Patrick,

Lets try a couple of more options, if this isn''t too much of a trouble?
Mainly change the dom0_mem=max:512MB to dom0_mem=max:4GB and then
dom0_mem=max:6GB.

You ought to have no trouble at 4GB. 6GB will fail if this is a DMA related
issue.
> Since the netconsole now also works, I assume some kind of IRQ problem
> which is strengthened by the fact, that the keyboard didn''t work
either
> (for scrolling and searching for problems) without the memory limit option.
The log says:
> [    2.209240] ehci_hcd 0000:00:13.5: applying AMD SB600/SB700 USB freeze
workaround
That might be the reason your keyboard didn''t work. Unless your
keyboard is PS/2?
> 
> > OK. I need the kernel boot log to figure out what might be a potential
problem.
> > Let''s try to get your netconsole working or your serial
console? Does this
> > machine even have a serial port? Maybe you can use that to capture the
kernel
> > output?
> 
> I attached the bootlog with the 512 memory-limit enabled and the xen dmesg.
Thank you.
> 
> Unfortunately I have to drive 200km to install a serial cable to the
> system. So if there is an IRQ problem would the serial console really
> help or would it suffer from the same irq problem?
>
If it is an IRQ problem it would probably mess up the serial console right as it
is happening. But there are ways to make Xen not do IRQs for serial console.

Here is what I have:
xen.gz  com1=115200,8n1,0xcf00,0  console=com1,vga guest_loglvl=all 
dom0_mem=max:512MB

The last argument defines the IRQ. If it is set to ''0'' it will
poll the device instead of
depending on the PCI card to provide the output. The 0xcf00 is the non-standard
location
of the COM1 (most systems have that on 0x3f8 or so).

Looking at NewEgg for your motherboard it looks to _not_ have any serial output.
You would need to get a PCI Serial card for this thought.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,
> Lets try a couple of more options, if this isn''t too much of a
trouble?
That''s fine with me.
> You ought to have no trouble at 4GB. 6GB will fail if this is a DMA related
> issue.
It''s exactly as you said. 4GB works, 6GB fails.
> The log says:
>> [    2.209240] ehci_hcd 0000:00:13.5: applying AMD SB600/SB700 USB
freeze workaround
> That might be the reason your keyboard didn''t work. Unless your
keyboard is PS/2?

I also tested the same kernel on an SB700 machine and there it works
just fine including SATA and USB.
The keyboard is a remote management card and is indeed connected via USB.
> Looking at NewEgg for your motherboard it looks to _not_ have any serial
output.
> You would need to get a PCI Serial card for this thought.
The board has a serial port but no external connector.
I have a serial port bracket which I can install if it helps on.

Is there anything else I can try meanwhile?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Sep 22, 2009 at 10:08:25AM -0400, Konrad Rzeszutek Wilk
wrote:
> On Tue, Sep 22, 2009 at 11:00:23AM +0200, Patrick Scharrenberg wrote:
> > Konrad Rzeszutek Wilk wrote:
> > > Ooooh. Can you give some more details on the hardware flavor?
CPU? Memory?
> > Its an AMD system on an ASUS M2A-VM mainboard with an Athlon 64 X2 CPU
> > and 8GB of memory.
> > 
> > > What happens if you use dom0_mem=max:512MB or pass in
''irqpoll'' to the
> > > Linux kernel?
> > irqpoll does not change anything but the memory limit does the trick.
> > The system then boots fine, I attached the dmesg log.
> 
> Patrick,
> 
> Lets try a couple of more options, if this isn''t too much of a
trouble?
> Mainly change the dom0_mem=max:512MB to dom0_mem=max:4GB and then
dom0_mem=max:6GB.
> 
> You ought to have no trouble at 4GB. 6GB will fail if this is a DMA related
> issue.
Patrick,

I''ve gotten my hands on machine with SB700 and it exhibits similar
problems. The
SB700 AHCI controller stops working if I have more than 4GB in the machine.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Sep 23, 2009 at 09:37:55AM +0200, Patrick Scharrenberg
wrote:
> Hi,
> 
> > Lets try a couple of more options, if this isn''t too much of
a trouble?
> That''s fine with me.
> 
> > You ought to have no trouble at 4GB. 6GB will fail if this is a DMA
related
> > issue.
> It''s exactly as you said. 4GB works, 6GB fails.
> 
> > The log says:
> >> [    2.209240] ehci_hcd 0000:00:13.5: applying AMD SB600/SB700 USB
freeze workaround
> > That might be the reason your keyboard didn''t work. Unless
your
> keyboard is PS/2?
> 
> I also tested the same kernel on an SB700 machine and there it works
> just fine including SATA and USB.
Weird. I have a machine with SB700 and if it has more than 4GB it fails.
Your SB700 machine does have more than 4GB, right?
> The keyboard is a remote management card and is indeed connected via USB.
Nice. Just curious - what kind of card is this?
> 
> > Looking at NewEgg for your motherboard it looks to _not_ have any
serial output.
> > You would need to get a PCI Serial card for this thought.
> 
> The board has a serial port but no external connector.
> I have a serial port bracket which I can install if it helps on.
> 
> Is there anything else I can try meanwhile?
Let me start working on the machine I have here to troubleshoot.
I would suggest that in the meantime you use the dom0_mem workaround
and see if there are other problems besides the one you found?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Jeremy

are there plans to get any of the (non-bugfix) changes to upstream?
I think the 2.6.32 merge window will close very soon right?


Best regards,
    Christian


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/23/09 05:06, Konrad Rzeszutek Wilk wrote:
> I''ve gotten my hands on machine with SB700 and it exhibits similar
problems. The
> SB700 AHCI controller stops working if I have more than 4GB in the machine.
>   
You mean just with this kernel?  I presume it works OK normally.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Sep 23, 2009 at 12:22:32PM -0700, Jeremy Fitzhardinge
wrote:
> On 09/23/09 05:06, Konrad Rzeszutek Wilk wrote:
> > I''ve gotten my hands on machine with SB700 and it exhibits
similar problems. The
> > SB700 AHCI controller stops working if I have more than 4GB in the
machine.
> >   
> 
> You mean just with this kernel?  I presume it works OK normally.
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1514
has the details.

It looks that the calls to ioremap_nocache return an address
that is not synchronized with the physical address.

This is exhibited only when dom0 has more than 4GB, so if you do dom_mem=max:4GB
the machine boots succesfully.

If I boot the machine without Xen, with Linux seeing 8GB, 4GB, 6GB, or whatnot,
it
boots succesfully.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/23/09 12:32, Konrad Rzeszutek Wilk wrote:
> On Wed, Sep 23, 2009 at 12:22:32PM -0700, Jeremy Fitzhardinge wrote:
>   
>> On 09/23/09 05:06, Konrad Rzeszutek Wilk wrote:
>>     
>>> I''ve gotten my hands on machine with SB700 and it exhibits
similar problems. The
>>> SB700 AHCI controller stops working if I have more than 4GB in the
machine.
>>>   
>>>       
>> You mean just with this kernel?  I presume it works OK normally.
>>     
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1514
> has the details.
>
> It looks that the calls to ioremap_nocache return an address
> that is not synchronized with the physical address.
>   
Synchronized in what sense?  ioremap_* is all common code, so I would
expect it to fail or not fail.  I guess the possibilities are that that
the physaddr is getting truncated to 32-bits somewhere, or _PAGE_PCD is
getting masked out (either from __supported_pte_flags, or elsewhere in
the process).

But you mention in the bug that the appears to be problems with the AGP
aperture.  Do you get the "Aperture pointing to e820 RAM. Ignoring"
message when booting native?

We use the real BIOS-provided e820 map and then trim it according to the
provided memory size, so there should always be the same holes in the
E820 map that BIOS provides.

arch/x86/kernel/aperture_64.c has the test:

		if (!no_iommu &&
		    max_pfn > MAX_DMA32_PFN &&
		    !printed_gart_size_msg) {
			printk(KERN_ERR "you are using iommu with agp, but GART size is less
than 64M\n");
			printk(KERN_ERR "please increase GART size in your BIOS setup\n");
			printk(KERN_ERR "if BIOS doesn''t have that option, contact your
HW vendor!\n");
			printed_gart_size_msg = 1;
		}

I guess the "!no_iommu" clause is triggering because we have swiotlb
set, but I wonder if that''s specifically testing for the presence of a
GART IOMMU?

How does ioremap_nocache fit into this?  Is the mapping failing in some
way and causing this code to fail?  Or am I misunderstanding?
> This is exhibited only when dom0 has more than 4GB, so if you do
dom_mem=max:4GB
> the machine boots succesfully.
>   
The test above also tests "max_pfn > MAX_DMA32_PFN" which limiting
memory would avoid.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/23/09 06:16, Christian Tramnitz wrote:
> are there plans to get any of the (non-bugfix) changes to upstream?
> I think the 2.6.32 merge window will close very soon right?
No plans to put anything into .32.  We need to have a solid story about
how to handle IOAPIC setup before pushing the rest, I think.  I''ve just
restarted work on that, but I need to work out how to reconcile it with
the recent MSI work.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/23/09 13:09, Jeremy Fitzhardinge wrote:
> 		if (!no_iommu &&
> 		    max_pfn > MAX_DMA32_PFN &&
> 		    !printed_gart_size_msg) {
> 			printk(KERN_ERR "you are using iommu with agp, but GART size is
less than 64M\n");
> 			printk(KERN_ERR "please increase GART size in your BIOS
setup\n");
> 			printk(KERN_ERR "if BIOS doesn''t have that option, contact
your HW vendor!\n");
> 			printed_gart_size_msg = 1;
> 		}
>   
Oh, that''s the wrong error message, but the other one has similar
predicates.  Hm, but it also skips the test if (swiotlb &&
!valid_agp)...
> I guess the "!no_iommu" clause is triggering because we have
swiotlb
> set, but I wonder if that''s specifically testing for the presence
of a
> GART IOMMU?
>
> How does ioremap_nocache fit into this?  Is the mapping failing in some
> way and causing this code to fail?  Or am I misunderstanding?
>
>   
>> This is exhibited only when dom0 has more than 4GB, so if you do
dom_mem=max:4GB
>> the machine boots succesfully.
>>   
>>     
> The test above also tests "max_pfn > MAX_DMA32_PFN" which
limiting
> memory would avoid.
>   
    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Sep 23, 2009 at 01:30:44PM -0700, Jeremy Fitzhardinge
wrote:
> On 09/23/09 13:09, Jeremy Fitzhardinge wrote:
> > 		if (!no_iommu &&
> > 		    max_pfn > MAX_DMA32_PFN &&
> > 		    !printed_gart_size_msg) {
> > 			printk(KERN_ERR "you are using iommu with agp, but GART size
is less than 64M\n");
> > 			printk(KERN_ERR "please increase GART size in your BIOS
setup\n");
> > 			printk(KERN_ERR "if BIOS doesn''t have that option,
contact your HW vendor!\n");
> > 			printed_gart_size_msg = 1;
> > 		}
> >   
> 
> Oh, that''s the wrong error message, but the other one has similar
> predicates.  Hm, but it also skips the test if (swiotlb &&
!valid_agp)...
Right. We don''t set the swiotlb. The reason being if you do set it then
the original SWIOTLB kicks in.

The weird part is that the function you copied-n-pasted (gart_iommu_hole_init)
only detectes and allocates a buffer. It does not set the dma_ops at all.
Setting of the dma_ops is done via the gart_iommu_init() call which is done
much later. But with Xen-SWIOTLB already initialized, the gart_iommu_init()
quits right away.

So the kernel sets the dma_ops to the Xen SWIOTLB, and it
allocates an extra 64MB chunk of memory for the GART, which is not
used, and ... somehow all of the ioremap_nocache functions stop working
correctly. Maybe the ioremap_nocache does use some of that memory that
the gart_iommu_hole_init allocated?

With this patch, the GART is forcefully disabled, and the kernel boots fine
(with 6GB, 8GB, etc).

diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
index 6b76948..1101a9f 100644
--- a/arch/x86/kernel/pci-dma.c
+++ b/arch/x86/kernel/pci-dma.c
@@ -122,6 +122,8 @@ void __init pci_iommu_alloc(void)
 	 * The order of these functions is important for
 	 * fall-back/fail-over reasons
 	 */
+	xen_swiotlb_init();
+
 	gart_iommu_hole_init();
 
 	detect_calgary();
@@ -130,8 +132,6 @@ void __init pci_iommu_alloc(void)
 
 	amd_iommu_detect();
 
-	xen_swiotlb_init();
-
 	pci_swiotlb_init();
 }
 
diff --git a/arch/x86/xen/pci-swiotlb.c b/arch/x86/xen/pci-swiotlb.c
index 5e2c856..00f2260 100644
--- a/arch/x86/xen/pci-swiotlb.c
+++ b/arch/x86/xen/pci-swiotlb.c
@@ -43,6 +43,10 @@
 #include <xen/page.h>
 #include <xen/xen-ops.h>
 
+
+#include <linux/pci.h>
+#include <asm/gart.h>
+
 #define OFFSET(val,align) ((unsigned long)	\
 	                   ( (val) & ( (align) - 1)))
 
@@ -985,5 +989,9 @@ void __init xen_swiotlb_init(void)
 		xen_swiotlb_init_with_default_size(64 * (1<<20));	/* default to 64MB */
 		dma_ops = &xen_swiotlb_dma_ops;
 		iommu_detected = 1;
+#ifdef CONFIG_GART_IOMMU
+		gart_iommu_aperture_disabled = 1;
+#endif
+
 	}
 }


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/23/09 14:24, Konrad Rzeszutek Wilk wrote:
> The weird part is that the function you copied-n-pasted
(gart_iommu_hole_init)
> only detectes and allocates a buffer. It does not set the dma_ops at all.
> Setting of the dma_ops is done via the gart_iommu_init() call which is done
> much later. But with Xen-SWIOTLB already initialized, the gart_iommu_init()
> quits right away.
>   
Perhaps the fix is to make gart_iommu_hole_init() quit if iommu_detected
too?  Though it is called after xen_swiotlb_init()...
> So the kernel sets the dma_ops to the Xen SWIOTLB, and it
> allocates an extra 64MB chunk of memory for the GART, which is not
> used, and ... somehow all of the ioremap_nocache functions stop working
> correctly. Maybe the ioremap_nocache does use some of that memory that
> the gart_iommu_hole_init allocated?
>   
Can''t see how it would affect it.  ioremap allocates a new virtual
space
for the mapping and then just plugs in the pfns for the pages you want
to map.  They end up getting _PAGE_IOMAP set in the pte flags, which
causes the xen/mmu.c backend to use the address as-is (ie, as an mfn),
so the mapping will be constructed properly.  Well, that''s the theory;
but I''d expect we''d be seeing a lot more havok of ioremap is
either
mapping the wrong pages or using the wrong caching.
> With this patch, the GART is forcefully disabled, and the kernel boots fine
> (with 6GB, 8GB, etc).
>   
OK, I''ll put it in for now.  Will we have issues with other forms of
iommu?

Another thought, could we actually use the gart iommu instead of swiotlb
if it is available?  I think it leads to exactly the same set of issues
as extending normal swiotlb for Xen''s use (ie, inserting pfn->mfn
conversion into the correct places, and perhaps allocating the memory
properly).  Worth thinking about; it may shine light on better ways to
fix up swiotlb.

Thanks,
    J


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, 2009-09-24 at 04:13 +0800, Jeremy Fitzhardinge
wrote:
> On 09/23/09 06:16, Christian Tramnitz wrote:
> > are there plans to get any of the (non-bugfix) changes to upstream?
> > I think the 2.6.32 merge window will close very soon right?
> 
> No plans to put anything into .32.  We need to have a solid story about
> how to handle IOAPIC setup before pushing the rest, I think.  I''ve
just
> restarted work on that, but I need to work out how to reconcile it with
> the recent MSI work.
> 
>     J
What''s your current thinking on IOAPIC setup, still similar to that
new-routing branch? I''d like to know about it so the MSI can adapt.

And for a trap and emulation based MSI, the current logic may have
to be changed. Currently I''m still trying to find an elegant solution,
so I think you can do the IOAPIC work without much burden of MSI.

Thanks,
Qing

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jeremy, 
   After reading your branch new_interrupt_routing,  I think the main changes
about two hypercalls and their purposes maybe unnecessary.  I also implemented
the similar logic to remove ioapic changes from pv_ops dom0 and just re-used and
extended existing interfaces for that.  As to the new-introduced hypercall
PHYSDEVOP_route_gsi,  the existing hypercall PHYSDEVOP_map_pirq can cover its
functionality through some extension. And for the hypercall
PHYSDEVOP_acpi_irq_model, seems it is redundant and unncessary, because
irq_model can be parsed through the related acpi tables, so hypervisor and dom0
can reach the agreement automatically after parsing the tables.

The attached two patches are based on latest Xen and pv_ops_dom0, and they
should works for you with latest Xen and pv_ops dom0. Since they are only for
furuther discussion, so one dirty hack about probe_gsi also exists in the
current code.
Xiantao

-----Original Message-----
From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] 
Sent: Thursday, September 24, 2009 4:14 AM
To: Christian Tramnitz
Cc: xen-devel@lists.xensource.com; He, Qing; Zhang, Xiantao
Subject: Re: [Xen-devel] Re: Announcing xen/master: pvops git trees rearranged

On 09/23/09 06:16, Christian Tramnitz wrote:
> are there plans to get any of the (non-bugfix) changes to upstream?
> I think the 2.6.32 merge window will close very soon right?
No plans to put anything into .32.  We need to have a solid story about
how to handle IOAPIC setup before pushing the rest, I think.  I''ve just
restarted work on that, but I need to work out how to reconcile it with
the recent MSI work.

    J



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Sep 23, 2009 at 02:56:16PM -0700, Jeremy Fitzhardinge
wrote:
> On 09/23/09 14:24, Konrad Rzeszutek Wilk wrote:
> > The weird part is that the function you copied-n-pasted
(gart_iommu_hole_init)
> > only detectes and allocates a buffer. It does not set the dma_ops at
all.
> > Setting of the dma_ops is done via the gart_iommu_init() call which is
done
> > much later. But with Xen-SWIOTLB already initialized, the
gart_iommu_init()
> > quits right away.
> >   
> 
> Perhaps the fix is to make gart_iommu_hole_init() quit if iommu_detected
> too?  Though it is called after xen_swiotlb_init()...
That is a good idea too. That would avoid that ugly
#ifdef.
> 
> > So the kernel sets the dma_ops to the Xen SWIOTLB, and it
> > allocates an extra 64MB chunk of memory for the GART, which is not
> > used, and ... somehow all of the ioremap_nocache functions stop
working
> > correctly. Maybe the ioremap_nocache does use some of that memory that
> > the gart_iommu_hole_init allocated?
> >   
> Can''t see how it would affect it.  ioremap allocates a new virtual
space
> for the mapping and then just plugs in the pfns for the pages you want
> to map.  They end up getting _PAGE_IOMAP set in the pte flags, which
> causes the xen/mmu.c backend to use the address as-is (ie, as an mfn),
> so the mapping will be constructed properly.  Well, that''s the
theory;
> but I''d expect we''d be seeing a lot more havok of ioremap
is either
> mapping the wrong pages or using the wrong caching.
There was a lot of havoc - all of the PCI BARs were useless. Is the MFN
(from the pfn_to_mfn on this address) suppose to have a specific value?
> 
> > With this patch, the GART is forcefully disabled, and the kernel boots
fine
> > (with 6GB, 8GB, etc).
> >   
> 
> OK, I''ll put it in for now.  Will we have issues with other forms
of iommu?
There are three other types: AMD IOMMU (a real IOMMU), Intel''s IOMMU,
and the IBM''s Calgary IOMMU.

For all of those setting, no_iommu=1 should do the trick. But in reality
I need to double-check that:


diff --git a/arch/x86/xen/pci-swiotlb.c b/arch/x86/xen/pci-swiotlb.c
index 00f2260..390f698 100644
--- a/arch/x86/xen/pci-swiotlb.c
+++ b/arch/x86/xen/pci-swiotlb.c
@@ -989,6 +989,8 @@ void __init xen_swiotlb_init(void)
 		xen_swiotlb_init_with_default_size(64 * (1<<20));	/* default to 64MB */
 		dma_ops = &xen_swiotlb_dma_ops;
 		iommu_detected = 1;
+		no_iommu = 1; /* Forces the other IOMMU (if they are detected) to
+				to quit, rather than initialize. */
 #ifdef CONFIG_GART_IOMMU
 		gart_iommu_aperture_disabled = 1;
 #endif

<sigh>I think I need to rethink this swiotlb-Xen part. This is starting
to look like a hack.
> 
> Another thought, could we actually use the gart iommu instead of swiotlb
> if it is available?  I think it leads to exactly the same set of issues
> as extending normal swiotlb for Xen''s use (ie, inserting
pfn->mfn
> conversion into the correct places, and perhaps allocating the memory
> properly).  Worth thinking about; it may shine light on better ways to
> fix up swiotlb.
Yes! That was my next step - see if it is possible to use it and if so
extend it for that purpose (and without any ghastly #ifdef).

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jeremy Fitzhardinge wrote:
> No plans to put anything into .32.  We need to have a solid story about
> how to handle IOAPIC setup before pushing the rest, I think.  I''ve
just
> restarted work on that, but I need to work out how to reconcile it with
> the recent MSI work.
> 
>     J
Yes, I guess this increases the likelihood of things being merged...

On another note, what about xen-tip/next? Wasn''t there pciback (or was 
it pcifront) already incorporated (or was that wishful thinking)? I 
haven''t seen any changes there for month, is it not updated anymore?


Best regards,
    Christian


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

2009/9/24 Christian Tramnitz <chris.ace@gmx.net>:
> Jeremy Fitzhardinge wrote:
>>
>> No plans to put anything into .32.  We need to have a solid story about
>> how to handle IOAPIC setup before pushing the rest, I think.
>
> Yes, I guess this increases the likelihood of things being merged...
One potentially disturbing snippet on LWN today ...

http://lwn.net/Articles/353852/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/24/09 05:44, Konrad Rzeszutek Wilk wrote:
> There was a lot of havoc - all of the PCI BARs were useless. Is the MFN
> (from the pfn_to_mfn on this address) suppose to have a specific value?
>   
Not sure.  pfn_to_mfn is never supposed to happen on ioremap phys addrs,
because of _PAGE_IOMAP in the pte.  Its probably worth checking that
_PAGE_IOMAP is actually getting set.
> For all of those setting, no_iommu=1 should do the trick. But in reality
> I need to double-check that:
>
>
> diff --git a/arch/x86/xen/pci-swiotlb.c b/arch/x86/xen/pci-swiotlb.c
> index 00f2260..390f698 100644
> --- a/arch/x86/xen/pci-swiotlb.c
> +++ b/arch/x86/xen/pci-swiotlb.c
> @@ -989,6 +989,8 @@ void __init xen_swiotlb_init(void)
>  		xen_swiotlb_init_with_default_size(64 * (1<<20));	/* default to
64MB */
>  		dma_ops = &xen_swiotlb_dma_ops;
>  		iommu_detected = 1;
> +		no_iommu = 1; /* Forces the other IOMMU (if they are detected) to
> +				to quit, rather than initialize. */
>  #ifdef CONFIG_GART_IOMMU
>  		gart_iommu_aperture_disabled = 1;
>  #endif
>
> <sigh>I think I need to rethink this swiotlb-Xen part. This is
starting
> to look like a hack.
>   
It isn''t great.  We need a way to either layer or arbitrate between
these different address translation mechanisms.
>> Another thought, could we actually use the gart iommu instead of
swiotlb
>> if it is available?  I think it leads to exactly the same set of issues
>> as extending normal swiotlb for Xen''s use (ie, inserting
pfn->mfn
>> conversion into the correct places, and perhaps allocating the memory
>> properly).  Worth thinking about; it may shine light on better ways to
>> fix up swiotlb.
>>     
> Yes! That was my next step - see if it is possible to use it and if so
> extend it for that purpose (and without any ghastly #ifdef).
>   
Good.

    J


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I don''t believe that, because the paravirt_ops provides a great
virtualization job without the need for a special CPU. There are too many
people around the world in this situation...I don''t have too much CPUs
with
VMX and in all my dom0s that have the VMX flag, I simple disable it in my
BIOS.

The mainline Linux with the pv_ops dom0 will arrive some day... for sure!

Cheers,
Thiago

2009/9/24 Andy Burns <xen.lists@burns.me.uk>
> 2009/9/24 Christian Tramnitz <chris.ace@gmx.net>:
>
> > Jeremy Fitzhardinge wrote:
> >>
> >> No plans to put anything into .32.  We need to have a solid story
about
> >> how to handle IOAPIC setup before pushing the rest, I think.
> >
> > Yes, I guess this increases the likelihood of things being merged...
>
> One potentially disturbing snippet on LWN today ...
>
> http://lwn.net/Articles/353852/
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Konrad,
> With this patch, the GART is forcefully disabled, and the kernel boots fine
> (with 6GB, 8GB, etc).
I tried your patch and my SB600 system also boots fine. Great. Thanks!
Now I''m going to do some tests on this kernel the next days and will
report.

Patrick

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

And what about the OpenSolaris, NetBSD and FreeBSD paravirtualized kernels?!Who
needs the full virtualization?!

:-D

Thiago

2009/9/24 Thiago Camargo Martins Cordeiro <thiagocmartinsc@gmail.com>
> I don''t believe that, because the paravirt_ops provides a great
> virtualization job without the need for a special CPU. There are too many
> people around the world in this situation...I don''t have too much
CPUs
> with VMX and in all my dom0s that have the VMX flag, I simple disable it in
> my BIOS.
>
> The mainline Linux with the pv_ops dom0 will arrive some day... for sure!
>
> Cheers,
> Thiago
>
> 2009/9/24 Andy Burns <xen.lists@burns.me.uk>
>
> 2009/9/24 Christian Tramnitz <chris.ace@gmx.net>:
>>
>> > Jeremy Fitzhardinge wrote:
>> >>
>> >> No plans to put anything into .32.  We need to have a solid
story about
>> >> how to handle IOAPIC setup before pushing the rest, I think.
>> >
>> > Yes, I guess this increases the likelihood of things being
merged...
>>
>> One potentially disturbing snippet on LWN today ...
>>
>> http://lwn.net/Articles/353852/
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/23/09 20:17, Qing He wrote:
> What''s your current thinking on IOAPIC setup, still similar to
that
> new-routing branch? I''d like to know about it so the MSI can
adapt.
>   
Yes, that''s my goal.
> And for a trap and emulation based MSI, the current logic may have
> to be changed. Currently I''m still trying to find an elegant
solution,
> so I think you can do the IOAPIC work without much burden of MSI.
>   
OK.  I still need to resolve some conflicts between new-routing and your
MSI-related changes though.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/24/09 06:20, Christian Tramnitz wrote:
> Yes, I guess this increases the likelihood of things being merged...
Right.
> On another note, what about xen-tip/next? Wasn''t there pciback (or
was
> it pcifront) already incorporated (or was that wishful thinking)?
There are a lot of pieces needed for pci back/front in the tree already,
but it just needs some work to glue them all together.  I think Konrad
is going to look at it shortly.
> I haven''t seen any changes there for month, is it not updated
anymore?
No, xen-tip has been dead for a while.   All the action is on xen/master.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/24/09 10:47, Andy Burns wrote:
> One potentially disturbing snippet on LWN today ...
>
> http://lwn.net/Articles/353852/
>   
I wouldn''t read too much into that.  At some point in the (distant)
future we may decide that pvops doesn''t have enough users to justify
supporting it any more and pull it out, like any other kernel feature
(or perhaps more specifically, nobody''s interested in maintaining it
any
more).  But we''re a long way from there.  After all, people are still
actively maintaining 80386 support...

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Sep 24, 2009 at 11:23:16AM -0700, Jeremy Fitzhardinge
wrote:
> On 09/24/09 05:44, Konrad Rzeszutek Wilk wrote:
> > There was a lot of havoc - all of the PCI BARs were useless. Is the
MFN
> > (from the pfn_to_mfn on this address) suppose to have a specific
value?
> >   
> 
> Not sure.  pfn_to_mfn is never supposed to happen on ioremap phys addrs,
> because of _PAGE_IOMAP in the pte.  Its probably worth checking that
> _PAGE_IOMAP is actually getting set.
<sigh> I found the problem. I did not power off the machine after doing
a non-Xen boot where the GART was used. I am not entirely sure what
the GART was doing, but pretty much all writew/readw were not doing/going
where they were suppose to.
> 
> > For all of those setting, no_iommu=1 should do the trick. But in
reality
> > I need to double-check that:
> >
> >
> > diff --git a/arch/x86/xen/pci-swiotlb.c b/arch/x86/xen/pci-swiotlb.c
> > index 00f2260..390f698 100644
> > --- a/arch/x86/xen/pci-swiotlb.c
> > +++ b/arch/x86/xen/pci-swiotlb.c
> > @@ -989,6 +989,8 @@ void __init xen_swiotlb_init(void)
> >  		xen_swiotlb_init_with_default_size(64 * (1<<20));	/* default
to 64MB */
> >  		dma_ops = &xen_swiotlb_dma_ops;
> >  		iommu_detected = 1;
> > +		no_iommu = 1; /* Forces the other IOMMU (if they are detected) to
> > +				to quit, rather than initialize. */
> >  #ifdef CONFIG_GART_IOMMU
> >  		gart_iommu_aperture_disabled = 1;
> >  #endif
> >
> > <sigh>I think I need to rethink this swiotlb-Xen part. This is
starting
> > to look like a hack.
> >   
> 
> It isn''t great.  We need a way to either layer or arbitrate
between
> these different address translation mechanisms.
I am sending another patch that I think is more nicer, and it takes care of the
other IOMMUs.
> 
> >> Another thought, could we actually use the gart iommu instead of
swiotlb
> >> if it is available?  I think it leads to exactly the same set of
issues
> >> as extending normal swiotlb for Xen''s use (ie, inserting
pfn->mfn
> >> conversion into the correct places, and perhaps allocating the
memory
> >> properly).  Worth thinking about; it may shine light on better
ways to
> >> fix up swiotlb.
> >>     
> > Yes! That was my next step - see if it is possible to use it and if so
> > extend it for that purpose (and without any ghastly #ifdef).
> >   
> 
> Good.
Still thinking about this ..

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Inline the pv_ops dom0 patch. 

diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
index 70f5ea9..b45dedf 100644
--- a/arch/x86/include/asm/io_apic.h
+++ b/arch/x86/include/asm/io_apic.h
@@ -188,9 +188,5 @@ static inline void probe_nr_irqs_gsi(void)	{ }
 #endif
 
 void xen_io_apic_init(void);
-unsigned int xen_io_apic_read(unsigned apic, unsigned reg);
-void xen_io_apic_write(unsigned int apic,
-		       unsigned int reg, unsigned int value);
-
 
 #endif /* _ASM_X86_IO_APIC_H */
diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c
index d47c54f..783678c 100644
--- a/arch/x86/kernel/acpi/boot.c
+++ b/arch/x86/kernel/acpi/boot.c
@@ -954,6 +954,9 @@ int __init acpi_probe_gsi(void)
 	int idx;
 	int gsi;
 	int max_gsi = 0;
+    
+	if (xen_initial_domain()) /* Dirty hack!! */
+        return 256;
 
 	if (acpi_disabled)
 		return 0;
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 77151ce..0c87499 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -386,9 +386,6 @@ static inline unsigned int io_apic_read(unsigned int apic,
unsigned int reg)
 {
 	struct io_apic __iomem *io_apic;
 
-	if (xen_initial_domain())
-		return xen_io_apic_read(apic, reg);
-
 	io_apic = io_apic_base(apic);
 	writel(reg, &io_apic->index);
 	return readl(&io_apic->data);
@@ -398,11 +395,6 @@ static inline void io_apic_write(unsigned int apic,
unsigned int reg, unsigned i
 {
 	struct io_apic __iomem *io_apic;
 
-	if (xen_initial_domain()) {
-		xen_io_apic_write(apic, reg, value);
-		return;
-	}
-
 	io_apic = io_apic_base(apic);
 	writel(reg, &io_apic->index);
 	writel(value, &io_apic->data);
@@ -418,11 +410,6 @@ static inline void io_apic_modify(unsigned int apic,
unsigned int reg, unsigned
 {
 	struct io_apic __iomem *io_apic;
 
-	if (xen_initial_domain()) {
-		xen_io_apic_write(apic, reg, value);
-		return;
-	}
-
 	io_apic = io_apic_base(apic);
 
 	if (sis_apic_bug)
@@ -3162,9 +3149,6 @@ static int __init ioapic_init_sysfs(void)
 	struct sys_device * dev;
 	int i, size, error;
 
-	if (xen_initial_domain())
-		return 0;
-
 	error = sysdev_class_register(&ioapic_sysdev_class);
 	if (error)
 		return error;
@@ -4183,11 +4167,6 @@ void __init ioapic_init_mappings(void)
 	struct resource *ioapic_res;
 	int i;
 
-	if (xen_initial_domain()) {
-		xen_io_apic_init();
-		return;
-	}
-
 	ioapic_res = ioapic_setup_resources();
 	for (i = 0; i < nr_ioapics; i++) {
 		if (smp_found_config) {
diff --git a/arch/x86/xen/apic.c b/arch/x86/xen/apic.c
index ee0db39..805ce70 100644
--- a/arch/x86/xen/apic.c
+++ b/arch/x86/xen/apic.c
@@ -17,31 +17,6 @@ void __init xen_io_apic_init(void)
 	enable_IO_APIC();
 }
 
-unsigned int xen_io_apic_read(unsigned apic, unsigned reg)
-{
-	struct physdev_apic apic_op;
-	int ret;
-
-	apic_op.apic_physbase = mp_ioapics[apic].apicaddr;
-	apic_op.reg = reg;
-	ret = HYPERVISOR_physdev_op(PHYSDEVOP_apic_read, &apic_op);
-	if (ret)
-		BUG();
-	return apic_op.value;
-}
-
-
-void xen_io_apic_write(unsigned int apic, unsigned int reg, unsigned int value)
-{
-	struct physdev_apic apic_op;
-
-	apic_op.apic_physbase = mp_ioapics[apic].apicaddr;
-	apic_op.reg = reg;
-	apic_op.value = value;
-	if (HYPERVISOR_physdev_op(PHYSDEVOP_apic_write, &apic_op))
-		BUG();
-}
-
 void xen_init_apic(void)
 {
 	if (!xen_initial_domain())
diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index a654a49..0ca0c2c 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -1938,6 +1938,8 @@ __init pgd_t *xen_setup_kernel_pagetable(pgd_t *pgd,
 }
 #endif	/* CONFIG_X86_64 */
 
+static unsigned char dummy_ioapic_mapping[PAGE_SIZE] __page_aligned_bss;
+
 static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot)
 {
 	pte_t pte;
@@ -1973,7 +1975,7 @@ static void xen_set_fixmap(unsigned idx, phys_addr_t phys,
pgprot_t prot)
 		 * We just don''t map the IO APIC - all access is via
 		 * hypercalls.  Keep the address in the pte for reference.
 		 */
-		pte = pfn_pte(phys, PAGE_NONE);
+		pte = __pte(__pa(dummy_ioapic_mapping) | __PAGE_KERNEL);
 		break;
 #endif
 	case FIX_PARAVIRT_BOOTMAP:
diff --git a/arch/x86/xen/pci.c b/arch/x86/xen/pci.c
index 44d91ad..9ec0d08 100644
--- a/arch/x86/xen/pci.c
+++ b/arch/x86/xen/pci.c
@@ -15,52 +15,38 @@
 
 #include "xen-ops.h"
 
-static void xen_set_io_apic_routing(int irq, int trigger, int polarity)
-{
-	int ioapic, ioapic_pin;
-	int vector, gsi;
-	struct IO_APIC_route_entry entry;
-
-	gsi = xen_gsi_from_irq(irq);
-	vector = xen_vector_from_irq(irq);
-
-	ioapic = mp_find_ioapic(gsi);
-	if (ioapic == -1) {
-		printk(KERN_WARNING "xen_set_ioapic_routing: irq %d gsi %d ioapic
%d\n",
-			irq, gsi, ioapic);
-		return;
-	}
-
-	ioapic_pin = mp_find_ioapic_pin(ioapic, gsi);
-
-	printk(KERN_INFO "xen_set_ioapic_routing: irq %d gsi %d vector %d ioapic
%d pin %d triggering %d polarity %d\n",
-		irq, gsi, vector, ioapic, ioapic_pin, trigger, polarity);
-
-	setup_ioapic_entry(ioapic, -1, &entry, ~0, trigger, polarity, vector,
-			   ioapic_pin);
-	ioapic_write_entry(ioapic, ioapic_pin, entry);
-}
-
 int xen_register_gsi(u32 gsi, int triggering, int polarity)
 {
-	int irq;
+	int rc, irq;
+	struct physdev_map_pirq map_irq;
+	domid_t domid = DOMID_SELF;
 
 	if (!xen_domain())
 		return -1;
 
-	printk(KERN_DEBUG "xen: registering gsi %u triggering %d polarity
%d\n",
-	       gsi, triggering, polarity);
+	BUG_ON(gsi >= (1 << 16));
 
 	irq = xen_allocate_pirq(gsi, (triggering == ACPI_EDGE_SENSITIVE)
-				     ? "ioapic-edge" : "ioapic-level");
+					 ? "ioapic-edge" : "ioapic-level");
 
 	printk(KERN_DEBUG "xen: --> irq=%d\n", irq);
 
-	if (irq >= 0)
-		xen_set_io_apic_routing(irq,
-					triggering == ACPI_EDGE_SENSITIVE ? 0 : 1,
-					polarity == ACPI_ACTIVE_HIGH ? 0 : 1);
-
+	triggering = (triggering == ACPI_EDGE_SENSITIVE ? 0 : 1);
+	polarity =  (polarity == ACPI_ACTIVE_HIGH ? 0 : 1);
+	printk( "xen: registering gsi %u triggering %d polarity %d\n",
+		   gsi, triggering, polarity);
+
+	memset(&map_irq, 0, sizeof(map_irq));
+	map_irq.domid = domid;
+	map_irq.type = MAP_PIRQ_TYPE_GSI;
+	map_irq.index = gsi | (triggering << 16) | (polarity << 24);
+	map_irq.pirq = irq;
+
+	rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);
+	if (rc) {
+		printk(KERN_WARNING "xen map irq failed %d\n", rc);
+		irq = -1;
+	}
 	return irq;
 }
 
diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index 68c287c..52e4294 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -74,7 +74,7 @@ enum xen_irq_type {
  * event channel - irq->event channel mapping
  * cpu - cpu this event channel is bound to
  * index - type-specific information:
- *    PIRQ - vector, with MSB being "needs EIO"
+ *    PIRQ - with MSB being "needs EIO"
  *    VIRQ - virq number
  *    IPI - IPI vector
  *    EVTCHN -
@@ -90,7 +90,6 @@ struct irq_info
 		enum ipi_vector ipi;
 		struct {
 			unsigned short nr;
-			unsigned char vector;
 			unsigned char flags;
 		} pirq;
 	} u;
@@ -144,11 +143,10 @@ static struct irq_info mk_virq_info(unsigned short evtchn,
unsigned short virq)
 			.cpu = 0, .u.virq = virq };
 }
 
-static struct irq_info mk_pirq_info(unsigned short evtchn,
-				    unsigned short pirq, unsigned short vector)
+static struct irq_info mk_pirq_info(unsigned short evtchn, unsigned short pirq)
 {
 	return (struct irq_info) { .type = IRQT_PIRQ, .evtchn = evtchn,
-			.cpu = 0, .u.pirq = { .nr = pirq, .vector = vector } };
+			.cpu = 0, .u.pirq = { .nr = pirq } };
 }
 
 /*
@@ -170,16 +168,6 @@ unsigned irq_from_evtchn(unsigned int evtchn)
 }
 EXPORT_SYMBOL_GPL(irq_from_evtchn);
 
-static enum ipi_vector ipi_from_irq(unsigned irq)
-{
-	struct irq_info *info = info_for_irq(irq);
-
-	BUG_ON(info == NULL);
-	BUG_ON(info->type != IRQT_IPI);
-
-	return info->u.ipi;
-}
-
 static unsigned virq_from_irq(unsigned irq)
 {
 	struct irq_info *info = info_for_irq(irq);
@@ -200,16 +188,6 @@ static unsigned gsi_from_irq(unsigned irq)
 	return info->u.pirq.nr;
 }
 
-static unsigned vector_from_irq(unsigned irq)
-{
-	struct irq_info *info = info_for_irq(irq);
-
-	BUG_ON(info == NULL);
-	BUG_ON(info->type != IRQT_PIRQ);
-
-	return info->u.pirq.vector;
-}
-
 static enum xen_irq_type type_from_irq(unsigned irq)
 {
 	return info_for_irq(irq)->type;
@@ -539,14 +517,13 @@ static int find_irq_by_gsi(unsigned gsi)
 }
 
 /*
- * Allocate a physical irq, along with a vector.  We don''t assign an
+ * Allocate a physical irq.  We don''t assign an
  * event channel until the irq actually started up.  Return an
  * existing irq if we''ve already got one for the gsi.
  */
 int xen_allocate_pirq(unsigned gsi, char *name)
 {
 	int irq;
-	struct physdev_irq irq_op;
 
 	spin_lock(&irq_mapping_update_lock);
 
@@ -567,14 +544,7 @@ int xen_allocate_pirq(unsigned gsi, char *name)
 	set_irq_chip_and_handler_name(irq, &xen_pirq_chip,
 				      handle_level_irq, name);
 
-	irq_op.irq = gsi;
-	if (HYPERVISOR_physdev_op(PHYSDEVOP_alloc_irq_vector, &irq_op)) {
-		dynamic_irq_cleanup(irq);
-		irq = -ENOSPC;
- 		goto out;
- 	}
-
-	irq_info[irq] = mk_pirq_info(0, gsi, irq_op.vector);
+	irq_info[irq] = mk_pirq_info(0, gsi);
 out:
 	spin_unlock(&irq_mapping_update_lock);
 	return irq;
@@ -657,7 +627,7 @@ int xen_create_msi_irq(struct pci_dev *dev, struct msi_desc
*msidesc, int type)
 		goto out;
 	}
 
-	irq_info[irq] = mk_pirq_info(0, map_irq.pirq, map_irq.index);
+	irq_info[irq] = mk_pirq_info(0, map_irq.pirq);
 	set_irq_chip_and_handler_name(irq, &xen_pirq_chip,
 			handle_level_irq,
 			(type == PCI_CAP_ID_MSIX) ? "msi-x":"msi");
@@ -668,11 +638,6 @@ out:
 }
 #endif
 
-int xen_vector_from_irq(unsigned irq)
-{
-	return vector_from_irq(irq);
-}
-
 int xen_gsi_from_irq(unsigned irq)
 {
 	return gsi_from_irq(irq);
@@ -752,6 +717,15 @@ static int bind_interdomain_evtchn_to_irq(unsigned int
remote_domain,
         return err ? : bind_evtchn_to_irq(bind_interdomain.local_port);
 }
 
+static enum ipi_vector ipi_from_irq(unsigned irq)
+{
+	struct irq_info *info = info_for_irq(irq);
+
+	BUG_ON(info == NULL);
+	BUG_ON(info->type != IRQT_IPI);
+
+	return info->u.ipi;
+}
 
 int bind_virq_to_irq(unsigned int virq, unsigned int cpu)
 {
 

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Xiantao
Sent: Thursday, September 24, 2009 4:15 PM
To: Jeremy Fitzhardinge; Christian Tramnitz
Cc: xen-devel@lists.xensource.com; He, Qing
Subject: RE: [Xen-devel] Re: Announcing xen/master: pvops git trees rearranged

 Jeremy, 
   After reading your branch new_interrupt_routing,  I think the main changes
about two hypercalls and their purposes maybe unnecessary.  I also implemented
the similar logic to remove ioapic changes from pv_ops dom0 and just re-used and
extended existing interfaces for that.  As to the new-introduced hypercall
PHYSDEVOP_route_gsi,  the existing hypercall PHYSDEVOP_map_pirq can cover its
functionality through some extension. And for the hypercall
PHYSDEVOP_acpi_irq_model, seems it is redundant and unncessary, because
irq_model can be parsed through the related acpi tables, so hypervisor and dom0
can reach the agreement automatically after parsing the tables.

The attached two patches are based on latest Xen and pv_ops_dom0, and they
should works for you with latest Xen and pv_ops dom0. Since they are only for
furuther discussion, so one dirty hack about probe_gsi also exists in the
current code.
Xiantao

-----Original Message-----
From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] 
Sent: Thursday, September 24, 2009 4:14 AM
To: Christian Tramnitz
Cc: xen-devel@lists.xensource.com; He, Qing; Zhang, Xiantao
Subject: Re: [Xen-devel] Re: Announcing xen/master: pvops git trees rearranged

On 09/23/09 06:16, Christian Tramnitz wrote:
> are there plans to get any of the (non-bugfix) changes to upstream?
> I think the 2.6.32 merge window will close very soon right?
No plans to put anything into .32.  We need to have a solid story about
how to handle IOAPIC setup before pushing the rest, I think.  I''ve just
restarted work on that, but I need to work out how to reconcile it with
the recent MSI work.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Sep 18, 2009 at 06:19:41PM -0700, Jeremy Fitzhardinge
wrote:
> 
> This is definitely a work-in-progress kernel.  I''d appreciate all
bug
> *and* success reports so I can get some idea of how many people are
> using this thing, and how often there are problems.  Patches gratefully
> accepted.
> 
I just tried the latest pv_ops dom0 git tree (11 Oct 2009) on x86_64 AHCI box.

The good news is that the dom0 kernel boots up, but there are some error
messages.

Using the default options (modeset) the VGA console doesn''t work, it
goes blank (display says "power save") in the beginning of dom0 kernel
boot:
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.1-2009-10-11.txt

When using "nomodeset" dom0 kernel option the VGA text console works:
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.1-nomodeset-2009-10-11.txt

Both of the above logs have really long errors/tracebacks about:
[ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ]

The latest Fedora 12/rawhide myoung xendom0 kernel has the same errors:
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.3-1.2.71.xendom0.fc12.x86_64.txt

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sun, Oct 11, 2009 at 06:39:00PM +0300, Pasi Kärkkäinen
wrote:
> On Fri, Sep 18, 2009 at 06:19:41PM -0700, Jeremy Fitzhardinge wrote:
> > 
> > This is definitely a work-in-progress kernel.  I''d appreciate
all bug
> > *and* success reports so I can get some idea of how many people are
> > using this thing, and how often there are problems.  Patches
gratefully
> > accepted.
> > 
> 
> I just tried the latest pv_ops dom0 git tree (11 Oct 2009) on x86_64 AHCI
box.
> 
> The good news is that the dom0 kernel boots up, but there are some error
> messages.
> 
> Using the default options (modeset) the VGA console doesn''t work,
it
> goes blank (display says "power save") in the beginning of dom0
kernel boot:
>
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.1-2009-10-11.txt
This line:
[drm:radeon_ring_test] *ERROR* radeon: ring test failed
(sracth(0x15E4)=0xCAFEDEAD)

Is a pretty good pointer at what the fault is. If you look at git commit
93e7c3850b8431e19c9cba91413066bfd2360671 you will see the band-aid Jeremy added.
It looks though as if not all of the radeon drivers allocate their ring buffer
memory via
drm_sg_alloc calls thought. Not sure how the r100 (and the corresponding X
driver) does it.
The long/erro traceback about the HARDIRQ is a red-herring in this case.

Here is a couple of things that I would like you to try, if you can:

1). Pass in ''drm.debug=255'' and send the output. It should
have tons of extra
output. 

2). Send in the Xorg.log (or whatever output the program in the userland that
 starts the modesetting produces).  I don''t have much knowledge in how
modesetting works,
 so this might require some digging.

2). When you boot the kernel without Xen, you don''t see this error,
right?

3) What does lspci -vvv output show for the video card in question?


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Oct 12, 2009 at 04:02:48PM -0400, Konrad Rzeszutek Wilk
wrote:
> On Sun, Oct 11, 2009 at 06:39:00PM +0300, Pasi Kärkkäinen wrote:
> > On Fri, Sep 18, 2009 at 06:19:41PM -0700, Jeremy Fitzhardinge wrote:
> > > 
> > > This is definitely a work-in-progress kernel.  I''d
appreciate all bug
> > > *and* success reports so I can get some idea of how many people
are
> > > using this thing, and how often there are problems.  Patches
gratefully
> > > accepted.
> > > 
> > 
> > I just tried the latest pv_ops dom0 git tree (11 Oct 2009) on x86_64
AHCI box.
> > 
> > The good news is that the dom0 kernel boots up, but there are some
error
> > messages.
> > 
> > Using the default options (modeset) the VGA console doesn''t
work, it
> > goes blank (display says "power save") in the beginning of
dom0 kernel boot:
> >
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.1-2009-10-11.txt
> 
> This line:
> [drm:radeon_ring_test] *ERROR* radeon: ring test failed
(sracth(0x15E4)=0xCAFEDEAD)
> 
> Is a pretty good pointer at what the fault is. If you look at git commit
> 93e7c3850b8431e19c9cba91413066bfd2360671 you will see the band-aid Jeremy
added.
> It looks though as if not all of the radeon drivers allocate their ring
buffer memory via
> drm_sg_alloc calls thought. Not sure how the r100 (and the corresponding X
driver) does it.
> The long/erro traceback about the HARDIRQ is a red-herring in this case.
> 
> Here is a couple of things that I would like you to try, if you can:
> 
Sure.
> 1). Pass in ''drm.debug=255'' and send the output. It
should have tons of extra
> output. 
> 
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.1-2009-10-14-drmdebug.txt

Unknown boot option `drm.debug=255'': ignoring

Also it seems this error:
[ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ]

.. comes from the USB stuff.

I tried Fedora xendom0 kernel rpm, and the kernel graphics modesetting
seems to work there! (Fedora kernel contains newer graphics/drm drivers).

But the same USB related error is there with the fedora kernel:
[ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ]

http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.3-1.2.71.xendom0.fc12.x86_64-2009-10-14.txt

> 2). Send in the Xorg.log (or whatever output the program in the userland
that
>  starts the modesetting produces).  I don''t have much knowledge in
how modesetting works,
>  so this might require some digging.
> 
Hmm.. yeah, I''m not sure either which is the first program setting up
graphics mode using kernel modesetting (KMS) in Fedora.. 

I extracted the initrd image and checked the ''init'' script:

echo "Loading drm module"
modprobe -q drm
echo "Loading ttm module"
modprobe -q ttm
echo "Loading radeon module"
modprobe -q radeon
/lib/udev/console_init tty0
plymouth --show-splash

So I guess plymouth is asking for a graphics mode..

> 2). When you boot the kernel without Xen, you don''t see this
error, right?
> 
Yeah, it works OK on baremetal without Xen.
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.1-baremetal.txt
> 3) What does lspci -vvv output show for the video card in question?
> 
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/lspci-vvv-2.6.31.1-baremetal.txt
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/lspci-vvv-2.6.31.1-dom0.txt

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Oct 15, 2009 at 12:14:15AM +0300, Pasi Kärkkäinen
wrote:
> On Mon, Oct 12, 2009 at 04:02:48PM -0400, Konrad Rzeszutek Wilk wrote:
> > On Sun, Oct 11, 2009 at 06:39:00PM +0300, Pasi Kärkkäinen wrote:
> > > On Fri, Sep 18, 2009 at 06:19:41PM -0700, Jeremy Fitzhardinge
wrote:
> > > > 
> > > > This is definitely a work-in-progress kernel.  I''d
appreciate all bug
> > > > *and* success reports so I can get some idea of how many
people are
> > > > using this thing, and how often there are problems.  Patches
gratefully
> > > > accepted.
> > > > 
> > > 
> > > I just tried the latest pv_ops dom0 git tree (11 Oct 2009) on
x86_64 AHCI box.
> > > 
> > > The good news is that the dom0 kernel boots up, but there are
some error
> > > messages.
> > > 
> > > Using the default options (modeset) the VGA console
doesn''t work, it
> > > goes blank (display says "power save") in the beginning
of dom0 kernel boot:
> > >
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.1-2009-10-11.txt
> > 
> > This line:
> > [drm:radeon_ring_test] *ERROR* radeon: ring test failed
(sracth(0x15E4)=0xCAFEDEAD)
> > 
> > Is a pretty good pointer at what the fault is. If you look at git
commit
> > 93e7c3850b8431e19c9cba91413066bfd2360671 you will see the band-aid
Jeremy added.
> > It looks though as if not all of the radeon drivers allocate their
ring buffer memory via
> > drm_sg_alloc calls thought. Not sure how the r100 (and the
corresponding X driver) does it.
> > The long/erro traceback about the HARDIRQ is a red-herring in this
case.
> > 
> > Here is a couple of things that I would like you to try, if you can:
> > 
> 
> Sure.
> 
> > 1). Pass in ''drm.debug=255'' and send the output. It
should have tons of extra
> > output. 
> > 
> 
>
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.1-2009-10-14-drmdebug.txt
> 
> Unknown boot option `drm.debug=255'': ignoring
I forgot to mention that you probably need to have CONFIG_DRM set to
''y'' instead of ''m''
for this to work. Or you could hack up the initrd (modprobe.conf) and make drm
load
with the ''debug=255'' parameter.

.. snip ..
> seems to work there! (Fedora kernel contains newer graphics/drm drivers).
> 
> But the same USB related error is there with the fedora kernel:
> [ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ]
> 
>
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.3-1.2.71.xendom0.fc12.x86_64-2009-10-14.txt
Nah. Still has the same problem:

[drm:r100_ring_test] *ERROR* radeon: ring test failed
(sracth(0x15E4)=0xCAFEDEAD)
[drm:r100_cp_init] *ERROR* radeon: cp isn''t working (-
> 
> 
> > 2). Send in the Xorg.log (or whatever output the program in the
userland that
> >  starts the modesetting produces).  I don''t have much
knowledge in how modesetting works,
> >  so this might require some digging.
> > 
> 
> Hmm.. yeah, I''m not sure either which is the first program setting
up
> graphics mode using kernel modesetting (KMS) in Fedora.. 
> 
> I extracted the initrd image and checked the ''init''
script:
> 
> echo "Loading drm module"
> modprobe -q drm
> echo "Loading ttm module"
> modprobe -q ttm
> echo "Loading radeon module"
> modprobe -q radeon
> /lib/udev/console_init tty0
add here:
export LIBGL_DEBUG=verbose
> plymouth --show-splash
> 
> So I guess plymouth is asking for a graphics mode..
Add this to your kernel command line: plymouth:debug

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Dmesg report for 2.6.31.4 been built on F12 and loaded under Xen 3.4.1
(installed via rawhide 3.4.1-5 ) Dom0 on top of F12 ( yum updated)

[drm] Initialized drm 1.1.0 20060810
[drm] radeon default to kernel modesetting.
[drm] radeon kernel modesetting enabled.
xen: registering gsi 16 triggering 0 polarity 1
xen_allocate_pirq: returning irq 16 for gsi 16
xen: --> irq=16
xen_set_ioapic_routing: irq 16 gsi 16 vector 152 ioapic 0 pin 16 triggering 1
polarity 1
radeon 0000:01:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
radeon 0000:01:00.0: setting latency timer to 64
[drm] radeon: Initializing kernel modesetting.
[drm:radeon_driver_load_kms] *ERROR* Failed to initialize radeon, disabling
IOCTL
radeon 0000:01:00.0: PCI INT A disabled
radeon: probe of 0000:01:00.0 failed with error -22

.   .   .   .   .   .


=====================================================[ INFO: HARDIRQ-safe ->
HARDIRQ-unsafe lock order detected ]
2.6.31.4 #2
------------------------------------------------------
khubd/28 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
 (&retval->lock){......}, at: [<ffffffff81126fa4>]
dma_pool_alloc+0x46/0x312

and this task is already holding:
 (&ehci->lock){-.....}, at: [<ffffffff814359e4>]
ehci_urb_enqueue+0xb4/0xd7c
which would create a new lock dependency:
 (&ehci->lock){-.....} -> (&retval->lock){......}

but this new dependency connects a HARDIRQ-irq-safe lock:
 (&ehci->lock){-.....}
... which became HARDIRQ-irq-safe at:
  [<ffffffff810996d8>] __lock_acquire+0x256/0xc11
  [<ffffffff8109a181>] lock_acquire+0xee/0x12e
  [<ffffffff81579e9f>] _spin_lock+0x45/0x8e
  [<ffffffff814345ec>] ehci_irq+0x41/0x441
  [<ffffffff814195d5>] usb_hcd_irq+0x59/0xcc
  [<ffffffff810c8200>] handle_IRQ_event+0x62/0x148
  [<ffffffff810ca797>] handle_level_irq+0x90/0xf9
  [<ffffffff81018038>] handle_irq+0x9a/0xba
  [<ffffffff81302342>] xen_evtchn_do_upcall+0x10c/0x1bd
  [<ffffffff8101623e>] xen_do_hypervisor_callback+0x1e/0x30
  [<ffffffffffffffff>] 0xffffffffffffffff

to a HARDIRQ-irq-unsafe lock:
 (purge_lock){+.+...}
... which became HARDIRQ-irq-unsafe at:
...  [<ffffffff8109974d>] __lock_acquire+0x2cb/0xc11
  [<ffffffff8109a181>] lock_acquire+0xee/0x12e
  [<ffffffff81579e9f>] _spin_lock+0x45/0x8e
  [<ffffffff81120137>] __purge_vmap_area_lazy+0x63/0x198
  [<ffffffff81121a15>] vm_unmap_aliases+0x18f/0x1b2
  [<ffffffff8100e400>] xen_alloc_ptpage+0x47/0x75
  [<ffffffff8100e46b>] xen_alloc_pte+0x13/0x15
  [<ffffffff81115495>] __pte_alloc_kernel+0x6f/0xdd
  [<ffffffff81120f42>] vmap_page_range_noflush+0x1c5/0x315
  [<ffffffff811210d3>] map_vm_area+0x41/0x6b
  [<ffffffff8112122c>] __vmalloc_area_node+0x12f/0x167
  [<ffffffff811212f4>] __vmalloc_node+0x90/0xb5
  [<ffffffff81121169>] __vmalloc_area_node+0x6c/0x167
  [<ffffffff811212f4>] __vmalloc_node+0x90/0xb5
  [<ffffffff8112156b>] __vmalloc+0x28/0x3e
  [<ffffffff81adb40a>] alloc_large_system_hash+0x12f/0x1fb
  [<ffffffff81addc9a>] vfs_caches_init+0xb8/0x140
  [<ffffffff81ab5a69>] start_kernel+0x3ef/0x44c
  [<ffffffff81ab4d70>] x86_64_start_reservations+0xbb/0xd6
  [<ffffffff81ab93b7>] xen_start_kernel+0x5d5/0x5dc
  [<ffffffffffffffff>] 0xffffffffffffffff

other info that might help us debug this:

2 locks held by khubd/28:
 #0:  (usb_address0_mutex){+.+...}, at: [<ffffffff81414344>]
hub_port_init+0x8c/0x81e
 #1:  (&ehci->lock){-.....}, at: [<ffffffff814359e4>]
ehci_urb_enqueue+0xb4/0xd7c

the HARDIRQ-irq-safe lock''s dependencies:
-> (&ehci->lock){-.....} ops: 0 {
   IN-HARDIRQ-W at:
                        [<ffffffff810996d8>] __lock_acquire+0x256/0xc11
                        [<ffffffff8109a181>] lock_acquire+0xee/0x12e
                        [<ffffffff81579e9f>] _spin_lock+0x45/0x8e
                        [<ffffffff814345ec>] ehci_irq+0x41/0x441
                        [<ffffffff814195d5>] usb_hcd_irq+0x59/0xcc
                        [<ffffffff810c8200>] handle_IRQ_event+0x62/0x148
                        [<ffffffff810ca797>] handle_level_irq+0x90/0xf9
                        [<ffffffff81018038>] handle_irq+0x9a/0xba
                        [<ffffffff81302342>]
xen_evtchn_do_upcall+0x10c/0x1bd
                        [<ffffffff8101623e>]
xen_do_hypervisor_callback+0x1e/0x30
                        [<ffffffffffffffff>] 0xffffffffffffffff
.  .  .  .

The most recent build 2.6.31.1 on F12 produced clean dmesg output.
 Builds 2.6.31.4 ( same commit on top) on F11 and Ubuntu 9.04 Server seem
clean.

Boris.

--- On Thu, 10/15/09, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
wrote:

From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Subject: Re: [Xen-devel] Announcing xen/master: pvops git trees rearranged
To: "Pasi Kärkkäinen" <pasik@iki.fi>
Cc: "Jeremy Fitzhardinge" <jeremy@goop.org>,
"Xen-devel" <xen-devel@lists.xensource.com>
Date: Thursday, October 15, 2009, 4:04 PM

On Thu, Oct 15, 2009 at 12:14:15AM +0300, Pasi Kärkkäinen
wrote:
> On Mon, Oct 12, 2009 at 04:02:48PM -0400, Konrad Rzeszutek Wilk wrote:
> > On Sun, Oct 11, 2009 at 06:39:00PM +0300, Pasi Kärkkäinen wrote:
> > > On Fri, Sep 18, 2009 at 06:19:41PM -0700, Jeremy Fitzhardinge
wrote:
> > > > 
> > > > This is definitely a work-in-progress kernel.  I''d
appreciate all bug
> > > > *and* success reports so I can get some idea of how many
people are
> > > > using this thing, and how often there are problems.  Patches
gratefully
> > > > accepted.
> > > > 
> > > 
> > > I just tried the latest pv_ops dom0 git tree (11 Oct 2009) on
x86_64 AHCI box.
> > > 
> > > The good news is that the dom0 kernel boots up, but there are
some error
> > > messages.
> > > 
> > > Using the default options (modeset) the VGA console
doesn''t work, it
> > > goes blank (display says "power save") in the beginning
of dom0 kernel boot:
> > >
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.1-2009-10-11.txt
> > 
> > This line:
> > [drm:radeon_ring_test] *ERROR* radeon: ring test failed
(sracth(0x15E4)=0xCAFEDEAD)
> > 
> > Is a pretty good pointer at what the fault is. If you look at git
commit
> > 93e7c3850b8431e19c9cba91413066bfd2360671 you will see the band-aid
Jeremy added.
> > It looks though as if not all of the radeon drivers allocate their
ring buffer memory via
> > drm_sg_alloc calls thought. Not sure how the r100 (and the
corresponding X driver) does it.
> > The long/erro traceback about the HARDIRQ is a red-herring in this
case.
> > 
> > Here is a couple of things that I would like you to try, if you can:
> > 
> 
> Sure.
> 
> > 1). Pass in ''drm.debug=255'' and send the output. It
should have tons of extra
> > output. 
> > 
> 
>
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.1-2009-10-14-drmdebug.txt
> 
> Unknown boot option `drm.debug=255'': ignoring
I forgot to mention that you probably need to have CONFIG_DRM set to
''y'' instead of ''m''
for this to work. Or you could hack up the initrd (modprobe.conf) and make drm
load
with the ''debug=255'' parameter.

.. snip ..
> seems to work there! (Fedora kernel contains newer graphics/drm drivers).
> 
> But the same USB related error is there with the fedora kernel:
> [ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ]
> 
>
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.3-1.2.71.xendom0.fc12.x86_64-2009-10-14.txt
Nah. Still has the same problem:

[drm:r100_ring_test] *ERROR* radeon: ring test failed
(sracth(0x15E4)=0xCAFEDEAD)
[drm:r100_cp_init] *ERROR* radeon: cp isn''t working (-
> 
> 
> > 2). Send in the Xorg.log (or whatever output the program in the
userland that
> >  starts the modesetting produces).  I don''t have much
knowledge in how modesetting works,
> >  so this might require some digging.
> > 
> 
> Hmm.. yeah, I''m not sure either which is the first program setting
up
> graphics mode using kernel modesetting (KMS) in Fedora.. 
> 
> I extracted the initrd image and checked the ''init''
script:
> 
> echo "Loading drm module"
> modprobe -q drm
> echo "Loading ttm module"
> modprobe -q ttm
> echo "Loading radeon module"
> modprobe -q radeon
> /lib/udev/console_init tty0
add here:
export LIBGL_DEBUG=verbose
> plymouth --show-splash
> 
> So I guess plymouth is asking for a graphics mode..
Add this to your kernel command line: plymouth:debug

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Oct 15, 2009 at 04:04:09PM -0400, Konrad Rzeszutek Wilk
wrote:
> > > 
> > > Here is a couple of things that I would like you to try, if you
can:
> > > 
> > 
> > Sure.
> > 
> > > 1). Pass in ''drm.debug=255'' and send the
output. It should have tons of extra
> > > output. 
> > > 
> > 
> >
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.1-2009-10-14-drmdebug.txt
> > 
> > Unknown boot option `drm.debug=255'': ignoring
> 
> I forgot to mention that you probably need to have CONFIG_DRM set to
''y'' instead of ''m''
> for this to work. Or you could hack up the initrd (modprobe.conf) and make
drm load
> with the ''debug=255'' parameter.
>
It seems there are two separate issues; the radeon/drm failure, and then the USB
problem.
More about USB later in this mail.

I hacked the initrd and added ''debug=255'' when modprobing
drm.ko.
Log below.
> 
> > seems to work there! (Fedora kernel contains newer graphics/drm
drivers).
> > 
> > But the same USB related error is there with the fedora kernel:
> > [ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ]
> > 
> >
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.3-1.2.71.xendom0.fc12.x86_64-2009-10-14.txt
> 
> Nah. Still has the same problem:
> 
> [drm:r100_ring_test] *ERROR* radeon: ring test failed
(sracth(0x15E4)=0xCAFEDEAD)
> [drm:r100_cp_init] *ERROR* radeon: cp isn''t working (-
> 
True, I missed that. But the modesetting actually works on that fedora kernel,
so it''s
probably driver version related, since fedora has newer gfx stuff than
upstream kernel (some of the drm/radeon developers are fedora developers so the
patches
end up in fedora kernels before upstream kernels).
> > 
> > 
> > > 2). Send in the Xorg.log (or whatever output the program in the
userland that
> > >  starts the modesetting produces).  I don''t have much
knowledge in how modesetting works,
> > >  so this might require some digging.
> > > 
> > 
> > Hmm.. yeah, I''m not sure either which is the first program
setting up
> > graphics mode using kernel modesetting (KMS) in Fedora.. 
> > 
> > I extracted the initrd image and checked the ''init''
script:
> > 
> > echo "Loading drm module"
> > modprobe -q drm
> > echo "Loading ttm module"
> > modprobe -q ttm
> > echo "Loading radeon module"
> > modprobe -q radeon
> > /lib/udev/console_init tty0
> 
> add here:
> export LIBGL_DEBUG=verbose
> 
Done.
> > plymouth --show-splash
> > 
> > So I guess plymouth is asking for a graphics mode..
> 
> Add this to your kernel command line: plymouth:debug
Done.

Log with the modifications:
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.1-2009-10-16-drmdebug.txt

Still same problems related to radeon:
[drm:radeon_ring_test] *ERROR* radeon: ring test failed
(sracth(0x15E4)=0xCAFEDEAD)
[drm:r100_cp_init] *ERROR* radeon: cp isn''t working (-22).


Then more about USB stuff:

usb 1-5: new high speed USB device using ehci_hcd and address 2
<huge list of backtraces>

I think those USB devices are the IPMI management devices for remote KVM
(keyboard/video/mouse).. shouldn''t make any difference I guess..


-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > >
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.3-1.2.71.xendom0.fc12.x86_64-2009-10-14.txt
> > 
> > Nah. Still has the same problem:
> > 
> > [drm:r100_ring_test] *ERROR* radeon: ring test failed
(sracth(0x15E4)=0xCAFEDEAD)
> > [drm:r100_cp_init] *ERROR* radeon: cp isn''t working (-
I looked a bit at the code and tried to reproduce this, but found out that the
hardware I''ve is
a bit too modern for mode-setting to work (no support yet for RS780).

I looked at the code yesterday and I think I''ve found the
the failure. But I don''t have yet a test bed for this, so if you
are willing to be guinea pig, please test this patch (also attached):


diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index b8b6c4a..4445364 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -27,7 +27,7 @@
 /*
  * Authors: Thomas Hellstrom <thellstrom-at-vmware-dot-com>
  */
-
+#include <linux/dma-mapping.h>
 #include <linux/vmalloc.h>
 #include <linux/sched.h>
 #include <linux/highmem.h>
@@ -138,6 +138,9 @@ static void ttm_tt_free_page_directory(struct ttm_tt *ttm)
 static struct page *ttm_tt_alloc_page(unsigned page_flags)
 {
 	gfp_t gfp_flags = GFP_USER;
+	void *addr;
+	dma_addr_t phys;
+	struct page *page;
 
 	if (page_flags & TTM_PAGE_FLAG_ZERO_ALLOC)
 		gfp_flags |= __GFP_ZERO;
@@ -147,7 +150,24 @@ static struct page *ttm_tt_alloc_page(unsigned page_flags)
 	else
 		gfp_flags |= __GFP_HIGHMEM;
 
-	return alloc_page(gfp_flags);
+	addr = dma_alloc_coherent(NULL, PAGE_SIZE, &phys, gfp_flags);
+	BUG_ON(!addr);
+	page = virt_to_page(addr);
+	get_page(page);
+	return page;
+}
+
+static void ttm_tt_free_page(struct page *page)
+{
+	void *addr;
+
+ 	if (page == NULL)
+		return;
+
+	put_page(page);
+	addr = page_address(page);
+
+	dma_free_coherent(NULL, PAGE_SIZE, addr, virt_to_bus(addr));
 }
 
 static void ttm_tt_free_user_pages(struct ttm_tt *ttm)
@@ -180,7 +200,7 @@ static void ttm_tt_free_user_pages(struct ttm_tt *ttm)
 
 		ttm->pages[i] = NULL;
 		ttm_mem_global_free(ttm->bdev->mem_glob, PAGE_SIZE, false);
-		put_page(page);
+		ttm_tt_free_page(page);
 	}
 	ttm->state = tt_unpopulated;
 	ttm->first_himem_page = ttm->num_pages;
@@ -218,7 +238,7 @@ static struct page *__ttm_tt_get_page(struct ttm_tt *ttm,
int index)
 	}
 	return p;
 out_err:
-	put_page(p);
+	ttm_tt_free_page(p);
 	return NULL;
 }
 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Oct 20, 2009 at 12:58:05PM -0400, Konrad Rzeszutek Wilk
wrote:
> > > >
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.3-1.2.71.xendom0.fc12.x86_64-2009-10-14.txt
> > > 
> > > Nah. Still has the same problem:
> > > 
> > > [drm:r100_ring_test] *ERROR* radeon: ring test failed
(sracth(0x15E4)=0xCAFEDEAD)
> > > [drm:r100_cp_init] *ERROR* radeon: cp isn''t working (-
> 
> I looked a bit at the code and tried to reproduce this, but found out that
the hardware I''ve is
> a bit too modern for mode-setting to work (no support yet for RS780).
> 
> I looked at the code yesterday and I think I''ve found the
> the failure. But I don''t have yet a test bed for this, so if you
> are willing to be guinea pig, please test this patch (also attached):
> 
I updated the pv_ops dom0 git tree to the latest 2.6.31.4 tree as of
today, and also applied your ttm.patch.

Modesetting works now, and there are no drm/radeon errors.
USB related errors/tracebacks remain though.

http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.4-2009-10-21-with-ttm-patch.txt

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I updated the pv_ops dom0 git tree to the latest 2.6.31.4 tree as of
> today, and also applied your ttm.patch.
> 
> Modesetting works now, and there are no drm/radeon errors.
Thank you for testing it.
> USB related errors/tracebacks remain though.
You saw this on bare-metal too, right?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Oct 21, 2009 at 02:31:30PM -0400, Konrad Rzeszutek Wilk
wrote:
> > I updated the pv_ops dom0 git tree to the latest 2.6.31.4 tree as of
> > today, and also applied your ttm.patch.
> > 
> > Modesetting works now, and there are no drm/radeon errors.
> 
> Thank you for testing it.
> 
No problems. Thanks for the patch :)
> > USB related errors/tracebacks remain though.
> 
> You saw this on bare-metal too, right?
>
Nope. The same kernel booted on baremetal (without Xen) works OK without
the USB errors/tracebacks:
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.4-2009-10-21-baremetal.txt

I only get the USB tracebacks on pv_ops dom0 kernel.

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>
> Nope. The same kernel booted on baremetal (without Xen) works OK without
> the USB errors/tracebacks:
>
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.4-2009-10-21-baremetal.txt
>
> I only get the USB tracebacks on pv_ops dom0 kernel.
>   
Yeah, that''s a known bug at the moment.  It should be harmless.

(I need to fix up vm_unmap_aliases() to be callable in interrupt context.)

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Oct 21, 2009 at 12:50:08PM -0700, Jeremy Fitzhardinge
wrote:
> >
> > Nope. The same kernel booted on baremetal (without Xen) works OK
without
> > the USB errors/tracebacks:
> >
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.4-2009-10-21-baremetal.txt
> >
> > I only get the USB tracebacks on pv_ops dom0 kernel.
> >   
> 
> Yeah, that''s a known bug at the moment.  It should be harmless.
> 
> (I need to fix up vm_unmap_aliases() to be callable in interrupt context.)
> 
Ok, thanks for the info.

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Oct 21, 2009 at 02:31:30PM -0400, Konrad Rzeszutek Wilk
wrote:
> > I updated the pv_ops dom0 git tree to the latest 2.6.31.4 tree as of
> > today, and also applied your ttm.patch.
> > 
> > Modesetting works now, and there are no drm/radeon errors.
> 
> Thank you for testing it.
> 
Btw are you going to post this for inclusion in drm/ttm trees?

Or should it be committed to pvops tree?

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Oct 27, 2009 at 05:46:51PM +0200, Pasi Kärkkäinen
wrote:
> On Wed, Oct 21, 2009 at 02:31:30PM -0400, Konrad Rzeszutek Wilk wrote:
> > > I updated the pv_ops dom0 git tree to the latest 2.6.31.4 tree as
of
> > > today, and also applied your ttm.patch.
> > > 
> > > Modesetting works now, and there are no drm/radeon errors.
> > 
> > Thank you for testing it.
> > 
> 
> Btw are you going to post this for inclusion in drm/ttm trees?
I am not really comfortable with it. It has the same drawbacks
as the fix for the drm_scatter, where we blindly assume
phys_to_bus(virt_to_phys(X)) will give us the same value as
what dma_alloc_coherent provides. We should save that bus address
somewhere...

Saving it somewhere (perhaps in some of the structs the drm_ttm allocates)
could do it. But we should probably differentiate between memory
that is being allocated for DMA transfers vs other things so that
we don''t over-exercise the dma_alloc_coherent. Thought maybe
the memory returned via drm_tt calls are only used for DMA transfers.

We can figure this out.  Pasi, I don''t have a modesetting working
machine,
but you do. Can you compile your pv_ops with the fix I provided earlier,
along with enabling CONFIG_DMA_API_DEBUG=y. Once mode-setting is turned
on and your machine is humming along (maybe even run glxgears), compile
the attached module and load it. You should get a kernel dump
off all devices that are using the DMA buffers. Can you e-mail me that back
please?





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Oct 27, 2009 at 01:00:19PM -0400, Konrad Rzeszutek Wilk
wrote:
> On Tue, Oct 27, 2009 at 05:46:51PM +0200, Pasi Kärkkäinen wrote:
> > On Wed, Oct 21, 2009 at 02:31:30PM -0400, Konrad Rzeszutek Wilk wrote:
> > > > I updated the pv_ops dom0 git tree to the latest 2.6.31.4
tree as of
> > > > today, and also applied your ttm.patch.
> > > > 
> > > > Modesetting works now, and there are no drm/radeon errors.
> > > 
> > > Thank you for testing it.
> > > 
> > 
> > Btw are you going to post this for inclusion in drm/ttm trees?
> 
> I am not really comfortable with it. It has the same drawbacks
> as the fix for the drm_scatter, where we blindly assume
> phys_to_bus(virt_to_phys(X)) will give us the same value as
> what dma_alloc_coherent provides. We should save that bus address
> somewhere...
> 
> Saving it somewhere (perhaps in some of the structs the drm_ttm allocates)
> could do it. But we should probably differentiate between memory
> that is being allocated for DMA transfers vs other things so that
> we don''t over-exercise the dma_alloc_coherent. Thought maybe
> the memory returned via drm_tt calls are only used for DMA transfers.
>
Ok..
> We can figure this out.  Pasi, I don''t have a modesetting working
machine,
> but you do. Can you compile your pv_ops with the fix I provided earlier,
> along with enabling CONFIG_DMA_API_DEBUG=y. Once mode-setting is turned
> on and your machine is humming along (maybe even run glxgears), compile
> the attached module and load it. You should get a kernel dump
> off all devices that are using the DMA buffers. Can you e-mail me that back
> please?
> 
Yeah, I''ll do that and get back to you.

-- Pasi



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > We can figure this out.  Pasi, I don''t have a modesetting
working machine,
> > but you do. Can you compile your pv_ops with the fix I provided
earlier,
> > along with enabling CONFIG_DMA_API_DEBUG=y. Once mode-setting is
turned
> > on and your machine is humming along (maybe even run glxgears),
compile
> > the attached module and load it. You should get a kernel dump
> > off all devices that are using the DMA buffers. Can you e-mail me that
back
> > please?
> > 
> > 
> 
> I''ve never actually tried X on this box.. it seems X
doesn''t start.
As in, even on baremetal it does not work?
> 
>
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.4-2009-10-27.txt
>
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/Xorg.0.log-2009-10-27
This is weird. The EDID reporting seems to be stuck in a endless loop:

(II) RADEON(0): EDID for output DVI-0
(II) RADEON(0): EDID for output VGA-0
...
(II) RADEON(0): EDID for output DVI-0
(II) RADEON(0): EDID for output VGA-0
..
and then last one where it is cut off (probably when you rebooted the box?)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Oct 27, 2009 at 01:00:19PM -0400, Konrad Rzeszutek Wilk
wrote:
> On Tue, Oct 27, 2009 at 05:46:51PM +0200, Pasi Kärkkäinen wrote:
> > On Wed, Oct 21, 2009 at 02:31:30PM -0400, Konrad Rzeszutek Wilk wrote:
> > > > I updated the pv_ops dom0 git tree to the latest 2.6.31.4
tree as of
> > > > today, and also applied your ttm.patch.
> > > > 
> > > > Modesetting works now, and there are no drm/radeon errors.
> > > 
> > > Thank you for testing it.
> > > 
> > 
> > Btw are you going to post this for inclusion in drm/ttm trees?
> 
> I am not really comfortable with it. It has the same drawbacks
> as the fix for the drm_scatter, where we blindly assume
> phys_to_bus(virt_to_phys(X)) will give us the same value as
> what dma_alloc_coherent provides. We should save that bus address
> somewhere...
> 
> Saving it somewhere (perhaps in some of the structs the drm_ttm allocates)
> could do it. But we should probably differentiate between memory
> that is being allocated for DMA transfers vs other things so that
> we don''t over-exercise the dma_alloc_coherent. Thought maybe
> the memory returned via drm_tt calls are only used for DMA transfers.
> 
> We can figure this out.  Pasi, I don''t have a modesetting working
machine,
> but you do. Can you compile your pv_ops with the fix I provided earlier,
> along with enabling CONFIG_DMA_API_DEBUG=y. Once mode-setting is turned
> on and your machine is humming along (maybe even run glxgears), compile
> the attached module and load it. You should get a kernel dump
> off all devices that are using the DMA buffers. Can you e-mail me that back
> please?
> 
> 
I''ve never actually tried X on this box.. it seems X doesn''t
start.

http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.4-2009-10-27.txt
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/Xorg.0.log-2009-10-27

X never comes up.. screen goes blank/black, but there still is some signal on
the monitor..
and I can''t switch VTs from the console keyboard.

"kill <pid_of_X>" doesn''t help.. it seems only reboot
helps.

Hmm.. I wonder how to debug this. I guess X is stuck trying to determine
modelines from the (non-connected) DVI display.. actually I don''t even
have
DVI connector on the motherboard..

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > As in, even on baremetal it does not work?
> 
> Heh. should have tried that already. Too tired atm :)
> 
> Anyway, I tried baremetal now. X comes up, I can see the desktop, and
> then the monitor goes out of signal.
> 
> "dmesg" is full of:
> 
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> X used greatest stack depth: 3000 bytes left
> 
> Could that be caused by the patch? I booted the same ttm-patched kernel
I fear so. 
> on baremetal without Xen..
By next week I should have a working machine where I can do the mode-setting
and dig some deeper in this.

Thanks so far for testing some ideas out.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Oct 27, 2009 at 03:41:14PM -0400, Konrad Rzeszutek Wilk
wrote:
> > > We can figure this out.  Pasi, I don''t have a
modesetting working machine,
> > > but you do. Can you compile your pv_ops with the fix I provided
earlier,
> > > along with enabling CONFIG_DMA_API_DEBUG=y. Once mode-setting is
turned
> > > on and your machine is humming along (maybe even run glxgears),
compile
> > > the attached module and load it. You should get a kernel dump
> > > off all devices that are using the DMA buffers. Can you e-mail me
that back
> > > please?
> > > 
> > > 
> > 
> > I''ve never actually tried X on this box.. it seems X
doesn''t start.
> 
> As in, even on baremetal it does not work?
Heh. should have tried that already. Too tired atm :)

Anyway, I tried baremetal now. X comes up, I can see the desktop, and
then the monitor goes out of signal.

"dmesg" is full of:

[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
X used greatest stack depth: 3000 bytes left

Could that be caused by the patch? I booted the same ttm-patched kernel
on baremetal without Xen..
> > 
> >
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.4-2009-10-27.txt
> >
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/Xorg.0.log-2009-10-27
> 
> This is weird. The EDID reporting seems to be stuck in a endless loop:
> 
> (II) RADEON(0): EDID for output DVI-0
> (II) RADEON(0): EDID for output VGA-0
> ...
> (II) RADEON(0): EDID for output DVI-0
> (II) RADEON(0): EDID for output VGA-0
> ..
> and then last one where it is cut off (probably when you rebooted the box?)

Xorg.0.log for baremetal looks pretty much the same:
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/Xorg.0.log-2009-10-27-baremetal

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Oct 27, 2009 at 10:18:32PM +0200, Pasi Kärkkäinen
wrote:
> On Tue, Oct 27, 2009 at 03:41:14PM -0400, Konrad Rzeszutek Wilk wrote:
> > > > We can figure this out.  Pasi, I don''t have a
modesetting working machine,
> > > > but you do. Can you compile your pv_ops with the fix I
provided earlier,
> > > > along with enabling CONFIG_DMA_API_DEBUG=y. Once
mode-setting is turned
> > > > on and your machine is humming along (maybe even run
glxgears), compile
> > > > the attached module and load it. You should get a kernel
dump
> > > > off all devices that are using the DMA buffers. Can you
e-mail me that back
> > > > please?
> > > > 
> > > > 
> > > 
> > > I''ve never actually tried X on this box.. it seems X
doesn''t start.
> > 
> > As in, even on baremetal it does not work?
> 
> Heh. should have tried that already. Too tired atm :)
> 
> Anyway, I tried baremetal now. X comes up, I can see the desktop, and
> then the monitor goes out of signal.
> 
> "dmesg" is full of:
> 
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> X used greatest stack depth: 3000 bytes left
> 
> Could that be caused by the patch? I booted the same ttm-patched kernel
> on baremetal without Xen..
> 
Using the 2.6.31.5-96.fc12.x86_64 fedora kernel rpm (baremetal) 
X works OK. No TTM errors in dmesg.

Xorg.0.log looks similar.. last line is:

(II) RADEON(0): EDID for output DVI-0

So I guess the Xorg.0.log part is how it should be.. 

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Oct 27, 2009 at 04:13:06PM -0400, Konrad Rzeszutek Wilk
wrote:
> > > As in, even on baremetal it does not work?
> > 
> > Heh. should have tried that already. Too tired atm :)
> > 
> > Anyway, I tried baremetal now. X comes up, I can see the desktop, and
> > then the monitor goes out of signal.
> > 
> > "dmesg" is full of:
> > 
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > X used greatest stack depth: 3000 bytes left
> > 
> > Could that be caused by the patch? I booted the same ttm-patched
kernel
> 
> I fear so. 
> 
> > on baremetal without Xen..
> 
> By next week I should have a working machine where I can do the
mode-setting
> and dig some deeper in this.
> 
> Thanks so far for testing some ideas out.
>
Ok. Just let me know if you need me to test something else/more.

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 10/19/09 04:05, Zhang, Xiantao wrote:
> Attached the patches, and also created the patch for Xen-3.4-testing tree. 
Maybe Keir needs to check-in this patch before Xen-3.4.2 to keep new dom0 be
able to run on Xen-3.4-2.  Add Keir to the list. :-)
>   
Hi,

I was just looking at this again and I realize there''s been some
non-trival changes which cause conflicts with this patch.  Would you
have a chance to revise the patch for the current xen/master? 
Alternatively, would it be possible to base the patch on the
xen/dom0/apic topic branch?  It''s possible this is awkward because of
conflicts from the PCI front/back branches, in which case it may need
some extra patches for those branches.

For now I''ve applied your current patch to the branch
xen/dom0/apic-xiantao, rooted on the last xen/master branch where it
applies cleanly.

Thanks,
    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 11/11/09 16:47, Jeremy Fitzhardinge wrote:
> For now I''ve applied your current patch to the branch
> xen/dom0/apic-xiantao, rooted on the last xen/master branch where it
> applies cleanly.
>   
And xen/master-xiantao is my attempt to merge it into master.  It turned
out less complex than I''d thought, but I haven''t tested it
yet.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 11/11/09 17:00, Jeremy Fitzhardinge wrote:
> On 11/11/09 16:47, Jeremy Fitzhardinge wrote:
>   
>> For now I''ve applied your current patch to the branch
>> xen/dom0/apic-xiantao, rooted on the last xen/master branch where it
>> applies cleanly.
>>   
>>     
> And xen/master-xiantao is my attempt to merge it into master.  It turned
> out less complex than I''d thought, but I haven''t tested
it yet.
>   
And it works, at least for simple stuff (haven''t tried passthrough or
MSI yet).

Keir, how do you feel about this change for Xen?

x86: Change the interface physdev_map_pirq to support new dom0.

It also keeps compatibility with old dom0. 

Signed-off-by: Xiantao Zhang <xiantao.zhang@intel.com>

diff -r 8f81bdd57afe xen/arch/x86/irq.c
--- a/xen/arch/x86/irq.c	Thu Sep 03 09:51:37 2009 +0100
+++ b/xen/arch/x86/irq.c	Thu Sep 24 15:36:19 2009 +0800
@@ -55,6 +55,11 @@ DEFINE_PER_CPU(vector_irq_t, vector_irq)
 
 DEFINE_PER_CPU(struct cpu_user_regs *, __irq_regs);
 
+int check_irq_status(int irq)
+{
+    return irq_status[irq] != IRQ_UNUSED ? 1 : 0;
+}
+
 /* Must be called when irq disabled */
 void lock_vector_lock(void)
 {
@@ -588,6 +593,9 @@ int setup_irq(unsigned int irq, struct i
     desc->depth   = 0;
     desc->status &= ~IRQ_DISABLED;
     desc->handler->startup(irq);
+
+    if ( !check_irq_status(irq) )
+        irq_status[irq] = IRQ_USED;
 
     spin_unlock_irqrestore(&desc->lock,flags);
 
@@ -1277,6 +1285,8 @@ int map_domain_pirq(
 
     ASSERT(spin_is_locked(&pcidevs_lock));
     ASSERT(spin_is_locked(&d->event_lock));
+    
+    desc = irq_to_desc(irq);
 
     if ( !IS_PRIV(current->domain) )
         return -EPERM;
@@ -1288,6 +1298,13 @@ int map_domain_pirq(
         return -EINVAL;
     }
 
+    if ( desc->action )
+    {
+        dprintk(XENLOG_G_WARNING, "Attempt to map in-use IRQ by Xen,"
+                        " irq:%d!\n", irq);
+        return 0;
+    }
+
     old_irq = domain_pirq_to_irq(d, pirq);
     old_pirq = domain_irq_to_pirq(d, irq);
 
@@ -1307,7 +1324,6 @@ int map_domain_pirq(
         return ret;
     }
 
-    desc = irq_to_desc(irq);
 
     if ( type == MAP_PIRQ_TYPE_MSI )
     {
diff -r 8f81bdd57afe xen/arch/x86/physdev.c
--- a/xen/arch/x86/physdev.c	Thu Sep 03 09:51:37 2009 +0100
+++ b/xen/arch/x86/physdev.c	Thu Sep 24 15:45:17 2009 +0800
@@ -30,7 +30,7 @@ static int physdev_map_pirq(struct physd
 static int physdev_map_pirq(struct physdev_map_pirq *map)
 {
     struct domain *d;
-    int pirq, irq, ret = 0;
+    int pirq = 0, irq, ret = 0;
     struct msi_info _msi;
     void *map_data = NULL;
 
@@ -55,23 +55,28 @@ static int physdev_map_pirq(struct physd
     switch ( map->type )
     {
         case MAP_PIRQ_TYPE_GSI:
-            if ( map->index < 0 || map->index >= nr_irqs_gsi )
+        {
+            int gsi, triggering, polarity;
+            
+            gsi = map->index & 0xffff;
+            triggering = !!(map->index & (1 << 16));
+            polarity = !!(map->index & (1 << 24));
+            irq = pirq = map->pirq;
+            
+            if ( gsi < 0 || gsi >= nr_irqs_gsi )
             {
-                dprintk(XENLOG_G_ERR, "dom%d: map invalid irq %d\n",
-                        d->domain_id, map->index);
+                dprintk(XENLOG_G_ERR, "dom%d: map invalid gsi %d\n",
+                        d->domain_id, gsi);
                 ret = -EINVAL;
                 goto free_domain;
             }
-            irq = domain_pirq_to_irq(current->domain, map->index);
-            if ( !irq )
-            {
-                dprintk(XENLOG_G_ERR, "dom%d: map pirq with incorrect
irq!\n",
-                        d->domain_id);
-                ret = -EINVAL;
-                goto free_domain;
-            }
-            break;
-
+            if ( !check_irq_status(irq) ) {
+                mp_register_gsi(gsi, triggering, polarity);
+                printk("Register gsi:%d for dom:%d, irq:%d\n", gsi,
+                      d->domain_id, irq);
+            }
+            break;
+        }
         case MAP_PIRQ_TYPE_MSI:
             irq = map->index;
             if ( irq == -1 )
@@ -103,7 +108,6 @@ static int physdev_map_pirq(struct physd
     spin_lock(&pcidevs_lock);
     /* Verify or get pirq. */
     spin_lock(&d->event_lock);
-    pirq = domain_irq_to_pirq(d, irq);
     if ( map->pirq < 0 )
     {
         if ( pirq )
diff -r 8f81bdd57afe xen/include/asm-x86/irq.h
--- a/xen/include/asm-x86/irq.h	Thu Sep 03 09:51:37 2009 +0100
+++ b/xen/include/asm-x86/irq.h	Sat Sep 05 16:09:07 2009 +0800
@@ -123,6 +123,8 @@ int __assign_irq_vector(int irq, struct 
 
 int bind_irq_vector(int irq, int vector, cpumask_t domain);
 
+int check_irq_status(int irq);
+
 #define domain_pirq_to_irq(d, pirq) ((d)->arch.pirq_irq[pirq])
 #define domain_irq_to_pirq(d, irq) ((d)->arch.irq_pirq[irq])
 

	J


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jeremy Fitzhardinge wrote:
> On 11/11/09 17:00, Jeremy Fitzhardinge wrote:
>> On 11/11/09 16:47, Jeremy Fitzhardinge wrote:
>> 
>>> For now I''ve applied your current patch to the branch
>>> xen/dom0/apic-xiantao, rooted on the last xen/master branch where
>>> it applies cleanly. 
>>> 
>>> 
>> And xen/master-xiantao is my attempt to merge it into master.  It
>> turned out less complex than I''d thought, but I
haven''t tested it
>> yet. 
>> 
> 
> And it works, at least for simple stuff (haven''t tried passthrough
or
> MSI yet).
Hi, Jeremy
   I had tested VT-d and MSI when made the patch.  But anyway, we had better do
a full functional testing since the code base varies a lot.  :)
Xiantao
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 12/11/2009 23:51, "Jeremy Fitzhardinge" <jeremy@goop.org>
wrote:
> nd it works, at least for simple stuff (haven''t tried passthrough
or
> MSI yet).
> 
> Keir, how do you feel about this change for Xen?
Sure, I''ll check it in.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 11/12/09 23:24, Keir Fraser wrote:
> On 12/11/2009 23:51, "Jeremy Fitzhardinge"
<jeremy@goop.org> wrote:
>
>   
>> nd it works, at least for simple stuff (haven''t tried
passthrough or
>> MSI yet).
>>
>> Keir, how do you feel about this change for Xen?
>>     
> Sure, I''ll check it in.
>   
How would you feel about backporting this into a stable release?  If we
start using this interrupt setup mechanism it will be a red-letter day
for dom0/xen compatibility.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 13/11/2009 23:57, "Jeremy Fitzhardinge" <jeremy@goop.org>
wrote:
>> Sure, I''ll check it in.
>>   
> 
> How would you feel about backporting this into a stable release?  If we
> start using this interrupt setup mechanism it will be a red-letter day
> for dom0/xen compatibility.
There probably won''t be another stable release until 4.0 now. There
will be
a 3.4.3 about the same time though.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, Keir/Jeremy
   After you picking the two patches into upstream, we found it may break old
dom0 when assigned a level-triggered devices to a HVM domain.  The casue is
that, old dom0 can''t provide trigger mode and polarity when they do
map_domain_pirq.  In attached patches, they introduce a bit to indicate whether
old dom0 or not.

xen-introduce-a-bit-to-identify-dom0-type.patch:  for hypervisor. 
0001-x86-Introduce-a-bit-MAP_COMPAT-mode-for-MAP_PIRQ_TY.patch: for pv ops dom0.

Xiantao


Keir Fraser wrote:
> On 13/11/2009 23:57, "Jeremy Fitzhardinge"
<jeremy@goop.org> wrote:
> 
>>> Sure, I''ll check it in.
>>> 
>> 
>> How would you feel about backporting this into a stable release?  If
>> we start using this interrupt setup mechanism it will be a
>> red-letter day for dom0/xen compatibility.
> 
> There probably won''t be another stable release until 4.0 now.
There
> will be a 3.4.3 about the same time though.
> 
>  -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 11/16/09 02:38, Zhang, Xiantao wrote:
> Hi, Keir/Jeremy
>    After you picking the two patches into upstream, we found it may break
old dom0 when assigned a level-triggered devices to a HVM domain.  The casue is
that, old dom0 can''t provide trigger mode and polarity when they do
map_domain_pirq.  In attached patches, they introduce a bit to indicate whether
old dom0 or not.
>
> xen-introduce-a-bit-to-identify-dom0-type.patch:  for hypervisor. 
> 0001-x86-Introduce-a-bit-MAP_COMPAT-mode-for-MAP_PIRQ_TY.patch: for pv ops
dom0.
>   
Is there any way for the dom0 kernel to tell whether the hypervisor is
implementing the new ABI, so it can choose how to set up interrupts.

MAP_COMPAT_BIT doesn''t seem like a very good name, because it implies
that setting it reverts to "compatible" behaviour.  I assume that
leaving it clear enables the historical behaviour and setting it enables
the new one (since old kernels won''t be setting it).

    J
> Xiantao
>
>
> Keir Fraser wrote:
>   
>> On 13/11/2009 23:57, "Jeremy Fitzhardinge"
<jeremy@goop.org> wrote:
>>
>>     
>>>> Sure, I''ll check it in.
>>>>
>>>>         
>>> How would you feel about backporting this into a stable release? 
If
>>> we start using this interrupt setup mechanism it will be a
>>> red-letter day for dom0/xen compatibility.
>>>       
>> There probably won''t be another stable release until 4.0 now.
There
>> will be a 3.4.3 about the same time though.
>>
>>  -- Keir
>>     
>   

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jeremy Fitzhardinge wrote:
> On 11/16/09 02:38, Zhang, Xiantao wrote:
>> Hi, Keir/Jeremy
>>    After you picking the two patches into upstream, we found it may
>> break old dom0 when assigned a level-triggered devices to a HVM
>> domain.  The casue is that, old dom0 can''t provide trigger
mode and
>> polarity when they do map_domain_pirq.  In attached patches, they
>> introduce a bit to indicate whether old dom0 or not.    
>> 
>> xen-introduce-a-bit-to-identify-dom0-type.patch:  for hypervisor.
>> 0001-x86-Introduce-a-bit-MAP_COMPAT-mode-for-MAP_PIRQ_TY.patch: for
>> pv ops dom0. 
>> 
> 
> Is there any way for the dom0 kernel to tell whether the hypervisor is
> implementing the new ABI, so it can choose how to set up interrupts.
> 
> MAP_COMPAT_BIT doesn''t seem like a very good name, because it
implies
> that setting it reverts to "compatible" behaviour.  I assume that
> leaving it clear enables the historical behaviour and setting it
> enables the new one (since old kernels won''t be setting it).
Maybe better change to MAP_NEW_ABI_BIT ?  Since the hypervisor patch
didn''t change old code path after introducing this bit, so it is very
easy and safe to backport to xen-3.4-testing tree, and make new dom0 be able to
run top of it.  :)
Xiantao
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 17/11/2009 03:13, "Zhang, Xiantao" <xiantao.zhang@intel.com>
wrote:
>> Is there any way for the dom0 kernel to tell whether the hypervisor is
>> implementing the new ABI, so it can choose how to set up interrupts.
>> 
>> MAP_COMPAT_BIT doesn''t seem like a very good name, because it
implies
>> that setting it reverts to "compatible" behaviour.  I assume
that
>> leaving it clear enables the historical behaviour and setting it
>> enables the new one (since old kernels won''t be setting it).
> 
> Maybe better change to MAP_NEW_ABI_BIT ?  Since the hypervisor patch
didn''t
> change old code path after introducing this bit, so it is very easy and
safe
> to backport to xen-3.4-testing tree, and make new dom0 be able to run top
of
> it.  :)
It''s kind of a shame to need this though. Is there no way for the
hypervisor
to work out automatically whether an older dom0 is running? Or work out the
trigger/level stuff for itself (after all it parses the relevant bios tables
just like dom0)?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 11/16/09 19:13, Zhang, Xiantao wrote:
> Jeremy Fitzhardinge wrote:
>> Is there any way for the dom0 kernel to tell whether the hypervisor is
>> implementing the new ABI, so it can choose how to set up interrupts.
>>
>> MAP_COMPAT_BIT doesn''t seem like a very good name, because it
implies
>> that setting it reverts to "compatible" behaviour.  I assume
that
>> leaving it clear enables the historical behaviour and setting it
>> enables the new one (since old kernels won''t be setting it).
>>     
> Maybe better change to MAP_NEW_ABI_BIT ?  Since the hypervisor patch
didn''t change old code path after introducing this bit, so it is very
easy and safe to backport to xen-3.4-testing tree, and make new dom0 be able to
run top of it.  :)
>   
Better to give it a name which actually describes what it means/does. 
NEW_ABI isn''t very descriptive when it becomes *the* ABI (and what
happens when there''s a newer one?).

    J
> Xiantao
>   

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 11/16/09 19:45, Keir Fraser wrote:
> It''s kind of a shame to need this though. Is there no way for the
hypervisor
> to work out automatically whether an older dom0 is running? Or work out the
> trigger/level stuff for itself (after all it parses the relevant bios
tables
> just like dom0)?
If Xen can set the interrupt triggering by itself, why would it ever
need dom0 to do it?  Couldn''t it just preconfigure all the pins, and
then wait for dom0 to provide/request the pirq<->evtchn mapping?

    J


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 17/11/2009 05:20, "Jeremy Fitzhardinge" <jeremy@goop.org>
wrote:
> On 11/16/09 19:45, Keir Fraser wrote:
>> It''s kind of a shame to need this though. Is there no way for
the hypervisor
>> to work out automatically whether an older dom0 is running? Or work out
the
>> trigger/level stuff for itself (after all it parses the relevant bios
tables
>> just like dom0)?
> 
> If Xen can set the interrupt triggering by itself, why would it ever
> need dom0 to do it?  Couldn''t it just preconfigure all the pins,
and
> then wait for dom0 to provide/request the pirq<->evtchn mapping?
Very fair question. As long as Xen knows which GSI is being addressed (which
I''m sure it can work out), the rest should follow. I don''t
think even things
like PCI hotplug complicate this -- IOAPIC pins themselves are static.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jeremy Fitzhardinge wrote:
> On 11/16/09 19:45, Keir Fraser wrote:
>> It''s kind of a shame to need this though. Is there no way for
the
>> hypervisor to work out automatically whether an older dom0 is
>> running? Or work out the trigger/level stuff for itself (after all
>> it parses the relevant bios tables just like dom0)?
> 
> If Xen can set the interrupt triggering by itself, why would it ever
> need dom0 to do it?  Couldn''t it just preconfigure all the pins,
and
> then wait for dom0 to provide/request the pirq<->evtchn mapping?

After reviewing the logic, I think we can use DOMID_SELF to identify new dom0. 
In linux-2.6.18 dom0, only qemu uses this hypercall for device assginment, so
map->domid shouldn''t be dom0.  If old dom0/qemu with this hypercall,
keeps the logic unchanged, and only change the logic for new dom0 when call into
it.   Attached the patch.
Xiantao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 17/11/2009 12:46, "Zhang, Xiantao" <xiantao.zhang@intel.com>
wrote:
>> If Xen can set the interrupt triggering by itself, why would it ever
>> need dom0 to do it?  Couldn''t it just preconfigure all the
pins, and
>> then wait for dom0 to provide/request the pirq<->evtchn mapping?
> 
> After reviewing the logic, I think we can use DOMID_SELF to identify new
dom0.
> In linux-2.6.18 dom0, only qemu uses this hypercall for device assginment,
so
> map->domid shouldn''t be dom0.  If old dom0/qemu with this
hypercall, keeps the
> logic unchanged, and only change the logic for new dom0 when call into it.
> Attached the patch.
Don''t like it (subtle semantic difference based on domid field is very
odd
and could bite us later), and actually now I look closer I don''t like
the
original patch much either, if only because it does clearly change the
interface for MAP_PIRQ_TYPE_GSI by adding trigger/polarity flags (and not
documented or exported properly in the physdev.h header file).

I need to take a step back and work out what in fact you''re trying to
achieve. I''m going to revert the original patch from xen-unstable. If
such
an interface extension is really required, I think at least the new
interface should be a new subtype of MAP_PIRQ_TYPE_xxx, properly described
in the header file. But like I say, I don''t know what the patch is even
really trying to do or overcome.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 17/11/2009 12:46, "Zhang, Xiantao"
<xiantao.zhang@intel.com> wrote:
> 
>>> If Xen can set the interrupt triggering by itself, why would it
ever
>>> need dom0 to do it?  Couldn''t it just preconfigure all the
pins, and
>>> then wait for dom0 to provide/request the pirq<->evtchn
mapping?
>> 
>> After reviewing the logic, I think we can use DOMID_SELF to identify
>> new dom0. In linux-2.6.18 dom0, only qemu uses this hypercall for
>> device assginment, so map->domid shouldn''t be dom0.  If old
>> dom0/qemu with this hypercall, keeps the logic unchanged, and only
>> change the logic for new dom0 when call into it. Attached the patch.
> 
> Don''t like it (subtle semantic difference based on domid field is
> very odd and could bite us later), and actually now I look closer I
> don''t like the original patch much either, if only because it does
> clearly change the interface for MAP_PIRQ_TYPE_GSI by adding
> trigger/polarity flags (and not documented or exported properly in
> the physdev.h header file). 
> 
> I need to take a step back and work out what in fact you''re trying
to
> achieve. I''m going to revert the original patch from xen-unstable.
If
> such an interface extension is really required, I think at least the
> new interface should be a new subtype of MAP_PIRQ_TYPE_xxx, properly
> described in the header file. But like I say, I don''t know what
the
> patch is even really trying to do or overcome. 
Originally, this patch is target to get rid of ioapic changes in dom0. Before
this patch, GSI irq should be mapped and setup through dom0 programming ioapic
entries, but it depends on using ioapic logic in dom0. And if we remove ioapic
logic from dom0,  we need to find new way how to setup GSI irq.  And this patch
comes out under this situation.  The idea is from that in Xen the interface
MAP_PIRQ_TYPE_MSI is used to build the pirq and irq mapping for MSI IRQ for each
domain. Since MSI IRQ can be setup through this hypercall, and I think  we also
can leverage the interface MAP_PIRQ_TYPE _GSI to build the mapping for GSI irq.
Further analysis showes that this interface is only used for assigning devices
to HVM domain in qemu, and I think it should be Okay for dom0 building the
mapping between its pirq and irq. One different thing for GSI irq is that more
info should be provided in the call, since GSI IRQ has different trigger-mode
and polarity (originally it is provided by ioapic write in dom0). Certainly, I
also think we need to document the related info, and if you agree to the change,
I am happy to add it.
Xiantao
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 11/17/09 06:17, Zhang, Xiantao wrote:
> Originally, this patch is target to get rid of ioapic changes in dom0.
Before this patch, GSI irq should be mapped and setup through dom0 programming
ioapic entries, but it depends on using ioapic logic in dom0. And if we remove
ioapic logic from dom0,  we need to find new way how to setup GSI irq.  And this
patch comes out under this situation.  The idea is from that in Xen the
interface MAP_PIRQ_TYPE_MSI is used to build the pirq and irq mapping for MSI
IRQ for each domain. Since MSI IRQ can be setup through this hypercall, and I
think  we also can leverage the interface MAP_PIRQ_TYPE _GSI to build the
mapping for GSI irq. Further analysis showes that this interface is only used
for assigning devices to HVM domain in qemu, and I think it should be Okay for
dom0 building the mapping between its pirq and irq. One different thing for GSI
irq is that more info should be provided in the call, since GSI IRQ has
different trigger-mode and polarity (originally it is provided by ioapic write
in dom0). Certainly, I also think we need to document the related info, and if
you agree to the change, I am happy to add it.
>   
I don''t think there''s any need to overload the existing
interface
though.  If we''re adding new functionality then we can add a new
interface for it (but with luck we can reuse most of the existing code
to implement it).

If you''re already considering a "treat this differently" flag
in the
argument, then that''s a strong pointer that a new interface is
warranted.

But also consider the question whether Xen needs to get triggering info
from dom0 at all, or whether it can correctly derive that info from its
own parsing of the relevant tables.

    J


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Xiantao,

Responding inline below with stupid/basic questions. Just want to go back to
first principles with this and work out which of us is on the wrong track.

On 17/11/2009 14:17, "Zhang, Xiantao" <xiantao.zhang@intel.com>
wrote:
> Originally, this patch is target to get rid of ioapic changes in dom0.
Before
> this patch, GSI irq should be mapped and setup through dom0 programming
ioapic
> entries, but it depends on using ioapic logic in dom0.
So dom0 doesn''t write the IOAPIC *at all*? Okay.
> And if we remove ioapic
> logic from dom0,  we need to find new way how to setup GSI irq.  And this
> patch comes out under this situation.
Why does this need to be done under dom0 control? All GSIs are parseable by
Xen by itself aren''t they, from MPBIOS tables or ACPI MADT? So at least
Xen
should be able to work out for itself APIC pin -> GSI mappings, and pol/trig
settings.
> The idea is from that in Xen the
> interface MAP_PIRQ_TYPE_MSI is used to build the pirq and irq mapping for
MSI
> IRQ for each domain. Since MSI IRQ can be setup through this hypercall, and
I
> think  we also can leverage the interface MAP_PIRQ_TYPE _GSI to build the
> mapping for GSI irq.
For modern dom0 don''t we already assume that dom0 pirq == irq == gsi
(see
comments in ioapic_guest_write)? Perhaps we should just have that
relationship set up by default: I think only NetBSD dom0 has different, and
it will establish different relationship via legacy method of
PHYSDEVOP_alloc_irq_vector and paravirtualised IOAPIC writes?
> Further analysis showes that this interface is only used
> for assigning devices to HVM domain in qemu, and I think it should be Okay
for
> dom0 building the mapping between its pirq and irq. One different thing for
> GSI irq is that more info should be provided in the call, since GSI IRQ has
> different trigger-mode and polarity (originally it is provided by ioapic
write
> in dom0). Certainly, I also think we need to document the related info, and
if
> you agree to the change, I am happy to add it.
Maybe this follows if my above assumptions/arguments are broken.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>> And if we remove ioapic
>> logic from dom0,  we need to find new way how to setup GSI irq.  And
>> this patch comes out under this situation.
> 
> Why does this need to be done under dom0 control? All GSIs are
> parseable by Xen by itself aren''t they, from MPBIOS tables or ACPI
> MADT? So 
> at least Xen
> should be able to work out for itself APIC pin -> GSI
> mappings, and pol/trig
> settings.
My 2 cents for this topic, although I''m still trying to figure out the
whole picture of the patch and discussion thread.

The ACPI MADT table gives the relationship between IOAPIC and gsi, while DSDT
table''s _PRT gives the relationship between PCI devices and GSI.

Two way provided in ACPI _PRT table. If the source filed in _PRT entry does not
refert to any device, it means a GSI. I didn''t find any hint in ACPI
spec on the polarity/level should be configured. Currently kernel assume the
polarity/level is fixed as low/level, which I suspect is according to PCI spec.

If the _PRT table present the interrupt as PCI Interrupt Link Device, that means
the interrupt attribute is configurable, OSPM need figure out the polarity/level
information through _CRS/_PRS method in these objectes.

For method 1 (i.e. source filed does not refer to device), maybe we can assume
the attribute is fixed and Keir''s suggestion will work, while I suspect
if Xen can do anyting for second type.

Quickly check Xiantao''s patch, I suspect if it will work for 2nd
situation.

BTW, I don''t know know any system which use 2nd type when working in
APIC mode.

--jyh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> Xiantao,
> 
> Responding inline below with stupid/basic questions. Just want to go
> back to first principles with this and work out which of us is on the
> wrong track. 
> 
> On 17/11/2009 14:17, "Zhang, Xiantao"
<xiantao.zhang@intel.com> wrote:
> 
>> Originally, this patch is target to get rid of ioapic changes in
>> dom0. Before this patch, GSI irq should be mapped and setup through
>> dom0 programming ioapic entries, but it depends on using ioapic
>> logic in dom0. 
> 
> So dom0 doesn''t write the IOAPIC *at all*? Okay.
Dom0 won''t write ioapic in new dom0, and all operations related to
ioapic should be dummy, won''t trap to hypervisor any more.
>> And if we remove ioapic
>> logic from dom0,  we need to find new way how to setup GSI irq.  And
>> this patch comes out under this situation.
> 
> Why does this need to be done under dom0 control? All GSIs are
> parseable by Xen by itself aren''t they, from MPBIOS tables or ACPI
> MADT? So at least Xen should be able to work out for itself APIC pin
> -> GSI mappings, and pol/trig settings.
I agree with you at this point, and Xen should has the ability to parse the
info, but current Xen should doesn''t have.
A clean way is to do this in hyervisor and dom0 only ask the request about GSI
IRQ, like MSI does today.
>> The idea is from that in Xen the
>> interface MAP_PIRQ_TYPE_MSI is used to build the pirq and irq
>> mapping for MSI IRQ for each domain. Since MSI IRQ can be setup
>> through this hypercall, and I think  we also can leverage the
>> interface MAP_PIRQ_TYPE _GSI to build the mapping for GSI irq.
> 
> For modern dom0 don''t we already assume that dom0 pirq == irq ==
gsi
> (see comments in ioapic_guest_write)? Perhaps we should just have that
> relationship set up by default: I think only NetBSD dom0 has
> different, and it will establish different relationship via legacy
> method of PHYSDEVOP_alloc_irq_vector and paravirtualised IOAPIC
> writes?
The assumption should be right for dom0 today, but it still needs to register
this info to dom0''s private data(d->arch.{pirq_irq, irq_pirq).
And I think maybe we should clean up the logic and let hypervisor knows the
assumption, when consulting this relationship.

Xiantao


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jeremy Fitzhardinge wrote:
> On 11/17/09 06:17, Zhang, Xiantao wrote:
>> Originally, this patch is target to get rid of ioapic changes in
>> dom0. Before this patch, GSI irq should be mapped and setup through
>> dom0 programming ioapic entries, but it depends on using ioapic
>> logic in dom0. And if we remove ioapic logic from dom0,  we need to
>> find new way how to setup GSI irq.  And this patch comes out under
>> this situation.  The idea is from that in Xen the interface
>> MAP_PIRQ_TYPE_MSI is used to build the pirq and irq mapping for MSI
>> IRQ for each domain. Since MSI IRQ can be setup through this
>> hypercall, and I think  we also can leverage the interface
>> MAP_PIRQ_TYPE _GSI to build the mapping for GSI irq. Further
>> analysis showes that this interface is only used for assigning
>> devices to HVM domain in qemu, and I think it should be Okay for
>> dom0 building the mapping between its pirq and irq. One different
>> thing for GSI irq is that more info should be provided in the call,
>> since GSI IRQ has different trigger-mode and polarity (originally it
>> is provided by ioapic write in dom0). Certainly, I also think we
>> need to document the related info, and if you agree to the change, I
>> am happy to add it.                 
>> 
> 
> I don''t think there''s any need to overload the existing
interface
> though.  If we''re adding new functionality then we can add a new
> interface for it (but with luck we can reuse most of the existing code
> to implement it).
> If you''re already considering a "treat this differently"
flag in the
> argument, then that''s a strong pointer that a new interface is
> warranted. 
Agree, and I also don''t object to add a similar interface.  
Since this existing interface is only used for hvm domain before, and  just want
to re-use it for dom.

> But also consider the question whether Xen needs to get triggering
> info from dom0 at all, or whether it can correctly derive that info
> from its own parsing of the relevant tables.
Xen should have the ability to parse the info, but may need more logic porting
to hyervisor.

Xiantao


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 18/11/2009 03:25, "Zhang, Xiantao" <xiantao.zhang@intel.com>
wrote:
>> For modern dom0 don''t we already assume that dom0 pirq == irq
== gsi
>> (see comments in ioapic_guest_write)? Perhaps we should just have that
>> relationship set up by default: I think only NetBSD dom0 has
>> different, and it will establish different relationship via legacy
>> method of PHYSDEVOP_alloc_irq_vector and paravirtualised IOAPIC
>> writes?
> 
> The assumption should be right for dom0 today, but it still needs to
register
> this info to dom0''s private data(d->arch.{pirq_irq, irq_pirq).
> And I think maybe we should clean up the logic and let hypervisor knows the
> assumption, when consulting this relationship.
Well, it strikes me that existing MAP_PIRQ_TYPE_GSI fills this role already,
as it is, doesn''t it? Seems to me that is its whole purpose. :-)

Shoehorning trig/pol information into it as well is kind of nasty. And I
think on any PC system it should suffice to assume GSI 0-15 are ISA
edge-triggered active-high, GSI 16+ are PCI level-triggered active-low, and
any exceptions are parsed out of MADT or MPBIOS. We pretty much have all
that code, it just might need plumbing back in a little bit. Yunhong points
out that ACPI DSDT can have overriding objects in the _PRT, but I don''t
know
it ever actually gets used on real-world PC systems. So I would try without,
but if we do end up needing to get this info from dom0, I think it should be
via a new physdev_op.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Nov 17, 2009 at 08:46:19PM +0800, Zhang, Xiantao
wrote:
> Jeremy Fitzhardinge wrote:
> > On 11/16/09 19:45, Keir Fraser wrote:
> >> It''s kind of a shame to need this though. Is there no way
for the
> >> hypervisor to work out automatically whether an older dom0 is
> >> running? Or work out the trigger/level stuff for itself (after all
> >> it parses the relevant bios tables just like dom0)?
> > 
> > If Xen can set the interrupt triggering by itself, why would it ever
> > need dom0 to do it?  Couldn''t it just preconfigure all the
pins, and
> > then wait for dom0 to provide/request the pirq<->evtchn mapping?
> 
> 
> After reviewing the logic, I think we can use DOMID_SELF to identify new
dom0.  In linux-2.6.18 dom0, only qemu uses this hypercall for device
assginment, so map->domid shouldn''t be dom0.  If old dom0/qemu with
this hypercall, keeps the logic unchanged, and only change the logic for new
dom0 when call into it.   Attached the patch.
What about privileged domains that are not domain 0? Won''t that
mess this up?


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Nov 18, 2009 at 11:37:51AM +0800, Zhang, Xiantao
wrote:
> Jeremy Fitzhardinge wrote:
> > On 11/17/09 06:17, Zhang, Xiantao wrote:
> >> Originally, this patch is target to get rid of ioapic changes in
> >> dom0. Before this patch, GSI irq should be mapped and setup
through
> >> dom0 programming ioapic entries, but it depends on using ioapic
> >> logic in dom0. And if we remove ioapic logic from dom0,  we need
to
> >> find new way how to setup GSI irq.  And this patch comes out under
> >> this situation.  The idea is from that in Xen the interface
> >> MAP_PIRQ_TYPE_MSI is used to build the pirq and irq mapping for
MSI
> >> IRQ for each domain. Since MSI IRQ can be setup through this
> >> hypercall, and I think  we also can leverage the interface
> >> MAP_PIRQ_TYPE _GSI to build the mapping for GSI irq. Further
> >> analysis showes that this interface is only used for assigning
> >> devices to HVM domain in qemu, and I think it should be Okay for
> >> dom0 building the mapping between its pirq and irq. One different
> >> thing for GSI irq is that more info should be provided in the
call,
> >> since GSI IRQ has different trigger-mode and polarity (originally
it
> >> is provided by ioapic write in dom0). Certainly, I also think we
> >> need to document the related info, and if you agree to the change,
I
> >> am happy to add it.                 
> >> 
> > 
> > I don''t think there''s any need to overload the
existing interface
> > though.  If we''re adding new functionality then we can add a
new
> > interface for it (but with luck we can reuse most of the existing code
> > to implement it).
> 
> > If you''re already considering a "treat this
differently" flag in the
> > argument, then that''s a strong pointer that a new interface
is
> > warranted. 
> 
> Agree, and I also don''t object to add a similar interface.  
> Since this existing interface is only used for hvm domain before, and  just
want to re-use it for dom.
I am pretty sure it is used for PV domains too. Look in
''xen_create_msi_irq'', which
extracts the domain that has a PCI device for pass-through and on behalf of that
domain makes the MAP_PIRQ_TYPE_MSI call.

Furthermore, it is also used by Dom0 for MSI devices.

What do you mean by ''re-use it for dom''?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk wrote:
> On Tue, Nov 17, 2009 at 08:46:19PM +0800, Zhang, Xiantao wrote:
>> Jeremy Fitzhardinge wrote:
>>> On 11/16/09 19:45, Keir Fraser wrote:
>>>> It''s kind of a shame to need this though. Is there no
way for the
>>>> hypervisor to work out automatically whether an older dom0 is
>>>> running? Or work out the trigger/level stuff for itself (after
all
>>>> it parses the relevant bios tables just like dom0)?
>>> 
>>> If Xen can set the interrupt triggering by itself, why would it
ever
>>> need dom0 to do it?  Couldn''t it just preconfigure all the
pins, and
>>> then wait for dom0 to provide/request the pirq<->evtchn
mapping?
>> 
>> 
>> After reviewing the logic, I think we can use DOMID_SELF to identify
>> new dom0.  In linux-2.6.18 dom0, only qemu uses this hypercall for
>> device assginment, so map->domid shouldn''t be dom0.  If old
>> dom0/qemu with this hypercall, keeps the logic unchanged, and only
>> change the logic for new dom0 when call into it.   Attached the
>> patch.     
> 
> What about privileged domains that are not domain 0? Won''t that
> mess this up?
Do you mean driver domain, I think the driver domain should use DOMID_SELF for
the hypercall. :)
Xiantao
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk wrote:
> On Wed, Nov 18, 2009 at 11:37:51AM +0800, Zhang, Xiantao wrote:
>> Jeremy Fitzhardinge wrote:
>>> On 11/17/09 06:17, Zhang, Xiantao wrote:
>>>> Originally, this patch is target to get rid of ioapic changes
in
>>>> dom0. Before this patch, GSI irq should be mapped and setup
through
>>>> dom0 programming ioapic entries, but it depends on using ioapic
>>>> logic in dom0. And if we remove ioapic logic from dom0,  we
need to
>>>> find new way how to setup GSI irq.  And this patch comes out
under
>>>> this situation.  The idea is from that in Xen the interface
>>>> MAP_PIRQ_TYPE_MSI is used to build the pirq and irq mapping for
MSI
>>>> IRQ for each domain. Since MSI IRQ can be setup through this
>>>> hypercall, and I think  we also can leverage the interface
>>>> MAP_PIRQ_TYPE _GSI to build the mapping for GSI irq. Further
>>>> analysis showes that this interface is only used for assigning
>>>> devices to HVM domain in qemu, and I think it should be Okay
for
>>>> dom0 building the mapping between its pirq and irq. One
different
>>>> thing for GSI irq is that more info should be provided in the
call,
>>>> since GSI IRQ has different trigger-mode and polarity
(originally
>>>> it is provided by ioapic write in dom0). Certainly, I also
think we
>>>> need to document the related info, and if you agree to the
change,
>>>> I am happy to add it. 
>>>> 
>>> 
>>> I don''t think there''s any need to overload the
existing interface
>>> though.  If we''re adding new functionality then we can add
a new
>>> interface for it (but with luck we can reuse most of the existing
>>> code to implement it).
>> 
>>> If you''re already considering a "treat this
differently" flag in the
>>> argument, then that''s a strong pointer that a new
interface is
>>> warranted.
>> 
>> Agree, and I also don''t object to add a similar interface.
>> Since this existing interface is only used for hvm domain before,
>> and  just want to re-use it for dom. 
> 
> I am pretty sure it is used for PV domains too. Look in
> ''xen_create_msi_irq'', which extracts the domain that has
a PCI device
> for pass-through and on behalf of that domain makes the
> MAP_PIRQ_TYPE_MSI call. 
> 
> Furthermore, it is also used by Dom0 for MSI devices.
We mean the call MAP_PIRQ_TYPE_GSI, not MAP_PIRQ_TYPE_MSI. 
> What do you mean by ''re-use it for dom''?
Oh, seems the char "0" is missing, should be dom0. 
Xiantao


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 18/11/2009 03:25, "Zhang, Xiantao"
<xiantao.zhang@intel.com> wrote:
> 
>>> For modern dom0 don''t we already assume that dom0 pirq ==
irq == gsi
>>> (see comments in ioapic_guest_write)? Perhaps we should just have
>>> that relationship set up by default: I think only NetBSD dom0 has
>>> different, and it will establish different relationship via legacy
>>> method of PHYSDEVOP_alloc_irq_vector and paravirtualised IOAPIC
>>> writes?
>> 
>> The assumption should be right for dom0 today, but it still needs to
>> register this info to dom0''s private
data(d->arch.{pirq_irq,
>> irq_pirq). 
>> And I think maybe we should clean up the logic and let hypervisor
>> knows the assumption, when consulting this relationship.
> 
> Well, it strikes me that existing MAP_PIRQ_TYPE_GSI fills this role
> already, as it is, doesn''t it? Seems to me that is its whole
purpose.
> :-) 
> 
> Shoehorning trig/pol information into it as well is kind of nasty.
> And I think on any PC system it should suffice to assume GSI 0-15 are
> ISA edge-triggered active-high, GSI 16+ are PCI level-triggered
> active-low, and any exceptions are parsed out of MADT or MPBIOS. We
> pretty much have all that code, it just might need plumbing back in a
> little bit. Yunhong points out that ACPI DSDT can have overriding
> objects in the _PRT, but I don''t know it ever actually gets used
on
> real-world PC systems. So I would try without, but if we do end up
> needing to get this info from dom0, I think it should be via a new
> physdev_op.
At least dom0 parses this info from DSDT, so we can''t have the
assuption whether it is used or not, I think. And I also agree to add a new
physdev_op to handle this case, and it should be better way to go.
Based on this idea, I worked out the patch, attached!  In this patch, we
introduced a new physdev_op PHYSDEVOP_setup_gsi for each GSI setup, and each
domain can require to map each GSI in this case.
In addition, I believe it is very safe to port the hypervisor patch to xen-3.4-x
tree and keeps pv_ops dom0 running on it, since no logic is changed.  BTW, I
also tested apic and non-apic cases, they works fine after applying the patches.
Xiantao





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 11/24/09 02:04, Zhang, Xiantao wrote:
>> Shoehorning trig/pol information into it as well is kind of nasty.
>> And I think on any PC system it should suffice to assume GSI 0-15 are
>> ISA edge-triggered active-high, GSI 16+ are PCI level-triggered
>> active-low, and any exceptions are parsed out of MADT or MPBIOS. We
>> pretty much have all that code, it just might need plumbing back in a
>> little bit. Yunhong points out that ACPI DSDT can have overriding
>> objects in the _PRT, but I don''t know it ever actually gets
used on
>> real-world PC systems. So I would try without, but if we do end up
>> needing to get this info from dom0, I think it should be via a new
>> physdev_op.
>>     
> At least dom0 parses this info from DSDT, so we can''t have the
assuption whether it is used or not, I think.
Could you clarify this?  Are you saying that Xen can''t use the DSDT
because dom0 does?
>  And I also agree to add a new physdev_op to handle this case, and it
should be better way to go.
> Based on this idea, I worked out the patch, attached!  In this patch, we
introduced a new physdev_op PHYSDEVOP_setup_gsi for each GSI setup, and each
domain can require to map each GSI in this case.
> In addition, I believe it is very safe to port the hypervisor patch to
xen-3.4-x tree and keeps pv_ops dom0 running on it, since no logic is changed. 
BTW, I also tested apic and non-apic cases, they works fine after applying the
patches.
Could you resend the Linux patch as a delta against your previous
patch?  It makes it easier both to see how things are evolving, and also
so I can reuse previous merges. 

What branch/version is your diff against?

Do we still need the dummy APIC mappings?  Can anything end up touching
them?

Thanks,
    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> At least dom0 parses this info from DSDT, so we can''t have the
assuption whether it is used or not, I think. And I also agree to add a new
physdev_op to handle this case, and it should be better way to go.
> Based on this idea, I worked out the patch, attached!  In this patch, we
introduced a new physdev_op PHYSDEVOP_setup_gsi for each GSI setup, and each
domain can require to map each GSI in this case.
> In addition, I believe it is very safe to port the hypervisor patch to
xen-3.4-x tree and keeps pv_ops dom0 running on it, since no logic is changed. 
BTW, I also tested apic and non-apic cases, they works fine after applying the
patches.
But I don''t think you tested PCI front and PCI back.

Mainly these lines worry me (can you inline the patch next time too, please):

+               map_irq.domid = DOMID_SELF;
+               map_irq.type = MAP_PIRQ_TYPE_GSI;
+               map_irq.index = gsi;
+               map_irq.pirq = irq;
+               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);

For PCI passthrough to work, the domid needs to be for the guest domain, while
in this case it is set to Dom0.

There is already a method of extracting the domain id for PCI devices passed to
the guest. Look in the ''xen_create_msi_irq'' function.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 11/24/09 11:44, Konrad Rzeszutek Wilk wrote:
>> At least dom0 parses this info from DSDT, so we can''t have the
assuption whether it is used or not, I think. And I also agree to add a new
physdev_op to handle this case, and it should be better way to go.
>> Based on this idea, I worked out the patch, attached!  In this patch,
we introduced a new physdev_op PHYSDEVOP_setup_gsi for each GSI setup, and each
domain can require to map each GSI in this case.
>> In addition, I believe it is very safe to port the hypervisor patch to
xen-3.4-x tree and keeps pv_ops dom0 running on it, since no logic is changed. 
BTW, I also tested apic and non-apic cases, they works fine after applying the
patches.
>>     
> But I don''t think you tested PCI front and PCI back.
>
> Mainly these lines worry me (can you inline the patch next time too,
please):
>   
(Inline+attach, or an inline attachment rather than plain inline, is
best.  Plain inline with quoted-printable encoding is awkward to deal with.)
> +               map_irq.domid = DOMID_SELF;
> +               map_irq.type = MAP_PIRQ_TYPE_GSI;
> +               map_irq.index = gsi;
> +               map_irq.pirq = irq;
> +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq,
&map_irq);
>
> For PCI passthrough to work, the domid needs to be for the guest domain,
while
> in this case it is set to Dom0.
>
> There is already a method of extracting the domain id for PCI devices
passed to
> the guest. Look in the ''xen_create_msi_irq'' function.
>   
Hm, I''m not very keen on having xen_create_msi_irq do its own traversal
of xenstore; it should take the domid as a parameter and its caller can
do the walk if necessary.  The direct call makes too much of a direct
dependency between two otherwise unrelated subsystems.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jeremy Fitzhardinge wrote:
> On 11/24/09 02:04, Zhang, Xiantao wrote:
>>> Shoehorning trig/pol information into it as well is kind of nasty.
>>> And I think on any PC system it should suffice to assume GSI 0-15
>>> are ISA edge-triggered active-high, GSI 16+ are PCI level-triggered
>>> active-low, and any exceptions are parsed out of MADT or MPBIOS. We
>>> pretty much have all that code, it just might need plumbing back in
>>> a little bit. Yunhong points out that ACPI DSDT can have overriding
>>> objects in the _PRT, but I don''t know it ever actually
gets used on
>>> real-world PC systems. So I would try without, but if we do end up
>>> needing to get this info from dom0, I think it should be via a new
>>> physdev_op. 
>>> 
>> At least dom0 parses this info from DSDT, so we can''t have the
>> assuption whether it is used or not, I think. 
> 
> Could you clarify this?  Are you saying that Xen can''t use the
DSDT
> because dom0 does?
Dsdt parsing needs AML interpreter, and Xen hasn''t this interpreter, so
it can''t get the info.
>>  And I also agree to add a new physdev_op to handle this case, and
>> it should be better way to go. 
>> Based on this idea, I worked out the patch, attached!  In this
>> patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each
>> GSI setup, and each domain can require to map each GSI in this case.
>> In addition, I believe it is very safe to port the hypervisor patch
>> to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no
>> logic is changed.  BTW, I also tested apic and non-apic cases, they
>> works fine after applying the patches.   
> 
> Could you resend the Linux patch as a delta against your previous
> patch?  It makes it easier both to see how things are evolving, and
> also so I can reuse previous merges.
> 
> What branch/version is your diff against?
I''m against the branch origion/xen/master instead of
origin/xen/master-xiantao.  And the base commit is
"b9160b12ecc71 ". 
> Do we still need the dummy APIC mappings?  Can anything end up
> touching them?
Yes, we still have the dummy mapping in this patch, since ioapic logic and
ioapic info(such as GSI info, the info device link with ioapic) parsing are not
split separately in current Linux.
If we want to end up this, we have to modify much logic in upstream. 
Xiantao


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk wrote:
>> At least dom0 parses this info from DSDT, so we can''t have the
>> assuption whether it is used or not, I think. And I also agree to
>> add a new physdev_op to handle this case, and it should be better
>> way to go.   
>> Based on this idea, I worked out the patch, attached!  In this
>> patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each
>> GSI setup, and each domain can require to map each GSI in this case.
>> In addition, I believe it is very safe to port the hypervisor patch
>> to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no
>> logic is changed.  BTW, I also tested apic and non-apic cases, they
>> works fine after applying the patches.   
> 
> But I don''t think you tested PCI front and PCI back.
> 
> Mainly these lines worry me (can you inline the patch next time too,
> please): 
> 
> +               map_irq.domid = DOMID_SELF;
> +               map_irq.type = MAP_PIRQ_TYPE_GSI;
> +               map_irq.index = gsi;
> +               map_irq.pirq = irq;
> +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq,
> &map_irq); 
> 
> For PCI passthrough to work, the domid needs to be for the guest
> domain, while in this case it is set to Dom0.
> There is already a method of extracting the domain id for PCI devices
> passed to the guest. Look in the ''xen_create_msi_irq''
function.
Could you detail the concern ?  This hypercall is only related to GSI, not MSI,
why it has side-effect about pci passthrough ?

Xiantao


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Nov 25, 2009 at 10:43:47AM +0800, Zhang, Xiantao
wrote:
> Konrad Rzeszutek Wilk wrote:
> >> At least dom0 parses this info from DSDT, so we can''t
have the
> >> assuption whether it is used or not, I think. And I also agree to
> >> add a new physdev_op to handle this case, and it should be better
> >> way to go.   
> >> Based on this idea, I worked out the patch, attached!  In this
> >> patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each
> >> GSI setup, and each domain can require to map each GSI in this
case.
> >> In addition, I believe it is very safe to port the hypervisor
patch
> >> to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no
> >> logic is changed.  BTW, I also tested apic and non-apic cases,
they
> >> works fine after applying the patches.   
> > 
> > But I don''t think you tested PCI front and PCI back.
> > 
> > Mainly these lines worry me (can you inline the patch next time too,
> > please): 
> > 
> > +               map_irq.domid = DOMID_SELF;
> > +               map_irq.type = MAP_PIRQ_TYPE_GSI;
> > +               map_irq.index = gsi;
> > +               map_irq.pirq = irq;
> > +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq,
> > &map_irq); 
> > 
> > For PCI passthrough to work, the domid needs to be for the guest
> > domain, while in this case it is set to Dom0.
> > There is already a method of extracting the domain id for PCI devices
> > passed to the guest. Look in the
''xen_create_msi_irq'' function.
> 
> Could you detail the concern ?  This hypercall is only related to GSI, not
MSI, why it has side-effect about pci passthrough ?
This is for PV guests _without_ using QEMU. They are using the PCI backend to
"enable"
a device (drivers/xen/pciback and drivers/pci/xen-pcifront.c).
The front and back-end communicate the IRQ number (GSI) to the guest
when enabling a INTx PCI device (not MSI ones).

Then the PV guest can bind the IRQ (GSI) number to its own event channel and
have fully working PCI device.

With your change, the privileged domain pins the device to itself, not to
other domains.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Nov 24, 2009 at 03:35:22PM -0800, Jeremy Fitzhardinge
wrote:
> On 11/24/09 11:44, Konrad Rzeszutek Wilk wrote:
> >> At least dom0 parses this info from DSDT, so we can''t
have the assuption whether it is used or not, I think. And I also agree to add a
new physdev_op to handle this case, and it should be better way to go.
> >> Based on this idea, I worked out the patch, attached!  In this
patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each GSI setup,
and each domain can require to map each GSI in this case.
> >> In addition, I believe it is very safe to port the hypervisor
patch to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no logic is
changed.  BTW, I also tested apic and non-apic cases, they works fine after
applying the patches.
> >>     
> > But I don''t think you tested PCI front and PCI back.
> >
> > Mainly these lines worry me (can you inline the patch next time too,
please):
> >   
> 
> (Inline+attach, or an inline attachment rather than plain inline, is
> best.  Plain inline with quoted-printable encoding is awkward to deal
with.)
> 
> > +               map_irq.domid = DOMID_SELF;
> > +               map_irq.type = MAP_PIRQ_TYPE_GSI;
> > +               map_irq.index = gsi;
> > +               map_irq.pirq = irq;
> > +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq,
&map_irq);
> >
> > For PCI passthrough to work, the domid needs to be for the guest
domain, while
> > in this case it is set to Dom0.
> >
> > There is already a method of extracting the domain id for PCI devices
passed to
> > the guest. Look in the ''xen_create_msi_irq''
function.
> >   
> 
> Hm, I''m not very keen on having xen_create_msi_irq do its own
traversal
> of xenstore; it should take the domid as a parameter and its caller can
> do the walk if necessary.  The direct call makes too much of a direct
It would be the easiest way. Unfortunatly the call sequence is the
following:

pciback_do_op
 pciback_enable_msi (dev)
  ... arch_setup_msi_irqs
  ...   arch_setup_msi_irq
  ...     xen_setup_msi_irqs

The caller of the pciback_enable_msi knows of the domain id, but the rest
of the stack does not.  One way of doing this is to provide hooks in event.c,
say:

 [de|]register_msi_owner 

(this is akin to how it was done in 2.6.18.hg)

where the device is associated with a specific domain id. We would
call those register/deregister when a new device has changed ownership
to a domain.

Back to the call trace. When we reach xen_setup_msi_irqs we can find the
relation.
Lookup the device (on this ''registration list'') and see if it
is assigned to a
specific domain.

But in essence it is similar in spirit to the mechanism of the xenbus_walk
- that is to find out what domain id is associated with this device.

With the patches posted by Xiantao this behavior would have to be
emulated for PCI INTx devices as well, or some flavour of it.

So the question is, Jeremy, would you prefer to have a
''[de|]register_device_owner''
call that both MSI and INTx can use and some way of traversing the list this
creates and finding out if there is an owner for the device? Or just leave
it with the xenbus_walk interface?


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk wrote:
> On Wed, Nov 25, 2009 at 10:43:47AM +0800, Zhang, Xiantao wrote:
>> Konrad Rzeszutek Wilk wrote:
>>>> At least dom0 parses this info from DSDT, so we can''t
have the
>>>> assuption whether it is used or not, I think. And I also agree
to
>>>> add a new physdev_op to handle this case, and it should be
better
>>>> way to go. Based on this idea, I worked out the patch,
attached!
>>>> In this patch, we introduced a new physdev_op
PHYSDEVOP_setup_gsi
>>>> for each GSI setup, and each domain can require to map each GSI
in
>>>> this case. In addition, I believe it is very safe to port the
>>>> hypervisor patch to xen-3.4-x tree and keeps pv_ops dom0
running
>>>> on it, since no logic is changed.  BTW, I also tested apic and
>>>> non-apic cases, they works fine after applying the patches.
>>> 
>>> But I don''t think you tested PCI front and PCI back.
>>> 
>>> Mainly these lines worry me (can you inline the patch next time
>>> too, please): 
>>> 
>>> +               map_irq.domid = DOMID_SELF;
>>> +               map_irq.type = MAP_PIRQ_TYPE_GSI;
>>> +               map_irq.index = gsi;
>>> +               map_irq.pirq = irq;
>>> +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq,
>>> &map_irq); 
>>> 
>>> For PCI passthrough to work, the domid needs to be for the guest
>>> domain, while in this case it is set to Dom0.
>>> There is already a method of extracting the domain id for PCI
>>> devices passed to the guest. Look in the
''xen_create_msi_irq''
>>> function. 
>> 
>> Could you detail the concern ?  This hypercall is only related to
>> GSI, not MSI, why it has side-effect about pci passthrough ? 
> 
> This is for PV guests _without_ using QEMU. They are using the PCI
> backend to "enable" a device (drivers/xen/pciback and
> drivers/pci/xen-pcifront.c). 
> The front and back-end communicate the IRQ number (GSI) to the guest
> when enabling a INTx PCI device (not MSI ones).
> 
> Then the PV guest can bind the IRQ (GSI) number to its own event
> channel and 
> have fully working PCI device.
> 
> With your change, the privileged domain pins the device to itself,
> not to 
> other domains.
But I think dom0 should own the device first during boot, and then assign it to
PV guest when this device is required by pcifront?  Basically, we don''t
know which devices should be reserved for non-previleged domains, right ? So I
think the GSI should be initialized and bind to dom0 when dom0 boots?  Once the
devices is assigned to PV guests, it maybe need to do the unmapping operation
about the GSI from dom0 and do mapping for the domU.

BTW, I just met a strange issue with the  function xen_create_msi_irq you
mentioned, and it blocks initialization of SR-IOV devices'' VFs , and I
think it should be a bug.
 
In the function xen_create_msi_irq, there is one line as following to get the
domid of the specified device, but a strange domid(0xfff0) is returned by this
call, could you help to check whether this strange domid is from?
domid = rc = xenbus_walk( "/local/domain/0", get_domid_for_dev, dev);
^^^^^^^
Xiantao 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Nov 25, 2009 at 11:21:45PM +0800, Zhang, Xiantao
wrote:
> Konrad Rzeszutek Wilk wrote:
> > On Wed, Nov 25, 2009 at 10:43:47AM +0800, Zhang, Xiantao wrote:
> >> Konrad Rzeszutek Wilk wrote:
> >>>> At least dom0 parses this info from DSDT, so we
can''t have the
> >>>> assuption whether it is used or not, I think. And I also
agree to
> >>>> add a new physdev_op to handle this case, and it should be
better
> >>>> way to go. Based on this idea, I worked out the patch,
attached!
> >>>> In this patch, we introduced a new physdev_op
PHYSDEVOP_setup_gsi
> >>>> for each GSI setup, and each domain can require to map
each GSI in
> >>>> this case. In addition, I believe it is very safe to port
the
> >>>> hypervisor patch to xen-3.4-x tree and keeps pv_ops dom0
running
> >>>> on it, since no logic is changed.  BTW, I also tested apic
and
> >>>> non-apic cases, they works fine after applying the
patches.
> >>> 
> >>> But I don''t think you tested PCI front and PCI back.
> >>> 
> >>> Mainly these lines worry me (can you inline the patch next
time
> >>> too, please): 
> >>> 
> >>> +               map_irq.domid = DOMID_SELF;
> >>> +               map_irq.type = MAP_PIRQ_TYPE_GSI;
> >>> +               map_irq.index = gsi;
> >>> +               map_irq.pirq = irq;
> >>> +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq,
> >>> &map_irq); 
> >>> 
> >>> For PCI passthrough to work, the domid needs to be for the
guest
> >>> domain, while in this case it is set to Dom0.
> >>> There is already a method of extracting the domain id for PCI
> >>> devices passed to the guest. Look in the
''xen_create_msi_irq''
> >>> function. 
> >> 
> >> Could you detail the concern ?  This hypercall is only related to
> >> GSI, not MSI, why it has side-effect about pci passthrough ? 
> > 
> > This is for PV guests _without_ using QEMU. They are using the PCI
> > backend to "enable" a device (drivers/xen/pciback and
> > drivers/pci/xen-pcifront.c). 
> > The front and back-end communicate the IRQ number (GSI) to the guest
> > when enabling a INTx PCI device (not MSI ones).
> > 
> > Then the PV guest can bind the IRQ (GSI) number to its own event
> > channel and 
> > have fully working PCI device.
> > 
> > With your change, the privileged domain pins the device to itself,
> > not to 
> > other domains.
> 
> But I think dom0 should own the device first during boot, and then assign
it to PV guest when this device is required by pcifront?  Basically, we
don''t know which devices should be reserved for non-previleged domains,
right ? So I think the GSI should be initialized and bind to dom0 when dom0
boots?  Once the devices is assigned to PV guests, it maybe need to do the
unmapping operation about the GSI from dom0 and do mapping for the domU.
During boot the device can be owned by pciback or by the modele for which a
PCI entry exist. Look for pciback.hide entry.

There are two modes of execution:
 1). First being what you described wherein the device initially belogs to Dom0.
     The user unbinds it from the PCI device and binds it to the pciback module.
     At that point, the device is disabled and ready for PV guests. When a PV
guest
     starts pciback module makes the pci_enable_device call and sets the IRQ,
etc
     for the device (for MSI, it obviously gets the IDT value from the
hypervisor
     call).
 2). Dom0 boots where the user specified on the command line
pciback.hide=(0000:01:03.1).
     The pciback owns the device (is binded to it) and the native module that
would
     load for this PCI device is not called.
    
It is correct that the unmapping/mapping and the ownership needs to be dynamic.
As user
could bind to the pciback module, give it to guest, kill the guest, then map the
PCI device
back to dom0, and after that repeat the whole thing.
 
> 
> BTW, I just met a strange issue with the  function xen_create_msi_irq you
mentioned, and it blocks initialization of SR-IOV devices'' VFs , and I
think it should be a bug.
Hmm.. I am not sure if this is the appropiate place for it. You see
this driver is designed for the machines that don''t have SR-IOV, VFs,
or VT-d to be able to passthrough a PCI device to the PV guest. You can
use this to run PV guests on Pentium 4 style machine.

I think that the SR-IOV devices would go through a different call stack
to enable the device? Either way, I recently got my hands on a SR-IOV machine
and will see how this works.

Has the 2.6.31.5 been working before with SR-IOV cards or is this
your first experiment with this.
>  
> In the function xen_create_msi_irq, there is one line as following to get
the domid of the specified device, but a strange domid(0xfff0) is returned by
this call, could you help to check whether this strange domid is from?
> domid = rc = xenbus_walk( "/local/domain/0", get_domid_for_dev,
dev);
That is due to casting (domid is short int, rc is int).
You need to check if rc is negative, and if so set the domid to DOMID_SELF.
The next line below does this:

 if (domid <= 0)
                domid = DOMID_SELF;

Ohh wait, looks like the fix for this never got pushed upstream. That
should have been ''if (rc <= 0)''. Yikes.

rc is -16 (EBUSY). That implies that xenstored has not
been initialized on Dom0.

> ^^^^^^^
> Xiantao 
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 11/25/09 07:21, Zhang, Xiantao wrote:
> In the function xen_create_msi_irq, there is one line as following to get
the domid of the specified device, but a strange domid(0xfff0) is returned by
this call, could you help to check whether this strange domid is from?
> domid = rc = xenbus_walk( "/local/domain/0", get_domid_for_dev,
dev);
>   
That''s DOMID_SELF.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 11/25/09 06:10, Konrad Rzeszutek Wilk wrote:
> On Tue, Nov 24, 2009 at 03:35:22PM -0800, Jeremy Fitzhardinge wrote:
>   
>> On 11/24/09 11:44, Konrad Rzeszutek Wilk wrote:
>>     
>>>> At least dom0 parses this info from DSDT, so we can''t
have the assuption whether it is used or not, I think. And I also agree to add a
new physdev_op to handle this case, and it should be better way to go.
>>>> Based on this idea, I worked out the patch, attached!  In this
patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each GSI setup,
and each domain can require to map each GSI in this case.
>>>> In addition, I believe it is very safe to port the hypervisor
patch to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no logic is
changed.  BTW, I also tested apic and non-apic cases, they works fine after
applying the patches.
>>>>     
>>>>         
>>> But I don''t think you tested PCI front and PCI back.
>>>
>>> Mainly these lines worry me (can you inline the patch next time
too, please):
>>>   
>>>       
>> (Inline+attach, or an inline attachment rather than plain inline, is
>> best.  Plain inline with quoted-printable encoding is awkward to deal
with.)
>>
>>     
>>> +               map_irq.domid = DOMID_SELF;
>>> +               map_irq.type = MAP_PIRQ_TYPE_GSI;
>>> +               map_irq.index = gsi;
>>> +               map_irq.pirq = irq;
>>> +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq,
&map_irq);
>>>
>>> For PCI passthrough to work, the domid needs to be for the guest
domain, while
>>> in this case it is set to Dom0.
>>>
>>> There is already a method of extracting the domain id for PCI
devices passed to
>>> the guest. Look in the ''xen_create_msi_irq''
function.
>>>   
>>>       
>> Hm, I''m not very keen on having xen_create_msi_irq do its own
traversal
>> of xenstore; it should take the domid as a parameter and its caller can
>> do the walk if necessary.  The direct call makes too much of a direct
>>     
> It would be the easiest way. Unfortunatly the call sequence is the
> following:
>
> pciback_do_op
>  pciback_enable_msi (dev)
>   ... arch_setup_msi_irqs
>   ...   arch_setup_msi_irq
>   ...     xen_setup_msi_irqs
>
> The caller of the pciback_enable_msi knows of the domain id, but the rest
> of the stack does not.  One way of doing this is to provide hooks in
event.c, say:
>
>  [de|]register_msi_owner 
>
> (this is akin to how it was done in 2.6.18.hg)
>
> where the device is associated with a specific domain id. We would
> call those register/deregister when a new device has changed ownership
> to a domain.
>
> Back to the call trace. When we reach xen_setup_msi_irqs we can find the
relation.
> Lookup the device (on this ''registration list'') and see
if it is assigned to a
> specific domain.
>
> But in essence it is similar in spirit to the mechanism of the xenbus_walk
> - that is to find out what domain id is associated with this device.
>
> With the patches posted by Xiantao this behavior would have to be
> emulated for PCI INTx devices as well, or some flavour of it.
>
> So the question is, Jeremy, would you prefer to have a
''[de|]register_device_owner''
> call that both MSI and INTx can use and some way of traversing the list
this
> creates and finding out if there is an owner for the device? Or just leave
> it with the xenbus_walk interface?
>   
I don''t particularly object to xenbus_walk per-se, its just that I
don''t
think xen_create_msi_irq() should call it directly.  It would be OK for
xen_setup_msi_irqs() to call it and pass the results into
xen_create_msi_irq().

Would a registration list be any cleaner?  Presumably you''d just keep a
list of devices being controlled by other domains, so non-presence on
the list means DOMID_SELF, so you''d only need to worry about
registration on pciback paths.  If that could be done without needing to
touch common code (or do the horrible hacks that some of the earlier MSI
patches did), then maybe its worthwhile.

xenbus_walk() ends up actually talking to xenstored, so its fairly
expensive, and adds an opportunity for things to go wrong (it is
effectively doing a call out to dom0 usermode from the depths of the pci
code, which doesn''t sound like a terribly healthy activity, but I guess
it has worked OK so far).

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jeremy Fitzhardinge wrote:
> On 11/25/09 07:21, Zhang, Xiantao wrote:
>> In the function xen_create_msi_irq, there is one line as following
>> to get the domid of the specified device, but a strange
>> domid(0xfff0) is returned by this call, could you help to check
>> whether this strange domid is from? domid = rc = xenbus_walk(
>> "/local/domain/0", get_domid_for_dev, dev);   
>> 
> 
> That''s DOMID_SELF.
No, DOMID_SELF is 0x7ff0, not 0xfff0, and it should be -EBUSY, and they are
accidentally similar. :)
Xiantao 




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk wrote:
> On Wed, Nov 25, 2009 at 11:21:45PM +0800, Zhang, Xiantao wrote:
>> Konrad Rzeszutek Wilk wrote:
>>> On Wed, Nov 25, 2009 at 10:43:47AM +0800, Zhang, Xiantao wrote:
>>>> Konrad Rzeszutek Wilk wrote:
>>>>>> At least dom0 parses this info from DSDT, so we
can''t have the
>>>>>> assuption whether it is used or not, I think. And I
also agree to
>>>>>> add a new physdev_op to handle this case, and it should
be better
>>>>>> way to go. Based on this idea, I worked out the patch,
attached!
>>>>>> In this patch, we introduced a new physdev_op
PHYSDEVOP_setup_gsi
>>>>>> for each GSI setup, and each domain can require to map
each GSI
>>>>>> in this case. In addition, I believe it is very safe to
port the
>>>>>> hypervisor patch to xen-3.4-x tree and keeps pv_ops
dom0 running
>>>>>> on it, since no logic is changed.  BTW, I also tested
apic and
>>>>>> non-apic cases, they works fine after applying the
patches.
>>>>> 
>>>>> But I don''t think you tested PCI front and PCI
back.
>>>>> 
>>>>> Mainly these lines worry me (can you inline the patch next
time
>>>>> too, please): 
>>>>> 
>>>>> +               map_irq.domid = DOMID_SELF;
>>>>> +               map_irq.type = MAP_PIRQ_TYPE_GSI;
>>>>> +               map_irq.index = gsi;
>>>>> +               map_irq.pirq = irq;
>>>>> +               rc =
HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq,
>>>>> &map_irq); 
>>>>> 
>>>>> For PCI passthrough to work, the domid needs to be for the
guest
>>>>> domain, while in this case it is set to Dom0.
>>>>> There is already a method of extracting the domain id for
PCI
>>>>> devices passed to the guest. Look in the
''xen_create_msi_irq''
>>>>> function.
>>>> 
>>>> Could you detail the concern ?  This hypercall is only related
to
>>>> GSI, not MSI, why it has side-effect about pci passthrough ?
>>> 
>>> This is for PV guests _without_ using QEMU. They are using the PCI
>>> backend to "enable" a device (drivers/xen/pciback and
>>> drivers/pci/xen-pcifront.c). The front and back-end communicate the
>>> IRQ number (GSI) to the guest when enabling a INTx PCI device (not
>>> MSI ones). 
>>> 
>>> Then the PV guest can bind the IRQ (GSI) number to its own event
>>> channel and have fully working PCI device.
>>> 
>>> With your change, the privileged domain pins the device to itself,
>>> not to other domains.
>> 
>> But I think dom0 should own the device first during boot, and then
>> assign it to PV guest when this device is required by pcifront? 
>> Basically, we don''t know which devices should be reserved for
>> non-previleged domains, right ? So I think the GSI should be
>> initialized and bind to dom0 when dom0 boots?  Once the devices is
>> assigned to PV guests, it maybe need to do the unmapping operation
>> about the GSI from dom0 and do mapping for the domU.      
> 
> During boot the device can be owned by pciback or by the modele for
> which a 
> PCI entry exist. Look for pciback.hide entry.
> 
> There are two modes of execution:
>  1). First being what you described wherein the device initially
>      belogs to Dom0. The user unbinds it from the PCI device and
>      binds it to the pciback module. At that point, the device is
>      disabled and ready for PV guests. When a PV guest starts pciback
>      module makes the pci_enable_device call and sets the IRQ, etc
>      for the device (for MSI, it obviously gets the IDT value from
>  the hypervisor call). 2). Dom0 boots where the user specified on the
>      command line pciback.hide=(0000:01:03.1). The pciback owns the
>      device (is binded to it) and the native module that would load
> for this PCI device is not called. 
> 
> It is correct that the unmapping/mapping and the ownership needs to
> be dynamic. As user could bind to the pciback module, give it to
> guest, kill the guest, then map the PCI device back to dom0, and
> after that repeat the whole thing.
If only the above two cases are needed to support, I think my patch should meet
the requirement. This is because once the device is hidden by the pciback, the
pirq unmap logic should be callled in both cases. So it the mapping for dom0
should be cleared in the operation, so there is not the issue you figured.  :-)
Xiantao 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 
> I don''t particularly object to xenbus_walk per-se, its just that I
don''t
> think xen_create_msi_irq() should call it directly.  It would be OK for
> xen_setup_msi_irqs() to call it and pass the results into
> xen_create_msi_irq().
> 
> Would a registration list be any cleaner?  Presumably you''d just
keep a
> list of devices being controlled by other domains, so non-presence on
> the list means DOMID_SELF, so you''d only need to worry about
> registration on pciback paths.  If that could be done without needing to
> touch common code (or do the horrible hacks that some of the earlier MSI
> patches did), then maybe its worthwhile.
I think that would work nicely. Should have a patch cooked up shortly
after I am done with the xen-fbfront drier in DomU.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > During boot the device can be owned by pciback or by the modele for
> > which a 
> > PCI entry exist. Look for pciback.hide entry.
> > 
> > There are two modes of execution:
> >  1). First being what you described wherein the device initially
> >      belogs to Dom0. The user unbinds it from the PCI device and
> >      binds it to the pciback module. At that point, the device is
> >      disabled and ready for PV guests. When a PV guest starts pciback
> >      module makes the pci_enable_device call and sets the IRQ, etc
> >      for the device (for MSI, it obviously gets the IDT value from
> >  the hypervisor call). 2). Dom0 boots where the user specified on the
> >      command line pciback.hide=(0000:01:03.1). The pciback owns the
> >      device (is binded to it) and the native module that would load
> > for this PCI device is not called. 
> > 
> > It is correct that the unmapping/mapping and the ownership needs to
> > be dynamic. As user could bind to the pciback module, give it to
> > guest, kill the guest, then map the PCI device back to dom0, and
> > after that repeat the whole thing.
> 
> If only the above two cases are needed to support, I think my patch should
meet the requirement. This is because once the device is hidden by the pciback,
the pirq unmap logic should be callled in both cases. So it the mapping for dom0
should be cleared in the operation, so there is not the issue you figured.  :-)
Ah, what about maping the device to a guest? That is not neccessary? Or are you
saying that the PV guest should be making a hypervisor call to
PHYSDEVOP_map_pirq and
it will map the device itself? I thought those calls in the Hypervisor
were guarded by a IS_PRIV(xx) which would expel a PV guest. The other
complication is
that the PV guests have their own PCI subsystem (look in arch/x86/pci/xen.c)
which only calls ''xen_allocate_pirq'' to setup the chip_data to
xen_pirq_chip.

The next call the PV guest does is to startup_pirq.
> Xiantao 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk wrote:
>>> During boot the device can be owned by pciback or by the modele for
>>> which a PCI entry exist. Look for pciback.hide entry.
>>> 
>>> There are two modes of execution:
>>>  1). First being what you described wherein the device initially
>>>      belogs to Dom0. The user unbinds it from the PCI device and
>>>      binds it to the pciback module. At that point, the device is
>>>      disabled and ready for PV guests. When a PV guest starts
>>>      pciback module makes the pci_enable_device call and sets the
>>>      IRQ, etc for the device (for MSI, it obviously gets the IDT
>>>  value from the hypervisor call). 2). Dom0 boots where the user
>>>      specified on the command line pciback.hide=(0000:01:03.1). The
>>>      pciback owns the device (is binded to it) and the native
>>> module that would load 
>>> for this PCI device is not called.
>>> 
>>> It is correct that the unmapping/mapping and the ownership needs to
>>> be dynamic. As user could bind to the pciback module, give it to
>>> guest, kill the guest, then map the PCI device back to dom0, and
>>> after that repeat the whole thing.
>> 
>> If only the above two cases are needed to support, I think my patch
>> should meet the requirement. This is because once the device is
>> hidden by the pciback, the pirq unmap logic should be callled in
>> both cases. So it the mapping for dom0 should be cleared in the
>> operation, so there is not the issue you figured.  :-)    
> 
> Ah, what about maping the device to a guest? That is not neccessary?
> Or are you saying that the PV guest should be making a hypervisor
> call to PHYSDEVOP_map_pirq and it will map the device itself? I
> thought those calls in the Hypervisor 
> were guarded by a IS_PRIV(xx) which would expel a PV guest. The other
> complication is that the PV guests have their own PCI subsystem (look
> in arch/x86/pci/xen.c) 
> which only calls ''xen_allocate_pirq'' to setup the
chip_data to
> xen_pirq_chip. 
> 
> The next call the PV guest does is to startup_pirq.
I am also confusing about how to assign a device to a PV guest. Originally  I
thought the logic should be it should be hidden first through pciback in dom0,
and pv guest uses its pci subsystem (pcifront end) to require pciback to mapping
this device.  In the first step, once pciback.hide logic is called in dom0, 
this device should be released by dom0 and avaliable for PV guests at this time.
And the in subsequent step, pciback or dom0''s pci system should help PV
guests to do the irq mapping, otherwise I can''t see how the irqs from
the assigned device are delivered to PV guests.

Xiantao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I am also confusing about how to assign a device to a PV guest. Originally 
I thought the logic should be it should be hidden first through pciback in dom0,
and pv guest uses its pci subsystem (pcifront end) to require pciback to mapping
this device.  In the first step, once pciback.hide logic is called in dom0, 
this device should be released by dom0 and avaliable for PV guests at this time.
And the in subsequent step, pciback or dom0''s pci system should help PV
guests to do the irq mapping, otherwise I can''t see how the irqs from
the assigned device are delivered to PV guests.
That is correct. Here is an URL of the correct steps:
http://wiki.xensource.com/xenwiki/Assign_hardware_to_DomU_with_PCIBack_as_module
>From the unprivileged side (domU), when it makes a call to
pci_enable_device, it gets routed to pcifront_bus_write.
On the privileged side (dom0), pciback picks up the write
and routes it to ''command_write'' (conf_space_header.c). It
does the pci_enable_device in the dom0 side. The pci_enable_device
ends up calling xen_allocate_pirq which gets the IRQ from
PHYSDEVOP_alloc_irq_vector. That IRQ is saved in dev->irq and
is visible to the DomU.

Also during the pci_enable_device (in the DomU side), pcibios_enable_device
gets called - which in domU is called
''xen_pcifront_enable_irq''.
The xen_pcifront_enable_irq allocates an irq_desc with xen_pirq_chip
structure. The GSI it requests is actually the IRQ number from dev->irq.

To summarize, dom0 on behalf of domU, calls PHYSDEVOP_alloc_irq_vector
for the device in question. Saves the GSI in dev->irq which is visible to
the DomU. DomU sets up a xen_pirq_chip structure for the device and
starts/stop/etc through that function structure. Please note that 
there is nothing in PHYSDEVOP_alloc..  about which domain owns the device.
That is only done with PHYSDEVOP_map_pirq calls.

With your patch instead of PHYSDEVOP_alloc_irq_vector, it would be
PHYSDEVOP_map_pirq.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jeremy Fitzhardinge schrieb:

Hello,
> Well, my current pvops/dom0 tree is finally (reasonably) stable.  There
> was a fairly nasty bug which ended up corrupting dom0 memory when doing
> IO on behalf of domains, but that is finally fixed.
> 
runs this Kernel with xen-3.3.1 or does it need xen.-3.4.x?
My xen-3.3.1 tells me that my Kernel ist noc ELF binary.

Regards,
Stefan Kuhne



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Dec 04, 2009 at 05:07:32PM +0100, Stefan Kuhne
wrote:
> Jeremy Fitzhardinge schrieb:
> 
> Hello,
> 
> > Well, my current pvops/dom0 tree is finally (reasonably) stable. 
There
> > was a fairly nasty bug which ended up corrupting dom0 memory when
doing
> > IO on behalf of domains, but that is finally fixed.
> > 
> runs this Kernel with xen-3.3.1 or does it need xen.-3.4.x?
> My xen-3.3.1 tells me that my Kernel ist noc ELF binary.
> 
Please read this wiki page:
http://wiki.xensource.com/xenwiki/XenParavirtOps

Especially the "Xen requirements for using pv_ops dom0 kernel"
chapter.

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 12/04/09 08:07, Stefan Kuhne wrote:
> runs this Kernel with xen-3.3.1 or does it need xen.-3.4.x?
> My xen-3.3.1 tells me that my Kernel ist noc ELF binary.
>    
If you''re using an old Xen, you need to use the vmlinux(.gz) file as 
your kernel image.  Newer versions of Xen can directly load the bzImage.

     J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Oct 27, 2009 at 04:13:06PM -0400, Konrad Rzeszutek Wilk
wrote:
> > > As in, even on baremetal it does not work?
> > 
> > Heh. should have tried that already. Too tired atm :)
> > 
> > Anyway, I tried baremetal now. X comes up, I can see the desktop, and
> > then the monitor goes out of signal.
> > 
> > "dmesg" is full of:
> > 
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > X used greatest stack depth: 3000 bytes left
> > 
> > Could that be caused by the patch? I booted the same ttm-patched
kernel
> 
> I fear so. 
> 
> > on baremetal without Xen..
> 
> By next week I should have a working machine where I can do the
mode-setting
> and dig some deeper in this.
> 
> Thanks so far for testing some ideas out.

Hello again and happy new year!

Konrad: Have you had time to look at this yet? 

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Jan 01, 2010 at 07:21:37PM +0200, Pasi Kärkkäinen
wrote:
> On Tue, Oct 27, 2009 at 04:13:06PM -0400, Konrad Rzeszutek Wilk wrote:
> > > > As in, even on baremetal it does not work?
> > > 
> > > Heh. should have tried that already. Too tired atm :)
> > > 
> > > Anyway, I tried baremetal now. X comes up, I can see the desktop,
and
> > > then the monitor goes out of signal.
> > > 
> > > "dmesg" is full of:
> > > 
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > X used greatest stack depth: 3000 bytes left
> > > 
> > > Could that be caused by the patch? I booted the same ttm-patched
kernel
> > 
> > I fear so. 
> > 
> > > on baremetal without Xen..
> > 
> > By next week I should have a working machine where I can do the
mode-setting
> > and dig some deeper in this.
> > 
> > Thanks so far for testing some ideas out.
> 
> 
> Hello again and happy new year!
Heya!
> 
> Konrad: Have you had time to look at this yet? 
Uhh, ... umm.. Thank you for reminding me to look at it. I am right now
doing a experimental swiotlb that I want to post to iommu-devel and see what
they think about it.

After that I am going back to revisit this bug.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Jan 04, 2010 at 08:37:28AM -0500, Konrad Rzeszutek Wilk
wrote:
> On Fri, Jan 01, 2010 at 07:21:37PM +0200, Pasi Kärkkäinen wrote:
> > On Tue, Oct 27, 2009 at 04:13:06PM -0400, Konrad Rzeszutek Wilk wrote:
> > > > > As in, even on baremetal it does not work?
> > > > 
> > > > Heh. should have tried that already. Too tired atm :)
> > > > 
> > > > Anyway, I tried baremetal now. X comes up, I can see the
desktop, and
> > > > then the monitor goes out of signal.
> > > > 
> > > > "dmesg" is full of:
> > > > 
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > X used greatest stack depth: 3000 bytes left
> > > > 
> > > > Could that be caused by the patch? I booted the same
ttm-patched kernel
> > > 
> > > I fear so. 
> > > 
> > > > on baremetal without Xen..
> > > 
> > > By next week I should have a working machine where I can do the
mode-setting
> > > and dig some deeper in this.
> > > 
> > > Thanks so far for testing some ideas out.
> > 
> > 
> > Hello again and happy new year!
> 
> Heya!
> > 
> > Konrad: Have you had time to look at this yet? 
> 
> Uhh, ... umm.. Thank you for reminding me to look at it. I am right now
> doing a experimental swiotlb that I want to post to iommu-devel and see
what
> they think about it.
> 
> After that I am going back to revisit this bug.
>
Ok, thanks for the update :)
Let me know if/when you have some patch to test.

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Ok, thanks for the update :)
> Let me know if/when you have some patch to test.
Trying to refresh my memory. The issue was that under Xen, the DRM
modesetting with your radeon card (ES1000) would fail during the
ring buffer initialization. I provided a test-path that made it go
away, but now you get those Free Memory leak.

So is the Memory leak happening after running X for some time or
pretty much immediately?

Thanks.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Jan 14, 2010 at 03:05:48PM -0500, Konrad Rzeszutek Wilk
wrote:
> > Ok, thanks for the update :)
> > Let me know if/when you have some patch to test.
> 
> Trying to refresh my memory. The issue was that under Xen, the DRM
> modesetting with your radeon card (ES1000) would fail during the
> ring buffer initialization. I provided a test-path that made it go
> away, but now you get those Free Memory leak.
> 
Let''s see.

You sent me this patch:
http://lists.xensource.com/archives/html/xen-devel/2009-10/msg00986.html

And then kernel modesetting started working:
http://lists.xensource.com/archives/html/xen-devel/2009-10/msg01014.html

Then you asked me to try running X and getting some debug stuff, but it
seems X still doesn''t work for me:
http://lists.xensource.com/archives/html/xen-devel/2009-10/msg01309.html

Some analysis and it seems the patch causes TTM "Erroneous page count.
Leaking pages." errors:
http://lists.xensource.com/archives/html/xen-devel/2009-10/msg01316.html
> So is the Memory leak happening after running X for some time or
> pretty much immediately?
> 
Starting X causes the errors.

If I don''t start X then the system runs fine without errors. Setting
the
graphics mode during boot/initrd works OK.

I can re-rest things if needed.

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel