Xen devel - Mar 2013 - GPU passthrough issue when VM is configured with 4G memory

Hi,all
I have tried to passthrough GPU card(Nvidia quadro 4000) on the latest Xen
unstable version (QEMU is using Qemu-upsteam-unstable, not traditional Qemu).
This issue as below:
       Windows7 64-bit guest will blue screen when GPU passthrough configure 4g
memory,blue screen code is 50, and SUSE 11 64-bit guest will always stay at the
grub screen.  I noticed that it will relocate RAM that overlaps PCI space in
pci_setup()(tools/hvmloader/pci.c). If VM memory is configured with 3G, it
won''t cause relocate RAM that overlaps PCI space in pci_setup(), and
GPU pass-through is no problem. So it seems this issue is related to
"relocate RAM" in pci_setup().
       In the failure case (VM memory is 4G), I used "memtest" to
check memory of the VM which configured with more than 4G memory, the last 256M
has errors.

BTW, Xen 4.1.2 doesn''t have this issue.

Any ideas about this issue? Thanks in advance.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Mon, Mar 04, 2013 at 08:10:10AM +0000, Gonglei (Arei)
wrote:
>    Hi,all
> 
Hello,
>    I have tried to passthrough GPU card(Nvidia quadro 4000) on the latest
Xen
>    unstable version (QEMU is using Qemu-upsteam-unstable, not traditional
>    Qemu). This issue as below:
> 
I don''t think qemu-upstream has GPU/VGA passthrough support yet.

>           Windows7 64-bit guest will blue screen when GPU passthrough
>    configure 4g memory,blue screen code is 50, and SUSE 11 64-bit guest
will
>    always stay at the grub screen.  I noticed that it will relocate RAM
that
>    overlaps PCI space in pci_setup()(tools/hvmloader/pci.c). If VM memory
is
>    configured with 3G, it won''t cause relocate RAM that overlaps
PCI space in
>    pci_setup(), and GPU pass-through is no problem. So it seems this issue
is
>    related to "relocate RAM" in pci_setup().
> 
>           In the failure case (VM memory is 4G), I used "memtest"
to check
>    memory of the VM which configured with more than 4G memory, the last
256M
>    has errors.
> 
> 
> 
>    BTW, Xen 4.1.2 doesn''t have this issue.
> 
I assume with Xen 4.1.2 you''re using qemu-traditional.. ? 

Try using qemu-traditional also with xen-unstable.

-- Pasi
> 
> 
>    Any ideas about this issue? Thanks in advance.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

> 
> >    I have tried to passthrough GPU card(Nvidia quadro 4000) on the
latest
> Xen
> >    unstable version (QEMU is using Qemu-upsteam-unstable, not
traditional
> >    Qemu). This issue as below:
> >
> 
> I don''t think qemu-upstream has GPU/VGA passthrough support yet.
Qemu-upstream already supports GPU/VGA pass-through. If we configure VM memory
with 3G, GPU pass-through works well.
> 
> 
> >           Windows7 64-bit guest will blue screen when GPU passthrough
> >    configure 4g memory,blue screen code is 50, and SUSE 11 64-bit
guest
> will
> >    always stay at the grub screen.  I noticed that it will relocate
RAM that
> >    overlaps PCI space in pci_setup()(tools/hvmloader/pci.c). If VM
memory
> is
> >    configured with 3G, it won''t cause relocate RAM that
overlaps PCI space
> in
> >    pci_setup(), and GPU pass-through is no problem. So it seems this
issue
> is
> >    related to "relocate RAM" in pci_setup().
> >
> >           In the failure case (VM memory is 4G), I used
"memtest" to
> check
> >    memory of the VM which configured with more than 4G memory, the
last
> 256M
> >    has errors.
> >
> >
> >
> >    BTW, Xen 4.1.2 doesn''t have this issue.
> >
> 
> I assume with Xen 4.1.2 you''re using qemu-traditional.. ?
 Yes, we tried Xen 4.1.2 with qemu-traditional. 
> 
> Try using qemu-traditional also with xen-unstable.
> 
OK, we will have a try. But seems it''s not qemu''s problem, we
can make GPU pass-through succeed if we didn''t do
XENMAPSPACE_gmfn_range hypercall in pci_setup() with 4G memory. 

--Weidong
> -- Pasi
> 
> >
> >
> >    Any ideas about this issue? Thanks in advance.
> >
> >
> 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel

On Mon, Mar 4, 2013 at 8:10 AM, Gonglei (Arei) <arei.gonglei@huawei.com>
wrote:
> Hi,all
>
> I have tried to passthrough GPU card(Nvidia quadro 4000) on the latest Xen
> unstable version (QEMU is using Qemu-upsteam-unstable, not traditional
> Qemu). This issue as below:
>
>        Windows7 64-bit guest will blue screen when GPU passthrough
configure
> 4g memory,blue screen code is 50, and SUSE 11 64-bit guest will always stay
> at the grub screen.  I noticed that it will relocate RAM that overlaps PCI
> space in pci_setup()(tools/hvmloader/pci.c). If VM memory is configured
with
> 3G, it won''t cause relocate RAM that overlaps PCI space in
pci_setup(), and
> GPU pass-through is no problem. So it seems this issue is related to
> "relocate RAM" in pci_setup().
So one issue XenServer found with passing through GPUs is that there
are bugs in some PCI bridges that completely break VT-d.  The issue
was that if the *guest* physical address space overlapped the *host*
physical address of a different device, that the PCI bridges would
send traffic from the passed-through card intended for the guest to
another card instead.  The work-around was to make the hole in the
guest MMIO space the same size as the host MMIO hole.  I''m not sure if
that made it upstream or not -- let me check...

 -George

On Tue, Mar 05, 2013 at 12:44:36PM +0000, Hanweidong
wrote:
> > 
> > >    I have tried to passthrough GPU card(Nvidia quadro 4000) on
the latest
> > Xen
> > >    unstable version (QEMU is using Qemu-upsteam-unstable, not
traditional
> > >    Qemu). This issue as below:
> > >
> > 
> > I don''t think qemu-upstream has GPU/VGA passthrough support
yet.
> 
> Qemu-upstream already supports GPU/VGA pass-through.
>
Really? With Xen? 
I haven''t seen the patches.. 
> If we configure VM memory with 3G, GPU pass-through works well.
> 
Right..

> > 
> > 
> > >           Windows7 64-bit guest will blue screen when GPU
passthrough
> > >    configure 4g memory,blue screen code is 50, and SUSE 11 64-bit
guest
> > will
> > >    always stay at the grub screen.  I noticed that it will
relocate RAM that
> > >    overlaps PCI space in pci_setup()(tools/hvmloader/pci.c). If
VM memory
> > is
> > >    configured with 3G, it won''t cause relocate RAM that
overlaps PCI space
> > in
> > >    pci_setup(), and GPU pass-through is no problem. So it seems
this issue
> > is
> > >    related to "relocate RAM" in pci_setup().
> > >
> > >           In the failure case (VM memory is 4G), I used
"memtest" to
> > check
> > >    memory of the VM which configured with more than 4G memory,
the last
> > 256M
> > >    has errors.
> > >
> > >
> > >
> > >    BTW, Xen 4.1.2 doesn''t have this issue.
> > >
> > 
> > I assume with Xen 4.1.2 you''re using qemu-traditional.. ?
> 
>  Yes, we tried Xen 4.1.2 with qemu-traditional. 
> 
> > 
> > Try using qemu-traditional also with xen-unstable.
> > 
> 
> OK, we will have a try. But seems it''s not qemu''s
problem, we can make GPU pass-through succeed if we didn''t do
> XENMAPSPACE_gmfn_range hypercall in pci_setup() with 4G memory. 
> 
Yep. Please send patches when you figure it out!


-- Pasi


> --Weidong
> 
> > -- Pasi
> > 
> > >
> > >
> > >    Any ideas about this issue? Thanks in advance.
> > >
> > >
> > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xen.org
> > > http://lists.xen.org/xen-devel
>

I can recreate the issue:

Whereas xen-unstable does VGA passthrough fine with more then 4G RAM
with traditional qemu, xen''s qemu  upstream does only work with memory
< 4G.


2013/3/5 Pasi Kärkkäinen <pasik@iki.fi>:
> On Tue, Mar 05, 2013 at 12:44:36PM +0000, Hanweidong wrote:
>> >
>> > >    I have tried to passthrough GPU card(Nvidia quadro 4000)
on the latest
>> > Xen
>> > >    unstable version (QEMU is using Qemu-upsteam-unstable, not
traditional
>> > >    Qemu). This issue as below:
>> > >
>> >
>> > I don''t think qemu-upstream has GPU/VGA passthrough
support yet.
>>
>> Qemu-upstream already supports GPU/VGA pass-through.
>>
>
> Really? With Xen?
> I haven''t seen the patches..
>
>> If we configure VM memory with 3G, GPU pass-through works well.
>>
>
> Right..
>
>
>> >
>> >
>> > >           Windows7 64-bit guest will blue screen when GPU
passthrough
>> > >    configure 4g memory,blue screen code is 50, and SUSE 11
64-bit guest
>> > will
>> > >    always stay at the grub screen.  I noticed that it will
relocate RAM that
>> > >    overlaps PCI space in pci_setup()(tools/hvmloader/pci.c).
If VM memory
>> > is
>> > >    configured with 3G, it won''t cause relocate RAM
that overlaps PCI space
>> > in
>> > >    pci_setup(), and GPU pass-through is no problem. So it
seems this issue
>> > is
>> > >    related to "relocate RAM" in pci_setup().
>> > >
>> > >           In the failure case (VM memory is 4G), I used
"memtest" to
>> > check
>> > >    memory of the VM which configured with more than 4G
memory, the last
>> > 256M
>> > >    has errors.
>> > >
>> > >
>> > >
>> > >    BTW, Xen 4.1.2 doesn''t have this issue.
>> > >
>> >
>> > I assume with Xen 4.1.2 you''re using qemu-traditional.. ?
>>
>>  Yes, we tried Xen 4.1.2 with qemu-traditional.
>>
>> >
>> > Try using qemu-traditional also with xen-unstable.
>> >
>>
>> OK, we will have a try. But seems it''s not qemu''s
problem, we can make GPU pass-through succeed if we didn''t do
>> XENMAPSPACE_gmfn_range hypercall in pci_setup() with 4G memory.
>>
>
> Yep. Please send patches when you figure it out!
>
>
> -- Pasi
>
>
>
>> --Weidong
>>
>> > -- Pasi
>> >
>> > >
>> > >
>> > >    Any ideas about this issue? Thanks in advance.
>> > >
>> > >
>> >
>> > > _______________________________________________
>> > > Xen-devel mailing list
>> > > Xen-devel@lists.xen.org
>> > > http://lists.xen.org/xen-devel
>>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

On Tue, Mar 05, 2013 at 03:21:46PM +0100, Matthias
wrote:
> I can recreate the issue:
> 
> Whereas xen-unstable does VGA passthrough fine with more then 4G RAM
> with traditional qemu, xen''s qemu  upstream does only work with
memory
> < 4G.
> 
Does "normal" PCI passthrough work with qemu upstream + xen-unstable
and >4G RAM?
Say, a NIC, or a USB controller, or a soundcard.

-- Pasi
> 
> 2013/3/5 Pasi Kärkkäinen <pasik@iki.fi>:
> > On Tue, Mar 05, 2013 at 12:44:36PM +0000, Hanweidong wrote:
> >> >
> >> > >    I have tried to passthrough GPU card(Nvidia quadro
4000) on the latest
> >> > Xen
> >> > >    unstable version (QEMU is using
Qemu-upsteam-unstable, not traditional
> >> > >    Qemu). This issue as below:
> >> > >
> >> >
> >> > I don''t think qemu-upstream has GPU/VGA passthrough
support yet.
> >>
> >> Qemu-upstream already supports GPU/VGA pass-through.
> >>
> >
> > Really? With Xen?
> > I haven''t seen the patches..
> >
> >> If we configure VM memory with 3G, GPU pass-through works well.
> >>
> >
> > Right..
> >
> >
> >> >
> >> >
> >> > >           Windows7 64-bit guest will blue screen when
GPU passthrough
> >> > >    configure 4g memory,blue screen code is 50, and SUSE
11 64-bit guest
> >> > will
> >> > >    always stay at the grub screen.  I noticed that it
will relocate RAM that
> >> > >    overlaps PCI space in
pci_setup()(tools/hvmloader/pci.c). If VM memory
> >> > is
> >> > >    configured with 3G, it won''t cause relocate
RAM that overlaps PCI space
> >> > in
> >> > >    pci_setup(), and GPU pass-through is no problem. So
it seems this issue
> >> > is
> >> > >    related to "relocate RAM" in pci_setup().
> >> > >
> >> > >           In the failure case (VM memory is 4G), I used
"memtest" to
> >> > check
> >> > >    memory of the VM which configured with more than 4G
memory, the last
> >> > 256M
> >> > >    has errors.
> >> > >
> >> > >
> >> > >
> >> > >    BTW, Xen 4.1.2 doesn''t have this issue.
> >> > >
> >> >
> >> > I assume with Xen 4.1.2 you''re using
qemu-traditional.. ?
> >>
> >>  Yes, we tried Xen 4.1.2 with qemu-traditional.
> >>
> >> >
> >> > Try using qemu-traditional also with xen-unstable.
> >> >
> >>
> >> OK, we will have a try. But seems it''s not
qemu''s problem, we can make GPU pass-through succeed if we
didn''t do
> >> XENMAPSPACE_gmfn_range hypercall in pci_setup() with 4G memory.
> >>
> >
> > Yep. Please send patches when you figure it out!
> >
> >
> > -- Pasi
> >
> >
> >
> >> --Weidong
> >>
> >> > -- Pasi
> >> >
> >> > >
> >> > >
> >> > >    Any ideas about this issue? Thanks in advance.
> >> > >
> >> > >
> >> >
> >> > > _______________________________________________
> >> > > Xen-devel mailing list
> >> > > Xen-devel@lists.xen.org
> >> > > http://lists.xen.org/xen-devel
> >>
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel

Yes, normal PCI Passthrough works fine with like 5GB of Ram (tested
with 3 USB controller and 1 Audio device).

But the moment I use vga passthrough (using an AMD card with secondary
passthrough), vnc stays black all the way till i get the bluescreen.

2013/3/5 Pasi Kärkkäinen <pasik@iki.fi>:
> On Tue, Mar 05, 2013 at 03:21:46PM +0100, Matthias wrote:
>> I can recreate the issue:
>>
>> Whereas xen-unstable does VGA passthrough fine with more then 4G RAM
>> with traditional qemu, xen''s qemu  upstream does only work
with memory
>> < 4G.
>>
>
> Does "normal" PCI passthrough work with qemu upstream +
xen-unstable and >4G RAM?
> Say, a NIC, or a USB controller, or a soundcard.
>
> -- Pasi
>
>>
>> 2013/3/5 Pasi Kärkkäinen <pasik@iki.fi>:
>> > On Tue, Mar 05, 2013 at 12:44:36PM +0000, Hanweidong wrote:
>> >> >
>> >> > >    I have tried to passthrough GPU card(Nvidia
quadro 4000) on the latest
>> >> > Xen
>> >> > >    unstable version (QEMU is using
Qemu-upsteam-unstable, not traditional
>> >> > >    Qemu). This issue as below:
>> >> > >
>> >> >
>> >> > I don''t think qemu-upstream has GPU/VGA
passthrough support yet.
>> >>
>> >> Qemu-upstream already supports GPU/VGA pass-through.
>> >>
>> >
>> > Really? With Xen?
>> > I haven''t seen the patches..
>> >
>> >> If we configure VM memory with 3G, GPU pass-through works
well.
>> >>
>> >
>> > Right..
>> >
>> >
>> >> >
>> >> >
>> >> > >           Windows7 64-bit guest will blue screen
when GPU passthrough
>> >> > >    configure 4g memory,blue screen code is 50, and
SUSE 11 64-bit guest
>> >> > will
>> >> > >    always stay at the grub screen.  I noticed that
it will relocate RAM that
>> >> > >    overlaps PCI space in
pci_setup()(tools/hvmloader/pci.c). If VM memory
>> >> > is
>> >> > >    configured with 3G, it won''t cause
relocate RAM that overlaps PCI space
>> >> > in
>> >> > >    pci_setup(), and GPU pass-through is no problem.
So it seems this issue
>> >> > is
>> >> > >    related to "relocate RAM" in
pci_setup().
>> >> > >
>> >> > >           In the failure case (VM memory is 4G), I
used "memtest" to
>> >> > check
>> >> > >    memory of the VM which configured with more than
4G memory, the last
>> >> > 256M
>> >> > >    has errors.
>> >> > >
>> >> > >
>> >> > >
>> >> > >    BTW, Xen 4.1.2 doesn''t have this issue.
>> >> > >
>> >> >
>> >> > I assume with Xen 4.1.2 you''re using
qemu-traditional.. ?
>> >>
>> >>  Yes, we tried Xen 4.1.2 with qemu-traditional.
>> >>
>> >> >
>> >> > Try using qemu-traditional also with xen-unstable.
>> >> >
>> >>
>> >> OK, we will have a try. But seems it''s not
qemu''s problem, we can make GPU pass-through succeed if we
didn''t do
>> >> XENMAPSPACE_gmfn_range hypercall in pci_setup() with 4G
memory.
>> >>
>> >
>> > Yep. Please send patches when you figure it out!
>> >
>> >
>> > -- Pasi
>> >
>> >
>> >
>> >> --Weidong
>> >>
>> >> > -- Pasi
>> >> >
>> >> > >
>> >> > >
>> >> > >    Any ideas about this issue? Thanks in advance.
>> >> > >
>> >> > >
>> >> >
>> >> > > _______________________________________________
>> >> > > Xen-devel mailing list
>> >> > > Xen-devel@lists.xen.org
>> >> > > http://lists.xen.org/xen-devel
>> >>
>> >
>> > _______________________________________________
>> > Xen-devel mailing list
>> > Xen-devel@lists.xen.org
>> > http://lists.xen.org/xen-devel

> -----Original Message-----
> From: Pasi Kärkkäinen [mailto:pasik@iki.fi]
> Sent: 2013年3月5日 21:21
> To: Hanweidong
> Cc: Gonglei (Arei); xen-devel@lists.xen.org; Yangxiaowei; Yanqiangjun;
> Luonengjun; Wangzhenguo
> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> On Tue, Mar 05, 2013 at 12:44:36PM +0000, Hanweidong wrote:
> > >
> > > >    I have tried to passthrough GPU card(Nvidia quadro 4000)
on
> the latest
> > > Xen
> > > >    unstable version (QEMU is using Qemu-upsteam-unstable,
not
> traditional
> > > >    Qemu). This issue as below:
> > > >
> > >
> > > I don't think qemu-upstream has GPU/VGA passthrough support
yet.
> >
> > Qemu-upstream already supports GPU/VGA pass-through.
> >
> 
> Really? With Xen?
> I haven't seen the patches..
Yes, QEMU-upstream doesn't have VGA pass-through. We just assign GPU as
a common pci device to VM, and it can work after loading driver.
> 
> > If we configure VM memory with 3G, GPU pass-through works well.
> >
> 
> Right..
> 
> 
> > >
> > >
> > > >           Windows7 64-bit guest will blue screen when GPU
> passthrough
> > > >    configure 4g memory,blue screen code is 50, and SUSE 11
64-bit
> guest
> > > will
> > > >    always stay at the grub screen.  I noticed that it will
> relocate RAM that
> > > >    overlaps PCI space in pci_setup()(tools/hvmloader/pci.c).
If
> VM memory
> > > is
> > > >    configured with 3G, it won't cause relocate RAM that
overlaps
> PCI space
> > > in
> > > >    pci_setup(), and GPU pass-through is no problem. So it
seems
> this issue
> > > is
> > > >    related to "relocate RAM" in pci_setup().
> > > >
> > > >           In the failure case (VM memory is 4G), I used
"memtest"
> to
> > > check
> > > >    memory of the VM which configured with more than 4G
memory,
> the last
> > > 256M
> > > >    has errors.
> > > >
> > > >
> > > >
> > > >    BTW, Xen 4.1.2 doesn't have this issue.
> > > >
> > >
> > > I assume with Xen 4.1.2 you're using qemu-traditional.. ?
> >
> >  Yes, we tried Xen 4.1.2 with qemu-traditional.
> >
> > >
> > > Try using qemu-traditional also with xen-unstable.
> > >
> >
> > OK, we will have a try. But seems it's not qemu's problem, we
can
> make GPU pass-through succeed if we didn't do
> > XENMAPSPACE_gmfn_range hypercall in pci_setup() with 4G memory.
> >
> 
> Yep. Please send patches when you figure it out!
> 
We found GPU pass-through worked if using qemu-traditional. So it looks 
there is some relationship between XENMAPSPACE_gmfn_range hypercall and 
qemu. In addition, we found it's fine to assign GPU to Win7 32bit guest
 with 4G memory.

--Weidong
> 
> -- Pasi
> 
> 
> 
> > --Weidong
> >
> > > -- Pasi
> > >
> > > >
> > > >
> > > >    Any ideas about this issue? Thanks in advance.
> > > >
> > > >
> > >
> > > > _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.xen.org
> > > > http://lists.xen.org/xen-devel
> >
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

> -----Original Message-----
> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of George
> Dunlap
> Sent: 2013年3月5日 20:59
> To: Gonglei (Arei)
> Cc: xen-devel@lists.xen.org; Yangxiaowei; Yanqiangjun; Luonengjun;
> Wangzhenguo; Hanweidong
> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> On Mon, Mar 4, 2013 at 8:10 AM, Gonglei (Arei)
<arei.gonglei@huawei.com>
> wrote:
> > Hi,all
> >
> > I have tried to passthrough GPU card(Nvidia quadro 4000) on the
> latest Xen
> > unstable version (QEMU is using Qemu-upsteam-unstable, not
> traditional
> > Qemu). This issue as below:
> >
> >        Windows7 64-bit guest will blue screen when GPU passthrough
> configure
> > 4g memory,blue screen code is 50, and SUSE 11 64-bit guest will
> always stay
> > at the grub screen.  I noticed that it will relocate RAM that
> overlaps PCI
> > space in pci_setup()(tools/hvmloader/pci.c). If VM memory is
> configured with
> > 3G, it won't cause relocate RAM that overlaps PCI space in
> pci_setup(), and
> > GPU pass-through is no problem. So it seems this issue is related to
> > "relocate RAM" in pci_setup().
> 
> So one issue XenServer found with passing through GPUs is that there
> are bugs in some PCI bridges that completely break VT-d.  The issue
> was that if the *guest* physical address space overlapped the *host*
> physical address of a different device, that the PCI bridges would
> send traffic from the passed-through card intended for the guest to
> another card instead.  The work-around was to make the hole in the
> guest MMIO space the same size as the host MMIO hole.  I'm not sure if
> that made it upstream or not -- let me check...
> 
Hi George,

Could you post your patch and let us have a try with it? Thanks!

--Weidong 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 06/03/13 11:38, Hanweidong wrote:
>> -----Original Message-----
>> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of George
>> Dunlap
>> Sent: 2013年3月5日 20:59
>> To: Gonglei (Arei)
>> Cc: xen-devel@lists.xen.org; Yangxiaowei; Yanqiangjun; Luonengjun;
>> Wangzhenguo; Hanweidong
>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
>> with 4G memory
>>
>> On Mon, Mar 4, 2013 at 8:10 AM, Gonglei (Arei)
<arei.gonglei@huawei.com>
>> wrote:
>>> Hi,all
>>>
>>> I have tried to passthrough GPU card(Nvidia quadro 4000) on the
>> latest Xen
>>> unstable version (QEMU is using Qemu-upsteam-unstable, not
>> traditional
>>> Qemu). This issue as below:
>>>
>>>        Windows7 64-bit guest will blue screen when GPU passthrough
>> configure
>>> 4g memory,blue screen code is 50, and SUSE 11 64-bit guest will
>> always stay
>>> at the grub screen.  I noticed that it will relocate RAM that
>> overlaps PCI
>>> space in pci_setup()(tools/hvmloader/pci.c). If VM memory is
>> configured with
>>> 3G, it won't cause relocate RAM that overlaps PCI space in
>> pci_setup(), and
>>> GPU pass-through is no problem. So it seems this issue is related
to
>>> "relocate RAM" in pci_setup().
>> So one issue XenServer found with passing through GPUs is that there
>> are bugs in some PCI bridges that completely break VT-d.  The issue
>> was that if the *guest* physical address space overlapped the *host*
>> physical address of a different device, that the PCI bridges would
>> send traffic from the passed-through card intended for the guest to
>> another card instead.  The work-around was to make the hole in the
>> guest MMIO space the same size as the host MMIO hole.  I'm not sure
if
>> that made it upstream or not -- let me check...
>>
> Hi George,
>
> Could you post your patch and let us have a try with it? Thanks!
So the patch got checked in, but there still may be some more work if
you want to use it. :-)

The patch modifies xc_hvm_build_args structure to include a field called
"mmio_size". If this is set to zero, it will default to
HVM_BELOW_4G_MMIO_LENGTH; otherwise, it will be the size of the default
MMIO hole set during the build process. The guest BIOS may modify this
at boot time to make it bigger, but it doesn't make it smaller.

Since this was designed for xapi, however, which calls libxc directly,
we didn't add any options to xend / xl / libxl to set this option.

The easiest way to test it probably is just to hard-code
HVM_BELOW_4G_MMIO_LENGTH to a new value (from the description, setting
it to 1GiB should be sufficient).

Then if you want to use it in production, you probably want to either:
1. Try it with the latest version of XCP (which I think has an option
you can set)
2. Implement a config option for xl that allows you to set the MMIO hole
size.

#2 should be a relatively straightforward matter of "plumbing", and
would be a welcome contribution. :-)

If you do implement #2, it might be nice to have an option of
"mmio_hole_size=host", which will set the guest mmio hole to the same
size as the host. That's what we implemented for XenServer, to make sure
there would never be any collisions.

-George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Wed, Mar 06, 2013 at 12:43:09PM +0000, George Dunlap
wrote:
> On 06/03/13 11:38, Hanweidong wrote:
> >> -----Original Message-----
> >> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of
George
> >> Dunlap
> >> Sent: 2013??3??5?? 20:59
> >> To: Gonglei (Arei)
> >> Cc: xen-devel@lists.xen.org; Yangxiaowei; Yanqiangjun; Luonengjun;
> >> Wangzhenguo; Hanweidong
> >> Subject: Re: [Xen-devel] GPU passthrough issue when VM is
configured
> >> with 4G memory
> >>
> >> On Mon, Mar 4, 2013 at 8:10 AM, Gonglei (Arei)
<arei.gonglei@huawei.com>
> >> wrote:
> >>> Hi,all
> >>>
> >>> I have tried to passthrough GPU card(Nvidia quadro 4000) on
the
> >> latest Xen
> >>> unstable version (QEMU is using Qemu-upsteam-unstable, not
> >> traditional
> >>> Qemu). This issue as below:
> >>>
> >>>        Windows7 64-bit guest will blue screen when GPU
passthrough
> >> configure
> >>> 4g memory,blue screen code is 50, and SUSE 11 64-bit guest
will
> >> always stay
> >>> at the grub screen.  I noticed that it will relocate RAM that
> >> overlaps PCI
> >>> space in pci_setup()(tools/hvmloader/pci.c). If VM memory is
> >> configured with
> >>> 3G, it won''t cause relocate RAM that overlaps PCI
space in
> >> pci_setup(), and
> >>> GPU pass-through is no problem. So it seems this issue is
related to
> >>> "relocate RAM" in pci_setup().
> >> So one issue XenServer found with passing through GPUs is that
there
> >> are bugs in some PCI bridges that completely break VT-d.  The
issue
> >> was that if the *guest* physical address space overlapped the
*host*
> >> physical address of a different device, that the PCI bridges would
> >> send traffic from the passed-through card intended for the guest
to
> >> another card instead.  The work-around was to make the hole in the
> >> guest MMIO space the same size as the host MMIO hole. 
I''m not sure if
> >> that made it upstream or not -- let me check...
> >>
> > Hi George,
> >
> > Could you post your patch and let us have a try with it? Thanks!
> 
> So the patch got checked in, but there still may be some more work if
> you want to use it. :-)
> 
> The patch modifies xc_hvm_build_args structure to include a field called
> "mmio_size". If this is set to zero, it will default to
> HVM_BELOW_4G_MMIO_LENGTH; otherwise, it will be the size of the default
> MMIO hole set during the build process. The guest BIOS may modify this
> at boot time to make it bigger, but it doesn''t make it smaller.
> 
> Since this was designed for xapi, however, which calls libxc directly,
> we didn''t add any options to xend / xl / libxl to set this option.
> 
> The easiest way to test it probably is just to hard-code
> HVM_BELOW_4G_MMIO_LENGTH to a new value (from the description, setting
> it to 1GiB should be sufficient).
> 
> Then if you want to use it in production, you probably want to either:
> 1. Try it with the latest version of XCP (which I think has an option
> you can set)
> 2. Implement a config option for xl that allows you to set the MMIO hole
> size.
> 
> #2 should be a relatively straightforward matter of "plumbing",
and
> would be a welcome contribution. :-)
> 
> If you do implement #2, it might be nice to have an option of
> "mmio_hole_size=host", which will set the guest mmio hole to the
same
> size as the host. That''s what we implemented for XenServer, to
make sure
> there would never be any collisions.
> 
does the e820_host= option affect this? 

-- Pasi

On Wed, Mar 06, 2013 at 04:04:39PM +0200, Pasi Kärkkäinen
wrote:
> On Wed, Mar 06, 2013 at 12:43:09PM +0000, George Dunlap wrote:
> > On 06/03/13 11:38, Hanweidong wrote:
> > >> -----Original Message-----
> > >> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf
Of George
> > >> Dunlap
> > >> Sent: 2013??3??5?? 20:59
> > >> To: Gonglei (Arei)
> > >> Cc: xen-devel@lists.xen.org; Yangxiaowei; Yanqiangjun;
Luonengjun;
> > >> Wangzhenguo; Hanweidong
> > >> Subject: Re: [Xen-devel] GPU passthrough issue when VM is
configured
> > >> with 4G memory
> > >>
> > >> On Mon, Mar 4, 2013 at 8:10 AM, Gonglei (Arei)
<arei.gonglei@huawei.com>
> > >> wrote:
> > >>> Hi,all
> > >>>
> > >>> I have tried to passthrough GPU card(Nvidia quadro 4000)
on the
> > >> latest Xen
> > >>> unstable version (QEMU is using Qemu-upsteam-unstable,
not
> > >> traditional
> > >>> Qemu). This issue as below:
> > >>>
> > >>>        Windows7 64-bit guest will blue screen when GPU
passthrough
> > >> configure
> > >>> 4g memory,blue screen code is 50, and SUSE 11 64-bit
guest will
> > >> always stay
> > >>> at the grub screen.  I noticed that it will relocate RAM
that
> > >> overlaps PCI
> > >>> space in pci_setup()(tools/hvmloader/pci.c). If VM memory
is
> > >> configured with
> > >>> 3G, it won''t cause relocate RAM that overlaps
PCI space in
> > >> pci_setup(), and
> > >>> GPU pass-through is no problem. So it seems this issue is
related to
> > >>> "relocate RAM" in pci_setup().
> > >> So one issue XenServer found with passing through GPUs is
that there
> > >> are bugs in some PCI bridges that completely break VT-d.  The
issue
> > >> was that if the *guest* physical address space overlapped the
*host*
> > >> physical address of a different device, that the PCI bridges
would
> > >> send traffic from the passed-through card intended for the
guest to
> > >> another card instead.  The work-around was to make the hole
in the
> > >> guest MMIO space the same size as the host MMIO hole. 
I''m not sure if
> > >> that made it upstream or not -- let me check...
> > >>
> > > Hi George,
> > >
> > > Could you post your patch and let us have a try with it? Thanks!
> > 
> > So the patch got checked in, but there still may be some more work if
> > you want to use it. :-)
> > 
> > The patch modifies xc_hvm_build_args structure to include a field
called
> > "mmio_size". If this is set to zero, it will default to
> > HVM_BELOW_4G_MMIO_LENGTH; otherwise, it will be the size of the
default
> > MMIO hole set during the build process. The guest BIOS may modify this
> > at boot time to make it bigger, but it doesn''t make it
smaller.
> > 
> > Since this was designed for xapi, however, which calls libxc directly,
> > we didn''t add any options to xend / xl / libxl to set this
option.
> > 
> > The easiest way to test it probably is just to hard-code
> > HVM_BELOW_4G_MMIO_LENGTH to a new value (from the description, setting
> > it to 1GiB should be sufficient).
> > 
> > Then if you want to use it in production, you probably want to either:
> > 1. Try it with the latest version of XCP (which I think has an option
> > you can set)
> > 2. Implement a config option for xl that allows you to set the MMIO
hole
> > size.
> > 
> > #2 should be a relatively straightforward matter of
"plumbing", and
> > would be a welcome contribution. :-)
> > 
> > If you do implement #2, it might be nice to have an option of
> > "mmio_hole_size=host", which will set the guest mmio hole to
the same
> > size as the host. That''s what we implemented for XenServer,
to make sure
> > there would never be any collisions.
> > 
> 
> does the e820_host= option affect this? 
No. That is for PV guests only.

> -----Original Message-----
> From: George Dunlap [mailto:george.dunlap@eu.citrix.com]
> Sent: 2013年3月6日 20:43
> To: Hanweidong
> Cc: Gonglei (Arei); xen-devel@lists.xen.org; Yangxiaowei; Yanqiangjun;
> Luonengjun; Wangzhenguo
> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> On 06/03/13 11:38, Hanweidong wrote:
> >> -----Original Message-----
> >> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of
> George
> >> Dunlap
> >> Sent: 2013年3月5日 20:59
> >> To: Gonglei (Arei)
> >> Cc: xen-devel@lists.xen.org; Yangxiaowei; Yanqiangjun; Luonengjun;
> >> Wangzhenguo; Hanweidong
> >> Subject: Re: [Xen-devel] GPU passthrough issue when VM is
configured
> >> with 4G memory
> >>
> >> On Mon, Mar 4, 2013 at 8:10 AM, Gonglei (Arei)
> <arei.gonglei@huawei.com>
> >> wrote:
> >>> Hi,all
> >>>
> >>> I have tried to passthrough GPU card(Nvidia quadro 4000) on
the
> >> latest Xen
> >>> unstable version (QEMU is using Qemu-upsteam-unstable, not
> >> traditional
> >>> Qemu). This issue as below:
> >>>
> >>>        Windows7 64-bit guest will blue screen when GPU
passthrough
> >> configure
> >>> 4g memory,blue screen code is 50, and SUSE 11 64-bit guest
will
> >> always stay
> >>> at the grub screen.  I noticed that it will relocate RAM that
> >> overlaps PCI
> >>> space in pci_setup()(tools/hvmloader/pci.c). If VM memory is
> >> configured with
> >>> 3G, it won't cause relocate RAM that overlaps PCI space in
> >> pci_setup(), and
> >>> GPU pass-through is no problem. So it seems this issue is
related
> to
> >>> "relocate RAM" in pci_setup().
> >> So one issue XenServer found with passing through GPUs is that
there
> >> are bugs in some PCI bridges that completely break VT-d.  The
issue
> >> was that if the *guest* physical address space overlapped the
*host*
> >> physical address of a different device, that the PCI bridges would
> >> send traffic from the passed-through card intended for the guest
to
> >> another card instead.  The work-around was to make the hole in the
> >> guest MMIO space the same size as the host MMIO hole.  I'm not
sure
> if
> >> that made it upstream or not -- let me check...
> >>
> > Hi George,
> >
> > Could you post your patch and let us have a try with it? Thanks!
> 
> So the patch got checked in, but there still may be some more work if
> you want to use it. :-)
> 
> The patch modifies xc_hvm_build_args structure to include a field
> called
> "mmio_size". If this is set to zero, it will default to
> HVM_BELOW_4G_MMIO_LENGTH; otherwise, it will be the size of the default
> MMIO hole set during the build process. The guest BIOS may modify this
> at boot time to make it bigger, but it doesn't make it smaller.
> 
> Since this was designed for xapi, however, which calls libxc directly,
> we didn't add any options to xend / xl / libxl to set this option.
> 
> The easiest way to test it probably is just to hard-code
> HVM_BELOW_4G_MMIO_LENGTH to a new value (from the description, setting
> it to 1GiB should be sufficient).
> 
> Then if you want to use it in production, you probably want to either:
> 1. Try it with the latest version of XCP (which I think has an option
> you can set)
> 2. Implement a config option for xl that allows you to set the MMIO
> hole
> size.
> 
> #2 should be a relatively straightforward matter of "plumbing",
and
> would be a welcome contribution. :-)
> 
> If you do implement #2, it might be nice to have an option of
> "mmio_hole_size=host", which will set the guest mmio hole to the
same
> size as the host. That's what we implemented for XenServer, to make
> sure
> there would never be any collisions.
> 
We rooted caused this issue: in pci_setup, it relocates pci_mem_start from 
0xF0000000 to 0XE0000000 because GPU has big more MMIO, but in QEMU, 
xen_ram_init directly uses the macro HVM_BELOW_4G_RAM_END and 
HVM_BELOW_4G_MMIO_LENGTH, doesn't do corresponding relocation like pci_setup
does.

We tried to hardcod HVM_BELOW_4G_RAM_END to 0XE0000000, GPU pass-through
Worked.

--Weidong


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 07/03/13 07:51, Hanweidong wrote:
>> -----Original Message-----
>> From: George Dunlap [mailto:george.dunlap@eu.citrix.com]
>> Sent: 2013年3月6日 20:43
>> To: Hanweidong
>> Cc: Gonglei (Arei); xen-devel@lists.xen.org; Yangxiaowei; Yanqiangjun;
>> Luonengjun; Wangzhenguo
>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
>> with 4G memory
>>
>> On 06/03/13 11:38, Hanweidong wrote:
>>>> -----Original Message-----
>>>> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of
>> George
>>>> Dunlap
>>>> Sent: 2013年3月5日 20:59
>>>> To: Gonglei (Arei)
>>>> Cc: xen-devel@lists.xen.org; Yangxiaowei; Yanqiangjun;
Luonengjun;
>>>> Wangzhenguo; Hanweidong
>>>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is
configured
>>>> with 4G memory
>>>>
>>>> On Mon, Mar 4, 2013 at 8:10 AM, Gonglei (Arei)
>> <arei.gonglei@huawei.com>
>>>> wrote:
>>>>> Hi,all
>>>>>
>>>>> I have tried to passthrough GPU card(Nvidia quadro 4000) on
the
>>>> latest Xen
>>>>> unstable version (QEMU is using Qemu-upsteam-unstable, not
>>>> traditional
>>>>> Qemu). This issue as below:
>>>>>
>>>>>        Windows7 64-bit guest will blue screen when GPU
passthrough
>>>> configure
>>>>> 4g memory,blue screen code is 50, and SUSE 11 64-bit guest
will
>>>> always stay
>>>>> at the grub screen.  I noticed that it will relocate RAM
that
>>>> overlaps PCI
>>>>> space in pci_setup()(tools/hvmloader/pci.c). If VM memory
is
>>>> configured with
>>>>> 3G, it won't cause relocate RAM that overlaps PCI space
in
>>>> pci_setup(), and
>>>>> GPU pass-through is no problem. So it seems this issue is
related
>> to
>>>>> "relocate RAM" in pci_setup().
>>>> So one issue XenServer found with passing through GPUs is that
there
>>>> are bugs in some PCI bridges that completely break VT-d.  The
issue
>>>> was that if the *guest* physical address space overlapped the
*host*
>>>> physical address of a different device, that the PCI bridges
would
>>>> send traffic from the passed-through card intended for the
guest to
>>>> another card instead.  The work-around was to make the hole in
the
>>>> guest MMIO space the same size as the host MMIO hole.  I'm
not sure
>> if
>>>> that made it upstream or not -- let me check...
>>>>
>>> Hi George,
>>>
>>> Could you post your patch and let us have a try with it? Thanks!
>> So the patch got checked in, but there still may be some more work if
>> you want to use it. :-)
>>
>> The patch modifies xc_hvm_build_args structure to include a field
>> called
>> "mmio_size". If this is set to zero, it will default to
>> HVM_BELOW_4G_MMIO_LENGTH; otherwise, it will be the size of the default
>> MMIO hole set during the build process. The guest BIOS may modify this
>> at boot time to make it bigger, but it doesn't make it smaller.
>>
>> Since this was designed for xapi, however, which calls libxc directly,
>> we didn't add any options to xend / xl / libxl to set this option.
>>
>> The easiest way to test it probably is just to hard-code
>> HVM_BELOW_4G_MMIO_LENGTH to a new value (from the description, setting
>> it to 1GiB should be sufficient).
>>
>> Then if you want to use it in production, you probably want to either:
>> 1. Try it with the latest version of XCP (which I think has an option
>> you can set)
>> 2. Implement a config option for xl that allows you to set the MMIO
>> hole
>> size.
>>
>> #2 should be a relatively straightforward matter of
"plumbing", and
>> would be a welcome contribution. :-)
>>
>> If you do implement #2, it might be nice to have an option of
>> "mmio_hole_size=host", which will set the guest mmio hole to
the same
>> size as the host. That's what we implemented for XenServer, to make
>> sure
>> there would never be any collisions.
>>
> We rooted caused this issue: in pci_setup, it relocates pci_mem_start from 
> 0xF0000000 to 0XE0000000 because GPU has big more MMIO, but in QEMU, 
> xen_ram_init directly uses the macro HVM_BELOW_4G_RAM_END and 
> HVM_BELOW_4G_MMIO_LENGTH, doesn't do corresponding relocation like
pci_setup
> does.
>
> We tried to hardcod HVM_BELOW_4G_RAM_END to 0XE0000000, GPU pass-through
> Worked.
I'm not super familiar with the HVM qemu / guest BIOS stuff, but it
sounds like that's a bug in pass-through that needs to be sorted out.
Anthony, can you comment?

-George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

> -----Original Message-----
> From: George Dunlap [mailto:george.dunlap@eu.citrix.com]
> Sent: 2013年3月7日 18:16
> To: Hanweidong
> Cc: Gonglei (Arei); xen-devel@lists.xen.org; Yangxiaowei; Yanqiangjun;
> Luonengjun; Wangzhenguo; Anthony Perard; Stefano Stabellini
> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> On 07/03/13 07:51, Hanweidong wrote:
> >> -----Original Message-----
> >> From: George Dunlap [mailto:george.dunlap@eu.citrix.com]
> >> Sent: 2013年3月6日 20:43
> >> To: Hanweidong
> >> Cc: Gonglei (Arei); xen-devel@lists.xen.org; Yangxiaowei;
> Yanqiangjun;
> >> Luonengjun; Wangzhenguo
> >> Subject: Re: [Xen-devel] GPU passthrough issue when VM is
configured
> >> with 4G memory
> >>
> >> On 06/03/13 11:38, Hanweidong wrote:
> >>>> -----Original Message-----
> >>>> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On
Behalf Of
> >> George
> >>>> Dunlap
> >>>> Sent: 2013年3月5日 20:59
> >>>> To: Gonglei (Arei)
> >>>> Cc: xen-devel@lists.xen.org; Yangxiaowei; Yanqiangjun;
Luonengjun;
> >>>> Wangzhenguo; Hanweidong
> >>>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is
> configured
> >>>> with 4G memory
> >>>>
> >>>> On Mon, Mar 4, 2013 at 8:10 AM, Gonglei (Arei)
> >> <arei.gonglei@huawei.com>
> >>>> wrote:
> >>>>> Hi,all
> >>>>>
> >>>>> I have tried to passthrough GPU card(Nvidia quadro
4000) on the
> >>>> latest Xen
> >>>>> unstable version (QEMU is using Qemu-upsteam-unstable,
not
> >>>> traditional
> >>>>> Qemu). This issue as below:
> >>>>>
> >>>>>        Windows7 64-bit guest will blue screen when GPU
> passthrough
> >>>> configure
> >>>>> 4g memory,blue screen code is 50, and SUSE 11 64-bit
guest will
> >>>> always stay
> >>>>> at the grub screen.  I noticed that it will relocate
RAM that
> >>>> overlaps PCI
> >>>>> space in pci_setup()(tools/hvmloader/pci.c). If VM
memory is
> >>>> configured with
> >>>>> 3G, it won't cause relocate RAM that overlaps PCI
space in
> >>>> pci_setup(), and
> >>>>> GPU pass-through is no problem. So it seems this issue
is related
> >> to
> >>>>> "relocate RAM" in pci_setup().
> >>>> So one issue XenServer found with passing through GPUs is
that
> there
> >>>> are bugs in some PCI bridges that completely break VT-d. 
The
> issue
> >>>> was that if the *guest* physical address space overlapped
the
> *host*
> >>>> physical address of a different device, that the PCI
bridges would
> >>>> send traffic from the passed-through card intended for the
guest
> to
> >>>> another card instead.  The work-around was to make the
hole in the
> >>>> guest MMIO space the same size as the host MMIO hole. 
I'm not
> sure
> >> if
> >>>> that made it upstream or not -- let me check...
> >>>>
> >>> Hi George,
> >>>
> >>> Could you post your patch and let us have a try with it?
Thanks!
> >> So the patch got checked in, but there still may be some more work
> if
> >> you want to use it. :-)
> >>
> >> The patch modifies xc_hvm_build_args structure to include a field
> >> called
> >> "mmio_size". If this is set to zero, it will default to
> >> HVM_BELOW_4G_MMIO_LENGTH; otherwise, it will be the size of the
> default
> >> MMIO hole set during the build process. The guest BIOS may modify
> this
> >> at boot time to make it bigger, but it doesn't make it
smaller.
> >>
> >> Since this was designed for xapi, however, which calls libxc
> directly,
> >> we didn't add any options to xend / xl / libxl to set this
option.
> >>
> >> The easiest way to test it probably is just to hard-code
> >> HVM_BELOW_4G_MMIO_LENGTH to a new value (from the description,
> setting
> >> it to 1GiB should be sufficient).
> >>
> >> Then if you want to use it in production, you probably want to
> either:
> >> 1. Try it with the latest version of XCP (which I think has an
> option
> >> you can set)
> >> 2. Implement a config option for xl that allows you to set the
MMIO
> >> hole
> >> size.
> >>
> >> #2 should be a relatively straightforward matter of
"plumbing", and
> >> would be a welcome contribution. :-)
> >>
> >> If you do implement #2, it might be nice to have an option of
> >> "mmio_hole_size=host", which will set the guest mmio
hole to the
> same
> >> size as the host. That's what we implemented for XenServer, to
make
> >> sure
> >> there would never be any collisions.
> >>
> > We rooted caused this issue: in pci_setup, it relocates pci_mem_start
> from
> > 0xF0000000 to 0XE0000000 because GPU has big more MMIO, but in QEMU,
> > xen_ram_init directly uses the macro HVM_BELOW_4G_RAM_END and
> > HVM_BELOW_4G_MMIO_LENGTH, doesn't do corresponding relocation like
> pci_setup
> > does.
> >
> > We tried to hardcod HVM_BELOW_4G_RAM_END to 0XE0000000, GPU pass-
> through
> > Worked.
> 
> I'm not super familiar with the HVM qemu / guest BIOS stuff, but it
> sounds like that's a bug in pass-through that needs to be sorted out.
> Anthony, can you comment?
> 
I don't think it's a pass-through issue. We also encountered similar 
issue when using ivshmem device with 512M mmio. The problem is hvmloader
adjusted pci_mem_start according to actual mmio size of VM, but QEMU does 
not know the actual mmio size of VM, it just uses the HVM_BELOW_4G_RAM_END 
and HVM_BELOW_4G_MMIO_LENGTH in xen_ram_init. This results in memory layout 
mismatch between hvmloader and QEMU. I think it needs to make QEMU know 
actual mmio size, and then it can make the same memory layout as hvmloader. 

--Weidong

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Tue, Mar 12, 2013 at 5:45 AM, Hanweidong <hanweidong@huawei.com>
wrote:
> I don''t think it''s a pass-through issue. We also
encountered similar
> issue when using ivshmem device with 512M mmio. The problem is hvmloader
> adjusted pci_mem_start according to actual mmio size of VM, but QEMU does
> not know the actual mmio size of VM, it just uses the HVM_BELOW_4G_RAM_END
> and HVM_BELOW_4G_MMIO_LENGTH in xen_ram_init. This results in memory layout
> mismatch between hvmloader and QEMU. I think it needs to make QEMU know
> actual mmio size, and then it can make the same memory layout as hvmloader.
So I''m not an expert in this codepath, but it was my understanding
that resizing the MMIO hole is actually the job of the guest BIOS: it
is supposed to query the PCI devices to find out how much memory they
need, and then relocate memory as appropriate.  XenServer never had
any problem with the MMIO hole not being big enough when passing cards
to the guest; the only problem we''ve ever had (as far as I know) is
the issue I mentioned before, where there was a bug in some pci bridge
cards that breaks VT-d if the guest physical address overlaps the host
physical address range of another device.

Can you maybe boot Linux in an HVM domain and do an lspci or look at
Linux''s e820 map, and check to see whether the BIOS has in fact
relocated the guest memory, or whether the MMIO hole does indeed start
only at HVM_BELOW_4G_RAM_END?

 -George

> -----Original Message-----
> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of George
> Dunlap
> Sent: 2013年3月12日 18:40
> To: Hanweidong
> Cc: Stefano Stabellini; Yanqiangjun; Luonengjun; Wangzhenguo;
> Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> On Tue, Mar 12, 2013 at 5:45 AM, Hanweidong <hanweidong@huawei.com>
> wrote:
> > I don't think it's a pass-through issue. We also encountered
similar
> > issue when using ivshmem device with 512M mmio. The problem is
> hvmloader
> > adjusted pci_mem_start according to actual mmio size of VM, but QEMU
> does
> > not know the actual mmio size of VM, it just uses the
> HVM_BELOW_4G_RAM_END
> > and HVM_BELOW_4G_MMIO_LENGTH in xen_ram_init. This results in memory
> layout
> > mismatch between hvmloader and QEMU. I think it needs to make QEMU
> know
> > actual mmio size, and then it can make the same memory layout as
> hvmloader.
> 
> So I'm not an expert in this codepath, but it was my understanding
> that resizing the MMIO hole is actually the job of the guest BIOS: it
> is supposed to query the PCI devices to find out how much memory they
> need, and then relocate memory as appropriate.  XenServer never had
> any problem with the MMIO hole not being big enough when passing cards
> to the guest; the only problem we've ever had (as far as I know) is
> the issue I mentioned before, where there was a bug in some pci bridge
> cards that breaks VT-d if the guest physical address overlaps the host
> physical address range of another device.
> 
> Can you maybe boot Linux in an HVM domain and do an lspci or look at
> Linux's e820 map, and check to see whether the BIOS has in fact
> relocated the guest memory, or whether the MMIO hole does indeed start
> only at HVM_BELOW_4G_RAM_END?
> 
We tried Linux guest, but it stopped at grub screen, cannot boot up. Posted 
"xl dmesg" output as below:

(XEN) HVM13: HVM Loader
(XEN) HVM13: Detected Xen v4.3-unstable
(XEN) HVM13: Xenbus rings @0xfeffc000, event channel 3
(XEN) HVM13: System requested SeaBIOS
(XEN) HVM13: CPU speed is 2500 MHz
(XEN) irq.c:270: Dom13 PCI link 0 changed 0 -> 5
(XEN) HVM13: PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:270: Dom13 PCI link 1 changed 0 -> 10
(XEN) HVM13: PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:270: Dom13 PCI link 2 changed 0 -> 11
(XEN) HVM13: PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:270: Dom13 PCI link 3 changed 0 -> 5
(XEN) HVM13: PCI-ISA link 3 routed to IRQ5
(XEN) HVM13: pci dev 01:2 INTD->IRQ5
(XEN) HVM13: pci dev 01:3 INTA->IRQ10
(XEN) HVM13: pci dev 03:0 INTA->IRQ5
(XEN) HVM13: pci dev 04:0 INTA->IRQ5
(XEN) HVM13: pci dev 05:0 INTA->IRQ10
(XEN) HVM13: pci dev 05:0 bar 14 size lx: 08000000
(XEN) memory_map:add: dom13 gfn=e0000 mfn=e8000 nr=8000
(XEN) memory_map:add: dom13 gfn=e8000 mfn=f0000 nr=4000
(XEN) HVM13: pci dev 05:0 bar 1c size lx: 04000000
(XEN) HVM13: pci dev 02:0 bar 10 size lx: 02000000
(XEN) memory_map:add: dom13 gfn=ee000 mfn=bc000 nr=2000
(XEN) HVM13: pci dev 05:0 bar 10 size lx: 02000000
(XEN) HVM13: pci dev 03:0 bar 14 size lx: 01000000
(XEN) HVM13: pci dev 05:0 bar 30 size lx: 00080000
(XEN) HVM13: pci dev 02:0 bar 30 size lx: 00010000
(XEN) HVM13: pci dev 04:0 bar 30 size lx: 00010000
(XEN) HVM13: pci dev 02:0 bar 14 size lx: 00001000
(XEN) HVM13: pci dev 03:0 bar 10 size lx: 00000100
(XEN) HVM13: pci dev 04:0 bar 10 size lx: 00000100
(XEN) HVM13: pci dev 04:0 bar 14 size lx: 00000100
(XEN) HVM13: pci dev 05:0 bar 24 size lx: 00000080
(XEN) ioport_map:add: dom13 gport=c200 mport=2000 nr=80
(XEN) HVM13: pci dev 01:2 bar 20 size lx: 00000020
(XEN) HVM13: pci dev 01:1 bar 20 size lx: 00000010
(XEN) HVM13: Multiprocessor initialisation:
(XEN) HVM13:  - CPU0 ... 46-bit phys ... fixed MTRRs ... var MTRRs [3/8] ...
done.
(XEN) HVM13: Testing HVM environment:
(XEN) HVM13:  - REP INSB across page boundaries ... passed
(XEN) HVM13:  - GS base MSRs and SWAPGS ... passed
(XEN) HVM13: Passed 2 of 2 tests
(XEN) HVM13: Writing SMBIOS tables ...
(XEN) HVM13: Loading SeaBIOS ...
(XEN) HVM13: Creating MP tables ...
(XEN) HVM13: Loading ACPI ...
(XEN) HVM13: vm86 TSS at fc00a000
(XEN) HVM13: BIOS map:
(XEN) HVM13:  10000-100d3: Scratch space
(XEN) HVM13:  e0000-fffff: Main BIOS
(XEN) HVM13: E820 table:
(XEN) HVM13:  [00]: 00000000:00000000 - 00000000:000a0000: RAM
(XEN) HVM13:  HOLE: 00000000:000a0000 - 00000000:000e0000
(XEN) HVM13:  [01]: 00000000:000e0000 - 00000000:00100000: RESERVED
(XEN) HVM13:  [02]: 00000000:00100000 - 00000000:e0000000: RAM
(XEN) HVM13:  HOLE: 00000000:e0000000 - 00000000:fc000000
(XEN) HVM13:  [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
(XEN) HVM13:  [04]: 00000001:00000000 - 00000001:1f800000: RAM
(XEN) HVM13: Invoking SeaBIOS ...
(XEN) HVM13: SeaBIOS (version rel-1.7.1-0-g51755c3-20130227_164001-linux-DPMZPO)
(XEN) HVM13: 
(XEN) HVM13: Found Xen hypervisor signature at 40000000
(XEN) HVM13: xen: copy e820...
(XEN) HVM13: Ram Size=0xe0000000 (0x000000001f800000 high)
(XEN) HVM13: Relocating low data from 0x000e2490 to 0x000ef790 (size 2156)
(XEN) HVM13: Relocating init from 0x000e2cfc to 0xdffe20f0 (size 56804)
(XEN) HVM13: CPU Mhz=2501
(XEN) HVM13: Found 9 PCI devices (max PCI bus is 00)
(XEN) HVM13: Allocated Xen hypercall page at dffff000
(XEN) HVM13: Detected Xen v4.3-unstable
(XEN) HVM13: Found 1 cpu(s) max supported 1 cpu(s)
(XEN) HVM13: xen: copy BIOS tables...
(XEN) HVM13: Copying SMBIOS entry point from 0x00010010 to 0x000fdb10
(XEN) HVM13: Copying MPTABLE from 0xfc001140/fc001150 to 0x000fda30
(XEN) HVM13: Copying PIR from 0x00010030 to 0x000fd9b0
(XEN) HVM13: Copying ACPI RSDP from 0x000100b0 to 0x000fd980
(XEN) HVM13: Scan for VGA option rom
(XEN) HVM13: Running option rom at c000:0003
(XEN) stdvga.c:147:d13 entering stdvga and caching modes
(XEN) HVM13: Turning on vga text mode console
(XEN) HVM13: SeaBIOS (version rel-1.7.1-0-g51755c3-20130227_164001-linux-DPMZPO)
(XEN) HVM13: 
(XEN) HVM13: UHCI init on dev 00:01.2 (io=c280)
(XEN) HVM13: Found 1 lpt ports
(XEN) HVM13: Found 1 serial ports
(XEN) HVM13: ATA controller 1 at 1f0/3f4/c2a0 (irq 14 dev 9)
(XEN) HVM13: ATA controller 2 at 170/374/c2a8 (irq 15 dev 9)
(XEN) HVM13: ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (20480 MiBytes)
(XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0
(XEN) HVM13: DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
(XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
(XEN) HVM13: PS2 keyboard initialized
(XEN) HVM13: All threads complete.
(XEN) HVM13: Scan for option roms
(XEN) HVM13: Running option rom at c900:0003
(XEN) HVM13: pmm call arg1=1
(XEN) HVM13: pmm call arg1=0
(XEN) HVM13: pmm call arg1=1
(XEN) HVM13: pmm call arg1=0
(XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@4
(XEN) HVM13: Press F12 for boot menu.
(XEN) HVM13: 
(XEN) HVM13: drive 0x000fd930: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
s=41943040
(XEN) HVM13: 
(XEN) HVM13: Space available for UMB: 000ca000-000ee800
(XEN) HVM13: Returned 61440 bytes of ZoneHigh
(XEN) HVM13: e820 map has 7 items:
(XEN) HVM13:   0: 0000000000000000 - 000000000009fc00 = 1 RAM
(XEN) HVM13:   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
(XEN) HVM13:   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
(XEN) HVM13:   3: 0000000000100000 - 00000000dffff000 = 1 RAM
(XEN) HVM13:   4: 00000000dffff000 - 00000000e0000000 = 2 RESERVED
(XEN) HVM13:   5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
(XEN) HVM13:   6: 0000000100000000 - 000000011f800000 = 1 RAM
(XEN) HVM13: enter handle_19:
(XEN) HVM13:   NULL
(XEN) HVM13: Booting from Hard Disk...
(XEN) HVM13: Booting from 0000:7c00
(XEN) stdvga.c:151:d13 leaving stdvga
(XEN) stdvga.c:147:d13 entering stdvga and caching modes

MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses below code to init 
RAM in xen_ram_init: 

    ...
    block_len = ram_size;
    if (ram_size >= HVM_BELOW_4G_RAM_END) {
        /* Xen does not allocate the memory continuously, and keep a hole at
         * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
         */
        block_len += HVM_BELOW_4G_MMIO_LENGTH;
    }
    memory_region_init_ram(&ram_memory, "xen.ram", block_len);
    vmstate_register_ram_global(&ram_memory);

    if (ram_size >= HVM_BELOW_4G_RAM_END) {
        above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
        below_4g_mem_size = HVM_BELOW_4G_RAM_END;
    } else {
        below_4g_mem_size = ram_size;
    }
    ...

HVM_BELOW_4G_RAM_END is f0000000. If we change HVM_BELOW_4G_RAM_END to e0000000,
Which it's consistent with hvmloader when assigning a GPU, and then guest
worked
for us. So we wondering that xen_ram_init in QEMU should be consistent with 
hvmloader. 

In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1() as below. 
Should keep these places handle the consistent mmio hole or not?

    if (ram_size >= 0xe0000000 ) {
        above_4g_mem_size = ram_size - 0xe0000000;
        below_4g_mem_size = 0xe0000000;
    } else {
        above_4g_mem_size = 0;
        below_4g_mem_size = ram_size;
    }

--weidong
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Le 13.03.2013 14:23, Hanweidong a écrit :
>> -----Original Message-----
>> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of 
>> George
>> Dunlap
>> Sent: 2013年3月12日 18:40
>> To: Hanweidong
>> Cc: Stefano Stabellini; Yanqiangjun; Luonengjun; Wangzhenguo;
>> Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-devel@lists.xen.org
>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
>> with 4G memory
>>
>> On Tue, Mar 12, 2013 at 5:45 AM, Hanweidong
<hanweidong@huawei.com>
>> wrote:
>> > I don't think it's a pass-through issue. We also
encountered
>> similar
>> > issue when using ivshmem device with 512M mmio. The problem is
>> hvmloader
>> > adjusted pci_mem_start according to actual mmio size of VM, but 
>> QEMU
>> does
>> > not know the actual mmio size of VM, it just uses the
>> HVM_BELOW_4G_RAM_END
>> > and HVM_BELOW_4G_MMIO_LENGTH in xen_ram_init. This results in 
>> memory
>> layout
>> > mismatch between hvmloader and QEMU. I think it needs to make QEMU
>> know
>> > actual mmio size, and then it can make the same memory layout as
>> hvmloader.
>>
>> So I'm not an expert in this codepath, but it was my understanding
>> that resizing the MMIO hole is actually the job of the guest BIOS: 
>> it
>> is supposed to query the PCI devices to find out how much memory 
>> they
>> need, and then relocate memory as appropriate.  XenServer never had
>> any problem with the MMIO hole not being big enough when passing 
>> cards
>> to the guest; the only problem we've ever had (as far as I know) is
>> the issue I mentioned before, where there was a bug in some pci 
>> bridge
>> cards that breaks VT-d if the guest physical address overlaps the 
>> host
>> physical address range of another device.
>>
>> Can you maybe boot Linux in an HVM domain and do an lspci or look at
>> Linux's e820 map, and check to see whether the BIOS has in fact
>> relocated the guest memory, or whether the MMIO hole does indeed 
>> start
>> only at HVM_BELOW_4G_RAM_END?
>>
>
> We tried Linux guest, but it stopped at grub screen, cannot boot up. 
> Posted
> "xl dmesg" output as below:
>
> (XEN) HVM13: HVM Loader
> (XEN) HVM13: Detected Xen v4.3-unstable
> (XEN) HVM13: Xenbus rings @0xfeffc000, event channel 3
> (XEN) HVM13: System requested SeaBIOS
> (XEN) HVM13: CPU speed is 2500 MHz
> (XEN) irq.c:270: Dom13 PCI link 0 changed 0 -> 5
> (XEN) HVM13: PCI-ISA link 0 routed to IRQ5
> (XEN) irq.c:270: Dom13 PCI link 1 changed 0 -> 10
> (XEN) HVM13: PCI-ISA link 1 routed to IRQ10
> (XEN) irq.c:270: Dom13 PCI link 2 changed 0 -> 11
> (XEN) HVM13: PCI-ISA link 2 routed to IRQ11
> (XEN) irq.c:270: Dom13 PCI link 3 changed 0 -> 5
> (XEN) HVM13: PCI-ISA link 3 routed to IRQ5
> (XEN) HVM13: pci dev 01:2 INTD->IRQ5
> (XEN) HVM13: pci dev 01:3 INTA->IRQ10
> (XEN) HVM13: pci dev 03:0 INTA->IRQ5
> (XEN) HVM13: pci dev 04:0 INTA->IRQ5
> (XEN) HVM13: pci dev 05:0 INTA->IRQ10
> (XEN) HVM13: pci dev 05:0 bar 14 size lx: 08000000
> (XEN) memory_map:add: dom13 gfn=e0000 mfn=e8000 nr=8000
> (XEN) memory_map:add: dom13 gfn=e8000 mfn=f0000 nr=4000
> (XEN) HVM13: pci dev 05:0 bar 1c size lx: 04000000
> (XEN) HVM13: pci dev 02:0 bar 10 size lx: 02000000
> (XEN) memory_map:add: dom13 gfn=ee000 mfn=bc000 nr=2000
> (XEN) HVM13: pci dev 05:0 bar 10 size lx: 02000000
> (XEN) HVM13: pci dev 03:0 bar 14 size lx: 01000000
> (XEN) HVM13: pci dev 05:0 bar 30 size lx: 00080000
> (XEN) HVM13: pci dev 02:0 bar 30 size lx: 00010000
> (XEN) HVM13: pci dev 04:0 bar 30 size lx: 00010000
> (XEN) HVM13: pci dev 02:0 bar 14 size lx: 00001000
> (XEN) HVM13: pci dev 03:0 bar 10 size lx: 00000100
> (XEN) HVM13: pci dev 04:0 bar 10 size lx: 00000100
> (XEN) HVM13: pci dev 04:0 bar 14 size lx: 00000100
> (XEN) HVM13: pci dev 05:0 bar 24 size lx: 00000080
> (XEN) ioport_map:add: dom13 gport=c200 mport=2000 nr=80
> (XEN) HVM13: pci dev 01:2 bar 20 size lx: 00000020
> (XEN) HVM13: pci dev 01:1 bar 20 size lx: 00000010
> (XEN) HVM13: Multiprocessor initialisation:
> (XEN) HVM13:  - CPU0 ... 46-bit phys ... fixed MTRRs ... var MTRRs
> [3/8] ... done.
> (XEN) HVM13: Testing HVM environment:
> (XEN) HVM13:  - REP INSB across page boundaries ... passed
> (XEN) HVM13:  - GS base MSRs and SWAPGS ... passed
> (XEN) HVM13: Passed 2 of 2 tests
> (XEN) HVM13: Writing SMBIOS tables ...
> (XEN) HVM13: Loading SeaBIOS ...
> (XEN) HVM13: Creating MP tables ...
> (XEN) HVM13: Loading ACPI ...
> (XEN) HVM13: vm86 TSS at fc00a000
> (XEN) HVM13: BIOS map:
> (XEN) HVM13:  10000-100d3: Scratch space
> (XEN) HVM13:  e0000-fffff: Main BIOS
> (XEN) HVM13: E820 table:
> (XEN) HVM13:  [00]: 00000000:00000000 - 00000000:000a0000: RAM
> (XEN) HVM13:  HOLE: 00000000:000a0000 - 00000000:000e0000
> (XEN) HVM13:  [01]: 00000000:000e0000 - 00000000:00100000: RESERVED
> (XEN) HVM13:  [02]: 00000000:00100000 - 00000000:e0000000: RAM
> (XEN) HVM13:  HOLE: 00000000:e0000000 - 00000000:fc000000
> (XEN) HVM13:  [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
> (XEN) HVM13:  [04]: 00000001:00000000 - 00000001:1f800000: RAM
> (XEN) HVM13: Invoking SeaBIOS ...
> (XEN) HVM13: SeaBIOS (version 
> rel-1.7.1-0-g51755c3-20130227_164001-linux-DPMZPO)
> (XEN) HVM13:
> (XEN) HVM13: Found Xen hypervisor signature at 40000000
> (XEN) HVM13: xen: copy e820...
> (XEN) HVM13: Ram Size=0xe0000000 (0x000000001f800000 high)
> (XEN) HVM13: Relocating low data from 0x000e2490 to 0x000ef790 (size 
> 2156)
> (XEN) HVM13: Relocating init from 0x000e2cfc to 0xdffe20f0 (size 
> 56804)
> (XEN) HVM13: CPU Mhz=2501
> (XEN) HVM13: Found 9 PCI devices (max PCI bus is 00)
> (XEN) HVM13: Allocated Xen hypercall page at dffff000
> (XEN) HVM13: Detected Xen v4.3-unstable
> (XEN) HVM13: Found 1 cpu(s) max supported 1 cpu(s)
> (XEN) HVM13: xen: copy BIOS tables...
> (XEN) HVM13: Copying SMBIOS entry point from 0x00010010 to 0x000fdb10
> (XEN) HVM13: Copying MPTABLE from 0xfc001140/fc001150 to 0x000fda30
> (XEN) HVM13: Copying PIR from 0x00010030 to 0x000fd9b0
> (XEN) HVM13: Copying ACPI RSDP from 0x000100b0 to 0x000fd980
> (XEN) HVM13: Scan for VGA option rom
> (XEN) HVM13: Running option rom at c000:0003
> (XEN) stdvga.c:147:d13 entering stdvga and caching modes
> (XEN) HVM13: Turning on vga text mode console
> (XEN) HVM13: SeaBIOS (version 
> rel-1.7.1-0-g51755c3-20130227_164001-linux-DPMZPO)
> (XEN) HVM13:
> (XEN) HVM13: UHCI init on dev 00:01.2 (io=c280)
> (XEN) HVM13: Found 1 lpt ports
> (XEN) HVM13: Found 1 serial ports
> (XEN) HVM13: ATA controller 1 at 1f0/3f4/c2a0 (irq 14 dev 9)
> (XEN) HVM13: ATA controller 2 at 170/374/c2a8 (irq 15 dev 9)
> (XEN) HVM13: ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (20480 MiBytes)
> (XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0
> (XEN) HVM13: DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
> (XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
> (XEN) HVM13: PS2 keyboard initialized
> (XEN) HVM13: All threads complete.
> (XEN) HVM13: Scan for option roms
> (XEN) HVM13: Running option rom at c900:0003
> (XEN) HVM13: pmm call arg1=1
> (XEN) HVM13: pmm call arg1=0
> (XEN) HVM13: pmm call arg1=1
> (XEN) HVM13: pmm call arg1=0
> (XEN) HVM13: Searching bootorder for: /pci@i0cf8/*@4
> (XEN) HVM13: Press F12 for boot menu.
> (XEN) HVM13:
> (XEN) HVM13: drive 0x000fd930: PCHS=16383/16/63 translation=lba
> LCHS=1024/255/63 s=41943040
> (XEN) HVM13:
> (XEN) HVM13: Space available for UMB: 000ca000-000ee800
> (XEN) HVM13: Returned 61440 bytes of ZoneHigh
> (XEN) HVM13: e820 map has 7 items:
> (XEN) HVM13:   0: 0000000000000000 - 000000000009fc00 = 1 RAM
> (XEN) HVM13:   1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
> (XEN) HVM13:   2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
> (XEN) HVM13:   3: 0000000000100000 - 00000000dffff000 = 1 RAM
> (XEN) HVM13:   4: 00000000dffff000 - 00000000e0000000 = 2 RESERVED
> (XEN) HVM13:   5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
> (XEN) HVM13:   6: 0000000100000000 - 000000011f800000 = 1 RAM
> (XEN) HVM13: enter handle_19:
> (XEN) HVM13:   NULL
> (XEN) HVM13: Booting from Hard Disk...
> (XEN) HVM13: Booting from 0000:7c00
> (XEN) stdvga.c:151:d13 leaving stdvga
> (XEN) stdvga.c:147:d13 entering stdvga and caching modes
>
> MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses below 
> code to init
> RAM in xen_ram_init:
>
>     ...
>     block_len = ram_size;
>     if (ram_size >= HVM_BELOW_4G_RAM_END) {
>         /* Xen does not allocate the memory continuously, and keep a 
> hole at
>          * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
>          */
>         block_len += HVM_BELOW_4G_MMIO_LENGTH;
>     }
>     memory_region_init_ram(&ram_memory, "xen.ram",
block_len);
>     vmstate_register_ram_global(&ram_memory);
>
>     if (ram_size >= HVM_BELOW_4G_RAM_END) {
>         above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
>         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
>     } else {
>         below_4g_mem_size = ram_size;
>     }
>     ...
>
> HVM_BELOW_4G_RAM_END is f0000000. If we change HVM_BELOW_4G_RAM_END
> to e0000000,
> Which it's consistent with hvmloader when assigning a GPU, and then
> guest worked
> for us. So we wondering that xen_ram_init in QEMU should be 
> consistent with
> hvmloader.
>
> In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1() as 
> below.
> Should keep these places handle the consistent mmio hole or not?
>
>     if (ram_size >= 0xe0000000 ) {
>         above_4g_mem_size = ram_size - 0xe0000000;
>         below_4g_mem_size = 0xe0000000;
>     } else {
>         above_4g_mem_size = 0;
>         below_4g_mem_size = ram_size;
>     }
>
> --weidong
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
Hello,

What can of bug we can encounter by hardcoding the HVM_BELOW_4G_RAM_END 
to e0000000 ?

By doing this I'm able to boot on Win7 with gpu passthrough of an 
nvidia 560TI an 12Go of ram.

Regards and thanks for all your work !

Youenn.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Sun, Mar 17, 2013 at 12:41:01AM +0100, youenn.gestin@grim-is-a-geek.fr
wrote:
> 
> Hello,
> 
> What can of bug we can encounter by hardcoding the
> HVM_BELOW_4G_RAM_END to e0000000 ?
> 
> By doing this I''m able to boot on Win7 with gpu passthrough of an
> nvidia 560TI an 12Go of ram.
> 
> Regards and thanks for all your work !
> 
Some questions about your Nvidia GPU passthrough setup:

- What version of Xen are you using? 
- Are you using any additional patches for Nvidia GPU passthrough? 

Thanks,

-- Pasi

On Wed, 13 Mar 2013, Hanweidong wrote:
> MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses below code to
init
> RAM in xen_ram_init: 
> 
>     ...
>     block_len = ram_size;
>     if (ram_size >= HVM_BELOW_4G_RAM_END) {
>         /* Xen does not allocate the memory continuously, and keep a hole
at
>          * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
>          */
>         block_len += HVM_BELOW_4G_MMIO_LENGTH;
>     }
>     memory_region_init_ram(&ram_memory, "xen.ram",
block_len);
>     vmstate_register_ram_global(&ram_memory);
> 
>     if (ram_size >= HVM_BELOW_4G_RAM_END) {
>         above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
>         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
>     } else {
>         below_4g_mem_size = ram_size;
>     }
>     ...
> 
> HVM_BELOW_4G_RAM_END is f0000000. If we change HVM_BELOW_4G_RAM_END to
e0000000,
> Which it''s consistent with hvmloader when assigning a GPU, and
then guest worked
> for us. So we wondering that xen_ram_init in QEMU should be consistent with
> hvmloader. 
> 
> In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1() as below.
> Should keep these places handle the consistent mmio hole or not?
> 
>     if (ram_size >= 0xe0000000 ) {
>         above_4g_mem_size = ram_size - 0xe0000000;
>         below_4g_mem_size = 0xe0000000;
>     } else {
>         above_4g_mem_size = 0;
>         below_4g_mem_size = ram_size;
>     }
The guys at Intel sent a couple of patches recently to fix this issue:

http://marc.info/?l=xen-devel&m=136150317011027
http://marc.info/?l=qemu-devel&m=136177475215360&w=2

Do they solve your problem?

Xudong and Xiantao,
are you going to send an update of the second patch to QEMU?

Thanks,
-Xudong

> -----Original Message-----
> From: Stefano Stabellini [mailto:stefano.stabellini@eu.citrix.com]
> Sent: Monday, March 18, 2013 8:02 PM
> To: Hanweidong
> Cc: George Dunlap; Stefano Stabellini; Yanqiangjun; Luonengjun;
Wangzhenguo;
> Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-devel@lists.xen.org; Hao,
> Xudong; Zhang, Xiantao
> Subject: RE: [Xen-devel] GPU passthrough issue when VM is configured with
4G
> memory
> 
> On Wed, 13 Mar 2013, Hanweidong wrote:
> > MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses below
> code to init
> > RAM in xen_ram_init:
> >
> >     ...
> >     block_len = ram_size;
> >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> >         /* Xen does not allocate the memory continuously, and keep a
hole
> at
> >          * HVM_BELOW_4G_MMIO_START of
> HVM_BELOW_4G_MMIO_LENGTH
> >          */
> >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
> >     }
> >     memory_region_init_ram(&ram_memory, "xen.ram",
block_len);
> >     vmstate_register_ram_global(&ram_memory);
> >
> >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> >         above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
> >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
> >     } else {
> >         below_4g_mem_size = ram_size;
> >     }
> >     ...
> >
> > HVM_BELOW_4G_RAM_END is f0000000. If we change
> HVM_BELOW_4G_RAM_END to e0000000,
> > Which it''s consistent with hvmloader when assigning a GPU,
and then guest
> worked
> > for us. So we wondering that xen_ram_init in QEMU should be consistent
> with
> > hvmloader.
> >
> > In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1() as
> below.
> > Should keep these places handle the consistent mmio hole or not?
> >
> >     if (ram_size >= 0xe0000000 ) {
> >         above_4g_mem_size = ram_size - 0xe0000000;
> >         below_4g_mem_size = 0xe0000000;
> >     } else {
> >         above_4g_mem_size = 0;
> >         below_4g_mem_size = ram_size;
> >     }
> 
> The guys at Intel sent a couple of patches recently to fix this issue:
> 
> http://marc.info/?l=xen-devel&m=136150317011027
> http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> 
> Do they solve your problem?
> 
> Xudong and Xiantao,
> are you going to send an update of the second patch to QEMU?
Hi, Stefano

We''d like to send an update of second patch(implement this register as
a write-only one). However, we have other priority things if it must be
implement by xenbus, so how do you think?

Le 17.03.2013 18:32, Pasi Kärkkäinen a écrit :
> On Sun, Mar 17, 2013 at 12:41:01AM +0100, 
> youenn.gestin@grim-is-a-geek.fr wrote:
>>
>> Hello,
>>
>> What can of bug we can encounter by hardcoding the
>> HVM_BELOW_4G_RAM_END to e0000000 ?
>>
>> By doing this I'm able to boot on Win7 with gpu passthrough of an
>> nvidia 560TI an 12Go of ram.
>>
>> Regards and thanks for all your work !
>>
>
> Some questions about your Nvidia GPU passthrough setup:
>
> - What version of Xen are you using?
> - Are you using any additional patches for Nvidia GPU passthrough?
>
> Thanks,
>
> -- Pasi
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
Hello Pasi,

I'm using xen 4.2.0

You can find the patchs i used here whose mostly came from David 
Techer's blog : 
http://www.grim-is-a-geek.fr/xen-gfx-passthrough-patchs.bz2
And information about my xen Dom_0/Dom_U config here : 
http://grim-is-a-geek.fr/index.php?option=com_content&view=article&id=52:xen-configuration-for-win7-and-nvidia-gpu-passthrough&catid=35:linux&Itemid=77

Regards,
Youenn.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

> -----Original Message-----
> From: Stefano Stabellini [mailto:stefano.stabellini@eu.citrix.com]
> Sent: 2013年3月18日 20:02
> To: Hanweidong
> Cc: George Dunlap; Stefano Stabellini; Yanqiangjun; Luonengjun;
> Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> devel@lists.xen.org; xudong.hao@intel.com; xiantao.zhang@intel.com
> Subject: RE: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> On Wed, 13 Mar 2013, Hanweidong wrote:
> > MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses below
> code to init
> > RAM in xen_ram_init:
> >
> >     ...
> >     block_len = ram_size;
> >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> >         /* Xen does not allocate the memory continuously, and keep a
> hole at
> >          * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
> >          */
> >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
> >     }
> >     memory_region_init_ram(&ram_memory, "xen.ram",
block_len);
> >     vmstate_register_ram_global(&ram_memory);
> >
> >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> >         above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
> >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
> >     } else {
> >         below_4g_mem_size = ram_size;
> >     }
> >     ...
> >
> > HVM_BELOW_4G_RAM_END is f0000000. If we change HVM_BELOW_4G_RAM_END
> to e0000000,
> > Which it's consistent with hvmloader when assigning a GPU, and
then
> guest worked
> > for us. So we wondering that xen_ram_init in QEMU should be
> consistent with
> > hvmloader.
> >
> > In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1() as
> below.
> > Should keep these places handle the consistent mmio hole or not?
> >
> >     if (ram_size >= 0xe0000000 ) {
> >         above_4g_mem_size = ram_size - 0xe0000000;
> >         below_4g_mem_size = 0xe0000000;
> >     } else {
> >         above_4g_mem_size = 0;
> >         below_4g_mem_size = ram_size;
> >     }
> 
> The guys at Intel sent a couple of patches recently to fix this issue:
> 
> http://marc.info/?l=xen-devel&m=136150317011027
> http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> 
> Do they solve your problem?
I think it's the solution. We will verify it. Thanks.

--weidong

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Hi Youenn,

Good to know that it works for you :)

You should be well advised to share expected information for both your
motherboard and your Nvidia graphic card.

(VGA PassThrough, Nvidia) works for Win7 with very rare exceptions.


Kind regards.

David.

________________________________
 De : "youenn.gestin@grim-is-a-geek.fr"
<youenn.gestin@grim-is-a-geek.fr>
À : xen-devel@lists.xen.org 
Envoyé le : Lundi 18 mars 2013 23h43
Objet : Re: [Xen-devel] GPU passthrough issue when VM is configured with 4G
memory
 
Le 17.03.2013 18:32, Pasi Kärkkäinen a écrit :
> On Sun, Mar 17, 2013 at 12:41:01AM +0100, 
> youenn.gestin@grim-is-a-geek.fr wrote:
>>
>> Hello,
>>
>> What can of bug we can encounter by hardcoding the
>> HVM_BELOW_4G_RAM_END to e0000000 ?
>>
>> By doing this I''m able to boot on Win7 with gpu passthrough of
an
>> nvidia 560TI an 12Go of ram.
>>
>> Regards and thanks for all your work !
>>
>
> Some questions about your Nvidia GPU passthrough setup:
>
> - What version of Xen are you using?
> - Are you using any additional patches for Nvidia GPU passthrough?
>
> Thanks,
>
> -- Pasi
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
Hello Pasi,

I''m using xen 4.2.0

You can find the patchs i used here whose mostly came from David 
Techer''s blog : 
http://www.grim-is-a-geek.fr/xen-gfx-passthrough-patchs.bz2
And information about my xen Dom_0/Dom_U config here : 
http://grim-is-a-geek.fr/index.php?option=com_content&view=article&id=52:xen-configuration-for-win7-and-nvidia-gpu-passthrough&catid=35:linux&Itemid=77

Regards,
Youenn.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

> -----Original Message-----
> From: Stefano Stabellini [mailto:stefano.stabellini@eu.citrix.com]
> Sent: 2013年3月18日 20:02
> To: Hanweidong
> Cc: George Dunlap; Stefano Stabellini; Yanqiangjun; Luonengjun;
> Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> devel@lists.xen.org; xudong.hao@intel.com; xiantao.zhang@intel.com
> Subject: RE: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> On Wed, 13 Mar 2013, Hanweidong wrote:
> > MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses below
> code to init
> > RAM in xen_ram_init:
> >
> >     ...
> >     block_len = ram_size;
> >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> >         /* Xen does not allocate the memory continuously, and keep a
> hole at
> >          * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
> >          */
> >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
> >     }
> >     memory_region_init_ram(&ram_memory, "xen.ram",
block_len);
> >     vmstate_register_ram_global(&ram_memory);
> >
> >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> >         above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
> >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
> >     } else {
> >         below_4g_mem_size = ram_size;
> >     }
> >     ...
> >
> > HVM_BELOW_4G_RAM_END is f0000000. If we change HVM_BELOW_4G_RAM_END
> to e0000000,
> > Which it's consistent with hvmloader when assigning a GPU, and
then
> guest worked
> > for us. So we wondering that xen_ram_init in QEMU should be
> consistent with
> > hvmloader.
> >
> > In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1() as
> below.
> > Should keep these places handle the consistent mmio hole or not?
> >
> >     if (ram_size >= 0xe0000000 ) {
> >         above_4g_mem_size = ram_size - 0xe0000000;
> >         below_4g_mem_size = 0xe0000000;
> >     } else {
> >         above_4g_mem_size = 0;
> >         below_4g_mem_size = ram_size;
> >     }
> 
> The guys at Intel sent a couple of patches recently to fix this issue:
> 
> http://marc.info/?l=xen-devel&m=136150317011027
> http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> 
> Do they solve your problem?
These two patches didn't solve our problem. 

--weidong
> 
> Xudong and Xiantao,
> are you going to send an update of the second patch to QEMU?
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Tue, Mar 26, 2013 at 09:37:53AM +0000, Hanweidong
wrote:
> > >
> > > HVM_BELOW_4G_RAM_END is f0000000. If we change
HVM_BELOW_4G_RAM_END
> > to e0000000,
> > > Which it''s consistent with hvmloader when assigning a
GPU, and then
> > guest worked
> > > for us. So we wondering that xen_ram_init in QEMU should be
> > consistent with
> > > hvmloader.
> > >
> > > In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1()
as
> > below.
> > > Should keep these places handle the consistent mmio hole or not?
> > >
> > >     if (ram_size >= 0xe0000000 ) {
> > >         above_4g_mem_size = ram_size - 0xe0000000;
> > >         below_4g_mem_size = 0xe0000000;
> > >     } else {
> > >         above_4g_mem_size = 0;
> > >         below_4g_mem_size = ram_size;
> > >     }
> > 
> > The guys at Intel sent a couple of patches recently to fix this issue:
> > 
> > http://marc.info/?l=xen-devel&m=136150317011027
> > http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> > 
> > Do they solve your problem?
> 
> These two patches didn''t solve our problem. 
>
Any updates on this? It''d be nice to get this fixed before Xen 4.3. 

Thanks,

-- Pasi
 
> --weidong
> 
> > 
> > Xudong and Xiantao,
> > are you going to send an update of the second patch to QEMU?

Those patches works fine for me with my ATI card both for Windows 7 64 bit or
Linux Mint 14 64-Bit.

Both were tested with 6GB and 8GB of RAM and vcpus = 6

To do ''make stubdom'' I have to find a way to neuter iopl()
call from tools/qemu-xen-traditional-dir-remote/hw/pt-graphics.c

All details provided here

http://www.davidgis.fr/blog/index.php?2013/04/05/937-xen-43-unstable-vga-passthrough-hd-7970-windows-7-64-bits-with-more-than-3gb-for-ram






________________________________
 De : Pasi Kärkkäinen <pasik@iki.fi>
À : Hanweidong <hanweidong@huawei.com> 
Cc : Stefano Stabellini <stefano.stabellini@eu.citrix.com>; George Dunlap
<George.Dunlap@eu.citrix.com>; "xudong.hao@intel.com"
<xudong.hao@intel.com>; Yanqiangjun <yanqiangjun@huawei.com>;
Luonengjun <luonengjun@huawei.com>; Wangzhenguo
<wangzhenguo@huawei.com>; Yangxiaowei <xiaowei.yang@huawei.com>;
Gonglei (Arei) <arei.gonglei@huawei.com>; Anthony Perard
<anthony.perard@citrix.com>; "xen-devel@lists.xen.org"
<xen-devel@lists.xen.org>; "xiantao.zhang@intel.com"
<xiantao.zhang@intel.com>
Envoyé le : Lundi 15 avril 2013 23h22
Objet : Re: [Xen-devel] GPU passthrough issue when VM is configured with 4G
memory
 

On Tue, Mar 26, 2013 at 09:37:53AM +0000, Hanweidong
wrote:
> > >
> > > HVM_BELOW_4G_RAM_END is f0000000. If we change
HVM_BELOW_4G_RAM_END
> > to e0000000,
> > > Which it''s consistent with hvmloader when assigning a
GPU, and then
> > guest worked
> > > for us. So we wondering that xen_ram_init in QEMU should be
> > consistent with
> > > hvmloader.
> > >
> > > In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1()
as
> > below.
> > > Should keep these places handle the consistent mmio hole or not?
> > >
> > >     if (ram_size >= 0xe0000000 ) {
> > >         above_4g_mem_size = ram_size - 0xe0000000;
> > >         below_4g_mem_size = 0xe0000000;
> > >     } else {
> > >         above_4g_mem_size = 0;
> > >         below_4g_mem_size = ram_size;
> > >     }
> > 
> > The guys at Intel sent a couple of patches recently to fix this issue:
> > 
> > http://marc.info/?l=xen-devel&m=136150317011027
> > http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> > 
> > Do they solve your problem?
> 
> These two patches didn''t solve our problem. 
>
Any updates on this? It''d be nice to get this fixed before Xen 4.3. 

Thanks,

-- Pasi
> --weidong
> 
> > 
> > Xudong and Xiantao,
> > are you going to send an update of the second patch to QEMU?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Tue, Apr 16, 2013 at 01:44:23AM +0100, David TECHER
wrote:
>    Those patches works fine for me with my ATI card both for Windows 7 64
bit
>    or Linux Mint 14 64-Bit.
> 
>    Both were tested with 6GB and 8GB of RAM and vcpus = 6
>
Hmm.. so the patches work OK for David, but fail for Weidong.
I wonder what''s the difference in your setups? 

And thanks David for testing!


-- Pasi
 
>    To do ''make stubdom'' I have to find a way to neuter
iopl() call from
>    tools/qemu-xen-traditional-dir-remote/hw/pt-graphics.c
> 
>    All details provided here
>   
http://www.davidgis.fr/blog/index.php?2013/04/05/937-xen-43-unstable-vga-passthrough-hd-7970-windows-7-64-bits-with-more-than-3gb-for-ram
> 
>   
--------------------------------------------------------------------------
> 
>    De : Pasi Kärkkäinen <pasik@iki.fi>
>    À : Hanweidong <hanweidong@huawei.com>
>    Cc : Stefano Stabellini <stefano.stabellini@eu.citrix.com>; George
Dunlap
>    <George.Dunlap@eu.citrix.com>; "xudong.hao@intel.com"
>    <xudong.hao@intel.com>; Yanqiangjun
<yanqiangjun@huawei.com>; Luonengjun
>    <luonengjun@huawei.com>; Wangzhenguo
<wangzhenguo@huawei.com>; Yangxiaowei
>    <xiaowei.yang@huawei.com>; Gonglei (Arei)
<arei.gonglei@huawei.com>;
>    Anthony Perard <anthony.perard@citrix.com>;
"xen-devel@lists.xen.org"
>    <xen-devel@lists.xen.org>; "xiantao.zhang@intel.com"
>    <xiantao.zhang@intel.com>
>    Envoyé le : Lundi 15 avril 2013 23h22
>    Objet : Re: [Xen-devel] GPU passthrough issue when VM is configured with
>    4G memory
>    On Tue, Mar 26, 2013 at 09:37:53AM +0000, Hanweidong wrote:
>    > > >
>    > > > HVM_BELOW_4G_RAM_END is f0000000. If we change
HVM_BELOW_4G_RAM_END
>    > > to e0000000,
>    > > > Which it''s consistent with hvmloader when
assigning a GPU, and then
>    > > guest worked
>    > > > for us. So we wondering that xen_ram_init in QEMU should
be
>    > > consistent with
>    > > > hvmloader.
>    > > >
>    > > > In addition, we found QEMU uses hardcode 0xe0000000 in
pc_init1() as
>    > > below.
>    > > > Should keep these places handle the consistent mmio hole
or not?
>    > > >
>    > > >    if (ram_size >= 0xe0000000 ) {
>    > > >        above_4g_mem_size = ram_size - 0xe0000000;
>    > > >        below_4g_mem_size = 0xe0000000;
>    > > >    } else {
>    > > >        above_4g_mem_size = 0;
>    > > >        below_4g_mem_size = ram_size;
>    > > >    }
>    > >
>    > > The guys at Intel sent a couple of patches recently to fix
this issue:
>    > >
>    > > [1]http://marc.info/?l=xen-devel&m=136150317011027
>    > >
[2]http://marc.info/?l=qemu-devel&m=136177475215360&w=2
>    > >
>    > > Do they solve your problem?
>    >
>    > These two patches didn''t solve our problem.
>    >
> 
>    Any updates on this? It''d be nice to get this fixed before Xen
4.3.
> 
>    Thanks,
> 
>    -- Pasi
> 
>    > --weidong
>    >
>    > >
>    > > Xudong and Xiantao,
>    > > are you going to send an update of the second patch to QEMU?
> 
>    _______________________________________________
>    Xen-devel mailing list
>    [3]Xen-devel@lists.xen.org
>    [4]http://lists.xen.org/xen-devel
> 
> References
> 
>    Visible links
>    1. http://marc.info/?l=xen-devel&m=136150317011027
>    2. http://marc.info/?l=qemu-devel&m=136177475215360&w=2
>    3. mailto:Xen-devel@lists.xen.org
>    4. http://lists.xen.org/xen-devel

Hi Pasi,

FYI and to be sure that we understand each other  I just applied the following
patches in order

- to have VGA passthrough with ATI :
ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch

- to have more that 4GB of RAM: 
http://marc.info/?l=qemu-devel&m=136177475215360

Here are the details (a simple copy-paste from the prior link)..As previously
noticed I have to build studom as latest step by commenting out any call to
iopl() function.
To be honest I think the method I used could be improved. Unfortunately I am not
a developer.

Download Xen sources
===================rev=267773
hg clone -r $rev http://xenbits.xensource.com/staging/xen-unstable.hg/
xen-unstable.hg-rev-XXXXX
cd xen-unstable.hg-rev-XXXXX

Configure
========CURL=$(which curl-config) XML=$(which xml2-config) ./configure

Make a first build for tools and cleanup the folder
================================================== *   Make a first build for
tools

    cd tools
    make -j4

 *  Clean up the folder

    make clean

Download and apply the patches
============================= *   Download and apply the 1st patch (patch for
RAM > 3GB)

    cd qemu-xen-dir-remote/
    wget
"http://marc.info/?l=qemu-devel&m=136177475215360&q=raw" -O -
| patch -p1

 *   Download and apply the 2nd patch (patch for ATI)

    cd ../..
    wget ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch -O - |
sed -e "s:qemu-xen-traditional:qemu-xen-traditional-dir-remote:g" |
patch -p1

 *   We have to modify tools/qemu-xen-traditional-dir-remote/hw/pt-graphics.c
file so we can run build stubdom later

    for i in 0 3;do sed -i "s:^.*iopl(${i});$://iopl(${i});:g"
tools/qemu-xen-traditional-dir-remote/hw/pt-graphics.c;done

    NOTICE if this workaround is not applied then you should have this error

    /opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/mini-os-x86_64-ioemu/mini-os.o:
In function `ati_hw_out'':
    /opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:82:
undefined reference to `iopl''
    /opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:84:
undefined reference to `iopl''
    /opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/mini-os-x86_64-ioemu/mini-os.o:
In function `ati_hw_in'':
    /opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:72:
undefined reference to `iopl''
    /opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:74:
undefined reference to `iopl''

Build and install xen and stubdom
================================  *  Build

    make -j4 xen && make -j4 stubdom

  *  Install

    make install-xen && make install-stubdom

Build and install tools
======================  *  Cleanup tools

    cd tools
    make clean

 *  Reverse the workaround by commenting out any call to iopl() function

    cd ..
    for i in 0 3;do sed -i "s:^.*iopl(${i});$:iopl(${i});:g"
tools/qemu-xen-traditional-dir-remote/hw/pt-graphics.c;done

 *   Build and install tools

    make -j4 tools
    make install-tools PYTHON_PREFIX_ARG





________________________________
 De : Pasi Kärkkäinen <pasik@iki.fi>
À : David TECHER <davidtecher@yahoo.fr> 
Cc : Stefano Stabellini <stefano.stabellini@eu.citrix.com>; George Dunlap
<George.Dunlap@eu.citrix.com>; "xudong.hao@intel.com"
<xudong.hao@intel.com>; Yanqiangjun <yanqiangjun@huawei.com>;
Luonengjun <luonengjun@huawei.com>; Wangzhenguo
<wangzhenguo@huawei.com>; Yangxiaowei <xiaowei.yang@huawei.com>;
Gonglei (Arei) <arei.gonglei@huawei.com>; Anthony Perard
<anthony.perard@citrix.com>; "xen-devel@lists.xen.org"
<xen-devel@lists.xen.org>; Hanweidong <hanweidong@huawei.com>;
"xiantao.zhang@intel.com" <xiantao.zhang@intel.com>
Envoyé le : Mardi 16 avril 2013 5h54
Objet : Re: [Xen-devel] GPU passthrough issue when VM is configured with 4G
memory
 

On Tue, Apr 16, 2013 at 01:44:23AM +0100, David TECHER
wrote:
>    Those patches works fine for me with my ATI card both for Windows 7 64
bit
>    or Linux Mint 14 64-Bit.
> 
>    Both were tested with 6GB and 8GB of RAM and vcpus = 6
>
Hmm.. so the patches work OK for David, but fail for Weidong.
I wonder what''s the difference in your setups? 

And thanks David for testing!


-- Pasi
>    To do ''make stubdom'' I have to find a way to neuter
iopl() call from
>    tools/qemu-xen-traditional-dir-remote/hw/pt-graphics.c
> 
>    All details provided here
>   
http://www.davidgis.fr/blog/index.php?2013/04/05/937-xen-43-unstable-vga-passthrough-hd-7970-windows-7-64-bits-with-more-than-3gb-for-ram
> 
>   
--------------------------------------------------------------------------
> 
>    De : Pasi Kärkkäinen <pasik@iki.fi>
>    À : Hanweidong <hanweidong@huawei.com>
>    Cc : Stefano Stabellini <stefano.stabellini@eu.citrix.com>; George
Dunlap
>    <George.Dunlap@eu.citrix.com>; "xudong.hao@intel.com"
>    <xudong.hao@intel.com>; Yanqiangjun
<yanqiangjun@huawei.com>; Luonengjun
>    <luonengjun@huawei.com>; Wangzhenguo
<wangzhenguo@huawei.com>; Yangxiaowei
>    <xiaowei.yang@huawei.com>; Gonglei (Arei)
<arei.gonglei@huawei.com>;
>    Anthony Perard <anthony.perard@citrix.com>;
"xen-devel@lists.xen.org"
>    <xen-devel@lists.xen.org>; "xiantao.zhang@intel.com"
>    <xiantao.zhang@intel.com>
>    Envoyé le : Lundi 15 avril 2013 23h22
>    Objet : Re: [Xen-devel] GPU passthrough issue when VM is configured with
>    4G memory
>    On Tue, Mar 26, 2013 at 09:37:53AM +0000, Hanweidong wrote:
>    > > >
>    > > > HVM_BELOW_4G_RAM_END is f0000000. If we change
HVM_BELOW_4G_RAM_END
>    > > to e0000000,
>    > > > Which it''s consistent with hvmloader when
assigning a GPU, and then
>    > > guest worked
>    > > > for us. So we wondering that xen_ram_init in QEMU should
be
>    > > consistent with
>    > > > hvmloader.
>    > > >
>    > > > In addition, we found QEMU uses hardcode 0xe0000000 in
pc_init1() as
>    > > below.
>    > > > Should keep these places handle the consistent mmio hole
or not?
>    > > >
>    > > >    if (ram_size >= 0xe0000000 ) {
>    > > >        above_4g_mem_size = ram_size - 0xe0000000;
>    > > >        below_4g_mem_size = 0xe0000000;
>    > > >    } else {
>    > > >        above_4g_mem_size = 0;
>    > > >        below_4g_mem_size = ram_size;
>    > > >    }
>    > >
>    > > The guys at Intel sent a couple of patches recently to fix
this issue:
>    > >
>    > > [1]http://marc.info/?l=xen-devel&m=136150317011027
>    > >
[2]http://marc.info/?l=qemu-devel&m=136177475215360&w=2
>    > >
>    > > Do they solve your problem?
>    >
>    > These two patches didn''t solve our problem.
>    >
> 
>    Any updates on this? It''d be nice to get this fixed before Xen
4.3.
> 
>    Thanks,
> 
>    -- Pasi
> 
>    > --weidong
>    >
>    > >
>    > > Xudong and Xiantao,
>    > > are you going to send an update of the second patch to QEMU?
> 
>    _______________________________________________
>    Xen-devel mailing list
>    [3]Xen-devel@lists.xen.org
>    [4]http://lists.xen.org/xen-devel
> 
> References
> 
>    Visible links
>    1. http://marc.info/?l=xen-devel&m=136150317011027
>    2. http://marc.info/?l=qemu-devel&m=136177475215360&w=2
>    3. mailto:Xen-devel@lists.xen.org
>    4. http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 15/04/13 22:22, Pasi Kärkkäinen wrote:
> On Tue, Mar 26, 2013 at 09:37:53AM +0000, Hanweidong wrote:
>>>> HVM_BELOW_4G_RAM_END is f0000000. If we change
HVM_BELOW_4G_RAM_END
>>> to e0000000,
>>>> Which it''s consistent with hvmloader when assigning a
GPU, and then
>>> guest worked
>>>> for us. So we wondering that xen_ram_init in QEMU should be
>>> consistent with
>>>> hvmloader.
>>>>
>>>> In addition, we found QEMU uses hardcode 0xe0000000 in
pc_init1() as
>>> below.
>>>> Should keep these places handle the consistent mmio hole or
not?
>>>>
>>>>      if (ram_size >= 0xe0000000 ) {
>>>>          above_4g_mem_size = ram_size - 0xe0000000;
>>>>          below_4g_mem_size = 0xe0000000;
>>>>      } else {
>>>>          above_4g_mem_size = 0;
>>>>          below_4g_mem_size = ram_size;
>>>>      }
>>> The guys at Intel sent a couple of patches recently to fix this
issue:
>>>
>>> http://marc.info/?l=xen-devel&m=136150317011027
>>> http://marc.info/?l=qemu-devel&m=136177475215360&w=2
>>>
>>> Do they solve your problem?
>> These two patches didn''t solve our problem.
>>
> Any updates on this? It''d be nice to get this fixed before Xen
4.3.
It looks like the qemu guys didn''t like the patch; they said there was 
already an interface to get this information.  I didn''t see whether Xu 
Dong followed that up or not.

I''m going to list this as a bug, hopefully we can get it sorted out.

  -George

>
> Thanks,
>
> -- Pasi
>   
>> --weidong
>>
>>> Xudong and Xiantao,
>>> are you going to send an update of the second patch to QEMU?

I assigned the nvidia quadro 2000 card as the normal pci pass-through instead of
VGA pass-through. The card’s mmio size is about 256M. what’s the mmio size of
your ATI card?

-weidong

From: David TECHER [mailto:davidtecher@yahoo.fr]
Sent: 2013年4月16日 17:22
To: Pasi Kärkkäinen
Cc: Stefano Stabellini; George Dunlap; xudong.hao@intel.com; Yanqiangjun;
Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard;
xen-devel@lists.xen.org; Hanweidong; xiantao.zhang@intel.com
Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured with 4G
memory

Hi Pasi,

FYI and to be sure that we understand each other  I just applied the following
patches in order

- to have VGA passthrough with ATI :
ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch

- to have more that 4GB of RAM: 
http://marc.info/?l=qemu-devel&m=136177475215360

Here are the details (a simple copy-paste from the prior link)..As previously
noticed I have to build studom as latest step by commenting out any call to
iopl() function.
To be honest I think the method I used could be improved. Unfortunately I am not
a developer.

Download Xen sources
===================rev=267773
hg clone -r $rev http://xenbits.xensource.com/staging/xen-unstable.hg/
xen-unstable.hg-rev-XXXXX
cd xen-unstable.hg-rev-XXXXX

Configure
========CURL=$(which curl-config) XML=$(which xml2-config) ./configure

Make a first build for tools and cleanup the folder
================================================== *   Make a first build for
tools

    cd tools
    make -j4

 *  Clean up the folder

    make clean

Download and apply the patches
============================= *   Download and apply the 1st patch (patch for
RAM > 3GB)

    cd qemu-xen-dir-remote/
    wget
"http://marc.info/?l=qemu-devel&m=136177475215360&q=raw" -O -
| patch -p1

 *   Download and apply the 2nd patch (patch for ATI)

    cd ../..
    wget ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch -O - |
sed -e "s:qemu-xen-traditional:qemu-xen-traditional-dir-remote:g" |
patch -p1

 *   We have to modify tools/qemu-xen-traditional-dir-remote/hw/pt-graphics.c
file so we can run build stubdom later

    for i in 0 3;do sed -i "s:^.*iopl(${i});$://iopl(${i});:g"
tools/qemu-xen-traditional-dir-remote/hw/pt-graphics.c;done

    NOTICE if this workaround is not applied then you should have this error

    /opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/mini-os-x86_64-ioemu/mini-os.o:
In function `ati_hw_out':
    /opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:82:
undefined reference to `iopl'
    /opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:84:
undefined reference to `iopl'
    /opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/mini-os-x86_64-ioemu/mini-os.o:
In function `ati_hw_in':
    /opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:72:
undefined reference to `iopl'
    /opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:74:
undefined reference to `iopl'

Build and install xen and stubdom
================================  *  Build

    make -j4 xen && make -j4 stubdom

  *  Install

    make install-xen && make install-stubdom

Build and install tools
======================  *  Cleanup tools

    cd tools
    make clean

 *  Reverse the workaround by commenting out any call to iopl() function

    cd ..
    for i in 0 3;do sed -i "s:^.*iopl(${i});$:iopl(${i});:g"
tools/qemu-xen-traditional-dir-remote/hw/pt-graphics.c;done

 *   Build and install tools

    make -j4 tools
    make install-tools PYTHON_PREFIX_ARG


________________________________
De : Pasi Kärkkäinen <pasik@iki.fi>
À : David TECHER <davidtecher@yahoo.fr>
Cc : Stefano Stabellini <stefano.stabellini@eu.citrix.com>; George Dunlap
<George.Dunlap@eu.citrix.com>; "xudong.hao@intel.com"
<xudong.hao@intel.com>; Yanqiangjun <yanqiangjun@huawei.com>;
Luonengjun <luonengjun@huawei.com>; Wangzhenguo
<wangzhenguo@huawei.com>; Yangxiaowei <xiaowei.yang@huawei.com>;
Gonglei (Arei) <arei.gonglei@huawei.com>; Anthony Perard
<anthony.perard@citrix.com>; "xen-devel@lists.xen.org"
<xen-devel@lists.xen.org>; Hanweidong <hanweidong@huawei.com>;
"xiantao.zhang@intel.com" <xiantao.zhang@intel.com>
Envoyé le : Mardi 16 avril 2013 5h54
Objet : Re: [Xen-devel] GPU passthrough issue when VM is configured with 4G
memory

On Tue, Apr 16, 2013 at 01:44:23AM +0100, David TECHER
wrote:
>    Those patches works fine for me with my ATI card both for Windows 7 64
bit
>    or Linux Mint 14 64-Bit.
>
>    Both were tested with 6GB and 8GB of RAM and vcpus = 6
>
Hmm.. so the patches work OK for David, but fail for Weidong.
I wonder what's the difference in your setups?

And thanks David for testing!


-- Pasi
>    To do 'make stubdom' I have to find a way to neuter iopl() call
from
>    tools/qemu-xen-traditional-dir-remote/hw/pt-graphics.c
>
>    All details provided here
>   
http://www.davidgis.fr/blog/index.php?2013/04/05/937-xen-43-unstable-vga-passthrough-hd-7970-windows-7-64-bits-with-more-than-3gb-for-ram
>
>   
--------------------------------------------------------------------------
>
>    De : Pasi Kärkkäinen <pasik@iki.fi<mailto:pasik@iki.fi>>
>    À : Hanweidong
<hanweidong@huawei.com<mailto:hanweidong@huawei.com>>
>    Cc : Stefano Stabellini
<stefano.stabellini@eu.citrix.com<mailto:stefano.stabellini@eu.citrix.com>>;
George Dunlap
>   
<George.Dunlap@eu.citrix.com<mailto:George.Dunlap@eu.citrix.com>>;
"xudong.hao@intel.com<mailto:xudong.hao@intel.com>"
>    <xudong.hao@intel.com<mailto:xudong.hao@intel.com>>;
Yanqiangjun <yanqiangjun@huawei.com<mailto:yanqiangjun@huawei.com>>;
Luonengjun
>    <luonengjun@huawei.com<mailto:luonengjun@huawei.com>>;
Wangzhenguo <wangzhenguo@huawei.com<mailto:wangzhenguo@huawei.com>>;
Yangxiaowei
>    <xiaowei.yang@huawei.com<mailto:xiaowei.yang@huawei.com>>;
Gonglei (Arei)
<arei.gonglei@huawei.com<mailto:arei.gonglei@huawei.com>>;
>    Anthony Perard
<anthony.perard@citrix.com<mailto:anthony.perard@citrix.com>>;
"xen-devel@lists.xen.org<mailto:xen-devel@lists.xen.org>"
>    <xen-devel@lists.xen.org<mailto:xen-devel@lists.xen.org>>;
"xiantao.zhang@intel.com<mailto:xiantao.zhang@intel.com>"
>    <xiantao.zhang@intel.com<mailto:xiantao.zhang@intel.com>>
>    Envoyé le : Lundi 15 avril 2013 23h22
>    Objet : Re: [Xen-devel] GPU passthrough issue when VM is configured with
>    4G memory
>    On Tue, Mar 26, 2013 at 09:37:53AM +0000, Hanweidong wrote:
>    > > >
>    > > > HVM_BELOW_4G_RAM_END is f0000000. If we change
HVM_BELOW_4G_RAM_END
>    > > to e0000000,
>    > > > Which it's consistent with hvmloader when assigning a
GPU, and then
>    > > guest worked
>    > > > for us. So we wondering that xen_ram_init in QEMU should
be
>    > > consistent with
>    > > > hvmloader.
>    > > >
>    > > > In addition, we found QEMU uses hardcode 0xe0000000 in
pc_init1() as
>    > > below.
>    > > > Should keep these places handle the consistent mmio hole
or not?
>    > > >
>    > > >    if (ram_size >= 0xe0000000 ) {
>    > > >        above_4g_mem_size = ram_size - 0xe0000000;
>    > > >        below_4g_mem_size = 0xe0000000;
>    > > >    } else {
>    > > >        above_4g_mem_size = 0;
>    > > >        below_4g_mem_size = ram_size;
>    > > >    }
>    > >
>    > > The guys at Intel sent a couple of patches recently to fix
this issue:
>    > >
>    > > [1]http://marc.info/?l=xen-devel&m=136150317011027
>    > >
[2]http://marc.info/?l=qemu-devel&m=136177475215360&w=2
>    > >
>    > > Do they solve your problem?
>    >
>    > These two patches didn't solve our problem.
>    >
>
>    Any updates on this? It'd be nice to get this fixed before Xen 4.3.
>
>    Thanks,
>
>    -- Pasi
>
>    > --weidong
>    >
>    > >
>    > > Xudong and Xiantao,
>    > > are you going to send an update of the second patch to QEMU?
>
>    _______________________________________________
>    Xen-devel mailing list
>    [3]Xen-devel@lists.xen.org<mailto:Xen-devel@lists.xen.org>
>    [4]http://lists.xen.org/xen-devel
>
> References
>
>    Visible links
>    1. http://marc.info/?l=xen-devel&m=136150317011027
>    2. http://marc.info/?l=qemu-devel&m=136177475215360&w=2
>    3. mailto:Xen-devel@lists.xen.org<mailto:Xen-devel@lists.xen.org>
>    4. http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org<mailto:Xen-devel@lists.xen.org>
http://lists.xen.org/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Tue, 2013-04-16 at 13:37 +0100, George Dunlap wrote:
> It looks like the qemu guys didn''t like the patch;
Neither did I, I disagree with adding a magic register to an emulation
of an *real* piece of hardware (as opposed to some made up virtual one).
Particularly when, as you''ve noted, there are other way to get this
information.

Ian.

> -----Original Message-----
> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> bounces@lists.xen.org] On Behalf Of Hanweidong
> Sent: 2013年3月26日 17:38
> To: Stefano Stabellini
> Cc: George Dunlap; xudong.hao@intel.com; Yanqiangjun; Luonengjun;
> Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> devel@lists.xen.org; xiantao.zhang@intel.com
> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> 
> > -----Original Message-----
> > From: Stefano Stabellini [mailto:stefano.stabellini@eu.citrix.com]
> > Sent: 2013年3月18日 20:02
> > To: Hanweidong
> > Cc: George Dunlap; Stefano Stabellini; Yanqiangjun; Luonengjun;
> > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> > devel@lists.xen.org; xudong.hao@intel.com; xiantao.zhang@intel.com
> > Subject: RE: [Xen-devel] GPU passthrough issue when VM is configured
> > with 4G memory
> >
> > On Wed, 13 Mar 2013, Hanweidong wrote:
> > > MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses
below
> > code to init
> > > RAM in xen_ram_init:
> > >
> > >     ...
> > >     block_len = ram_size;
> > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > >         /* Xen does not allocate the memory continuously, and
keep
> a
> > hole at
> > >          * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
> > >          */
> > >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
> > >     }
> > >     memory_region_init_ram(&ram_memory, "xen.ram",
block_len);
> > >     vmstate_register_ram_global(&ram_memory);
> > >
> > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > >         above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
> > >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
> > >     } else {
> > >         below_4g_mem_size = ram_size;
> > >     }
> > >     ...
> > >
> > > HVM_BELOW_4G_RAM_END is f0000000. If we change
HVM_BELOW_4G_RAM_END
> > to e0000000,
> > > Which it's consistent with hvmloader when assigning a GPU,
and then
> > guest worked
> > > for us. So we wondering that xen_ram_init in QEMU should be
> > consistent with
> > > hvmloader.
> > >
> > > In addition, we found QEMU uses hardcode 0xe0000000 in pc_init1()
> as
> > below.
> > > Should keep these places handle the consistent mmio hole or not?
> > >
> > >     if (ram_size >= 0xe0000000 ) {
> > >         above_4g_mem_size = ram_size - 0xe0000000;
> > >         below_4g_mem_size = 0xe0000000;
> > >     } else {
> > >         above_4g_mem_size = 0;
> > >         below_4g_mem_size = ram_size;
> > >     }
> >
> > The guys at Intel sent a couple of patches recently to fix this issue:
> >
> > http://marc.info/?l=xen-devel&m=136150317011027
> > http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> >
> > Do they solve your problem?
> 
> These two patches didn't solve our problem.
> 
I debugged this issue with above two patches. I want to share some information
and discuss solution here. This issue is actually caused by that a VM has a
large pci hole (mmio size) which results in QEMU sets memory regions
inconsistently with hvmloader (QEMU uses hardcode 0xe0000000 in pc_init1 and
xen_ram_init). I created a virtual device with 1GB mmio size to debug this
issue. Firstly, QEMU set memory regions except pci hole region in pc_init1() and
xen_ram_init(), then hvmloader calculated pci_mem_start as 0x80000000, and wrote
it to TOM register, which triggered QEMU to update pci hole region with
0x80000000 using i440fx_update_pci_mem_hole(). Finally the windows 7 VM
(configured 8G) crashed with BSOD code 0x00000024. If I hardcode in QEMU
pc_init1 and xen_ram_init to match hvmloader's. Then the problem was gone.

Althrough above two patches will pass actual pci hole start address to QEMU, but
it's too late, QEMU pc_init1() and xen_ram_init() already set the other
memory regions, and obviously the pci hole might overlap with ram regions in
this case. So I think hvmloader should setup pci devices and calculate pci hole
first, then QEMU can map memory regions correctly from the beginning.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

> > > > In addition, we found QEMU uses hardcode 0xe0000000 in
pc_init1()
> > as
> > > below.
> > > > Should keep these places handle the consistent mmio hole or
not?
> > > >
> > > >     if (ram_size >= 0xe0000000 ) {
> > > >         above_4g_mem_size = ram_size - 0xe0000000;
> > > >         below_4g_mem_size = 0xe0000000;
> > > >     } else {
> > > >         above_4g_mem_size = 0;
> > > >         below_4g_mem_size = ram_size;
> > > >     }
> > >
> > > The guys at Intel sent a couple of patches recently to fix this
issue:
> > >
> > > http://marc.info/?l=xen-devel&m=136150317011027
> > > http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> > >
> > > Do they solve your problem?
> >
> > These two patches didn''t solve our problem.
> >
> 
> I debugged this issue with above two patches. I want to share some
> information and discuss solution here. This issue is actually caused by
that a
> VM has a large pci hole (mmio size) which results in QEMU sets memory
> regions inconsistently with hvmloader (QEMU uses hardcode 0xe0000000 in
> pc_init1 and xen_ram_init). I created a virtual device with 1GB mmio size
to
> debug this issue. Firstly, QEMU set memory regions except pci hole region
in
> pc_init1() and xen_ram_init(), then hvmloader calculated pci_mem_start as
> 0x80000000, and wrote it to TOM register, which triggered QEMU to update
> pci hole region with 0x80000000 using i440fx_update_pci_mem_hole(). Finally
> the windows 7 VM (configured 8G) crashed with BSOD code 0x00000024. If I
> hardcode in QEMU pc_init1 and xen_ram_init to match hvmloader''s.
Then the
> problem was gone.
> 
> Althrough above two patches will pass actual pci hole start address to
QEMU,
> but it''s too late, QEMU pc_init1() and xen_ram_init() already set
the other
> memory regions, and obviously the pci hole might overlap with ram regions
in
> this case. 
How about update ram region as well when pci memory hole updating?
> So I think hvmloader should setup pci devices and calculate pci hole
> first, then QEMU can map memory regions correctly from the beginning.
> 
Every device may be different bar size, the start address of pci hole will be
different by hvmloader calculating result, so it still need a channel to report
this address to qemu.

Thanks,
-Xudong

On Thu, Apr 25, 2013 at 4:46 AM, Hanweidong <hanweidong@huawei.com>
wrote:
>
>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
>> bounces@lists.xen.org] On Behalf Of Hanweidong
>> Sent: 2013年3月26日 17:38
>> To: Stefano Stabellini
>> Cc: George Dunlap; xudong.hao@intel.com; Yanqiangjun; Luonengjun;
>> Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
>> devel@lists.xen.org; xiantao.zhang@intel.com
>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
>> with 4G memory
>>
>>
>> > -----Original Message-----
>> > From: Stefano Stabellini [mailto:stefano.stabellini@eu.citrix.com]
>> > Sent: 2013年3月18日 20:02
>> > To: Hanweidong
>> > Cc: George Dunlap; Stefano Stabellini; Yanqiangjun; Luonengjun;
>> > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
>> > devel@lists.xen.org; xudong.hao@intel.com; xiantao.zhang@intel.com
>> > Subject: RE: [Xen-devel] GPU passthrough issue when VM is
configured
>> > with 4G memory
>> >
>> > On Wed, 13 Mar 2013, Hanweidong wrote:
>> > > MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses
below
>> > code to init
>> > > RAM in xen_ram_init:
>> > >
>> > >     ...
>> > >     block_len = ram_size;
>> > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
>> > >         /* Xen does not allocate the memory continuously, and
keep
>> a
>> > hole at
>> > >          * HVM_BELOW_4G_MMIO_START of
HVM_BELOW_4G_MMIO_LENGTH
>> > >          */
>> > >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
>> > >     }
>> > >     memory_region_init_ram(&ram_memory,
"xen.ram", block_len);
>> > >     vmstate_register_ram_global(&ram_memory);
>> > >
>> > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
>> > >         above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
>> > >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
>> > >     } else {
>> > >         below_4g_mem_size = ram_size;
>> > >     }
>> > >     ...
>> > >
>> > > HVM_BELOW_4G_RAM_END is f0000000. If we change
HVM_BELOW_4G_RAM_END
>> > to e0000000,
>> > > Which it's consistent with hvmloader when assigning a
GPU, and then
>> > guest worked
>> > > for us. So we wondering that xen_ram_init in QEMU should be
>> > consistent with
>> > > hvmloader.
>> > >
>> > > In addition, we found QEMU uses hardcode 0xe0000000 in
pc_init1()
>> as
>> > below.
>> > > Should keep these places handle the consistent mmio hole or
not?
>> > >
>> > >     if (ram_size >= 0xe0000000 ) {
>> > >         above_4g_mem_size = ram_size - 0xe0000000;
>> > >         below_4g_mem_size = 0xe0000000;
>> > >     } else {
>> > >         above_4g_mem_size = 0;
>> > >         below_4g_mem_size = ram_size;
>> > >     }
>> >
>> > The guys at Intel sent a couple of patches recently to fix this
issue:
>> >
>> > http://marc.info/?l=xen-devel&m=136150317011027
>> > http://marc.info/?l=qemu-devel&m=136177475215360&w=2
>> >
>> > Do they solve your problem?
>>
>> These two patches didn't solve our problem.
>>
>
> I debugged this issue with above two patches. I want to share some
information and discuss solution here. This issue is actually caused by that a
VM has a large pci hole (mmio size) which results in QEMU sets memory regions
inconsistently with hvmloader (QEMU uses hardcode 0xe0000000 in pc_init1 and
xen_ram_init). I created a virtual device with 1GB mmio size to debug this
issue. Firstly, QEMU set memory regions except pci hole region in pc_init1() and
xen_ram_init(), then hvmloader calculated pci_mem_start as 0x80000000, and wrote
it to TOM register, which triggered QEMU to update pci hole region with
0x80000000 using i440fx_update_pci_mem_hole(). Finally the windows 7 VM
(configured 8G) crashed with BSOD code 0x00000024. If I hardcode in QEMU
pc_init1 and xen_ram_init to match hvmloader's. Then the problem was gone.
>
> Althrough above two patches will pass actual pci hole start address to
QEMU, but it's too late, QEMU pc_init1() and xen_ram_init() already set the
other memory regions, and obviously the pci hole might overlap with ram regions
in this case. So I think hvmloader should setup pci devices and calculate pci
hole first, then QEMU can map memory regions correctly from the beginning.
So just to check to make sure I understand correctly: the problem is
that qemu has mapped the guests' pages before the memory hole is made
larger?

In that case, wouldn't it make sense to flush qemu's mapcache whenever
the memory layout changes?  Such a thing should be possible now, e.g.,
when doing ballooning, right?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

> -----Original Message-----
> From: Hao, Xudong [mailto:xudong.hao@intel.com]
> Sent: 2013年4月25日 16:13
> To: Hanweidong; Stefano Stabellini
> Cc: George Dunlap; Yanqiangjun; Luonengjun; Wangzhenguo; Yangxiaowei;
> Gonglei (Arei); Anthony Perard; xen-devel@lists.xen.org; Zhang, Xiantao
> Subject: RE: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> > > > > In addition, we found QEMU uses hardcode 0xe0000000 in
> pc_init1()
> > > as
> > > > below.
> > > > > Should keep these places handle the consistent mmio
hole or not?
> > > > >
> > > > >     if (ram_size >= 0xe0000000 ) {
> > > > >         above_4g_mem_size = ram_size - 0xe0000000;
> > > > >         below_4g_mem_size = 0xe0000000;
> > > > >     } else {
> > > > >         above_4g_mem_size = 0;
> > > > >         below_4g_mem_size = ram_size;
> > > > >     }
> > > >
> > > > The guys at Intel sent a couple of patches recently to fix
this
> issue:
> > > >
> > > > http://marc.info/?l=xen-devel&m=136150317011027
> > > > http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> > > >
> > > > Do they solve your problem?
> > >
> > > These two patches didn't solve our problem.
> > >
> >
> > I debugged this issue with above two patches. I want to share some
> > information and discuss solution here. This issue is actually caused
> by that a
> > VM has a large pci hole (mmio size) which results in QEMU sets memory
> > regions inconsistently with hvmloader (QEMU uses hardcode 0xe0000000
> in
> > pc_init1 and xen_ram_init). I created a virtual device with 1GB mmio
> size to
> > debug this issue. Firstly, QEMU set memory regions except pci hole
> region in
> > pc_init1() and xen_ram_init(), then hvmloader calculated
> pci_mem_start as
> > 0x80000000, and wrote it to TOM register, which triggered QEMU to
> update
> > pci hole region with 0x80000000 using i440fx_update_pci_mem_hole().
> Finally
> > the windows 7 VM (configured 8G) crashed with BSOD code 0x00000024.
> If I
> > hardcode in QEMU pc_init1 and xen_ram_init to match hvmloader's.
Then
> the
> > problem was gone.
> >
> > Althrough above two patches will pass actual pci hole start address
> to QEMU,
> > but it's too late, QEMU pc_init1() and xen_ram_init() already set
the
> other
> > memory regions, and obviously the pci hole might overlap with ram
> regions in
> > this case.
> 
> How about update ram region as well when pci memory hole updating?
will have a try to see what will happen.
> 
> > So I think hvmloader should setup pci devices and calculate pci hole
> > first, then QEMU can map memory regions correctly from the beginning.
> >
> 
> Every device may be different bar size, the start address of pci hole
> will be different by hvmloader calculating result, so it still need a
> channel to report this address to qemu.
> 
Yes, it needs a channel to tell qemu. It's best if the channel can pass the
actual pci hole to QEMU before QEMU maps memory regions.

weidong
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

> -----Original Message-----
> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of George
> Dunlap
> Sent: 2013年4月25日 18:29
> To: Hanweidong
> Cc: Stefano Stabellini; xudong.hao@intel.com; Yanqiangjun; Luonengjun;
> Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> devel@lists.xen.org; xiantao.zhang@intel.com
> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> On Thu, Apr 25, 2013 at 4:46 AM, Hanweidong <hanweidong@huawei.com>
> wrote:
> >
> >
> >> -----Original Message-----
> >> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> >> bounces@lists.xen.org] On Behalf Of Hanweidong
> >> Sent: 2013年3月26日 17:38
> >> To: Stefano Stabellini
> >> Cc: George Dunlap; xudong.hao@intel.com; Yanqiangjun; Luonengjun;
> >> Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> >> devel@lists.xen.org; xiantao.zhang@intel.com
> >> Subject: Re: [Xen-devel] GPU passthrough issue when VM is
configured
> >> with 4G memory
> >>
> >>
> >> > -----Original Message-----
> >> > From: Stefano Stabellini
[mailto:stefano.stabellini@eu.citrix.com]
> >> > Sent: 2013年3月18日 20:02
> >> > To: Hanweidong
> >> > Cc: George Dunlap; Stefano Stabellini; Yanqiangjun;
Luonengjun;
> >> > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard;
xen-
> >> > devel@lists.xen.org; xudong.hao@intel.com;
xiantao.zhang@intel.com
> >> > Subject: RE: [Xen-devel] GPU passthrough issue when VM is
> configured
> >> > with 4G memory
> >> >
> >> > On Wed, 13 Mar 2013, Hanweidong wrote:
> >> > > MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU
uses
> below
> >> > code to init
> >> > > RAM in xen_ram_init:
> >> > >
> >> > >     ...
> >> > >     block_len = ram_size;
> >> > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> >> > >         /* Xen does not allocate the memory
continuously, and
> keep
> >> a
> >> > hole at
> >> > >          * HVM_BELOW_4G_MMIO_START of
HVM_BELOW_4G_MMIO_LENGTH
> >> > >          */
> >> > >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
> >> > >     }
> >> > >     memory_region_init_ram(&ram_memory,
"xen.ram", block_len);
> >> > >     vmstate_register_ram_global(&ram_memory);
> >> > >
> >> > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> >> > >         above_4g_mem_size = ram_size -
HVM_BELOW_4G_RAM_END;
> >> > >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
> >> > >     } else {
> >> > >         below_4g_mem_size = ram_size;
> >> > >     }
> >> > >     ...
> >> > >
> >> > > HVM_BELOW_4G_RAM_END is f0000000. If we change
> HVM_BELOW_4G_RAM_END
> >> > to e0000000,
> >> > > Which it's consistent with hvmloader when assigning
a GPU, and
> then
> >> > guest worked
> >> > > for us. So we wondering that xen_ram_init in QEMU should
be
> >> > consistent with
> >> > > hvmloader.
> >> > >
> >> > > In addition, we found QEMU uses hardcode 0xe0000000 in
pc_init1()
> >> as
> >> > below.
> >> > > Should keep these places handle the consistent mmio hole
or not?
> >> > >
> >> > >     if (ram_size >= 0xe0000000 ) {
> >> > >         above_4g_mem_size = ram_size - 0xe0000000;
> >> > >         below_4g_mem_size = 0xe0000000;
> >> > >     } else {
> >> > >         above_4g_mem_size = 0;
> >> > >         below_4g_mem_size = ram_size;
> >> > >     }
> >> >
> >> > The guys at Intel sent a couple of patches recently to fix
this
> issue:
> >> >
> >> > http://marc.info/?l=xen-devel&m=136150317011027
> >> > http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> >> >
> >> > Do they solve your problem?
> >>
> >> These two patches didn't solve our problem.
> >>
> >
> > I debugged this issue with above two patches. I want to share some
> information and discuss solution here. This issue is actually caused by
> that a VM has a large pci hole (mmio size) which results in QEMU sets
> memory regions inconsistently with hvmloader (QEMU uses hardcode
> 0xe0000000 in pc_init1 and xen_ram_init). I created a virtual device
> with 1GB mmio size to debug this issue. Firstly, QEMU set memory
> regions except pci hole region in pc_init1() and xen_ram_init(), then
> hvmloader calculated pci_mem_start as 0x80000000, and wrote it to TOM
> register, which triggered QEMU to update pci hole region with
> 0x80000000 using i440fx_update_pci_mem_hole(). Finally the windows 7 VM
> (configured 8G) crashed with BSOD code 0x00000024. If I hardcode in
> QEMU pc_init1 and xen_ram_init to match hvmloader's. Then the problem
> was gone.
> >
> > Althrough above two patches will pass actual pci hole start address
> to QEMU, but it's too late, QEMU pc_init1() and xen_ram_init() already
> set the other memory regions, and obviously the pci hole might overlap
> with ram regions in this case. So I think hvmloader should setup pci
> devices and calculate pci hole first, then QEMU can map memory regions
> correctly from the beginning.
> 
> So just to check to make sure I understand correctly: the problem is
> that qemu has mapped the guests' pages before the memory hole is made
> larger?
> 
Yes. 
> In that case, wouldn't it make sense to flush qemu's mapcache
whenever
> the memory layout changes?  Such a thing should be possible now, e.g.,
> when doing ballooning, right?
> 
Good point. Will have a try.

weidong


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Thu, Apr 25, 2013 at 02:24:11PM +0000, Hanweidong
wrote:
> > 
> > So just to check to make sure I understand correctly: the problem is
> > that qemu has mapped the guests'' pages before the memory hole
is made
> > larger?
> > 
> 
> Yes. 
> 
> > In that case, wouldn''t it make sense to flush qemu''s
mapcache whenever
> > the memory layout changes?  Such a thing should be possible now, e.g.,
> > when doing ballooning, right?
> > 
> 
> Good point. Will have a try.
> 
Any luck with debugging this issue? 

I''m setting up a new box for testing GPU passthrough so I''m
happy to test
any patches you might have..

Thanks,

-- Pasi

> -----Original Message-----
> From: Pasi Kärkkäinen [mailto:pasik@iki.fi]
> Sent: 2013年5月10日 0:50
> To: Hanweidong
> Cc: George Dunlap; Stefano Stabellini; xudong.hao@intel.com;
> Yanqiangjun; Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei (Arei);
> Anthony Perard; xen-devel@lists.xen.org; xiantao.zhang@intel.com
> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> On Thu, Apr 25, 2013 at 02:24:11PM +0000, Hanweidong wrote:
> > >
> > > So just to check to make sure I understand correctly: the problem
> is
> > > that qemu has mapped the guests' pages before the memory hole
is
> made
> > > larger?
> > >
> >
> > Yes.
> >
> > > In that case, wouldn't it make sense to flush qemu's
mapcache
> whenever
> > > the memory layout changes?  Such a thing should be possible now,
> e.g.,
> > > when doing ballooning, right?
> > >
> >
> > Good point. Will have a try.
> >
> 
> Any luck with debugging this issue?
Flushing qemu's mapcache doesn't help this issue. Unlike ballooning,
this case needs to adjust memory region layout. Seems it also needs to update
ram regions besides pci hole region, don't know it's feasible to update
ram regions (initialized in xen_ram_init()) when getting actual pci hole start
address from hvmloader.
> 
> I'm setting up a new box for testing GPU passthrough so I'm happy
to
> test
> any patches you might have..
> 
Actually, it's not really related to GPU passthrough. You can reproduce it
when a VM has a big pci hole size (such as 512MB), e.g. create a VM with a
virtual device which has a 512MB pci BAR.

Weidong
 
> Thanks,
> 
> -- Pasi
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

I think it is this problem that has been driving me nuts for the
 last 2 weeks (big thread on xen-users) while I've been trying to get 
 VGA passthrough to work
 on my system. The difference is that on my system the domU suffers
 a GPU related crash when more than 2GB of RAM are assigned to domU.

 Of course, when you see mentions of this issue occurring
with
> 4GB) RAM, seeing the problem with <=4GB of RAM in domU doesn't
 immediately connect as being the same issue, which can lead to
 somebody like me chasing their tail for days with no one
 configuration change making any consistent difference. Of course,
 being a memory stomp by nature, the result are rarely 100%
 predictable and consistently reproducible.

 Is there a way to explicitly configure the guest's memory hole
 to cover everything between, say, 2GB and 4GB? It's not a
 replacement to getting the BARs right, but it might be easier
 easier as a quick workaround for people who need a workaround now.
 Would it work if the memory map was equivalent of something
 like:

 mem=2GB@0 mem=2GB@4GB?

 That should work provided all the BARs are somewhere in the
 2GB-4GB region. Granted, that may not always be the case, but
 if it covers 90% of cases it would be a good workaround until
 a proper fix is available.

 Gordan

 On Fri, 17 May 2013 07:10:17 +0000, Hanweidong <hanweidong@huawei.com> 
 wrote:
>> -----Original Message-----
>> From: Pasi Kärkkäinen [mailto:pasik@iki.fi]
>> Sent: 2013年5月10日 0:50
>> To: Hanweidong
>> Cc: George Dunlap; Stefano Stabellini; xudong.hao@intel.com;
>> Yanqiangjun; Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei (Arei);
>> Anthony Perard; xen-devel@lists.xen.org; xiantao.zhang@intel.com
>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
>> with 4G memory
>>
>> On Thu, Apr 25, 2013 at 02:24:11PM +0000, Hanweidong wrote:
>> > >
>> > > So just to check to make sure I understand correctly: the 
>> problem
>> is
>> > > that qemu has mapped the guests' pages before the memory
hole is
>> made
>> > > larger?
>> > >
>> >
>> > Yes.
>> >
>> > > In that case, wouldn't it make sense to flush qemu's
mapcache
>> whenever
>> > > the memory layout changes?  Such a thing should be possible
now,
>> e.g.,
>> > > when doing ballooning, right?
>> > >
>> >
>> > Good point. Will have a try.
>> >
>>
>> Any luck with debugging this issue?
>
> Flushing qemu's mapcache doesn't help this issue. Unlike
ballooning,
> this case needs to adjust memory region layout. Seems it also needs 
> to
> update ram regions besides pci hole region, don't know it's
feasible
> to update ram regions (initialized in xen_ram_init()) when getting
> actual pci hole start address from hvmloader.
>
>>
>> I'm setting up a new box for testing GPU passthrough so I'm
happy to
>> test
>> any patches you might have..
>>
>
> Actually, it's not really related to GPU passthrough. You can
> reproduce it when a VM has a big pci hole size (such as 512MB), e.g.
> create a VM with a virtual device which has a 512MB pci BAR.
>
> Weidong
>
>> Thanks,
>>
>> -- Pasi
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Fri, May 17, 2013 at 07:10:17AM +0000, Hanweidong
wrote:
> 
> > -----Original Message-----
> > From: Pasi Kärkkäinen [mailto:pasik@iki.fi]
> > Sent: 2013???5???10??? 0:50
> > To: Hanweidong
> > Cc: George Dunlap; Stefano Stabellini; xudong.hao@intel.com;
> > Yanqiangjun; Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei (Arei);
> > Anthony Perard; xen-devel@lists.xen.org; xiantao.zhang@intel.com
> > Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> > with 4G memory
> > 
> > On Thu, Apr 25, 2013 at 02:24:11PM +0000, Hanweidong wrote:
> > > >
> > > > So just to check to make sure I understand correctly: the
problem
> > is
> > > > that qemu has mapped the guests'' pages before the
memory hole is
> > made
> > > > larger?
> > > >
> > >
> > > Yes.
> > >
> > > > In that case, wouldn''t it make sense to flush
qemu''s mapcache
> > whenever
> > > > the memory layout changes?  Such a thing should be possible
now,
> > e.g.,
> > > > when doing ballooning, right?
> > > >
> > >
> > > Good point. Will have a try.
> > >
> > 
> > Any luck with debugging this issue?
> 
> Flushing qemu''s mapcache doesn''t help this issue. Unlike
ballooning, this case needs to adjust memory region layout. Seems it also needs
to update ram regions besides pci hole region, don''t know it''s
feasible to update ram regions (initialized in xen_ram_init()) when getting
actual pci hole start address from hvmloader.
> 
Ok. 
> > 
> > I''m setting up a new box for testing GPU passthrough so
I''m happy to
> > test
> > any patches you might have..
> > 
> 
> Actually, it''s not really related to GPU passthrough. You can
reproduce it when a VM has a big pci hole size (such as 512MB), e.g. create a VM
with a virtual device which has a 512MB pci BAR.
> 
George: Should we track this issue in the 4.3 status emails? It seems
it''s a generic PCI passthrough issue.


-- Pasi
> Weidong
>  
> > Thanks,
> > 
> > -- Pasi
>

On 17/05/13 08:10, Hanweidong wrote:
> Actually, it''s not really related to GPU passthrough. You can
reproduce it when a VM has a big pci hole size (such as 512MB), e.g. create a VM
with a virtual device which has a 512MB pci BAR.
It seems like this is probably an artchitectural thing that will require 
someone familiar with qemu to sort out.  Anthony and Stefano, could one 
of you two step up to getting it sorted by 4.3?

  -George

On Mon, 20 May 2013, George Dunlap wrote:
> On 17/05/13 08:10, Hanweidong wrote:
> > Actually, it''s not really related to GPU passthrough. You can
reproduce it
> > when a VM has a big pci hole size (such as 512MB), e.g. create a VM
with a
> > virtual device which has a 512MB pci BAR.
> 
> It seems like this is probably an artchitectural thing that will require
> someone familiar with qemu to sort out.  Anthony and Stefano, could one of
you
> two step up to getting it sorted by 4.3?
I am not likely to be able to work on this in the near future

I''d also like to stress this is not only an issue for > 4GB of RAM
in
domU - I am seeing the issue with > 2GB of RAM in domU.

Gordan

On 05/20/2013 12:29 PM, George Dunlap wrote:
> On 17/05/13 08:10, Hanweidong wrote:
>> Actually, it''s not really related to GPU passthrough. You can
>> reproduce it when a VM has a big pci hole size (such as 512MB), e.g.
>> create a VM with a virtual device which has a 512MB pci BAR.
>
> It seems like this is probably an artchitectural thing that will require
> someone familiar with qemu to sort out.  Anthony and Stefano, could one
> of you two step up to getting it sorted by 4.3?
>
>   -George
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

> -----Original Message-----
> From: Gordan Bobic [mailto:gordan@bobich.net]
> Sent: 2013年5月21日 2:43
> To: George Dunlap
> Cc: Hanweidong; Stefano Stabellini; xudong.hao@intel.com; Yanqiangjun;
> Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard;
> xen-devel@lists.xen.org; xiantao.zhang@intel.com
> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> I'd also like to stress this is not only an issue for > 4GB of RAM
in
> domU - I am seeing the issue with > 2GB of RAM in domU.
What's the total mmio size of you domU? When RAM of domU overlaps with
pci_mem_start, hvmloader will relocate RAM, and it will cause problem due to
hvmloader and QEMU don't setup the memory layout consistently. I suspect the
mmio size of your domU is close to 2GB. When you configured RAM > 2G, then
RAM of your domU overlapped with pci_mem_start, and resulted in failure.

weidong
> 
> Gordan
> 
> On 05/20/2013 12:29 PM, George Dunlap wrote:
> > On 17/05/13 08:10, Hanweidong wrote:
> >> Actually, it's not really related to GPU passthrough. You can
> >> reproduce it when a VM has a big pci hole size (such as 512MB),
e.g.
> >> create a VM with a virtual device which has a 512MB pci BAR.
> >
> > It seems like this is probably an artchitectural thing that will
> require
> > someone familiar with qemu to sort out.  Anthony and Stefano, could
> one
> > of you two step up to getting it sorted by 4.3?
> >
> >   -George
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Tue, 21 May 2013 04:09:25 +0000, Hanweidong <hanweidong@huawei.com> 
 wrote:
>> -----Original Message-----
>> From: Gordan Bobic [mailto:gordan@bobich.net]
>> Sent: 2013年5月21日 2:43
>> To: George Dunlap
>> Cc: Hanweidong; Stefano Stabellini; xudong.hao@intel.com; 
>> Yanqiangjun;
>> Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony 
>> Perard;
>> xen-devel@lists.xen.org; xiantao.zhang@intel.com
>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
>> with 4G memory
>>
>> I'd also like to stress this is not only an issue for > 4GB of
RAM
>> in
>> domU - I am seeing the issue with > 2GB of RAM in domU.
>
> What's the total mmio size of you domU?
 How can I find that out?
> When RAM of domU overlaps
> with pci_mem_start, hvmloader will relocate RAM, and it will cause
> problem due to hvmloader and QEMU don't setup the memory layout
> consistently. I suspect the mmio size of your domU is close to 2GB.
> When you configured RAM > 2G, then RAM of your domU overlapped with
> pci_mem_start, and resulted in failure.
 Is there a way to force a hole between, say, 2GB and 4GB explicitly
 to avoid the PCI memory being clobbered?

 Gordan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Wed, May 22, 2013 at 11:17 AM, Gordan Bobic <gordan@bobich.net> wrote:
> On Tue, 21 May 2013 04:09:25 +0000, Hanweidong
<hanweidong@huawei.com>
> wrote:
>
>> -----Original Message-----
>>> From: Gordan Bobic [mailto:gordan@bobich.net]
>>> Sent: 2013年5月21日 2:43
>>> To: George Dunlap
>>> Cc: Hanweidong; Stefano Stabellini; xudong.hao@intel.com;
Yanqiangjun;
>>> Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony
Perard;
>>> xen-devel@lists.xen.org; xiantao.zhang@intel.com
>>> Subject: Re: [Xen-devel] GPU passthrough issue when VM is
configured
>>> with 4G memory
>>>
>>> I''d also like to stress this is not only an issue for >
4GB of RAM in
>>> domU - I am seeing the issue with > 2GB of RAM in domU.
>>>
>>
>> What''s the total mmio size of you domU?
>>
>
> How can I find that out?
>
>
>  When RAM of domU overlaps
>> with pci_mem_start, hvmloader will relocate RAM, and it will cause
>> problem due to hvmloader and QEMU don''t setup the memory
layout
>> consistently. I suspect the mmio size of your domU is close to 2GB.
>> When you configured RAM > 2G, then RAM of your domU overlapped with
>> pci_mem_start, and resulted in failure.
>>
>
> Is there a way to force a hole between, say, 2GB and 4GB explicitly
> to avoid the PCI memory being clobbered?
>
> Gordan

I''ll second that question!

Gordan, one thing I didn''t think of until now: the card I was having
trouble with, a Radeon 6990, is a 4GB card, laid out as two 2GB GPUs.  The
card with which I don''t have issues (aside from the occasional need to
reboot the whole system to get it to work right), a Radeon 5850, is a 1GB
card.  If as this thread has suggested, along with what I''ve read about
the
PCI Hole, the memory on the GPU determines the "clobber size" (yes I
made
that term up :P), when I get back home (I''m on vacation) I''d
love to try
my luck at PCI Hole remapping.

-Andrew


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 05/22/2013 07:11 PM, Andrew Bobulsky wrote:
> On Wed, May 22, 2013 at 11:17 AM, Gordan Bobic <gordan@bobich.net
> <mailto:gordan@bobich.net>> wrote:
>
>     On Tue, 21 May 2013 04:09:25 +0000, Hanweidong
>     <hanweidong@huawei.com <mailto:hanweidong@huawei.com>>
wrote:
>
>             -----Original Message-----
>             From: Gordan Bobic [mailto:gordan@bobich.net
>             <mailto:gordan@bobich.net>]
>             Sent: 2013年5月21日 2:43
>             To: George Dunlap
>             Cc: Hanweidong; Stefano Stabellini; xudong.hao@intel.com
>             <mailto:xudong.hao@intel.com>; Yanqiangjun;
>             Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei (Arei);
>             Anthony Perard;
>             xen-devel@lists.xen.org <mailto:xen-devel@lists.xen.org>;
>             xiantao.zhang@intel.com <mailto:xiantao.zhang@intel.com>
>             Subject: Re: [Xen-devel] GPU passthrough issue when VM is
>             configured
>             with 4G memory
>
>             I'd also like to stress this is not only an issue for >
4GB
>             of RAM in
>             domU - I am seeing the issue with > 2GB of RAM in domU.
>
>
>         What's the total mmio size of you domU?
>
>
>     How can I find that out?
>
>
>         When RAM of domU overlaps
>         with pci_mem_start, hvmloader will relocate RAM, and it will cause
>         problem due to hvmloader and QEMU don't setup the memory layout
>         consistently. I suspect the mmio size of your domU is close to 2GB.
>         When you configured RAM > 2G, then RAM of your domU overlapped
with
>         pci_mem_start, and resulted in failure.
>
>
>     Is there a way to force a hole between, say, 2GB and 4GB explicitly
>     to avoid the PCI memory being clobbered?
>
> I'll second that question!
> Gordan, one thing I didn't think of until now: the card I was having
> trouble with, a Radeon 6990, is a 4GB card, laid out as two 2GB GPUs.
> The card with which I don't have issues (aside from the occasional need
> to reboot the whole system to get it to work right), a Radeon 5850, is a
> 1GB card.  If as this thread has suggested, along with what I've read
> about the PCI Hole, the memory on the GPU determines the "clobber
size"
> (yes I made that term up :P),
I'm pretty sure the GPU RAM size is not relevant. I've seen this with 
the 1GB Nvidia Quadro 2000, 1GB ATI 7450 and the 6GB 7970. The GPU 
memory size does not affect the aperture size or location. I for one 
would love it if all of the GPU RAM was in fact mapped as a BAR - that 
would open up some very funky approaches to processing large amounts of 
data, e.g. by just mapping the data straight to the GPU. It would also 
make it trivially easy to use all that RAM as a RAM disk when the GPU 
isn't being used (e.g. you can use the RAM for fast swap for normal 
desktop tasks, and free it up when gaming). but that's a whole different 
topic.

Gordan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

> -----Original Message-----
> From: Gordan Bobic [mailto:gordan@bobich.net]
> Sent: 2013年5月22日 23:18
> To: Hanweidong
> Cc: George Dunlap; Stefano Stabellini; xudong.hao@intel.com;
> Yanqiangjun; Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei (Arei);
> Anthony Perard; xen-devel@lists.xen.org; xiantao.zhang@intel.com
> Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
>  On Tue, 21 May 2013 04:09:25 +0000, Hanweidong
<hanweidong@huawei.com>
>  wrote:
> >> -----Original Message-----
> >> From: Gordan Bobic [mailto:gordan@bobich.net]
> >> Sent: 2013年5月21日 2:43
> >> To: George Dunlap
> >> Cc: Hanweidong; Stefano Stabellini; xudong.hao@intel.com;
> >> Yanqiangjun;
> >> Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony
> >> Perard;
> >> xen-devel@lists.xen.org; xiantao.zhang@intel.com
> >> Subject: Re: [Xen-devel] GPU passthrough issue when VM is
configured
> >> with 4G memory
> >>
> >> I'd also like to stress this is not only an issue for > 4GB
of RAM
> >> in
> >> domU - I am seeing the issue with > 2GB of RAM in domU.
> >
> > What's the total mmio size of you domU?
> 
>  How can I find that out?
You can print mmio_total in pci_setup() in tools/firmware/hvmloader/pci.c.
> 
> > When RAM of domU overlaps
> > with pci_mem_start, hvmloader will relocate RAM, and it will cause
> > problem due to hvmloader and QEMU don't setup the memory layout
> > consistently. I suspect the mmio size of your domU is close to 2GB.
> > When you configured RAM > 2G, then RAM of your domU overlapped with
> > pci_mem_start, and resulted in failure.
> 
>  Is there a way to force a hole between, say, 2GB and 4GB explicitly
>  to avoid the PCI memory being clobbered?
No graceful way to do it, but you can hardcode it by modifying hvmloader and
qemu code.

weidong
> 
>  Gordan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Thu, 25 Apr 2013, Hanweidong wrote:
> > -----Original Message-----
> > From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> > bounces@lists.xen.org] On Behalf Of Hanweidong
> > Sent: 2013年3月26日 17:38
> > To: Stefano Stabellini
> > Cc: George Dunlap; xudong.hao@intel.com; Yanqiangjun; Luonengjun;
> > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> > devel@lists.xen.org; xiantao.zhang@intel.com
> > Subject: Re: [Xen-devel] GPU passthrough issue when VM is configured
> > with 4G memory
> > 
> > 
> > > -----Original Message-----
> > > From: Stefano Stabellini
[mailto:stefano.stabellini@eu.citrix.com]
> > > Sent: 2013年3月18日 20:02
> > > To: Hanweidong
> > > Cc: George Dunlap; Stefano Stabellini; Yanqiangjun; Luonengjun;
> > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> > > devel@lists.xen.org; xudong.hao@intel.com;
xiantao.zhang@intel.com
> > > Subject: RE: [Xen-devel] GPU passthrough issue when VM is
configured
> > > with 4G memory
> > >
> > > On Wed, 13 Mar 2013, Hanweidong wrote:
> > > > MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU uses
below
> > > code to init
> > > > RAM in xen_ram_init:
> > > >
> > > >     ...
> > > >     block_len = ram_size;
> > > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > > >         /* Xen does not allocate the memory continuously,
and keep
> > a
> > > hole at
> > > >          * HVM_BELOW_4G_MMIO_START of
HVM_BELOW_4G_MMIO_LENGTH
> > > >          */
> > > >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
> > > >     }
> > > >     memory_region_init_ram(&ram_memory,
"xen.ram", block_len);
> > > >     vmstate_register_ram_global(&ram_memory);
> > > >
> > > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > > >         above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
> > > >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
> > > >     } else {
> > > >         below_4g_mem_size = ram_size;
> > > >     }
> > > >     ...
> > > >
> > > > HVM_BELOW_4G_RAM_END is f0000000. If we change
HVM_BELOW_4G_RAM_END
> > > to e0000000,
> > > > Which it''s consistent with hvmloader when assigning
a GPU, and then
> > > guest worked
> > > > for us. So we wondering that xen_ram_init in QEMU should be
> > > consistent with
> > > > hvmloader.
> > > >
> > > > In addition, we found QEMU uses hardcode 0xe0000000 in
pc_init1()
> > as
> > > below.
> > > > Should keep these places handle the consistent mmio hole or
not?
> > > >
> > > >     if (ram_size >= 0xe0000000 ) {
> > > >         above_4g_mem_size = ram_size - 0xe0000000;
> > > >         below_4g_mem_size = 0xe0000000;
> > > >     } else {
> > > >         above_4g_mem_size = 0;
> > > >         below_4g_mem_size = ram_size;
> > > >     }
> > >
> > > The guys at Intel sent a couple of patches recently to fix this
issue:
> > >
> > > http://marc.info/?l=xen-devel&m=136150317011027
> > > http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> > >
> > > Do they solve your problem?
> > 
> > These two patches didn''t solve our problem.
> > 
> 
> I debugged this issue with above two patches. I want to share some
information and discuss solution here. This issue is actually caused by that a
VM has a large pci hole (mmio size) which results in QEMU sets memory regions
inconsistently with hvmloader (QEMU uses hardcode 0xe0000000 in pc_init1 and
xen_ram_init). I created a virtual device with 1GB mmio size to debug this
issue. Firstly, QEMU set memory regions except pci hole region in pc_init1() and
xen_ram_init(), then hvmloader calculated pci_mem_start as 0x80000000, and wrote
it to TOM register, which triggered QEMU to update pci hole region with
0x80000000 using i440fx_update_pci_mem_hole(). Finally the windows 7 VM
(configured 8G) crashed with BSOD code 0x00000024. If I hardcode in QEMU
pc_init1 and xen_ram_init to match hvmloader''s. Then the problem was
gone.
> 
> Althrough above two patches will pass actual pci hole start address to
QEMU, but it''s too late, QEMU pc_init1() and xen_ram_init() already set
the other memory regions, and obviously the pci hole might overlap with ram
regions in this case. So I think hvmloader should setup pci devices and
calculate pci hole first, then QEMU can map memory regions correctly from the
beginning.
> 
Thank you very much for your detailed analysis of the problem.

After reading this, I wonder how is possible that qemu-xen-traditional
does not have this issue, considering that AFAIK there is no way for
hvmloader to tell qemu-xen-traditional where the PCI hole starts.

The only difference between upstream QEMU and qemu-xen-traditional is
that the former would start the PCI hole at 0xf0000000 while the latter
would start the PCI hole at 0xe0000000.

So I would expect that your test, where hvmloader is updating the PCI
hole region to start at 0x80000000, would fail on qemu-xen-traditional
too.

Of course having the PCI hole starting unconditionally at 0xf0000000
makes it much easier to run into problems than starting it at
0xe0000000.


Assuming that everything above is correct, this is what I would do:

1) modify upstream QEMU to start the PCI hole at 0xe0000000, to match
qemu-xen-unstable in terms of configuration and not to introduce any
regressions. Do this for the Xen 4.3 release.

2) for Xen 4.4 rework the two patches above and improve
i440fx_update_pci_mem_hole: resizing the pci_hole subregion is not
enough, it also needs to be able to resize the system memory region
(xen.ram) to make room for the bigger pci_hole
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

> -----Original Message-----
> From: Stefano Stabellini [mailto:stefano.stabellini@eu.citrix.com]
> Sent: 2013年5月30日 0:18
> To: Hanweidong
> Cc: Stefano Stabellini; George Dunlap; xudong.hao@intel.com;
> Yanqiangjun; Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei (Arei);
> Anthony Perard; xen-devel@lists.xen.org; xiantao.zhang@intel.com
> Subject: RE: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memory
> 
> On Thu, 25 Apr 2013, Hanweidong wrote:
> > > -----Original Message-----
> > > From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> > > bounces@lists.xen.org] On Behalf Of Hanweidong
> > > Sent: 2013年3月26日 17:38
> > > To: Stefano Stabellini
> > > Cc: George Dunlap; xudong.hao@intel.com; Yanqiangjun; Luonengjun;
> > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> > > devel@lists.xen.org; xiantao.zhang@intel.com
> > > Subject: Re: [Xen-devel] GPU passthrough issue when VM is
> configured
> > > with 4G memory
> > >
> > >
> > > > -----Original Message-----
> > > > From: Stefano Stabellini
[mailto:stefano.stabellini@eu.citrix.com]
> > > > Sent: 2013年3月18日 20:02
> > > > To: Hanweidong
> > > > Cc: George Dunlap; Stefano Stabellini; Yanqiangjun;
Luonengjun;
> > > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard;
xen-
> > > > devel@lists.xen.org; xudong.hao@intel.com;
> xiantao.zhang@intel.com
> > > > Subject: RE: [Xen-devel] GPU passthrough issue when VM is
> configured
> > > > with 4G memory
> > > >
> > > > On Wed, 13 Mar 2013, Hanweidong wrote:
> > > > > MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU
uses
> below
> > > > code to init
> > > > > RAM in xen_ram_init:
> > > > >
> > > > >     ...
> > > > >     block_len = ram_size;
> > > > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > > > >         /* Xen does not allocate the memory
continuously, and
> keep
> > > a
> > > > hole at
> > > > >          * HVM_BELOW_4G_MMIO_START of
HVM_BELOW_4G_MMIO_LENGTH
> > > > >          */
> > > > >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
> > > > >     }
> > > > >     memory_region_init_ram(&ram_memory,
"xen.ram", block_len);
> > > > >     vmstate_register_ram_global(&ram_memory);
> > > > >
> > > > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > > > >         above_4g_mem_size = ram_size -
HVM_BELOW_4G_RAM_END;
> > > > >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
> > > > >     } else {
> > > > >         below_4g_mem_size = ram_size;
> > > > >     }
> > > > >     ...
> > > > >
> > > > > HVM_BELOW_4G_RAM_END is f0000000. If we change
> HVM_BELOW_4G_RAM_END
> > > > to e0000000,
> > > > > Which it's consistent with hvmloader when assigning
a GPU, and
> then
> > > > guest worked
> > > > > for us. So we wondering that xen_ram_init in QEMU
should be
> > > > consistent with
> > > > > hvmloader.
> > > > >
> > > > > In addition, we found QEMU uses hardcode 0xe0000000 in
> pc_init1()
> > > as
> > > > below.
> > > > > Should keep these places handle the consistent mmio
hole or not?
> > > > >
> > > > >     if (ram_size >= 0xe0000000 ) {
> > > > >         above_4g_mem_size = ram_size - 0xe0000000;
> > > > >         below_4g_mem_size = 0xe0000000;
> > > > >     } else {
> > > > >         above_4g_mem_size = 0;
> > > > >         below_4g_mem_size = ram_size;
> > > > >     }
> > > >
> > > > The guys at Intel sent a couple of patches recently to fix
this
> issue:
> > > >
> > > > http://marc.info/?l=xen-devel&m=136150317011027
> > > > http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> > > >
> > > > Do they solve your problem?
> > >
> > > These two patches didn't solve our problem.
> > >
> >
> > I debugged this issue with above two patches. I want to share some
> information and discuss solution here. This issue is actually caused by
> that a VM has a large pci hole (mmio size) which results in QEMU sets
> memory regions inconsistently with hvmloader (QEMU uses hardcode
> 0xe0000000 in pc_init1 and xen_ram_init). I created a virtual device
> with 1GB mmio size to debug this issue. Firstly, QEMU set memory
> regions except pci hole region in pc_init1() and xen_ram_init(), then
> hvmloader calculated pci_mem_start as 0x80000000, and wrote it to TOM
> register, which triggered QEMU to update pci hole region with
> 0x80000000 using i440fx_update_pci_mem_hole(). Finally the windows 7 VM
> (configured 8G) crashed with BSOD code 0x00000024. If I hardcode in
> QEMU pc_init1 and xen_ram_init to match hvmloader's. Then the problem
> was gone.
> >
> > Althrough above two patches will pass actual pci hole start address
> to QEMU, but it's too late, QEMU pc_init1() and xen_ram_init() already
> set the other memory regions, and obviously the pci hole might overlap
> with ram regions in this case. So I think hvmloader should setup pci
> devices and calculate pci hole first, then QEMU can map memory regions
> correctly from the beginning.
> >
> 
> Thank you very much for your detailed analysis of the problem.
> 
> After reading this, I wonder how is possible that qemu-xen-traditional
> does not have this issue, considering that AFAIK there is no way for
> hvmloader to tell qemu-xen-traditional where the PCI hole starts.
> 
> The only difference between upstream QEMU and qemu-xen-traditional is
> that the former would start the PCI hole at 0xf0000000 while the latter
> would start the PCI hole at 0xe0000000.
> 
> So I would expect that your test, where hvmloader is updating the PCI
> hole region to start at 0x80000000, would fail on qemu-xen-traditional
> too.
Yes, I think so. 
> 
> Of course having the PCI hole starting unconditionally at 0xf0000000
> makes it much easier to run into problems than starting it at
> 0xe0000000.
> 
> 
> Assuming that everything above is correct, this is what I would do:
> 
> 1) modify upstream QEMU to start the PCI hole at 0xe0000000, to match
> qemu-xen-unstable in terms of configuration and not to introduce any
> regressions. Do this for the Xen 4.3 release.
It's a quick improvement before implementing a thorough solution.

weidong
> 
> 2) for Xen 4.4 rework the two patches above and improve
> i440fx_update_pci_mem_hole: resizing the pci_hole subregion is not
> enough, it also needs to be able to resize the system memory region
> (xen.ram) to make room for the bigger pci_hole

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Thu, 30 May 2013, Hanweidong wrote:
> > -----Original Message-----
> > From: Stefano Stabellini [mailto:stefano.stabellini@eu.citrix.com]
> > Sent: 2013年5月30日 0:18
> > To: Hanweidong
> > Cc: Stefano Stabellini; George Dunlap; xudong.hao@intel.com;
> > Yanqiangjun; Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei (Arei);
> > Anthony Perard; xen-devel@lists.xen.org; xiantao.zhang@intel.com
> > Subject: RE: [Xen-devel] GPU passthrough issue when VM is configured
> > with 4G memory
> > 
> > On Thu, 25 Apr 2013, Hanweidong wrote:
> > > > -----Original Message-----
> > > > From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> > > > bounces@lists.xen.org] On Behalf Of Hanweidong
> > > > Sent: 2013年3月26日 17:38
> > > > To: Stefano Stabellini
> > > > Cc: George Dunlap; xudong.hao@intel.com; Yanqiangjun;
Luonengjun;
> > > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard;
xen-
> > > > devel@lists.xen.org; xiantao.zhang@intel.com
> > > > Subject: Re: [Xen-devel] GPU passthrough issue when VM is
> > configured
> > > > with 4G memory
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Stefano Stabellini
[mailto:stefano.stabellini@eu.citrix.com]
> > > > > Sent: 2013年3月18日 20:02
> > > > > To: Hanweidong
> > > > > Cc: George Dunlap; Stefano Stabellini; Yanqiangjun;
Luonengjun;
> > > > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony
Perard; xen-
> > > > > devel@lists.xen.org; xudong.hao@intel.com;
> > xiantao.zhang@intel.com
> > > > > Subject: RE: [Xen-devel] GPU passthrough issue when VM
is
> > configured
> > > > > with 4G memory
> > > > >
> > > > > On Wed, 13 Mar 2013, Hanweidong wrote:
> > > > > > MMIO HOLE was adjusted to e0000000 - fc000000. But
QEMU uses
> > below
> > > > > code to init
> > > > > > RAM in xen_ram_init:
> > > > > >
> > > > > >     ...
> > > > > >     block_len = ram_size;
> > > > > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > > > > >         /* Xen does not allocate the memory
continuously, and
> > keep
> > > > a
> > > > > hole at
> > > > > >          * HVM_BELOW_4G_MMIO_START of
HVM_BELOW_4G_MMIO_LENGTH
> > > > > >          */
> > > > > >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
> > > > > >     }
> > > > > >     memory_region_init_ram(&ram_memory,
"xen.ram", block_len);
> > > > > >     vmstate_register_ram_global(&ram_memory);
> > > > > >
> > > > > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > > > > >         above_4g_mem_size = ram_size -
HVM_BELOW_4G_RAM_END;
> > > > > >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
> > > > > >     } else {
> > > > > >         below_4g_mem_size = ram_size;
> > > > > >     }
> > > > > >     ...
> > > > > >
> > > > > > HVM_BELOW_4G_RAM_END is f0000000. If we change
> > HVM_BELOW_4G_RAM_END
> > > > > to e0000000,
> > > > > > Which it''s consistent with hvmloader when
assigning a GPU, and
> > then
> > > > > guest worked
> > > > > > for us. So we wondering that xen_ram_init in QEMU
should be
> > > > > consistent with
> > > > > > hvmloader.
> > > > > >
> > > > > > In addition, we found QEMU uses hardcode
0xe0000000 in
> > pc_init1()
> > > > as
> > > > > below.
> > > > > > Should keep these places handle the consistent
mmio hole or not?
> > > > > >
> > > > > >     if (ram_size >= 0xe0000000 ) {
> > > > > >         above_4g_mem_size = ram_size - 0xe0000000;
> > > > > >         below_4g_mem_size = 0xe0000000;
> > > > > >     } else {
> > > > > >         above_4g_mem_size = 0;
> > > > > >         below_4g_mem_size = ram_size;
> > > > > >     }
> > > > >
> > > > > The guys at Intel sent a couple of patches recently to
fix this
> > issue:
> > > > >
> > > > > http://marc.info/?l=xen-devel&m=136150317011027
> > > > >
http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> > > > >
> > > > > Do they solve your problem?
> > > >
> > > > These two patches didn''t solve our problem.
> > > >
> > >
> > > I debugged this issue with above two patches. I want to share
some
> > information and discuss solution here. This issue is actually caused
by
> > that a VM has a large pci hole (mmio size) which results in QEMU sets
> > memory regions inconsistently with hvmloader (QEMU uses hardcode
> > 0xe0000000 in pc_init1 and xen_ram_init). I created a virtual device
> > with 1GB mmio size to debug this issue. Firstly, QEMU set memory
> > regions except pci hole region in pc_init1() and xen_ram_init(), then
> > hvmloader calculated pci_mem_start as 0x80000000, and wrote it to TOM
> > register, which triggered QEMU to update pci hole region with
> > 0x80000000 using i440fx_update_pci_mem_hole(). Finally the windows 7
VM
> > (configured 8G) crashed with BSOD code 0x00000024. If I hardcode in
> > QEMU pc_init1 and xen_ram_init to match hvmloader''s. Then the
problem
> > was gone.
> > >
> > > Althrough above two patches will pass actual pci hole start
address
> > to QEMU, but it''s too late, QEMU pc_init1() and
xen_ram_init() already
> > set the other memory regions, and obviously the pci hole might overlap
> > with ram regions in this case. So I think hvmloader should setup pci
> > devices and calculate pci hole first, then QEMU can map memory regions
> > correctly from the beginning.
> > >
> > 
> > Thank you very much for your detailed analysis of the problem.
> > 
> > After reading this, I wonder how is possible that qemu-xen-traditional
> > does not have this issue, considering that AFAIK there is no way for
> > hvmloader to tell qemu-xen-traditional where the PCI hole starts.
> > 
> > The only difference between upstream QEMU and qemu-xen-traditional is
> > that the former would start the PCI hole at 0xf0000000 while the
latter
> > would start the PCI hole at 0xe0000000.
> > 
> > So I would expect that your test, where hvmloader is updating the PCI
> > hole region to start at 0x80000000, would fail on qemu-xen-traditional
> > too.
> 
> Yes, I think so. 
> 
> > 
> > Of course having the PCI hole starting unconditionally at 0xf0000000
> > makes it much easier to run into problems than starting it at
> > 0xe0000000.
> > 
> > 
> > Assuming that everything above is correct, this is what I would do:
> > 
> > 1) modify upstream QEMU to start the PCI hole at 0xe0000000, to match
> > qemu-xen-unstable in terms of configuration and not to introduce any
> > regressions. Do this for the Xen 4.3 release.
> 
> It''s a quick improvement before implementing a thorough solution.
Cool.
Can you confirm that the following patch solves your original problem?


diff --git a/xen-all.c b/xen-all.c
index daf43b9..259f862 100644
--- a/xen-all.c
+++ b/xen-all.c
@@ -35,6 +35,9 @@
     do { } while (0)
 #endif
 
+#define QEMU_XEN_BELOW_4G_RAM_END       0xe0000000
+#define QEMU_XEN_BELOW_4G_MMIO_LENGTH   ((1ULL << 32) -
QEMU_XEN_BELOW_4G_RAM_END)
+
 static MemoryRegion ram_memory, ram_640k, ram_lo, ram_hi;
 static MemoryRegion *framebuffer;
 static bool xen_in_migration;
@@ -160,18 +163,18 @@ static void xen_ram_init(ram_addr_t ram_size)
     ram_addr_t block_len;
 
     block_len = ram_size;
-    if (ram_size >= HVM_BELOW_4G_RAM_END) {
+    if (ram_size >= QEMU_XEN_BELOW_4G_RAM_END) {
         /* Xen does not allocate the memory continuously, and keep a hole at
-         * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
+         * QEMU_XEN_BELOW_4G_RAM_END of QEMU_XEN_BELOW_4G_MMIO_LENGTH
          */
-        block_len += HVM_BELOW_4G_MMIO_LENGTH;
+        block_len += QEMU_XEN_BELOW_4G_MMIO_LENGTH;
     }
     memory_region_init_ram(&ram_memory, "xen.ram", block_len);
     vmstate_register_ram_global(&ram_memory);
 
-    if (ram_size >= HVM_BELOW_4G_RAM_END) {
-        above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
-        below_4g_mem_size = HVM_BELOW_4G_RAM_END;
+    if (ram_size >= QEMU_XEN_BELOW_4G_RAM_END) {
+        above_4g_mem_size = ram_size - QEMU_XEN_BELOW_4G_RAM_END;
+        below_4g_mem_size = QEMU_XEN_BELOW_4G_RAM_END;
     } else {
         below_4g_mem_size = ram_size;
     }
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

> -----Original Message-----
> From: Stefano Stabellini [mailto:stefano.stabellini@eu.citrix.com]
> Sent: 2013年5月30日 18:28
> To: Hanweidong
> Cc: Stefano Stabellini; George Dunlap; xudong.hao@intel.com;
> Yanqiangjun; Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei (Arei);
> Anthony Perard; xen-devel@lists.xen.org; xiantao.zhang@intel.com
> Subject: RE: [Xen-devel] GPU passthrough issue when VM is configured
> with 4G memoryo
> 
> On Thu, 30 May 2013, Hanweidong wrote:
> > > -----Original Message-----
> > > From: Stefano Stabellini
[mailto:stefano.stabellini@eu.citrix.com]
> > > Sent: 2013年5月30日 0:18
> > > To: Hanweidong
> > > Cc: Stefano Stabellini; George Dunlap; xudong.hao@intel.com;
> > > Yanqiangjun; Luonengjun; Wangzhenguo; Yangxiaowei; Gonglei
(Arei);
> > > Anthony Perard; xen-devel@lists.xen.org; xiantao.zhang@intel.com
> > > Subject: RE: [Xen-devel] GPU passthrough issue when VM is
> configured
> > > with 4G memory
> > >
> > > On Thu, 25 Apr 2013, Hanweidong wrote:
> > > > > -----Original Message-----
> > > > > From: xen-devel-bounces@lists.xen.org
[mailto:xen-devel-
> > > > > bounces@lists.xen.org] On Behalf Of Hanweidong
> > > > > Sent: 2013年3月26日 17:38
> > > > > To: Stefano Stabellini
> > > > > Cc: George Dunlap; xudong.hao@intel.com; Yanqiangjun;
> Luonengjun;
> > > > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony
Perard; xen-
> > > > > devel@lists.xen.org; xiantao.zhang@intel.com
> > > > > Subject: Re: [Xen-devel] GPU passthrough issue when VM
is
> > > configured
> > > > > with 4G memory
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Stefano Stabellini
> [mailto:stefano.stabellini@eu.citrix.com]
> > > > > > Sent: 2013年3月18日 20:02
> > > > > > To: Hanweidong
> > > > > > Cc: George Dunlap; Stefano Stabellini;
Yanqiangjun;
> Luonengjun;
> > > > > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony
Perard;
> xen-
> > > > > > devel@lists.xen.org; xudong.hao@intel.com;
> > > xiantao.zhang@intel.com
> > > > > > Subject: RE: [Xen-devel] GPU passthrough issue
when VM is
> > > configured
> > > > > > with 4G memory
> > > > > >
> > > > > > On Wed, 13 Mar 2013, Hanweidong wrote:
> > > > > > > MMIO HOLE was adjusted to e0000000 -
fc000000. But QEMU
> uses
> > > below
> > > > > > code to init
> > > > > > > RAM in xen_ram_init:
> > > > > > >
> > > > > > >     ...
> > > > > > >     block_len = ram_size;
> > > > > > >     if (ram_size >= HVM_BELOW_4G_RAM_END)
{
> > > > > > >         /* Xen does not allocate the memory
continuously,
> and
> > > keep
> > > > > a
> > > > > > hole at
> > > > > > >          * HVM_BELOW_4G_MMIO_START of
> HVM_BELOW_4G_MMIO_LENGTH
> > > > > > >          */
> > > > > > >         block_len +=
HVM_BELOW_4G_MMIO_LENGTH;
> > > > > > >     }
> > > > > > >     memory_region_init_ram(&ram_memory,
"xen.ram",
> block_len);
> > > > > > >    
vmstate_register_ram_global(&ram_memory);
> > > > > > >
> > > > > > >     if (ram_size >= HVM_BELOW_4G_RAM_END)
{
> > > > > > >         above_4g_mem_size = ram_size -
HVM_BELOW_4G_RAM_END;
> > > > > > >         below_4g_mem_size =
HVM_BELOW_4G_RAM_END;
> > > > > > >     } else {
> > > > > > >         below_4g_mem_size = ram_size;
> > > > > > >     }
> > > > > > >     ...
> > > > > > >
> > > > > > > HVM_BELOW_4G_RAM_END is f0000000. If we
change
> > > HVM_BELOW_4G_RAM_END
> > > > > > to e0000000,
> > > > > > > Which it's consistent with hvmloader when
assigning a GPU,
> and
> > > then
> > > > > > guest worked
> > > > > > > for us. So we wondering that xen_ram_init in
QEMU should be
> > > > > > consistent with
> > > > > > > hvmloader.
> > > > > > >
> > > > > > > In addition, we found QEMU uses hardcode
0xe0000000 in
> > > pc_init1()
> > > > > as
> > > > > > below.
> > > > > > > Should keep these places handle the
consistent mmio hole or
> not?
> > > > > > >
> > > > > > >     if (ram_size >= 0xe0000000 ) {
> > > > > > >         above_4g_mem_size = ram_size -
0xe0000000;
> > > > > > >         below_4g_mem_size = 0xe0000000;
> > > > > > >     } else {
> > > > > > >         above_4g_mem_size = 0;
> > > > > > >         below_4g_mem_size = ram_size;
> > > > > > >     }
> > > > > >
> > > > > > The guys at Intel sent a couple of patches
recently to fix
> this
> > > issue:
> > > > > >
> > > > > >
http://marc.info/?l=xen-devel&m=136150317011027
> > > > > >
http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> > > > > >
> > > > > > Do they solve your problem?
> > > > >
> > > > > These two patches didn't solve our problem.
> > > > >
> > > >
> > > > I debugged this issue with above two patches. I want to
share
> some
> > > information and discuss solution here. This issue is actually
> caused by
> > > that a VM has a large pci hole (mmio size) which results in QEMU
> sets
> > > memory regions inconsistently with hvmloader (QEMU uses hardcode
> > > 0xe0000000 in pc_init1 and xen_ram_init). I created a virtual
> device
> > > with 1GB mmio size to debug this issue. Firstly, QEMU set memory
> > > regions except pci hole region in pc_init1() and xen_ram_init(),
> then
> > > hvmloader calculated pci_mem_start as 0x80000000, and wrote it to
> TOM
> > > register, which triggered QEMU to update pci hole region with
> > > 0x80000000 using i440fx_update_pci_mem_hole(). Finally the
windows
> 7 VM
> > > (configured 8G) crashed with BSOD code 0x00000024. If I hardcode
in
> > > QEMU pc_init1 and xen_ram_init to match hvmloader's. Then the
> problem
> > > was gone.
> > > >
> > > > Althrough above two patches will pass actual pci hole start
> address
> > > to QEMU, but it's too late, QEMU pc_init1() and
xen_ram_init()
> already
> > > set the other memory regions, and obviously the pci hole might
> overlap
> > > with ram regions in this case. So I think hvmloader should setup
> pci
> > > devices and calculate pci hole first, then QEMU can map memory
> regions
> > > correctly from the beginning.
> > > >
> > >
> > > Thank you very much for your detailed analysis of the problem.
> > >
> > > After reading this, I wonder how is possible that qemu-xen-
> traditional
> > > does not have this issue, considering that AFAIK there is no way
> for
> > > hvmloader to tell qemu-xen-traditional where the PCI hole starts.
> > >
> > > The only difference between upstream QEMU and
qemu-xen-traditional
> is
> > > that the former would start the PCI hole at 0xf0000000 while the
> latter
> > > would start the PCI hole at 0xe0000000.
> > >
> > > So I would expect that your test, where hvmloader is updating the
> PCI
> > > hole region to start at 0x80000000, would fail on qemu-xen-
> traditional
> > > too.
> >
> > Yes, I think so.
> >
> > >
> > > Of course having the PCI hole starting unconditionally at
> 0xf0000000
> > > makes it much easier to run into problems than starting it at
> > > 0xe0000000.
> > >
> > >
> > > Assuming that everything above is correct, this is what I would
do:
> > >
> > > 1) modify upstream QEMU to start the PCI hole at 0xe0000000, to
> match
> > > qemu-xen-unstable in terms of configuration and not to introduce
> any
> > > regressions. Do this for the Xen 4.3 release.
> >
> > It's a quick improvement before implementing a thorough solution.
> 
> Cool.
> Can you confirm that the following patch solves your original problem?
> 
Actually I already modified code like your below patch. It worked for me when I
passthrough one GPU whose mmio size is about 200MB.

There is hardcode 0xe0000000 in pc_init1() in pc_piix.c file. I suggest to
replace it by QEMU_XEN_BELOW_4G_RAM_END. I think the memory layout calculation
should be consistent between xen_ram_init() and pc_init1().

weidong
	
> 
> diff --git a/xen-all.c b/xen-all.c
> index daf43b9..259f862 100644
> --- a/xen-all.c
> +++ b/xen-all.c
> @@ -35,6 +35,9 @@
>      do { } while (0)
>  #endif
> 
> +#define QEMU_XEN_BELOW_4G_RAM_END       0xe0000000
> +#define QEMU_XEN_BELOW_4G_MMIO_LENGTH   ((1ULL << 32) -
> QEMU_XEN_BELOW_4G_RAM_END)
> +
>  static MemoryRegion ram_memory, ram_640k, ram_lo, ram_hi;
>  static MemoryRegion *framebuffer;
>  static bool xen_in_migration;
> @@ -160,18 +163,18 @@ static void xen_ram_init(ram_addr_t ram_size)
>      ram_addr_t block_len;
> 
>      block_len = ram_size;
> -    if (ram_size >= HVM_BELOW_4G_RAM_END) {
> +    if (ram_size >= QEMU_XEN_BELOW_4G_RAM_END) {
>          /* Xen does not allocate the memory continuously, and keep a
> hole at
> -         * HVM_BELOW_4G_MMIO_START of HVM_BELOW_4G_MMIO_LENGTH
> +         * QEMU_XEN_BELOW_4G_RAM_END of QEMU_XEN_BELOW_4G_MMIO_LENGTH
>           */
> -        block_len += HVM_BELOW_4G_MMIO_LENGTH;
> +        block_len += QEMU_XEN_BELOW_4G_MMIO_LENGTH;
>      }
>      memory_region_init_ram(&ram_memory, "xen.ram",
block_len);
>      vmstate_register_ram_global(&ram_memory);
> 
> -    if (ram_size >= HVM_BELOW_4G_RAM_END) {
> -        above_4g_mem_size = ram_size - HVM_BELOW_4G_RAM_END;
> -        below_4g_mem_size = HVM_BELOW_4G_RAM_END;
> +    if (ram_size >= QEMU_XEN_BELOW_4G_RAM_END) {
> +        above_4g_mem_size = ram_size - QEMU_XEN_BELOW_4G_RAM_END;
> +        below_4g_mem_size = QEMU_XEN_BELOW_4G_RAM_END;
>      } else {
>          below_4g_mem_size = ram_size;
>      }
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Wed, May 29, 2013 at 05:18:24PM +0100, Stefano Stabellini
wrote:
> On Thu, 25 Apr 2013, Hanweidong wrote:
> > > -----Original Message-----
> > > From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> > > bounces@lists.xen.org] On Behalf Of Hanweidong
> > > Sent: 2013年3月26日 17:38
> > > To: Stefano Stabellini
> > > Cc: George Dunlap; xudong.hao@intel.com; Yanqiangjun; Luonengjun;
> > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard; xen-
> > > devel@lists.xen.org; xiantao.zhang@intel.com
> > > Subject: Re: [Xen-devel] GPU passthrough issue when VM is
configured
> > > with 4G memory
> > > 
> > > 
> > > > -----Original Message-----
> > > > From: Stefano Stabellini
[mailto:stefano.stabellini@eu.citrix.com]
> > > > Sent: 2013年3月18日 20:02
> > > > To: Hanweidong
> > > > Cc: George Dunlap; Stefano Stabellini; Yanqiangjun;
Luonengjun;
> > > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard;
xen-
> > > > devel@lists.xen.org; xudong.hao@intel.com;
xiantao.zhang@intel.com
> > > > Subject: RE: [Xen-devel] GPU passthrough issue when VM is
configured
> > > > with 4G memory
> > > >
> > > > On Wed, 13 Mar 2013, Hanweidong wrote:
> > > > > MMIO HOLE was adjusted to e0000000 - fc000000. But QEMU
uses below
> > > > code to init
> > > > > RAM in xen_ram_init:
> > > > >
> > > > >     ...
> > > > >     block_len = ram_size;
> > > > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > > > >         /* Xen does not allocate the memory
continuously, and keep
> > > a
> > > > hole at
> > > > >          * HVM_BELOW_4G_MMIO_START of
HVM_BELOW_4G_MMIO_LENGTH
> > > > >          */
> > > > >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
> > > > >     }
> > > > >     memory_region_init_ram(&ram_memory,
"xen.ram", block_len);
> > > > >     vmstate_register_ram_global(&ram_memory);
> > > > >
> > > > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > > > >         above_4g_mem_size = ram_size -
HVM_BELOW_4G_RAM_END;
> > > > >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
> > > > >     } else {
> > > > >         below_4g_mem_size = ram_size;
> > > > >     }
> > > > >     ...
> > > > >
> > > > > HVM_BELOW_4G_RAM_END is f0000000. If we change
HVM_BELOW_4G_RAM_END
> > > > to e0000000,
> > > > > Which it's consistent with hvmloader when assigning
a GPU, and then
> > > > guest worked
> > > > > for us. So we wondering that xen_ram_init in QEMU
should be
> > > > consistent with
> > > > > hvmloader.
> > > > >
> > > > > In addition, we found QEMU uses hardcode 0xe0000000 in
pc_init1()
> > > as
> > > > below.
> > > > > Should keep these places handle the consistent mmio
hole or not?
> > > > >
> > > > >     if (ram_size >= 0xe0000000 ) {
> > > > >         above_4g_mem_size = ram_size - 0xe0000000;
> > > > >         below_4g_mem_size = 0xe0000000;
> > > > >     } else {
> > > > >         above_4g_mem_size = 0;
> > > > >         below_4g_mem_size = ram_size;
> > > > >     }
> > > >
> > > > The guys at Intel sent a couple of patches recently to fix
this issue:
> > > >
> > > > http://marc.info/?l=xen-devel&m=136150317011027
> > > > http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> > > >
> > > > Do they solve your problem?
> > > 
> > > These two patches didn't solve our problem.
> > > 
> > 
> > I debugged this issue with above two patches. I want to share some
information and discuss solution here. This issue is actually caused by that a
VM has a large pci hole (mmio size) which results in QEMU sets memory regions
inconsistently with hvmloader (QEMU uses hardcode 0xe0000000 in pc_init1 and
xen_ram_init). I created a virtual device with 1GB mmio size to debug this
issue. Firstly, QEMU set memory regions except pci hole region in pc_init1() and
xen_ram_init(), then hvmloader calculated pci_mem_start as 0x80000000, and wrote
it to TOM register, which triggered QEMU to update pci hole region with
0x80000000 using i440fx_update_pci_mem_hole(). Finally the windows 7 VM
(configured 8G) crashed with BSOD code 0x00000024. If I hardcode in QEMU
pc_init1 and xen_ram_init to match hvmloader's. Then the problem was gone.
> > 
> > Althrough above two patches will pass actual pci hole start address to
QEMU, but it's too late, QEMU pc_init1() and xen_ram_init() already set the
other memory regions, and obviously the pci hole might overlap with ram regions
in this case. So I think hvmloader should setup pci devices and calculate pci
hole first, then QEMU can map memory regions correctly from the beginning.
> > 
> 
> Thank you very much for your detailed analysis of the problem.
> 
> After reading this, I wonder how is possible that qemu-xen-traditional
> does not have this issue, considering that AFAIK there is no way for
> hvmloader to tell qemu-xen-traditional where the PCI hole starts.
> 
> The only difference between upstream QEMU and qemu-xen-traditional is
> that the former would start the PCI hole at 0xf0000000 while the latter
> would start the PCI hole at 0xe0000000.
> 
> So I would expect that your test, where hvmloader is updating the PCI
> hole region to start at 0x80000000, would fail on qemu-xen-traditional
> too.
> 
> Of course having the PCI hole starting unconditionally at 0xf0000000
> makes it much easier to run into problems than starting it at
> 0xe0000000.
> 
> 
> Assuming that everything above is correct, this is what I would do:
> 
> 1) modify upstream QEMU to start the PCI hole at 0xe0000000, to match
> qemu-xen-unstable in terms of configuration and not to introduce any
> regressions. Do this for the Xen 4.3 release.
> 
> 2) for Xen 4.4 rework the two patches above and improve
> i440fx_update_pci_mem_hole: resizing the pci_hole subregion is not
> enough, it also needs to be able to resize the system memory region
> (xen.ram) to make room for the bigger pci_hole

Would that make migration more difficult - meaning if you have now two
different QEMU versions where the PCI hole is different on them? Or is
that not an issue and QEMU handles setting the layout nicely? Or is
the 0xe0000000 the norm in Xen 4.1, and Xen 4.2?

I am assuming you unplug the PCI device before you migrate of course.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Mon, 3 Jun 2013, Konrad Rzeszutek Wilk wrote:
> On Wed, May 29, 2013 at 05:18:24PM +0100, Stefano Stabellini wrote:
> > On Thu, 25 Apr 2013, Hanweidong wrote:
> > > > -----Original Message-----
> > > > From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> > > > bounces@lists.xen.org] On Behalf Of Hanweidong
> > > > Sent: 2013年3月26日 17:38
> > > > To: Stefano Stabellini
> > > > Cc: George Dunlap; xudong.hao@intel.com; Yanqiangjun;
Luonengjun;
> > > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony Perard;
xen-
> > > > devel@lists.xen.org; xiantao.zhang@intel.com
> > > > Subject: Re: [Xen-devel] GPU passthrough issue when VM is
configured
> > > > with 4G memory
> > > > 
> > > > 
> > > > > -----Original Message-----
> > > > > From: Stefano Stabellini
[mailto:stefano.stabellini@eu.citrix.com]
> > > > > Sent: 2013年3月18日 20:02
> > > > > To: Hanweidong
> > > > > Cc: George Dunlap; Stefano Stabellini; Yanqiangjun;
Luonengjun;
> > > > > Wangzhenguo; Yangxiaowei; Gonglei (Arei); Anthony
Perard; xen-
> > > > > devel@lists.xen.org; xudong.hao@intel.com;
xiantao.zhang@intel.com
> > > > > Subject: RE: [Xen-devel] GPU passthrough issue when VM
is configured
> > > > > with 4G memory
> > > > >
> > > > > On Wed, 13 Mar 2013, Hanweidong wrote:
> > > > > > MMIO HOLE was adjusted to e0000000 - fc000000. But
QEMU uses below
> > > > > code to init
> > > > > > RAM in xen_ram_init:
> > > > > >
> > > > > >     ...
> > > > > >     block_len = ram_size;
> > > > > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > > > > >         /* Xen does not allocate the memory
continuously, and keep
> > > > a
> > > > > hole at
> > > > > >          * HVM_BELOW_4G_MMIO_START of
HVM_BELOW_4G_MMIO_LENGTH
> > > > > >          */
> > > > > >         block_len += HVM_BELOW_4G_MMIO_LENGTH;
> > > > > >     }
> > > > > >     memory_region_init_ram(&ram_memory,
"xen.ram", block_len);
> > > > > >     vmstate_register_ram_global(&ram_memory);
> > > > > >
> > > > > >     if (ram_size >= HVM_BELOW_4G_RAM_END) {
> > > > > >         above_4g_mem_size = ram_size -
HVM_BELOW_4G_RAM_END;
> > > > > >         below_4g_mem_size = HVM_BELOW_4G_RAM_END;
> > > > > >     } else {
> > > > > >         below_4g_mem_size = ram_size;
> > > > > >     }
> > > > > >     ...
> > > > > >
> > > > > > HVM_BELOW_4G_RAM_END is f0000000. If we change
HVM_BELOW_4G_RAM_END
> > > > > to e0000000,
> > > > > > Which it''s consistent with hvmloader when
assigning a GPU, and then
> > > > > guest worked
> > > > > > for us. So we wondering that xen_ram_init in QEMU
should be
> > > > > consistent with
> > > > > > hvmloader.
> > > > > >
> > > > > > In addition, we found QEMU uses hardcode
0xe0000000 in pc_init1()
> > > > as
> > > > > below.
> > > > > > Should keep these places handle the consistent
mmio hole or not?
> > > > > >
> > > > > >     if (ram_size >= 0xe0000000 ) {
> > > > > >         above_4g_mem_size = ram_size - 0xe0000000;
> > > > > >         below_4g_mem_size = 0xe0000000;
> > > > > >     } else {
> > > > > >         above_4g_mem_size = 0;
> > > > > >         below_4g_mem_size = ram_size;
> > > > > >     }
> > > > >
> > > > > The guys at Intel sent a couple of patches recently to
fix this issue:
> > > > >
> > > > > http://marc.info/?l=xen-devel&m=136150317011027
> > > > >
http://marc.info/?l=qemu-devel&m=136177475215360&w=2
> > > > >
> > > > > Do they solve your problem?
> > > > 
> > > > These two patches didn''t solve our problem.
> > > > 
> > > 
> > > I debugged this issue with above two patches. I want to share
some information and discuss solution here. This issue is actually caused by
that a VM has a large pci hole (mmio size) which results in QEMU sets memory
regions inconsistently with hvmloader (QEMU uses hardcode 0xe0000000 in pc_init1
and xen_ram_init). I created a virtual device with 1GB mmio size to debug this
issue. Firstly, QEMU set memory regions except pci hole region in pc_init1() and
xen_ram_init(), then hvmloader calculated pci_mem_start as 0x80000000, and wrote
it to TOM register, which triggered QEMU to update pci hole region with
0x80000000 using i440fx_update_pci_mem_hole(). Finally the windows 7 VM
(configured 8G) crashed with BSOD code 0x00000024. If I hardcode in QEMU
pc_init1 and xen_ram_init to match hvmloader''s. Then the problem was
gone.
> > > 
> > > Althrough above two patches will pass actual pci hole start
address to QEMU, but it''s too late, QEMU pc_init1() and xen_ram_init()
already set the other memory regions, and obviously the pci hole might overlap
with ram regions in this case. So I think hvmloader should setup pci devices and
calculate pci hole first, then QEMU can map memory regions correctly from the
beginning.
> > > 
> > 
> > Thank you very much for your detailed analysis of the problem.
> > 
> > After reading this, I wonder how is possible that qemu-xen-traditional
> > does not have this issue, considering that AFAIK there is no way for
> > hvmloader to tell qemu-xen-traditional where the PCI hole starts.
> > 
> > The only difference between upstream QEMU and qemu-xen-traditional is
> > that the former would start the PCI hole at 0xf0000000 while the
latter
> > would start the PCI hole at 0xe0000000.
> > 
> > So I would expect that your test, where hvmloader is updating the PCI
> > hole region to start at 0x80000000, would fail on qemu-xen-traditional
> > too.
> > 
> > Of course having the PCI hole starting unconditionally at 0xf0000000
> > makes it much easier to run into problems than starting it at
> > 0xe0000000.
> > 
> > 
> > Assuming that everything above is correct, this is what I would do:
> > 
> > 1) modify upstream QEMU to start the PCI hole at 0xe0000000, to match
> > qemu-xen-unstable in terms of configuration and not to introduce any
> > regressions. Do this for the Xen 4.3 release.
> > 
> > 2) for Xen 4.4 rework the two patches above and improve
> > i440fx_update_pci_mem_hole: resizing the pci_hole subregion is not
> > enough, it also needs to be able to resize the system memory region
> > (xen.ram) to make room for the bigger pci_hole
> 
> 
> Would that make migration more difficult - meaning if you have now two
> different QEMU versions where the PCI hole is different on them? Or is
> that not an issue and QEMU handles setting the layout nicely? Or is
> the 0xe0000000 the norm in Xen 4.1, and Xen 4.2?
>
> I am assuming you unplug the PCI device before you migrate of course.

the change in configuration is only for qemu-xen and upstream QEMU and
Xen 4.3 is the first release that defaults to it, so I don''t think we
need to maintain save/restore compatibility yet. But from the next one
is going to be unavoidable.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Hello,

George: I think this one is missing from the Xen 4.4 status emails? (see below)

On Wed, May 29, 2013 at 05:18:24PM +0100, Stefano Stabellini
wrote:
> 
> Thank you very much for your detailed analysis of the problem.
> 
> After reading this, I wonder how is possible that qemu-xen-traditional
> does not have this issue, considering that AFAIK there is no way for
> hvmloader to tell qemu-xen-traditional where the PCI hole starts.
> 
> The only difference between upstream QEMU and qemu-xen-traditional is
> that the former would start the PCI hole at 0xf0000000 while the latter
> would start the PCI hole at 0xe0000000.
> 
> So I would expect that your test, where hvmloader is updating the PCI
> hole region to start at 0x80000000, would fail on qemu-xen-traditional
> too.
> 
> Of course having the PCI hole starting unconditionally at 0xf0000000
> makes it much easier to run into problems than starting it at
> 0xe0000000.
> 
> 
> Assuming that everything above is correct, this is what I would do:
> 
> 1) modify upstream QEMU to start the PCI hole at 0xe0000000, to match
> qemu-xen-unstable in terms of configuration and not to introduce any
> regressions. Do this for the Xen 4.3 release.
> 
> 2) for Xen 4.4 rework the two patches above and improve
> i440fx_update_pci_mem_hole: resizing the pci_hole subregion is not
> enough, it also needs to be able to resize the system memory region
> (xen.ram) to make room for the bigger pci_hole

I think this second part hasn''t been done/fixed yet? 
Feel free to correct me if it has been done already :) 

Thanks,

-- Pasi