Xen devel - Jul 2010 - Frame buffer mmap not working in pvops dom0

I''m trying to confirm the fix to the VESA fbdev mmap issue that was
brought up a few months ago
(http://marc.info/?l=xen-devel&m=126842551306571&w=2). The wiki page at
http://wiki.xensource.com/xenwiki/XenPVOPSDRM says that this bug should
be fixed, but doesn''t point to a patch for the fix. I am still able to
reproduce the issue both on real hardware and by running Xen under qemu
(using cirrusfb on the dom0). Eamon (the original reporter) has also not
been able to confirm a fix.

I''m currently testing using Xen 4.1 built from hg 21831:6bebaf40e925
and
a pvops dom0 from xen/stable-2.6.32.x revid c0a00fbe.

So far, I''ve been able to determine that an mmap requesting multiple
pages from /dev/fb0 will result in page table entries all pointing to
the same physical page, which is not in the framebuffer address space.
Writing to the mapped page ends up corrupting parts of kernel memory.
I''d be happy to run further tests, try patches, or provide more
information if needed.

-- 

Daniel De Graaf
National Security Agency

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Jul 21, 2010 at 09:47:57AM -0400, Daniel De Graaf
wrote:
> I''m trying to confirm the fix to the VESA fbdev mmap issue that
was
> brought up a few months ago
> (http://marc.info/?l=xen-devel&m=126842551306571&w=2). The wiki
page at
> http://wiki.xensource.com/xenwiki/XenPVOPSDRM says that this bug should
> be fixed, but doesn''t point to a patch for the fix. I am still
able to
> reproduce the issue both on real hardware and by running Xen under qemu
> (using cirrusfb on the dom0). Eamon (the original reporter) has also not
> been able to confirm a fix.
> 
> I''m currently testing using Xen 4.1 built from hg
21831:6bebaf40e925 and
> a pvops dom0 from xen/stable-2.6.32.x revid c0a00fbe.
> 
> So far, I''ve been able to determine that an mmap requesting
multiple
> pages from /dev/fb0 will result in page table entries all pointing to
> the same physical page, which is not in the framebuffer address space.
> Writing to the mapped page ends up corrupting parts of kernel memory.
> I''d be happy to run further tests, try patches, or provide more
> information if needed.
> 
I guess many (most?) graphics related fixes are in Konrad''s git tree..

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Jul 21, 2010 at 05:16:13PM +0300, Pasi Kärkkäinen
wrote:
> On Wed, Jul 21, 2010 at 09:47:57AM -0400, Daniel De Graaf wrote:
> > I''m trying to confirm the fix to the VESA fbdev mmap issue
that was
> > brought up a few months ago
> > (http://marc.info/?l=xen-devel&m=126842551306571&w=2). The
wiki page at
Weird. I don''t remember seeing that e-mail..
> > http://wiki.xensource.com/xenwiki/XenPVOPSDRM says that this bug
should
> > be fixed, but doesn''t point to a patch for the fix. I am
still able to
> > reproduce the issue both on real hardware and by running Xen under
qemu
> > (using cirrusfb on the dom0). Eamon (the original reporter) has also
not
> > been able to confirm a fix.
> > 
> > I''m currently testing using Xen 4.1 built from hg
21831:6bebaf40e925 and
> > a pvops dom0 from xen/stable-2.6.32.x revid c0a00fbe.
> > 
> > So far, I''ve been able to determine that an mmap requesting
multiple
> > pages from /dev/fb0 will result in page table entries all pointing to
> > the same physical page, which is not in the framebuffer address space.
> > Writing to the mapped page ends up corrupting parts of kernel memory.
> > I''d be happy to run further tests, try patches, or provide
more
> > information if needed.
Goodies. Let me fix up a tree that cleanly merges with Jeremy''s
xen/next
(or xen/stable-2.6.32.x) and give you a go with that. And then from
there we can come up with a fix.

Can you tell me how you came up with the analysis? (that should speed up
finding the culprit). Any serial/dmesg outputs would be appreciated.
> > 
> 
> I guess many (most?) graphics related fixes are in Konrad''s git
tree..
<nods>

Daniel,

I honestly don''t remember which patch I thought fixed it. But I do know
that when the /dev/fb0 was backed by DRM (nvidia) it worked (I used the 
below program).

Granted now that I tested it with fbxine and it hangs with a Xen error.
Let me start using Eamon''s program.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Jul 21, 2010 at 10:42:09AM -0400, Konrad Rzeszutek Wilk
wrote:
> On Wed, Jul 21, 2010 at 05:16:13PM +0300, Pasi Kärkkäinen wrote:
> > On Wed, Jul 21, 2010 at 09:47:57AM -0400, Daniel De Graaf wrote:
> > > I''m trying to confirm the fix to the VESA fbdev mmap
issue that was
> > > brought up a few months ago
> > > (http://marc.info/?l=xen-devel&m=126842551306571&w=2).
The wiki page at
> 
> Weird. I don''t remember seeing that e-mail..
> 
> > > http://wiki.xensource.com/xenwiki/XenPVOPSDRM says that this bug
should
> > > be fixed, but doesn''t point to a patch for the fix. I am
still able to
> > > reproduce the issue both on real hardware and by running Xen
under qemu
> > > (using cirrusfb on the dom0). Eamon (the original reporter) has
also not
> > > been able to confirm a fix.
> > > 
> > > I''m currently testing using Xen 4.1 built from hg
21831:6bebaf40e925 and
> > > a pvops dom0 from xen/stable-2.6.32.x revid c0a00fbe.
> > > 
> > > So far, I''ve been able to determine that an mmap
requesting multiple
> > > pages from /dev/fb0 will result in page table entries all
pointing to
> > > the same physical page, which is not in the framebuffer address
space.
> > > Writing to the mapped page ends up corrupting parts of kernel
memory.
> > > I''d be happy to run further tests, try patches, or
provide more
> > > information if needed.
> 
> Goodies. Let me fix up a tree that cleanly merges with Jeremy''s
xen/next
> (or xen/stable-2.6.32.x) and give you a go with that. And then from
> there we can come up with a fix.
> 
I''d prefer xen/stable-2.6.32.x, since that''s what most people
are using..
We''ll get more testers that way.

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 07/21/2010 10:42 AM, Konrad Rzeszutek Wilk wrote:
> On Wed, Jul 21, 2010 at 05:16:13PM +0300, Pasi Kärkkäinen wrote:
>> On Wed, Jul 21, 2010 at 09:47:57AM -0400, Daniel De Graaf wrote:
>>> I''m trying to confirm the fix to the VESA fbdev mmap issue
that was
>>> brought up a few months ago
>>> (http://marc.info/?l=xen-devel&m=126842551306571&w=2). The
wiki page at
> 
> Weird. I don''t remember seeing that e-mail..
> 
>>> http://wiki.xensource.com/xenwiki/XenPVOPSDRM says that this bug
should
>>> be fixed, but doesn''t point to a patch for the fix. I am
still able to
>>> reproduce the issue both on real hardware and by running Xen under
qemu
>>> (using cirrusfb on the dom0). Eamon (the original reporter) has
also not
>>> been able to confirm a fix.
>>>
>>> I''m currently testing using Xen 4.1 built from hg
21831:6bebaf40e925 and
>>> a pvops dom0 from xen/stable-2.6.32.x revid c0a00fbe.
>>>
>>> So far, I''ve been able to determine that an mmap
requesting multiple
>>> pages from /dev/fb0 will result in page table entries all pointing
to
>>> the same physical page, which is not in the framebuffer address
space.
>>> Writing to the mapped page ends up corrupting parts of kernel
memory.
>>> I''d be happy to run further tests, try patches, or provide
more
>>> information if needed.
> 
> Goodies. Let me fix up a tree that cleanly merges with Jeremy''s
xen/next
> (or xen/stable-2.6.32.x) and give you a go with that. And then from
> there we can come up with a fix.
> 
> Can you tell me how you came up with the analysis? (that should speed up
> finding the culprit). Any serial/dmesg outputs would be appreciated.
> 
I have been dumping the page tables (using the attached pt-dump script,
as qemu''s "info tlb" only works on i386) from a paused qemu
instance
that is running a simple mmap-and-spin program (also attached). All 100
pages map to physical memory address 39a4c000.
>From a bit more debugging, I''ve been able to trace the correct
address
(0xf0000000) being lost when it is passed by xen_make_pte to
pte_pfn_to_mfn and eventually to get_phys_to_machine(0xf0000) which
returns -1. Still not sure where the final physical address is coming
from, but I''m guessing this is part of the problem.
> 
> Daniel,
> 
> I honestly don''t remember which patch I thought fixed it. But I do
know
> that when the /dev/fb0 was backed by DRM (nvidia) it worked (I used the 
> below program).
> 
> Granted now that I tested it with fbxine and it hangs with a Xen error.
> Let me start using Eamon''s program.
> 
The attached program has no visible effect on the screen when run, so
it''s likely also not working here.

-- 

Daniel De Graaf
National Security Agency



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I have been dumping the page tables (using the attached pt-dump script,
> as qemu''s "info tlb" only works on i386) from a paused
qemu instance
> that is running a simple mmap-and-spin program (also attached). All 100
> pages map to physical memory address 39a4c000.
The qemu output then isn''t going to a VNC window but something else. I
presume the something else is the SDL piece? Were there any special flags
to enable this on QEMU?
> 
> >From a bit more debugging, I''ve been able to trace the correct
address
> (0xf0000000) being lost when it is passed by xen_make_pte to
> pte_pfn_to_mfn and eventually to get_phys_to_machine(0xf0000) which
> returns -1. Still not sure where the final physical address is coming
> from, but I''m guessing this is part of the problem.
That looks like the VM_IO flag (_PAGE_IOMAP on the PTE) is not set somewhere.
Do you have an idea what piece of kernel code gets triggered when QEMU does
''mmap'' on the /dev/fb0?

On my machine where I use KMS/DRM it ends up calling
''ttm_fb_mmap''. But
for your system, where it looks that you are using the "old"
framebuffer
code it might be something entirely different.

.. snip..
> The attached program has no visible effect on the screen when run, so
> it''s likely also not working here.
<nods> Looks quite similar to what I''ve been using.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> That looks like the VM_IO flag (_PAGE_IOMAP on the PTE) is not set
somewhere.
> Do you have an idea what piece of kernel code gets triggered when QEMU does
> ''mmap'' on the /dev/fb0?
> 
> On my machine where I use KMS/DRM it ends up calling
''ttm_fb_mmap''. But
err.. ttm_fbdev_mmap

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 07/21/2010 03:00 PM, Konrad Rzeszutek Wilk wrote:
>> I have been dumping the page tables (using the attached pt-dump script,
>> as qemu''s "info tlb" only works on i386) from a
paused qemu instance
>> that is running a simple mmap-and-spin program (also attached). All 100
>> pages map to physical memory address 39a4c000.
> 
> The qemu output then isn''t going to a VNC window but something
else. I
> presume the something else is the SDL piece? Were there any special flags
> to enable this on QEMU?
The exact command line I''m using is:

qemu-kvm -daemonize -usbdevice tablet -m 1024 \
	-monitor unix:fc13-mon,server,nowait \
        -net nic -net tap /dev/lvm0/FC13-64

This uses SDL graphics, which is qemu''s default. Within qemu, I use
vga=ask on the xen command line in grub.
>> >From a bit more debugging, I''ve been able to trace the
correct address
>> (0xf0000000) being lost when it is passed by xen_make_pte to
>> pte_pfn_to_mfn and eventually to get_phys_to_machine(0xf0000) which
>> returns -1. Still not sure where the final physical address is coming
>> from, but I''m guessing this is part of the problem.
> 
> That looks like the VM_IO flag (_PAGE_IOMAP on the PTE) is not set
somewhere.
> Do you have an idea what piece of kernel code gets triggered when QEMU does
> ''mmap'' on the /dev/fb0?
> 
> On my machine where I use KMS/DRM it ends up calling
''ttm_fb_mmap''. But
> for your system, where it looks that you are using the "old"
framebuffer
> code it might be something entirely different.
> 
The code path is fb_mmap with a NULL fbops->fb_mmap, so it just
delegates to the default code. Specifically, io_remap_pfn_range is where
the bad mapping is requested.

I have a patch that fixes the issue, but I''m not sure under what
conditions the _PAGE_IOMAP bit needs to be set.

--- a/arch/x86/include/asm/fb.h
+++ b/arch/x86/include/asm/fb.h
@@ -10,6 +10,7 @@ static inline void fb_pgprotect
 {
        if (boot_cpu_data.x86 > 3)
                pgprot_val(vma->vm_page_prot) |= _PAGE_PCD;
+       pgprot_val(vma->vm_page_prot) |= _PAGE_IOMAP;
 }

 #ifdef CONFIG_X86_32

-- 

Daniel De Graaf
National Security Agency

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> The code path is fb_mmap with a NULL fbops->fb_mmap, so it just
> delegates to the default code. Specifically, io_remap_pfn_range is where
> the bad mapping is requested.
> 
> I have a patch that fixes the issue, but I''m not sure under what
> conditions the _PAGE_IOMAP bit needs to be set.
Oh wow. That easy, eh?
> --- a/arch/x86/include/asm/fb.h
> +++ b/arch/x86/include/asm/fb.h
> @@ -10,6 +10,7 @@ static inline void fb_pgprotect
>  {
>         if (boot_cpu_data.x86 > 3)
>                 pgprot_val(vma->vm_page_prot) |= _PAGE_PCD;
> +       pgprot_val(vma->vm_page_prot) |= _PAGE_IOMAP;
>  }
> 
>  #ifdef CONFIG_X86_32

I would say this patch is more sensible as the VM_IO flag had been
set already, it just never got propagated:


diff --git a/drivers/video/fbmem.c b/drivers/video/fbmem.c
index 731fce6..187171b 100644
--- a/drivers/video/fbmem.c
+++ b/drivers/video/fbmem.c
@@ -1362,6 +1362,7 @@ fb_mmap(struct file *file, struct vm_area_struct * vma)
 	vma->vm_pgoff = off >> PAGE_SHIFT;
 	/* This is an IO map - tell maydump to skip this VMA */
 	vma->vm_flags |= VM_IO | VM_RESERVED;
+	vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
 	fb_pgprotect(file, vma, off);
 	if (io_remap_pfn_range(vma, vma->vm_start, off >> PAGE_SHIFT,
 			     vma->vm_end - vma->vm_start, vma->vm_page_prot))


If that fixes your problem, are you OK with me sticking a Signed-off-by:
from you on this patch?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 07/21/2010 03:50 PM, Konrad Rzeszutek Wilk wrote:
>> The code path is fb_mmap with a NULL fbops->fb_mmap, so it just
>> delegates to the default code. Specifically, io_remap_pfn_range is
where
>> the bad mapping is requested.
>>
>> I have a patch that fixes the issue, but I''m not sure under
what
>> conditions the _PAGE_IOMAP bit needs to be set.
> 
> Oh wow. That easy, eh?
> 
>> --- a/arch/x86/include/asm/fb.h
>> +++ b/arch/x86/include/asm/fb.h
>> @@ -10,6 +10,7 @@ static inline void fb_pgprotect
>>  {
>>         if (boot_cpu_data.x86 > 3)
>>                 pgprot_val(vma->vm_page_prot) |= _PAGE_PCD;
>> +       pgprot_val(vma->vm_page_prot) |= _PAGE_IOMAP;
>>  }
>>
>>  #ifdef CONFIG_X86_32
> 
> 
> I would say this patch is more sensible as the VM_IO flag had been
> set already, it just never got propagated:
> 
> 
> diff --git a/drivers/video/fbmem.c b/drivers/video/fbmem.c
> index 731fce6..187171b 100644
> --- a/drivers/video/fbmem.c
> +++ b/drivers/video/fbmem.c
> @@ -1362,6 +1362,7 @@ fb_mmap(struct file *file, struct vm_area_struct *
vma)
>  	vma->vm_pgoff = off >> PAGE_SHIFT;
>  	/* This is an IO map - tell maydump to skip this VMA */
>  	vma->vm_flags |= VM_IO | VM_RESERVED;
> +	vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
>  	fb_pgprotect(file, vma, off);
>  	if (io_remap_pfn_range(vma, vma->vm_start, off >> PAGE_SHIFT,
>  			     vma->vm_end - vma->vm_start, vma->vm_page_prot))
> 
> 
> If that fixes your problem, are you OK with me sticking a Signed-off-by:
> from you on this patch?
> 
Yep, your version also fixes the problem, and I agree it''s more
sensible. I''m OK with signing off on that patch.

-- 

Daniel De Graaf
National Security Agency

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 07/21/2010 03:50 PM, Konrad Rzeszutek Wilk wrote:
>> The code path is fb_mmap with a NULL fbops->fb_mmap, so it just
>> delegates to the default code. Specifically, io_remap_pfn_range is
where
>> the bad mapping is requested.
>>
>> I have a patch that fixes the issue, but I''m not sure under
what
>> conditions the _PAGE_IOMAP bit needs to be set.
>>     
> Oh wow. That easy, eh?
>
>   
>> --- a/arch/x86/include/asm/fb.h
>> +++ b/arch/x86/include/asm/fb.h
>> @@ -10,6 +10,7 @@ static inline void fb_pgprotect
>>  {
>>         if (boot_cpu_data.x86 > 3)
>>                 pgprot_val(vma->vm_page_prot) |= _PAGE_PCD;
>> +       pgprot_val(vma->vm_page_prot) |= _PAGE_IOMAP;
>>  }
>>
>>  #ifdef CONFIG_X86_32
>>     
>
> I would say this patch is more sensible as the VM_IO flag had been
> set already, it just never got propagated:
>
>
> diff --git a/drivers/video/fbmem.c b/drivers/video/fbmem.c
> index 731fce6..187171b 100644
> --- a/drivers/video/fbmem.c
> +++ b/drivers/video/fbmem.c
> @@ -1362,6 +1362,7 @@ fb_mmap(struct file *file, struct vm_area_struct *
vma)
>  	vma->vm_pgoff = off >> PAGE_SHIFT;
>  	/* This is an IO map - tell maydump to skip this VMA */
>  	vma->vm_flags |= VM_IO | VM_RESERVED;
> +	vma->vm_page_prot = vm_get_page_prot(vma->vm_flags);
>  	fb_pgprotect(file, vma, off);
>  	if (io_remap_pfn_range(vma, vma->vm_start, off >> PAGE_SHIFT,
>  			     vma->vm_end - vma->vm_start, vma->vm_page_prot))
>
>
> If that fixes your problem, are you OK with me sticking a Signed-off-by:
> from you on this patch?
>   

Either patch fixes the problem for me.

Tested-by: Eamon Walsh <ewalsh@tycho.nsa.gov>



-- 

Eamon Walsh 
National Security Agency


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > If that fixes your problem, are you OK with me sticking a
Signed-off-by:
> > from you on this patch?
> > 
> 
> Yep, your version also fixes the problem, and I agree it''s more
> sensible. I''m OK with signing off on that patch.
Oh, and here is the S-O-B that I missed sticking on the git tree.

Anyhow, I found out that his patch also fixes the Nvidia fb problem,
which is quite nice.

By any chance did you try to use Xserver on top of this patch? In the
past I had gotten this (from Xorg.0.log file):

==) Using config directory: "/etc/xorg.conf.d"
(XEN) mm.c:1747:d0 Bad L1 flags 400000
(XEN) mm.c:1747:d0 Bad L1 flags 400000
(XEN) mm.c:1747:d0 Bad L1 flags 400000
...
(EE) FBDEV(0): FBIOPUTCMAP: Invalid argument
(EE) FBDEV(0): FBIOPUTCMAP: Invalid argument

.. and a continuing stream of FBIOPUTCMAP''s.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> ==) Using config directory: "/etc/xorg.conf.d"
> (XEN) mm.c:1747:d0 Bad L1 flags 400000
> (XEN) mm.c:1747:d0 Bad L1 flags 400000
> (XEN) mm.c:1747:d0 Bad L1 flags 400000
With your patch, those messages above are gone.
> ...
> (EE) FBDEV(0): FBIOPUTCMAP: Invalid argument
> (EE) FBDEV(0): FBIOPUTCMAP: Invalid argument
I get this even on baremetal, so I doubt this Xen problem. But 
I am curious how the fbdev works then under Xen fbdev device.
Or maybe it is a xorg-video-vesa it uses ...

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 07/28/2010 10:29 AM, Konrad Rzeszutek Wilk wrote:
>>> If that fixes your problem, are you OK with me sticking a
Signed-off-by:
>>> from you on this patch?
>>>
>>
>> Yep, your version also fixes the problem, and I agree it''s
more
>> sensible. I''m OK with signing off on that patch.
> 
> Oh, and here is the S-O-B that I missed sticking on the git tree.
> 
> Anyhow, I found out that his patch also fixes the Nvidia fb problem,
> which is quite nice.
> 
> By any chance did you try to use Xserver on top of this patch? In the
> past I had gotten this (from Xorg.0.log file):
> 
> ==) Using config directory: "/etc/xorg.conf.d"
> (XEN) mm.c:1747:d0 Bad L1 flags 400000
> (XEN) mm.c:1747:d0 Bad L1 flags 400000
> (XEN) mm.c:1747:d0 Bad L1 flags 400000
> ...
> (EE) FBDEV(0): FBIOPUTCMAP: Invalid argument
> (EE) FBDEV(0): FBIOPUTCMAP: Invalid argument
> 
> .. and a continuing stream of FBIOPUTCMAP''s.
> 
I had been testing using framebuffer only; I just tested Xorg, and it
appears to be working - I don''t see any errors in my Xorg.0.log or on
the Xen console. Haven''t tried running anything more interesting than
xterm, though.

I''m not certain if I''ve ended up testing the Xen fbdev device,
or just
using the guest''s i915 driver within the guest. This is with an empty
xorg.conf{,.d/}.

The only issue I''ve been having is that I haven''t been able to
get the
system to work more than once per powerup, even with a soft reboot of
the host (going back to the real BIOS). There''s obviously some state
not
being reset by the startup that is tripping it on later boots, but I
haven''t yet been able to find out what it is.

-- 

Daniel De Graaf
National Security Agency

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel