Xen devel - Jan 2007 - IVTV and xen: silent intermittent failures reading from /dev/video0

[Disclaimer: I realize that I probably won''t get this problem solved as
it''s probably an unsual setup. I thought I''d ask, though.]

P roblem: ''cat /dev/video0 > test.mpg'' (a simple IVTV test
of my hardware mpeg2 encoder) doesn''t work properly under Xen. When
this command is run, test.mpg should grow approximately 1MB/sec in perpetuity,
and does so when the machine is booted under a ''normal'' (i.e.
non-Xen) kernel. When I boot under Xen, test.mpg grows properly for a while
(say, 2 to 10 minutes) and then stops growing. Typical final file sizes are 50MB
to 500MB. Note that there are *no* errors in dmesg, syslog, etc. The cat command
doesn''t exit, either. It seems to be blocked on read. This same
behavior is observed whether IVTV is in dom0 or domu.

More information: 

- I can usually hasten the failure by doing something with the Xen machine that
causes lots of disk and/or network activity. Failure-hastening activity can
occur in any Xen domain, not just the one with /dev/video0 or Domain 0.
- When ''cat /dev/video0'' is running and IVTV is working,
''xm top'' shows 3-4% cpu usage in the IVTV domu. Inside the
IVTV domu, however, I see 1% cpu usage or so. As soon as the failure occurs,
''xm top'' shows that the IVTV domu cpu usage has dropped to 1%.

This failure occurs identically whether IVTV is in Domain 0 or in an
unprivileged domain. No difference.

My setup: 

- Hardware: Nforce2 motherboard, 2GB ram, 1.8GHz Athlon XP, Hauppauge PVR-150 
- Xen: 3.0.2 
- Kernel: 2.6.16-xen 
- ivtv: 0.6.6 

What I''ve tried so far: 

1. Verified that this is not a hardware problem (I''ve done stability
testing, prime95, memtest86, etc. The system works perfectly and the IVTV device
is rock-solid on a non-Xen Linux 2.6.16 kernel.)
2. Moved the encoder (IVTV) to its own IRQ 
3. Set PCI latency timers to reasonable values for all devices (e.g. 128) 
4. Verified that there are *no* errors in dmesg, xm dmesg, syslogs on domu,
syslogs on dom0, etc.
5. Verified that there are no throughput problems with disks. The output of
''cat'' is redirected to a dedicated disk on a dedicated
controller where I can get 30MB/sec writing speeds sustained.
6. Verified that I''m using the proper firmware for my encoder card 
7. Verified that swiotlb=force is not the problem. Using swiotlb=force is
required to get IVTV working under Xen kernels but not under
''normal'' kernels. However, passing it under a
''normal'' 2.6.16 kernel doesn''t cause any problems --
the IVTV device works fine.
8. Tuned the Xen scheduler to reduce latency for Domain 0 and the domain with
/dev/video0. This helped a little, but not much.

Consequences of problem: 

My recordings ''hiccup'' every 2-10 minutes or so. The hiccup is
a jump in the recording of probably just a few seconds. I believe that mythtv
detects that reading from the device has failed and restarts the read. These
interruptions are annoying but I don''t have a machine to dedicate to
this purpose, so I''m stuck with them. If there''s anything that
I can do to help diagnose this problem please let me know.

Xen rocks, by the way. Thanks very much for your work on this excellent
platform, and for your time reading this post.

Dan 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 20/1/07 3:30 am, "Dan Schmick" <dschmick@dschmick.com> wrote:
> My recordings ''hiccup'' every 2-10 minutes or so.  The
hiccup is a jump in the
> recording of probably just a few seconds.  I believe that mythtv detects
that
> reading from the device has failed and restarts the read. These
interruptions
> are annoying but I don''t have a machine to dedicate to this
purpose, so I''m
> stuck with them.  If there''s anything that I can do to help
diagnose this
> problem please let me know.
What happens if you pin the ivtv domain to a CPU that no other domains run
on? E.g., ivtv on cpu1 and all others on cpu0? Perhaps preemption latency is
an issue. 

 -- Keir
 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 	My recordings ''hiccup'' every 2-10 minutes or so.  The
hiccup is
a
> jump in the recording of probably just a few seconds.  I believe that
> mythtv detects that reading from the device has failed and restarts
the
> read. These interruptions are annoying but I don''t have a machine
to
> dedicate to this purpose, so I''m stuck with them.  If
there''s anything
that
> I can do to help diagnose this problem please let me know.
There''s been lots of scheduler improvements since 3.0.2 -- I''d
suggest
upgrading first.

Ian
 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian,

Upgrading to 3.0.3 didn''t solve the problem, but upgrading to 3.0.4 did
(so far, so good, anyway).

I''ve been beating my head against this for a week.  Many thanks!

Dan

----- Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> wrote:
> > 	My recordings ''hiccup'' every 2-10 minutes or so. 
The hiccup is
> a
> > jump in the recording of probably just a few seconds.  I believe
> that
> > mythtv detects that reading from the device has failed and restarts
> the
> > read. These interruptions are annoying but I don''t have a
machine
> to
> > dedicate to this purpose, so I''m stuck with them.  If
there''s
> anything
> that
> > I can do to help diagnose this problem please let me know.
> 
> There''s been lots of scheduler improvements since 3.0.2 --
I''d
> suggest
> upgrading first.
> 
> Ian
>  
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel