Xen users - Jan 2009 - [XEN 3.3 - PCI passthrough] - interaction latencies with /dev/ttyUSB0

Hi everybody!
I''m a pretty new XEN user (3.3) and I need your help on a weird
situation.
In fact I cannot conclude if it could be a XEN problem or not. So I hope you
will be able to guide me.
You will find below what my project is and as you guess ...the problem!


OBJECTIVE :
==========   - validate XEN solution for potential projects
   - setup a paravirtualized environment
   - embedd on a domU, software interacting with a USB connected desktop
RFID reader


ENVIRONMENT :
============   - XEN 3.3
   - paravirtualized environment
   - dom0 : linux 2.6.24-19-xen (ubuntu) installed with no graphical
interface
   - domU : linux 2.6.24-19-xen (ubuntu) installed with gnome and vncserver
   - PCI passthrough to dedicate all USB controllers to the single domU
   - domU is installed in a file-backed VBDs

BASIC RESULT:
============   - dom0 and domU boots well
   - PCI passthrough works
   - system is stable
   - on domU, lsusb shows the device connected

USB DEVICE :
===========   - RFID desktop reader (STID)
   - serial communication ("/dev/ttyUSB0) 9600 8 n 2
   - very short instructions sent and received asynchronously (5 bytes) ;
minimum system stress...

EMBEDDED SOFTWARE :
==================   - very basic test software that polls the device for an
event
   - if event detected, send a sound instruction to the device
   - communication with the device is done via the serial dev file
"/dev/ttyUSB0"
   - tested on non virtualized environment : no latency on execution and
test succeeds


PROBLEM :
========   - running the soft from the console (xm console) without GUI
introduces
hudge execution latencies
   - if we connect our appliance from a remote PC via vncviewer, gnome is
displayed and run the same soft from an Xterm makes the execution much more
faster
     and coud almost be compared to non-virtualized environment perfs.
   - Note that I''ve tried to connect via SSH without and then with X
exporting, and latencies are still noticed
   - it seems that latencies are only seen when interacting with the device
   - Finally, we do not want to embedd any window manager such as gnome/KDE
for footprints concerns
   - soft have been written in C and JAVA but results are the same

HELPPPPP:
========As you see, it is a very strange problem and I can''t find out
who''s
responsible for that :
   - is it a XEN problem?
   - is it a console problem?
   - is it a PCI passthrough problem?
   - does gnome/vnc introduces implicits optimizations?


Furthermore, in order to find some clues:
   - can you confirm me that XEN 3.3 does not support USB2.0 (not really
needed here ...)
   - I often have a system crash connecting USB mass storage devices (even
dom0 get frozen) ; is it a known bug? I haven''t found anything on
forums.
   - based on what I''ve read (and test) I cannot use newer XEN kernels
if I
want to keep PCI passthrough working
   - are there some "best practices" to optimize USB passthrough

I''ve process a lot of diff, compare, readings and I have to tell you
I''ve no
more idea to solve this silly problem. Any help is thus welcomed!!

Thanks a lot for your help!

Cordially,
Jérémie


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Thanks for your help Ken,

I''m gonna have a look to the USB over IP solution.
Nevertheless, in addition to my post, I ran further tests and noticed that
when processing test on dom0 (no passthrough),  everything works fine (no
more latencies).
So, I''m about to conclude that the problem is due to PCI passthrough.
I''m a litlle bit surprised, cause I thought that doing this would have
given
direct USB access.

Is it known that pci passthrough pull down performances?

Furthertmore, I''ve read that XEN 3.4 will introduce virtualized USB
driver.
So can we concluide that no more support will be done on PCI passthrough?

Finally, do you have any perf issues running USB over IP?

Tks a lot!

Jérémie

2009/1/9 Ken Cobler <kcobler@coblercorp.com>
> jer jer wrote:
>
>> Hi everybody!
>> I''m a pretty new XEN user (3.3) and I need your help on a
weird situation.
>> In fact I cannot conclude if it could be a XEN problem or not. So I
hope
>> you will be able to guide me.
>> You will find below what my project is and as you guess ...the problem!
>>
>>
>> OBJECTIVE :
>> ==========>>   - validate XEN solution for potential projects
>>   - setup a paravirtualized environment
>>   - embedd on a domU, software interacting with a USB connected desktop
>> RFID reader
>>
>>
>> ENVIRONMENT :
>> ============>>   - XEN 3.3
>>   - paravirtualized environment
>>   - dom0 : linux 2.6.24-19-xen (ubuntu) installed with no graphical
>> interface
>>   - domU : linux 2.6.24-19-xen (ubuntu) installed with gnome and
vncserver
>>   - PCI passthrough to dedicate all USB controllers to the single domU
>>   - domU is installed in a file-backed VBDs
>>
>> BASIC RESULT:
>> ============>>   - dom0 and domU boots well
>>   - PCI passthrough works
>>   - system is stable
>>   - on domU, lsusb shows the device connected
>>
>> USB DEVICE :
>> ===========>>   - RFID desktop reader (STID)
>>   - serial communication ("/dev/ttyUSB0) 9600 8 n 2
>>   - very short instructions sent and received asynchronously (5 bytes)
;
>> minimum system stress...
>>
>> EMBEDDED SOFTWARE :
>> ==================>>   - very basic test software that polls the
device for an event
>>   - if event detected, send a sound instruction to the device
>>   - communication with the device is done via the serial dev file
>> "/dev/ttyUSB0"
>>   - tested on non virtualized environment : no latency on execution and
>> test succeeds
>>
>>
>> PROBLEM :
>> ========>>   - running the soft from the console (xm console)
without GUI introduces
>> hudge execution latencies
>>   - if we connect our appliance from a remote PC via vncviewer, gnome
is
>> displayed and run the same soft from an Xterm makes the execution much
more
>> faster
>>     and coud almost be compared to non-virtualized environment perfs.
>>   - Note that I''ve tried to connect via SSH without and then
with X
>> exporting, and latencies are still noticed
>>   - it seems that latencies are only seen when interacting with the
device
>>   - Finally, we do not want to embedd any window manager such as
gnome/KDE
>> for footprints concerns
>>   - soft have been written in C and JAVA but results are the same
>>
>> HELPPPPP:
>> ========>> As you see, it is a very strange problem and I
can''t find out who''s
>> responsible for that :
>>   - is it a XEN problem?
>>   - is it a console problem?
>>   - is it a PCI passthrough problem?
>>   - does gnome/vnc introduces implicits optimizations?
>>
>>
>> Furthermore, in order to find some clues:
>>   - can you confirm me that XEN 3.3 does not support USB2.0 (not really
>> needed here ...)
>>   - I often have a system crash connecting USB mass storage devices
(even
>> dom0 get frozen) ; is it a known bug? I haven''t found anything
on forums.
>>   - based on what I''ve read (and test) I cannot use newer XEN
kernels if I
>> want to keep PCI passthrough working
>>   - are there some "best practices" to optimize USB
passthrough
>>
>> I''ve process a lot of diff, compare, readings and I have to
tell you I''ve
>> no more idea to solve this silly problem. Any help is thus welcomed!!
>>
>> Thanks a lot for your help!
>>
>> Cordially,
>> Jérémie
>>
> I''ve had difficulty with USB and Xen (using Windows HVM guests). 
I''ve
> tried the PCI approach as well.
>
> Xen uses QEMU to run HVM guests. QEMU can only allow accept one USB device
> to the HVM guest.  I was unable to successfully allocate a complete USB
host
> controller to the domU (that was recognized by domU).
> PCI approach requires that dom0 cannot grab a PCI card and leave it for the
> domU (which is handled through boot kernel parameters).  I was never
> successful in getting that to work from the domU side.  I could get dom0 to
> release the PCI device, but could not get domU to recognize the device.
>
> My latest approach that I am working on is using USB over IP devices.
>  Since both dom0 and domU have a network interface, I could attach the USB
> device to dom0 and let domU get it through the USB over IP.   There are a
> couple of implementations for Windows and some for Linux, but, very limited
> on ability to do both (which is what I need for my application).
>
> In your case, since both dom0 and domU are Linux, you might consider this
> solution from IncentivePros
> http://www.incentivespro.com/downloads.html#usb-server
>
> The USB server + USB Client Beta (at the bottom of the page) is an open
> source Linux to Linux USB over IP solution.
>
> Ken Cobler
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

jer jer wrote:
> Thanks for your help Ken,
>
> I''m gonna have a look to the USB over IP solution.
> Nevertheless, in addition to my post, I ran further tests and noticed 
> that when processing test on dom0 (no passthrough),  everything works 
> fine (no more latencies).
> So, I''m about to conclude that the problem is due to PCI
passthrough.
> I''m a litlle bit surprised, cause I thought that doing this would
have
> given direct USB access.
>
> Is it known that pci passthrough pull down performances?
>
> Furthertmore, I''ve read that XEN 3.4 will introduce virtualized
USB
> driver. So can we concluide that no more support will be done on PCI 
> passthrough?
>
> Finally, do you have any perf issues running USB over IP?
>
> Tks a lot!
>
It would be great to see complete USB virtualization in Xen. 

However, I don''t think that would take away from PCI passthrough.  
Simply, there are other devices that are PCI devices and not USB, that 
could be passed to a domU.


With regards to USB over IP, I have only implemented a Digiboard 
hardware product called AnywhereUSB, 
http://www.digi.com/products/usb/anywhereusb.jsp

The only concern someone may have about AnywhereUSB is it will only 
support USB 1.0 speeds for all devices.  In my application, that was not 
an issue.


Ken Cobler

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, Jan 12, 2009 at 8:16 AM, Ken Cobler <kcobler@coblercorp.com>
wrote:
> It would be great to see complete USB virtualization in Xen.
> However, I don''t think that would take away from PCI passthrough. 
Simply,
> there are other devices that are PCI devices and not USB, that could be
> passed to a domU.
>
Maybe these links will help:
http://www.virtuatopia.com/index.php/Adding_USB_Devices_to_a_Xen_HVM_domainU_Guest
http://www.linux-kvm.com/content/virt-manager-and-usb-device-support

Yes, the second one is based on KVM, but Xen HVM and KVM both use qemu
behind the scenes.

Cheers,
Todd

-- 
Todd Deshane
http://todddeshane.net
http://runningxen.com

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Todd Deshane wrote:
> On Mon, Jan 12, 2009 at 8:16 AM, Ken Cobler <kcobler@coblercorp.com>
wrote:
>
>   
>> It would be great to see complete USB virtualization in Xen.
>> However, I don''t think that would take away from PCI
passthrough.  Simply,
>> there are other devices that are PCI devices and not USB, that could be
>> passed to a domU.
>>
>>     
>
> Maybe these links will help:
>
http://www.virtuatopia.com/index.php/Adding_USB_Devices_to_a_Xen_HVM_domainU_Guest
> http://www.linux-kvm.com/content/virt-manager-and-usb-device-support
>
> Yes, the second one is based on KVM, but Xen HVM and KVM both use qemu
> behind the scenes.
>
> Cheers,
> Todd
>
>   
There''s a limitation in QEMU for only 1 USB device to be passed 
(regardless of passing through vm config or to a running domU guest).  
If the domU is passed the usb tablet device to make smoother mouse 
support, than no other USB device can be passed.

Ken Cobler


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

The fact that you say it works fine with X-Windows running and badly
from the console makes me think you''ve got a software problem
unrelated to Xen.  If Xen+X-Windows  works well, then it''s hard to
imagine it being a problem with the USB/PCI.  Does it work well at the
command line outside of Xen?  If so, I''d spend time looking at the
difference between console+xen, gui+xen, console-xen etc, rather than
the USB & PCI passthrough.   If it doesn''t work well at the console
outside of Xen, then you know it''s not a Xen problem.

-Dylan

On Fri, Jan 9, 2009 at 6:29 AM, jer jer <jeremie.garcia.trash@gmail.com>
wrote:
> Hi everybody!
> I''m a pretty new XEN user (3.3) and I need your help on a weird
situation.
> In fact I cannot conclude if it could be a XEN problem or not. So I hope
you
> will be able to guide me.
> You will find below what my project is and as you guess ...the problem!
>
>
> OBJECTIVE :
> ==========>    - validate XEN solution for potential projects
>    - setup a paravirtualized environment
>    - embedd on a domU, software interacting with a USB connected desktop
> RFID reader
>
>
> ENVIRONMENT :
> ============>    - XEN 3.3
>    - paravirtualized environment
>    - dom0 : linux 2.6.24-19-xen (ubuntu) installed with no graphical
> interface
>    - domU : linux 2.6.24-19-xen (ubuntu) installed with gnome and vncserver
>    - PCI passthrough to dedicate all USB controllers to the single domU
>    - domU is installed in a file-backed VBDs
>
> BASIC RESULT:
> ============>    - dom0 and domU boots well
>    - PCI passthrough works
>    - system is stable
>    - on domU, lsusb shows the device connected
>
> USB DEVICE :
> ===========>    - RFID desktop reader (STID)
>    - serial communication ("/dev/ttyUSB0) 9600 8 n 2
>    - very short instructions sent and received asynchronously (5 bytes) ;
> minimum system stress...
>
> EMBEDDED SOFTWARE :
> ==================>    - very basic test software that polls the device
for an event
>    - if event detected, send a sound instruction to the device
>    - communication with the device is done via the serial dev file
> "/dev/ttyUSB0"
>    - tested on non virtualized environment : no latency on execution and
> test succeeds
>
>
> PROBLEM :
> ========>    - running the soft from the console (xm console) without
GUI introduces
> hudge execution latencies
>    - if we connect our appliance from a remote PC via vncviewer, gnome is
> displayed and run the same soft from an Xterm makes the execution much more
> faster
>      and coud almost be compared to non-virtualized environment perfs.
>    - Note that I''ve tried to connect via SSH without and then with
X
> exporting, and latencies are still noticed
>    - it seems that latencies are only seen when interacting with the device
>    - Finally, we do not want to embedd any window manager such as gnome/KDE
> for footprints concerns
>    - soft have been written in C and JAVA but results are the same
>
> HELPPPPP:
> ========> As you see, it is a very strange problem and I can''t
find out who''s
> responsible for that :
>    - is it a XEN problem?
>    - is it a console problem?
>    - is it a PCI passthrough problem?
>    - does gnome/vnc introduces implicits optimizations?
>
>
> Furthermore, in order to find some clues:
>    - can you confirm me that XEN 3.3 does not support USB2.0 (not really
> needed here ...)
>    - I often have a system crash connecting USB mass storage devices (even
> dom0 get frozen) ; is it a known bug? I haven''t found anything on
forums.
>    - based on what I''ve read (and test) I cannot use newer XEN
kernels if I
> want to keep PCI passthrough working
>    - are there some "best practices" to optimize USB passthrough
>
> I''ve process a lot of diff, compare, readings and I have to tell
you I''ve no
> more idea to solve this silly problem. Any help is thus welcomed!!
>
> Thanks a lot for your help!
>
> Cordially,
> Jérémie
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Thanks all for your help!
In fact, I''ve tried the soft outside XEN from console and it works
fine.
If I resume :
  - no XEN ==> works fine from console or GUI
  - with XEN on dom0 ==> works fine from console or GUI
  - with XEN on domU with GUI (VNC) ==> works ok but is slower than previous
environments
  - with XEN on domU with console ==> huge latencies
  - with XEN on domU SSH connected ==> huge latencies


2009/1/12 Dylan Martin <dmartin@sccd.ctc.edu>

The fact that you say it works fine with X-Windows running and
badly
> from the console makes me think you''ve got a software problem
> unrelated to Xen.  If Xen+X-Windows  works well, then it''s hard to
> imagine it being a problem with the USB/PCI.  Does it work well at the
> command line outside of Xen?  If so, I''d spend time looking at the
> difference between console+xen, gui+xen, console-xen etc, rather than
> the USB & PCI passthrough.   If it doesn''t work well at the
console
> outside of Xen, then you know it''s not a Xen problem.
>
> -Dylan
>
> On Fri, Jan 9, 2009 at 6:29 AM, jer jer
<jeremie.garcia.trash@gmail.com>
> wrote:
> > Hi everybody!
> > I''m a pretty new XEN user (3.3) and I need your help on a
weird
> situation.
> > In fact I cannot conclude if it could be a XEN problem or not. So I
hope
> you
> > will be able to guide me.
> > You will find below what my project is and as you guess ...the
problem!
> >
> >
> > OBJECTIVE :
> > ==========> >    - validate XEN solution for potential projects
> >    - setup a paravirtualized environment
> >    - embedd on a domU, software interacting with a USB connected
desktop
> > RFID reader
> >
> >
> > ENVIRONMENT :
> > ============> >    - XEN 3.3
> >    - paravirtualized environment
> >    - dom0 : linux 2.6.24-19-xen (ubuntu) installed with no graphical
> > interface
> >    - domU : linux 2.6.24-19-xen (ubuntu) installed with gnome and
> vncserver
> >    - PCI passthrough to dedicate all USB controllers to the single
domU
> >    - domU is installed in a file-backed VBDs
> >
> > BASIC RESULT:
> > ============> >    - dom0 and domU boots well
> >    - PCI passthrough works
> >    - system is stable
> >    - on domU, lsusb shows the device connected
> >
> > USB DEVICE :
> > ===========> >    - RFID desktop reader (STID)
> >    - serial communication ("/dev/ttyUSB0) 9600 8 n 2
> >    - very short instructions sent and received asynchronously (5
bytes) ;
> > minimum system stress...
> >
> > EMBEDDED SOFTWARE :
> > ==================> >    - very basic test software that polls
the device for an event
> >    - if event detected, send a sound instruction to the device
> >    - communication with the device is done via the serial dev file
> > "/dev/ttyUSB0"
> >    - tested on non virtualized environment : no latency on execution
and
> > test succeeds
> >
> >
> > PROBLEM :
> > ========> >    - running the soft from the console (xm console)
without GUI
> introduces
> > hudge execution latencies
> >    - if we connect our appliance from a remote PC via vncviewer, gnome
is
> > displayed and run the same soft from an Xterm makes the execution much
> more
> > faster
> >      and coud almost be compared to non-virtualized environment perfs.
> >    - Note that I''ve tried to connect via SSH without and then
with X
> > exporting, and latencies are still noticed
> >    - it seems that latencies are only seen when interacting with the
> device
> >    - Finally, we do not want to embedd any window manager such as
> gnome/KDE
> > for footprints concerns
> >    - soft have been written in C and JAVA but results are the same
> >
> > HELPPPPP:
> > ========> > As you see, it is a very strange problem and I
can''t find out who''s
> > responsible for that :
> >    - is it a XEN problem?
> >    - is it a console problem?
> >    - is it a PCI passthrough problem?
> >    - does gnome/vnc introduces implicits optimizations?
> >
> >
> > Furthermore, in order to find some clues:
> >    - can you confirm me that XEN 3.3 does not support USB2.0 (not
really
> > needed here ...)
> >    - I often have a system crash connecting USB mass storage devices
> (even
> > dom0 get frozen) ; is it a known bug? I haven''t found
anything on forums.
> >    - based on what I''ve read (and test) I cannot use newer
XEN kernels if
> I
> > want to keep PCI passthrough working
> >    - are there some "best practices" to optimize USB
passthrough
> >
> > I''ve process a lot of diff, compare, readings and I have to
tell you I''ve
> no
> > more idea to solve this silly problem. Any help is thus welcomed!!
> >
> > Thanks a lot for your help!
> >
> > Cordially,
> > Jérémie
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xensource.com
> > http://lists.xensource.com/xen-users
> >
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users