Hi I''m running the following setup: - Gentoo Linux 2.6.21-xen (on both dom0 and all domUs) - Xen 3.2.1 - Firewall (shorewall) running in a domU - Bridged networking The problem I''m having is that whenever i download a large file (3mb+) from the internet on the dom0 any networking on the host machine dies and the dom0 and all its domU''s become completely unavailible for about 5 minutes whereafter the network comes back up again and everything starts working. I have no problems transfering large files between the dom0 and the rest of my LAN. Only data transfers between the dom0 and a domU seems to be problematic. In the past i have had similar problems when transfering files between the dom0 and a domU via a NFS. I solved them by specifying rsize=1536 wsize=1536 when mounting the NFS volume on a domU. I have tried reducing the MTU from 1500 til 1400 on all interfaces without any luck. Does anyone have a clue of whats going on? Thanks in advance Truls Asheim _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Joseph L. Casale
2008-Aug-25 02:08 UTC
RE: [Xen-users] Network dies when downloading on dom0
>I''m running the following setup: > - Gentoo Linux 2.6.21-xen (on both dom0 and all domUs) > - Xen 3.2.1 > - Firewall (shorewall) running in a domU > - Bridged networkingWith the exception of Dom0/U OS, I run the exact setup :)>The problem I''m having is that whenever i download a large file (3mb+) >from the internet on the dom0 any networking on the host machine dies >and the dom0 and all its domU''s become completely unavailible for about >5 minutes whereafter the network comes back up again and everything >starts working.I had these same issues, do you run iptables in Dom0? I recall an issue when doing so but I never tested the fix I just disabled it as Dom0 was physically isolated. jlc _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Aug 24, 2008, at 8:39 PM, Truls Asheim wrote:> The problem I''m having is that whenever i download a large file (3mb+) > from the internet on the dom0 any networking on the host machine dies > and the dom0 and all its domU''s become completely unavailible for > about > 5 minutes whereafter the network comes back up again and everything > starts working. > > Does anyone have a clue of whats going on?I have a similar setup on Ubuntu 8.0.4 and I am experiencing the same problem with the exact same symptoms. What I have observed is that if the the rate of the incoming traffic is above certain threshold, ping times to the public internet starts to build up. So I am assuming there is some problem with VIF code, where too much traffic will incurr into backlogs until it fills up whatever queues there are. After the queues are full, you have to wait for a long while until everything is back to normal. The worse part, it that while the VIFs are backlogged and unresponsive, you cannot do something major like destroying the domUs or rebooting the dom0 because some of the processes seem to be in uninterruptible sleep and then the entire system locks up warranting a cold reboot (Which in my case caused a resync of our RAID array, enough reason for me to avoid trying to troubleshoot this problem even more). I have managed to mitigate the problem by limiting the transfer rate on all the VIF, using the ''rate'' directive in all the config files. Also, I have enabled QoS on Shorewall to limit the traffic rate on the public interface. In my case our connection is a 7Mbps/896kbps DSL line and by trial and error I determined that with a limit of 5Mbps I can avoid the DoS at least. I haven''t been able to find a solution to this problem and get the full download rate of our DSL line. Also, I don''t know how well the rate limiting code for VIF in Xen works, because using that option alone, didn''t fix much, just prolonged the amount of traffic that could go through the interface until it died. Any ideas on how to solve this issue are welcome. Pepe _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Aug 24, 2008, at 9:08 PM, Joseph L. Casale wrote:> I had these same issues, do you run iptables in Dom0? I recall an > issue > when doing so but I never tested the fix I just disabled it as Dom0 > was > physically isolated.When you say disable, do you mean remove any rules or disable it from the Kernel? I have the same problem without any rules in the dom0 (Except the ones specified by Xen; to accept all traffic for the VIFs), so that doesn''t seem to help much. Pepe _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Joseph L. Casale
2008-Aug-25 20:02 UTC
RE: [Xen-users] Network dies when downloading on dom0
>When you say disable, do you mean remove any rules or disable it from >the Kernel?I didn''t bother to test the fix in the wiki, I just #chkconfig ip(6)tables off.>I have the same problem without any rules in the dom0 (Except the ones >specified by Xen; to accept all traffic for the VIFs), so that doesn''t >seem to help much.Specified *by* xen? Looking at your earlier post you are running ubuntu, and I have no experience with this. Do they package xen with some special firewall rules? Just stop iptables and run some tests... jlc _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Aug 25, 2008, at 3:02 PM, Joseph L. Casale wrote:> Specified *by* xen? Looking at your earlier post you are running > ubuntu, and I have > no experience with this. Do they package xen with some special > firewall rules?Example: ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in vif2.0> Just stop iptables and run some tests...By stopping you mean clearing out all the rules. Not all the distros provide init-style scripts to manage IPTables, Ubuntu doesn''t at least. Pepe _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Joseph L. Casale
2008-Aug-25 20:36 UTC
RE: [Xen-users] Network dies when downloading on dom0
>By stopping you mean clearing out all the rules. Not all the distros >provide init-style scripts to manage IPTables, Ubuntu doesn''t at least. > >PepeYeah, I don''t know squat about ubuntu... Maybe someone can chime in? All I know is my RH experience with iptabels on Dom0 is an issue unless its setup right, but I never checked if the fix in the wiki was accurate I just disabled it completely as Dom0 was physically segmented. jlc _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 25/08/08 14:36, Joseph L. Casale wrote:> >By stopping you mean clearing out all the rules. Not all the distros > >provide init-style scripts to manage IPTables, Ubuntu doesn''t at least. > > > >Pepe > > Yeah, I don''t know squat about ubuntu... Maybe someone can chime in? > All I know is my RH experience with iptabels on Dom0 is an issue unless > its setup right, but I never checked if the fix in the wiki was accurate > I just disabled it completely as Dom0 was physically segmented. > > jlc >I have the same iptables rules created by xen as Pepe Barbe. I have tried flushing them and I have tried unloading the iptables kernel modules but neither of those things seems to make any diffrence. Is there any chance I could make you post your RH iptables initscript so we can see excacly what it does? And also, what fix in the wiki are you talking about? (I dont seem to be able to access the xensource page right now) Truls _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Aug 25, 2008, at 7:54 PM, Truls Asheim wrote:> And also, what fix in the wiki are you talking about? (I dont seem > to be > able to access the xensource page right now)I am guessing here, but have you disabed TX checksum offloading? It doesn''t work for me, but you should try it: > ethtool -K eth0 tx off Pepe _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, The command: ethtool -K eth0 tx off I''ve got the same problem, it worked for me. But the difference was that this problem happened randomly. On Aug 25, 2008, at 10:10 PM, Pepe Barbe wrote:> > On Aug 25, 2008, at 7:54 PM, Truls Asheim wrote: > >> And also, what fix in the wiki are you talking about? (I dont seem >> to be >> able to access the xensource page right now) > > I am guessing here, but have you disabed TX checksum offloading? It > doesn''t work for me, but you should try it: > > > ethtool -K eth0 tx off > > Pepe > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello, Did you run that command inside of Dom0 or DomU? Thanks Chris On Tue, Aug 26, 2008 at 8:35 AM, Daniel Kühl <dklima@gmail.com> wrote:> Hi, > > The command: ethtool -K eth0 tx off > > I''ve got the same problem, it worked for me. But the difference was that > this problem happened randomly. > > > > On Aug 25, 2008, at 10:10 PM, Pepe Barbe wrote: > > >> On Aug 25, 2008, at 7:54 PM, Truls Asheim wrote: >> >> And also, what fix in the wiki are you talking about? (I dont seem to be >>> able to access the xensource page right now) >>> >> >> I am guessing here, but have you disabed TX checksum offloading? It >> doesn''t work for me, but you should try it: >> >> > ethtool -K eth0 tx off >> >> Pepe >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, just passing by this thread, I''ld suggest setting tx off on both dom0 and domU. Another problem I noticed sometimes on a few platforms on high dom0 traffic are missing interrupts. On SMP machines I could always solve this by installing irqbalance inside dom0. cheers, Stephan Chris Holland schrieb:> Hello, > > Did you run that command inside of Dom0 or DomU? > > Thanks > Chris > > On Tue, Aug 26, 2008 at 8:35 AM, Daniel Kühl <dklima@gmail.com> wrote: > >> Hi, >> >> The command: ethtool -K eth0 tx off >> >> I''ve got the same problem, it worked for me. But the difference was that >> this problem happened randomly. >> >> >> >> On Aug 25, 2008, at 10:10 PM, Pepe Barbe wrote: >> >> >>> On Aug 25, 2008, at 7:54 PM, Truls Asheim wrote: >>> >>> And also, what fix in the wiki are you talking about? (I dont seem to be >>>> able to access the xensource page right now) >>>> >>> I am guessing here, but have you disabed TX checksum offloading? It >>> doesn''t work for me, but you should try it: >>> >>>> ethtool -K eth0 tx off >>> Pepe >>> >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com >>> http://lists.xensource.com/xen-users >>> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> > > > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- Stephan Seitz Senior System Administrator *netz-haut* e.K. multimediale kommunikation zweierweg 22 97074 würzburg fon: +49 931 2876247 fax: +49 931 2876248 web: www.netz-haut.de <http://www.netz-haut.de/> registriergericht: amtsgericht würzburg, hra 5054 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Aug 26, 2008, at 10:21 AM, Stephan Seitz wrote:> just passing by this thread, I''ld suggest setting tx off on both dom0 > and domU.I always did it on domUs. I tried on dom0 and it improved things marginally. The network holds longer for high traffic but the delay still manages to increase from 80 ms up to 1000ms. Usually after that it just dies.> Another problem I noticed sometimes on a few platforms on high dom0 > traffic are missing interrupts. On SMP machines I could always solve > this by installing irqbalance inside dom0.I tried this, but didn''t help either. Similar behavior as before. What can I do to see if I am missing interrupts? The only thing that manages to keep our ping delay times constant about 120ms (Which is the primary symptom of the network about to die) is to cap the traffic in our domU router. Something interesting I noticed, is that I tried to limit the speed of the VIF between the router and the dom0 and it didn''t help. Only limiting the speed on the WAN interface which is Physical interface hidden from the dom0 and added to the domU. Could there be some issue because it is not a Virtual Interface? A driver problem perhaps? Thanks, Pepe _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
What kinds of speed restrictions are needed to keep things going? My internet connection is 10Mbit/s with 40Mbit/s burst speed and I don''t want to loose that speed. Thanks Chris On Tue, Aug 26, 2008 at 11:40 AM, Pepe Barbe <elventear@gmail.com> wrote:> > On Aug 26, 2008, at 10:21 AM, Stephan Seitz wrote: > > just passing by this thread, I''ld suggest setting tx off on both dom0 >> and domU. >> > > I always did it on domUs. I tried on dom0 and it improved things > marginally. The network holds longer for high traffic but the delay still > manages to increase from 80 ms up to 1000ms. Usually after that it just > dies. > > Another problem I noticed sometimes on a few platforms on high dom0 >> traffic are missing interrupts. On SMP machines I could always solve >> this by installing irqbalance inside dom0. >> > > I tried this, but didn''t help either. Similar behavior as before. What can > I do to see if I am missing interrupts? > > The only thing that manages to keep our ping delay times constant about > 120ms (Which is the primary symptom of the network about to die) is to cap > the traffic in our domU router. > > Something interesting I noticed, is that I tried to limit the speed of the > VIF between the router and the dom0 and it didn''t help. Only limiting the > speed on the WAN interface which is Physical interface hidden from the dom0 > and added to the domU. Could there be some issue because it is not a Virtual > Interface? A driver problem perhaps? > > Thanks, > Pepe >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Aug 26, 2008, at 2:07 PM, Chris Holland wrote:> What kinds of speed restrictions are needed to keep things going? My > internet connection is 10Mbit/s with 40Mbit/s burst speed and I > don''t want > to loose that speed.I suggest that you do some testing to see if you observe similar things as I do. My observations are highly empirical and I haven''t really had time to figure out things, nor I have a clue what else to try currently. As I said, through trial and error I came up with a connection limit that avoids killing the network. I don''t know if the same values would apply to you. Pepe _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users