Hi! Since my upgrade from xen 2-testing to xen 3-testing I have the problem that domain0 consumes more and more memory on domU networktraffic. It is about 50 bytes per packet that gets lost. I reproduced the bug on another machine. In normal bridged setup all works fine. I changed the setup in the following way: - added an ip address to the bridge (in another subnet (B)) - changed domU ip address to a net B address - added masquerading for eth0 in the postrouting-chain in dom0 After that I pinged a host outside this machine. Sending and receiving 100.000 pakets results to a dom0-memory-consumption increasement of about 5 MB! I have to restart this machine once a day for not running out of memory. What can I do? Patrick _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Sending and receiving 100.000 pakets results to a > dom0-memory-consumption increasement of about 5 MB!When you''ve shipped a few million packets through the machine please post the output of /proc/slabinfo Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I changed the setup in the following way: > - added an ip address to the bridge (in another subnet (B)) > - changed domU ip address to a net B address > - added masquerading for eth0 in the postrouting-chain in dom0What iptables rules do you use to set this up? Is the gateway address for domU the net B address interface you add to the bridge? It''s worth trying ''ethtool -K eth0 tx off'' in domU to turn off checksum offload, just in case that is causing problems. -- Keir> After that I pinged a host outside this machine. > > Sending and receiving 100.000 pakets results to a > dom0-memory-consumption increasement of about 5 MB! > > I have to restart this machine once a day for not running out of > memory. > > What can I do? > > Patrick > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>> I changed the setup in the following way: >> - added an ip address to the bridge (in another subnet (B)) >> - changed domU ip address to a net B address >> - added masquerading for eth0 in the postrouting-chain in dom0 > > What iptables rules do you use to set this up?I used "iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE" for masquerading.> Is the gateway address for domU the net B address interface you add to > the bridge?Yes that right, i forgot to mention that. I have one official address on dom0 eth0 and a privat class B network for the domU''s. The address used on the bridge is from this private network and the gateway for the domU''s> It''s worth trying ''ethtool -K eth0 tx off'' in domU to turn off > checksum offload, just in case that is causing problems.Checksum offload is turned off in all domU''s. The problem only appears if the pakets are masqueraded. Pinging the gateway-address (the one on the bridge-if) or the dom0-eth0-address causes no memory consumption Patrick _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 4 Jan 2006, at 14:47, Patrick Scharrenberg wrote:>> What iptables rules do you use to set this up? > I used "iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE" for > masquerading. > >> Is the gateway address for domU the net B address interface you add >> to the bridge? > Yes that right, i forgot to mention that. > > I have one official address on dom0 eth0 and a privat class B network > for the domU''s. > The address used on the bridge is from this private network and the > gateway for the domU''sWhat interface do you attach the domU gateway address to? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Is the gateway address for domU the net B address interface you add >>> to the bridge? >> Yes that right, i forgot to mention that. >> >> I have one official address on dom0 eth0 and a privat class B network >> for the domU''s. >> The address used on the bridge is from this private network and the >> gateway for the domU''s > > What interface do you attach the domU gateway address to? >it''s the xen-bridge-interface where I set the domU gateway address. Patrick _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 4 Jan 2006, at 15:35, Patrick Scharrenberg wrote:>>>> Is the gateway address for domU the net B address interface you add >>>> to the bridge? >>> Yes that right, i forgot to mention that. >>> >>> I have one official address on dom0 eth0 and a privat class B network >>> for the domU''s. >>> The address used on the bridge is from this private network and the >>> gateway for the domU''s >> >> What interface do you attach the domU gateway address to? >> > it''s the xen-bridge-interface where I set the domU gateway address.I can''t repro the problem. My setup is that the domU vif is the only interface I attach to the bridge (eth0 is not attached). I give the bridge address 10.0.0.1, enable IP forwarding, and set up IP masquerade on eth0. I set domU IP address to 10.0.02 gw 10.0.0.1 and disable transmit checksumming. I then did a TCP bulk transfer to an external host (a few million packets) and observed no memory leak in domain 0. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>> Sending and receiving 100.000 pakets results to a >> dom0-memory-consumption increasement of about 5 MB! >> > > When you''ve shipped a few million packets through the machine please > post the output of /proc/slabinfo >Okay, I shipped arround 3million packets and here are the results (see attached files). slabinfo_a is the output before the tests and slabinfo_b is after the tests. Here the output of free before.. total used free shared buffers cached Mem: 700416 61288 639128 0 2984 19268 -/+ buffers/cache: 39036 661380 Swap: 0 0 0 and after the test: total used free shared buffers cached Mem: 700416 258388 442028 0 2996 19392 -/+ buffers/cache: 236000 464416 Swap: 0 0 0 I used floodping to generate the packets and found that the interface stats for peth0 in dom0 are not correct either. domU eth0 shows: eth0 Link encap:Ethernet HWaddr 00:16:3E:05:B2:64 inet addr:192.168.37.2 Bcast:192.168.37.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3000075 errors:0 dropped:0 overruns:0 frame:0 TX packets:3000028 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:294006770 (280.3 MiB) TX bytes:294001488 (280.3 MiB) rx bytes and tx bytes is nearly the same. But in dom0: peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF UP BROADCAST RUNNING NOARP MULTICAST MTU:1500 Metric:1 RX packets:3000123 errors:0 dropped:0 overruns:0 frame:0 TX packets:3000078 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:294010467 (280.3 MiB) TX bytes:306006234 (291.8 MiB) Interrupt:17 Memory:e8000000-0 rx bytes and tx bytes differ in about 11 MB! In dom0 nothing else ran, and on the (crossovered) pinged host, rx and tx are also the same: eth1 Link encap:Ethernet HWaddr 00:0E:0C:22:0E:55 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:4700954 errors:0 dropped:0 overruns:0 frame:0 TX packets:4701065 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:479480546 (457.2 MiB) TX bytes:479508173 (457.2 MiB) Base address:0x9400 Memory:e6000000-e6020000 (the higher values result from a second test) I hope this helps!? Patrick _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser schrieb:> > On 4 Jan 2006, at 15:35, Patrick Scharrenberg wrote: > >>>>> Is the gateway address for domU the net B address interface you add >>>>> to the bridge? >>>> Yes that right, i forgot to mention that. >>>> >>>> I have one official address on dom0 eth0 and a privat class B network >>>> for the domU''s. >>>> The address used on the bridge is from this private network and the >>>> gateway for the domU''s >>> >>> What interface do you attach the domU gateway address to? >>> >> it''s the xen-bridge-interface where I set the domU gateway address. > > I can''t repro the problem. My setup is that the domU vif is the only > interface I attach to the bridge (eth0 is not attached). I give the > bridge address 10.0.0.1, enable IP forwarding, and set up IP > masquerade on eth0. I set domU IP address to 10.0.02 gw 10.0.0.1 and > disable transmit checksumming. I then did a TCP bulk transfer to an > external host (a few million packets) and observed no memory leak in > domain 0. >Hmm.. thats strange, since I did nothing special. Here once more what I did, maybe (hopefully) we misunderstood: eth0 has my internet ip address. I used the network-bridge-script to setup the (xen-) networking, set up ip 192.168.37.254 on xenbr0. Then I startet domU. After that I added masquerading to eth0. I have the following interfaces in the bridge: peth0, vif0.0, vif1.0 I''m using debian sarge as os. Maybe they set something special in their networking-scripts? It is a test-machine, so i can give you the root-pw, if nothing helps. Patrick> -- Keir > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 4 Jan 2006, at 16:22, Patrick Scharrenberg wrote:> > I used floodping to generate the packets and found that the interface > stats for peth0 in dom0 are not correct either.If all your domU interfaces will be NATed then there''s no need to be using Ethernet bridging at all. You probably don''t want to use the /etc/xen/scripts/network-bridge and vif-bridge scripts, replacing them with your own which would place a domU-specific virtual gateway address on each vifx.0 in dom0. Then enable IP forwarding and IP masquerade on eth0 and you should be away, with no Ethernet bridging required. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 4 Jan 2006, at 16:39, Patrick Scharrenberg wrote:> I used the network-bridge-script to setup the (xen-) networking, > set up ip 192.168.37.254 on xenbr0. > Then I startet domU. > After that I added masquerading to eth0.Possibly you can sidestep the whole issue, and end up with faster and simpler networking setup in dom0, by not using network-bridge and vif-bridge scripts at all. Instead make your own (much much simpler) scripts to set up NAT. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hey Keir!>> I used the network-bridge-script to setup the (xen-) networking, >> set up ip 192.168.37.254 on xenbr0. >> Then I startet domU. >> After that I added masquerading to eth0. > > Possibly you can sidestep the whole issue, and end up with faster and > simpler networking setup in dom0, by not using network-bridge and > vif-bridge scripts at all. Instead make your own (much much simpler) > scripts to set up NAT.I found another odd thing regarding the vif-interfaces that my has something to do with the problem mentioned before: I have a high rate of dropped tx packets on all vif''s except for vif 0.0. As mentioned before, the config without eth0->peth0 renaming and without vif for dom0 and the rest of the vif''s in the bridge works well. Patrick _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel