Hi all,
I have 2 virtual machines running and everything runs just fine except
that I have some
network issues. Both machines are in the same ip network and are
attached to the same
bridging interface:
# brctl show
bridge name bridge id STP enabled interfaces
xen-br0 8000.0013216b6112 no eth1
vif1.0
vif2.0
If I look at the ifconfig stats I see TX packet drops on the vif interfaces:
eth1 Link encap:Ethernet HWaddr 00:13:21:6B:61:12
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2900161 errors:0 dropped:0 overruns:0 frame:0
TX packets:1437145 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:215418139 (205.4 MiB) TX bytes:106086424 (101.1 MiB)
Interrupt:26
vif1.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5585927 errors:0 dropped:0 overruns:0 frame:0
TX packets:5649573 errors:0 dropped:13923 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1385954050 (1.2 GiB) TX bytes:2747246776 (2.5 GiB)
vif2.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4932512 errors:0 dropped:0 overruns:0 frame:0
TX packets:6466388 errors:0 dropped:9191 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2700280469 (2.5 GiB) TX bytes:1447794114 (1.3 GiB)
xen-br0 Link encap:Ethernet HWaddr 00:13:21:6B:61:12
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:154640 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:14863871 (14.1 MiB) TX bytes:0 (0.0 b)
Could someone explain these packet drops? The packet drops occur mostly
under
higher load (CPU and/or network) and can be reproduced by communicating
between
the two virtual machines (over the bridge without involving the "real"
network) but
also if I try to connect from the outside to the virtual machine I have
packet loss.
I noticed the lines
/* Disable queuing. */
dev->tx_queue_len = 0;
in interface.c (netback source of XEN). Could this lead into problems if
the bridge
does not react fast enough to receive packets from the vifs in high load
situations?
cheers,
Torsten
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
> vif1.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:5585927 errors:0 dropped:0 overruns:0 frame:0 > TX packets:5649573 errors:0 dropped:13923 overruns:0 carrier:0FWIW, this appears to be the same situation I reported under a recent thread, "intermittent network hangs." Can you confirm that the symtoms of these packet drops are the same for you? John -- John Madden UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John Madden schrieb:>>vif1.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:5585927 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:5649573 errors:0 dropped:13923 overruns:0 carrier:0 >> >> > >FWIW, this appears to be the same situation I reported under a recent thread, >"intermittent network hangs." > >Can you confirm that the symtoms of these packet drops are the same for you? > >Yes, I think so... but I did not notice the hangs during a session but mostly during the TCP SYN handshake :-o So in my test scenario the "wget --mirror" hangs mostly during the connect and not during the transfer of some files... strange problem... did you get the routed environment working? I don''t understand how the proxy-arp for only one ip address works (my host is on eth0 in another ip network than the xen-bridge (eth1)). Torsten _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Yes, I think so... but I did not notice the hangs during a session but > mostly during the TCP SYN handshake :-o > So in my test scenario the "wget --mirror" hangs mostly during the > connect and not during the transfer of someI see hangs mostly in the middle of ssh sessions -- start up a continuous ping and watch the packet loss.> files... strange problem... did you get the routed environment working?I haven''t had the time or opportunity (it''s a production box). I''m mostly concerned with having it not operational and having to fall back to the bridged config anyway. John -- John Madden UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John Madden schrieb:>>Yes, I think so... but I did not notice the hangs during a session but >>mostly during the TCP SYN handshake :-o >>So in my test scenario the "wget --mirror" hangs mostly during the >>connect and not during the transfer of some >> >> > >I see hangs mostly in the middle of ssh sessions -- start up a continuous ping and >watch the packet loss. > >Checked it and it''s true. I see ICMP packet loss, too and it coincidents with the percentage of TX packet drops of ~ 0.2%>>files... strange problem... did you get the routed environment working? >> >> > >I haven''t had the time or opportunity (it''s a production box). I''m mostly >concerned with having it not operational and having to fall back to the bridged >config anyway. > >Same for me... and I really like the bridged config... perhaps someone has an idea/solution/fix/workaround. BTW I use FC4 with updated Kernel 2.6.12.1 and updated xen (05-30-2005). Torsten _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John Madden schrieb:>>Yes, I think so... but I did not notice the hangs during a session but >>mostly during the TCP SYN handshake :-o >>So in my test scenario the "wget --mirror" hangs mostly during the >>connect and not during the transfer of some >> >> > >I see hangs mostly in the middle of ssh sessions -- start up a continuous ping and >watch the packet loss. > >Just found a document in the xen distribution (docs/src/interface.tex). The subsection about networking is interesting: e.g. "Each virtual interface uses two ``descriptor rings'''', one for transmit, the other for receive. Each descriptor identifies a block of contiguous physical memory allocated to the domain." and "If a domain does not keep its receive ring stocked with empty buffers then packets destined to it may be dropped. This provides some defence against receive livelock problems because an overload domain will cease to receive further data. Similarly, on the transmit path, it provides the application with feedback on the rate at which packets are able to leave the system." In xen/include/public/io/netif.h the sizes of these buffers are defined as #define NETIF_TX_RING_SIZE 256 #define NETIF_RX_RING_SIZE 256 Perhaps this is too small for new performance machines. My machine is a Dual Xeon 3.2GHz. If I have time, I will compile new XEN kernels with greater buffers and see what will happen. cheers, Torsten _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> #define NETIF_TX_RING_SIZE 256 > #define NETIF_RX_RING_SIZE 256Interesting...> Perhaps this is too small for new performance machines. My machine is a > Dual Xeon 3.2GHz.Same here -- PE1750.> If I have time, I will compile new XEN kernels with greater buffers and > see what will happen.Please do, particularly if you can afford the reboot. :) John -- John Madden UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
>> If I have time, I will compile new XEN kernels with greater buffers and >> see what will happen. > > Please do, particularly if you can afford the reboot. :)The problem is that the data structure in which the ring buffer is organized has to fit into one memory page (4096 bytes). So we are limited to a buffer size of 340 entries which means 340 packets (~30% bigger). The result are almost the same; it gets a little bit better: loss rates <0.1% But I think it''s really not a bug but a feature. You have to ensure that the virtual machines are not locked up with too old packets in a long queue during high network load (this would lead to an unreachable virtual server which cannot answer recent packets). So this is really a traffic shaping routine (very very basic); it would be better e.g. to delay tcp packets if the buffer gets crowded since tcp stacks would react on the delay and adjust their sending speed; but this is not possible because the host machine does not "see" tcp traffic but only bridges the frames. This leads to another problem: if your guests are in a network with high broadcast load the buffers get filled with theses broadcasts too. So after all, IMHO it''s a general design problem of virtual machines which have to emulate the interrupts of real hardware. cheers, Torsten PS: What type of applications do you run on the productive xen machines? What is your overall experience (beside the network issue ;-)) ? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> So this is really a traffic > shaping routine (very very basic); it would be better e.g. to delay tcp > packets if the buffer gets crowded since tcp stacks would react on the > delay and adjust their sending speed; but this is not possible because the > host machine does not "see" tcp traffic but only bridges the frames. This > leads to another problem: if your guests are in a network with high > broadcast load the buffers get filled with theses broadcasts too. > So after all, IMHO it''s a general design problem of virtual machines which > have to emulate the interrupts of real hardware.In other words, routing rather than bridging may be advisable? Routers at least have queues and are designed to handle these sorts of situations, right? At the very least, Linux QoS principles could be applied on dom0 to prioritize packet flow to the more critical domU''s...> PS: What type of applications do you run on the productive xen machines? > What is your overall experience (beside the network issue ;-)) ?We''re running our internal FTP server, a web development machine, our enterprise Helpdesk application, and eventually the nagios installation will actually be doing something. Feel free to quote me on it: Xen has changed the way I do my job. It''s every bit as disruptive a technology as the SAN we installed earlier this year. It has been and will continue to be a key element in the consolidation of hardware, simpliciation of infrastructure, and ease of management. It''s quickly becoming one of those "how did I live without this software" pieces of software. ...And I''ll happily go on all day about it. :) That said, customers don''t like packet loss and it doesn''t matter how good Xen is if their ssh sessions hang every few minutes. John -- John Madden UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
>In other words, routing rather than bridging may be advisable? Routers at least >have queues and are designed to handle these sorts of situations, right? At the >very least, Linux QoS principles could be applied on dom0 to prioritize packet >flow to the more critical domU''s... > >Perhaps this would help... I will try to test it after the weekend. The routing process might notice that the interface (queue) is full and would queue the packets in its own router queues... priotization might be helpful aswell. BTW tc works on a bridged vif, too.... I will give it both a try on monday. Did you find any more infos on the routing setup for xen?>>PS: What type of applications do you run on the productive xen machines? >>What is your overall experience (beside the network issue ;-)) ? >> >> > >It''s quickly becoming one of those "how did I live without this software" pieces of >software. ...And I''ll happily go on all day about it. :) > >That''s good to hear. We are planning to use it for ftp server, nntp host, smtp mailing and perhaps http proxy. I think only for the last one the packet loss could be a problem because of direct user interaction... but hey, it''s internet: packet loss is normal ;-)) Torsten _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Torsten Vielhak wrote:> The problem is that the data structure in which the ring buffer is > organized has to fit into one memory page (4096 bytes). So we are limited > to a buffer size of 340 entries which means 340 packets (~30% bigger). The > result are almost the same; it gets a little bit better: loss rates <0.1% > But I think it''s really not a bug but a feature. You have to ensure that > the virtual machines are not locked up with too old packets in a long > queue during high network load (this would lead to an unreachable virtual > server which cannot answer recent packets). So this is really a traffic > shaping routine (very very basic); it would be better e.g. to delay tcp > packets if the buffer gets crowded since tcp stacks would react on the > delay and adjust their sending speed; but this is not possible because the > host machine does not "see" tcp traffic but only bridges the frames. ThisActually, tcp does react to getting acks received later, dropped packets, and in general, congestion on the network. Sending will slow down. Even sending of acks gets slowed when the user process isn''t scheduled often enough to drain the receive buffer fast enough. So, there are mechanisms in TCP to react to the environment, whether network related or caused by virtualization. The degree to which this is handled might, of course, not be good enough for your performance needs.> leads to another problem: if your guests are in a network with high > broadcast load the buffers get filled with theses broadcasts too. > So after all, IMHO it''s a general design problem of virtual machines which > have to emulate the interrupts of real hardware.All of the above said, there is some thinking that has to go into improving the current architecture to handle high loads better. thanks, Nivedita _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John Madden wrote:> That said, customers don''t like packet loss and it doesn''t matter how good Xen is > if their ssh sessions hang every few minutes.Yep, we are trying to fix this. thanks, Nivedita _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users