Santos, Jose Renato G (Jose Renato Santos)
2005-Apr-05 15:23 UTC
RE: [Xen-devel] MPI benchmark performance gap between native linux anddomU
Keir, In which version the ''truesize'' field was changed to report less than a page? We were using 2.0.3 when we found this problem. I agree this trick will prevent the early overflow of the receive buffer. However, I am thinking if there is no other side effect of lying about the true size of the buffer to the kernel. Would bad things happen if the kernel believes that is using less memory than it is really using. For example, would it be possible for the kernel to exhaust memory for network intensive application with a large number of open connections ? Renato>> -----Original Message----- >> From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] >> Sent: Tuesday, April 05, 2005 1:59 AM >> To: Santos, Jose Renato G (Jose Renato Santos) >> Cc: Aravind Menon; Turner, Yoshio; >> Xen-devel@lists.xensource.com; xuehai zhang; G John Janakiraman >> Subject: Re: [Xen-devel] MPI benchmark performance gap >> between native linux anddomU >> >> >> >> On 5 Apr 2005, at 03:07, Santos, Jose Renato G (Jose Renato Santos) >> wrote: >> >> > Here is a brief explanation of the problem we found and >> the solution >> > that worked for us. >> > Xenolinux allocates a full page (4KB) to store socket buffers >> > instead of using just MTU bytes as in traditional linux. This is >> > necessary to enable page exchanges between the guest and the I/O >> > domains. The side effect of this is that memory space used >> for socket >> > buffers is not very efficient. >> >> This is true, but these days we lie to the network stack >> about how big >> the skb data area is. The ''truesize'' field, which is what I think is >> used for socket buffer accounting, will be around 1600 >> bytes, not 4096. >> So I would expect the old trick of reducing the receive >> windows not to >> work: but if it does then that is very interesting! >> >> -- Keir >> >>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2005-Apr-05 15:47 UTC
Re: [Xen-devel] MPI benchmark performance gap between native linux anddomU
On 5 Apr 2005, at 16:23, Santos, Jose Renato G (Jose Renato Santos) wrote:> In which version the ''truesize'' field was changed to report less than > a page? > We were using 2.0.3 when we found this problem. > I agree this trick will prevent the early overflow of the receive > buffer. > However, I am thinking if there is no other side effect of lying > about > the true size of the buffer to the kernel. > Would bad things happen if the kernel believes that is using less > memory than it is really using. > For example, would it be possible for the kernel to exhaust memory > for > network intensive application with a large number of open connections ?I guess it would be easier to provoke trouble, but in any case the default advertised window and socket buffer allocation are not affected dynamically by system-wide memory pressure. Per-sockbuf limits are set to a ''suitable default'' at boot-time according to amount of RAM detected, but after that they have to be manually reset by the user. So I don''t think we are breaking any carefully-tuned dynamically-balanced memory allocation algorithms here. :-) By setting the true size (4kB) we are far more likely to throw network performance off, as the TCP stack will not have been tuned with such large packet overheads in mind. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel