Hi everyone. I am running Samba 3.2.0-22.1 (as packaged by OpenSUSE in 11.0) on a storage server connected to multiple windows based clients over a gigabit ethernet link. The server is a quad core Intel CPU and is equipped with an Intel e1000 based gigabit ethernet controller and plugged into a common gigabit ethernet switch with the windows clients. I am seeing performance issues on transfers over the gigabit ethernet network, and was trying to play with the socket options settings in the smb.conf file to improve transfers rates, but no matter what I set the SO_RCVBUF and SO_SNDBUF values too, the transfer rates are unchanged, even if I set the buffer sizes down to 512, which should have the effect of at least slowing things down dramatically, leading me to question if Samba is actually using these settings at all. TCP_NODELAY is set, but it doesn't seem to matter much if I include it or not on the socket options line, and the line is definitely not commented out, as if I misspell something on that line, samba terminates with an error when I try and restart the daemon. Samba is getting roughly 30 MB/s tranfer rates from the linux server to a windows vista and a windows XP client, and the disks on both windows machines are RAID0 (4 and 2 disk RAID0 sets respectively), so I don't think I am running into filesystem performance issues on the target. Moving from the windows systems to Samba, I see about 45 MB/sec transfers rates. The raid array on the samba server consist of 2 6 disk raid5 sets with fast disks on them, running lvm and XFS for a filesysteem. I can do a dd of a multigigabyte file to /dev/null and get roughly 500-600 MB/'s transfer rates through the filesystem, so I don't think the raid array and file system is a bottleneck. I have run netperf tests between the server and the clients to see if I had some network plumbing problems. With default socket settings for netperf (8182 buffer size), I get about 300 mbps transfer rates between the clients and the server (which matches approximately the 30 MB/s transfer rates). With 65536 byte buffers, that number goes to 970 or so Mbps, so I think the interface cards, TCP stack, switches are all ok. If I do an FTP from the server to a client, I get 45-50 MB/s transfer rates, so I think the problem is somewhere in Samba. Again, if I change the SO_SNDBUF and SO_RCVBUF values, either up or down, or keep them unset, I get almost no variance in transfer rates. Vista is set to use autotuning in it's TCP configuration, and window scaling and RFC 1323 options are enabled, but I see the same exact performance on XP as well, so I don't think it's a client issue. I get somewhat faster copying from a windows 2003 server on the same LAN to the same clients, even though it is not equipped with a raid array and is just reading from one disk with an unoptimized NTFS filesystem. Is there a bug here or am I missing something in the configuration? thanks, Mike
On Thu, Aug 28, 2008 at 12:53:00PM -0700, Mike Myers wrote:> Hi everyone. I am running Samba 3.2.0-22.1 (as packaged by OpenSUSE in > 11.0) on a storage server connected to multiple windows based clients > over a gigabit ethernet link. The server is a quad core Intel CPU and > is equipped with an Intel e1000 based gigabit ethernet controller and plugged into a common gigabit ethernet switch with the windows clients. > > I am seeing performance issues on transfers over the gigabit ethernet network, > and was trying to play with the socket options settings in the smb.conf > file to improve transfers rates, but no matter what I set the SO_RCVBUF > and SO_SNDBUF values too, the transfer rates are unchanged, even if I > set the buffer sizes down to 512, which should have the effect of at > least slowing things down dramatically, leading me to question if Samba > is actually using these settings at all. TCP_NODELAY is set, but it > doesn't seem to matter much if I include it or not on the socket > options line, and the line is definitely not commented out, as if I > misspell something on that line, samba terminates with an error when I > try and restart the daemon.Samba is definately setting these options.> Samba is getting roughly 30 MB/s tranfer rates from the linux server to a windows vista and a windows XP client, > and the disks on both windows machines are RAID0 (4 and 2 disk RAID0 > sets respectively), so I don't think I am running into filesystem > performance issues on the target. Moving from the windows systems to Samba, I see about 45 MB/sec transfers rates. > > The > raid array on the samba server consist of 2 6 disk raid5 sets with fast > disks on them, running lvm and XFS for a filesysteem. I can do a dd of > a multigigabyte file to /dev/null and get roughly 500-600 MB/'s > transfer rates through the filesystem, so I don't think the raid array > and file system is a bottleneck. > > I have run netperf tests > between the server and the clients to see if I had some network > plumbing problems. With default socket settings for netperf (8182 > buffer size), I get about 300 mbps transfer rates between the clients > and the server (which matches approximately the 30 MB/s transfer > rates). With 65536 byte buffers, that number goes to 970 or so Mbps, > so I think the interface cards, TCP stack, switches are all ok.Can you try using smbclient to do a large file transfer from another client Linux box and time that please ? That eliminates the Windows clients from the equation, and allows us to test only with things we can examine directly. Jeremy.
On Thu, Aug 28, 2008 at 12:53:00PM -0700, Mike Myers wrote:> Hi everyone. I am running Samba 3.2.0-22.1 (as packaged by OpenSUSE in > 11.0) on a storage server connected to multiple windows based clients > over a gigabit ethernet link. The server is a quad core Intel CPU and > is equipped with an Intel e1000 based gigabit ethernet controller and plugged into a common gigabit ethernet switch with the windows clients.Is this still the case? In addition to Jeremy's suggestion I'd like to see you testing this with a straight connection between the two systems. But after reading the rest of your posting it's not very liekly to help. For openSUSE 11.0 we'll also soon see an official update to 3.2.3. Meanwhile you might use the packages provided by the openSUSE Build Service. See http://en.opensuse.org/Samba#openSUSE_Build_Service for how to access them.> Samba is getting roughly 30 MB/s tranfer rates from the linux server to a windows vista and a windows XP client, > and the disks on both windows machines are RAID0 (4 and 2 disk RAID0 > sets respectively), so I don't think I am running into filesystem > performance issues on the target. Moving from the windows systems to Samba, I see about 45 MB/sec transfers rates.The write case is faster than read. Strange to me.> The > raid array on the samba server consist of 2 6 disk raid5 sets with fast > disks on them, running lvm and XFS for a filesysteem. I can do a dd of > a multigigabyte file to /dev/null and get roughly 500-600 MB/'s > transfer rates through the filesystem, so I don't think the raid array > and file system is a bottleneck.Do you have some space on a local disk with ext2, no raid, no lvm? Only to ensure your issue isn't caused by one of these components.> I have run netperf tests > between the server and the clients to see if I had some network > plumbing problems. With default socket settings for netperf (8182 > buffer size), I get about 300 mbps transfer rates between the clients > and the server (which matches approximately the 30 MB/s transfer > rates). With 65536 byte buffers, that number goes to 970 or so Mbps, > so I think the interface cards, TCP stack, switches are all ok.Therefore this might be caused by the file system in use. Please try to test the way as suggested above. If it isn't solved already. Lars -- Lars M?ller [?la?(r)z ?m?l?] Samba Team SUSE Linux, Maxfeldstra?e 5, 90409 N?rnberg, Germany -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://lists.samba.org/archive/samba/attachments/20080829/59997dfb/attachment.bin
I still see the problem. I don't have another real linux system to test, but will install another copy on a different system and configure it for dual boot. I can't really do that easily on on the same hardware as the existing windows machines with raid, since there is no non-raid boot partition on those. I do have a laptop and some other machines that I can do dual-boot on, but I am concerned that the local filesystem will be a bottleneck as it's just a single disk. My experience is that you really can't test the maxmium speed for a server with a fast raid configuration to a client that has a non-raid disk - the local target for the file just can't keep up, but it may be able to go faster than 30 MB/s... It may take me another day or so to ready such a configuration for test. I will go ahead and set up a new samba share off the system disk which isn't part of the raid5 sets, and uses resierfs and try that too. The disk I use should be able to do north of 60 MB/s in raw transfer rate on sequential reads, so if resier doesn't get too much in the way, it should be able to supply samba at a rate of more than the 30 MB/s I see for transfers off the raid volume. I am still curious as to why changing the socket buffer settings to a wide variety of options has absolutely no effect on perfromance. Steve Thompson seems to have noticed the same issue, and my experience tells me that if I set the SO_SNDBUF and SO_RCVBUF to something like 512 on an app, that it slaughters performance. It certainly does for netperf, so I can't understand if Samba is really using those options why setting it to 512 doesn't slow the speed down like it does for netperf. I don't mean to be presumptuous, but these data points re: no speed variance even in the face of tiny buffer settings indicates that something is wrong with the options being exercised, so I am curious as to why you think the options are being used? Jeremy and Lars, in your test environment, can you try setting the SO_SNDBUF and SO_RCVBUF down to 512 and see if you also do not notice a variance in performance? Thanks much, Mike ----- Original Message ---- From: Lars M?ller <lmuelle@suse.de> To: Mike Myers <mikesm559@yahoo.com> Cc: samba@lists.samba.org Sent: Friday, August 29, 2008 4:36:30 AM Subject: Re: [Samba] Samba ignoring socket options? On Thu, Aug 28, 2008 at 12:53:00PM -0700, Mike Myers wrote:> Hi everyone. I am running Samba 3.2.0-22.1 (as packaged by OpenSUSE in > 11.0) on a storage server connected to multiple windows based clients > over a gigabit ethernet link. The server is a quad core Intel CPU and > is equipped with an Intel e1000 based gigabit ethernet controller and plugged into a common gigabit ethernet switch with the windows clients.Is this still the case? In addition to Jeremy's suggestion I'd like to see you testing this with a straight connection between the two systems. But after reading the rest of your posting it's not very liekly to help. For openSUSE 11.0 we'll also soon see an official update to 3.2.3. Meanwhile you might use the packages provided by the openSUSE Build Service. See http://en.opensuse.org/Samba#openSUSE_Build_Service for how to access them.> Samba is getting roughly 30 MB/s tranfer rates from the linux server to a windows vista and a windows XP client, > and the disks on both windows machines are RAID0 (4 and 2 disk RAID0 > sets respectively), so I don't think I am running into filesystem > performance issues on the target. Moving from the windows systems to Samba, I see about 45 MB/sec transfers rates.The write case is faster than read. Strange to me.> The > raid array on the samba server consist of 2 6 disk raid5 sets with fast > disks on them, running lvm and XFS for a filesysteem. I can do a dd of > a multigigabyte file to /dev/null and get roughly 500-600 MB/'s > transfer rates through the filesystem, so I don't think the raid array > and file system is a bottleneck.Do you have some space on a local disk with ext2, no raid, no lvm? Only to ensure your issue isn't caused by one of these components.> I have run netperf tests > between the server and the clients to see if I had some network > plumbing problems. With default socket settings for netperf (8182 > buffer size), I get about 300 mbps transfer rates between the clients > and the server (which matches approximately the 30 MB/s transfer > rates). With 65536 byte buffers, that number goes to 970 or so Mbps, > so I think the interface cards, TCP stack, switches are all ok.Therefore this might be caused by the file system in use. Please try to test the way as suggested above. If it isn't solved already. Lars -- Lars M?ller [?la?(r)z ?m?l?] Samba Team SUSE Linux, Maxfeldstra?e 5, 90409 N?rnberg, Germany
Ok, Tests from the system disk which isn't raided seem to be about the same performance as from the raid volume. One other item of note, the system has 8 GB of ram in it, so I notice if I dd a 1 GB file file to /dev/null on the local system, it runs about 500-600 MB/s transfer rate. If I do it again shortly afterwards, I get about 3 GB/s transfer rate, no doubt because of the disk and filesystem caching going on ( the whole file gets buffered in RAM). The point is that I doubt samba is running into local filesystem performance problems, as I repeatedly transfer the same 1 GB file from the server to the clients, so I expect it's all cached in RAM during most of the transfers. I'll go ahead and get my laptop configured to do dual boot an d see what an SMB client sees... Thanks, Mike ----- Original Message ---- From: Lars M?ller <lmuelle@suse.de> To: Mike Myers <mikesm559@yahoo.com> Cc: samba@lists.samba.org Sent: Friday, August 29, 2008 4:36:30 AM Subject: Re: [Samba] Samba ignoring socket options? On Thu, Aug 28, 2008 at 12:53:00PM -0700, Mike Myers wrote:> Hi everyone. I am running Samba 3.2.0-22.1 (as packaged by OpenSUSE in > 11.0) on a storage server connected to multiple windows based clients > over a gigabit ethernet link. The server is a quad core Intel CPU and > is equipped with an Intel e1000 based gigabit ethernet controller and plugged into a common gigabit ethernet switch with the windows clients.Is this still the case? In addition to Jeremy's suggestion I'd like to see you testing this with a straight connection between the two systems. But after reading the rest of your posting it's not very liekly to help. For openSUSE 11.0 we'll also soon see an official update to 3.2.3. Meanwhile you might use the packages provided by the openSUSE Build Service. See http://en.opensuse.org/Samba#openSUSE_Build_Service for how to access them.> Samba is getting roughly 30 MB/s tranfer rates from the linux server to a windows vista and a windows XP client, > and the disks on both windows machines are RAID0 (4 and 2 disk RAID0 > sets respectively), so I don't think I am running into filesystem > performance issues on the target. Moving from the windows systems to Samba, I see about 45 MB/sec transfers rates.The write case is faster than read. Strange to me.> The > raid array on the samba server consist of 2 6 disk raid5 sets with fast > disks on them, running lvm and XFS for a filesysteem. I can do a dd of > a multigigabyte file to /dev/null and get roughly 500-600 MB/'s > transfer rates through the filesystem, so I don't think the raid array > and file system is a bottleneck.Do you have some space on a local disk with ext2, no raid, no lvm? Only to ensure your issue isn't caused by one of these components.> I have run netperf tests > between the server and the clients to see if I had some network > plumbing problems. With default socket settings for netperf (8182 > buffer size), I get about 300 mbps transfer rates between the clients > and the server (which matches approximately the 30 MB/s transfer > rates). With 65536 byte buffers, that number goes to 970 or so Mbps, > so I think the interface cards, TCP stack, switches are all ok.Therefore this might be caused by the file system in use. Please try to test the way as suggested above. If it isn't solved already. Lars -- Lars M?ller [?la?(r)z ?m?l?] Samba Team SUSE Linux, Maxfeldstra?e 5, 90409 N?rnberg, Germany