Chris Duffy
2011-Jan-17 16:14 UTC
[Samba] Question on how to get Samba to use larger pread/write calls.
We are testing Samba 3 (and 4) on Fedora Core 13, 10Gbit connection with a Mac OS 10.6.4 system as the client. We will be adding some Windows machines sooner or later with 10Gbit interfaces. We are seeing 100-150MBytes/sec read or write performance between the Mac and the FC13 system over 10Gbit interface but it should be capable of 400-500MBytes/sec. We have a local raid on the FC13 system that runs 1GByte/sec locally using an Areca 1880-ix-16 raid card (6Gbit version). It has 16 fast Hitachi disks in a Raid5 format using xfs filesystem. The problem here is that samba is poking the Areca at 128KByte I/O's on preads and writes, i.e. shown to us using strace on the smbd daemons that are running. Using vmstat/iostat/sar utilities, we see 100% utilization of the Areca card because the average wait time is real high and the average queue length to it is also high......too many small I/O's..... This is not the case if I run "fio" or "dd" locally to/from the Areca's raid using 1-4MByte I/O's. I see fast I/O... I do not see any way to increase the size of Samba's pread/write's in the smb.conf documentation. I am sure it may be just a matter of getting the source code and making some changes to allow larger sized IO's but........and of course I suppose Windows clients may complain but.... I remember that back in the old Irix days with the group of engineers in Australia I worked with, we had Samba screaming fast but not sure if they tweaked the version of Samba on Irix to do this. Can you guys come up with a way for us to allow the reads/writes to/from the disks to be tunable up to say 4MB in size? Thx chris duffy
Volker Lendecke
2011-Jan-18 06:34 UTC
[Samba] Question on how to get Samba to use larger pread/write calls.
On Mon, Jan 17, 2011 at 10:14:04AM -0600, Chris Duffy wrote:> We are testing Samba 3 (and 4) on Fedora Core 13, > 10Gbit connection with a Mac OS 10.6.4 system > as the client. We will be adding some Windows > machines sooner or later with 10Gbit interfaces. > > We are seeing 100-150MBytes/sec read or write > performance between the Mac and the FC13 system > over 10Gbit interface but it should be capable of > 400-500MBytes/sec. We have a local raid > on the FC13 system that runs 1GByte/sec locally > using an Areca 1880-ix-16 raid card (6Gbit version). > It has 16 fast Hitachi disks in a Raid5 format > using xfs filesystem. > > The problem here is that samba is poking the Areca > at 128KByte I/O's on preads and writes, i.e. > shown to us using strace on the smbd daemons > that are running. Using vmstat/iostat/sar utilities, > we see 100% utilization of the Areca card because the > average wait time is real high and the average > queue length to it is also high......too many > small I/O's..... > This is not the case if I run "fio" or "dd" locally > to/from the Areca's raid using 1-4MByte I/O's. > I see fast I/O... > > I do not see any way to increase the size of Samba's > pread/write's in the smb.conf documentation. I > am sure it may be just a matter of getting the source > code and making some changes to allow larger > sized IO's but........and of course I suppose Windows > clients may complain but.... > > I remember that back in the old Irix days with the > group of engineers in Australia I worked with, > we had Samba screaming fast but not sure if they > tweaked the version of Samba on Irix to do this. > > Can you guys come up with a way for us to allow > the reads/writes to/from the disks to be tunable > up to say 4MB in size?You might want to play with "write cache size". This will only tune writes though. And it will only work for oplocked files. I'm not 100% sure that OS/X plays nicely wrt oplocks. For reads, we need to take a much closer look at your real workload and see if we need to use some kind of preopen, prefetch or so module. We need to closely work with the kernel buffer cache and potential readahead kernel algorithms. With best regards, Volker Lendecke -- SerNet GmbH, Bahnhofsallee 1b, 37081 G?ttingen phone: +49-551-370000-0, fax: +49-551-370000-9 AG G?ttingen, HRB 2816, GF: Dr. Johannes Loxen
James Peach
2011-Jan-18 17:50 UTC
[Samba] Question on how to get Samba to use larger pread/write calls.
On 17 January 2011 08:14, Chris Duffy <cjd at small-tree.com> wrote:> We are testing Samba 3 (and 4) on Fedora Core 13, > 10Gbit connection with a Mac OS 10.6.4 system > as the client. ?We will be adding some Windows > machines sooner or later with 10Gbit interfaces. > > We are seeing 100-150MBytes/sec read or write > performance between the Mac and the FC13 system > over 10Gbit interface but it should be capable of > 400-500MBytes/sec. ?We have a local raid > on the FC13 system that runs 1GByte/sec locally > using an Areca 1880-ix-16 raid card (6Gbit version). > It has 16 fast Hitachi disks in a Raid5 format > using xfs filesystem. > > The problem here is that samba is poking the Areca > at 128KByte I/O's on preads and writes, i.e. > shown to us using strace on the smbd daemons > that are running. ?Using vmstat/iostat/sar utilities, > we see 100% utilization of the Areca card because the > average wait time is real high and the average > queue length to it is also high......too many > small I/O's..... > This is not the case if I run "fio" or "dd" locally > to/from the Areca's raid using 1-4MByte I/O's. > I see fast I/O... > > I do not see any way to increase the size of Samba's > pread/write's in the smb.conf documentation. ?I > am sure it may be just a matter of getting the source > code and making some changes to allow larger > sized IO's but........and of course I suppose Windows > clients may complain but.... > > I remember that back in the old Irix days with the > group of engineers in Australia I worked with, > we had Samba screaming fast but not sure if they > tweaked the version of Samba on Irix to do this.Hi Chris :) IIRC, we used a carefully tweaked read ahead module to make sure that I/Os were hitting the disks properly aligned and of reasonable size. http://www.samba.org/samba/docs/man/manpages-3/vfs_readahead.8.html> > Can you guys come up with a way for us to allow > the reads/writes to/from the disks to be tunable > up to say 4MB in size? > > Thx > chris duffy > > -- > To unsubscribe from this list go to the following URL and read the > instructions: ?https://lists.samba.org/mailman/options/samba >-- James Peach | jorgar at gmail.com