Hi all, In our office we use a program that makes many small writes to a file. We have been trying to use this program with our NAS, which hosts a Samba share on the network. However, the performance of the program is staggeringly slow on Window 7, 8.1, and 10. To make the problem easily repeatable, I created a small C++ program to just write 1 million lines to a file: https://github.com/mevatron/nas-write-test Next, I performed a packet capture of the two OS conversations with Wireshark. I found the Linux SMB implementation buffer many hundreds of lines into each write; whereas, Windows decides to immediately flush it's buffer to disk on each line write, which obviously causes tremendous protocol overhead and thus abysmal performance. I have tried adjusting Lanmanworkstation parameters: https://technet.microsoft.com/en-us/library/cc978438.aspx The ones that looked promising are shown below: MaxCollectionCount "Specifies the amount of data that must be present in the character-mode buffer of a named pipe to trigger a write operation. If the amount of data in the buffer meets or exceeds this value, it is written immediately. Otherwise, it is retained in the buffer until either more data is added or the value of the CollectionTime entry expires. Increasing the value of this entry can improve the performance of named-pipe applications, but it does not affect applications, such as SQL Server applications, that do their own buffering." CollectionTime "Specifies the maximum time that application data waiting to be written can remain in the character-mode buffer of a named pipe . The redirector writes data from the character-mode buffer when the amount of buffered data reaches the size threshold established by the value of the MaxCollectionCount entry. However, if the application has written at least one byte of data to the buffer but not enough data to reach the size threshold, the contents of the buffer are written when the time specified by this value expires. Increasing the value of this entry can improve the performance of named pipe applications. However, it does not affect applications, such as SQL Server applications, which do their own buffering." Unfortunately, I am unable to get these parameters to have any effect. I have also determined through testing that this problem persists even when bypassing Samba on the NAS and just writing directly between to Windows boxes. So, I don't think it's an issue with Samba, but with the Windows SMB client. I was hoping someone with more knowledge of SMB has seen this before or has deeper insights into what I can do to solve it. I have a more visual write up of all of this on ServerFault here: https://serverfault.com/questions/808633/small-writes-to-smb-network-share-are-slow-on-windows-fast-over-cifs-linux-moun I have a bounty of 100 points if you would like to collect :D Thanks for your time! Will
On Fri, Oct 06, 2017 at 06:09:32PM +0000, Will Lucas via samba wrote:> Hi all, > > In our office we use a program that makes many small writes to a file. We > have been trying to use this program with our NAS, which hosts a Samba > share on the network. However, the performance of the program is > staggeringly slow on Window 7, 8.1, and 10. > > To make the problem easily repeatable, I created a small C++ program to > just write 1 million lines to a file: > https://github.com/mevatron/nas-write-test > > Next, I performed a packet capture of the two OS conversations with > Wireshark. I found the Linux SMB implementation buffer many hundreds of > lines into each write; whereas, Windows decides to immediately flush it's > buffer to disk on each line write, which obviously causes tremendous > protocol overhead and thus abysmal performance. > > I was hoping someone with more knowledge of SMB has seen this before or has > deeper insights into what I can do to solve it. > > I have a more visual write up of all of this on ServerFault here: > https://serverfault.com/questions/808633/small-writes-to-smb-network-share-are-slow-on-windows-fast-over-cifs-linux-mounDoes the wireshark trace show the Windows client asking for and getting a RWH lease under SMB2 ? If so, then there's no reason it can't be caching the entire file locally. Seems strange behaviour from the Windows redirector here.
On Fri, Oct 06, 2017 at 06:09:32PM +0000, Will Lucas via samba wrote:> Hi all, > > In our office we use a program that makes many small writes to a file. We > have been trying to use this program with our NAS, which hosts a Samba > share on the network. However, the performance of the program is > staggeringly slow on Window 7, 8.1, and 10. > > To make the problem easily repeatable, I created a small C++ program to > just write 1 million lines to a file: > https://github.com/mevatron/nas-write-test > > Next, I performed a packet capture of the two OS conversations with > Wireshark. I found the Linux SMB implementation buffer many hundreds of > lines into each write; whereas, Windows decides to immediately flush it's > buffer to disk on each line write, which obviously causes tremendous > protocol overhead and thus abysmal performance.Or alternatively, just use the Linux SMB client and don't port the software to Windows :-).
On Fri, Oct 06, 2017 at 06:09:32PM +0000, Will Lucas via samba wrote:> Hi all, > > In our office we use a program that makes many small writes to a file. WeAlso, try running this program under Wine on Linux to a Linux-mounted SMB share...
> > > Does the wireshark trace show the Windows client asking for and > getting a RWH lease under SMB2 ? If so, then there's no reason > it can't be caching the entire file locally. Seems strange > behaviour from the Windows redirector here. >I have uploaded the Linux packet capture here (14MB): https://drive.google.com/open?id=0B6UHr3GQEkQwWXJ1NjVwMkJXOEU Also, here is the corresponding Windows packet capture as well (375MB): https://drive.google.com/open?id=0B6UHr3GQEkQwbFhOY0lCcEFjZkU> Or alternatively, just use the Linux SMB client and don't port thesoftware to Windows :-). You have no idea how much I wish that was possible :D I'm trying to slowly convert my colleagues to Linux :) Thanks for the quick reply! Will
On Fri, 2017-10-06 at 18:09 +0000, Will Lucas via samba wrote:> Hi all, >...> Next, I performed a packet capture of the two OS conversations with > Wireshark. I found the Linux SMB implementation buffer many hundreds > of > lines into each write; whereas, Windows decides to immediately flush > it's > buffer to disk on each line write, which obviously causes tremendous > protocol overhead and thus abysmal performance. > > I have tried adjusting Lanmanworkstation parameters: > https://technet.microsoft.com/en-us/library/cc978438.aspx > > The ones that looked promising are shown below: > > MaxCollectionCount > "Specifies the amount of data that must be present in the character- > mode > buffer of a named pipe to trigger a write operation. If the amount of > data > in the buffer meets or exceeds this value, it is written immediately. > Otherwise, it is retained in the buffer until either more data is > added or > the value of the CollectionTime entry expires. > Increasing the value of this entry can improve the performance of > named-pipe applications, but it does not affect applications, such as > SQL > Server applications, that do their own buffering."... I'm no expert here but how does say SQL Server indicate that it does its own buffering? I think you have managed to go as far as you can with your approach and have hit a fundamental Windowism. You clearly know what you are doing and have spent quite a while on hitting a brickwall, that needs (Windows) source code to get to the bottom of. Might I suggest that you re-evaluate how your program works? Getting "Seems strange behaviour from the Windows redirector here." from Mr A does imply that it may not be easy to fix. A first thought is, can you buffer reads at the client end before sending them to the network? Cheers Jon