samba - Sep 2018 - Can't copy large files to Windows with SMB2/3 on 10G network

We have a 10G network with a mix of Windows 10 and MacOS 10.12 machines, 
and a CentOS 7 machine.

Copying large files "fast" from the CentOS machine to Windows, using 
SMB2 or SMB3 fails almost every time, at the end of the copy.
- With SMB1, it seems to work, but is much slower.
- Copying to a smb share on a Mac seems to work, but is also much slower.
- Using a normal Gigabit network card instead of 10G, it always works.

By "large files" I mean 50 to 100 GB.
By "fast" I mean over 300 MBytes per second (usually 500 to 700 MB/s.)
as reported by various monitors on the Linux client or the windows servers.
By "slower" I mean under 150 MB/s.

I tried both smbclient and mount.cifs:
- smbclient [no -m option] is OK and slow (uses default SMB1/NT1?)
- mount.cifs -o vers=1.0 is OK and slow
- smbclient -m SMB2 (or -m SMB3) fails
- mount.cifs -o vers=3.0 (or vers=2.1 or vers=2.0) fails.

I also tried 2 different 10G cards on the Linux client (Intel and Asus).

The few times when the fast copy worked, it was to a Windows share on a 
very fast RAID (faster than 10G). But even copying to that specific fast 
disk didn't work every time.

Example errors when it fails:

cp: failed to close ‘/mnt/dcp/temp/file.mxf’: Input/output error
or
rsync: close failed on "/mnt/dcp2/temp/.file.mxf.41mPmS": Input/output
error (5)
rsync error: error in file IO (code 11) at receiver.c(859) [receiver=3.1.2]


When I turned on debugging, I got an avalanche of "Status code returned 
0xc000009a STATUS_INSUFFICIENT_RESOURCES" in the log, and tons of 
"callbacks suppressed". Example:

...
[Fri Aug 31 17:21:53 2018] Status code returned 0xc000009a 
STATUS_INSUFFICIENT_RESOURCES
[Fri Aug 31 17:21:53 2018] Status code returned 0xc000009a 
STATUS_INSUFFICIENT_RESOURCES
[Fri Aug 31 17:22:21 2018] cifs_demultiplex_thread: 927 callbacks suppressed
[Fri Aug 31 17:22:21 2018] fs/cifs/connect.c: RFC1002 header 0x50
[Fri Aug 31 17:22:21 2018] smb2_check_message: 929 callbacks suppressed
[Fri Aug 31 17:22:21 2018] fs/cifs/smb2misc.c: smb2_check_message 
length: 0x54, smb_buf_length: 0x50
[Fri Aug 31 17:22:21 2018] smb2_calc_size: 929 callbacks suppressed
[Fri Aug 31 17:22:21 2018] fs/cifs/smb2misc.c: SMB2 len 85
[Fri Aug 31 17:22:21 2018] smb2_check_message: 3 callbacks suppressed
[Fri Aug 31 17:22:21 2018] fs/cifs/smb2misc.c: Calculated size 85 length 
84 mismatch mid 3265029
[Fri Aug 31 17:22:31 2018] fs/cifs/smb2pdu.c: In echo request
[Fri Aug 31 17:22:31 2018] __smb_send_rqst: 916 callbacks suppressed
...


I tried this all with (on the same machine) :
- Samba 4.7.1 on CentOS 7.5 with kernel 3.10.0-862.11.6.el7.x86_64
- Samba 4.3.11 on Ubuntu 16.04 with kernel 4.4.0-134-generic

Many Google searches didn't bring up anything which seems relevant, so I 
hope someone here has ideas...

Thanks for any help or pointers,

MI

On Sun, Sep 16, 2018 at 09:53:59AM +0200, MI via samba
wrote:
> We have a 10G network with a mix of Windows 10 and MacOS 10.12 machines,
and
> a CentOS 7 machine.
> 
> Copying large files "fast" from the CentOS machine to Windows,
using SMB2 or
> SMB3 fails almost every time, at the end of the copy.
> - With SMB1, it seems to work, but is much slower.
> - Copying to a smb share on a Mac seems to work, but is also much slower.
> - Using a normal Gigabit network card instead of 10G, it always works.
> 
> By "large files" I mean 50 to 100 GB.
> By "fast" I mean over 300 MBytes per second (usually 500 to 700
MB/s.) as
> reported by various monitors on the Linux client or the windows servers.
> By "slower" I mean under 150 MB/s.
> 
> I tried both smbclient and mount.cifs:
> - smbclient [no -m option] is OK and slow (uses default SMB1/NT1?)
> - mount.cifs -o vers=1.0 is OK and slow
> - smbclient -m SMB2 (or -m SMB3) fails
The above is the interesting case, as this is much
easier to debug (in user space) than the kernel
client.

Can you give more details on *exactly* HOW smbclient -mSMB3
fails ?

>> - smbclient [no -m option] is OK and slow (uses default SMB1/NT1?)
>> - mount.cifs -o vers=1.0 is OK and slow
>> - smbclient -m SMB2 (or -m SMB3) fails
> The above is the interesting case, as this is much
> easier to debug (in user space) than the kernel
> client.
>
> Can you give more details on *exactly* HOW smbclient -mSMB3
> fails ?
This is really strange: today I was not able to reproduce the problem. 
Neither with smbclient, nor with mount.cifs.

Tried with 2 of the 3 Windows machines which showed the original 
problem, but today all copies with file sizes from 50 GB up to 150 GB 
went fine, using SMB 3.11.

Well, it looks like we might never know what the problem was.

Thanks anyway,

MI