On Thu, 27 Oct 2016 16:57:13 -0700, Jeremy Allison <jra at samba.org> wrote :> On Fri, Oct 28, 2016 at 01:42:35AM +0200, Saint Germain via samba > wrote: > > On Thu, 27 Oct 2016 16:27:05 -0700, Jeremy Allison <jra at samba.org> > > > if you see any of these then it happened. > > > > Ok I understand how to check that it happened, but how can I make > > sure that it doesn't happen ? > > How can I ensure that clone range request sizes are integer > > divisible by 4096 on my server ? > > Should I change something on the BTRFS filesystem, or shift my > > partition or something ? > > Nope. It's not a critical error. In smbclient, scopy > requests a copy of the entire filesize. The libsmbclient > layer splits this into copychunk requests of default > size 1mb - which is certainly integer divisible by > 4096. > > The only time you'll get the server copy is if the > last chunk isn't divisible by 4096. It's not a problem.Thank you very much for all the help and explanations. It's much clearer now. So if I understand correctly, as long as the client send the FSCTL_SRV_COPYCHUNK, it should be ok. The FSCTL_SRV_COPYCHUNK instructions seems to be included in SMB2 in smbclient starting from version 4.3. On linux, most current utilities like Nautilus (gvfs) still use SMB1 to connect, so there is no chance to trigger a server-side copy. On Mac, it seems to be using SMB3, so it should be ok. On Windows, I'll have to check. Thanks for your help !
On Fri, Oct 28, 2016 at 02:02:57AM +0200, Saint Germain via samba wrote:> > > On Thu, 27 Oct 2016 16:57:13 -0700, Jeremy Allison <jra at samba.org> > wrote : > > > On Fri, Oct 28, 2016 at 01:42:35AM +0200, Saint Germain via samba > > wrote: > > > On Thu, 27 Oct 2016 16:27:05 -0700, Jeremy Allison <jra at samba.org> > > > > if you see any of these then it happened. > > > > > > Ok I understand how to check that it happened, but how can I make > > > sure that it doesn't happen ? > > > How can I ensure that clone range request sizes are integer > > > divisible by 4096 on my server ? > > > Should I change something on the BTRFS filesystem, or shift my > > > partition or something ? > > > > Nope. It's not a critical error. In smbclient, scopy > > requests a copy of the entire filesize. The libsmbclient > > layer splits this into copychunk requests of default > > size 1mb - which is certainly integer divisible by > > 4096. > > > > The only time you'll get the server copy is if the > > last chunk isn't divisible by 4096. It's not a problem. > > Thank you very much for all the help and explanations. > It's much clearer now. > > So if I understand correctly, as long as the client send the > FSCTL_SRV_COPYCHUNK, it should be ok. > > The FSCTL_SRV_COPYCHUNK instructions seems to be included in SMB2 in > smbclient starting from version 4.3. > > On linux, most current utilities like Nautilus (gvfs) still use SMB1 to > connect, so there is no chance to trigger a server-side copy.Well if you set the default to be SMB2 in your smb.conf then gvfs will connect using SMB2. However, gvfs would have to get the 'splice' function signature using smbc_getFunctionSplice() and then call it in order to use this. I'm not sure if gvfs does that.
On Fri, Oct 28, 2016 at 02:02:57AM +0200, Saint Germain via samba wrote:> > On linux, most current utilities like Nautilus (gvfs) still use SMB1 to > connect, so there is no chance to trigger a server-side copy.Oh, just checked the current gvfs source code to copy an smb file. It does: static gboolean copy_file (GVfsBackendSmb *backend, GVfsJob *job, const char *from_uri, const char *to_uri) { SMBCFILE *from_file, *to_file; char buffer[4096]; size_t buffer_size; ssize_t res; char *p; gboolean succeeded; smbc_open_fn smbc_open; smbc_read_fn smbc_read; smbc_write_fn smbc_write; smbc_close_fn smbc_close; from_file = NULL; to_file = NULL; succeeded = FALSE; smbc_open = smbc_getFunctionOpen (backend->smb_context); smbc_read = smbc_getFunctionRead (backend->smb_context); smbc_write = smbc_getFunctionWrite (backend->smb_context); smbc_close = smbc_getFunctionClose (backend->smb_context); Which clearly doesn't use the splice call. That's broken. If there's anyone on the Gnome gvfs team please let me know and let's get this fixed. Looking around inside that source code I also see: do_read (GVfsBackend *backend, GVfsJobRead *job, GVfsBackendHandle handle, char *buffer, gsize bytes_requested) { GVfsBackendSmb *op_backend = G_VFS_BACKEND_SMB (backend); ssize_t res; smbc_read_fn smbc_read; /* libsmbclient limits blocksize to (64*1024)-2 for Windows servers, * let's do the same here to achieve reasonable performance. (#588391) * * TODO: port to pull mechanism (#592468) */ if (bytes_requested > 65534) bytes_requested = 65534; smbc_read = smbc_getFunctionRead (op_backend->smb_context); res = smbc_read (op_backend->smb_context, (SMBCFILE *)handle, buffer, bytes_requested); which is also completely broken and wrong. The read call can take any arbitrary size and the both the libsmbclient SMB1 and SMB2 engines will break this down into as many simultaneous on the wire calls as needed to pipeline the reads / writes. Restricting this to 65534 is *insane* and slow. That whole chunk of code restricting the size needs to be removed.
On Thu, Oct 27, 2016 at 05:22:14PM -0700, Jeremy Allison via samba wrote:> > Looking around inside that source code I also see: > > do_read (GVfsBackend *backend, > GVfsJobRead *job, > GVfsBackendHandle handle, > char *buffer, > gsize bytes_requested) > { > GVfsBackendSmb *op_backend = G_VFS_BACKEND_SMB (backend); > ssize_t res; > smbc_read_fn smbc_read; > > /* libsmbclient limits blocksize to (64*1024)-2 for Windows servers, > * let's do the same here to achieve reasonable performance. (#588391) > * > * TODO: port to pull mechanism (#592468) > */ > if (bytes_requested > 65534) > bytes_requested = 65534; > > smbc_read = smbc_getFunctionRead (op_backend->smb_context); > res = smbc_read (op_backend->smb_context, (SMBCFILE *)handle, buffer, bytes_requested); > > which is also completely broken and wrong. The read call > can take any arbitrary size and the both the libsmbclient > SMB1 and SMB2 engines will break this down into as many > simultaneous on the wire calls as needed to pipeline > the reads / writes. > > Restricting this to 65534 is *insane* and slow. That > whole chunk of code restricting the size needs to be > removed.Oh it's my fault from 2009: https://bugzilla.gnome.org/show_bug.cgi?id=588391#c10 :-). I'll log another bug to get the whole thing removed now we pipeline correctly.