On Fri, May 01, 2020 at 09:27:58PM +0200, A L via samba wrote:> > Jeremy. > I did not use a command line, but rather File Explorer > 1) In File explorer, go to \\SAMBA\share_io_uring\ > 2) Select folder "test2-ro" and choose copy > 3) Paste to a local drive > > The amount of files not working seems to vary, but usually within the 10 > first copied files. You can download a copy of the source test files from > https://mirrors.tnonline.net/?dir=samba including all checksums. > > The file https://mirrors.tnonline.net/samba/test2-ro_no_io_uring.7z contains > 1000 copies of the source file > The file https://mirrors.tnonline.net/samba/test2-ro_with_io_uring.7z > contains the same files as they ended up after copying to the Windows > client. Here you can see where the holes (bunch of zeroed data) in the files > are created at different offsets. > > I hope this helps!OK, I just did this locally with 100 10MB duplicate files. Completely correct SHA256 output for every file on both Windows and Linux after the copy. Note that these are *separate* files on ext4, not created with --reflink (ext4 doesn't support it). The Windows client first tries to do the FSCTL_OFFLOAD_READ (which our btrfs module does support, but not the default ext4 module) and then falls back to regular SMB2 async read, ramping up from 32k initially to 1MB reads. I'm starting to think the problem is in your btrfs filesystem, not the Samba uring module. Before I spend more time on this I'd like you to create a standard ext4 filesystem, create all the files with cp *without* using reflink (which ext4 doesn't support) so we know there's no COW shenanigans and then copy from *that* filesystem to the Windows client using the iouring module. I'm not a betting man, but if I was I'd bet the problem dissapears :-).
On 2020-05-01 22:04, Jeremy Allison via samba wrote:> On Fri, May 01, 2020 at 09:27:58PM +0200, A L via samba wrote: >>> Jeremy. >> I did not use a command line, but rather File Explorer >> 1) In File explorer, go to \\SAMBA\share_io_uring\ >> 2) Select folder "test2-ro" and choose copy >> 3) Paste to a local drive >> >> The amount of files not working seems to vary, but usually within the 10 >> first copied files. You can download a copy of the source test files from >> https://mirrors.tnonline.net/?dir=samba including all checksums. >> >> The file https://mirrors.tnonline.net/samba/test2-ro_no_io_uring.7z contains >> 1000 copies of the source file >> The file https://mirrors.tnonline.net/samba/test2-ro_with_io_uring.7z >> contains the same files as they ended up after copying to the Windows >> client. Here you can see where the holes (bunch of zeroed data) in the files >> are created at different offsets. >> >> I hope this helps! > OK, I just did this locally with 100 10MB duplicate files. > > Completely correct SHA256 output for every file on both > Windows and Linux after the copy. > > Note that these are *separate* files on ext4, not created > with --reflink (ext4 doesn't support it).I forgot to mention that when copied the files to a SSD, then I did not use any reflinks or hard links, just plain copies.> The Windows client first tries to do the FSCTL_OFFLOAD_READ > (which our btrfs module does support, but not the default > ext4 module) and then falls back to regular SMB2 async > read, ramping up from 32k initially to 1MB reads.Is this the "vfs module = btrfs". This was disabled too in the latest tests.> I'm starting to think the problem is in your btrfs > filesystem, not the Samba uring module. > > Before I spend more time on this I'd like you to > create a standard ext4 filesystem, create all the > files with cp *without* using reflink (which ext4 > doesn't support) so we know there's no COW shenanigans > and then copy from *that* filesystem to the Windows client > using the iouring module. > > I'm not a betting man, but if I was I'd bet the problem > dissapears :-). >So, I found a 160GB USB disk that I formatted to ext4 and exported it as: ######## [share_io_uring_ext4] ??? comment = ext4-test io_uring ??? path = /mnt/ext4 ??? browseable = yes ??? read only = yes ??? guest only = Yes ??? guest ok = Yes ??? vfs objects = io_uring [share_no_io_uring_ext4] ??? comment = ext4-test no_io_uring ??? path = /mnt/ext4 ??? browseable = yes ??? read only = yes ??? guest only = Yes ??? guest ok = Yes ??? vfs objects ######## I also tried various combinations of io_uring settings. io_uring:sqpoll = true/false io_uring:num_entries = 2, 4, 8, 64, 128 (default), 256 Between each test i do "echo 3 > /proc/sys/vm/drop_caches" and restart the samba service. One difference I noticed is that there is now only one [io_wqe_worker-0] io thread active when copying from this USB disk. This is probably because it is so slow, about 25MB/s maximum read speed. This seems to have an impact on where the corruptions happen in the files. All files have a corrupt/zeroed block at the end of the file, 99% of the time at the same place, as seen here https://paste.tnonline.net/files/Zdq7YKOfzogc_ext4_copy_io_uring.png So, now I think we have at least ruled out Btrfs as filesystem as culprit with the io_uring. You mentioned a wire trace of the SMB traffic. Can you provide some examples on how I do that? Perhaps I should open a bug on the Bugzilla tracker and continue this discussion there? Regards, Anders
On Sat, May 02, 2020 at 11:34:41AM +0200, A L wrote:> ######## > I also tried various combinations of io_uring settings. > > io_uring:sqpoll = true/false > io_uring:num_entries = 2, 4, 8, 64, 128 (default), 256 > > Between each test i do "echo 3 > /proc/sys/vm/drop_caches" and restart the > samba service. > > One difference I noticed is that there is now only one [io_wqe_worker-0] io > thread active when copying from this USB disk. This is probably because it > is so slow, about 25MB/s maximum read speed. This seems to have an impact on > where the corruptions happen in the files. All files have a corrupt/zeroed > block at the end of the file, 99% of the time at the same place, as seen > here https://paste.tnonline.net/files/Zdq7YKOfzogc_ext4_copy_io_uring.png > > So, now I think we have at least ruled out Btrfs as filesystem as culprit > with the io_uring. > > You mentioned a wire trace of the SMB traffic. Can you provide some examples > on how I do that?Wireshark. Capture on TCP port 445.> Perhaps I should open a bug on the Bugzilla tracker and continue this > discussion there?Thanks.
On Sat, May 2, 2020, 05:36 A L via samba <samba at lists.samba.org> wrote:> > On 2020-05-01 22:04, Jeremy Allison via samba wrote: > > On Fri, May 01, 2020 at 09:27:58PM +0200, A L via samba wrote: > >>> Jeremy. > >> I did not use a command line, but rather File Explorer > >> 1) In File explorer, go to \\SAMBA\share_io_uring\ > >> 2) Select folder "test2-ro" and choose copy > >> 3) Paste to a local drive > >> > >> The amount of files not working seems to vary, but usually within the 10 > >> first copied files. You can download a copy of the source test files > from > >> https://mirrors.tnonline.net/?dir=samba including all checksums. > >> > >> The file https://mirrors.tnonline.net/samba/test2-ro_no_io_uring.7z > contains > >> 1000 copies of the source file > >> The file https://mirrors.tnonline.net/samba/test2-ro_with_io_uring.7z > >> contains the same files as they ended up after copying to the Windows > >> client. Here you can see where the holes (bunch of zeroed data) in the > files > >> are created at different offsets. > >> > >> I hope this helps! > > OK, I just did this locally with 100 10MB duplicate files. > > > > Completely correct SHA256 output for every file on both > > Windows and Linux after the copy. > > > > Note that these are *separate* files on ext4, not created > > with --reflink (ext4 doesn't support it). > I forgot to mention that when copied the files to a SSD, then I did not > use any reflinks or hard links, just plain copies. > > The Windows client first tries to do the FSCTL_OFFLOAD_READ > > (which our btrfs module does support, but not the default > > ext4 module) and then falls back to regular SMB2 async > > read, ramping up from 32k initially to 1MB reads. > Is this the "vfs module = btrfs". This was disabled too in the latest > tests. > > I'm starting to think the problem is in your btrfs > > filesystem, not the Samba uring module. > > > > Before I spend more time on this I'd like you to > > create a standard ext4 filesystem, create all the > > files with cp *without* using reflink (which ext4 > > doesn't support) so we know there's no COW shenanigans > > and then copy from *that* filesystem to the Windows client > > using the iouring module. > > > > I'm not a betting man, but if I was I'd bet the problem > > dissapears :-). > > > So, I found a 160GB USB disk that I formatted to ext4 and exported it as: > ######## > [share_io_uring_ext4] > comment = ext4-test io_uring > path = /mnt/ext4 > browseable = yes > read only = yes > guest only = Yes > guest ok = Yes > vfs objects = io_uring > > [share_no_io_uring_ext4] > comment = ext4-test no_io_uring > path = /mnt/ext4 > browseable = yes > read only = yes > guest only = Yes > guest ok = Yes > vfs objects > ######## > I also tried various combinations of io_uring settings. > > io_uring:sqpoll = true/false > io_uring:num_entries = 2, 4, 8, 64, 128 (default), 256 > > Between each test i do "echo 3 > /proc/sys/vm/drop_caches" and restart > the samba service. > > One difference I noticed is that there is now only one [io_wqe_worker-0] > io thread active when copying from this USB disk. This is probably > because it is so slow, about 25MB/s maximum read speed. This seems to > have an impact on where the corruptions happen in the files. All files > have a corrupt/zeroed block at the end of the file, 99% of the time at > the same place, as seen here > https://paste.tnonline.net/files/Zdq7YKOfzogc_ext4_copy_io_uring.png > > So, now I think we have at least ruled out Btrfs as filesystem as > culprit with the io_uring. > > You mentioned a wire trace of the SMB traffic. Can you provide some > examples on how I do that? > > Perhaps I should open a bug on the Bugzilla tracker and continue this > discussion there? >If you do, please include a link in this thread. Thanks. Regards,> Anders > > -- > To unsubscribe from this list go to the following URL and read the > instructions: https://lists.samba.org/mailman/options/samba >