Cedric Blancher
2025-Mar-05 08:57 UTC
Support for transferring sparse files via scp/sftp correctly?
On Tue, 4 Mar 2025 at 21:22, Chris Rapier <rapier at psc.edu> wrote:> > > > On 3/4/25 05:34, Philipp Marek via openssh-unix-dev wrote: > >> Does OpenSSH scp/sftp mode transfer sparse files correctly, i.e. are > >> holes skipped and not transferred as chunks of 0 bytes? [1] > >> > >> We're asking about sparse files in the >= 1PB range, which consists of > >> multi-TB holes with around 600-2000GB of valid data. > > > > > > Perhaps rsync would be a good fit here, > > it supports --sparse. > > _______________________________________________ > > openssh-unix-dev mailing list > > openssh-unix-dev at mindrot.org > > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev > > I think one of the issues you are going to face is that SEEK_DATA and > SEEK_HOLE don't seem to be currently supported under OpenBSD. Since > that's the home OS for OpenSSH this could create portability issues. > While you can get around that with the judicious use of defines it means > that the feature set will start to shift between different OSes.OpenBSD unfortunately does not implement so many other APIs. But other OS do implement SEEK_DATA+SEEK_HOLE, including FreeBSD, Linux, Solaris, Illumos and even Cygwin. Even NFS has a SEEK to lookup holes and data sections in files. SEEK_HOLE+SEEK_DATA are also now part of the POSIX standard, so IMO it is time to face the bug that sparse files are not handled correctly and fix it> > Personally I think it's a good idea and I may explore it for HPN-SSH but > I think it's going to be a hard sell for the OpenBSD community.Selling points: - Users need it - Sparse files were always part of UNIX - It's part of the POSIX standard: https://pubs.opengroup.org/onlinepubs/9799919799/functions/lseek.html Ced -- Cedric Blancher <cedric.blancher at gmail.com> [https://plus.google.com/u/0/+CedricBlancher/] Institute Pasteur
Damien Miller
2025-Mar-07 05:38 UTC
Support for transferring sparse files via scp/sftp correctly?
On Wed, 5 Mar 2025, Cedric Blancher wrote:> On Tue, 4 Mar 2025 at 21:22, Chris Rapier <rapier at psc.edu> wrote: > > > > > > > > On 3/4/25 05:34, Philipp Marek via openssh-unix-dev wrote: > > >> Does OpenSSH scp/sftp mode transfer sparse files correctly, i.e. are > > >> holes skipped and not transferred as chunks of 0 bytes? [1] > > >> > > >> We're asking about sparse files in the >= 1PB range, which consists of > > >> multi-TB holes with around 600-2000GB of valid data. > > > > > > > > > Perhaps rsync would be a good fit here, > > > it supports --sparse. > > > _______________________________________________ > > > openssh-unix-dev mailing list > > > openssh-unix-dev at mindrot.org > > > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev > > > > I think one of the issues you are going to face is that SEEK_DATA and > > SEEK_HOLE don't seem to be currently supported under OpenBSD. Since > > that's the home OS for OpenSSH this could create portability issues. > > While you can get around that with the judicious use of defines it means > > that the feature set will start to shift between different OSes. > > OpenBSD unfortunately does not implement so many other APIs. But other > OS do implement SEEK_DATA+SEEK_HOLE, including FreeBSD, Linux, > Solaris, Illumos and even Cygwin. Even NFS has a SEEK to lookup holes > and data sections in files. > SEEK_HOLE+SEEK_DATA are also now part of the POSIX standard, so IMO it > is time to face the bug that sparse files are not handled correctly > and fix itYou and the others on this thread are IIRC the first people in sftp's 24 year history to ever ask for sparse file support. Its absence is not a bug and adding it will almost certainly require new protocol extensions. Being pushy with vounteer developers, telling us what our priorities should be, assigning us work, etc. will not have the result you want. If you want this to happen, I recommend starting by figuring out what protocol extensions need to be made, and how to support sparse files on system without SEEK_DATA/HOLE - it should be pretty to do this on upload without these flags and without extensions. -d
Apparently Analagous Threads
- Support for transferring sparse files via scp/sftp correctly?
- Support for transferring sparse files via scp/sftp correctly?
- Support for transferring sparse files via scp/sftp correctly?
- Support for transferring sparse files via scp/sftp correctly?
- Re: [PATCH nbdkit v5 FINAL 15/19] file: Implement extents.