openssh unix dev - Mar 2025 - Support for transferring sparse files via scp/sftp correctly?

On Tue, 4 Mar 2025 at 11:34, Philipp Marek <philipp at marek.priv.at>
wrote:
>
> > Does OpenSSH scp/sftp mode transfer sparse files correctly, i.e. are
> > holes skipped and not transferred as chunks of 0 bytes? [1]
> >
> > We're asking about sparse files in the >= 1PB range, which
consists of
> > multi-TB holes with around 600-2000GB of valid data.
>
> Perhaps rsync would be a good fit here,
> it supports --sparse.
No, if we would use external tools then mounting NFSv4.2 filesystem
via
https://github.com/kofemann/ms-nfs41-client/blob/master/cygwin/utils/sshnfs/sshnfs.ksh
would be the tool of choice.

I'm talking about NATIVE sparse file support in scp/sftp via
SEEK_DATA/SEEK_HOLE on POSIX
(https://pubs.opengroup.org/onlinepubs/9799919799/functions/lseek.html)
or Windows  FSCTL_QUERY_ALLOCATED_RANGES
(https://learn.microsoft.com/en-us/windows/win32/api/winioctl/ni-winioctl-fsctl_query_allocated_ranges).
We need that because people have tried to copy sparse file before, and
it either RUINED the files (by making them non-sparse, the holes were
filled-in with 0 byte data), or copies took forever because the holes
in sparse files are very large (e.g. multi TB size).

Lionel

On Tue, Mar 4, 2025, 05:59 Lionel Cons <lionelcons1972 at gmail.com>
wrote:
> On Tue, 4 Mar 2025 at 11:34, Philipp Marek <philipp at marek.priv.at>
wrote:
> >
> > > Does OpenSSH scp/sftp mode transfer sparse files correctly, i.e.
are
> > > holes skipped and not transferred as chunks of 0 bytes? [1]
> > >
> > > We're asking about sparse files in the >= 1PB range, which
consists of
> > > multi-TB holes with around 600-2000GB of valid data.
> >
> > Perhaps rsync would be a good fit here,
> > it supports --sparse.
>
> No, if we would use external tools then mounting NFSv4.2 filesystem
> via
>
https://github.com/kofemann/ms-nfs41-client/blob/master/cygwin/utils/sshnfs/sshnfs.ksh
> would be the tool of choice.
>
Would an external tool be useful if it's already present? Most modern
versions of tar can efficiently pack and unpack sparse files (with the
appropriate command line option). If it matters, I'd check whether BSD tar
and GNU tar are compatible when encoding sparse files, though.

-MD
>