openssh unix dev - Mar 2025 - Support for transferring sparse files via scp/sftp correctly?

On Tue, Mar 4, 2025, 05:59 Lionel Cons <lionelcons1972 at gmail.com>
wrote:
> On Tue, 4 Mar 2025 at 11:34, Philipp Marek <philipp at marek.priv.at>
wrote:
> >
> > > Does OpenSSH scp/sftp mode transfer sparse files correctly, i.e.
are
> > > holes skipped and not transferred as chunks of 0 bytes? [1]
> > >
> > > We're asking about sparse files in the >= 1PB range, which
consists of
> > > multi-TB holes with around 600-2000GB of valid data.
> >
> > Perhaps rsync would be a good fit here,
> > it supports --sparse.
>
> No, if we would use external tools then mounting NFSv4.2 filesystem
> via
>
https://github.com/kofemann/ms-nfs41-client/blob/master/cygwin/utils/sshnfs/sshnfs.ksh
> would be the tool of choice.
>
Would an external tool be useful if it's already present? Most modern
versions of tar can efficiently pack and unpack sparse files (with the
appropriate command line option). If it matters, I'd check whether BSD tar
and GNU tar are compatible when encoding sparse files, though.

-MD
>

On Tue, 4 Mar 2025 at 13:53, Mitchell Dorrell <mwd at psc.edu>
wrote:
>
> On Tue, Mar 4, 2025, 05:59 Lionel Cons <lionelcons1972 at gmail.com>
wrote:
>>
>> On Tue, 4 Mar 2025 at 11:34, Philipp Marek <philipp at
marek.priv.at> wrote:
>> >
>> > > Does OpenSSH scp/sftp mode transfer sparse files correctly,
i.e. are
>> > > holes skipped and not transferred as chunks of 0 bytes? [1]
>> > >
>> > > We're asking about sparse files in the >= 1PB range,
which consists of
>> > > multi-TB holes with around 600-2000GB of valid data.
>> >
>> > Perhaps rsync would be a good fit here,
>> > it supports --sparse.
>>
>> No, if we would use external tools then mounting NFSv4.2 filesystem
>> via
https://github.com/kofemann/ms-nfs41-client/blob/master/cygwin/utils/sshnfs/sshnfs.ksh
>> would be the tool of choice.
>
>
> Would an external tool be useful if it's already present? Most modern
versions of tar can efficiently pack and unpack sparse files (with the
appropriate command line option). If it matters, I'd check whether BSD tar
and GNU tar are compatible when encoding sparse files, though.
Yes, I can wrap each sparse file in a container: ZIP-packed QEMU VM
image with btrfs containing the needed sparse file.

Really: Built In sparse file support, which is on by default, makes
more sense, as we do not have to maintain&update&administer lots of
tools just to get the job done. It's also less error-prone.

FYI Sparse files are nothing new or magic, they have been around since
the dawn of filesystems, and even WinXP&WinServer2000 have sparse file
support.

Lionel