openssh unix dev - Mar 2025 - Support for transferring sparse files via scp/sftp correctly?

Does OpenSSH scp/sftp mode transfer sparse files correctly, i.e. are
holes skipped and not transferred as chunks of 0 bytes? [1]

We're asking about sparse files in the >= 1PB range, which consists of
multi-TB holes with around 600-2000GB of valid data.

[1] Note the difference: Sparse files can have data sections which
consist of valid 0 bytes. Holes in sparse files represent regions
where no data space is allocated, but they read as 0 bytes

Lionel

> Does OpenSSH scp/sftp mode transfer sparse files correctly, i.e. are
> holes skipped and not transferred as chunks of 0 bytes? [1]
> 
> We're asking about sparse files in the >= 1PB range, which consists
of
> multi-TB holes with around 600-2000GB of valid data.

Perhaps rsync would be a good fit here,
it supports --sparse.

On Tue, 4 Mar 2025 at 11:05, Lionel Cons <lionelcons1972 at gmail.com>
wrote:
>
> Does OpenSSH scp/sftp mode transfer sparse files correctly, i.e. are
> holes skipped and not transferred as chunks of 0 bytes? [1]
>
> We're asking about sparse files in the >= 1PB range, which consists
of
> multi-TB holes with around 600-2000GB of valid data.
>
> [1] Note the difference: Sparse files can have data sections which
> consist of valid 0 bytes. Holes in sparse files represent regions
> where no data space is allocated, but they read as 0 bytes
>
I think proper sparse file support via SEEK_DATA+SEEK_HOLE is a very
good idea. Sparse files are abundantly in use (bioinformatiks in our
case), and it is very annoying to always have to wrap file transfers
in tar --sparse first. Error prone too.
Better sftp should support sparse files directly.

Ced
-- 
Cedric Blancher <cedric.blancher at gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

On Tue, 4 Mar 2025 at 11:01, Lionel Cons <lionelcons1972 at gmail.com>
wrote:
>
> Does OpenSSH scp/sftp mode transfer sparse files correctly, i.e. are
> holes skipped and not transferred as chunks of 0 bytes? [1]
>
> We're asking about sparse files in the >= 1PB range, which consists
of
> multi-TB holes with around 600-2000GB of valid data.
>
> [1] Note the difference: Sparse files can have data sections which
> consist of valid 0 bytes. Holes in sparse files represent regions
> where no data space is allocated, but they read as 0 bytes
>
> Lionel
I filed https://bugzilla.mindrot.org/show_bug.cgi?id=3798 for sparse
file copy support

Lionel