openssh bugs - Mar 2025 - [Bug 3798] New: Add native sparse file support (SEEK_HOLE, SEEK_DATA) to scp/sftp

https://bugzilla.mindrot.org/show_bug.cgi?id=3798

            Bug ID: 3798
           Summary: Add native sparse file support (SEEK_HOLE, SEEK_DATA)
                    to scp/sftp
           Product: Portable OpenSSH
           Version: 9.9p2
          Hardware: All
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: sftp-server
          Assignee: unassigned-bugs at mindrot.org
          Reporter: Lionelcons1972 at gmail.com

Please add NATIVE sparse file support in scp/sftp via
SEEK_DATA/SEEK_HOLE on
POSIX(https://pubs.opengroup.org/onlinepubs/9799919799/functions/lseek.html)
or Windows  FSCTL_QUERY_ALLOCATED_RANGES
(https://learn.microsoft.com/en-us/windows/win32/api/winioctl/ni-winioctl-fsctl_query_allocated_ranges).

We need that because people have tried to copy sparse file before, and
it either RUINED the files (by making them non-sparse, the holes were
filled-in with 0 byte data), or copies took forever because the holes
in sparse files are very large (e.g. multi TB size).

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3798

Lionel Cons <Lionelcons1972 at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |Lionelcons1972 at gmail.com

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3798

Lionel Cons <Lionelcons1972 at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|unassigned-bugs at mindrot.org |dtucker at dtucker.net

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3798

Christoph Anton Mitterer <calestyo at scientia.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |calestyo at scientia.org

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3798

--- Comment #1 from Lionel Cons <Lionelcons1972 at gmail.com>
---
>From openssh dev mailing list:
I like the following idea to have a new sftp GET_PLUS request, which is
designed like NFSv4.2 READ_PLUS:

On Wed, Mar 5, 2025 at 11:14?AM Darren Tucker <dtucker at dtucker.net>
wrote:
> On Tue, Mar 04, 2025 at 02:43:10PM +0100, Lionel Cons wrote:
> [...]
> > Really: Built In sparse file support, which is on by default, makes
> > more sense, as we do not have to maintain&update&administer
lots of
> > tools just to get the job done. It's also less error-prone.
> >
> > FYI Sparse files are nothing new or magic, they have been around since
> > the dawn of filesystems, and even WinXP&WinServer2000 have sparse
file
> > support.
>
> I wasn't aware that the SEEK_HOLE and SEEK_DATA had even been
> standardised, although it looks like that was only some time last year.
> As others have noted it's still not universally available.
>
> Having looked at it:
[snip]
>  - I don't see sufficient information available in the sftp protocol
>    from the server to the client to support it for client "get".
>    Certainly the secsh-filefxer-02[0] (ie v3) version that OpenSSH
>    implements doesn't, but even the most recent -13 drafts only seem
>    to support only a per-file boolean that indicates if it's on or off.
>    I don't see a way for a client to determine the location and/or size
of
>    any holes in a remote file in order to replicate them on a downloaded
>    file.  The only way I can see it could be supported is by adding a
>    vendor extension (which would need to be supported by both client
>    and server) that could supply the information about holes/extents,
>    which would be a larger undertaking.
FYI NFSv4.2 added a READ_PLUS operation to implement reading sparse
files efficiently:
Basically each time the NFSv.2 client does a READ_PLUS (see
https://datatracker.ietf.org/doc/html/rfc7862#page-86) the server
returns an array of elements. Each element can either be a "data"
element with { data_offset, len, data } XOR an "hole" element with {
hole_offset, hole_len }.
IMHO the sftp protocol could do the same.

There is even one optimisation:
The standards people also debated whether to support a 3rd return
type, an "application data block" (see
https://datatracker.ietf.org/doc/html/rfc7862#section-15.12), which
consists of a { data_offset, data_total_length, data_pattern_bytes,
data_pattern_bytes_len, number_of_data_patterns }.
The idea is that if an application stores repeated patterns in a file
the server can return this as "ADB". Use case examples include filling
files with 0xdeadbeef to invalidate data, or fill everything with
'\0'-bytes (which are valid data, holes in sparse files mean "no
here
here").
But that is just an optimization, and IMHO the current sftp/scp-work
support for sparse files should handle only support for holes+data -
but maybe keep the protocol flexible enough to (later!) add ADB
support.
> +/*
> + * Check a potentially-sparse file for location of holes and data,
starting
> + * from "offset".  If the next hole points to EOF, there are no
remaining holes.
> + */
> +static void
> +sftp_check_sparse_file(int fd, off_t offset, off_t *data_offset,
> +    off_t *hole_offset)
> +{
> +#if defined(SEEK_HOLE) && defined(SEEK_DATA)
> +       if ((*hole_offset = lseek(fd, offset, SEEK_HOLE)) == -1)
> +               fatal_f("lseek(SEEK_HOLE): %s", strerror(errno));
> +       if ((*data_offset = lseek(fd, offset, SEEK_DATA)) == -1)
> +               fatal_f("lseek(SEEK_DATA): %s", strerror(errno));
> +#else
> +       /* No sparse file support, assume data spans start to end. */
> +       *data_offset = offset;
> +       if ((*hole_offset = lseek(fd, offset, SEEK_END)) == -1)
> +               fatal_f("lseek(SEEK_SET): %s", strerror(errno));
> +#endif
> +       if (lseek(fd, offset, SEEK_SET) == -1) /* restore cursor */
> +               fatal_f("lseek(SEEK_SET): %s", strerror(errno));
> +       debug3_f("offset %llu data_offset %llu hole_offset %llu",
> +           (unsigned long long)offset, (unsigned long long)*data_offset,
> +           (unsigned long long)*hole_offset);
[snip]

Notes (I didn't test the code yet):
* Code for this must cover:
- Normal, non-sparse files
- Sparse files which only consists of a single hole, no data
- Sparse files which begin with a hole
- Sparse files which begin with data
- Sparse files which end with a hole
- Sparse files which end with data
- Sparse files with 60000 holes (no joke, at SUN we had customers who
had many more holes in files)

----

Bye,
Roland

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3798

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|dtucker at dtucker.net         |unassigned-bugs at
mindrot.org

-- 
You are receiving this mail because:
You are watching the assignee of the bug.