samba - Feb 2023 - Missing Files/Missing Folders from an NFS Share

On Thu, Feb 16, 2023 at 08:46:26PM +0100, Conor Armstrong
wrote:
>   I'm trying to get to the?bottom of where the normal directory
functions
>   differ in results for block storage compared to NFS.?
>   I created a short c script to look at the offsets generated for the
>   directory when block storage backed in comparison to NFS backed.?
>   block storage backed are always positive:
>
>   File: 5050, Offset: 9213006461214587617
>
>   File: 6567, Offset: 9218690536865144001
>
>   File: 4033, Offset: 9220247894244709239
>
>   ...
>
>   NFS backed are negative:
>
>   File: 2535, Offset: -376809161603969956
>
>   File: 4236, Offset: -376809161586095012
>
>   File: 3507, Offset: -376809161585570724
>
>   ...
>
>   The negative NFS negative offsets seem to be managed already
>   in?source3/smbd/dir.c in the?map_dir_offset_to_wire() function which
>   adjusts for signed/unsigned longs.
>   You mention that?NFS doesn't have a working
>   seekdir()/telldir()/rewinddir(). Other than the different offset (cookie)
>   management, do you know of any other differences?
>   I had a look at the wireshark dumps. I'm getting a STATUS_PENDING
shortly
>   followed by a STATUS_CANCELLED for the NFS share which are absent on the
>   block storage share.
Samba relies on being able to do offset = telldir() and then
seekdir(offset) returning us to the same position in the
directory listing.

If you think about this you'll see why. Imaginary case,
directory with files a-z. Client does findfirst with 1k
buffer, server enumerates files one by one, but when
it gets to "m" it finds it doesn't have enough room
to marshall the client response.

No worries, we saved off the 'offset = telldir()' before
doing the readdir that returned filename "m". So we
seek back to the offset before the filename "m" in
this code:

         if (NT_STATUS_EQUAL(status, STATUS_MORE_ENTRIES)) {
                 dptr_SeekDir(dirptr, prev_dirpos);
                 return status;
         }

and return the findfirst reply with files a-l.

Now when the listing continues, it'll do so with
filename "m" etc. until the listing is done.

If:

offset = telldir()
seekdir(offset)

doesn't return us to the same position in the directory
listing, then we are broken and will not return the complete
list of files.

This is what Ralph is saying when he says that NFS is
broken w.r.t listing directories. What NFS server are
you using ?

This looks relevent:

https://lore.kernel.org/all/CAAmbk-e-YQAPo6QyNB0aJyc9qzUShmEC+x5eTR7wqp1ABWADsg
at mail.gmail.com/T/

Client then does findnext,

On Thu, Feb 16, 2023 at 12:20:27PM -0800, Jeremy Allison via samba
wrote:
>
>doesn't return us to the same position in the directory
>listing, then we are broken and will not return the complete
>list of files.
>
>This is what Ralph is saying when he says that NFS is
>broken w.r.t listing directories. What NFS server are
>you using ?
Oh I just remembered, you're using the Amazon NFS
server.
>This looks relevent:
>
>https://lore.kernel.org/all/CAAmbk-e-YQAPo6QyNB0aJyc9qzUShmEC+x5eTR7wqp1ABWADsg
at mail.gmail.com/T/
Indeed. Note the comment from Trond (NFS client
maintainer in the Linux kernel).

---------------------------------------------------------------
On Thu, 2022-09-08 at 10:45 +0100, Chris Chilvers wrote:
> I should have flagged this as a bug on the first email.
> 
> I've run into an interesting issue using VAST data, their NFS
> implementation
> makes use of the full 64 bit unsigned range allowed by NFS cookie for
> READDIR
> and READDIRPLUS.
> 
> The issue is that this 64 unsigned value is used for the directory's
> file
> position (d_off), which is a 64 bit signed value. This can cause
> readdir and
> telldir to return negative values.
Known issue that we had to deal with 25 years ago for 32 bit systems
when glibc first decided to make lseek() return signed values (struct
old_linux_direct still has it as an unsigned value).

So VAST have had a long time to learn not to do this...

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust at hammerspace.com
---------------------------------------------------------------

Looks like Amazon may have the same bug.