Andriy Gapon
2021-Feb-22 15:13 UTC
lots of "no such file or directory" errors in zfs filesystem
On 22/02/2021 16:20, Chris Anderson wrote:> On Mon, Feb 22, 2021 at 1:36 AM Andriy Gapon <avg at freebsd.org > <mailto:avg at freebsd.org>> wrote: > > On 22/02/2021 09:31, Chris Anderson wrote: > > None of these files are especially important to me, however I was wondering > > if there would be any benefit to the community from trying to debug this > > issue further to understand what might be going wrong. > > Yes. > > > Could you offer any guidance about what kind of debugging information I could > collect that would be of use?You can start with picking a single file that demonstrates the problem. Then, ls -li the-file zdb -dddd file's-filesystem file's-inode-number The filesystem can be found out from df output, the inode number is in ls -li output -- if the command prints anything at all. If it does not, then do ls -lid on the file's directory and then zdb -dddd for the directory's inode number. In the output there should be the file name and its number (I think that it's in hex, but not sure). -- Andriy Gapon
Chris Anderson
2021-Feb-23 03:25 UTC
lots of "no such file or directory" errors in zfs filesystem
On Mon, Feb 22, 2021 at 9:13 AM Andriy Gapon <avg at freebsd.org> wrote:> On 22/02/2021 16:20, Chris Anderson wrote: > > On Mon, Feb 22, 2021 at 1:36 AM Andriy Gapon <avg at freebsd.org > > <mailto:avg at freebsd.org>> wrote: > > > > On 22/02/2021 09:31, Chris Anderson wrote: > > > None of these files are especially important to me, however I was > wondering > > > if there would be any benefit to the community from trying to > debug this > > > issue further to understand what might be going wrong. > > > > Yes. > > > > > > Could you offer any guidance about what kind of debugging information I > could > > collect that would be of use? > > You can start with picking a single file that demonstrates the problem. > Then, > ls -li the-file > zdb -dddd file's-filesystem file's-inode-number > The filesystem can be found out from df output, the inode number is in ls > -li > output -- if the command prints anything at all. > If it does not, then do ls -lid on the file's directory and then zdb -dddd > for > the directory's inode number. In the output there should be the file name > and > its number (I think that it's in hex, but not sure). >so I can't ls -i the file since that triggers the no such file warning. if I run zdb -dddd on the inode of a directory which contains one of those missing files, I can get the inode of the file from that, but I don't get anything particularly interesting in the output. most of the files that are missing are in directories with a large number of files (the largest has 180k) but I managed to find a directory which had a single file entry that is missing: Dataset tank/home/cva [ZPL], ID 196, cr_txg 163, 109G, 908537 objects, rootbp DVA[0]=<0:13210311000:1000> DVA[1]=<0:18b9a02c000:1000> [L0 DMU objset] fletcher4 uncompressed LE contiguous unique double size=800L/800P birth=46916371L/46916371P fill=908537 cksum=11fdd21d1d:13cb24c87a6e:da0c9bf1b5df3:715ab2ec45b7b09 Object lvl iblk dblk dsize dnsize lsize %full type 38268 1 128K 1K 0 512 1K 100.00 ZFS directory 264 bonus ZFS znode dnode flags: USED_BYTES USERUSED_ACCOUNTED dnode maxblkid: 0 uid 1001 gid 1001 atime Sun Aug 6 02:00:41 2017 mtime Wed Apr 15 12:12:42 2020 ctime Wed Apr 15 12:12:42 2020 crtime Sat Aug 5 15:10:07 2017 gen 23881023 mode 40755 size 3 parent 38176 links 2 pflags 40800000144 xattr 0 rdev 0x0000000000000000 microzap: 1024 bytes, 1 entries hash_test.go = 38274 (type: Regular File) # zdb -dddd tank/home/cva 38274 Dataset tank/home/cva [ZPL], ID 196, cr_txg 163, 109G, 908537 objects, rootbp DVA[0]=<0:13210311000:1000> DVA[1]=<0:18b9a02c000:1000> [L0 DMU objset] fletcher4 uncompressed LE contiguous unique double size=800L/800P birth=46916371L/46916371P fill=908537 cksum=11fdd21d1d:13cb24c87a6e:da0c9bf1b5df3:715ab2ec45b7b09 Object lvl iblk dblk dsize dnsize lsize %full type zdb: dmu_bonus_hold(38274) failed, errno 2> > -- > Andriy Gapon >