Gluster users - Aug 2018 - Bug with hardlink limitation in 3.12.13 ?

Hello,

I got yesterday unexpected error "No space left on device" on my new
gluster volume caused by too many hardlinks.
This happened while I done "rsync --aAHXxv ..." replication from old
gluster to new gluster servers - each running latest version 3.12.13
(for changing volume schema from 2x2 to 3x1 with quorum and a fresh
Debian Stretch setup instead Jessie).

When I deduplicated it around half a year ago with "rdfind"
hardlinking
was working fine (I think that was glusterfs around version 3.12.8 -
3.12.10 ?)

My search for documentation found only the parameter
"storage.max-hardlinks" with default of 100 for version 4.0.
I checked it in my gluster 3.12.13 but here the parameter is not yet
implemented.

I tested/proofed it by running my small test on underlaying ext4
filesystem brick directly and on gluster volume using same ext4
filesystem of the brick:

Testline for it:
??? ??? ??? mkdir test; cd test; echo "hello" > test; for I in
$(seq 1
100); do ln test test-$I ; done

* on ext4 fs (old brick: xfs) I could do 100 hardlinks without problems
(from documentation I found ext has 65.000 hardlinks compiled in )
* on actual GlusterFS (same on my old and new gluster volumes) I could
do only up to 45 hardlinks now

But from deduplication around 6 months ago I could find e.g. a file with
240 hardlinks setup and there is no problem using these referenced files
(caused by multiple languages / multiple uploads per language ,
production/staging system cloned... ).

My actual workaround has to be using duplicated content but it would be
great if this could be fixed in next versions ;)

(Saltstack didn't support yet successful setup of glusterfs 4.0
peers/volumes; something in output of "gluster --xml --mode=script"
call
must be weird but I haven't seen any differences so far)

Bests


Reiner

On 08/31/2018 07:15 AM, Reiner Keller wrote:
> Hello,
> 
> I got yesterday unexpected error "No space left on device" on my
new
> gluster volume caused by too many hardlinks.
> This happened while I done "rsync --aAHXxv ..." replication from
old
> gluster to new gluster servers - each running latest version 3.12.13
> (for changing volume schema from 2x2 to 3x1 with quorum and a fresh
> Debian Stretch setup instead Jessie).
I suspect you have hit this:
https://bugzilla.redhat.com/show_bug.cgi?id=1602262#c5

I further suspect your older setup was 3.10 based and not 3.12 based.

There is an additional feature added in 3.12 that stores GFID to path
conversion details using xattrs (see "GFID to path" in
https://docs.gluster.org/en/latest/release-notes/3.12.0/#major-changes-and-features
)

Due to which xattr storage limit is reached/breached on ext4 based bricks.

To check if you are facing similar issue to the one in the bug provided
above, I would check if the brick logs throw up the no space error on a
gfid2path set failure.

To get around the problem, I would suggest using xfs as the backing FS
for the brick (considering you have close to 250 odd hardlinks to a
file). I would not attempt to disable the gfid2path feature, as that is
useful in getting to the real file just given a GFID and is already part
of core on disk Gluster metadata (It can be shut off, but I would
refrain from it).
> 
> When I deduplicated it around half a year ago with "rdfind"
hardlinking
> was working fine (I think that was glusterfs around version 3.12.8 -
> 3.12.10 ?)
> 
> My search for documentation found only the parameter
> "storage.max-hardlinks" with default of 100 for version 4.0.
> I checked it in my gluster 3.12.13 but here the parameter is not yet
> implemented.
> 
> I tested/proofed it by running my small test on underlaying ext4
> filesystem brick directly and on gluster volume using same ext4
> filesystem of the brick:
> 
> Testline for it:
> ??? ??? ??? mkdir test; cd test; echo "hello" > test; for I in
$(seq 1
> 100); do ln test test-$I ; done
> 
> * on ext4 fs (old brick: xfs) I could do 100 hardlinks without problems
> (from documentation I found ext has 65.000 hardlinks compiled in )
> * on actual GlusterFS (same on my old and new gluster volumes) I could
> do only up to 45 hardlinks now
> 
> But from deduplication around 6 months ago I could find e.g. a file with
> 240 hardlinks setup and there is no problem using these referenced files
> (caused by multiple languages / multiple uploads per language ,
> production/staging system cloned... ).
> 
> My actual workaround has to be using duplicated content but it would be
> great if this could be fixed in next versions ;)
> 
> (Saltstack didn't support yet successful setup of glusterfs 4.0
> peers/volumes; something in output of "gluster --xml
--mode=script" call
> must be weird but I haven't seen any differences so far)
> 
> Bests
> 
> 
> Reiner
> 
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>