Gluster users - Mar 2017 - Issue with duplicated files in gluster 3.10

Hi and thanks for your time.
This setting is "off" by default. We enabled that some time last week
after
the issue has been detected without any noticeable fix.
Is this setting willing to avoid the cration of dupes or to fix the
situation live?

Thanks
Luca & Andrea

On Mon, 13 Mar 2017 at 14:03 Ravishankar N <ravishankar at redhat.com>
wrote:
> On 03/13/2017 03:08 PM, Luca Gervasi wrote:
>
> Hello Ravishankar,
> we had to change the directory as we fixed that one, so please check these
> links which refers to a new (broken) path:
>
> https://nopaste.me/view/1ee13a63 LS debug log
> https://nopaste.me/view/80ac1e13 getfattr
> https://nopaste.me/view/eafb0b44 volume status
>
> Thanks, could you check if setting performance.parallel-readdir to
'off'
> solves the issue? If yes, do you mind raising a bug and letting us know the
> BZ ID?
> Please note that parallel-readdir option is still experimental.
> -Ravi
>
>
> Thanks in advance.
>
> Luca & Andrea
>
>
> On Sat, 11 Mar 2017 at 02:01 Ravishankar N <ravishankar at
redhat.com> wrote:
>
> On 03/10/2017 10:32 PM, Luca Gervasi wrote:
>
> Hi,
> I'm Andrea's collegue. I'd like to add that we have no
trusted.afr xattr
> on the root folder
>
> Just to confirm, this would be 'includes2013' right?
>
> where those files are located and every file seems to be clean on each
> brick.
> You can find another example file's xattr here:
> https://nopaste.me/view/3c2014ac
> Here a listing: https://nopaste.me/view/eb4430a2
> This behavior causes the directory which contains those files undeletable
> (we had to clear them up on brick level, clearing all the hardlinks too).
> This issue is visible on fuse mounted volumes while it's not noticeable
> when mounted in NFS through ganesha.
>
> Could you provide the complete output of `gluster volume info`? I want to
> find out which bricks constitute a replica pair.
>
> Also could you change the diagnostics.client-log-level to DEBUG
> temporarily, do an `ls <directory where you see duplicate entries>`
on the
> fuse mount  and share the corresponding mount log?
> Thanks,
> Ravi
>
> Thanks a lot.
>
> Luca Gervasi
>
>
>
> On Fri, 10 Mar 2017 at 17:41 Andrea Fogazzi <foggy at fogazzi.com>
wrote:
>
> Hi community,
>
> we ran an extensive issue on our installation of gluster 3.10, which we
> did upgraded from 3.8.8 (it's a distribute+replicate, 5 nodes, 3 bricks
in
> replica 2+1 quorum); recently we noticed a frequent issue where files get
> duplicated on the some of the directories; this is visible on the fuse
> mount points (RW), but not on the NFS/Ganesha (RO) mount points.
>
>
> A sample of an ll output:
>
>
> ---------T 1 48 web_rw     0 Mar 10 11:57 paginazione.shtml
> -rw-rw-r-- 1 48 web_rw   272 Feb 18 22:00 paginazione.shtml
>
> As you can see, the file is listed twice, but only one of the two is good
> (the name is identical, we verified that no spurious/hidden characters are
> present in the name); the issue maybe is related on how we uploaded the
> files on the file system, via incremental rsync on the fuse mount.
>
> Do anyone have suggestion on how it can happen, how to solve existing
> duplication or how to prevent to happen anymore.
>
> Thanks in advance.
> Best regards,
> andrea
>
> Options Reconfigured:
> performance.cache-invalidation: true
> cluster.favorite-child-policy: mtime
> features.cache-invalidation: 1
> network.inode-lru-limit: 90000
> performance.cache-size: 1024MB
> storage.linux-aio: on
> nfs.outstanding-rpc-limit: 64
> storage.build-pgfid: on
> cluster.server-quorum-type: server
> cluster.self-heal-daemon: enable
> performance.nfs.io-cache: on
> performance.client-io-threads: on
> performance.nfs.stat-prefetch: on
> performance.nfs.io-threads: on
> diagnostics.latency-measurement: on
> diagnostics.count-fop-hits: on
> performance.md-cache-timeout: 1
> performance.io-thread-count: 16
> performance.high-prio-threads: 32
> performance.normal-prio-threads: 32
> performance.low-prio-threads: 32
> performance.least-prio-threads: 1
> nfs.acl: off
> nfs.rpc-auth-unix: off
> diagnostics.client-log-level: ERROR
> diagnostics.brick-log-level: ERROR
> cluster.lookup-unhashed: auto
> performance.nfs.quick-read: on
> performance.nfs.read-ahead: on
> cluster.quorum-type: auto
> cluster.locking-scheme: granular
> cluster.data-self-heal-algorithm: full
> transport.address-family: inet
> performance.readdir-ahead: on
> nfs.disable: on
> cluster.lookup-optimize: on
> cluster.readdir-optimize: on
> performance.read-ahead: off
> performance.write-behind-window-size: 1MB
> client.event-threads: 4
> server.event-threads: 16
> cluster.granular-entry-heal: enable
> performance.parallel-readdir: on
> cluster.server-quorum-ratio: 51
>
>
>
> Andrea Fogazzi
>
>
>
> _______________________________________________
> Gluster-users mailing listGluster-users at
gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170313/d627a2ce/attachment.html>

On 03/13/2017 06:41 PM, Luca Gervasi wrote:
> Is this setting willing to avoid the cration of dupes or to fix the 
> situation live?

The linkto files are not duplicates. They're created when you rename a 
file and the file hashes to a different suvolume of dht. When you issue 
readdir, dht normally 'filters out' the linkto files so that they
don't
appear on the mount.
parallel-readdir option has a bug which interferes with dht's filtering, 
so disabling this option until the bug is fixed should work on your live 
system.
-Ravi