Ravishankar N
2017-Mar-11 01:01 UTC
[Gluster-users] Issue with duplicated files in gluster 3.10
On 03/10/2017 10:32 PM, Luca Gervasi wrote:> Hi, > I'm Andrea's collegue. I'd like to add that we have no trusted.afr > xattr on the root folderJust to confirm, this would be 'includes2013' right?> where those files are located and every file seems to be clean on each > brick. > You can find another example file's xattr here: > https://nopaste.me/view/3c2014ac > Here a listing: https://nopaste.me/view/eb4430a2 > This behavior causes the directory which contains those files > undeletable (we had to clear them up on brick level, clearing all the > hardlinks too). > This issue is visible on fuse mounted volumes while it's not > noticeable when mounted in NFS through ganesha. >Could you provide the complete output of `gluster volume info`? I want to find out which bricks constitute a replica pair. Also could you change the diagnostics.client-log-level to DEBUG temporarily, do an `ls <directory where you see duplicate entries>` on the fuse mount and share the corresponding mount log? Thanks, Ravi> Thanks a lot. > > Luca Gervasi > > > > On Fri, 10 Mar 2017 at 17:41 Andrea Fogazzi <foggy at fogazzi.com > <mailto:foggy at fogazzi.com>> wrote: > > Hi community, > > we ran an extensive issue on our installation of gluster 3.10, > which we did upgraded from 3.8.8 (it's a distribute+replicate, 5 > nodes, 3 bricks in replica 2+1 quorum); recently we noticed a > frequent issue where files get duplicated on the some of the > directories; this is visible on the fuse mount points (RW), but > not on the NFS/Ganesha (RO) mount points. > > > A sample of an ll output: > > > ---------T 1 48 web_rw 0 Mar 10 11:57 paginazione.shtml > -rw-rw-r-- 1 48 web_rw 272 Feb 18 22:00 paginazione.shtml > > As you can see, the file is listed twice, but only one of the two > is good (the name is identical, we verified that no > spurious/hidden characters are present in the name); the issue > maybe is related on how we uploaded the files on the file system, > via incremental rsync on the fuse mount. > > Do anyone have suggestion on how it can happen, how to solve > existing duplication or how to prevent to happen anymore. > > Thanks in advance. > Best regards, > andrea > > Options Reconfigured: > performance.cache-invalidation: true > cluster.favorite-child-policy: mtime > features.cache-invalidation: 1 > network.inode-lru-limit: 90000 > performance.cache-size: 1024MB > storage.linux-aio: on > nfs.outstanding-rpc-limit: 64 > storage.build-pgfid: on > cluster.server-quorum-type: server > cluster.self-heal-daemon: enable > performance.nfs.io-cache: on > performance.client-io-threads: on > performance.nfs.stat-prefetch: on > performance.nfs.io-threads: on > diagnostics.latency-measurement: on > diagnostics.count-fop-hits: on > performance.md-cache-timeout: 1 > performance.io-thread-count: 16 > performance.high-prio-threads: 32 > performance.normal-prio-threads: 32 > performance.low-prio-threads: 32 > performance.least-prio-threads: 1 > nfs.acl: off > nfs.rpc-auth-unix: off > diagnostics.client-log-level: ERROR > diagnostics.brick-log-level: ERROR > cluster.lookup-unhashed: auto > performance.nfs.quick-read: on > performance.nfs.read-ahead: on > cluster.quorum-type: auto > cluster.locking-scheme: granular > cluster.data-self-heal-algorithm: full > transport.address-family: inet > performance.readdir-ahead: on > nfs.disable: on > cluster.lookup-optimize: on > cluster.readdir-optimize: on > performance.read-ahead: off > performance.write-behind-window-size: 1MB > client.event-threads: 4 > server.event-threads: 16 > cluster.granular-entry-heal: enable > performance.parallel-readdir: on > cluster.server-quorum-ratio: 51 > > > > Andrea Fogazzi > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170311/6eecc43d/attachment.html>
Luca Gervasi
2017-Mar-13 09:38 UTC
[Gluster-users] Issue with duplicated files in gluster 3.10
Hello Ravishankar, we had to change the directory as we fixed that one, so please check these links which refers to a new (broken) path: https://nopaste.me/view/1ee13a63 LS debug log https://nopaste.me/view/80ac1e13 getfattr https://nopaste.me/view/eafb0b44 volume status Thanks in advance. Luca & Andrea On Sat, 11 Mar 2017 at 02:01 Ravishankar N <ravishankar at redhat.com> wrote:> On 03/10/2017 10:32 PM, Luca Gervasi wrote: > > Hi, > I'm Andrea's collegue. I'd like to add that we have no trusted.afr xattr > on the root folder > > Just to confirm, this would be 'includes2013' right? > > where those files are located and every file seems to be clean on each > brick. > You can find another example file's xattr here: > https://nopaste.me/view/3c2014ac > Here a listing: https://nopaste.me/view/eb4430a2 > This behavior causes the directory which contains those files undeletable > (we had to clear them up on brick level, clearing all the hardlinks too). > This issue is visible on fuse mounted volumes while it's not noticeable > when mounted in NFS through ganesha. > > Could you provide the complete output of `gluster volume info`? I want to > find out which bricks constitute a replica pair. > > Also could you change the diagnostics.client-log-level to DEBUG > temporarily, do an `ls <directory where you see duplicate entries>` on the > fuse mount and share the corresponding mount log? > Thanks, > Ravi > > Thanks a lot. > > Luca Gervasi > > > > On Fri, 10 Mar 2017 at 17:41 Andrea Fogazzi <foggy at fogazzi.com> wrote: > > Hi community, > > we ran an extensive issue on our installation of gluster 3.10, which we > did upgraded from 3.8.8 (it's a distribute+replicate, 5 nodes, 3 bricks in > replica 2+1 quorum); recently we noticed a frequent issue where files get > duplicated on the some of the directories; this is visible on the fuse > mount points (RW), but not on the NFS/Ganesha (RO) mount points. > > > A sample of an ll output: > > > ---------T 1 48 web_rw 0 Mar 10 11:57 paginazione.shtml > -rw-rw-r-- 1 48 web_rw 272 Feb 18 22:00 paginazione.shtml > > As you can see, the file is listed twice, but only one of the two is good > (the name is identical, we verified that no spurious/hidden characters are > present in the name); the issue maybe is related on how we uploaded the > files on the file system, via incremental rsync on the fuse mount. > > Do anyone have suggestion on how it can happen, how to solve existing > duplication or how to prevent to happen anymore. > > Thanks in advance. > Best regards, > andrea > > Options Reconfigured: > performance.cache-invalidation: true > cluster.favorite-child-policy: mtime > features.cache-invalidation: 1 > network.inode-lru-limit: 90000 > performance.cache-size: 1024MB > storage.linux-aio: on > nfs.outstanding-rpc-limit: 64 > storage.build-pgfid: on > cluster.server-quorum-type: server > cluster.self-heal-daemon: enable > performance.nfs.io-cache: on > performance.client-io-threads: on > performance.nfs.stat-prefetch: on > performance.nfs.io-threads: on > diagnostics.latency-measurement: on > diagnostics.count-fop-hits: on > performance.md-cache-timeout: 1 > performance.io-thread-count: 16 > performance.high-prio-threads: 32 > performance.normal-prio-threads: 32 > performance.low-prio-threads: 32 > performance.least-prio-threads: 1 > nfs.acl: off > nfs.rpc-auth-unix: off > diagnostics.client-log-level: ERROR > diagnostics.brick-log-level: ERROR > cluster.lookup-unhashed: auto > performance.nfs.quick-read: on > performance.nfs.read-ahead: on > cluster.quorum-type: auto > cluster.locking-scheme: granular > cluster.data-self-heal-algorithm: full > transport.address-family: inet > performance.readdir-ahead: on > nfs.disable: on > cluster.lookup-optimize: on > cluster.readdir-optimize: on > performance.read-ahead: off > performance.write-behind-window-size: 1MB > client.event-threads: 4 > server.event-threads: 16 > cluster.granular-entry-heal: enable > performance.parallel-readdir: on > cluster.server-quorum-ratio: 51 > > > > Andrea Fogazzi > > > > _______________________________________________ > Gluster-users mailing listGluster-users at gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170313/9e80321d/attachment.html>