Andrea Fogazzi
2017-Mar-10 16:13 UTC
[Gluster-users] Issue with duplicated files in gluster 3.10
Hi community, we ran an extensive issue on our installation of gluster 3.10, which we did upgraded from 3.8.8 (it's a distribute+replicate, 5 nodes, 3 bricks in replica 2+1 quorum); recently we noticed a frequent issue where files get duplicated on the some of the directories; this is visible on the fuse mount points (RW), but not on the NFS/Ganesha (RO) mount points. A sample of an ll output: ---------T 1 48 web_rw 0 Mar 10 11:57 paginazione.shtml -rw-rw-r-- 1 48 web_rw 272 Feb 18 22:00 paginazione.shtml As you can see, the file is listed twice, but only one of the two is good (the name is identical, we verified that no spurious/hidden characters are present in the name); the issue maybe is related on how we uploaded the files on the file system, via incremental rsync on the fuse mount. Do anyone have suggestion on how it can happen, how to solve existing duplication or how to prevent to happen anymore. Thanks in advance. Best regards, andrea Options Reconfigured: performance.cache-invalidation: true cluster.favorite-child-policy: mtime features.cache-invalidation: 1 network.inode-lru-limit: 90000 performance.cache-size: 1024MB storage.linux-aio: on nfs.outstanding-rpc-limit: 64 storage.build-pgfid: on cluster.server-quorum-type: server cluster.self-heal-daemon: enable performance.nfs.io-cache: on performance.client-io-threads: on performance.nfs.stat-prefetch: on performance.nfs.io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on performance.md-cache-timeout: 1 performance.io-thread-count: 16 performance.high-prio-threads: 32 performance.normal-prio-threads: 32 performance.low-prio-threads: 32 performance.least-prio-threads: 1 nfs.acl: off nfs.rpc-auth-unix: off diagnostics.client-log-level: ERROR diagnostics.brick-log-level: ERROR cluster.lookup-unhashed: auto performance.nfs.quick-read: on performance.nfs.read-ahead: on cluster.quorum-type: auto cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full transport.address-family: inet performance.readdir-ahead: on nfs.disable: on cluster.lookup-optimize: on cluster.readdir-optimize: on performance.read-ahead: off performance.write-behind-window-size: 1MB client.event-threads: 4 server.event-threads: 16 cluster.granular-entry-heal: enable performance.parallel-readdir: on cluster.server-quorum-ratio: 51 Andrea Fogazzi -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170310/a6d9c70c/attachment.html>
Luca Gervasi
2017-Mar-10 17:02 UTC
[Gluster-users] Issue with duplicated files in gluster 3.10
Hi, I'm Andrea's collegue. I'd like to add that we have no trusted.afr xattr on the root folder where those files are located and every file seems to be clean on each brick. You can find another example file's xattr here: https://nopaste.me/view/3c2014ac Here a listing: https://nopaste.me/view/eb4430a2 This behavior causes the directory which contains those files undeletable (we had to clear them up on brick level, clearing all the hardlinks too). This issue is visible on fuse mounted volumes while it's not noticeable when mounted in NFS through ganesha. Thanks a lot. Luca Gervasi On Fri, 10 Mar 2017 at 17:41 Andrea Fogazzi <foggy at fogazzi.com> wrote:> Hi community, > > we ran an extensive issue on our installation of gluster 3.10, which we > did upgraded from 3.8.8 (it's a distribute+replicate, 5 nodes, 3 bricks in > replica 2+1 quorum); recently we noticed a frequent issue where files get > duplicated on the some of the directories; this is visible on the fuse > mount points (RW), but not on the NFS/Ganesha (RO) mount points. > > > A sample of an ll output: > > > ---------T 1 48 web_rw 0 Mar 10 11:57 paginazione.shtml > -rw-rw-r-- 1 48 web_rw 272 Feb 18 22:00 paginazione.shtml > > As you can see, the file is listed twice, but only one of the two is good > (the name is identical, we verified that no spurious/hidden characters are > present in the name); the issue maybe is related on how we uploaded the > files on the file system, via incremental rsync on the fuse mount. > > Do anyone have suggestion on how it can happen, how to solve existing > duplication or how to prevent to happen anymore. > > Thanks in advance. > Best regards, > andrea > > Options Reconfigured: > performance.cache-invalidation: true > cluster.favorite-child-policy: mtime > features.cache-invalidation: 1 > network.inode-lru-limit: 90000 > performance.cache-size: 1024MB > storage.linux-aio: on > nfs.outstanding-rpc-limit: 64 > storage.build-pgfid: on > cluster.server-quorum-type: server > cluster.self-heal-daemon: enable > performance.nfs.io-cache: on > performance.client-io-threads: on > performance.nfs.stat-prefetch: on > performance.nfs.io-threads: on > diagnostics.latency-measurement: on > diagnostics.count-fop-hits: on > performance.md-cache-timeout: 1 > performance.io-thread-count: 16 > performance.high-prio-threads: 32 > performance.normal-prio-threads: 32 > performance.low-prio-threads: 32 > performance.least-prio-threads: 1 > nfs.acl: off > nfs.rpc-auth-unix: off > diagnostics.client-log-level: ERROR > diagnostics.brick-log-level: ERROR > cluster.lookup-unhashed: auto > performance.nfs.quick-read: on > performance.nfs.read-ahead: on > cluster.quorum-type: auto > cluster.locking-scheme: granular > cluster.data-self-heal-algorithm: full > transport.address-family: inet > performance.readdir-ahead: on > nfs.disable: on > cluster.lookup-optimize: on > cluster.readdir-optimize: on > performance.read-ahead: off > performance.write-behind-window-size: 1MB > client.event-threads: 4 > server.event-threads: 16 > cluster.granular-entry-heal: enable > performance.parallel-readdir: on > cluster.server-quorum-ratio: 51 > > > > Andrea Fogazzi > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170310/d58ef599/attachment.html>