Hi Strahil, Thank you for your reply and your suggestions. I'm not sure which logs would be most relevant to be checking to diagnose this issue, we have the brick logs, the cluster mount logs, the shd logs or something else? I have posted a few that I have seen repeated a few times already. I will continue to post anything further that I see. I am working on migrating data to some new storage, so this will slowly free up space, although this is a production cluster and new data is being uploaded every day, sometimes faster than I can migrate it off. I have several other similar clusters and none of them have the same problem, one the others is actually at 98-99% right now (big problem, I know) but still performs perfectly fine compared to this cluster, I am not sure low space is the root cause here. I currently have 13 VMs accessing this cluster, I have checked each one and all of them use one of the two options below to mount the cluster in fstab HOSTNAME:/gvAA01 /mountpoint glusterfs defaults,_netdev,rw,log-level=WARNING,direct-io-mode=disable,use-readdirp=no 0 0 HOSTNAME:/gvAA01 /mountpoint glusterfs defaults,_netdev,rw,log-level=WARNING,direct-io-mode=disable I also have a few other VMs which use NFS to access the cluster, and these machines appear to be significantly quicker, initially I get a similar delay with NFS but if I cancel the first "ls" and try it again I get < 1 sec lookups, this can take over 10 minutes by FUSE/gluster client, but the same trick of cancelling and trying again doesn't work for FUSE/gluster. Sometimes the NFS queries have no delay at all, so this is a bit strange to me. HOSTNAME:/gvAA01 /mountpoint/ nfs defaults,_netdev,vers=3,async,noatime 0 0 Example: user at VM:~$ time ls /cluster/folder ^C real 9m49.383s user 0m0.001s sys 0m0.010s user at VM:~$ time ls /cluster/folder <results> real 0m0.069s user 0m0.001s sys 0m0.007s --- I have checked the profiling as you suggested, I let it run for around a minute, then cancelled it and saved the profile info. root at HOSTNAME:/var/log/glusterfs# gluster volume profile gvAA01 start Starting volume profile on gvAA01 has been successful root at HOSTNAME:/var/log/glusterfs# time ls /cluster/folder ^C real 1m1.660s user 0m0.000s sys 0m0.002s root at HOSTNAME:/var/log/glusterfs# gluster volume profile gvAA01 info >> ~/profile.txt root at HOSTNAME:/var/log/glusterfs# gluster volume profile gvAA01 stop I will attach the results to this email as it's over 1000 lines. Unfortunately, I'm not sure what I'm looking at but possibly somebody will be able to help me make sense of it and let me know if it highlights any specific issues. Happy to try any further suggestions. Thank you, -Patrick On Sun, Apr 21, 2019 at 7:55 PM Strahil <hunter86_bg at yahoo.com> wrote:> By the way, can you provide the 'volume info' and the mount options on all > clients? > Maybe , there is an option that uses a lot of resources due to some > client's mount options. > > Best Regards, > Strahil Nikolov > On Apr 21, 2019 10:55, Patrick Rennie <patrickmrennie at gmail.com> wrote: > > Just another small update, I'm continuing to watch my brick logs and I > just saw these errors come up in the recent events too. I am going to > continue to post any errors I see in the hope of finding the right one to > try and fix.. > This is from the logs on brick1, seems to be occurring on both nodes on > brick1, although at different times. I'm not sure what this means, can > anyone shed any light? > I guess I am looking for some kind of specific error which may indicate > something is broken or stuck and locking up and causing the extreme latency > I'm seeing in the cluster. > > [2019-04-21 07:25:55.064497] E [rpcsvc.c:1364:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x7c700c, Program: GlusterFS > 3.3, ProgVers: 330, Proc: 29) to rpc-transport (tcp.gvAA01-server) > [2019-04-21 07:25:55.064612] E [server.c:195:server_submit_reply] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e58a) > [0x7f3b3e93158a] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17d45) > [0x7f3b3e4c5d45] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) > [0x7f3b3e4b72cd] ) 0-: Reply submission failed > [2019-04-21 07:25:55.064675] E [rpcsvc.c:1364:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x7c70af, Program: GlusterFS > 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) > [2019-04-21 07:25:55.064705] E [server.c:195:server_submit_reply] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) > [0x7f3b3e9318fa] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) > [0x7f3b3e4c5f35] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) > [0x7f3b3e4b72cd] ) 0-: Reply submission failed > [2019-04-21 07:25:55.064742] E [rpcsvc.c:1364:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x7c723c, Program: GlusterFS > 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) > [2019-04-21 07:25:55.064768] E [server.c:195:server_submit_reply] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) > [0x7f3b3e9318fa] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) > [0x7f3b3e4c5f35] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) > [0x7f3b3e4b72cd] ) 0-: Reply submission failed > [2019-04-21 07:25:55.064812] E [rpcsvc.c:1364:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x7c72b4, Program: GlusterFS > 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) > [2019-04-21 07:25:55.064837] E [server.c:195:server_submit_reply] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) > [0x7f3b3e9318fa] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) > [0x7f3b3e4c5f35] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) > [0x7f3b3e4b72cd] ) 0-: Reply submission failed > [2019-04-21 07:25:55.064880] E [rpcsvc.c:1364:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x7c740b, Program: GlusterFS > 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) > [2019-04-21 07:25:55.064905] E [server.c:195:server_submit_reply] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) > [0x7f3b3e9318fa] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) > [0x7f3b3e4c5f35] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) > [0x7f3b3e4b72cd] ) 0-: Reply submission failed > [2019-04-21 07:25:55.064939] E [rpcsvc.c:1364:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x7c7441, Program: GlusterFS > 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) > [2019-04-21 07:25:55.064962] E [server.c:195:server_submit_reply] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) > [0x7f3b3e9318fa] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) > [0x7f3b3e4c5f35] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) > [0x7f3b3e4b72cd] ) 0-: Reply submission failed > [2019-04-21 07:25:55.064996] E [rpcsvc.c:1364:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x7c74d5, Program: GlusterFS > 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) > [2019-04-21 07:25:55.065020] E [server.c:195:server_submit_reply] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) > [0x7f3b3e9318fa] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) > [0x7f3b3e4c5f35] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) > [0x7f3b3e4b72cd] ) 0-: Reply submission failed > [2019-04-21 07:25:55.065052] E [rpcsvc.c:1364:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x7c7551, Program: GlusterFS > 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) > [2019-04-21 07:25:55.065076] E [server.c:195:server_submit_reply] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) > [0x7f3b3e9318fa] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) > [0x7f3b3e4c5f35] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) > [0x7f3b3e4b72cd] ) 0-: Reply submission failed > [2019-04-21 07:25:55.065110] E [rpcsvc.c:1364:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x7c76d1, Program: GlusterFS > 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) > [2019-04-21 07:25:55.065133] E [server.c:195:server_submit_reply] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) > [0x7f3b3e9318fa] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) > [0x7f3b3e4c5f35] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) > [0x7f3b3e4b72cd] ) 0-: Reply submission failed > > Thanks again, > > -Patrick > > On Sun, Apr 21, 2019 at 3:50 PM Patrick Rennie <patrickmrennie at gmail.com> > wrote: > > Hi Darrell, > > Thanks again for your advice, I've left it for a while but unfortunately > it's still just as slow and causing more problems for our operations now. I > will need to try and take some steps to at least bring performance back to > normal while continuing to investigate the issue longer term. I can > definitely see one node with heavier CPU than the other, almost double, > which I am OK with, but I think the heal process is going to take forever, > trying to check the "gluster volume heal info" shows thousands and > thousands of files which may need healing, I have no idea how many in total > the command is still running after hours, so I am not sure what has gone so > wrong to cause this. > > I've checked cluster.op-version and cluster.max-op-version and it looks > like I'm on the latest version there. > > I have no idea how long the healing is going to take on this cluster, we > have around 560TB of data on here, but I don't think I can wait that long > to try and restore performance to normal. > > Can anyone think of anything else I can try in the meantime to work out > what's causing the extreme latency? > > I've been going through cluster client the logs of some of our VMs and on > some of our FTP servers I found this in the cluster mount log, but I am not > seeing it on any of our other servers, just our FTP servers. > > [2019-04-21 07:16:19.925388] E [MSGID: 101046] > [dht-common.c:1904:dht_revalidate_cbk] 0-gvAA01-dht: dict is null > [2019-04-21 07:19:43.413834] W [MSGID: 114031] > [client-rpc-fops.c:2203:client3_3_setattr_cbk] 0-gvAA01-client-19: remote > operation failed [No such file or directory] > [2019-04-21 07:19:43.414153] W [MSGID: 114031] > [client-rpc-fops.c:2203:client3_3_setattr_cbk] 0-gvAA01-client-20: remote > operation failed [No such file or directory] > [2019-04-21 07:23:33.154717] E [MSGID: 101046] > [dht-common.c:1904:dht_revalidate_cbk] 0-gvAA01-dht: dict is null > [2019-04-21 07:33:24.943913] E [MSGID: 101046] > [dht-common.c:1904:dht_revalidate_cbk] 0-gvAA01-dht: dict is null > > Any ideas what this could mean? I am basically just grasping at straws > here. > > I am going to hold off on the version upgrade until I know there are no > files which need healing, which could be a while, from some reading I've > done there shouldn't be any issues with this as both are on v3.12.x > > I've free'd up a small amount of space, but I still need to work on this > further. > > I've read of a command "find .glusterfs -type f -links -2 -exec rm {} \;" > which could be run on each brick and it would potentially clean up any > files which were deleted straight from the bricks, but not via the client, > I have a feeling this could help me free up about 5-10TB per brick from > what I've been told about the history of this cluster. Can anyone confirm > if this is actually safe to run? > > At this stage, I'm open to any suggestions as to how to proceed, thanks > again for any advice. > > Cheers, > > - Patrick > > On Sun, Apr 21, 2019 at 1:22 AM Darrell Budic <budic at onholyground.com> > wrote: > > Patrick, > > Sounds like progress. Be aware that gluster is expected to max out the > CPUs on at least one of your servers while healing. This is normal and > won?t adversely affect overall performance (any more than having bricks in > need of healing, at any rate) unless you?re overdoing it. shd threads <= 4 > should not do that on your hardware. Other tunings may have also increased > overall performance, so you may see higher CPU than previously anyway. I?d > recommend upping those thread counts and letting it heal as fast as > possible, especially if these are dedicated Gluster storage servers (Ie: > not also running VMs, etc). You should see ?normal? CPU use one heals are > completed. I see ~15-30% overall normally, 95-98% while healing (x my 20 > cores). It?s also likely to be different between your servers, in a pure > replica, one tends to max and one tends to be a little higher, in a > distributed-replica, I?d expect more than one to run harder while healing. > > Keep the differences between doing an ls on a brick and doing an ls on a > gluster mount in mind. When you do a ls on a gluster volume, it isn?t just > doing a ls on one brick, it?s effectively doing it on ALL of your bricks, > and they all have to return data before the ls succeeds. In a distributed > volume, it?s figuring out where on each volume things live and getting the > stat() from each to assemble the whole thing. And if things are in need of > healing, it will take even longer to decide which version is current and > use it (shd triggers a heal anytime it encounters this). Any of these > things being slow slows down the overall response. > > At this point, I?d get some sleep too, and let your cluster heal while you > do. I?d really want it fully healed before I did any updates anyway, so let > it use CPU and get itself sorted out. Expect it to do a round of healing > after you upgrade each machine too, this is normal so don?t let the CPU > spike surprise you, It?s just catching up from the downtime incurred by the > update and/or reboot if you did one. > > That reminds me, check your gluster cluster.op-version and > cluster.max-op-version (gluster vol get all all | grep op-version). If > op-version isn?t at the max-op-verison, set it to it so you?re taking > advantage of the latest features available to your version. > > -Darrell > > On Apr 20, 2019, at 11:54 AM, Patrick Rennie <patrickmrennie at gmail.com> > wrote: > > Hi Darrell, > > Thanks again for your advice, I've applied the acltype=posixacl on my > zpools and I think that has reduced some of the noise from my brick logs. > I also bumped up some of the thread counts you suggested but my CPU load > skyrocketed, so I dropped it back down to something slightly lower, but > still higher than it was before, and will see how that goes for a while. > > Although low space is a definite issue, if I run an ls anywhere on my > bricks directly it's instant, <1 second, and still takes several minutes > via gluster, so there is still a problem in my gluster configuration > somewhere. We don't have any snapshots, but I am trying to work out if any > data on there is safe to delete, or if there is any way I can safely find > and delete data which has been removed directly from the bricks in the > past. I also have lz4 compression already enabled on each zpool which does > help a bit, we get between 1.05 and 1.08x compression on this data. > I've tried to go through each client and checked it's cluster mount logs > and also my brick logs and looking for errors, so far nothing is jumping > out at me, but there are some warnings and errors here and there, I am > trying to work out what they mean. > > It's already 1 am here and unfortunately, I'm still awake working on this > issue, but I think that I will have to leave the version upgrades until > tomorrow. > > Thanks again for your advice so far. If anyone has any ideas on where I > can look for errors other than brick logs or the cluster mount logs to help > resolve this issue, it would be much appreciated. > > Cheers, > > - Patrick > > On Sat, Apr 20, 2019 at 11:57 PM Darrell Budic <budic at onholyground.com> > wrote: > > See inline: > > On Apr 20, 2019, at 10:09 AM, Patrick Rennie <patrickmrennie at gmail.com> > wrote: > > Hi Darrell, > > Thanks for your reply, this issue seems to be getting worse over the last > few days, really has me tearing my hair out. I will do as you have > suggested and get started on upgrading from 3.12.14 to 3.12.15. > I've checked the zfs properties and all bricks have "xattr=sa" set, but > none of them has "acltype=posixacl" set, currently the acltype property > shows "off", if I make these changes will it apply retroactively to the > existing data? I'm unfamiliar with what this will change so I may need to > look into that before I proceed. > > > It is safe to apply that now, any new set/get calls will then use it if > new posixacls exist, and use older if not. ZFS is good that way. It should > clear up your posix_acl and posix errors over time. > > I understand performance is going to slow down as the bricks get full, I > am currently trying to free space and migrate data to some newer storage, I > have fresh several hundred TB storage I just setup recently but with these > performance issues it's really slow. I also believe there is significant > data which has been deleted directly from the bricks in the past, so if I > can reclaim this space in a safe manner then I will have at least around > 10-15% free space. > > > Full ZFS volumes will have a much larger impact on performance than you?d > think, I?d prioritize this. If you have been taking zfs snapshots, consider > deleting them to get the overall volume free space back up. And just to be > sure it?s been said, delete from within the mounted volumes, don?t delete > directly from the bricks (gluster will just try and heal it later, > compounding your issues). Does not apply to deleting other data from the > ZFS volume if it?s not part of the brick directory, of course. > > These servers have dual 8 core Xeon (E5-2620v4) and 512GB of RAM so > generally they have plenty of resources available, currently only using > around 330/512GB of memory. > > I will look into what your suggested settings will change, and then will > probably go ahead with your recommendations, for our specs as stated above, > what would you suggest for performance.io-thread-count ? > > > I run single 2630v4s on my servers, which have a smaller storage footprint > than yours. I?d go with 32 for performance.io-thread-count. I?d try 4 for > the shd thread settings on that gear. Your memory use sounds fine, so no > worries there. > > Our workload is nothing too extreme, we have a few VMs which write backup > data to this storage nightly for our clients, our VMs don't live on this > cluster, but just write to it. > > > If they are writing compressible data, you?ll get immediate benefit by > setting compression=lz4 on your ZFS volumes. It won?t help any old data, of > course, but it will compress new data going forward. This is another one > that?s safe to enable on the fly. > > I've been going through all of the logs I can, below are some slightly > sanitized errors I've come across, but I'm not sure what to make of them. > The main error I am seeing is the first one below, across several of my > bricks, but possibly only for specific folders on the cluster, I'm not 100% > about that yet though. > > [2019-04-20 05:56:59.512649] E [MSGID: 113001] > [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on > /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not > supported] > [2019-04-20 05:59:06.084333] E [MSGID: 113001] > [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on > /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not > supported] > [2019-04-20 05:59:43.289030] E [MSGID: 113001] > [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on > /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not > supported] > [2019-04-20 05:59:50.582257] E [MSGID: 113001] > [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on > /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not > supported] > [2019-04-20 06:01:42.501701] E [MSGID: 113001] > [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on > /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not > supported] > [2019-04-20 06:01:51.665354] W [posix.c:4929:posix_getxattr] > 0-gvAA01-posix: Extended attributes not supported (try remounting brick > with 'user_xattr' flag) > > > [2019-04-20 13:12:36.131856] E [MSGID: 113002] > [posix-helpers.c:893:posix_gfid_set] 0-gvAA01-posix: gfid is null for > /xxxxxxxxxxxxxxxxxxxx [Invalid argument] > [2019-04-20 13:12:36.131959] E [MSGID: 113002] [posix.c:362:posix_lookup] > 0-gvAA01-posix: buf->ia_gfid is null for > /brick2/xxxxxxxxxxxxxxxxxxxx_62906_tmp [No data available] > [2019-04-20 13:12:36.132016] E [MSGID: 115050] > [server-rpc-fops.c:175:server_lookup_cbk] 0-gvAA01-server: 24274759: LOOKUP > /xxxxxxxxxxxxxxxxxxxx (a7c9b4a0-b7ee-4d01-a79e-576013c8ac87/Cloud > Backup_clone1.vbm_62906_tmp), client: > 00-A-16217-2019/04/08-21:23:03:692424-gvAA01-client-4-0-3, error-xlator: > gvAA01-posix [No data available] > [2019-04-20 13:12:38.093719] E [MSGID: 115050] > [server-rpc-fops.c:175:server_lookup_cbk] 0-gvAA01-server: 24276491: LOOKUP > /xxxxxxxxxxxxxxxxxxxx (a7c9b4a0-b7ee-4d01-a79e-576013c8ac87/Cloud > Backup_clone1.vbm_62906_tmp), client: > 00-A-16217-2019/04/08-21:23:03:692424-gvAA01-client-4-0-3, error-xlator: > gvAA01-posix [No data available] > [2019-04-20 13:12:38.093660] E [MSGID: 113002] > [posix-helpers.c:893:posix_gfid_set] 0-gvAA01-posix: gfid is null for > /xxxxxxxxxxxxxxxxxxxx [Invalid argument] > [2019-04-20 13:12:38.093696] E [MSGID: 113002] [posix.c:362:posix_lookup] > 0-gvAA01-posix: buf->ia_gfid is null for /brick2/xxxxxxxxxxxxxxxxxxxx [No > data available] > > > posixacls should clear those up, as mentioned. > > > [2019-04-20 14:25:59.654576] E [inodelk.c:404:__inode_unlock_lock] > 0-gvAA01-locks: Matching lock not found for unlock 0-9223372036854775807, > by 980fdbbd367f0000 on 0x7fc4f0161440 > [2019-04-20 14:25:59.654668] E [MSGID: 115053] > [server-rpc-fops.c:295:server_inodelk_cbk] 0-gvAA01-server: 6092928: > INODELK /xxxxxxxxxxxxxxxxxxxx.cdr$ (25b14631-a179-4274-8243-6e272d4f2ad8), > client: > cb-per-worker18-53637-2019/04/19-14:25:37:927673-gvAA01-client-1-0-4, > error-xlator: gvAA01-locks [Invalid argument] > > > [2019-04-20 13:35:07.495495] E [rpcsvc.c:1364:rpcsvc_submit_generic] > 0-rpc-service: failed to submit message (XID: 0x247c644, Program: GlusterFS > 3.3, ProgVers: 330, Proc: 27) to rpc-transport (tcp.gvAA01-server) > [2019-04-20 13:35:07.495619] E [server.c:195:server_submit_reply] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/debug/io-stats.so(+0x1696a) > [0x7ff4ae6f796a] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/protocol/server.so(+0x2d6e8) > [0x7ff4ae2a96e8] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/protocol/server.so(+0x928d) > [0x7ff4ae28528d] ) 0-: Reply submission failed > > > Fix the posix acls and see if these clear up over time as well, I?m > unclear on what the overall effect of running without the posix acls will > be to total gluster health. Your biggest problem sounds like you need to > free up space on the volumes and get the overall volume health back up to > par and see if that doesn?t resolve the symptoms you?re seeing. > > > > Thank you again for your assistance. It is greatly appreciated. > > - Patrick > > > > On Sat, Apr 20, 2019 at 10:50 PM Darrell Budic <budic at onholyground.com> > wrote: > > Patrick, > > I would definitely upgrade your two nodes from 3.12.14 to 3.12.15. You > also mention ZFS, and that error you show makes me think you need to check > to be sure you have ?xattr=sa? and ?acltype=posixacl? set on your ZFS > volumes. > > You also observed your bricks are crossing the 95% full line, ZFS > performance will degrade significantly the closer you get to full. In my > experience, this starts somewhere between 10% and 5% free space remaining, > so you?re in that realm. > > How?s your free memory on the servers doing? Do you have your zfs arc > cache limited to something less than all the RAM? It shares pretty well, > but I?ve encountered situations where other things won?t try and take ram > back properly if they think it?s in use, so ZFS never gets the opportunity > to give it up. > > Since your volume is a disperse-replica, you might try tuning > disperse.shd-max-threads, default is 1, I?d try it at 2, 4, or even more if > the CPUs are beefy enough. And setting server.event-threads to 4 and > client.event-threads to 8 has proven helpful in many cases. After you get > upgraded to 3.12.15, enabling performance.stat-prefetch may help as well. I > don?t know if it matters, but I?d also recommend resetting > performance.least-prio-threads to the default of 1 (or try 2 or 4) and/or > also setting performance.io-thread-count to 32 if those have beefy CPUs. > > Beyond those general ideas, more info about your hardware (CPU and RAM) > and workload (VMs, direct storage for web servers or enders, etc) may net > you some more ideas. Then you?re going to have to do more digging into > brick logs looking for errors and/or warnings to see what?s going on. > > -Darrell > > > On Apr 20, 2019, at 8:22 AM, Patrick Rennie <patrickmrennie at gmail.com> > wrote: > > Hello Gluster Users, > > I am hoping someone can help me with resolving an ongoing issue I've been > having, I'm new to mailing lists so forgive me if I have gotten anything > wrong. We have noticed our performance deteriorating over the last few > weeks, easily measured by trying to do an ls on one of our top-level > folders, and timing it, which usually would take 2-5 seconds, and now takes > up to 20 minutes, which obviously renders our cluster basically unusable. > This has been intermittent in the past but is now almost constant and I am > not sure how to work out the exact cause. We have noticed some errors in > the brick logs, and have noticed that if we kill the right brick process, > performance instantly returns back to normal, this is not always the same > brick, but it indicates to me something in the brick processes or > background tasks may be causing extreme latency. Due to this ability to fix > it by killing the right brick process off, I think it's a specific file, or > folder, or operation which may be hanging and causing the increased > latency, but I am not sure how to work it out. One last thing to add is > that our bricks are getting quite full (~95% full), we are trying to > migrate data off to new storage but that is going slowly, not helped by > this issue. I am currently trying to run a full heal as there appear to be > many files needing healing, and I have all brick processes running so they > have an opportunity to heal, but this means performance is very poor. It > currently takes over 15-20 minutes to do an ls of one of our top-level > folders, which just contains 60-80 other folders, this should take 2-5 > seconds. This is all being checked by FUSE mount locally on the storage > node itself, but it is the same for other clients and VMs accessing the > cluster. Initially, it seemed our NFS mounts were not affected and operated > at normal speed, but testing over the last day has shown that our NFS > clients are also extremely slow, so it doesn't seem specific to FUSE as I > first thought it might be. > > I am not sure how to proceed from here, I am fairly new to gluster having > inherited this setup from my predecessor and trying to keep it going. I > have included some info below to try and help with diagnosis, please let me > know if any further info would be helpful. I would really appreciate any > advice on what I could try to work out the cause. Thank you in advance for > reading this, and any suggestions you might be able to offer. > > - Patrick > > This is an example of the main error I see in our brick logs, there have > been others, I can post them when I see them again too: > [2019-04-20 04:54:43.055680] E [MSGID: 113001] > [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on > /brick1/<filename> library: system.posix_acl_default [Operation not > supported] > [2019-04-20 05:01:29.476313] W [posix.c:4929:posix_getxattr] > 0-gvAA01-posix: Extended attributes not supported (try remounting brick > with 'user_xattr' flag) > > Our setup consists of 2 storage nodes and an arbiter node. I have noticed > our nodes are on slightly different versions, I'm not sure if this could be > an issue. We have 9 bricks on each node, made up of ZFS RAIDZ2 pools - > total capacity is around 560TB. > We have bonded 10gbps NICS on each node, and I have tested bandwidth with > iperf and found that it's what would be expected from this config. > Individual brick performance seems ok, I've tested several bricks using dd > and can write a 10GB files at 1.7GB/s. > > # dd if=/dev/zero of=/brick1/test/test.file bs=1M count=10000 > 10000+0 records in > 10000+0 records out > 10485760000 bytes (10 GB, 9.8 GiB) copied, 6.20303 s, 1.7 GB/s > > Node 1: > # glusterfs --version > glusterfs 3.12.15 > > Node 2: > # glusterfs --version > glusterfs 3.12.14 > > Arbiter: > # glusterfs --version > glusterfs 3.12.14 > > Here is our gluster volume status: > > # gluster volume status > Status of volume: gvAA01 > Gluster process TCP Port RDMA Port Online > Pid > > ------------------------------------------------------------------------------ > Brick 01-B:/brick1/gvAA01/brick 49152 0 Y 7219 > Brick 02-B:/brick1/gvAA01/brick 49152 0 Y 21845 > Brick 00-A:/arbiterAA01/gvAA01/bri > ck1 49152 0 Y > 6931 > Brick 01-B:/brick2/gvAA01/brick 49153 0 Y 7239 > Brick 02-B:/brick2/gvAA01/brick 49153 0 Y 9916 > Brick 00-A:/arbiterAA01/gvAA01/bri > ck2 49153 0 Y > 6939 > Brick 01-B:/brick3/gvAA01/brick 49154 0 Y 7235 > Brick 02-B:/brick3/gvAA01/brick 49154 0 Y 21858 > Brick 00-A:/arbiterAA01/gvAA01/bri > ck3 49154 0 Y > 6947 > Brick 01-B:/brick4/gvAA01/brick 49155 0 Y 31840 > Brick 02-B:/brick4/gvAA01/brick 49155 0 Y 9933 > Brick 00-A:/arbiterAA01/gvAA01/bri > ck4 49155 0 Y > 6956 > Brick 01-B:/brick5/gvAA01/brick 49156 0 Y 7233 > Brick 02-B:/brick5/gvAA01/brick 49156 0 Y 9942 > Brick 00-A:/arbiterAA01/gvAA01/bri > ck5 49156 0 Y > 6964 > Brick 01-B:/brick6/gvAA01/brick 49157 0 Y 7234 > Brick 02-B:/brick6/gvAA01/brick 49157 0 Y 9952 > Brick 00-A:/arbiterAA01/gvAA01/bri > ck6 49157 0 Y > 6974 > Brick 01-B:/brick7/gvAA01/brick 49158 0 Y 7248 > Brick 02-B:/brick7/gvAA01/brick 49158 0 Y 9960 > Brick 00-A:/arbiterAA01/gvAA01/bri > ck7 49158 0 Y > 6984 > Brick 01-B:/brick8/gvAA01/brick 49159 0 Y 7253 > Brick 02-B:/brick8/gvAA01/brick 49159 0 Y 9970 > Brick 00-A:/arbiterAA01/gvAA01/bri > ck8 49159 0 Y > 6993 > Brick 01-B:/brick9/gvAA01/brick 49160 0 Y 7245 > Brick 02-B:/brick9/gvAA01/brick 49160 0 Y 9984 > Brick 00-A:/arbiterAA01/gvAA01/bri > ck9 49160 0 Y > 7001 > NFS Server on localhost 2049 0 Y > 17276 > Self-heal Daemon on localhost N/A N/A Y > 25245 > NFS Server on 02-B 2049 0 Y 9089 > Self-heal Daemon on 02-B N/A N/A Y 17838 > NFS Server on 00-a 2049 0 Y 15660 > Self-heal Daemon on 00-a N/A N/A Y 16218 > > Task Status of Volume gvAA01 > > ------------------------------------------------------------------------------ > There are no active volume tasks > > And gluster volume info: > > # gluster volume info > > Volume Name: gvAA01 > Type: Distributed-Replicate > Volume ID: ca4ece2c-13fe-414b-856c-2878196d6118 > Status: Started > Snapshot Count: 0 > Number of Bricks: 9 x (2 + 1) = 27 > Transport-type: tcp > Bricks: > Brick1: 01-B:/brick1/gvAA01/brick > Brick2: 02-B:/brick1/gvAA01/brick > Brick3: 00-A:/arbiterAA01/gvAA01/brick1 (arbiter) > Brick4: 01-B:/brick2/gvAA01/brick > Brick5: 02-B:/brick2/gvAA01/brick > Brick6: 00-A:/arbiterAA01/gvAA01/brick2 (arbiter) > Brick7: 01-B:/brick3/gvAA01/brick > Brick8: 02-B:/brick3/gvAA01/brick > Brick9: 00-A:/arbiterAA01/gvAA01/brick3 (arbiter) > Brick10: 01-B:/brick4/gvAA01/brick > Brick11: 02-B:/brick4/gvAA01/brick > Brick12: 00-A:/arbiterAA01/gvAA01/brick4 (arbiter) > Brick13: 01-B:/brick5/gvAA01/brick > Brick14: 02-B:/brick5/gvAA01/brick > Brick15: 00-A:/arbiterAA01/gvAA01/brick5 (arbiter) > Brick16: 01-B:/brick6/gvAA01/brick > Brick17: 02-B:/brick6/gvAA01/brick > Brick18: 00-A:/arbiterAA01/gvAA01/brick6 (arbiter) > Brick19: 01-B:/brick7/gvAA01/brick > Brick20: 02-B:/brick7/gvAA01/brick > Brick21: 00-A:/arbiterAA01/gvAA01/brick7 (arbiter) > Brick22: 01-B:/brick8/gvAA01/brick > Brick23: 02-B:/brick8/gvAA01/brick > Brick24: 00-A:/arbiterAA01/gvAA01/brick8 (arbiter) > Brick25: 01-B:/brick9/gvAA01/brick > Brick26: 02-B:/brick9/gvAA01/brick > Brick27: 00-A:/arbiterAA01/gvAA01/brick9 (arbiter) > Options Reconfigured: > cluster.shd-max-threads: 4 > performance.least-prio-threads: 16 > cluster.readdir-optimize: on > performance.quick-read: off > performance.stat-prefetch: off > cluster.data-self-heal: on > cluster.lookup-unhashed: auto > cluster.lookup-optimize: on > cluster.favorite-child-policy: mtime > server.allow-insecure: on > transport.address-family: inet > client.bind-insecure: on > cluster.entry-self-heal: off > cluster.metadata-self-heal: off > performance.md-cache-timeout: 600 > cluster.self-heal-daemon: enable > performance.readdir-ahead: on > diagnostics.brick-log-level: INFO > nfs.disable: off > > Thank you for any assistance. > > - Patrick > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190421/97818bce/attachment.html> -------------- next part -------------- Brick: HOSTNAME-02-B:/brick1/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 32b+ 64b+ 128b+ No. of Reads: 1 0 0 No. of Writes: 138 7 45 Block Size: 256b+ 512b+ 1024b+ No. of Reads: 0 0 0 No. of Writes: 1 588 8321 Block Size: 2048b+ 4096b+ 8192b+ No. of Reads: 0 2 5 No. of Writes: 9294 88957 21544 Block Size: 16384b+ 32768b+ 65536b+ No. of Reads: 13 22 41 No. of Writes: 253312 24305 207953 Block Size: 131072b+ No. of Reads: 415121 No. of Writes: 632261 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 1656 FORGET 0.00 0.00 us 0.00 us 0.00 us 3098 RELEASE 0.00 0.00 us 0.00 us 0.00 us 30293 RELEASEDIR 0.00 38.00 us 38.00 us 38.00 us 1 FLUSH 0.00 881.00 us 881.00 us 881.00 us 1 UNLINK 0.00 252705.00 us 252705.00 us 252705.00 us 1 LK 0.00 45581.06 us 275.00 us 760038.00 us 17 SETXATTR 0.00 178264.40 us 1180.00 us 567795.00 us 5 MKNOD 0.01 52932.15 us 269.00 us 710452.00 us 34 SETATTR 0.01 56835.03 us 22.00 us 875479.00 us 34 GETXATTR 0.01 33009.92 us 55.00 us 690160.00 us 59 READ 0.01 130547.06 us 199.00 us 746238.00 us 17 REMOVEXATTR 0.01 101512.69 us 45.00 us 898760.00 us 35 READDIR 0.02 176200.21 us 79.00 us 767224.00 us 24 READDIRP 0.03 149224.76 us 1210.00 us 847273.00 us 50 MKDIR 0.04 477780.00 us 33.00 us 894975.00 us 20 FSTAT 0.04 294297.09 us 766.00 us 1055615.00 us 33 XATTROP 0.10 121732.71 us 19.00 us 1134288.00 us 211 ENTRYLK 0.10 61611.54 us 470.00 us 1132038.00 us 439 FSYNC 0.11 308381.30 us 56.00 us 1117021.00 us 97 OPENDIR 0.26 352045.54 us 25.00 us 1117112.00 us 200 STATFS 0.54 59824.41 us 66.00 us 1050503.00 us 2449 WRITE 0.62 42784.77 us 135.00 us 982688.00 us 3920 FXATTROP 1.09 48299.64 us 12.00 us 1475231.00 us 6113 FINODELK 1.99 74036.50 us 337.00 us 1504736.00 us 7270 RCHECKSUM 5.02 91592.07 us 14.00 us 1727644.00 us 14800 INODELK 18.22 391339.18 us 13.00 us 1118801.00 us 12565 STAT 71.77 447238.86 us 65.00 us 1520575.00 us 43295 LOOKUP Duration: 78214 seconds Data Read: 54416941626 bytes Data Written: 113658424203 bytes Interval 1 Stats: Block Size: 4096b+ 16384b+ 32768b+ No. of Reads: 0 0 0 No. of Writes: 10 1 37 Block Size: 65536b+ 131072b+ No. of Reads: 0 99 No. of Writes: 364 2494 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 115 RELEASEDIR 0.00 159976.00 us 1747.00 us 318205.00 us 2 MKNOD 0.01 43052.35 us 72.00 us 711323.00 us 17 READDIR 0.01 95923.62 us 306.00 us 760038.00 us 8 SETXATTR 0.01 66875.50 us 283.00 us 703396.00 us 16 SETATTR 0.01 99516.19 us 22.00 us 875479.00 us 16 GETXATTR 0.01 186339.50 us 268.00 us 670244.00 us 10 READDIRP 0.01 38121.54 us 55.00 us 690160.00 us 49 READ 0.02 276285.38 us 199.00 us 746238.00 us 8 REMOVEXATTR 0.03 234487.19 us 766.00 us 992806.00 us 16 XATTROP 0.04 160636.43 us 1337.00 us 847273.00 us 35 MKDIR 0.05 445612.14 us 33.00 us 885814.00 us 14 FSTAT 0.10 318281.76 us 56.00 us 1117021.00 us 42 OPENDIR 0.11 77332.15 us 512.00 us 1132038.00 us 184 FSYNC 0.13 140282.40 us 19.00 us 1134288.00 us 121 ENTRYLK 0.26 371711.56 us 25.00 us 1117112.00 us 94 STATFS 0.50 65120.02 us 77.00 us 1048875.00 us 1037 WRITE 0.52 44151.04 us 145.00 us 982688.00 us 1588 FXATTROP 0.96 53136.48 us 15.00 us 1131465.00 us 2444 FINODELK 2.04 75382.00 us 337.00 us 1135632.00 us 3653 RCHECKSUM 4.89 89311.00 us 14.00 us 1727644.00 us 7403 INODELK 19.46 432093.99 us 13.00 us 1118801.00 us 6093 STAT 70.86 461973.08 us 65.00 us 1520575.00 us 20751 LOOKUP Duration: 246 seconds Data Read: 12976128 bytes Data Written: 374076928 bytes Brick: HOSTNAME-02-B:/brick7/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 32b+ 64b+ 512b+ No. of Reads: 0 0 0 No. of Writes: 3 1 2 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 0 No. of Writes: 5 6 1174 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 0 0 1 No. of Writes: 354 415 1133 Block Size: 65536b+ 131072b+ No. of Reads: 0 5403 No. of Writes: 7199 16939 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 59 FORGET 0.00 0.00 us 0.00 us 0.00 us 414 RELEASE 0.00 0.00 us 0.00 us 0.00 us 812 RELEASEDIR 0.00 51.00 us 51.00 us 51.00 us 1 LK 0.00 811.00 us 811.00 us 811.00 us 1 LINK 0.00 1962.67 us 116.00 us 5155.00 us 3 OPEN 0.00 19450.00 us 19450.00 us 19450.00 us 1 RENAME 0.00 425385.00 us 3485.00 us 847285.00 us 2 MKNOD 0.00 514930.00 us 94.00 us 1029766.00 us 2 FTRUNCATE 0.01 355550.20 us 457.00 us 713035.00 us 5 UNLINK 0.01 1192925.00 us 955005.00 us 1430845.00 us 2 XATTROP 0.01 493677.80 us 1865.00 us 1133142.00 us 5 CREATE 0.02 429648.22 us 26.00 us 1005252.00 us 9 FLUSH 0.02 375096.13 us 31.00 us 743624.00 us 15 FSTAT 0.03 553294.91 us 124.00 us 2047492.00 us 11 SETATTR 0.04 241321.34 us 210.00 us 1130863.00 us 35 READDIRP 0.11 308798.10 us 37.00 us 1090701.00 us 80 OPENDIR 0.14 437895.92 us 20.00 us 2390710.00 us 76 ENTRYLK 0.24 274161.42 us 20.00 us 1131003.00 us 206 STATFS 0.26 380771.70 us 111.00 us 2217146.00 us 156 FSYNC 0.62 608724.21 us 98.00 us 2805007.00 us 234 RCHECKSUM 0.68 436962.29 us 65.00 us 2218008.00 us 359 READ 0.88 292235.22 us 16.00 us 1860745.00 us 696 FINODELK 1.16 517693.60 us 20.00 us 3188822.00 us 516 INODELK 3.10 1007821.69 us 185.00 us 8062558.00 us 710 FXATTROP 5.47 459404.39 us 60.00 us 2424925.00 us 2747 WRITE 19.30 343135.28 us 11.00 us 1508447.00 us 12965 STAT 67.89 449890.07 us 56.00 us 3189202.00 us 34791 LOOKUP Duration: 2760 seconds Data Read: 708235477 bytes Data Written: 3108339591 bytes Interval 1 Stats: Block Size: 32b+ 2048b+ 4096b+ No. of Reads: 0 0 0 No. of Writes: 1 2 87 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 0 0 0 No. of Writes: 33 46 155 Block Size: 65536b+ 131072b+ No. of Reads: 0 384 No. of Writes: 898 1821 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 7 FORGET 0.00 0.00 us 0.00 us 0.00 us 5 RELEASE 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.00 811.00 us 811.00 us 811.00 us 1 LINK 0.00 3485.00 us 3485.00 us 3485.00 us 1 MKNOD 0.00 1962.67 us 116.00 us 5155.00 us 3 OPEN 0.01 364139.50 us 35.00 us 728244.00 us 2 FLUSH 0.01 955005.00 us 955005.00 us 955005.00 us 1 XATTROP 0.01 514930.00 us 94.00 us 1029766.00 us 2 FTRUNCATE 0.01 568496.50 us 3851.00 us 1133142.00 us 2 CREATE 0.02 443739.00 us 457.00 us 713035.00 us 4 UNLINK 0.03 340935.25 us 31.00 us 700342.00 us 8 FSTAT 0.03 701433.25 us 124.00 us 2047492.00 us 4 SETATTR 0.03 319603.10 us 1904.00 us 1130863.00 us 10 READDIRP 0.10 277164.95 us 40.00 us 1090701.00 us 39 OPENDIR 0.18 558491.82 us 25.00 us 2390710.00 us 34 ENTRYLK 0.25 275750.60 us 20.00 us 1131003.00 us 96 STATFS 0.26 362599.55 us 111.00 us 2217146.00 us 77 FSYNC 0.78 761765.55 us 98.00 us 2805007.00 us 110 RCHECKSUM 0.81 519364.02 us 65.00 us 2218008.00 us 168 READ 0.86 292915.32 us 16.00 us 1860745.00 us 313 FINODELK 1.36 618592.78 us 23.00 us 3188822.00 us 235 INODELK 3.03 1037876.55 us 220.00 us 8062558.00 us 313 FXATTROP 6.12 518399.40 us 67.00 us 2424925.00 us 1265 WRITE 21.23 369324.98 us 11.00 us 1496678.00 us 6163 STAT 64.89 406497.92 us 60.00 us 3189202.00 us 17113 LOOKUP Duration: 246 seconds Data Read: 50331648 bytes Data Written: 349692935 bytes Brick: HOSTNAME-02-B:/brick4/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 2b+ 4b+ 8b+ No. of Reads: 1 0 1 No. of Writes: 2 1 7 Block Size: 16b+ 32b+ 64b+ No. of Reads: 4 392 51 No. of Writes: 12 135 57 Block Size: 128b+ 256b+ 512b+ No. of Reads: 443 127 10 No. of Writes: 140 304 1444 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 90 261 364 No. of Writes: 219068 121020 345512 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 514 856 1478 No. of Writes: 118098 171851 108390 Block Size: 65536b+ 131072b+ 262144b+ No. of Reads: 2306 8288758 0 No. of Writes: 371871 702539 10 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 45329 FORGET 0.00 0.00 us 0.00 us 0.00 us 48418 RELEASE 0.00 0.00 us 0.00 us 0.00 us 311777 RELEASEDIR 0.00 38.00 us 38.00 us 38.00 us 1 LK 0.00 49.50 us 45.00 us 54.00 us 2 ENTRYLK 0.00 191.00 us 191.00 us 191.00 us 1 SETATTR 0.00 114.50 us 82.00 us 212.00 us 10 OPEN 0.00 53.83 us 28.00 us 258.00 us 29 FINODELK 0.00 32180.00 us 32180.00 us 32180.00 us 1 MKNOD 0.05 13870.29 us 44.00 us 565518.00 us 80 OPENDIR 0.11 84408.74 us 164.00 us 483823.00 us 27 READDIRP 0.15 15813.07 us 27.00 us 642324.00 us 200 STATFS 0.21 10819.69 us 30.00 us 2037193.00 us 404 FSTAT 0.34 49.82 us 10.00 us 159830.00 us 141050 INODELK 0.47 155572.08 us 67.00 us 469682.00 us 62 WRITE 1.93 376836.99 us 162.00 us 3309818.00 us 105 FXATTROP 9.35 2726.16 us 388.00 us 708391.00 us 70513 RCHECKSUM 17.39 6686.00 us 36.00 us 689525.00 us 53448 READ 18.05 29067.22 us 14.00 us 1708922.00 us 12766 STAT 51.94 33265.70 us 59.00 us 3308011.00 us 32089 LOOKUP Duration: 87061 seconds Data Read: 1086749393549 bytes Data Written: 147537171706 bytes Interval 1 Stats: Block Size: 8192b+ 32768b+ 65536b+ No. of Reads: 0 0 0 No. of Writes: 1 3 11 Block Size: 131072b+ No. of Reads: 85818 No. of Writes: 47 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.00 77.00 us 33.00 us 258.00 us 7 FINODELK 0.00 109.12 us 46.00 us 397.00 us 34 OPENDIR 0.06 120138.50 us 837.00 us 483823.00 us 6 READDIRP 0.10 12370.53 us 31.00 us 642324.00 us 94 STATFS 0.29 54.74 us 10.00 us 159830.00 us 62641 INODELK 0.32 12202.91 us 30.00 us 2037193.00 us 313 FSTAT 0.42 166359.60 us 67.00 us 469682.00 us 30 WRITE 1.76 424134.63 us 162.00 us 3309818.00 us 49 FXATTROP 9.44 3566.54 us 396.00 us 708391.00 us 31320 RCHECKSUM 21.56 41576.42 us 14.00 us 1708922.00 us 6138 STAT 22.70 6488.02 us 36.00 us 689525.00 us 41414 READ 43.34 33595.04 us 81.00 us 3308011.00 us 15268 LOOKUP Duration: 246 seconds Data Read: 11248336896 bytes Data Written: 7489024 bytes Brick: HOSTNAME-02-B:/brick8/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 32b+ 512b+ 1024b+ No. of Reads: 0 1 0 No. of Writes: 2 415 835 Block Size: 2048b+ 4096b+ 8192b+ No. of Reads: 4 2 4 No. of Writes: 1788 22766 9151 Block Size: 16384b+ 32768b+ 65536b+ No. of Reads: 12 28 129 No. of Writes: 12333 23668 124056 Block Size: 131072b+ 262144b+ No. of Reads: 1565084 0 No. of Writes: 331920 2 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 99 FORGET 0.00 0.00 us 0.00 us 0.00 us 1590 RELEASE 0.00 0.00 us 0.00 us 0.00 us 32065 RELEASEDIR 0.00 158.00 us 158.00 us 158.00 us 1 SETATTR 0.00 42.75 us 34.00 us 57.00 us 4 ENTRYLK 0.00 95.00 us 91.00 us 99.00 us 2 OPEN 0.00 36.90 us 31.00 us 49.00 us 10 INODELK 0.00 70.67 us 23.00 us 139.00 us 6 GETXATTR 0.03 3341.00 us 3341.00 us 3341.00 us 1 UNLINK 0.03 4044.00 us 4044.00 us 4044.00 us 1 MKNOD 0.09 135.14 us 2.00 us 1219.00 us 83 OPENDIR 0.30 187.68 us 24.00 us 7544.00 us 200 STATFS 0.67 4177.55 us 263.00 us 28261.00 us 20 READDIRP 1.02 21242.50 us 153.00 us 122468.00 us 6 READDIR 14.15 138.01 us 16.00 us 21061.00 us 12765 STAT 83.70 336.20 us 35.00 us 30375.00 us 31007 LOOKUP Duration: 87061 seconds Data Read: 205153204736 bytes Data Written: 59695163000 bytes Interval 1 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.00 41.00 us 33.00 us 49.00 us 2 INODELK 0.00 158.00 us 158.00 us 158.00 us 1 SETATTR 0.00 42.75 us 34.00 us 57.00 us 4 ENTRYLK 0.05 3341.00 us 3341.00 us 3341.00 us 1 UNLINK 0.06 116.15 us 43.00 us 427.00 us 34 OPENDIR 0.06 4044.00 us 4044.00 us 4044.00 us 1 MKNOD 0.25 176.27 us 31.00 us 7544.00 us 94 STATFS 0.34 4370.40 us 3093.00 us 5741.00 us 5 READDIRP 14.33 152.05 us 17.00 us 21061.00 us 6138 STAT 84.90 366.58 us 73.00 us 30375.00 us 15084 LOOKUP Duration: 246 seconds Data Read: 0 bytes Data Written: 0 bytes Brick: HOSTNAME-02-B:/brick5/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 0 0 0 No. of Writes: 675 2015 5669 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 3 0 2 No. of Writes: 86933 18437 13563 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 13 11 2226145 No. of Writes: 28736 150681 348421 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 114 FORGET 0.00 0.00 us 0.00 us 0.00 us 1644 RELEASE 0.00 0.00 us 0.00 us 0.00 us 29197 RELEASEDIR 0.00 93.00 us 93.00 us 93.00 us 1 OPEN 0.00 121.00 us 111.00 us 131.00 us 2 SETATTR 0.00 62.43 us 22.00 us 134.00 us 7 GETXATTR 0.00 25.81 us 17.00 us 74.00 us 70 ENTRYLK 0.00 5628.50 us 2325.00 us 8932.00 us 2 MKNOD 0.01 124665.17 us 231.00 us 624079.00 us 6 READDIR 0.02 101333.57 us 165.00 us 520857.00 us 30 READDIRP 0.14 237618.40 us 1.00 us 763629.00 us 85 OPENDIR 0.24 170075.12 us 22.00 us 762654.00 us 200 STATFS 0.28 1494.00 us 387.00 us 658594.00 us 26497 RCHECKSUM 0.38 67376.52 us 67.00 us 592219.00 us 782 WRITE 0.39 36555.77 us 136.00 us 970840.00 us 1506 FXATTROP 0.41 76461.54 us 1824.00 us 784619.00 us 751 FSYNC 0.44 27318.09 us 14.00 us 779529.00 us 2256 FINODELK 1.39 3660.22 us 13.00 us 727062.00 us 53075 INODELK 18.68 205180.63 us 12.00 us 767356.00 us 12736 STAT 77.62 351003.58 us 30.00 us 971220.00 us 30942 LOOKUP Duration: 78214 seconds Data Read: 291787269120 bytes Data Written: 65192541184 bytes Interval 1 Stats: Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 0 No. of Writes: 2 1 5 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 0 0 0 No. of Writes: 10 19 31 Block Size: 65536b+ 131072b+ No. of Reads: 0 0 No. of Writes: 206 686 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.02 264573.67 us 103764.00 us 499127.00 us 6 READDIRP 0.09 197846.12 us 43.00 us 651148.00 us 34 OPENDIR 0.23 173971.46 us 27.00 us 663534.00 us 94 STATFS 0.25 1340.23 us 403.00 us 609386.00 us 13504 RCHECKSUM 0.33 30710.89 us 136.00 us 970840.00 us 774 FXATTROP 0.33 59568.81 us 79.00 us 592219.00 us 400 WRITE 0.40 74721.05 us 1894.00 us 645840.00 us 386 FSYNC 0.42 25870.04 us 17.00 us 641253.00 us 1160 FINODELK 1.35 3568.56 us 13.00 us 677567.00 us 27047 INODELK 20.17 235917.45 us 12.00 us 668351.00 us 6109 STAT 76.39 365037.23 us 61.00 us 971220.00 us 14951 LOOKUP Duration: 246 seconds Data Read: 0 bytes Data Written: 116390912 bytes Brick: HOSTNAME-02-B:/brick6/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 64b+ 512b+ 1024b+ No. of Reads: 2 1 0 No. of Writes: 0 249 554 Block Size: 2048b+ 4096b+ 8192b+ No. of Reads: 0 0 7 No. of Writes: 1147 5601 4339 Block Size: 16384b+ 32768b+ 65536b+ No. of Reads: 9 14 52 No. of Writes: 8060 16044 85807 Block Size: 131072b+ 262144b+ No. of Reads: 57503 0 No. of Writes: 233443 2 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 177 FORGET 0.00 0.00 us 0.00 us 0.00 us 267 RELEASE 0.00 0.00 us 0.00 us 0.00 us 32347 RELEASEDIR 0.00 77.80 us 18.00 us 140.00 us 5 GETXATTR 0.01 110.21 us 28.00 us 884.00 us 14 ENTRYLK 0.01 2082.00 us 2082.00 us 2082.00 us 1 RENAME 0.10 185.46 us 2.00 us 1256.00 us 84 OPENDIR 0.20 148.97 us 30.00 us 2192.00 us 200 STATFS 0.64 16299.00 us 163.00 us 95266.00 us 6 READDIR 0.83 5265.21 us 315.00 us 28553.00 us 24 READDIRP 15.27 181.48 us 14.00 us 30885.00 us 12765 STAT 82.92 409.24 us 29.00 us 31576.00 us 30731 LOOKUP Duration: 87061 seconds Data Read: 7542971594 bytes Data Written: 41717849088 bytes Interval 1 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.01 275.25 us 65.00 us 884.00 us 4 ENTRYLK 0.03 2082.00 us 2082.00 us 2082.00 us 1 RENAME 0.10 217.26 us 63.00 us 1044.00 us 34 OPENDIR 0.17 134.26 us 30.00 us 939.00 us 94 STATFS 0.33 4005.17 us 2273.00 us 6785.00 us 6 READDIRP 14.30 171.33 us 16.00 us 19199.00 us 6138 STAT 85.06 413.91 us 68.00 us 31576.00 us 15114 LOOKUP Duration: 246 seconds Data Read: 0 bytes Data Written: 0 bytes Brick: HOSTNAME-02-B:/brick2/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 8b+ 32b+ 64b+ No. of Reads: 0 0 0 No. of Writes: 1 54 46 Block Size: 128b+ 256b+ 512b+ No. of Reads: 0 0 0 No. of Writes: 180 277 721 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 3 No. of Writes: 20590 17995 62409 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 4 26 28 No. of Writes: 68162 107367 71851 Block Size: 65536b+ 131072b+ 262144b+ No. of Reads: 59 542702 0 No. of Writes: 218072 310390 8 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 573 FORGET 0.00 0.00 us 0.00 us 0.00 us 2644 RELEASE 0.00 0.00 us 0.00 us 0.00 us 28568 RELEASEDIR 0.00 33.00 us 33.00 us 33.00 us 1 FLUSH 0.00 67.50 us 56.00 us 79.00 us 2 LK 0.00 167.00 us 167.00 us 167.00 us 1 OPEN 0.01 144.96 us 42.00 us 1139.00 us 80 OPENDIR 0.01 111.08 us 24.00 us 2868.00 us 200 STATFS 0.03 157.12 us 73.00 us 9048.00 us 429 WRITE 0.86 135.98 us 15.00 us 13652.00 us 12765 STAT 1.18 81686.03 us 231.00 us 851745.00 us 29 READDIRP 3.98 45.61 us 14.00 us 11734.00 us 175875 INODELK 8.95 577.68 us 74.00 us 224862.00 us 31188 LOOKUP 84.98 1946.12 us 330.00 us 811977.00 us 87936 RCHECKSUM Duration: 78214 seconds Data Read: 71140696438 bytes Data Written: 71129235470 bytes Interval 1 Stats: Block Size: 131072b+ No. of Reads: 0 No. of Writes: 551 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 1 FORGET 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.01 150.06 us 42.00 us 1139.00 us 34 OPENDIR 0.01 90.35 us 31.00 us 735.00 us 94 STATFS 0.04 202.23 us 81.00 us 9048.00 us 144 WRITE 1.11 133.66 us 16.00 us 9674.00 us 6138 STAT 1.90 234569.83 us 1225.00 us 851745.00 us 6 READDIRP 5.01 46.95 us 14.00 us 11257.00 us 78964 INODELK 11.66 568.83 us 74.00 us 114583.00 us 15164 LOOKUP 80.26 1503.60 us 330.00 us 558433.00 us 39481 RCHECKSUM Duration: 246 seconds Data Read: 0 bytes Data Written: 72220672 bytes Brick: HOSTNAME-02-B:/brick9/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 64b+ 512b+ 1024b+ No. of Reads: 1 2 0 No. of Writes: 0 1488 3133 Block Size: 2048b+ 4096b+ 8192b+ No. of Reads: 2 12 6 No. of Writes: 6241 20337 17112 Block Size: 16384b+ 32768b+ 65536b+ No. of Reads: 41 40 109 No. of Writes: 36536 97167 400752 Block Size: 131072b+ 262144b+ No. of Reads: 1392662 0 No. of Writes: 734025 9 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 245 FORGET 0.00 0.00 us 0.00 us 0.00 us 1040 RELEASE 0.00 0.00 us 0.00 us 0.00 us 32052 RELEASEDIR 0.00 40.50 us 32.00 us 49.00 us 2 FLUSH 0.00 64.50 us 54.00 us 75.00 us 2 ENTRYLK 0.00 45.25 us 14.00 us 92.00 us 4 GETXATTR 0.00 66.33 us 43.00 us 94.00 us 3 LK 0.00 230.00 us 230.00 us 230.00 us 1 OPEN 0.06 14688.00 us 14688.00 us 14688.00 us 1 UNLINK 0.13 358.22 us 2.00 us 6730.00 us 83 OPENDIR 0.16 178.78 us 26.00 us 4082.00 us 200 STATFS 0.57 21934.83 us 164.00 us 122756.00 us 6 READDIR 2.74 24249.73 us 122.00 us 185638.00 us 26 READDIRP 16.00 287.83 us 14.00 us 53088.00 us 12765 STAT 80.33 596.96 us 60.00 us 55814.00 us 30911 LOOKUP Duration: 87061 seconds Data Read: 182552716865 bytes Data Written: 146735579648 bytes Interval 1 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.00 64.50 us 54.00 us 75.00 us 2 ENTRYLK 0.14 14688.00 us 14688.00 us 14688.00 us 1 UNLINK 0.16 174.14 us 26.00 us 3573.00 us 94 STATFS 0.17 541.12 us 45.00 us 6730.00 us 34 OPENDIR 2.16 45567.60 us 951.00 us 139415.00 us 5 READDIRP 14.97 257.11 us 14.00 us 30619.00 us 6138 STAT 82.40 576.00 us 70.00 us 49717.00 us 15086 LOOKUP Duration: 246 seconds Data Read: 0 bytes Data Written: 0 bytes Brick: HOSTNAME-02-B:/brick3/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 4b+ 16b+ 32b+ No. of Reads: 0 0 0 No. of Writes: 1 1 13 Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 0 0 No. of Writes: 10 125 37 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 0 0 1 No. of Writes: 324 326351 4542 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 4 2 9 No. of Writes: 56358 18819 27128 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 17 42 3816011 No. of Writes: 35096 76153 521853 Block Size: 262144b+ No. of Reads: 0 No. of Writes: 12 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 175 FORGET 0.00 0.00 us 0.00 us 0.00 us 2519 RELEASE 0.00 0.00 us 0.00 us 0.00 us 28490 RELEASEDIR 0.00 27.00 us 27.00 us 27.00 us 1 FLUSH 0.00 512.00 us 512.00 us 512.00 us 1 UNLINK 0.00 414686.00 us 414686.00 us 414686.00 us 1 LK 0.00 253232.00 us 39.00 us 506425.00 us 2 ENTRYLK 0.05 497784.80 us 124.00 us 1570288.00 us 30 READDIRP 0.11 7015.25 us 336.00 us 1565069.00 us 4505 RCHECKSUM 0.13 494818.47 us 52.00 us 1557805.00 us 80 OPENDIR 0.15 21618.97 us 19.00 us 1410202.00 us 2051 FSTAT 0.23 68568.80 us 1781.00 us 1573119.00 us 984 FSYNC 0.27 415087.87 us 24.00 us 1423645.00 us 196 STATFS 0.33 10851.24 us 16.00 us 1570142.00 us 9075 INODELK 0.60 85769.69 us 152.00 us 2328777.00 us 2065 FXATTROP 0.63 104161.26 us 53.00 us 1573331.00 us 1808 WRITE 0.71 24037.55 us 37.00 us 1570577.00 us 8788 READ 1.34 123004.33 us 13.00 us 1572980.00 us 3244 FINODELK 20.01 469342.63 us 11.00 us 1570713.00 us 12689 STAT 75.44 675578.60 us 98.00 us 1963137.00 us 33238 LOOKUP Duration: 78214 seconds Data Read: 500177719808 bytes Data Written: 79883172260 bytes Interval 1 Stats: Block Size: 128b+ 1024b+ 2048b+ No. of Reads: 0 0 0 No. of Writes: 1 566 14 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 0 0 0 No. of Writes: 623 191 123 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 0 0 10368 No. of Writes: 81 126 445 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.00 512.00 us 512.00 us 512.00 us 1 UNLINK 0.00 253232.00 us 39.00 us 506425.00 us 2 ENTRYLK 0.02 567538.67 us 61455.00 us 1096240.00 us 6 READDIRP 0.10 425877.85 us 52.00 us 1412618.00 us 34 OPENDIR 0.10 6947.28 us 340.00 us 1416193.00 us 2133 RCHECKSUM 0.17 24302.80 us 20.00 us 1393421.00 us 1016 FSTAT 0.20 79981.08 us 1784.00 us 1444418.00 us 364 FSYNC 0.27 432331.46 us 24.00 us 1393263.00 us 90 STATFS 0.34 11557.54 us 16.00 us 1441854.00 us 4303 INODELK 0.51 94985.64 us 162.00 us 2328777.00 us 775 FXATTROP 0.63 128079.62 us 58.00 us 1442703.00 us 712 WRITE 0.73 23597.75 us 37.00 us 1416135.00 us 4456 READ 0.86 109550.04 us 13.00 us 1443792.00 us 1140 FINODELK 19.86 470022.73 us 14.00 us 1412768.00 us 6122 STAT 76.20 702701.46 us 98.00 us 1963137.00 us 15711 LOOKUP Duration: 246 seconds Data Read: 1358954496 bytes Data Written: 85018520 bytes Brick: HOSTNAME-01-B:/brick8/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 0 148 9 No. of Writes: 2198 856 1414 Block Size: 8b+ 16b+ 32b+ No. of Reads: 17 52 2454 No. of Writes: 2883 23076 27561 Block Size: 64b+ 128b+ 256b+ No. of Reads: 2418 1055 843 No. of Writes: 148757 245049 274675 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 8942 12954 30505 No. of Writes: 1985063 409642431 64134867 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 112381 168949 285944 No. of Writes: 92567046 63502163 116489833 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 743469 1210030 816372704 No. of Writes: 17318963 70068548 142121633 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 0 0 No. of Writes: 1088 11 111 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 1179035 FORGET 0.00 0.00 us 0.00 us 0.00 us 2293524 RELEASE 0.00 0.00 us 0.00 us 0.00 us 26725606 RELEASEDIR 0.00 292.00 us 292.00 us 292.00 us 1 LINK 0.00 36.47 us 24.00 us 45.00 us 15 FLUSH 0.00 428.50 us 358.00 us 499.00 us 2 RENAME 0.00 166.80 us 121.00 us 220.00 us 10 SETATTR 0.00 70.97 us 35.00 us 326.00 us 30 LK 0.00 58.15 us 37.00 us 105.00 us 55 FINODELK 0.00 1624.25 us 770.00 us 3488.00 us 8 MKNOD 0.00 61.19 us 27.00 us 334.00 us 214 ENTRYLK 0.01 242.92 us 154.00 us 560.00 us 78 SETXATTR 0.03 85.31 us 28.00 us 2293.00 us 894 STATFS 0.05 26925.20 us 294.00 us 90304.00 us 5 UNLINK 0.06 3003.38 us 297.00 us 57010.00 us 55 FXATTROP 0.07 2474.06 us 475.00 us 36988.00 us 78 MKDIR 0.10 68.06 us 27.00 us 2641.00 us 4293 FSTAT 0.11 82114.75 us 351.00 us 310420.00 us 4 READDIR 0.21 97.59 us 3.00 us 3330.00 us 6260 OPENDIR 0.39 134.22 us 47.00 us 26252.00 us 8479 OPEN 0.44 128.85 us 12.00 us 25805.00 us 9866 STAT 0.80 151.41 us 75.00 us 69114.00 us 15222 WRITE 0.83 69.77 us 18.00 us 599472.00 us 34079 INODELK 3.07 476.92 us 48.00 us 59897.00 us 18573 GETXATTR 8.90 505.04 us 47.00 us 120915.00 us 50768 READ 10.30 284.59 us 20.00 us 120764.00 us 104241 LOOKUP 74.63 19961.45 us 52.00 us 5262484.00 us 10772 READDIRP Duration: 4279793 seconds Data Read: 107167648980220 bytes Data Written: 31737926706873 bytes Interval 5 Stats: Block Size: 131072b+ No. of Reads: 2903 No. of Writes: 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.00 85.00 us 62.00 us 108.00 us 2 ENTRYLK 0.00 185.00 us 185.00 us 185.00 us 1 SETATTR 0.03 1168.50 us 77.00 us 2260.00 us 2 INODELK 0.07 93.63 us 40.00 us 165.00 us 62 FSTAT 0.07 177.49 us 54.00 us 955.00 us 35 OPENDIR 0.14 135.89 us 28.00 us 2293.00 us 91 STATFS 0.49 43269.00 us 43269.00 us 43269.00 us 1 UNLINK 2.25 1365.04 us 164.00 us 22604.00 us 145 READDIRP 5.95 153.65 us 18.00 us 9302.00 us 3397 STAT 36.13 2655.90 us 61.00 us 101625.00 us 1194 READ 54.87 321.63 us 74.00 us 33476.00 us 14974 LOOKUP Duration: 246 seconds Data Read: 380502016 bytes Data Written: 0 bytes Brick: HOSTNAME-01-B:/brick6/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 0 1 0 No. of Writes: 2161 831 1435 Block Size: 8b+ 16b+ 32b+ No. of Reads: 0 0 2903 No. of Writes: 4166 5588026 44908 Block Size: 64b+ 128b+ 256b+ No. of Reads: 3536 5 80 No. of Writes: 87870 241581 376594 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 8020 7205 19984 No. of Writes: 7805307 539972646 80512909 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 62273 100742 210754 No. of Writes: 102909970 76529387 166357551 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 551103 735158 761717336 No. of Writes: 15712591 54848367 117383734 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 5 360612 No. of Writes: 3717 5306 1855286 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 835383 FORGET 0.00 0.00 us 0.00 us 0.00 us 817859 RELEASE 0.00 0.00 us 0.00 us 0.00 us 28832755 RELEASEDIR 0.00 42.75 us 30.00 us 67.00 us 4 FLUSH 0.00 65.38 us 43.00 us 120.00 us 8 LK 0.00 172.33 us 123.00 us 238.00 us 9 SETATTR 0.00 548.14 us 264.00 us 1315.00 us 7 UNLINK 0.00 1554.89 us 596.00 us 4101.00 us 9 MKNOD 0.00 59.64 us 24.00 us 520.00 us 238 ENTRYLK 0.00 245.50 us 150.00 us 493.00 us 78 SETXATTR 0.00 20433.00 us 20433.00 us 20433.00 us 1 RENAME 0.00 2955.50 us 81.00 us 22895.00 us 8 FSTAT 0.01 7511.17 us 216.00 us 42274.00 us 6 READDIR 0.01 85.94 us 26.00 us 2189.00 us 894 STATFS 0.01 4187.81 us 2124.00 us 20241.00 us 21 FSYNC 0.04 1532.93 us 223.00 us 66019.00 us 202 FXATTROP 0.05 4863.85 us 568.00 us 145709.00 us 78 MKDIR 0.11 131.42 us 61.00 us 3438.00 us 6721 OPEN 0.11 91.46 us 2.00 us 29957.00 us 10315 OPENDIR 0.17 142.09 us 15.00 us 12445.00 us 9865 STAT 0.35 107.33 us 18.00 us 774166.00 us 27066 INODELK 1.21 480.50 us 23.00 us 121342.00 us 21084 GETXATTR 2.71 145604.99 us 38.00 us 1277202.00 us 155 FINODELK 3.14 247.45 us 14.00 us 64731.00 us 105762 LOOKUP 3.36 282.18 us 68.00 us 131903.00 us 99206 READ 15.81 7013.42 us 49.00 us 4783006.00 us 18800 READDIRP 72.91 61747.58 us 84.00 us 612823.00 us 9846 WRITE Duration: 4279793 seconds Data Read: 100321376892903 bytes Data Written: 29652026734449 bytes Interval 5 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.01 157.00 us 50.00 us 436.00 us 4 ENTRYLK 0.12 177.06 us 58.00 us 1407.00 us 35 OPENDIR 0.24 138.63 us 39.00 us 1793.00 us 91 STATFS 0.38 20433.00 us 20433.00 us 20433.00 us 1 RENAME 2.46 1564.36 us 149.00 us 18439.00 us 84 READDIRP 8.49 133.50 us 19.00 us 7417.00 us 3397 STAT 88.30 313.58 us 81.00 us 20208.00 us 15040 LOOKUP Duration: 246 seconds Data Read: 0 bytes Data Written: 0 bytes Brick: HOSTNAME-01-B:/brick4/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 2b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 3 1 7 Block Size: 16b+ 32b+ 64b+ No. of Reads: 0 3 0 No. of Writes: 14 176 55 Block Size: 128b+ 256b+ 512b+ No. of Reads: 0 0 6 No. of Writes: 551 409 1327 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 2 25 20 No. of Writes: 163855 98807 281768 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 52 81 142 No. of Writes: 105053 164786 104540 Block Size: 65536b+ 131072b+ 262144b+ No. of Reads: 311 2267877 0 No. of Writes: 353531 7948711 10 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 61063 FORGET 0.00 0.00 us 0.00 us 0.00 us 14884 RELEASE 0.00 0.00 us 0.00 us 0.00 us 95159 RELEASEDIR 0.00 70.00 us 70.00 us 70.00 us 1 LK 0.00 59.50 us 56.00 us 63.00 us 2 ENTRYLK 0.00 174.00 us 174.00 us 174.00 us 1 SETATTR 0.00 142.19 us 70.00 us 229.00 us 26 OPEN 0.00 264.95 us 47.00 us 3405.00 us 81 OPENDIR 0.01 5896.86 us 27.00 us 127468.00 us 22 FINODELK 0.02 199117.00 us 199117.00 us 199117.00 us 1 MKNOD 0.12 5566.34 us 20.00 us 545166.00 us 197 STATFS 0.25 312.26 us 26.00 us 700937.00 us 7444 FSTAT 0.86 63127.77 us 162.00 us 952344.00 us 125 READDIRP 0.91 77367.29 us 212.00 us 1039173.00 us 107 FXATTROP 1.42 209840.71 us 121.00 us 620314.00 us 62 WRITE 1.48 95.93 us 15.00 us 722187.00 us 141085 INODELK 4.15 2543.50 us 57.00 us 721863.00 us 14886 READ 13.69 15652.88 us 16.00 us 721267.00 us 7986 STAT 17.78 2302.89 us 363.00 us 834931.00 us 70497 RCHECKSUM 59.29 16876.75 us 68.00 us 1036627.00 us 32076 LOOKUP Duration: 77227 seconds Data Read: 297294236329 bytes Data Written: 1094314598941 bytes Interval 1 Stats: Block Size: 8192b+ 32768b+ 65536b+ No. of Reads: 0 0 0 No. of Writes: 1 3 11 Block Size: 131072b+ No. of Reads: 17261 No. of Writes: 47 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.00 193.43 us 54.00 us 1791.00 us 35 OPENDIR 0.02 554.58 us 20.00 us 31421.00 us 91 STATFS 0.06 12897.40 us 32.00 us 127468.00 us 10 FINODELK 1.02 312.28 us 26.00 us 700937.00 us 7443 FSTAT 1.47 44044.13 us 162.00 us 917890.00 us 76 READDIRP 2.55 92.91 us 16.00 us 722187.00 us 62715 INODELK 2.99 227428.40 us 121.00 us 534197.00 us 30 WRITE 3.12 139535.71 us 259.00 us 1039173.00 us 51 FXATTROP 8.24 5538.18 us 18.00 us 721267.00 us 3397 STAT 16.69 2558.28 us 57.00 us 721863.00 us 14886 READ 22.42 1632.14 us 363.00 us 718808.00 us 31356 RCHECKSUM 41.42 6177.50 us 76.00 us 1036627.00 us 15304 LOOKUP Duration: 246 seconds Data Read: 2262433792 bytes Data Written: 7489024 bytes Brick: HOSTNAME-01-B:/brick5/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 0 0 2 No. of Writes: 482 547 864 Block Size: 8b+ 16b+ 32b+ No. of Reads: 12 39 3645 No. of Writes: 1717 18223 16043 Block Size: 64b+ 128b+ 256b+ No. of Reads: 2441 542 390 No. of Writes: 607834 219970 204446 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 6262 5164 13211 No. of Writes: 1064280 203635430 61786149 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 70712 123308 170598 No. of Writes: 80206120 104132098 110568574 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 493854 698297 533801800 No. of Writes: 18319266 76209603 132443069 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 0 2 No. of Writes: 837 1 11 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 583048 FORGET 0.00 0.00 us 0.00 us 0.00 us 1791579 RELEASE 0.00 0.00 us 0.00 us 0.00 us 25457262 RELEASEDIR 0.00 322.17 us 205.00 us 808.00 us 6 SETATTR 0.00 2292.58 us 31.00 us 16199.00 us 12 FLUSH 0.00 28625.50 us 10170.00 us 47081.00 us 2 UNLINK 0.00 2983.42 us 51.00 us 37006.00 us 24 LK 0.00 15029.17 us 1096.00 us 44572.00 us 6 MKNOD 0.00 4302.60 us 150.00 us 66649.00 us 78 SETXATTR 0.01 13873.86 us 808.00 us 107723.00 us 78 MKDIR 0.01 755.09 us 121.00 us 211768.00 us 1698 FXATTROP 0.01 4013.43 us 23.00 us 71921.00 us 334 ENTRYLK 0.02 220483.93 us 257.00 us 516216.00 us 15 READDIR 0.03 4518.96 us 31.00 us 76937.00 us 894 STATFS 0.04 7217.03 us 1872.00 us 1382417.00 us 811 FSYNC 0.09 5690.68 us 2.00 us 2641373.00 us 2147 OPENDIR 0.10 1368.02 us 15.00 us 134024.00 us 9865 STAT 0.16 1421.97 us 48.00 us 3165390.00 us 15489 OPEN 0.17 2595.75 us 26.00 us 2281303.00 us 8854 GETXATTR 0.21 2879.52 us 22.00 us 132848.00 us 9957 FSTAT 0.21 11982.66 us 54.00 us 2340814.00 us 2438 READDIRP 1.42 2157.46 us 39.00 us 3980667.00 us 90417 LOOKUP 2.01 1851.28 us 13.00 us 4353557.00 us 149524 INODELK 2.09 6592.01 us 333.00 us 2340514.00 us 43669 RCHECKSUM 3.92 8208.32 us 37.00 us 284665.00 us 65801 READ 13.14 726280.54 us 26.00 us 4778309.00 us 2490 FINODELK 76.36 304426.35 us 47.00 us 3078703.00 us 34522 WRITE Duration: 4279793 seconds Data Read: 70063314642216 bytes Data Written: 30894365505026 bytes Interval 5 Stats: Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 0 No. of Writes: 2 1 5 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 0 0 0 No. of Writes: 10 19 31 Block Size: 65536b+ 131072b+ No. of Reads: 0 19354 No. of Writes: 206 675 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.00 142.03 us 51.00 us 558.00 us 35 OPENDIR 0.00 207.80 us 37.00 us 4278.00 us 91 STATFS 0.00 79.99 us 31.00 us 141.00 us 290 FSTAT 0.01 1539.89 us 170.00 us 8853.00 us 54 READDIRP 0.02 260.63 us 121.00 us 1374.00 us 776 FXATTROP 0.11 3223.89 us 2100.00 us 5998.00 us 388 FSYNC 0.14 444.37 us 15.00 us 15289.00 us 3397 STAT 0.83 593.88 us 76.00 us 47077.00 us 15531 LOOKUP 2.89 83325.97 us 86.00 us 502479.00 us 387 WRITE 5.95 8661.33 us 47.00 us 267392.00 us 7668 READ 8.17 3374.06 us 18.00 us 3834382.00 us 27019 INODELK 10.53 8711.57 us 411.00 us 496077.00 us 13488 RCHECKSUM 71.35 688576.36 us 37.00 us 4310436.00 us 1156 FINODELK Duration: 246 seconds Data Read: 2536767488 bytes Data Written: 114949120 bytes Brick: HOSTNAME-01-B:/brick1/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 32b+ 128b+ 512b+ No. of Reads: 10 16 0 No. of Writes: 2 0 3 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 2 2 4 No. of Writes: 4 1 230 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 2 8 10 No. of Writes: 62 111 342 Block Size: 65536b+ 131072b+ No. of Reads: 12 39099 No. of Writes: 1999 14756 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 110 FORGET 0.00 0.00 us 0.00 us 0.00 us 1552 RELEASE 0.00 0.00 us 0.00 us 0.00 us 5486 RELEASEDIR 0.00 33.00 us 33.00 us 33.00 us 1 FLUSH 0.00 75.00 us 75.00 us 75.00 us 1 LK 0.00 177.50 us 93.00 us 313.00 us 10 FSTAT 0.00 23790.00 us 23790.00 us 23790.00 us 1 UNLINK 0.00 308.52 us 28.00 us 9424.00 us 197 STATFS 0.01 5786.87 us 124.00 us 196553.00 us 102 READDIRP 0.01 166.50 us 54.00 us 37190.00 us 3921 FXATTROP 0.02 1713.96 us 155.00 us 57875.00 us 440 FSYNC 0.03 13994.76 us 26.00 us 115223.00 us 86 GETXATTR 0.03 9081.56 us 33.00 us 119977.00 us 134 OPENDIR 0.04 58848.03 us 100.00 us 709195.00 us 33 READDIR 0.05 262.36 us 15.00 us 16663.00 us 8244 STAT 0.08 40910.06 us 598.00 us 174924.00 us 88 XATTROP 0.16 20063.18 us 25.00 us 165004.00 us 355 ENTRYLK 0.21 17832.65 us 77.00 us 171068.00 us 545 READ 0.31 7303.16 us 65.00 us 559590.00 us 1961 WRITE 2.79 2933.18 us 67.00 us 208762.00 us 43381 LOOKUP 5.41 33917.36 us 430.00 us 842225.00 us 7262 RCHECKSUM 13.39 41023.35 us 20.00 us 13484708.00 us 14865 INODELK 77.46 603032.78 us 17.00 us 14543086.00 us 5850 FINODELK Duration: 2952 seconds Data Read: 5126633408 bytes Data Written: 2195986825 bytes Interval 1 Stats: Block Size: 4096b+ 16384b+ 32768b+ No. of Reads: 0 0 0 No. of Writes: 10 1 37 Block Size: 65536b+ 131072b+ No. of Reads: 0 678 No. of Writes: 364 1888 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 17 FORGET 0.00 0.00 us 0.00 us 0.00 us 159 RELEASEDIR 0.00 175.83 us 93.00 us 313.00 us 6 FSTAT 0.00 372.52 us 34.00 us 9424.00 us 91 STATFS 0.01 155.47 us 54.00 us 15012.00 us 1586 FXATTROP 0.02 1662.29 us 603.00 us 19925.00 us 185 FSYNC 0.02 5773.04 us 124.00 us 196553.00 us 56 READDIRP 0.03 11697.47 us 56.00 us 119977.00 us 55 OPENDIR 0.04 19956.53 us 41.00 us 115223.00 us 36 GETXATTR 0.04 48883.81 us 100.00 us 119392.00 us 16 READDIR 0.06 344.76 us 15.00 us 16663.00 us 3514 STAT 0.10 39258.90 us 800.00 us 140015.00 us 52 XATTROP 0.17 19739.70 us 28.00 us 165004.00 us 169 ENTRYLK 0.29 19225.55 us 85.00 us 154901.00 us 299 READ 0.38 9418.15 us 74.00 us 180442.00 us 795 WRITE 3.27 3127.60 us 74.00 us 208762.00 us 20788 LOOKUP 6.44 35274.85 us 431.00 us 310417.00 us 3637 RCHECKSUM 13.94 37449.35 us 25.00 us 12546026.00 us 7412 INODELK 75.20 635507.62 us 19.00 us 14543086.00 us 2356 FINODELK Duration: 246 seconds Data Read: 88866816 bytes Data Written: 294647296 bytes Brick: HOSTNAME-01-B:/brick9/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 0 55 9 No. of Writes: 1491 1000 2102 Block Size: 8b+ 16b+ 32b+ No. of Reads: 64 50 4812 No. of Writes: 4696 7222905 44892 Block Size: 64b+ 128b+ 256b+ No. of Reads: 4319 4022 3248 No. of Writes: 293891 418033 542883 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 12528 18194 54796 No. of Writes: 8494202 664144673 121431414 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 216483 292636 465186 No. of Writes: 206999000 164921303 247436952 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 1428505 2116990 1026270560 No. of Writes: 33487910 131100929 250121202 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 0 0 No. of Writes: 3112 1649 486445 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 1466412 FORGET 0.00 0.00 us 0.00 us 0.00 us 1649265 RELEASE 0.00 0.00 us 0.00 us 0.00 us 26523053 RELEASEDIR 0.00 41.00 us 18.00 us 134.00 us 13 FLUSH 0.00 61.28 us 37.00 us 155.00 us 25 LK 0.00 6961.00 us 6961.00 us 6961.00 us 1 RENAME 0.00 1194.50 us 237.00 us 4764.00 us 6 READDIR 0.00 28213.00 us 28213.00 us 28213.00 us 1 UNLINK 0.00 6334.43 us 126.00 us 25589.00 us 7 SETATTR 0.00 12989.29 us 800.00 us 37294.00 us 7 MKNOD 0.00 1259.36 us 128.00 us 32443.00 us 78 SETXATTR 0.01 1803.81 us 25.00 us 44751.00 us 204 ENTRYLK 0.01 7197.82 us 506.00 us 90051.00 us 78 MKDIR 0.04 3359.54 us 21.00 us 216237.00 us 894 STATFS 0.05 25889.78 us 600.00 us 236221.00 us 130 FSYNC 0.06 1592.46 us 2.00 us 381435.00 us 2539 OPENDIR 0.06 463.18 us 60.00 us 539032.00 us 8960 OPEN 0.12 841.19 us 14.00 us 217750.00 us 9865 STAT 0.25 1861.16 us 20.00 us 247927.00 us 9467 FSTAT 0.28 544.06 us 17.00 us 1376223.00 us 36022 INODELK 0.30 13330.61 us 159.00 us 399369.00 us 1583 FXATTROP 0.33 2010.68 us 29.00 us 537519.00 us 11519 GETXATTR 0.66 35582.06 us 18.00 us 983226.00 us 1290 FINODELK 1.93 1499.23 us 47.00 us 1543777.00 us 89791 LOOKUP 4.19 87790.99 us 52.00 us 11504765.00 us 3329 READDIRP 13.62 2616.76 us 35.00 us 534914.00 us 363070 READ 78.09 65133.76 us 49.00 us 1024146.00 us 83658 WRITE Duration: 4279793 seconds Data Read: 134801101778949 bytes Data Written: 58578267669735 bytes Interval 5 Stats: Block Size: 131072b+ No. of Reads: 1129 No. of Writes: 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.00 81.00 us 77.00 us 85.00 us 2 ENTRYLK 0.10 197.94 us 61.00 us 1277.00 us 35 OPENDIR 0.17 127.99 us 37.00 us 1322.00 us 91 STATFS 0.40 827.09 us 58.00 us 24347.00 us 33 FSTAT 0.41 28213.00 us 28213.00 us 28213.00 us 1 UNLINK 7.75 156.57 us 17.00 us 16695.00 us 3397 STAT 8.23 3870.01 us 138.00 us 284848.00 us 146 READDIRP 10.90 1623.22 us 50.00 us 23865.00 us 461 READ 72.04 327.04 us 87.00 us 60571.00 us 15125 LOOKUP Duration: 246 seconds Data Read: 147980288 bytes Data Written: 0 bytes Brick: HOSTNAME-01-B:/brick3/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 0 12 6 No. of Writes: 2567 781 1340 Block Size: 8b+ 16b+ 32b+ No. of Reads: 31 66 4036 No. of Writes: 2948 440590 38913 Block Size: 64b+ 128b+ 256b+ No. of Reads: 2734 1728 1167 No. of Writes: 75799 347793 470159 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 9774 16256 29003 No. of Writes: 2265947 446846351 86298960 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 118753 170352 279731 No. of Writes: 147940171 119810004 134459511 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 1011892 1204426 639618284 No. of Writes: 29300584 134801413 251115548 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 3 368642 No. of Writes: 1808 14 312 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 556431 FORGET 0.00 0.00 us 0.00 us 0.00 us 646004 RELEASE 0.00 0.00 us 0.00 us 0.00 us 24454045 RELEASEDIR 0.00 308.00 us 308.00 us 308.00 us 1 RENAME 0.00 39.80 us 27.00 us 67.00 us 15 FLUSH 0.00 452.50 us 304.00 us 601.00 us 2 READDIR 0.00 62.24 us 43.00 us 88.00 us 29 LK 0.00 167.55 us 133.00 us 195.00 us 11 SETATTR 0.00 230.03 us 138.00 us 610.00 us 78 SETXATTR 0.00 252.74 us 27.00 us 40634.00 us 205 ENTRYLK 0.00 4938.91 us 750.00 us 39272.00 us 11 MKNOD 0.00 24506.75 us 379.00 us 59086.00 us 4 UNLINK 0.00 64.45 us 26.00 us 1244.00 us 1721 FSTAT 0.00 93.36 us 3.00 us 6609.00 us 5953 OPENDIR 0.00 725.79 us 29.00 us 556857.00 us 894 STATFS 0.00 151.83 us 73.00 us 40777.00 us 4636 OPEN 0.00 9566.51 us 531.00 us 85953.00 us 78 MKDIR 0.05 192.95 us 40.00 us 96090.00 us 46282 READ 0.05 2198.09 us 349.00 us 123915.00 us 4724 RCHECKSUM 0.12 2289.39 us 15.00 us 574911.00 us 9870 STAT 0.16 2874.39 us 29.00 us 725497.00 us 10731 GETXATTR 0.22 5318.90 us 112.00 us 613165.00 us 7887 FXATTROP 0.62 630.24 us 49.00 us 436551.00 us 188339 WRITE 1.04 19727.28 us 54.00 us 2411484.00 us 10085 READDIRP 1.71 3781.30 us 21.00 us 613196.00 us 86277 LOOKUP 1.86 120602.70 us 146.00 us 22966912.00 us 2935 FSYNC 4.43 29711.50 us 19.00 us 36688198.00 us 28412 INODELK 89.72 1630926.05 us 19.00 us 39459053.00 us 10488 FINODELK Duration: 4279793 seconds Data Read: 84395109496663 bytes Data Written: 55381384251629 bytes Interval 5 Stats: Block Size: 128b+ 1024b+ 2048b+ No. of Reads: 0 0 0 No. of Writes: 1 566 14 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 0 0 0 No. of Writes: 623 191 123 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 6 2 19 No. of Writes: 81 126 445 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.00 71.00 us 44.00 us 98.00 us 3 ENTRYLK 0.00 117.00 us 102.00 us 132.00 us 2 FSTAT 0.00 160.97 us 54.00 us 989.00 us 35 OPENDIR 0.00 182.71 us 37.00 us 2711.00 us 91 STATFS 0.00 37863.00 us 37863.00 us 37863.00 us 1 UNLINK 0.01 10687.38 us 93.00 us 96090.00 us 24 READ 0.04 906.65 us 112.00 us 140471.00 us 793 FXATTROP 0.04 10671.96 us 130.00 us 156901.00 us 74 READDIRP 0.05 248.06 us 17.00 us 26637.00 us 3400 STAT 0.06 2929.57 us 1830.00 us 42843.00 us 365 FSYNC 0.20 1782.90 us 377.00 us 114465.00 us 2135 RCHECKSUM 0.62 699.57 us 87.00 us 44929.00 us 16450 LOOKUP 0.91 23643.10 us 80.00 us 227392.00 us 716 WRITE 7.92 34236.55 us 19.00 us 21085778.00 us 4296 INODELK 90.15 1542280.19 us 21.00 us 23438747.00 us 1086 FINODELK Duration: 246 seconds Data Read: 2984448 bytes Data Written: 85018520 bytes Brick: HOSTNAME-01-B:/brick2/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 0 6 1 No. of Writes: 5690 1158 1507 Block Size: 8b+ 16b+ 32b+ No. of Reads: 3 8 4220 No. of Writes: 4257 653093 38884 Block Size: 64b+ 128b+ 256b+ No. of Reads: 2057 143 237 No. of Writes: 138246 284967 301583 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 9964 14318 24878 No. of Writes: 2799071 412706426 75565460 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 111870 177302 261376 No. of Writes: 190929249 79365618 255744283 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 1009823 1146396 686781597 No. of Writes: 27530847 83970418 194993443 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 1 0 No. of Writes: 1208 19 69 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 723412 FORGET 0.00 0.00 us 0.00 us 0.00 us 731255 RELEASE 0.00 0.00 us 0.00 us 0.00 us 24362990 RELEASEDIR 0.00 374.71 us 20.00 us 4282.00 us 21 FLUSH 0.00 262.48 us 35.00 us 2950.00 us 40 LK 0.00 2522.33 us 138.00 us 13040.00 us 15 SETATTR 0.00 8113.44 us 298.00 us 28903.00 us 9 RENAME 0.00 400.89 us 71.00 us 85670.00 us 1193 OPEN 0.00 6982.42 us 138.00 us 31124.00 us 78 SETXATTR 0.00 40219.80 us 955.00 us 114403.00 us 15 MKNOD 0.00 807.49 us 25.00 us 37564.00 us 894 STATFS 0.01 586.83 us 27.00 us 52770.00 us 1780 FSTAT 0.01 94259.50 us 1264.00 us 259989.00 us 12 UNLINK 0.01 238.64 us 15.00 us 418394.00 us 5290 ENTRYLK 0.03 20173.97 us 302.00 us 1014766.00 us 195 XATTROP 0.05 90005.90 us 1022.00 us 480876.00 us 78 MKDIR 0.06 908.45 us 15.00 us 46094.00 us 10388 STAT 0.06 2539.71 us 38.00 us 981906.00 us 3719 OPENDIR 0.29 11452.34 us 20.00 us 5427128.00 us 3916 GETXATTR 0.95 1026081.59 us 632.00 us 18320824.00 us 141 FSYNC 1.19 554.78 us 14.00 us 9904525.00 us 327694 INODELK 1.67 3739.81 us 67.00 us 9923314.00 us 68290 LOOKUP 1.76 2587364.42 us 45.00 us 134512764.00 us 104 READDIR 4.39 208181.93 us 58.00 us 39750208.00 us 3224 READDIRP 4.59 4377.20 us 259.00 us 10053127.00 us 160191 RCHECKSUM 7.95 5493.00 us 38.00 us 1437578.00 us 221339 READ 21.05 110957.49 us 151.00 us 15390646.00 us 29002 FXATTROP 27.49 29140.19 us 47.00 us 5380701.00 us 144248 WRITE 28.45 103576.62 us 12.00 us 39311904.00 us 41993 FINODELK Duration: 4279793 seconds Data Read: 90182870915545 bytes Data Written: 43304871600094 bytes Interval 5 Stats: Block Size: 131072b+ No. of Reads: 585 No. of Writes: 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.00 390.00 us 390.00 us 390.00 us 1 FSTAT 0.01 363.34 us 54.00 us 4960.00 us 35 OPENDIR 0.01 175.43 us 30.00 us 3792.00 us 91 STATFS 0.22 3670.76 us 65.00 us 173128.00 us 148 READ 0.54 18823.38 us 146.00 us 428923.00 us 72 READDIRP 1.75 55.66 us 16.00 us 63524.00 us 78812 INODELK 2.19 1615.88 us 16.00 us 28474.00 us 3397 STAT 12.31 2043.55 us 96.00 us 127094.00 us 15129 LOOKUP 82.99 5290.98 us 359.00 us 998606.00 us 39404 RCHECKSUM Duration: 246 seconds Data Read: 76677120 bytes Data Written: 0 bytes Brick: HOSTNAME-01-B:/brick7/gvAA01/brick ----------------------------------------- Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 0 13 4 No. of Writes: 1196 734 1040 Block Size: 8b+ 16b+ 32b+ No. of Reads: 1 36 3299 No. of Writes: 2062 300726 37753 Block Size: 64b+ 128b+ 256b+ No. of Reads: 2023 496 525 No. of Writes: 78098 422006 308797 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 18849 14540 35638 No. of Writes: 1846989 429384177 84605233 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 150787 230288 420816 No. of Writes: 137132537 56105000 83554076 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 1193840 1821978 1328519682 No. of Writes: 23571537 108072617 188243753 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 0 0 No. of Writes: 1439 45 467 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 584854 FORGET 0.00 0.00 us 0.00 us 0.00 us 847666 RELEASE 0.00 0.00 us 0.00 us 0.00 us 25363464 RELEASEDIR 0.00 64.12 us 40.00 us 118.00 us 8 LK 0.00 55.38 us 28.00 us 195.00 us 13 FLUSH 0.00 584.50 us 296.00 us 873.00 us 2 XATTROP 0.00 191.79 us 117.00 us 302.00 us 14 SETATTR 0.00 861.00 us 700.00 us 921.00 us 6 MKNOD 0.00 15796.00 us 15796.00 us 15796.00 us 1 RENAME 0.00 242.94 us 137.00 us 827.00 us 78 SETXATTR 0.00 80.85 us 29.00 us 2109.00 us 280 ENTRYLK 0.00 40533.00 us 40533.00 us 40533.00 us 1 LINK 0.00 12168.00 us 3663.00 us 29808.00 us 5 CREATE 0.00 85.45 us 27.00 us 4758.00 us 894 STATFS 0.00 1005.23 us 84.00 us 6375.00 us 157 FSYNC 0.00 2083.33 us 570.00 us 29605.00 us 78 MKDIR 0.00 46997.20 us 1143.00 us 78119.00 us 5 UNLINK 0.00 57905.17 us 236.00 us 344520.00 us 6 READDIR 0.01 76.43 us 28.00 us 86185.00 us 8213 FSTAT 0.02 155.31 us 76.00 us 61948.00 us 9793 OPEN 0.02 7686.27 us 146.00 us 89509.00 us 231 RCHECKSUM 0.04 100.69 us 3.00 us 41331.00 us 31020 OPENDIR 0.09 3927.64 us 81.00 us 852406.00 us 1732 FXATTROP 0.13 238.47 us 21.00 us 36324.00 us 39834 GETXATTR 0.20 217.49 us 42.00 us 104309.00 us 68968 READ 0.55 1549.97 us 62.00 us 186554.00 us 25900 WRITE 1.14 8343.75 us 13.00 us 877090.00 us 10010 STAT 2.36 2870.24 us 55.00 us 133764.00 us 60231 READDIRP 3.39 1648.07 us 15.00 us 1386407.00 us 150815 LOOKUP 18.82 34568.94 us 20.00 us 255108468.00 us 39875 INODELK 73.20 2950233.38 us 22.00 us 238932201.00 us 1817 FINODELK Duration: 4279793 seconds Data Read: 174378591384147 bytes Data Written: 42067381218610 bytes Interval 5 Stats: Block Size: 32b+ 2048b+ 4096b+ No. of Reads: 1 0 0 No. of Writes: 0 2 90 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 0 0 1 No. of Writes: 34 59 174 Block Size: 65536b+ 131072b+ No. of Reads: 0 102 No. of Writes: 1045 1863 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 8 FORGET 0.00 0.00 us 0.00 us 0.00 us 5 RELEASE 0.00 0.00 us 0.00 us 0.00 us 93 RELEASEDIR 0.00 48.00 us 35.00 us 61.00 us 2 FLUSH 0.00 182.00 us 79.00 us 285.00 us 2 FSTAT 0.00 188.50 us 184.00 us 193.00 us 2 SETATTR 0.00 873.00 us 873.00 us 873.00 us 1 XATTROP 0.00 492.00 us 154.00 us 1073.00 us 3 OPEN 0.00 314.67 us 128.00 us 658.00 us 6 GETXATTR 0.00 115.02 us 29.00 us 1537.00 us 44 ENTRYLK 0.00 168.39 us 45.00 us 861.00 us 38 OPENDIR 0.00 166.43 us 39.00 us 4758.00 us 91 STATFS 0.00 40533.00 us 40533.00 us 40533.00 us 1 LINK 0.00 23496.50 us 17185.00 us 29808.00 us 2 CREATE 0.00 1991.03 us 209.00 us 17197.00 us 35 READ 0.00 1044.09 us 84.00 us 3060.00 us 74 FSYNC 0.00 1268.33 us 159.00 us 8139.00 us 70 READDIRP 0.00 39216.75 us 1143.00 us 60962.00 us 4 UNLINK 0.01 149.32 us 16.00 us 55158.00 us 3479 STAT 0.01 7967.72 us 146.00 us 76151.00 us 106 RCHECKSUM 0.02 2188.39 us 81.00 us 87414.00 us 500 FXATTROP 0.08 296.47 us 80.00 us 20575.00 us 16886 LOOKUP 0.24 10693.63 us 85.00 us 186554.00 us 1365 WRITE 20.68 5275482.42 us 32.00 us 255108468.00 us 239 INODELK 78.95 8146074.61 us 22.00 us 238932201.00 us 591 FINODELK Duration: 246 seconds Data Read: 13408908 bytes Data Written: 373263308 bytes Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick2 ----------------------------------------------- Cumulative Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 4595631725 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 5210160 FORGET 0.00 0.00 us 0.00 us 0.00 us 19833857 RELEASE 0.00 0.00 us 0.00 us 0.00 us 121845659 RELEASEDIR 0.00 47.77 us 29.00 us 72.00 us 22 FLUSH 0.00 67.21 us 32.00 us 161.00 us 42 LK 0.00 190.20 us 142.00 us 368.00 us 15 SETATTR 0.00 355.78 us 255.00 us 408.00 us 9 RENAME 0.01 660.58 us 239.00 us 988.00 us 12 UNLINK 0.01 223.95 us 142.00 us 652.00 us 78 SETXATTR 0.02 1351.87 us 991.00 us 1807.00 us 15 MKNOD 0.11 1625.97 us 1013.00 us 3225.00 us 78 MKDIR 0.14 134.56 us 73.00 us 1296.00 us 1194 OPEN 0.18 1095.03 us 265.00 us 17673.00 us 195 XATTROP 0.25 56.99 us 18.00 us 3303.00 us 5290 ENTRYLK 0.57 286.73 us 46.00 us 28784.00 us 2379 OPENDIR 2.47 69.67 us 17.00 us 31954.00 us 42231 FINODELK 3.17 26765.84 us 73.00 us 243526.00 us 141 FSYNC 6.30 73.60 us 22.00 us 18488.00 us 101758 WRITE 9.51 389.77 us 167.00 us 21472.00 us 29019 FXATTROP 17.26 62.64 us 16.00 us 37178.00 us 327699 INODELK 59.99 1046.54 us 81.00 us 83922.00 us 68182 LOOKUP Duration: 19297631 seconds Data Read: 0 bytes Data Written: 4595631725 bytes Interval 5 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.11 1109.06 us 64.00 us 28784.00 us 35 OPENDIR 14.97 64.35 us 16.00 us 36247.00 us 78809 INODELK 84.92 1903.39 us 101.00 us 57323.00 us 15117 LOOKUP Duration: 246 seconds Data Read: 0 bytes Data Written: 0 bytes Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick5 ----------------------------------------------- Cumulative Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 3725579480 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 4675824 FORGET 0.00 0.00 us 0.00 us 0.00 us 3501480 RELEASE 0.00 0.00 us 0.00 us 0.00 us 121321048 RELEASEDIR 0.00 43.60 us 26.00 us 60.00 us 5 GETXATTR 0.00 183.50 us 136.00 us 254.00 us 6 SETATTR 0.00 94.50 us 34.00 us 360.00 us 12 FLUSH 0.00 637.50 us 378.00 us 897.00 us 2 UNLINK 0.00 152.08 us 54.00 us 1647.00 us 24 LK 0.01 923.40 us 352.00 us 2505.00 us 5 READDIR 0.02 1971.33 us 952.00 us 4079.00 us 6 MKNOD 0.03 278.36 us 158.00 us 1290.00 us 78 SETXATTR 0.03 70.37 us 27.00 us 906.00 us 334 ENTRYLK 0.06 147.48 us 83.00 us 1130.00 us 311 OPEN 0.18 1747.03 us 991.00 us 14702.00 us 78 MKDIR 0.28 178.04 us 33.00 us 9265.00 us 1190 OPENDIR 0.31 92.93 us 27.00 us 28425.00 us 2510 FINODELK 2.11 1980.44 us 117.00 us 116330.00 us 812 FSYNC 2.48 1109.55 us 141.00 us 193978.00 us 1702 FXATTROP 3.54 77.89 us 21.00 us 19627.00 us 34622 WRITE 7.06 60.51 us 21.00 us 43675.00 us 88861 INODELK 83.89 470.13 us 46.00 us 227967.00 us 135916 LOOKUP Duration: 19297631 seconds Data Read: 0 bytes Data Written: 3725579480 bytes Interval 5 Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 949 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.09 532.37 us 59.00 us 4354.00 us 35 OPENDIR 0.15 79.14 us 42.00 us 813.00 us 388 WRITE 0.44 75.07 us 27.00 us 1936.00 us 1164 FINODELK 0.94 241.59 us 141.00 us 3508.00 us 776 FXATTROP 1.86 957.12 us 124.00 us 9367.00 us 388 FSYNC 8.59 63.47 us 21.00 us 43675.00 us 27031 INODELK 87.92 1125.41 us 103.00 us 60160.00 us 15599 LOOKUP Duration: 246 seconds Data Read: 0 bytes Data Written: 949 bytes Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick9 ----------------------------------------------- Cumulative Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 13824276920 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 11714931 FORGET 0.00 0.00 us 0.00 us 0.00 us 12574659 RELEASE 0.00 0.00 us 0.00 us 0.00 us 121307214 RELEASEDIR 0.00 612.00 us 612.00 us 612.00 us 1 RENAME 0.00 53.69 us 32.00 us 75.00 us 13 FLUSH 0.00 146.43 us 118.00 us 157.00 us 7 SETATTR 0.00 1053.00 us 1053.00 us 1053.00 us 1 UNLINK 0.00 90.76 us 39.00 us 575.00 us 25 LK 0.01 1011.83 us 280.00 us 3498.00 us 6 READDIR 0.01 1154.50 us 873.00 us 1578.00 us 6 MKNOD 0.01 65.37 us 32.00 us 609.00 us 202 ENTRYLK 0.02 212.12 us 124.00 us 561.00 us 78 SETXATTR 0.05 408.86 us 31.00 us 989.00 us 111 GETXATTR 0.07 66.08 us 25.00 us 1177.00 us 1094 FINODELK 0.11 1413.50 us 793.00 us 6939.00 us 78 MKDIR 0.17 145.79 us 3.00 us 16855.00 us 1188 OPENDIR 0.90 566.03 us 178.00 us 101200.00 us 1583 FXATTROP 1.17 132.19 us 70.00 us 15892.00 us 8843 OPEN 2.28 63.93 us 25.00 us 14064.00 us 35560 INODELK 3.30 25327.51 us 77.00 us 275527.00 us 130 FSYNC 6.02 71.86 us 20.00 us 29539.00 us 83532 WRITE 85.86 950.72 us 45.00 us 234956.00 us 89984 LOOKUP Duration: 19297631 seconds Data Read: 0 bytes Data Written: 13824276920 bytes Interval 5 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.00 107.50 us 84.00 us 131.00 us 2 ENTRYLK 0.00 1053.00 us 1053.00 us 1053.00 us 1 UNLINK 0.13 946.06 us 58.00 us 16855.00 us 35 OPENDIR 99.87 1700.08 us 89.00 us 67917.00 us 15086 LOOKUP Duration: 246 seconds Data Read: 0 bytes Data Written: 0 bytes Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick1 ----------------------------------------------- Cumulative Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 2407865216 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 3885977 FORGET 0.00 0.00 us 0.00 us 0.00 us 1908766 RELEASE 0.00 0.00 us 0.00 us 0.00 us 121336583 RELEASEDIR 0.00 41.33 us 37.00 us 44.00 us 3 FLUSH 0.00 107.40 us 60.00 us 216.00 us 5 LK 0.00 197.00 us 144.00 us 367.00 us 9 SETATTR 0.00 559.25 us 269.00 us 1318.00 us 4 UNLINK 0.01 117.55 us 34.00 us 426.00 us 49 GETXATTR 0.01 729.10 us 111.00 us 2755.00 us 10 FSTAT 0.02 283.04 us 169.00 us 1903.00 us 78 SETXATTR 0.04 71.01 us 24.00 us 3460.00 us 547 ENTRYLK 0.05 5719.78 us 1045.00 us 39386.00 us 9 MKNOD 0.10 1199.26 us 338.00 us 2419.00 us 88 XATTROP 0.10 18965.50 us 244.00 us 111009.00 us 6 READDIR 0.13 1842.74 us 740.00 us 17409.00 us 78 MKDIR 0.22 196.96 us 3.00 us 13973.00 us 1241 OPENDIR 0.37 848.02 us 97.00 us 22427.00 us 478 FSYNC 0.58 142.08 us 80.00 us 9771.00 us 4495 OPEN 0.71 79.24 us 30.00 us 8519.00 us 9794 WRITE 0.72 197.05 us 83.00 us 36661.00 us 4020 FXATTROP 2.42 80.26 us 16.00 us 20655.00 us 33176 INODELK 4.57 791.78 us 19.00 us 854250.00 us 6345 FINODELK 89.97 1130.79 us 47.00 us 716773.00 us 87554 LOOKUP Duration: 19297631 seconds Data Read: 0 bytes Data Written: 2407865216 bytes Interval 5 Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 2300 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 159 RELEASEDIR 0.02 993.33 us 161.00 us 2755.00 us 6 FSTAT 0.04 85.40 us 24.00 us 3460.00 us 169 ENTRYLK 0.14 957.84 us 61.00 us 13973.00 us 55 OPENDIR 0.16 76.25 us 38.00 us 1284.00 us 795 WRITE 0.17 1246.94 us 468.00 us 2063.00 us 52 XATTROP 0.36 755.13 us 97.00 us 10173.00 us 185 FSYNC 0.68 165.35 us 92.00 us 6470.00 us 1586 FXATTROP 1.75 90.50 us 16.00 us 7633.00 us 7475 INODELK 6.20 972.86 us 23.00 us 854250.00 us 2464 FINODELK 90.50 1686.49 us 89.00 us 78332.00 us 20759 LOOKUP Duration: 246 seconds Data Read: 0 bytes Data Written: 2300 bytes Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick8 ----------------------------------------------- Cumulative Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 5492133233 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6710878 FORGET 0.00 0.00 us 0.00 us 0.00 us 5709109 RELEASE 0.00 0.00 us 0.00 us 0.00 us 121308841 RELEASEDIR 0.00 381.00 us 381.00 us 381.00 us 1 LINK 0.00 321.50 us 309.00 us 334.00 us 2 RENAME 0.00 50.13 us 31.00 us 92.00 us 15 FLUSH 0.00 68.93 us 43.00 us 136.00 us 30 LK 0.00 207.60 us 137.00 us 379.00 us 10 SETATTR 0.00 456.20 us 247.00 us 837.00 us 5 UNLINK 0.00 67.16 us 39.00 us 233.00 us 56 FINODELK 0.01 1313.12 us 943.00 us 1632.00 us 8 MKNOD 0.01 61.85 us 34.00 us 384.00 us 214 ENTRYLK 0.02 260.12 us 141.00 us 1053.00 us 78 SETXATTR 0.04 688.39 us 327.00 us 2844.00 us 56 FXATTROP 0.11 1360.77 us 862.00 us 3735.00 us 78 MKDIR 0.22 177.44 us 49.00 us 11813.00 us 1185 OPENDIR 0.90 147.72 us 78.00 us 17203.00 us 5732 OPEN 1.30 79.82 us 37.00 us 11963.00 us 15230 WRITE 1.65 67.26 us 23.00 us 13182.00 us 23091 INODELK 95.71 796.62 us 69.00 us 131536.00 us 112767 LOOKUP Duration: 19297631 seconds Data Read: 0 bytes Data Written: 5492133233 bytes Interval 5 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.00 106.00 us 84.00 us 128.00 us 2 ENTRYLK 0.00 379.00 us 379.00 us 379.00 us 1 SETATTR 0.00 513.00 us 513.00 us 513.00 us 1 UNLINK 0.00 665.50 us 662.00 us 669.00 us 2 INODELK 0.15 1173.54 us 86.00 us 11813.00 us 35 OPENDIR 99.84 1831.00 us 105.00 us 93579.00 us 15150 LOOKUP Duration: 246 seconds Data Read: 0 bytes Data Written: 0 bytes Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick3 ----------------------------------------------- Cumulative Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 4572691669 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 4594149 FORGET 0.00 0.00 us 0.00 us 0.00 us 3305802 RELEASE 0.00 0.00 us 0.00 us 0.00 us 121331753 RELEASEDIR 0.00 351.00 us 351.00 us 351.00 us 1 RENAME 0.00 52.20 us 31.00 us 74.00 us 15 FLUSH 0.00 150.27 us 116.00 us 192.00 us 11 SETATTR 0.00 70.48 us 40.00 us 155.00 us 29 LK 0.00 698.00 us 294.00 us 1204.00 us 4 UNLINK 0.01 101.02 us 45.00 us 170.00 us 123 GETXATTR 0.02 226.96 us 132.00 us 661.00 us 78 SETXATTR 0.03 2449.45 us 981.00 us 14588.00 us 11 MKNOD 0.03 133.80 us 33.00 us 14384.00 us 205 ENTRYLK 0.08 13713.00 us 250.00 us 79575.00 us 6 READDIR 0.11 1403.08 us 856.00 us 3307.00 us 78 MKDIR 0.15 130.34 us 3.00 us 7373.00 us 1188 OPENDIR 0.63 135.12 us 75.00 us 3913.00 us 4698 OPEN 1.58 151.39 us 20.00 us 720477.00 us 10441 FINODELK 2.03 70.30 us 26.00 us 31988.00 us 28857 INODELK 3.61 457.95 us 148.00 us 96986.00 us 7889 FXATTROP 13.93 74.03 us 19.00 us 38536.00 us 188198 WRITE 15.54 5296.92 us 87.00 us 285482.00 us 2936 FSYNC 62.24 723.17 us 19.00 us 1037869.00 us 86100 LOOKUP Duration: 19297631 seconds Data Read: 0 bytes Data Written: 4572691669 bytes Interval 5 Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 2170 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.00 105.67 us 65.00 us 164.00 us 3 ENTRYLK 0.01 950.00 us 950.00 us 950.00 us 1 UNLINK 0.07 398.43 us 64.00 us 5324.00 us 35 OPENDIR 0.34 88.92 us 36.00 us 4283.00 us 716 WRITE 0.52 88.23 us 22.00 us 6820.00 us 1112 FINODELK 1.21 289.14 us 156.00 us 6432.00 us 793 FXATTROP 1.53 796.04 us 104.00 us 2668.00 us 365 FSYNC 1.55 68.03 us 29.00 us 1880.00 us 4317 INODELK 94.77 1086.38 us 76.00 us 101220.00 us 16536 LOOKUP Duration: 246 seconds Data Read: 0 bytes Data Written: 2170 bytes Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick6 ----------------------------------------------- Cumulative Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 3299040333 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 4224301 FORGET 0.00 0.00 us 0.00 us 0.00 us 2341147 RELEASE 0.00 0.00 us 0.00 us 0.00 us 121317598 RELEASEDIR 0.00 75.25 us 39.00 us 117.00 us 4 FLUSH 0.00 70.62 us 47.00 us 100.00 us 8 LK 0.00 156.11 us 125.00 us 189.00 us 9 SETATTR 0.00 1980.00 us 1980.00 us 1980.00 us 1 RENAME 0.00 364.29 us 311.00 us 434.00 us 7 UNLINK 0.01 134.92 us 31.00 us 662.00 us 53 GETXATTR 0.01 1307.44 us 1005.00 us 1850.00 us 9 MKNOD 0.01 79.36 us 26.00 us 665.00 us 157 FINODELK 0.02 275.58 us 162.00 us 1593.00 us 78 SETXATTR 0.03 108.84 us 33.00 us 4176.00 us 238 ENTRYLK 0.12 1469.38 us 947.00 us 7741.00 us 78 MKDIR 0.16 17107.56 us 88.00 us 150018.00 us 9 READDIR 0.22 170.89 us 3.00 us 12533.00 us 1191 OPENDIR 0.24 1117.26 us 216.00 us 79180.00 us 202 FXATTROP 0.85 38142.10 us 498.00 us 126241.00 us 21 FSYNC 0.89 85.41 us 21.00 us 27516.00 us 9833 WRITE 0.98 138.12 us 74.00 us 3527.00 us 6670 OPEN 1.87 65.50 us 24.00 us 18157.00 us 26861 INODELK 94.56 840.11 us 19.00 us 150836.00 us 105706 LOOKUP Duration: 19297631 seconds Data Read: 0 bytes Data Written: 3299040333 bytes Interval 5 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.01 1980.00 us 1980.00 us 1980.00 us 1 RENAME 0.04 2164.25 us 113.00 us 4176.00 us 4 ENTRYLK 0.14 780.77 us 62.00 us 12533.00 us 35 OPENDIR 99.81 1300.71 us 65.00 us 96686.00 us 15135 LOOKUP Duration: 246 seconds Data Read: 0 bytes Data Written: 0 bytes Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick4 ----------------------------------------------- Cumulative Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 4188179800 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 4843776 FORGET 0.00 0.00 us 0.00 us 0.00 us 4734700 RELEASE 0.00 0.00 us 0.00 us 0.00 us 122187144 RELEASEDIR 0.00 453.00 us 453.00 us 453.00 us 1 RENAME 0.00 2935.00 us 2935.00 us 2935.00 us 1 MKNOD 0.00 314.83 us 275.00 us 348.00 us 12 LINK 0.00 46.91 us 22.00 us 394.00 us 116 FLUSH 0.00 78.42 us 37.00 us 725.00 us 78 LK 0.00 152.00 us 66.00 us 2303.00 us 41 FTRUNCATE 0.01 120.96 us 27.00 us 211.00 us 80 GETXATTR 0.01 1123.27 us 400.00 us 1955.00 us 11 XATTROP 0.01 264.98 us 89.00 us 4166.00 us 59 SETATTR 0.01 522.77 us 219.00 us 1988.00 us 31 UNLINK 0.01 3116.33 us 335.00 us 15814.00 us 6 READDIR 0.02 194.27 us 87.00 us 1585.00 us 146 OPEN 0.02 404.47 us 146.00 us 10053.00 us 78 SETXATTR 0.03 94.93 us 24.00 us 12262.00 us 407 ENTRYLK 0.04 380.95 us 82.00 us 30028.00 us 169 FSTAT 0.08 1825.35 us 1237.00 us 3241.00 us 62 CREATE 0.11 2054.41 us 1177.00 us 7853.00 us 78 MKDIR 0.11 137.97 us 4.00 us 4662.00 us 1188 OPENDIR 0.93 140.91 us 18.00 us 368752.00 us 9912 FINODELK 2.28 404.58 us 88.00 us 26154.00 us 8431 FXATTROP 20.16 9694.89 us 64.00 us 288694.00 us 3111 FSYNC 20.76 68.66 us 19.00 us 45520.00 us 452355 WRITE 26.67 59.16 us 16.00 us 26108.00 us 674404 INODELK 28.72 687.59 us 46.00 us 85079.00 us 62473 LOOKUP Duration: 19297631 seconds Data Read: 0 bytes Data Written: 4188179800 bytes Interval 5 Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 62 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR 0.01 71.83 us 46.00 us 116.00 us 30 WRITE 0.03 150.81 us 33.00 us 1289.00 us 37 FINODELK 0.08 489.54 us 63.00 us 4603.00 us 35 OPENDIR 0.20 874.41 us 240.00 us 2977.00 us 51 FXATTROP 19.05 66.56 us 17.00 us 14272.00 us 62733 INODELK 80.63 1162.50 us 99.00 us 85079.00 us 15201 LOOKUP Duration: 246 seconds Data Read: 0 bytes Data Written: 62 bytes Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick7 ----------------------------------------------- Cumulative Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 4459447660 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 4903888 FORGET 0.00 0.00 us 0.00 us 0.00 us 4906100 RELEASE 0.00 0.00 us 0.00 us 0.00 us 121313760 RELEASEDIR 0.00 157.00 us 157.00 us 157.00 us 1 FSTAT 0.00 334.00 us 334.00 us 334.00 us 1 LINK 0.00 492.00 us 492.00 us 492.00 us 1 RENAME 0.00 81.56 us 50.00 us 121.00 us 9 LK 0.00 600.00 us 433.00 us 767.00 us 2 XATTROP 0.00 284.69 us 26.00 us 1664.00 us 13 FLUSH 0.00 343.79 us 162.00 us 1811.00 us 14 SETATTR 0.00 1061.67 us 236.00 us 3527.00 us 6 UNLINK 0.01 1371.00 us 997.00 us 2107.00 us 6 MKNOD 0.01 109.50 us 33.00 us 166.00 us 103 GETXATTR 0.01 3619.60 us 1046.00 us 6783.00 us 5 CREATE 0.02 258.63 us 161.00 us 1183.00 us 78 SETXATTR 0.05 214.24 us 27.00 us 6443.00 us 280 ENTRYLK 0.09 1455.38 us 615.00 us 7156.00 us 78 MKDIR 0.11 24019.00 us 296.00 us 140960.00 us 6 READDIR 0.17 181.83 us 4.00 us 11536.00 us 1191 OPENDIR 0.20 1608.87 us 48.00 us 21829.00 us 157 FSYNC 1.05 138.38 us 73.00 us 1645.00 us 9762 OPEN 1.27 945.52 us 99.00 us 35935.00 us 1730 FXATTROP 2.19 71.12 us 25.00 us 12922.00 us 39768 INODELK 2.31 115.41 us 18.00 us 17045.00 us 25869 WRITE 10.45 6550.60 us 19.00 us 5168504.00 us 2062 FINODELK 82.07 703.56 us 22.00 us 235243.00 us 150709 LOOKUP Duration: 19297631 seconds Data Read: 0 bytes Data Written: 4459447660 bytes Interval 5 Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 3267 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 9 FORGET 0.00 0.00 us 0.00 us 0.00 us 5 RELEASE 0.00 0.00 us 0.00 us 0.00 us 93 RELEASEDIR 0.00 157.00 us 157.00 us 157.00 us 1 FSTAT 0.00 334.00 us 334.00 us 334.00 us 1 LINK 0.00 187.00 us 170.00 us 204.00 us 2 SETATTR 0.00 272.50 us 54.00 us 491.00 us 2 FLUSH 0.00 767.00 us 767.00 us 767.00 us 1 XATTROP 0.01 654.00 us 153.00 us 1645.00 us 3 OPEN 0.03 1135.40 us 236.00 us 3527.00 us 5 UNLINK 0.04 3975.50 us 1713.00 us 6238.00 us 2 CREATE 0.05 305.97 us 56.00 us 2057.00 us 38 OPENDIR 0.12 607.39 us 27.00 us 6443.00 us 44 ENTRYLK 0.59 1786.74 us 48.00 us 21829.00 us 74 FSYNC 0.73 661.93 us 33.00 us 12922.00 us 245 INODELK 3.02 1345.12 us 101.00 us 31614.00 us 500 FXATTROP 3.23 526.99 us 18.00 us 14746.00 us 1365 WRITE 22.47 6998.94 us 19.00 us 2596341.00 us 715 FINODELK 69.70 916.77 us 78.00 us 51010.00 us 16930 LOOKUP Duration: 246 seconds Data Read: 0 bytes Data Written: 3267 bytes
I think just worked out why NFS lookups are sometimes slow and sometimes fast as the hostname uses round robin DNS lookups, if I change to a specific host, 01-B, it's always quick, and if I change to the other brick host, 02-B, it's always slow. Maybe that will help to narrow this down? On Sun, Apr 21, 2019 at 10:24 PM Patrick Rennie <patrickmrennie at gmail.com> wrote:> Hi Strahil, > > Thank you for your reply and your suggestions. I'm not sure which logs > would be most relevant to be checking to diagnose this issue, we have the > brick logs, the cluster mount logs, the shd logs or something else? I have > posted a few that I have seen repeated a few times already. I will continue > to post anything further that I see. > I am working on migrating data to some new storage, so this will slowly > free up space, although this is a production cluster and new data is being > uploaded every day, sometimes faster than I can migrate it off. I have > several other similar clusters and none of them have the same problem, one > the others is actually at 98-99% right now (big problem, I know) but still > performs perfectly fine compared to this cluster, I am not sure low space > is the root cause here. > > I currently have 13 VMs accessing this cluster, I have checked each one > and all of them use one of the two options below to mount the cluster in > fstab > > HOSTNAME:/gvAA01 /mountpoint glusterfs > defaults,_netdev,rw,log-level=WARNING,direct-io-mode=disable,use-readdirp=no > 0 0 > HOSTNAME:/gvAA01 /mountpoint glusterfs > defaults,_netdev,rw,log-level=WARNING,direct-io-mode=disable > > I also have a few other VMs which use NFS to access the cluster, and these > machines appear to be significantly quicker, initially I get a similar > delay with NFS but if I cancel the first "ls" and try it again I get < 1 > sec lookups, this can take over 10 minutes by FUSE/gluster client, but the > same trick of cancelling and trying again doesn't work for FUSE/gluster. > Sometimes the NFS queries have no delay at all, so this is a bit strange to > me. > HOSTNAME:/gvAA01 /mountpoint/ nfs > defaults,_netdev,vers=3,async,noatime 0 0 > > Example: > user at VM:~$ time ls /cluster/folder > ^C > > real 9m49.383s > user 0m0.001s > sys 0m0.010s > > user at VM:~$ time ls /cluster/folder > <results> > > real 0m0.069s > user 0m0.001s > sys 0m0.007s > > --- > > I have checked the profiling as you suggested, I let it run for around a > minute, then cancelled it and saved the profile info. > > root at HOSTNAME:/var/log/glusterfs# gluster volume profile gvAA01 start > Starting volume profile on gvAA01 has been successful > root at HOSTNAME:/var/log/glusterfs# time ls /cluster/folder > ^C > > real 1m1.660s > user 0m0.000s > sys 0m0.002s > > root at HOSTNAME:/var/log/glusterfs# gluster volume profile gvAA01 info >> > ~/profile.txt > root at HOSTNAME:/var/log/glusterfs# gluster volume profile gvAA01 stop > > I will attach the results to this email as it's over 1000 lines. > Unfortunately, I'm not sure what I'm looking at but possibly somebody will > be able to help me make sense of it and let me know if it highlights any > specific issues. > > Happy to try any further suggestions. Thank you, > > -Patrick > > On Sun, Apr 21, 2019 at 7:55 PM Strahil <hunter86_bg at yahoo.com> wrote: > >> By the way, can you provide the 'volume info' and the mount options on >> all clients? >> Maybe , there is an option that uses a lot of resources due to some >> client's mount options. >> >> Best Regards, >> Strahil Nikolov >> On Apr 21, 2019 10:55, Patrick Rennie <patrickmrennie at gmail.com> wrote: >> >> Just another small update, I'm continuing to watch my brick logs and I >> just saw these errors come up in the recent events too. I am going to >> continue to post any errors I see in the hope of finding the right one to >> try and fix.. >> This is from the logs on brick1, seems to be occurring on both nodes on >> brick1, although at different times. I'm not sure what this means, can >> anyone shed any light? >> I guess I am looking for some kind of specific error which may indicate >> something is broken or stuck and locking up and causing the extreme latency >> I'm seeing in the cluster. >> >> [2019-04-21 07:25:55.064497] E [rpcsvc.c:1364:rpcsvc_submit_generic] >> 0-rpc-service: failed to submit message (XID: 0x7c700c, Program: GlusterFS >> 3.3, ProgVers: 330, Proc: 29) to rpc-transport (tcp.gvAA01-server) >> [2019-04-21 07:25:55.064612] E [server.c:195:server_submit_reply] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e58a) >> [0x7f3b3e93158a] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17d45) >> [0x7f3b3e4c5d45] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) >> [0x7f3b3e4b72cd] ) 0-: Reply submission failed >> [2019-04-21 07:25:55.064675] E [rpcsvc.c:1364:rpcsvc_submit_generic] >> 0-rpc-service: failed to submit message (XID: 0x7c70af, Program: GlusterFS >> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) >> [2019-04-21 07:25:55.064705] E [server.c:195:server_submit_reply] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) >> [0x7f3b3e9318fa] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) >> [0x7f3b3e4c5f35] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) >> [0x7f3b3e4b72cd] ) 0-: Reply submission failed >> [2019-04-21 07:25:55.064742] E [rpcsvc.c:1364:rpcsvc_submit_generic] >> 0-rpc-service: failed to submit message (XID: 0x7c723c, Program: GlusterFS >> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) >> [2019-04-21 07:25:55.064768] E [server.c:195:server_submit_reply] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) >> [0x7f3b3e9318fa] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) >> [0x7f3b3e4c5f35] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) >> [0x7f3b3e4b72cd] ) 0-: Reply submission failed >> [2019-04-21 07:25:55.064812] E [rpcsvc.c:1364:rpcsvc_submit_generic] >> 0-rpc-service: failed to submit message (XID: 0x7c72b4, Program: GlusterFS >> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) >> [2019-04-21 07:25:55.064837] E [server.c:195:server_submit_reply] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) >> [0x7f3b3e9318fa] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) >> [0x7f3b3e4c5f35] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) >> [0x7f3b3e4b72cd] ) 0-: Reply submission failed >> [2019-04-21 07:25:55.064880] E [rpcsvc.c:1364:rpcsvc_submit_generic] >> 0-rpc-service: failed to submit message (XID: 0x7c740b, Program: GlusterFS >> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) >> [2019-04-21 07:25:55.064905] E [server.c:195:server_submit_reply] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) >> [0x7f3b3e9318fa] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) >> [0x7f3b3e4c5f35] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) >> [0x7f3b3e4b72cd] ) 0-: Reply submission failed >> [2019-04-21 07:25:55.064939] E [rpcsvc.c:1364:rpcsvc_submit_generic] >> 0-rpc-service: failed to submit message (XID: 0x7c7441, Program: GlusterFS >> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) >> [2019-04-21 07:25:55.064962] E [server.c:195:server_submit_reply] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) >> [0x7f3b3e9318fa] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) >> [0x7f3b3e4c5f35] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) >> [0x7f3b3e4b72cd] ) 0-: Reply submission failed >> [2019-04-21 07:25:55.064996] E [rpcsvc.c:1364:rpcsvc_submit_generic] >> 0-rpc-service: failed to submit message (XID: 0x7c74d5, Program: GlusterFS >> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) >> [2019-04-21 07:25:55.065020] E [server.c:195:server_submit_reply] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) >> [0x7f3b3e9318fa] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) >> [0x7f3b3e4c5f35] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) >> [0x7f3b3e4b72cd] ) 0-: Reply submission failed >> [2019-04-21 07:25:55.065052] E [rpcsvc.c:1364:rpcsvc_submit_generic] >> 0-rpc-service: failed to submit message (XID: 0x7c7551, Program: GlusterFS >> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) >> [2019-04-21 07:25:55.065076] E [server.c:195:server_submit_reply] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) >> [0x7f3b3e9318fa] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) >> [0x7f3b3e4c5f35] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) >> [0x7f3b3e4b72cd] ) 0-: Reply submission failed >> [2019-04-21 07:25:55.065110] E [rpcsvc.c:1364:rpcsvc_submit_generic] >> 0-rpc-service: failed to submit message (XID: 0x7c76d1, Program: GlusterFS >> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server) >> [2019-04-21 07:25:55.065133] E [server.c:195:server_submit_reply] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa) >> [0x7f3b3e9318fa] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35) >> [0x7f3b3e4c5f35] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd) >> [0x7f3b3e4b72cd] ) 0-: Reply submission failed >> >> Thanks again, >> >> -Patrick >> >> On Sun, Apr 21, 2019 at 3:50 PM Patrick Rennie <patrickmrennie at gmail.com> >> wrote: >> >> Hi Darrell, >> >> Thanks again for your advice, I've left it for a while but unfortunately >> it's still just as slow and causing more problems for our operations now. I >> will need to try and take some steps to at least bring performance back to >> normal while continuing to investigate the issue longer term. I can >> definitely see one node with heavier CPU than the other, almost double, >> which I am OK with, but I think the heal process is going to take forever, >> trying to check the "gluster volume heal info" shows thousands and >> thousands of files which may need healing, I have no idea how many in total >> the command is still running after hours, so I am not sure what has gone so >> wrong to cause this. >> >> I've checked cluster.op-version and cluster.max-op-version and it looks >> like I'm on the latest version there. >> >> I have no idea how long the healing is going to take on this cluster, we >> have around 560TB of data on here, but I don't think I can wait that long >> to try and restore performance to normal. >> >> Can anyone think of anything else I can try in the meantime to work out >> what's causing the extreme latency? >> >> I've been going through cluster client the logs of some of our VMs and on >> some of our FTP servers I found this in the cluster mount log, but I am not >> seeing it on any of our other servers, just our FTP servers. >> >> [2019-04-21 07:16:19.925388] E [MSGID: 101046] >> [dht-common.c:1904:dht_revalidate_cbk] 0-gvAA01-dht: dict is null >> [2019-04-21 07:19:43.413834] W [MSGID: 114031] >> [client-rpc-fops.c:2203:client3_3_setattr_cbk] 0-gvAA01-client-19: remote >> operation failed [No such file or directory] >> [2019-04-21 07:19:43.414153] W [MSGID: 114031] >> [client-rpc-fops.c:2203:client3_3_setattr_cbk] 0-gvAA01-client-20: remote >> operation failed [No such file or directory] >> [2019-04-21 07:23:33.154717] E [MSGID: 101046] >> [dht-common.c:1904:dht_revalidate_cbk] 0-gvAA01-dht: dict is null >> [2019-04-21 07:33:24.943913] E [MSGID: 101046] >> [dht-common.c:1904:dht_revalidate_cbk] 0-gvAA01-dht: dict is null >> >> Any ideas what this could mean? I am basically just grasping at straws >> here. >> >> I am going to hold off on the version upgrade until I know there are no >> files which need healing, which could be a while, from some reading I've >> done there shouldn't be any issues with this as both are on v3.12.x >> >> I've free'd up a small amount of space, but I still need to work on this >> further. >> >> I've read of a command "find .glusterfs -type f -links -2 -exec rm {} \;" >> which could be run on each brick and it would potentially clean up any >> files which were deleted straight from the bricks, but not via the client, >> I have a feeling this could help me free up about 5-10TB per brick from >> what I've been told about the history of this cluster. Can anyone confirm >> if this is actually safe to run? >> >> At this stage, I'm open to any suggestions as to how to proceed, thanks >> again for any advice. >> >> Cheers, >> >> - Patrick >> >> On Sun, Apr 21, 2019 at 1:22 AM Darrell Budic <budic at onholyground.com> >> wrote: >> >> Patrick, >> >> Sounds like progress. Be aware that gluster is expected to max out the >> CPUs on at least one of your servers while healing. This is normal and >> won?t adversely affect overall performance (any more than having bricks in >> need of healing, at any rate) unless you?re overdoing it. shd threads <= 4 >> should not do that on your hardware. Other tunings may have also increased >> overall performance, so you may see higher CPU than previously anyway. I?d >> recommend upping those thread counts and letting it heal as fast as >> possible, especially if these are dedicated Gluster storage servers (Ie: >> not also running VMs, etc). You should see ?normal? CPU use one heals are >> completed. I see ~15-30% overall normally, 95-98% while healing (x my 20 >> cores). It?s also likely to be different between your servers, in a pure >> replica, one tends to max and one tends to be a little higher, in a >> distributed-replica, I?d expect more than one to run harder while healing. >> >> Keep the differences between doing an ls on a brick and doing an ls on a >> gluster mount in mind. When you do a ls on a gluster volume, it isn?t just >> doing a ls on one brick, it?s effectively doing it on ALL of your bricks, >> and they all have to return data before the ls succeeds. In a distributed >> volume, it?s figuring out where on each volume things live and getting the >> stat() from each to assemble the whole thing. And if things are in need of >> healing, it will take even longer to decide which version is current and >> use it (shd triggers a heal anytime it encounters this). Any of these >> things being slow slows down the overall response. >> >> At this point, I?d get some sleep too, and let your cluster heal while >> you do. I?d really want it fully healed before I did any updates anyway, so >> let it use CPU and get itself sorted out. Expect it to do a round of >> healing after you upgrade each machine too, this is normal so don?t let the >> CPU spike surprise you, It?s just catching up from the downtime incurred by >> the update and/or reboot if you did one. >> >> That reminds me, check your gluster cluster.op-version and >> cluster.max-op-version (gluster vol get all all | grep op-version). If >> op-version isn?t at the max-op-verison, set it to it so you?re taking >> advantage of the latest features available to your version. >> >> -Darrell >> >> On Apr 20, 2019, at 11:54 AM, Patrick Rennie <patrickmrennie at gmail.com> >> wrote: >> >> Hi Darrell, >> >> Thanks again for your advice, I've applied the acltype=posixacl on my >> zpools and I think that has reduced some of the noise from my brick logs. >> I also bumped up some of the thread counts you suggested but my CPU load >> skyrocketed, so I dropped it back down to something slightly lower, but >> still higher than it was before, and will see how that goes for a while. >> >> Although low space is a definite issue, if I run an ls anywhere on my >> bricks directly it's instant, <1 second, and still takes several minutes >> via gluster, so there is still a problem in my gluster configuration >> somewhere. We don't have any snapshots, but I am trying to work out if any >> data on there is safe to delete, or if there is any way I can safely find >> and delete data which has been removed directly from the bricks in the >> past. I also have lz4 compression already enabled on each zpool which does >> help a bit, we get between 1.05 and 1.08x compression on this data. >> I've tried to go through each client and checked it's cluster mount logs >> and also my brick logs and looking for errors, so far nothing is jumping >> out at me, but there are some warnings and errors here and there, I am >> trying to work out what they mean. >> >> It's already 1 am here and unfortunately, I'm still awake working on this >> issue, but I think that I will have to leave the version upgrades until >> tomorrow. >> >> Thanks again for your advice so far. If anyone has any ideas on where I >> can look for errors other than brick logs or the cluster mount logs to help >> resolve this issue, it would be much appreciated. >> >> Cheers, >> >> - Patrick >> >> On Sat, Apr 20, 2019 at 11:57 PM Darrell Budic <budic at onholyground.com> >> wrote: >> >> See inline: >> >> On Apr 20, 2019, at 10:09 AM, Patrick Rennie <patrickmrennie at gmail.com> >> wrote: >> >> Hi Darrell, >> >> Thanks for your reply, this issue seems to be getting worse over the last >> few days, really has me tearing my hair out. I will do as you have >> suggested and get started on upgrading from 3.12.14 to 3.12.15. >> I've checked the zfs properties and all bricks have "xattr=sa" set, but >> none of them has "acltype=posixacl" set, currently the acltype property >> shows "off", if I make these changes will it apply retroactively to the >> existing data? I'm unfamiliar with what this will change so I may need to >> look into that before I proceed. >> >> >> It is safe to apply that now, any new set/get calls will then use it if >> new posixacls exist, and use older if not. ZFS is good that way. It should >> clear up your posix_acl and posix errors over time. >> >> I understand performance is going to slow down as the bricks get full, I >> am currently trying to free space and migrate data to some newer storage, I >> have fresh several hundred TB storage I just setup recently but with these >> performance issues it's really slow. I also believe there is significant >> data which has been deleted directly from the bricks in the past, so if I >> can reclaim this space in a safe manner then I will have at least around >> 10-15% free space. >> >> >> Full ZFS volumes will have a much larger impact on performance than you?d >> think, I?d prioritize this. If you have been taking zfs snapshots, consider >> deleting them to get the overall volume free space back up. And just to be >> sure it?s been said, delete from within the mounted volumes, don?t delete >> directly from the bricks (gluster will just try and heal it later, >> compounding your issues). Does not apply to deleting other data from the >> ZFS volume if it?s not part of the brick directory, of course. >> >> These servers have dual 8 core Xeon (E5-2620v4) and 512GB of RAM so >> generally they have plenty of resources available, currently only using >> around 330/512GB of memory. >> >> I will look into what your suggested settings will change, and then will >> probably go ahead with your recommendations, for our specs as stated above, >> what would you suggest for performance.io-thread-count ? >> >> >> I run single 2630v4s on my servers, which have a smaller storage >> footprint than yours. I?d go with 32 for performance.io-thread-count. >> I?d try 4 for the shd thread settings on that gear. Your memory use sounds >> fine, so no worries there. >> >> Our workload is nothing too extreme, we have a few VMs which write backup >> data to this storage nightly for our clients, our VMs don't live on this >> cluster, but just write to it. >> >> >> If they are writing compressible data, you?ll get immediate benefit by >> setting compression=lz4 on your ZFS volumes. It won?t help any old data, of >> course, but it will compress new data going forward. This is another one >> that?s safe to enable on the fly. >> >> I've been going through all of the logs I can, below are some slightly >> sanitized errors I've come across, but I'm not sure what to make of them. >> The main error I am seeing is the first one below, across several of my >> bricks, but possibly only for specific folders on the cluster, I'm not 100% >> about that yet though. >> >> [2019-04-20 05:56:59.512649] E [MSGID: 113001] >> [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on >> /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not >> supported] >> [2019-04-20 05:59:06.084333] E [MSGID: 113001] >> [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on >> /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not >> supported] >> [2019-04-20 05:59:43.289030] E [MSGID: 113001] >> [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on >> /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not >> supported] >> [2019-04-20 05:59:50.582257] E [MSGID: 113001] >> [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on >> /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not >> supported] >> [2019-04-20 06:01:42.501701] E [MSGID: 113001] >> [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on >> /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not >> supported] >> [2019-04-20 06:01:51.665354] W [posix.c:4929:posix_getxattr] >> 0-gvAA01-posix: Extended attributes not supported (try remounting brick >> with 'user_xattr' flag) >> >> >> [2019-04-20 13:12:36.131856] E [MSGID: 113002] >> [posix-helpers.c:893:posix_gfid_set] 0-gvAA01-posix: gfid is null for >> /xxxxxxxxxxxxxxxxxxxx [Invalid argument] >> [2019-04-20 13:12:36.131959] E [MSGID: 113002] [posix.c:362:posix_lookup] >> 0-gvAA01-posix: buf->ia_gfid is null for >> /brick2/xxxxxxxxxxxxxxxxxxxx_62906_tmp [No data available] >> [2019-04-20 13:12:36.132016] E [MSGID: 115050] >> [server-rpc-fops.c:175:server_lookup_cbk] 0-gvAA01-server: 24274759: LOOKUP >> /xxxxxxxxxxxxxxxxxxxx (a7c9b4a0-b7ee-4d01-a79e-576013c8ac87/Cloud >> Backup_clone1.vbm_62906_tmp), client: >> 00-A-16217-2019/04/08-21:23:03:692424-gvAA01-client-4-0-3, error-xlator: >> gvAA01-posix [No data available] >> [2019-04-20 13:12:38.093719] E [MSGID: 115050] >> [server-rpc-fops.c:175:server_lookup_cbk] 0-gvAA01-server: 24276491: LOOKUP >> /xxxxxxxxxxxxxxxxxxxx (a7c9b4a0-b7ee-4d01-a79e-576013c8ac87/Cloud >> Backup_clone1.vbm_62906_tmp), client: >> 00-A-16217-2019/04/08-21:23:03:692424-gvAA01-client-4-0-3, error-xlator: >> gvAA01-posix [No data available] >> [2019-04-20 13:12:38.093660] E [MSGID: 113002] >> [posix-helpers.c:893:posix_gfid_set] 0-gvAA01-posix: gfid is null for >> /xxxxxxxxxxxxxxxxxxxx [Invalid argument] >> [2019-04-20 13:12:38.093696] E [MSGID: 113002] [posix.c:362:posix_lookup] >> 0-gvAA01-posix: buf->ia_gfid is null for /brick2/xxxxxxxxxxxxxxxxxxxx [No >> data available] >> >> >> posixacls should clear those up, as mentioned. >> >> >> [2019-04-20 14:25:59.654576] E [inodelk.c:404:__inode_unlock_lock] >> 0-gvAA01-locks: Matching lock not found for unlock 0-9223372036854775807, >> by 980fdbbd367f0000 on 0x7fc4f0161440 >> [2019-04-20 14:25:59.654668] E [MSGID: 115053] >> [server-rpc-fops.c:295:server_inodelk_cbk] 0-gvAA01-server: 6092928: >> INODELK /xxxxxxxxxxxxxxxxxxxx.cdr$ (25b14631-a179-4274-8243-6e272d4f2ad8), >> client: >> cb-per-worker18-53637-2019/04/19-14:25:37:927673-gvAA01-client-1-0-4, >> error-xlator: gvAA01-locks [Invalid argument] >> >> >> [2019-04-20 13:35:07.495495] E [rpcsvc.c:1364:rpcsvc_submit_generic] >> 0-rpc-service: failed to submit message (XID: 0x247c644, Program: GlusterFS >> 3.3, ProgVers: 330, Proc: 27) to rpc-transport (tcp.gvAA01-server) >> [2019-04-20 13:35:07.495619] E [server.c:195:server_submit_reply] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/debug/io-stats.so(+0x1696a) >> [0x7ff4ae6f796a] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/protocol/server.so(+0x2d6e8) >> [0x7ff4ae2a96e8] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/protocol/server.so(+0x928d) >> [0x7ff4ae28528d] ) 0-: Reply submission failed >> >> >> Fix the posix acls and see if these clear up over time as well, I?m >> unclear on what the overall effect of running without the posix acls will >> be to total gluster health. Your biggest problem sounds like you need to >> free up space on the volumes and get the overall volume health back up to >> par and see if that doesn?t resolve the symptoms you?re seeing. >> >> >> >> Thank you again for your assistance. It is greatly appreciated. >> >> - Patrick >> >> >> >> On Sat, Apr 20, 2019 at 10:50 PM Darrell Budic <budic at onholyground.com> >> wrote: >> >> Patrick, >> >> I would definitely upgrade your two nodes from 3.12.14 to 3.12.15. You >> also mention ZFS, and that error you show makes me think you need to check >> to be sure you have ?xattr=sa? and ?acltype=posixacl? set on your ZFS >> volumes. >> >> You also observed your bricks are crossing the 95% full line, ZFS >> performance will degrade significantly the closer you get to full. In my >> experience, this starts somewhere between 10% and 5% free space remaining, >> so you?re in that realm. >> >> How?s your free memory on the servers doing? Do you have your zfs arc >> cache limited to something less than all the RAM? It shares pretty well, >> but I?ve encountered situations where other things won?t try and take ram >> back properly if they think it?s in use, so ZFS never gets the opportunity >> to give it up. >> >> Since your volume is a disperse-replica, you might try tuning >> disperse.shd-max-threads, default is 1, I?d try it at 2, 4, or even more if >> the CPUs are beefy enough. And setting server.event-threads to 4 and >> client.event-threads to 8 has proven helpful in many cases. After you get >> upgraded to 3.12.15, enabling performance.stat-prefetch may help as well. I >> don?t know if it matters, but I?d also recommend resetting >> performance.least-prio-threads to the default of 1 (or try 2 or 4) and/or >> also setting performance.io-thread-count to 32 if those have beefy CPUs. >> >> Beyond those general ideas, more info about your hardware (CPU and RAM) >> and workload (VMs, direct storage for web servers or enders, etc) may net >> you some more ideas. Then you?re going to have to do more digging into >> brick logs looking for errors and/or warnings to see what?s going on. >> >> -Darrell >> >> >> On Apr 20, 2019, at 8:22 AM, Patrick Rennie <patrickmrennie at gmail.com> >> wrote: >> >> Hello Gluster Users, >> >> I am hoping someone can help me with resolving an ongoing issue I've been >> having, I'm new to mailing lists so forgive me if I have gotten anything >> wrong. We have noticed our performance deteriorating over the last few >> weeks, easily measured by trying to do an ls on one of our top-level >> folders, and timing it, which usually would take 2-5 seconds, and now takes >> up to 20 minutes, which obviously renders our cluster basically unusable. >> This has been intermittent in the past but is now almost constant and I am >> not sure how to work out the exact cause. We have noticed some errors in >> the brick logs, and have noticed that if we kill the right brick process, >> performance instantly returns back to normal, this is not always the same >> brick, but it indicates to me something in the brick processes or >> background tasks may be causing extreme latency. Due to this ability to fix >> it by killing the right brick process off, I think it's a specific file, or >> folder, or operation which may be hanging and causing the increased >> latency, but I am not sure how to work it out. One last thing to add is >> that our bricks are getting quite full (~95% full), we are trying to >> migrate data off to new storage but that is going slowly, not helped by >> this issue. I am currently trying to run a full heal as there appear to be >> many files needing healing, and I have all brick processes running so they >> have an opportunity to heal, but this means performance is very poor. It >> currently takes over 15-20 minutes to do an ls of one of our top-level >> folders, which just contains 60-80 other folders, this should take 2-5 >> seconds. This is all being checked by FUSE mount locally on the storage >> node itself, but it is the same for other clients and VMs accessing the >> cluster. Initially, it seemed our NFS mounts were not affected and operated >> at normal speed, but testing over the last day has shown that our NFS >> clients are also extremely slow, so it doesn't seem specific to FUSE as I >> first thought it might be. >> >> I am not sure how to proceed from here, I am fairly new to gluster having >> inherited this setup from my predecessor and trying to keep it going. I >> have included some info below to try and help with diagnosis, please let me >> know if any further info would be helpful. I would really appreciate any >> advice on what I could try to work out the cause. Thank you in advance for >> reading this, and any suggestions you might be able to offer. >> >> - Patrick >> >> This is an example of the main error I see in our brick logs, there have >> been others, I can post them when I see them again too: >> [2019-04-20 04:54:43.055680] E [MSGID: 113001] >> [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on >> /brick1/<filename> library: system.posix_acl_default [Operation not >> supported] >> [2019-04-20 05:01:29.476313] W [posix.c:4929:posix_getxattr] >> 0-gvAA01-posix: Extended attributes not supported (try remounting brick >> with 'user_xattr' flag) >> >> Our setup consists of 2 storage nodes and an arbiter node. I have noticed >> our nodes are on slightly different versions, I'm not sure if this could be >> an issue. We have 9 bricks on each node, made up of ZFS RAIDZ2 pools - >> total capacity is around 560TB. >> We have bonded 10gbps NICS on each node, and I have tested bandwidth with >> iperf and found that it's what would be expected from this config. >> Individual brick performance seems ok, I've tested several bricks using >> dd and can write a 10GB files at 1.7GB/s. >> >> # dd if=/dev/zero of=/brick1/test/test.file bs=1M count=10000 >> 10000+0 records in >> 10000+0 records out >> 10485760000 bytes (10 GB, 9.8 GiB) copied, 6.20303 s, 1.7 GB/s >> >> Node 1: >> # glusterfs --version >> glusterfs 3.12.15 >> >> Node 2: >> # glusterfs --version >> glusterfs 3.12.14 >> >> Arbiter: >> # glusterfs --version >> glusterfs 3.12.14 >> >> Here is our gluster volume status: >> >> # gluster volume status >> Status of volume: gvAA01 >> Gluster process TCP Port RDMA Port Online >> Pid >> >> ------------------------------------------------------------------------------ >> Brick 01-B:/brick1/gvAA01/brick 49152 0 Y 7219 >> Brick 02-B:/brick1/gvAA01/brick 49152 0 Y 21845 >> Brick 00-A:/arbiterAA01/gvAA01/bri >> ck1 49152 0 Y >> 6931 >> Brick 01-B:/brick2/gvAA01/brick 49153 0 Y 7239 >> Brick 02-B:/brick2/gvAA01/brick 49153 0 Y 9916 >> Brick 00-A:/arbiterAA01/gvAA01/bri >> ck2 49153 0 Y >> 6939 >> Brick 01-B:/brick3/gvAA01/brick 49154 0 Y 7235 >> Brick 02-B:/brick3/gvAA01/brick 49154 0 Y 21858 >> Brick 00-A:/arbiterAA01/gvAA01/bri >> ck3 49154 0 Y >> 6947 >> Brick 01-B:/brick4/gvAA01/brick 49155 0 Y 31840 >> Brick 02-B:/brick4/gvAA01/brick 49155 0 Y 9933 >> Brick 00-A:/arbiterAA01/gvAA01/bri >> ck4 49155 0 Y >> 6956 >> Brick 01-B:/brick5/gvAA01/brick 49156 0 Y 7233 >> Brick 02-B:/brick5/gvAA01/brick 49156 0 Y 9942 >> Brick 00-A:/arbiterAA01/gvAA01/bri >> ck5 49156 0 Y >> 6964 >> Brick 01-B:/brick6/gvAA01/brick 49157 0 Y 7234 >> Brick 02-B:/brick6/gvAA01/brick 49157 0 Y 9952 >> Brick 00-A:/arbiterAA01/gvAA01/bri >> ck6 49157 0 Y >> 6974 >> Brick 01-B:/brick7/gvAA01/brick 49158 0 Y 7248 >> Brick 02-B:/brick7/gvAA01/brick 49158 0 Y 9960 >> Brick 00-A:/arbiterAA01/gvAA01/bri >> ck7 49158 0 Y >> 6984 >> Brick 01-B:/brick8/gvAA01/brick 49159 0 Y 7253 >> Brick 02-B:/brick8/gvAA01/brick 49159 0 Y 9970 >> Brick 00-A:/arbiterAA01/gvAA01/bri >> ck8 49159 0 Y >> 6993 >> Brick 01-B:/brick9/gvAA01/brick 49160 0 Y 7245 >> Brick 02-B:/brick9/gvAA01/brick 49160 0 Y 9984 >> Brick 00-A:/arbiterAA01/gvAA01/bri >> ck9 49160 0 Y >> 7001 >> NFS Server on localhost 2049 0 Y >> 17276 >> Self-heal Daemon on localhost N/A N/A Y >> 25245 >> NFS Server on 02-B 2049 0 Y 9089 >> Self-heal Daemon on 02-B N/A N/A Y 17838 >> NFS Server on 00-a 2049 0 Y 15660 >> Self-heal Daemon on 00-a N/A N/A Y 16218 >> >> Task Status of Volume gvAA01 >> >> ------------------------------------------------------------------------------ >> There are no active volume tasks >> >> And gluster volume info: >> >> # gluster volume info >> >> Volume Name: gvAA01 >> Type: Distributed-Replicate >> Volume ID: ca4ece2c-13fe-414b-856c-2878196d6118 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 9 x (2 + 1) = 27 >> Transport-type: tcp >> Bricks: >> Brick1: 01-B:/brick1/gvAA01/brick >> Brick2: 02-B:/brick1/gvAA01/brick >> Brick3: 00-A:/arbiterAA01/gvAA01/brick1 (arbiter) >> Brick4: 01-B:/brick2/gvAA01/brick >> Brick5: 02-B:/brick2/gvAA01/brick >> Brick6: 00-A:/arbiterAA01/gvAA01/brick2 (arbiter) >> Brick7: 01-B:/brick3/gvAA01/brick >> Brick8: 02-B:/brick3/gvAA01/brick >> Brick9: 00-A:/arbiterAA01/gvAA01/brick3 (arbiter) >> Brick10: 01-B:/brick4/gvAA01/brick >> Brick11: 02-B:/brick4/gvAA01/brick >> Brick12: 00-A:/arbiterAA01/gvAA01/brick4 (arbiter) >> Brick13: 01-B:/brick5/gvAA01/brick >> Brick14: 02-B:/brick5/gvAA01/brick >> Brick15: 00-A:/arbiterAA01/gvAA01/brick5 (arbiter) >> Brick16: 01-B:/brick6/gvAA01/brick >> Brick17: 02-B:/brick6/gvAA01/brick >> Brick18: 00-A:/arbiterAA01/gvAA01/brick6 (arbiter) >> Brick19: 01-B:/brick7/gvAA01/brick >> Brick20: 02-B:/brick7/gvAA01/brick >> Brick21: 00-A:/arbiterAA01/gvAA01/brick7 (arbiter) >> Brick22: 01-B:/brick8/gvAA01/brick >> Brick23: 02-B:/brick8/gvAA01/brick >> Brick24: 00-A:/arbiterAA01/gvAA01/brick8 (arbiter) >> Brick25: 01-B:/brick9/gvAA01/brick >> Brick26: 02-B:/brick9/gvAA01/brick >> Brick27: 00-A:/arbiterAA01/gvAA01/brick9 (arbiter) >> Options Reconfigured: >> cluster.shd-max-threads: 4 >> performance.least-prio-threads: 16 >> cluster.readdir-optimize: on >> performance.quick-read: off >> performance.stat-prefetch: off >> cluster.data-self-heal: on >> cluster.lookup-unhashed: auto >> cluster.lookup-optimize: on >> cluster.favorite-child-policy: mtime >> server.allow-insecure: on >> transport.address-family: inet >> client.bind-insecure: on >> cluster.entry-self-heal: off >> cluster.metadata-self-heal: off >> performance.md-cache-timeout: 600 >> cluster.self-heal-daemon: enable >> performance.readdir-ahead: on >> diagnostics.brick-log-level: INFO >> nfs.disable: off >> >> Thank you for any assistance. >> >> - Patrick >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> >> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190421/1833d9b7/attachment.html>