Hi Hari, Sorry for not providing you more details from the start. Here below you will find all the relevant log entries and info. Regarding the quota.conf file I have found one for my volume but it is a binary file. Is it supposed to be binary or text? Regards, M. *** gluster volume info myvolume *** Volume Name: myvolume Type: Replicate Volume ID: e7a40a1b-45c9-4d3c-bb19-0c59b4eceec5 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: gfs1a.domain.local:/data/myvolume/brick Brick2: gfs1b.domain.local:/data/myvolume/brick Brick3: gfs1c.domain.local:/srv/glusterfs/myvolume/brick (arbiter) Options Reconfigured: server.event-threads: 4 client.event-threads: 4 performance.readdir-ahead: on nfs.disable: on features.quota: on features.inode-quota: on features.quota-deem-statfs: on transport.address-family: inet *** /var/log/glusterfs/glusterd.log *** [2018-02-13 08:16:09.929568] E [MSGID: 101042] [compat.c:569:gf_umount_lazy] 0-management: Lazy unmount of /var/run/gluster/myvolume_quota_list/ [2018-02-13 08:16:28.596527] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume myvolume [2018-02-13 08:16:28.601097] I [MSGID: 106419] [glusterd-utils.c:6110:glusterd_add_inode_size_to_dict] 0-management: the brick on data/myvolume (zfs) uses dynamic inode sizes *** /var/log/glusterfs/cmd_history.log *** [2018-02-13 08:16:28.605478] : volume status myvolume detail : SUCCESS *** /var/log/glusterfs/quota-mount-myvolume.log *** [2018-02-13 08:16:09.934117] I [MSGID: 100030] [glusterfsd.c:2503:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.10.7 (args: /usr/sbin/glusterfs --volfile-server localhost --volfile-id myvolume -l /var/log/glusterfs/quota-mount-myvolume.log -p /var/run/gluster/myvolume_quota_list.pid --client-pid -5 /var/run/gluster/myvolume_quota_list/) [2018-02-13 08:16:09.940432] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2018-02-13 08:16:09.940491] E [socket.c:2327:socket_connect_finish] 0-glusterfs: connection to ::1:24007 failed (Connection refused); disconnecting socket [2018-02-13 08:16:09.940519] I [glusterfsd-mgmt.c:2134:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: localhost [2018-02-13 08:16:13.943827] I [afr.c:94:fix_quorum_options] 0-myvolume-replicate-0: reindeer: incoming qtype = none [2018-02-13 08:16:13.943857] I [afr.c:116:fix_quorum_options] 0-myvolume-replicate-0: reindeer: quorum_count = 2147483647 [2018-02-13 08:16:13.945194] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2018-02-13 08:16:13.945237] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4 [2018-02-13 08:16:13.945247] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3 [2018-02-13 08:16:13.945941] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-myvolume-readdir-ahead: option 'parallel-readdir' is not recognized [2018-02-13 08:16:13.946342] I [MSGID: 114020] [client.c:2352:notify] 0-myvolume-client-0: parent translators are ready, attempting connect on transport [2018-02-13 08:16:13.946789] I [MSGID: 114020] [client.c:2352:notify] 0-myvolume-client-1: parent translators are ready, attempting connect on transport [2018-02-13 08:16:13.947151] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-myvolume-client-0: changing port to 49153 (from 0) [2018-02-13 08:16:13.947846] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-myvolume-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-02-13 08:16:13.948073] I [MSGID: 114020] [client.c:2352:notify] 0-myvolume-client-2: parent translators are ready, attempting connect on transport [2018-02-13 08:16:13.948579] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-myvolume-client-1: changing port to 49154 (from 0) [2018-02-13 08:16:13.948699] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvolume-client-0: Connected to myvolume-client-0, attached to remote volume '/data/myvolume/brick'. [2018-02-13 08:16:13.948747] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvolume-client-0: Server and Client lk-version numbers are not same, reopening the fds [2018-02-13 08:16:13.948842] I [MSGID: 108005] [afr-common.c:4813:afr_notify] 0-myvolume-replicate-0: Subvolume 'myvolume-client-0' came back up; going online. Final graph: +------------------------------------------------------------------------------+ 1: volume myvolume-client-0 2: type protocol/client 3: option opversion 31004 4: option clnt-lk-version 1 5: option volfile-checksum 0 6: option volfile-key myvolume 7: option client-version 3.10.7 8: option process-uuid gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 9: option fops-version 1298437 10: option ping-timeout 42 11: option remote-host gfs1a.domain.local 12: option remote-subvolume /data/myvolume/brick 13: option transport-type socket 14: option transport.address-family inet 15: option username bea3e634-e174-4bb3-a1d6-25b09d03b536 16: option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6 17: option event-threads 4 18: option transport.tcp-user-timeout 0 19: option transport.socket.keepalive-time 20 20: option transport.socket.keepalive-interval 2 21: option transport.socket.keepalive-count 9 [2018-02-13 08:16:13.949147] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvolume-client-0: Server lk version = 1 22: option send-gids true 23: end-volume 24: 25: volume myvolume-client-1 26: type protocol/client 27: option ping-timeout 42 28: option remote-host gfs1b.domain.local 29: option remote-subvolume /data/myvolume/brick 30: option transport-type socket 31: option transport.address-family inet 32: option username bea3e634-e174-4bb3-a1d6-25b09d03b536 33: option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6 34: option event-threads 4 35: option transport.tcp-user-timeout 0 36: option transport.socket.keepalive-time 20 37: option transport.socket.keepalive-interval 2 38: option transport.socket.keepalive-count 9 39: option send-gids true 40: end-volume 41: 42: volume myvolume-client-2 43: type protocol/client 44: option ping-timeout 42 45: option remote-host gfs1c.domain.local 46: option remote-subvolume /srv/glusterfs/myvolume/brick 47: option transport-type socket 48: option transport.address-family inet 49: option username bea3e634-e174-4bb3-a1d6-25b09d03b536 50: option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6 51: option event-threads 4 52: option transport.tcp-user-timeout 0 53: option transport.socket.keepalive-time 20 54: option transport.socket.keepalive-interval 2 55: option transport.socket.keepalive-count 9 56: option send-gids true 57: end-volume 58: 59: volume myvolume-replicate-0 60: type cluster/replicate 61: option afr-pending-xattr myvolume-client-0,myvolume-client-1,myvolume-client-2 62: option arbiter-count 1 63: option use-compound-fops off 64: subvolumes myvolume-client-0 myvolume-client-1 myvolume-client-2 65: end-volume [2018-02-13 08:16:13.949442] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-myvolume-client-2: changing port to 49153 (from 0) 66: 67: volume myvolume-dht 68: type cluster/distribute 69: option lock-migration off 70: subvolumes myvolume-replicate-0 71: end-volume 72: 73: volume myvolume-write-behind 74: type performance/write-behind 75: subvolumes myvolume-dht 76: end-volume 77: 78: volume myvolume-read-ahead 79: type performance/read-ahead 80: subvolumes myvolume-write-behind 81: end-volume 82: 83: volume myvolume-readdir-ahead 84: type performance/readdir-ahead 85: option parallel-readdir off 86: option rda-request-size 131072 87: option rda-cache-limit 10MB 88: subvolumes myvolume-read-ahead 89: end-volume 90: 91: volume myvolume-io-cache 92: type performance/io-cache 93: subvolumes myvolume-readdir-ahead 94: end-volume 95: 96: volume myvolume-quick-read 97: type performance/quick-read 98: subvolumes myvolume-io-cache 99: end-volume 100: 101: volume myvolume-open-behind 102: type performance/open-behind 103: subvolumes myvolume-quick-read 104: end-volume 105: 106: volume myvolume-md-cache 107: type performance/md-cache 108: subvolumes myvolume-open-behind 109: end-volume 110: 111: volume myvolume 112: type debug/io-stats 113: option log-level INFO 114: option latency-measurement off 115: option count-fop-hits off 116: subvolumes myvolume-md-cache 117: end-volume 118: 119: volume meta-autoload 120: type meta 121: subvolumes myvolume 122: end-volume 123: +------------------------------------------------------------------------------+ [2018-02-13 08:16:13.949813] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-myvolume-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-02-13 08:16:13.950686] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-myvolume-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-02-13 08:16:13.950698] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvolume-client-1: Connected to myvolume-client-1, attached to remote volume '/data/myvolume/brick'. [2018-02-13 08:16:13.950759] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvolume-client-1: Server and Client lk-version numbers are not same, reopening the fds [2018-02-13 08:16:13.951009] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvolume-client-1: Server lk version = 1 [2018-02-13 08:16:13.951320] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvolume-client-2: Connected to myvolume-client-2, attached to remote volume '/srv/glusterfs/myvolume/brick'. [2018-02-13 08:16:13.951348] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvolume-client-2: Server and Client lk-version numbers are not same, reopening the fds [2018-02-13 08:16:13.952626] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvolume-client-2: Server lk version = 1 [2018-02-13 08:16:13.952704] I [fuse-bridge.c:4138:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.23 [2018-02-13 08:16:13.952742] I [fuse-bridge.c:4823:fuse_graph_sync] 0-fuse: switched to graph 0 [2018-02-13 08:16:13.954337] I [MSGID: 108031] [afr-common.c:2357:afr_local_discovery_cbk] 0-myvolume-replicate-0: selecting local read_child myvolume-client-0 [2018-02-13 08:16:14.020681] I [fuse-bridge.c:5081:fuse_thread_proc] 0-fuse: unmounting /var/run/gluster/myvolume_quota_list/ [2018-02-13 08:16:14.020981] W [glusterfsd.c:1360:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064) [0x7f92aaeef064] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55d8ddb85f75] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d8ddb85d97] ) 0-: received signum (15), shutting down [2018-02-13 08:16:14.021017] I [fuse-bridge.c:5804:fini] 0-fuse: Unmounting '/var/run/gluster/myvolume_quota_list/'. *** /var/log/glusterfs/quotad.log *** [2018-02-13 08:16:14.023519] W [MSGID: 108027] [afr-common.c:2718:afr_discover_done] 0-myvolume-replicate-0: no read subvols for (null) [2018-02-13 08:16:14.064410] W [dict.c:599:dict_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) [0x7f63153e7f3d] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) [0x7f63153e8d72] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] [2018-02-13 08:16:14.074683] W [dict.c:599:dict_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) [0x7f63153e7f3d] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) [0x7f63153e8d72] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] [2018-02-13 08:16:14.082040] W [dict.c:599:dict_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) [0x7f63153e7f3d] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) [0x7f63153e8d72] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] [2018-02-13 08:16:14.086950] W [dict.c:599:dict_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) [0x7f63153e7f3d] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) [0x7f63153e8d72] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] [2018-02-13 08:16:14.087939] W [dict.c:599:dict_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) [0x7f63153e7f3d] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) [0x7f63153e8d72] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] [2018-02-13 08:16:14.089775] W [dict.c:599:dict_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) [0x7f63153e7f3d] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) [0x7f63153e8d72] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] [2018-02-13 08:16:14.092937] W [dict.c:599:dict_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) [0x7f63153e7f3d] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) [0x7f63153e8d72] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] The message "W [MSGID: 108027] [afr-common.c:2718:afr_discover_done] 0-myvolume-replicate-0: no read subvols for (null)" repeated 34 times between [2018-02-13 08:16:14.023519] and [2018-02-13 08:17:26.560943] *** /var/log/glusterfs/cli.log *** [2018-02-13 08:16:14.064404] E [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get quota limits for 3df709ee-641d-46a2-bd61-889583e3033c [2018-02-13 08:16:14.074693] E [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get quota limits for a27818fe-0248-40fe-bb23-d43d61010478 [2018-02-13 08:16:14.082067] E [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get quota limits for daf97388-bcec-4cc0-a8ef-5b93f05b30f6 [2018-02-13 08:16:14.086929] E [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get quota limits for 3c768b36-2625-4509-87ef-fe5214cb9b01 [2018-02-13 08:16:14.087905] E [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get quota limits for f8cf47d4-4f54-43c5-ab0d-75b45b4677a3 [2018-02-13 08:16:14.089788] E [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get quota limits for b4c81a39-2152-45c5-95d3-b796d88226fe [2018-02-13 08:16:14.092919] E [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get quota limits for 16ac4cde-a5d4-451f-adcc-422a542fea24 [2018-02-13 08:16:14.092980] I [input.c:31:cli_batch] 0-: Exiting with: 0 *** /var/log/glusterfs/bricks/data-myvolume-brick.log *** [2018-02-13 08:16:13.948065] I [addr.c:182:gf_auth] 0-/data/myvolume/brick: allowed = "*", received addr = "127.0.0.1" [2018-02-13 08:16:13.948105] I [login.c:76:gf_auth] 0-auth/login: allowed user names: bea3e634-e174-4bb3-a1d6-25b09d03b536 [2018-02-13 08:16:13.948125] I [MSGID: 115029] [server-handshake.c:695:server_setvolume] 0-myvolume-server: accepted client from gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 (version: 3.10.7) [2018-02-13 08:16:14.022257] I [MSGID: 115036] [server.c:559:server_rpc_notify] 0-myvolume-server: disconnecting connection from gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 [2018-02-13 08:16:14.022465] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-myvolume-server: Shutting down connection gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 -------- Original Message -------- On February 13, 2018 12:47 AM, Hari Gowtham <hgowtham at redhat.com> wrote:> Hi, > > Can you provide more information like, the volume configuration, quota.conf file and the log files. > > On Sat, Feb 10, 2018 at 1:05 AM, mabi <mabi at protonmail.ch> wrote: >> Hello, >> >> I am running GlusterFS 3.10.7 and just noticed by doing a "gluster volume quota <volname> list" that my quotas on that volume are broken. The command returns no output and no errors but by looking in /var/log/glusterfs.cli I found the following errors: >> >> [2018-02-09 19:31:24.242324] E [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get quota limits for 3df709ee-641d-46a2-bd61-889583e3033c >> [2018-02-09 19:31:24.249790] E [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get quota limits for a27818fe-0248-40fe-bb23-d43d61010478 >> [2018-02-09 19:31:24.252378] E [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get quota limits for daf97388-bcec-4cc0-a8ef-5b93f05b30f6 >> [2018-02-09 19:31:24.256775] E [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get quota limits for 3c768b36-2625-4509-87ef-fe5214cb9b01 >> [2018-02-09 19:31:24.257434] E [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get quota limits for f8cf47d4-4f54-43c5-ab0d-75b45b4677a3 >> [2018-02-09 19:31:24.259126] E [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get quota limits for b4c81a39-2152-45c5-95d3-b796d88226fe >> [2018-02-09 19:31:24.261664] E [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get quota limits for 16ac4cde-a5d4-451f-adcc-422a542fea24 >> [2018-02-09 19:31:24.261719] I [input.c:31:cli_batch] 0-: Exiting with: 0 >> >> How can I fix my quota on that volume again? I had around 30 quotas set on different directories of that volume. >> >> Thanks in advance. >> >> Regards, >> M. >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users > > -- > Regards, > Hari Gowtham.-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180213/d9dc2d89/attachment.html>
Hi, A part of the log won't be enough to debug the issue. Need the whole log messages till date. You can send it as attachments. Yes the quota.conf is a binary file. And I need the volume status output too. On Tue, Feb 13, 2018 at 1:56 PM, mabi <mabi at protonmail.ch> wrote:> Hi Hari, > Sorry for not providing you more details from the start. Here below you will > find all the relevant log entries and info. Regarding the quota.conf file I > have found one for my volume but it is a binary file. Is it supposed to be > binary or text? > Regards, > M. > > *** gluster volume info myvolume *** > > Volume Name: myvolume > Type: Replicate > Volume ID: e7a40a1b-45c9-4d3c-bb19-0c59b4eceec5 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x (2 + 1) = 3 > Transport-type: tcp > Bricks: > Brick1: gfs1a.domain.local:/data/myvolume/brick > Brick2: gfs1b.domain.local:/data/myvolume/brick > Brick3: gfs1c.domain.local:/srv/glusterfs/myvolume/brick (arbiter) > Options Reconfigured: > server.event-threads: 4 > client.event-threads: 4 > performance.readdir-ahead: on > nfs.disable: on > features.quota: on > features.inode-quota: on > features.quota-deem-statfs: on > transport.address-family: inet > > > *** /var/log/glusterfs/glusterd.log *** > > [2018-02-13 08:16:09.929568] E [MSGID: 101042] [compat.c:569:gf_umount_lazy] > 0-management: Lazy unmount of /var/run/gluster/myvolume_quota_list/ > [2018-02-13 08:16:28.596527] I [MSGID: 106499] > [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: > Received status volume req for volume myvolume > [2018-02-13 08:16:28.601097] I [MSGID: 106419] > [glusterd-utils.c:6110:glusterd_add_inode_size_to_dict] 0-management: the > brick on data/myvolume (zfs) uses dynamic inode sizes > > > *** /var/log/glusterfs/cmd_history.log *** > > [2018-02-13 08:16:28.605478] : volume status myvolume detail : SUCCESS > > > *** /var/log/glusterfs/quota-mount-myvolume.log *** > > [2018-02-13 08:16:09.934117] I [MSGID: 100030] [glusterfsd.c:2503:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.10.7 > (args: /usr/sbin/glusterfs --volfile-server localhost --volfile-id myvolume > -l /var/log/glusterfs/quota-mount-myvolume.log -p > /var/run/gluster/myvolume_quota_list.pid --client-pid -5 > /var/run/gluster/myvolume_quota_list/) > [2018-02-13 08:16:09.940432] I [MSGID: 101190] > [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 1 > [2018-02-13 08:16:09.940491] E [socket.c:2327:socket_connect_finish] > 0-glusterfs: connection to ::1:24007 failed (Connection refused); > disconnecting socket > [2018-02-13 08:16:09.940519] I [glusterfsd-mgmt.c:2134:mgmt_rpc_notify] > 0-glusterfsd-mgmt: disconnected from remote-host: localhost > [2018-02-13 08:16:13.943827] I [afr.c:94:fix_quorum_options] > 0-myvolume-replicate-0: reindeer: incoming qtype = none > [2018-02-13 08:16:13.943857] I [afr.c:116:fix_quorum_options] > 0-myvolume-replicate-0: reindeer: quorum_count = 2147483647 > [2018-02-13 08:16:13.945194] I [MSGID: 101190] > [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 2 > [2018-02-13 08:16:13.945237] I [MSGID: 101190] > [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 4 > [2018-02-13 08:16:13.945247] I [MSGID: 101190] > [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 3 > [2018-02-13 08:16:13.945941] W [MSGID: 101174] > [graph.c:361:_log_if_unknown_option] 0-myvolume-readdir-ahead: option > 'parallel-readdir' is not recognized > [2018-02-13 08:16:13.946342] I [MSGID: 114020] [client.c:2352:notify] > 0-myvolume-client-0: parent translators are ready, attempting connect on > transport > [2018-02-13 08:16:13.946789] I [MSGID: 114020] [client.c:2352:notify] > 0-myvolume-client-1: parent translators are ready, attempting connect on > transport > [2018-02-13 08:16:13.947151] I [rpc-clnt.c:2000:rpc_clnt_reconfig] > 0-myvolume-client-0: changing port to 49153 (from 0) > [2018-02-13 08:16:13.947846] I [MSGID: 114057] > [client-handshake.c:1451:select_server_supported_programs] > 0-myvolume-client-0: Using Program GlusterFS 3.3, Num (1298437), Version > (330) > [2018-02-13 08:16:13.948073] I [MSGID: 114020] [client.c:2352:notify] > 0-myvolume-client-2: parent translators are ready, attempting connect on > transport > [2018-02-13 08:16:13.948579] I [rpc-clnt.c:2000:rpc_clnt_reconfig] > 0-myvolume-client-1: changing port to 49154 (from 0) > [2018-02-13 08:16:13.948699] I [MSGID: 114046] > [client-handshake.c:1216:client_setvolume_cbk] 0-myvolume-client-0: > Connected to myvolume-client-0, attached to remote volume > '/data/myvolume/brick'. > [2018-02-13 08:16:13.948747] I [MSGID: 114047] > [client-handshake.c:1227:client_setvolume_cbk] 0-myvolume-client-0: Server > and Client lk-version numbers are not same, reopening the fds > [2018-02-13 08:16:13.948842] I [MSGID: 108005] > [afr-common.c:4813:afr_notify] 0-myvolume-replicate-0: Subvolume > 'myvolume-client-0' came back up; going online. > Final graph: > +------------------------------------------------------------------------------+ > 1: volume myvolume-client-0 > 2: type protocol/client > 3: option opversion 31004 > 4: option clnt-lk-version 1 > 5: option volfile-checksum 0 > 6: option volfile-key myvolume > 7: option client-version 3.10.7 > 8: option process-uuid > gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 > 9: option fops-version 1298437 > 10: option ping-timeout 42 > 11: option remote-host gfs1a.domain.local > 12: option remote-subvolume /data/myvolume/brick > 13: option transport-type socket > 14: option transport.address-family inet > 15: option username bea3e634-e174-4bb3-a1d6-25b09d03b536 > 16: option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6 > 17: option event-threads 4 > 18: option transport.tcp-user-timeout 0 > 19: option transport.socket.keepalive-time 20 > 20: option transport.socket.keepalive-interval 2 > 21: option transport.socket.keepalive-count 9 > [2018-02-13 08:16:13.949147] I [MSGID: 114035] > [client-handshake.c:202:client_set_lk_version_cbk] 0-myvolume-client-0: > Server lk version = 1 > 22: option send-gids true > 23: end-volume > 24: > 25: volume myvolume-client-1 > 26: type protocol/client > 27: option ping-timeout 42 > 28: option remote-host gfs1b.domain.local > 29: option remote-subvolume /data/myvolume/brick > 30: option transport-type socket > 31: option transport.address-family inet > 32: option username bea3e634-e174-4bb3-a1d6-25b09d03b536 > 33: option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6 > 34: option event-threads 4 > 35: option transport.tcp-user-timeout 0 > 36: option transport.socket.keepalive-time 20 > 37: option transport.socket.keepalive-interval 2 > 38: option transport.socket.keepalive-count 9 > 39: option send-gids true > 40: end-volume > 41: > 42: volume myvolume-client-2 > 43: type protocol/client > 44: option ping-timeout 42 > 45: option remote-host gfs1c.domain.local > 46: option remote-subvolume /srv/glusterfs/myvolume/brick > 47: option transport-type socket > 48: option transport.address-family inet > 49: option username bea3e634-e174-4bb3-a1d6-25b09d03b536 > 50: option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6 > 51: option event-threads 4 > 52: option transport.tcp-user-timeout 0 > 53: option transport.socket.keepalive-time 20 > 54: option transport.socket.keepalive-interval 2 > 55: option transport.socket.keepalive-count 9 > 56: option send-gids true > 57: end-volume > 58: > 59: volume myvolume-replicate-0 > 60: type cluster/replicate > 61: option afr-pending-xattr > myvolume-client-0,myvolume-client-1,myvolume-client-2 > 62: option arbiter-count 1 > 63: option use-compound-fops off > 64: subvolumes myvolume-client-0 myvolume-client-1 myvolume-client-2 > 65: end-volume > [2018-02-13 08:16:13.949442] I [rpc-clnt.c:2000:rpc_clnt_reconfig] > 0-myvolume-client-2: changing port to 49153 (from 0) > 66: > 67: volume myvolume-dht > 68: type cluster/distribute > 69: option lock-migration off > 70: subvolumes myvolume-replicate-0 > 71: end-volume > 72: > 73: volume myvolume-write-behind > 74: type performance/write-behind > 75: subvolumes myvolume-dht > 76: end-volume > 77: > 78: volume myvolume-read-ahead > 79: type performance/read-ahead > 80: subvolumes myvolume-write-behind > 81: end-volume > 82: > 83: volume myvolume-readdir-ahead > 84: type performance/readdir-ahead > 85: option parallel-readdir off > 86: option rda-request-size 131072 > 87: option rda-cache-limit 10MB > 88: subvolumes myvolume-read-ahead > 89: end-volume > 90: > 91: volume myvolume-io-cache > 92: type performance/io-cache > 93: subvolumes myvolume-readdir-ahead > 94: end-volume > 95: > 96: volume myvolume-quick-read > 97: type performance/quick-read > 98: subvolumes myvolume-io-cache > 99: end-volume > 100: > 101: volume myvolume-open-behind > 102: type performance/open-behind > 103: subvolumes myvolume-quick-read > 104: end-volume > 105: > 106: volume myvolume-md-cache > 107: type performance/md-cache > 108: subvolumes myvolume-open-behind > 109: end-volume > 110: > 111: volume myvolume > 112: type debug/io-stats > 113: option log-level INFO > 114: option latency-measurement off > 115: option count-fop-hits off > 116: subvolumes myvolume-md-cache > 117: end-volume > 118: > 119: volume meta-autoload > 120: type meta > 121: subvolumes myvolume > 122: end-volume > 123: > +------------------------------------------------------------------------------+ > [2018-02-13 08:16:13.949813] I [MSGID: 114057] > [client-handshake.c:1451:select_server_supported_programs] > 0-myvolume-client-1: Using Program GlusterFS 3.3, Num (1298437), Version > (330) > [2018-02-13 08:16:13.950686] I [MSGID: 114057] > [client-handshake.c:1451:select_server_supported_programs] > 0-myvolume-client-2: Using Program GlusterFS 3.3, Num (1298437), Version > (330) > [2018-02-13 08:16:13.950698] I [MSGID: 114046] > [client-handshake.c:1216:client_setvolume_cbk] 0-myvolume-client-1: > Connected to myvolume-client-1, attached to remote volume > '/data/myvolume/brick'. > [2018-02-13 08:16:13.950759] I [MSGID: 114047] > [client-handshake.c:1227:client_setvolume_cbk] 0-myvolume-client-1: Server > and Client lk-version numbers are not same, reopening the fds > [2018-02-13 08:16:13.951009] I [MSGID: 114035] > [client-handshake.c:202:client_set_lk_version_cbk] 0-myvolume-client-1: > Server lk version = 1 > [2018-02-13 08:16:13.951320] I [MSGID: 114046] > [client-handshake.c:1216:client_setvolume_cbk] 0-myvolume-client-2: > Connected to myvolume-client-2, attached to remote volume > '/srv/glusterfs/myvolume/brick'. > [2018-02-13 08:16:13.951348] I [MSGID: 114047] > [client-handshake.c:1227:client_setvolume_cbk] 0-myvolume-client-2: Server > and Client lk-version numbers are not same, reopening the fds > [2018-02-13 08:16:13.952626] I [MSGID: 114035] > [client-handshake.c:202:client_set_lk_version_cbk] 0-myvolume-client-2: > Server lk version = 1 > [2018-02-13 08:16:13.952704] I [fuse-bridge.c:4138:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel > 7.23 > [2018-02-13 08:16:13.952742] I [fuse-bridge.c:4823:fuse_graph_sync] 0-fuse: > switched to graph 0 > [2018-02-13 08:16:13.954337] I [MSGID: 108031] > [afr-common.c:2357:afr_local_discovery_cbk] 0-myvolume-replicate-0: > selecting local read_child myvolume-client-0 > [2018-02-13 08:16:14.020681] I [fuse-bridge.c:5081:fuse_thread_proc] 0-fuse: > unmounting /var/run/gluster/myvolume_quota_list/ > [2018-02-13 08:16:14.020981] W [glusterfsd.c:1360:cleanup_and_exit] > (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064) [0x7f92aaeef064] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55d8ddb85f75] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d8ddb85d97] ) 0-: > received signum (15), shutting down > [2018-02-13 08:16:14.021017] I [fuse-bridge.c:5804:fini] 0-fuse: Unmounting > '/var/run/gluster/myvolume_quota_list/'. > > > *** /var/log/glusterfs/quotad.log *** > > [2018-02-13 08:16:14.023519] W [MSGID: 108027] > [afr-common.c:2718:afr_discover_done] 0-myvolume-replicate-0: no read > subvols for (null) > [2018-02-13 08:16:14.064410] W [dict.c:599:dict_unref] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) > [0x7f63153e7f3d] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) > [0x7f63153e8d72] > -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) > [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] > [2018-02-13 08:16:14.074683] W [dict.c:599:dict_unref] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) > [0x7f63153e7f3d] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) > [0x7f63153e8d72] > -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) > [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] > [2018-02-13 08:16:14.082040] W [dict.c:599:dict_unref] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) > [0x7f63153e7f3d] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) > [0x7f63153e8d72] > -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) > [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] > [2018-02-13 08:16:14.086950] W [dict.c:599:dict_unref] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) > [0x7f63153e7f3d] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) > [0x7f63153e8d72] > -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) > [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] > [2018-02-13 08:16:14.087939] W [dict.c:599:dict_unref] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) > [0x7f63153e7f3d] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) > [0x7f63153e8d72] > -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) > [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] > [2018-02-13 08:16:14.089775] W [dict.c:599:dict_unref] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) > [0x7f63153e7f3d] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) > [0x7f63153e8d72] > -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) > [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] > [2018-02-13 08:16:14.092937] W [dict.c:599:dict_unref] > (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) > [0x7f63153e7f3d] > -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) > [0x7f63153e8d72] > -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) > [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] > The message "W [MSGID: 108027] [afr-common.c:2718:afr_discover_done] > 0-myvolume-replicate-0: no read subvols for (null)" repeated 34 times > between [2018-02-13 08:16:14.023519] and [2018-02-13 08:17:26.560943] > > > *** /var/log/glusterfs/cli.log *** > > [2018-02-13 08:16:14.064404] E > [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get > quota limits for 3df709ee-641d-46a2-bd61-889583e3033c > [2018-02-13 08:16:14.074693] E > [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get > quota limits for a27818fe-0248-40fe-bb23-d43d61010478 > [2018-02-13 08:16:14.082067] E > [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get > quota limits for daf97388-bcec-4cc0-a8ef-5b93f05b30f6 > [2018-02-13 08:16:14.086929] E > [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get > quota limits for 3c768b36-2625-4509-87ef-fe5214cb9b01 > [2018-02-13 08:16:14.087905] E > [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get > quota limits for f8cf47d4-4f54-43c5-ab0d-75b45b4677a3 > [2018-02-13 08:16:14.089788] E > [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get > quota limits for b4c81a39-2152-45c5-95d3-b796d88226fe > [2018-02-13 08:16:14.092919] E > [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get > quota limits for 16ac4cde-a5d4-451f-adcc-422a542fea24 > [2018-02-13 08:16:14.092980] I [input.c:31:cli_batch] 0-: Exiting with: 0 > > > *** /var/log/glusterfs/bricks/data-myvolume-brick.log *** > > [2018-02-13 08:16:13.948065] I [addr.c:182:gf_auth] 0-/data/myvolume/brick: > allowed = "*", received addr = "127.0.0.1" > [2018-02-13 08:16:13.948105] I [login.c:76:gf_auth] 0-auth/login: allowed > user names: bea3e634-e174-4bb3-a1d6-25b09d03b536 > [2018-02-13 08:16:13.948125] I [MSGID: 115029] > [server-handshake.c:695:server_setvolume] 0-myvolume-server: accepted client > from gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 (version: > 3.10.7) > [2018-02-13 08:16:14.022257] I [MSGID: 115036] > [server.c:559:server_rpc_notify] 0-myvolume-server: disconnecting connection > from gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 > [2018-02-13 08:16:14.022465] I [MSGID: 101055] > [client_t.c:436:gf_client_unref] 0-myvolume-server: Shutting down connection > gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 > > > > -------- Original Message -------- > On February 13, 2018 12:47 AM, Hari Gowtham <hgowtham at redhat.com> wrote: > > Hi, > > Can you provide more information like, the volume configuration, quota.conf > file and the log files. > > On Sat, Feb 10, 2018 at 1:05 AM, mabi <mabi at protonmail.ch> wrote: >> Hello, >> >> I am running GlusterFS 3.10.7 and just noticed by doing a "gluster volume >> quota <volname> list" that my quotas on that volume are broken. The command >> returns no output and no errors but by looking in /var/log/glusterfs.cli I >> found the following errors: >> >> [2018-02-09 19:31:24.242324] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for 3df709ee-641d-46a2-bd61-889583e3033c >> [2018-02-09 19:31:24.249790] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for a27818fe-0248-40fe-bb23-d43d61010478 >> [2018-02-09 19:31:24.252378] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for daf97388-bcec-4cc0-a8ef-5b93f05b30f6 >> [2018-02-09 19:31:24.256775] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for 3c768b36-2625-4509-87ef-fe5214cb9b01 >> [2018-02-09 19:31:24.257434] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for f8cf47d4-4f54-43c5-ab0d-75b45b4677a3 >> [2018-02-09 19:31:24.259126] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for b4c81a39-2152-45c5-95d3-b796d88226fe >> [2018-02-09 19:31:24.261664] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for 16ac4cde-a5d4-451f-adcc-422a542fea24 >> [2018-02-09 19:31:24.261719] I [input.c:31:cli_batch] 0-: Exiting with: 0 >> >> How can I fix my quota on that volume again? I had around 30 quotas set on >> different directories of that volume. >> >> Thanks in advance. >> >> Regards, >> M. >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users > > > > -- > Regards, > Hari Gowtham. > >-- Regards, Hari Gowtham.
Hi Hari, Sure no problem, I will send you in a minute another mail where you can download all the relevant log files including the quota.conf binary file. Let me know if you need anything else. In the mean time here below is the output of a volume status. Best regards, M. Status of volume: myvolume Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gfs1a.domain.local:/data/myvolume /brick 49153 0 Y 3214 Brick gfs1b.domain.local:/data/myvolume /brick 49154 0 Y 3256 Brick gfs1c.domain.local:/srv/glusterf s/myvolume/brick 49153 0 Y 515 Self-heal Daemon on localhost N/A N/A Y 3186 Quota Daemon on localhost N/A N/A Y 3195 Self-heal Daemon on gfs1b.domain.local N/A N/A Y 3217 Quota Daemon on gfs1b.domain.local N/A N/A Y 3229 Self-heal Daemon on gfs1c.domain.local N/A N/A Y 486 Quota Daemon on gfs1c.domain.local N/A N/A Y 495 Task Status of Volume myvolume ------------------------------------------------------------------------------ There are no active volume tasks ? -------- Original Message -------- On February 13, 2018 10:09 AM, Hari Gowtham <hgowtham at redhat.com> wrote:>Hi, > > A part of the log won't be enough to debug the issue. > Need the whole log messages till date. > You can send it as attachments. > > Yes the quota.conf is a binary file. > > And I need the volume status output too. > > On Tue, Feb 13, 2018 at 1:56 PM, mabi mabi at protonmail.ch wrote: >>Hi Hari, >> Sorry for not providing you more details from the start. Here below you will >> find all the relevant log entries and info. Regarding the quota.conf file I >> have found one for my volume but it is a binary file. Is it supposed to be >> binary or text? >> Regards, >> M. >>*** gluster volume info myvolume *** >>Volume Name: myvolume >> Type: Replicate >> Volume ID: e7a40a1b-45c9-4d3c-bb19-0c59b4eceec5 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x (2 + 1) = 3 >> Transport-type: tcp >> Bricks: >> Brick1: gfs1a.domain.local:/data/myvolume/brick >> Brick2: gfs1b.domain.local:/data/myvolume/brick >> Brick3: gfs1c.domain.local:/srv/glusterfs/myvolume/brick (arbiter) >> Options Reconfigured: >> server.event-threads: 4 >> client.event-threads: 4 >> performance.readdir-ahead: on >> nfs.disable: on >> features.quota: on >> features.inode-quota: on >> features.quota-deem-statfs: on >> transport.address-family: inet >>*** /var/log/glusterfs/glusterd.log *** >>[2018-02-13 08:16:09.929568] E [MSGID: 101042] [compat.c:569:gf_umount_lazy] >> 0-management: Lazy unmount of /var/run/gluster/myvolume_quota_list/ >> [2018-02-13 08:16:28.596527] I [MSGID: 106499] >> [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: >> Received status volume req for volume myvolume >> [2018-02-13 08:16:28.601097] I [MSGID: 106419] >> [glusterd-utils.c:6110:glusterd_add_inode_size_to_dict] 0-management: the >> brick on data/myvolume (zfs) uses dynamic inode sizes >>*** /var/log/glusterfs/cmd_history.log *** >>[2018-02-13 08:16:28.605478] : volume status myvolume detail : SUCCESS >>*** /var/log/glusterfs/quota-mount-myvolume.log *** >>[2018-02-13 08:16:09.934117] I [MSGID: 100030] [glusterfsd.c:2503:main] >> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.10.7 >> (args: /usr/sbin/glusterfs --volfile-server localhost --volfile-id myvolume >> -l /var/log/glusterfs/quota-mount-myvolume.log -p >> /var/run/gluster/myvolume_quota_list.pid --client-pid -5 >> /var/run/gluster/myvolume_quota_list/) >> [2018-02-13 08:16:09.940432] I [MSGID: 101190] >> [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with >> index 1 >> [2018-02-13 08:16:09.940491] E [socket.c:2327:socket_connect_finish] >> 0-glusterfs: connection to ::1:24007 failed (Connection refused); >> disconnecting socket >> [2018-02-13 08:16:09.940519] I [glusterfsd-mgmt.c:2134:mgmt_rpc_notify] >> 0-glusterfsd-mgmt: disconnected from remote-host: localhost >> [2018-02-13 08:16:13.943827] I [afr.c:94:fix_quorum_options] >> 0-myvolume-replicate-0: reindeer: incoming qtype = none >> [2018-02-13 08:16:13.943857] I [afr.c:116:fix_quorum_options] >> 0-myvolume-replicate-0: reindeer: quorum_count = 2147483647 >> [2018-02-13 08:16:13.945194] I [MSGID: 101190] >> [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with >> index 2 >> [2018-02-13 08:16:13.945237] I [MSGID: 101190] >> [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with >> index 4 >> [2018-02-13 08:16:13.945247] I [MSGID: 101190] >> [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with >> index 3 >> [2018-02-13 08:16:13.945941] W [MSGID: 101174] >> [graph.c:361:_log_if_unknown_option] 0-myvolume-readdir-ahead: option >> 'parallel-readdir' is not recognized >> [2018-02-13 08:16:13.946342] I [MSGID: 114020] [client.c:2352:notify] >> 0-myvolume-client-0: parent translators are ready, attempting connect on >> transport >> [2018-02-13 08:16:13.946789] I [MSGID: 114020] [client.c:2352:notify] >> 0-myvolume-client-1: parent translators are ready, attempting connect on >> transport >> [2018-02-13 08:16:13.947151] I [rpc-clnt.c:2000:rpc_clnt_reconfig] >> 0-myvolume-client-0: changing port to 49153 (from 0) >> [2018-02-13 08:16:13.947846] I [MSGID: 114057] >> [client-handshake.c:1451:select_server_supported_programs] >> 0-myvolume-client-0: Using Program GlusterFS 3.3, Num (1298437), Version >> (330) >> [2018-02-13 08:16:13.948073] I [MSGID: 114020] [client.c:2352:notify] >> 0-myvolume-client-2: parent translators are ready, attempting connect on >> transport >> [2018-02-13 08:16:13.948579] I [rpc-clnt.c:2000:rpc_clnt_reconfig] >> 0-myvolume-client-1: changing port to 49154 (from 0) >> [2018-02-13 08:16:13.948699] I [MSGID: 114046] >> [client-handshake.c:1216:client_setvolume_cbk] 0-myvolume-client-0: >> Connected to myvolume-client-0, attached to remote volume >> '/data/myvolume/brick'. >> [2018-02-13 08:16:13.948747] I [MSGID: 114047] >> [client-handshake.c:1227:client_setvolume_cbk] 0-myvolume-client-0: Server >> and Client lk-version numbers are not same, reopening the fds >> [2018-02-13 08:16:13.948842] I [MSGID: 108005] >> [afr-common.c:4813:afr_notify] 0-myvolume-replicate-0: Subvolume >> 'myvolume-client-0' came back up; going online. >> Final graph: >> +------------------------------------------------------------------------------+ >> 1: volume myvolume-client-0 >> 2: type protocol/client >> 3: option opversion 31004 >> 4: option clnt-lk-version 1 >> 5: option volfile-checksum 0 >> 6: option volfile-key myvolume >> 7: option client-version 3.10.7 >> 8: option process-uuid >> gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 >> 9: option fops-version 1298437 >> 10: option ping-timeout 42 >> 11: option remote-host gfs1a.domain.local >> 12: option remote-subvolume /data/myvolume/brick >> 13: option transport-type socket >> 14: option transport.address-family inet >> 15: option username bea3e634-e174-4bb3-a1d6-25b09d03b536 >> 16: option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6 >> 17: option event-threads 4 >> 18: option transport.tcp-user-timeout 0 >> 19: option transport.socket.keepalive-time 20 >> 20: option transport.socket.keepalive-interval 2 >> 21: option transport.socket.keepalive-count 9 >> [2018-02-13 08:16:13.949147] I [MSGID: 114035] >> [client-handshake.c:202:client_set_lk_version_cbk] 0-myvolume-client-0: >> Server lk version = 1 >> 22: option send-gids true >> 23: end-volume >> 24: >> 25: volume myvolume-client-1 >> 26: type protocol/client >> 27: option ping-timeout 42 >> 28: option remote-host gfs1b.domain.local >> 29: option remote-subvolume /data/myvolume/brick >> 30: option transport-type socket >> 31: option transport.address-family inet >> 32: option username bea3e634-e174-4bb3-a1d6-25b09d03b536 >> 33: option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6 >> 34: option event-threads 4 >> 35: option transport.tcp-user-timeout 0 >> 36: option transport.socket.keepalive-time 20 >> 37: option transport.socket.keepalive-interval 2 >> 38: option transport.socket.keepalive-count 9 >> 39: option send-gids true >> 40: end-volume >> 41: >> 42: volume myvolume-client-2 >> 43: type protocol/client >> 44: option ping-timeout 42 >> 45: option remote-host gfs1c.domain.local >> 46: option remote-subvolume /srv/glusterfs/myvolume/brick >> 47: option transport-type socket >> 48: option transport.address-family inet >> 49: option username bea3e634-e174-4bb3-a1d6-25b09d03b536 >> 50: option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6 >> 51: option event-threads 4 >> 52: option transport.tcp-user-timeout 0 >> 53: option transport.socket.keepalive-time 20 >> 54: option transport.socket.keepalive-interval 2 >> 55: option transport.socket.keepalive-count 9 >> 56: option send-gids true >> 57: end-volume >> 58: >> 59: volume myvolume-replicate-0 >> 60: type cluster/replicate >> 61: option afr-pending-xattr >> myvolume-client-0,myvolume-client-1,myvolume-client-2 >> 62: option arbiter-count 1 >> 63: option use-compound-fops off >> 64: subvolumes myvolume-client-0 myvolume-client-1 myvolume-client-2 >> 65: end-volume >> [2018-02-13 08:16:13.949442] I [rpc-clnt.c:2000:rpc_clnt_reconfig] >> 0-myvolume-client-2: changing port to 49153 (from 0) >> 66: >> 67: volume myvolume-dht >> 68: type cluster/distribute >> 69: option lock-migration off >> 70: subvolumes myvolume-replicate-0 >> 71: end-volume >> 72: >> 73: volume myvolume-write-behind >> 74: type performance/write-behind >> 75: subvolumes myvolume-dht >> 76: end-volume >> 77: >> 78: volume myvolume-read-ahead >> 79: type performance/read-ahead >> 80: subvolumes myvolume-write-behind >> 81: end-volume >> 82: >> 83: volume myvolume-readdir-ahead >> 84: type performance/readdir-ahead >> 85: option parallel-readdir off >> 86: option rda-request-size 131072 >> 87: option rda-cache-limit 10MB >> 88: subvolumes myvolume-read-ahead >> 89: end-volume >> 90: >> 91: volume myvolume-io-cache >> 92: type performance/io-cache >> 93: subvolumes myvolume-readdir-ahead >> 94: end-volume >> 95: >> 96: volume myvolume-quick-read >> 97: type performance/quick-read >> 98: subvolumes myvolume-io-cache >> 99: end-volume >> 100: >> 101: volume myvolume-open-behind >> 102: type performance/open-behind >> 103: subvolumes myvolume-quick-read >> 104: end-volume >> 105: >> 106: volume myvolume-md-cache >> 107: type performance/md-cache >> 108: subvolumes myvolume-open-behind >> 109: end-volume >> 110: >> 111: volume myvolume >> 112: type debug/io-stats >> 113: option log-level INFO >> 114: option latency-measurement off >> 115: option count-fop-hits off >> 116: subvolumes myvolume-md-cache >> 117: end-volume >> 118: >> 119: volume meta-autoload >> 120: type meta >> 121: subvolumes myvolume >> 122: end-volume >> 123: >> +------------------------------------------------------------------------------+ >> [2018-02-13 08:16:13.949813] I [MSGID: 114057] >> [client-handshake.c:1451:select_server_supported_programs] >> 0-myvolume-client-1: Using Program GlusterFS 3.3, Num (1298437), Version >> (330) >> [2018-02-13 08:16:13.950686] I [MSGID: 114057] >> [client-handshake.c:1451:select_server_supported_programs] >> 0-myvolume-client-2: Using Program GlusterFS 3.3, Num (1298437), Version >> (330) >> [2018-02-13 08:16:13.950698] I [MSGID: 114046] >> [client-handshake.c:1216:client_setvolume_cbk] 0-myvolume-client-1: >> Connected to myvolume-client-1, attached to remote volume >> '/data/myvolume/brick'. >> [2018-02-13 08:16:13.950759] I [MSGID: 114047] >> [client-handshake.c:1227:client_setvolume_cbk] 0-myvolume-client-1: Server >> and Client lk-version numbers are not same, reopening the fds >> [2018-02-13 08:16:13.951009] I [MSGID: 114035] >> [client-handshake.c:202:client_set_lk_version_cbk] 0-myvolume-client-1: >> Server lk version = 1 >> [2018-02-13 08:16:13.951320] I [MSGID: 114046] >> [client-handshake.c:1216:client_setvolume_cbk] 0-myvolume-client-2: >> Connected to myvolume-client-2, attached to remote volume >> '/srv/glusterfs/myvolume/brick'. >> [2018-02-13 08:16:13.951348] I [MSGID: 114047] >> [client-handshake.c:1227:client_setvolume_cbk] 0-myvolume-client-2: Server >> and Client lk-version numbers are not same, reopening the fds >> [2018-02-13 08:16:13.952626] I [MSGID: 114035] >> [client-handshake.c:202:client_set_lk_version_cbk] 0-myvolume-client-2: >> Server lk version = 1 >> [2018-02-13 08:16:13.952704] I [fuse-bridge.c:4138:fuse_init] >> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel >> 7.23 >> [2018-02-13 08:16:13.952742] I [fuse-bridge.c:4823:fuse_graph_sync] 0-fuse: >> switched to graph 0 >> [2018-02-13 08:16:13.954337] I [MSGID: 108031] >> [afr-common.c:2357:afr_local_discovery_cbk] 0-myvolume-replicate-0: >> selecting local read_child myvolume-client-0 >> [2018-02-13 08:16:14.020681] I [fuse-bridge.c:5081:fuse_thread_proc] 0-fuse: >> unmounting /var/run/gluster/myvolume_quota_list/ >> [2018-02-13 08:16:14.020981] W [glusterfsd.c:1360:cleanup_and_exit] >> (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064) [0x7f92aaeef064] >> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55d8ddb85f75] >> -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d8ddb85d97] ) 0-: >> received signum (15), shutting down >> [2018-02-13 08:16:14.021017] I [fuse-bridge.c:5804:fini] 0-fuse: Unmounting >> '/var/run/gluster/myvolume_quota_list/'. >>*** /var/log/glusterfs/quotad.log *** >>[2018-02-13 08:16:14.023519] W [MSGID: 108027] >> [afr-common.c:2718:afr_discover_done] 0-myvolume-replicate-0: no read >> subvols for (null) >> [2018-02-13 08:16:14.064410] W [dict.c:599:dict_unref] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) >> [0x7f63153e7f3d] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) >> [0x7f63153e8d72] >> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) >> [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] >> [2018-02-13 08:16:14.074683] W [dict.c:599:dict_unref] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) >> [0x7f63153e7f3d] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) >> [0x7f63153e8d72] >> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) >> [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] >> [2018-02-13 08:16:14.082040] W [dict.c:599:dict_unref] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) >> [0x7f63153e7f3d] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) >> [0x7f63153e8d72] >> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) >> [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] >> [2018-02-13 08:16:14.086950] W [dict.c:599:dict_unref] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) >> [0x7f63153e7f3d] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) >> [0x7f63153e8d72] >> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) >> [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] >> [2018-02-13 08:16:14.087939] W [dict.c:599:dict_unref] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) >> [0x7f63153e7f3d] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) >> [0x7f63153e8d72] >> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) >> [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] >> [2018-02-13 08:16:14.089775] W [dict.c:599:dict_unref] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) >> [0x7f63153e7f3d] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) >> [0x7f63153e8d72] >> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) >> [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] >> [2018-02-13 08:16:14.092937] W [dict.c:599:dict_unref] >> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x1f3d) >> [0x7f63153e7f3d] >> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.7/xlator/features/quotad.so(+0x2d72) >> [0x7f63153e8d72] >> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) >> [0x7f631b9bd870] ) 0-dict: dict is NULL [Invalid argument] >> The message "W [MSGID: 108027] [afr-common.c:2718:afr_discover_done] >> 0-myvolume-replicate-0: no read subvols for (null)" repeated 34 times >> between [2018-02-13 08:16:14.023519] and [2018-02-13 08:17:26.560943] >>*** /var/log/glusterfs/cli.log *** >>[2018-02-13 08:16:14.064404] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for 3df709ee-641d-46a2-bd61-889583e3033c >> [2018-02-13 08:16:14.074693] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for a27818fe-0248-40fe-bb23-d43d61010478 >> [2018-02-13 08:16:14.082067] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for daf97388-bcec-4cc0-a8ef-5b93f05b30f6 >> [2018-02-13 08:16:14.086929] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for 3c768b36-2625-4509-87ef-fe5214cb9b01 >> [2018-02-13 08:16:14.087905] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for f8cf47d4-4f54-43c5-ab0d-75b45b4677a3 >> [2018-02-13 08:16:14.089788] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for b4c81a39-2152-45c5-95d3-b796d88226fe >> [2018-02-13 08:16:14.092919] E >> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >> quota limits for 16ac4cde-a5d4-451f-adcc-422a542fea24 >> [2018-02-13 08:16:14.092980] I [input.c:31:cli_batch] 0-: Exiting with: 0 >>*** /var/log/glusterfs/bricks/data-myvolume-brick.log *** >>[2018-02-13 08:16:13.948065] I [addr.c:182:gf_auth] 0-/data/myvolume/brick: >> allowed = "*", received addr = "127.0.0.1" >> [2018-02-13 08:16:13.948105] I [login.c:76:gf_auth] 0-auth/login: allowed >> user names: bea3e634-e174-4bb3-a1d6-25b09d03b536 >> [2018-02-13 08:16:13.948125] I [MSGID: 115029] >> [server-handshake.c:695:server_setvolume] 0-myvolume-server: accepted client >> from gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 (version: >> 3.10.7) >> [2018-02-13 08:16:14.022257] I [MSGID: 115036] >> [server.c:559:server_rpc_notify] 0-myvolume-server: disconnecting connection >> from gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 >> [2018-02-13 08:16:14.022465] I [MSGID: 101055] >> [client_t.c:436:gf_client_unref] 0-myvolume-server: Shutting down connection >> gfs1a-14348-2018/02/13-08:16:09:933625-myvolume-client-0-0-0 >>-------- Original Message -------- >> On February 13, 2018 12:47 AM, Hari Gowtham hgowtham at redhat.com wrote: >>Hi, >>Can you provide more information like, the volume configuration, quota.conf >> file and the log files. >>On Sat, Feb 10, 2018 at 1:05 AM, mabi mabi at protonmail.ch wrote: >>>Hello, >>>I am running GlusterFS 3.10.7 and just noticed by doing a "gluster volume >>> quota <volname> list" that my quotas on that volume are broken. The command >>> returns no output and no errors but by looking in /var/log/glusterfs.cli I >>> found the following errors: >>>[2018-02-09 19:31:24.242324] E >>> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >>> quota limits for 3df709ee-641d-46a2-bd61-889583e3033c >>> [2018-02-09 19:31:24.249790] E >>> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >>> quota limits for a27818fe-0248-40fe-bb23-d43d61010478 >>> [2018-02-09 19:31:24.252378] E >>> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >>> quota limits for daf97388-bcec-4cc0-a8ef-5b93f05b30f6 >>> [2018-02-09 19:31:24.256775] E >>> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >>> quota limits for 3c768b36-2625-4509-87ef-fe5214cb9b01 >>> [2018-02-09 19:31:24.257434] E >>> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >>> quota limits for f8cf47d4-4f54-43c5-ab0d-75b45b4677a3 >>> [2018-02-09 19:31:24.259126] E >>> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >>> quota limits for b4c81a39-2152-45c5-95d3-b796d88226fe >>> [2018-02-09 19:31:24.261664] E >>> [cli-cmd-volume.c:1674:cli_cmd_quota_handle_list_all] 0-cli: Failed to get >>> quota limits for 16ac4cde-a5d4-451f-adcc-422a542fea24 >>> [2018-02-09 19:31:24.261719] I [input.c:31:cli_batch] 0-: Exiting with: 0 >>>How can I fix my quota on that volume again? I had around 30 quotas set on >>> different directories of that volume. >>>Thanks in advance. >>>Regards, >>> M. >>> >>>Gluster-users mailing list >>>Gluster-users at gluster.org >>>http://lists.gluster.org/mailman/listinfo/gluster-users >>>-- >> Regards, >> Hari Gowtham. >> > > > > >Regards, > Hari Gowtham. >