Bekk, Klaus (IKP)
2015-Jan-12 08:03 UTC
[Gluster-users] Bricks become not available with quota on
Hi all, we are running 4 server with Ubuntu 14.04 as glusterfs server with 3.5.2-ubuntu1~trusty1. With the following configuration root at ikpsrv01:/var/log/glusterfs# gluster volume info Volume Name: gltestn Type: Distribute Volume ID: e50bd30d-dbc8-4279-ac9d-2e557328a643 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: ikpsrv01:/gltest2/gl Volume Name: glwww1 Type: Replicate Volume ID: 811e4c2a-4520-4e61-bb84-955afbdb9dff Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: ikpsrv03:/glwwwn/gl Brick2: ikpsrv01:/glwwwn2/gl Options Reconfigured: features.quota: on Volume Name: corsika Type: Distribute Volume ID: d6e07f3e-4b35-4736-9b71-afc3224f0f29 Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: ikpsrv03:/glcors3-1/gl Brick2: ikpsrv03:/glcors3-2/gl Brick3: ikpsrv02:/glcors2-1/gl Brick4: ikpsrv02:/glcors2-2/gl Volume Name: gldata Type: Distribute Volume ID: 82485d03-9ef4-4151-a2cc-881072e871a1 Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: ikpsrv02:/gldata2-1/gl Brick2: ikpsrv01:/gldata1-2/gl Options Reconfigured: features.quota: on Volume Name: gltest Type: Distribute Volume ID: 9e69afcc-d421-41c9-824b-7d9a31240072 Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: ikpsrv01:/gltest3/gl Brick2: ikpsrv01:/gltest4/gl Options Reconfigured: features.quota: off Volume Name: glusersold Type: Distribute Volume ID: 91a79e1a-4ff3-4191-b0ae-73357105774c Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: ikpsrv02:/glusers2-2/gl Volume Name: glusers Type: Distribute Volume ID: f48b1941-9c19-4311-bd9e-497660fbcc80 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: ikpsrv02:/glusers2-1/gl Options Reconfigured: diagnostics.client-log-level: CRITICAL diagnostics.brick-log-level: CRITICAL features.quota: off When starting quotas on any of the volumes the brick will be not available after some time. So we have to disable quota on this volume and stop and start this volume. The command Gluster volume quota gltestn enable gives the following output in quotad.log [2015-01-07 12:01:55.522229] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fb455bd9fbd] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7fb455ead182] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7fb4569a7265]))) 0-: received signum (15), shutting down [2015-01-07 12:01:56.534931] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2 (/usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l /var/log/glusterfs/quotad.log -S /var/run/fa869d5552bf340ec7506747e21b2841.socket --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off) [2015-01-07 12:01:56.537061] I [socket.c:3561:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled [2015-01-07 12:01:56.537134] I [socket.c:3576:socket_init] 0-socket.glusterfsd: using system polling thread [2015-01-07 12:01:56.537355] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled [2015-01-07 12:01:56.537383] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread [2015-01-07 12:01:56.541480] I [graph.c:254:gf_add_cmdline_options] 0-glwww1-replicate-0: adding option 'entry-self-heal' for volume 'glwww1-replicate-0' with value 'off' [2015-01-07 12:01:56.541516] I [graph.c:254:gf_add_cmdline_options] 0-glwww1-replicate-0: adding option 'metadata-self-heal' for volume 'glwww1-replicate-0' with value 'off' [2015-01-07 12:01:56.541536] I [graph.c:254:gf_add_cmdline_options] 0-glwww1-replicate-0: adding option 'data-self-heal' for volume 'glwww1-replicate-0' with value 'off' [2015-01-07 12:01:56.543122] I [socket.c:3561:socket_init] 0-socket.quotad: SSL support is NOT enabled [2015-01-07 12:01:56.543157] I [socket.c:3576:socket_init] 0-socket.quotad: using system polling thread [2015-01-07 12:01:56.543404] I [dht-shared.c:311:dht_init_regex] 0-glwww1: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ [2015-01-07 12:01:56.547124] I [socket.c:3561:socket_init] 0-glwww1-client-1: SSL support is NOT enabled [2015-01-07 12:01:56.547158] I [socket.c:3576:socket_init] 0-glwww1-client-1: using system polling thread [2015-01-07 12:01:56.547728] I [socket.c:3561:socket_init] 0-glwww1-client-0: SSL support is NOT enabled [2015-01-07 12:01:56.547754] I [socket.c:3576:socket_init] 0-glwww1-client-0: using system polling thread [2015-01-07 12:01:56.547820] I [dht-shared.c:311:dht_init_regex] 0-gltestn: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ [2015-01-07 12:01:56.549471] I [socket.c:3561:socket_init] 0-gltestn-client-0: SSL support is NOT enabled [2015-01-07 12:01:56.549501] I [socket.c:3576:socket_init] 0-gltestn-client-0: using system polling thread [2015-01-07 12:01:56.549565] I [dht-shared.c:311:dht_init_regex] 0-gldata: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ [2015-01-07 12:01:56.551155] I [socket.c:3561:socket_init] 0-gldata-client-1: SSL support is NOT enabled [2015-01-07 12:01:56.551184] I [socket.c:3576:socket_init] 0-gldata-client-1: using system polling thread [2015-01-07 12:01:56.551744] I [socket.c:3561:socket_init] 0-gldata-client-0: SSL support is NOT enabled [2015-01-07 12:01:56.551770] I [socket.c:3576:socket_init] 0-gldata-client-0: using system polling thread [2015-01-07 12:01:56.551790] W [graph.c:329:_log_if_unknown_option] 0-quotad: option 'rpc-auth.auth-glusterfs' is not recognized [2015-01-07 12:01:56.551819] W [graph.c:329:_log_if_unknown_option] 0-quotad: option 'rpc-auth.auth-unix' is not recognized [2015-01-07 12:01:56.551838] W [graph.c:329:_log_if_unknown_option] 0-quotad: option 'rpc-auth.auth-null' is not recognized [2015-01-07 12:01:56.551854] W [graph.c:329:_log_if_unknown_option] 0-quotad: option 'glwww1.volume-id' is not recognized [2015-01-07 12:01:56.551870] W [graph.c:329:_log_if_unknown_option] 0-quotad: option 'gltestn.volume-id' is not recognized [2015-01-07 12:01:56.551885] W [graph.c:329:_log_if_unknown_option] 0-quotad: option 'gldata.volume-id' is not recognized [2015-01-07 12:01:56.551922] I [client.c:2294:notify] 0-glwww1-client-0: parent translators are ready, attempting connect on transport [2015-01-07 12:01:56.552579] I [client.c:2294:notify] 0-glwww1-client-1: parent translators are ready, attempting connect on transport [2015-01-07 12:01:56.553179] I [client.c:2294:notify] 0-gltestn-client-0: parent translators are ready, attempting connect on transport [2015-01-07 12:01:56.553757] I [client.c:2294:notify] 0-gldata-client-0: parent translators are ready, attempting connect on transport [2015-01-07 12:01:56.554372] I [client.c:2294:notify] 0-gldata-client-1: parent translators are ready, attempting connect on transport Final graph: +------------------------------------------------------------------------------+ 1: volume glwww1-client-0 2: type protocol/client 3: option remote-host ikpsrv03 4: option remote-subvolume /glwwwn/gl 5: option transport-type socket 6: option username 7c5931b4-d41e-40f7-978e-4b5a40f49481 7: option password 6ab1661b-4752-42a4-80cc-ca0bd0167a9f 8: end-volume 9: 10: volume glwww1-client-1 11: type protocol/client 12: option remote-host ikpsrv01 13: option remote-subvolume /glwwwn2/gl 14: option transport-type socket 15: option username 7c5931b4-d41e-40f7-978e-4b5a40f49481 16: option password 6ab1661b-4752-42a4-80cc-ca0bd0167a9f 17: end-volume 18: 19: volume glwww1-replicate-0 20: type cluster/replicate 21: option data-self-heal off 22: option metadata-self-heal off 23: option entry-self-heal off 24: subvolumes glwww1-client-0 glwww1-client-1 25: end-volume 26: 27: volume glwww1 28: type cluster/distribute 29: subvolumes glwww1-replicate-0 30: end-volume 31: 32: volume gltestn-client-0 33: type protocol/client 34: option remote-host ikpsrv01 35: option remote-subvolume /gltest2/gl 36: option transport-type socket 37: option username 76c8bf85-a5c5-445d-8e04-e596b6eb173b 38: option password ceec81fd-5afd-4d6b-a413-00b2688f4c62 39: end-volume 40: 41: volume gltestn 42: type cluster/distribute 43: subvolumes gltestn-client-0 44: end-volume 45: 46: volume gldata-client-0 47: type protocol/client 48: option remote-host ikpsrv02 49: option remote-subvolume /gldata2-1/gl 50: option transport-type socket 51: option username e9dcdbec-1110-4ab2-b667-63625126b246 52: option password 05a9e37d-67a0-476b-88df-93d612a4e08e 53: end-volume 54: 55: volume gldata-client-1 56: type protocol/client 57: option remote-host ikpsrv01 58: option remote-subvolume /gldata1-2/gl 59: option transport-type socket 60: option username e9dcdbec-1110-4ab2-b667-63625126b246 61: option password 05a9e37d-67a0-476b-88df-93d612a4e08e 62: end-volume 63: 64: volume gldata 65: type cluster/distribute 66: subvolumes gldata-client-0 gldata-client-1 67: end-volume 68: 69: volume quotad 70: type features/quotad 71: option rpc-auth.auth-glusterfs on 72: option rpc-auth.auth-unix on 73: option rpc-auth.auth-null on 74: option transport.socket.listen-path /tmp/quotad.socket 75: option transport-type socket 76: option transport.address-family unix 77: option glwww1.volume-id glwww1 78: option gltestn.volume-id gltestn 79: option gldata.volume-id gldata 80: subvolumes glwww1 gltestn gldata 81: end-volume 82: +------------------------------------------------------------------------------+ [2015-01-07 12:01:56.555618] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gldata-client-0: changing port to 49160 (from 0) [2015-01-07 12:01:56.556529] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-glwww1-client-1: changing port to 49165 (from 0) [2015-01-07 12:01:56.556597] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gltestn-client-0: changing port to 49166 (from 0) [2015-01-07 12:01:56.556643] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gldata-client-1: changing port to 49178 (from 0) [2015-01-07 12:01:56.558382] W [client-callback.c:31:client_cbk_fetchspec] 0-glwww1-client-0: this function should not be called [2015-01-07 12:01:56.558484] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed [2015-01-07 12:01:56.558601] I [client-handshake.c:1677:select_server_supported_programs] 0-gldata-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-01-07 12:01:56.559594] I [client-handshake.c:1677:select_server_supported_programs] 0-gltestn-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-01-07 12:01:56.559686] I [client-handshake.c:1677:select_server_supported_programs] 0-glwww1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-01-07 12:01:56.559768] I [client-handshake.c:1677:select_server_supported_programs] 0-gldata-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-01-07 12:01:56.559852] I [client-handshake.c:1462:client_setvolume_cbk] 0-gldata-client-0: Connected to 141.52.67.86:49160, attached to remote volume '/gldata2-1/gl'. [2015-01-07 12:01:56.559873] I [client-handshake.c:1474:client_setvolume_cbk] 0-gldata-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-01-07 12:01:56.559996] I [client-handshake.c:1462:client_setvolume_cbk] 0-gltestn-client-0: Connected to 141.52.67.85:49166, attached to remote volume '/gltest2/gl'. [2015-01-07 12:01:56.560017] I [client-handshake.c:1474:client_setvolume_cbk] 0-gltestn-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-01-07 12:01:56.560089] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gldata-client-0: Server lk version = 1 [2015-01-07 12:01:56.560139] I [client-handshake.c:1462:client_setvolume_cbk] 0-glwww1-client-1: Connected to 141.52.67.85:49165, attached to remote volume '/glwwwn2/gl'. [2015-01-07 12:01:56.560158] I [client-handshake.c:1474:client_setvolume_cbk] 0-glwww1-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-01-07 12:01:56.560204] I [afr-common.c:4131:afr_notify] 0-glwww1-replicate-0: Subvolume 'glwww1-client-1' came back up; going online. [2015-01-07 12:01:56.560262] I [client-handshake.c:1462:client_setvolume_cbk] 0-gldata-client-1: Connected to 141.52.67.85:49178, attached to remote volume '/gldata1-2/gl'. [2015-01-07 12:01:56.560301] I [client-handshake.c:1474:client_setvolume_cbk] 0-gldata-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-01-07 12:01:56.560385] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gltestn-client-0: Server lk version = 1 [2015-01-07 12:01:56.560467] I [client-handshake.c:450:client_set_lk_version_cbk] 0-glwww1-client-1: Server lk version = 1 [2015-01-07 12:01:56.560546] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gldata-client-1: Server lk version = 1 [2015-01-07 12:01:57.255911] I [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2015-01-07 12:01:57.557002] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-glwww1-client-0: changing port to 49153 (from 0) [2015-01-07 12:01:57.557924] I [client-handshake.c:1677:select_server_supported_programs] 0-glwww1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-01-07 12:01:57.558360] I [client-handshake.c:1462:client_setvolume_cbk] 0-glwww1-client-0: Connected to 141.52.67.96:49153, attached to remote volume '/glwwwn/gl'. [2015-01-07 12:01:57.558405] I [client-handshake.c:1474:client_setvolume_cbk] 0-glwww1-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-01-07 12:01:57.558706] I [client-handshake.c:450:client_set_lk_version_cbk] 0-glwww1-client-0: Server lk version = 1 what means 2015-01-07 12:01:56.559594] I [client-handshake.c:1677:select_server_supported_programs] 0-gltestn-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) In the log output. Our glusterfs clients listed with "gluster status VOLNAME clients" all have 3.5.2 glusterfs version. Can someone help? kind regards Klaus Bekk ---------------------------------------------------------- Karlsruher Institut of Technologie (KIT) Institut f?r Kernphysik (IKP) Dr. Klaus Bekk Hermann-von-Helmholtz-Platz 1 Campus Nord Geb?ude 425 76344 Eggenstein-Leopoldshafen Telefon: +49 721 608-23382 Fax: +49 721 608-23321 E-Mail: Klaus.Bekk at kit.edu http://www.kit.edu/ KIT - University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association ---------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150112/e6b47ba8/attachment.html>