Atin Mukherjee
2017-Aug-18 08:08 UTC
[Gluster-users] Glusterd not working with systemd in redhat 7
On Fri, Aug 18, 2017 at 12:22 PM, Atin Mukherjee <amukherj at redhat.com> wrote:> You're hitting a race here. By the time glusterd tries to resolve the > address of one of the remote bricks of a particular volume, the n/w > interface is not up by that time. We have fixed this issue in mainline and > 3.12 branch through the following commit: >Correction. This patch is only available in master and I've backported it to 3.12 now, https://review.gluster.org/#/c/18061.> commit 1477fa442a733d7b1a5ea74884cac8f29fbe7e6a > Author: Gaurav Yadav <gyadav at redhat.com> > Date: Tue Jul 18 16:23:18 2017 +0530 > > glusterd : glusterd fails to start when peer's network interface is > down > > Problem: > glusterd fails to start on nodes where glusterd tries to come up even > before network is up. > > Fix: > On startup glusterd tries to resolve brick path which is based on > hostname/ip, but in the above scenario when network interface is not > up, glusterd is not able to resolve the brick path using ip_address or > hostname With this fix glusterd will use UUID to resolve brick path. > > Change-Id: Icfa7b2652417135530479d0aa4e2a82b0476f710 > BUG: 1472267 > Signed-off-by: Gaurav Yadav <gyadav at redhat.com> > Reviewed-on: https://review.gluster.org/17813 > Smoke: Gluster Build System <jenkins at build.gluster.org> > Reviewed-by: Prashanth Pai <ppai at redhat.com> > CentOS-regression: Gluster Build System <jenkins at build.gluster.org> > Reviewed-by: Atin Mukherjee <amukherj at redhat.com> > > > > Note : 3.12 release is planned by end of this month. > > ~Atin > > On Thu, Aug 17, 2017 at 2:45 PM, ismael mondiu <mondiu at hotmail.com> wrote: > >> Hi Team, >> >> I noticed that glusterd is never starting when i reboot my Redhat 7.1 >> server. >> >> Service is enabled but don't works. >> >> I tested with gluster 3.10.4 & gluster 3.10.5 and the problem still >> exists. >> >> >> When i started the service manually this works. >> >> I'va also tested on Redhat 6.6 server and gluster 3.10.4 and this works >> fine. >> >> The problem seems to be related to Redhat 7.1 >> >> >> This is ? known issue ? if yes , can you tell me what's is the workaround? >> >> >> Thanks >> >> >> Some logs here >> >> >> [root@~]# systemctl status glusterd >> ? glusterd.service - GlusterFS, a clustered file-system server >> Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; >> vendor preset: disabled) >> Active: failed (Result: exit-code) since Thu 2017-08-17 11:04:00 CEST; >> 2min 9s ago >> Process: 851 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid >> --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=1/FAILURE) >> >> Aug 17 11:03:59 dvihcasc0r systemd[1]: Starting GlusterFS, a clustered >> file-system server... >> Aug 17 11:04:00 dvihcasc0r systemd[1]: glusterd.service: control process >> exited, code=exited status=1 >> Aug 17 11:04:00 dvihcasc0r systemd[1]: Failed to start GlusterFS, a >> clustered file-system server. >> Aug 17 11:04:00 dvihcasc0r systemd[1]: Unit glusterd.service entered >> failed state. >> Aug 17 11:04:00 dvihcasc0r systemd[1]: glusterd.service failed. >> >> >> ************************************************************ >> **************************** >> >> /var/log/glusterfs/glusterd.log >> >> ************************************************************ >> ******************************** >> >> >> 2017-08-17 09:04:00.202529] I [MSGID: 106478] [glusterd.c:1449:init] >> 0-management: Maximum allowed open file descriptors set to 65536 >> [2017-08-17 09:04:00.202573] I [MSGID: 106479] [glusterd.c:1496:init] >> 0-management: Using /var/lib/glusterd as working directory >> [2017-08-17 09:04:00.365134] E [rpc-transport.c:283:rpc_transport_load] >> 0-rpc-transport: /usr/lib64/glusterfs/3.10.5/rpc-transport/rdma.so: >> cannot open shared object file: No such file or directory >> [2017-08-17 09:04:00.365161] W [rpc-transport.c:287:rpc_transport_load] >> 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not >> valid or not found on this machine >> [2017-08-17 09:04:00.365195] W [rpcsvc.c:1661:rpcsvc_create_listener] >> 0-rpc-service: cannot create listener, initing the transport failed >> [2017-08-17 09:04:00.365206] E [MSGID: 106243] [glusterd.c:1720:init] >> 0-management: creation of 1 listeners failed, continuing with succeeded >> transport >> [2017-08-17 09:04:00.464314] I [MSGID: 106228] >> [glusterd.c:500:glusterd_check_gsync_present] 0-glusterd: >> geo-replication module not installed in the system [No such file or >> directory] >> [2017-08-17 09:04:00.510412] I [MSGID: 106513] >> [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: >> retrieved op-version: 31004 >> [2017-08-17 09:04:00.711413] I [MSGID: 106194] >> [glusterd-store.c:3776:glusterd_store_retrieve_missed_snaps_list] >> 0-management: No missed snaps list. >> [2017-08-17 09:04:00.756731] E [MSGID: 106187] >> [glusterd-store.c:4559:glusterd_resolve_all_bricks] 0-glusterd: resolve >> brick failed in restore >> [2017-08-17 09:04:00.756787] E [MSGID: 101019] [xlator.c:503:xlator_init] >> 0-management: Initialization of volume 'management' failed, review your >> volfile again >> [2017-08-17 09:04:00.756802] E [MSGID: 101066] >> [graph.c:325:glusterfs_graph_init] 0-management: initializing translator >> failed >> [2017-08-17 09:04:00.756816] E [MSGID: 101176] >> [graph.c:681:glusterfs_graph_activate] 0-graph: init failed >> [2017-08-17 09:04:00.766584] W [glusterfsd.c:1332:cleanup_and_exit] >> (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xfd) [0x7f9bdef4cabd] >> -->/usr/sbin/glusterd(glusterfs_process_volfp+0x1b1) [0x7f9bdef4c961] >> -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x7f9bdef4be4b] ) 0-: >> received signum (1), shutting down >> >> ************************************************************ >> ****************************** >> >> [root@~]# uptime >> 11:13:55 up 10 min, 1 user, load average: 0.00, 0.02, 0.04 >> >> >> ************************************************************ >> ****************************** >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170818/4b03d8e3/attachment.html>
Raghavendra Talur
2017-Aug-18 08:14 UTC
[Gluster-users] Glusterd not working with systemd in redhat 7
On Fri, Aug 18, 2017 at 1:38 PM, Atin Mukherjee <amukherj at redhat.com> wrote:> > > On Fri, Aug 18, 2017 at 12:22 PM, Atin Mukherjee <amukherj at redhat.com> > wrote: >> >> You're hitting a race here. By the time glusterd tries to resolve the >> address of one of the remote bricks of a particular volume, the n/w >> interface is not up by that time. We have fixed this issue in mainline and >> 3.12 branch through the following commit: > > > Correction. This patch is only available in master and I've backported it to > 3.12 now, https://review.gluster.org/#/c/18061.Atin, User has 3.10.x, we will have to backport it to 3.10 also. Talur> >> >> commit 1477fa442a733d7b1a5ea74884cac8f29fbe7e6a >> Author: Gaurav Yadav <gyadav at redhat.com> >> Date: Tue Jul 18 16:23:18 2017 +0530 >> >> glusterd : glusterd fails to start when peer's network interface is >> down >> >> Problem: >> glusterd fails to start on nodes where glusterd tries to come up even >> before network is up. >> >> Fix: >> On startup glusterd tries to resolve brick path which is based on >> hostname/ip, but in the above scenario when network interface is not >> up, glusterd is not able to resolve the brick path using ip_address or >> hostname With this fix glusterd will use UUID to resolve brick path. >> >> Change-Id: Icfa7b2652417135530479d0aa4e2a82b0476f710 >> BUG: 1472267 >> Signed-off-by: Gaurav Yadav <gyadav at redhat.com> >> Reviewed-on: https://review.gluster.org/17813 >> Smoke: Gluster Build System <jenkins at build.gluster.org> >> Reviewed-by: Prashanth Pai <ppai at redhat.com> >> CentOS-regression: Gluster Build System <jenkins at build.gluster.org> >> Reviewed-by: Atin Mukherjee <amukherj at redhat.com> >> >> >> >> Note : 3.12 release is planned by end of this month. >> >> ~Atin >> >> On Thu, Aug 17, 2017 at 2:45 PM, ismael mondiu <mondiu at hotmail.com> wrote: >>> >>> Hi Team, >>> >>> I noticed that glusterd is never starting when i reboot my Redhat 7.1 >>> server. >>> >>> Service is enabled but don't works. >>> >>> I tested with gluster 3.10.4 & gluster 3.10.5 and the problem still >>> exists. >>> >>> >>> When i started the service manually this works. >>> >>> I'va also tested on Redhat 6.6 server and gluster 3.10.4 and this works >>> fine. >>> >>> The problem seems to be related to Redhat 7.1 >>> >>> >>> This is ? known issue ? if yes , can you tell me what's is the >>> workaround? >>> >>> >>> Thanks >>> >>> >>> Some logs here >>> >>> >>> [root@~]# systemctl status glusterd >>> ? glusterd.service - GlusterFS, a clustered file-system server >>> Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; >>> vendor preset: disabled) >>> Active: failed (Result: exit-code) since Thu 2017-08-17 11:04:00 CEST; >>> 2min 9s ago >>> Process: 851 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid >>> --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=1/FAILURE) >>> >>> Aug 17 11:03:59 dvihcasc0r systemd[1]: Starting GlusterFS, a clustered >>> file-system server... >>> Aug 17 11:04:00 dvihcasc0r systemd[1]: glusterd.service: control process >>> exited, code=exited status=1 >>> Aug 17 11:04:00 dvihcasc0r systemd[1]: Failed to start GlusterFS, a >>> clustered file-system server. >>> Aug 17 11:04:00 dvihcasc0r systemd[1]: Unit glusterd.service entered >>> failed state. >>> Aug 17 11:04:00 dvihcasc0r systemd[1]: glusterd.service failed. >>> >>> >>> >>> **************************************************************************************** >>> >>> /var/log/glusterfs/glusterd.log >>> >>> >>> ******************************************************************************************** >>> >>> >>> 2017-08-17 09:04:00.202529] I [MSGID: 106478] [glusterd.c:1449:init] >>> 0-management: Maximum allowed open file descriptors set to 65536 >>> [2017-08-17 09:04:00.202573] I [MSGID: 106479] [glusterd.c:1496:init] >>> 0-management: Using /var/lib/glusterd as working directory >>> [2017-08-17 09:04:00.365134] E [rpc-transport.c:283:rpc_transport_load] >>> 0-rpc-transport: /usr/lib64/glusterfs/3.10.5/rpc-transport/rdma.so: cannot >>> open shared object file: No such file or directory >>> [2017-08-17 09:04:00.365161] W [rpc-transport.c:287:rpc_transport_load] >>> 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not >>> valid or not found on this machine >>> [2017-08-17 09:04:00.365195] W [rpcsvc.c:1661:rpcsvc_create_listener] >>> 0-rpc-service: cannot create listener, initing the transport failed >>> [2017-08-17 09:04:00.365206] E [MSGID: 106243] [glusterd.c:1720:init] >>> 0-management: creation of 1 listeners failed, continuing with succeeded >>> transport >>> [2017-08-17 09:04:00.464314] I [MSGID: 106228] >>> [glusterd.c:500:glusterd_check_gsync_present] 0-glusterd: geo-replication >>> module not installed in the system [No such file or directory] >>> [2017-08-17 09:04:00.510412] I [MSGID: 106513] >>> [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved >>> op-version: 31004 >>> [2017-08-17 09:04:00.711413] I [MSGID: 106194] >>> [glusterd-store.c:3776:glusterd_store_retrieve_missed_snaps_list] >>> 0-management: No missed snaps list. >>> [2017-08-17 09:04:00.756731] E [MSGID: 106187] >>> [glusterd-store.c:4559:glusterd_resolve_all_bricks] 0-glusterd: resolve >>> brick failed in restore >>> [2017-08-17 09:04:00.756787] E [MSGID: 101019] [xlator.c:503:xlator_init] >>> 0-management: Initialization of volume 'management' failed, review your >>> volfile again >>> [2017-08-17 09:04:00.756802] E [MSGID: 101066] >>> [graph.c:325:glusterfs_graph_init] 0-management: initializing translator >>> failed >>> [2017-08-17 09:04:00.756816] E [MSGID: 101176] >>> [graph.c:681:glusterfs_graph_activate] 0-graph: init failed >>> [2017-08-17 09:04:00.766584] W [glusterfsd.c:1332:cleanup_and_exit] >>> (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xfd) [0x7f9bdef4cabd] >>> -->/usr/sbin/glusterd(glusterfs_process_volfp+0x1b1) [0x7f9bdef4c961] >>> -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x7f9bdef4be4b] ) 0-: received >>> signum (1), shutting down >>> >>> >>> ****************************************************************************************** >>> >>> [root@~]# uptime >>> 11:13:55 up 10 min, 1 user, load average: 0.00, 0.02, 0.04 >>> >>> >>> >>> ****************************************************************************************** >>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://lists.gluster.org/mailman/listinfo/gluster-users >> >> > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users
Atin Mukherjee
2017-Aug-18 08:21 UTC
[Gluster-users] Glusterd not working with systemd in redhat 7
On Fri, 18 Aug 2017 at 13:45, Raghavendra Talur <rtalur at redhat.com> wrote:> On Fri, Aug 18, 2017 at 1:38 PM, Atin Mukherjee <amukherj at redhat.com> > wrote: > > > > > > On Fri, Aug 18, 2017 at 12:22 PM, Atin Mukherjee <amukherj at redhat.com> > > wrote: > >> > >> You're hitting a race here. By the time glusterd tries to resolve the > >> address of one of the remote bricks of a particular volume, the n/w > >> interface is not up by that time. We have fixed this issue in mainline > and > >> 3.12 branch through the following commit: > > > > > > Correction. This patch is only available in master and I've backported > it to > > 3.12 now, https://review.gluster.org/#/c/18061. > > Atin, > > User has 3.10.x, we will have to backport it to 3.10 also.Can be done but we always encourage users to try out the latest bits :)> > Talur > > > >> > >> commit 1477fa442a733d7b1a5ea74884cac8f29fbe7e6a > >> Author: Gaurav Yadav <gyadav at redhat.com> > >> Date: Tue Jul 18 16:23:18 2017 +0530 > >> > >> glusterd : glusterd fails to start when peer's network interface is > >> down > >> > >> Problem: > >> glusterd fails to start on nodes where glusterd tries to come up > even > >> before network is up. > >> > >> Fix: > >> On startup glusterd tries to resolve brick path which is based on > >> hostname/ip, but in the above scenario when network interface is not > >> up, glusterd is not able to resolve the brick path using ip_address > or > >> hostname With this fix glusterd will use UUID to resolve brick path. > >> > >> Change-Id: Icfa7b2652417135530479d0aa4e2a82b0476f710 > >> BUG: 1472267 > >> Signed-off-by: Gaurav Yadav <gyadav at redhat.com> > >> Reviewed-on: https://review.gluster.org/17813 > >> Smoke: Gluster Build System <jenkins at build.gluster.org> > >> Reviewed-by: Prashanth Pai <ppai at redhat.com> > >> CentOS-regression: Gluster Build System <jenkins at build.gluster.org> > >> Reviewed-by: Atin Mukherjee <amukherj at redhat.com> > >> > >> > >> > >> Note : 3.12 release is planned by end of this month. > >> > >> ~Atin > >> > >> On Thu, Aug 17, 2017 at 2:45 PM, ismael mondiu <mondiu at hotmail.com> > wrote: > >>> > >>> Hi Team, > >>> > >>> I noticed that glusterd is never starting when i reboot my Redhat 7.1 > >>> server. > >>> > >>> Service is enabled but don't works. > >>> > >>> I tested with gluster 3.10.4 & gluster 3.10.5 and the problem still > >>> exists. > >>> > >>> > >>> When i started the service manually this works. > >>> > >>> I'va also tested on Redhat 6.6 server and gluster 3.10.4 and this works > >>> fine. > >>> > >>> The problem seems to be related to Redhat 7.1 > >>> > >>> > >>> This is ? known issue ? if yes , can you tell me what's is the > >>> workaround? > >>> > >>> > >>> Thanks > >>> > >>> > >>> Some logs here > >>> > >>> > >>> [root@~]# systemctl status glusterd > >>> ? glusterd.service - GlusterFS, a clustered file-system server > >>> Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; > >>> vendor preset: disabled) > >>> Active: failed (Result: exit-code) since Thu 2017-08-17 11:04:00 > CEST; > >>> 2min 9s ago > >>> Process: 851 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid > >>> --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, > status=1/FAILURE) > >>> > >>> Aug 17 11:03:59 dvihcasc0r systemd[1]: Starting GlusterFS, a clustered > >>> file-system server... > >>> Aug 17 11:04:00 dvihcasc0r systemd[1]: glusterd.service: control > process > >>> exited, code=exited status=1 > >>> Aug 17 11:04:00 dvihcasc0r systemd[1]: Failed to start GlusterFS, a > >>> clustered file-system server. > >>> Aug 17 11:04:00 dvihcasc0r systemd[1]: Unit glusterd.service entered > >>> failed state. > >>> Aug 17 11:04:00 dvihcasc0r systemd[1]: glusterd.service failed. > >>> > >>> > >>> > >>> > **************************************************************************************** > >>> > >>> /var/log/glusterfs/glusterd.log > >>> > >>> > >>> > ******************************************************************************************** > >>> > >>> > >>> 2017-08-17 09:04:00.202529] I [MSGID: 106478] [glusterd.c:1449:init] > >>> 0-management: Maximum allowed open file descriptors set to 65536 > >>> [2017-08-17 09:04:00.202573] I [MSGID: 106479] [glusterd.c:1496:init] > >>> 0-management: Using /var/lib/glusterd as working directory > >>> [2017-08-17 09:04:00.365134] E [rpc-transport.c:283:rpc_transport_load] > >>> 0-rpc-transport: /usr/lib64/glusterfs/3.10.5/rpc-transport/rdma.so: > cannot > >>> open shared object file: No such file or directory > >>> [2017-08-17 09:04:00.365161] W [rpc-transport.c:287:rpc_transport_load] > >>> 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not > >>> valid or not found on this machine > >>> [2017-08-17 09:04:00.365195] W [rpcsvc.c:1661:rpcsvc_create_listener] > >>> 0-rpc-service: cannot create listener, initing the transport failed > >>> [2017-08-17 09:04:00.365206] E [MSGID: 106243] [glusterd.c:1720:init] > >>> 0-management: creation of 1 listeners failed, continuing with succeeded > >>> transport > >>> [2017-08-17 09:04:00.464314] I [MSGID: 106228] > >>> [glusterd.c:500:glusterd_check_gsync_present] 0-glusterd: > geo-replication > >>> module not installed in the system [No such file or directory] > >>> [2017-08-17 09:04:00.510412] I [MSGID: 106513] > >>> [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: > retrieved > >>> op-version: 31004 > >>> [2017-08-17 09:04:00.711413] I [MSGID: 106194] > >>> [glusterd-store.c:3776:glusterd_store_retrieve_missed_snaps_list] > >>> 0-management: No missed snaps list. > >>> [2017-08-17 09:04:00.756731] E [MSGID: 106187] > >>> [glusterd-store.c:4559:glusterd_resolve_all_bricks] 0-glusterd: resolve > >>> brick failed in restore > >>> [2017-08-17 09:04:00.756787] E [MSGID: 101019] > [xlator.c:503:xlator_init] > >>> 0-management: Initialization of volume 'management' failed, review your > >>> volfile again > >>> [2017-08-17 09:04:00.756802] E [MSGID: 101066] > >>> [graph.c:325:glusterfs_graph_init] 0-management: initializing > translator > >>> failed > >>> [2017-08-17 09:04:00.756816] E [MSGID: 101176] > >>> [graph.c:681:glusterfs_graph_activate] 0-graph: init failed > >>> [2017-08-17 09:04:00.766584] W [glusterfsd.c:1332:cleanup_and_exit] > >>> (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xfd) [0x7f9bdef4cabd] > >>> -->/usr/sbin/glusterd(glusterfs_process_volfp+0x1b1) [0x7f9bdef4c961] > >>> -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x7f9bdef4be4b] ) 0-: > received > >>> signum (1), shutting down > >>> > >>> > >>> > ****************************************************************************************** > >>> > >>> [root@~]# uptime > >>> 11:13:55 up 10 min, 1 user, load average: 0.00, 0.02, 0.04 > >>> > >>> > >>> > >>> > ****************************************************************************************** > >>> > >>> > >>> > >>> _______________________________________________ > >>> Gluster-users mailing list > >>> Gluster-users at gluster.org > >>> http://lists.gluster.org/mailman/listinfo/gluster-users > >> > >> > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-- --Atin -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170818/da4b45ca/attachment.html>