Soumya Koduri
2015-Sep-22 17:26 UTC
[Gluster-users] Fwd: nfs-ganesha HA with arbiter volume
On 09/22/2015 05:06 PM, Tiemen Ruiten wrote:> That's correct and my original question was actually if a two node + > arbiter setup is possible. The documentation provided by Soumya only > mentions two servers in the example ganesha-ha.sh script. Perhaps that > could be updated as well then, to not give the wrong impression. >It does work with 2-node as well. In the script, there is already a check to verify if the number of servers < 3, it automatically disables quorum. Quorum cannot be enabled for a 2-node setup for obvious reasons. If one node fails, other node just takes over the IP. Thanks, Soumya> I could try to change the script to disable quorum, but wouldn't that > defeat the purpose? What will happen in case one node goes down > unexpectedly? > > On 22 September 2015 at 12:47, Kaleb Keithley <kkeithle at redhat.com > <mailto:kkeithle at redhat.com>> wrote: > > > Hi, > > IIRC, the setup is two nodes gluster+ganesha nodes plus the arbiter > node for gluster quorum. > > Have I remembered that correctly? > > The Ganesha HA in 3.7 requires a minimum of three servers running > ganesha and pacemaker. Two might work if you change the > ganesha-ha.sh to not enable pacemaker quorum, but I haven't tried > that myself. I'll try and find time in the next couple of days to > update the documentation or write a blog post. > > > > ----- Original Message ---- > > > > > > > > On 21/09/15 21:21, Tiemen Ruiten wrote: > > > Whoops, replied off-list. > > > > > > Additionally I noticed that the generated corosync config is not > > > valid, as there is no interface section: > > > > > > /etc/corosync/corosync.conf > > > > > > totem { > > > version: 2 > > > secauth: off > > > cluster_name: rd-ganesha-ha > > > transport: udpu > > > } > > > > > > nodelist { > > > node { > > > ring0_addr: cobalt > > > nodeid: 1 > > > } > > > node { > > > ring0_addr: iron > > > nodeid: 2 > > > } > > > } > > > > > > quorum { > > > provider: corosync_votequorum > > > two_node: 1 > > > } > > > > > > logging { > > > to_syslog: yes > > > } > > > > > > > > > > > > > May be Kaleb can help you out. > > > > > > ---------- Forwarded message ---------- > > > From: *Tiemen Ruiten* <t.ruiten at rdmedia.com <mailto:t.ruiten at rdmedia.com> > <mailto:t.ruiten at rdmedia.com <mailto:t.ruiten at rdmedia.com>>> > > > Date: 21 September 2015 at 17:16 > > > Subject: Re: [Gluster-users] nfs-ganesha HA with arbiter volume > > > To: Jiffin Tony Thottan <jthottan at redhat.com > <mailto:jthottan at redhat.com> <mailto:jthottan at redhat.com > <mailto:jthottan at redhat.com>>> > > > > > > > > > Could you point me to the latest documentation? I've been > struggling > > > to find something up-to-date. I believe I have all the > prerequisites: > > > > > > - shared storage volume exists and is mounted > > > - all nodes in hosts files > > > - Gluster-NFS disabled > > > - corosync, pacemaker and nfs-ganesha rpm's installed > > > > > > Anything I missed? > > > > > > Everything has been installed by RPM so is in the default > locations: > > > /usr/libexec/ganesha/ganesha-ha.sh > > > /etc/ganesha/ganesha.conf (empty) > > > /etc/ganesha/ganesha-ha.conf > > > > > > > Looks fine for me. > > > > > After I started the pcsd service manually, nfs-ganesha could be > > > enabled successfully, but there was no virtual IP present on the > > > interfaces and looking at the system log, I noticed corosync > failed to > > > start: > > > > > > - on the host where I issued the gluster nfs-ganesha enable > command: > > > > > > Sep 21 17:07:18 iron systemd: Starting NFS-Ganesha file server... > > > Sep 21 17:07:19 iron systemd: Started NFS-Ganesha file server. > > > Sep 21 17:07:19 iron rpc.statd[2409]: Received SM_UNMON_ALL request > > > from iron.int.rdmedia.com <http://iron.int.rdmedia.com> > <http://iron.int.rdmedia.com> while not > > > monitoring any hosts > > > Sep 21 17:07:20 iron systemd: Starting Corosync Cluster Engine... > > > Sep 21 17:07:20 iron corosync[3426]: [MAIN ] Corosync Cluster > Engine > > > ('2.3.4'): started and ready to provide service. > > > Sep 21 17:07:20 iron corosync[3426]: [MAIN ] Corosync built-in > > > features: dbus systemd xmlconf snmp pie relro bindnow > > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] Initializing > transport > > > (UDP/IP Unicast). > > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] Initializing > > > transmit/receive security (NSS) crypto: none hash: none > > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] The network interface > > > [10.100.30.38] is now up. > > > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine > loaded: > > > corosync configuration map access [0] > > > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name: cmap > > > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine > loaded: > > > corosync configuration service [1] > > > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name: cfg > > > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine > loaded: > > > corosync cluster closed process group service v1.01 [2] > > > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name: cpg > > > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine > loaded: > > > corosync profile loading service [4] > > > Sep 21 17:07:20 iron corosync[3427]: [QUORUM] Using quorum provider > > > corosync_votequorum > > > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ] Waiting for all > cluster > > > members. Current votes: 1 expected_votes: 2 > > > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine > loaded: > > > corosync vote quorum service v1.0 [5] > > > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name: > votequorum > > > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine > loaded: > > > corosync cluster quorum service v0.1 [3] > > > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name: quorum > > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] adding new UDPU > member > > > {10.100.30.38} > > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] adding new UDPU > member > > > {10.100.30.37} > > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] A new membership > > > (10.100.30.38:104 <http://10.100.30.38:104> > <http://10.100.30.38:104>) was formed. Members joined: 1 > > > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ] Waiting for all cluster > > > members. Current votes: 1 expected_votes: 2 > > > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ] Waiting for all cluster > > > members. Current votes: 1 expected_votes: 2 > > > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ] Waiting for all cluster > > > members. Current votes: 1 expected_votes: 2 > > > Sep 21 17:07:20 iron corosync[3427]: [QUORUM] Members[1]: 1 > > > Sep 21 17:07:20 iron corosync[3427]: [MAIN ] Completed service > > > synchronization, ready to provide service. > > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] A new membership > > > (10.100.30.37:108 <http://10.100.30.37:108> > <http://10.100.30.37:108>) was formed. Members joined: 1 > > > Sep 21 17:08:21 iron corosync: Starting Corosync Cluster Engine > > > (corosync): [FAILED] > > > Sep 21 17:08:21 iron systemd: corosync.service: control process > > > exited, code=exited status=1 > > > Sep 21 17:08:21 iron systemd: Failed to start Corosync Cluster > Engine. > > > Sep 21 17:08:21 iron systemd: Unit corosync.service entered > failed state. > > > > > > > > > - on the other host: > > > > > > Sep 21 17:07:19 cobalt systemd: Starting Preprocess NFS > configuration... > > > Sep 21 17:07:19 cobalt systemd: Starting RPC Port Mapper. > > > Sep 21 17:07:19 cobalt systemd: Reached target RPC Port Mapper. > > > Sep 21 17:07:19 cobalt systemd: Starting Host and Network Name > Lookups. > > > Sep 21 17:07:19 cobalt systemd: Reached target Host and Network > Name > > > Lookups. > > > Sep 21 17:07:19 cobalt systemd: Starting RPC bind service... > > > Sep 21 17:07:19 cobalt systemd: Started Preprocess NFS > configuration. > > > Sep 21 17:07:19 cobalt systemd: Started RPC bind service. > > > Sep 21 17:07:19 cobalt systemd: Starting NFS status monitor for > > > NFSv2/3 locking.... > > > Sep 21 17:07:19 cobalt rpc.statd[2662]: Version 1.3.0 starting > > > Sep 21 17:07:19 cobalt rpc.statd[2662]: Flags: TI-RPC > > > Sep 21 17:07:19 cobalt systemd: Started NFS status monitor for > NFSv2/3 > > > locking.. > > > Sep 21 17:07:19 cobalt systemd: Starting NFS-Ganesha file server... > > > Sep 21 17:07:19 cobalt systemd: Started NFS-Ganesha file server. > > > Sep 21 17:07:19 cobalt kernel: warning: `ganesha.nfsd' uses 32-bit > > > capabilities (legacy support in use) > > > Sep 21 17:07:19 cobalt logger: setting up rd-ganesha-ha > > > Sep 21 17:07:19 cobalt rpc.statd[2662]: Received SM_UNMON_ALL > request > > > from cobalt.int.rdmedia.com <http://cobalt.int.rdmedia.com> > <http://cobalt.int.rdmedia.com> while not > > > monitoring any hosts > > > Sep 21 17:07:19 cobalt logger: setting up cluster rd-ganesha-ha > with > > > the following cobalt iron > > > Sep 21 17:07:20 cobalt systemd: Stopped Pacemaker High Availability > > > Cluster Manager. > > > Sep 21 17:07:20 cobalt systemd: Stopped Corosync Cluster Engine. > > > Sep 21 17:07:20 cobalt systemd: Reloading. > > > Sep 21 17:07:20 cobalt systemd: > > > [/usr/lib/systemd/system/dm-event.socket:10] Unknown lvalue > > > 'RemoveOnStop' in section 'Socket' > > > Sep 21 17:07:20 cobalt systemd: > > > [/usr/lib/systemd/system/lvm2-lvmetad.socket:9] Unknown lvalue > > > 'RemoveOnStop' in section 'Socket' > > > Sep 21 17:07:20 cobalt systemd: Reloading. > > > Sep 21 17:07:20 cobalt systemd: > > > [/usr/lib/systemd/system/dm-event.socket:10] Unknown lvalue > > > 'RemoveOnStop' in section 'Socket' > > > Sep 21 17:07:20 cobalt systemd: > > > [/usr/lib/systemd/system/lvm2-lvmetad.socket:9] Unknown lvalue > > > 'RemoveOnStop' in section 'Socket' > > > Sep 21 17:07:20 cobalt systemd: Starting Corosync Cluster Engine... > > > Sep 21 17:07:20 cobalt corosync[2816]: [MAIN ] Corosync Cluster > > > Engine ('2.3.4'): started and ready to provide service. > > > Sep 21 17:07:20 cobalt corosync[2816]: [MAIN ] Corosync built-in > > > features: dbus systemd xmlconf snmp pie relro bindnow > > > Sep 21 17:07:20 cobalt corosync[2817]: [TOTEM ] Initializing > transport > > > (UDP/IP Unicast). > > > Sep 21 17:07:20 cobalt corosync[2817]: [TOTEM ] Initializing > > > transmit/receive security (NSS) crypto: none hash: none > > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] The network > interface > > > [10.100.30.37] is now up. > > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine > loaded: > > > corosync configuration map access [0] > > > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name: cmap > > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine > loaded: > > > corosync configuration service [1] > > > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name: cfg > > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine > loaded: > > > corosync cluster closed process group service v1.01 [2] > > > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name: cpg > > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine > loaded: > > > corosync profile loading service [4] > > > Sep 21 17:07:21 cobalt corosync[2817]: [QUORUM] Using quorum > provider > > > corosync_votequorum > > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all > > > cluster members. Current votes: 1 expected_votes: 2 > > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine > loaded: > > > corosync vote quorum service v1.0 [5] > > > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name: > votequorum > > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine > loaded: > > > corosync cluster quorum service v0.1 [3] > > > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name: quorum > > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] adding new UDPU > member > > > {10.100.30.37} > > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] adding new UDPU > member > > > {10.100.30.38} > > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] A new membership > > > (10.100.30.37:100 <http://10.100.30.37:100> > <http://10.100.30.37:100>) was formed. Members joined: 1 > > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all > > > cluster members. Current votes: 1 expected_votes: 2 > > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all > > > cluster members. Current votes: 1 expected_votes: 2 > > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all > > > cluster members. Current votes: 1 expected_votes: 2 > > > Sep 21 17:07:21 cobalt corosync[2817]: [QUORUM] Members[1]: 1 > > > Sep 21 17:07:21 cobalt corosync[2817]: [MAIN ] Completed service > > > synchronization, ready to provide service. > > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] A new membership > > > (10.100.30.37:108 <http://10.100.30.37:108> > <http://10.100.30.37:108>) was formed. Members joined: 1 > > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all > > > cluster members. Current votes: 1 expected_votes: 2 > > > Sep 21 17:07:21 cobalt corosync[2817]: [QUORUM] Members[1]: 1 > > > Sep 21 17:07:21 cobalt corosync[2817]: [MAIN ] Completed service > > > synchronization, ready to provide service. > > > Sep 21 17:08:50 cobalt systemd: corosync.service operation > timed out. > > > Terminating. > > > Sep 21 17:08:50 cobalt corosync: Starting Corosync Cluster Engine > > > (corosync): > > > Sep 21 17:08:50 cobalt systemd: Failed to start Corosync > Cluster Engine. > > > Sep 21 17:08:50 cobalt systemd: Unit corosync.service entered > failed > > > state. > > > Sep 21 17:08:55 cobalt logger: warning: pcs property set > > > no-quorum-policy=ignore failed > > > Sep 21 17:08:55 cobalt logger: warning: pcs property set > > > stonith-enabled=false failed > > > Sep 21 17:08:55 cobalt logger: warning: pcs resource create > nfs_start > > > ganesha_nfsd ha_vol_mnt=/var/run/gluster/shared_storage --clone > failed > > > Sep 21 17:08:56 cobalt logger: warning: pcs resource delete > > > nfs_start-clone failed > > > Sep 21 17:08:56 cobalt logger: warning: pcs resource create nfs-mon > > > ganesha_mon --clone failed > > > Sep 21 17:08:56 cobalt logger: warning: pcs resource create > nfs-grace > > > ganesha_grace --clone failed > > > Sep 21 17:08:57 cobalt logger: warning pcs resource create > > > cobalt-cluster_ip-1 ocf:heartbeat:IPaddr ip= cidr_netmask=32 op > > > monitor interval=15s failed > > > Sep 21 17:08:57 cobalt logger: warning: pcs resource create > > > cobalt-trigger_ip-1 ocf:heartbeat:Dummy failed > > > Sep 21 17:08:57 cobalt logger: warning: pcs constraint > colocation add > > > cobalt-cluster_ip-1 with cobalt-trigger_ip-1 failed > > > Sep 21 17:08:57 cobalt logger: warning: pcs constraint order > > > cobalt-trigger_ip-1 then nfs-grace-clone failed > > > Sep 21 17:08:57 cobalt logger: warning: pcs constraint order > > > nfs-grace-clone then cobalt-cluster_ip-1 failed > > > Sep 21 17:08:57 cobalt logger: warning pcs resource create > > > iron-cluster_ip-1 ocf:heartbeat:IPaddr ip= cidr_netmask=32 op > monitor > > > interval=15s failed > > > Sep 21 17:08:57 cobalt logger: warning: pcs resource create > > > iron-trigger_ip-1 ocf:heartbeat:Dummy failed > > > Sep 21 17:08:57 cobalt logger: warning: pcs constraint > colocation add > > > iron-cluster_ip-1 with iron-trigger_ip-1 failed > > > Sep 21 17:08:57 cobalt logger: warning: pcs constraint order > > > iron-trigger_ip-1 then nfs-grace-clone failed > > > Sep 21 17:08:58 cobalt logger: warning: pcs constraint order > > > nfs-grace-clone then iron-cluster_ip-1 failed > > > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location > > > cobalt-cluster_ip-1 rule score=-INFINITY ganesha-active ne 1 failed > > > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location > > > cobalt-cluster_ip-1 prefers iron=1000 failed > > > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location > > > cobalt-cluster_ip-1 prefers cobalt=2000 failed > > > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location > > > iron-cluster_ip-1 rule score=-INFINITY ganesha-active ne 1 failed > > > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location > > > iron-cluster_ip-1 prefers cobalt=1000 failed > > > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location > > > iron-cluster_ip-1 prefers iron=2000 failed > > > Sep 21 17:08:58 cobalt logger: warning pcs cluster cib-push > > > /tmp/tmp.nXTfyA1GMR failed > > > Sep 21 17:08:58 cobalt logger: warning: scp ganesha-ha.conf to > cobalt > > > failed > > > > > > BTW, I'm using CentOS 7. There are multiple network interfaces > on the > > > servers, could that be a problem? > > > > > > > > > > > > > > > On 21 September 2015 at 11:48, Jiffin Tony Thottan > > > <jthottan at redhat.com <mailto:jthottan at redhat.com> > <mailto:jthottan at redhat.com <mailto:jthottan at redhat.com>>> wrote: > > > > > > > > > > > > On 21/09/15 13:56, Tiemen Ruiten wrote: > > >> Hello Soumya, Kaleb, list, > > >> > > >> This Friday I created the gluster_shared_storage volume > manually, > > >> I just tried it with the command you supplied, but both > have the > > >> same result: > > >> > > >> from etc-glusterfs-glusterd.vol.log on the node where I issued > > >> the command: > > >> > > >> [2015-09-21 07:59:47.756845] I [MSGID: 106474] > > >> [glusterd-ganesha.c:403:check_host_list] 0-management: ganesha > > >> host found Hostname is cobalt > > >> [2015-09-21 07:59:48.071755] I [MSGID: 106474] > > >> [glusterd-ganesha.c:349:is_ganesha_host] 0-management: ganesha > > >> host found Hostname is cobalt > > >> [2015-09-21 07:59:48.653879] E [MSGID: 106470] > > >> [glusterd-ganesha.c:264:glusterd_op_set_ganesha] 0-management: > > >> Initial NFS-Ganesha set up failed > > > > > > As far as what I understand from the logs, it called > > > setup_cluser()[calls `ganesha-ha.sh` script ] but script > failed. > > > Can u please provide following details : > > > -Location of ganesha.sh file?? > > > -Location of ganesha-ha.conf, ganesha.conf files ? > > > > > > > > > And also can u cross check whether all the prerequisites > before HA > > > setup satisfied ? > > > > > > -- > > > With Regards, > > > Jiffin > > > > > > > > >> [2015-09-21 07:59:48.653912] E [MSGID: 106123] > > >> [glusterd-syncop.c:1404:gd_commit_op_phase] 0-management: > Commit > > >> of operation 'Volume (null)' failed on localhost : Failed > to set > > >> up HA config for NFS-Ganesha. Please check the log file > for details > > >> [2015-09-21 07:59:45.402458] I [MSGID: 106006] > > >> [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] > > >> 0-management: nfs has disconnected from glusterd. > > >> [2015-09-21 07:59:48.071578] I [MSGID: 106474] > > >> [glusterd-ganesha.c:403:check_host_list] 0-management: ganesha > > >> host found Hostname is cobalt > > >> > > >> from etc-glusterfs-glusterd.vol.log on the other node: > > >> > > >> [2015-09-21 08:12:50.111877] E [MSGID: 106062] > > >> [glusterd-op-sm.c:3698:glusterd_op_ac_unlock] 0-management: > > >> Unable to acquire volname > > >> [2015-09-21 08:14:50.548087] E [MSGID: 106062] > > >> [glusterd-op-sm.c:3635:glusterd_op_ac_lock] 0-management: > Unable > > >> to acquire volname > > >> [2015-09-21 08:14:50.654746] I [MSGID: 106132] > > >> [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs > > >> already stopped > > >> [2015-09-21 08:14:50.655095] I [MSGID: 106474] > > >> [glusterd-ganesha.c:403:check_host_list] 0-management: ganesha > > >> host found Hostname is cobalt > > >> [2015-09-21 08:14:51.287156] E [MSGID: 106062] > > >> [glusterd-op-sm.c:3698:glusterd_op_ac_unlock] 0-management: > > >> Unable to acquire volname > > >> > > >> > > >> from etc-glusterfs-glusterd.vol.log on the arbiter node: > > >> > > >> [2015-09-21 08:18:50.934713] E [MSGID: 101075] > > >> [common-utils.c:3127:gf_is_local_addr] 0-management: error in > > >> getaddrinfo: Name or service not known > > >> [2015-09-21 08:18:51.504694] E [MSGID: 106062] > > >> [glusterd-op-sm.c:3698:glusterd_op_ac_unlock] 0-management: > > >> Unable to acquire volname > > >> > > >> I have put the hostnames of all servers in my /etc/hosts file, > > >> including the arbiter node. > > >> > > >> > > >> On 18 September 2015 at 16:52, Soumya Koduri > <skoduri at redhat.com <mailto:skoduri at redhat.com> > > >> <mailto:skoduri at redhat.com <mailto:skoduri at redhat.com>>> > wrote: > > >> > > >> Hi Tiemen, > > >> > > >> One of the pre-requisites before setting up > nfs-ganesha HA is > > >> to create and mount shared_storage volume. Use below > CLI for that > > >> > > >> "gluster volume set all cluster.enable-shared-storage > enable" > > >> > > >> It shall create the volume and mount in all the nodes > > >> (including the arbiter node). Note this volume shall be > > >> mounted on all the nodes of the gluster storage pool > (though > > >> in this case it may not be part of nfs-ganesha cluster). > > >> > > >> So instead of manually creating those directory paths, > please > > >> use above CLI and try re-configuring the setup. > > >> > > >> Thanks, > > >> Soumya > > >> > > >> On 09/18/2015 07:29 PM, Tiemen Ruiten wrote: > > >> > > >> Hello Kaleb, > > >> > > >> I don't: > > >> > > >> # Name of the HA cluster created. > > >> # must be unique within the subnet > > >> HA_NAME="rd-ganesha-ha" > > >> # > > >> # The gluster server from which to mount the > shared data > > >> volume. > > >> HA_VOL_SERVER="iron" > > >> # > > >> # N.B. you may use short names or long names; you > may not > > >> use IP addrs. > > >> # Once you select one, stay with it as it will be > mildly > > >> unpleasant to > > >> # clean up if you switch later on. Ensure that all > names > > >> - short and/or > > >> # long - are in DNS or /etc/hosts on all machines > in the > > >> cluster. > > >> # > > >> # The subset of nodes of the Gluster Trusted Pool that > > >> form the ganesha > > >> # HA cluster. Hostname is specified. > > >> HA_CLUSTER_NODES="cobalt,iron" > > >> #HA_CLUSTER_NODES="server1.lab.redhat.com > <http://server1.lab.redhat.com> > > >> <http://server1.lab.redhat.com> > > >> > <http://server1.lab.redhat.com>,server2.lab.redhat.com > <http://server2.lab.redhat.com> > > >> <http://server2.lab.redhat.com> > > >> <http://server2.lab.redhat.com>,..." > > >> # > > >> # Virtual IPs for each of the nodes specified above. > > >> VIP_server1="10.100.30.101" > > >> VIP_server2="10.100.30.102" > > >> #VIP_server1_lab_redhat_com="10.0.2.1" > > >> #VIP_server2_lab_redhat_com="10.0.2.2" > > >> > > >> hosts cobalt & iron are the data nodes, the arbiter > > >> ip/hostname (neon) > > >> isn't mentioned anywhere in this config file. > > >> > > >> > > >> On 18 September 2015 at 15:56, Kaleb S. KEITHLEY > > >> <kkeithle at redhat.com <mailto:kkeithle at redhat.com> > <mailto:kkeithle at redhat.com <mailto:kkeithle at redhat.com>> > > >> <mailto:kkeithle at redhat.com > <mailto:kkeithle at redhat.com> > > >> <mailto:kkeithle at redhat.com > <mailto:kkeithle at redhat.com>>>> wrote: > > >> > > >> On 09/18/2015 09:46 AM, Tiemen Ruiten wrote: > > >> > Hello, > > >> > > > >> > I have a Gluster cluster with a single > replica 3, > > >> arbiter 1 volume (so > > >> > two nodes with actual data, one arbiter node). I > > >> would like to setup > > >> > NFS-Ganesha HA for this volume but I'm > having some > > >> difficulties. > > >> > > > >> > - I needed to create a directory > > >> /var/run/gluster/shared_storage > > >> > manually on all nodes, or the command 'gluster > > >> nfs-ganesha enable would > > >> > fail with the following error: > > >> > [2015-09-18 13:13:34.690416] E [MSGID: 106032] > > >> > [glusterd-ganesha.c:708:pre_setup] 0-THIS->name: > > >> mkdir() failed on path > > >> > /var/run/gluster/shared_storage/nfs-ganesha, [No > > >> such file or directory] > > >> > > > >> > - Then I found out that the command connects > to the > > >> arbiter node as > > >> > well, but obviously I don't want to set up > > >> NFS-Ganesha there. Is it > > >> > actually possible to setup NFS-Ganesha HA > with an > > >> arbiter node? If it's > > >> > possible, is there any documentation on how > to do that? > > >> > > > >> > > >> Please send the /etc/ganesha/ganesha-ha.conf file > > >> you're using. > > >> > > >> Probably you have included the arbiter in your HA > > >> config; that would be > > >> a mistake. > > >> > > >> -- > > >> > > >> Kaleb > > >> > > >> > > >> > > >> > > >> -- > > >> Tiemen Ruiten > > >> Systems Engineer > > >> R&D Media > > >> > > >> > > >> _______________________________________________ > > >> Gluster-users mailing list > > >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > <mailto:Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>> > > >>http://www.gluster.org/mailman/listinfo/gluster-users > > >> > > >> > > >> > > >> > > >> -- > > >> Tiemen Ruiten > > >> Systems Engineer > > >> R&D Media > > >> > > >> > > >> _______________________________________________ > > >> Gluster-users mailing list > > >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > <mailto:Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>> > > >>http://www.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > <mailto:Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>> > > >http://www.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > > > > > > > -- > > > Tiemen Ruiten > > > Systems Engineer > > > R&D Media > > > > > > > > > > > > -- > > > Tiemen Ruiten > > > Systems Engineer > > > R&D Media > > > > > > > > > _______________________________________________ > > > Gluster-users mailing list > > >Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > > >http://www.gluster.org/mailman/listinfo/gluster-users > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://www.gluster.org/mailman/listinfo/gluster-users > > > > > -- > Tiemen Ruiten > Systems Engineer > R&D Media > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >
Tiemen Ruiten
2015-Sep-22 19:44 UTC
[Gluster-users] Fwd: nfs-ganesha HA with arbiter volume
Allright, thank you Soumya. I actually did do the cleanup every time (gluster nfs-ganesha disable), but it didn't always finish succesfully. Sometimes it would just time out. I'll try with the second command tomorrow. Good to know that it should work with two nodes as well. On 22 September 2015 at 19:26, Soumya Koduri <skoduri at redhat.com> wrote:> > > On 09/22/2015 05:06 PM, Tiemen Ruiten wrote: > >> That's correct and my original question was actually if a two node + >> arbiter setup is possible. The documentation provided by Soumya only >> mentions two servers in the example ganesha-ha.sh script. Perhaps that >> could be updated as well then, to not give the wrong impression. >> >> It does work with 2-node as well. In the script, there is already a check > to verify if the number of servers < 3, it automatically disables quorum. > Quorum cannot be enabled for a 2-node setup for obvious reasons. If one > node fails, other node just takes over the IP. > > Thanks, > Soumya > > I could try to change the script to disable quorum, but wouldn't that >> defeat the purpose? What will happen in case one node goes down >> unexpectedly? >> >> On 22 September 2015 at 12:47, Kaleb Keithley <kkeithle at redhat.com >> <mailto:kkeithle at redhat.com>> wrote: >> >> >> Hi, >> >> IIRC, the setup is two nodes gluster+ganesha nodes plus the arbiter >> node for gluster quorum. >> >> Have I remembered that correctly? >> >> The Ganesha HA in 3.7 requires a minimum of three servers running >> ganesha and pacemaker. Two might work if you change the >> ganesha-ha.sh to not enable pacemaker quorum, but I haven't tried >> that myself. I'll try and find time in the next couple of days to >> update the documentation or write a blog post. >> >> >> >> ----- Original Message ---- >> > >> > >> > >> > On 21/09/15 21:21, Tiemen Ruiten wrote: >> > > Whoops, replied off-list. >> > > >> > > Additionally I noticed that the generated corosync config is not >> > > valid, as there is no interface section: >> > > >> > > /etc/corosync/corosync.conf >> > > >> > > totem { >> > > version: 2 >> > > secauth: off >> > > cluster_name: rd-ganesha-ha >> > > transport: udpu >> > > } >> > > >> > > nodelist { >> > > node { >> > > ring0_addr: cobalt >> > > nodeid: 1 >> > > } >> > > node { >> > > ring0_addr: iron >> > > nodeid: 2 >> > > } >> > > } >> > > >> > > quorum { >> > > provider: corosync_votequorum >> > > two_node: 1 >> > > } >> > > >> > > logging { >> > > to_syslog: yes >> > > } >> > > >> > > >> > > >> > >> > May be Kaleb can help you out. >> > > >> > > ---------- Forwarded message ---------- >> > > From: *Tiemen Ruiten* <t.ruiten at rdmedia.com <mailto: >> t.ruiten at rdmedia.com> >> <mailto:t.ruiten at rdmedia.com <mailto:t.ruiten at rdmedia.com>>> >> > > Date: 21 September 2015 at 17:16 >> > > Subject: Re: [Gluster-users] nfs-ganesha HA with arbiter volume >> > > To: Jiffin Tony Thottan <jthottan at redhat.com >> <mailto:jthottan at redhat.com> <mailto:jthottan at redhat.com >> <mailto:jthottan at redhat.com>>> >> > > >> > > >> > > Could you point me to the latest documentation? I've been >> struggling >> > > to find something up-to-date. I believe I have all the >> prerequisites: >> > > >> > > - shared storage volume exists and is mounted >> > > - all nodes in hosts files >> > > - Gluster-NFS disabled >> > > - corosync, pacemaker and nfs-ganesha rpm's installed >> > > >> > > Anything I missed? >> > > >> > > Everything has been installed by RPM so is in the default >> locations: >> > > /usr/libexec/ganesha/ganesha-ha.sh >> > > /etc/ganesha/ganesha.conf (empty) >> > > /etc/ganesha/ganesha-ha.conf >> > > >> > >> > Looks fine for me. >> > >> > > After I started the pcsd service manually, nfs-ganesha could be >> > > enabled successfully, but there was no virtual IP present on the >> > > interfaces and looking at the system log, I noticed corosync >> failed to >> > > start: >> > > >> > > - on the host where I issued the gluster nfs-ganesha enable >> command: >> > > >> > > Sep 21 17:07:18 iron systemd: Starting NFS-Ganesha file server... >> > > Sep 21 17:07:19 iron systemd: Started NFS-Ganesha file server. >> > > Sep 21 17:07:19 iron rpc.statd[2409]: Received SM_UNMON_ALL >> request >> > > from iron.int.rdmedia.com <http://iron.int.rdmedia.com> >> <http://iron.int.rdmedia.com> while not >> > > monitoring any hosts >> > > Sep 21 17:07:20 iron systemd: Starting Corosync Cluster Engine... >> > > Sep 21 17:07:20 iron corosync[3426]: [MAIN ] Corosync Cluster >> Engine >> > > ('2.3.4'): started and ready to provide service. >> > > Sep 21 17:07:20 iron corosync[3426]: [MAIN ] Corosync built-in >> > > features: dbus systemd xmlconf snmp pie relro bindnow >> > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] Initializing >> transport >> > > (UDP/IP Unicast). >> > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] Initializing >> > > transmit/receive security (NSS) crypto: none hash: none >> > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] The network >> interface >> > > [10.100.30.38] is now up. >> > > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine >> loaded: >> > > corosync configuration map access [0] >> > > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name: cmap >> > > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine >> loaded: >> > > corosync configuration service [1] >> > > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name: cfg >> > > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine >> loaded: >> > > corosync cluster closed process group service v1.01 [2] >> > > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name: cpg >> > > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine >> loaded: >> > > corosync profile loading service [4] >> > > Sep 21 17:07:20 iron corosync[3427]: [QUORUM] Using quorum >> provider >> > > corosync_votequorum >> > > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ] Waiting for all >> cluster >> > > members. Current votes: 1 expected_votes: 2 >> > > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine >> loaded: >> > > corosync vote quorum service v1.0 [5] >> > > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name: >> votequorum >> > > Sep 21 17:07:20 iron corosync[3427]: [SERV ] Service engine >> loaded: >> > > corosync cluster quorum service v0.1 [3] >> > > Sep 21 17:07:20 iron corosync[3427]: [QB ] server name: quorum >> > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] adding new UDPU >> member >> > > {10.100.30.38} >> > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] adding new UDPU >> member >> > > {10.100.30.37} >> > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] A new membership >> > > (10.100.30.38:104 <http://10.100.30.38:104> >> <http://10.100.30.38:104>) was formed. Members joined: 1 >> > > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ] Waiting for all >> cluster >> > > members. Current votes: 1 expected_votes: 2 >> > > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ] Waiting for all >> cluster >> > > members. Current votes: 1 expected_votes: 2 >> > > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ] Waiting for all >> cluster >> > > members. Current votes: 1 expected_votes: 2 >> > > Sep 21 17:07:20 iron corosync[3427]: [QUORUM] Members[1]: 1 >> > > Sep 21 17:07:20 iron corosync[3427]: [MAIN ] Completed service >> > > synchronization, ready to provide service. >> > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] A new membership >> > > (10.100.30.37:108 <http://10.100.30.37:108> >> <http://10.100.30.37:108>) was formed. Members joined: 1 >> > > Sep 21 17:08:21 iron corosync: Starting Corosync Cluster Engine >> > > (corosync): [FAILED] >> > > Sep 21 17:08:21 iron systemd: corosync.service: control process >> > > exited, code=exited status=1 >> > > Sep 21 17:08:21 iron systemd: Failed to start Corosync Cluster >> Engine. >> > > Sep 21 17:08:21 iron systemd: Unit corosync.service entered >> failed state. >> > > >> > > >> > > - on the other host: >> > > >> > > Sep 21 17:07:19 cobalt systemd: Starting Preprocess NFS >> configuration... >> > > Sep 21 17:07:19 cobalt systemd: Starting RPC Port Mapper. >> > > Sep 21 17:07:19 cobalt systemd: Reached target RPC Port Mapper. >> > > Sep 21 17:07:19 cobalt systemd: Starting Host and Network Name >> Lookups. >> > > Sep 21 17:07:19 cobalt systemd: Reached target Host and Network >> Name >> > > Lookups. >> > > Sep 21 17:07:19 cobalt systemd: Starting RPC bind service... >> > > Sep 21 17:07:19 cobalt systemd: Started Preprocess NFS >> configuration. >> > > Sep 21 17:07:19 cobalt systemd: Started RPC bind service. >> > > Sep 21 17:07:19 cobalt systemd: Starting NFS status monitor for >> > > NFSv2/3 locking.... >> > > Sep 21 17:07:19 cobalt rpc.statd[2662]: Version 1.3.0 starting >> > > Sep 21 17:07:19 cobalt rpc.statd[2662]: Flags: TI-RPC >> > > Sep 21 17:07:19 cobalt systemd: Started NFS status monitor for >> NFSv2/3 >> > > locking.. >> > > Sep 21 17:07:19 cobalt systemd: Starting NFS-Ganesha file >> server... >> > > Sep 21 17:07:19 cobalt systemd: Started NFS-Ganesha file server. >> > > Sep 21 17:07:19 cobalt kernel: warning: `ganesha.nfsd' uses >> 32-bit >> > > capabilities (legacy support in use) >> > > Sep 21 17:07:19 cobalt logger: setting up rd-ganesha-ha >> > > Sep 21 17:07:19 cobalt rpc.statd[2662]: Received SM_UNMON_ALL >> request >> > > from cobalt.int.rdmedia.com <http://cobalt.int.rdmedia.com> >> <http://cobalt.int.rdmedia.com> while not >> > > monitoring any hosts >> > > Sep 21 17:07:19 cobalt logger: setting up cluster rd-ganesha-ha >> with >> > > the following cobalt iron >> > > Sep 21 17:07:20 cobalt systemd: Stopped Pacemaker High >> Availability >> > > Cluster Manager. >> > > Sep 21 17:07:20 cobalt systemd: Stopped Corosync Cluster Engine. >> > > Sep 21 17:07:20 cobalt systemd: Reloading. >> > > Sep 21 17:07:20 cobalt systemd: >> > > [/usr/lib/systemd/system/dm-event.socket:10] Unknown lvalue >> > > 'RemoveOnStop' in section 'Socket' >> > > Sep 21 17:07:20 cobalt systemd: >> > > [/usr/lib/systemd/system/lvm2-lvmetad.socket:9] Unknown lvalue >> > > 'RemoveOnStop' in section 'Socket' >> > > Sep 21 17:07:20 cobalt systemd: Reloading. >> > > Sep 21 17:07:20 cobalt systemd: >> > > [/usr/lib/systemd/system/dm-event.socket:10] Unknown lvalue >> > > 'RemoveOnStop' in section 'Socket' >> > > Sep 21 17:07:20 cobalt systemd: >> > > [/usr/lib/systemd/system/lvm2-lvmetad.socket:9] Unknown lvalue >> > > 'RemoveOnStop' in section 'Socket' >> > > Sep 21 17:07:20 cobalt systemd: Starting Corosync Cluster >> Engine... >> > > Sep 21 17:07:20 cobalt corosync[2816]: [MAIN ] Corosync Cluster >> > > Engine ('2.3.4'): started and ready to provide service. >> > > Sep 21 17:07:20 cobalt corosync[2816]: [MAIN ] Corosync built-in >> > > features: dbus systemd xmlconf snmp pie relro bindnow >> > > Sep 21 17:07:20 cobalt corosync[2817]: [TOTEM ] Initializing >> transport >> > > (UDP/IP Unicast). >> > > Sep 21 17:07:20 cobalt corosync[2817]: [TOTEM ] Initializing >> > > transmit/receive security (NSS) crypto: none hash: none >> > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] The network >> interface >> > > [10.100.30.37] is now up. >> > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine >> loaded: >> > > corosync configuration map access [0] >> > > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name: cmap >> > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine >> loaded: >> > > corosync configuration service [1] >> > > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name: cfg >> > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine >> loaded: >> > > corosync cluster closed process group service v1.01 [2] >> > > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name: cpg >> > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine >> loaded: >> > > corosync profile loading service [4] >> > > Sep 21 17:07:21 cobalt corosync[2817]: [QUORUM] Using quorum >> provider >> > > corosync_votequorum >> > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all >> > > cluster members. Current votes: 1 expected_votes: 2 >> > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine >> loaded: >> > > corosync vote quorum service v1.0 [5] >> > > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name: >> votequorum >> > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV ] Service engine >> loaded: >> > > corosync cluster quorum service v0.1 [3] >> > > Sep 21 17:07:21 cobalt corosync[2817]: [QB ] server name: >> quorum >> > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] adding new UDPU >> member >> > > {10.100.30.37} >> > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] adding new UDPU >> member >> > > {10.100.30.38} >> > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] A new membership >> > > (10.100.30.37:100 <http://10.100.30.37:100> >> <http://10.100.30.37:100>) was formed. Members joined: 1 >> > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all >> > > cluster members. Current votes: 1 expected_votes: 2 >> > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all >> > > cluster members. Current votes: 1 expected_votes: 2 >> > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all >> > > cluster members. Current votes: 1 expected_votes: 2 >> > > Sep 21 17:07:21 cobalt corosync[2817]: [QUORUM] Members[1]: 1 >> > > Sep 21 17:07:21 cobalt corosync[2817]: [MAIN ] Completed service >> > > synchronization, ready to provide service. >> > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] A new membership >> > > (10.100.30.37:108 <http://10.100.30.37:108> >> <http://10.100.30.37:108>) was formed. Members joined: 1 >> > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ] Waiting for all >> > > cluster members. Current votes: 1 expected_votes: 2 >> > > Sep 21 17:07:21 cobalt corosync[2817]: [QUORUM] Members[1]: 1 >> > > Sep 21 17:07:21 cobalt corosync[2817]: [MAIN ] Completed service >> > > synchronization, ready to provide service. >> > > Sep 21 17:08:50 cobalt systemd: corosync.service operation >> timed out. >> > > Terminating. >> > > Sep 21 17:08:50 cobalt corosync: Starting Corosync Cluster Engine >> > > (corosync): >> > > Sep 21 17:08:50 cobalt systemd: Failed to start Corosync >> Cluster Engine. >> > > Sep 21 17:08:50 cobalt systemd: Unit corosync.service entered >> failed >> > > state. >> > > Sep 21 17:08:55 cobalt logger: warning: pcs property set >> > > no-quorum-policy=ignore failed >> > > Sep 21 17:08:55 cobalt logger: warning: pcs property set >> > > stonith-enabled=false failed >> > > Sep 21 17:08:55 cobalt logger: warning: pcs resource create >> nfs_start >> > > ganesha_nfsd ha_vol_mnt=/var/run/gluster/shared_storage --clone >> failed >> > > Sep 21 17:08:56 cobalt logger: warning: pcs resource delete >> > > nfs_start-clone failed >> > > Sep 21 17:08:56 cobalt logger: warning: pcs resource create >> nfs-mon >> > > ganesha_mon --clone failed >> > > Sep 21 17:08:56 cobalt logger: warning: pcs resource create >> nfs-grace >> > > ganesha_grace --clone failed >> > > Sep 21 17:08:57 cobalt logger: warning pcs resource create >> > > cobalt-cluster_ip-1 ocf:heartbeat:IPaddr ip= cidr_netmask=32 op >> > > monitor interval=15s failed >> > > Sep 21 17:08:57 cobalt logger: warning: pcs resource create >> > > cobalt-trigger_ip-1 ocf:heartbeat:Dummy failed >> > > Sep 21 17:08:57 cobalt logger: warning: pcs constraint >> colocation add >> > > cobalt-cluster_ip-1 with cobalt-trigger_ip-1 failed >> > > Sep 21 17:08:57 cobalt logger: warning: pcs constraint order >> > > cobalt-trigger_ip-1 then nfs-grace-clone failed >> > > Sep 21 17:08:57 cobalt logger: warning: pcs constraint order >> > > nfs-grace-clone then cobalt-cluster_ip-1 failed >> > > Sep 21 17:08:57 cobalt logger: warning pcs resource create >> > > iron-cluster_ip-1 ocf:heartbeat:IPaddr ip= cidr_netmask=32 op >> monitor >> > > interval=15s failed >> > > Sep 21 17:08:57 cobalt logger: warning: pcs resource create >> > > iron-trigger_ip-1 ocf:heartbeat:Dummy failed >> > > Sep 21 17:08:57 cobalt logger: warning: pcs constraint >> colocation add >> > > iron-cluster_ip-1 with iron-trigger_ip-1 failed >> > > Sep 21 17:08:57 cobalt logger: warning: pcs constraint order >> > > iron-trigger_ip-1 then nfs-grace-clone failed >> > > Sep 21 17:08:58 cobalt logger: warning: pcs constraint order >> > > nfs-grace-clone then iron-cluster_ip-1 failed >> > > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location >> > > cobalt-cluster_ip-1 rule score=-INFINITY ganesha-active ne 1 >> failed >> > > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location >> > > cobalt-cluster_ip-1 prefers iron=1000 failed >> > > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location >> > > cobalt-cluster_ip-1 prefers cobalt=2000 failed >> > > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location >> > > iron-cluster_ip-1 rule score=-INFINITY ganesha-active ne 1 failed >> > > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location >> > > iron-cluster_ip-1 prefers cobalt=1000 failed >> > > Sep 21 17:08:58 cobalt logger: warning: pcs constraint location >> > > iron-cluster_ip-1 prefers iron=2000 failed >> > > Sep 21 17:08:58 cobalt logger: warning pcs cluster cib-push >> > > /tmp/tmp.nXTfyA1GMR failed >> > > Sep 21 17:08:58 cobalt logger: warning: scp ganesha-ha.conf to >> cobalt >> > > failed >> > > >> > > BTW, I'm using CentOS 7. There are multiple network interfaces >> on the >> > > servers, could that be a problem? >> > > >> > > >> > > >> > > >> > > On 21 September 2015 at 11:48, Jiffin Tony Thottan >> > > <jthottan at redhat.com <mailto:jthottan at redhat.com> >> <mailto:jthottan at redhat.com <mailto:jthottan at redhat.com>>> wrote: >> > > >> > > >> > > >> > > On 21/09/15 13:56, Tiemen Ruiten wrote: >> > >> Hello Soumya, Kaleb, list, >> > >> >> > >> This Friday I created the gluster_shared_storage volume >> manually, >> > >> I just tried it with the command you supplied, but both >> have the >> > >> same result: >> > >> >> > >> from etc-glusterfs-glusterd.vol.log on the node where I >> issued >> > >> the command: >> > >> >> > >> [2015-09-21 07:59:47.756845] I [MSGID: 106474] >> > >> [glusterd-ganesha.c:403:check_host_list] 0-management: >> ganesha >> > >> host found Hostname is cobalt >> > >> [2015-09-21 07:59:48.071755] I [MSGID: 106474] >> > >> [glusterd-ganesha.c:349:is_ganesha_host] 0-management: >> ganesha >> > >> host found Hostname is cobalt >> > >> [2015-09-21 07:59:48.653879] E [MSGID: 106470] >> > >> [glusterd-ganesha.c:264:glusterd_op_set_ganesha] >> 0-management: >> > >> Initial NFS-Ganesha set up failed >> > > >> > > As far as what I understand from the logs, it called >> > > setup_cluser()[calls `ganesha-ha.sh` script ] but script >> failed. >> > > Can u please provide following details : >> > > -Location of ganesha.sh file?? >> > > -Location of ganesha-ha.conf, ganesha.conf files ? >> > > >> > > >> > > And also can u cross check whether all the prerequisites >> before HA >> > > setup satisfied ? >> > > >> > > -- >> > > With Regards, >> > > Jiffin >> > > >> > > >> > >> [2015-09-21 07:59:48.653912] E [MSGID: 106123] >> > >> [glusterd-syncop.c:1404:gd_commit_op_phase] 0-management: >> Commit >> > >> of operation 'Volume (null)' failed on localhost : Failed >> to set >> > >> up HA config for NFS-Ganesha. Please check the log file >> for details >> > >> [2015-09-21 07:59:45.402458] I [MSGID: 106006] >> > >> [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] >> > >> 0-management: nfs has disconnected from glusterd. >> > >> [2015-09-21 07:59:48.071578] I [MSGID: 106474] >> > >> [glusterd-ganesha.c:403:check_host_list] 0-management: >> ganesha >> > >> host found Hostname is cobalt >> > >> >> > >> from etc-glusterfs-glusterd.vol.log on the other node: >> > >> >> > >> [2015-09-21 08:12:50.111877] E [MSGID: 106062] >> > >> [glusterd-op-sm.c:3698:glusterd_op_ac_unlock] 0-management: >> > >> Unable to acquire volname >> > >> [2015-09-21 08:14:50.548087] E [MSGID: 106062] >> > >> [glusterd-op-sm.c:3635:glusterd_op_ac_lock] 0-management: >> Unable >> > >> to acquire volname >> > >> [2015-09-21 08:14:50.654746] I [MSGID: 106132] >> > >> [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: >> nfs >> > >> already stopped >> > >> [2015-09-21 08:14:50.655095] I [MSGID: 106474] >> > >> [glusterd-ganesha.c:403:check_host_list] 0-management: >> ganesha >> > >> host found Hostname is cobalt >> > >> [2015-09-21 08:14:51.287156] E [MSGID: 106062] >> > >> [glusterd-op-sm.c:3698:glusterd_op_ac_unlock] 0-management: >> > >> Unable to acquire volname >> > >> >> > >> >> > >> from etc-glusterfs-glusterd.vol.log on the arbiter node: >> > >> >> > >> [2015-09-21 08:18:50.934713] E [MSGID: 101075] >> > >> [common-utils.c:3127:gf_is_local_addr] 0-management: error >> in >> > >> getaddrinfo: Name or service not known >> > >> [2015-09-21 08:18:51.504694] E [MSGID: 106062] >> > >> [glusterd-op-sm.c:3698:glusterd_op_ac_unlock] 0-management: >> > >> Unable to acquire volname >> > >> >> > >> I have put the hostnames of all servers in my /etc/hosts >> file, >> > >> including the arbiter node. >> > >> >> > >> >> > >> On 18 September 2015 at 16:52, Soumya Koduri >> <skoduri at redhat.com <mailto:skoduri at redhat.com> >> > >> <mailto:skoduri at redhat.com <mailto:skoduri at redhat.com>>> >> >> wrote: >> > >> >> > >> Hi Tiemen, >> > >> >> > >> One of the pre-requisites before setting up >> nfs-ganesha HA is >> > >> to create and mount shared_storage volume. Use below >> CLI for that >> > >> >> > >> "gluster volume set all cluster.enable-shared-storage >> enable" >> > >> >> > >> It shall create the volume and mount in all the nodes >> > >> (including the arbiter node). Note this volume shall be >> > >> mounted on all the nodes of the gluster storage pool >> (though >> > >> in this case it may not be part of nfs-ganesha cluster). >> > >> >> > >> So instead of manually creating those directory paths, >> please >> > >> use above CLI and try re-configuring the setup. >> > >> >> > >> Thanks, >> > >> Soumya >> > >> >> > >> On 09/18/2015 07:29 PM, Tiemen Ruiten wrote: >> > >> >> > >> Hello Kaleb, >> > >> >> > >> I don't: >> > >> >> > >> # Name of the HA cluster created. >> > >> # must be unique within the subnet >> > >> HA_NAME="rd-ganesha-ha" >> > >> # >> > >> # The gluster server from which to mount the >> shared data >> > >> volume. >> > >> HA_VOL_SERVER="iron" >> > >> # >> > >> # N.B. you may use short names or long names; you >> may not >> > >> use IP addrs. >> > >> # Once you select one, stay with it as it will be >> mildly >> > >> unpleasant to >> > >> # clean up if you switch later on. Ensure that all >> names >> > >> - short and/or >> > >> # long - are in DNS or /etc/hosts on all machines >> in the >> > >> cluster. >> > >> # >> > >> # The subset of nodes of the Gluster Trusted Pool >> that >> > >> form the ganesha >> > >> # HA cluster. Hostname is specified. >> > >> HA_CLUSTER_NODES="cobalt,iron" >> > >> #HA_CLUSTER_NODES="server1.lab.redhat.com >> <http://server1.lab.redhat.com> >> > >> <http://server1.lab.redhat.com> >> > >> >> <http://server1.lab.redhat.com>,server2.lab.redhat.com >> <http://server2.lab.redhat.com> >> > >> <http://server2.lab.redhat.com> >> > >> <http://server2.lab.redhat.com>,..." >> > >> # >> > >> # Virtual IPs for each of the nodes specified above. >> > >> VIP_server1="10.100.30.101" >> > >> VIP_server2="10.100.30.102" >> > >> #VIP_server1_lab_redhat_com="10.0.2.1" >> > >> #VIP_server2_lab_redhat_com="10.0.2.2" >> > >> >> > >> hosts cobalt & iron are the data nodes, the arbiter >> > >> ip/hostname (neon) >> > >> isn't mentioned anywhere in this config file. >> > >> >> > >> >> > >> On 18 September 2015 at 15:56, Kaleb S. KEITHLEY >> > >> <kkeithle at redhat.com <mailto:kkeithle at redhat.com> >> <mailto:kkeithle at redhat.com <mailto:kkeithle at redhat.com>> >> > >> <mailto:kkeithle at redhat.com >> <mailto:kkeithle at redhat.com> >> > >> <mailto:kkeithle at redhat.com >> <mailto:kkeithle at redhat.com>>>> wrote: >> > >> >> > >> On 09/18/2015 09:46 AM, Tiemen Ruiten wrote: >> > >> > Hello, >> > >> > >> > >> > I have a Gluster cluster with a single >> replica 3, >> > >> arbiter 1 volume (so >> > >> > two nodes with actual data, one arbiter >> node). I >> > >> would like to setup >> > >> > NFS-Ganesha HA for this volume but I'm >> having some >> > >> difficulties. >> > >> > >> > >> > - I needed to create a directory >> > >> /var/run/gluster/shared_storage >> > >> > manually on all nodes, or the command 'gluster >> > >> nfs-ganesha enable would >> > >> > fail with the following error: >> > >> > [2015-09-18 13:13:34.690416] E [MSGID: 106032] >> > >> > [glusterd-ganesha.c:708:pre_setup] >> 0-THIS->name: >> > >> mkdir() failed on path >> > >> > /var/run/gluster/shared_storage/nfs-ganesha, >> [No >> > >> such file or directory] >> > >> > >> > >> > - Then I found out that the command connects >> to the >> > >> arbiter node as >> > >> > well, but obviously I don't want to set up >> > >> NFS-Ganesha there. Is it >> > >> > actually possible to setup NFS-Ganesha HA >> with an >> > >> arbiter node? If it's >> > >> > possible, is there any documentation on how >> to do that? >> > >> > >> > >> >> > >> Please send the /etc/ganesha/ganesha-ha.conf >> file >> > >> you're using. >> > >> >> > >> Probably you have included the arbiter in your >> HA >> > >> config; that would be >> > >> a mistake. >> > >> >> > >> -- >> > >> >> > >> Kaleb >> > >> >> > >> >> > >> >> > >> >> > >> -- >> > >> Tiemen Ruiten >> > >> Systems Engineer >> > >> R&D Media >> > >> >> > >> >> > >> _______________________________________________ >> > >> Gluster-users mailing list >> > >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >> <mailto:Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>> >> > >>http://www.gluster.org/mailman/listinfo/gluster-users >> > >> >> > >> >> > >> >> > >> >> > >> -- >> > >> Tiemen Ruiten >> > >> Systems Engineer >> > >> R&D Media >> > >> >> > >> >> > >> _______________________________________________ >> > >> Gluster-users mailing list >> > >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >> <mailto:Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>> >> > >>http://www.gluster.org/mailman/listinfo/gluster-users >> > > >> > > >> > > _______________________________________________ >> > > Gluster-users mailing list >> > > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >> <mailto:Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>> >> >> > >http://www.gluster.org/mailman/listinfo/gluster-users >> > > >> > > >> > > >> > > >> > > -- >> > > Tiemen Ruiten >> > > Systems Engineer >> > > R&D Media >> > > >> > > >> > > >> > > -- >> > > Tiemen Ruiten >> > > Systems Engineer >> > > R&D Media >> > > >> > > >> > > _______________________________________________ >> > > Gluster-users mailing list >> > >Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >> > >http://www.gluster.org/mailman/listinfo/gluster-users >> > >> > >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >> http://www.gluster.org/mailman/listinfo/gluster-users >> >> >> >> >> -- >> Tiemen Ruiten >> Systems Engineer >> R&D Media >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> >>-- Tiemen Ruiten Systems Engineer R&D Media -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150922/7de80ce5/attachment.html>