Gaurav Chhabra
2017-Aug-30 09:59 UTC
[Gluster-users] Unable to use Heketi setup to install Gluster for Kubernetes
Hi,
I have the following setup in place:
1 node : RancherOS having Rancher application for Kubernetes setup
2 nodes : RancherOS having Rancher agent
1 node : CentOS 7 workstation having kubectl installed and folder
cloned/downloaded from https://github.com/gluster/gluster-kubernetes using
which i run Heketi setup (gk-deploy -g)
I also have rancher-glusterfs-server container running with the following
configuration:
------------------------------------------------------------------
[root at node-1 rancher]# cat gluster-server.sh
#!/bin/bash
sudo docker run --name=gluster-server -d \
--env 'SERVICE_NAME=gluster' \
--restart always \
--env 'GLUSTER_DATA=/srv/docker/gitlab' \
--publish 2222:22 \
webcenter/rancher-glusterfs-server
------------------------------------------------------------------
In /etc/heketi/heketi.json, following is the only modified portion:
------------------------------------------------------------------
"executor": "ssh",
"_sshexec_comment": "SSH username and private key file
information",
"sshexec": {
"keyfile": "/var/lib/heketi/.ssh/id_rsa",
"user": "root",
"port": "22",
"fstab": "/etc/fstab"
},
------------------------------------------------------------------
Status before running gk-deploy:
[root at workstation deploy]# kubectl get nodes,pods,services,deployments
NAME STATUS AGE VERSION
no/node-1.c.kubernetes-174104.internal Ready 2d v1.7.2-rancher1
no/node-2.c.kubernetes-174104.internal Ready 2d v1.7.2-rancher1
no/node-3.c.kubernetes-174104.internal Ready 2d v1.7.2-rancher1
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kubernetes 10.43.0.1 <none> 443/TCP 2d
Now when i run 'gk-deploy -g', in the Rancher console, i see the
following
error:
Readiness probe failed: Failed to get D-Bus connection: Operation not
permitted
>From the attached gk-deploy_log i see that it failed at:
Waiting for GlusterFS pods to start ... pods not found.
In the kube-templates/glusterfs-daemonset.yaml file, i see this for
Readiness probe section:
------------------------------------------------------------------
readinessProbe:
timeoutSeconds: 3
initialDelaySeconds: 40
exec:
command:
- "/bin/bash"
- "-c"
- systemctl status glusterd.service
periodSeconds: 25
successThreshold: 1
failureThreshold: 15
------------------------------------------------------------------
Status after running gk-deploy:
[root at workstation deploy]# kubectl get nodes,pods,deployments,services
NAME STATUS AGE VERSION
no/node-1.c.kubernetes-174104.internal Ready 2d v1.7.2-rancher1
no/node-2.c.kubernetes-174104.internal Ready 2d v1.7.2-rancher1
no/node-3.c.kubernetes-174104.internal Ready 2d v1.7.2-rancher1
NAME READY STATUS RESTARTS AGE
po/glusterfs-0s440 0/1 Running 0 1m
po/glusterfs-j7dgr 0/1 Running 0 1m
po/glusterfs-p6jl3 0/1 Running 0 1m
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kubernetes 10.43.0.1 <none> 443/TCP 2d
Also, from prerequisite perspective, i was also seeing this mentioned:
The following kernel modules must be loaded:
* dm_snapshot
* dm_mirror
* dm_thin_pool
Where exactly is this to be checked? On all Gluster server nodes? How can i
check whether it's there?
I have attached topology.json and gk-deploy log for reference.
Does this issue has anything to do with the host OS (RancherOS) that i am
using for Gluster nodes? Any idea how i can fix this? Any help will really
be appreciated.
?Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170830/6c90513e/attachment.html>
-------------- next part --------------
[root at workstation deploy]# ./gk-deploy -g
Welcome to the deployment tool for GlusterFS on Kubernetes and OpenShift.
Before getting started, this script has some requirements of the execution
environment and of the container platform that you should verify.
The client machine that will run this script must have:
* Administrative access to an existing Kubernetes or OpenShift cluster
* Access to a python interpreter 'python'
Each of the nodes that will host GlusterFS must also have appropriate firewall
rules for the required GlusterFS ports:
* 2222 - sshd (if running GlusterFS in a pod)
* 24007 - GlusterFS Management
* 24008 - GlusterFS RDMA
* 49152 to 49251 - Each brick for every volume on the host requires its own
port. For every new brick, one new port will be used starting at 49152. We
recommend a default range of 49152-49251 on each host, though you can adjust
this to fit your needs.
The following kernel modules must be loaded:
* dm_snapshot
* dm_mirror
* dm_thin_pool
For systems with SELinux, the following settings need to be considered:
* virt_sandbox_use_fusefs should be enabled on each node to allow writing to
remote GlusterFS volumes
In addition, for an OpenShift deployment you must:
* Have 'cluster_admin' role on the administrative account doing the
deployment
* Add the 'default' and 'router' Service Accounts to the
'privileged' SCC
* Have a router deployed that is configured to allow apps to access services
running in the cluster
Do you wish to proceed with deployment?
[Y]es, [N]o? [Default: Y]: Y
Using Kubernetes CLI.
Using namespace "default".
Checking for pre-existing resources...
GlusterFS pods ... not found.
deploy-heketi pod ... not found.
heketi pod ... not found.
Creating initial resources ... Error from server (AlreadyExists): error when
creating
"/root/gluster-kubernetes/deploy/kube-templates/heketi-service-account.yaml":
serviceaccounts "heketi-service-account" already exists
Error from server (AlreadyExists): clusterrolebindings.rbac.authorization.k8s.io
"heketi-sa-view" already exists
clusterrolebinding "heketi-sa-view" not labeled
OK
node "node-1.c.kubernetes-174104.internal" labeled
node "node-2.c.kubernetes-174104.internal" labeled
node "node-3.c.kubernetes-174104.internal" labeled
daemonset "glusterfs" created
Waiting for GlusterFS pods to start ... pods not found.
[root at workstation deploy]#
-------------- next part --------------
A non-text attachment was scrubbed...
Name: topology.json
Type: application/json
Size: 790 bytes
Desc: not available
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170830/6c90513e/attachment.json>
Maybe Matching Threads
Wisdom of the Ancients
