Gluster users - Aug 2017 - Unable to use Heketi setup to install Gluster for Kubernetes

Hi,


I have the following setup in place:

1 node    : RancherOS having Rancher application for Kubernetes setup
2 nodes  : RancherOS having Rancher agent
1 node   : CentOS 7 workstation having kubectl installed and folder
cloned/downloaded from https://github.com/gluster/gluster-kubernetes using
which i run Heketi setup (gk-deploy -g)

I also have rancher-glusterfs-server container running with the following
configuration:
------------------------------------------------------------------
[root at node-1 rancher]# cat gluster-server.sh
#!/bin/bash

sudo docker run --name=gluster-server -d \
        --env 'SERVICE_NAME=gluster' \
        --restart always \
        --env 'GLUSTER_DATA=/srv/docker/gitlab' \
        --publish 2222:22 \
        webcenter/rancher-glusterfs-server
------------------------------------------------------------------

In /etc/heketi/heketi.json, following is the only modified portion:
------------------------------------------------------------------
    "executor": "ssh",

    "_sshexec_comment": "SSH username and private key file
information",
    "sshexec": {
      "keyfile": "/var/lib/heketi/.ssh/id_rsa",
      "user": "root",
      "port": "22",
      "fstab": "/etc/fstab"
    },
------------------------------------------------------------------

Status before running gk-deploy:

[root at workstation deploy]# kubectl get nodes,pods,services,deployments
NAME                                     STATUS    AGE       VERSION
no/node-1.c.kubernetes-174104.internal   Ready     2d        v1.7.2-rancher1
no/node-2.c.kubernetes-174104.internal   Ready     2d        v1.7.2-rancher1
no/node-3.c.kubernetes-174104.internal   Ready     2d        v1.7.2-rancher1

NAME             CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
svc/kubernetes   10.43.0.1    <none>        443/TCP   2d


Now when i run 'gk-deploy -g', in the Rancher console, i see the
following
error:
Readiness probe failed: Failed to get D-Bus connection: Operation not
permitted
>From the attached gk-deploy_log i see that it failed at:
Waiting for GlusterFS pods to start ... pods not found.

In the kube-templates/glusterfs-daemonset.yaml file, i see this for
Readiness probe section:
------------------------------------------------------------------
        readinessProbe:
          timeoutSeconds: 3
          initialDelaySeconds: 40
          exec:
            command:
            - "/bin/bash"
            - "-c"
            - systemctl status glusterd.service
          periodSeconds: 25
          successThreshold: 1
          failureThreshold: 15
------------------------------------------------------------------


Status after running gk-deploy:

[root at workstation deploy]# kubectl get nodes,pods,deployments,services
NAME                                     STATUS    AGE       VERSION
no/node-1.c.kubernetes-174104.internal   Ready     2d        v1.7.2-rancher1
no/node-2.c.kubernetes-174104.internal   Ready     2d        v1.7.2-rancher1
no/node-3.c.kubernetes-174104.internal   Ready     2d        v1.7.2-rancher1

NAME                 READY     STATUS    RESTARTS   AGE
po/glusterfs-0s440   0/1       Running   0          1m
po/glusterfs-j7dgr   0/1       Running   0          1m
po/glusterfs-p6jl3   0/1       Running   0          1m

NAME             CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
svc/kubernetes   10.43.0.1    <none>        443/TCP   2d


Also, from prerequisite perspective, i was also seeing this mentioned:

The following kernel modules must be loaded:
 * dm_snapshot
 * dm_mirror
 * dm_thin_pool

Where exactly is this to be checked? On all Gluster server nodes? How can i
check whether it's there?

I have attached topology.json and gk-deploy log for reference.

Does this issue has anything to do with the host OS (RancherOS) that i am
using for Gluster nodes? Any idea how i can fix this? Any help will really
be appreciated.


?Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170830/6c90513e/attachment.html>
-------------- next part --------------
[root at workstation deploy]# ./gk-deploy -g
Welcome to the deployment tool for GlusterFS on Kubernetes and OpenShift.

Before getting started, this script has some requirements of the execution
environment and of the container platform that you should verify.

The client machine that will run this script must have:
 * Administrative access to an existing Kubernetes or OpenShift cluster
 * Access to a python interpreter 'python'

Each of the nodes that will host GlusterFS must also have appropriate firewall
rules for the required GlusterFS ports:
 * 2222  - sshd (if running GlusterFS in a pod)
 * 24007 - GlusterFS Management
 * 24008 - GlusterFS RDMA
 * 49152 to 49251 - Each brick for every volume on the host requires its own
   port. For every new brick, one new port will be used starting at 49152. We
   recommend a default range of 49152-49251 on each host, though you can adjust
   this to fit your needs.

The following kernel modules must be loaded:
 * dm_snapshot
 * dm_mirror
 * dm_thin_pool

For systems with SELinux, the following settings need to be considered:
 * virt_sandbox_use_fusefs should be enabled on each node to allow writing to
   remote GlusterFS volumes

In addition, for an OpenShift deployment you must:
 * Have 'cluster_admin' role on the administrative account doing the
deployment
 * Add the 'default' and 'router' Service Accounts to the
'privileged' SCC
 * Have a router deployed that is configured to allow apps to access services
   running in the cluster

Do you wish to proceed with deployment?

[Y]es, [N]o? [Default: Y]: Y
Using Kubernetes CLI.
Using namespace "default".
Checking for pre-existing resources...
  GlusterFS pods ... not found.
  deploy-heketi pod ... not found.
  heketi pod ... not found.
Creating initial resources ... Error from server (AlreadyExists): error when
creating
"/root/gluster-kubernetes/deploy/kube-templates/heketi-service-account.yaml":
serviceaccounts "heketi-service-account" already exists
Error from server (AlreadyExists): clusterrolebindings.rbac.authorization.k8s.io
"heketi-sa-view" already exists
clusterrolebinding "heketi-sa-view" not labeled
OK
node "node-1.c.kubernetes-174104.internal" labeled
node "node-2.c.kubernetes-174104.internal" labeled
node "node-3.c.kubernetes-174104.internal" labeled
daemonset "glusterfs" created
Waiting for GlusterFS pods to start ... pods not found.
[root at workstation deploy]#
-------------- next part --------------
A non-text attachment was scrubbed...
Name: topology.json
Type: application/json
Size: 790 bytes
Desc: not available
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170830/6c90513e/attachment.json>