Gluster users - Dec 2013 - Error when trying to connect to a gluster volume (with libvirt/libgfapi)

On a CentOS 6.5 server, I'm trying to use the newly added native
libvirt/libgfapi support (rather than fuse) for VM storage.

I (believe) I have my VM properly configured:

[root at kvmhost ~]# virsh dumpxml testvm

{--snip--}
<disk type='network' device='disk'>
      <driver name='qemu' type='raw'/>
      <source protocol='gluster'
name='kvmimages/testvm.img'>
        <host name='glusterhost' port='0'/>
      </source>
     <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x05'
function='0x0'/>
    </disk>
{--snip--}

[root at kvmhost ~]

However, I get the following error on my KVM host:

[root at kvmhost ~]# VIRSH_DEBUG=0 virsh start testvm
start: found option <domain>: testvm
start: <domain> trying as domain NAME
libvir: QEMU Driver error : Unable to read from monitor: Connection
reset by peer
error: Failed to start domain testvm
error: Unable to read from monitor: Connection reset by peer

[root at kvmhost ~]#

On my gluster server, I get the following errors:

[root at glusterhost ~]# tail /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
[2013-12-02 17:12:09.031364] I
[glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
Received get vol req
[2013-12-02 17:12:09.033233] I
[glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
Received get vol req
[2013-12-02 17:12:26.462655] E [rpcsvc.c:521:rpcsvc_handle_rpc_call]
0-glusterd: Request received from non-privileged port. Failing request
[2013-12-02 17:12:29.466987] E [rpcsvc.c:521:rpcsvc_handle_rpc_call]
0-glusterd: Request received from non-privileged port. Failing request
[2013-12-02 17:12:54.991104] E [rpcsvc.c:521:rpcsvc_handle_rpc_call]
0-glusterd: Request received from non-privileged port. Failing request
[2013-12-02 17:12:57.995071] E [rpcsvc.c:521:rpcsvc_handle_rpc_call]
0-glusterd: Request received from non-privileged port. Failing request
[2013-12-02 17:13:22.763704] I
[glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
Received get vol req
[2013-12-02 17:13:22.764385] I
[glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
Received get vol req
[2013-12-02 17:21:17.553601] E [rpcsvc.c:521:rpcsvc_handle_rpc_call]
0-glusterd: Request received from non-privileged port. Failing request
[2013-12-02 17:21:20.557649] E [rpcsvc.c:521:rpcsvc_handle_rpc_call]
0-glusterd: Request received from non-privileged port. Failing request
[root at glusterhost ~]#

I checked and I _do_ have allow-insecure set:

[root at glusterhost ~]# gluster volume info

Volume Name: kvmimages
Type: Distribute
Volume ID: 43a1d032-5339-42e0-b20f-690600c810ce
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: glusterhost:/gluster/kvmimages
Options Reconfigured:
server.allow-insecure: on
performance.open-behind: off
[root at glusterhost ~]#

Any suggestions?  Do I have my KVM VM configured improperly?  Or do I
need to make a gluster volume change?

Does anyone else have libvirt/libgfapi working with gluster 3.4.1?

-Jacob

On 12/02/2013 10:57 PM, Jacob Yundt wrote:
> On a CentOS 6.5 server, I'm trying to use the newly added native
> libvirt/libgfapi support (rather than fuse) for VM storage.
>
> I (believe) I have my VM properly configured:
>
> [root at kvmhost ~]# virsh dumpxml testvm
>
> {--snip--}
> <disk type='network' device='disk'>
>        <driver name='qemu' type='raw'/>
>        <source protocol='gluster'
name='kvmimages/testvm.img'>
>          <host name='glusterhost' port='0'/>
>        </source>
>       <target dev='vda' bus='virtio'/>
>        <address type='pci' domain='0x0000'
bus='0x00' slot='0x05'
> function='0x0'/>
>      </disk>
> {--snip--}
>
> [root at kvmhost ~]
>
> However, I get the following error on my KVM host:
>
> [root at kvmhost ~]# VIRSH_DEBUG=0 virsh start testvm
> start: found option <domain>: testvm
> start: <domain> trying as domain NAME
> libvir: QEMU Driver error : Unable to read from monitor: Connection
> reset by peer
> error: Failed to start domain testvm
> error: Unable to read from monitor: Connection reset by peer
>
> [root at kvmhost ~]#
>
> On my gluster server, I get the following errors:
>
> [root at glusterhost ~]# tail
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log
> [2013-12-02 17:12:09.031364] I
> [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
> Received get vol req
> [2013-12-02 17:12:09.033233] I
> [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd:
> Received get vol req
> [2013-12-02 17:12:26.462655] E [rpcsvc.c:521:rpcsvc_handle_rpc_call]
> 0-glusterd: Request received from non-privileged port. Failing request

As documented in release notes for 3.4.0:

<snip>


The following configuration changes are necessary for qemu and samba 
integration with libgfapi to work seamlessly:

        1) gluster volume set <volname> server.allow-insecure on

        2) Edit /etc/glusterfs/glusterd.vol to contain this line:
               option rpc-auth-allow-insecure on

       Post 2), restarting glusterd would be necessary.

</snip>

You seem to have got 1) covered. You would need to perform 2) to get 
this working.

HTH,
Vijay

Jacob Yundt <jyundt at gmail.com> kirjoitti 2.12.2013 kello 19.27:
> 
> I checked and I _do_ have allow-insecure set:
> 
Did you remember to put "option rpc-auth-allow-insecure on" into
/etc/glusterfs/glusterd.vol aswell?

-samuli