thr3ads.net - Gluster users - [Gluster-users] Gluster + Cinder [Sep 2013]

If this information is useful, please help other people find it:
Share via:

Derek Yarnell

2013-Sep-27 00:12 UTC

[Gluster-users] Gluster + Cinder

I am trying to integrate Gluster (3.4.0-8) with a RDO deployment (Havana
pre-release) of OpenStack with Cinder.  I have a simple 3 node
depolyment (admin host, network host, hypervisor host).  The
cinder-volume is running on the admin host and has mounted the glusterfs
volume successfully.

Excuse me if I am confused on some of these points but the documentation
on Gluster and RHOS doesn't seem to cover the exact specifics of all
these components interactions.

So I can get Cinder and Gluster integrated enough to be able to create
volumes, delete volumes etc.  My understanding is that from the
cinder-volume node it will take that volume and export it to the
instance via iSCSI using the tgtd daemons.  The thing is that when I
attach the volume I get a local libvirt error below[1].  It seems that
libvirt is trying to attach a locally mounted volume file.  This is not
what I expected and I was wondering if anyone has done this integration
that might share some insights.

[1]
2013-09-26 20:03:43.320 8718 ERROR nova.openstack.common.rpc.amqp
[req-acb8d4fa-ab55-45e6-934f-5e723a958acd
92741ec5808647f787a7ea8317d1ca43 064d1904dec8428da2e30e3726b9b943]
Exception during message handling
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
Traceback (most recent call last):
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py",
line 461, in _process_data
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
**args)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py",
line 172, in dispatch
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 89, in
wrapped
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
payload)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in
wrapped
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
return f(self, context, *args, **kw)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 243,
in
decorated_function
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp     pass
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 229,
in
decorated_function
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
return function(self, context, *args, **kwargs)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 271,
in
decorated_function
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp     e,
sys.exc_info())
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 258,
in
decorated_function
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
return function(self, context, *args, **kwargs)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3575,
in attach_volume
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
context, instance, mountpoint)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3570,
in attach_volume
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
mountpoint, instance)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3612,
in _attach_volume
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
connector)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3602,
in _attach_volume
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
mountpoint)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
1072, in attach_volume
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
disk_dev)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
1059, in attach_volume
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
virt_dom.attachDeviceFlags(conf.to_xml(), flags)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 187, in
doit
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in
proxy_call
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp     rv
= execute(f,*args,**kwargs)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in
tworker
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp     rv
= meth(*args,**kwargs)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib64/python2.6/site-packages/libvirt.py", line 419, in
attachDeviceFlags
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp     if
ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed',
dom=self)
2013-09-26 20:03:43.320 8718 TRACE nova.openstack.common.rpc.amqp
libvirtError: cannot open file
'/var/lib/nova/mnt/0fb9dadec76b71926e1cdb9ca60e1dfe/volume-da84a50f-c0f3-4f55-91e4-6cd3bb5ddc6a':
No such file or directory


-- 
---
Derek T. Yarnell
University of Maryland
Institute for Advanced Computer Studies

Gowrishankar Rajaiyan

2013-Sep-27 05:28 UTC

head link

[Gluster-users] Gluster + Cinder

On 09/27/2013 05:42 AM, Derek Yarnell wrote:> I am trying to integrate Gluster (3.4.0-8) with a RDO deployment (Havana
> pre-release) of OpenStack with Cinder.  I have a simple 3 node
> depolyment (admin host, network host, hypervisor host).  The
> cinder-volume is running on the admin host and has mounted the glusterfs
> volume successfully.
>
> Excuse me if I am confused on some of these points but the documentation
> on Gluster and RHOS doesn't seem to cover the exact specifics of all
> these components interactions.
>
> So I can get Cinder and Gluster integrated enough to be able to create
> volumes, delete volumes etc.  My understanding is that from the
> cinder-volume node it will take that volume and export it to the
> instance via iSCSI using the tgtd daemons.  The thing is that when I
> attach the volume I get a local libvirt error below[1].  It seems that
> libvirt is trying to attach a locally mounted volume file.  This is not
> what I expected and I was wondering if anyone has done this integration
> that might share some insights.
>
Just checking if the pre-requisites are covered:
1. "setting storage.owner-gid: 165" and "storage.owner-uid:
165" on the
gluster volume for cinder.

2. The appropriate SELinux booleans[1] are set.

[1] Ref: https://bugzilla.redhat.com/show_bug.cgi?id=995139#c0


I just happened to install RDO (Havana pre-release). Will try this out 
and respond.

-shanks

Maciej Gałkiewicz

2013-Sep-27 05:50 UTC

head link

[Gluster-users] Gluster + Cinder

On 27 September 2013 02:12, Derek Yarnell <derek at umiacs.umd.edu>
wrote:> So I can get Cinder and Gluster integrated enough to be able to create
> volumes, delete volumes etc.  My understanding is that from the
> cinder-volume node it will take that volume and export it to the
> instance via iSCSI using the tgtd daemons.  The thing is that when I
> attach the volume I get a local libvirt error below[1].  It seems that
> libvirt is trying to attach a locally mounted volume file.  This is not
> what I expected and I was wondering if anyone has done this integration
> that might share some insights.
iSCSI is not used at all. All compute nodes mount glusterfs volume
where all cinder volumes are stored and export them to VMs as regular
files. This article should clear things up a little bit:
https://shellycloud.com/blog/2013/09/why-glusterfs-should-not-be-implemented-with-openstack

Notice that you are using Havana and the blog post is describing
situation in Grizzly with some insights in Havana.

regards
-- 
Maciej Ga?kiewicz
Shelly Cloud Sp. z o. o., Sysadmin
http://shellycloud.com/, macias at shellycloud.com
KRS: 0000440358 REGON: 101504426

Gluster users - Sep 2013 - Gluster + Cinder

[Gluster-users] Gluster + Cinder

[Gluster-users] Gluster + Cinder

[Gluster-users] Gluster + Cinder