Gluster users - Jun 2012 - Can't run KVM Virtual Machines on a Gluster volume

I just built a 2 node(4 bricks), Distributed-Replicated and everything mounts
fine.
Each node mounts using GlusterFS client on its hostname (mount -t glusterfs
hostname:VOLUME /virtual-machines)
When creating a new Virtual Machine using virt-manager it creates the file on
the storage, but when trying to power it On, it doesn't work and gives back
an error message.(See below. Yes the folder has full permission to All to
write.)

Has anyone actually was able to run it fine  on Gluster 3.3 ? I get the same
results on VMware ESXi, but I thought that as for KVM using GlusterFS client
that would work fine and better. If I mount the volume as NFS the exact same
thing happens.
Quiet frustrating !!!

Fernando

Unable to complete install: 'internal error Process exited while reading
console log output: char device redirected to /dev/pts/1
qemu-kvm: -drive
file=/virtual-machines/Ubuntu_12.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none:
could not open disk image /virtual-machines/Ubuntu_12.img: Permission denied
'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 44, in
cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 1903, in
do_install
    guest.start_install(False, meter=meter)
  File "/usr/lib/python2.6/site-packages/virtinst/Guest.py", line
1223, in start_install
    noboot)
  File "/usr/lib/python2.6/site-packages/virtinst/Guest.py", line
1291, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2064, in
createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed',
conn=self)
libvirtError: internal error Process exited while reading console log output:
char device redirected to /dev/pts/1
qemu-kvm: -drive
file=/virtual-machines/Ubuntu_12.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none:
could not open disk image /virtual-machines/Ubuntu_12.img: Permission denied


Regards,

Fernando Frediani


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120623/7347e5fc/attachment.html>

Thanks for the reply Adam, but I've tried also Raw and it had the exact same
error when trying to Power On the machine. Does it make any diference if
it's IDE or VirtIO and if cache is None or Writethrough ?

Regards,

Fernando
________________________________________
From: Adam Tygart [mozes at k-state.edu]
Sent: 24 June 2012 00:14
To: Fernando Frediani (Qube)
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] Can't run KVM Virtual Machines on a Gluster
volume

Fernando,

The qcow2 disk format requires the file to be opened with O_DIRECT.
This is unsupported in FUSE with a linux kernel below 3.4. You can use
the raw disk format, or use https://github.com/avati/liboindirect to
catch opens with O_DIRECT and change them to O_SYNC.

--
Adam Tygart

On Sat, Jun 23, 2012 at 3:16 PM, Fernando Frediani (Qube)
<fernando.frediani at qubenet.net> wrote:
> I just built a 2 node(4 bricks), Distributed-Replicated and everything
> mounts fine.
>
> Each node mounts using GlusterFS client on its hostname (mount ?t glusterfs
> hostname:VOLUME /virtual-machines)
>
> When creating a new Virtual Machine using virt-manager it creates the file
> on the storage, but when trying to power it On, it doesn?t work and gives
> back an error message.(See below. Yes the folder has full permission to All
> to write.)
>
>
>
> Has anyone actually was able to run it fine  on Gluster 3.3 ? I get the
same
> results on VMware ESXi, but I thought that as for KVM using GlusterFS
client
> that would work fine and better. If I mount the volume as NFS the exact
same
> thing happens.
>
> Quiet frustrating !!!
>
>
>
> Fernando
>
>
>
> Unable to complete install: 'internal error Process exited while
reading
> console log output: char device redirected to /dev/pts/1
>
> qemu-kvm: -drive
>
file=/virtual-machines/Ubuntu_12.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none:
> could not open disk image /virtual-machines/Ubuntu_12.img: Permission
denied
>
> '
>
>
>
> Traceback (most recent call last):
>
>   File "/usr/share/virt-manager/virtManager/asyncjob.py", line
44, in
> cb_wrapper
>
>     callback(asyncjob, *args, **kwargs)
>
>   File "/usr/share/virt-manager/virtManager/create.py", line
1903, in
> do_install
>
>     guest.start_install(False, meter=meter)
>
>   File "/usr/lib/python2.6/site-packages/virtinst/Guest.py", line
1223, in
> start_install
>
>     noboot)
>
>   File "/usr/lib/python2.6/site-packages/virtinst/Guest.py", line
1291, in
> _create_guest
>
>     dom = self.conn.createLinux(start_xml or final_xml, 0)
>
>   File "/usr/lib64/python2.6/site-packages/libvirt.py", line
2064, in
> createLinux
>
>     if ret is None:raise libvirtError('virDomainCreateLinux()
failed',
> conn=self)
>
> libvirtError: internal error Process exited while reading console log
> output: char device redirected to /dev/pts/1
>
> qemu-kvm: -drive
>
file=/virtual-machines/Ubuntu_12.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none:
> could not open disk image /virtual-machines/Ubuntu_12.img: Permission
denied
>
>
>
>
>
> Regards,
>
>
>
> Fernando Frediani
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>

On 06/23/2012 04:16 PM, Fernando Frediani (Qube) wrote:
> I just built a 2 node(4 bricks), Distributed-Replicated and everything
> mounts fine.
>
> Each node mounts using GlusterFS client on its hostname (mount ?t
> glusterfs hostname:VOLUME /virtual-machines)
>
> When creating a new Virtual Machine using virt-manager it creates the
> file on the storage, but when trying to power it On, it doesn?t work and
> gives back an error message.(See below. Yes the folder has full
> permission to All to write.)
>
Can you please post the complete client and server log files? What user 
is the virt-manager running as?

-Vijay

Use cache=writeback
---------- Messaggio inoltrato ----------
Da: "Marco Agostini" <comunelevico at gmail.com>
Data: 23/giu/2012 22:53
Oggetto: Re: [Gluster-users] Can't run KVM Virtual Machines on a Gluster
volume
A: "Fernando Frediani (Qube)" <fernando.frediani at qubenet.net>


Il giorno 23/giu/2012 22:16, "Fernando Frediani (Qube)" <
fernando.frediani at qubenet.net> ha scritto:
>
> qemu-kvm: -drive
file=/virtual-machines/Ubuntu_12.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none:
could not open disk image /virtual-machines/Ubuntu_12.img

Try to change cache=writeback
It's a limit using fuse.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120624/b0a2c6ce/attachment.html>

On Sat, Jun 23, 2012 at 08:16:47PM +0000, Fernando Frediani (Qube)
wrote:
>    I just built a 2 node(4 bricks), Distributed-Replicated and everything
>    mounts fine.
> 
>    Each node mounts using GlusterFS client on its hostname (mount ?t
>    glusterfs hostname:VOLUME /virtual-machines)
> 
>    When creating a new Virtual Machine using virt-manager it creates the
>    file on the storage, but when trying to power it On, it doesn?t work
>    and gives back an error message.(See below. Yes the folder has full
>    permission to All to write.)
Failing to access something, when the something has permissions, usually
implies a problem with apparmor / SElinux.

If this is an Ubuntu platform, then use libvirt to start the VM (i.e. define
it using an XML file), and it should automatically create an apparmor policy
dynamically.  Otherwise, try turning off apparmor / SElinux temporarily to
see whether or that's the problem.  You could also try mounting your volume
directly onto /var/lib/libvirt/images.
>    Has anyone actually was able to run it fine  on Gluster 3.3
I have. Ubuntu 12.04.