Martin Toth
2016-Nov-25 15:11 UTC
[Gluster-users] Unable to start/deploy VMs after Qemu/Gluster upgrade to 2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1
Hello all, we are using your qemu packages to deploy qemu VMs on our gluster via gfsapi. Recent upgrade broken our qemu and we are not able to deploy / start VMs anymore. Gluster is running OK, mounted with FUSE, everything looks ok, there is probably some problem with qemu while accessing gluster with gfsapi. These are our current versions (Qemu is from https://launchpad.net/~monotek/+archive/ubuntu/qemu-glusterfs-3.7 ): ii glusterfs-client 3.7.17-ubuntu1~trusty2 amd64 clustered file-system (client package) ii glusterfs-common 3.7.17-ubuntu1~trusty2 amd64 GlusterFS common libraries and translator modules ii glusterfs-server 3.7.17-ubuntu1~trusty2 amd64 clustered file-system (server package) ii qemu-keymaps 2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 all QEMU keyboard maps ii qemu-kvm 2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 amd64 QEMU Full virtualization ii qemu-system-common 2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 amd64 QEMU full system emulation binaries (common files) ii qemu-system-x86 2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 amd64 QEMU full system emulation binaries (x86) ii qemu-utils 2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1 amd64 QEMU utilities We see error attached lower in mail. Do you have any suggestions what can cause this problem ? Thanks in advance for your help. Regards, Martin Volume Name: vmvol Type: Replicate Volume ID: a72b5c9e-b8ff-488e-b10f-5ba4b71e62b8 Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: node1.storage.internal:/gluster/vmvol/brick1 Brick2: node2.storage.internal:/gluster/vmvol/brick1 Brick3: node3.storage.internal:/gluster/vmvol/brick1 Options Reconfigured: cluster.self-heal-daemon: on nfs.disable: on cluster.server-quorum-type: server cluster.quorum-type: auto network.remote-dio: enable performance.stat-prefetch: on performance.io-cache: off performance.read-ahead: off performance.quick-read: off cluster.eager-lock: enable storage.owner-gid: 9869 storage.owner-uid: 9869 server.allow-insecure: on performance.readdir-ahead: on 2016-11-25 12:54:06.121+0000: starting up LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name one-67 -S -machine pc-i440fx-trusty,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 3c703b26-3f57-44d0-8d76-bb281fd8902c -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-67.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=gluster://node1:24007/vmvol/67/disk.1,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 -drive file=/var/lib/one//datastores/112/67/disk.0,if=none,id=drive-ide0-0-1,readonly=on,format=raw,cache=none -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1,bootindex=3 -drive file=/var/lib/one//datastores/112/67/disk.2,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=24,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=02:00:0a:c8:64:1f,bus=pci.0,addr=0x3,bootindex=1 -vnc 0.0.0.0:67 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 [2016-11-25 12:54:06.254217] I [MSGID: 104045] [glfs-master.c:96:notify] 0-gfapi: New graph 6e6f6465-322d-3231-3330-392d32303136 (0) coming up [2016-11-25 12:54:06.254246] I [MSGID: 114020] [client.c:2113:notify] 0-vmvol-client-0: parent translators are ready, attempting connect on transport [2016-11-25 12:54:06.254473] I [MSGID: 114020] [client.c:2113:notify] 0-vmvol-client-1: parent translators are ready, attempting connect on transport [2016-11-25 12:54:06.254672] I [MSGID: 114020] [client.c:2113:notify] 0-vmvol-client-2: parent translators are ready, attempting connect on transport [2016-11-25 12:54:06.254844] I [rpc-clnt.c:1960:rpc_clnt_reconfig] 0-vmvol-client-0: changing port to 49152 (from 0) [2016-11-25 12:54:06.255303] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmvol-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-25 12:54:06.255391] I [rpc-clnt.c:1960:rpc_clnt_reconfig] 0-vmvol-client-2: changing port to 49152 (from 0) [2016-11-25 12:54:06.255844] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmvol-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-25 12:54:06.256259] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmvol-client-2: Connected to vmvol-client-2, attached to remote volume '/gluster/vmvol/brick1'. [2016-11-25 12:54:06.256268] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmvol-client-2: Server and Client lk-version numbers are not same, reopening the fds [2016-11-25 12:54:06.256288] I [MSGID: 108005] [afr-common.c:4299:afr_notify] 0-vmvol-replicate-0: Subvolume 'vmvol-client-2' came back up; going online. [2016-11-25 12:54:06.256464] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmvol-client-2: Server lk version = 1 [2016-11-25 12:54:06.262716] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmvol-client-0: Connected to vmvol-client-0, attached to remote volume '/gluster/vmvol/brick1'. [2016-11-25 12:54:06.262729] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmvol-client-0: Server and Client lk-version numbers are not same, reopening the fds [2016-11-25 12:54:06.262863] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmvol-client-0: Server lk version = 1 [2016-11-25 12:54:06.267906] I [rpc-clnt.c:1960:rpc_clnt_reconfig] 0-vmvol-client-1: changing port to 49152 (from 0) [2016-11-25 12:54:06.268148] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-vmvol-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2016-11-25 12:54:06.287724] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-vmvol-client-1: Connected to vmvol-client-1, attached to remote volume '/gluster/vmvol/brick1'. [2016-11-25 12:54:06.287734] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-vmvol-client-1: Server and Client lk-version numbers are not same, reopening the fds [2016-11-25 12:54:06.313439] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-vmvol-client-1: Server lk version = 1 [2016-11-25 12:54:06.314113] I [MSGID: 108031] [afr-common.c:2071:afr_local_discovery_cbk] 0-vmvol-replicate-0: selecting local read_child vmvol-client-1 [2016-11-25 12:54:06.314928] I [MSGID: 104041] [glfs-resolve.c:890:__glfs_active_subvol] 0-vmvol: switched to graph 6e6f6465-322d-3231-3330-392d32303136 (0) [2016-11-25 12:54:16.331479] I [MSGID: 114021] [client.c:2122:notify] 0-vmvol-client-0: current graph is no longer active, destroying rpc_client [2016-11-25 12:54:16.331507] I [MSGID: 114021] [client.c:2122:notify] 0-vmvol-client-1: current graph is no longer active, destroying rpc_client [2016-11-25 12:54:16.331517] I [MSGID: 114018] [client.c:2037:client_rpc_notify] 0-vmvol-client-0: disconnected from vmvol-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2016-11-25 12:54:16.331531] I [MSGID: 114018] [client.c:2037:client_rpc_notify] 0-vmvol-client-1: disconnected from vmvol-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2016-11-25 12:54:16.331534] I [MSGID: 114021] [client.c:2122:notify] 0-vmvol-client-2: current graph is no longer active, destroying rpc_client [2016-11-25 12:54:16.331543] W [MSGID: 108001] [afr-common.c:4379:afr_notify] 0-vmvol-replicate-0: Client-quorum is not met [2016-11-25 12:54:16.331667] E [rpc-clnt.c:370:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f4dcee94502] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f4dcec642be] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f4dcec643ce] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f4dcec65aa4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f4dcec66360] ))))) 0-vmvol-client-2: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2016-11-25 12:54:16.331128 (xid=0x2f62) [2016-11-25 12:54:16.331684] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-vmvol-client-2: remote operation failed [Transport endpoint is not connected] [2016-11-25 12:54:16.331701] I [MSGID: 114018] [client.c:2037:client_rpc_notify] 0-vmvol-client-2: disconnected from vmvol-client-2. Client process will keep trying to connect to glusterd until brick's port is available [2016-11-25 12:54:16.331710] E [MSGID: 108006] [afr-common.c:4321:afr_notify] 0-vmvol-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2016-11-25 12:54:16.331712] E [MSGID: 114031] [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-0: remote operation failed [Transport endpoint is not connected] [2016-11-25 12:54:16.331727] E [MSGID: 114031] [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-1: remote operation failed [Transport endpoint is not connected] [2016-11-25 12:54:16.331735] E [MSGID: 114031] [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-2: remote operation failed [Transport endpoint is not connected] [2016-11-25 12:54:16.331749] E [MSGID: 114031] [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-0: remote operation failed [Transport endpoint is not connected] [2016-11-25 12:54:16.331756] E [MSGID: 114031] [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-1: remote operation failed [Transport endpoint is not connected] [2016-11-25 12:54:16.331762] E [MSGID: 114031] [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-vmvol-client-2: remote operation failed [Transport endpoint is not connected] [2016-11-25 12:54:16.331768] W [inode.c:1814:inode_table_destroy] (-->/usr/lib/x86_64-linux-gnu/libgfapi.so.0(glfs_fini+0x3cf) [0x7f4dd57ef49f] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy_all+0x51) [0x7f4dceebe4d1] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_table_destroy+0xd4) [0x7f4dceebe3d4] ) 0-gfapi: Active inode(0x7f4db83101a4) with refcount(1) found during cleanup [2016-11-25 12:54:16.331834] E [inode.c:468:__inode_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.17/xlator/cluster/replicate.so(afr_local_cleanup+0x128) [0x7f4dbb5dafd8] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x21) [0x7f4dceebc7d1] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x4a0b6) [0x7f4dceebc0b6] ) 0-: Assertion failed: inode->ref [2016-11-25 12:54:16.331876] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-gfapi: size=84 max=2 total=2 [2016-11-25 12:54:16.331924] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-gfapi: size=156 max=3 total=3 [2016-11-25 12:54:16.332037] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-gfapi: size=108 max=2 total=2 [2016-11-25 12:54:16.332046] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-vmvol-client-0: size=1300 max=2 total=3038 [2016-11-25 12:54:16.332054] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-vmvol-client-1: size=1300 max=2 total=3039 [2016-11-25 12:54:16.332060] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-vmvol-client-2: size=1300 max=3 total=3038 [2016-11-25 12:54:16.332068] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-vmvol-replicate-0: size=10556 max=4 total=3038 [2016-11-25 12:54:16.332107] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-vmvol-dht: size=1148 max=0 total=0 [2016-11-25 12:54:16.332141] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-vmvol-dht: size=2316 max=2 total=6 [2016-11-25 12:54:16.332200] I [MSGID: 101053] [mem-pool.c:636:mem_pool_destroy] 0-vmvol-readdir-ahead: size=60 max=0 total=0 [2016-11-25 12:54:16.332212] I [io-stats.c:2951:fini] 0-vmvol: io-stats translator unloaded [2016-11-25 12:54:16.332245] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 1 [2016-11-25 12:54:16.332259] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 2 qemu-system-x86_64: -drive file=gluster://node1:24007/vmvol/67/disk.1,if=none,id=drive-ide0-0-0,format=qcow2,cache=none: could not open disk image gluster://node1:24007/vmvol/67/disk.1: Could not read L1 table: Bad file descriptor 2016-11-25 12:54:17.199+0000: shutting down
Lindsay Mathieson
2016-Nov-25 15:47 UTC
[Gluster-users] Unable to start/deploy VMs after Qemu/Gluster upgrade to 2.0.0+dfsg-2ubuntu1.28glusterfs3.7.17trusty1
On 26/11/2016 1:11 AM, Martin Toth wrote:> Qemu is fromhttps://launchpad.net/~monotek/+archive/ubuntu/qemu-glusterfs-3.7Why? Proxmox qemu already has gluster support built in. -- Lindsay Mathieson -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161126/35e5d721/attachment.html>