01.07.2016 11:21, Dmitry Melekhov ?????:> OK, in our case we solved this problem by installing 3.7.12 again and > restarting libvirtd after that. > Now we have all this working. > As I said this is centos7. > > I guess we did not restart libvirtd during upgrade to 3.7.11 too, so I > don't know why libvirt doesn't work > with 3.7.11 libs... :-( >Oops, by "solved" I mean that now we can start VMs :-D But they don't work, they dies: [2016-07-01 06:57:51.836023] I [MSGID: 108019] [afr-transaction.c:1339:afr_post_blocking_inodelk_cbk] 0-pool-replicate-0: Blocking inodelks failed. block I/O error in device 'drive-ide0-0-0': Operation not permitted (1) [2016-07-01 06:57:51.837022] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-pool-client-7: remote operation failed [Invalid argument] [2016-07-01 06:57:51.837086] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-pool-client-6: remote operation failed [Invalid argument] [2016-07-01 06:57:51.837306] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-pool-client-8: remote operation failed [Invalid argument] [2016-07-01 06:57:51.837658] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-pool-client-6: remote operation failed [Invalid argument] [2016-07-01 06:57:51.837940] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-pool-client-7: remote operation failed [Invalid argument] [2016-07-01 06:57:51.838533] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-pool-client-8: remote operation failed [Invalid argument] [2016-07-01 06:57:51.838583] W [MSGID: 108019] [afr-lk-common.c:1082:is_blocking_locks_count_sufficient] 0-pool-replicate-0: Unable to obtain blocking inode lock on even one child for gfid:c45544e7-f449-48cf-bbc1 -cfd9f3ebc401. End on gluster side in brick log: [2016-07-01 07:31:44.013224] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325937: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013306] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325938: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013410] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325939: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013485] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325940: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013535] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325941: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013628] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325943: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013703] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325944: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013732] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325945: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013734] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325946: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013782] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325947: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013793] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325948: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013895] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325949: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013918] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325950: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] So, gfapi is broken...
01.07.2016 11:35, Dmitry Melekhov ?????:> 01.07.2016 11:21, Dmitry Melekhov ?????: >> OK, in our case we solved this problem by installing 3.7.12 again and >> restarting libvirtd after that. >> Now we have all this working. >> As I said this is centos7. >> >> I guess we did not restart libvirtd during upgrade to 3.7.11 too, so >> I don't know why libvirt doesn't work >> with 3.7.11 libs... :-( >> >OK, finally downgrade to 3.7.11 is successful. Had to kill all gluster processess before downgrade, for some reason systemd systemctl does not stop them, and upgrade scripts from rpms too... Thank you!