Gluster users - Dec 2014 - [ovirt-users] Gluster command [<UNKNOWN>] failed on server...

Based on the logs I can guess that glusterd is being started before
the network has come up and that the addresses given to bricks do not
directly match the addresses used in during peer probe.

The gluster_after_reboot log has the line "[2014-11-25
06:46:09.972113] E [glusterd-store.c:2632:glusterd_resolve_all_bricks]
0-glusterd: resolve brick failed in restore".

Brick resolution fails when glusterd cannot match the address for the
brick, with one of the peers. Brick resolution happens in two phases,
1. We first try to identify the peer by performing string comparisions
with the brick address and the peer addresses (The peer names will be
the names/addresses that were given when the peer was probed).
2. If we don't find a match from step 1, we will then resolve all the
brick address and the peer addresses into addrinfo structs, and then
compare these structs to find a match. This process should generally
find a match if available. This will fail only if the network is not
up yet as we cannot resolve addresses.

The above steps are applicable only to glusterfs versions >=3.6. They
were introduced to reduce problems with peer identification, like the
one you encountered

Since both of the steps failed to find a match in one run, but
succeeded later, we can come to the conclusion that,
a) the bricks don't have the exact same string used in peer probe for
their addresses as step 1 failed, and
b) the network was not up in the initial run, as step 2 failed during
the initial run, but passed in the second run.

Please let me know if my conclusion is correct.

If it is, you can solve your problem in two ways.
1. Use the same string for doing the peer probe and for the brick
address during volume create/add-brick. Ideally, we suggest you use
properly resolvable FQDNs everywhere. If that is not possible, then
use only IP addresses. Try to avoid short names.
2. During boot up, make sure to launch glusterd only after the network
is up. This will allow the new peer identification mechanism to do its
job correctly.


If you have already followed these steps and yet still hit the
problem, then please provide more information (setup, logs, etc.). It
could be much different problem that you are facing.

~kaushal

On Wed, Nov 26, 2014 at 4:01 PM, Punit Dambiwal <hypunit at gmail.com>
wrote:
> Is there any one can help on this ??
>
> Thanks,
> punit
>
> On Wed, Nov 26, 2014 at 9:42 AM, Punit Dambiwal <hypunit at
gmail.com> wrote:
>>
>> Hi,
>>
>> My Glusterfs version is :- glusterfs-3.6.1-1.el7
>>
>> On Wed, Nov 26, 2014 at 1:59 AM, Kanagaraj Mayilsamy <kmayilsa at
redhat.com>
>> wrote:
>>>
>>> [+Gluster-users at gluster.org]
>>>
>>> "Initialization of volume 'management' failed, review
your volfile
>>> again", glusterd throws this error when the service is started
automatically
>>> after the reboot. But the service is successfully started later
manually by
>>> the user.
>>>
>>> can somebody from gluster-users please help on this?
>>>
>>> glusterfs version: 3.5.1
>>>
>>> Thanks,
>>> Kanagaraj
>>>
>>> ----- Original Message -----
>>> > From: "Punit Dambiwal" <hypunit at gmail.com>
>>> > To: "Kanagaraj" <kmayilsa at redhat.com>
>>> > Cc: users at ovirt.org
>>> > Sent: Tuesday, November 25, 2014 7:24:45 PM
>>> > Subject: Re: [ovirt-users] Gluster command [<UNKNOWN>]
failed on
>>> > server...
>>> >
>>> > Hi Kanagraj,
>>> >
>>> > Please check the attached log files....i didn't find any
thing
>>> > special....
>>> >
>>> > On Tue, Nov 25, 2014 at 12:12 PM, Kanagaraj <kmayilsa at
redhat.com>
>>> > wrote:
>>> >
>>> > >  Do you see any errors in
>>> > > /var/log/glusterfs/etc-glusterfs-glusterd.vol.log or
vdsm.log when
>>> > > the
>>> > > service is trying to start automatically after the
reboot?
>>> > >
>>> > > Thanks,
>>> > > Kanagaraj
>>> > >
>>> > >
>>> > > On 11/24/2014 08:13 PM, Punit Dambiwal wrote:
>>> > >
>>> > > Hi Kanagaraj,
>>> > >
>>> > >  Yes...once i will start the gluster service and then
vdsmd ...the
>>> > > host
>>> > > can connect to cluster...but the question is why it's
not started
>>> > > even it
>>> > > has chkconfig enabled...
>>> > >
>>> > >  I have tested it in two host cluster
environment...(Centos 6.6 and
>>> > > centos 7.0) on both hypervisior cluster..it's failed
to reconnect in
>>> > > to
>>> > > cluster after reboot....
>>> > >
>>> > >  In both the environment glusterd enabled for next
boot....but it's
>>> > > failed with the same error....seems it's bug in
either gluster or
>>> > > Ovirt ??
>>> > >
>>> > >  Please help me to find the workaround here if can not
resolve
>>> > > it...as
>>> > > without this the Host machine can not connect after
reboot....that
>>> > > means
>>> > > engine will consider it as down and every time need to
manually start
>>> > > the
>>> > > gluster service and vdsmd... ??
>>> > >
>>> > >  Thanks,
>>> > > Punit
>>> > >
>>> > > On Mon, Nov 24, 2014 at 10:20 PM, Kanagaraj <kmayilsa
at redhat.com>
>>> > > wrote:
>>> > >
>>> > >>  From vdsm.log "error: Connection failed. Please
check if gluster
>>> > >> daemon
>>> > >> is operational."
>>> > >>
>>> > >> Starting glusterd service should fix this issue.
'service glusterd
>>> > >> start'
>>> > >> But i am wondering why the glusterd was not started
automatically
>>> > >> after
>>> > >> the reboot.
>>> > >>
>>> > >> Thanks,
>>> > >> Kanagaraj
>>> > >>
>>> > >>
>>> > >>
>>> > >> On 11/24/2014 07:18 PM, Punit Dambiwal wrote:
>>> > >>
>>> > >> Hi Kanagaraj,
>>> > >>
>>> > >>  Please find the attached VDSM logs :-
>>> > >>
>>> > >>  ----------------
>>> > >>  Thread-13::DEBUG::2014-11-24
>>> > >>
>>> > >>
21:41:17,182::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
>>> > >> Owner.cancelAll requests {}
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >>
21:41:17,182::task::993::Storage.TaskManager.Task::(_decref)
>>> > >> Task=`1691d409-9b27-4585-8281-5ec26154367a`::ref 0
aborting False
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >>
21:41:32,393::task::595::Storage.TaskManager.Task::(_updateState)
>>> > >> Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving
from state init
>>> > >> ->
>>> > >> state preparing
>>> > >> Thread-13::INFO::2014-11-24
>>> > >> 21:41:32,393::logUtils::44::dispatcher::(wrapper) Run
and protect:
>>> > >> repoStats(options=None)
>>> > >> Thread-13::INFO::2014-11-24
>>> > >> 21:41:32,393::logUtils::47::dispatcher::(wrapper) Run
and protect:
>>> > >> repoStats, Return response: {}
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >>
21:41:32,393::task::1191::Storage.TaskManager.Task::(prepare)
>>> > >>
Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::finished: {}
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >>
21:41:32,394::task::595::Storage.TaskManager.Task::(_updateState)
>>> > >> Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving
from state
>>> > >> preparing
>>> > >> ->
>>> > >> state finished
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >>
>>> > >>
21:41:32,394::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
>>> > >> Owner.releaseAll requests {} resources {}
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >>
>>> > >>
21:41:32,394::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
>>> > >> Owner.cancelAll requests {}
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >>
21:41:32,394::task::993::Storage.TaskManager.Task::(_decref)
>>> > >> Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::ref 0
aborting False
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,550::BindingXMLRPC::1132::vds::(wrapper)
client
>>> > >> [10.10.10.2]::call
>>> > >> getCapabilities with () {}
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,553::utils::738::root::(execCmd)
>>> > >> /sbin/ip route show to 0.0.0.0/0 table all (cwd None)
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,560::utils::758::root::(execCmd)
>>> > >> SUCCESS: <err> = ''; <rc> = 0
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,588::caps::728::root::(_getKeyPackages) rpm
package
>>> > >> ('gluster-swift',) not found
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,592::caps::728::root::(_getKeyPackages) rpm
package
>>> > >> ('gluster-swift-object',) not found
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,593::caps::728::root::(_getKeyPackages) rpm
package
>>> > >> ('gluster-swift-plugin',) not found
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,598::caps::728::root::(_getKeyPackages) rpm
package
>>> > >> ('gluster-swift-account',) not found
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,598::caps::728::root::(_getKeyPackages) rpm
package
>>> > >> ('gluster-swift-proxy',) not found
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,598::caps::728::root::(_getKeyPackages) rpm
package
>>> > >> ('gluster-swift-doc',) not found
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,599::caps::728::root::(_getKeyPackages) rpm
package
>>> > >> ('gluster-swift-container',) not found
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,599::caps::728::root::(_getKeyPackages) rpm
package
>>> > >> ('glusterfs-geo-replication',) not found
>>> > >> Thread-13::DEBUG::2014-11-24
21:41:41,600::caps::646::root::(get)
>>> > >> VirtioRNG DISABLED: libvirt version 0.10.2-29.el6_5.9
required >>>> > >> 0.10.2-31
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,603::BindingXMLRPC::1139::vds::(wrapper)
return
>>> > >> getCapabilities
>>> > >> with {'status': {'message':
'Done', 'code': 0}, 'info':
>>> > >> {'HBAInventory':
>>> > >> {'iSCSI': [{'InitiatorName':
>>> > >> 'iqn.1994-05.com.redhat:32151ce183c8'}],
>>> > >> 'FC':
>>> > >> []}, 'packages2': {'kernel':
{'release': '431.el6.x86_64',
>>> > >> 'buildtime':
>>> > >> 1385061309.0, 'version': '2.6.32'},
'glusterfs-rdma': {'release':
>>> > >> '1.el6',
>>> > >> 'buildtime': 1403622628L, 'version':
'3.5.1'}, 'glusterfs-fuse':
>>> > >> {'release': '1.el6',
'buildtime': 1403622628L, 'version': '3.5.1'},
>>> > >> 'spice-server': {'release':
'6.el6_5.2', 'buildtime': 1402324637L,
>>> > >> 'version': '0.12.4'}, 'vdsm':
{'release': '1.gitdb83943.el6',
>>> > >> 'buildtime':
>>> > >> 1412784567L, 'version': '4.16.7'},
'qemu-kvm': {'release':
>>> > >> '2.415.el6_5.10', 'buildtime':
1402435700L, 'version': '0.12.1.2'},
>>> > >> 'qemu-img': {'release':
'2.415.el6_5.10', 'buildtime': 1402435700L,
>>> > >> 'version': '0.12.1.2'},
'libvirt': {'release': '29.el6_5.9',
>>> > >> 'buildtime':
>>> > >> 1402404612L, 'version': '0.10.2'},
'glusterfs': {'release': '1.el6',
>>> > >> 'buildtime': 1403622628L, 'version':
'3.5.1'}, 'mom': {'release':
>>> > >> '2.el6',
>>> > >> 'buildtime': 1403794344L, 'version':
'0.4.1'}, 'glusterfs-server':
>>> > >> {'release': '1.el6',
'buildtime': 1403622628L, 'version': '3.5.1'}},
>>> > >> 'numaNodeDistance': {'1': [20, 10],
'0': [10, 20]}, 'cpuModel':
>>> > >> 'Intel(R)
>>> > >> Xeon(R) CPU           X5650  @ 2.67GHz',
'liveMerge': 'false',
>>> > >> 'hooks':
>>> > >> {},
>>> > >> 'cpuSockets': '2', 'vmTypes':
['kvm'], 'selinux': {'mode': '1'},
>>> > >> 'kdumpStatus': 0,
'supportedProtocols': ['2.2', '2.3'],
'networks':
>>> > >> {'ovirtmgmt': {'iface':
u'bond0.10', 'addr': '43.252.176.16',
>>> > >> 'bridged':
>>> > >> False, 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'mtu': '1500',
>>> > >> 'bootproto4': 'none',
'netmask': '255.255.255.0', 'ipv4addrs': ['
>>> > >> 43.252.176.16/24'
<http://43.252.176.16/24%27>], 'interface':
>>> > >> u'bond0.10', 'ipv6gateway':
'::', 'gateway': '43.25.17.1'},
>>> > >> 'Internal':
>>> > >> {'iface': 'Internal', 'addr':
'', 'cfg': {'DEFROUTE': 'no',
>>> > >> 'HOTPLUG':
>>> > >> 'no', 'MTU': '9000',
'DELAY': '0', 'NM_CONTROLLED': 'no',
>>> > >> 'BOOTPROTO':
>>> > >> 'none', 'STP': 'off',
'DEVICE': 'Internal', 'TYPE': 'Bridge',
>>> > >> 'ONBOOT':
>>> > >> 'no'}, 'bridged': True,
'ipv6addrs':
>>> > >> ['fe80::210:18ff:fecd:daac/64'],
>>> > >> 'gateway': '', 'bootproto4':
'none', 'netmask': '', 'stp': 'off',
>>> > >> 'ipv4addrs': [], 'mtu':
'9000', 'ipv6gateway': '::', 'ports':
>>> > >> ['bond1.100']}, 'storage':
{'iface': u'bond1', 'addr': '10.10.10.6',
>>> > >> 'bridged': False, 'ipv6addrs':
['fe80::210:18ff:fecd:daac/64'],
>>> > >> 'mtu':
>>> > >> '9000', 'bootproto4': 'none',
'netmask': '255.255.255.0',
>>> > >> 'ipv4addrs': ['
>>> > >> 10.10.10.6/24' <http://10.10.10.6/24%27>],
'interface': u'bond1',
>>> > >> 'ipv6gateway': '::',
'gateway': ''}, 'VMNetwork': {'iface':
>>> > >> 'VMNetwork',
>>> > >> 'addr': '', 'cfg':
{'DEFROUTE': 'no', 'HOTPLUG': 'no',
'MTU':
>>> > >> '1500',
>>> > >> 'DELAY': '0',
'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none',
'STP':
>>> > >> 'off',
>>> > >> 'DEVICE': 'VMNetwork',
'TYPE': 'Bridge', 'ONBOOT': 'no'},
'bridged':
>>> > >> True,
>>> > >> 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'gateway': '',
>>> > >> 'bootproto4':
>>> > >> 'none', 'netmask': '',
'stp': 'off', 'ipv4addrs': [], 'mtu':
'1500',
>>> > >> 'ipv6gateway': '::', 'ports':
['bond0.36']}}, 'bridges':
>>> > >> {'Internal':
>>> > >> {'addr': '', 'cfg':
{'DEFROUTE': 'no', 'HOTPLUG': 'no',
'MTU':
>>> > >> '9000',
>>> > >> 'DELAY': '0',
'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none',
'STP':
>>> > >> 'off',
>>> > >> 'DEVICE': 'Internal', 'TYPE':
'Bridge', 'ONBOOT': 'no'},
>>> > >> 'ipv6addrs':
>>> > >> ['fe80::210:18ff:fecd:daac/64'],
'mtu': '9000', 'netmask': '',
>>> > >> 'stp':
>>> > >> 'off', 'ipv4addrs': [],
'ipv6gateway': '::', 'gateway': '',
'opts':
>>> > >> {'topology_change_detected': '0',
'multicast_last_member_count':
>>> > >> '2',
>>> > >> 'hash_elasticity': '4',
'multicast_query_response_interval': '999',
>>> > >> 'multicast_snooping': '1',
'multicast_startup_query_interval':
>>> > >> '3124',
>>> > >> 'hello_timer': '31',
'multicast_querier_interval': '25496',
>>> > >> 'max_age':
>>> > >> '1999', 'hash_max': '512',
'stp_state': '0', 'root_id':
>>> > >> '8000.001018cddaac', 'priority':
'32768',
>>> > >> 'multicast_membership_interval':
>>> > >> '25996', 'root_path_cost':
'0', 'root_port': '0',
>>> > >> 'multicast_querier':
>>> > >> '0',
>>> > >> 'multicast_startup_query_count': '2',
'hello_time': '199',
>>> > >> 'topology_change': '0',
'bridge_id': '8000.001018cddaac',
>>> > >> 'topology_change_timer': '0',
'ageing_time': '29995', 'gc_timer':
>>> > >> '31',
>>> > >> 'group_addr': '1:80:c2:0:0:0',
'tcn_timer': '0',
>>> > >> 'multicast_query_interval': '12498',
>>> > >> 'multicast_last_member_interval':
>>> > >> '99', 'multicast_router':
'1', 'forward_delay': '0'}, 'ports':
>>> > >> ['bond1.100']}, 'VMNetwork':
{'addr': '', 'cfg': {'DEFROUTE': 'no',
>>> > >> 'HOTPLUG': 'no', 'MTU':
'1500', 'DELAY': '0', 'NM_CONTROLLED':
'no',
>>> > >> 'BOOTPROTO': 'none', 'STP':
'off', 'DEVICE': 'VMNetwork', 'TYPE':
>>> > >> 'Bridge',
>>> > >> 'ONBOOT': 'no'}, 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'],
>>> > >> 'mtu':
>>> > >> '1500', 'netmask': '',
'stp': 'off', 'ipv4addrs': [], 'ipv6gateway':
>>> > >> '::',
>>> > >> 'gateway': '', 'opts':
{'topology_change_detected': '0',
>>> > >> 'multicast_last_member_count': '2',
'hash_elasticity': '4',
>>> > >> 'multicast_query_response_interval':
'999', 'multicast_snooping':
>>> > >> '1',
>>> > >> 'multicast_startup_query_interval':
'3124', 'hello_timer': '131',
>>> > >> 'multicast_querier_interval':
'25496', 'max_age': '1999',
>>> > >> 'hash_max':
>>> > >> '512', 'stp_state': '0',
'root_id': '8000.60eb6920b46c', 'priority':
>>> > >> '32768',
'multicast_membership_interval': '25996',
'root_path_cost':
>>> > >> '0',
>>> > >> 'root_port': '0',
'multicast_querier': '0',
>>> > >> 'multicast_startup_query_count': '2',
'hello_time': '199',
>>> > >> 'topology_change': '0',
'bridge_id': '8000.60eb6920b46c',
>>> > >> 'topology_change_timer': '0',
'ageing_time': '29995', 'gc_timer':
>>> > >> '31',
>>> > >> 'group_addr': '1:80:c2:0:0:0',
'tcn_timer': '0',
>>> > >> 'multicast_query_interval': '12498',
>>> > >> 'multicast_last_member_interval':
>>> > >> '99', 'multicast_router':
'1', 'forward_delay': '0'}, 'ports':
>>> > >> ['bond0.36']}}, 'uuid':
'44454C4C-4C00-1057-8053-B7C04F504E31',
>>> > >> 'lastClientIface': 'bond1',
'nics': {'eth3': {'permhwaddr':
>>> > >> '00:10:18:cd:da:ae', 'addr':
'', 'cfg': {'SLAVE': 'yes',
>>> > >> 'NM_CONTROLLED':
>>> > >> 'no', 'MTU': '9000',
'HWADDR': '00:10:18:cd:da:ae', 'MASTER':
>>> > >> 'bond1',
>>> > >> 'DEVICE': 'eth3', 'ONBOOT':
'no'}, 'ipv6addrs': [], 'mtu': '9000',
>>> > >> 'netmask': '', 'ipv4addrs':
[], 'hwaddr': '00:10:18:cd:da:ac',
>>> > >> 'speed':
>>> > >> 1000}, 'eth2': {'permhwaddr':
'00:10:18:cd:da:ac', 'addr': '',
>>> > >> 'cfg':
>>> > >> {'SLAVE': 'yes',
'NM_CONTROLLED': 'no', 'MTU': '9000',
'HWADDR':
>>> > >> '00:10:18:cd:da:ac', 'MASTER':
'bond1', 'DEVICE': 'eth2', 'ONBOOT':
>>> > >> 'no'},
>>> > >> 'ipv6addrs': [], 'mtu':
'9000', 'netmask': '', 'ipv4addrs': [],
>>> > >> 'hwaddr':
>>> > >> '00:10:18:cd:da:ac', 'speed': 1000},
'eth1': {'permhwaddr':
>>> > >> '60:eb:69:20:b4:6d', 'addr':
'', 'cfg': {'SLAVE': 'yes',
>>> > >> 'NM_CONTROLLED':
>>> > >> 'no', 'MTU': '1500',
'HWADDR': '60:eb:69:20:b4:6d', 'MASTER':
>>> > >> 'bond0',
>>> > >> 'DEVICE': 'eth1', 'ONBOOT':
'yes'}, 'ipv6addrs': [], 'mtu': '1500',
>>> > >> 'netmask': '', 'ipv4addrs':
[], 'hwaddr': '60:eb:69:20:b4:6c',
>>> > >> 'speed':
>>> > >> 1000}, 'eth0': {'permhwaddr':
'60:eb:69:20:b4:6c', 'addr': '',
>>> > >> 'cfg':
>>> > >> {'SLAVE': 'yes',
'NM_CONTROLLED': 'no', 'MTU': '1500',
'HWADDR':
>>> > >> '60:eb:69:20:b4:6c', 'MASTER':
'bond0', 'DEVICE': 'eth0', 'ONBOOT':
>>> > >> 'yes'},
>>> > >> 'ipv6addrs': [], 'mtu':
'1500', 'netmask': '', 'ipv4addrs': [],
>>> > >> 'hwaddr':
>>> > >> '60:eb:69:20:b4:6c', 'speed': 1000}},
'software_revision': '1',
>>> > >> 'clusterLevels': ['3.0',
'3.1', '3.2', '3.3', '3.4', '3.5'],
>>> > >> 'cpuFlags':
>>> > >>
>>> > >>
u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,popcnt,aes,lahf_lm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270',
>>> > >> 'ISCSIInitiatorName':
'iqn.1994-05.com.redhat:32151ce183c8',
>>> > >> 'netConfigDirty': 'False',
'supportedENGINEs': ['3.0', '3.1', '3.2',
>>> > >> '3.3',
>>> > >> '3.4', '3.5'],
'autoNumaBalancing': 2, 'reservedMem': '321',
>>> > >> 'bondings':
>>> > >> {'bond4': {'addr': '',
'cfg': {}, 'mtu': '1500', 'netmask': '',
>>> > >> 'slaves':
>>> > >> [], 'hwaddr': '00:00:00:00:00:00'},
'bond0': {'addr': '', 'cfg':
>>> > >> {'HOTPLUG': 'no', 'MTU':
'1500', 'NM_CONTROLLED': 'no',
>>> > >> 'BONDING_OPTS':
>>> > >> 'mode=4 miimon=100', 'DEVICE':
'bond0', 'ONBOOT': 'yes'},
>>> > >> 'ipv6addrs':
>>> > >> ['fe80::62eb:69ff:fe20:b46c/64'],
'mtu': '1500', 'netmask': '',
>>> > >> 'ipv4addrs': [], 'hwaddr':
'60:eb:69:20:b4:6c', 'slaves': ['eth0',
>>> > >> 'eth1'],
>>> > >> 'opts': {'miimon': '100',
'mode': '4'}}, 'bond1': {'addr':
>>> > >> '10.10.10.6',
>>> > >> 'cfg': {'DEFROUTE': 'no',
'IPADDR': '10.10.10.6', 'HOTPLUG': 'no',
>>> > >> 'MTU':
>>> > >> '9000', 'NM_CONTROLLED':
'no', 'NETMASK': '255.255.255.0',
>>> > >> 'BOOTPROTO':
>>> > >> 'none', 'BONDING_OPTS': 'mode=4
miimon=100', 'DEVICE': 'bond1',
>>> > >> 'ONBOOT':
>>> > >> 'no'}, 'ipv6addrs':
['fe80::210:18ff:fecd:daac/64'], 'mtu': '9000',
>>> > >> 'netmask': '255.255.255.0',
'ipv4addrs': ['10.10.10.6/24'
>>> > >> <http://10.10.10.6/24%27>], 'hwaddr':
'00:10:18:cd:da:ac', 'slaves':
>>> > >> ['eth2', 'eth3'], 'opts':
{'miimon': '100', 'mode': '4'}},
'bond2':
>>> > >> {'addr': '', 'cfg': {},
'mtu': '1500', 'netmask': '', 'slaves':
[],
>>> > >> 'hwaddr': '00:00:00:00:00:00'},
'bond3': {'addr': '', 'cfg': {},
>>> > >> 'mtu':
>>> > >> '1500', 'netmask': '',
'slaves': [], 'hwaddr':
>>> > >> '00:00:00:00:00:00'}},
>>> > >> 'software_version': '4.16',
'memSize': '24019', 'cpuSpeed':
>>> > >> '2667.000',
>>> > >> 'numaNodes': {u'1':
{'totalMemory': '12288', 'cpus': [6, 7, 8, 9,
>>> > >> 10, 11,
>>> > >> 18, 19, 20, 21, 22, 23]}, u'0':
{'totalMemory': '12278', 'cpus': [0,
>>> > >> 1, 2,
>>> > >> 3, 4, 5, 12, 13, 14, 15, 16, 17]}},
'version_name': 'Snow Man',
>>> > >> 'vlans':
>>> > >> {'bond0.10': {'iface':
'bond0', 'addr': '43.25.17.16', 'cfg':
>>> > >> {'DEFROUTE':
>>> > >> 'yes', 'VLAN': 'yes',
'IPADDR': '43.25.17.16', 'HOTPLUG': 'no',
>>> > >> 'GATEWAY':
>>> > >> '43.25.17.1', 'NM_CONTROLLED':
'no', 'NETMASK': '255.255.255.0',
>>> > >> 'BOOTPROTO': 'none',
'DEVICE': 'bond0.10', 'MTU': '1500',
'ONBOOT':
>>> > >> 'yes'},
>>> > >> 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid': 10, 'mtu':
>>> > >> '1500',
>>> > >> 'netmask': '255.255.255.0',
'ipv4addrs': ['43.25.17.16/24']
>>> > >> <http://43.25.17.16/24%27%5D>},
'bond0.36': {'iface': 'bond0',
>>> > >> 'addr':
>>> > >> '', 'cfg': {'BRIDGE':
'VMNetwork', 'VLAN': 'yes', 'HOTPLUG':
'no',
>>> > >> 'MTU':
>>> > >> '1500', 'NM_CONTROLLED':
'no', 'DEVICE': 'bond0.36', 'ONBOOT':
>>> > >> 'no'},
>>> > >> 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid': 36, 'mtu':
>>> > >> '1500',
>>> > >> 'netmask': '', 'ipv4addrs':
[]}, 'bond1.100': {'iface': 'bond1',
>>> > >> 'addr':
>>> > >> '', 'cfg': {'BRIDGE':
'Internal', 'VLAN': 'yes', 'HOTPLUG':
'no',
>>> > >> 'MTU':
>>> > >> '9000', 'NM_CONTROLLED':
'no', 'DEVICE': 'bond1.100', 'ONBOOT':
>>> > >> 'no'},
>>> > >> 'ipv6addrs':
['fe80::210:18ff:fecd:daac/64'], 'vlanid': 100, 'mtu':
>>> > >> '9000',
>>> > >> 'netmask': '', 'ipv4addrs':
[]}}, 'cpuCores': '12', 'kvmEnabled':
>>> > >> 'true',
>>> > >> 'guestOverhead': '65',
'cpuThreads': '24', 'emulatedMachines':
>>> > >> [u'rhel6.5.0', u'pc',
u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0',
>>> > >> u'rhel6.1.0', u'rhel6.0.0',
u'rhel5.5.0', u'rhel5.4.4',
>>> > >> u'rhel5.4.0'],
>>> > >> 'operatingSystem': {'release':
'5.el6.centos.11.1', 'version': '6',
>>> > >> 'name':
>>> > >> 'RHEL'}, 'lastClient':
'10.10.10.2'}}
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,620::BindingXMLRPC::1132::vds::(wrapper)
client
>>> > >> [10.10.10.2]::call
>>> > >> getHardwareInfo with () {}
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,621::BindingXMLRPC::1139::vds::(wrapper)
return
>>> > >> getHardwareInfo
>>> > >> with {'status': {'message':
'Done', 'code': 0}, 'info':
>>> > >> {'systemProductName': 'CS24-TY',
'systemSerialNumber': '7LWSPN1',
>>> > >> 'systemFamily': 'Server',
'systemVersion': 'A00', 'systemUUID':
>>> > >> '44454c4c-4c00-1057-8053-b7c04f504e31',
'systemManufacturer':
>>> > >> 'Dell'}}
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >> 21:41:41,733::BindingXMLRPC::1132::vds::(wrapper)
client
>>> > >> [10.10.10.2]::call
>>> > >> hostsList with () {} flowID [222e8036]
>>> > >> Thread-13::ERROR::2014-11-24
>>> > >> 21:41:44,753::BindingXMLRPC::1148::vds::(wrapper)
vdsm exception
>>> > >> occured
>>> > >> Traceback (most recent call last):
>>> > >>   File
"/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1135, in wrapper
>>> > >>     res = f(*args, **kwargs)
>>> > >>   File "/usr/share/vdsm/gluster/api.py",
line 54, in wrapper
>>> > >>     rv = func(*args, **kwargs)
>>> > >>   File "/usr/share/vdsm/gluster/api.py",
line 251, in hostsList
>>> > >>     return {'hosts':
self.svdsmProxy.glusterPeerStatus()}
>>> > >>   File "/usr/share/vdsm/supervdsm.py", line
50, in __call__
>>> > >>     return callMethod()
>>> > >>   File "/usr/share/vdsm/supervdsm.py", line
48, in <lambda>
>>> > >>     **kwargs)
>>> > >>   File "<string>", line 2, in
glusterPeerStatus
>>> > >>   File
"/usr/lib64/python2.6/multiprocessing/managers.py", line 740,
>>> > >> in
>>> > >> _callmethod
>>> > >>     raise convert_to_error(kind, result)
>>> > >> GlusterCmdExecFailedException: Command execution
failed
>>> > >> error: Connection failed. Please check if gluster
daemon is
>>> > >> operational.
>>> > >> return code: 1
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >>
21:41:50,949::task::595::Storage.TaskManager.Task::(_updateState)
>>> > >> Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving
from state init
>>> > >> ->
>>> > >> state preparing
>>> > >> Thread-13::INFO::2014-11-24
>>> > >> 21:41:50,950::logUtils::44::dispatcher::(wrapper) Run
and protect:
>>> > >> repoStats(options=None)
>>> > >> Thread-13::INFO::2014-11-24
>>> > >> 21:41:50,950::logUtils::47::dispatcher::(wrapper) Run
and protect:
>>> > >> repoStats, Return response: {}
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >>
21:41:50,950::task::1191::Storage.TaskManager.Task::(prepare)
>>> > >>
Task=`c9042986-c978-4b08-adb2-616f5299e115`::finished: {}
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >>
21:41:50,950::task::595::Storage.TaskManager.Task::(_updateState)
>>> > >> Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving
from state
>>> > >> preparing
>>> > >> ->
>>> > >> state finished
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >>
>>> > >>
21:41:50,951::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
>>> > >> Owner.releaseAll requests {} resources {}
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >>
>>> > >>
21:41:50,951::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
>>> > >> Owner.cancelAll requests {}
>>> > >> Thread-13::DEBUG::2014-11-24
>>> > >>
21:41:50,951::task::993::Storage.TaskManager.Task::(_decref)
>>> > >> Task=`c9042986-c978-4b08-adb2-616f5299e115`::ref 0
aborting False
>>> > >>  -------------------------------
>>> > >>
>>> > >>  [root at compute4 ~]# service glusterd status
>>> > >> glusterd is stopped
>>> > >> [root at compute4 ~]# chkconfig --list | grep
glusterd
>>> > >> glusterd        0:off   1:off   2:on    3:on    4:on 
5:on
>>> > >> 6:off
>>> > >> [root at compute4 ~]#
>>> > >>
>>> > >>  Thanks,
>>> > >> Punit
>>> > >>
>>> > >> On Mon, Nov 24, 2014 at 6:36 PM, Kanagaraj
<kmayilsa at redhat.com>
>>> > >> wrote:
>>> > >>
>>> > >>>  Can you send the corresponding error in vdsm.log
from the host?
>>> > >>>
>>> > >>> Also check if glusterd service is running.
>>> > >>>
>>> > >>> Thanks,
>>> > >>> Kanagaraj
>>> > >>>
>>> > >>>
>>> > >>> On 11/24/2014 03:39 PM, Punit Dambiwal wrote:
>>> > >>>
>>> > >>>   Hi,
>>> > >>>
>>> > >>>  After reboot my Hypervisior host can not
activate again in the
>>> > >>> cluster
>>> > >>> and failed with the following error :-
>>> > >>>
>>> > >>>  Gluster command [<UNKNOWN>] failed on
server...
>>> > >>>
>>> > >>>  Engine logs :-
>>> > >>>
>>> > >>>  2014-11-24 18:05:28,397 INFO
>>> > >>>
>>> > >>>
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>>> > >>> (DefaultQuartzScheduler_Worker-64) START,
>>> > >>> GlusterVolumesListVDSCommand(HostName = Compute4,
HostId >>> > >>> 33648a90-200c-45ca-89d5-1ce305d79a6a), log
id: 5f251c90
>>> > >>> 2014-11-24 18:05:30,609 INFO
>>> > >>>
>>> > >>>
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>>> > >>> (DefaultQuartzScheduler_Worker-64) FINISH,
>>> > >>> GlusterVolumesListVDSCommand,
>>> > >>> return:
>>> > >>>
>>> > >>>
{26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity
at d95203e0},
>>> > >>> log id: 5f251c90
>>> > >>> 2014-11-24 18:05:33,768 INFO
>>> > >>>  [org.ovirt.engine.core.bll.ActivateVdsCommand]
>>> > >>> (ajp--127.0.0.1-8702-8)
>>> > >>> [287d570d] Lock Acquired to object EngineLock
[exclusiveLocks= key:
>>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a value: VDS
>>> > >>> , sharedLocks= ]
>>> > >>> 2014-11-24 18:05:33,795 INFO
>>> > >>>  [org.ovirt.engine.core.bll.ActivateVdsCommand]
>>> > >>> (org.ovirt.thread.pool-8-thread-45) [287d570d]
Running command:
>>> > >>> ActivateVdsCommand internal: false. Entities
affected :  ID:
>>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type:
VDSAction group
>>> > >>> MANIPULATE_HOST
>>> > >>> with role type ADMIN
>>> > >>> 2014-11-24 18:05:33,796 INFO
>>> > >>>  [org.ovirt.engine.core.bll.ActivateVdsCommand]
>>> > >>> (org.ovirt.thread.pool-8-thread-45) [287d570d]
Before acquiring
>>> > >>> lock in
>>> > >>> order to prevent monitoring for host Compute5
from data-center
>>> > >>> SV_WTC
>>> > >>> 2014-11-24 18:05:33,797 INFO
>>> > >>>  [org.ovirt.engine.core.bll.ActivateVdsCommand]
>>> > >>> (org.ovirt.thread.pool-8-thread-45) [287d570d]
Lock acquired, from
>>> > >>> now a
>>> > >>> monitoring of host will be skipped for host
Compute5 from
>>> > >>> data-center
>>> > >>> SV_WTC
>>> > >>> 2014-11-24 18:05:33,817 INFO
>>> > >>> 
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
>>> > >>> (org.ovirt.thread.pool-8-thread-45) [287d570d]
START,
>>> > >>> SetVdsStatusVDSCommand(HostName = Compute5,
HostId >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a,
status=Unassigned,
>>> > >>> nonOperationalReason=NONE,
stopSpmFailureLogged=false), log id:
>>> > >>> 1cbc7311
>>> > >>> 2014-11-24 18:05:33,820 INFO
>>> > >>> 
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
>>> > >>> (org.ovirt.thread.pool-8-thread-45) [287d570d]
FINISH,
>>> > >>> SetVdsStatusVDSCommand, log id: 1cbc7311
>>> > >>> 2014-11-24 18:05:34,086 INFO
>>> > >>>  [org.ovirt.engine.core.bll.ActivateVdsCommand]
>>> > >>> (org.ovirt.thread.pool-8-thread-45) Activate
finished. Lock
>>> > >>> released.
>>> > >>> Monitoring can run now for host Compute5 from
data-center SV_WTC
>>> > >>> 2014-11-24 18:05:34,088 INFO
>>> > >>>
>>> > >>>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> > >>> (org.ovirt.thread.pool-8-thread-45) Correlation
ID: 287d570d, Job
>>> > >>> ID:
>>> > >>> 5ef8e4d6-b2bc-469e-8e81-7ef74b2a001a, Call Stack:
null, Custom
>>> > >>> Event ID:
>>> > >>> -1, Message: Host Compute5 was activated by
admin.
>>> > >>> 2014-11-24 18:05:34,090 INFO
>>> > >>>  [org.ovirt.engine.core.bll.ActivateVdsCommand]
>>> > >>> (org.ovirt.thread.pool-8-thread-45) Lock freed to
object EngineLock
>>> > >>> [exclusiveLocks= key:
0bf6b00f-7947-4411-b55a-cc5eea2b381a value:
>>> > >>> VDS
>>> > >>> , sharedLocks= ]
>>> > >>> 2014-11-24 18:05:35,792 INFO
>>> > >>>
>>> > >>>
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>>> > >>> (DefaultQuartzScheduler_Worker-55) [3706e836]
START,
>>> > >>> GlusterVolumesListVDSCommand(HostName = Compute4,
HostId >>> > >>> 33648a90-200c-45ca-89d5-1ce305d79a6a), log
id: 48a0c832
>>> > >>> 2014-11-24 18:05:37,064 INFO
>>> > >>>
>>> > >>>
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
>>> > >>> (DefaultQuartzScheduler_Worker-69) START,
>>> > >>> GetHardwareInfoVDSCommand(HostName = Compute5,
HostId >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a,
>>> > >>>
vds=Host[Compute5,0bf6b00f-7947-4411-b55a-cc5eea2b381a]), log id:
>>> > >>> 6d560cc2
>>> > >>> 2014-11-24 18:05:37,074 INFO
>>> > >>>
>>> > >>>
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
>>> > >>> (DefaultQuartzScheduler_Worker-69) FINISH,
>>> > >>> GetHardwareInfoVDSCommand, log
>>> > >>> id: 6d560cc2
>>> > >>> 2014-11-24 18:05:37,093 WARN
>>> > >>>  [org.ovirt.engine.core.vdsbroker.VdsManager]
>>> > >>> (DefaultQuartzScheduler_Worker-69) Host Compute5
is running with
>>> > >>> disabled
>>> > >>> SELinux.
>>> > >>> 2014-11-24 18:05:37,127 INFO
>>> > >>>
>>> > >>>
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
>>> > >>> (DefaultQuartzScheduler_Worker-69) [2b4a51cf]
Running command:
>>> > >>> HandleVdsCpuFlagsOrClusterChangedCommand
internal: true. Entities
>>> > >>> affected
>>> > >>> :  ID: 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type:
VDS
>>> > >>> 2014-11-24 18:05:37,147 INFO
>>> > >>>
>>> > >>>
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
>>> > >>> (DefaultQuartzScheduler_Worker-69) [2b4a51cf]
START,
>>> > >>> GlusterServersListVDSCommand(HostName = Compute5,
HostId >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a), log
id: 4faed87
>>> > >>> 2014-11-24 18:05:37,164 INFO
>>> > >>>
>>> > >>>
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
>>> > >>> (DefaultQuartzScheduler_Worker-69) [2b4a51cf]
FINISH,
>>> > >>> GlusterServersListVDSCommand, log id: 4faed87
>>> > >>> 2014-11-24 18:05:37,189 INFO
>>> > >>> 
[org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
>>> > >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5]
Running command:
>>> > >>> SetNonOperationalVdsCommand internal: true.
Entities affected :
>>> > >>> ID:
>>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
>>> > >>> 2014-11-24 18:05:37,206 INFO
>>> > >>> 
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
>>> > >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5]
START,
>>> > >>> SetVdsStatusVDSCommand(HostName = Compute5,
HostId >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a,
status=NonOperational,
>>> > >>> nonOperationalReason=GLUSTER_COMMAND_FAILED,
>>> > >>> stopSpmFailureLogged=false),
>>> > >>> log id: fed5617
>>> > >>> 2014-11-24 18:05:37,209 INFO
>>> > >>> 
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
>>> > >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5]
FINISH,
>>> > >>> SetVdsStatusVDSCommand, log id: fed5617
>>> > >>> 2014-11-24 18:05:37,223 ERROR
>>> > >>>
>>> > >>>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> > >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5]
Correlation ID:
>>> > >>> 4a84c4e5,
>>> > >>> Job
>>> > >>> ID: 4bfd4a6d-c3ef-468f-a40e-a3a6ca13011b, Call
Stack: null, Custom
>>> > >>> Event
>>> > >>> ID: -1, Message: Gluster command
[<UNKNOWN>] failed on server
>>> > >>> Compute5.
>>> > >>> 2014-11-24 18:05:37,243 INFO
>>> > >>>
>>> > >>>
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> > >>> (DefaultQuartzScheduler_Worker-69) [4a84c4e5]
Correlation ID: null,
>>> > >>> Call
>>> > >>> Stack: null, Custom Event ID: -1, Message: Status
of host Compute5
>>> > >>> was
>>> > >>> set
>>> > >>> to NonOperational.
>>> > >>> 2014-11-24 18:05:37,272 INFO
>>> > >>> 
[org.ovirt.engine.core.bll.HandleVdsVersionCommand]
>>> > >>> (DefaultQuartzScheduler_Worker-69) [a0c8a7f]
Running command:
>>> > >>> HandleVdsVersionCommand internal: true. Entities
affected :  ID:
>>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
>>> > >>> 2014-11-24 18:05:37,274 INFO
>>> > >>> 
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>>> > >>> (DefaultQuartzScheduler_Worker-69) [a0c8a7f] Host
>>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a : Compute5
is already in
>>> > >>> NonOperational status for reason
GLUSTER_COMMAND_FAILED.
>>> > >>> SetNonOperationalVds command is skipped.
>>> > >>> 2014-11-24 18:05:38,065 INFO
>>> > >>>
>>> > >>>
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>>> > >>> (DefaultQuartzScheduler_Worker-55) [3706e836]
FINISH,
>>> > >>> GlusterVolumesListVDSCommand, return:
>>> > >>>
>>> > >>>
{26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity
at 4e72a1b1},
>>> > >>> log id: 48a0c832
>>> > >>> 2014-11-24 18:05:43,243 INFO
>>> > >>>
>>> > >>>
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>>> > >>> (DefaultQuartzScheduler_Worker-35) START,
>>> > >>> GlusterVolumesListVDSCommand(HostName = Compute4,
HostId >>> > >>> 33648a90-200c-45ca-89d5-1ce305d79a6a), log
id: 3ce13ebc
>>> > >>> ^C
>>> > >>> [root at ccr01 ~]#
>>> > >>>
>>> > >>>  Thanks,
>>> > >>> Punit
>>> > >>>
>>> > >>>
>>> > >>>  _______________________________________________
>>> > >>> Users mailing
>>> > >>> listUsers at
ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>
>>> > >>
>>> > >
>>> > >
>>> >
>>
>>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

Hi Kaushal,

Thanks for the detailed reply....let me explain my setup first :-

1. Ovirt Engine
2. 4* host as well as storage machine (Host and gluster combined)
3. Every host has 24 bricks...

Now whenever the host machine reboot...it can come up but can not join the
cluster again and through the following error "Gluster command
[<UNKNOWN>]
failed on server.."

Please check my comment in line :-

1. Use the same string for doing the peer probe and for the brick address
during volume create/add-brick. Ideally, we suggest you use properly
resolvable FQDNs everywhere. If that is not possible, then use only IP
addresses. Try to avoid short names.
---------------
[root at cpu05 ~]# gluster peer status
Number of Peers: 3

Hostname: cpu03.stack.com
Uuid: 5729b8c4-e80d-4353-b456-6f467bddbdfb
State: Peer in Cluster (Connected)

Hostname: cpu04.stack.com
Uuid: d272b790-c4b2-4bed-ba68-793656e6d7b0
State: Peer in Cluster (Connected)
Other names:
10.10.0.8

Hostname: cpu02.stack.com
Uuid: 8d8a7041-950e-40d0-85f9-58d14340ca25
State: Peer in Cluster (Connected)
[root at cpu05 ~]#
----------------
2. During boot up, make sure to launch glusterd only after the network is
up. This will allow the new peer identification mechanism to do its
job correctly.
>> I think the service itself doing the same job....
[root at cpu05 ~]# cat /usr/lib/systemd/system/glusterd.service
[Unit]
Description=GlusterFS, a clustered file-system server
After=network.target rpcbind.service
Before=network-online.target

[Service]
Type=forking
PIDFile=/var/run/glusterd.pid
LimitNOFILE=65536
ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid
KillMode=process

[Install]
WantedBy=multi-user.target
[root at cpu05 ~]#
--------------------

gluster logs :-

[2014-11-24 09:22:22.147471] I [MSGID: 100030] [glusterfsd.c:2018:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.6.1
(args: /usr/sbin/glusterd -p /var/run/glusterd.pid)
[2014-11-24 09:22:22.151565] I [glusterd.c:1214:init] 0-management: Maximum
allowed open file descriptors set to 65536
[2014-11-24 09:22:22.151599] I [glusterd.c:1259:init] 0-management: Using
/var/lib/glusterd as working directory
[2014-11-24 09:22:22.155216] W [rdma.c:4195:__gf_rdma_ctx_create]
0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
[2014-11-24 09:22:22.155264] E [rdma.c:4483:init] 0-rdma.management: Failed
to initialize IB Device
[2014-11-24 09:22:22.155285] E [rpc-transport.c:333:rpc_transport_load]
0-rpc-transport: 'rdma' initialization failed
[2014-11-24 09:22:22.155354] W [rpcsvc.c:1524:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the transport failed
[2014-11-24 09:22:22.156290] I
[glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication
module not installed in the system
[2014-11-24 09:22:22.161318] I
[glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved
op-version: 30600
[2014-11-24 09:22:22.821800] I
[glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management:
connect returned 0
[2014-11-24 09:22:22.825810] I
[glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management:
connect returned 0
[2014-11-24 09:22:22.828705] I
[glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management:
connect returned 0
[2014-11-24 09:22:22.828771] I [rpc-clnt.c:969:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2014-11-24 09:22:22.832670] I [rpc-clnt.c:969:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2014-11-24 09:22:22.835919] I [rpc-clnt.c:969:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2014-11-24 09:22:22.840209] E
[glusterd-store.c:4248:glusterd_resolve_all_bricks] 0-glusterd: resolve
brick failed in restore
[2014-11-24 09:22:22.840233] E [xlator.c:425:xlator_init] 0-management:
Initialization of volume 'management' failed, review your volfile again
[2014-11-24 09:22:22.840245] E [graph.c:322:glusterfs_graph_init]
0-management: initializing translator failed
[2014-11-24 09:22:22.840264] E [graph.c:525:glusterfs_graph_activate]
0-graph: init failed
[2014-11-24 09:22:22.840754] W [glusterfsd.c:1194:cleanup_and_exit] (-->
0-: received signum (0), shutting down

Thanks,
Punit




On Wed, Nov 26, 2014 at 7:14 PM, Kaushal M <kshlmster at gmail.com> wrote:
> Based on the logs I can guess that glusterd is being started before
> the network has come up and that the addresses given to bricks do not
> directly match the addresses used in during peer probe.
>
> The gluster_after_reboot log has the line "[2014-11-25
> 06:46:09.972113] E [glusterd-store.c:2632:glusterd_resolve_all_bricks]
> 0-glusterd: resolve brick failed in restore".
>
> Brick resolution fails when glusterd cannot match the address for the
> brick, with one of the peers. Brick resolution happens in two phases,
> 1. We first try to identify the peer by performing string comparisions
> with the brick address and the peer addresses (The peer names will be
> the names/addresses that were given when the peer was probed).
> 2. If we don't find a match from step 1, we will then resolve all the
> brick address and the peer addresses into addrinfo structs, and then
> compare these structs to find a match. This process should generally
> find a match if available. This will fail only if the network is not
> up yet as we cannot resolve addresses.
>
> The above steps are applicable only to glusterfs versions >=3.6. They
> were introduced to reduce problems with peer identification, like the
> one you encountered
>
> Since both of the steps failed to find a match in one run, but
> succeeded later, we can come to the conclusion that,
> a) the bricks don't have the exact same string used in peer probe for
> their addresses as step 1 failed, and
> b) the network was not up in the initial run, as step 2 failed during
> the initial run, but passed in the second run.
>
> Please let me know if my conclusion is correct.
>
> If it is, you can solve your problem in two ways.
> 1. Use the same string for doing the peer probe and for the brick
> address during volume create/add-brick. Ideally, we suggest you use
> properly resolvable FQDNs everywhere. If that is not possible, then
> use only IP addresses. Try to avoid short names.
> 2. During boot up, make sure to launch glusterd only after the network
> is up. This will allow the new peer identification mechanism to do its
> job correctly.
>
>
> If you have already followed these steps and yet still hit the
> problem, then please provide more information (setup, logs, etc.). It
> could be much different problem that you are facing.
>
> ~kaushal
>
> On Wed, Nov 26, 2014 at 4:01 PM, Punit Dambiwal <hypunit at
gmail.com> wrote:
> > Is there any one can help on this ??
> >
> > Thanks,
> > punit
> >
> > On Wed, Nov 26, 2014 at 9:42 AM, Punit Dambiwal <hypunit at
gmail.com>
> wrote:
> >>
> >> Hi,
> >>
> >> My Glusterfs version is :- glusterfs-3.6.1-1.el7
> >>
> >> On Wed, Nov 26, 2014 at 1:59 AM, Kanagaraj Mayilsamy <
> kmayilsa at redhat.com>
> >> wrote:
> >>>
> >>> [+Gluster-users at gluster.org]
> >>>
> >>> "Initialization of volume 'management' failed,
review your volfile
> >>> again", glusterd throws this error when the service is
started
> automatically
> >>> after the reboot. But the service is successfully started
later
> manually by
> >>> the user.
> >>>
> >>> can somebody from gluster-users please help on this?
> >>>
> >>> glusterfs version: 3.5.1
> >>>
> >>> Thanks,
> >>> Kanagaraj
> >>>
> >>> ----- Original Message -----
> >>> > From: "Punit Dambiwal" <hypunit at
gmail.com>
> >>> > To: "Kanagaraj" <kmayilsa at redhat.com>
> >>> > Cc: users at ovirt.org
> >>> > Sent: Tuesday, November 25, 2014 7:24:45 PM
> >>> > Subject: Re: [ovirt-users] Gluster command
[<UNKNOWN>] failed on
> >>> > server...
> >>> >
> >>> > Hi Kanagraj,
> >>> >
> >>> > Please check the attached log files....i didn't find
any thing
> >>> > special....
> >>> >
> >>> > On Tue, Nov 25, 2014 at 12:12 PM, Kanagaraj <kmayilsa
at redhat.com>
> >>> > wrote:
> >>> >
> >>> > >  Do you see any errors in
> >>> > > /var/log/glusterfs/etc-glusterfs-glusterd.vol.log or
vdsm.log when
> >>> > > the
> >>> > > service is trying to start automatically after the
reboot?
> >>> > >
> >>> > > Thanks,
> >>> > > Kanagaraj
> >>> > >
> >>> > >
> >>> > > On 11/24/2014 08:13 PM, Punit Dambiwal wrote:
> >>> > >
> >>> > > Hi Kanagaraj,
> >>> > >
> >>> > >  Yes...once i will start the gluster service and
then vdsmd ...the
> >>> > > host
> >>> > > can connect to cluster...but the question is why
it's not started
> >>> > > even it
> >>> > > has chkconfig enabled...
> >>> > >
> >>> > >  I have tested it in two host cluster
environment...(Centos 6.6 and
> >>> > > centos 7.0) on both hypervisior cluster..it's
failed to reconnect
> in
> >>> > > to
> >>> > > cluster after reboot....
> >>> > >
> >>> > >  In both the environment glusterd enabled for next
boot....but it's
> >>> > > failed with the same error....seems it's bug in
either gluster or
> >>> > > Ovirt ??
> >>> > >
> >>> > >  Please help me to find the workaround here if can
not resolve
> >>> > > it...as
> >>> > > without this the Host machine can not connect after
reboot....that
> >>> > > means
> >>> > > engine will consider it as down and every time need
to manually
> start
> >>> > > the
> >>> > > gluster service and vdsmd... ??
> >>> > >
> >>> > >  Thanks,
> >>> > > Punit
> >>> > >
> >>> > > On Mon, Nov 24, 2014 at 10:20 PM, Kanagaraj
<kmayilsa at redhat.com>
> >>> > > wrote:
> >>> > >
> >>> > >>  From vdsm.log "error: Connection failed.
Please check if gluster
> >>> > >> daemon
> >>> > >> is operational."
> >>> > >>
> >>> > >> Starting glusterd service should fix this issue.
'service glusterd
> >>> > >> start'
> >>> > >> But i am wondering why the glusterd was not
started automatically
> >>> > >> after
> >>> > >> the reboot.
> >>> > >>
> >>> > >> Thanks,
> >>> > >> Kanagaraj
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >> On 11/24/2014 07:18 PM, Punit Dambiwal wrote:
> >>> > >>
> >>> > >> Hi Kanagaraj,
> >>> > >>
> >>> > >>  Please find the attached VDSM logs :-
> >>> > >>
> >>> > >>  ----------------
> >>> > >>  Thread-13::DEBUG::2014-11-24
> >>> > >>
> >>> > >>
>
21:41:17,182::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
> >>> > >> Owner.cancelAll requests {}
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
21:41:17,182::task::993::Storage.TaskManager.Task::(_decref)
> >>> > >> Task=`1691d409-9b27-4585-8281-5ec26154367a`::ref
0 aborting False
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
21:41:32,393::task::595::Storage.TaskManager.Task::(_updateState)
> >>> > >>
Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving from state
> init
> >>> > >> ->
> >>> > >> state preparing
> >>> > >> Thread-13::INFO::2014-11-24
> >>> > >>
21:41:32,393::logUtils::44::dispatcher::(wrapper) Run and protect:
> >>> > >> repoStats(options=None)
> >>> > >> Thread-13::INFO::2014-11-24
> >>> > >>
21:41:32,393::logUtils::47::dispatcher::(wrapper) Run and protect:
> >>> > >> repoStats, Return response: {}
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
21:41:32,393::task::1191::Storage.TaskManager.Task::(prepare)
> >>> > >>
Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::finished: {}
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
21:41:32,394::task::595::Storage.TaskManager.Task::(_updateState)
> >>> > >>
Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving from state
> >>> > >> preparing
> >>> > >> ->
> >>> > >> state finished
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
> >>> > >>
>
21:41:32,394::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
> >>> > >> Owner.releaseAll requests {} resources {}
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
> >>> > >>
>
21:41:32,394::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
> >>> > >> Owner.cancelAll requests {}
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
21:41:32,394::task::993::Storage.TaskManager.Task::(_decref)
> >>> > >> Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::ref
0 aborting False
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
21:41:41,550::BindingXMLRPC::1132::vds::(wrapper) client
> >>> > >> [10.10.10.2]::call
> >>> > >> getCapabilities with () {}
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >> 21:41:41,553::utils::738::root::(execCmd)
> >>> > >> /sbin/ip route show to 0.0.0.0/0 table all (cwd
None)
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >> 21:41:41,560::utils::758::root::(execCmd)
> >>> > >> SUCCESS: <err> = ''; <rc> =
0
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >> 21:41:41,588::caps::728::root::(_getKeyPackages)
rpm package
> >>> > >> ('gluster-swift',) not found
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >> 21:41:41,592::caps::728::root::(_getKeyPackages)
rpm package
> >>> > >> ('gluster-swift-object',) not found
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >> 21:41:41,593::caps::728::root::(_getKeyPackages)
rpm package
> >>> > >> ('gluster-swift-plugin',) not found
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >> 21:41:41,598::caps::728::root::(_getKeyPackages)
rpm package
> >>> > >> ('gluster-swift-account',) not found
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >> 21:41:41,598::caps::728::root::(_getKeyPackages)
rpm package
> >>> > >> ('gluster-swift-proxy',) not found
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >> 21:41:41,598::caps::728::root::(_getKeyPackages)
rpm package
> >>> > >> ('gluster-swift-doc',) not found
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >> 21:41:41,599::caps::728::root::(_getKeyPackages)
rpm package
> >>> > >> ('gluster-swift-container',) not found
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >> 21:41:41,599::caps::728::root::(_getKeyPackages)
rpm package
> >>> > >> ('glusterfs-geo-replication',) not found
> >>> > >> Thread-13::DEBUG::2014-11-24
21:41:41,600::caps::646::root::(get)
> >>> > >> VirtioRNG DISABLED: libvirt version
0.10.2-29.el6_5.9 required >> >>> > >> 0.10.2-31
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
21:41:41,603::BindingXMLRPC::1139::vds::(wrapper) return
> >>> > >> getCapabilities
> >>> > >> with {'status': {'message':
'Done', 'code': 0}, 'info':
> >>> > >> {'HBAInventory':
> >>> > >> {'iSCSI': [{'InitiatorName':
> >>> > >> 'iqn.1994-05.com.redhat:32151ce183c8'}],
> >>> > >> 'FC':
> >>> > >> []}, 'packages2': {'kernel':
{'release': '431.el6.x86_64',
> >>> > >> 'buildtime':
> >>> > >> 1385061309.0, 'version':
'2.6.32'}, 'glusterfs-rdma': {'release':
> >>> > >> '1.el6',
> >>> > >> 'buildtime': 1403622628L,
'version': '3.5.1'}, 'glusterfs-fuse':
> >>> > >> {'release': '1.el6',
'buildtime': 1403622628L, 'version':
> '3.5.1'},
> >>> > >> 'spice-server': {'release':
'6.el6_5.2', 'buildtime': 1402324637L,
> >>> > >> 'version': '0.12.4'},
'vdsm': {'release': '1.gitdb83943.el6',
> >>> > >> 'buildtime':
> >>> > >> 1412784567L, 'version':
'4.16.7'}, 'qemu-kvm': {'release':
> >>> > >> '2.415.el6_5.10', 'buildtime':
1402435700L, 'version':
> '0.12.1.2'},
> >>> > >> 'qemu-img': {'release':
'2.415.el6_5.10', 'buildtime':
> 1402435700L,
> >>> > >> 'version': '0.12.1.2'},
'libvirt': {'release': '29.el6_5.9',
> >>> > >> 'buildtime':
> >>> > >> 1402404612L, 'version':
'0.10.2'}, 'glusterfs': {'release':
> '1.el6',
> >>> > >> 'buildtime': 1403622628L,
'version': '3.5.1'}, 'mom': {'release':
> >>> > >> '2.el6',
> >>> > >> 'buildtime': 1403794344L,
'version': '0.4.1'}, 'glusterfs-server':
> >>> > >> {'release': '1.el6',
'buildtime': 1403622628L, 'version':
> '3.5.1'}},
> >>> > >> 'numaNodeDistance': {'1': [20,
10], '0': [10, 20]}, 'cpuModel':
> >>> > >> 'Intel(R)
> >>> > >> Xeon(R) CPU           X5650  @ 2.67GHz',
'liveMerge': 'false',
> >>> > >> 'hooks':
> >>> > >> {},
> >>> > >> 'cpuSockets': '2',
'vmTypes': ['kvm'], 'selinux': {'mode':
'1'},
> >>> > >> 'kdumpStatus': 0,
'supportedProtocols': ['2.2', '2.3'],
> 'networks':
> >>> > >> {'ovirtmgmt': {'iface':
u'bond0.10', 'addr': '43.252.176.16',
> >>> > >> 'bridged':
> >>> > >> False, 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'mtu':
> '1500',
> >>> > >> 'bootproto4': 'none',
'netmask': '255.255.255.0', 'ipv4addrs': ['
> >>> > >> 43.252.176.16/24'
<http://43.252.176.16/24%27>], 'interface':
> >>> > >> u'bond0.10', 'ipv6gateway':
'::', 'gateway': '43.25.17.1'},
> >>> > >> 'Internal':
> >>> > >> {'iface': 'Internal',
'addr': '', 'cfg': {'DEFROUTE': 'no',
> >>> > >> 'HOTPLUG':
> >>> > >> 'no', 'MTU': '9000',
'DELAY': '0', 'NM_CONTROLLED': 'no',
> >>> > >> 'BOOTPROTO':
> >>> > >> 'none', 'STP': 'off',
'DEVICE': 'Internal', 'TYPE': 'Bridge',
> >>> > >> 'ONBOOT':
> >>> > >> 'no'}, 'bridged': True,
'ipv6addrs':
> >>> > >> ['fe80::210:18ff:fecd:daac/64'],
> >>> > >> 'gateway': '',
'bootproto4': 'none', 'netmask': '',
'stp': 'off',
> >>> > >> 'ipv4addrs': [], 'mtu':
'9000', 'ipv6gateway': '::', 'ports':
> >>> > >> ['bond1.100']}, 'storage':
{'iface': u'bond1', 'addr':
> '10.10.10.6',
> >>> > >> 'bridged': False, 'ipv6addrs':
['fe80::210:18ff:fecd:daac/64'],
> >>> > >> 'mtu':
> >>> > >> '9000', 'bootproto4':
'none', 'netmask': '255.255.255.0',
> >>> > >> 'ipv4addrs': ['
> >>> > >> 10.10.10.6/24'
<http://10.10.10.6/24%27>], 'interface': u'bond1',
> >>> > >> 'ipv6gateway': '::',
'gateway': ''}, 'VMNetwork': {'iface':
> >>> > >> 'VMNetwork',
> >>> > >> 'addr': '', 'cfg':
{'DEFROUTE': 'no', 'HOTPLUG': 'no',
'MTU':
> >>> > >> '1500',
> >>> > >> 'DELAY': '0',
'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none',
'STP':
> >>> > >> 'off',
> >>> > >> 'DEVICE': 'VMNetwork',
'TYPE': 'Bridge', 'ONBOOT': 'no'},
> 'bridged':
> >>> > >> True,
> >>> > >> 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'gateway': '',
> >>> > >> 'bootproto4':
> >>> > >> 'none', 'netmask': '',
'stp': 'off', 'ipv4addrs': [], 'mtu':
> '1500',
> >>> > >> 'ipv6gateway': '::',
'ports': ['bond0.36']}}, 'bridges':
> >>> > >> {'Internal':
> >>> > >> {'addr': '', 'cfg':
{'DEFROUTE': 'no', 'HOTPLUG': 'no',
'MTU':
> >>> > >> '9000',
> >>> > >> 'DELAY': '0',
'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none',
'STP':
> >>> > >> 'off',
> >>> > >> 'DEVICE': 'Internal',
'TYPE': 'Bridge', 'ONBOOT': 'no'},
> >>> > >> 'ipv6addrs':
> >>> > >> ['fe80::210:18ff:fecd:daac/64'],
'mtu': '9000', 'netmask': '',
> >>> > >> 'stp':
> >>> > >> 'off', 'ipv4addrs': [],
'ipv6gateway': '::', 'gateway': '',
> 'opts':
> >>> > >> {'topology_change_detected':
'0', 'multicast_last_member_count':
> >>> > >> '2',
> >>> > >> 'hash_elasticity': '4',
'multicast_query_response_interval':
> '999',
> >>> > >> 'multicast_snooping': '1',
'multicast_startup_query_interval':
> >>> > >> '3124',
> >>> > >> 'hello_timer': '31',
'multicast_querier_interval': '25496',
> >>> > >> 'max_age':
> >>> > >> '1999', 'hash_max':
'512', 'stp_state': '0', 'root_id':
> >>> > >> '8000.001018cddaac', 'priority':
'32768',
> >>> > >> 'multicast_membership_interval':
> >>> > >> '25996', 'root_path_cost':
'0', 'root_port': '0',
> >>> > >> 'multicast_querier':
> >>> > >> '0',
> >>> > >> 'multicast_startup_query_count':
'2', 'hello_time': '199',
> >>> > >> 'topology_change': '0',
'bridge_id': '8000.001018cddaac',
> >>> > >> 'topology_change_timer': '0',
'ageing_time': '29995', 'gc_timer':
> >>> > >> '31',
> >>> > >> 'group_addr': '1:80:c2:0:0:0',
'tcn_timer': '0',
> >>> > >> 'multicast_query_interval':
'12498',
> >>> > >> 'multicast_last_member_interval':
> >>> > >> '99', 'multicast_router':
'1', 'forward_delay': '0'}, 'ports':
> >>> > >> ['bond1.100']}, 'VMNetwork':
{'addr': '', 'cfg': {'DEFROUTE':
> 'no',
> >>> > >> 'HOTPLUG': 'no', 'MTU':
'1500', 'DELAY': '0', 'NM_CONTROLLED':
> 'no',
> >>> > >> 'BOOTPROTO': 'none',
'STP': 'off', 'DEVICE': 'VMNetwork',
'TYPE':
> >>> > >> 'Bridge',
> >>> > >> 'ONBOOT': 'no'},
'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'],
> >>> > >> 'mtu':
> >>> > >> '1500', 'netmask': '',
'stp': 'off', 'ipv4addrs': [],
> 'ipv6gateway':
> >>> > >> '::',
> >>> > >> 'gateway': '', 'opts':
{'topology_change_detected': '0',
> >>> > >> 'multicast_last_member_count':
'2', 'hash_elasticity': '4',
> >>> > >> 'multicast_query_response_interval':
'999', 'multicast_snooping':
> >>> > >> '1',
> >>> > >> 'multicast_startup_query_interval':
'3124', 'hello_timer': '131',
> >>> > >> 'multicast_querier_interval':
'25496', 'max_age': '1999',
> >>> > >> 'hash_max':
> >>> > >> '512', 'stp_state': '0',
'root_id': '8000.60eb6920b46c',
> 'priority':
> >>> > >> '32768',
'multicast_membership_interval': '25996',
> 'root_path_cost':
> >>> > >> '0',
> >>> > >> 'root_port': '0',
'multicast_querier': '0',
> >>> > >> 'multicast_startup_query_count':
'2', 'hello_time': '199',
> >>> > >> 'topology_change': '0',
'bridge_id': '8000.60eb6920b46c',
> >>> > >> 'topology_change_timer': '0',
'ageing_time': '29995', 'gc_timer':
> >>> > >> '31',
> >>> > >> 'group_addr': '1:80:c2:0:0:0',
'tcn_timer': '0',
> >>> > >> 'multicast_query_interval':
'12498',
> >>> > >> 'multicast_last_member_interval':
> >>> > >> '99', 'multicast_router':
'1', 'forward_delay': '0'}, 'ports':
> >>> > >> ['bond0.36']}}, 'uuid':
'44454C4C-4C00-1057-8053-B7C04F504E31',
> >>> > >> 'lastClientIface': 'bond1',
'nics': {'eth3': {'permhwaddr':
> >>> > >> '00:10:18:cd:da:ae', 'addr':
'', 'cfg': {'SLAVE': 'yes',
> >>> > >> 'NM_CONTROLLED':
> >>> > >> 'no', 'MTU': '9000',
'HWADDR': '00:10:18:cd:da:ae', 'MASTER':
> >>> > >> 'bond1',
> >>> > >> 'DEVICE': 'eth3',
'ONBOOT': 'no'}, 'ipv6addrs': [], 'mtu':
'9000',
> >>> > >> 'netmask': '',
'ipv4addrs': [], 'hwaddr': '00:10:18:cd:da:ac',
> >>> > >> 'speed':
> >>> > >> 1000}, 'eth2': {'permhwaddr':
'00:10:18:cd:da:ac', 'addr': '',
> >>> > >> 'cfg':
> >>> > >> {'SLAVE': 'yes',
'NM_CONTROLLED': 'no', 'MTU': '9000',
'HWADDR':
> >>> > >> '00:10:18:cd:da:ac', 'MASTER':
'bond1', 'DEVICE': 'eth2',
> 'ONBOOT':
> >>> > >> 'no'},
> >>> > >> 'ipv6addrs': [], 'mtu':
'9000', 'netmask': '', 'ipv4addrs': [],
> >>> > >> 'hwaddr':
> >>> > >> '00:10:18:cd:da:ac', 'speed':
1000}, 'eth1': {'permhwaddr':
> >>> > >> '60:eb:69:20:b4:6d', 'addr':
'', 'cfg': {'SLAVE': 'yes',
> >>> > >> 'NM_CONTROLLED':
> >>> > >> 'no', 'MTU': '1500',
'HWADDR': '60:eb:69:20:b4:6d', 'MASTER':
> >>> > >> 'bond0',
> >>> > >> 'DEVICE': 'eth1',
'ONBOOT': 'yes'}, 'ipv6addrs': [], 'mtu':
> '1500',
> >>> > >> 'netmask': '',
'ipv4addrs': [], 'hwaddr': '60:eb:69:20:b4:6c',
> >>> > >> 'speed':
> >>> > >> 1000}, 'eth0': {'permhwaddr':
'60:eb:69:20:b4:6c', 'addr': '',
> >>> > >> 'cfg':
> >>> > >> {'SLAVE': 'yes',
'NM_CONTROLLED': 'no', 'MTU': '1500',
'HWADDR':
> >>> > >> '60:eb:69:20:b4:6c', 'MASTER':
'bond0', 'DEVICE': 'eth0',
> 'ONBOOT':
> >>> > >> 'yes'},
> >>> > >> 'ipv6addrs': [], 'mtu':
'1500', 'netmask': '', 'ipv4addrs': [],
> >>> > >> 'hwaddr':
> >>> > >> '60:eb:69:20:b4:6c', 'speed':
1000}}, 'software_revision': '1',
> >>> > >> 'clusterLevels': ['3.0',
'3.1', '3.2', '3.3', '3.4', '3.5'],
> >>> > >> 'cpuFlags':
> >>> > >>
> >>> > >>
>
u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,popcnt,aes,lahf_lm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270',
> >>> > >> 'ISCSIInitiatorName':
'iqn.1994-05.com.redhat:32151ce183c8',
> >>> > >> 'netConfigDirty': 'False',
'supportedENGINEs': ['3.0', '3.1',
> '3.2',
> >>> > >> '3.3',
> >>> > >> '3.4', '3.5'],
'autoNumaBalancing': 2, 'reservedMem': '321',
> >>> > >> 'bondings':
> >>> > >> {'bond4': {'addr': '',
'cfg': {}, 'mtu': '1500', 'netmask': '',
> >>> > >> 'slaves':
> >>> > >> [], 'hwaddr':
'00:00:00:00:00:00'}, 'bond0': {'addr': '',
'cfg':
> >>> > >> {'HOTPLUG': 'no', 'MTU':
'1500', 'NM_CONTROLLED': 'no',
> >>> > >> 'BONDING_OPTS':
> >>> > >> 'mode=4 miimon=100', 'DEVICE':
'bond0', 'ONBOOT': 'yes'},
> >>> > >> 'ipv6addrs':
> >>> > >> ['fe80::62eb:69ff:fe20:b46c/64'],
'mtu': '1500', 'netmask': '',
> >>> > >> 'ipv4addrs': [], 'hwaddr':
'60:eb:69:20:b4:6c', 'slaves': ['eth0',
> >>> > >> 'eth1'],
> >>> > >> 'opts': {'miimon':
'100', 'mode': '4'}}, 'bond1': {'addr':
> >>> > >> '10.10.10.6',
> >>> > >> 'cfg': {'DEFROUTE':
'no', 'IPADDR': '10.10.10.6', 'HOTPLUG':
'no',
> >>> > >> 'MTU':
> >>> > >> '9000', 'NM_CONTROLLED':
'no', 'NETMASK': '255.255.255.0',
> >>> > >> 'BOOTPROTO':
> >>> > >> 'none', 'BONDING_OPTS':
'mode=4 miimon=100', 'DEVICE': 'bond1',
> >>> > >> 'ONBOOT':
> >>> > >> 'no'}, 'ipv6addrs':
['fe80::210:18ff:fecd:daac/64'], 'mtu':
> '9000',
> >>> > >> 'netmask': '255.255.255.0',
'ipv4addrs': ['10.10.10.6/24'
> >>> > >> <http://10.10.10.6/24%27>],
'hwaddr': '00:10:18:cd:da:ac',
> 'slaves':
> >>> > >> ['eth2', 'eth3'],
'opts': {'miimon': '100', 'mode': '4'}},
> 'bond2':
> >>> > >> {'addr': '', 'cfg': {},
'mtu': '1500', 'netmask': '', 'slaves':
> [],
> >>> > >> 'hwaddr': '00:00:00:00:00:00'},
'bond3': {'addr': '', 'cfg': {},
> >>> > >> 'mtu':
> >>> > >> '1500', 'netmask': '',
'slaves': [], 'hwaddr':
> >>> > >> '00:00:00:00:00:00'}},
> >>> > >> 'software_version': '4.16',
'memSize': '24019', 'cpuSpeed':
> >>> > >> '2667.000',
> >>> > >> 'numaNodes': {u'1':
{'totalMemory': '12288', 'cpus': [6, 7, 8, 9,
> >>> > >> 10, 11,
> >>> > >> 18, 19, 20, 21, 22, 23]}, u'0':
{'totalMemory': '12278', 'cpus':
> [0,
> >>> > >> 1, 2,
> >>> > >> 3, 4, 5, 12, 13, 14, 15, 16, 17]}},
'version_name': 'Snow Man',
> >>> > >> 'vlans':
> >>> > >> {'bond0.10': {'iface':
'bond0', 'addr': '43.25.17.16', 'cfg':
> >>> > >> {'DEFROUTE':
> >>> > >> 'yes', 'VLAN': 'yes',
'IPADDR': '43.25.17.16', 'HOTPLUG': 'no',
> >>> > >> 'GATEWAY':
> >>> > >> '43.25.17.1', 'NM_CONTROLLED':
'no', 'NETMASK': '255.255.255.0',
> >>> > >> 'BOOTPROTO': 'none',
'DEVICE': 'bond0.10', 'MTU': '1500',
> 'ONBOOT':
> >>> > >> 'yes'},
> >>> > >> 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid': 10,
> 'mtu':
> >>> > >> '1500',
> >>> > >> 'netmask': '255.255.255.0',
'ipv4addrs': ['43.25.17.16/24']
> >>> > >> <http://43.25.17.16/24%27%5D>},
'bond0.36': {'iface': 'bond0',
> >>> > >> 'addr':
> >>> > >> '', 'cfg': {'BRIDGE':
'VMNetwork', 'VLAN': 'yes', 'HOTPLUG':
'no',
> >>> > >> 'MTU':
> >>> > >> '1500', 'NM_CONTROLLED':
'no', 'DEVICE': 'bond0.36', 'ONBOOT':
> >>> > >> 'no'},
> >>> > >> 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid': 36,
> 'mtu':
> >>> > >> '1500',
> >>> > >> 'netmask': '',
'ipv4addrs': []}, 'bond1.100': {'iface':
'bond1',
> >>> > >> 'addr':
> >>> > >> '', 'cfg': {'BRIDGE':
'Internal', 'VLAN': 'yes', 'HOTPLUG':
'no',
> >>> > >> 'MTU':
> >>> > >> '9000', 'NM_CONTROLLED':
'no', 'DEVICE': 'bond1.100', 'ONBOOT':
> >>> > >> 'no'},
> >>> > >> 'ipv6addrs':
['fe80::210:18ff:fecd:daac/64'], 'vlanid': 100,
> 'mtu':
> >>> > >> '9000',
> >>> > >> 'netmask': '',
'ipv4addrs': []}}, 'cpuCores': '12',
'kvmEnabled':
> >>> > >> 'true',
> >>> > >> 'guestOverhead': '65',
'cpuThreads': '24', 'emulatedMachines':
> >>> > >> [u'rhel6.5.0', u'pc',
u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0',
> >>> > >> u'rhel6.1.0', u'rhel6.0.0',
u'rhel5.5.0', u'rhel5.4.4',
> >>> > >> u'rhel5.4.0'],
> >>> > >> 'operatingSystem': {'release':
'5.el6.centos.11.1', 'version':
> '6',
> >>> > >> 'name':
> >>> > >> 'RHEL'}, 'lastClient':
'10.10.10.2'}}
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
21:41:41,620::BindingXMLRPC::1132::vds::(wrapper) client
> >>> > >> [10.10.10.2]::call
> >>> > >> getHardwareInfo with () {}
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
21:41:41,621::BindingXMLRPC::1139::vds::(wrapper) return
> >>> > >> getHardwareInfo
> >>> > >> with {'status': {'message':
'Done', 'code': 0}, 'info':
> >>> > >> {'systemProductName': 'CS24-TY',
'systemSerialNumber': '7LWSPN1',
> >>> > >> 'systemFamily': 'Server',
'systemVersion': 'A00', 'systemUUID':
> >>> > >> '44454c4c-4c00-1057-8053-b7c04f504e31',
'systemManufacturer':
> >>> > >> 'Dell'}}
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
21:41:41,733::BindingXMLRPC::1132::vds::(wrapper) client
> >>> > >> [10.10.10.2]::call
> >>> > >> hostsList with () {} flowID [222e8036]
> >>> > >> Thread-13::ERROR::2014-11-24
> >>> > >>
21:41:44,753::BindingXMLRPC::1148::vds::(wrapper) vdsm exception
> >>> > >> occured
> >>> > >> Traceback (most recent call last):
> >>> > >>   File
"/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1135, in
> wrapper
> >>> > >>     res = f(*args, **kwargs)
> >>> > >>   File
"/usr/share/vdsm/gluster/api.py", line 54, in wrapper
> >>> > >>     rv = func(*args, **kwargs)
> >>> > >>   File
"/usr/share/vdsm/gluster/api.py", line 251, in hostsList
> >>> > >>     return {'hosts':
self.svdsmProxy.glusterPeerStatus()}
> >>> > >>   File "/usr/share/vdsm/supervdsm.py",
line 50, in __call__
> >>> > >>     return callMethod()
> >>> > >>   File "/usr/share/vdsm/supervdsm.py",
line 48, in <lambda>
> >>> > >>     **kwargs)
> >>> > >>   File "<string>", line 2, in
glusterPeerStatus
> >>> > >>   File
"/usr/lib64/python2.6/multiprocessing/managers.py", line
> 740,
> >>> > >> in
> >>> > >> _callmethod
> >>> > >>     raise convert_to_error(kind, result)
> >>> > >> GlusterCmdExecFailedException: Command execution
failed
> >>> > >> error: Connection failed. Please check if
gluster daemon is
> >>> > >> operational.
> >>> > >> return code: 1
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
21:41:50,949::task::595::Storage.TaskManager.Task::(_updateState)
> >>> > >>
Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving from state
> init
> >>> > >> ->
> >>> > >> state preparing
> >>> > >> Thread-13::INFO::2014-11-24
> >>> > >>
21:41:50,950::logUtils::44::dispatcher::(wrapper) Run and protect:
> >>> > >> repoStats(options=None)
> >>> > >> Thread-13::INFO::2014-11-24
> >>> > >>
21:41:50,950::logUtils::47::dispatcher::(wrapper) Run and protect:
> >>> > >> repoStats, Return response: {}
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
21:41:50,950::task::1191::Storage.TaskManager.Task::(prepare)
> >>> > >>
Task=`c9042986-c978-4b08-adb2-616f5299e115`::finished: {}
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
21:41:50,950::task::595::Storage.TaskManager.Task::(_updateState)
> >>> > >>
Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving from state
> >>> > >> preparing
> >>> > >> ->
> >>> > >> state finished
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
> >>> > >>
>
21:41:50,951::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
> >>> > >> Owner.releaseAll requests {} resources {}
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
> >>> > >>
>
21:41:50,951::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
> >>> > >> Owner.cancelAll requests {}
> >>> > >> Thread-13::DEBUG::2014-11-24
> >>> > >>
21:41:50,951::task::993::Storage.TaskManager.Task::(_decref)
> >>> > >> Task=`c9042986-c978-4b08-adb2-616f5299e115`::ref
0 aborting False
> >>> > >>  -------------------------------
> >>> > >>
> >>> > >>  [root at compute4 ~]# service glusterd status
> >>> > >> glusterd is stopped
> >>> > >> [root at compute4 ~]# chkconfig --list | grep
glusterd
> >>> > >> glusterd        0:off   1:off   2:on    3:on   
4:on    5:on
> >>> > >> 6:off
> >>> > >> [root at compute4 ~]#
> >>> > >>
> >>> > >>  Thanks,
> >>> > >> Punit
> >>> > >>
> >>> > >> On Mon, Nov 24, 2014 at 6:36 PM, Kanagaraj
<kmayilsa at redhat.com>
> >>> > >> wrote:
> >>> > >>
> >>> > >>>  Can you send the corresponding error in
vdsm.log from the host?
> >>> > >>>
> >>> > >>> Also check if glusterd service is running.
> >>> > >>>
> >>> > >>> Thanks,
> >>> > >>> Kanagaraj
> >>> > >>>
> >>> > >>>
> >>> > >>> On 11/24/2014 03:39 PM, Punit Dambiwal
wrote:
> >>> > >>>
> >>> > >>>   Hi,
> >>> > >>>
> >>> > >>>  After reboot my Hypervisior host can not
activate again in the
> >>> > >>> cluster
> >>> > >>> and failed with the following error :-
> >>> > >>>
> >>> > >>>  Gluster command [<UNKNOWN>] failed on
server...
> >>> > >>>
> >>> > >>>  Engine logs :-
> >>> > >>>
> >>> > >>>  2014-11-24 18:05:28,397 INFO
> >>> > >>>
> >>> > >>>
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> >>> > >>> (DefaultQuartzScheduler_Worker-64) START,
> >>> > >>> GlusterVolumesListVDSCommand(HostName =
Compute4, HostId > >>> > >>>
33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 5f251c90
> >>> > >>> 2014-11-24 18:05:30,609 INFO
> >>> > >>>
> >>> > >>>
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> >>> > >>> (DefaultQuartzScheduler_Worker-64) FINISH,
> >>> > >>> GlusterVolumesListVDSCommand,
> >>> > >>> return:
> >>> > >>>
> >>> > >>>
>
{26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity
at d95203e0
> },
> >>> > >>> log id: 5f251c90
> >>> > >>> 2014-11-24 18:05:33,768 INFO
> >>> > >>> 
[org.ovirt.engine.core.bll.ActivateVdsCommand]
> >>> > >>> (ajp--127.0.0.1-8702-8)
> >>> > >>> [287d570d] Lock Acquired to object
EngineLock [exclusiveLocks> key:
> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a value:
VDS
> >>> > >>> , sharedLocks= ]
> >>> > >>> 2014-11-24 18:05:33,795 INFO
> >>> > >>> 
[org.ovirt.engine.core.bll.ActivateVdsCommand]
> >>> > >>> (org.ovirt.thread.pool-8-thread-45)
[287d570d] Running command:
> >>> > >>> ActivateVdsCommand internal: false. Entities
affected :  ID:
> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type:
VDSAction group
> >>> > >>> MANIPULATE_HOST
> >>> > >>> with role type ADMIN
> >>> > >>> 2014-11-24 18:05:33,796 INFO
> >>> > >>> 
[org.ovirt.engine.core.bll.ActivateVdsCommand]
> >>> > >>> (org.ovirt.thread.pool-8-thread-45)
[287d570d] Before acquiring
> >>> > >>> lock in
> >>> > >>> order to prevent monitoring for host
Compute5 from data-center
> >>> > >>> SV_WTC
> >>> > >>> 2014-11-24 18:05:33,797 INFO
> >>> > >>> 
[org.ovirt.engine.core.bll.ActivateVdsCommand]
> >>> > >>> (org.ovirt.thread.pool-8-thread-45)
[287d570d] Lock acquired,
> from
> >>> > >>> now a
> >>> > >>> monitoring of host will be skipped for host
Compute5 from
> >>> > >>> data-center
> >>> > >>> SV_WTC
> >>> > >>> 2014-11-24 18:05:33,817 INFO
> >>> > >>> 
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> >>> > >>> (org.ovirt.thread.pool-8-thread-45)
[287d570d] START,
> >>> > >>> SetVdsStatusVDSCommand(HostName = Compute5,
HostId > >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a,
status=Unassigned,
> >>> > >>> nonOperationalReason=NONE,
stopSpmFailureLogged=false), log id:
> >>> > >>> 1cbc7311
> >>> > >>> 2014-11-24 18:05:33,820 INFO
> >>> > >>> 
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> >>> > >>> (org.ovirt.thread.pool-8-thread-45)
[287d570d] FINISH,
> >>> > >>> SetVdsStatusVDSCommand, log id: 1cbc7311
> >>> > >>> 2014-11-24 18:05:34,086 INFO
> >>> > >>> 
[org.ovirt.engine.core.bll.ActivateVdsCommand]
> >>> > >>> (org.ovirt.thread.pool-8-thread-45) Activate
finished. Lock
> >>> > >>> released.
> >>> > >>> Monitoring can run now for host Compute5
from data-center SV_WTC
> >>> > >>> 2014-11-24 18:05:34,088 INFO
> >>> > >>>
> >>> > >>>
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >>> > >>> (org.ovirt.thread.pool-8-thread-45)
Correlation ID: 287d570d, Job
> >>> > >>> ID:
> >>> > >>> 5ef8e4d6-b2bc-469e-8e81-7ef74b2a001a, Call
Stack: null, Custom
> >>> > >>> Event ID:
> >>> > >>> -1, Message: Host Compute5 was activated by
admin.
> >>> > >>> 2014-11-24 18:05:34,090 INFO
> >>> > >>> 
[org.ovirt.engine.core.bll.ActivateVdsCommand]
> >>> > >>> (org.ovirt.thread.pool-8-thread-45) Lock
freed to object
> EngineLock
> >>> > >>> [exclusiveLocks= key:
0bf6b00f-7947-4411-b55a-cc5eea2b381a value:
> >>> > >>> VDS
> >>> > >>> , sharedLocks= ]
> >>> > >>> 2014-11-24 18:05:35,792 INFO
> >>> > >>>
> >>> > >>>
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> >>> > >>> (DefaultQuartzScheduler_Worker-55)
[3706e836] START,
> >>> > >>> GlusterVolumesListVDSCommand(HostName =
Compute4, HostId > >>> > >>>
33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 48a0c832
> >>> > >>> 2014-11-24 18:05:37,064 INFO
> >>> > >>>
> >>> > >>>
> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
> >>> > >>> (DefaultQuartzScheduler_Worker-69) START,
> >>> > >>> GetHardwareInfoVDSCommand(HostName =
Compute5, HostId > >>> > >>>
0bf6b00f-7947-4411-b55a-cc5eea2b381a,
> >>> > >>>
vds=Host[Compute5,0bf6b00f-7947-4411-b55a-cc5eea2b381a]), log id:
> >>> > >>> 6d560cc2
> >>> > >>> 2014-11-24 18:05:37,074 INFO
> >>> > >>>
> >>> > >>>
> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
> >>> > >>> (DefaultQuartzScheduler_Worker-69) FINISH,
> >>> > >>> GetHardwareInfoVDSCommand, log
> >>> > >>> id: 6d560cc2
> >>> > >>> 2014-11-24 18:05:37,093 WARN
> >>> > >>> 
[org.ovirt.engine.core.vdsbroker.VdsManager]
> >>> > >>> (DefaultQuartzScheduler_Worker-69) Host
Compute5 is running with
> >>> > >>> disabled
> >>> > >>> SELinux.
> >>> > >>> 2014-11-24 18:05:37,127 INFO
> >>> > >>>
> >>> > >>>
> [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[2b4a51cf] Running command:
> >>> > >>> HandleVdsCpuFlagsOrClusterChangedCommand
internal: true. Entities
> >>> > >>> affected
> >>> > >>> :  ID: 0bf6b00f-7947-4411-b55a-cc5eea2b381a
Type: VDS
> >>> > >>> 2014-11-24 18:05:37,147 INFO
> >>> > >>>
> >>> > >>>
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[2b4a51cf] START,
> >>> > >>> GlusterServersListVDSCommand(HostName =
Compute5, HostId > >>> > >>>
0bf6b00f-7947-4411-b55a-cc5eea2b381a), log id: 4faed87
> >>> > >>> 2014-11-24 18:05:37,164 INFO
> >>> > >>>
> >>> > >>>
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[2b4a51cf] FINISH,
> >>> > >>> GlusterServersListVDSCommand, log id:
4faed87
> >>> > >>> 2014-11-24 18:05:37,189 INFO
> >>> > >>> 
[org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[4a84c4e5] Running command:
> >>> > >>> SetNonOperationalVdsCommand internal: true.
Entities affected :
> >>> > >>> ID:
> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type:
VDS
> >>> > >>> 2014-11-24 18:05:37,206 INFO
> >>> > >>> 
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[4a84c4e5] START,
> >>> > >>> SetVdsStatusVDSCommand(HostName = Compute5,
HostId > >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a,
status=NonOperational,
> >>> > >>> nonOperationalReason=GLUSTER_COMMAND_FAILED,
> >>> > >>> stopSpmFailureLogged=false),
> >>> > >>> log id: fed5617
> >>> > >>> 2014-11-24 18:05:37,209 INFO
> >>> > >>> 
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[4a84c4e5] FINISH,
> >>> > >>> SetVdsStatusVDSCommand, log id: fed5617
> >>> > >>> 2014-11-24 18:05:37,223 ERROR
> >>> > >>>
> >>> > >>>
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[4a84c4e5] Correlation ID:
> >>> > >>> 4a84c4e5,
> >>> > >>> Job
> >>> > >>> ID: 4bfd4a6d-c3ef-468f-a40e-a3a6ca13011b,
Call Stack: null,
> Custom
> >>> > >>> Event
> >>> > >>> ID: -1, Message: Gluster command
[<UNKNOWN>] failed on server
> >>> > >>> Compute5.
> >>> > >>> 2014-11-24 18:05:37,243 INFO
> >>> > >>>
> >>> > >>>
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[4a84c4e5] Correlation ID:
> null,
> >>> > >>> Call
> >>> > >>> Stack: null, Custom Event ID: -1, Message:
Status of host
> Compute5
> >>> > >>> was
> >>> > >>> set
> >>> > >>> to NonOperational.
> >>> > >>> 2014-11-24 18:05:37,272 INFO
> >>> > >>> 
[org.ovirt.engine.core.bll.HandleVdsVersionCommand]
> >>> > >>> (DefaultQuartzScheduler_Worker-69) [a0c8a7f]
Running command:
> >>> > >>> HandleVdsVersionCommand internal: true.
Entities affected :  ID:
> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a Type:
VDS
> >>> > >>> 2014-11-24 18:05:37,274 INFO
> >>> > >>> 
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> >>> > >>> (DefaultQuartzScheduler_Worker-69) [a0c8a7f]
Host
> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a :
Compute5 is already in
> >>> > >>> NonOperational status for reason
GLUSTER_COMMAND_FAILED.
> >>> > >>> SetNonOperationalVds command is skipped.
> >>> > >>> 2014-11-24 18:05:38,065 INFO
> >>> > >>>
> >>> > >>>
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> >>> > >>> (DefaultQuartzScheduler_Worker-55)
[3706e836] FINISH,
> >>> > >>> GlusterVolumesListVDSCommand, return:
> >>> > >>>
> >>> > >>>
>
{26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity
at 4e72a1b1
> },
> >>> > >>> log id: 48a0c832
> >>> > >>> 2014-11-24 18:05:43,243 INFO
> >>> > >>>
> >>> > >>>
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> >>> > >>> (DefaultQuartzScheduler_Worker-35) START,
> >>> > >>> GlusterVolumesListVDSCommand(HostName =
Compute4, HostId > >>> > >>>
33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 3ce13ebc
> >>> > >>> ^C
> >>> > >>> [root at ccr01 ~]#
> >>> > >>>
> >>> > >>>  Thanks,
> >>> > >>> Punit
> >>> > >>>
> >>> > >>>
> >>> > >>> 
_______________________________________________
> >>> > >>> Users mailing
> >>> > >>> listUsers at
ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
> >>> > >>>
> >>> > >>>
> >>> > >>>
> >>> > >>
> >>> > >>
> >>> > >
> >>> > >
> >>> >
> >>
> >>
> >
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141127/0ef5b1d0/attachment.html>

Hi,

Can Any body help me on this ??

On Thu, Nov 27, 2014 at 9:29 AM, Punit Dambiwal <hypunit at gmail.com>
wrote:
> Hi Kaushal,
>
> Thanks for the detailed reply....let me explain my setup first :-
>
> 1. Ovirt Engine
> 2. 4* host as well as storage machine (Host and gluster combined)
> 3. Every host has 24 bricks...
>
> Now whenever the host machine reboot...it can come up but can not join the
> cluster again and through the following error "Gluster command
[<UNKNOWN>]
> failed on server.."
>
> Please check my comment in line :-
>
> 1. Use the same string for doing the peer probe and for the brick address
> during volume create/add-brick. Ideally, we suggest you use properly
> resolvable FQDNs everywhere. If that is not possible, then use only IP
> addresses. Try to avoid short names.
> ---------------
> [root at cpu05 ~]# gluster peer status
> Number of Peers: 3
>
> Hostname: cpu03.stack.com
> Uuid: 5729b8c4-e80d-4353-b456-6f467bddbdfb
> State: Peer in Cluster (Connected)
>
> Hostname: cpu04.stack.com
> Uuid: d272b790-c4b2-4bed-ba68-793656e6d7b0
> State: Peer in Cluster (Connected)
> Other names:
> 10.10.0.8
>
> Hostname: cpu02.stack.com
> Uuid: 8d8a7041-950e-40d0-85f9-58d14340ca25
> State: Peer in Cluster (Connected)
> [root at cpu05 ~]#
> ----------------
> 2. During boot up, make sure to launch glusterd only after the network is
> up. This will allow the new peer identification mechanism to do its
> job correctly.
> >> I think the service itself doing the same job....
>
> [root at cpu05 ~]# cat /usr/lib/systemd/system/glusterd.service
> [Unit]
> Description=GlusterFS, a clustered file-system server
> After=network.target rpcbind.service
> Before=network-online.target
>
> [Service]
> Type=forking
> PIDFile=/var/run/glusterd.pid
> LimitNOFILE=65536
> ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid
> KillMode=process
>
> [Install]
> WantedBy=multi-user.target
> [root at cpu05 ~]#
> --------------------
>
> gluster logs :-
>
> [2014-11-24 09:22:22.147471] I [MSGID: 100030] [glusterfsd.c:2018:main]
> 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.6.1
> (args: /usr/sbin/glusterd -p /var/run/glusterd.pid)
> [2014-11-24 09:22:22.151565] I [glusterd.c:1214:init] 0-management:
> Maximum allowed open file descriptors set to 65536
> [2014-11-24 09:22:22.151599] I [glusterd.c:1259:init] 0-management: Using
> /var/lib/glusterd as working directory
> [2014-11-24 09:22:22.155216] W [rdma.c:4195:__gf_rdma_ctx_create]
> 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such
device)
> [2014-11-24 09:22:22.155264] E [rdma.c:4483:init] 0-rdma.management:
> Failed to initialize IB Device
> [2014-11-24 09:22:22.155285] E [rpc-transport.c:333:rpc_transport_load]
> 0-rpc-transport: 'rdma' initialization failed
> [2014-11-24 09:22:22.155354] W [rpcsvc.c:1524:rpcsvc_transport_create]
> 0-rpc-service: cannot create listener, initing the transport failed
> [2014-11-24 09:22:22.156290] I
> [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication
> module not installed in the system
> [2014-11-24 09:22:22.161318] I
> [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved
> op-version: 30600
> [2014-11-24 09:22:22.821800] I
> [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management:
> connect returned 0
> [2014-11-24 09:22:22.825810] I
> [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management:
> connect returned 0
> [2014-11-24 09:22:22.828705] I
> [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management:
> connect returned 0
> [2014-11-24 09:22:22.828771] I [rpc-clnt.c:969:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2014-11-24 09:22:22.832670] I [rpc-clnt.c:969:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2014-11-24 09:22:22.835919] I [rpc-clnt.c:969:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2014-11-24 09:22:22.840209] E
> [glusterd-store.c:4248:glusterd_resolve_all_bricks] 0-glusterd: resolve
> brick failed in restore
> [2014-11-24 09:22:22.840233] E [xlator.c:425:xlator_init] 0-management:
> Initialization of volume 'management' failed, review your volfile
again
> [2014-11-24 09:22:22.840245] E [graph.c:322:glusterfs_graph_init]
> 0-management: initializing translator failed
> [2014-11-24 09:22:22.840264] E [graph.c:525:glusterfs_graph_activate]
> 0-graph: init failed
> [2014-11-24 09:22:22.840754] W [glusterfsd.c:1194:cleanup_and_exit] (-->
> 0-: received signum (0), shutting down
>
> Thanks,
> Punit
>
>
>
>
> On Wed, Nov 26, 2014 at 7:14 PM, Kaushal M <kshlmster at gmail.com>
wrote:
>
>> Based on the logs I can guess that glusterd is being started before
>> the network has come up and that the addresses given to bricks do not
>> directly match the addresses used in during peer probe.
>>
>> The gluster_after_reboot log has the line "[2014-11-25
>> 06:46:09.972113] E [glusterd-store.c:2632:glusterd_resolve_all_bricks]
>> 0-glusterd: resolve brick failed in restore".
>>
>> Brick resolution fails when glusterd cannot match the address for the
>> brick, with one of the peers. Brick resolution happens in two phases,
>> 1. We first try to identify the peer by performing string comparisions
>> with the brick address and the peer addresses (The peer names will be
>> the names/addresses that were given when the peer was probed).
>> 2. If we don't find a match from step 1, we will then resolve all
the
>> brick address and the peer addresses into addrinfo structs, and then
>> compare these structs to find a match. This process should generally
>> find a match if available. This will fail only if the network is not
>> up yet as we cannot resolve addresses.
>>
>> The above steps are applicable only to glusterfs versions >=3.6.
They
>> were introduced to reduce problems with peer identification, like the
>> one you encountered
>>
>> Since both of the steps failed to find a match in one run, but
>> succeeded later, we can come to the conclusion that,
>> a) the bricks don't have the exact same string used in peer probe
for
>> their addresses as step 1 failed, and
>> b) the network was not up in the initial run, as step 2 failed during
>> the initial run, but passed in the second run.
>>
>> Please let me know if my conclusion is correct.
>>
>> If it is, you can solve your problem in two ways.
>> 1. Use the same string for doing the peer probe and for the brick
>> address during volume create/add-brick. Ideally, we suggest you use
>> properly resolvable FQDNs everywhere. If that is not possible, then
>> use only IP addresses. Try to avoid short names.
>> 2. During boot up, make sure to launch glusterd only after the network
>> is up. This will allow the new peer identification mechanism to do its
>> job correctly.
>>
>>
>> If you have already followed these steps and yet still hit the
>> problem, then please provide more information (setup, logs, etc.). It
>> could be much different problem that you are facing.
>>
>> ~kaushal
>>
>> On Wed, Nov 26, 2014 at 4:01 PM, Punit Dambiwal <hypunit at
gmail.com>
>> wrote:
>> > Is there any one can help on this ??
>> >
>> > Thanks,
>> > punit
>> >
>> > On Wed, Nov 26, 2014 at 9:42 AM, Punit Dambiwal <hypunit at
gmail.com>
>> wrote:
>> >>
>> >> Hi,
>> >>
>> >> My Glusterfs version is :- glusterfs-3.6.1-1.el7
>> >>
>> >> On Wed, Nov 26, 2014 at 1:59 AM, Kanagaraj Mayilsamy <
>> kmayilsa at redhat.com>
>> >> wrote:
>> >>>
>> >>> [+Gluster-users at gluster.org]
>> >>>
>> >>> "Initialization of volume 'management'
failed, review your volfile
>> >>> again", glusterd throws this error when the service
is started
>> automatically
>> >>> after the reboot. But the service is successfully started
later
>> manually by
>> >>> the user.
>> >>>
>> >>> can somebody from gluster-users please help on this?
>> >>>
>> >>> glusterfs version: 3.5.1
>> >>>
>> >>> Thanks,
>> >>> Kanagaraj
>> >>>
>> >>> ----- Original Message -----
>> >>> > From: "Punit Dambiwal" <hypunit at
gmail.com>
>> >>> > To: "Kanagaraj" <kmayilsa at
redhat.com>
>> >>> > Cc: users at ovirt.org
>> >>> > Sent: Tuesday, November 25, 2014 7:24:45 PM
>> >>> > Subject: Re: [ovirt-users] Gluster command
[<UNKNOWN>] failed on
>> >>> > server...
>> >>> >
>> >>> > Hi Kanagraj,
>> >>> >
>> >>> > Please check the attached log files....i didn't
find any thing
>> >>> > special....
>> >>> >
>> >>> > On Tue, Nov 25, 2014 at 12:12 PM, Kanagaraj
<kmayilsa at redhat.com>
>> >>> > wrote:
>> >>> >
>> >>> > >  Do you see any errors in
>> >>> > >
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log or vdsm.log when
>> >>> > > the
>> >>> > > service is trying to start automatically after
the reboot?
>> >>> > >
>> >>> > > Thanks,
>> >>> > > Kanagaraj
>> >>> > >
>> >>> > >
>> >>> > > On 11/24/2014 08:13 PM, Punit Dambiwal wrote:
>> >>> > >
>> >>> > > Hi Kanagaraj,
>> >>> > >
>> >>> > >  Yes...once i will start the gluster service and
then vdsmd ...the
>> >>> > > host
>> >>> > > can connect to cluster...but the question is why
it's not started
>> >>> > > even it
>> >>> > > has chkconfig enabled...
>> >>> > >
>> >>> > >  I have tested it in two host cluster
environment...(Centos 6.6
>> and
>> >>> > > centos 7.0) on both hypervisior
cluster..it's failed to reconnect
>> in
>> >>> > > to
>> >>> > > cluster after reboot....
>> >>> > >
>> >>> > >  In both the environment glusterd enabled for
next boot....but
>> it's
>> >>> > > failed with the same error....seems it's bug
in either gluster or
>> >>> > > Ovirt ??
>> >>> > >
>> >>> > >  Please help me to find the workaround here if
can not resolve
>> >>> > > it...as
>> >>> > > without this the Host machine can not connect
after reboot....that
>> >>> > > means
>> >>> > > engine will consider it as down and every time
need to manually
>> start
>> >>> > > the
>> >>> > > gluster service and vdsmd... ??
>> >>> > >
>> >>> > >  Thanks,
>> >>> > > Punit
>> >>> > >
>> >>> > > On Mon, Nov 24, 2014 at 10:20 PM, Kanagaraj
<kmayilsa at redhat.com>
>> >>> > > wrote:
>> >>> > >
>> >>> > >>  From vdsm.log "error: Connection
failed. Please check if gluster
>> >>> > >> daemon
>> >>> > >> is operational."
>> >>> > >>
>> >>> > >> Starting glusterd service should fix this
issue. 'service
>> glusterd
>> >>> > >> start'
>> >>> > >> But i am wondering why the glusterd was not
started automatically
>> >>> > >> after
>> >>> > >> the reboot.
>> >>> > >>
>> >>> > >> Thanks,
>> >>> > >> Kanagaraj
>> >>> > >>
>> >>> > >>
>> >>> > >>
>> >>> > >> On 11/24/2014 07:18 PM, Punit Dambiwal
wrote:
>> >>> > >>
>> >>> > >> Hi Kanagaraj,
>> >>> > >>
>> >>> > >>  Please find the attached VDSM logs :-
>> >>> > >>
>> >>> > >>  ----------------
>> >>> > >>  Thread-13::DEBUG::2014-11-24
>> >>> > >>
>> >>> > >>
>>
21:41:17,182::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
>> >>> > >> Owner.cancelAll requests {}
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:17,182::task::993::Storage.TaskManager.Task::(_decref)
>> >>> > >>
Task=`1691d409-9b27-4585-8281-5ec26154367a`::ref 0 aborting False
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:32,393::task::595::Storage.TaskManager.Task::(_updateState)
>> >>> > >>
Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving from state
>> init
>> >>> > >> ->
>> >>> > >> state preparing
>> >>> > >> Thread-13::INFO::2014-11-24
>> >>> > >>
21:41:32,393::logUtils::44::dispatcher::(wrapper) Run and
>> protect:
>> >>> > >> repoStats(options=None)
>> >>> > >> Thread-13::INFO::2014-11-24
>> >>> > >>
21:41:32,393::logUtils::47::dispatcher::(wrapper) Run and
>> protect:
>> >>> > >> repoStats, Return response: {}
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:32,393::task::1191::Storage.TaskManager.Task::(prepare)
>> >>> > >>
Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::finished: {}
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:32,394::task::595::Storage.TaskManager.Task::(_updateState)
>> >>> > >>
Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::moving from state
>> >>> > >> preparing
>> >>> > >> ->
>> >>> > >> state finished
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
>> >>> > >>
>>
21:41:32,394::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
>> >>> > >> Owner.releaseAll requests {} resources {}
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
>> >>> > >>
>>
21:41:32,394::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
>> >>> > >> Owner.cancelAll requests {}
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:32,394::task::993::Storage.TaskManager.Task::(_decref)
>> >>> > >>
Task=`994c7bc3-a236-4d03-a732-e068c7ed9ed4`::ref 0 aborting False
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:41,550::BindingXMLRPC::1132::vds::(wrapper) client
>> >>> > >> [10.10.10.2]::call
>> >>> > >> getCapabilities with () {}
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >> 21:41:41,553::utils::738::root::(execCmd)
>> >>> > >> /sbin/ip route show to 0.0.0.0/0 table all
(cwd None)
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >> 21:41:41,560::utils::758::root::(execCmd)
>> >>> > >> SUCCESS: <err> = '';
<rc> = 0
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:41,588::caps::728::root::(_getKeyPackages) rpm package
>> >>> > >> ('gluster-swift',) not found
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:41,592::caps::728::root::(_getKeyPackages) rpm package
>> >>> > >> ('gluster-swift-object',) not found
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:41,593::caps::728::root::(_getKeyPackages) rpm package
>> >>> > >> ('gluster-swift-plugin',) not found
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:41,598::caps::728::root::(_getKeyPackages) rpm package
>> >>> > >> ('gluster-swift-account',) not found
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:41,598::caps::728::root::(_getKeyPackages) rpm package
>> >>> > >> ('gluster-swift-proxy',) not found
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:41,598::caps::728::root::(_getKeyPackages) rpm package
>> >>> > >> ('gluster-swift-doc',) not found
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:41,599::caps::728::root::(_getKeyPackages) rpm package
>> >>> > >> ('gluster-swift-container',) not
found
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:41,599::caps::728::root::(_getKeyPackages) rpm package
>> >>> > >> ('glusterfs-geo-replication',) not
found
>> >>> > >> Thread-13::DEBUG::2014-11-24
21:41:41,600::caps::646::root::(get)
>> >>> > >> VirtioRNG DISABLED: libvirt version
0.10.2-29.el6_5.9 required >>> >>> > >> 0.10.2-31
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:41,603::BindingXMLRPC::1139::vds::(wrapper) return
>> >>> > >> getCapabilities
>> >>> > >> with {'status': {'message':
'Done', 'code': 0}, 'info':
>> >>> > >> {'HBAInventory':
>> >>> > >> {'iSCSI': [{'InitiatorName':
>> >>> > >>
'iqn.1994-05.com.redhat:32151ce183c8'}],
>> >>> > >> 'FC':
>> >>> > >> []}, 'packages2': {'kernel':
{'release': '431.el6.x86_64',
>> >>> > >> 'buildtime':
>> >>> > >> 1385061309.0, 'version':
'2.6.32'}, 'glusterfs-rdma': {'release':
>> >>> > >> '1.el6',
>> >>> > >> 'buildtime': 1403622628L,
'version': '3.5.1'}, 'glusterfs-fuse':
>> >>> > >> {'release': '1.el6',
'buildtime': 1403622628L, 'version':
>> '3.5.1'},
>> >>> > >> 'spice-server': {'release':
'6.el6_5.2', 'buildtime':
>> 1402324637L,
>> >>> > >> 'version': '0.12.4'},
'vdsm': {'release': '1.gitdb83943.el6',
>> >>> > >> 'buildtime':
>> >>> > >> 1412784567L, 'version':
'4.16.7'}, 'qemu-kvm': {'release':
>> >>> > >> '2.415.el6_5.10',
'buildtime': 1402435700L, 'version':
>> '0.12.1.2'},
>> >>> > >> 'qemu-img': {'release':
'2.415.el6_5.10', 'buildtime':
>> 1402435700L,
>> >>> > >> 'version': '0.12.1.2'},
'libvirt': {'release': '29.el6_5.9',
>> >>> > >> 'buildtime':
>> >>> > >> 1402404612L, 'version':
'0.10.2'}, 'glusterfs': {'release':
>> '1.el6',
>> >>> > >> 'buildtime': 1403622628L,
'version': '3.5.1'}, 'mom': {'release':
>> >>> > >> '2.el6',
>> >>> > >> 'buildtime': 1403794344L,
'version': '0.4.1'},
>> 'glusterfs-server':
>> >>> > >> {'release': '1.el6',
'buildtime': 1403622628L, 'version':
>> '3.5.1'}},
>> >>> > >> 'numaNodeDistance': {'1':
[20, 10], '0': [10, 20]}, 'cpuModel':
>> >>> > >> 'Intel(R)
>> >>> > >> Xeon(R) CPU           X5650  @ 2.67GHz',
'liveMerge': 'false',
>> >>> > >> 'hooks':
>> >>> > >> {},
>> >>> > >> 'cpuSockets': '2',
'vmTypes': ['kvm'], 'selinux': {'mode':
'1'},
>> >>> > >> 'kdumpStatus': 0,
'supportedProtocols': ['2.2', '2.3'],
>> 'networks':
>> >>> > >> {'ovirtmgmt': {'iface':
u'bond0.10', 'addr': '43.252.176.16',
>> >>> > >> 'bridged':
>> >>> > >> False, 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'mtu':
>> '1500',
>> >>> > >> 'bootproto4': 'none',
'netmask': '255.255.255.0', 'ipv4addrs': ['
>> >>> > >> 43.252.176.16/24'
<http://43.252.176.16/24%27>], 'interface':
>> >>> > >> u'bond0.10', 'ipv6gateway':
'::', 'gateway': '43.25.17.1'},
>> >>> > >> 'Internal':
>> >>> > >> {'iface': 'Internal',
'addr': '', 'cfg': {'DEFROUTE': 'no',
>> >>> > >> 'HOTPLUG':
>> >>> > >> 'no', 'MTU': '9000',
'DELAY': '0', 'NM_CONTROLLED': 'no',
>> >>> > >> 'BOOTPROTO':
>> >>> > >> 'none', 'STP':
'off', 'DEVICE': 'Internal', 'TYPE':
'Bridge',
>> >>> > >> 'ONBOOT':
>> >>> > >> 'no'}, 'bridged': True,
'ipv6addrs':
>> >>> > >> ['fe80::210:18ff:fecd:daac/64'],
>> >>> > >> 'gateway': '',
'bootproto4': 'none', 'netmask': '',
'stp': 'off',
>> >>> > >> 'ipv4addrs': [], 'mtu':
'9000', 'ipv6gateway': '::', 'ports':
>> >>> > >> ['bond1.100']}, 'storage':
{'iface': u'bond1', 'addr':
>> '10.10.10.6',
>> >>> > >> 'bridged': False,
'ipv6addrs': ['fe80::210:18ff:fecd:daac/64'],
>> >>> > >> 'mtu':
>> >>> > >> '9000', 'bootproto4':
'none', 'netmask': '255.255.255.0',
>> >>> > >> 'ipv4addrs': ['
>> >>> > >> 10.10.10.6/24'
<http://10.10.10.6/24%27>], 'interface':
>> u'bond1',
>> >>> > >> 'ipv6gateway': '::',
'gateway': ''}, 'VMNetwork': {'iface':
>> >>> > >> 'VMNetwork',
>> >>> > >> 'addr': '', 'cfg':
{'DEFROUTE': 'no', 'HOTPLUG': 'no',
'MTU':
>> >>> > >> '1500',
>> >>> > >> 'DELAY': '0',
'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none',
'STP':
>> >>> > >> 'off',
>> >>> > >> 'DEVICE': 'VMNetwork',
'TYPE': 'Bridge', 'ONBOOT': 'no'},
>> 'bridged':
>> >>> > >> True,
>> >>> > >> 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'gateway': '',
>> >>> > >> 'bootproto4':
>> >>> > >> 'none', 'netmask':
'', 'stp': 'off', 'ipv4addrs': [],
'mtu':
>> '1500',
>> >>> > >> 'ipv6gateway': '::',
'ports': ['bond0.36']}}, 'bridges':
>> >>> > >> {'Internal':
>> >>> > >> {'addr': '', 'cfg':
{'DEFROUTE': 'no', 'HOTPLUG': 'no',
'MTU':
>> >>> > >> '9000',
>> >>> > >> 'DELAY': '0',
'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none',
'STP':
>> >>> > >> 'off',
>> >>> > >> 'DEVICE': 'Internal',
'TYPE': 'Bridge', 'ONBOOT': 'no'},
>> >>> > >> 'ipv6addrs':
>> >>> > >> ['fe80::210:18ff:fecd:daac/64'],
'mtu': '9000', 'netmask': '',
>> >>> > >> 'stp':
>> >>> > >> 'off', 'ipv4addrs': [],
'ipv6gateway': '::', 'gateway': '',
>> 'opts':
>> >>> > >> {'topology_change_detected':
'0', 'multicast_last_member_count':
>> >>> > >> '2',
>> >>> > >> 'hash_elasticity': '4',
'multicast_query_response_interval':
>> '999',
>> >>> > >> 'multicast_snooping': '1',
'multicast_startup_query_interval':
>> >>> > >> '3124',
>> >>> > >> 'hello_timer': '31',
'multicast_querier_interval': '25496',
>> >>> > >> 'max_age':
>> >>> > >> '1999', 'hash_max':
'512', 'stp_state': '0', 'root_id':
>> >>> > >> '8000.001018cddaac',
'priority': '32768',
>> >>> > >> 'multicast_membership_interval':
>> >>> > >> '25996', 'root_path_cost':
'0', 'root_port': '0',
>> >>> > >> 'multicast_querier':
>> >>> > >> '0',
>> >>> > >> 'multicast_startup_query_count':
'2', 'hello_time': '199',
>> >>> > >> 'topology_change': '0',
'bridge_id': '8000.001018cddaac',
>> >>> > >> 'topology_change_timer':
'0', 'ageing_time': '29995', 'gc_timer':
>> >>> > >> '31',
>> >>> > >> 'group_addr':
'1:80:c2:0:0:0', 'tcn_timer': '0',
>> >>> > >> 'multicast_query_interval':
'12498',
>> >>> > >> 'multicast_last_member_interval':
>> >>> > >> '99', 'multicast_router':
'1', 'forward_delay': '0'}, 'ports':
>> >>> > >> ['bond1.100']}, 'VMNetwork':
{'addr': '', 'cfg': {'DEFROUTE':
>> 'no',
>> >>> > >> 'HOTPLUG': 'no',
'MTU': '1500', 'DELAY': '0',
'NM_CONTROLLED':
>> 'no',
>> >>> > >> 'BOOTPROTO': 'none',
'STP': 'off', 'DEVICE': 'VMNetwork',
'TYPE':
>> >>> > >> 'Bridge',
>> >>> > >> 'ONBOOT': 'no'},
'ipv6addrs': ['fe80::62eb:69ff:fe20:b46c/64'],
>> >>> > >> 'mtu':
>> >>> > >> '1500', 'netmask':
'', 'stp': 'off', 'ipv4addrs': [],
>> 'ipv6gateway':
>> >>> > >> '::',
>> >>> > >> 'gateway': '',
'opts': {'topology_change_detected': '0',
>> >>> > >> 'multicast_last_member_count':
'2', 'hash_elasticity': '4',
>> >>> > >> 'multicast_query_response_interval':
'999', 'multicast_snooping':
>> >>> > >> '1',
>> >>> > >> 'multicast_startup_query_interval':
'3124', 'hello_timer': '131',
>> >>> > >> 'multicast_querier_interval':
'25496', 'max_age': '1999',
>> >>> > >> 'hash_max':
>> >>> > >> '512', 'stp_state':
'0', 'root_id': '8000.60eb6920b46c',
>> 'priority':
>> >>> > >> '32768',
'multicast_membership_interval': '25996',
>> 'root_path_cost':
>> >>> > >> '0',
>> >>> > >> 'root_port': '0',
'multicast_querier': '0',
>> >>> > >> 'multicast_startup_query_count':
'2', 'hello_time': '199',
>> >>> > >> 'topology_change': '0',
'bridge_id': '8000.60eb6920b46c',
>> >>> > >> 'topology_change_timer':
'0', 'ageing_time': '29995', 'gc_timer':
>> >>> > >> '31',
>> >>> > >> 'group_addr':
'1:80:c2:0:0:0', 'tcn_timer': '0',
>> >>> > >> 'multicast_query_interval':
'12498',
>> >>> > >> 'multicast_last_member_interval':
>> >>> > >> '99', 'multicast_router':
'1', 'forward_delay': '0'}, 'ports':
>> >>> > >> ['bond0.36']}}, 'uuid':
'44454C4C-4C00-1057-8053-B7C04F504E31',
>> >>> > >> 'lastClientIface': 'bond1',
'nics': {'eth3': {'permhwaddr':
>> >>> > >> '00:10:18:cd:da:ae', 'addr':
'', 'cfg': {'SLAVE': 'yes',
>> >>> > >> 'NM_CONTROLLED':
>> >>> > >> 'no', 'MTU': '9000',
'HWADDR': '00:10:18:cd:da:ae', 'MASTER':
>> >>> > >> 'bond1',
>> >>> > >> 'DEVICE': 'eth3',
'ONBOOT': 'no'}, 'ipv6addrs': [], 'mtu':
>> '9000',
>> >>> > >> 'netmask': '',
'ipv4addrs': [], 'hwaddr': '00:10:18:cd:da:ac',
>> >>> > >> 'speed':
>> >>> > >> 1000}, 'eth2':
{'permhwaddr': '00:10:18:cd:da:ac', 'addr': '',
>> >>> > >> 'cfg':
>> >>> > >> {'SLAVE': 'yes',
'NM_CONTROLLED': 'no', 'MTU': '9000',
'HWADDR':
>> >>> > >> '00:10:18:cd:da:ac',
'MASTER': 'bond1', 'DEVICE': 'eth2',
>> 'ONBOOT':
>> >>> > >> 'no'},
>> >>> > >> 'ipv6addrs': [], 'mtu':
'9000', 'netmask': '', 'ipv4addrs': [],
>> >>> > >> 'hwaddr':
>> >>> > >> '00:10:18:cd:da:ac',
'speed': 1000}, 'eth1': {'permhwaddr':
>> >>> > >> '60:eb:69:20:b4:6d', 'addr':
'', 'cfg': {'SLAVE': 'yes',
>> >>> > >> 'NM_CONTROLLED':
>> >>> > >> 'no', 'MTU': '1500',
'HWADDR': '60:eb:69:20:b4:6d', 'MASTER':
>> >>> > >> 'bond0',
>> >>> > >> 'DEVICE': 'eth1',
'ONBOOT': 'yes'}, 'ipv6addrs': [], 'mtu':
>> '1500',
>> >>> > >> 'netmask': '',
'ipv4addrs': [], 'hwaddr': '60:eb:69:20:b4:6c',
>> >>> > >> 'speed':
>> >>> > >> 1000}, 'eth0':
{'permhwaddr': '60:eb:69:20:b4:6c', 'addr': '',
>> >>> > >> 'cfg':
>> >>> > >> {'SLAVE': 'yes',
'NM_CONTROLLED': 'no', 'MTU': '1500',
'HWADDR':
>> >>> > >> '60:eb:69:20:b4:6c',
'MASTER': 'bond0', 'DEVICE': 'eth0',
>> 'ONBOOT':
>> >>> > >> 'yes'},
>> >>> > >> 'ipv6addrs': [], 'mtu':
'1500', 'netmask': '', 'ipv4addrs': [],
>> >>> > >> 'hwaddr':
>> >>> > >> '60:eb:69:20:b4:6c',
'speed': 1000}}, 'software_revision': '1',
>> >>> > >> 'clusterLevels': ['3.0',
'3.1', '3.2', '3.3', '3.4', '3.5'],
>> >>> > >> 'cpuFlags':
>> >>> > >>
>> >>> > >>
>>
u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,popcnt,aes,lahf_lm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270',
>> >>> > >> 'ISCSIInitiatorName':
'iqn.1994-05.com.redhat:32151ce183c8',
>> >>> > >> 'netConfigDirty': 'False',
'supportedENGINEs': ['3.0', '3.1',
>> '3.2',
>> >>> > >> '3.3',
>> >>> > >> '3.4', '3.5'],
'autoNumaBalancing': 2, 'reservedMem': '321',
>> >>> > >> 'bondings':
>> >>> > >> {'bond4': {'addr':
'', 'cfg': {}, 'mtu': '1500', 'netmask':
'',
>> >>> > >> 'slaves':
>> >>> > >> [], 'hwaddr':
'00:00:00:00:00:00'}, 'bond0': {'addr': '',
'cfg':
>> >>> > >> {'HOTPLUG': 'no',
'MTU': '1500', 'NM_CONTROLLED': 'no',
>> >>> > >> 'BONDING_OPTS':
>> >>> > >> 'mode=4 miimon=100',
'DEVICE': 'bond0', 'ONBOOT': 'yes'},
>> >>> > >> 'ipv6addrs':
>> >>> > >> ['fe80::62eb:69ff:fe20:b46c/64'],
'mtu': '1500', 'netmask': '',
>> >>> > >> 'ipv4addrs': [], 'hwaddr':
'60:eb:69:20:b4:6c', 'slaves':
>> ['eth0',
>> >>> > >> 'eth1'],
>> >>> > >> 'opts': {'miimon':
'100', 'mode': '4'}}, 'bond1': {'addr':
>> >>> > >> '10.10.10.6',
>> >>> > >> 'cfg': {'DEFROUTE':
'no', 'IPADDR': '10.10.10.6', 'HOTPLUG':
>> 'no',
>> >>> > >> 'MTU':
>> >>> > >> '9000', 'NM_CONTROLLED':
'no', 'NETMASK': '255.255.255.0',
>> >>> > >> 'BOOTPROTO':
>> >>> > >> 'none', 'BONDING_OPTS':
'mode=4 miimon=100', 'DEVICE': 'bond1',
>> >>> > >> 'ONBOOT':
>> >>> > >> 'no'}, 'ipv6addrs':
['fe80::210:18ff:fecd:daac/64'], 'mtu':
>> '9000',
>> >>> > >> 'netmask': '255.255.255.0',
'ipv4addrs': ['10.10.10.6/24'
>> >>> > >> <http://10.10.10.6/24%27>],
'hwaddr': '00:10:18:cd:da:ac',
>> 'slaves':
>> >>> > >> ['eth2', 'eth3'],
'opts': {'miimon': '100', 'mode': '4'}},
>> 'bond2':
>> >>> > >> {'addr': '', 'cfg':
{}, 'mtu': '1500', 'netmask': '',
'slaves':
>> [],
>> >>> > >> 'hwaddr':
'00:00:00:00:00:00'}, 'bond3': {'addr': '',
'cfg': {},
>> >>> > >> 'mtu':
>> >>> > >> '1500', 'netmask':
'', 'slaves': [], 'hwaddr':
>> >>> > >> '00:00:00:00:00:00'}},
>> >>> > >> 'software_version': '4.16',
'memSize': '24019', 'cpuSpeed':
>> >>> > >> '2667.000',
>> >>> > >> 'numaNodes': {u'1':
{'totalMemory': '12288', 'cpus': [6, 7, 8, 9,
>> >>> > >> 10, 11,
>> >>> > >> 18, 19, 20, 21, 22, 23]}, u'0':
{'totalMemory': '12278', 'cpus':
>> [0,
>> >>> > >> 1, 2,
>> >>> > >> 3, 4, 5, 12, 13, 14, 15, 16, 17]}},
'version_name': 'Snow Man',
>> >>> > >> 'vlans':
>> >>> > >> {'bond0.10': {'iface':
'bond0', 'addr': '43.25.17.16', 'cfg':
>> >>> > >> {'DEFROUTE':
>> >>> > >> 'yes', 'VLAN':
'yes', 'IPADDR': '43.25.17.16', 'HOTPLUG':
'no',
>> >>> > >> 'GATEWAY':
>> >>> > >> '43.25.17.1',
'NM_CONTROLLED': 'no', 'NETMASK':
'255.255.255.0',
>> >>> > >> 'BOOTPROTO': 'none',
'DEVICE': 'bond0.10', 'MTU': '1500',
>> 'ONBOOT':
>> >>> > >> 'yes'},
>> >>> > >> 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid': 10,
>> 'mtu':
>> >>> > >> '1500',
>> >>> > >> 'netmask': '255.255.255.0',
'ipv4addrs': ['43.25.17.16/24']
>> >>> > >> <http://43.25.17.16/24%27%5D>},
'bond0.36': {'iface': 'bond0',
>> >>> > >> 'addr':
>> >>> > >> '', 'cfg':
{'BRIDGE': 'VMNetwork', 'VLAN': 'yes',
'HOTPLUG':
>> 'no',
>> >>> > >> 'MTU':
>> >>> > >> '1500', 'NM_CONTROLLED':
'no', 'DEVICE': 'bond0.36', 'ONBOOT':
>> >>> > >> 'no'},
>> >>> > >> 'ipv6addrs':
['fe80::62eb:69ff:fe20:b46c/64'], 'vlanid': 36,
>> 'mtu':
>> >>> > >> '1500',
>> >>> > >> 'netmask': '',
'ipv4addrs': []}, 'bond1.100': {'iface':
'bond1',
>> >>> > >> 'addr':
>> >>> > >> '', 'cfg':
{'BRIDGE': 'Internal', 'VLAN': 'yes',
'HOTPLUG': 'no',
>> >>> > >> 'MTU':
>> >>> > >> '9000', 'NM_CONTROLLED':
'no', 'DEVICE': 'bond1.100', 'ONBOOT':
>> >>> > >> 'no'},
>> >>> > >> 'ipv6addrs':
['fe80::210:18ff:fecd:daac/64'], 'vlanid': 100,
>> 'mtu':
>> >>> > >> '9000',
>> >>> > >> 'netmask': '',
'ipv4addrs': []}}, 'cpuCores': '12',
'kvmEnabled':
>> >>> > >> 'true',
>> >>> > >> 'guestOverhead': '65',
'cpuThreads': '24', 'emulatedMachines':
>> >>> > >> [u'rhel6.5.0', u'pc',
u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0',
>> >>> > >> u'rhel6.1.0', u'rhel6.0.0',
u'rhel5.5.0', u'rhel5.4.4',
>> >>> > >> u'rhel5.4.0'],
>> >>> > >> 'operatingSystem':
{'release': '5.el6.centos.11.1', 'version':
>> '6',
>> >>> > >> 'name':
>> >>> > >> 'RHEL'}, 'lastClient':
'10.10.10.2'}}
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:41,620::BindingXMLRPC::1132::vds::(wrapper) client
>> >>> > >> [10.10.10.2]::call
>> >>> > >> getHardwareInfo with () {}
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:41,621::BindingXMLRPC::1139::vds::(wrapper) return
>> >>> > >> getHardwareInfo
>> >>> > >> with {'status': {'message':
'Done', 'code': 0}, 'info':
>> >>> > >> {'systemProductName':
'CS24-TY', 'systemSerialNumber': '7LWSPN1',
>> >>> > >> 'systemFamily': 'Server',
'systemVersion': 'A00', 'systemUUID':
>> >>> > >>
'44454c4c-4c00-1057-8053-b7c04f504e31', 'systemManufacturer':
>> >>> > >> 'Dell'}}
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:41,733::BindingXMLRPC::1132::vds::(wrapper) client
>> >>> > >> [10.10.10.2]::call
>> >>> > >> hostsList with () {} flowID [222e8036]
>> >>> > >> Thread-13::ERROR::2014-11-24
>> >>> > >>
21:41:44,753::BindingXMLRPC::1148::vds::(wrapper) vdsm exception
>> >>> > >> occured
>> >>> > >> Traceback (most recent call last):
>> >>> > >>   File
"/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1135, in
>> wrapper
>> >>> > >>     res = f(*args, **kwargs)
>> >>> > >>   File
"/usr/share/vdsm/gluster/api.py", line 54, in wrapper
>> >>> > >>     rv = func(*args, **kwargs)
>> >>> > >>   File
"/usr/share/vdsm/gluster/api.py", line 251, in hostsList
>> >>> > >>     return {'hosts':
self.svdsmProxy.glusterPeerStatus()}
>> >>> > >>   File
"/usr/share/vdsm/supervdsm.py", line 50, in __call__
>> >>> > >>     return callMethod()
>> >>> > >>   File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
>> >>> > >>     **kwargs)
>> >>> > >>   File "<string>", line 2,
in glusterPeerStatus
>> >>> > >>   File
"/usr/lib64/python2.6/multiprocessing/managers.py", line
>> 740,
>> >>> > >> in
>> >>> > >> _callmethod
>> >>> > >>     raise convert_to_error(kind, result)
>> >>> > >> GlusterCmdExecFailedException: Command
execution failed
>> >>> > >> error: Connection failed. Please check if
gluster daemon is
>> >>> > >> operational.
>> >>> > >> return code: 1
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:50,949::task::595::Storage.TaskManager.Task::(_updateState)
>> >>> > >>
Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving from state
>> init
>> >>> > >> ->
>> >>> > >> state preparing
>> >>> > >> Thread-13::INFO::2014-11-24
>> >>> > >>
21:41:50,950::logUtils::44::dispatcher::(wrapper) Run and
>> protect:
>> >>> > >> repoStats(options=None)
>> >>> > >> Thread-13::INFO::2014-11-24
>> >>> > >>
21:41:50,950::logUtils::47::dispatcher::(wrapper) Run and
>> protect:
>> >>> > >> repoStats, Return response: {}
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:50,950::task::1191::Storage.TaskManager.Task::(prepare)
>> >>> > >>
Task=`c9042986-c978-4b08-adb2-616f5299e115`::finished: {}
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:50,950::task::595::Storage.TaskManager.Task::(_updateState)
>> >>> > >>
Task=`c9042986-c978-4b08-adb2-616f5299e115`::moving from state
>> >>> > >> preparing
>> >>> > >> ->
>> >>> > >> state finished
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
>> >>> > >>
>>
21:41:50,951::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
>> >>> > >> Owner.releaseAll requests {} resources {}
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
>> >>> > >>
>>
21:41:50,951::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
>> >>> > >> Owner.cancelAll requests {}
>> >>> > >> Thread-13::DEBUG::2014-11-24
>> >>> > >>
21:41:50,951::task::993::Storage.TaskManager.Task::(_decref)
>> >>> > >>
Task=`c9042986-c978-4b08-adb2-616f5299e115`::ref 0 aborting False
>> >>> > >>  -------------------------------
>> >>> > >>
>> >>> > >>  [root at compute4 ~]# service glusterd
status
>> >>> > >> glusterd is stopped
>> >>> > >> [root at compute4 ~]# chkconfig --list |
grep glusterd
>> >>> > >> glusterd        0:off   1:off   2:on    3:on
4:on    5:on
>> >>> > >> 6:off
>> >>> > >> [root at compute4 ~]#
>> >>> > >>
>> >>> > >>  Thanks,
>> >>> > >> Punit
>> >>> > >>
>> >>> > >> On Mon, Nov 24, 2014 at 6:36 PM, Kanagaraj
<kmayilsa at redhat.com>
>> >>> > >> wrote:
>> >>> > >>
>> >>> > >>>  Can you send the corresponding error in
vdsm.log from the host?
>> >>> > >>>
>> >>> > >>> Also check if glusterd service is
running.
>> >>> > >>>
>> >>> > >>> Thanks,
>> >>> > >>> Kanagaraj
>> >>> > >>>
>> >>> > >>>
>> >>> > >>> On 11/24/2014 03:39 PM, Punit Dambiwal
wrote:
>> >>> > >>>
>> >>> > >>>   Hi,
>> >>> > >>>
>> >>> > >>>  After reboot my Hypervisior host can
not activate again in the
>> >>> > >>> cluster
>> >>> > >>> and failed with the following error :-
>> >>> > >>>
>> >>> > >>>  Gluster command [<UNKNOWN>]
failed on server...
>> >>> > >>>
>> >>> > >>>  Engine logs :-
>> >>> > >>>
>> >>> > >>>  2014-11-24 18:05:28,397 INFO
>> >>> > >>>
>> >>> > >>>
>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>> >>> > >>> (DefaultQuartzScheduler_Worker-64)
START,
>> >>> > >>> GlusterVolumesListVDSCommand(HostName =
Compute4, HostId >> >>> > >>>
33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 5f251c90
>> >>> > >>> 2014-11-24 18:05:30,609 INFO
>> >>> > >>>
>> >>> > >>>
>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>> >>> > >>> (DefaultQuartzScheduler_Worker-64)
FINISH,
>> >>> > >>> GlusterVolumesListVDSCommand,
>> >>> > >>> return:
>> >>> > >>>
>> >>> > >>>
>>
{26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity
at d95203e0
>> },
>> >>> > >>> log id: 5f251c90
>> >>> > >>> 2014-11-24 18:05:33,768 INFO
>> >>> > >>> 
[org.ovirt.engine.core.bll.ActivateVdsCommand]
>> >>> > >>> (ajp--127.0.0.1-8702-8)
>> >>> > >>> [287d570d] Lock Acquired to object
EngineLock [exclusiveLocks>> key:
>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a
value: VDS
>> >>> > >>> , sharedLocks= ]
>> >>> > >>> 2014-11-24 18:05:33,795 INFO
>> >>> > >>> 
[org.ovirt.engine.core.bll.ActivateVdsCommand]
>> >>> > >>> (org.ovirt.thread.pool-8-thread-45)
[287d570d] Running command:
>> >>> > >>> ActivateVdsCommand internal: false.
Entities affected :  ID:
>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a
Type: VDSAction group
>> >>> > >>> MANIPULATE_HOST
>> >>> > >>> with role type ADMIN
>> >>> > >>> 2014-11-24 18:05:33,796 INFO
>> >>> > >>> 
[org.ovirt.engine.core.bll.ActivateVdsCommand]
>> >>> > >>> (org.ovirt.thread.pool-8-thread-45)
[287d570d] Before acquiring
>> >>> > >>> lock in
>> >>> > >>> order to prevent monitoring for host
Compute5 from data-center
>> >>> > >>> SV_WTC
>> >>> > >>> 2014-11-24 18:05:33,797 INFO
>> >>> > >>> 
[org.ovirt.engine.core.bll.ActivateVdsCommand]
>> >>> > >>> (org.ovirt.thread.pool-8-thread-45)
[287d570d] Lock acquired,
>> from
>> >>> > >>> now a
>> >>> > >>> monitoring of host will be skipped for
host Compute5 from
>> >>> > >>> data-center
>> >>> > >>> SV_WTC
>> >>> > >>> 2014-11-24 18:05:33,817 INFO
>> >>> > >>> 
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
>> >>> > >>> (org.ovirt.thread.pool-8-thread-45)
[287d570d] START,
>> >>> > >>> SetVdsStatusVDSCommand(HostName =
Compute5, HostId >> >>> > >>>
0bf6b00f-7947-4411-b55a-cc5eea2b381a, status=Unassigned,
>> >>> > >>> nonOperationalReason=NONE,
stopSpmFailureLogged=false), log id:
>> >>> > >>> 1cbc7311
>> >>> > >>> 2014-11-24 18:05:33,820 INFO
>> >>> > >>> 
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
>> >>> > >>> (org.ovirt.thread.pool-8-thread-45)
[287d570d] FINISH,
>> >>> > >>> SetVdsStatusVDSCommand, log id: 1cbc7311
>> >>> > >>> 2014-11-24 18:05:34,086 INFO
>> >>> > >>> 
[org.ovirt.engine.core.bll.ActivateVdsCommand]
>> >>> > >>> (org.ovirt.thread.pool-8-thread-45)
Activate finished. Lock
>> >>> > >>> released.
>> >>> > >>> Monitoring can run now for host Compute5
from data-center SV_WTC
>> >>> > >>> 2014-11-24 18:05:34,088 INFO
>> >>> > >>>
>> >>> > >>>
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> >>> > >>> (org.ovirt.thread.pool-8-thread-45)
Correlation ID: 287d570d,
>> Job
>> >>> > >>> ID:
>> >>> > >>> 5ef8e4d6-b2bc-469e-8e81-7ef74b2a001a,
Call Stack: null, Custom
>> >>> > >>> Event ID:
>> >>> > >>> -1, Message: Host Compute5 was activated
by admin.
>> >>> > >>> 2014-11-24 18:05:34,090 INFO
>> >>> > >>> 
[org.ovirt.engine.core.bll.ActivateVdsCommand]
>> >>> > >>> (org.ovirt.thread.pool-8-thread-45) Lock
freed to object
>> EngineLock
>> >>> > >>> [exclusiveLocks= key:
0bf6b00f-7947-4411-b55a-cc5eea2b381a
>> value:
>> >>> > >>> VDS
>> >>> > >>> , sharedLocks= ]
>> >>> > >>> 2014-11-24 18:05:35,792 INFO
>> >>> > >>>
>> >>> > >>>
>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>> >>> > >>> (DefaultQuartzScheduler_Worker-55)
[3706e836] START,
>> >>> > >>> GlusterVolumesListVDSCommand(HostName =
Compute4, HostId >> >>> > >>>
33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 48a0c832
>> >>> > >>> 2014-11-24 18:05:37,064 INFO
>> >>> > >>>
>> >>> > >>>
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
>> >>> > >>> (DefaultQuartzScheduler_Worker-69)
START,
>> >>> > >>> GetHardwareInfoVDSCommand(HostName =
Compute5, HostId >> >>> > >>>
0bf6b00f-7947-4411-b55a-cc5eea2b381a,
>> >>> > >>>
vds=Host[Compute5,0bf6b00f-7947-4411-b55a-cc5eea2b381a]), log
>> id:
>> >>> > >>> 6d560cc2
>> >>> > >>> 2014-11-24 18:05:37,074 INFO
>> >>> > >>>
>> >>> > >>>
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand]
>> >>> > >>> (DefaultQuartzScheduler_Worker-69)
FINISH,
>> >>> > >>> GetHardwareInfoVDSCommand, log
>> >>> > >>> id: 6d560cc2
>> >>> > >>> 2014-11-24 18:05:37,093 WARN
>> >>> > >>> 
[org.ovirt.engine.core.vdsbroker.VdsManager]
>> >>> > >>> (DefaultQuartzScheduler_Worker-69) Host
Compute5 is running with
>> >>> > >>> disabled
>> >>> > >>> SELinux.
>> >>> > >>> 2014-11-24 18:05:37,127 INFO
>> >>> > >>>
>> >>> > >>>
>> [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
>> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[2b4a51cf] Running command:
>> >>> > >>> HandleVdsCpuFlagsOrClusterChangedCommand
internal: true.
>> Entities
>> >>> > >>> affected
>> >>> > >>> :  ID:
0bf6b00f-7947-4411-b55a-cc5eea2b381a Type: VDS
>> >>> > >>> 2014-11-24 18:05:37,147 INFO
>> >>> > >>>
>> >>> > >>>
>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
>> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[2b4a51cf] START,
>> >>> > >>> GlusterServersListVDSCommand(HostName =
Compute5, HostId >> >>> > >>>
0bf6b00f-7947-4411-b55a-cc5eea2b381a), log id: 4faed87
>> >>> > >>> 2014-11-24 18:05:37,164 INFO
>> >>> > >>>
>> >>> > >>>
>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
>> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[2b4a51cf] FINISH,
>> >>> > >>> GlusterServersListVDSCommand, log id:
4faed87
>> >>> > >>> 2014-11-24 18:05:37,189 INFO
>> >>> > >>> 
[org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
>> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[4a84c4e5] Running command:
>> >>> > >>> SetNonOperationalVdsCommand internal:
true. Entities affected :
>> >>> > >>> ID:
>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a
Type: VDS
>> >>> > >>> 2014-11-24 18:05:37,206 INFO
>> >>> > >>> 
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
>> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[4a84c4e5] START,
>> >>> > >>> SetVdsStatusVDSCommand(HostName =
Compute5, HostId >> >>> > >>>
0bf6b00f-7947-4411-b55a-cc5eea2b381a, status=NonOperational,
>> >>> > >>>
nonOperationalReason=GLUSTER_COMMAND_FAILED,
>> >>> > >>> stopSpmFailureLogged=false),
>> >>> > >>> log id: fed5617
>> >>> > >>> 2014-11-24 18:05:37,209 INFO
>> >>> > >>> 
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
>> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[4a84c4e5] FINISH,
>> >>> > >>> SetVdsStatusVDSCommand, log id: fed5617
>> >>> > >>> 2014-11-24 18:05:37,223 ERROR
>> >>> > >>>
>> >>> > >>>
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[4a84c4e5] Correlation ID:
>> >>> > >>> 4a84c4e5,
>> >>> > >>> Job
>> >>> > >>> ID:
4bfd4a6d-c3ef-468f-a40e-a3a6ca13011b, Call Stack: null,
>> Custom
>> >>> > >>> Event
>> >>> > >>> ID: -1, Message: Gluster command
[<UNKNOWN>] failed on server
>> >>> > >>> Compute5.
>> >>> > >>> 2014-11-24 18:05:37,243 INFO
>> >>> > >>>
>> >>> > >>>
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[4a84c4e5] Correlation ID:
>> null,
>> >>> > >>> Call
>> >>> > >>> Stack: null, Custom Event ID: -1,
Message: Status of host
>> Compute5
>> >>> > >>> was
>> >>> > >>> set
>> >>> > >>> to NonOperational.
>> >>> > >>> 2014-11-24 18:05:37,272 INFO
>> >>> > >>> 
[org.ovirt.engine.core.bll.HandleVdsVersionCommand]
>> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[a0c8a7f] Running command:
>> >>> > >>> HandleVdsVersionCommand internal: true.
Entities affected :  ID:
>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a
Type: VDS
>> >>> > >>> 2014-11-24 18:05:37,274 INFO
>> >>> > >>> 
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>> >>> > >>> (DefaultQuartzScheduler_Worker-69)
[a0c8a7f] Host
>> >>> > >>> 0bf6b00f-7947-4411-b55a-cc5eea2b381a :
Compute5 is already in
>> >>> > >>> NonOperational status for reason
GLUSTER_COMMAND_FAILED.
>> >>> > >>> SetNonOperationalVds command is skipped.
>> >>> > >>> 2014-11-24 18:05:38,065 INFO
>> >>> > >>>
>> >>> > >>>
>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>> >>> > >>> (DefaultQuartzScheduler_Worker-55)
[3706e836] FINISH,
>> >>> > >>> GlusterVolumesListVDSCommand, return:
>> >>> > >>>
>> >>> > >>>
>>
{26ae1672-ee09-4a38-8fd2-72dd9974cc2b=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity
at 4e72a1b1
>> },
>> >>> > >>> log id: 48a0c832
>> >>> > >>> 2014-11-24 18:05:43,243 INFO
>> >>> > >>>
>> >>> > >>>
>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>> >>> > >>> (DefaultQuartzScheduler_Worker-35)
START,
>> >>> > >>> GlusterVolumesListVDSCommand(HostName =
Compute4, HostId >> >>> > >>>
33648a90-200c-45ca-89d5-1ce305d79a6a), log id: 3ce13ebc
>> >>> > >>> ^C
>> >>> > >>> [root at ccr01 ~]#
>> >>> > >>>
>> >>> > >>>  Thanks,
>> >>> > >>> Punit
>> >>> > >>>
>> >>> > >>>
>> >>> > >>> 
_______________________________________________
>> >>> > >>> Users mailing
>> >>> > >>> listUsers at ovirt.orghttp://
>> lists.ovirt.org/mailman/listinfo/users
>> >>> > >>>
>> >>> > >>>
>> >>> > >>>
>> >>> > >>
>> >>> > >>
>> >>> > >
>> >>> > >
>> >>> >
>> >>
>> >>
>> >
>> >
>> > _______________________________________________
>> > Gluster-users mailing list
>> > Gluster-users at gluster.org
>> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141201/db4f1945/attachment.html>