Bill James
2017-Jan-13 17:41 UTC
[Gluster-users] vdsm IOProcessClient WARNING Timeout waiting for communication thread for client
resending without logs, except vdsm.log since list limit is too small. On 1/13/17 8:50 AM, Bill James wrote:> We have an ovirt system with 3 clusters, all running centos7. > ovirt engine is running on separate host, > ovirt-engine-3.6.4.1-1.el7.centos.noarch > 2 of the clusters are running newer version of ovirt, 3 nodes each, > ovirt-engine-4.0.3-1.el7.centos.noarch, glusterfs-3.7.16-1.el7.x86_64, > vdsm-4.18.11-1.el7.centos.x86_64. > 1 cluster is still running the older version, > ovirt-engine-3.6.4.1-1.el7.centos.noarch. > > Yes we are in the process of upgrading the whole system to ovirt4.0, > but takes time.... > > One of the 2 clusters running ovirt4 is complaining of timeouts, vdsm > talking to gluster. No warnings on the 2 other clusters. > > > > Thread-720062::DEBUG::2017-01-13 > 07:29:46,814::outOfProcess::87::Storage.oop::(getProcessPool) Creating > ioprocess /rhev/data-center/mnt/glusterSD/ovirt1-gl.dmz.p > rod.j2noc.com:_gv1 > Thread-720062::INFO::2017-01-13 > 07:29:46,814::__init__::325::IOProcessClient::(__init__) Starting > client ioprocess-5874 > Thread-720062::DEBUG::2017-01-13 > 07:29:46,814::__init__::334::IOProcessClient::(_run) Starting > ioprocess for client ioprocess-5874 > Thread-720062::DEBUG::2017-01-13 > 07:29:46,832::__init__::386::IOProcessClient::(_startCommunication) > Starting communication thread for client ioprocess-5874 > *Thread-720062::WARNING::2017-01-13 > 07:29:46,847::__init__::401::IOProcessClient::(_startCommunication) > Timeout waiting for communication thread for client ioproc**ess-5874* > > > [2017-01-12 07:27:58.685680] I [MSGID: 106488] > [glusterd-handler.c:1533:__glusterd_handle_cli_get_volume] 0-glusterd: > Received get vol req > The message "I [MSGID: 106488] > [glusterd-handler.c:1533:__glusterd_handle_cli_get_volume] 0-glusterd: > Received get vol req" repeated 31 times between [2017-01-12 > 07:27:58.685680] and [2017-01-12 07:29:46.971939] > > > attached logs: engine.log supervdsm.log vdsm.log > etc-glusterfs-glusterd.vol.log cli.log >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170113/f68fdc0e/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: vdsm.log.gz Type: application/x-gzip Size: 1677515 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170113/f68fdc0e/attachment-0001.gz>
Nir Soffer
2017-Jan-13 18:37 UTC
[Gluster-users] [ovirt-users] vdsm IOProcessClient WARNING Timeout waiting for communication thread for client
On Fri, Jan 13, 2017 at 7:41 PM, Bill James <bill.james at j2.com> wrote:> resending without logs, except vdsm.log since list limit is too small. > > > > On 1/13/17 8:50 AM, Bill James wrote: > > We have an ovirt system with 3 clusters, all running centos7. > ovirt engine is running on separate host, > ovirt-engine-3.6.4.1-1.el7.centos.noarch > 2 of the clusters are running newer version of ovirt, 3 nodes each, > ovirt-engine-4.0.3-1.el7.centos.noarch, glusterfs-3.7.16-1.el7.x86_64, > vdsm-4.18.11-1.el7.centos.x86_64. > 1 cluster is still running the older version, > ovirt-engine-3.6.4.1-1.el7.centos.noarch.Which ioprocess version?> > Yes we are in the process of upgrading the whole system to ovirt4.0, but > takes time.... > > One of the 2 clusters running ovirt4 is complaining of timeouts, vdsm > talking to gluster. No warnings on the 2 other clusters. > > > > Thread-720062::DEBUG::2017-01-13 > 07:29:46,814::outOfProcess::87::Storage.oop::(getProcessPool) Creating > ioprocess /rhev/data-center/mnt/glusterSD/ovirt1-gl.dmz.p > rod.j2noc.com:_gv1 > Thread-720062::INFO::2017-01-13 > 07:29:46,814::__init__::325::IOProcessClient::(__init__) Starting client > ioprocess-5874 > Thread-720062::DEBUG::2017-01-13 > 07:29:46,814::__init__::334::IOProcessClient::(_run) Starting ioprocess for > client ioprocess-5874 > Thread-720062::DEBUG::2017-01-13 > 07:29:46,832::__init__::386::IOProcessClient::(_startCommunication) Starting > communication thread for client ioprocess-5874 > Thread-720062::WARNING::2017-01-13 > 07:29:46,847::__init__::401::IOProcessClient::(_startCommunication) Timeout > waiting for communication thread for client ioprocess-5874This warning is harmless, it means that ioprocess thread did not start in 1 second. This probably means that the host is overloaded, typically new threads start instantly. Anyway I think we are using too short timeout. Can you open an ioprocess bug for this?> > > [2017-01-12 07:27:58.685680] I [MSGID: 106488] > [glusterd-handler.c:1533:__glusterd_handle_cli_get_volume] 0-glusterd: > Received get vol req > The message "I [MSGID: 106488] > [glusterd-handler.c:1533:__glusterd_handle_cli_get_volume] 0-glusterd: > Received get vol req" repeated 31 times between [2017-01-12 07:27:58.685680] > and [2017-01-12 07:29:46.971939] > > > attached logs: engine.log supervdsm.log vdsm.log > etc-glusterfs-glusterd.vol.log cli.log > > > > _______________________________________________ > Users mailing list > Users at ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >