thr3ads.net - Gluster users - [Gluster-users] Message repeated over and over after upgrade from 4.1 to 5.3: W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument] [Feb 2019]

If this information is useful, please help other people find it:
Share via:

Raghavendra Gowdappa

2019-Jan-31 02:37 UTC

[Gluster-users] Message repeated over and over after upgrade from 4.1 to 5.3: W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument]

On Thu, Jan 31, 2019 at 2:14 AM Artem Russakovskii <archon810 at
gmail.com>
wrote:
> Also, not sure if related or not, but I got a ton of these "Failed to
> dispatch handler" in my logs as well. Many people have been commenting
> about this issue here https://bugzilla.redhat.com/show_bug.cgi?id=1651246.
>
https://review.gluster.org/#/c/glusterfs/+/22046/ addresses this.

> ==> mnt-SITE_data1.log <=>> [2019-01-30 20:38:20.783713] W
[dict.c:761:dict_ref]
>>
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
>> [0x7fd966fcd329]
>> -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
>> [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
>> [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument]
>> ==> mnt-SITE_data3.log <=>> The message "E [MSGID:
101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to
dispatch
>> handler" repeated 413 times between [2019-01-30 20:36:23.881090]
and
>> [2019-01-30 20:38:20.015593]
>> The message "I [MSGID: 108031]
>> [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data3-replicate-0:
>> selecting local read_child SITE_data3-client-0" repeated 42 times
between
>> [2019-01-30 20:36:23.290287] and [2019-01-30 20:38:20.280306]
>> ==> mnt-SITE_data1.log <=>> The message "I [MSGID:
108031]
>> [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0:
>> selecting local read_child SITE_data1-client-0" repeated 50 times
between
>> [2019-01-30 20:36:22.247367] and [2019-01-30 20:38:19.459789]
>> The message "E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to
dispatch
>> handler" repeated 2654 times between [2019-01-30 20:36:22.667327]
and
>> [2019-01-30 20:38:20.546355]
>> [2019-01-30 20:38:21.492319] I [MSGID: 108031]
>> [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0:
>> selecting local read_child SITE_data1-client-0
>> ==> mnt-SITE_data3.log <=>> [2019-01-30 20:38:22.349689] I
[MSGID: 108031]
>> [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data3-replicate-0:
>> selecting local read_child SITE_data3-client-0
>> ==> mnt-SITE_data1.log <=>> [2019-01-30 20:38:22.762941] E
[MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to
dispatch
>> handler
>
>
> I'm hoping raising the issue here on the mailing list may bring some
> additional eyeballs and get them both fixed.
>
> Thanks.
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
> <http://twitter.com/ArtemR>
>
>
> On Wed, Jan 30, 2019 at 12:26 PM Artem Russakovskii <archon810 at
gmail.com>
> wrote:
>
>> I found a similar issue here:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1313567. There's a
comment
>> from 3 days ago from someone else with 5.3 who started seeing the spam.
>>
>> Here's the command that repeats over and over:
>> [2019-01-30 20:23:24.481581] W [dict.c:761:dict_ref]
>>
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
>> [0x7fd966fcd329]
>> -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
>> [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
>> [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument]
>>
>+Milind Changire <mchangir at redhat.com> Can you check why this message
is
logged and send a fix?

>> Is there any fix for this issue?
>>
>> Thanks.
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com>, APK
Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net | +ArtemRussakovskii
>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>> <http://twitter.com/ArtemR>
>>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20190131/5502a3a6/attachment.html>

Artem Russakovskii

2019-Jan-31 18:09 UTC

head link

[Gluster-users] Message repeated over and over after upgrade from 4.1 to 5.3: W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument]

Within 24 hours after updating from rock solid 4.1 to 5.3, I already got a
crash which others have mentioned in
https://bugzilla.redhat.com/show_bug.cgi?id=1313567 and had to unmount,
kill gluster, and remount:


[2019-01-31 09:38:04.317604] W [dict.c:761:dict_ref]
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
[0x7fcccafcd329]
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
[0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
[0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument]
[2019-01-31 09:38:04.319308] W [dict.c:761:dict_ref]
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
[0x7fcccafcd329]
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
[0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
[0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument]
[2019-01-31 09:38:04.320047] W [dict.c:761:dict_ref]
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
[0x7fcccafcd329]
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
[0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
[0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument]
[2019-01-31 09:38:04.320677] W [dict.c:761:dict_ref]
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
[0x7fcccafcd329]
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
[0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
[0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument]
The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk]
2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-3"
repeated 5 times between [2019-01-31 09:37:54.751905] and [2019-01-31
09:38:03.958061]
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch
handler" repeated 72 times between [2019-01-31 09:37:53.746741] and
[2019-01-31 09:38:04.696993]
pending frames:
frame : type(1) op(READ)
frame : type(1) op(OPEN)
frame : type(0) op(0)
patchset: git://git.gluster.org/glusterfs.git
signal received: 6
time of crash:
2019-01-31 09:38:04
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 5.3
/usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fccd706664c]
/usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fccd7070cb6]
/lib64/libc.so.6(+0x36160)[0x7fccd622d160]
/lib64/libc.so.6(gsignal+0x110)[0x7fccd622d0e0]
/lib64/libc.so.6(abort+0x151)[0x7fccd622e6c1]
/lib64/libc.so.6(+0x2e6fa)[0x7fccd62256fa]
/lib64/libc.so.6(+0x2e772)[0x7fccd6225772]
/lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fccd65bb0b8]
/usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x32c4d)[0x7fcccbb01c4d]
/usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x65778)[0x7fcccbdd1778]
/usr/lib64/libgfrpc.so.0(+0xe820)[0x7fccd6e31820]
/usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fccd6e31b6f]
/usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fccd6e2e063]
/usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fccd0b7e0b2]
/usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fccd70c44c3]
/lib64/libpthread.so.0(+0x7559)[0x7fccd65b8559]
/lib64/libc.so.6(clone+0x3f)[0x7fccd62ef81f]
---------

Do the pending patches fix the crash or only the repeated warnings? I'm
running glusterfs on OpenSUSE 15.0 installed via
http://download.opensuse.org/repositories/home:/glusterfs:/Leap15-5/openSUSE_Leap_15.0/,
not too sure how to make it core dump.

If it's not fixed by the patches above, has anyone already opened a ticket
for the crashes that I can join and monitor? This is going to create a
massive problem for us since production systems are crashing.

Thanks.

Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | +ArtemRussakovskii
<https://plus.google.com/+ArtemRussakovskii> | @ArtemR
<http://twitter.com/ArtemR>


On Wed, Jan 30, 2019 at 6:37 PM Raghavendra Gowdappa <rgowdapp at
redhat.com>
wrote:
>
>
> On Thu, Jan 31, 2019 at 2:14 AM Artem Russakovskii <archon810 at
gmail.com>
> wrote:
>
>> Also, not sure if related or not, but I got a ton of these "Failed
to
>> dispatch handler" in my logs as well. Many people have been
commenting
>> about this issue here
https://bugzilla.redhat.com/show_bug.cgi?id=1651246
>> .
>>
>
> https://review.gluster.org/#/c/glusterfs/+/22046/ addresses this.
>
>
>> ==> mnt-SITE_data1.log <=>>> [2019-01-30
20:38:20.783713] W [dict.c:761:dict_ref]
>>>
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
>>> [0x7fd966fcd329]
>>>
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
>>> [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
>>> [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument]
>>> ==> mnt-SITE_data3.log <=>>> The message "E
[MSGID: 101191]
>>> [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to
dispatch
>>> handler" repeated 413 times between [2019-01-30
20:36:23.881090] and
>>> [2019-01-30 20:38:20.015593]
>>> The message "I [MSGID: 108031]
>>> [afr-common.c:2543:afr_local_discovery_cbk]
2-SITE_data3-replicate-0:
>>> selecting local read_child SITE_data3-client-0" repeated 42
times between
>>> [2019-01-30 20:36:23.290287] and [2019-01-30 20:38:20.280306]
>>> ==> mnt-SITE_data1.log <=>>> The message "I
[MSGID: 108031]
>>> [afr-common.c:2543:afr_local_discovery_cbk]
2-SITE_data1-replicate-0:
>>> selecting local read_child SITE_data1-client-0" repeated 50
times between
>>> [2019-01-30 20:36:22.247367] and [2019-01-30 20:38:19.459789]
>>> The message "E [MSGID: 101191]
>>> [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to
dispatch
>>> handler" repeated 2654 times between [2019-01-30
20:36:22.667327] and
>>> [2019-01-30 20:38:20.546355]
>>> [2019-01-30 20:38:21.492319] I [MSGID: 108031]
>>> [afr-common.c:2543:afr_local_discovery_cbk]
2-SITE_data1-replicate-0:
>>> selecting local read_child SITE_data1-client-0
>>> ==> mnt-SITE_data3.log <=>>> [2019-01-30
20:38:22.349689] I [MSGID: 108031]
>>> [afr-common.c:2543:afr_local_discovery_cbk]
2-SITE_data3-replicate-0:
>>> selecting local read_child SITE_data3-client-0
>>> ==> mnt-SITE_data1.log <=>>> [2019-01-30
20:38:22.762941] E [MSGID: 101191]
>>> [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to
dispatch
>>> handler
>>
>>
>> I'm hoping raising the issue here on the mailing list may bring
some
>> additional eyeballs and get them both fixed.
>>
>> Thanks.
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com>, APK
Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net | +ArtemRussakovskii
>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>> <http://twitter.com/ArtemR>
>>
>>
>> On Wed, Jan 30, 2019 at 12:26 PM Artem Russakovskii <archon810 at
gmail.com>
>> wrote:
>>
>>> I found a similar issue here:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1313567. There's a
comment
>>> from 3 days ago from someone else with 5.3 who started seeing the
spam.
>>>
>>> Here's the command that repeats over and over:
>>> [2019-01-30 20:23:24.481581] W [dict.c:761:dict_ref]
>>>
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
>>> [0x7fd966fcd329]
>>>
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
>>> [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
>>> [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument]
>>>
>>
> +Milind Changire <mchangir at redhat.com> Can you check why this
message is
> logged and send a fix?
>
>
>>> Is there any fix for this issue?
>>>
>>> Thanks.
>>>
>>> Sincerely,
>>> Artem
>>>
>>> --
>>> Founder, Android Police <http://www.androidpolice.com>, APK
Mirror
>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>> beerpla.net | +ArtemRussakovskii
>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>> <http://twitter.com/ArtemR>
>>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20190131/7149bc89/attachment.html>

Amar Tumballi Suryanarayan

2019-Feb-01 03:24 UTC

head link

[Gluster-users] Message repeated over and over after upgrade from 4.1 to 5.3: W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument]

Hi Artem,

Opened https://bugzilla.redhat.com/show_bug.cgi?id=1671603 (ie, as a clone
of other bugs where recent discussions happened), and marked it as a
blocker for glusterfs-5.4 release.

We already have fixes for log flooding - https://review.gluster.org/22128,
and are the process of identifying and fixing the issue seen with crash.

Can you please tell if the crashes happened as soon as upgrade ? or was
there any particular pattern you observed before the crash.

-Amar


On Thu, Jan 31, 2019 at 11:40 PM Artem Russakovskii <archon810 at
gmail.com>
wrote:
> Within 24 hours after updating from rock solid 4.1 to 5.3, I already got a
> crash which others have mentioned in
> https://bugzilla.redhat.com/show_bug.cgi?id=1313567 and had to unmount,
> kill gluster, and remount:
>
>
> [2019-01-31 09:38:04.317604] W [dict.c:761:dict_ref]
> (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
> [0x7fcccafcd329]
> -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
> [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
> [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument]
> [2019-01-31 09:38:04.319308] W [dict.c:761:dict_ref]
> (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
> [0x7fcccafcd329]
> -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
> [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
> [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument]
> [2019-01-31 09:38:04.320047] W [dict.c:761:dict_ref]
> (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
> [0x7fcccafcd329]
> -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
> [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
> [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument]
> [2019-01-31 09:38:04.320677] W [dict.c:761:dict_ref]
> (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
> [0x7fcccafcd329]
> -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
> [0x7fcccb1deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
> [0x7fccd705b218] ) 2-dict: dict is NULL [Invalid argument]
> The message "I [MSGID: 108031]
[afr-common.c:2543:afr_local_discovery_cbk]
> 2-SITE_data1-replicate-0: selecting local read_child
SITE_data1-client-3"
> repeated 5 times between [2019-01-31 09:37:54.751905] and [2019-01-31
> 09:38:03.958061]
> The message "E [MSGID: 101191]
> [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch
> handler" repeated 72 times between [2019-01-31 09:37:53.746741] and
> [2019-01-31 09:38:04.696993]
> pending frames:
> frame : type(1) op(READ)
> frame : type(1) op(OPEN)
> frame : type(0) op(0)
> patchset: git://git.gluster.org/glusterfs.git
> signal received: 6
> time of crash:
> 2019-01-31 09:38:04
> configuration details:
> argp 1
> backtrace 1
> dlfcn 1
> libpthread 1
> llistxattr 1
> setfsid 1
> spinlock 1
> epoll.h 1
> xattr.h 1
> st_atim.tv_nsec 1
> package-string: glusterfs 5.3
> /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fccd706664c]
> /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fccd7070cb6]
> /lib64/libc.so.6(+0x36160)[0x7fccd622d160]
> /lib64/libc.so.6(gsignal+0x110)[0x7fccd622d0e0]
> /lib64/libc.so.6(abort+0x151)[0x7fccd622e6c1]
> /lib64/libc.so.6(+0x2e6fa)[0x7fccd62256fa]
> /lib64/libc.so.6(+0x2e772)[0x7fccd6225772]
> /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fccd65bb0b8]
>
>
/usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x32c4d)[0x7fcccbb01c4d]
>
>
/usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x65778)[0x7fcccbdd1778]
> /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fccd6e31820]
> /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fccd6e31b6f]
> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fccd6e2e063]
> /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fccd0b7e0b2]
> /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fccd70c44c3]
> /lib64/libpthread.so.0(+0x7559)[0x7fccd65b8559]
> /lib64/libc.so.6(clone+0x3f)[0x7fccd62ef81f]
> ---------
>
> Do the pending patches fix the crash or only the repeated warnings? I'm
> running glusterfs on OpenSUSE 15.0 installed via
>
http://download.opensuse.org/repositories/home:/glusterfs:/Leap15-5/openSUSE_Leap_15.0/,
> not too sure how to make it core dump.
>
> If it's not fixed by the patches above, has anyone already opened a
ticket
> for the crashes that I can join and monitor? This is going to create a
> massive problem for us since production systems are crashing.
>
> Thanks.
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
> <http://twitter.com/ArtemR>
>
>
> On Wed, Jan 30, 2019 at 6:37 PM Raghavendra Gowdappa <rgowdapp at
redhat.com>
> wrote:
>
>>
>>
>> On Thu, Jan 31, 2019 at 2:14 AM Artem Russakovskii <archon810 at
gmail.com>
>> wrote:
>>
>>> Also, not sure if related or not, but I got a ton of these
"Failed to
>>> dispatch handler" in my logs as well. Many people have been
commenting
>>> about this issue here
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1651246.
>>>
>>
>> https://review.gluster.org/#/c/glusterfs/+/22046/ addresses this.
>>
>>
>>> ==> mnt-SITE_data1.log <=>>>> [2019-01-30
20:38:20.783713] W [dict.c:761:dict_ref]
>>>>
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
>>>> [0x7fd966fcd329]
>>>>
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
>>>> [0x7fd9671deaf5]
-->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
>>>> [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument]
>>>> ==> mnt-SITE_data3.log <=>>>> The message
"E [MSGID: 101191]
>>>> [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed
to dispatch
>>>> handler" repeated 413 times between [2019-01-30
20:36:23.881090] and
>>>> [2019-01-30 20:38:20.015593]
>>>> The message "I [MSGID: 108031]
>>>> [afr-common.c:2543:afr_local_discovery_cbk]
2-SITE_data3-replicate-0:
>>>> selecting local read_child SITE_data3-client-0" repeated
42 times between
>>>> [2019-01-30 20:36:23.290287] and [2019-01-30 20:38:20.280306]
>>>> ==> mnt-SITE_data1.log <=>>>> The message
"I [MSGID: 108031]
>>>> [afr-common.c:2543:afr_local_discovery_cbk]
2-SITE_data1-replicate-0:
>>>> selecting local read_child SITE_data1-client-0" repeated
50 times between
>>>> [2019-01-30 20:36:22.247367] and [2019-01-30 20:38:19.459789]
>>>> The message "E [MSGID: 101191]
>>>> [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed
to dispatch
>>>> handler" repeated 2654 times between [2019-01-30
20:36:22.667327] and
>>>> [2019-01-30 20:38:20.546355]
>>>> [2019-01-30 20:38:21.492319] I [MSGID: 108031]
>>>> [afr-common.c:2543:afr_local_discovery_cbk]
2-SITE_data1-replicate-0:
>>>> selecting local read_child SITE_data1-client-0
>>>> ==> mnt-SITE_data3.log <=>>>> [2019-01-30
20:38:22.349689] I [MSGID: 108031]
>>>> [afr-common.c:2543:afr_local_discovery_cbk]
2-SITE_data3-replicate-0:
>>>> selecting local read_child SITE_data3-client-0
>>>> ==> mnt-SITE_data1.log <=>>>> [2019-01-30
20:38:22.762941] E [MSGID: 101191]
>>>> [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed
to dispatch
>>>> handler
>>>
>>>
>>> I'm hoping raising the issue here on the mailing list may bring
some
>>> additional eyeballs and get them both fixed.
>>>
>>> Thanks.
>>>
>>> Sincerely,
>>> Artem
>>>
>>> --
>>> Founder, Android Police <http://www.androidpolice.com>, APK
Mirror
>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>> beerpla.net | +ArtemRussakovskii
>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>> <http://twitter.com/ArtemR>
>>>
>>>
>>> On Wed, Jan 30, 2019 at 12:26 PM Artem Russakovskii <archon810
at gmail.com>
>>> wrote:
>>>
>>>> I found a similar issue here:
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1313567.
There's a comment
>>>> from 3 days ago from someone else with 5.3 who started seeing
the spam.
>>>>
>>>> Here's the command that repeats over and over:
>>>> [2019-01-30 20:23:24.481581] W [dict.c:761:dict_ref]
>>>>
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
>>>> [0x7fd966fcd329]
>>>>
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
>>>> [0x7fd9671deaf5]
-->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
>>>> [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument]
>>>>
>>>
>> +Milind Changire <mchangir at redhat.com> Can you check why this
message is
>> logged and send a fix?
>>
>>
>>>> Is there any fix for this issue?
>>>>
>>>> Thanks.
>>>>
>>>> Sincerely,
>>>> Artem
>>>>
>>>> --
>>>> Founder, Android Police <http://www.androidpolice.com>,
APK Mirror
>>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>>> beerpla.net | +ArtemRussakovskii
>>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>>> <http://twitter.com/ArtemR>
>>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users


-- 
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20190201/6b643a72/attachment.html>

Gluster users - Feb 2019 - Message repeated over and over after upgrade from 4.1 to 5.3: W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)