Artem Russakovskii
2019-Jan-30 20:26 UTC
[Gluster-users] Message repeated over and over after upgrade from 4.1 to 5.3: W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument]
I found a similar issue here: https://bugzilla.redhat.com/show_bug.cgi?id=1313567. There's a comment from 3 days ago from someone else with 5.3 who started seeing the spam. Here's the command that repeats over and over: [2019-01-30 20:23:24.481581] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument] Is there any fix for this issue? Thanks. Sincerely, Artem -- Founder, Android Police <http://www.androidpolice.com>, APK Mirror <http://www.apkmirror.com/>, Illogical Robot LLC beerpla.net | +ArtemRussakovskii <https://plus.google.com/+ArtemRussakovskii> | @ArtemR <http://twitter.com/ArtemR> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190130/be5efa4e/attachment.html>
Artem Russakovskii
2019-Jan-30 20:43 UTC
[Gluster-users] Message repeated over and over after upgrade from 4.1 to 5.3: W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument]
Also, not sure if related or not, but I got a ton of these "Failed to dispatch handler" in my logs as well. Many people have been commenting about this issue here https://bugzilla.redhat.com/show_bug.cgi?id=1651246. ==> mnt-SITE_data1.log <=> [2019-01-30 20:38:20.783713] W [dict.c:761:dict_ref]> (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) > [0x7fd966fcd329] > -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) > [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) > [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument] > ==> mnt-SITE_data3.log <=> The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch > handler" repeated 413 times between [2019-01-30 20:36:23.881090] and > [2019-01-30 20:38:20.015593] > The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] > 2-SITE_data3-replicate-0: selecting local read_child SITE_data3-client-0" > repeated 42 times between [2019-01-30 20:36:23.290287] and [2019-01-30 > 20:38:20.280306] > ==> mnt-SITE_data1.log <=> The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] > 2-SITE_data1-replicate-0: selecting local read_child SITE_data1-client-0" > repeated 50 times between [2019-01-30 20:36:22.247367] and [2019-01-30 > 20:38:19.459789] > The message "E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch > handler" repeated 2654 times between [2019-01-30 20:36:22.667327] and > [2019-01-30 20:38:20.546355] > [2019-01-30 20:38:21.492319] I [MSGID: 108031] > [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data1-replicate-0: > selecting local read_child SITE_data1-client-0 > ==> mnt-SITE_data3.log <=> [2019-01-30 20:38:22.349689] I [MSGID: 108031] > [afr-common.c:2543:afr_local_discovery_cbk] 2-SITE_data3-replicate-0: > selecting local read_child SITE_data3-client-0 > ==> mnt-SITE_data1.log <=> [2019-01-30 20:38:22.762941] E [MSGID: 101191] > [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch > handlerI'm hoping raising the issue here on the mailing list may bring some additional eyeballs and get them both fixed. Thanks. Sincerely, Artem -- Founder, Android Police <http://www.androidpolice.com>, APK Mirror <http://www.apkmirror.com/>, Illogical Robot LLC beerpla.net | +ArtemRussakovskii <https://plus.google.com/+ArtemRussakovskii> | @ArtemR <http://twitter.com/ArtemR> On Wed, Jan 30, 2019 at 12:26 PM Artem Russakovskii <archon810 at gmail.com> wrote:> I found a similar issue here: > https://bugzilla.redhat.com/show_bug.cgi?id=1313567. There's a comment > from 3 days ago from someone else with 5.3 who started seeing the spam. > > Here's the command that repeats over and over: > [2019-01-30 20:23:24.481581] W [dict.c:761:dict_ref] > (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) > [0x7fd966fcd329] > -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) > [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) > [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument] > > Is there any fix for this issue? > > Thanks. > > Sincerely, > Artem > > -- > Founder, Android Police <http://www.androidpolice.com>, APK Mirror > <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net | +ArtemRussakovskii > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR > <http://twitter.com/ArtemR> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190130/38fdabf3/attachment.html>
lejeczek
2019-Oct-10 16:00 UTC
[Gluster-users] Message repeated over and over after upgrade from 4.1 to 5.3: W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument]
On 30/01/2019 20:26, Artem Russakovskii wrote:> I found a similar issue > here:?https://bugzilla.redhat.com/show_bug.cgi?id=1313567. There's a > comment from 3 days ago from someone else with 5.3 who started seeing > the spam. > > Here's the command that repeats over and over: > [2019-01-30 20:23:24.481581] W [dict.c:761:dict_ref] > (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) > [0x7fd966fcd329] > -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) > [0x7fd9671deaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) > [0x7fd9731ea218] ) 2-dict: dict is NULL [Invalid argument] > > Is there any fix for this issue? > > Thanks. > > Sincerely, > Artem > > -- > Founder, Android Police <http://www.androidpolice.com>,?APK Mirror > <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii > <https://plus.google.com/+ArtemRussakovskii> |?@ArtemR > <http://twitter.com/ArtemR> > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-usersI get no crashes but with 6.5 I see these: ... [2019-10-10 15:52:08.528208] I [io-stats.c:4027:fini] 0-USER-HOME: io-stats translator unloaded [2019-10-10 15:52:10.441283] E [MSGID: 101046] [dht-common.c:11247:dht_pt_fgetxattr_cbk] 0-USER-HOME-dht: dict is null [2019-10-10 15:52:10.441387] E [MSGID: 101046] [dht-common.c:11248:dht_pt_fgetxattr_cbk] 0-USER-HOME-dht: dict is null [2019-10-10 15:52:10.555957] E [MSGID: 108006] [afr-common.c:5318:__afr_handle_child_down_event] 0-USER-HOME-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up. [2019-10-10 15:52:10.557136] I [io-stats.c:4027:fini] 0-USER-HOME: io-stats translator unloaded The message "E [MSGID: 101046] [dht-common.c:11220:dht_pt_getxattr_cbk] 0-USER-HOME-dht: dict is null" repeated 8 times between [2019-10-10 15:52:07.263547] and [2019-10-10 15:52:07.649220] The message "E [MSGID: 101046] [dht-common.c:11221:dht_pt_getxattr_cbk] 0-USER-HOME-dht: dict is null" repeated 8 times between [2019-10-10 15:52:07.263620] and [2019-10-10 15:52:07.649223] [2019-10-10 15:56:11.291652] E [MSGID: 101046] [dht-common.c:11247:dht_pt_fgetxattr_cbk] 0-USER-HOME-dht: dict is null [2019-10-10 15:56:11.291742] E [MSGID: 101046] [dht-common.c:11248:dht_pt_fgetxattr_cbk] 0-USER-HOME-dht: dict is null [2019-10-10 15:56:11.974495] E [MSGID: 101046] [dht-common.c:11220:dht_pt_getxattr_cbk] 0-USER-HOME-dht: dict is null [2019-10-10 15:56:11.974568] E [MSGID: 101046] [dht-common.c:11221:dht_pt_getxattr_cbk] 0-USER-HOME-dht: dict is null The message "E [MSGID: 101046] [dht-common.c:11220:dht_pt_getxattr_cbk] 0-USER-HOME-dht: dict is null" repeated 8 times between [2019-10-10 15:56:11.974495] and [2019-10-10 15:56:23.911313] The message "E [MSGID: 101046] [dht-common.c:11221:dht_pt_getxattr_cbk] 0-USER-HOME-dht: dict is null" repeated 8 times between [2019-10-10 15:56:11.974568] and [2019-10-10 15:56:23.911316] ... ? And in case it might have something to do with above log errors, interestingly, if quotas are in use on paths in the volume then windows shares (Samba) deny to copy in new data saying that 0 bytes is free(which is false) -------------- next part -------------- A non-text attachment was scrubbed... Name: pEpkey.asc Type: application/pgp-keys Size: 1757 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191010/64b52bd0/attachment.bin>