Olaf Buitelaar
2019-Jun-20 12:52 UTC
[Gluster-users] glusterd crashes on Assertion failed: rsp.op == txn_op_info.op
Hi Sanju, you can download the coredump here; http://edgecastcdn.net/0004FA/files/core_dump.zip (around 20MB) Thanks Olaf Op do 20 jun. 2019 om 08:35 schreef Sanju Rakonde <srakonde at redhat.com>:> Olaf, > > Can you please paste complete backtrace from the core file, so that we can > analyse what is wrong here. > > On Wed, Jun 19, 2019 at 10:31 PM Olaf Buitelaar <olaf.buitelaar at gmail.com> > wrote: > >> Hi Atin, >> >> Thank you for pointing out this bug report, however no rebalancing task >> was running during this event. So maybe something else is causing this? >> According the report this should be fixed in gluster 6, unfortunate ovirt >> doesn't seem to officially support that version, so i'm stuck on the 5 >> branch for now. >> Any chance this will be back ported? >> >> Thanks Olaf >> >> >> Op wo 19 jun. 2019 om 17:57 schreef Atin Mukherjee <amukherj at redhat.com>: >> >>> Please see - https://bugzilla.redhat.com/show_bug.cgi?id=1655827 >>> >>> >>> >>> On Wed, Jun 19, 2019 at 5:52 PM Olaf Buitelaar <olaf.buitelaar at gmail.com> >>> wrote: >>> >>>> Dear All, >>>> >>>> Has anybody seen this error on gluster 5.6; >>>> [glusterd-rpc-ops.c:1388:__glusterd_commit_op_cbk] >>>> (-->/lib64/libgfrpc.so.0(+0xec60) [0x7fbfb7801c60] >>>> -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x79b7a) >>>> [0x7fbfac50db7a] >>>> -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x77393) >>>> [0x7fbfac50b393] ) 0-: Assertion failed: rsp.op == txn_op_info.op >>>> >>>> checking the code; >>>> https://github.com/gluster/glusterfs/blob/6fd8281ac9af58609979f660ece58c2ed1100e72/xlators/mgmt/glusterd/src/glusterd-rpc-ops.c#L1388 >>>> >>>> doesn't seem to reveal much on what could causing this. >>>> >>>> It's the second time this occurs. >>>> >>>> Attached the full stack. >>>> >>>> Thanks Olaf >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >>> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > > > -- > Thanks, > Sanju >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190620/9a2b4f8e/attachment.html>
Olaf Buitelaar
2019-Jun-20 14:00 UTC
[Gluster-users] glusterd crashes on Assertion failed: rsp.op == txn_op_info.op
Hi Sanju, going through the stacks i noticed that this function was in between; glusterd_volume_rebalance_use_rsp_dict So it might after all have todo something with the rebalancing logic. I've checked the cmd_history.log and exactly on the time of crash time command was executed; [2019-06-19 07:25:03.108360] : volume rebalance ovirt-data status : SUCCESS preceding a couple of other status checks of rebalancing. The complete batch of 2 mins before, all reported success. These commands are executed by ovirt about every 2 minutes, to pull for the status of gluster. I'm sure no actual rebalancing tasks were running, also checked the last time that was @2019-06-08 21:13:02 and was completed successfully Hopefully this is additional useful info. Thanks Olaf Op do 20 jun. 2019 om 14:52 schreef Olaf Buitelaar <olaf.buitelaar at gmail.com>:> Hi Sanju, > > you can download the coredump here; > http://edgecastcdn.net/0004FA/files/core_dump.zip (around 20MB) > > Thanks Olaf > > Op do 20 jun. 2019 om 08:35 schreef Sanju Rakonde <srakonde at redhat.com>: > >> Olaf, >> >> Can you please paste complete backtrace from the core file, so that we >> can analyse what is wrong here. >> >> On Wed, Jun 19, 2019 at 10:31 PM Olaf Buitelaar <olaf.buitelaar at gmail.com> >> wrote: >> >>> Hi Atin, >>> >>> Thank you for pointing out this bug report, however no rebalancing task >>> was running during this event. So maybe something else is causing this? >>> According the report this should be fixed in gluster 6, unfortunate >>> ovirt doesn't seem to officially support that version, so i'm stuck on the >>> 5 branch for now. >>> Any chance this will be back ported? >>> >>> Thanks Olaf >>> >>> >>> Op wo 19 jun. 2019 om 17:57 schreef Atin Mukherjee <amukherj at redhat.com >>> >: >>> >>>> Please see - https://bugzilla.redhat.com/show_bug.cgi?id=1655827 >>>> >>>> >>>> >>>> On Wed, Jun 19, 2019 at 5:52 PM Olaf Buitelaar < >>>> olaf.buitelaar at gmail.com> wrote: >>>> >>>>> Dear All, >>>>> >>>>> Has anybody seen this error on gluster 5.6; >>>>> [glusterd-rpc-ops.c:1388:__glusterd_commit_op_cbk] >>>>> (-->/lib64/libgfrpc.so.0(+0xec60) [0x7fbfb7801c60] >>>>> -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x79b7a) >>>>> [0x7fbfac50db7a] >>>>> -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x77393) >>>>> [0x7fbfac50b393] ) 0-: Assertion failed: rsp.op == txn_op_info.op >>>>> >>>>> checking the code; >>>>> https://github.com/gluster/glusterfs/blob/6fd8281ac9af58609979f660ece58c2ed1100e72/xlators/mgmt/glusterd/src/glusterd-rpc-ops.c#L1388 >>>>> >>>>> doesn't seem to reveal much on what could causing this. >>>>> >>>>> It's the second time this occurs. >>>>> >>>>> Attached the full stack. >>>>> >>>>> Thanks Olaf >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> >> -- >> Thanks, >> Sanju >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190620/47d3f0aa/attachment.html>