On Thu, Apr 30, 2020 at 6:31 AM Artem Russakovskii <archon810 at gmail.com> wrote:> Hi, > > Every time I ls large dirs in our 1x4 replicate gluster volume, I get a > ton of these in the logs. > > If I run the same ls right away again, they won't repeat, but inevitably, > in a couple of hours or days, they show up again. > > What is the meaning of these messages and why am I getting them so > frequently? > > [2020-04-30 00:57:39.020757] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 9f879856-36fc-49f9-900f-0e5d742b2e54). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021036] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = d6c7f283-3607-4a4d-9099-6f76792d5bdd). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021067] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = d8f88e6d-d80e-40fc-a386-9dc35212ad91). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021079] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 704b3e6e-9f55-4166-8a0f-4283fb513593). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021103] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 2de3da38-c56d-4196-b001-e4fa7d901383). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021118] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = c0b2c50e-1abd-4c86-9958-2c6f87d469d1). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021400] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 5365eb39-f120-4a1c-9920-1e375c030843). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021436] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = f7f46524-28d8-4c57-b244-43f953ec3656). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021448] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 6cfd94ad-3b82-4a6f-9f0d-03cc0c59e5f1). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021455] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 285061fb-f1b7-49c0-ae10-5d8d8d5cd569). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021463] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 38734b38-381f-4825-b65a-4292c5075faa). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021470] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 60c15124-18da-40e7-8776-cd78303b79ad). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021481] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = e26783bf-bee7-4a59-ae3e-505ada95c5f9). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022163] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 6f7da42f-22bd-4e96-828d-02c1c1690afc). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022192] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 6d17f907-9fb3-4d13-9a2f-8266b0e3bead). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022205] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 64f40fc3-efb3-4359-98df-ba3be29b6326). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022214] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = d9081c5d-40e1-4521-a4b5-78fd60d71237). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022735] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = b627c175-d6eb-44de-9eed-a13efea52f88). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022771] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 9dce8380-ca5d-4947-831c-0db1aae85e09). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022788] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = a9c24397-a521-4d1b-8543-42b331fa6794). Holes=1 overlaps=0 >Generally, these logs indicate that there were directories missing ("hole 1") on one of the bricks. These are harmless message and you can ignore them.> >>Thanks.> > Sincerely, > Artem > > -- > Founder, Android Police <http://www.androidpolice.com>, APK Mirror > <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net | @ArtemR <http://twitter.com/ArtemR> > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200519/0f7a4b12/attachment.html>
> On 19-May-2020, at 12:05 PM, Susant Palai <spalai at redhat.com> wrote: > > > > On Thu, Apr 30, 2020 at 6:31 AM Artem Russakovskii <archon810 at gmail.com <mailto:archon810 at gmail.com>> wrote: > Hi, > > Every time I ls large dirs in our 1x4 replicate gluster volume, I get a ton of these in the logs. > > If I run the same ls right away again, they won't repeat, but inevitably, in a couple of hours or days, they show up again. > > What is the meaning of these messages and why am I getting them so frequently? > > [2020-04-30 00:57:39.020757] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 9f879856-36fc-49f9-900f-0e5d742b2e54). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021036] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = d6c7f283-3607-4a4d-9099-6f76792d5bdd). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021067] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = d8f88e6d-d80e-40fc-a386-9dc35212ad91). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021079] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 704b3e6e-9f55-4166-8a0f-4283fb513593). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021103] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 2de3da38-c56d-4196-b001-e4fa7d901383). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021118] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = c0b2c50e-1abd-4c86-9958-2c6f87d469d1). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021400] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 5365eb39-f120-4a1c-9920-1e375c030843). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021436] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = f7f46524-28d8-4c57-b244-43f953ec3656). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021448] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 6cfd94ad-3b82-4a6f-9f0d-03cc0c59e5f1). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021455] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 285061fb-f1b7-49c0-ae10-5d8d8d5cd569). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021463] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 38734b38-381f-4825-b65a-4292c5075faa). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021470] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 60c15124-18da-40e7-8776-cd78303b79ad). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021481] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = e26783bf-bee7-4a59-ae3e-505ada95c5f9). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022163] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 6f7da42f-22bd-4e96-828d-02c1c1690afc). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022192] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 6d17f907-9fb3-4d13-9a2f-8266b0e3bead). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022205] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 64f40fc3-efb3-4359-98df-ba3be29b6326). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022214] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = d9081c5d-40e1-4521-a4b5-78fd60d71237). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022735] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = b627c175-d6eb-44de-9eed-a13efea52f88). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022771] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 9dce8380-ca5d-4947-831c-0db1aae85e09). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022788] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = a9c24397-a521-4d1b-8543-42b331fa6794). Holes=1 overlaps=0 > > Generally, these logs indicate that there were directories missing ("hole = 1") on one of the bricks. These are harmless message and you can ignore them.Good opportunity to improve the log messages. Can we disable this logging if this can be ignored? Is it when distribute count is 1 or these messages are always harmless?> > > > > Thanks. > > Sincerely, > Artem > > -- > Founder, Android Police <http://www.androidpolice.com/>, APK Mirror <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net <http://beerpla.net/> | @ArtemR <http://twitter.com/ArtemR> > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 <https://bluejeans.com/441850968> > > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > https://lists.gluster.org/mailman/listinfo/gluster-users <https://lists.gluster.org/mailman/listinfo/gluster-users> > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-usersAravinda Vishwanathapura https://kadalu.io -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200519/bcd57d33/attachment.html>
Gionatan Danti
2020-May-19 10:56 UTC
[Gluster-users] Tons of dht: Found anomalies in (null)
Il 2020-05-19 08:35 Susant Palai ha scritto:> Generally, these logs indicate that there were directories missing > ("hole = 1") on one of the bricks. These are harmless message and you > can ignore them.I would guess that a message warning about missing directories on a brick is not to be ignore - ie it warns about something wrong. Am I missing something? -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it [1] email: g.danti at assyoma.it - info at assyoma.it GPG public key ID: FF5F32A8