Artem Russakovskii
2020-Apr-30 01:00 UTC
[Gluster-users] Tons of dht: Found anomalies in (null)
Hi, Every time I ls large dirs in our 1x4 replicate gluster volume, I get a ton of these in the logs. If I run the same ls right away again, they won't repeat, but inevitably, in a couple of hours or days, they show up again. What is the meaning of these messages and why am I getting them so frequently? [2020-04-30 00:57:39.020757] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 9f879856-36fc-49f9-900f-0e5d742b2e54). Holes=1 overlaps=0 [2020-04-30 00:57:39.021036] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = d6c7f283-3607-4a4d-9099-6f76792d5bdd). Holes=1 overlaps=0 [2020-04-30 00:57:39.021067] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = d8f88e6d-d80e-40fc-a386-9dc35212ad91). Holes=1 overlaps=0 [2020-04-30 00:57:39.021079] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 704b3e6e-9f55-4166-8a0f-4283fb513593). Holes=1 overlaps=0 [2020-04-30 00:57:39.021103] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 2de3da38-c56d-4196-b001-e4fa7d901383). Holes=1 overlaps=0 [2020-04-30 00:57:39.021118] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = c0b2c50e-1abd-4c86-9958-2c6f87d469d1). Holes=1 overlaps=0 [2020-04-30 00:57:39.021400] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 5365eb39-f120-4a1c-9920-1e375c030843). Holes=1 overlaps=0 [2020-04-30 00:57:39.021436] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = f7f46524-28d8-4c57-b244-43f953ec3656). Holes=1 overlaps=0 [2020-04-30 00:57:39.021448] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 6cfd94ad-3b82-4a6f-9f0d-03cc0c59e5f1). Holes=1 overlaps=0 [2020-04-30 00:57:39.021455] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 285061fb-f1b7-49c0-ae10-5d8d8d5cd569). Holes=1 overlaps=0 [2020-04-30 00:57:39.021463] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 38734b38-381f-4825-b65a-4292c5075faa). Holes=1 overlaps=0 [2020-04-30 00:57:39.021470] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 60c15124-18da-40e7-8776-cd78303b79ad). Holes=1 overlaps=0 [2020-04-30 00:57:39.021481] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = e26783bf-bee7-4a59-ae3e-505ada95c5f9). Holes=1 overlaps=0 [2020-04-30 00:57:39.022163] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 6f7da42f-22bd-4e96-828d-02c1c1690afc). Holes=1 overlaps=0 [2020-04-30 00:57:39.022192] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 6d17f907-9fb3-4d13-9a2f-8266b0e3bead). Holes=1 overlaps=0 [2020-04-30 00:57:39.022205] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 64f40fc3-efb3-4359-98df-ba3be29b6326). Holes=1 overlaps=0 [2020-04-30 00:57:39.022214] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = d9081c5d-40e1-4521-a4b5-78fd60d71237). Holes=1 overlaps=0 [2020-04-30 00:57:39.022735] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = b627c175-d6eb-44de-9eed-a13efea52f88). Holes=1 overlaps=0 [2020-04-30 00:57:39.022771] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = 9dce8380-ca5d-4947-831c-0db1aae85e09). Holes=1 overlaps=0 [2020-04-30 00:57:39.022788] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies in (null) (gfid = a9c24397-a521-4d1b-8543-42b331fa6794). Holes=1 overlaps=0 Thanks. Sincerely, Artem -- Founder, Android Police <http://www.androidpolice.com>, APK Mirror <http://www.apkmirror.com/>, Illogical Robot LLC beerpla.net | @ArtemR <http://twitter.com/ArtemR> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200429/029b1ce4/attachment.html>
Artem Russakovskii
2020-May-18 22:09 UTC
[Gluster-users] Tons of dht: Found anomalies in (null)
Hi, Any updates on this one? It seems to be related to the listing performance issue I also emailed about, and resolving whatever is causing these could be key to resolving the dir listing slowdowns. Sincerely, Artem -- Founder, Android Police <http://www.androidpolice.com>, APK Mirror <http://www.apkmirror.com/>, Illogical Robot LLC beerpla.net | @ArtemR <http://twitter.com/ArtemR> On Wed, Apr 29, 2020 at 6:00 PM Artem Russakovskii <archon810 at gmail.com> wrote:> Hi, > > Every time I ls large dirs in our 1x4 replicate gluster volume, I get a > ton of these in the logs. > > If I run the same ls right away again, they won't repeat, but inevitably, > in a couple of hours or days, they show up again. > > What is the meaning of these messages and why am I getting them so > frequently? > > [2020-04-30 00:57:39.020757] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 9f879856-36fc-49f9-900f-0e5d742b2e54). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021036] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = d6c7f283-3607-4a4d-9099-6f76792d5bdd). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021067] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = d8f88e6d-d80e-40fc-a386-9dc35212ad91). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021079] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 704b3e6e-9f55-4166-8a0f-4283fb513593). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021103] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 2de3da38-c56d-4196-b001-e4fa7d901383). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021118] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = c0b2c50e-1abd-4c86-9958-2c6f87d469d1). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021400] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 5365eb39-f120-4a1c-9920-1e375c030843). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021436] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = f7f46524-28d8-4c57-b244-43f953ec3656). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021448] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 6cfd94ad-3b82-4a6f-9f0d-03cc0c59e5f1). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021455] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 285061fb-f1b7-49c0-ae10-5d8d8d5cd569). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021463] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 38734b38-381f-4825-b65a-4292c5075faa). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021470] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 60c15124-18da-40e7-8776-cd78303b79ad). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021481] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = e26783bf-bee7-4a59-ae3e-505ada95c5f9). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022163] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 6f7da42f-22bd-4e96-828d-02c1c1690afc). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022192] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 6d17f907-9fb3-4d13-9a2f-8266b0e3bead). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022205] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 64f40fc3-efb3-4359-98df-ba3be29b6326). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022214] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = d9081c5d-40e1-4521-a4b5-78fd60d71237). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022735] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = b627c175-d6eb-44de-9eed-a13efea52f88). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022771] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 9dce8380-ca5d-4947-831c-0db1aae85e09). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022788] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = a9c24397-a521-4d1b-8543-42b331fa6794). Holes=1 overlaps=0 > > Thanks. > > Sincerely, > Artem > > -- > Founder, Android Police <http://www.androidpolice.com>, APK Mirror > <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net | @ArtemR <http://twitter.com/ArtemR> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200518/4b874247/attachment.html>
On Thu, Apr 30, 2020 at 6:31 AM Artem Russakovskii <archon810 at gmail.com> wrote:> Hi, > > Every time I ls large dirs in our 1x4 replicate gluster volume, I get a > ton of these in the logs. > > If I run the same ls right away again, they won't repeat, but inevitably, > in a couple of hours or days, they show up again. > > What is the meaning of these messages and why am I getting them so > frequently? > > [2020-04-30 00:57:39.020757] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 9f879856-36fc-49f9-900f-0e5d742b2e54). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021036] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = d6c7f283-3607-4a4d-9099-6f76792d5bdd). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021067] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = d8f88e6d-d80e-40fc-a386-9dc35212ad91). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021079] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 704b3e6e-9f55-4166-8a0f-4283fb513593). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021103] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 2de3da38-c56d-4196-b001-e4fa7d901383). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021118] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = c0b2c50e-1abd-4c86-9958-2c6f87d469d1). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021400] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 5365eb39-f120-4a1c-9920-1e375c030843). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021436] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = f7f46524-28d8-4c57-b244-43f953ec3656). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021448] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 6cfd94ad-3b82-4a6f-9f0d-03cc0c59e5f1). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021455] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 285061fb-f1b7-49c0-ae10-5d8d8d5cd569). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021463] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 38734b38-381f-4825-b65a-4292c5075faa). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021470] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 60c15124-18da-40e7-8776-cd78303b79ad). Holes=1 overlaps=0 > [2020-04-30 00:57:39.021481] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = e26783bf-bee7-4a59-ae3e-505ada95c5f9). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022163] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 6f7da42f-22bd-4e96-828d-02c1c1690afc). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022192] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 6d17f907-9fb3-4d13-9a2f-8266b0e3bead). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022205] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 64f40fc3-efb3-4359-98df-ba3be29b6326). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022214] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = d9081c5d-40e1-4521-a4b5-78fd60d71237). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022735] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = b627c175-d6eb-44de-9eed-a13efea52f88). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022771] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = 9dce8380-ca5d-4947-831c-0db1aae85e09). Holes=1 overlaps=0 > [2020-04-30 00:57:39.022788] I [MSGID: 109063] > [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies > in (null) (gfid = a9c24397-a521-4d1b-8543-42b331fa6794). Holes=1 overlaps=0 >Generally, these logs indicate that there were directories missing ("hole 1") on one of the bricks. These are harmless message and you can ignore them.> >>Thanks.> > Sincerely, > Artem > > -- > Founder, Android Police <http://www.androidpolice.com>, APK Mirror > <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net | @ArtemR <http://twitter.com/ArtemR> > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200519/0f7a4b12/attachment.html>