thr3ads.net - Gluster users - [Gluster-users] GFID attir is missing after adding large amounts of data [Aug 2017]

If this information is useful, please help other people find it:
Share via:

Christoph Schäbel

2017-Aug-28 07:55 UTC

[Gluster-users] GFID attir is missing after adding large amounts of data

Hi Cluster Community,

we are seeing some problems when adding multiple terrabytes of data to a 2 node
replicated GlusterFS installation.

The version is 3.8.11 on CentOS 7.
The machines are connected via 10Gbit LAN and are running 24/7. The OS is
virtualized on VMWare.

After a restart of node-1 we see that the log files are growing to multiple
Gigabytes a day.

Also there seem to be problems with the replication.
The setup worked fine until sometime after we added the additional data (around
3 TB in size) to node-1. We added the data to a mountpoint via the client, not
directly to the brick.
What we did is add tar files via a client-mount and then untar them while in the
client-mount folder.
The brick (/mnt/brick1/gv0) is using the XFS filesystem.

When checking the file attributes of one of the files mentioned in the brick
logs, i can see that the gfid attribute is missing on node-1. On node-2 the file
does not even exist.

getfattr -m . -d -e hex
mnt/brick1/gv0/.glusterfs/40/59/40598e46-9868-4d7c-b494-7b978e67370a/type=type1/part-r-00002-4846e211-c81d-4c08-bb5e-f22fa5a4b404.gz.parquet

# file:
mnt/brick1/gv0/.glusterfs/40/59/40598e46-9868-4d7c-b494-7b978e67370a/type=type1/part-r-00002-4846e211-c81d-4c08-bb5e-f22fa5a4b404.gz.parquet
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000

We repeated this scenario a second time with a fresh setup and got the same
results.

Does anyone know what we are doing wrong ?

Is there maybe a problem with glusterfs and tar ?


Log excerpts:


glustershd.log

[2017-07-26 15:31:36.290908] I [MSGID: 108026]
[afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: performing
entry selfheal on fe5c42ac-5fda-47d4-8221-484c8d826c06
[2017-07-26 15:31:36.294289] W [MSGID: 114031]
[client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote operation
failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2017-07-26 15:31:36.298287] I [MSGID: 108026]
[afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: performing
entry selfheal on e31ae2ca-a3d2-4a27-a6ce-9aae24608141
[2017-07-26 15:31:36.300695] W [MSGID: 114031]
[client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote operation
failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2017-07-26 15:31:36.303626] I [MSGID: 108026]
[afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: performing
entry selfheal on 2cc9dafe-64d3-454a-a647-20deddfaebfe
[2017-07-26 15:31:36.305763] W [MSGID: 114031]
[client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote operation
failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2017-07-26 15:31:36.308639] I [MSGID: 108026]
[afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: performing
entry selfheal on cbabf9ed-41be-4d08-9cdb-5734557ddbea
[2017-07-26 15:31:36.310819] W [MSGID: 114031]
[client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote operation
failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]
[2017-07-26 15:31:36.315057] I [MSGID: 108026]
[afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: performing
entry selfheal on 8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69
[2017-07-26 15:31:36.317196] W [MSGID: 114031]
[client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote operation
failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available]



bricks/mnt-brick1-gv0.log

2017-07-26 15:31:36.287831] E [MSGID: 115050]
[server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153546: LOOKUP
<gfid:d99930df-6b47-4b55-9af3-c767afd6584c>/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
(d99930df-6b47-4b55-9af3-c767afd6584c/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
==> (No data available) [No data available]
[2017-07-26 15:31:36.294202] E [MSGID: 113002] [posix.c:266:posix_lookup]
0-gv0-posix: buf->ia_gfid is null for
/mnt/brick1/gv0/.glusterfs/e7/2d/e72d9005-b958-432b-b4a9-37aaadd9d2df/type=type1/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
[No data available]
[2017-07-26 15:31:36.294235] E [MSGID: 115050]
[server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153564: LOOKUP
<gfid:fe5c42ac-5fda-47d4-8221-484c8d826c06>/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
(fe5c42ac-5fda-47d4-8221-484c8d826c06/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
==> (No data available) [No data available]
[2017-07-26 15:31:36.300611] E [MSGID: 113002] [posix.c:266:posix_lookup]
0-gv0-posix: buf->ia_gfid is null for
/mnt/brick1/gv0/.glusterfs/33/d4/33d47146-bc30-49dd-ada8-475bb75435bf/type=type2/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
[No data available]
[2017-07-26 15:31:36.300645] E [MSGID: 115050]
[server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153582: LOOKUP
<gfid:e31ae2ca-a3d2-4a27-a6ce-9aae24608141>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
(e31ae2ca-a3d2-4a27-a6ce-9aae24608141/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
==> (No data available) [No data available]
[2017-07-26 15:31:36.305671] E [MSGID: 113002] [posix.c:266:posix_lookup]
0-gv0-posix: buf->ia_gfid is null for
/mnt/brick1/gv0/.glusterfs/33/d4/33d47146-bc30-49dd-ada8-475bb75435bf/type=type1/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
[No data available]
[2017-07-26 15:31:36.305711] E [MSGID: 115050]
[server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153600: LOOKUP
<gfid:2cc9dafe-64d3-454a-a647-20deddfaebfe>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
(2cc9dafe-64d3-454a-a647-20deddfaebfe/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
==> (No data available) [No data available]
[2017-07-26 15:31:36.310735] E [MSGID: 113002] [posix.c:266:posix_lookup]
0-gv0-posix: buf->ia_gfid is null for
/mnt/brick1/gv0/.glusterfs/df/71/df715321-3078-47c8-bf23-dec47abe46d7/type=type2/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
[No data available]
[2017-07-26 15:31:36.310767] E [MSGID: 115050]
[server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153618: LOOKUP
<gfid:cbabf9ed-41be-4d08-9cdb-5734557ddbea>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
(cbabf9ed-41be-4d08-9cdb-5734557ddbea/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
==> (No data available) [No data available]
[2017-07-26 15:31:36.317113] E [MSGID: 113002] [posix.c:266:posix_lookup]
0-gv0-posix: buf->ia_gfid is null for
/mnt/brick1/gv0/.glusterfs/df/71/df715321-3078-47c8-bf23-dec47abe46d7/type=type3/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
[No data available]
[2017-07-26 15:31:36.317146] E [MSGID: 115050]
[server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153636: LOOKUP
<gfid:8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
(8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
==> (No data available) [No data available]


Regards,
Christoph

Ben Turner

2017-Aug-29 04:35 UTC

head link

[Gluster-users] GFID attir is missing after adding large amounts of data

This is strange, a couple of questions:

1.  What volume type is this?  What tuning have you done?  gluster v info output
would be helpful here.

2.  How big are your bricks?

3.  Can you write me a quick reproducer so I can try this in the lab?  Is it
just a single multi TB file you are untarring or many?  If you give me the steps
to repro, and I hit it, we can get a bug open.

4.  Other than this are you seeing any other problems?  What if you untar a
smaller file(s)?  Can you read and write to the volume with say DD without any
problems?

It sounds like you have some other issues affecting things here, there is no
reason why you shouldn't be able to untar and write multiple TBs of data to
gluster.  Go ahead and answer those questions and I'll see what I can do to
help you out.

-b

----- Original Message -----> From: "Christoph Sch?bel" <christoph.schaebel at
dc-square.de>
> To: gluster-users at gluster.org
> Sent: Monday, August 28, 2017 3:55:31 AM
> Subject: [Gluster-users] GFID attir is missing after adding large amounts
of	data
> 
> Hi Cluster Community,
> 
> we are seeing some problems when adding multiple terrabytes of data to a 2
> node replicated GlusterFS installation.
> 
> The version is 3.8.11 on CentOS 7.
> The machines are connected via 10Gbit LAN and are running 24/7. The OS is
> virtualized on VMWare.
> 
> After a restart of node-1 we see that the log files are growing to multiple
> Gigabytes a day.
> 
> Also there seem to be problems with the replication.
> The setup worked fine until sometime after we added the additional data
> (around 3 TB in size) to node-1. We added the data to a mountpoint via the
> client, not directly to the brick.
> What we did is add tar files via a client-mount and then untar them while
in
> the client-mount folder.
> The brick (/mnt/brick1/gv0) is using the XFS filesystem.
> 
> When checking the file attributes of one of the files mentioned in the
brick
> logs, i can see that the gfid attribute is missing on node-1. On node-2 the
> file does not even exist.
> 
> getfattr -m . -d -e hex
>
mnt/brick1/gv0/.glusterfs/40/59/40598e46-9868-4d7c-b494-7b978e67370a/type=type1/part-r-00002-4846e211-c81d-4c08-bb5e-f22fa5a4b404.gz.parquet
> 
> # file:
>
mnt/brick1/gv0/.glusterfs/40/59/40598e46-9868-4d7c-b494-7b978e67370a/type=type1/part-r-00002-4846e211-c81d-4c08-bb5e-f22fa5a4b404.gz.parquet
>
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
> 
> We repeated this scenario a second time with a fresh setup and got the same
> results.
> 
> Does anyone know what we are doing wrong ?
> 
> Is there maybe a problem with glusterfs and tar ?
> 
> 
> Log excerpts:
> 
> 
> glustershd.log
> 
> [2017-07-26 15:31:36.290908] I [MSGID: 108026]
> [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> performing entry selfheal on fe5c42ac-5fda-47d4-8221-484c8d826c06
> [2017-07-26 15:31:36.294289] W [MSGID: 114031]
> [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> data available]
> [2017-07-26 15:31:36.298287] I [MSGID: 108026]
> [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> performing entry selfheal on e31ae2ca-a3d2-4a27-a6ce-9aae24608141
> [2017-07-26 15:31:36.300695] W [MSGID: 114031]
> [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> data available]
> [2017-07-26 15:31:36.303626] I [MSGID: 108026]
> [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> performing entry selfheal on 2cc9dafe-64d3-454a-a647-20deddfaebfe
> [2017-07-26 15:31:36.305763] W [MSGID: 114031]
> [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> data available]
> [2017-07-26 15:31:36.308639] I [MSGID: 108026]
> [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> performing entry selfheal on cbabf9ed-41be-4d08-9cdb-5734557ddbea
> [2017-07-26 15:31:36.310819] W [MSGID: 114031]
> [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> data available]
> [2017-07-26 15:31:36.315057] I [MSGID: 108026]
> [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> performing entry selfheal on 8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69
> [2017-07-26 15:31:36.317196] W [MSGID: 114031]
> [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No
> data available]
> 
> 
> 
> bricks/mnt-brick1-gv0.log
> 
> 2017-07-26 15:31:36.287831] E [MSGID: 115050]
> [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153546: LOOKUP
>
<gfid:d99930df-6b47-4b55-9af3-c767afd6584c>/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
>
(d99930df-6b47-4b55-9af3-c767afd6584c/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> ==> (No data available) [No data available]
> [2017-07-26 15:31:36.294202] E [MSGID: 113002] [posix.c:266:posix_lookup]
> 0-gv0-posix: buf->ia_gfid is null for
>
/mnt/brick1/gv0/.glusterfs/e7/2d/e72d9005-b958-432b-b4a9-37aaadd9d2df/type=type1/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> [No data available]
> [2017-07-26 15:31:36.294235] E [MSGID: 115050]
> [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153564: LOOKUP
>
<gfid:fe5c42ac-5fda-47d4-8221-484c8d826c06>/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
>
(fe5c42ac-5fda-47d4-8221-484c8d826c06/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> ==> (No data available) [No data available]
> [2017-07-26 15:31:36.300611] E [MSGID: 113002] [posix.c:266:posix_lookup]
> 0-gv0-posix: buf->ia_gfid is null for
>
/mnt/brick1/gv0/.glusterfs/33/d4/33d47146-bc30-49dd-ada8-475bb75435bf/type=type2/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> [No data available]
> [2017-07-26 15:31:36.300645] E [MSGID: 115050]
> [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153582: LOOKUP
>
<gfid:e31ae2ca-a3d2-4a27-a6ce-9aae24608141>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
>
(e31ae2ca-a3d2-4a27-a6ce-9aae24608141/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> ==> (No data available) [No data available]
> [2017-07-26 15:31:36.305671] E [MSGID: 113002] [posix.c:266:posix_lookup]
> 0-gv0-posix: buf->ia_gfid is null for
>
/mnt/brick1/gv0/.glusterfs/33/d4/33d47146-bc30-49dd-ada8-475bb75435bf/type=type1/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> [No data available]
> [2017-07-26 15:31:36.305711] E [MSGID: 115050]
> [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153600: LOOKUP
>
<gfid:2cc9dafe-64d3-454a-a647-20deddfaebfe>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
>
(2cc9dafe-64d3-454a-a647-20deddfaebfe/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> ==> (No data available) [No data available]
> [2017-07-26 15:31:36.310735] E [MSGID: 113002] [posix.c:266:posix_lookup]
> 0-gv0-posix: buf->ia_gfid is null for
>
/mnt/brick1/gv0/.glusterfs/df/71/df715321-3078-47c8-bf23-dec47abe46d7/type=type2/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> [No data available]
> [2017-07-26 15:31:36.310767] E [MSGID: 115050]
> [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153618: LOOKUP
>
<gfid:cbabf9ed-41be-4d08-9cdb-5734557ddbea>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
>
(cbabf9ed-41be-4d08-9cdb-5734557ddbea/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> ==> (No data available) [No data available]
> [2017-07-26 15:31:36.317113] E [MSGID: 113002] [posix.c:266:posix_lookup]
> 0-gv0-posix: buf->ia_gfid is null for
>
/mnt/brick1/gv0/.glusterfs/df/71/df715321-3078-47c8-bf23-dec47abe46d7/type=type3/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> [No data available]
> [2017-07-26 15:31:36.317146] E [MSGID: 115050]
> [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153636: LOOKUP
>
<gfid:8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
>
(8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> ==> (No data available) [No data available]
> 
> 
> Regards,
> Christoph
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>

Ben Turner

2017-Aug-29 04:36 UTC

head link

[Gluster-users] GFID attir is missing after adding large amounts of data

Also include gluster v status, I want to check the status of your bricks and SHD
processes.

-b

----- Original Message -----> From: "Ben Turner" <bturner at redhat.com>
> To: "Christoph Sch?bel" <christoph.schaebel at
dc-square.de>
> Cc: gluster-users at gluster.org
> Sent: Tuesday, August 29, 2017 12:35:05 AM
> Subject: Re: [Gluster-users] GFID attir is missing after adding large
amounts of	data
> 
> This is strange, a couple of questions:
> 
> 1.  What volume type is this?  What tuning have you done?  gluster v info
> output would be helpful here.
> 
> 2.  How big are your bricks?
> 
> 3.  Can you write me a quick reproducer so I can try this in the lab?  Is
it
> just a single multi TB file you are untarring or many?  If you give me the
> steps to repro, and I hit it, we can get a bug open.
> 
> 4.  Other than this are you seeing any other problems?  What if you untar a
> smaller file(s)?  Can you read and write to the volume with say DD without
> any problems?
> 
> It sounds like you have some other issues affecting things here, there is
no
> reason why you shouldn't be able to untar and write multiple TBs of
data to
> gluster.  Go ahead and answer those questions and I'll see what I can
do to
> help you out.
> 
> -b
> 
> ----- Original Message -----
> > From: "Christoph Sch?bel" <christoph.schaebel at
dc-square.de>
> > To: gluster-users at gluster.org
> > Sent: Monday, August 28, 2017 3:55:31 AM
> > Subject: [Gluster-users] GFID attir is missing after adding large
amounts
> > of	data
> > 
> > Hi Cluster Community,
> > 
> > we are seeing some problems when adding multiple terrabytes of data to
a 2
> > node replicated GlusterFS installation.
> > 
> > The version is 3.8.11 on CentOS 7.
> > The machines are connected via 10Gbit LAN and are running 24/7. The OS
is
> > virtualized on VMWare.
> > 
> > After a restart of node-1 we see that the log files are growing to
multiple
> > Gigabytes a day.
> > 
> > Also there seem to be problems with the replication.
> > The setup worked fine until sometime after we added the additional
data
> > (around 3 TB in size) to node-1. We added the data to a mountpoint via
the
> > client, not directly to the brick.
> > What we did is add tar files via a client-mount and then untar them
while
> > in
> > the client-mount folder.
> > The brick (/mnt/brick1/gv0) is using the XFS filesystem.
> > 
> > When checking the file attributes of one of the files mentioned in the
> > brick
> > logs, i can see that the gfid attribute is missing on node-1. On
node-2 the
> > file does not even exist.
> > 
> > getfattr -m . -d -e hex
> >
mnt/brick1/gv0/.glusterfs/40/59/40598e46-9868-4d7c-b494-7b978e67370a/type=type1/part-r-00002-4846e211-c81d-4c08-bb5e-f22fa5a4b404.gz.parquet
> > 
> > # file:
> >
mnt/brick1/gv0/.glusterfs/40/59/40598e46-9868-4d7c-b494-7b978e67370a/type=type1/part-r-00002-4846e211-c81d-4c08-bb5e-f22fa5a4b404.gz.parquet
> >
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
> > 
> > We repeated this scenario a second time with a fresh setup and got the
same
> > results.
> > 
> > Does anyone know what we are doing wrong ?
> > 
> > Is there maybe a problem with glusterfs and tar ?
> > 
> > 
> > Log excerpts:
> > 
> > 
> > glustershd.log
> > 
> > [2017-07-26 15:31:36.290908] I [MSGID: 108026]
> > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> > performing entry selfheal on fe5c42ac-5fda-47d4-8221-484c8d826c06
> > [2017-07-26 15:31:36.294289] W [MSGID: 114031]
> > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[No
> > data available]
> > [2017-07-26 15:31:36.298287] I [MSGID: 108026]
> > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> > performing entry selfheal on e31ae2ca-a3d2-4a27-a6ce-9aae24608141
> > [2017-07-26 15:31:36.300695] W [MSGID: 114031]
> > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[No
> > data available]
> > [2017-07-26 15:31:36.303626] I [MSGID: 108026]
> > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> > performing entry selfheal on 2cc9dafe-64d3-454a-a647-20deddfaebfe
> > [2017-07-26 15:31:36.305763] W [MSGID: 114031]
> > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[No
> > data available]
> > [2017-07-26 15:31:36.308639] I [MSGID: 108026]
> > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> > performing entry selfheal on cbabf9ed-41be-4d08-9cdb-5734557ddbea
> > [2017-07-26 15:31:36.310819] W [MSGID: 114031]
> > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[No
> > data available]
> > [2017-07-26 15:31:36.315057] I [MSGID: 108026]
> > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0:
> > performing entry selfheal on 8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69
> > [2017-07-26 15:31:36.317196] W [MSGID: 114031]
> > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote
> > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
[No
> > data available]
> > 
> > 
> > 
> > bricks/mnt-brick1-gv0.log
> > 
> > 2017-07-26 15:31:36.287831] E [MSGID: 115050]
> > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153546:
LOOKUP
> >
<gfid:d99930df-6b47-4b55-9af3-c767afd6584c>/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> >
(d99930df-6b47-4b55-9af3-c767afd6584c/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> > ==> (No data available) [No data available]
> > [2017-07-26 15:31:36.294202] E [MSGID: 113002]
[posix.c:266:posix_lookup]
> > 0-gv0-posix: buf->ia_gfid is null for
> >
/mnt/brick1/gv0/.glusterfs/e7/2d/e72d9005-b958-432b-b4a9-37aaadd9d2df/type=type1/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > [No data available]
> > [2017-07-26 15:31:36.294235] E [MSGID: 115050]
> > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153564:
LOOKUP
> >
<gfid:fe5c42ac-5fda-47d4-8221-484c8d826c06>/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> >
(fe5c42ac-5fda-47d4-8221-484c8d826c06/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> > ==> (No data available) [No data available]
> > [2017-07-26 15:31:36.300611] E [MSGID: 113002]
[posix.c:266:posix_lookup]
> > 0-gv0-posix: buf->ia_gfid is null for
> >
/mnt/brick1/gv0/.glusterfs/33/d4/33d47146-bc30-49dd-ada8-475bb75435bf/type=type2/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > [No data available]
> > [2017-07-26 15:31:36.300645] E [MSGID: 115050]
> > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153582:
LOOKUP
> >
<gfid:e31ae2ca-a3d2-4a27-a6ce-9aae24608141>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> >
(e31ae2ca-a3d2-4a27-a6ce-9aae24608141/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> > ==> (No data available) [No data available]
> > [2017-07-26 15:31:36.305671] E [MSGID: 113002]
[posix.c:266:posix_lookup]
> > 0-gv0-posix: buf->ia_gfid is null for
> >
/mnt/brick1/gv0/.glusterfs/33/d4/33d47146-bc30-49dd-ada8-475bb75435bf/type=type1/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > [No data available]
> > [2017-07-26 15:31:36.305711] E [MSGID: 115050]
> > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153600:
LOOKUP
> >
<gfid:2cc9dafe-64d3-454a-a647-20deddfaebfe>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> >
(2cc9dafe-64d3-454a-a647-20deddfaebfe/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> > ==> (No data available) [No data available]
> > [2017-07-26 15:31:36.310735] E [MSGID: 113002]
[posix.c:266:posix_lookup]
> > 0-gv0-posix: buf->ia_gfid is null for
> >
/mnt/brick1/gv0/.glusterfs/df/71/df715321-3078-47c8-bf23-dec47abe46d7/type=type2/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > [No data available]
> > [2017-07-26 15:31:36.310767] E [MSGID: 115050]
> > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153618:
LOOKUP
> >
<gfid:cbabf9ed-41be-4d08-9cdb-5734557ddbea>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> >
(cbabf9ed-41be-4d08-9cdb-5734557ddbea/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> > ==> (No data available) [No data available]
> > [2017-07-26 15:31:36.317113] E [MSGID: 113002]
[posix.c:266:posix_lookup]
> > 0-gv0-posix: buf->ia_gfid is null for
> >
/mnt/brick1/gv0/.glusterfs/df/71/df715321-3078-47c8-bf23-dec47abe46d7/type=type3/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> > [No data available]
> > [2017-07-26 15:31:36.317146] E [MSGID: 115050]
> > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153636:
LOOKUP
> >
<gfid:8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet
> >
(8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet)
> > ==> (No data available) [No data available]
> > 
> > 
> > Regards,
> > Christoph
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> > 
>

Reasonably Related Threads

Search for more possibly parallel threads

Gluster users - Aug 2017 - GFID attir is missing after adding large amounts of data

[Gluster-users] GFID attir is missing after adding large amounts of data

[Gluster-users] GFID attir is missing after adding large amounts of data

[Gluster-users] GFID attir is missing after adding large amounts of data

Reasonably Related Threads