search for: fops_v2

...ile descriptor in bad state}] [2024-01-20 07:23:58.561177 +0000] E [MSGID: 108028] [afr-open.c:361:afr_is_reopen_allowed_cbk] 0-workdata-replicate-3: Failed getlk for faf59566-10f5-4ddd-8b0c-a87bc6a334fb [File descriptor in bad state] [2024-01-20 07:23:58.562151 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-11: remote operation failed. [{path=<gfid:faf59566-10f5-4ddd-8b0c-a87bc6a334fb>}, {gfid=faf59566-10f5-4ddd-8b0c-a87b c6a334fb}, {errno=2}, {error=No such file or directory}] [2024-01-20 07:23:58.562296 +0000] W [MSGID: 114061] [client-common.c:53...

ok, finally managed to get all servers, volumes etc runnung, but took a couple of restarts, cksum checks etc. One problem: a volume doesn't heal automatically or doesn't heal at all. gluster volume status Status of volume: workdata Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------

...ile descriptor in bad state}] [2024-01-20 07:23:58.561177 +0000] E [MSGID: 108028] [afr-open.c:361:afr_is_reopen_allowed_cbk] 0-workdata-replicate-3: Failed getlk for faf59566-10f5-4ddd-8b0c-a87bc6a334fb [File descriptor in bad state] [2024-01-20 07:23:58.562151 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-11: remote operation failed. [{path=<gfid:faf59566-10f5-4ddd-8b0c-a87bc6a334fb>}, {gfid=faf59566-10f5-4ddd-8b0c-a87b c6a334fb}, {errno=2}, {error=No such file or directory}] [2024-01-20 07:23:58.562296 +0000] W [MSGID: 114061] [client-common.c:53...

...> [2024-01-20 07:23:58.561177 +0000] E [MSGID: 108028] > [afr-open.c:361:afr_is_reopen_allowed_cbk] 0-workdata-replicate-3: > Failed getlk for faf59566-10f5-4ddd-8b0c-a87bc6a334fb [File descriptor > in bad state] > [2024-01-20 07:23:58.562151 +0000] W [MSGID: 114031] > [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-11: > remote operation failed. > [{path=<gfid:faf59566-10f5-4ddd-8b0c-a87bc6a334fb>}, > {gfid=faf59566-10f5-4ddd-8b0c-a87b > c6a334fb}, {errno=2}, {error=No such file or directory}] > [2024-01-20 07:23:58.562296 +0000] W [MSGID: 11...

...e. does anyone have any suggestions on how to go about doing this? I've built 3 vms to be a new test cluster, but if I copy over a file from the 3 nodes and try to read it, I can't and get errors in /var/log/glusterfs/foo.log: [2023-08-12 03:50:47.638134 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv-client-0: remote operation failed. [{path=/helmetpart.scad}, {gfid=00000000-0000-0000-0000-000000000000} , {errno=61}, {error=No data available}] [2023-08-12 03:50:49.834859 +0000] E [MSGID: 122066] [ec-common.c:1301:ec_prepare_update_cbk] 0-gv-disperse-0: Unable t...

...> [2024-01-20 07:23:58.561177 +0000] E [MSGID: 108028] > [afr-open.c:361:afr_is_reopen_allowed_cbk] 0-workdata-replicate-3: > Failed getlk for faf59566-10f5-4ddd-8b0c-a87bc6a334fb [File descriptor > in bad state] > [2024-01-20 07:23:58.562151 +0000] W [MSGID: 114031] > [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-11: > remote operation failed. > [{path=<gfid:faf59566-10f5-4ddd-8b0c-a87bc6a334fb>}, > {gfid=faf59566-10f5-4ddd-8b0c-a87b > c6a334fb}, {errno=2}, {error=No such file or directory}] > [2024-01-20 07:23:58.562296 +0000] W [MSGID: 11...

...on > > > 10.4, but doesn't with version 11. What could be wrong? > > > > > > I only see these errors on one of the "good" servers in glustershd.log: > > > > > > [2024-01-18 06:08:57.328480 +0000] W [MSGID: 114031] > > > [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-0: > > > remote operation failed. > > > [{path=<gfid:cb39a1e4-2a4c-4727-861d-3ed9ef00681b>}, > > > {gfid=cb39a1e4-2a4c-4727-861d-3ed9e > > > f00681b}, {errno=2}, {error=No such file or directory}] > > &gt...

...was starting instantly e.g. after server reboot with version > 10.4, but doesn't with version 11. What could be wrong? > > I only see these errors on one of the "good" servers in glustershd.log: > > [2024-01-18 06:08:57.328480 +0000] W [MSGID: 114031] > [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-0: > remote operation failed. > [{path=<gfid:cb39a1e4-2a4c-4727-861d-3ed9ef00681b>}, > {gfid=cb39a1e4-2a4c-4727-861d-3ed9e > f00681b}, {errno=2}, {error=No such file or directory}] > [2024-01-18 06:08:57.594051 +0000] W [MSGID: 114...

...sum up to 60K per brick. Heal was starting instantly e.g. after server reboot with version 10.4, but doesn't with version 11. What could be wrong? I only see these errors on one of the "good" servers in glustershd.log: [2024-01-18 06:08:57.328480 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-0: remote operation failed. [{path=<gfid:cb39a1e4-2a4c-4727-861d-3ed9ef00681b>}, {gfid=cb39a1e4-2a4c-4727-861d-3ed9e f00681b}, {errno=2}, {error=No such file or directory}] [2024-01-18 06:08:57.594051 +0000] W [MSGID: 114031] [client-rpc-fops_v2....

...was starting instantly e.g. after server reboot with version > 10.4, but doesn't with version 11. What could be wrong? > > I only see these errors on one of the "good" servers in glustershd.log: > > [2024-01-18 06:08:57.328480 +0000] W [MSGID: 114031] > [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-0: > remote operation failed. > [{path=<gfid:cb39a1e4-2a4c-4727-861d-3ed9ef00681b>}, > {gfid=cb39a1e4-2a4c-4727-861d-3ed9e > f00681b}, {errno=2}, {error=No such file or directory}] > [2024-01-18 06:08:57.594051 +0000] W [MSGID: 114...

...directories do not exist. There are some logs below. [2024-06-18 14:02:52.931157 +0000] W [MSGID: 109037] [dht-common.c:4711:dht_getxattr_get_real_filename_cbk] 0-tank-dht: Failed to get real filename. [No such file or directory] [2024-06-18 14:02:52.934395 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2097:client4_0_create_cbk] 0-tank-client-5: remote operation failed. [{path=/dir59/BallData.mat}, {errno=2}, {error=No such file or directory}] [2024-06-18 14:04:52.221379 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2097:client4_0_create_cbk] 0-tank-client-5: remote operation failed. [{path...

...after server reboot with version > > 10.4, but doesn't with version 11. What could be wrong? > > > > I only see these errors on one of the "good" servers in glustershd.log: > > > > [2024-01-18 06:08:57.328480 +0000] W [MSGID: 114031] > > [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-0: > > remote operation failed. > > [{path=<gfid:cb39a1e4-2a4c-4727-861d-3ed9ef00681b>}, > > {gfid=cb39a1e4-2a4c-4727-861d-3ed9e > > f00681b}, {errno=2}, {error=No such file or directory}] > > [2024-01-18 06:08:57.59...

...t with version >>> 10.4, but doesn't with version 11. What could be wrong? >>> >>> I only see these errors on one of the "good" servers in glustershd.log: >>> >>> [2024-01-18 06:08:57.328480 +0000] W [MSGID: 114031] >>> [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-0: >>> remote operation failed. >>> [{path=<gfid:cb39a1e4-2a4c-4727-861d-3ed9ef00681b>}, >>> {gfid=cb39a1e4-2a4c-4727-861d-3ed9e >>> f00681b}, {errno=2}, {error=No such file or directory}] >>> [2024-01...

...after server reboot with version > > 10.4, but doesn't with version 11. What could be wrong? > > > > I only see these errors on one of the "good" servers in glustershd.log: > > > > [2024-01-18 06:08:57.328480 +0000] W [MSGID: 114031] > > [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-workdata-client-0: > > remote operation failed. > > [{path=<gfid:cb39a1e4-2a4c-4727-861d-3ed9ef00681b>}, > > {gfid=cb39a1e4-2a4c-4727-861d-3ed9e > > f00681b}, {errno=2}, {error=No such file or directory}] > > [2024-01-18 06:08:57.59...

...Number of Bricks: 1 x (2 + 1) = 3 Bricks: Brick1: node-gfs1:/gluster_bricks/data/data1 Brick2: node-gfs2:/gluster_bricks/data/data1 Brick3: node-gfs3:/gluster_bricks/data/data1 (arbiter) gluster volume heal data info Number of entries: 39 /var/log/glusterfs/glustershd.log The log appears client-rpc-fops_v2.c:785:client4_0_fsync_cbk] 0-data-client-0: remote operation failed. [{errno=5}, {error=Input/output error}] How should I solve these unrepaired entries -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/2...

...10 to 11 last week). Running a rebalance command should a lot of errors and failures in the status report of the rebalance. I also get quite a lot of this error in the logfile (settings is the small file I?m talking about) > > [2023-08-17 21:57:31.910494 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-apptivegrid-client-0: remote operation failed. [{path=/62/03/8c/62038cd116e9a6857794aa14/settings}, {gfid=1d38410a-1c14-4346-a7e5-68856ed310e9}, {errno=2}, {error=No such file or directory}] > > and this > > [2023-08-17 21:57:31.902676 +0000] I [MSGID:...

...he issue returns after about 1 day of backups running and before the backup cycle is complete. It has been happening the last 2 weekends we have run backups to these nodes. #example log msg from /var/log/glusterfs/home-volbackups.log [2023-05-01 21:43:15.450502 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:670:client4_0_writev_cbk] 0-volbackups-client-18: remote operation failed. [{errno=28}, {error=No space left on device}] Each glusterfs node has a single brick and mounts a single distributed volume as a glusterfs client locally and receives backup files to the volume each weekend. We distri...

...he issue returns after about 1 day of backups running and before the backup cycle is complete. It has been happening the last 2 weekends we have run backups to these nodes. #example log msg from /var/log/glusterfs/home-volbackups.log [2023-05-01 21:43:15.450502 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:670:client4_0_writev_cbk] 0-volbackups-client-18: remote operation failed. [{errno=28}, {error=No space left on device}] Each glusterfs node has a single brick and mounts a single distributed volume as a glusterfs client locally and receives backup files to the volume each weekend.? We distrib...

...e returns after about 1 day of backups running and before the backup cycle is complete. It has been happening the last 2 weekends we have run backups to these nodes. #example log msg from /var/log/glusterfs/home-volbackups.log [2023-05-01 21:43:15.450502 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:670:client4_0_writev_cbk] 0-volbackups-client-18: remote operation failed. [{errno=28}, {error=No space left on device}] Each glusterfs node has a single brick and mounts a single distributed volume as a glusterfs client locally and receives backup files to the volume each weekend. W...