search for: client3_3_lookup_cbk

...ate_cbk] 0-gv0-dht: revalidate lookup of / returned with op_ret 0 [Structure needs cleaning] [2017-09-20 13:34:23.347612] D [MSGID: 0] [dht-common.c:2699:dht_lookup] 0-gv0-dht: Calling fresh lookup for /tempdir3 on gv0-replicate-0 [2017-09-20 13:34:23.348013] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3fff88001080, gv0-client-1 returned -1 error: No such file or directory [No such file or directory] [2017-09-20 13:34:23.348013] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x3fff88001080, gv0-client-0 returned -1 err...

Looks like it is an issue with architecture compatibility in RPC layer (ie, with XDRs and how it is used). Just glance the logs of the client process where you saw the errors, which could give some hints. If you don't understand the logs, share them, so we will try to look into it. -Amar On Wed, Sep 20, 2017 at 2:40 AM, Walter Deignan <WDeignan at uline.com> wrote: > I recently

...4766] I [MSGID: 100011] [glusterfsd.c:1414:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2017-11-15 08:18:02.085718] I [glusterfsd-mgmt.c:1789:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2017-11-15 19:13:42.005307] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-data01-client-0: remote operation failed. Path: <gfid:7e8513f4-d4e2-4e66-b0ba-2dbe4c803c54> (7e8513f4-d4e2-4e66-b0ba-2dbe4c803c54) [No such file or directory] The message "W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-data01-client-0: remote operation failed. Path:...

I recently compiled the 3.10-5 client from source on a few PPC64 systems running RHEL 7.3. They are mounting a Gluster volume which is hosted on more traditional x86 servers. Everything seems to be working properly except for creating new directories from the PPC64 clients. The mkdir command gives a "Input/output error" and for the first few minutes the new directory is

...fs and tar ? Log excerpts: glustershd.log [2017-07-26 15:31:36.290908] I [MSGID: 108026] [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: performing entry selfheal on fe5c42ac-5fda-47d4-8221-484c8d826c06 [2017-07-26 15:31:36.294289] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available] [2017-07-26 15:31:36.298287] I [MSGID: 108026] [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: performing entry selfheal on e31ae2ca-a3d2-4a27-a6ce-9aae24608141 [20...

On 11/16/2017 04:12 PM, Nithya Balachandran wrote: > > > On 15 November 2017 at 19:57, Frederic Harmignies > <frederic.harmignies at elementai.com > <mailto:frederic.harmignies at elementai.com>> wrote: > > Hello, we have 2x files that are missing from one of the bricks. > No idea how to fix this. > > Details: > > # gluster volume

...glustershd.log > > [2017-07-26 15:31:36.290908] I [MSGID: 108026] > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: > performing entry selfheal on fe5c42ac-5fda-47d4-8221-484c8d826c06 > [2017-07-26 15:31:36.294289] W [MSGID: 114031] > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No > data available] > [2017-07-26 15:31:36.298287] I [MSGID: 108026] > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: > performing entry selfheal on e31ae2ca-a3d2-4...

...gt;> I repeated the process and did another rebalance while the VM running and >> everything went fine. >> >> But the logs in the client throwing lots of warning messages: >> >> >> [2017-05-29 13:14:59.416382] W [MSGID: 114031] >> [client-rpc-fops.c:2928:client3_3_lookup_cbk] 2-gfs_vol2-client-2: >> remote operation failed. Path: /50294ed6-db7a-418d-965f-9b44c >> 69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f >> (93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or directory] >> [2017-05-29 13:14:59.41...

...> normally without issues. > > I repeated the process and did another rebalance while the VM running and > everything went fine. > > But the logs in the client throwing lots of warning messages: > > > [2017-05-29 13:14:59.416382] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] > 2-gfs_vol2-client-2: remote operation failed. Path: > /50294ed6-db7a-418d-965f-9b44c69a83fd/images/d59487fe- > f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f > (93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or directory] > [2017-05-29 13:14:59.416427] W [MS...

...t direction. Log excerpts: glustershd.log [2017-07-26 15:31:36.290908] I [MSGID: 108026] [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: performing entry selfheal on fe5c42ac-5fda-47d4-8221-484c8d826c06 [2017-07-26 15:31:36.294289] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No data available] [2017-07-26 15:31:36.298287] I [MSGID: 108026] [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: performing entry selfheal on e31ae2ca-a3d2-4a27-a6ce-9aae24608141 [20...

...ss and did another rebalance while the VM running >>> and everything went fine. >>> >>> But the logs in the client throwing lots of warning messages: >>> >>> >>> [2017-05-29 13:14:59.416382] W [MSGID: 114031] >>> [client-rpc-fops.c:2928:client3_3_lookup_cbk] 2-gfs_vol2-client-2: >>> remote operation failed. Path: /50294ed6-db7a-418d-965f-9b44c >>> 69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f >>> (93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or directory] >>> [2017-0...

...while the VM running >>>> and everything went fine. >>>> >>>> But the logs in the client throwing lots of warning messages: >>>> >>>> >>>> [2017-05-29 13:14:59.416382] W [MSGID: 114031] >>>> [client-rpc-fops.c:2928:client3_3_lookup_cbk] 2-gfs_vol2-client-2: >>>> remote operation failed. Path: /50294ed6-db7a-418d-965f-9b44c >>>> 69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f >>>> (93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or directory] >>...

...ocess i rebooted the VM, and it did start normally without issues. I repeated the process and did another rebalance while the VM running and everything went fine. But the logs in the client throwing lots of warning messages: [2017-05-29 13:14:59.416382] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 2-gfs_vol2-client-2: remote operation failed. Path: /50294ed6-db7a-418d-965f-9b44c69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f (93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or directory] [2017-05-29 13:14:59.416427] W [MSGID: 114031] [client-rpc-f...

...gt;> I repeated the process and did another rebalance while the VM running and >> everything went fine. >> >> But the logs in the client throwing lots of warning messages: >> >> >> [2017-05-29 13:14:59.416382] W [MSGID: 114031] >> [client-rpc-fops.c:2928:client3_3_lookup_cbk] 2-gfs_vol2-client-2: >> remote operation failed. Path: /50294ed6-db7a-418d-965f-9b44c >> 69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f >> (93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or directory] >> [2017-05-29 13:14:59.41...

...v/azipfile.zip (hash=myvol-private-replicate-0/cache=<nul>) [2018-04-09 06:58:53.692440] W [MSGID: 114031] [client-rpc-fops.c:670:client3_3_rmdir_cbk] 0-myvol-private-client-2: remote operation failed [Directory not empty] [2018-04-09 06:58:53.714129] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-myvol-private-client-1: remote operation failed. Path: <gfid:13880e8c-13da-442f-8180-fa40b6f5327c> (13880e8c-13da-442f-8180-fa40b6f5327c) [No such file or directory] [2018-04-09 06:58:53.714161] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-myvol-private-client-0: remot...

...called at 2017-06-17 09:07:59.914614 (xid=0xa4141) [2017-06-17 09:08:26.033788] E [MSGID: 114031] [client-rpc-fops.c:1596:client3_3_finodelk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected] [2017-06-17 09:08:26.033799] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gluster1-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected] [2017-06-17 09:08:26.034008] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.s...

...yvol-private-replicate-0/cache=<nul>) > > [2018-04-09 06:58:53.692440] W [MSGID: 114031] [client-rpc-fops.c:670:client3_3_rmdir_cbk] 0-myvol-private-client-2: remote operation failed [Directory not empty] > > [2018-04-09 06:58:53.714129] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-myvol-private-client-1: remote operation failed. Path: gfid:13880e8c-13da-442f-8180-fa40b6f5327c (13880e8c-13da-442f-8180-fa40b6f5327c) [No such file or directory] > > [2018-04-09 06:58:53.714161] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-myvol-private-client-0: re...

...times between [2017-12-21 16:36:37.171846] and [2017-12-21 16:36:37.172225] [2017-12-21 16:36:37.172457] D [MSGID: 0] [dht-common.c:2692:dht_lookup] 0-video-backup-dht: Calling fresh lookup for /path/file.txt on video-backup-client-4 [2017-12-21 16:36:37.173166] D [MSGID: 0] [client-rpc-fops.c:2910:client3_3_lookup_cbk] 0-video-backup-client-4: gfid changed for /path/file.txt [2017-12-21 16:36:37.173212] D [MSGID: 0] [client-rpc-fops.c:2941:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x7fe39ebdc42c, video-backup-client-4 returned -1 error: Stale file handle [Stale file handle] [2017-12-21 16:36:37.173233]...

...te-0/cache=<nul>) > > > > [2018-04-09 06:58:53.692440] W [MSGID: 114031] [client-rpc-fops.c:670:client3_3_rmdir_cbk] 0-myvol-private-client-2: remote operation failed [Directory not empty] > > > > [2018-04-09 06:58:53.714129] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-myvol-private-client-1: remote operation failed. Path: gfid:13880e8c-13da-442f-8180-fa40b6f5327c (13880e8c-13da-442f-8180-fa40b6f5327c) [No such file or directory] > > > > [2018-04-09 06:58:53.714161] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-myvol-private-cl...

...gt; > > > > [2018-04-09 06:58:53.692440] W [MSGID: 114031] [client-rpc-fops.c:670:client3_3_rmdir_cbk] 0-myvol-private-client-2: remote operation failed [Directory not empty] > > > > > > > > [2018-04-09 06:58:53.714129] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-myvol-private-client-1: remote operation failed. Path: gfid:13880e8c-13da-442f-8180-fa40b6f5327c (13880e8c-13da-442f-8180-fa40b6f5327c) [No such file or directory] > > > > > > > > [2018-04-09 06:58:53.714161] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk...