hi everyone. I see this: $ gluster volume heal QEMU_VMs info? Brick swir-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.QEMU_VMs Status: Connected Number of entries: 0 ? Brick rider-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.QEMU_VMs <gfid:a53e6a37-7f65-4c73-b9b4-baa87817a1b0>/default.sock? <gfid:e0121f76-2452-44dc-b1a6-82b46cc9ec79>/private/sbus-dp_disk.812? <gfid:d1d2e7c1-f7aa-4f32-b9d2-ab1a3a2e88e6>? <gfid:e0121f76-2452-44dc-b1a6-82b46cc9ec79>/pam? <gfid:d3e2716d-7205-4bb1-ab3f-24fbc3c253d8>? <gfid:ddda7488-2034-4861-a0c1-87fff25fd039>? <gfid:6a6dc874-6e8c-4db4-8a6c-8fb33188ba01>? <gfid:e75ce03d-e6eb-480d-9514-e530185b65a2>? <gfid:a53e6a37-7f65-4c73-b9b4-baa87817a1b0>? <gfid:e0121f76-2452-44dc-b1a6-82b46cc9ec79>? <gfid:e0121f76-2452-44dc-b1a6-82b46cc9ec79>/private? Status: Connected Number of entries: 11 ? Brick whale-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.QEMU_VMs Status: Connected Number of entries: 0 And when I start this volume, in log on the brick which shows gfids: ... [2020-01-06 14:28:24.119506] E [MSGID: 114031] [client-rpc-fops_v2.c:150:client4_0_mknod_cbk] 0-QEMU_VMs-client-5: remote operation failed. Path: <gfid:3f0239ac-e027-4a0c-b271-431e76ad97b1> [Permission denied] [2020-01-06 14:38:25.047537] I [MSGID: 108026] [afr-self-heal-entry.c:898:afr_selfheal_entry_do] 0-QEMU_VMs-replicate-0: performing entry selfheal on a53e6a37-7f65-4c73-b9b4-baa87817a1b0 [2020-01-06 14:38:25.061063] E [MSGID: 114031] [client-rpc-fops_v2.c:150:client4_0_mknod_cbk] 0-QEMU_VMs-client-3: remote operation failed. Path: <gfid:c1c4431d-5671-4383-9d11-9ed4b80beeff> [Permission denied] [2020-01-06 14:38:25.065119] E [MSGID: 114031] [client-rpc-fops_v2.c:150:client4_0_mknod_cbk] 0-QEMU_VMs-client-5: remote operation failed. Path: <gfid:c1c4431d-5671-4383-9d11-9ed4b80beeff> [Permission denied] [2020-01-06 14:38:25.075853] I [MSGID: 108026] [afr-self-heal-entry.c:898:afr_selfheal_entry_do] 0-QEMU_VMs-replicate-0: performing entry selfheal on e0121f76-2452-44dc-b1a6-82b46cc9ec79 [2020-01-06 14:38:25.100700] E [MSGID: 114031] [client-rpc-fops_v2.c:150:client4_0_mknod_cbk] 0-QEMU_VMs-client-3: remote operation failed. Path: <gfid:3f0239ac-e027-4a0c-b271-431e76ad97b1> [Permission denied] [2020-01-06 14:38:25.104633] E [MSGID: 114031] [client-rpc-fops_v2.c:150:client4_0_mknod_cbk] 0-QEMU_VMs-client-5: remote operation failed. Path: <gfid:3f0239ac-e027-4a0c-b271-431e76ad97b1> [Permission denied] [2020-01-06 14:38:25.114980] I [MSGID: 108026] [afr-self-heal-entry.c:898:afr_selfheal_entry_do] 0-QEMU_VMs-replicate-0: performing entry selfheal on 85a05801-547f-4f15-b2ff-e105049c445a [2020-01-06 14:38:25.127755] E [MSGID: 114031] [client-rpc-fops_v2.c:150:client4_0_mknod_cbk] 0-QEMU_VMs-client-3: remote operation failed. Path: <gfid:6f01098f-e8db-4f63-a661-86b4d02d937f> [Permission denied] [2020-01-06 14:38:25.131402] E [MSGID: 114031] [client-rpc-fops_v2.c:150:client4_0_mknod_cbk] 0-QEMU_VMs-client-5: remote operation failed. Path: <gfid:6f01098f-e8db-4f63-a661-86b4d02d937f> [Permission denied] ? ... Any suggestions as to how troubleshoot & fix are greatly appreciated. many thanks, L. -------------- next part -------------- A non-text attachment was scrubbed... Name: pEpkey.asc Type: application/pgp-keys Size: 1757 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200106/723618eb/attachment.bin>
On 06/01/20 8:12 pm, lejeczek wrote:> And when I start this volume, in log on the brick which shows gfids:I assume these messages are from the self-heal daemon's log (glustershd.log). Correct me if I am mistaken.> > ... > > [2020-01-06 14:28:24.119506] E [MSGID: 114031] > [client-rpc-fops_v2.c:150:client4_0_mknod_cbk] 0-QEMU_VMs-client-5: > remote operation failed. Path: > <gfid:3f0239ac-e027-4a0c-b271-431e76ad97b1> [Permission denied]Can you provide the following information: 1. gluster version 2. gluster volume info $volname 3. `getfattr -d -m . -e hex`? and `stat` outputs of any one gfid/file for which you see the EACCES from /all the bricks/ of the replica? 4. Do you see errors in the brick log as well? Does server4_mknod_cbk()? also complain of EACCES errors? 5. If yes, can you attach gdb to the brick process, put a break point in server4_mknod_cbk() and provide the function backtrace once it is hit? Thanks, Ravi -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200107/6566aa7e/attachment.html>