4 hours later, no files have been demoted
[root at Glus1 ~]# gluster volume status FFPrimary detail
Status of volume: FFPrimary
Hot Bricks:
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick3
TCP Port : 49155
RDMA Port : 0
Online : Y
Pid : 24177
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 24.3GB
Total Disk Space : 476.7GB
Inode Count : 50877088
Free Inodes : 50874696
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick2
TCP Port : 49155
RDMA Port : 0
Online : Y
Pid : 17994
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 15.5GB
Total Disk Space : 476.7GB
Inode Count : 32560288
Free Inodes : 32557896
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick1
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 23573
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 13.8GB
Total Disk Space : 476.7GB
Inode Count : 29027000
Free Inodes : 29024515
Cold Bricks:
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick5
TCP Port : 49152
RDMA Port : 0
Online : Y
Pid : 23442
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969488
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick6
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 17856
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969489
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick7
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 24018
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969488
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick8
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 23518
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969607
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick9
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 17943
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969607
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick10
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 24108
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969604
[root at Glus1 ~]# gluster volume info
Volume Name: FFPrimary
Type: Tier
Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: tcp
Hot Tier :
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 3 = 6
Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
Options Reconfigured:
cluster.tier-promote-frequency: 1800
cluster.tier-demote-frequency: 120
cluster.watermark-low: 60
cluster.watermark-hi: 80
performance.flush-behind: on
performance.cache-max-file-size: 128MB
performance.cache-size: 25GB
diagnostics.count-fop-hits: off
diagnostics.latency-measurement: off
cluster.tier-mode: cache
features.ctr-enabled: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
[root at Glus1 ~]# gluster volume tier FFPrimary status
Node Promoted files Demoted files Status
run time in h:m:s
--------- --------- --------- ---------
---------
localhost 49 0 in progress
5159:15:50
Glus2 0 0 in progress
5159:15:50
Glus3 0 2075 in progress
5159:15:52
Tiering Migration Functionality: FFPrimary: success
[root at Glus1 ~]#
On Sat, Sep 29, 2018 at 11:56 AM David Brown <dbccemtp at gmail.com>
wrote:
> Thank you Hari,
>
> I have set:
> cluster.tier-promote-frequency: 1800
> cluster.tier-demote-frequency: 120
>
> I will let you know if it makes a difference after some time. So far (10
> minutes), nothing has changed.
> I would agree with you, that by looking at the result of 'gluster
volume
> tier FFPrimary status' it would seem that demoting is happening.
However,
> for the last 24hrs, nothing has changed in the tier status report except
> the time. Could it be stuck? How would I know? Is there a way to restart it
> without restarting the cluster?
>
>
>
>
>
> On Sat, Sep 29, 2018 at 11:08 AM Hari Gowtham <hgowtham at
redhat.com> wrote:
>
>> Hi,
>>
>> I can see that the demotion is happening from the status provided by
you.
>> Do verify it.
>> I would recommend you to change the cluster.tier-demote-frequency to
120
>> and cluster.tier-promote-frequency to 1800 to increase the demotions
>> until the
>> hot tier is emptied to a certain extent. Later you can use the values
>> existing now.
>> On Sat, Sep 29, 2018 at 5:39 PM David Brown <dbccemtp at
gmail.com> wrote:
>> >
>> > Hey Everyone,
>> >
>> > I have a 3 node GlusterFS cluster that uses NVMe hot tier and a
HDD
>> cold tier.
>> > I recently ran into some problems when the hot tier became full
with
>> df-h showing 100%.
>> >
>> > I did not have a watermark-hi set, but it is my understanding that
90%
>> is the default. In an attempt to get the cluster to demote some files,
I
>> set cluster.watermark-hi: 80 but it is still not demoting.
>> >
>> >
>> > [root at Glus1 ~]# gluster volume info
>> >
>> > Volume Name: FFPrimary
>> > Type: Tier
>> > Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
>> > Status: Started
>> > Snapshot Count: 0
>> > Number of Bricks: 9
>> > Transport-type: tcp
>> > Hot Tier :
>> > Hot Tier Type : Replicate
>> > Number of Bricks: 1 x 3 = 3
>> > Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
>> > Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
>> > Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
>> > Cold Tier:
>> > Cold Tier Type : Distributed-Replicate
>> > Number of Bricks: 2 x 3 = 6
>> > Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
>> > Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
>> > Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
>> > Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
>> > Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
>> > Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
>> > Options Reconfigured:
>> > cluster.tier-promote-frequency: 120
>> > cluster.tier-demote-frequency: 1800
>> > cluster.watermark-low: 60
>> > cluster.watermark-hi: 80
>> > performance.flush-behind: on
>> > performance.cache-max-file-size: 128MB
>> > performance.cache-size: 25GB
>> > diagnostics.count-fop-hits: off
>> > diagnostics.latency-measurement: off
>> > cluster.tier-mode: cache
>> > features.ctr-enabled: on
>> > transport.address-family: inet
>> > nfs.disable: on
>> > performance.client-io-threads: off
>> > [root at Glus1 ~]# gluster volume tier FFPrimary status
>> > Node Promoted files Demoted files
Status
>> run time in h:m:s
>> > --------- --------- ---------
>> --------- ---------
>> > localhost 49 0 in
>> progress 5151:30:45
>> > Glus2 0 0 in progress
>> 5151:30:45
>> > Glus3 0 2075 in progress
>> 5151:30:47
>> > Tiering Migration Functionality: FFPrimary: success
>> > [root at Glus1 ~]#
>> >
>> > What can cause GlusterFS to stop demoting files and allow it to
>> completely fill the Hot Tier?
>> >
>> > Thank you!
>> >
>> >
>> >
>> > _______________________________________________
>> > Gluster-users mailing list
>> > Gluster-users at gluster.org
>> > https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>> --
>> Regards,
>> Hari Gowtham.
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180929/373776ec/attachment.html>
Just found this in the tierd.log Not sure what it means or how to fix it tho, but I assume it may be the cause of my problem with files not being demoted from the hot tier.. [2018-09-30 12:25:56.438821] E [MSGID: 114031] [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> [Transport endpoint is not connected] [2018-09-30 12:25:56.440940] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388 (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.444633] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.448347] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9 (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.451919] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test-march-3-bc-file-501 (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.456198] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //krishna (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.459702] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test1 (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.463164] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316 (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.466601] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.470129] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.473758] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20 (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.477237] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.480696] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135 (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.484354] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.487884] I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa) [2018-09-30 12:25:56.490973] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.494618] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8 (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.498059] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.501428] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //DBTest (c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:56.504736] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //testCliff (cac8d0c8-2a10-4fa1-8b94-bf47981c01ff). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] The message "I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for 6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)" repeated 4 times between [2018-09-30 12:25:53.164873] and [2018-09-30 12:25:57.515127] [2018-09-30 12:25:57.520933] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory] [2018-09-30 12:25:57.521545] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected] [2018-09-30 12:25:57.521854] E [MSGID: 114031] [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> [Transport endpoint is not connected] [2018-09-30 12:25:57.524079] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388 (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.527993] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.531829] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9 (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.535529] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test-march-3-bc-file-501 (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.539852] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //krishna (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.543468] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test1 (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.547051] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316 (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.550694] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.554498] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.558086] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20 (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.561568] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.565094] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135 (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.568562] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.571974] I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa) [2018-09-30 12:25:57.574925] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.578639] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8 (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.582118] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:57.585547] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //DBTest (c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] The message "E [MSGID: 109037] [tier.c:2532:tier_run] 0-FFPrimary-tier-dht: Demotion failed" repeated 3 times between [2018-09-30 12:25:54.323716] and [2018-09-30 12:25:57.592361] [2018-09-30 12:25:58.002724] I [MSGID: 0] [dht-rebalance.c:4194:gf_defrag_total_file_size] 0-FFPrimary-tier-dht: local subvol: FFPrimary-cold-dht,cnt = 272288612352 [2018-09-30 12:25:58.018840] I [MSGID: 0] [dht-rebalance.c:4198:gf_defrag_total_file_size] 0-FFPrimary-tier-dht: Total size files = 272288612352 [2018-09-30 12:25:58.018854] I [dht-rebalance.c:4325:dht_file_counter_thread] 0-dht: tmp data size =272288612352 The message "E [MSGID: 109037] [tier.c:720:tier_lookup_file] 0-FFPrimary-tier-dht: Failed to lookup file 6bed3d1e-17b8-11e8-a5dc-902b3450f388 [No such file or directory]" repeated 4 times between [2018-09-30 12:25:54.247352] and [2018-09-30 12:25:58.599151] [2018-09-30 12:25:58.599162] I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for 6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d) [2018-09-30 12:25:58.604808] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory] [2018-09-30 12:25:58.605428] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected] [2018-09-30 12:25:58.605750] E [MSGID: 114031] [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> [Transport endpoint is not connected] [2018-09-30 12:25:58.607865] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388 (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.611433] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.615059] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9 (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.618669] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test-march-3-bc-file-501 (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.622667] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //krishna (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.626029] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test1 (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.629459] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316 (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.632994] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.636669] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.640155] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20 (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.643516] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.648787] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135 (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.652106] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.655577] I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa) [2018-09-30 12:25:58.658482] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.661703] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8 (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] [2018-09-30 12:25:58.665051] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected] The message "W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //e8a6aba1-e5ce-4ded-b474-1c5bf49b1285 (edffea55-296b-4d1b-8114-b4f8dd10920a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]" repeated 3 times between [2018-09-30 12:25:55.406782] and [2018-09-30 12:25:58.674842] [2018-09-30 12:25:58.675351] E [MSGID: 109037] [tier.c:2532:tier_run] 0-FFPrimary-tier-dht: Demotion failed [root at Glus1 FFPrimary]# On Sat, Sep 29, 2018 at 3:54 PM David Brown <dbccemtp at gmail.com> wrote:> 4 hours later, no files have been demoted > > [root at Glus1 ~]# gluster volume status FFPrimary detail > Status of volume: FFPrimary > Hot Bricks: > > ------------------------------------------------------------------------------ > Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick3 > TCP Port : 49155 > RDMA Port : 0 > Online : Y > Pid : 24177 > File System : xfs > Device : /dev/nvme0n1 > Mount Options : rw,seclabel,relatime,attr2,inode64,noquota > Inode Size : 512 > Disk Space Free : 24.3GB > Total Disk Space : 476.7GB > Inode Count : 50877088 > Free Inodes : 50874696 > > ------------------------------------------------------------------------------ > Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick2 > TCP Port : 49155 > RDMA Port : 0 > Online : Y > Pid : 17994 > File System : xfs > Device : /dev/nvme0n1 > Mount Options : rw,seclabel,relatime,attr2,inode64,noquota > Inode Size : 512 > Disk Space Free : 15.5GB > Total Disk Space : 476.7GB > Inode Count : 32560288 > Free Inodes : 32557896 > > ------------------------------------------------------------------------------ > Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick1 > TCP Port : 49154 > RDMA Port : 0 > Online : Y > Pid : 23573 > File System : xfs > Device : /dev/nvme0n1 > Mount Options : rw,seclabel,relatime,attr2,inode64,noquota > Inode Size : 512 > Disk Space Free : 13.8GB > Total Disk Space : 476.7GB > Inode Count : 29027000 > Free Inodes : 29024515 > Cold Bricks: > > ------------------------------------------------------------------------------ > Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick5 > TCP Port : 49152 > RDMA Port : 0 > Online : Y > Pid : 23442 > File System : xfs > Device : /dev/sdb1 > Mount Options : rw,seclabel,relatime,attr2,inode64,noquota > Inode Size : 512 > Disk Space Free : 2.5TB > Total Disk Space : 2.7TB > Inode Count : 292971904 > Free Inodes : 292969488 > > ------------------------------------------------------------------------------ > Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick6 > TCP Port : 49153 > RDMA Port : 0 > Online : Y > Pid : 17856 > File System : xfs > Device : /dev/sdb1 > Mount Options : rw,seclabel,relatime,attr2,inode64,noquota > Inode Size : 512 > Disk Space Free : 2.5TB > Total Disk Space : 2.7TB > Inode Count : 292971904 > Free Inodes : 292969489 > > ------------------------------------------------------------------------------ > Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick7 > TCP Port : 49153 > RDMA Port : 0 > Online : Y > Pid : 24018 > File System : xfs > Device : /dev/sdb1 > Mount Options : rw,seclabel,relatime,attr2,inode64,noquota > Inode Size : 512 > Disk Space Free : 2.5TB > Total Disk Space : 2.7TB > Inode Count : 292971904 > Free Inodes : 292969488 > > ------------------------------------------------------------------------------ > Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick8 > TCP Port : 49153 > RDMA Port : 0 > Online : Y > Pid : 23518 > File System : xfs > Device : /dev/sdc1 > Mount Options : rw,seclabel,relatime,attr2,inode64,noquota > Inode Size : 512 > Disk Space Free : 2.7TB > Total Disk Space : 2.7TB > Inode Count : 292971904 > Free Inodes : 292969607 > > ------------------------------------------------------------------------------ > Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick9 > TCP Port : 49154 > RDMA Port : 0 > Online : Y > Pid : 17943 > File System : xfs > Device : /dev/sdc1 > Mount Options : rw,seclabel,relatime,attr2,inode64,noquota > Inode Size : 512 > Disk Space Free : 2.7TB > Total Disk Space : 2.7TB > Inode Count : 292971904 > Free Inodes : 292969607 > > ------------------------------------------------------------------------------ > Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick10 > TCP Port : 49154 > RDMA Port : 0 > Online : Y > Pid : 24108 > File System : xfs > Device : /dev/sdc1 > Mount Options : rw,seclabel,relatime,attr2,inode64,noquota > Inode Size : 512 > Disk Space Free : 2.7TB > Total Disk Space : 2.7TB > Inode Count : 292971904 > Free Inodes : 292969604 > > [root at Glus1 ~]# gluster volume info > > Volume Name: FFPrimary > Type: Tier > Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e > Status: Started > Snapshot Count: 0 > Number of Bricks: 9 > Transport-type: tcp > Hot Tier : > Hot Tier Type : Replicate > Number of Bricks: 1 x 3 = 3 > Brick1: Glus3:/data/glusterfs/FFPrimary/brick3 > Brick2: Glus2:/data/glusterfs/FFPrimary/brick2 > Brick3: Glus1:/data/glusterfs/FFPrimary/brick1 > Cold Tier: > Cold Tier Type : Distributed-Replicate > Number of Bricks: 2 x 3 = 6 > Brick4: Glus1:/data/glusterfs/FFPrimary/brick5 > Brick5: Glus2:/data/glusterfs/FFPrimary/brick6 > Brick6: Glus3:/data/glusterfs/FFPrimary/brick7 > Brick7: Glus1:/data/glusterfs/FFPrimary/brick8 > Brick8: Glus2:/data/glusterfs/FFPrimary/brick9 > Brick9: Glus3:/data/glusterfs/FFPrimary/brick10 > Options Reconfigured: > cluster.tier-promote-frequency: 1800 > cluster.tier-demote-frequency: 120 > cluster.watermark-low: 60 > cluster.watermark-hi: 80 > performance.flush-behind: on > performance.cache-max-file-size: 128MB > performance.cache-size: 25GB > diagnostics.count-fop-hits: off > diagnostics.latency-measurement: off > cluster.tier-mode: cache > features.ctr-enabled: on > transport.address-family: inet > nfs.disable: on > performance.client-io-threads: off > > [root at Glus1 ~]# gluster volume tier FFPrimary status > Node Promoted files Demoted files Status > run time in h:m:s > --------- --------- --------- --------- > --------- > localhost 49 0 in > progress 5159:15:50 > Glus2 0 0 in progress > 5159:15:50 > Glus3 0 2075 in progress > 5159:15:52 > Tiering Migration Functionality: FFPrimary: success > [root at Glus1 ~]# > > > > On Sat, Sep 29, 2018 at 11:56 AM David Brown <dbccemtp at gmail.com> wrote: > >> Thank you Hari, >> >> I have set: >> cluster.tier-promote-frequency: 1800 >> cluster.tier-demote-frequency: 120 >> >> I will let you know if it makes a difference after some time. So far (10 >> minutes), nothing has changed. >> I would agree with you, that by looking at the result of 'gluster volume >> tier FFPrimary status' it would seem that demoting is happening. However, >> for the last 24hrs, nothing has changed in the tier status report except >> the time. Could it be stuck? How would I know? Is there a way to restart it >> without restarting the cluster? >> >> >> >> >> >> On Sat, Sep 29, 2018 at 11:08 AM Hari Gowtham <hgowtham at redhat.com> >> wrote: >> >>> Hi, >>> >>> I can see that the demotion is happening from the status provided by you. >>> Do verify it. >>> I would recommend you to change the cluster.tier-demote-frequency to 120 >>> and cluster.tier-promote-frequency to 1800 to increase the demotions >>> until the >>> hot tier is emptied to a certain extent. Later you can use the values >>> existing now. >>> On Sat, Sep 29, 2018 at 5:39 PM David Brown <dbccemtp at gmail.com> wrote: >>> > >>> > Hey Everyone, >>> > >>> > I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD >>> cold tier. >>> > I recently ran into some problems when the hot tier became full with >>> df-h showing 100%. >>> > >>> > I did not have a watermark-hi set, but it is my understanding that 90% >>> is the default. In an attempt to get the cluster to demote some files, I >>> set cluster.watermark-hi: 80 but it is still not demoting. >>> > >>> > >>> > [root at Glus1 ~]# gluster volume info >>> > >>> > Volume Name: FFPrimary >>> > Type: Tier >>> > Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e >>> > Status: Started >>> > Snapshot Count: 0 >>> > Number of Bricks: 9 >>> > Transport-type: tcp >>> > Hot Tier : >>> > Hot Tier Type : Replicate >>> > Number of Bricks: 1 x 3 = 3 >>> > Brick1: Glus3:/data/glusterfs/FFPrimary/brick3 >>> > Brick2: Glus2:/data/glusterfs/FFPrimary/brick2 >>> > Brick3: Glus1:/data/glusterfs/FFPrimary/brick1 >>> > Cold Tier: >>> > Cold Tier Type : Distributed-Replicate >>> > Number of Bricks: 2 x 3 = 6 >>> > Brick4: Glus1:/data/glusterfs/FFPrimary/brick5 >>> > Brick5: Glus2:/data/glusterfs/FFPrimary/brick6 >>> > Brick6: Glus3:/data/glusterfs/FFPrimary/brick7 >>> > Brick7: Glus1:/data/glusterfs/FFPrimary/brick8 >>> > Brick8: Glus2:/data/glusterfs/FFPrimary/brick9 >>> > Brick9: Glus3:/data/glusterfs/FFPrimary/brick10 >>> > Options Reconfigured: >>> > cluster.tier-promote-frequency: 120 >>> > cluster.tier-demote-frequency: 1800 >>> > cluster.watermark-low: 60 >>> > cluster.watermark-hi: 80 >>> > performance.flush-behind: on >>> > performance.cache-max-file-size: 128MB >>> > performance.cache-size: 25GB >>> > diagnostics.count-fop-hits: off >>> > diagnostics.latency-measurement: off >>> > cluster.tier-mode: cache >>> > features.ctr-enabled: on >>> > transport.address-family: inet >>> > nfs.disable: on >>> > performance.client-io-threads: off >>> > [root at Glus1 ~]# gluster volume tier FFPrimary status >>> > Node Promoted files Demoted files Status >>> run time in h:m:s >>> > --------- --------- --------- >>> --------- --------- >>> > localhost 49 0 in >>> progress 5151:30:45 >>> > Glus2 0 0 in progress >>> 5151:30:45 >>> > Glus3 0 2075 in progress >>> 5151:30:47 >>> > Tiering Migration Functionality: FFPrimary: success >>> > [root at Glus1 ~]# >>> > >>> > What can cause GlusterFS to stop demoting files and allow it to >>> completely fill the Hot Tier? >>> > >>> > Thank you! >>> > >>> > >>> > >>> > _______________________________________________ >>> > Gluster-users mailing list >>> > Gluster-users at gluster.org >>> > https://lists.gluster.org/mailman/listinfo/gluster-users >>> >>> >>> >>> -- >>> Regards, >>> Hari Gowtham. >>> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180930/bc279410/attachment.html>
Hi David, Just for your information, as a project, we are not currently taking any development focus on Tiering feature! Please refer to email thread @ https://lists.gluster.org/pipermail//gluster-devel/2018-July/055017.html It is recommended to use 'dmcache' on your disks to get best performance out of your backend instead! Also note that this may get retired in next upcoming release: https://review.gluster.org/21331 Hope this email will save you lot of time! Regards, Amar On Sun, Sep 30, 2018 at 6:04 PM David Brown <dbccemtp at gmail.com> wrote:> Just found this in the tierd.log Not sure what it means or how to fix it > tho, but I assume it may be the cause of my problem with files not being > demoted from the hot tier.. > > > [2018-09-30 12:25:56.438821] E [MSGID: 114031] > [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote > operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> > [Transport endpoint is not connected] > [2018-09-30 12:25:56.440940] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388 > (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.444633] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e > (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.448347] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9 > (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.451919] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //test-march-3-bc-file-501 > (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.456198] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //krishna > (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.459702] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //test1 > (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.463164] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316 > (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.466601] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a > (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.470129] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a > (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.473758] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20 > (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.477237] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb > (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.480696] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135 > (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.484354] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d > (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.487884] I [MSGID: 109038] > [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion > failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa) > [2018-09-30 12:25:56.490973] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b > (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.494618] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8 > (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.498059] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b > (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.501428] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //DBTest > (c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:56.504736] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //testCliff > (cac8d0c8-2a10-4fa1-8b94-bf47981c01ff). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > The message "I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] > 0-FFPrimary-tier-dht: Demotion failed for > 6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)" > repeated 4 times between [2018-09-30 12:25:53.164873] and [2018-09-30 > 12:25:57.515127] > [2018-09-30 12:25:57.520933] W [MSGID: 114031] > [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote > operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> > (10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory] > [2018-09-30 12:25:57.521545] W [MSGID: 114031] > [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote > operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> > (10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected] > [2018-09-30 12:25:57.521854] E [MSGID: 114031] > [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote > operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> > [Transport endpoint is not connected] > [2018-09-30 12:25:57.524079] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388 > (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.527993] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e > (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.531829] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9 > (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.535529] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //test-march-3-bc-file-501 > (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.539852] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //krishna > (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.543468] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //test1 > (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.547051] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316 > (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.550694] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a > (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.554498] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a > (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.558086] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20 > (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.561568] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb > (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.565094] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135 > (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.568562] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d > (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.571974] I [MSGID: 109038] > [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion > failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa) > [2018-09-30 12:25:57.574925] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b > (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.578639] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8 > (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.582118] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b > (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:57.585547] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //DBTest > (c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > The message "E [MSGID: 109037] [tier.c:2532:tier_run] > 0-FFPrimary-tier-dht: Demotion failed" repeated 3 times between [2018-09-30 > 12:25:54.323716] and [2018-09-30 12:25:57.592361] > [2018-09-30 12:25:58.002724] I [MSGID: 0] > [dht-rebalance.c:4194:gf_defrag_total_file_size] 0-FFPrimary-tier-dht: > local subvol: FFPrimary-cold-dht,cnt = 272288612352 > [2018-09-30 12:25:58.018840] I [MSGID: 0] > [dht-rebalance.c:4198:gf_defrag_total_file_size] 0-FFPrimary-tier-dht: > Total size files = 272288612352 > [2018-09-30 12:25:58.018854] I > [dht-rebalance.c:4325:dht_file_counter_thread] 0-dht: tmp data size > =272288612352 > The message "E [MSGID: 109037] [tier.c:720:tier_lookup_file] > 0-FFPrimary-tier-dht: Failed to lookup file > 6bed3d1e-17b8-11e8-a5dc-902b3450f388 > [No such file or directory]" repeated 4 times between [2018-09-30 > 12:25:54.247352] and [2018-09-30 12:25:58.599151] > [2018-09-30 12:25:58.599162] I [MSGID: 109038] > [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion > failed for > 6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d) > [2018-09-30 12:25:58.604808] W [MSGID: 114031] > [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote > operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> > (10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory] > [2018-09-30 12:25:58.605428] W [MSGID: 114031] > [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote > operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> > (10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected] > [2018-09-30 12:25:58.605750] E [MSGID: 114031] > [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote > operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> > [Transport endpoint is not connected] > [2018-09-30 12:25:58.607865] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388 > (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.611433] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e > (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.615059] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9 > (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.618669] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //test-march-3-bc-file-501 > (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.622667] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //krishna > (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.626029] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //test1 > (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.629459] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316 > (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.632994] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a > (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.636669] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a > (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.640155] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20 > (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.643516] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb > (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.648787] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135 > (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.652106] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d > (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.655577] I [MSGID: 109038] > [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion > failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa) > [2018-09-30 12:25:58.658482] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b > (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.661703] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8 > (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > [2018-09-30 12:25:58.665051] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b > (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected] > The message "W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: > remote operation failed. Path: //e8a6aba1-e5ce-4ded-b474-1c5bf49b1285 > (edffea55-296b-4d1b-8114-b4f8dd10920a). Key: trusted.glusterfs.node-uuid > [Transport endpoint is not connected]" repeated 3 times between [2018-09-30 > 12:25:55.406782] and [2018-09-30 12:25:58.674842] > [2018-09-30 12:25:58.675351] E [MSGID: 109037] [tier.c:2532:tier_run] > 0-FFPrimary-tier-dht: Demotion failed > [root at Glus1 FFPrimary]# > > On Sat, Sep 29, 2018 at 3:54 PM David Brown <dbccemtp at gmail.com> wrote: > >> 4 hours later, no files have been demoted >> >> [root at Glus1 ~]# gluster volume status FFPrimary detail >> Status of volume: FFPrimary >> Hot Bricks: >> >> ------------------------------------------------------------------------------ >> Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick3 >> TCP Port : 49155 >> RDMA Port : 0 >> Online : Y >> Pid : 24177 >> File System : xfs >> Device : /dev/nvme0n1 >> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota >> Inode Size : 512 >> Disk Space Free : 24.3GB >> Total Disk Space : 476.7GB >> Inode Count : 50877088 >> Free Inodes : 50874696 >> >> ------------------------------------------------------------------------------ >> Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick2 >> TCP Port : 49155 >> RDMA Port : 0 >> Online : Y >> Pid : 17994 >> File System : xfs >> Device : /dev/nvme0n1 >> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota >> Inode Size : 512 >> Disk Space Free : 15.5GB >> Total Disk Space : 476.7GB >> Inode Count : 32560288 >> Free Inodes : 32557896 >> >> ------------------------------------------------------------------------------ >> Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick1 >> TCP Port : 49154 >> RDMA Port : 0 >> Online : Y >> Pid : 23573 >> File System : xfs >> Device : /dev/nvme0n1 >> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota >> Inode Size : 512 >> Disk Space Free : 13.8GB >> Total Disk Space : 476.7GB >> Inode Count : 29027000 >> Free Inodes : 29024515 >> Cold Bricks: >> >> ------------------------------------------------------------------------------ >> Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick5 >> TCP Port : 49152 >> RDMA Port : 0 >> Online : Y >> Pid : 23442 >> File System : xfs >> Device : /dev/sdb1 >> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota >> Inode Size : 512 >> Disk Space Free : 2.5TB >> Total Disk Space : 2.7TB >> Inode Count : 292971904 >> Free Inodes : 292969488 >> >> ------------------------------------------------------------------------------ >> Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick6 >> TCP Port : 49153 >> RDMA Port : 0 >> Online : Y >> Pid : 17856 >> File System : xfs >> Device : /dev/sdb1 >> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota >> Inode Size : 512 >> Disk Space Free : 2.5TB >> Total Disk Space : 2.7TB >> Inode Count : 292971904 >> Free Inodes : 292969489 >> >> ------------------------------------------------------------------------------ >> Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick7 >> TCP Port : 49153 >> RDMA Port : 0 >> Online : Y >> Pid : 24018 >> File System : xfs >> Device : /dev/sdb1 >> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota >> Inode Size : 512 >> Disk Space Free : 2.5TB >> Total Disk Space : 2.7TB >> Inode Count : 292971904 >> Free Inodes : 292969488 >> >> ------------------------------------------------------------------------------ >> Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick8 >> TCP Port : 49153 >> RDMA Port : 0 >> Online : Y >> Pid : 23518 >> File System : xfs >> Device : /dev/sdc1 >> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota >> Inode Size : 512 >> Disk Space Free : 2.7TB >> Total Disk Space : 2.7TB >> Inode Count : 292971904 >> Free Inodes : 292969607 >> >> ------------------------------------------------------------------------------ >> Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick9 >> TCP Port : 49154 >> RDMA Port : 0 >> Online : Y >> Pid : 17943 >> File System : xfs >> Device : /dev/sdc1 >> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota >> Inode Size : 512 >> Disk Space Free : 2.7TB >> Total Disk Space : 2.7TB >> Inode Count : 292971904 >> Free Inodes : 292969607 >> >> ------------------------------------------------------------------------------ >> Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick10 >> TCP Port : 49154 >> RDMA Port : 0 >> Online : Y >> Pid : 24108 >> File System : xfs >> Device : /dev/sdc1 >> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota >> Inode Size : 512 >> Disk Space Free : 2.7TB >> Total Disk Space : 2.7TB >> Inode Count : 292971904 >> Free Inodes : 292969604 >> >> [root at Glus1 ~]# gluster volume info >> >> Volume Name: FFPrimary >> Type: Tier >> Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 9 >> Transport-type: tcp >> Hot Tier : >> Hot Tier Type : Replicate >> Number of Bricks: 1 x 3 = 3 >> Brick1: Glus3:/data/glusterfs/FFPrimary/brick3 >> Brick2: Glus2:/data/glusterfs/FFPrimary/brick2 >> Brick3: Glus1:/data/glusterfs/FFPrimary/brick1 >> Cold Tier: >> Cold Tier Type : Distributed-Replicate >> Number of Bricks: 2 x 3 = 6 >> Brick4: Glus1:/data/glusterfs/FFPrimary/brick5 >> Brick5: Glus2:/data/glusterfs/FFPrimary/brick6 >> Brick6: Glus3:/data/glusterfs/FFPrimary/brick7 >> Brick7: Glus1:/data/glusterfs/FFPrimary/brick8 >> Brick8: Glus2:/data/glusterfs/FFPrimary/brick9 >> Brick9: Glus3:/data/glusterfs/FFPrimary/brick10 >> Options Reconfigured: >> cluster.tier-promote-frequency: 1800 >> cluster.tier-demote-frequency: 120 >> cluster.watermark-low: 60 >> cluster.watermark-hi: 80 >> performance.flush-behind: on >> performance.cache-max-file-size: 128MB >> performance.cache-size: 25GB >> diagnostics.count-fop-hits: off >> diagnostics.latency-measurement: off >> cluster.tier-mode: cache >> features.ctr-enabled: on >> transport.address-family: inet >> nfs.disable: on >> performance.client-io-threads: off >> >> [root at Glus1 ~]# gluster volume tier FFPrimary status >> Node Promoted files Demoted files Status >> run time in h:m:s >> --------- --------- --------- --------- >> --------- >> localhost 49 0 in >> progress 5159:15:50 >> Glus2 0 0 in progress >> 5159:15:50 >> Glus3 0 2075 in progress >> 5159:15:52 >> Tiering Migration Functionality: FFPrimary: success >> [root at Glus1 ~]# >> >> >> >> On Sat, Sep 29, 2018 at 11:56 AM David Brown <dbccemtp at gmail.com> wrote: >> >>> Thank you Hari, >>> >>> I have set: >>> cluster.tier-promote-frequency: 1800 >>> cluster.tier-demote-frequency: 120 >>> >>> I will let you know if it makes a difference after some time. So far (10 >>> minutes), nothing has changed. >>> I would agree with you, that by looking at the result of 'gluster volume >>> tier FFPrimary status' it would seem that demoting is happening. However, >>> for the last 24hrs, nothing has changed in the tier status report except >>> the time. Could it be stuck? How would I know? Is there a way to restart it >>> without restarting the cluster? >>> >>> >>> >>> >>> >>> On Sat, Sep 29, 2018 at 11:08 AM Hari Gowtham <hgowtham at redhat.com> >>> wrote: >>> >>>> Hi, >>>> >>>> I can see that the demotion is happening from the status provided by >>>> you. >>>> Do verify it. >>>> I would recommend you to change the cluster.tier-demote-frequency to 120 >>>> and cluster.tier-promote-frequency to 1800 to increase the demotions >>>> until the >>>> hot tier is emptied to a certain extent. Later you can use the values >>>> existing now. >>>> On Sat, Sep 29, 2018 at 5:39 PM David Brown <dbccemtp at gmail.com> wrote: >>>> > >>>> > Hey Everyone, >>>> > >>>> > I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD >>>> cold tier. >>>> > I recently ran into some problems when the hot tier became full with >>>> df-h showing 100%. >>>> > >>>> > I did not have a watermark-hi set, but it is my understanding that >>>> 90% is the default. In an attempt to get the cluster to demote some files, >>>> I set cluster.watermark-hi: 80 but it is still not demoting. >>>> > >>>> > >>>> > [root at Glus1 ~]# gluster volume info >>>> > >>>> > Volume Name: FFPrimary >>>> > Type: Tier >>>> > Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e >>>> > Status: Started >>>> > Snapshot Count: 0 >>>> > Number of Bricks: 9 >>>> > Transport-type: tcp >>>> > Hot Tier : >>>> > Hot Tier Type : Replicate >>>> > Number of Bricks: 1 x 3 = 3 >>>> > Brick1: Glus3:/data/glusterfs/FFPrimary/brick3 >>>> > Brick2: Glus2:/data/glusterfs/FFPrimary/brick2 >>>> > Brick3: Glus1:/data/glusterfs/FFPrimary/brick1 >>>> > Cold Tier: >>>> > Cold Tier Type : Distributed-Replicate >>>> > Number of Bricks: 2 x 3 = 6 >>>> > Brick4: Glus1:/data/glusterfs/FFPrimary/brick5 >>>> > Brick5: Glus2:/data/glusterfs/FFPrimary/brick6 >>>> > Brick6: Glus3:/data/glusterfs/FFPrimary/brick7 >>>> > Brick7: Glus1:/data/glusterfs/FFPrimary/brick8 >>>> > Brick8: Glus2:/data/glusterfs/FFPrimary/brick9 >>>> > Brick9: Glus3:/data/glusterfs/FFPrimary/brick10 >>>> > Options Reconfigured: >>>> > cluster.tier-promote-frequency: 120 >>>> > cluster.tier-demote-frequency: 1800 >>>> > cluster.watermark-low: 60 >>>> > cluster.watermark-hi: 80 >>>> > performance.flush-behind: on >>>> > performance.cache-max-file-size: 128MB >>>> > performance.cache-size: 25GB >>>> > diagnostics.count-fop-hits: off >>>> > diagnostics.latency-measurement: off >>>> > cluster.tier-mode: cache >>>> > features.ctr-enabled: on >>>> > transport.address-family: inet >>>> > nfs.disable: on >>>> > performance.client-io-threads: off >>>> > [root at Glus1 ~]# gluster volume tier FFPrimary status >>>> > Node Promoted files Demoted files >>>> Status run time in h:m:s >>>> > --------- --------- --------- >>>> --------- --------- >>>> > localhost 49 0 in >>>> progress 5151:30:45 >>>> > Glus2 0 0 in progress >>>> 5151:30:45 >>>> > Glus3 0 2075 in progress >>>> 5151:30:47 >>>> > Tiering Migration Functionality: FFPrimary: success >>>> > [root at Glus1 ~]# >>>> > >>>> > What can cause GlusterFS to stop demoting files and allow it to >>>> completely fill the Hot Tier? >>>> > >>>> > Thank you! >>>> > >>>> > >>>> > >>>> > _______________________________________________ >>>> > Gluster-users mailing list >>>> > Gluster-users at gluster.org >>>> > https://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Hari Gowtham. >>>> >>> _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181009/ddc96d3e/attachment.html>