Matthew Benstead
2019-Jul-26 21:01 UTC
[Gluster-users] Brick missing trusted.glusterfs.dht xattr
Ok thank-you for explaining everything - that makes sense. Currently the brick file systems are pretty evenly distributed so I probably won't run the fix-layout right now. Would this state have any impact on geo-replication? I'm trying to geo-replicate this volume, but am getting a weird error: "Changelog register failed error=[Errno 21] Is a directory" I assume this is related to something else, but I wasn't sure. Thanks, ?-Matthew -- Matthew Benstead System Administrator Pacific Climate Impacts Consortium <https://pacificclimate.org/> University of Victoria, UH1 PO Box 1800, STN CSC Victoria, BC, V8W 2Y2 Phone: +1-250-721-8432 Email: matthewb at uvic.ca On 7/26/19 12:02 AM, Nithya Balachandran wrote:> > > On Fri, 26 Jul 2019 at 01:56, Matthew Benstead <matthewb at uvic.ca > <mailto:matthewb at uvic.ca>> wrote: > > Hi Nithya, > > Hmm... I don't remember if I did, but based on what I'm seeing it > sounds like I probably didn't run rebalance or fix-layout. > > It looks like folders that haven't had any new files created have > a dht of 0, while other folders have non-zero values. > > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/ | grep dht > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/home | grep dht > trusted.glusterfs.dht=0x00000000000000000000000000000000 > [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/home/matthewb | grep dht > trusted.glusterfs.dht=0x00000001000000004924921a6db6dbc7 > > If I just run the fix-layout command will it re-create all of the > dht values or just the missing ones? > > > A fix-layout will recalculate the layouts entirely so files all the > values will change. No files will be moved. > A rebalance will recalculate the layouts like the fix-layout but will > also move files to their new locations based on the new layout ranges. > This could take a lot of time depending on the number of > files/directories on the volume. If you do this, I would recommend > that you turn off lookup-optimize until the rebalance is over. > ? > > Since the brick is already fairly size balanced could I get away > with running fix-layout but not rebalance? Or would the new dht > layout mean slower accesses since the files may be expected on > different bricks? > > ? > The first access for a file will be slower. The next one will be > faster as the location will be cached in the client's in-memory > structures. > You may not need to run either a fix-layout or a rebalance if new file > creations will be in directories created after the add-brick. Gluster > will automatically include all 7 bricks for those directories. > > Regards, > Nithya > > > Thanks, > ?-Matthew > > -- > Matthew Benstead > System Administrator > Pacific Climate Impacts Consortium <https://pacificclimate.org/> > University of Victoria, UH1 > PO Box 1800, STN CSC > Victoria, BC, V8W 2Y2 > Phone: +1-250-721-8432 > Email: matthewb at uvic.ca <mailto:matthewb at uvic.ca> > > On 7/24/19 9:30 PM, Nithya Balachandran wrote: >> >> >> On Wed, 24 Jul 2019 at 22:12, Matthew Benstead <matthewb at uvic.ca >> <mailto:matthewb at uvic.ca>> wrote: >> >> So looking more closely at the trusted.glusterfs.dht >> attributes from the bricks it looks like they cover the >> entire range... and there is no range left for gluster07. >> >> The first 6 bricks range from 0x00000000 to 0xffffffff - >> so... is there a way to re-calculate what the dht values >> should be? Each of the bricks should have a gap >> >> Gluster05 00000000 -> 2aaaaaa9 >> Gluster06 2aaaaaaa -> 55555553 >> Gluster01 55555554 -> 7ffffffd >> Gluster02 7ffffffe -> aaaaaaa7 >> Gluster03 aaaaaaa8 -> d5555551 >> Gluster04 d5555552 -> ffffffff >> Gluster07 None >> >> If we split the range into 7 servers that would be a gap of >> about 0x24924924 for each server. >> >> Now in terms of the gluster07 brick, about 2 years ago the >> RAID array the brick was stored on became corrupted. I ran >> the remove-brick force command, then provisioned a new >> server, ran the add-brick command and then restored the >> missing files from backup by copying them back to the main >> gluster mount (not the brick). >> >> >> Did you run a rebalance after performing the add-brick? Without a >> rebalance/fix-layout , the layout for existing directories on the >> volume will not? be updated to use the new brick as well. >> >> That the layout does not include the new brick in the root dir is >> in itself is not a problem. Do you create a lot of files directly >> in the root of the volume? If yes, you might want to run a >> rebalance. Otherwise, if you mostly create files in newly added >> directories, you can probably ignore this. You can check the >> layout for directories on the volume and see if they incorporate >> the brick7. >> >> I would expect a lookup on the root to have set an xattr on the >> brick with an empty layout range . The fact that the xattr does >> not exist at all on the brick is what I am looking into. >> >> >> It looks like prior to that event this was the layout - which >> would make sense given the equal size of the 7 bricks: >> >> gluster02.pcic.uvic.ca <http://gluster02.pcic.uvic.ca> | >> SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x000000010000000048bfff206d1ffe5f >> >> gluster05.pcic.uvic.ca <http://gluster05.pcic.uvic.ca> | >> SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000b5dffce0da3ffc1f >> >> gluster04.pcic.uvic.ca <http://gluster04.pcic.uvic.ca> | >> SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000917ffda0b5dffcdf >> >> gluster03.pcic.uvic.ca <http://gluster03.pcic.uvic.ca> | >> SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x00000001000000006d1ffe60917ffd9f >> >> gluster01.pcic.uvic.ca <http://gluster01.pcic.uvic.ca> | >> SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000245fffe048bfff1f >> >> gluster07.pcic.uvic.ca <http://gluster07.pcic.uvic.ca> | >> SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x000000010000000000000000245fffdf >> >> gluster06.pcic.uvic.ca <http://gluster06.pcic.uvic.ca> | >> SUCCESS | rc=0 >> >> # file: /mnt/raid6-storage/storage >> trusted.glusterfs.dht=0x0000000100000000da3ffc20ffffffff >> >> Which yields the following: >> >> 00000000 -> 245fffdf??? Gluster07 >> 245fffe0 -> 48bfff1f??? Gluster01 >> 48bfff20 -> 6d1ffe5f??? Gluster02 >> 6d1ffe60 -> 917ffd9f??? Gluster03 >> 917ffda0 -> b5dffcdf??? Gluster04 >> b5dffce0 -> da3ffc1f??? Gluster05 >> da3ffc20 -> ffffffff??? Gluster06 >> >> Is there some way to get back to this? >> >> Thanks, >> ?-Matthew >> >> -- >> Matthew Benstead >> System Administrator >> Pacific Climate Impacts Consortium <https://pacificclimate.org/> >> University of Victoria, UH1 >> PO Box 1800, STN CSC >> Victoria, BC, V8W 2Y2 >> Phone: +1-250-721-8432 >> Email: matthewb at uvic.ca <mailto:matthewb at uvic.ca> >> >> On 7/18/19 7:20 AM, Matthew Benstead wrote: >>> Hi Nithya, >>> >>> No - it was added about a year and a half ago. I have tried >>> re-mounting the volume on the server, but it didn't add the >>> attr: >>> >>> [root at gluster07 ~]# umount /storage/ >>> [root at gluster07 ~]# cat /etc/fstab | grep "/storage" >>> 10.0.231.56:/storage /storage glusterfs >>> defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 >>> [root at gluster07 ~]# mount /storage/ >>> [root at gluster07 ~]# df -h /storage/ >>> Filesystem??????????? Size? Used Avail Use% Mounted on >>> 10.0.231.56:/storage? 255T? 194T?? 62T? 77% /storage >>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >>> /mnt/raid6-storage/storage/ >>> # file: /mnt/raid6-storage/storage/ >>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>> trusted.gfid=0x00000000000000000000000000000001 >>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 >>> trusted.glusterfs.quota.dirty=0x3000 >>> trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 >>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>> >>> Thanks, >>> ?-Matthew >>> >>> On 7/17/19 10:04 PM, Nithya Balachandran wrote: >>>> Hi Matthew, >>>> >>>> Was this node/brick added to the volume recently? If yes, >>>> try mounting the volume on a fresh mount point - that >>>> should create the xattr on this as well. >>>> >>>> Regards, >>>> Nithya >>>> >>>> On Wed, 17 Jul 2019 at 21:01, Matthew Benstead >>>> <matthewb at uvic.ca <mailto:matthewb at uvic.ca>> wrote: >>>> >>>> Hello, >>>> >>>> I've just noticed one brick in my 7 node distribute >>>> volume is missing >>>> the trusted.glusterfs.dht xattr...? How can I fix this? >>>> >>>> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. >>>> >>>> All of the other nodes are fine, but gluster07 from the >>>> list below does >>>> not have the attribute. >>>> >>>> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a >>>> "getfattr -m . >>>> --absolute-names -n trusted.glusterfs.dht -e hex >>>> /mnt/raid6-storage/storage" >>>> ... >>>> gluster05 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 >>>> >>>> gluster03 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 >>>> >>>> gluster04 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff >>>> >>>> gluster06 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 >>>> >>>> gluster02 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 >>>> >>>> gluster07 | FAILED | rc=1 >> >>>> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such >>>> attributenon-zero return code >>>> >>>> gluster01 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000555555547ffffffd >>>> >>>> Here are all of the attr's from the brick: >>>> >>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d >>>> -e hex >>>> /mnt/raid6-storage/storage/ >>>> # file: /mnt/raid6-storage/storage/ >>>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>> trusted.gfid=0x00000000000000000000000000000001 >>>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 >>>> trusted.glusterfs.quota.dirty=0x3000 >>>> trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 >>>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>>> >>>> >>>> And here is the volume information: >>>> >>>> [root at gluster07 ~]# gluster volume info storage >>>> >>>> Volume Name: storage >>>> Type: Distribute >>>> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >>>> Status: Started >>>> Snapshot Count: 0 >>>> Number of Bricks: 7 >>>> Transport-type: tcp >>>> Bricks: >>>> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >>>> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >>>> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >>>> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >>>> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >>>> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >>>> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >>>> Options Reconfigured: >>>> changelog.changelog: on >>>> features.quota-deem-statfs: on >>>> features.read-only: off >>>> features.inode-quota: on >>>> features.quota: on >>>> performance.readdir-ahead: on >>>> nfs.disable: on >>>> geo-replication.indexing: on >>>> geo-replication.ignore-pid-check: on >>>> transport.address-family: inet >>>> >>>> Thanks, >>>> ?-Matthew >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> <mailto:Gluster-users at gluster.org> >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190726/e106a785/attachment.html>
Nithya Balachandran
2019-Jul-29 03:43 UTC
[Gluster-users] Brick missing trusted.glusterfs.dht xattr
On Sat, 27 Jul 2019 at 02:31, Matthew Benstead <matthewb at uvic.ca> wrote:> Ok thank-you for explaining everything - that makes sense. > > Currently the brick file systems are pretty evenly distributed so I > probably won't run the fix-layout right now. > > Would this state have any impact on geo-replication? I'm trying to > geo-replicate this volume, but am getting a weird error: "Changelog > register failed error=[Errno 21] Is a directory" >It should not. Sunny, can you comment on this? Regards, Nithya> > I assume this is related to something else, but I wasn't sure. > > Thanks, > -Matthew > > -- > Matthew Benstead > System Administrator > Pacific Climate Impacts Consortium <https://pacificclimate.org/> > University of Victoria, UH1 > PO Box 1800, STN CSC > Victoria, BC, V8W 2Y2 > Phone: +1-250-721-8432 > Email: matthewb at uvic.ca > On 7/26/19 12:02 AM, Nithya Balachandran wrote: > > > > On Fri, 26 Jul 2019 at 01:56, Matthew Benstead <matthewb at uvic.ca> wrote: > >> Hi Nithya, >> >> Hmm... I don't remember if I did, but based on what I'm seeing it sounds >> like I probably didn't run rebalance or fix-layout. >> >> It looks like folders that haven't had any new files created have a dht >> of 0, while other folders have non-zero values. >> >> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >> /mnt/raid6-storage/storage/ | grep dht >> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >> /mnt/raid6-storage/storage/home | grep dht >> trusted.glusterfs.dht=0x00000000000000000000000000000000 >> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >> /mnt/raid6-storage/storage/home/matthewb | grep dht >> trusted.glusterfs.dht=0x00000001000000004924921a6db6dbc7 >> >> If I just run the fix-layout command will it re-create all of the dht >> values or just the missing ones? >> > > A fix-layout will recalculate the layouts entirely so files all the values > will change. No files will be moved. > A rebalance will recalculate the layouts like the fix-layout but will also > move files to their new locations based on the new layout ranges. This > could take a lot of time depending on the number of files/directories on > the volume. If you do this, I would recommend that you turn off > lookup-optimize until the rebalance is over. > > >> Since the brick is already fairly size balanced could I get away with >> running fix-layout but not rebalance? Or would the new dht layout mean >> slower accesses since the files may be expected on different bricks? >> > > The first access for a file will be slower. The next one will be faster as > the location will be cached in the client's in-memory structures. > You may not need to run either a fix-layout or a rebalance if new file > creations will be in directories created after the add-brick. Gluster will > automatically include all 7 bricks for those directories. > > Regards, > Nithya > > >> Thanks, >> -Matthew >> >> -- >> Matthew Benstead >> System Administrator >> Pacific Climate Impacts Consortium <https://pacificclimate.org/> >> University of Victoria, UH1 >> PO Box 1800, STN CSC >> Victoria, BC, V8W 2Y2 >> Phone: +1-250-721-8432 >> Email: matthewb at uvic.ca >> On 7/24/19 9:30 PM, Nithya Balachandran wrote: >> >> >> >> On Wed, 24 Jul 2019 at 22:12, Matthew Benstead <matthewb at uvic.ca> wrote: >> >>> So looking more closely at the trusted.glusterfs.dht attributes from the >>> bricks it looks like they cover the entire range... and there is no range >>> left for gluster07. >>> >>> The first 6 bricks range from 0x00000000 to 0xffffffff - so... is there >>> a way to re-calculate what the dht values should be? Each of the bricks >>> should have a gap >>> >>> Gluster05 00000000 -> 2aaaaaa9 >>> Gluster06 2aaaaaaa -> 55555553 >>> Gluster01 55555554 -> 7ffffffd >>> Gluster02 7ffffffe -> aaaaaaa7 >>> Gluster03 aaaaaaa8 -> d5555551 >>> Gluster04 d5555552 -> ffffffff >>> Gluster07 None >>> >>> If we split the range into 7 servers that would be a gap of about >>> 0x24924924 for each server. >>> >>> Now in terms of the gluster07 brick, about 2 years ago the RAID array >>> the brick was stored on became corrupted. I ran the remove-brick force >>> command, then provisioned a new server, ran the add-brick command and then >>> restored the missing files from backup by copying them back to the main >>> gluster mount (not the brick). >>> >>> >> Did you run a rebalance after performing the add-brick? Without a >> rebalance/fix-layout , the layout for existing directories on the volume >> will not be updated to use the new brick as well. >> >> That the layout does not include the new brick in the root dir is in >> itself is not a problem. Do you create a lot of files directly in the root >> of the volume? If yes, you might want to run a rebalance. Otherwise, if you >> mostly create files in newly added directories, you can probably ignore >> this. You can check the layout for directories on the volume and see if >> they incorporate the brick7. >> >> I would expect a lookup on the root to have set an xattr on the brick >> with an empty layout range . The fact that the xattr does not exist at all >> on the brick is what I am looking into. >> >> >> It looks like prior to that event this was the layout - which would make >>> sense given the equal size of the 7 bricks: >>> >>> gluster02.pcic.uvic.ca | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x000000010000000048bfff206d1ffe5f >>> >>> gluster05.pcic.uvic.ca | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000b5dffce0da3ffc1f >>> >>> gluster04.pcic.uvic.ca | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000917ffda0b5dffcdf >>> >>> gluster03.pcic.uvic.ca | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x00000001000000006d1ffe60917ffd9f >>> >>> gluster01.pcic.uvic.ca | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000245fffe048bfff1f >>> >>> gluster07.pcic.uvic.ca | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x000000010000000000000000245fffdf >>> >>> gluster06.pcic.uvic.ca | SUCCESS | rc=0 >> >>> # file: /mnt/raid6-storage/storage >>> trusted.glusterfs.dht=0x0000000100000000da3ffc20ffffffff >>> >>> Which yields the following: >>> >>> 00000000 -> 245fffdf Gluster07 >>> 245fffe0 -> 48bfff1f Gluster01 >>> 48bfff20 -> 6d1ffe5f Gluster02 >>> 6d1ffe60 -> 917ffd9f Gluster03 >>> 917ffda0 -> b5dffcdf Gluster04 >>> b5dffce0 -> da3ffc1f Gluster05 >>> da3ffc20 -> ffffffff Gluster06 >>> >>> Is there some way to get back to this? >>> >>> Thanks, >>> -Matthew >>> >>> -- >>> Matthew Benstead >>> System Administrator >>> Pacific Climate Impacts Consortium <https://pacificclimate.org/> >>> University of Victoria, UH1 >>> PO Box 1800, STN CSC >>> Victoria, BC, V8W 2Y2 >>> Phone: +1-250-721-8432 >>> Email: matthewb at uvic.ca >>> On 7/18/19 7:20 AM, Matthew Benstead wrote: >>> >>> Hi Nithya, >>> >>> No - it was added about a year and a half ago. I have tried re-mounting >>> the volume on the server, but it didn't add the attr: >>> >>> [root at gluster07 ~]# umount /storage/ >>> [root at gluster07 ~]# cat /etc/fstab | grep "/storage" >>> 10.0.231.56:/storage /storage glusterfs >>> defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0 >>> [root at gluster07 ~]# mount /storage/ >>> [root at gluster07 ~]# df -h /storage/ >>> Filesystem Size Used Avail Use% Mounted on >>> 10.0.231.56:/storage 255T 194T 62T 77% /storage >>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >>> /mnt/raid6-storage/storage/ >>> # file: /mnt/raid6-storage/storage/ >>> >>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>> trusted.gfid=0x00000000000000000000000000000001 >>> >>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0 >>> trusted.glusterfs.quota.dirty=0x3000 >>> >>> trusted.glusterfs.quota.size.2=0x00001b71d5279e000000000000763e32000000000005cd53 >>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>> >>> Thanks, >>> -Matthew >>> >>> On 7/17/19 10:04 PM, Nithya Balachandran wrote: >>> >>> Hi Matthew, >>> >>> Was this node/brick added to the volume recently? If yes, try mounting >>> the volume on a fresh mount point - that should create the xattr on this as >>> well. >>> >>> Regards, >>> Nithya >>> >>> On Wed, 17 Jul 2019 at 21:01, Matthew Benstead <matthewb at uvic.ca> wrote: >>> >>>> Hello, >>>> >>>> I've just noticed one brick in my 7 node distribute volume is missing >>>> the trusted.glusterfs.dht xattr...? How can I fix this? >>>> >>>> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7. >>>> >>>> All of the other nodes are fine, but gluster07 from the list below does >>>> not have the attribute. >>>> >>>> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a "getfattr -m . >>>> --absolute-names -n trusted.glusterfs.dht -e hex >>>> /mnt/raid6-storage/storage" >>>> ... >>>> gluster05 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 >>>> >>>> gluster03 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 >>>> >>>> gluster04 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff >>>> >>>> gluster06 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 >>>> >>>> gluster02 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 >>>> >>>> gluster07 | FAILED | rc=1 >> >>>> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such >>>> attributenon-zero return code >>>> >>>> gluster01 | SUCCESS | rc=0 >> >>>> # file: /mnt/raid6-storage/storage >>>> trusted.glusterfs.dht=0x0000000100000000555555547ffffffd >>>> >>>> Here are all of the attr's from the brick: >>>> >>>> [root at gluster07 ~]# getfattr --absolute-names -m . -d -e hex >>>> /mnt/raid6-storage/storage/ >>>> # file: /mnt/raid6-storage/storage/ >>>> >>>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>> trusted.gfid=0x00000000000000000000000000000001 >>>> >>>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee800001fdf9 >>>> trusted.glusterfs.quota.dirty=0x3000 >>>> >>>> trusted.glusterfs.quota.size.2=0x00001b69498a1400000000000076332e000000000005cd03 >>>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2 >>>> >>>> >>>> And here is the volume information: >>>> >>>> [root at gluster07 ~]# gluster volume info storage >>>> >>>> Volume Name: storage >>>> Type: Distribute >>>> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >>>> Status: Started >>>> Snapshot Count: 0 >>>> Number of Bricks: 7 >>>> Transport-type: tcp >>>> Bricks: >>>> Brick1: 10.0.231.50:/mnt/raid6-storage/storage >>>> Brick2: 10.0.231.51:/mnt/raid6-storage/storage >>>> Brick3: 10.0.231.52:/mnt/raid6-storage/storage >>>> Brick4: 10.0.231.53:/mnt/raid6-storage/storage >>>> Brick5: 10.0.231.54:/mnt/raid6-storage/storage >>>> Brick6: 10.0.231.55:/mnt/raid6-storage/storage >>>> Brick7: 10.0.231.56:/mnt/raid6-storage/storage >>>> Options Reconfigured: >>>> changelog.changelog: on >>>> features.quota-deem-statfs: on >>>> features.read-only: off >>>> features.inode-quota: on >>>> features.quota: on >>>> performance.readdir-ahead: on >>>> nfs.disable: on >>>> geo-replication.indexing: on >>>> geo-replication.ignore-pid-check: on >>>> transport.address-family: inet >>>> >>>> Thanks, >>>> -Matthew >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>> >>> >>> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190729/256f7560/attachment.html>