Vijaikumar M
2015-Mar-16 11:25 UTC
[Gluster-users] Issue with removing directories after disabling quotas
Hi JF, This may not be a quota issue. Can you please check if there are any linkto files exists on the brick? On node 'stor104/106', can we get the below output #find /zfs/brick0/brick | xargs ls -l #find /zfs/brick1/brick | xargs ls -l #find /zfs/brick2/brick | xargs ls -l #find /zfs/brick0/brick | xargs getfattr -d -m . -e hex #find /zfs/brick1/brick | xargs getfattr -d -m . -e hex #find /zfs/brick2/brick | xargs getfattr -d -m . -e hex Thanks, Vijay On Monday 16 March 2015 04:18 PM, JF Le Fill?tre wrote:> Forgot to mention: the non-empty directories list files like this: > > -?????????? ? ? ? ? ? Hal8723APhyReg.h > -?????????? ? ? ? ? ? Hal8723UHWImg_CE.h > -?????????? ? ? ? ? ? hal_com.h > -?????????? ? ? ? ? ? HalDMOutSrc8723A.h > -?????????? ? ? ? ? ? HalHWImg8723A_BB.h > -?????????? ? ? ? ? ? HalHWImg8723A_FW.h > -?????????? ? ? ? ? ? HalHWImg8723A_MAC.h > -?????????? ? ? ? ? ? HalHWImg8723A_RF.h > -?????????? ? ? ? ? ? hal_intf.h > -?????????? ? ? ? ? ? HalPwrSeqCmd.h > -?????????? ? ? ? ? ? ieee80211.h > -?????????? ? ? ? ? ? odm_debug.h > > Thanks, > JF > > > On 16/03/15 11:45, JF Le Fill?tre wrote: >> Hello all, >> >> So, another day another issue. I was trying to play with quotas on my volume: >> >> ===============================================================================>> [root at stor104 ~]# gluster volume status >> Status of volume: live >> Gluster process Port Online Pid >> ------------------------------------------------------------------------------ >> Brick stor104:/zfs/brick0/brick 49167 Y 13446 >> Brick stor104:/zfs/brick1/brick 49168 Y 13457 >> Brick stor104:/zfs/brick2/brick 49169 Y 13468 >> Brick stor106:/zfs/brick0/brick 49159 Y 14158 >> Brick stor106:/zfs/brick1/brick 49160 Y 14169 >> Brick stor106:/zfs/brick2/brick 49161 Y 14180 >> NFS Server on localhost 2049 Y 13483 >> Quota Daemon on localhost N/A Y 13490 >> NFS Server on stor106 2049 Y 14195 >> Quota Daemon on stor106 N/A Y 14202 >> >> Task Status of Volume live >> ------------------------------------------------------------------------------ >> Task : Rebalance >> ID : 6bd03709-1f48-49a9-a215-d0a6e6f3ab1e >> Status : completed >> ===============================================================================>> >> >> Not sure if the "Quota Daemon on localhost -> N/A" is normal, but that's another topic. >> >> While the quotas were enabled, I did some test copying a whole tree of small files (the Linux kernel sources) to the volume to see which performance I would get, and it was really low. So I decided to disable quotas again: >> >> >> ===============================================================================>> [root at stor104 ~]# gluster volume status >> Status of volume: live >> Gluster process Port Online Pid >> ------------------------------------------------------------------------------ >> Brick stor104:/zfs/brick0/brick 49167 Y 13754 >> Brick stor104:/zfs/brick1/brick 49168 Y 13765 >> Brick stor104:/zfs/brick2/brick 49169 Y 13776 >> Brick stor106:/zfs/brick0/brick 49159 Y 14282 >> Brick stor106:/zfs/brick1/brick 49160 Y 14293 >> Brick stor106:/zfs/brick2/brick 49161 Y 14304 >> NFS Server on localhost 2049 Y 13790 >> NFS Server on stor106 2049 Y 14319 >> >> Task Status of Volume live >> ------------------------------------------------------------------------------ >> Task : Rebalance >> ID : 6bd03709-1f48-49a9-a215-d0a6e6f3ab1e >> Status : completed >> ===============================================================================>> >> >> I remounted the volume from the client and tried deleting the directory containing the sources, which gave me a very long list of this: >> >> >> ===============================================================================>> rm: cannot remove ?/glusterfs/live/linux-3.18.7/tools/testing/selftests/ftrace/test.d/kprobe?: Directory not empty >> rm: cannot remove ?/glusterfs/live/linux-3.18.7/tools/testing/selftests/ptrace?: Directory not empty >> rm: cannot remove ?/glusterfs/live/linux-3.18.7/tools/testing/selftests/rcutorture/configs/rcu/v0.0?: Directory not empty >> rm: cannot remove ?/glusterfs/live/linux-3.18.7/tools/testing/selftests/rcutorture/configs/rcu/v3.5?: Directory not empty >> rm: cannot remove ?/glusterfs/live/linux-3.18.7/tools/testing/selftests/powerpc?: Directory not empty >> rm: cannot remove ?/glusterfs/live/linux-3.18.7/tools/perf/scripts/python/Perf-Trace-Util?: Directory not empty >> rm: cannot remove ?/glusterfs/live/linux-3.18.7/tools/perf/Documentation?: Directory not empty >> rm: cannot remove ?/glusterfs/live/linux-3.18.7/tools/perf/ui/tui?: Directory not empty >> rm: cannot remove ?/glusterfs/live/linux-3.18.7/tools/perf/util/include?: Directory not empty >> rm: cannot remove ?/glusterfs/live/linux-3.18.7/tools/lib?: Directory not empty >> rm: cannot remove ?/glusterfs/live/linux-3.18.7/tools/virtio?: Directory not empty >> rm: cannot remove ?/glusterfs/live/linux-3.18.7/virt/kvm?: Directory not empty >> ===============================================================================>> >> >> I did my homework on Google, yet the information I found was that the cause for this is that the contents of the bricks have been modified locally. It's definitely not the case here, I have *not* touched the contents of the bricks. >> >> So my question is: is it possible that disabling the quotas may have had some side effects on the metadata of Gluster? If so, what can I do to force Gluster to rescan all local directories and "import" local files? >> >> GlusterFS version: 3.6.2 >> >> The setup of my volume: >> >> >> ===============================================================================>> [root at stor104 ~]# gluster volume info >> >> Volume Name: live >> Type: Distribute >> Volume ID: >> Status: Started >> Number of Bricks: 6 >> Transport-type: tcp >> Bricks: >> Brick1: stor104:/zfs/brick0/brick >> Brick2: stor104:/zfs/brick1/brick >> Brick3: stor104:/zfs/brick2/brick >> Brick4: stor106:/zfs/brick0/brick >> Brick5: stor106:/zfs/brick1/brick >> Brick6: stor106:/zfs/brick2/brick >> Options Reconfigured: >> features.quota: off >> performance.readdir-ahead: on >> nfs.volume-access: read-only >> cluster.data-self-heal-algorithm: full >> performance.strict-write-ordering: off >> performance.strict-o-direct: off >> performance.force-readdirp: off >> performance.write-behind-window-size: 4MB >> performance.io-thread-count: 32 >> performance.flush-behind: on >> performance.client-io-threads: on >> performance.cache-size: 32GB >> performance.cache-refresh-timeout: 60 >> performance.cache-max-file-size: 4MB >> nfs.disable: off >> cluster.eager-lock: on >> cluster.min-free-disk: 1% >> server.allow-insecure: on >> diagnostics.client-log-level: ERROR >> diagnostics.brick-log-level: ERROR >> ===============================================================================>> >> >> It is mounted from the client with those fstab options: >> >> ===============================================================================>> stor104:live defaults,backupvolfile-server=stor106,direct-io-mode=disable,noauto >> ===============================================================================>> >> Attached are the log files from stor104 >> >> Thanks a lot for any help! >> JF >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >>
JF Le Fillâtre
2015-Mar-16 11:34 UTC
[Gluster-users] Issue with removing directories after disabling quotas
Hello Vijay, Mmmm sorry, I jumped the gun and deleted by hand all the files present on the bricks, so I can't see if there's any link anywhere... It's not the first time I have seen this. Very early in my tests I had another case of files still present on the bricks but not visible in Gluster, which I solved the same way. At that point I chalked it up to my limited knowledge of GlusterFS and I assumed that I had misconfigured it. What do you suspect that it may be, and what do I have to look for if it ever happens again? Thanks! JF On 16/03/15 12:25, Vijaikumar M wrote:> Hi JF, > > This may not be a quota issue. Can you please check if there are any > linkto files exists on the brick? > On node 'stor104/106', can we get the below output > > #find /zfs/brick0/brick | xargs ls -l > #find /zfs/brick1/brick | xargs ls -l > #find /zfs/brick2/brick | xargs ls -l > > #find /zfs/brick0/brick | xargs getfattr -d -m . -e hex > #find /zfs/brick1/brick | xargs getfattr -d -m . -e hex > #find /zfs/brick2/brick | xargs getfattr -d -m . -e hex > > > Thanks, > Vijay > > > > On Monday 16 March 2015 04:18 PM, JF Le Fill?tre wrote: >> Forgot to mention: the non-empty directories list files like this: >> >> -?????????? ? ? ? ? ? Hal8723APhyReg.h >> -?????????? ? ? ? ? ? Hal8723UHWImg_CE.h >> -?????????? ? ? ? ? ? hal_com.h >> -?????????? ? ? ? ? ? HalDMOutSrc8723A.h >> -?????????? ? ? ? ? ? HalHWImg8723A_BB.h >> -?????????? ? ? ? ? ? HalHWImg8723A_FW.h >> -?????????? ? ? ? ? ? HalHWImg8723A_MAC.h >> -?????????? ? ? ? ? ? HalHWImg8723A_RF.h >> -?????????? ? ? ? ? ? hal_intf.h >> -?????????? ? ? ? ? ? HalPwrSeqCmd.h >> -?????????? ? ? ? ? ? ieee80211.h >> -?????????? ? ? ? ? ? odm_debug.h >> >> Thanks, >> JF >> >> >> On 16/03/15 11:45, JF Le Fill?tre wrote: >>> Hello all, >>> >>> So, another day another issue. I was trying to play with quotas on my >>> volume: >>> >>> ===============================================================================>>> >>> [root at stor104 ~]# gluster volume status >>> Status of volume: live >>> Gluster process Port Online Pid >>> ------------------------------------------------------------------------------ >>> >>> Brick stor104:/zfs/brick0/brick 49167 Y 13446 >>> Brick stor104:/zfs/brick1/brick 49168 Y 13457 >>> Brick stor104:/zfs/brick2/brick 49169 Y 13468 >>> Brick stor106:/zfs/brick0/brick 49159 Y 14158 >>> Brick stor106:/zfs/brick1/brick 49160 Y 14169 >>> Brick stor106:/zfs/brick2/brick 49161 Y 14180 >>> NFS Server on localhost 2049 Y 13483 >>> Quota Daemon on localhost N/A Y 13490 >>> NFS Server on stor106 2049 Y 14195 >>> Quota Daemon on stor106 N/A Y 14202 >>> Task Status of Volume live >>> ------------------------------------------------------------------------------ >>> >>> Task : Rebalance >>> ID : 6bd03709-1f48-49a9-a215-d0a6e6f3ab1e >>> Status : completed >>> ===============================================================================>>> >>> >>> >>> Not sure if the "Quota Daemon on localhost -> N/A" is normal, but >>> that's another topic. >>> >>> While the quotas were enabled, I did some test copying a whole tree >>> of small files (the Linux kernel sources) to the volume to see which >>> performance I would get, and it was really low. So I decided to >>> disable quotas again: >>> >>> >>> ===============================================================================>>> >>> [root at stor104 ~]# gluster volume status >>> Status of volume: live >>> Gluster process Port Online Pid >>> ------------------------------------------------------------------------------ >>> >>> Brick stor104:/zfs/brick0/brick 49167 Y 13754 >>> Brick stor104:/zfs/brick1/brick 49168 Y 13765 >>> Brick stor104:/zfs/brick2/brick 49169 Y 13776 >>> Brick stor106:/zfs/brick0/brick 49159 Y 14282 >>> Brick stor106:/zfs/brick1/brick 49160 Y 14293 >>> Brick stor106:/zfs/brick2/brick 49161 Y 14304 >>> NFS Server on localhost 2049 Y 13790 >>> NFS Server on stor106 2049 Y 14319 >>> Task Status of Volume live >>> ------------------------------------------------------------------------------ >>> >>> Task : Rebalance >>> ID : 6bd03709-1f48-49a9-a215-d0a6e6f3ab1e >>> Status : completed >>> ===============================================================================>>> >>> >>> >>> I remounted the volume from the client and tried deleting the >>> directory containing the sources, which gave me a very long list of >>> this: >>> >>> >>> ===============================================================================>>> >>> rm: cannot remove >>> ?/glusterfs/live/linux-3.18.7/tools/testing/selftests/ftrace/test.d/kprobe?: >>> Directory not empty >>> rm: cannot remove >>> ?/glusterfs/live/linux-3.18.7/tools/testing/selftests/ptrace?: >>> Directory not empty >>> rm: cannot remove >>> ?/glusterfs/live/linux-3.18.7/tools/testing/selftests/rcutorture/configs/rcu/v0.0?: >>> Directory not empty >>> rm: cannot remove >>> ?/glusterfs/live/linux-3.18.7/tools/testing/selftests/rcutorture/configs/rcu/v3.5?: >>> Directory not empty >>> rm: cannot remove >>> ?/glusterfs/live/linux-3.18.7/tools/testing/selftests/powerpc?: >>> Directory not empty >>> rm: cannot remove >>> ?/glusterfs/live/linux-3.18.7/tools/perf/scripts/python/Perf-Trace-Util?: >>> Directory not empty >>> rm: cannot remove >>> ?/glusterfs/live/linux-3.18.7/tools/perf/Documentation?: Directory >>> not empty >>> rm: cannot remove ?/glusterfs/live/linux-3.18.7/tools/perf/ui/tui?: >>> Directory not empty >>> rm: cannot remove >>> ?/glusterfs/live/linux-3.18.7/tools/perf/util/include?: Directory not >>> empty >>> rm: cannot remove ?/glusterfs/live/linux-3.18.7/tools/lib?: Directory >>> not empty >>> rm: cannot remove ?/glusterfs/live/linux-3.18.7/tools/virtio?: >>> Directory not empty >>> rm: cannot remove ?/glusterfs/live/linux-3.18.7/virt/kvm?: Directory >>> not empty >>> ===============================================================================>>> >>> >>> >>> I did my homework on Google, yet the information I found was that the >>> cause for this is that the contents of the bricks have been modified >>> locally. It's definitely not the case here, I have *not* touched the >>> contents of the bricks. >>> >>> So my question is: is it possible that disabling the quotas may have >>> had some side effects on the metadata of Gluster? If so, what can I >>> do to force Gluster to rescan all local directories and "import" >>> local files? >>> >>> GlusterFS version: 3.6.2 >>> >>> The setup of my volume: >>> >>> >>> ===============================================================================>>> >>> [root at stor104 ~]# gluster volume info >>> Volume Name: live >>> Type: Distribute >>> Volume ID: >>> Status: Started >>> Number of Bricks: 6 >>> Transport-type: tcp >>> Bricks: >>> Brick1: stor104:/zfs/brick0/brick >>> Brick2: stor104:/zfs/brick1/brick >>> Brick3: stor104:/zfs/brick2/brick >>> Brick4: stor106:/zfs/brick0/brick >>> Brick5: stor106:/zfs/brick1/brick >>> Brick6: stor106:/zfs/brick2/brick >>> Options Reconfigured: >>> features.quota: off >>> performance.readdir-ahead: on >>> nfs.volume-access: read-only >>> cluster.data-self-heal-algorithm: full >>> performance.strict-write-ordering: off >>> performance.strict-o-direct: off >>> performance.force-readdirp: off >>> performance.write-behind-window-size: 4MB >>> performance.io-thread-count: 32 >>> performance.flush-behind: on >>> performance.client-io-threads: on >>> performance.cache-size: 32GB >>> performance.cache-refresh-timeout: 60 >>> performance.cache-max-file-size: 4MB >>> nfs.disable: off >>> cluster.eager-lock: on >>> cluster.min-free-disk: 1% >>> server.allow-insecure: on >>> diagnostics.client-log-level: ERROR >>> diagnostics.brick-log-level: ERROR >>> ===============================================================================>>> >>> >>> >>> It is mounted from the client with those fstab options: >>> >>> ===============================================================================>>> >>> stor104:live >>> defaults,backupvolfile-server=stor106,direct-io-mode=disable,noauto >>> ===============================================================================>>> >>> >>> Attached are the log files from stor104 >>> >>> Thanks a lot for any help! >>> JF >>> >>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >-- Jean-Fran?ois Le Fill?tre ------------------------------- HPC Systems Administrator LCSB - University of Luxembourg ------------------------------- PGP KeyID 0x134657C6