I've set up everything and have moved close to 2Tb of data to the glusterFS volume without incident. I thought that I try a recursive long listing just to make sure everything was working properly (in anticipation of doing just that when I turn on AFR soon). Doing ls -alR on mounted GlusterFS volume. Everything looks good, but every so often I get output that looks like: -rw------- 1 zope zope 30512 Jan 27 2008 efb1f22f2312af6ed5fbef014ee3 ?--------- ? ? ? ? ? /mnt/BACKUPS/blobdata/00/66/1ac485 0e731fed2e0eb2fdbc8d05 ?--------- ? ? ? ? ? /mnt/BACKUPS/blobdata/00/66/1d7d22 6f5f6845d62fddcadb4ca9 ?--------- ? ? ? ? ? /mnt/BACKUPS/blobdata/00/66/284371 d43c24d0d3267109c674e7 If I stop the listing and check the files individually I get: [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/66/1ac4850e731fed2e0 eb2fdbc8d05 -rw------- 1 zope zope 23328 Aug 12 2008 /mnt/BACKUPS/blobdata/00/66/1ac4850e73 1fed2e0eb2fdbc8d05 [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/66/1d7d226f5f6845d62 fddcadb4ca9 -rw-rw-r-- 1 zope zope 1616 Nov 2 2007 /mnt/BACKUPS/blobdata/00/66/1d7d226f5f6 845d62fddcadb4ca9 [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/66/284371d43c24d0d32 67109c674e7 -rw-rw-r-- 1 zope zope 34928 Dec 28 2007 /mnt/BACKUPS/blobdata/00/66/284371d43c 24d0d3267109c674e7 [zope at zodb01 prodscripts]$ Nothing appears to be wrong. Hint: These errors ALWAYS appear to affect the last few files listed in an individual directory (that is it nevery happens in the middle of a folder). Occasionally I get and error like the following: ls: /mnt/BACKUPS/blobdata/00/84/7590a35e615c736cc36324a41227: Permission denied Stopping the listing to check this file results in: [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/84/7590a35e615c736cc 36324a41227 -rw-rw-r-- 1 zope zope 512 Nov 22 2007 /mnt/BACKUPS/blobdata/00/84/7590a35e615c 736cc36324a41227 [zope at zodb01 prodscripts]$ Again, nothing appears to be wrong. Hint: These errors ALWAYS appear to affect the last file or files listed in an individual directory (that is it nevery happens in the middle of a folder). Sometimes it affects entire folders not just individual files: ls: /mnt/BACKUPS/blobdata/00/e1: Permission denied Stopping the recursive listing and just executing on this single folder works perfectly. [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/e1 total 15976 drwxrwxr-x 2 zope zope 8192 Dec 15 16:46 . drwxrwxr-x 258 zope zope 16384 Dec 15 16:46 .. -rw------- 1 zope zope 909456 Sep 15 2008 00fe85dffc1830fe10f34942c2c5 -rw------- 1 zope zope 399136 Oct 15 2008 0effcac8a7fb6819f17fcaea296f -rw------- 1 zope zope 149456 Jan 27 2008 1ebd7783357503e7b553dcfae23a -rw------- 1 zope zope 19472 Sep 29 10:27 2d3ada857f0e3692146f98cc923b -rw------- 1 zope zope 1440 Jun 21 10:18 302ad12cda83dcc7a7f241f9d7d4 -rw------- 1 zope zope 896 Mar 5 2008 3298c88694ee904cfd0ade6715fa -rw------- 1 zope zope 16 Jun 21 22:06 3ebb81573577e37fff643fbb7dcc -rw-rw-r-- 1 zope zope 132624 Dec 28 2007 415255d42503872b3a88cfe70d6c -rw------- 1 zope zope 416 Jun 26 2008 489e5ef508e56e2d76ba67c306ea -rw------- 1 zope zope 29200 Mar 17 2008 4a9f16e818309cc601bd886631cd -rw------- 1 zope zope 554272 Jan 26 2008 4ac36056b24a9b46e659abc44f49 -rw-rw-r-- 1 zope zope 468160 Jan 8 2008 5204e6d98d70e74240ec91263a46 -rw------- 1 zope zope 401616 Jan 26 2008 5255ecacc6709b64b7633cfb9199 -rw------- 1 zope zope 22096 Aug 4 02:49 590c4138783609d0d11b5d7d04a7 -rw------- 1 zope zope 2362752 Nov 8 19:48 631542f36a07601d678975374428 -rw------- 1 zope zope 1248 Feb 8 2009 6335e89fef0bdb300491a246e489 -rw------- 1 zope zope 875024 Jun 29 2008 6439dde68e0d91909518dce73e65 -rw------- 1 zope zope 1248 Feb 20 2008 6818b0f5686f2f4f7d892f42c5bf -rw------- 1 zope zope 27184 Jul 2 2008 693c281e4f7ae9ced89d0c326411 -rw------- 1 zope zope 1057392 Jan 26 2008 6a0ac5e30a06f3ee7723e4a1eb13 -rw-rw-r-- 1 zope zope 2871776 Dec 28 2007 6d932ed95810a51d7a5dd3852ad9 -rw-rw-r-- 1 zope zope 2996896 Nov 2 2007 73000cfd294042e5fcdc322e1f3d -rw------- 1 zope zope 3392 Oct 28 20:23 77638a97c972a1c1a395324d3d68 -rw-rw-r-- 1 zope zope 21584 Dec 29 2007 78fdd154c8c45aa793c2a4fdfac5 -rw------- 1 zope zope 21008 Jul 26 19:29 79ec9ef2f3937effc6af0f25d5c8 -rw-rw-r-- 1 zope zope 19872 Jan 8 2008 8044fb8c6b68f4484716c9757bfa -rw-rw-r-- 1 zope zope 21168 Jan 10 2008 83074f956ab297c62b3c1a41574c -rw------- 1 zope zope 784 Oct 16 08:56 8c72c89d9d2d86efc8fe1ed71cfe -rw-rw-r-- 1 zope zope 144 Dec 29 2007 99c5f01f3ed0dbdafed07ae38205 -rw------- 1 zope zope 16 Feb 12 2008 a2a02f4aa6ffc7f36103576c36e2 -rw------- 1 zope zope 26640 Sep 12 14:03 a2e55ba90942c56738e5c704c545 -rw------- 1 zope zope 95760 Jul 2 2008 a3c63b279c7edc4b4c797b8f332e -rw-rw-r-- 1 zope zope 77712 Jan 8 2008 ab93c709881a668b6f9f8a92c964 -rw-rw-r-- 1 zope zope 16544 Jan 8 2008 aed5db4a22f81825165b86a99aa1 -rw------- 1 zope zope 5040 Aug 20 2008 c9e589908be1c448737a5fca842c -rw------- 1 zope zope 3184 Jun 30 2008 cd45ed25b3ccea39d0091bbad4d3 -rw-rw-r-- 1 zope zope 534976 Dec 29 2007 cd4a58908bb8b8a8c84c004e258d -rw-rw-r-- 1 zope zope 144 Jan 10 2008 d84cfb4389497fd679bbb7ce4077 -rw------- 1 zope zope 4800 May 30 2008 ed0c4a25941795c2d216355af1d4 -rw------- 1 zope zope 1362768 Oct 7 06:12 edec7b082feab6b5ab923094954c -rw------- 1 zope zope 37392 Aug 4 17:48 ef358bc52a503dc9302526af3ce1 -rw------- 1 zope zope 1616 Jun 26 2008 f221db1d18f01ebe93caf29fe730 -rw------- 1 zope zope 400 Jan 20 2008 f4c43212930051c45da5163fba15 -rw------- 1 zope zope 10064 Aug 24 2008 f612a2284de15ad71b166132bb8b -rw-rw-r-- 1 zope zope 416704 Dec 28 2007 fbfca80d116b0d092aacd6712260 -rw------- 1 zope zope 640 Nov 26 16:13 fec8f41e8306758f238983948c64 -rw------- 1 zope zope 30224 Mar 16 2008 ffbda6d769a1bb8e73a8e966914f Ideas? Larry
On 12/18/2009 03:10 PM, Larry Bates wrote:> I've set up everything and have moved close to 2Tb of data to the glusterFS > volume without incident. I thought that I try a recursive long listing just > to make sure everything was working properly (in anticipation of doing just > that when I turn on AFR soon). > > Doing ls -alR on mounted GlusterFS volume. Everything looks good, but every > so often I get output that looks like: > > -rw------- 1 zope zope 30512 Jan 27 2008 efb1f22f2312af6ed5fbef014ee3 > ?--------- ? ? ? ? ? /mnt/BACKUPS/blobdata/00/66/1ac485 > 0e731fed2e0eb2fdbc8d05 > ?--------- ? ? ? ? ? /mnt/BACKUPS/blobdata/00/66/1d7d22 > 6f5f6845d62fddcadb4ca9 > ?--------- ? ? ? ? ? /mnt/BACKUPS/blobdata/00/66/284371 > d43c24d0d3267109c674e7 >The above sort of behaviour normally indicates that readdir() returned an entry and stat() returned an error such as "file does not exist." Result is a name with no information. Cheers, mark -- Mark Mielke<mark at mielke.cc>
Tejas N. Bhise
2009-Dec-21 17:40 UTC
[Gluster-users] ls -alR problems on mounted glusterFS
Thank you for pointing out this issue, Larry. We will open a bug for this and try to reproduce this inhouse for debugging. Will let you know how it goes. While working on it, one of us might get in touch with you to understand your environment, file/directory layout or filenames etc better, in case that is required for the debugging. Regards, Tejas. ----- Original Message ----- From: "Larry Bates" <larry.bates at vitalesafe.com> To: gluster-users at gluster.org Sent: Saturday, December 19, 2009 1:40:57 AM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi Subject: [Gluster-users] ls -alR problems on mounted glusterFS I've set up everything and have moved close to 2Tb of data to the glusterFS volume without incident. I thought that I try a recursive long listing just to make sure everything was working properly (in anticipation of doing just that when I turn on AFR soon). Doing ls -alR on mounted GlusterFS volume. Everything looks good, but every so often I get output that looks like: -rw------- 1 zope zope 30512 Jan 27 2008 efb1f22f2312af6ed5fbef014ee3 ?--------- ? ? ? ? ? /mnt/BACKUPS/blobdata/00/66/1ac485 0e731fed2e0eb2fdbc8d05 ?--------- ? ? ? ? ? /mnt/BACKUPS/blobdata/00/66/1d7d22 6f5f6845d62fddcadb4ca9 ?--------- ? ? ? ? ? /mnt/BACKUPS/blobdata/00/66/284371 d43c24d0d3267109c674e7 If I stop the listing and check the files individually I get: [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/66/1ac4850e731fed2e0 eb2fdbc8d05 -rw------- 1 zope zope 23328 Aug 12 2008 /mnt/BACKUPS/blobdata/00/66/1ac4850e73 1fed2e0eb2fdbc8d05 [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/66/1d7d226f5f6845d62 fddcadb4ca9 -rw-rw-r-- 1 zope zope 1616 Nov 2 2007 /mnt/BACKUPS/blobdata/00/66/1d7d226f5f6 845d62fddcadb4ca9 [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/66/284371d43c24d0d32 67109c674e7 -rw-rw-r-- 1 zope zope 34928 Dec 28 2007 /mnt/BACKUPS/blobdata/00/66/284371d43c 24d0d3267109c674e7 [zope at zodb01 prodscripts]$ Nothing appears to be wrong. Hint: These errors ALWAYS appear to affect the last few files listed in an individual directory (that is it nevery happens in the middle of a folder). Occasionally I get and error like the following: ls: /mnt/BACKUPS/blobdata/00/84/7590a35e615c736cc36324a41227: Permission denied Stopping the listing to check this file results in: [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/84/7590a35e615c736cc 36324a41227 -rw-rw-r-- 1 zope zope 512 Nov 22 2007 /mnt/BACKUPS/blobdata/00/84/7590a35e615c 736cc36324a41227 [zope at zodb01 prodscripts]$ Again, nothing appears to be wrong. Hint: These errors ALWAYS appear to affect the last file or files listed in an individual directory (that is it nevery happens in the middle of a folder). Sometimes it affects entire folders not just individual files: ls: /mnt/BACKUPS/blobdata/00/e1: Permission denied Stopping the recursive listing and just executing on this single folder works perfectly. [zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/e1 total 15976 drwxrwxr-x 2 zope zope 8192 Dec 15 16:46 . drwxrwxr-x 258 zope zope 16384 Dec 15 16:46 .. -rw------- 1 zope zope 909456 Sep 15 2008 00fe85dffc1830fe10f34942c2c5 -rw------- 1 zope zope 399136 Oct 15 2008 0effcac8a7fb6819f17fcaea296f -rw------- 1 zope zope 149456 Jan 27 2008 1ebd7783357503e7b553dcfae23a -rw------- 1 zope zope 19472 Sep 29 10:27 2d3ada857f0e3692146f98cc923b -rw------- 1 zope zope 1440 Jun 21 10:18 302ad12cda83dcc7a7f241f9d7d4 -rw------- 1 zope zope 896 Mar 5 2008 3298c88694ee904cfd0ade6715fa -rw------- 1 zope zope 16 Jun 21 22:06 3ebb81573577e37fff643fbb7dcc -rw-rw-r-- 1 zope zope 132624 Dec 28 2007 415255d42503872b3a88cfe70d6c -rw------- 1 zope zope 416 Jun 26 2008 489e5ef508e56e2d76ba67c306ea -rw------- 1 zope zope 29200 Mar 17 2008 4a9f16e818309cc601bd886631cd -rw------- 1 zope zope 554272 Jan 26 2008 4ac36056b24a9b46e659abc44f49 -rw-rw-r-- 1 zope zope 468160 Jan 8 2008 5204e6d98d70e74240ec91263a46 -rw------- 1 zope zope 401616 Jan 26 2008 5255ecacc6709b64b7633cfb9199 -rw------- 1 zope zope 22096 Aug 4 02:49 590c4138783609d0d11b5d7d04a7 -rw------- 1 zope zope 2362752 Nov 8 19:48 631542f36a07601d678975374428 -rw------- 1 zope zope 1248 Feb 8 2009 6335e89fef0bdb300491a246e489 -rw------- 1 zope zope 875024 Jun 29 2008 6439dde68e0d91909518dce73e65 -rw------- 1 zope zope 1248 Feb 20 2008 6818b0f5686f2f4f7d892f42c5bf -rw------- 1 zope zope 27184 Jul 2 2008 693c281e4f7ae9ced89d0c326411 -rw------- 1 zope zope 1057392 Jan 26 2008 6a0ac5e30a06f3ee7723e4a1eb13 -rw-rw-r-- 1 zope zope 2871776 Dec 28 2007 6d932ed95810a51d7a5dd3852ad9 -rw-rw-r-- 1 zope zope 2996896 Nov 2 2007 73000cfd294042e5fcdc322e1f3d -rw------- 1 zope zope 3392 Oct 28 20:23 77638a97c972a1c1a395324d3d68 -rw-rw-r-- 1 zope zope 21584 Dec 29 2007 78fdd154c8c45aa793c2a4fdfac5 -rw------- 1 zope zope 21008 Jul 26 19:29 79ec9ef2f3937effc6af0f25d5c8 -rw-rw-r-- 1 zope zope 19872 Jan 8 2008 8044fb8c6b68f4484716c9757bfa -rw-rw-r-- 1 zope zope 21168 Jan 10 2008 83074f956ab297c62b3c1a41574c -rw------- 1 zope zope 784 Oct 16 08:56 8c72c89d9d2d86efc8fe1ed71cfe -rw-rw-r-- 1 zope zope 144 Dec 29 2007 99c5f01f3ed0dbdafed07ae38205 -rw------- 1 zope zope 16 Feb 12 2008 a2a02f4aa6ffc7f36103576c36e2 -rw------- 1 zope zope 26640 Sep 12 14:03 a2e55ba90942c56738e5c704c545 -rw------- 1 zope zope 95760 Jul 2 2008 a3c63b279c7edc4b4c797b8f332e -rw-rw-r-- 1 zope zope 77712 Jan 8 2008 ab93c709881a668b6f9f8a92c964 -rw-rw-r-- 1 zope zope 16544 Jan 8 2008 aed5db4a22f81825165b86a99aa1 -rw------- 1 zope zope 5040 Aug 20 2008 c9e589908be1c448737a5fca842c -rw------- 1 zope zope 3184 Jun 30 2008 cd45ed25b3ccea39d0091bbad4d3 -rw-rw-r-- 1 zope zope 534976 Dec 29 2007 cd4a58908bb8b8a8c84c004e258d -rw-rw-r-- 1 zope zope 144 Jan 10 2008 d84cfb4389497fd679bbb7ce4077 -rw------- 1 zope zope 4800 May 30 2008 ed0c4a25941795c2d216355af1d4 -rw------- 1 zope zope 1362768 Oct 7 06:12 edec7b082feab6b5ab923094954c -rw------- 1 zope zope 37392 Aug 4 17:48 ef358bc52a503dc9302526af3ce1 -rw------- 1 zope zope 1616 Jun 26 2008 f221db1d18f01ebe93caf29fe730 -rw------- 1 zope zope 400 Jan 20 2008 f4c43212930051c45da5163fba15 -rw------- 1 zope zope 10064 Aug 24 2008 f612a2284de15ad71b166132bb8b -rw-rw-r-- 1 zope zope 416704 Dec 28 2007 fbfca80d116b0d092aacd6712260 -rw------- 1 zope zope 640 Nov 26 16:13 fec8f41e8306758f238983948c64 -rw------- 1 zope zope 30224 Mar 16 2008 ffbda6d769a1bb8e73a8e966914f Ideas? Larry _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Seems like you are right. I just ran another test and I am no longer seeing these problems. I did make change to config file (added writebehind translator to client) and remounted GlusterFS in between. It had been hours since the files had been copied to GlusterFS before the first test so if it is transient problems some cached info isn't getting flushed. FYI, Larry -----Original Message----- From: Francois Berenger [mailto:berenger at riken.jp] Sent: Monday, December 21, 2009 6:23 PM To: Larry Bates Subject: Re: [Gluster-users] ls -alR problems on mounted glusterFS I don't know if it is of any help but I have already seen what you describe on NFS mounted volumes. It is like the freshly written files are in a transient state, but after some time all their info is available via ls -l for example.
what operations are you doing on mount point? how easy it is to reproduce memory leak? On Thu, Dec 24, 2009 at 5:08 AM, Larry Bates <larry.bates at vitalesafe.com>wrote:> I restarted the mirror and once again the glusterfs client process is > just growing. Echoing the 3 to /proc/sys/vm/drop_caches eems to shriink the > memory footprint but only by a small amount. > > BTW - it is the client process that seems to grow infinitely. Note: I have > one machine that is acting as both a client and a server (which is where the > client process is growing) and another machine that is a dedicated GFS > server. > > > > Here are my configs: > > > > Client: > > > > # > > # Add client feature and attach to remote subvolumes of gfs001 > > # > > volume gfs001brick1 > > type protocol/client > > option transport-type tcp # for TCP/IP transport > > option remote-host gfs001 # IP address of the remote volume > > option remote-subvolume vol1 # name of the remote volume > > #option transport.socket.nodelay on # undocumented option for speed > > end-volume > > > > volume gfs001brick2 > > type protocol/client > > option transport-type tcp # for TCP/IP transport > > option remote-host gfs001 # IP address of the remote volume > > option remote-subvolume vol2 # name of the remote volume > > #option transport.socket.nodelay on # undocumented option for speed > > end-volume > > > > volume coraidbrick13 > > type protocol/client > > option transport-type tcp > > option remote-host 10.0.0.71 > > option remote-subvolume coraidvol13 > > #option transport.socket.nodelay on > > end-volume > > > > volume coraidbrick14 > > type protocol/client > > option transport-type tcp > > option remote-host 10.0.0.71 > > option remote-subvolume coraidvol14 > > #option transport.socket.nodelay on > > end-volume > > > > # > > # Replicate volumes > > # > > volume afr-vol1 > > type cluster/replicate > > subvolumes gfs001brick1 coraidbrick13 > > end-volume > > > > volume afr-vol2 > > type cluster/replicate > > subvolumes gfs001brick2 coraidbrick14 > > end-volume > > > > # > > # Distribute files across bricks > > # > > volume dht-vol > > type cluster/distribute > > subvolumes afr-vol1 afr-vol2 > > option min-free-disk 2% # 2% of 1.8Tb volumes is 36Gb > > end-volume > > > > # > > # Add quick-read for small files > > # > > volume quickread > > type performance/quick-read > > option cache-timeout 1 # default 1 second > > option max-file-size 256KB # default 64Kb > > subvolumes dht-vol > > end-volume > > > > Server: > > > > volume coraidbrick13 > > type storage/posix # POSIX FS translator > > option directory /mnt/glusterfs/e101.13 # Export this directory > > option background-unlink yes # unlink in background > > end-volume > > > > volume coraidbrick14 > > type storage/posix > > option directory /mnt/glusterfs/e101.14 > > option background-unlink yes > > end-volume > > > > volume iot1 > > type performance/io-threads > > option thread-count 4 > > subvolumes coraidbrick13 > > end-volume > > > > volume iot2 > > type performance/io-threads > > option thread-count 4 > > subvolumes coraidbrick14 > > end-volume > > > > volume coraidvol13 > > type features/locks > > subvolumes iot1 > > end-volume > > > > volume coraidvol14 > > type features/locks > > subvolumes iot2 > > end-volume > > > > ## Add network serving capability to volumes > > volume server > > type protocol/server > > option transport-type tcp # For TCP/IP transport > > subvolumes coraidvol13 coraidvol14 # Expose both volumes > > option auth.addr.coraidvol13.allow 10.0.0.* > > option auth.addr.coraidvol14.allow 10.0.0.* > > end-volume > > > > Thanks, > > Larry > > > > > > > > *From:* raghavendra.hg at gmail.com [mailto:raghavendra.hg at gmail.com] *On > Behalf Of *Raghavendra G > *Sent:* Wednesday, December 23, 2009 2:43 PM > *To:* Larry Bates > *Cc:* gluster-users > *Subject:* Re: [Gluster-users] glusterFS memory leak? > > > > Hi Larry, > > What is the client and server configuration? Instead of killing glusterfs > process, can you do "echo 3 > /proc/sys/vm/drop_caches" and check whether > memory usage comes down? > > regards, > > On Wed, Dec 23, 2009 at 9:01 PM, Larry Bates <larry.bates at vitalesafe.com> > wrote: > > I've successfully set up GlusterFS 3.0 with a single server. I brought on > a 2nd server and setup AFR and have been working through the mirroring > process. I started an "ls -alR" to trigger a complete mirror of all files > between the two servers. After running for about 16 hours I started getting > kernel out of memory errors. Looking at top I see: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 2561 root 15 0 13.0g 8.9g 860 S 7 91.6 60:56.02 glusterfs > > Seems that the client has used up all my memory (RAM and SWAP). Killing > the process returned all the memory to the OS. > > BTW - I'm working on a 2.3Tb store that contains about 2.5 million files in > 65K folders. > > Thoughts? > > Larry Bates > vitalEsafe, Inc. > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > > > > -- > Raghavendra G >-- Raghavendra G
Hi Larry, By any chance did you add the files to backend directory directly (instead of creating the files on mount point)? regards, On Sat, Dec 26, 2009 at 11:41 PM, Larry Bates <larry.bates at vitalesafe.com>wrote:> I'm finding the "recommended" method for healing a broken mirror (AFR) > > just isn't working reliably for me. After doing the following: > > > > ls -alR /mnt/storage/blobdata > > > > I find that not all the files have been copied to the second set of bricks. > Curiously > > MOST of the files have been copied, but a few have not. I've tried > everything I > > > can think of (ls -alR <on a single filename>, touch -a <on a single > filename>) > and > > nothing seems to trigger the remirror. Even running ls -alR a second time > doesn't > > pick up the missing files. The files DO NOT exist on the second set of > bricks. > > > > Thanks in advance, > > Regards, > > Larry Bates > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > >-- Raghavendra G