I rsync'd 20-TB over to my gluster system and noticed that I had some directories missing even though the rsync completed normally. The rsync logs showed that the missing files were transferred. I went to the bricks and did an 'ls -al /data/brick*/homegfs/dir/*' the files were on the bricks. After I did this 'ls', the files then showed up on the FUSE mounts. 1) Why are the files hidden on the fuse mount? 2) Why does the ls make them show up on the FUSE mount? 3) How can I prevent this from happening again? Note, I also mounted the gluster volume using NFS and saw the same behavior. The files/directories were not shown until I did the "ls" on the bricks. David ==============================David F. Robinson, Ph.D. President - Corvid Technologies 704.799.6944 x101 [office] 704.252.1310 [cell] 704.799.7974 [fax] David.Robinson at corvidtec.com http://www.corvidtechnologies.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150203/295d8d7c/attachment.html>
It sounds to me like the files were only copied to one replica, werent there for the initial for the initial ls which triggered a self heal, and were there for the last ls because they were healed. Is there any chance that one of the replicas was down during the rsync? It could be that you lost a brick during copy or something like that. To confirm I would look for disconnects in the brick logs as well as checking glusterfshd.log to verify the missing files were actually healed. -b On Tue, Feb 3, 2015 at 5:37 PM, David F. Robinson < david.robinson at corvidtec.com> wrote:> I rsync'd 20-TB over to my gluster system and noticed that I had some > directories missing even though the rsync completed normally. > The rsync logs showed that the missing files were transferred. > > I went to the bricks and did an 'ls -al /data/brick*/homegfs/dir/*' the > files were on the bricks. After I did this 'ls', the files then showed up > on the FUSE mounts. > > 1) Why are the files hidden on the fuse mount? > 2) Why does the ls make them show up on the FUSE mount? > 3) How can I prevent this from happening again? > > Note, I also mounted the gluster volume using NFS and saw the same > behavior. The files/directories were not shown until I did the "ls" on the > bricks. > > David > > > > ==============================> David F. Robinson, Ph.D. > President - Corvid Technologies > 704.799.6944 x101 [office] > 704.252.1310 [cell] > 704.799.7974 [fax] > David.Robinson at corvidtec.com > http://www.corvidtechnologies.com > > > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150203/f2219a26/attachment.html>
Like these? data-brick02a-homegfs.log:[2015-02-03 19:09:34.568842] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs02a.corvidtec.com-18563-2015/02/03-19:07:58:519134-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-03 19:09:41.286551] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01a.corvidtec.com-12804-2015/02/03-19:09:38:497808-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-03 19:16:35.906412] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs02b.corvidtec.com-27190-2015/02/03-19:15:53:458467-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-03 19:51:22.761293] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01a.corvidtec.com-25926-2015/02/03-19:51:02:89070-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-03 20:54:02.772180] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01b.corvidtec.com-4175-2015/02/02-16:44:31:179119-homegfs-client-2-0-1 data-brick02a-homegfs.log:[2015-02-03 22:44:47.458905] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01a.corvidtec.com-29467-2015/02/03-22:44:05:838129-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-03 22:47:42.830866] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01a.corvidtec.com-30069-2015/02/03-22:47:37:209436-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-03 22:48:26.785931] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01a.corvidtec.com-30256-2015/02/03-22:47:55:203659-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-03 22:53:25.530836] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01a.corvidtec.com-30658-2015/02/03-22:53:21:627538-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-03 22:56:14.033823] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01a.corvidtec.com-30893-2015/02/03-22:56:01:450507-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-03 22:56:55.622800] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01a.corvidtec.com-31080-2015/02/03-22:56:32:665370-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-03 22:59:11.445742] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01a.corvidtec.com-31383-2015/02/03-22:58:45:190874-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-03 23:06:26.482709] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01a.corvidtec.com-31720-2015/02/03-23:06:11:340012-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-03 23:10:54.807725] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01a.corvidtec.com-32083-2015/02/03-23:10:22:131678-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-03 23:13:35.545513] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01a.corvidtec.com-32284-2015/02/03-23:13:21:26552-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-03 23:14:19.065271] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01a.corvidtec.com-32471-2015/02/03-23:13:48:221126-homegfs-client-2-0-0 data-brick02a-homegfs.log:[2015-02-04 00:18:20.261428] I [server.c:518:server_rpc_notify] 0-homegfs-server: disconnecting connection from gfs01a.corvidtec.com-1369-2015/02/04-00:16:53:613570-homegfs-client-2-0-0 ------ Original Message ------ From: "Benjamin Turner" <bennyturns at gmail.com> To: "David F. Robinson" <david.robinson at corvidtec.com> Cc: "Gluster Devel" <gluster-devel at gluster.org>; "gluster-users at gluster.org" <gluster-users at gluster.org> Sent: 2/3/2015 7:12:34 PM Subject: Re: [Gluster-devel] missing files>It sounds to me like the files were only copied to one replica, werent >there for the initial for the initial ls which triggered a self heal, >and were there for the last ls because they were healed. Is there any >chance that one of the replicas was down during the rsync? It could be >that you lost a brick during copy or something like that. To confirm I >would look for disconnects in the brick logs as well as checking >glusterfshd.log to verify the missing files were actually healed. > >-b > >On Tue, Feb 3, 2015 at 5:37 PM, David F. Robinson ><david.robinson at corvidtec.com> wrote: >>I rsync'd 20-TB over to my gluster system and noticed that I had some >>directories missing even though the rsync completed normally. >>The rsync logs showed that the missing files were transferred. >> >>I went to the bricks and did an 'ls -al /data/brick*/homegfs/dir/*' >>the files were on the bricks. After I did this 'ls', the files then >>showed up on the FUSE mounts. >> >>1) Why are the files hidden on the fuse mount? >>2) Why does the ls make them show up on the FUSE mount? >>3) How can I prevent this from happening again? >> >>Note, I also mounted the gluster volume using NFS and saw the same >>behavior. The files/directories were not shown until I did the "ls" >>on the bricks. >> >>David >> >> >> >>==============================>>David F. Robinson, Ph.D. >>President - Corvid Technologies >>704.799.6944 x101 [office] >>704.252.1310 [cell] >>704.799.7974 [fax] >>David.Robinson at corvidtec.com >>http://www.corvidtechnologies.com >> >> >> >>_______________________________________________ >>Gluster-devel mailing list >>Gluster-devel at gluster.org >>http://www.gluster.org/mailman/listinfo/gluster-devel >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150204/af5deb6e/attachment.html>
Sorry. Thought about this a little more. I should have been clearer.
The files were on both bricks of the replica, not just one side. So,
both bricks had to have been up... The files/directories just don't show
up on the mount.
I was reading and saw a related bug
(https://bugzilla.redhat.com/show_bug.cgi?id=1159484). I saw it
suggested to run:
find <mount> -d -exec getfattr -h -n trusted.ec.heal {} \;
I get a bunch of errors for operation not supported:
[root at gfs02a homegfs]# find wks_backup -d -exec getfattr -h -n
trusted.ec.heal {} \;
find: warning: the -d option is deprecated; please use -depth instead,
because the latter is a POSIX-compliant feature.
wks_backup/homer_backup/backup: trusted.ec.heal: Operation not supported
wks_backup/homer_backup/logs/2014_05_20.log: trusted.ec.heal: Operation
not supported
wks_backup/homer_backup/logs/2014_05_21.log: trusted.ec.heal: Operation
not supported
wks_backup/homer_backup/logs/2014_05_18.log: trusted.ec.heal: Operation
not supported
wks_backup/homer_backup/logs/2014_05_19.log: trusted.ec.heal: Operation
not supported
wks_backup/homer_backup/logs/2014_05_22.log: trusted.ec.heal: Operation
not supported
wks_backup/homer_backup/logs: trusted.ec.heal: Operation not supported
wks_backup/homer_backup: trusted.ec.heal: Operation not supported
------ Original Message ------
From: "Benjamin Turner" <bennyturns at gmail.com>
To: "David F. Robinson" <david.robinson at corvidtec.com>
Cc: "Gluster Devel" <gluster-devel at gluster.org>;
"gluster-users at gluster.org" <gluster-users at gluster.org>
Sent: 2/3/2015 7:12:34 PM
Subject: Re: [Gluster-devel] missing files
>It sounds to me like the files were only copied to one replica, werent
>there for the initial for the initial ls which triggered a self heal,
>and were there for the last ls because they were healed. Is there any
>chance that one of the replicas was down during the rsync? It could be
>that you lost a brick during copy or something like that. To confirm I
>would look for disconnects in the brick logs as well as checking
>glusterfshd.log to verify the missing files were actually healed.
>
>-b
>
>On Tue, Feb 3, 2015 at 5:37 PM, David F. Robinson
><david.robinson at corvidtec.com> wrote:
>>I rsync'd 20-TB over to my gluster system and noticed that I had
some
>>directories missing even though the rsync completed normally.
>>The rsync logs showed that the missing files were transferred.
>>
>>I went to the bricks and did an 'ls -al
/data/brick*/homegfs/dir/*'
>>the files were on the bricks. After I did this 'ls', the files
then
>>showed up on the FUSE mounts.
>>
>>1) Why are the files hidden on the fuse mount?
>>2) Why does the ls make them show up on the FUSE mount?
>>3) How can I prevent this from happening again?
>>
>>Note, I also mounted the gluster volume using NFS and saw the same
>>behavior. The files/directories were not shown until I did the
"ls"
>>on the bricks.
>>
>>David
>>
>>
>>
>>==============================>>David F. Robinson, Ph.D.
>>President - Corvid Technologies
>>704.799.6944 x101 [office]
>>704.252.1310 [cell]
>>704.799.7974 [fax]
>>David.Robinson at corvidtec.com
>>http://www.corvidtechnologies.com
>>
>>
>>
>>_______________________________________________
>>Gluster-devel mailing list
>>Gluster-devel at gluster.org
>>http://www.gluster.org/mailman/listinfo/gluster-devel
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150204/1089c128/attachment.html>