tbenzvi at 3vgeomatics.com
2014-Dec-27 07:41 UTC
[Gluster-users] Hundreds of duplicate files
Thanks for your continued help Joe. A demonstration of the problem, in this case I was able to open the file in vim (a text file) without any issues, however sometimes duplicated text files open in vim as one line consisting of @ characters, and binary data files can also not be opened correctly for reading. However the duplicate listing is still an issue. Note that Dec 13 was the date of a server crash. [root at jongoo ~]# ll /sar/complete/vancouver/refdem/tif2flt.pro* -rw-rw-r-T 1 parwant users 1712 Dec 13 19:02 tif2flt.pro -rw-rw-r-- 1 parwant users 1712 Jun 17 2010 tif2flt.pro A few minutes later doing the same listing.. sticky bit disappeared and modification date changed [root at jongoo ~]# ll /sar/complete/vancouver/refdem/tif2flt.pro* -rw-rw-r-- 1 parwant users 1712 Jun 17 2010 /sar/complete/vancouver/refdem/tif2flt.pro -rw-rw-r-- 1 parwant users 1712 Jun 17 2010 /sar/complete/vancouver/refdem/tif2flt.pro [root at jongoo ~]# getfattr -m . -d -e hex /data/glusterfs/safari/brick00/brick/complete/vancouver/refdem/tif2flt.pro getfattr: Removing leading '/' from absolute path names # file: data/glusterfs/safari/brick00/brick/complete/vancouver/refdem/tif2flt.pro system.posix_acl_access=0x0200000001000600ffffffff04000600ffffffff10000600ffffffff20000400ffffffff trusted.SGI_ACL_FILE=0x0000000400000001ffffffff0006000000000004ffffffff0006000000000010ffffffff0006000000000020ffffffff00040000 trusted.gfid=0xdfe13dc088bf4a779488ef72f0a879cd trusted.glusterfs.dht.linkto=0x7361666172692d636c69656e742d3300 [root at ndovu ~]# getfattr -m . -d -e hex /data/glusterfs/safari/brick03/brick/complete/vancouver/refdem/tif2flt.pro getfattr: Removing leading '/' from absolute path names # file: data/glusterfs/safari/brick03/brick/complete/vancouver/refdem/tif2flt.pro system.posix_acl_access=0x0200000001000600ffffffff04000600ffffffff10000600ffffffff20000400ffffffff trusted.SGI_ACL_FILE=0x0000000400000001ffffffff0006000000000004ffffffff0006000000000010ffffffff0006000000000020ffffffff00040000 trusted.gfid=0xdfe13dc088bf4a779488ef72f0a879cd Log for brick 00: [2014-12-27 01:59:24.263095] I [server-handshake.c:575:server_setvolume] 0-safari-server: accepted client from jongoo-1910-2014/12/27-01:59:24:173469-safari-client-0-0-0 (version: 3.5.3) [2014-12-27 02:02:00.772454] I [server-handshake.c:575:server_setvolume] 0-safari-server: accepted client from jongoo-2015-2014/12/27-02:01:55:694478-safari-client-0-0-0 (version: 3.5.3) [2014-12-27 02:02:05.780497] I [server-handshake.c:575:server_setvolume] 0-safari-server: accepted client from ndovu-16310-2014/12/27-02:02:00:703051-safari-client-0-0-0 (version: 3.5.3) [2014-12-27 04:41:07.094149] I [server.c:520:server_rpc_notify] 0-safari-server: disconnecting connectionfrom ndovu-16310-2014/12/27-02:02:00:703051-safari-client-0-0-0 [2014-12-27 04:41:07.094187] I [client_t.c:417:gf_client_unref] 0-safari-server: Shutting down connection ndovu-16310-2014/12/27-02:02:00:703051-safari-client-0-0-0 [2014-12-27 04:41:56.979717] I [server.c:520:server_rpc_notify] 0-safari-server: disconnecting connectionfrom jongoo-2015-2014/12/27-02:01:55:694478-safari-client-0-0-0 [2014-12-27 04:41:56.979761] I [client_t.c:417:gf_client_unref] 0-safari-server: Shutting down connection jongoo-2015-2014/12/27-02:01:55:694478-safari-client-0-0-0 Log for brick 03: [2014-12-27 01:59:24.270123] I [server-handshake.c:575:server_setvolume] 0-safari-server: accepted client from jongoo-1910-2014/12/27-01:59:24:173469-safari-client-3-0-0 (version: 3.5.3) [2014-12-27 02:02:05.724212] I [server-handshake.c:575:server_setvolume] 0-safari-server: accepted client from jongoo-2015-2014/12/27-02:01:55:694478-safari-client-3-0-0 (version: 3.5.3) [2014-12-27 02:02:05.778098] I [server-handshake.c:575:server_setvolume] 0-safari-server: accepted client from ndovu-16310-2014/12/27-02:02:00:703051-safari-client-3-0-0 (version: 3.5.3) [2014-12-27 04:41:07.098381] I [server.c:520:server_rpc_notify] 0-safari-server: disconnecting connectionfrom ndovu-16310-2014/12/27-02:02:00:703051-safari-client-3-0-0 [2014-12-27 04:41:07.098417] I [client_t.c:417:gf_client_unref] 0-safari-server: Shutting down connection ndovu-16310-2014/12/27-02:02:00:703051-safari-client-3-0-0 [2014-12-27 04:41:56.984140] I [server.c:520:server_rpc_notify] 0-safari-server: disconnecting connectionfrom jongoo-2015-2014/12/27-02:01:55:694478-safari-client-3-0-0 [2014-12-27 04:41:56.984203] I [client_t.c:417:gf_client_unref] 0-safari-server: Shutting down connection jongoo-2015-2014/12/27-02:01:55:694478-safari-client-3-0-0 Log for mounted volume: [2014-12-27 01:59:24.180253] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.3 (/usr/sbin/glusterfs --volfile-server=ndovu --volfile-id=/safari /sar) [2014-12-27 01:59:24.199613] I [socket.c:3645:socket_init] 0-glusterfs: SSL support is NOT enabled [2014-12-27 01:59:24.199684] I [socket.c:3660:socket_init] 0-glusterfs: using system polling thread [2014-12-27 01:59:24.225167] I [dht-shared.c:314:dht_init_regex] 0-safari-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ [2014-12-27 01:59:24.226442] I [socket.c:3645:socket_init] 0-safari-client-4: SSL support is NOT enabled [2014-12-27 01:59:24.226475] I [socket.c:3660:socket_init] 0-safari-client-4: using system polling thread [2014-12-27 01:59:24.226773] I [socket.c:3645:socket_init] 0-safari-client-3: SSL support is NOT enabled [2014-12-27 01:59:24.226784] I [socket.c:3660:socket_init] 0-safari-client-3: using system polling thread [2014-12-27 01:59:24.227218] I [socket.c:3645:socket_init] 0-safari-client-2: SSL support is NOT enabled [2014-12-27 01:59:24.227239] I [socket.c:3660:socket_init] 0-safari-client-2: using system polling thread [2014-12-27 01:59:24.227650] I [socket.c:3645:socket_init] 0-safari-client-1: SSL support is NOT enabled [2014-12-27 01:59:24.227661] I [socket.c:3660:socket_init] 0-safari-client-1: using system polling thread [2014-12-27 01:59:24.227995] I [socket.c:3645:socket_init] 0-safari-client-0: SSL support is NOT enabled [2014-12-27 01:59:24.228016] I [socket.c:3660:socket_init] 0-safari-client-0: using system polling thread [2014-12-27 01:59:24.228058] I [client.c:2294:notify] 0-safari-client-0: parent translators are ready, attempting connect on transport [2014-12-27 01:59:24.232835] I [client.c:2294:notify] 0-safari-client-1: parent translators are ready, attempting connect on transport [2014-12-27 01:59:24.235585] I [client.c:2294:notify] 0-safari-client-2: parent translators are ready, attempting connect on transport [2014-12-27 01:59:24.240482] I [client.c:2294:notify] 0-safari-client-3: parent translators are ready, attempting connect on transport [2014-12-27 01:59:24.243816] I [client.c:2294:notify] 0-safari-client-4: parent translators are ready, attempting connect on transport Final graph: +------------------------------------------------------------------------------+ 1: volume safari-client-0 2: type protocol/client 3: option remote-host jongoo 4: option remote-subvolume /data/glusterfs/safari/brick00/brick 5: option transport-type socket 6: option send-gids true 7: end-volume 8: 9: volume safari-client-1 10: type protocol/client 11: option remote-host jongoo 12: option remote-subvolume /data/glusterfs/safari/brick01/brick 13: option transport-type socket 14: option send-gids true 15: end-volume 16: 17: volume safari-client-2 18: type protocol/client 19: option remote-host ndovu 20: option remote-subvolume /data/glusterfs/safari/brick02/brick 21: option transport-type socket 22: option send-gids true 23: end-volume 24: 25: volume safari-client-3 26: type protocol/client 27: option remote-host ndovu 28: option remote-subvolume /data/glusterfs/safari/brick03/brick 29: option transport-type socket 30: option send-gids true 31: end-volume 32: 33: volume safari-client-4 34: type protocol/client 35: option remote-host ndovu 36: option remote-subvolume /data/glusterfs/safari/brick04/brick 37: option transport-type socket 38: option send-gids true 39: end-volume 40: 41: volume safari-dht 42: type cluster/distribute 43: subvolumes safari-client-0 safari-client-1 safari-client-2 safari-client-3 safari-client-4 44: end-volume 45: 46: volume safari-write-behind 47: type performance/write-behind 48: subvolumes safari-dht 49: end-volume 50: 51: volume safari-read-ahead 52: type performance/read-ahead 53: subvolumes safari-write-behind 54: end-volume 55: 56: volume safari-io-cache 57: type performance/io-cache 58: subvolumes safari-read-ahead 59: end-volume 60: 61: volume safari-quick-read 62: type performance/quick-read 63: subvolumes safari-io-cache 64: end-volume 65: 66: volume safari-open-behind 67: type performance/open-behind 68: subvolumes safari-quick-read 69: end-volume 70: 71: volume safari-md-cache 72: type performance/md-cache 73: subvolumes safari-open-behind 74: end-volume 75: 76: volume safari 77: type debug/io-stats 78: option latency-measurement off 79: option count-fop-hits off 80: subvolumes safari-md-cache 81: end-volume 82: +------------------------------------------------------------------------------+ [2014-12-27 01:59:24.247044] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-safari-client-0: changing port to 49152 (from 0) [2014-12-27 01:59:24.252049] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-safari-client-1: changing port to 49153 (from 0) [2014-12-27 01:59:24.258273] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-safari-client-2: changing port to 49155 (from 0) [2014-12-27 01:59:24.262939] I [client-handshake.c:1677:select_server_supported_programs] 0-safari-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2014-12-27 01:59:24.263008] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-safari-client-3: changing port to 49156 (from 0) [2014-12-27 01:59:24.263032] I [client-handshake.c:1677:select_server_supported_programs] 0-safari-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2014-12-27 01:59:24.263070] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-safari-client-4: changing port to 49157 (from 0) [2014-12-27 01:59:24.271162] I [client-handshake.c:1462:client_setvolume_cbk] 0-safari-client-0: Connected to 192.168.1.53:49152, attached to remote volume '/data/glusterfs/safari/brick00/brick'. [2014-12-27 01:59:24.271212] I [client-handshake.c:1474:client_setvolume_cbk] 0-safari-client-0: Server and Client lk-version numbers are not same, reopening the fds [2014-12-27 01:59:24.271401] I [client-handshake.c:1462:client_setvolume_cbk] 0-safari-client-1: Connected to 192.168.1.53:49153, attached to remote volume '/data/glusterfs/safari/brick01/brick'. [2014-12-27 01:59:24.271426] I [client-handshake.c:1474:client_setvolume_cbk] 0-safari-client-1: Server and Client lk-version numbers are not same, reopening the fds [2014-12-27 01:59:24.271599] I [client-handshake.c:450:client_set_lk_version_cbk] 0-safari-client-0: Server lk version = 1 [2014-12-27 01:59:24.271617] I [client-handshake.c:450:client_set_lk_version_cbk] 0-safari-client-1: Server lk version = 1 [2014-12-27 01:59:24.271646] I [client-handshake.c:1677:select_server_supported_programs] 0-safari-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2014-12-27 01:59:24.271824] I [client-handshake.c:1677:select_server_supported_programs] 0-safari-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2014-12-27 01:59:24.271894] I [client-handshake.c:1677:select_server_supported_programs] 0-safari-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2014-12-27 01:59:24.272044] I [client-handshake.c:1462:client_setvolume_cbk] 0-safari-client-2: Connected to 192.168.1.51:49155, attached to remote volume '/data/glusterfs/safari/brick02/brick'. [2014-12-27 01:59:24.272088] I [client-handshake.c:1474:client_setvolume_cbk] 0-safari-client-2: Server and Client lk-version numbers are not same, reopening the fds [2014-12-27 01:59:24.272211] I [client-handshake.c:1462:client_setvolume_cbk] 0-safari-client-3: Connected to 192.168.1.51:49156, attached to remote volume '/data/glusterfs/safari/brick03/brick'. [2014-12-27 01:59:24.272233] I [client-handshake.c:1474:client_setvolume_cbk] 0-safari-client-3: Server and Client lk-version numbers are not same, reopening the fds [2014-12-27 01:59:24.272373] I [client-handshake.c:1462:client_setvolume_cbk] 0-safari-client-4: Connected to 192.168.1.51:49157, attached to remote volume '/data/glusterfs/safari/brick04/brick'. [2014-12-27 01:59:24.272396] I [client-handshake.c:1474:client_setvolume_cbk] 0-safari-client-4: Server and Client lk-version numbers are not same, reopening the fds [2014-12-27 01:59:24.278278] I [fuse-bridge.c:5016:fuse_graph_setup] 0-fuse: switched to graph 0 [2014-12-27 01:59:24.278394] I [client-handshake.c:450:client_set_lk_version_cbk] 0-safari-client-2: Server lk version = 1 [2014-12-27 01:59:24.278428] I [client-handshake.c:450:client_set_lk_version_cbk] 0-safari-client-4: Server lk version = 1 [2014-12-27 01:59:24.278502] I [client-handshake.c:450:client_set_lk_version_cbk] 0-safari-client-3: Server lk version = 1 [2014-12-27 01:59:24.278574] I [fuse-bridge.c:3953:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22 --------- Original Message --------- Subject: Re: [Gluster-users] Hundreds of duplicate files From: "Joe Julian" <joe at julianfamily.org> Date: 12/26/14 9:11 pm To: gluster-users at gluster.org Please truncate the client and brick logs then mount the fuse client and demonstrate the error, then share the brick logs and the client log up to that point. Also, share the extended attributes for both the dht linkfile and the actual file. On 12/26/2014 06:34 PM, tbenzvi at 3vgeomatics.com wrote: Hello everyone and happy holidays, I upgraded both servers so that they are now both running Gluster 3.5.3, in fact they are both running Fedora 20 with the same kernel version. We have only one client and that is the first server itself, with plans to change this in the future.. As per a previous suggestion, I also ran xfs_repair on each of the five bricks, which reported no errors. So to recap: Doing a file listing on the mounted Gluster volume shows the same filename appearing twice. Trying to access the file either gives me the link file (and an error trying to read it), or the file on the actual brick location (this is entirely random, sometimes it will not work and then a few seconds later trying to read the file again works). Additionally, there are some cases in which two versions (different content) with the same filename appear on two bricks on the different servers. I would much appreciate it if someone could shed some light on this issue. Best Regards, Tom --------- Original Message --------- Subject: Re: [Gluster-users] Hundreds of duplicate files From: "Joe Julian" <joe at julianfamily.org> Date: 12/21/14 10:34 pm To: gluster-users at gluster.org Have you tried upgrading the older server so all are running the same version? Even though it's supposed to work with mixed versions, the goal should always be to have everything running the same version (clients and servers). On 12/20/2014 09:37 PM, tbenzvi at 3vgeomatics.com wrote: Hi Joe, Thanks for the reply. That worked; I probably forgot to do this as root last time. Yet, the files still show up twice in a directory listing on the mounted volume. And it seems to be random whether reading the file will succeed or not. I've tried with several files and it sometimes works and sometimes fails; I assume this depends on whether it locates the actual file on the brick or the link file. Let me know if you have any idea what's going on. Output of the command: $ getfattr -m . -d -e hex /data/glusterfs/safari/brick01/brick/rsc/tsx/montreal_smaller/sm_asc/stack/slc/20130210.slc.ras getfattr: Removing leading '/' from absolute path names # file: data/glusterfs/safari/brick01/brick/rsc/tsx/montreal_smaller/sm_asc/stack/slc/20130210.slc.ras system.posix_acl_access=0x0200000001000600ffffffff04000600ffffffff10000600ffffffff20000400ffffffff trusted.SGI_ACL_FILE=0x0000000400000001ffffffff0006000000000004ffffffff0006000000000010ffffffff0006000000000020ffffffff00040000 trusted.gfid=0x52c2aed77d09412d8bfd7ca70e87b196 trusted.glusterfs.dht.linkto=0x7361666172692d636c69656e742d3200 Cheers, Tom --------- Original Message --------- Subject: Re: [Gluster-users] Hundreds of duplicate files From: "Joe Julian" <joe at julianfamily.org> Date: 12/20/14 8:53 pm To: gluster-users at gluster.org Try 'getfattr -m . -d -e hex' (dot instead of dash) and, of course, do that as root. On 12/20/2014 06:02 PM, tbenzvi at 3vgeomatics.com wrote: Hi everyone, We have a distributed Gluster volume on five bricks over two servers (first server running gluster 3.4.2, second server running gluster 3.5.1, both running Fedora 20) Starting last week, doing a file listing on the mounted volume shows many files with the same name appearing twice (and they are listed with the same inode). Doing a search for these files, I have found 290,000 of them!! If I do a listing of these files on the bricks themselves, it looks like most are link files (du will show the file on the first server as 0 bytes, and the sticky bit set). The file is fine on the second server. Unfortunately, running "getfattr -m - -e hex -d" on the file shows NO gluster-related attributes and I believe this is why both files appear in the listing. The files cannot be read by any programs as it is trying to read the link file. I assume the metadata became corrupted. This is a production server so we really need to know: 1. How did this happen, and how can we prevent it going forward? There was a server crash a week ago and I believe that was the cause. 2. How can we heal the Gluster volume/bricks and link files. If there is some straightforward way of restoring the link file pointer I can write a script to do it, obviously doing this manually will be impossible. Thanks very much for any and all help - much appreciated! Regards, Tom On Wed, Dec 17, 2014 at 4:07 AM, <tbenzvi at 3vgeomatics.com> wrote:> Hi everyone, we have noticed some extremely odd behaviour with our> distributed Gluster volume where duplicate files (same name, same or > different content) are being created and stored on multiple bricks. The only > consistent clue is that one of the duplicate files has the sticky bit set. I > am hoping someone will be able to shed some light on why this is happening > and how we can restore the volume as there appear to be hundreds of such > files. I will try to provide as much pertinent information as I can. > > We have a 130TB Gluster volume consisting of two 20TB bricks on server1, and > three 40TB bricks on a server2 which were added at a later date (and > rebalancing was done). The volume is mounted on server1, and accessed only > through this server but by many users. Both servers went down due to power > loss several days ago after which this problem was first noticed. We ran a > rebalance command on the volumes, this has not fixed the problem. > > > Gluster volume info: > Volume Name: safari > Type: Distribute > Volume ID: d48d0e6b-4389-4c2c-8fd1-cd2854121eda > Status: Started > Number of Bricks: 5 > Transport-type: tcp > Bricks: > Brick1: server1:/data/glusterfs/safari/brick00/brick > Brick2: server1:/data/glusterfs/safari/brick01/brick > Brick3: server2:/data/glusterfs/safari/brick02/brick > Brick4: server2:/data/glusterfs/safari/brick03/brick > Brick5: server2:/data/glusterfs/safari/brick04/brick > > > Size information: > /dev/sdc 37T 16T 22T 42% /data/glusterfs/safari/brick02 > /dev/sdd 37T 16T 22T 42% /data/glusterfs/safari/brick03 > /dev/sde 37T 17T 21T 45% /data/glusterfs/safari/brick04 > /dev/md126 11T 7.7T 2.8T 74% /data/glusterfs/safari/brick00 > /dev/md124 11T 8.0T 2.5T 77% /data/glusterfs/safari/brick01 > server2:/safari 130T 63T 68T 48% /sar > > > Example 1: > -Two files with the same name exist in one directory > -They have different contents and attributes > -A file listing on the mounted volume shows the same inode > -The newer file has sticky bit set > -Neither file is corrupted, they can both be viewed by using the absolute > path (on the bricks) > > File listing on the mounted volume > 13036730497538635177 -rw-rw-r-T 1 jon users 924 Dec 15 10:42 RSLC_tab > 13036730497538635177 -rw-rw-r-- 1 jon users 418 Mar 18 2013 RSLC_tab > > Listing of the files on the bricks: > 8925798411 -rw-rw-r-T+ 2 jon users 924 Dec 15 10:42 > /data/glusterfs/safari/brick00/brick/complete/shm/rs2/ottawa/mf6_asc/stack_org/RSLC_tab > 51541886672 -rw-rw-r--+ 2 1002 users 418 Mar 18 2013 > /data/glusterfs/safari/brick02/brick/complete/shm/rs2/ottawa/mf6_asc/stack_org/RSLC_tab > > > Example 2: > -Two files with the same name exist in one directory > -They have the same content and attributes > -No sticky bit is set when looking at file listing on the mounted volume > -Sticky bit is set for one while when looking at file listing on the bricks > -Files are corrupted > > File listing on the mounted volume: > 13012555852904096080 -rw-rw-r-- 1 tom users 2393848 Dec 8 2013 > ifg_lr/20130226_20130813.diff.phi.ras > 13012555852904096080 -rw-rw-r-- 1 tom users 2393848 Dec 8 2013 > ifg_lr/20130226_20130813.diff.phi.ras > > Listing of the files on the bricks: > 17058578 -rw-rw-r-T+ 2 tom users 2393848 Dec 13 17:11 > /data/glusterfs/safari/brick00/brick/rsc/rs2/calgary/u22_dsc/stack_org/ifg_lr/20130226_20130813.diff.phi.ras > 57986922129 -rw-rw-r--+ 2 1010 users 2393848 Dec 8 2013 > /data/glusterfs/safari/brick02/brick/rsc/rs2/calgary/u22_dsc/stack_org/ifg_lr/20130226_20130813.diff.phi.ras > > > Additionally, only some files in this directory are duplicated. The > duplicated files are corrupted (can not be viewed as Raster images: the > original file type) > The files which are not duplicated are not corrupted. > > File command: (notice duplicate and singleton files) > ifg_lr/20091021_20100218.diff.phi.ras: Sun raster image data, 1208 x 1981, > 8-bit, RGB colormap > ifg_lr/20091021_20101016.diff.phi.ras: data > ifg_lr/20091021_20101016.diff.phi.ras: data > ifg_lr/20091021_20101109.diff.phi.ras: Sun raster image data, 1208 x 1981, > 8-bit, RGB colormap > ifg_lr/20091021_20101203.diff.phi.ras: Sun raster image data, 1208 x 1981, > 8-bit, RGB colormap > ifg_lr/20091021_20101227.diff.phi.ras: Sun raster image data, 1208 x 1981, > 8-bit, RGB colormap > ifg_lr/20091021_20110120.diff.phi.ras: Sun raster image data, 1208 x 1981, > 8-bit, RGB colormap > ifg_lr/20091021_20110213.diff.phi.ras: data > ifg_lr/20091021_20110213.diff.phi.ras: data > ifg_lr/20091021_20110309.diff.phi.ras: data > ifg_lr/20091021_20110309.diff.phi.ras: sticky data > ifg_lr/20091021_20110402.diff.phi.ras: Sun raster image data, 1208 x 1981, > 8-bit, RGB colormap _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://www.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20141227/38d2adf1/attachment.html>
On 12/27/2014 01:11 PM, tbenzvi at 3vgeomatics.com wrote:> Thanks for your continued help Joe. > A demonstration of the problem, in this case I was able to open the file > in vim (a text file) without any issues, however sometimes duplicated > text files open in vim as one line consisting of @ characters, and > binary data files can also not be opened correctly for reading. > However the duplicate listing is still an issue. Note that Dec 13 was > the date of a server crash. > > [root at jongoo ~]# ll /sar/complete/vancouver/refdem/tif2flt.pro* > -rw-rw-r-T 1 parwant users 1712 Dec 13 19:02 tif2flt.pro > -rw-rw-r-- 1 parwant users 1712 Jun 17 2010 tif2flt.pro > > A few minutes later doing the same listing.. sticky bit disappeared and > modification date changed > > [root at jongoo ~]# ll /sar/complete/vancouver/refdem/tif2flt.pro* > -rw-rw-r-- 1 parwant users 1712 Jun 17 2010 > /sar/complete/vancouver/refdem/tif2flt.pro > -rw-rw-r-- 1 parwant users 1712 Jun 17 2010 > /sar/complete/vancouver/refdem/tif2flt.pro > > [root at jongoo ~]# getfattr -m . -d -e hex > /data/glusterfs/safari/brick00/brick/complete/vancouver/refdem/tif2flt.pro > getfattr: Removing leading '/' from absolute path names > # file: > data/glusterfs/safari/brick00/brick/complete/vancouver/refdem/tif2flt.pro > system.posix_acl_access=0x0200000001000600ffffffff04000600ffffffff10000600ffffffff20000400ffffffff > trusted.SGI_ACL_FILE=0x0000000400000001ffffffff0006000000000004ffffffff0006000000000010ffffffff0006000000000020ffffffff00040000 > trusted.gfid=0xdfe13dc088bf4a779488ef72f0a879cd > trusted.glusterfs.dht.linkto=0x7361666172692d636c69656e742d3300 > > [root at ndovu ~]# getfattr -m . -d -e hex > /data/glusterfs/safari/brick03/brick/complete/vancouver/refdem/tif2flt.pro > getfattr: Removing leading '/' from absolute path names > # file: > data/glusterfs/safari/brick03/brick/complete/vancouver/refdem/tif2flt.pro > system.posix_acl_access=0x0200000001000600ffffffff04000600ffffffff10000600ffffffff20000400ffffffff > trusted.SGI_ACL_FILE=0x0000000400000001ffffffff0006000000000004ffffffff0006000000000010ffffffff0006000000000020ffffffff00040000 > trusted.gfid=0xdfe13dc088bf4a779488ef72f0a879cd >Is rebalance running on this volume right now? If not, can you please move out the file copy with "trusted.glusterfs.dht.linkto" attribute out of the brick directory (/data/glusterfs/safari/brick00/brick/complete/vancouver/refdem/tif2flt.pro) to an alternate location & check the behavior? Thanks, Vijay