thr3ads.net - Gluster users - [Gluster-users] Issue with files on glusterfs becoming unreadable. [Jun 2009]

If this information is useful, please help other people find it:
Share via:

Elbert Lai

2009-Jun-11 17:33 UTC

[Gluster-users] Issue with files on glusterfs becoming unreadable.

elbert at host1:~$ dpkg -l|grep glusterfs
ii  glusterfs-client                   
1.3.8-0pre2                          GlusterFS fuse client
ii  glusterfs-server                   
1.3.8-0pre2                          GlusterFS fuse server
ii  libglusterfs0                      
1.3.8-0pre2                          GlusterFS libraries and  
translator modules

I have 2 hosts set up to use AFR with the package versions listed  
above. I have been experiencing an issue where a file that is copied  
to glusterfs is readable/writable for a while, then at some point it  
time, it ceases to be. Trying to access it only retrieves the error  
message, "cannot open `filename' for reading: Input/output error".

Files enter glusterfs either via the "cp" command from a client or via
"rsync". In the case of cp, the clients are all local and copying  
across a very fast connection. In the case of rsync, the 1 client is  
itself a gluster client. We are testing out a later version of  
gluster, and it rsync's across a vpn.

elbert at host2:~$ dpkg -l|grep glusterfs
ii  glusterfs-client                    2.0.1-1     clustered file- 
system
ii  glusterfs-server                    2.0.1-1     clustered file- 
system
ii  libglusterfs0                       2.0.1-1     GlusterFS  
libraries and translator modules
ii  libglusterfsclient0                 2.0.1-1     GlusterFS client  
library

========What causes files to become inaccessible? I read that fstat() had a  
bug in version 1.3.x whereas stat() did not, and that it was being  
worked on. Could this be related?

When a file becomes inaccessible, I have been manually removing the  
file from the mount point, then copying it back in via scp. Then the  
file becomes accessible. Below I've pasted a sample of what I'm seeing.
> elbert at tool3.:hourlogs$ cd myDir
> ls 1244682000.log
> elbert at tool3.:myDir$ ls 1244682000.log
> 1244682000.log
> elbert at tool3.:myDir$ stat 1244682000.log
>   File: `1244682000.log'
>   Size: 40265114  	Blocks: 78744      IO Block: 4096   regular file
> Device: 15h/21d	Inode: 42205749    Links: 1
> Access: (0755/-rwxr-xr-x)  Uid: ( 1003/   elbert)   Gid: ( 6000/      
> ops)
> Access: 2009-06-11 02:25:10.000000000 +0000
> Modify: 2009-06-11 02:26:02.000000000 +0000
> Change: 2009-06-11 02:26:02.000000000 +0000
> elbert at tool3.:myDir$ tail 1244682000.log
> tail: cannot open `1244682000.log' for reading: Input/output error
At this point, I am able to rm the file. Then, if I scp it back in, I  
am able to successfully tail it.

So,

I have observed cases where the files had a Size of 0, and otherwise  
they were in the same state. I'm not totally certain, but it looks  
like if a file gets into this state from rsync, either it gets  
deposited in this state immediately (before I try to read it), or else  
it quickly enters this state. Speaking generally, file sizes tend to  
be several MB up to 150 MB.

Here's my server config:
# Gluster Server configuration /etc/glusterfs/glusterfs-server.vol
# Configured for AFR & Unify features

volume brick
  type storage/posix
  option directory /var/gluster/data/
end-volume

volume brick-ns
  type storage/posix
  option directory /var/gluster/ns/
end-volume

volume server
  type protocol/server
  option transport-type tcp/server
  subvolumes brick brick-ns
  option auth.ip.brick.allow 165.193.245.*,10.11.*
  option auth.ip.brick-ns.allow 165.193.245.*,10.11.*
end-volume

Here's my client config:
# Gluster Client configuration /etc/glusterfs/glusterfs-client.vol
# Configured for AFR & Unify features

volume brick1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
  option remote-host 10.11.16.68    # IP address of the remote brick
  option remote-subvolume brick        # name of the remote volume
end-volume

volume brick2
  type protocol/client
  option transport-type tcp/client
  option remote-host 10.11.16.71
  option remote-subvolume brick
end-volume

volume brick3
  type protocol/client
  option transport-type tcp/client
  option remote-host 10.11.16.69
  option remote-subvolume brick
end-volume

volume brick4
  type protocol/client
  option transport-type tcp/client
  option remote-host 10.11.16.70
  option remote-subvolume brick
end-volume

volume brick5
  type protocol/client
  option transport-type tcp/client
  option remote-host 10.11.16.119
  option remote-subvolume brick
end-volume

volume brick6
  type protocol/client
  option transport-type tcp/client
  option remote-host 10.11.16.120
  option remote-subvolume brick
end-volume

volume brick-ns1
  type protocol/client
  option transport-type tcp/client
  option remote-host 10.11.16.68
  option remote-subvolume brick-ns  # Note the different remote volume  
name.
end-volume

volume brick-ns2
  type protocol/client
  option transport-type tcp/client
  option remote-host 10.11.16.71
  option remote-subvolume brick-ns  # Note the different remote volume  
name.
end-volume

volume afr1
  type cluster/afr
  subvolumes brick1 brick2
end-volume

volume afr2
  type cluster/afr
  subvolumes brick3 brick4
end-volume

volume afr3
  type cluster/afr
  subvolumes brick5 brick6
end-volume

volume afr-ns
  type cluster/afr
  subvolumes brick-ns1 brick-ns2
end-volume

volume unify
  type cluster/unify
  subvolumes afr1 afr2 afr3
  option namespace afr-ns

  # use the ALU scheduler
  option scheduler alu

  # This option makes brick5 to be readonly, where no new files are  
created.
  ##option alu.read-only-subvolumes brick5##

  # Don't create files one a volume with less than 5% free diskspace
  option alu.limits.min-free-disk  10%

  # Don't create files on a volume with more than 10000 files open
  option alu.limits.max-open-files 10000

  # When deciding where to place a file, first look at the disk-usage,  
then at
  # read-usage, write-usage, open files, and finally the disk-speed- 
usage.
  option alu.order disk-usage:read-usage:write-usage:open-files- 
usage:disk-speed-usage

  # Kick in if the discrepancy in disk-usage between volumes is more  
than 2GB
  option alu.disk-usage.entry-threshold 2GB

  # Don't stop writing to the least-used volume until the discrepancy  
is 1988MB
  option alu.disk-usage.exit-threshold  60MB

  # Kick in if the discrepancy in open files is 1024
  option alu.open-files-usage.entry-threshold 1024

  # Don't stop until 992 files have been written the least-used volume
  option alu.open-files-usage.exit-threshold 32

  # Kick in when the read-usage discrepancy is 20%
  option alu.read-usage.entry-threshold 20%

  # Don't stop until the discrepancy has been reduced to 16% (20% - 4%)
  option alu.read-usage.exit-threshold 4%

  # Kick in when the write-usage discrepancy is 20%
  option alu.write-usage.entry-threshold 20%

## Don't stop until the discrepancy has been reduced to 16%
  option alu.write-usage.exit-threshold 4%

  # Refresh the statistics used for decision-making every 10 seconds
  option alu.stat-refresh.interval 10sec

# Refresh the statistics used for decision-making after creating 10  
files
# option alu.stat-refresh.num-file-create 10
end-volume


#writebehind improves write performance a lot
volume writebehind
   type performance/write-behind
   option aggregate-size 131072 # in bytes
   subvolumes unify
end-volume

Has anyone seen this issue before? Any suggestions?

Thanks,
-elb-
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090611/da7e3a20/attachment.html>

Elbert Lai

2009-Jun-11 18:03 UTC

head link

[Gluster-users] Issue with files on glusterfs becoming unreadable.

Also, in the logfiles on the clients, it looks like I get these types  
of messages whenever I try to access a file that is no longer  
accessible.

2009-06-11 07:58:24 E [fuse-bridge.c:675:fuse_fd_cbk] glusterfs-fuse:  
22068570: /hourlogs/myDir0/1243432800.log => -1 (5)
2009-06-11 07:58:24 E [fuse-bridge.c:436:fuse_entry_cbk] glusterfs- 
fuse: 22068579: /hourlogs/myDir1/1243400400.log => -1 (116)
2009-06-11 07:58:24 E [unify.c:850:unify_open] unify: /hourlogs/ 
myDir1/1243400400.log: entry_count is 3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir1/1243400400.log: found on afr3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir1/1243400400.log: found on afr2
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir1/1243400400.log: found on afr-ns
2009-06-11 07:58:24 E [fuse-bridge.c:675:fuse_fd_cbk] glusterfs-fuse:  
22068580: /hourlogs/myDir1/1243400400.log => -1 (5)
2009-06-11 07:58:24 E [fuse-bridge.c:436:fuse_entry_cbk] glusterfs- 
fuse: 22068583: /hourlogs/myDir2/1243411200.log => -1 (116)
2009-06-11 07:58:24 E [unify.c:850:unify_open] unify: /hourlogs/ 
myDir2/1243411200.log: entry_count is 3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir2/1243411200.log: found on afr1
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir2/1243411200.log: found on afr3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir2/1243411200.log: found on afr-ns
2009-06-11 07:58:24 E [fuse-bridge.c:675:fuse_fd_cbk] glusterfs-fuse:  
22068584: /hourlogs/myDir2/1243411200.log => -1 (5)
2009-06-11 07:58:24 E [fuse-bridge.c:436:fuse_entry_cbk] glusterfs- 
fuse: 22068599: /hourlogs/myDir3/1243472400.log => -1 (116)
2009-06-11 07:58:24 E [unify.c:850:unify_open] unify: /hourlogs/ 
myDir3/1243472400.log: entry_count is 3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir3/1243472400.log: found on afr1
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir3/1243472400.log: found on afr3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir3/1243472400.log: found on afr-ns
2009-06-11 07:58:24 E [fuse-bridge.c:675:fuse_fd_cbk] glusterfs-fuse:  
22068600: /hourlogs/myDir3/1243472400.log => -1 (5)
2009-06-11 07:58:24 E [fuse-bridge.c:436:fuse_entry_cbk] glusterfs- 
fuse: 22068603: /hourlogs/myDir4/1243404000.log => -1 (116)
2009-06-11 07:58:24 E [unify.c:850:unify_open] unify: /hourlogs/ 
myDir4/1243404000.log: entry_count is 3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir4/1243404000.log: found on afr1
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir4/1243404000.log: found on afr-ns
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir4/1243404000.log: found on afr3
2009-06-11 07:58:24 E [fuse-bridge.c:675:fuse_fd_cbk] glusterfs-fuse:  
22068604: /hourlogs/myDir5/1243404000.log => -1 (5)
2009-06-11 07:58:24 E [fuse-bridge.c:436:fuse_entry_cbk] glusterfs- 
fuse: 22068619: /hourlogs/myDir5/1243447200.log => -1 (116)
2009-06-11 07:58:24 E [unify.c:850:unify_open] unify: /hourlogs/ 
myDir5/1243447200.log: entry_count is 4
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir5/1243447200.log: found on afr1
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir5/1243447200.log: found on afr3
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir5/1243447200.log: found on afr2
2009-06-11 07:58:24 E [unify.c:853:unify_open] unify: /hourlogs/ 
myDir5/1243447200.log: found on afr-ns
2009-06-11 07:58:24 E [fuse-bridge.c:675:fuse_fd_cbk] glusterfs-fuse:  
22068620: /hourlogs/myDir5/1243447200.log => -1 (5)

On Jun 11, 2009, at 10:33 AM, Elbert Lai wrote:
> elbert at host1:~$ dpkg -l|grep glusterfs
> ii  glusterfs-client                   
> 1.3.8-0pre2                          GlusterFS fuse client
> ii  glusterfs-server                   
> 1.3.8-0pre2                          GlusterFS fuse server
> ii  libglusterfs0                      
> 1.3.8-0pre2                          GlusterFS libraries and  
> translator modules
>
> I have 2 hosts set up to use AFR with the package versions listed  
> above. I have been experiencing an issue where a file that is copied  
> to glusterfs is readable/writable for a while, then at some point it  
> time, it ceases to be. Trying to access it only retrieves the error  
> message, "cannot open `filename' for reading: Input/output
error".
>
> Files enter glusterfs either via the "cp" command from a client
or
> via "rsync". In the case of cp, the clients are all local and  
> copying across a very fast connection. In the case of rsync, the 1  
> client is itself a gluster client. We are testing out a later  
> version of gluster, and it rsync's across a vpn.
>
> elbert at host2:~$ dpkg -l|grep glusterfs
> ii  glusterfs-client                    2.0.1-1     clustered file- 
> system
> ii  glusterfs-server                    2.0.1-1     clustered file- 
> system
> ii  libglusterfs0                       2.0.1-1     GlusterFS  
> libraries and translator modules
> ii  libglusterfsclient0                 2.0.1-1     GlusterFS client  
> library
>
> ========> What causes files to become inaccessible? I read that fstat()
had a
> bug in version 1.3.x whereas stat() did not, and that it was being  
> worked on. Could this be related?
>
> When a file becomes inaccessible, I have been manually removing the  
> file from the mount point, then copying it back in via scp. Then the  
> file becomes accessible. Below I've pasted a sample of what I'm  
> seeing.
>
>> elbert at tool3.:hourlogs$ cd myDir
>> ls 1244682000.log
>> elbert at tool3.:myDir$ ls 1244682000.log
>> 1244682000.log
>> elbert at tool3.:myDir$ stat 1244682000.log
>>   File: `1244682000.log'
>>   Size: 40265114  	Blocks: 78744      IO Block: 4096   regular file
>> Device: 15h/21d	Inode: 42205749    Links: 1
>> Access: (0755/-rwxr-xr-x)  Uid: ( 1003/   elbert)   Gid:  
>> ( 6000/     ops)
>> Access: 2009-06-11 02:25:10.000000000 +0000
>> Modify: 2009-06-11 02:26:02.000000000 +0000
>> Change: 2009-06-11 02:26:02.000000000 +0000
>> elbert at tool3.:myDir$ tail 1244682000.log
>> tail: cannot open `1244682000.log' for reading: Input/output error
>
> At this point, I am able to rm the file. Then, if I scp it back in,  
> I am able to successfully tail it.
>
> So,
>
> I have observed cases where the files had a Size of 0, and otherwise  
> they were in the same state. I'm not totally certain, but it looks  
> like if a file gets into this state from rsync, either it gets  
> deposited in this state immediately (before I try to read it), or  
> else it quickly enters this state. Speaking generally, file sizes  
> tend to be several MB up to 150 MB.
>
> Here's my server config:
> # Gluster Server configuration /etc/glusterfs/glusterfs-server.vol
> # Configured for AFR & Unify features
>
> volume brick
>  type storage/posix
>  option directory /var/gluster/data/
> end-volume
>
> volume brick-ns
>  type storage/posix
>  option directory /var/gluster/ns/
> end-volume
>
> volume server
>  type protocol/server
>  option transport-type tcp/server
>  subvolumes brick brick-ns
>  option auth.ip.brick.allow 165.193.245.*,10.11.*
>  option auth.ip.brick-ns.allow 165.193.245.*,10.11.*
> end-volume
>
> Here's my client config:
> # Gluster Client configuration /etc/glusterfs/glusterfs-client.vol
> # Configured for AFR & Unify features
>
> volume brick1
>  type protocol/client
>  option transport-type tcp/client     # for TCP/IP transport
>  option remote-host 10.11.16.68    # IP address of the remote brick
>  option remote-subvolume brick        # name of the remote volume
> end-volume
>
> volume brick2
>  type protocol/client
>  option transport-type tcp/client
>  option remote-host 10.11.16.71
>  option remote-subvolume brick
> end-volume
>
> volume brick3
>  type protocol/client
>  option transport-type tcp/client
>  option remote-host 10.11.16.69
>  option remote-subvolume brick
> end-volume
>
> volume brick4
>  type protocol/client
>  option transport-type tcp/client
>  option remote-host 10.11.16.70
>  option remote-subvolume brick
> end-volume
>
> volume brick5
>  type protocol/client
>  option transport-type tcp/client
>  option remote-host 10.11.16.119
>  option remote-subvolume brick
> end-volume
>
> volume brick6
>  type protocol/client
>  option transport-type tcp/client
>  option remote-host 10.11.16.120
>  option remote-subvolume brick
> end-volume
>
> volume brick-ns1
>  type protocol/client
>  option transport-type tcp/client
>  option remote-host 10.11.16.68
>  option remote-subvolume brick-ns  # Note the different remote  
> volume name.
> end-volume
>
> volume brick-ns2
>  type protocol/client
>  option transport-type tcp/client
>  option remote-host 10.11.16.71
>  option remote-subvolume brick-ns  # Note the different remote  
> volume name.
> end-volume
>
> volume afr1
>  type cluster/afr
>  subvolumes brick1 brick2
> end-volume
>
> volume afr2
>  type cluster/afr
>  subvolumes brick3 brick4
> end-volume
>
> volume afr3
>  type cluster/afr
>  subvolumes brick5 brick6
> end-volume
>
> volume afr-ns
>  type cluster/afr
>  subvolumes brick-ns1 brick-ns2
> end-volume
>
> volume unify
>  type cluster/unify
>  subvolumes afr1 afr2 afr3
>  option namespace afr-ns
>
>  # use the ALU scheduler
>  option scheduler alu
>
>  # This option makes brick5 to be readonly, where no new files are  
> created.
>  ##option alu.read-only-subvolumes brick5##
>
>  # Don't create files one a volume with less than 5% free diskspace
>  option alu.limits.min-free-disk  10%
>
>  # Don't create files on a volume with more than 10000 files open
>  option alu.limits.max-open-files 10000
>
>  # When deciding where to place a file, first look at the disk- 
> usage, then at
>  # read-usage, write-usage, open files, and finally the disk-speed- 
> usage.
>  option alu.order disk-usage:read-usage:write-usage:open-files- 
> usage:disk-speed-usage
>
>  # Kick in if the discrepancy in disk-usage between volumes is more  
> than 2GB
>  option alu.disk-usage.entry-threshold 2GB
>
>  # Don't stop writing to the least-used volume until the discrepancy  
> is 1988MB
>  option alu.disk-usage.exit-threshold  60MB
>
>  # Kick in if the discrepancy in open files is 1024
>  option alu.open-files-usage.entry-threshold 1024
>
>  # Don't stop until 992 files have been written the least-used volume
>  option alu.open-files-usage.exit-threshold 32
>
>  # Kick in when the read-usage discrepancy is 20%
>  option alu.read-usage.entry-threshold 20%
>
>  # Don't stop until the discrepancy has been reduced to 16% (20% - 4%)
>  option alu.read-usage.exit-threshold 4%
>
>  # Kick in when the write-usage discrepancy is 20%
>  option alu.write-usage.entry-threshold 20%
>
> ## Don't stop until the discrepancy has been reduced to 16%
>  option alu.write-usage.exit-threshold 4%
>
>  # Refresh the statistics used for decision-making every 10 seconds
>  option alu.stat-refresh.interval 10sec
>
> # Refresh the statistics used for decision-making after creating 10  
> files
> # option alu.stat-refresh.num-file-create 10
> end-volume
>
>
> #writebehind improves write performance a lot
> volume writebehind
>   type performance/write-behind
>   option aggregate-size 131072 # in bytes
>   subvolumes unify
> end-volume
>
> Has anyone seen this issue before? Any suggestions?
>
> Thanks,
> -elb-
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090611/5e41e87a/attachment.html>

Vikas Gorur

2009-Jun-11 20:47 UTC

head link

[Gluster-users] Issue with files on glusterfs becoming unreadable.

----- "Elbert Lai" <elbert.lai at eng.admob.com> wrote:
> elbert at host1:~$ dpkg -l|grep glusterfs
> ii  glusterfs-client                   
> 1.3.8-0pre2                          GlusterFS fuse client
> ii  glusterfs-server                   
> 1.3.8-0pre2                          GlusterFS fuse server
> ii  libglusterfs0                      
> 1.3.8-0pre2                          GlusterFS libraries and  
> translator modules
You are using quite an old version of GlusterFS. Many issues with AFR have been
fixed then. Can you try the newest release, 2.0.1?

Debian packages available at: http://packages.debian.org/sid/glusterfs-client
(Look there for the related packages.)

Vikas
-- 
Engineer - Z Research
http://gluster.com/

Apparently Analagous Threads

Search for more reasonably related threads

Gluster users - Jun 2009 - Issue with files on glusterfs becoming unreadable.

[Gluster-users] Issue with files on glusterfs becoming unreadable.

[Gluster-users] Issue with files on glusterfs becoming unreadable.

[Gluster-users] Issue with files on glusterfs becoming unreadable.

Apparently Analagous Threads