Gluster users - Dec 2010 - Possible workaround for a problem with a "permissions" or "file access" error

Hi folks

  We saw this at two different customer sites, one running 3.0.5 and one 
running 3.1.1.  This is a distribute layout, no replication.  Pretty 
intensive loads on the siCluster.

Every now and then there was a hiccup:

258% ls -l /x/y/.z/t.xml
-rw-r--r-- 1 a b 574 Dec 9 07:10
  /x/y/.z/t.xml
259% cat !$
cat /x/y/.z/t.xml
cat: /x/y/.z/t.xml: Input/output error
260% df -h !$
df -h /x/y/.z/t.xml
Filesystem Size Used Avail Use% Mounted on
a.b.c.d:/nfs 15T 100G 15T 1% /x

The file was really there.  And it was really inaccessible.  Remounts 
sometimes helped.  Logs contained access errors.  File system on backend 
was fine, no issues.  Network was fine, no issues.

Way back in 2.0.x days, there were caching bugs.  stat-prefetch and 
others were problematic.  I remembered this.  So we tried this at the 
3.0.5 customer site (set timeout to zero and remove stat-prefetch)

volume quickread
     type performance/quick-read
     option cache-timeout 0
     option max-file-size 64kB
     subvolumes iocache
end-volume

#volume statprefetch
#    type performance/stat-prefetch
#    subvolumes quickread
#end-volume

and it seemed to fix it (changed cache-timeout to 0 and commented out 
stat-prefetch).  Still testing, but it seems to have made that bug less 
problematic.

On 3.1.1, there are no config files.  And there is no comprehensive list 
of volume/translator switches, nor any real clue as to what the 
translator layers are in use.

So we experimented until we found this:

[root at manager ~]# gluster volume set nfs performance.cache-refresh-timeout 0
Set volume successful

[root at manager ~]# gluster volume set nfs performance.stat-prefetch 0
Set volume successful

These seem to have worked.  And we are testing to see if they fix the 
problem.

Has someone somewhere compiled a complete list of the settable config 
elements and their meanings?  I don't know if the settings simply set 
key-value pairs, or if they actually impact things.  A listing of these 
would be nice, though I found what we found by some creative guessing 
based upon older configurations.

Regards,

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615

On Mon, 20 Dec 2010 at 3:36pm, Joe Landman wrote
> Has someone somewhere compiled a complete list of the settable config 
> elements and their meanings?  I don't know if the settings simply set 
> key-value pairs, or if they actually impact things.  A listing of these
would
> be nice, though I found what we found by some creative guessing based upon 
> older configurations.
This came up recently, and the wiki has been updated with all the options 
and their default values:

http://www.gluster.com/community/documentation/index.php/Gluster_3.1:_Setting_Volume_Options

Also, an RFE for "gluster volume info" was submitted such that it
always
displays the volume options in use.  Currently it only displays values 
that have been changed from their defaults.

-- 
Joshua Baker-LePain
QB3 Shared Cluster Sysadmin
UCSF

Update:

On 12/20/2010 03:36 PM, Joe Landman wrote:
> So we experimented until we found this:
>
> [root at manager ~]# gluster volume set nfs
> performance.cache-refresh-timeout 0
> Set volume successful
>
> [root at manager ~]# gluster volume set nfs performance.stat-prefetch 0
> Set volume successful
>
> These seem to have worked. And we are testing to see if they fix the
> problem.
User observed the issue again.  They did a mount with noac on the client 
and indicated it was helping.

Will update later on.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615