thr3ads.net - Gluster users - [Gluster-users] 3.1.2 with "No such file" and "Invalid argument" errors [Jan 2011]

If this information is useful, please help other people find it:
Share via:

Steve Wilson

2011-Jan-28 17:49 UTC

[Gluster-users] 3.1.2 with "No such file" and "Invalid argument" errors

I'm running a pair of replicated/distributed GlusterFS 3.1.2 servers, 
each with 8 bricks.  Here's the command I used to create the data volume:

    gluster volume create post replica 2 transport tcp
    pablo:/gluster/01/brick stanley:/gluster/01/brick \
    pablo:/gluster/02/brick stanley:/gluster/02/brick \
    pablo:/gluster/03/brick stanley:/gluster/03/brick \
    pablo:/gluster/04/brick stanley:/gluster/04/brick \
    pablo:/gluster/05/brick stanley:/gluster/05/brick \
    pablo:/gluster/06/brick stanley:/gluster/06/brick \
    pablo:/gluster/07/brick stanley:/gluster/07/brick \
    pablo:/gluster/08/brick stanley:/gluster/08/brick

The clients mount the volume using the gluster FUSE client, again 
version 3.1.2.  Since GlusterFS doesn't yet support exporting 
sub-directories directly (it does now for NFS, though, I believe), I 
mount the volume as:
     /net/.post-volume
and then bind mount a couple of lower sub-directories as:
     /net/post (bound to /net/.post-volume/home)
     /net/post-scratch (bound to /net/.post-volume/scratch)

I am using this volume for user home directories.

That's the background information... now for the problem.  There are 
certain files that are almost guaranteed to go missing after less than a 
few hours of a user starting up a login session.  We use Ubuntu with GDM 
as a login greeter and both KDE and Gnome desktops.  Here are a few of 
the files that usually go missing:

    ls: .dmrc: No such file or directory
    ls: .pine-debug2: Invalid argument
    ls: .xsession-errors.old: No such file or directory
    ls: .recently-used.xbel: No such file or directory

Taking .dmrc, for example, I can stop the volume, then remove all 
occurrences of this file on all the bricks on both servers.  Then start 
up the volume again, start the user session again, and after a period of 
time (I'm not sure how long), the file comes up missing again.  When the 
file is missing, this is what it looks like on the bricks themselves:

server 1:

    root at stanley:/gluster# ls -l ./07/brick/home/stevew/.dmrc
    ---------T 1 stevew sysmgr 0 2011-01-28 08:24
    ./07/brick/home/stevew/.dmrc
    root at stanley:/gluster# ls -l ./02/brick/home/stevew/.dmrc
    -rw-r--r-- 1 stevew sysmgr 25 2011-01-28 08:25
    ./02/brick/home/stevew/.dmrc

server 2:

    root at pablo:/gluster# ls -l ./07/brick/home/stevew/.dmrc
    ---------T 1 stevew sysmgr 0 2011-01-28 08:24
    ./07/brick/home/stevew/.dmrc
    root at pablo:/gluster# ls -l ./02/brick/home/stevew/.dmrc
    -rw-r--r-- 1 stevew sysmgr 25 2011-01-28 08:25
    ./02/brick/home/stevew/.dmrc


Pertinent log messages are:

server 1:

    net-.post-volume.log:[2011-01-28 08:16:28.899544] W
    [fuse-bridge.c:2432:fuse_xattr_cbk] glusterfs-fuse: 6318974:
    GETXATTR() /home/stevew/.dmrc => -1 (Invalid argument)


server 2:

    net-.post-volume.log:[2011-01-28 08:10:58.74848] W
    [fuse-bridge.c:2432:fuse_xattr_cbk] glusterfs-fuse: 2428788:
    GETXATTR() /home/stevew/.dmrc => -1 (No such file or directory)


Any thoughts or ideas?

Thanks!

Steve
-- 

Steven M. Wilson, Systems and Network Manager
Markey Center for Structural Biology
Purdue University
(765) 496-1946

Raghavendra G

2011-Jan-31 05:27 UTC

head link

[Gluster-users] 3.1.2 with "No such file" and "Invalid argument" errors

Hi Steve,

Can you please send us complete client and server log files?

regards,
----- Original Message -----> From: "Steve Wilson" <stevew at purdue.edu>
> To: "Gluster General Discussion List" <gluster-users at
gluster.org>
> Sent: Friday, January 28, 2011 9:49:03 PM
> Subject: [Gluster-users] 3.1.2 with "No such file" and
"Invalid argument" errors
> I'm running a pair of replicated/distributed GlusterFS 3.1.2 servers,
> each with 8 bricks. Here's the command I used to create the data
> volume:
> 
> gluster volume create post replica 2 transport tcp
> pablo:/gluster/01/brick stanley:/gluster/01/brick \
> pablo:/gluster/02/brick stanley:/gluster/02/brick \
> pablo:/gluster/03/brick stanley:/gluster/03/brick \
> pablo:/gluster/04/brick stanley:/gluster/04/brick \
> pablo:/gluster/05/brick stanley:/gluster/05/brick \
> pablo:/gluster/06/brick stanley:/gluster/06/brick \
> pablo:/gluster/07/brick stanley:/gluster/07/brick \
> pablo:/gluster/08/brick stanley:/gluster/08/brick
> 
> The clients mount the volume using the gluster FUSE client, again
> version 3.1.2. Since GlusterFS doesn't yet support exporting
> sub-directories directly (it does now for NFS, though, I believe), I
> mount the volume as:
> /net/.post-volume
> and then bind mount a couple of lower sub-directories as:
> /net/post (bound to /net/.post-volume/home)
> /net/post-scratch (bound to /net/.post-volume/scratch)
> 
> I am using this volume for user home directories.
> 
> That's the background information... now for the problem. There are
> certain files that are almost guaranteed to go missing after less than
> a
> few hours of a user starting up a login session. We use Ubuntu with
> GDM
> as a login greeter and both KDE and Gnome desktops. Here are a few of
> the files that usually go missing:
> 
> ls: .dmrc: No such file or directory
> ls: .pine-debug2: Invalid argument
> ls: .xsession-errors.old: No such file or directory
> ls: .recently-used.xbel: No such file or directory
> 
> Taking .dmrc, for example, I can stop the volume, then remove all
> occurrences of this file on all the bricks on both servers. Then start
> up the volume again, start the user session again, and after a period
> of
> time (I'm not sure how long), the file comes up missing again. When
> the
> file is missing, this is what it looks like on the bricks themselves:
> 
> server 1:
> 
> root at stanley:/gluster# ls -l ./07/brick/home/stevew/.dmrc
> ---------T 1 stevew sysmgr 0 2011-01-28 08:24
> ./07/brick/home/stevew/.dmrc
> root at stanley:/gluster# ls -l ./02/brick/home/stevew/.dmrc
> -rw-r--r-- 1 stevew sysmgr 25 2011-01-28 08:25
> ./02/brick/home/stevew/.dmrc
> 
> server 2:
> 
> root at pablo:/gluster# ls -l ./07/brick/home/stevew/.dmrc
> ---------T 1 stevew sysmgr 0 2011-01-28 08:24
> ./07/brick/home/stevew/.dmrc
> root at pablo:/gluster# ls -l ./02/brick/home/stevew/.dmrc
> -rw-r--r-- 1 stevew sysmgr 25 2011-01-28 08:25
> ./02/brick/home/stevew/.dmrc
> 
> 
> Pertinent log messages are:
> 
> server 1:
> 
> net-.post-volume.log:[2011-01-28 08:16:28.899544] W
> [fuse-bridge.c:2432:fuse_xattr_cbk] glusterfs-fuse: 6318974:
> GETXATTR() /home/stevew/.dmrc => -1 (Invalid argument)
> 
> 
> server 2:
> 
> net-.post-volume.log:[2011-01-28 08:10:58.74848] W
> [fuse-bridge.c:2432:fuse_xattr_cbk] glusterfs-fuse: 2428788:
> GETXATTR() /home/stevew/.dmrc => -1 (No such file or directory)
> 
> 
> Any thoughts or ideas?
> 
> Thanks!
> 
> Steve
> --
> 
> Steven M. Wilson, Systems and Network Manager
> Markey Center for Structural Biology
> Purdue University
> (765) 496-1946
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Steve Wilson

2011-Feb-04 16:36 UTC

head link

[Gluster-users] 3.1.2 with "No such file" and "Invalid argument"

On 01/28/2011 12:49 PM, Steve Wilson wrote:> I'm running a pair of replicated/distributed GlusterFS 3.1.2 servers, 
> each with 8 bricks.  Here's the command I used to create the data
volume:
>
>    gluster volume create post replica 2 transport tcp
>    pablo:/gluster/01/brick stanley:/gluster/01/brick \
>    pablo:/gluster/02/brick stanley:/gluster/02/brick \
>    pablo:/gluster/03/brick stanley:/gluster/03/brick \
>    pablo:/gluster/04/brick stanley:/gluster/04/brick \
>    pablo:/gluster/05/brick stanley:/gluster/05/brick \
>    pablo:/gluster/06/brick stanley:/gluster/06/brick \
>    pablo:/gluster/07/brick stanley:/gluster/07/brick \
>    pablo:/gluster/08/brick stanley:/gluster/08/brick
>
> The clients mount the volume using the gluster FUSE client, again 
> version 3.1.2.  Since GlusterFS doesn't yet support exporting 
> sub-directories directly (it does now for NFS, though, I believe), I 
> mount the volume as:
>     /net/.post-volume
> and then bind mount a couple of lower sub-directories as:
>     /net/post (bound to /net/.post-volume/home)
>     /net/post-scratch (bound to /net/.post-volume/scratch)
>
> I am using this volume for user home directories.
>
> That's the background information... now for the problem.  There are 
> certain files that are almost guaranteed to go missing after less than 
> a few hours of a user starting up a login session.  We use Ubuntu with 
> GDM as a login greeter and both KDE and Gnome desktops.  Here are a 
> few of the files that usually go missing:
>
>    ls: .dmrc: No such file or directory
>    ls: .pine-debug2: Invalid argument
>    ls: .xsession-errors.old: No such file or directory
>    ls: .recently-used.xbel: No such file or directory
>
> Taking .dmrc, for example, I can stop the volume, then remove all 
> occurrences of this file on all the bricks on both servers.  Then 
> start up the volume again, start the user session again, and after a 
> period of time (I'm not sure how long), the file comes up missing 
> again.  When the file is missing, this is what it looks like on the 
> bricks themselves:
>
> server 1:
>
>    root at stanley:/gluster# ls -l ./07/brick/home/stevew/.dmrc
>    ---------T 1 stevew sysmgr 0 2011-01-28 08:24
>    ./07/brick/home/stevew/.dmrc
>    root at stanley:/gluster# ls -l ./02/brick/home/stevew/.dmrc
>    -rw-r--r-- 1 stevew sysmgr 25 2011-01-28 08:25
>    ./02/brick/home/stevew/.dmrc
>
> server 2:
>
>    root at pablo:/gluster# ls -l ./07/brick/home/stevew/.dmrc
>    ---------T 1 stevew sysmgr 0 2011-01-28 08:24
>    ./07/brick/home/stevew/.dmrc
>    root at pablo:/gluster# ls -l ./02/brick/home/stevew/.dmrc
>    -rw-r--r-- 1 stevew sysmgr 25 2011-01-28 08:25
>    ./02/brick/home/stevew/.dmrc
>
>
> Pertinent log messages are:
>
> server 1:
>
>    net-.post-volume.log:[2011-01-28 08:16:28.899544] W
>    [fuse-bridge.c:2432:fuse_xattr_cbk] glusterfs-fuse: 6318974:
>    GETXATTR() /home/stevew/.dmrc => -1 (Invalid argument)
>
>
> server 2:
>
>    net-.post-volume.log:[2011-01-28 08:10:58.74848] W
>    [fuse-bridge.c:2432:fuse_xattr_cbk] glusterfs-fuse: 2428788:
>    GETXATTR() /home/stevew/.dmrc => -1 (No such file or directory)
>
>
> Any thoughts or ideas?
>
> Thanks!
>
> Steve
>
>
Yesterday I disabled the performance.stat-prefetch volume option and 
this has improved the situation greatly.  I can now do a "ls -alR > 
/dev/null" and get no errors on the mounted volume.

Steve

Raghavendra G

2011-Feb-08 04:49 UTC

head link

[Gluster-users] 3.1.2 with "No such file" and "Invalid argument" errors

Hi Steve,

Are the back-end file systems working correctly? I am seeing lots of errors in
server log files while accessing back-end filesystem.

gluster-01-brick.log.1:[2011-01-26 03:43:07.353445] E [posix.c:2193:posix_open]
post-posix: open on /gluster/01/bri
ck/home/lev/deltah/aadimers/serd/converge/0..75000/serd_phi-psi_hist.4deg.0..75000_map.cmd:
Read-only file system
gluster-01-brick.log.1:[2011-01-26 03:43:07.353857] E
[posix.c:678:posix_setattr] post-posix: setattr (utimes) on /
gluster/01/brick/home/lev/deltah/aadimers/serd/converge/0..75000/serd_phi-psi_hist.4deg.0..75000_map.cmd
failed: Re
ad-only file system
gluster-01-brick.log.1:[2011-01-26 03:43:07.354827] E [posix.c:2318:posix_readv]
post-posix: read failed on fd=0x7f
28e50dc1c8: Input/output error
gluster-01-brick.log.1:[2011-01-26 03:43:07.357396] E [posix.c:2193:posix_open]
post-posix: open on /gluster/01/bri
ck/home/lev/deltah/aadimers/serd/converge/0..75000/serd_phi-psi_hist.4deg.0..75000_map.ps:
Read-only file system
gluster-01-brick.log.1:[2011-01-26 03:43:07.357794] E
[posix.c:678:posix_setattr] post-posix: setattr (utimes) on /
gluster/01/brick/home/lev/deltah/aadimers/serd/converge/0..75000/serd_phi-psi_hist.4deg.0..75000_map.ps
failed: Rea
d-only file system
gluster-01-brick.log.1:[2011-01-26 03:43:07.358865] E [posix.c:2318:posix_readv]
post-posix: read failed on fd=0x7f
28e50dc1c8: Input/output error
gluster-01-brick.log.1:[2011-01-26 03:43:07.359264] E [posix.c:2318:posix_readv]
post-posix: read failed on fd=0x7f
28e50dc1c8: Input/output error
gluster-01-brick.log.1:[2011-01-26 03:43:07.359548] E [posix.c:2318:posix_readv]
post-posix: read failed on fd=0x7f
28e50dc1c8: Input/output error
gluster-01-brick.log.1:[2011-01-26 03:43:07.367163] E [posix.c:2318:posix_readv]
post-posix: read failed on fd=0x7f

I am seeing other errors, which indicate that the backend is read-only
filesystem. Due to this distribute and replicate are not able to store the
metadata (using xattrs), which in turn is resulting in lots of split-brains and
layout NULL errors. Can you please check the backend file system?

regards,
----- Original Message -----> From: "Steve Wilson" <stevew at purdue.edu>
> To: "Lakshmipathi" <lakshmipathi at gluster.com>
> Cc: "Raghavendra G" <raghavendra at gluster.com>
> Sent: Thursday, February 3, 2011 7:21:36 PM
> Subject: Re: [Gluster-users] 3.1.2 with "No such file" and
"Invalid argument" errors
> Hi,
> 
> Thanks for looking into this. Any ideas so far? Or anything you'd like
> me to try?
> 
> Here's some other perhaps relevant information:
> * all bricks are formatted ext4 and mounted with the noatime option
> in addition to default options
> * servers and clients are running Ubuntu 10.04
> * I did try mounting the GlusterFS volume with direct-io-mode
> disabled but that didn't fix the problem
> 
> Thanks!
> 
> Steve
> 
> On 02/01/2011 07:35 AM, Lakshmipathi wrote:
> > Hi,
> > Could you please sent us client and server log files?
> >
> >
> 
> --
> Steven M. Wilson, Systems and Network Manager
> Markey Center for Structural Biology
> Purdue University
> (765) 496-1946

Gluster users - Jan 2011 - 3.1.2 with "No such file" and "Invalid argument" errors

[Gluster-users] 3.1.2 with "No such file" and "Invalid argument" errors

[Gluster-users] 3.1.2 with "No such file" and "Invalid argument" errors

[Gluster-users] 3.1.2 with "No such file" and "Invalid argument"

[Gluster-users] 3.1.2 with "No such file" and "Invalid argument" errors