Gluster users - Jul 2011 - gluster opens thousands of files on one peer

Hi,

I have a two server gluster 3.1.5 mirror setup.


One of the peers is behaving very slow - load is constantly around 6-15, 
and generally all connected clients work very slow.

As soon as I shut that peer down, clients are relatively fast.


What I noticed, that on one gluster peer, I only get around 100 files 
opened by the gluster process:

# lsof -n|grep -c gluster
96


On the slow peer, it's hundreds, tens of thousands, and growing constantly:


# while true; do lsof -n|grep -c gluster; sleep 10; done
169934
170181
170363
170655


If I restart glusterd on that peer, it starts to grow again.



On both peers, I have a big number of such entries:

[2011-07-06 03:22:00.20873] W [server-resolve.c:556:server_resolve] 
0-gluster-data-server: pure path resolution for /some/path/.../file (LOOKUP)


Where the path is a symbolic link to the gluster mount.


The unaffected peer has these entries in the log as well:

[2011-07-06 03:23:30.117245] E [posix.c:3672:do_xattrop] 
0-gluster-data-posix: getxattr failed on /some/path/.../file while doing 
xattrop: Numerical result out of range


Other than that, I don't really see anything unusual in the logs.



Why is gluster on one of the peers opening so many files?



-- 
Tomasz Chmielewski
http://wpkg.org

On Wed, Jul 6, 2011 at 6:58 AM, Tomasz Chmielewski <mangoo at wpkg.org>
wrote:
>
> One of the peers is behaving very slow - load is constantly around 6-15,
> and generally all connected clients work very slow.
>
> As soon as I shut that peer down, clients are relatively fast.
>
>
> What I noticed, that on one gluster peer, I only get around 100 files
> opened by the gluster process:
>
> # lsof -n|grep -c gluster
> 96
>
>
> On the slow peer, it's hundreds, tens of thousands, and growing
constantly:
>
>
> # while true; do lsof -n|grep -c gluster; sleep 10; done
> 169934
> 170181
> 170363
> 170655
>
>
> If I restart glusterd on that peer, it starts to grow again.
>
>
>
> On both peers, I have a big number of such entries:
>
> [2011-07-06 03:22:00.20873] W [server-resolve.c:556:server_**resolve]
> 0-gluster-data-server: pure path resolution for /some/path/.../file
(LOOKUP)
>
>
> Where the path is a symbolic link to the gluster mount.
>

Can you describe this a little more? What do you mean by those paths are
symbolic links to the gluster mount? They are supposed to be files within
the gluster mount. Do you have a gluster mount on that peer as well? If so,
does unmounting have any effect?

> The unaffected peer has these entries in the log as well:
>
> [2011-07-06 03:23:30.117245] E [posix.c:3672:do_xattrop]
> 0-gluster-data-posix: getxattr failed on /some/path/.../file while doing
> xattrop: Numerical result out of range
>

This is strange. There is almost no reason how you can be seeing this error
unless the extended attributes were modified outside gluster. Can you get a
dump of those attributes on those files with 'getfattr -d -m . -e hex
/some/path/../file' and post it here?

>
> Why is gluster on one of the peers opening so many files?
>
>
>
It is certainly strange that it's opening more files on only one peer.
Please give as much info about the setup as possible.

Avati
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110709/9fa6da0c/attachment.html>