Gluster users - Dec 2008 - Abnormally large number of Open files with Glusterfs

Hi,

I am using glusterfs to provide storage for a web server. The current server
in question had been running for around 20 days now without requiring any
intervention. However, the performance was degrading over the last week and
it became completely unresponsive today. The panic button was pressed when
we realized that not only glusterfs but all other processes on that machine
were not responsive and trying to restart any service resulted in the error
- "Too many files open."
Doing an lsof showed over 115000 files open of which glusterfs was
responsible for around 112000.  Checking the list of files opened by
glusterfs, i was surprisd to see that glusterfs had opened same files again
and again and again. More surprisingly, most of these file descriptors were
for directories rather than files. Is it a fall off from some book keeping?
Or are a new file descriptor is always opened by glusterfs irrespective of
whether a previous one already exists or not?

Also, can someone suggest a good time interval after which to restart
glusterfsd or if there is already a mechanism in place to ask glusterfsd to
release the open file descriptoes?

Thanks and regards
Chandranshu
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20081217/a764b5b8/attachment.html>

Chandranshu,
  Can you share the volume file for the server which is keeping the files
open?
  It will also be helpful to debug if you can share the log of the
corresponding server with us.

--
gowda


On Wed, Dec 17, 2008 at 4:39 PM, Chandranshu . <chandranshu at
gmail.com>wrote:
> Hi,
>
> I am using glusterfs to provide storage for a web server. The current
> server in question had been running for around 20 days now without
requiring
> any intervention. However, the performance was degrading over the last week
> and it became completely unresponsive today. The panic button was pressed
> when we realized that not only glusterfs but all other processes on that
> machine were not responsive and trying to restart any service resulted in
> the error - "Too many files open."
> Doing an lsof showed over 115000 files open of which glusterfs was
> responsible for around 112000.  Checking the list of files opened by
> glusterfs, i was surprisd to see that glusterfs had opened same files again
> and again and again. More surprisingly, most of these file descriptors were
> for directories rather than files. Is it a fall off from some book keeping?
> Or are a new file descriptor is always opened by glusterfs irrespective of
> whether a previous one already exists or not?
>
> Also, can someone suggest a good time interval after which to restart
> glusterfsd or if there is already a mechanism in place to ask glusterfsd to
> release the open file descriptoes?
>
> Thanks and regards
> Chandranshu
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>
>

-- 
hard work often pays off after time, but laziness always pays off now
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20081217/4648e9cf/attachment.html>

I am including the server and client configurations inline in the mail. The
log files are quite huge and we are looking into them ourselves. If you are
looking for something in particular, I can attach the specific logs.

Regards
Chandranshu

########################
##       glusterfs-server.vol                 ##
########################
volume brick
  type storage/posix
  option directory /var/glusterfs
end-volume

volume server
  type protocol/server
  subvolumes brick
  option transport-type tcp/server     # For TCP/IP transport
  option auth.ip.brick.allow *
end-volume
########## Server Vol ends ####

########################
##      glusterfs-client.vol                   ##
########################
volume client1
  type protocol/client
  option transport-type tcp/client
  option remote-host 192.168.4.53
  option remote-subvolume brick
 option transport-timeout 3
end-volume

volume client2
  type protocol/client
  option transport-type tcp/client
  option remote-host 192.168.4.55
  option remote-subvolume brick
 option transport-timeout 3
end-volume

volume afr
 type cluster/afr
 subvolumes client1 client2
# option replicate *:2
end-volume
####### Glusterfs Client ends #####

On Wed, Dec 17, 2008 at 5:49 PM, Basavanagowda Kanur <
basavanagowda at gmail.com> wrote:
> Chandranshu,
>   Can you share the volume file for the server which is keeping the files
> open?
>   It will also be helpful to debug if you can share the log of the
> corresponding server with us.
>
> --
> gowda
>
>
> On Wed, Dec 17, 2008 at 4:39 PM, Chandranshu . <chandranshu at
gmail.com>wrote:
>
>> Hi,
>>
>> I am using glusterfs to provide storage for a web server. The current
>> server in question had been running for around 20 days now without
requiring
>> any intervention. However, the performance was degrading over the last
week
>> and it became completely unresponsive today. The panic button was
pressed
>> when we realized that not only glusterfs but all other processes on
that
>> machine were not responsive and trying to restart any service resulted
in
>> the error - "Too many files open."
>> Doing an lsof showed over 115000 files open of which glusterfs was
>> responsible for around 112000.  Checking the list of files opened by
>> glusterfs, i was surprisd to see that glusterfs had opened same files
again
>> and again and again. More surprisingly, most of these file descriptors
were
>> for directories rather than files. Is it a fall off from some book
keeping?
>> Or are a new file descriptor is always opened by glusterfs irrespective
of
>> whether a previous one already exists or not?
>>
>> Also, can someone suggest a good time interval after which to restart
>> glusterfsd or if there is already a mechanism in place to ask
glusterfsd to
>> release the open file descriptoes?
>>
>> Thanks and regards
>> Chandranshu
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>
>>
>
>
> --
> hard work often pays off after time, but laziness always pays off now
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20081217/d81742d2/attachment.html>

I see this and on my system too.
Aprox 10% of fd is not present on disk.
No errors in logfile.
> 
> Chandranshu,
>   Can you share the volume file for the server which is keeping the files
> open?
>   It will also be helpful to debug if you can share the log of the
> corresponding server with us.
> 
> --
> gowda
> 
> 
> On Wed, Dec 17, 2008 at 4:39 PM, Chandranshu . <chandranshu at
gmail.com>wrote:
> 
> > Hi,
> >
> > I am using glusterfs to provide storage for a web server. The current
> > server in question had been running for around 20 days now without
requiring
> > any intervention. However, the performance was degrading over the last
week
> > and it became completely unresponsive today. The panic button was
pressed
> > when we realized that not only glusterfs but all other processes on
that
> > machine were not responsive and trying to restart any service resulted
in
> > the error - "Too many files open."
> > Doing an lsof showed over 115000 files open of which glusterfs was
> > responsible for around 112000.  Checking the list of files opened by
> > glusterfs, i was surprisd to see that glusterfs had opened same files
again
> > and again and again. More surprisingly, most of these file descriptors
were
> > for directories rather than files. Is it a fall off from some book
keeping?
> > Or are a new file descriptor is always opened by glusterfs
irrespective of
> > whether a previous one already exists or not?
> >
> > Also, can someone suggest a good time interval after which to restart
> > glusterfsd or if there is already a mechanism in place to ask
glusterfsd to
> > release the open file descriptoes?
> >
> > Thanks and regards
> > Chandranshu
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
> >
> >
> 
> 
> -- 
> hard work often pays off after time, but laziness always pays off now
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
> 
>

Hi Avati,

1. `glusterfs --version` gives following output.
glusterfs 1.3.8 built on May 14 2008 17:16:06
Repository revision: glusterfs--mainline--2.5--patch-760
Copyright (c) 2006, 2007, 2008 Z RESEARCH Inc. <http://www.zresearch.com>

2. No, the large number of open fds were seen on only one machine. The other
machine had very small number of FDs (1156 only).

Regards
Chandranshu

On Thu, Dec 18, 2008 at 2:34 PM, Anand Avati <anand.avati at gmail.com>
wrote:
> Chandra,
>  What version of glusterfs are you using? Did you see open fd's on
> both the servers?
>
> avati
>
> 2008/12/17 Chandranshu . <chandranshu at gmail.com>:
> > I am including the server and client configurations inline in the
mail.
> The
> > log files are quite huge and we are looking into them ourselves. If
you
> are
> > looking for something in particular, I can attach the specific logs.
> >
> > Regards
> > Chandranshu
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20081218/de7dac21/attachment.html>