Puppet users - Oct 2009 - The /var/lib/puppet/clientbucket directory grew 4GB in minutes, filling up /var on production server

Hi.


Last night the /var/lib/puppet/clientbucktet directory on one of our
production servers suddenly grew about 4 GB, filling up the entire /var
partition. Around that time, I see that there was a problem with one of our
CIFS mount points (i.e. our CIFS client had problems reaching our CIFS
server). We''re using puppet to set up the CIFS mount point.

Under the clientbucket folder, I have a great number of folders and
subfolders like this:
---
0/2/0/f/4/f/3/0/020f4f30aa1670825e81cae2faac4d1d/paths
0/2/0/e/b/d/3/2/020ebd32755675f65bab06b96d7f2265/paths
0/2/0/c/0/1/d/7/020c01d71491b1139b0f23b6ba81d410/paths
---

Investigating the contents of a few of the "paths"-files like the ones
above, if find that the contents seems to be related to the mount points. I
found this by grepping for the name of the mount point withing these filers.
I looks like each of the "paths"-file contains the name of exactly one
file
(seems to be different name within each of the "paths"-files) found
under
the mount point.

What exactly is the clientbucket folder used for? It''s not the same as
filebucket I presume.

Next: How come the "paths"-files above contain names of individual
files
found within the CIFS mount point? Anyone know why this is so, and why the
clientbucket folder so suddenly got filled up with those
"paths"-files?
And how can I make sure this doesn''t happen again (at least not on our
production servers)?


Best regards,
Kenneth Holter

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Oct 23, 2009, at 12:44 AM, Kenneth Holter wrote:
> Hi.
>
>
> Last night the /var/lib/puppet/clientbucktet directory on one of our  
> production servers suddenly grew about 4 GB, filling up the entire / 
> var partition. Around that time, I see that there was a problem with  
> one of our CIFS mount points (i.e. our CIFS client had problems  
> reaching our CIFS server). We''re using puppet to set up the CIFS  
> mount point.
>
> Under the clientbucket folder, I have a great number of folders and  
> subfolders like this:
> ---
> 0/2/0/f/4/f/3/0/020f4f30aa1670825e81cae2faac4d1d/paths
> 0/2/0/e/b/d/3/2/020ebd32755675f65bab06b96d7f2265/paths
> 0/2/0/c/0/1/d/7/020c01d71491b1139b0f23b6ba81d410/paths
> ---
>
> Investigating the contents of a few of the "paths"-files like the
> ones above, if find that the contents seems to be related to the  
> mount points. I found this by grepping for the name of the mount  
> point withing these filers. I looks like each of the "paths"-file
> contains the name of exactly one file (seems to be different name  
> within each of the "paths"-files) found under the mount point.
>
> What exactly is the clientbucket folder used for? It''s not the
same
> as filebucket I presume.
>
> Next: How come the "paths"-files above contain names of
individual
> files found within the CIFS mount point? Anyone know why this is so,  
> and why the clientbucket folder so suddenly got filled up with those  
> "paths"-files?
> And how can I make sure this doesn''t happen again (at least not on
> our production servers)?
I''ve only ever seen this happen in one other case, and it was related  
to NFS - something caused Puppet to back up a whole filesystem to the  
filebucket:

http://projects.reductivelabs.com/issues/2019

It''d be awesome if you can track down the problem, because
we''re
stymied.

As to why the individual file names are in the ''paths'' files -
that''s
what they do - hold the paths that have been backed up there.

-- 
Due to circumstances beyond your control, you are master of your fate
and captain of your soul.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Thansk for your reply. I''ll let you know if I find the cause of the
problems.

Regards,
Kenneth

On Mon, Oct 26, 2009 at 6:06 AM, Luke Kanies <luke@madstop.com> wrote:
>
> On Oct 23, 2009, at 12:44 AM, Kenneth Holter wrote:
>
> > Hi.
> >
> >
> > Last night the /var/lib/puppet/clientbucktet directory on one of our
> > production servers suddenly grew about 4 GB, filling up the entire /
> > var partition. Around that time, I see that there was a problem with
> > one of our CIFS mount points (i.e. our CIFS client had problems
> > reaching our CIFS server). We''re using puppet to set up the
CIFS
> > mount point.
> >
> > Under the clientbucket folder, I have a great number of folders and
> > subfolders like this:
> > ---
> > 0/2/0/f/4/f/3/0/020f4f30aa1670825e81cae2faac4d1d/paths
> > 0/2/0/e/b/d/3/2/020ebd32755675f65bab06b96d7f2265/paths
> > 0/2/0/c/0/1/d/7/020c01d71491b1139b0f23b6ba81d410/paths
> > ---
> >
> > Investigating the contents of a few of the "paths"-files
like the
> > ones above, if find that the contents seems to be related to the
> > mount points. I found this by grepping for the name of the mount
> > point withing these filers. I looks like each of the
"paths"-file
> > contains the name of exactly one file (seems to be different name
> > within each of the "paths"-files) found under the mount
point.
> >
> > What exactly is the clientbucket folder used for? It''s not
the same
> > as filebucket I presume.
> >
> > Next: How come the "paths"-files above contain names of
individual
> > files found within the CIFS mount point? Anyone know why this is so,
> > and why the clientbucket folder so suddenly got filled up with those
> > "paths"-files?
> > And how can I make sure this doesn''t happen again (at least
not on
> > our production servers)?
>
> I''ve only ever seen this happen in one other case, and it was
related
> to NFS - something caused Puppet to back up a whole filesystem to the
> filebucket:
>
> http://projects.reductivelabs.com/issues/2019
>
> It''d be awesome if you can track down the problem, because
we''re
> stymied.
>
> As to why the individual file names are in the ''paths''
files - that''s
> what they do - hold the paths that have been backed up there.
>
> --
> Due to circumstances beyond your control, you are master of your fate
> and captain of your soul.
> ---------------------------------------------------------------------
> Luke Kanies | http://reductivelabs.com | http://madstop.com
>
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---