Kenneth Holter
2009-Oct-23 07:44 UTC
[Puppet Users] The /var/lib/puppet/clientbucket directory grew 4GB in minutes, filling up /var on production server
Hi. Last night the /var/lib/puppet/clientbucktet directory on one of our production servers suddenly grew about 4 GB, filling up the entire /var partition. Around that time, I see that there was a problem with one of our CIFS mount points (i.e. our CIFS client had problems reaching our CIFS server). We''re using puppet to set up the CIFS mount point. Under the clientbucket folder, I have a great number of folders and subfolders like this: --- 0/2/0/f/4/f/3/0/020f4f30aa1670825e81cae2faac4d1d/paths 0/2/0/e/b/d/3/2/020ebd32755675f65bab06b96d7f2265/paths 0/2/0/c/0/1/d/7/020c01d71491b1139b0f23b6ba81d410/paths --- Investigating the contents of a few of the "paths"-files like the ones above, if find that the contents seems to be related to the mount points. I found this by grepping for the name of the mount point withing these filers. I looks like each of the "paths"-file contains the name of exactly one file (seems to be different name within each of the "paths"-files) found under the mount point. What exactly is the clientbucket folder used for? It''s not the same as filebucket I presume. Next: How come the "paths"-files above contain names of individual files found within the CIFS mount point? Anyone know why this is so, and why the clientbucket folder so suddenly got filled up with those "paths"-files? And how can I make sure this doesn''t happen again (at least not on our production servers)? Best regards, Kenneth Holter --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Luke Kanies
2009-Oct-26 05:06 UTC
[Puppet Users] Re: The /var/lib/puppet/clientbucket directory grew 4GB in minutes, filling up /var on production server
On Oct 23, 2009, at 12:44 AM, Kenneth Holter wrote:> Hi. > > > Last night the /var/lib/puppet/clientbucktet directory on one of our > production servers suddenly grew about 4 GB, filling up the entire / > var partition. Around that time, I see that there was a problem with > one of our CIFS mount points (i.e. our CIFS client had problems > reaching our CIFS server). We''re using puppet to set up the CIFS > mount point. > > Under the clientbucket folder, I have a great number of folders and > subfolders like this: > --- > 0/2/0/f/4/f/3/0/020f4f30aa1670825e81cae2faac4d1d/paths > 0/2/0/e/b/d/3/2/020ebd32755675f65bab06b96d7f2265/paths > 0/2/0/c/0/1/d/7/020c01d71491b1139b0f23b6ba81d410/paths > --- > > Investigating the contents of a few of the "paths"-files like the > ones above, if find that the contents seems to be related to the > mount points. I found this by grepping for the name of the mount > point withing these filers. I looks like each of the "paths"-file > contains the name of exactly one file (seems to be different name > within each of the "paths"-files) found under the mount point. > > What exactly is the clientbucket folder used for? It''s not the same > as filebucket I presume. > > Next: How come the "paths"-files above contain names of individual > files found within the CIFS mount point? Anyone know why this is so, > and why the clientbucket folder so suddenly got filled up with those > "paths"-files? > And how can I make sure this doesn''t happen again (at least not on > our production servers)?I''ve only ever seen this happen in one other case, and it was related to NFS - something caused Puppet to back up a whole filesystem to the filebucket: http://projects.reductivelabs.com/issues/2019 It''d be awesome if you can track down the problem, because we''re stymied. As to why the individual file names are in the ''paths'' files - that''s what they do - hold the paths that have been backed up there. -- Due to circumstances beyond your control, you are master of your fate and captain of your soul. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Kenneth Holter
2009-Oct-28 19:21 UTC
[Puppet Users] Re: The /var/lib/puppet/clientbucket directory grew 4GB in minutes, filling up /var on production server
Thansk for your reply. I''ll let you know if I find the cause of the problems. Regards, Kenneth On Mon, Oct 26, 2009 at 6:06 AM, Luke Kanies <luke@madstop.com> wrote:> > On Oct 23, 2009, at 12:44 AM, Kenneth Holter wrote: > > > Hi. > > > > > > Last night the /var/lib/puppet/clientbucktet directory on one of our > > production servers suddenly grew about 4 GB, filling up the entire / > > var partition. Around that time, I see that there was a problem with > > one of our CIFS mount points (i.e. our CIFS client had problems > > reaching our CIFS server). We''re using puppet to set up the CIFS > > mount point. > > > > Under the clientbucket folder, I have a great number of folders and > > subfolders like this: > > --- > > 0/2/0/f/4/f/3/0/020f4f30aa1670825e81cae2faac4d1d/paths > > 0/2/0/e/b/d/3/2/020ebd32755675f65bab06b96d7f2265/paths > > 0/2/0/c/0/1/d/7/020c01d71491b1139b0f23b6ba81d410/paths > > --- > > > > Investigating the contents of a few of the "paths"-files like the > > ones above, if find that the contents seems to be related to the > > mount points. I found this by grepping for the name of the mount > > point withing these filers. I looks like each of the "paths"-file > > contains the name of exactly one file (seems to be different name > > within each of the "paths"-files) found under the mount point. > > > > What exactly is the clientbucket folder used for? It''s not the same > > as filebucket I presume. > > > > Next: How come the "paths"-files above contain names of individual > > files found within the CIFS mount point? Anyone know why this is so, > > and why the clientbucket folder so suddenly got filled up with those > > "paths"-files? > > And how can I make sure this doesn''t happen again (at least not on > > our production servers)? > > I''ve only ever seen this happen in one other case, and it was related > to NFS - something caused Puppet to back up a whole filesystem to the > filebucket: > > http://projects.reductivelabs.com/issues/2019 > > It''d be awesome if you can track down the problem, because we''re > stymied. > > As to why the individual file names are in the ''paths'' files - that''s > what they do - hold the paths that have been backed up there. > > -- > Due to circumstances beyond your control, you are master of your fate > and captain of your soul. > --------------------------------------------------------------------- > Luke Kanies | http://reductivelabs.com | http://madstop.com > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---