Bernd Adamowicz
2012-Sep-13 15:55 UTC
WG: [Puppet Users] AW: Issue with large directory content
Hope, the list doesn''t blame me for spamming, but I''d just
like to provide some more information on my problem, still hoping someone can
help.
Using the ''iostat'' command while Puppet is stuck, I can
clearly see that there''s a very high I/O on the disk:
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
fioa 0.00 0.00 163.00 5.00 31280.00 40.00 186.43
0.04 0.25 0.25 4.20
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-5 0.00 0.00 163.00 5.00 31280.00 40.00 186.43
0.04 0.25 0.25 4.20
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
drbd0 0.00 0.00 163.00 5.00 31280.00 40.00 186.43
0.96 0.53 5.72 96.10
drbd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
* The ''drbd0'' device is an HP IO Accelerator card
* The ''rsec/s'' column for drbd0 shows the number of sectors
read from disk per second. This value is pretty low if Puppet is not running.
* The ''%util'' (CPU time during the I/O requests) column for
drbd0 shows that the device is almost in saturation
* This is the mount point for drbd0: ''/dev/ drbd0 on
/repository'' and ''/repository/sonatype-work'' is the
directory hosted by Puppet
Again, this is the configuration:
file {
"/repository/sonatype-work":
ensure => directory,
owner => $nexus_user,
group => $nexus_group,
mode => 0755,
recurse => false,
backup => false,
}
For me it still seems there''s a kind of recursive scanning done by
Puppet though ''recurse'' is set to false.
Thanks for helping,
Bernd
Von: puppet-users@googlegroups.com [mailto:puppet-users@googlegroups.com] Im
Auftrag von Bernd Adamowicz
Gesendet: Donnerstag, 13. September 2012 14:45
An: ''puppet-users@googlegroups.com''
Betreff: AW: [Puppet Users] AW: Issue with large directory content
This keeps being weird. Simply thought to wait until Puppet finishes, but had to
quit after one and a half hour. Also tried ''ensure =>
present'' instead of ''ensure => directory'' with no
success. No log output at all. Still investigating. But any ideas still highly
appreciated!
Bernd
Von: puppet-users@googlegroups.com [mailto:puppet-users@googlegroups.com] Im
Auftrag von Bernd Adamowicz
Gesendet: Donnerstag, 13. September 2012 10:46
An: puppet-users@googlegroups.com
Betreff: Re: [Puppet Users] AW: Issue with large directory content
Thanks for your answers so far.
But beware that the huge artifacts are *not* managed by Puppet (see recurse
=> false). Actually it''s a Maven repository filled by Nexus. Only
the top directory is managed by Puppet to have it in place and have correct
access rights. This worked well until I initially filled the repository with the
artifacts manually. That slowed down Puppet. Seems to me as Puppet will do some
recursive scanning, but that''s just an assumption, since Puppet is
running with almost 100% CPU load.
I turned debug on in Puppet but will not see anything even after a few minutes.
Presumably I would see something if I let Puppet run just long enough. However,
it''s a strange behaviour I''ve never experienced. And I think
my configuration is OK.
Bernd
On 09/13/2012 01:24 AM, Peter Brown wrote:
On 13 September 2012 00:12, Christopher Wood <christopher_wood@pobox.com>
wrote:> I don''t have enough information to say. You might want to run the
master and agent in debug mode to get more output, though.
>
> puppet agent --debug --verbose --no-daemonize
>
> Also, 100 GB? Any particular reason why you''re not installing this
using a content distribution system or a large number of RPMs?
Recursing through A 100Gb directory will definitely slow down your puppet run.
If the contents of the directory are reasonably static an RPM would be
the best idea.
If it''s not static a git or svn repo would be a better idea.
If you are tricky you can manage the checkouts of git or svn with
puppet as well.
I wrote a few tricky resources for this a while ago and they are
infinitely handy.
>
> On Wed, Sep 12, 2012 at 04:08:13PM +0200, Bernd Adamowicz wrote:
>> No ideas at all?
>>
>> > -----Ursprüngliche Nachricht-----
>> > Von: Bernd Adamowicz
>> > Gesendet: Dienstag, 11. September 2012 16:16
>> > An: puppet-users@googlegroups.com
>> > Betreff: Issue with large directory content
>> >
>> > Hi all,
>> >
>> > I got this directory configuration:
>> >
>> >
>> > file {
>> > "${codebase_ng::repository_mount}/${sonatype_work_dir}":
>> > ensure => directory,
>> > owner => $nexus_user,
>> > group => $nexus_group,
>> > mode => 0755,
>> > recurse => false,
>> > backup => false,
>> > }
>> >
>> > Today I added some 100GB of artifacts to a subdirectory of
>> > "${codebase_ng::repository_mount}/${sonatype_work_dir}".
Now the result
>> > is that the Puppet seems to run "forever". If I
uncomment this code,
>> > Puppet finishes in 15 seconds. So I presume Puppet is doing some
>> > recursive scanning of this directory. Could this be true? Is there
a
>> > know issue with large content of directories?
>> >
>> > Thanks in advance!
>> > Bernd
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Nigel Kersten
2012-Sep-13 18:45 UTC
Re: [Puppet Users] AW: Issue with large directory content
On Thu, Sep 13, 2012 at 8:55 AM, Bernd Adamowicz <Bernd.Adamowicz@esailors.de> wrote:> For me it still seems there''s a kind of recursive scanning done by Puppet though ''recurse'' is set to false.This does look like a bug, as you''ve got a directory with no remote source, no recursion, and all you want to do is to enforce owner/group/mode on it. That shouldn''t be an I/O intensive process at all. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Denmat
2012-Sep-13 21:05 UTC
Re: WG: [Puppet Users] AW: Issue with large directory content
Hi, Not sure if I can help but just have couple of questions. How long does it take to do a ''find'' on the directory or a ''stat''? Have you tried running the client with --verbose --debug --evaltrace --summarize? Are you trying to set any permissions inside that directory elsewhere in the manifest? I would expect recurse => false not to do anything too, but (and this should throw an error) can you set recurselimit? If so, does it make any difference? HTH Den On 14/09/2012, at 1:55, Bernd Adamowicz <Bernd.Adamowicz@esailors.de> wrote:> Hope, the list doesn''t blame me for spamming, but I''d just like to provide some more information on my problem, still hoping someone can help. > > Using the ''iostat'' command while Puppet is stuck, I can clearly see that there''s a very high I/O on the disk: > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util > sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > fioa 0.00 0.00 163.00 5.00 31280.00 40.00 186.43 0.04 0.25 0.25 4.20 > dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > dm-5 0.00 0.00 163.00 5.00 31280.00 40.00 186.43 0.04 0.25 0.25 4.20 > dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > drbd0 0.00 0.00 163.00 5.00 31280.00 40.00 186.43 0.96 0.53 5.72 96.10 > drbd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > * The ''drbd0'' device is an HP IO Accelerator card > * The ''rsec/s'' column for drbd0 shows the number of sectors read from disk per second. This value is pretty low if Puppet is not running. > * The ''%util'' (CPU time during the I/O requests) column for drbd0 shows that the device is almost in saturation > * This is the mount point for drbd0: ''/dev/ drbd0 on /repository'' and ''/repository/sonatype-work'' is the directory hosted by Puppet > > Again, this is the configuration: > > file { > "/repository/sonatype-work": > ensure => directory, > owner => $nexus_user, > group => $nexus_group, > mode => 0755, > recurse => false, > backup => false, > } > > For me it still seems there''s a kind of recursive scanning done by Puppet though ''recurse'' is set to false. > > Thanks for helping, > Bernd > > Von: puppet-users@googlegroups.com [mailto:puppet-users@googlegroups.com] Im Auftrag von Bernd Adamowicz > Gesendet: Donnerstag, 13. September 2012 14:45 > An: ''puppet-users@googlegroups.com'' > Betreff: AW: [Puppet Users] AW: Issue with large directory content > > This keeps being weird. Simply thought to wait until Puppet finishes, but had to quit after one and a half hour. Also tried ''ensure => present'' instead of ''ensure => directory'' with no success. No log output at all. Still investigating. But any ideas still highly appreciated! > > Bernd > > Von: puppet-users@googlegroups.com [mailto:puppet-users@googlegroups.com] Im Auftrag von Bernd Adamowicz > Gesendet: Donnerstag, 13. September 2012 10:46 > An: puppet-users@googlegroups.com > Betreff: Re: [Puppet Users] AW: Issue with large directory content > > Thanks for your answers so far. > > But beware that the huge artifacts are *not* managed by Puppet (see recurse => false). Actually it''s a Maven repository filled by Nexus. Only the top directory is managed by Puppet to have it in place and have correct access rights. This worked well until I initially filled the repository with the artifacts manually. That slowed down Puppet. Seems to me as Puppet will do some recursive scanning, but that''s just an assumption, since Puppet is running with almost 100% CPU load. > > I turned debug on in Puppet but will not see anything even after a few minutes. Presumably I would see something if I let Puppet run just long enough. However, it''s a strange behaviour I''ve never experienced. And I think my configuration is OK. > > Bernd > > On 09/13/2012 01:24 AM, Peter Brown wrote: > On 13 September 2012 00:12, Christopher Wood <christopher_wood@pobox.com> wrote: >> I don''t have enough information to say. You might want to run the master and agent in debug mode to get more output, though. >> >> puppet agent --debug --verbose --no-daemonize >> >> Also, 100 GB? Any particular reason why you''re not installing this using a content distribution system or a large number of RPMs? > > Recursing through A 100Gb directory will definitely slow down your puppet run. > If the contents of the directory are reasonably static an RPM would be > the best idea. > If it''s not static a git or svn repo would be a better idea. > If you are tricky you can manage the checkouts of git or svn with > puppet as well. > I wrote a few tricky resources for this a while ago and they are > infinitely handy. > >> >> On Wed, Sep 12, 2012 at 04:08:13PM +0200, Bernd Adamowicz wrote: >>> No ideas at all? >>> >>>> -----Ursprüngliche Nachricht----- >>>> Von: Bernd Adamowicz >>>> Gesendet: Dienstag, 11. September 2012 16:16 >>>> An: puppet-users@googlegroups.com >>>> Betreff: Issue with large directory content >>>> >>>> Hi all, >>>> >>>> I got this directory configuration: >>>> >>>> >>>> file { >>>> "${codebase_ng::repository_mount}/${sonatype_work_dir}": >>>> ensure => directory, >>>> owner => $nexus_user, >>>> group => $nexus_group, >>>> mode => 0755, >>>> recurse => false, >>>> backup => false, >>>> } >>>> >>>> Today I added some 100GB of artifacts to a subdirectory of >>>> "${codebase_ng::repository_mount}/${sonatype_work_dir}". Now the result >>>> is that the Puppet seems to run "forever". If I uncomment this code, >>>> Puppet finishes in 15 seconds. So I presume Puppet is doing some >>>> recursive scanning of this directory. Could this be true? Is there a >>>> know issue with large content of directories? >>>> >>>> Thanks in advance! >>>> Bernd > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.