Robin Lee Powell
2011-May-14 04:04 UTC
[Puppet Users] So, clientconfigs.sqlite3 is pretty big
I only have like 15 hosts, and the only stuff I''m aware that I''m saving via <<| |>>/@@ is ssh keys. It''s > 600 MiB. That seems really large? Should I just delete it every once in a while? -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ohad Levy
2011-May-14 10:13 UTC
Re: [Puppet Users] So, clientconfigs.sqlite3 is pretty big
On Sat, May 14, 2011 at 7:04 AM, Robin Lee Powell < rlpowell@digitalkingdom.org> wrote:> > I only have like 15 hosts, and the only stuff I''m aware that I''m > saving via <<| |>>/@@ is ssh keys. > > It''s > 600 MiB. > > That seems really large? > > Should I just delete it every once in a while? > > sqlite does not release disk space unless you vacuum it.Ohad> -Robin > > -- > http://singinst.org/ : Our last, best hope for a fantastic future. > Lojban (http://www.lojban.org/): The language in which "this parrot > is dead" is "ti poi spitaki cu morsi", but "this sentence is false" > is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Robin Lee Powell
2011-May-14 10:21 UTC
Re: [Puppet Users] So, clientconfigs.sqlite3 is pretty big
On Sat, May 14, 2011 at 01:13:19PM +0300, Ohad Levy wrote:> On Sat, May 14, 2011 at 7:04 AM, Robin Lee Powell < > rlpowell@digitalkingdom.org> wrote: > > > > > I only have like 15 hosts, and the only stuff I''m aware that I''m > > saving via <<| |>>/@@ is ssh keys. > > > > It''s > 600 MiB. > > > > That seems really large? > > > > Should I just delete it every once in a while? > > sqlite does not release disk space unless you vacuum it.OK. Shouldn''t puppet be responsible for doing that if it''s going to use sqlite on the backend, then? I don''t know much of anything about sqlite, and I''m surprised if I''d have to learn it to use puppet. Perhaps someone has a puppet snippet to handle this already? :) -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Stefan Schulte
2011-May-14 11:35 UTC
Re: [Puppet Users] So, clientconfigs.sqlite3 is pretty big
On Fri, May 13, 2011 at 09:04:35PM -0700, Robin Lee Powell wrote:> > I only have like 15 hosts, and the only stuff I''m aware that I''m > saving via <<| |>>/@@ is ssh keys. > > It''s > 600 MiB. > > That seems really large? > > Should I just delete it every once in a while? > > -Robin >You are talking about stored configuration right? Do you use thin_storeconfigs[1]? Otherwise puppet will not only store your exported ssh key resources in the database. [1]http://projects.puppetlabs.com/projects/1/wiki/Using_Stored_Configuration#Configuring+basic+storeconfigs -Stefan -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ohad Levy
2011-May-14 11:48 UTC
Re: [Puppet Users] So, clientconfigs.sqlite3 is pretty big
On Sat, May 14, 2011 at 1:21 PM, Robin Lee Powell < rlpowell@digitalkingdom.org> wrote:> On Sat, May 14, 2011 at 01:13:19PM +0300, Ohad Levy wrote: > > On Sat, May 14, 2011 at 7:04 AM, Robin Lee Powell < > > rlpowell@digitalkingdom.org> wrote: > > > > > > > > I only have like 15 hosts, and the only stuff I''m aware that I''m > > > saving via <<| |>>/@@ is ssh keys. > > > > > > It''s > 600 MiB. > > > > > > That seems really large? > > > > > > Should I just delete it every once in a while? > > > > sqlite does not release disk space unless you vacuum it. > > OK. Shouldn''t puppet be responsible for doing that if it''s going to > use sqlite on the backend, then? I don''t know much of anything > about sqlite, and I''m surprised if I''d have to learn it to use > puppet. > > Perhaps someone has a puppet snippet to handle this already? :) > > echo "VACUUM;" | sqlite3 <your db>to be on the safe side, back it up first :) Ohad> -Robin > > -- > http://singinst.org/ : Our last, best hope for a fantastic future. > Lojban (http://www.lojban.org/): The language in which "this parrot > is dead" is "ti poi spitaki cu morsi", but "this sentence is false" > is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Robin Lee Powell
2011-May-14 19:20 UTC
Re: [Puppet Users] So, clientconfigs.sqlite3 is pretty big
On Sat, May 14, 2011 at 01:35:51PM +0200, Stefan Schulte wrote:> On Fri, May 13, 2011 at 09:04:35PM -0700, Robin Lee Powell wrote: > > > > I only have like 15 hosts, and the only stuff I''m aware that I''m > > saving via <<| |>>/@@ is ssh keys. > > > > It''s > 600 MiB. > > > > That seems really large? > > > > Should I just delete it every once in a while? > > > > -Robin > > > > You are talking about stored configuration right?I have no idea; is that what''s in that file? -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Robin Lee Powell
2011-May-15 02:39 UTC
530 million backslashes (was Re: [Puppet Users] So, clientconfigs.sqlite3 is pretty big)
On Sat, May 14, 2011 at 01:35:51PM +0200, Stefan Schulte wrote:> On Fri, May 13, 2011 at 09:04:35PM -0700, Robin Lee Powell wrote: > > > > I only have like 15 hosts, and the only stuff I''m aware that I''m > > saving via <<| |>>/@@ is ssh keys. > > > > It''s > 600 MiB. > > > > That seems really large? > > > > Should I just delete it every once in a while? > > > > -Robin > > > > You are talking about stored configuration right? Do you use > thin_storeconfigs[1]?I do, in fact, use thin_storeconfigs. I seem to have hit something of a rather nasty bug here. I opened up the database and did a dump and discovered this: INSERT INTO "hosts" VALUES(2, ''tools01.c44031.blueboxgrid.com'', ''67.214.208.43'', ''--- "--- \"--- \\\"--- \\\\\\\"--- \\\\\\\\\\\\\\\ "--- \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\ \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\ \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\\\\\ \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ This goes on. For some time: [rlpowell@test02 ~]$ wc /tmp/dump 511 5877 536988618 /tmp/dump [rlpowell@test02 ~]$ cat /tmp/dump | tr ''\\'' ''\n'' | wc 536920329 5988 536988618 This means my database has 530 *MILLION* backslashes in it. Which strikes me as maybe not the best thing. Oddly </sarcasm>, this server crashes a lot. Which is how I found out this database was so big: I was stracing and it was doing nothing but reading lines of \\\\\\\ out of this file. I *think* I deleted this file already, and it got back to the same state within a couple of days, but I''m not sure so I''m trying again. So, I''m at a loss. Help? -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Nigel Kersten
2011-May-15 15:33 UTC
Re: 530 million backslashes (was Re: [Puppet Users] So, clientconfigs.sqlite3 is pretty big)
On Sun, May 15, 2011 at 2:39 AM, Robin Lee Powell <rlpowell@digitalkingdom.org> wrote:> On Sat, May 14, 2011 at 01:35:51PM +0200, Stefan Schulte wrote: >> On Fri, May 13, 2011 at 09:04:35PM -0700, Robin Lee Powell wrote: >> > >> > I only have like 15 hosts, and the only stuff I''m aware that I''m >> > saving via <<| |>>/@@ is ssh keys. >> > >> > It''s > 600 MiB. >> > >> > That seems really large? >> > >> > Should I just delete it every once in a while? >> > >> > -Robin >> > >> >> You are talking about stored configuration right? Do you use >> thin_storeconfigs[1]? > > I do, in fact, use thin_storeconfigs. > > I seem to have hit something of a rather nasty bug here. I opened > up the database and did a dump and discovered this: > > INSERT INTO "hosts" VALUES(2, ''tools01.c44031.blueboxgrid.com'', ''67.214.208.43'', ''--- "--- \"--- \\\"--- \\\\\\\"--- \\\\\\\\\\\\\\\ > "--- \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\ > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\ > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\\\\\ > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ > > This goes on. For some time: > > [rlpowell@test02 ~]$ wc /tmp/dump > 511 5877 536988618 /tmp/dump > [rlpowell@test02 ~]$ cat /tmp/dump | tr ''\\'' ''\n'' | wc > 536920329 5988 536988618 > > This means my database has 530 *MILLION* backslashes in it. > > Which strikes me as maybe not the best thing.This looks to be the issue that you''re hitting: http://projects.puppetlabs.com/issues/4487 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jacob Helwig
2011-May-15 16:57 UTC
Re: 530 million backslashes (was Re: [Puppet Users] So, clientconfigs.sqlite3 is pretty big)
On Sun, 15 May 2011 15:33:56 +0000, Nigel Kersten wrote:> > On Sun, May 15, 2011 at 2:39 AM, Robin Lee Powell > <rlpowell@digitalkingdom.org> wrote: > > On Sat, May 14, 2011 at 01:35:51PM +0200, Stefan Schulte wrote: > >> On Fri, May 13, 2011 at 09:04:35PM -0700, Robin Lee Powell wrote: > >> > > >> > I only have like 15 hosts, and the only stuff I''m aware that I''m > >> > saving via <<| |>>/@@ is ssh keys. > >> > > >> > It''s > 600 MiB. > >> > > >> > That seems really large? > >> > > >> > Should I just delete it every once in a while? > >> > > >> > -Robin > >> > > >> > >> You are talking about stored configuration right? Do you use > >> thin_storeconfigs[1]? > > > > I do, in fact, use thin_storeconfigs. > > > > I seem to have hit something of a rather nasty bug here. I opened > > up the database and did a dump and discovered this: > > > > INSERT INTO "hosts" VALUES(2, ''tools01.c44031.blueboxgrid.com'', ''67.214.208.43'', ''--- "--- \"--- \\\"--- \\\\\\\"--- \\\\\\\\\\\\\\\ > > "--- \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\ > > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\ > > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ > > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\\\\\ > > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ > > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ > > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ > > \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ > > > > This goes on. For some time: > > > > [rlpowell@test02 ~]$ wc /tmp/dump > > 511 5877 536988618 /tmp/dump > > [rlpowell@test02 ~]$ cat /tmp/dump | tr ''\\'' ''\n'' | wc > > 536920329 5988 536988618 > > > > This means my database has 530 *MILLION* backslashes in it. > > > > Which strikes me as maybe not the best thing. > > This looks to be the issue that you''re hitting: > http://projects.puppetlabs.com/issues/4487 >It definitely looks like you''re hitting 4487, as Nigel said. Right now the fix for that isn''t yet available in a released version of Puppet. If you''re comfortable running a pre-release version, the fix is in the 2.6.x branch of the git repository. It will be in the next point release of the 2.6.x series, if you don''t feel comfortable running a pre-release version of Puppet. -- Jacob Helwig
Robin Lee Powell
2011-May-15 22:56 UTC
Re: 530 million backslashes (was Re: [Puppet Users] So, clientconfigs.sqlite3 is pretty big)
On Sun, May 15, 2011 at 09:57:29AM -0700, Jacob Helwig wrote:> On Sun, 15 May 2011 15:33:56 +0000, Nigel Kersten wrote: > > > > This looks to be the issue that you''re hitting: > > http://projects.puppetlabs.com/issues/4487 > > > > It definitely looks like you''re hitting 4487, as Nigel said. > > Right now the fix for that isn''t yet available in a released > version of Puppet. If you''re comfortable running a pre-release > version, the fix is in the 2.6.x branch of the git repository. > > It will be in the next point release of the 2.6.x series, if you > don''t feel comfortable running a pre-release version of Puppet.Great stuff, thanks. Can I run current git on just the master to fix this? -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which "this parrot is dead" is "ti poi spitaki cu morsi", but "this sentence is false" is "na nei". My personal page: http://www.digitalkingdom.org/rlp/ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jacob Helwig
2011-May-16 00:08 UTC
Re: 530 million backslashes (was Re: [Puppet Users] So, clientconfigs.sqlite3 is pretty big)
On Sun, 15 May 2011 15:56:01 -0700, Robin Lee Powell wrote:> > On Sun, May 15, 2011 at 09:57:29AM -0700, Jacob Helwig wrote: > > On Sun, 15 May 2011 15:33:56 +0000, Nigel Kersten wrote: > > > > > > This looks to be the issue that you''re hitting: > > > http://projects.puppetlabs.com/issues/4487 > > > > > > > It definitely looks like you''re hitting 4487, as Nigel said. > > > > Right now the fix for that isn''t yet available in a released > > version of Puppet. If you''re comfortable running a pre-release > > version, the fix is in the 2.6.x branch of the git repository. > > > > It will be in the next point release of the 2.6.x series, if you > > don''t feel comfortable running a pre-release version of Puppet. > > Great stuff, thanks. Can I run current git on just the master to > fix this? > > -Robin >That should do it. It was a completely server-side fix. -- Jacob Helwig