Puppet users - Dec 2012 - Broken rsync mirroring for PuppetLabs APT repo

Hi,

I have sent this mail to info@puppetlabs.com and received no answer, so
I''m posting here as well; I hope somebody at PuppetLabs will be able to
help.

I have a local mirror of apt.puppetlabs.com which has been broken for at
least a few days: the apt/dists/ subdirectory only contains symlinks to
non-existent files.

[mirror@excellent dists]$ ls -l
total 0
lrwxrwxrwx 1 mirror mirror 29 21 déc.  01:45 hardy ->
hardy-20121220164502045137453
lrwxrwxrwx 1 mirror mirror 29 21 déc.  01:53 lucid ->
lucid-20121220165208876527646
lrwxrwxrwx 1 mirror mirror 29 21 déc.  01:57 natty ->
natty-20121220165607590234617
lrwxrwxrwx 1 mirror mirror 31 21 déc.  01:47 oneiric ->
oneiric-20121220164616495945899
lrwxrwxrwx 1 mirror mirror 31 21 déc.  01:54 precise ->
precise-20121220165341748274165
lrwxrwxrwx 1 mirror mirror 31 21 déc.  01:46 quantal ->
quantal-20121220164539080938625
lrwxrwxrwx 1 mirror mirror 27 21 déc.  01:49 sid ->
sid-20121220164825095308145
lrwxrwxrwx 1 mirror mirror 31 21 déc.  01:52 squeeze ->
squeeze-20121220165038896453411
lrwxrwxrwx 1 mirror mirror 30 21 déc.  01:50 stable ->
stable-20121220164951162557153
lrwxrwxrwx 1 mirror mirror 31 21 déc.  01:48 testing ->
testing-20121220164736915534425
lrwxrwxrwx 1 mirror mirror 32 21 déc.  01:58 unstable ->
unstable-20121220165727244887087
lrwxrwxrwx 1 mirror mirror 30 21 déc.  01:56 wheezy ->
wheezy-20121220165450654775488

This mirror is maintained with rsync. I have tried downloading a fresh
copy of the repo on another machine in case I have somehow setup rsync
wrong, and I can confirm the issue is not on my side.

-- 
A

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Arnaud Gomes-do-Vale wrote:
> Hi,
> 
> I have sent this mail to info@puppetlabs.com and received no answer, so
> I''m posting here as well; I hope somebody at PuppetLabs will be
able to
> help.
Arnaud

I saw your saw email to info@. It arrived over Christmas so I am afraid
people were thin on the ground at the office. I''ll ask our Delivery and
Ops teams to take a look and see if we can work out what''s wrong.

Thanks for letting us know!

Regards

James

-- 
James Turnbull
1-503-734-8571
To schedule a meeting with me: http://doodle.com/jamtur01
We are moving! New address: Puppet Labs, Inc., 926 NW 13th Ave., Suite
210, Portland, OR 97209.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Arnaud,
What flags are you passing to rsync? The README at
http://apt.puppetlabs.com/README.txt offers a sample rsync command
which includes the --copy-links flag. When I rsync using copy links,
my dists directory looks like:

total 0
drwxrwsr-x  7 matthaus  staff  238 Dec 20 16:45 hardy
drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:27 lucid
drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:28 natty
drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:30 oneiric
drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:37 precise
drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:38 quantal
drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:33 sid
drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:34 squeeze
drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:39 stable
drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:39 testing
drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:31 unstable
drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:36 wheezy

Without --copy-links, the dists directory looks like what you have described.

HTH

On Thu, Dec 27, 2012 at 5:59 AM, Arnaud Gomes-do-Vale
<Arnaud.Gomes@ircam.fr> wrote:
> Hi,
>
> I have sent this mail to info@puppetlabs.com and received no answer, so
> I''m posting here as well; I hope somebody at PuppetLabs will be
able to
> help.
>
> I have a local mirror of apt.puppetlabs.com which has been broken for at
> least a few days: the apt/dists/ subdirectory only contains symlinks to
> non-existent files.
>
> [mirror@excellent dists]$ ls -l
> total 0
> lrwxrwxrwx 1 mirror mirror 29 21 déc.  01:45 hardy ->
hardy-20121220164502045137453
> lrwxrwxrwx 1 mirror mirror 29 21 déc.  01:53 lucid ->
lucid-20121220165208876527646
> lrwxrwxrwx 1 mirror mirror 29 21 déc.  01:57 natty ->
natty-20121220165607590234617
> lrwxrwxrwx 1 mirror mirror 31 21 déc.  01:47 oneiric ->
oneiric-20121220164616495945899
> lrwxrwxrwx 1 mirror mirror 31 21 déc.  01:54 precise ->
precise-20121220165341748274165
> lrwxrwxrwx 1 mirror mirror 31 21 déc.  01:46 quantal ->
quantal-20121220164539080938625
> lrwxrwxrwx 1 mirror mirror 27 21 déc.  01:49 sid ->
sid-20121220164825095308145
> lrwxrwxrwx 1 mirror mirror 31 21 déc.  01:52 squeeze ->
squeeze-20121220165038896453411
> lrwxrwxrwx 1 mirror mirror 30 21 déc.  01:50 stable ->
stable-20121220164951162557153
> lrwxrwxrwx 1 mirror mirror 31 21 déc.  01:48 testing ->
testing-20121220164736915534425
> lrwxrwxrwx 1 mirror mirror 32 21 déc.  01:58 unstable ->
unstable-20121220165727244887087
> lrwxrwxrwx 1 mirror mirror 30 21 déc.  01:56 wheezy ->
wheezy-20121220165450654775488
>
> This mirror is maintained with rsync. I have tried downloading a fresh
> copy of the repo on another machine in case I have somehow setup rsync
> wrong, and I can confirm the issue is not on my side.
>
> --
> A
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>


-- 
Matthaus Owens
Release Manager, Puppet Labs

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Matthaus Owens <matthaus@puppetlabs.com> writes:
> What flags are you passing to rsync? The README at
> http://apt.puppetlabs.com/README.txt offers a sample rsync command
> which includes the --copy-links flag. When I rsync using copy links,
> my dists directory looks like:
Of course you''re right, I used my usual rsync combination of flags,
that
is "rsync -avH"; using "rsync -avLH" fixed the problem.

Thanks!
-- 
A

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 12/28/2012 12:48 PM, Matthaus Owens wrote:
> Arnaud,
> What flags are you passing to rsync? The README at
> http://apt.puppetlabs.com/README.txt offers a sample rsync command
> which includes the --copy-links flag. When I rsync using copy links,
> my dists directory looks like:
>
> total 0
> drwxrwsr-x  7 matthaus  staff  238 Dec 20 16:45 hardy
> drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:27 lucid
> drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:28 natty
> drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:30 oneiric
> drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:37 precise
> drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:38 quantal
> drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:33 sid
> drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:34 squeeze
> drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:39 stable
> drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:39 testing
> drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:31 unstable
> drwxrwsr-x  7 matthaus  staff  238 Dec 27 11:36 wheezy
>
> Without --copy-links, the dists directory looks like what you have
described.
The requirement for the --copy-links option is why I submitted Issue #18055 [1].
I am maintaining a few other mirrors like CentOS and Fedora EPEL and using this 
flag on those mirrors would greatly increase the disk space and bandwidth 
required by my mirrors. Both these mirrors use symlinks like 5 -> 5.8. Even
the
puppetlabs mirror uses symlinks in the yum/el directory! The result is that the 
puppetlabs yum mirror with --copy-links uses far more disk space and bandwidth 
than necessary when used as recommended.

Initially I had mirrored puppetlabs using the same script that I used for my 
other rsync mirrors, but due to the --copy-links requirement I have had to split
out just the puppetlabs apt mirror into a separate script. I am still using my 
regular rsync script without --copy-links to mirror the puppetlabs/yum portion 
to avoid wasting disk space and bandwidth.

[1] https://projects.puppetlabs.com/issues/18055

-- 
Konrad Scherer, Sr. Engineer, Linux Products Group, Wind River
direct 613-963-1342   fax 613-592-2283

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.