CentOS - Mar 2005 - How to make a mirror of CentOS sites?

Hi friends:

I have 20 CentOS production servers in my company, so I want to set up an update
mirror in my intranet, Does anybody know a script or something to do this?

regards

Israel

Hi friends:

I have 20 CentOS production servers in my company, so I want to set up an update
mirror in my intranet, Does anybody know a script or something to do this?

regards

Israel

Hi friends:

I have 20 CentOS production servers in my company, so I want to set up an update
mirror in my intranet, Does anybody know a script or something to do this?

regards

Israel

israel.garcia@cimex.com.cu wrote:
> Hi friends:
>
> I have 20 CentOS production servers in my company, so I want to set up
> an update mirror in my intranet, Does anybody know a script or
> something to do this?
I just do a rsync from a mirror close to me. The mirrors supporting
rsync can be found on the mirror page on <http://www.centos.org/>.

So there''s no great scripting involved ...

Ralph
--
Ralph Angenendt......ra@br-online.de | .."Text processing has made it
possible
Bayerischer Rundfunk...HA-Multimedia | ....to right-justify any idea, even one
Rundfunkplatz 1.......80300 Muenchen | .which cannot be justified on any other
Tl:089.5900.16023..Fx:089.5900.16240 | ..........grounds." -- J. Finnegan,
USC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url :
http://lists.caosity.org/pipermail/centos/attachments/20050322/85889cd8/attachment-0001.bin

On Mar 22, 2005, at 10:09 AM, <israel.garcia@cimex.com.cu> wrote:
> Hi friends:
>
> I have 20 CentOS production servers in my company, so I want to set up
> an update
> mirror in my intranet, Does anybody know a script or something to do
> this?
You could use Current to setup your own local up2date server.

http://current.tigris.org/

--
Jason Dixon
DixonGroup Consulting
http://www.dixongroup.net

israel.garcia@cimex.com.cu wrote:
>Hi friends:
>
>I have 20 CentOS production servers in my company, so I want to set up an
update
>mirror in my intranet, Does anybody know a script or something to do this?
>
>regards
>
>Israel
>
>_______________________________________________
>CentOS mailing list
>CentOS@caosity.org
>http://lists.caosity.org/mailman/listinfo/centos
>
>
>
I hope you''re not sending this from all 20 servers. :)

John Hinton <who has seen this three times now>

israel.garcia@cimex.com.cu wrote:
> Hi friends:
>
> I have 20 CentOS production servers in my company, so I want to set up
> an update mirror in my intranet, Does anybody know a script or
> something to do this?
> I just do a rsync from a mirror close to me. The mirrors supporting
> rsync can be found on the mirror page on <http://www.centos.org/>.
> So there''s no great scripting involved ..
>Ralph
Hi Ralph, thanks for your soon answer, but my access to internet is trough http
proxies, rsyn is not allowed. I think wget  can do this but I dont know how?

thanks

israel

israel.garcia@cimex.com.cu wrote:
> Hi friends:
>
> I have 20 CentOS production servers in my company, so I want to set up
> an update mirror in my intranet, Does anybody know a script or
> something to do this?
> I just do a rsync from a mirror close to me. The mirrors supporting
> rsync can be found on the mirror page on <http://www.centos.org/>.
> So there''s no great scripting involved ..
>Ralph
Hi Ralph, thanks for your soon answer, but my access to internet is trough http
proxies, rsyn is not allowed. I think wget  can do this but I dont know how?

thanks

israel

israel.garcia@cimex.com.cu wrote:
>israel.garcia@cimex.com.cu wrote:
>
>
>>Hi friends:
>>
>>I have 20 CentOS production servers in my company, so I want to set up
>>an update mirror in my intranet, Does anybody know a script or
>>something to do this?
>>
>>
>
>
>
>>I just do a rsync from a mirror close to me. The mirrors supporting
>>rsync can be found on the mirror page on <http://www.centos.org/>.
>>
>>
>
>
>
>>So there''s no great scripting involved ..
>>Ralph
>>
>>
>Hi Ralph, thanks for your soon answer, but my access to internet is trough
http
>proxies, rsyn is not allowed. I think wget  can do this but I dont know how?
>
>thanks
>
>israel
>
>_______________________________________________
>CentOS mailing list
>CentOS@caosity.org
>http://lists.caosity.org/mailman/listinfo/centos
>
>
wget -r?

--
Brian Trudeau,   I.T., Q.A. Inspector
Eastek International Corporation
330 Hastings Drive,   Buffalo Grove, IL 60089
Tel: (847) 353-8300 Ext. 213   Fax: (847) 353-8900
Web: http://www.eastek-intl.com   Email: btrudeau@eastek-intl.com
----
The information contained in this electronic mail transmission is intended by
Eastek International for the use of the named individual or entity to which it
is directed and may contain information that is confidential or privileged.

If you are not the intended recipient, you must not keep, use, disclose, copy or
distribute this email without the author''s prior permission. We have
taken precautions to minimize the risk of transmitting software viruses, but we
advise you to carry out your own virus checks on any attachment to this message.
We cannot accept liability for any loss or damage caused by software viruses or
other attachments.

If you have received this electronic mail transmission in error, please delete
it from your system without copying or forwarding it, and notify the sender of
the error by reply email so that the sender''s address records can be
corrected.  Thank you.

On Tue, Mar 22, 2005 at 10:42:00AM -0500, israel.garcia@cimex.com.cu wrote:
> Hi Ralph, thanks for your soon answer, but my access to internet is trough
http
> proxies, rsyn is not allowed. I think wget  can do this but I dont know
how?
>
use lftp(1)

lftp your_ftp_target <<EOF
cd ...../centos
mirror
EOF

Tru

On Tue, 2005-03-22 at 09:42, israel.garcia@cimex.com.cu
wrote:
> >
> > I have 20 CentOS production servers in my company, so I want to set up
> > an update mirror in my intranet, Does anybody know a script or
> > something to do this?
>
> > I just do a rsync from a mirror close to me. The mirrors supporting
> > rsync can be found on the mirror page on
<http://www.centos.org/>.
>
> Hi Ralph, thanks for your soon answer, but my access to internet is trough
http
> proxies, rsyn is not allowed. I think wget  can do this but I dont know
how?
In that case, just point all your machines to the same proxy and let the
automatic caching take care of things:
http_proxy=http://your_proxy_server:port yum update
You should be sure that the proxy server is set up to cache large files.

--
  Les Mikesell
   les@futuresource.com

> I hope you''re not sending this from all 20 servers. :)
>
> John Hinton <who has seen this three times now>
I have (stcl.edu is my domain)
Timothy
> _______________________________________________
> CentOS mailing list
> CentOS@caosity.org
> http://lists.caosity.org/mailman/listinfo/centos

On Tue, March 22, 2005 9:23 am, Tru Huynh said:
> On Tue, Mar 22, 2005 at 10:42:00AM -0500, israel.garcia@cimex.com.cu
> wrote:
>
>> Hi Ralph, thanks for your soon answer, but my access to internet is
>> trough http
>> proxies, rsyn is not allowed. I think wget  can do this but I dont know
>> how?
>>
> use lftp(1)
>
> lftp your_ftp_target <<EOF
> cd ...../centos
> mirror
> EOF
>
> Tru
> _______________________________________________
> CentOS mailing list
> CentOS@caosity.org
> http://lists.caosity.org/mailman/listinfo/centos
>
I second Tru''s recommendation for lftp (if rsync can''t be
used).

You can mirror either an http or ftp tree.  It won''t redownload items
that
already exist.

The only real difference is that if you need to redownload something (like
an ISO) it doesn''t recoup old data ... which rsync does do.

You really need to get rsync enabled ... as it greatly minimizes bandwidth
usage on files that change, but have the same name.

But if you can''t use rsync, then I think lftp is the most flexible
tool.


--
Johnny Hughes
<http://www.HughesJR.com/>

Jason Dixon said:
> On Mar 22, 2005, at 10:09 AM, <israel.garcia@cimex.com.cu> wrote:
>
>
>> Hi friends:
>>
>>
>> I have 20 CentOS production servers in my company, so I want to set up
>> an update mirror in my intranet, Does anybody know a script or
something
>> to do this?
>
> You could use Current to setup your own local up2date server.
>
>
> http://current.tigris.org/
Since up2date now supports apt and yum repos, and Current handles none of
the client management options of up2date yet (or so it appears skimming
the docs), I''d have to ask why?  A yum repo can do the same job with a
lot
less setup.

--
William Hooper

I believe Current also requires that you setup Postgresql.  Which for
a repo is unneeded.  The idea of Current was to use up2date when no
other software (yum/apt) was available.


--
Matt Shields
http://masnetworks.biz
http://sexydates4u.com
http://shieldslinux.com
http://shieldsproductions.com

On Tue, 22 Mar 2005 17:37:17 -0500 (EST), William Hooper
<whooperhsd3@earthlink.net> wrote:
>
> Jason Dixon said:
> > On Mar 22, 2005, at 10:09 AM, <israel.garcia@cimex.com.cu>
wrote:
> >
> >
> >> Hi friends:
> >>
> >>
> >> I have 20 CentOS production servers in my company, so I want to
set up
> >> an update mirror in my intranet, Does anybody know a script or
something
> >> to do this?
> >
> > You could use Current to setup your own local up2date server.
> >
> >
> > http://current.tigris.org/
>
> Since up2date now supports apt and yum repos, and Current handles none of
> the client management options of up2date yet (or so it appears skimming
> the docs), I''d have to ask why?  A yum repo can do the same job
with a lot
> less setup.
>
> --
> William Hooper
>
> _______________________________________________
> CentOS mailing list
> CentOS@caosity.org
> http://lists.caosity.org/mailman/listinfo/centos
>

I like and use yam from DAG''s repo for this job.

On Tue, 2005-03-22 at 17:54 -0500, Matt Shields wrote:
> I believe Current also requires that you setup Postgresql.  Which for
> a repo is unneeded.  The idea of Current was to use up2date when no
> other software (yum/apt) was available.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.caosity.org/pipermail/centos/attachments/20050322/54373315/attachment.htm

This is how we are doing it here. I setup an rsync job to sync the files
from one of the Centos mirrors to our local server, told apache to serve
up that directory, and fashioned a yum.conf which pointed at that URL.
This way we are able to track updates, but freeze the repository to do a
site-wide rollout over the course of a few weeks without worrying about
untested packages sneaking in.

I''m happy to share our script and/or config if you want more details.

-----Original Message-----
From: centos-bounces@caosity.org [mailto:centos-bounces@caosity.org] On
Behalf Of William Hooper
Sent: Tuesday, March 22, 2005 2:37 PM
To: centos@caosity.org
Subject: Re: [Centos] How to make a mirror of CentOS sites?


Jason Dixon said:
> On Mar 22, 2005, at 10:09 AM, <israel.garcia@cimex.com.cu> wrote:
>
>
>> Hi friends:
>>
>>
>> I have 20 CentOS production servers in my company, so I want to set
>> up an update mirror in my intranet, Does anybody know a script or
>> something to do this?
>
> You could use Current to setup your own local up2date server.
>
>
> http://current.tigris.org/
Since up2date now supports apt and yum repos, and Current handles none
of the client management options of up2date yet (or so it appears
skimming the docs), I''d have to ask why?  A yum repo can do the same
job
with a lot less setup.

--
William Hooper

_______________________________________________
CentOS mailing list
CentOS@caosity.org
http://lists.caosity.org/mailman/listinfo/centos

On Tue, 2005-03-22 at 09:26 -0500, Jason Dixon wrote:
> On Mar 22, 2005, at 10:09 AM, <israel.garcia@cimex.com.cu> wrote:
>
> > Hi friends:
> >
> > I have 20 CentOS production servers in my company, so I want to set up
> > an update
> > mirror in my intranet, Does anybody know a script or something to do
> > this?
>
> You could use Current to setup your own local up2date server.
>
> http://current.tigris.org/
>
A pain to setup and totally unneeded since up2date now supports yum
repos.  I ran it for RH 7.3 servers at work, but since switching them
over I just mirror a close repo and point the servers at the local
system.

Paul

On Tue, 2005-03-22 at 16:23 +0100, Tru Huynh wrote:
> On Tue, Mar 22, 2005 at 10:42:00AM -0500, israel.garcia@cimex.com.cu wrote:
>
> > Hi Ralph, thanks for your soon answer, but my access to internet is
trough http
> > proxies, rsyn is not allowed. I think wget  can do this but I dont
know how?
> >
> use lftp(1)
>
> lftp your_ftp_target <<EOF
> cd ...../centos
> mirror
> EOF
>

The script I use is as follows:
#!/bin/sh
#
# mirror-centos.sh - Mirror CentOS Linux Updates
#

# Variables
MIRROR="mirror.cs.wisc.edu"
WBDIR="pub/mirrors/linux/caosity.org/centos"
VERSIONS="4"
ARCHS="SRPMS i386 x86_64"
WBUPDATES="/home/ftp/pub/centos"
PROXY="http://firewall.company.com:8080/"

# Limit Transfer Rate to 10K bytes per second
MAXRATE="10k"

# Do the mirror
for d in $VERSIONS; do
  for y in $ARCHS; do
    cd $WBUPDATES
    ## env http_proxy=$PROXY \
    wget --non-verbose --no-host-directories --cut-dirs=8 --mirror \
     --passive-ftp --limit-rate=$MAXRATE \
     --directory-prefix=$WBUPDATES/$d/updates/$y \
     ftp://$MIRROR/$WBDIR/$d/updates/$y
     ##--output-file=/tmp/mirror-centos.log

    # the RPMS should be checked for valid sigs, not ready for that yet
    ##rpm --checksig $WBUPDATES/$d/updates/$y/*.rpm | grep ''NOT
OK''
  done
done


# Done
exit

On Tue, 2005-03-22 at 18:07, Foster, Mark wrote:
> This is how we are doing it here. I setup an rsync job to sync the files
> from one of the Centos mirrors to our local server, told apache to serve
> up that directory, and fashioned a yum.conf which pointed at that URL.
> This way we are able to track updates, but freeze the repository to do a
> site-wide rollout over the course of a few weeks without worrying about
> untested packages sneaking in.
>
> I''m happy to share our script and/or config if you want more
details.
I''ll have to do it that way one of these days...  Doing http through a
proxy cache takes care the bandwidth issue but it doesn''t give you
any control over the contents.  I really wish the package manager
programs had some kind of revision control concept even if it was
as simple as ''ignore updates newer than timestamp'' where
timestamp
is the point where you have tested, so you don''t have surprises
on production machines.

--
  Les Mikesell
   les@futuresource.com