i've maintained a local centos repository at work using rsync, but it
seems the corp honchos have decided to block rsync at our firewall, plus
its never been 100% reliable, I'd get aborts on protocol errors
sometimes several times before pulling down a complete new distro update.
i'm trying to figure out how to do this with lftp, and its got me
somewhat stymied.
I'm testing with --dry-run like...
lftp -c 'open http://mirrors.kernel.org/centos/ && mirror -c -X
\*/ia64 -X \*/s390 -X \*/alpha -X \*/SRPMS --dry-run'
but my -X excludes don't seem to work (I don't want/need itanium, s390,
alpha, or SRPMs in my repo).
and, it doesn't seem to be aware of the symlinks from, for example, /5/
to /5.6/ and seems to want to pull down two copies instead (I
understand that http doesn't make symlinks obvious). Should I use
ftp:// instead of http:// or something?
does someone have a script for maintaining a repo with lftp they'd like
to share? what I saw on the wiki wasn't very helpful.
--
john r pierce N 37, W 122
santa cruz ca mid-left coast
On Jun 27, 2011, at 6:26 PM, John R Pierce <pierce at hogranch.com> wrote:> i've maintained a local centos repository at work using rsync, but it > seems the corp honchos have decided to block rsync at our firewall, plus > its never been 100% reliable, I'd get aborts on protocol errors > sometimes several times before pulling down a complete new distro update. > > i'm trying to figure out how to do this with lftp, and its got me > somewhat stymied. > > > I'm testing with --dry-run like... > > lftp -c 'open http://mirrors.kernel.org/centos/ && mirror -c -X > \*/ia64 -X \*/s390 -X \*/alpha -X \*/SRPMS --dry-run' > > but my -X excludes don't seem to work (I don't want/need itanium, s390, > alpha, or SRPMs in my repo). > > and, it doesn't seem to be aware of the symlinks from, for example, /5/ > to /5.6/ and seems to want to pull down two copies instead (I > understand that http doesn't make symlinks obvious). Should I use > ftp:// instead of http:// or something? > > > does someone have a script for maintaining a repo with lftp they'd like > to share? what I saw on the wiki wasn't very helpful.I have a wget repo mirroring script. I'll dig it up and post it tonight when I get home. -Ross
On Mon, 27 Jun 2011, John R Pierce wrote:> does someone have a script for maintaining a repo with lftp they'd like > to share? what I saw on the wiki wasn't very helpful.[root at xps400 ~]# grep kernel *conf lftp-centos-4-updates.conf: ftp://mirrors.kernel.org/centos/4/updates/i386 \ lftp-centos-4-updates.conf: ftp://mirrors.kernel.org/centos/4/updates/x86_64\ lftp-centos-5-updates.conf: ftp://mirrors.kernel.org/centos/5/updates/i386 \ lftp-centos-5-updates.conf: ftp://mirrors.kernel.org/centos/5/updates/x86_64\ lftp-openwall.conf: ftp://mirrors.kernel.org/openwall/Owl/contrib/2.0/SRPMS/ \ [root at xps400 ~]# cat lftp-centos-5-updates.conf # # Get the Centos 5 updates # mirror -c -e \ ftp://mirrors.kernel.org/centos/5/updates/i386 \ /var/ftp/pub/mirror/centos/centos-5/5/updates/i386/RPMS # mirror -c -e \ ftp://mirrors.kernel.org/centos/5/updates/x86_64 \ /var/ftp/pub/mirror/centos/centos-5/5/updates/x86_64/RPMS # # The following line is for our yum-arch and # createrepo logic # # yum: /var/ftp/pub/mirror/centos/centos-4/4/updates/i386/RPMS # [root at xps400 ~]# This is warpped in a driver script that walks through the directory, lokking for files ending in .conf -- when it finds them it runs: lftp -f /root/lftp-centos-5-updates.conf which script is linked into /etc/cron.daily/ I do a lot more as well (lockfiles, deltas, emailled reporting of unexpected variances, and so forth) so extract its essence here A google search with: site:orcorc.blogspot.com lftp will turn up relevant links -I and -X globbing are arcane, and here are some examples I use [root at xps400 ~]# cat lftp-RHEL-enterprise-all-srpms.conf # # # mirror -c \ -I "*/SRPMS/*" \ ftp://ftp.redhat.com/pub/redhat/linux/enterprise/ \ /var/ftp/pub/mirror/redhat/rhel/at-release/SRPMSonly # mirror -c \ -I "*/SRPMS/*" \ ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/ \ /var/ftp/pub/mirror/redhat/rhel/updates/SRPMSonly # # new in 5 mirror -c \ -I "*/SRPMS/*" \ ftp://ftp.redhat.com/pub/redhat/linux/updates/rhn/ \ /var/ftp/pub/mirror/redhat/rhel/updates/rhn/SRPMSonly # mirror -c \ -I "*/SRPMS/*" \ ftp://ftp.redhat.com/pub/redhat/linux/eal/ \ /var/ftp/pub/mirror/redhat/rhel/eal/SRPMSonly # mirror -c \ -I "*/SRPMS/*" \ -X "*/4AS/*" \ -X "*/4Desktop/*" \ -X "*/4ES/*" \ -X "*/4WS/*" \ -X "*/5Client/*" \ -X "*/5Server/*" \ -X "*/RHHPC/*" \ -X "RHHPC/*" \ ftp://ftp.redhat.com/pub/redhat/linux/beta/ \ /var/ftp/pub/mirror/redhat/rhel/beta/SRPMSonly # # added X for RHHPC 2011-06-20 # That last -X specification seemed to be needed (although one out think the entry above it would provide the same effect). It does not, as 'lftp' uses a textual, rather than a logical parsing, and "*/RHHPC/*" != "RHHPC/*" as the match on the leading "*/" is not discarded I wish it used a reasonable regex language, but as John Boehner said last week in a different context: 'If ands, if's and but's were candies and nuts, it'd be Christmas every day' and I have no intention of forking lftp ;) -- Russ herrold
John R Pierce wrote: > does someone have a script for maintaining a repo with lftp they'd like> to share? what I saw on the wiki wasn't very helpful.I am using mrepo, and for number of repos use lftp, but from mrepo's config. It downloads packages and then creates repositories for the web server, and even allows creating one larger repo from several smaller. It also takes care of ISO mounting so you do not need base repo synced. Ljubomir