CentOS - Apr 2007 - Jigdoes of CentOS 4.4 and 5.0 i386/x86_64 CD/DVD available.

I've gone through the trouble (after some 10 rounds I finally have 
something I'm happy with) and generated jigdo/template files for 
CentOS 5.0 iso's (CD/DVD i386/x86_64) (and CentOS 4.4 while I was at it: 
CD/DVD/ServerCD SRPMS/i386/x86_64).

(and I started on this before the thread about creating the DVD's from the 
CD's started ;-) )

They are available at:
http://tcs.uj.edu.pl/~buildcentos/centos-jigdo/

(I skipped the LiveCD since jigdo offers no size improvement here -- 
already for the ServerCD the template file is 125 MBs because of the 
different .img files inside the iso)

These jigdo/template files should allow you to generate any of the listed 
cd/dvd isos from the publically available http://mirror.centos.org mirror 
site (which is DNS-aliased to something close to you).  Jigdo will also 
allow using a local source of files (ie. an existing CD, etc...)

You will need to install jigdo-file and jigdo-lite.

It is available:
- in package jigdo from dag's repository (rpmforge) for CentOS 4 (and
others)
- in package jigdo-file for debian/ubuntu (just apt-get it)
- from the source website (both sources, and windows binaries, and 
statically linked linux binaries) http://atterer.net/jigdo/

Once installed usage is (...should be...) trivial:

jigdo-lite --noask http://tcs.uj.edu.pl/~buildcentos/centos-jigdo/%.jigdo

which will generate the requested (replace the % with what you want, see 
the http://tcs.uj.edu.pl/~buildcentos/centos-jigdo/ listing) iso file in 
the directory in which you run it.

[If you have a local source of files you can skip the --noask and provide 
it when asked]

Next step: get this included on the centos mirrors ;-) [hint hint]

Afterwards: it would be nice to drop the iso files from the mirrors in 
their entirety and generate them on-the-fly using the JTE extensions 
package.  Should be significant space-savings.  [requires perl cgi in the 
hosting webserver, but should be very safe since it's all read-only 
access]

Even later: it would be awesome if we could generate the jigdo/template 
during mkisofs (requires using a patched mkisofs).  This would skip the 
slightly painful jigdo generation step (painful - because I have to 
download a local mirror of both the files and the CD/DVD images, much less 
painful to do it directly on the centos source/build system/mirrors)

Both later steps are 95% there (the JTE jigit website/extensions/mkisofs 
patches/etc) and mostly require putting together rpms for EL4/5.
(I'm going to be trying to put together a mkisofs-jte rpm which doesn't 
override the normal mkisofs rpm, but instead provides two files - 
/usr/bin/mkisofs-jte and it's manpage)

This should also allow more customized images (like the rolled up updates) 
without increasing mirror space usage.

Jigdo (combined with hardlinking identical files, which I believe is 
already done) should enable drastic reductions in centos HDD footprint.

Comments?

Cheers,
Maciej Zenczykowski

Maciej Zenczykowski wrote:
> I've gone through the trouble (after some 10 rounds I finally have 
> something I'm happy with) and generated jigdo/template files for CentOS
> 5.0 iso's (CD/DVD i386/x86_64) (and CentOS 4.4 while I was at it: 
> CD/DVD/ServerCD SRPMS/i386/x86_64).
> 
thanks for taking the time to get this done. however, one thing I'd like 
to point out is that there is now a -qa testing team in place, and 
issues such as this should really be run past them ( posting in 
centos-devel usually get their attention ), before posting it on the 
centos list or the public forums.

If its ok with you, and if you are happy to work with the CentOS 
developers and qa people to make this jidgo process more 'official' - 
could we move this to the centos-devel list and let the -qa people have 
a go at it for a bit ?


- KB
-- 
Karanbir Singh : http://www.karan.org/ : 2522219 at icq

Maciej Zenczykowski spake the following on 4/15/2007 9:46
PM:
> I've gone through the trouble (after some 10 rounds I finally have
> something I'm happy with) and generated jigdo/template files for CentOS
> 5.0 iso's (CD/DVD i386/x86_64) (and CentOS 4.4 while I was at it:
> CD/DVD/ServerCD SRPMS/i386/x86_64).
> 
> (and I started on this before the thread about creating the DVD's from
> the CD's started ;-) )
> 
> They are available at:
> http://tcs.uj.edu.pl/~buildcentos/centos-jigdo/
> 
> (I skipped the LiveCD since jigdo offers no size improvement here --
> already for the ServerCD the template file is 125 MBs because of the
> different .img files inside the iso)
> 
> These jigdo/template files should allow you to generate any of the
> listed cd/dvd isos from the publically available
> http://mirror.centos.org mirror site (which is DNS-aliased to something
> close to you).  Jigdo will also allow using a local source of files (ie.
> an existing CD, etc...)
> 
> You will need to install jigdo-file and jigdo-lite.
> 
> It is available:
> - in package jigdo from dag's repository (rpmforge) for CentOS 4 (and
> others)
> - in package jigdo-file for debian/ubuntu (just apt-get it)
> - from the source website (both sources, and windows binaries, and
> statically linked linux binaries) http://atterer.net/jigdo/
> 
> Once installed usage is (...should be...) trivial:
> 
> jigdo-lite --noask http://tcs.uj.edu.pl/~buildcentos/centos-jigdo/%.jigdo
> 
> which will generate the requested (replace the % with what you want, see
> the http://tcs.uj.edu.pl/~buildcentos/centos-jigdo/ listing) iso file in
> the directory in which you run it.
> 
> [If you have a local source of files you can skip the --noask and
> provide it when asked]
> 
> Next step: get this included on the centos mirrors ;-) [hint hint]
> 
> Afterwards: it would be nice to drop the iso files from the mirrors in
> their entirety and generate them on-the-fly using the JTE extensions
> package.  Should be significant space-savings.  [requires perl cgi in
> the hosting webserver, but should be very safe since it's all read-only
> access]
> 
> Even later: it would be awesome if we could generate the jigdo/template
> during mkisofs (requires using a patched mkisofs).  This would skip the
> slightly painful jigdo generation step (painful - because I have to
> download a local mirror of both the files and the CD/DVD images, much
> less painful to do it directly on the centos source/build system/mirrors)
> 
> Both later steps are 95% there (the JTE jigit website/extensions/mkisofs
> patches/etc) and mostly require putting together rpms for EL4/5.
> (I'm going to be trying to put together a mkisofs-jte rpm which
doesn't
> override the normal mkisofs rpm, but instead provides two files -
> /usr/bin/mkisofs-jte and it's manpage)
> 
> This should also allow more customized images (like the rolled up
> updates) without increasing mirror space usage.
> 
> Jigdo (combined with hardlinking identical files, which I believe is
> already done) should enable drastic reductions in centos HDD footprint.
> 
> Comments?
> 
Bitorrent has one advantage over jigdo AFAIR -- Jigdo uses the bandwidth of
the mirrors to get all downloads, but bittorrent uses the bandwidth of the
downloaders. This adds up to significant savings to the one hosting the files,
because storage is an occasional purchase, where bandwidth is a monthly expense.


-- 

MailScanner is like deodorant...
You hope everybody uses it, and
you notice quickly if they don't!!!!

On Tue, 17 Apr 2007, Maciej ?enczykowski wrote:
>> But --- does jigdo have any kind of indirection in place , eg to get a
>> list of mirrors and use them ??
>
> It does use a list of mirros for the debian.  I haven't really looked
> into it too deeply.
> Furthermore the jigdo-lite program is really just a shell script -
> should be easy to get it to behave however we want it (it just runs
> wget).
>
>> Or could yum be used as a wrapper around jigdo maybe to pull stuff down
>> using fastestmirror or maybe even dags stuff ??
>
> I would really like to see dynamic DNS resolution based on source IP
> giving out the address of the nearest centos compatible mirror - then
> it would 'just work'.
see below re cnames ...

we really need to push the bandwidth out to external mirrors - and 302's 
are messy ...
>> Otherwise it gets messy because mirrors have different file structures,
so
>> just using CNAME's wont work.
>
> And that's the problem... But we could possibly work with them to get
> this working - at least for http this is relatively trivial (just
> setup a virtual domain for whatever global name we would decide on
> like mirror.centos.org).
Except that is not what we do for mirror.centos.org - all those are our 
servers.
>
>> The trouble I see is that if a user doesnt have a local mirror or
>> local copy of a set if isos then downloading the packages individually
>> from a mirror may be less efficient than downloading an iso / set of
isos,
>> and certainly without mirror redirect and fastestmirror it is a non
starter
>> ...
>
> There aren't that many files on a cd, so the extra time spent opening
> closing connections shouldn't really be significant - unless you have
> an extremely big pipe.  But - yes - we would want each user to use the
> closest possible mirror.
>
>> The fedora guys have beern working on stuff that we also ought to look
at
>> , and we should look at metalinks while we are at it.
>
> No idea what this is... :-)
https://hosted.fedoraproject.org/projects/revisor

I dont really think it is the same thing ...

and http://www.metalinker.org/

eg http://www.metalinker.org/samples/CentOS-4.4-i386-binDVD.iso.metalink

Regards
Lance

--
uklinux.net -
The ISP of choice for the discerning Linux user.