CentOS - May 2012 - Repositories in CentOS 5.8

Hi,

I am running Cent OS 5.8 in production. Can someone please explain me about
various repositories available in CentOS 5.8 and which third party repos (
http://wiki.centos.org/AdditionalResources/Repositories) i should use it in
Production environment.

Help me understand the pros and cons.

Regards

Kaushal

On Tue, May 22, 2012 at 11:17 AM, Kaushal Shriyan
<kaushalshriyan at gmail.com> wrote:
>
> I am running Cent OS 5.8 in production. Can someone please explain me about
> various repositories available in CentOS 5.8 and which third party repos (
> http://wiki.centos.org/AdditionalResources/Repositories) i should use it in
> Production environment.
>
> Help me understand the pros and cons.
Usually you decide what packages that aren't included in the base
distribution that you need, and then pick the repository that has them
instead of the other way around.   However EPEL is generally the first
place to look and since it has a policy of not overwriting base
packages there is not much risk in using it.

The pros are that you get access to many more applications and
libraries without having to compile and update the software yourself.
 The cons are that in certain cases the repositories have modified or
newer versions of the same packages as the base distribution which can
cause conflicts in future updates.   There are usually ways to work
around the conflicts, but it is best to avoid them unless you have a
specific need for particular modified packages.

-- 
   Les Mikesell
     lesmikesell at gmail.com

On 22.5.2012 18:17, Kaushal Shriyan wrote:
> Hi,
> 
> I am running Cent OS 5.8 in production. Can someone please explain me about
> various repositories available in CentOS 5.8 and which third party repos (
> http://wiki.centos.org/AdditionalResources/Repositories) i should use it in
> Production environment.
> 
> Help me understand the pros and cons.
There are a lot of 3rd party repositories around, and my understanding
is that the only sane way is not to trust a whole repository but only
selected and therefore tested packages. Consequently though you will
have to maintain your own repository.
-- 
Kind Regards, Markus Falb

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 307 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20120522/9b817c8c/attachment.sig>

On Tue, May 22, 2012 at 12:15 PM, Markus Falb <markus.falb at fasel.at>
wrote:
>>
>> I am running Cent OS 5.8 in production. Can someone please explain me
about
>> various repositories available in CentOS 5.8 and which third party
repos (
>> http://wiki.centos.org/AdditionalResources/Repositories) i should use
it in
>> Production environment.
>>
>> Help me understand the pros and cons.
>
> There are a lot of 3rd party repositories around, and my understanding
> is that the only sane way is not to trust a whole repository but only
> selected and therefore tested packages. Consequently though you will
> have to maintain your own repository.
But with EPEL and others with policies to not overwrite base packages,
you won't get anything that you didn't explicitly install (assuming
you trust them to follow their policy...).    A possible exception is
that they consider RHEL as upstream so they might have a rare conflict
with something from centos extras or testing.

Also, you can make things a bit safer by setting 'enabled=0' in the
yum repo config file and then when you want to install or update a
package from there:
yum --enablerepo=repo_name install package_name

-- 
   Les Mikesell
     lesmikesell at gmail.com

On Tuesday, May 22, 2012 12:17:07 PM Kaushal Shriyan
wrote:
> I am running Cent OS 5.8 in production. Can someone please explain me about
> various repositories available in CentOS 5.8 and which third party repos (
> http://wiki.centos.org/AdditionalResources/Repositories) i should use it in
> Production environment.
You've already gotten some excellent advice about this, but I'd like to
add a few things, starting with a list of just a few factoids I've observed
about repos and repo mixing:

1.) EPEL doesn't make it a priority (or even a goal) to work well with any
other third-party repository;

2.) The mixability of other repos varies, with both repoforge/RPMforge and
ELrepo taking pains to not overwrite base repo packages unless you enable their
'extras' sub-repos (others may as well, I'm speaking from my own
experience, and the repos I use the most are EPEL, repoforge, and ELrepo, and so
I'm not even going to comment about the mixability of others, like remi,
IUS, ATrpms, or others since I have either insufficient or old information about
them);

3.) At least one useful repository that I use on certain production machines,
the CERT Forensics repo, relies on both EPEL and RPMforge/repoforge, so look
carefully at the mixing issues of each of your selected repos' upstream repo
dependencies;

4.) Random mixing of packages (like random downloads from pkgs.org) is certain
to cause problems;

5.) The fewer the number of repos you use the more stable your package set will
be.  That is especially true if you mix 'specialty' repos like OpenNMS
and PacketFence (or even slightly off the wall ones like LinuxTech (which has a
usable handbrake for CentOS 6, for instance)), or upstream repos like the one
from the PostgreSQL RPM building project to get the latest PostgreSQL on you
system;

6.) There is no such thing in repositories as 'one size fits all.'  What
will fit your needs depends a great deal on what 'production' is defined
to be in your specific instance.

(For a server in 'production' that is serving typical network service
loads (file, print, web, e-mail, databases, etc) you're going to need some
specific things (where 'things' is defined as the set of packages and
interdependencies between packages).  A 'production'
research/development desktop (we have a few here) will need different things; a
'production' embedded machine controller (we have a couple of those,
too) will need yet another different set of things.);

7.) You really need to look at the packages that you need for your application
and then individually investigate which repo or repos has the packages that you
need, built the way you need them.  And look at the longevity of the repo; both
RPMforge/repoforge and EPEL, for instance, have been around a while and are
pretty well maintained;

8.) The recommendations on the CentOS Wiki repositories page are very good
starting points, but what you specifically need in production is something
you'll need to determine for yourself after doing some testing with
different repos.  And I'd keep some testing machines or VM's available
to test various repos over time to see how they work or don't work with each
other, and you might even want to build your own repository, depending upon your
specific critieria;

9.) Don't mix from-source (./configure;make;make install) installed packages
and packages from repositories unless:
	a.) You know exactly what you're doing;
	b.) The from-source package builds all its own dependencies (like Plone does);
	c.) The from-source package's author won't support it otherwise.

10.) Learn to use yum and its tools effectively to keep mixing issues at bay
(priorities, plugins, and the command line parameters to enable and disable
individual repositories as needed are the ones to start with);

Now, a non-factoid observation: if you think about it, it's quite an amazing
thing that so many people are so willing to keep repositories of packages up to
date at no cost to the end-user, given the very definite benefit and value of
those updates (which is why I can't really complain if a repo is a little
out of date, or if two repos that aren't costing me any opex won't mix
just the way I want them to) and the very real cost to the maintainer, in terms
of time, stress, frustration, and money.

Having kept packages up to date for public consumption before, I understand all
too well the trials of a packager and the entitlement syndrome some users seem
to have.

And thus my last recommendation: 

11.) be prepared to do some work on your own to make different repositories work
together for you, and be patient with the maintainers of those repositories when
they don't work together the way you might like.  They don't have to
listen to you, but most will listen if you approach them the right way,
respectfully acknowledging their valuable contribution to your bottom line.

YMMV, FWIW, IMHO, HTH, etc.

On Wednesday, May 23, 2012 01:11:54 PM Les Mikesell
wrote:
> On Wed, May 23, 2012 at 10:26 AM, Lamar Owen <lowen at pari.edu>
wrote:
> > 1.) EPEL doesn't make it a priority (or even a goal) to work well
with any other third-party repository;
> True enough as a fact, but it sort-of sounds like a criticism for
> something that's not really practical. 
I'll just point to the EPEL section of the wiki.centos.org Repositories
page, and let that say almost all that really needs to be said, other than I
will say that I chose my wording for that sentence rather carefully, since I
didn't want it to sound like a complaint, and only hit send after making
multiple revisions.

On Wednesday, May 23, 2012 09:22:34 PM Kaushal Shriyan
wrote:
> Thanks for the explanation. Please help me understand when i install CentOS
> 5.8 on a fresh server, what are the available repositories by default 
yum repolist enabled

That command works on both CentOS 6.2 and CentOS 5.8.  Use that command after
install and you have your answer; use that command on any server you have and
you can see what repos were used to get the software set installed (the
yum-utils package includes other tools that can help you visualize the
dependency tree).
> and
> if i need any packages which are not there in the default repos, Do i need
> to enable third party repositories.
I will put forth the goal of enabling the fewest possible repos.

So you'd want to carefully think about which repo or repos to use if your
desired package is found in more than one, as well as the differences between
the way these packages are built.  I have two examples.

First, if you have a machine with an Intel GMA integrated video devices (say a
915, 945, or 965 chipset or even newer) you may need the ELrepo
xorg-x11-drv-intel instead of the provided one in CentOS.

Second, if you need the xrdp package (RDP remote desktop connections to a Linux
desktop), EPEL and repoforge have two different versions:

# repoquery --repoid=rpmforge xrdp
xrdp-0:0.4.0-1.el6.rf.i686
# repoquery --repoid=epel xrdp
xrdp-0:0.5.0-0.13.el6.i686

The 0.4.0 xrdp is some different from the 0.5.0 version, and those differences
by be significant for a particular use.  The repoquery tool is found in the
yum-utils package.

Now, having RPMforge and EPEL enabled on the same machine can be an adventure. 
As an example, suppose I have a server (an upstream EL6 server in this case)
which is serving remote desktop connections, being used for digital forensics
with the scalpel file carver, and I'm wanting to make it my small
consultancy's e-mail server and use the crm114 system to help with
anti-spam.

A yum install of the latest xrdp will pull from EPEL.  A yum install of the
latest scalpel will pull from EPEL.  So far so good.  Now:
[root at www ~]# yum install crm114
Loaded plugins: product-id, refresh-packagekit, rhnplugin, subscription-manager
Updating certificate-based repositories.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package crm114.i686 0:20100106-3.el6.rf will be installed
--> Processing Dependency: libtre.so.5 for package:
crm114-20100106-3.el6.rf.i686
--> Running transaction check
---> Package tre.i686 0:0.7.6-2.el6 will be updated
--> Processing Dependency: libtre.so.4 for package: scalpel-2.0-1.el6.i686
---> Package tre.i686 0:0.8.0-1.el6.rf will be an update
--> Finished Dependency Resolution
Error: Package: scalpel-2.0-1.el6.i686 (@epel)
           Requires: libtre.so.4
           Removing: tre-0.7.6-2.el6.i686 (@epel)
               libtre.so.4
           Updated By: tre-0.8.0-1.el6.rf.i686 (rpmforge)
               Not found
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
[root at www ~]#

Hmmm, looks like I need some more hardware or a VM to run the e-mail (as this is
a multiprocessor pre-em64t Xeon with loads of RAM and large disk, i686 is a
reasonable option, but KVM isn't by default available and no em64t Xeons are
available for the chipset and sockets on this particular SuperMicro
motherboard....and it's not fully depreciated yet, either, and newer
hardware is not in this year's budget, but I am open to donations....:-) ). 
Or I need to rebuild scalpel myself to be compatible with the RPMforge tre
package.  Or I can wait on EPEL to catch up on tre. Or I can use the RPMforge
scalpel (a lower version scalpel, but a higher version tre in this case):

[root at www ~]# repoquery --repoid=epel scalpel tre
scalpel-0:2.0-1.el6.i686
tre-0:0.7.6-2.el6.i686
[root at www ~]# repoquery --repoid=rpmforge scalpel tre
scalpel-0:1.60-1.el6.rf.i686
tre-0:0.8.0-1.el6.rf.i686
[root at www ~]#

Oh, but:
[root at www ~]# repoquery --repoid=forensics scalpel tre
scalpel-0:2.0-1.el6.i386
[root at www ~]#

(That's the CERT Forensics repo; I haven't checked to see which tre its
scalpel is built against.)

This is not a contrived example; this is my own box, and I wanted to do this
very thing not too long ago.  I haven't solved that problem yet, and
haven't installed crm114 either, due to time constraints.  It would depend
entirely on whether I really have to have the features in scalpel 2.0, or if
scalpel 1.60 is good enough (the scalpel website lists: "As for v2.0,
Scalpel supports regular expressions for headers and footers, minimum carve
sizes, multithreading and asynchronous I/O, and beta-level support for
GPU-accelerated file carving. ")  Hmmm, GPU accelerated file carving sounds
interesting, but my system doesn't have a GPU capable of helping much. 
Multithreading and async I/O, plus regexp for headers and footers.... while I
haven't needed those features yet, I might.

I think I've illustrated the process at this point.  I have a tradeoff
between not having crm114 at all and not having certain features that I may or
may not use in a program that I use once in a blue moon (but blue moons happen,
and I have used scalpel a few times in the past).... I'll probably end up
doing a yum remove of scalpel, excluding the EPEL and the CERT Forensics
versions of scalpel in the respective repo files in /etc/yum.repos.d, and then
reinstalling scalpel, which should pull the RPMforge one, and make the crm114
install possible.

On Thursday, May 24, 2012 12:42:59 PM Les Mikesell
wrote:
> But odds are pretty good that you could grab the scalpel src rpm from
> epel and fix it to rebuild against the newer libtre in a matter of
> minutes. - just changing the spec, not the source...  
Probably so, and I know how to do that, but I wasn't illustrating a specific
workaround, just illustrating the problem.  Technically, once you rebuild a
package yourself you have become yet another third party repository, even if it
isn't yummable, and you'll still have to pin the locally built and
installed version to prevent the other repo from updating that package.  I have
several locally rebuilt packages on that same system for other purposes, but
that wasn't what I was illustrating.
> Stuff like that
> does happen, but it's rare (what, one conflict out of thousands of
> packages?) 
It's only rare if you don't hit it.  

After writing that sentence, I started wondering just how rare it might be, and
could I throw a few lines of shell together to find out how rare it might or
might not be.  So I dug a little deeper.  And it makes a really good exercise
while taking a lunch break, so I decided to try to find out.

Since yum-utils does contain the interesting repodiff utility, I ran it, with
interesting and voluminous results.  I took the output of repodiff ran against
the baseurls of EPEL and RPMforge (using the -a to specify arches instead of
taking the default of running against source packages), pulled out the list of
updated packages, removed the version, release, and tags with a combination of
cut, sed, and a few manual edits, and ran repoquery against both EPEL and
RPMforge for the list (948 packages where the name is the same but the EVR was
different), cut the resulting lists at the colon epoch separator, dropped the
release, arch, dist, and repo tags from the versions, and compared the two lists
of version numbers alone.

Now, this is a snapshot in time, that is, the time I ran the repodiff and the
repoquery commands, and things can change, and mirrors do change over time,
especially with large repositories like these two.  And I didn't run this on
a CentOS 5.8 box, but an upstream EL6 box, so time for a subject line change.  I
have CentOS 6 boxes here, but none of them have both EPEL and RPMforge enabled,
but my upstream EL6 box does.

I found, first of all, two packages that RPMforge has in both a i386/i686 and a
.noarch form: facter and perl-AnyEvent; in both cases EPEL had but one package. 
So I dropped the oldest version of those two in RPMforge so that I could do a
straight version-to-version comparison.

A simple side-by-side diff of just the version numbers produced 418 differences
(424 lines of diff, twelve of which were marked with < and > instead of |,
so I subtracted half of  those).  Manually reading through the diff, I found one
difference where the version in one repo was 1.66.001 and in the other it was
1.66001, dropping a period, so that makes 417 packages where the name is the
same, but the version is different.  Some differences were small, some not so
small.

Some differences are such that the epoch was different; I didn't do a count
on those, since that's not an upstream version difference, but a packaging
version difference.  I found that when I was looking for why the EPEL list had
two fewer lines than the RPMforge list.

The bottom line: out of the about 6,000 packages in EPEL, there are 7% or so
that have the same name but a different version in RPMforge; out of the about
4,400 (4,381 listed by yum repolist) package in RPMforge, there are 9.5% or so
that have the same name but a different version in EPEL.  If anything you are
running relies on any of those 417 packages, you have a potential for problems.

So, it's not rare.  

And while I could share those lists, they will be out of date by tonight, and
most certainly by when you might need or want them.  Since there was a manual
edit portion involved due to repodiff not outputting the epoch and its perfectly
placed colon separator, it isn't very scriptable at this level.

I'm sure repodiff could be modified to use the actual version numbers and
names stored in the yum and rpm databases and ignoring the epoch, release, arch,
dist, and repo tags.  Of course, EPEL doesn't have a repo tag, so you have
to remember that, too.  Then you'd have another tool to help you deal with
repo collisions.  Perhaps such a tool has been written; I don't know about
it.  The repodiff tool as it stands is made for other purposes than this, and
has to be shoehorned in to do this sort of comparison.

Again, YMMV, FWIW, HTH, and all that.

And I of course as always reserve the right to be wrong.

On Thursday, May 24, 2012 03:26:02 PM Les Mikesell
wrote:
> But many, probably most of those cases are revs with forward/backward
> compatibility.  It's hard to generalize about that, though.  
Yep, it sure is.  Forward/backward compatibility is almost entirely in the hands
of the upstream projects (upstream from Red Hat).  Some do that kindof thing
well; others do not.  Open source not only gives choice to the user, but also
transfers the responsibility for choosing wisely to the user making the choice.
> Even in
> the scalpel case you mentioned the up-rev lib was likely compatible
> but just specified as requiring an exact version in the spec file.
The one that has bitten me the hardest is xrdp.  It will bite harder once a
release based on FreeRDP rather than rdesktop is released out of git.  The
compatibility depends entirely on the package's upstream policies,
developers, and stage of development (xrdp isn't out of beta, after all).
> And on the other side there are things like viewvc that are at the
> same rev in epel and rpmforge but have slightly different and
> incompatible configurations (and there is a reason I know that...).
Likewise amavisd-new.  It's not rare to have issues, and lots of people have
had them, thus my cautions.  Not trying to scare anyone off from any third party
repository; just want to give information that allows people to make informed
decisions, and show people how I at least arrived at my conclusions that work
for me with my particular setup; what my particular setup is isn't the
important thing, it's how and why I got there that I consider potentially
useful to others.

At the moment both EPEL and RPMforge are on a 2.6.x amavisd-new; 2.7 makes some
changes in the AM.PDP protocol that can break, for instance, amavisd-milter
(distinct from the much less useful amavis-milter).  Neither repo has
amavisd-milter, so that compatibility issue may not show up except to those who
actually use amavisd-milter instead of the much less useful amavis-milter.

I'll step out on a limb here and generalize somewhat; I would think that
most CentOS users use at least one third-party repository, if the traffic on
this list is any indication (and, again, I reserve the right to be wrong).  So
knowing how to properly determine how to use those repos (which was the OP's
question, after all) is very useful indeed, IMO.

And, again, FWIW, HTH, IMHO, YMMV, etc.

On Saturday, May 26, 2012 05:15:41 AM David Hrb?? wrote:
> Dne 25.5.2012 02:00, Lamar Owen napsal(a):
> > At the moment both EPEL and RPMforge are on a 2.6.x amavisd-new; 2.7
makes some changes in the AM.PDP protocol that can break, for instance,
amavisd-milter (distinct from the much less useful amavis-milter).  Neither repo
has amavisd-milter, so that compatibility issue may not show up except to those
who actually use amavisd-milter instead of the much less useful amavis-milter.
> 
> Lamar,
> Repoforge/RPMforge does provide amavisd-new-milter package...
> DH
David,

I understand that you are one of the RPMforge/repoforge packagers, and I thank
you for your efforts.

The amavisd-new-milter package does exist for CentOS 5.8; I cannot, however,
find an amavisd-new-milter package for CentOS 6 in either rpmforge or
rpmforge-extras.

Which is just as well, since this amavisd-new-milter is different from
amavisd-milter, which is currently at version 1.5.0, the version that is
compatible with amavisd-new 2.7.0 and up.  It's somewhat unfortunate to have
two very different things packaged with very similar names; the amavis-milter
that comes with amavisd-new is much less useful than the separate amavisd-milter
( http://amavisd-milter.sourceforge.net/ ; the one packaged with amavisd-new is
the one with a README at http://www.ijs.si/software/amavisd/README.milter.txt
that points to the Petr Rehor rewrite at amavisd-milter.sourceforge.net).

To my knowledge no repos have the amavisd-milter package available; I've
built my own for six years or so.  I've used both, and the
amavisd-new-milter (/usr/sbin/amavis-milter) is not nearly as useful as this
amavisd-milter.  In fact, for at least the last three years I've not been
able to get the amavis-milter that comes with amavisd-new to work at all,
whereas amavisd-milter (the Petr Rehor version at sourceforge) works very well
at version 1.5.0.

On Saturday, May 26, 2012 12:47:04 PM Les Mikesell
wrote:
> Have you looked at MimeDefang's ability to run all of your scanners
> out of one milter?  
Yes.  

Doing the same thing with amavisd-new on the few sendmail installs I still have
running; amavisd-new runs clam (or, at one site, the sophos scanner) and
spamassassin, and amavisd-milter does everything needed with one milter.  Using
essentially the same setup with a couple of postfix sites, but no milter in that
case.

On Monday, May 28, 2012 02:22:32 AM David Hrb?? wrote:
> Dne 26.5.2012 18:33, Lamar Owen napsal(a):
> > Which is just as well, since this amavisd-new-milter is different from
amavisd-milter, which is currently at version 1.5.0, the version that is
compatible with amavisd-new 2.7.0 and up.  It's somewhat unfortunate to have
two very different things packaged with very similar names; the amavis-milter
that comes with amavisd-new is much less useful than the separate amavisd-milter
( http://amavisd-milter.sourceforge.net/ ; the one packaged with amavisd-new is
the one with a README at http://www.ijs.si/software/amavisd/README.milter.txt
that points to the Petr Rehor rewrite at amavisd-milter.sourceforge.net).
> I did not know about this. Are you willing to share your amavis-milter
> spec file so we can include it in Repoforge?
It's pretty basic, really.  I'll dig it up; it's been a long time
since I've done anything with it.