freebsd stable - Jun 2013 - portupgrade(1) | portmaster(8) -- which is more effective for large upgrade?

Greetings,
 I haven't upgraded my tree(s) for awhile. My last attempt to rebuild after
an updating
src && ports, resulted in nearly installing the entire ports tree, which
is why I've
waited so long. Try as I might, I've had great difficulty finding something
that will
_only_ upgrade what I already have installed, _and_ respect the
"options" used during the
original make && make install, or those options expressed in make.conf.
As portupgrade(1) && portmaster(8) appear to be the most used in this
scenario,
I'm soliciting opinions on which of these works best, or if there is
something else to
better manage this situation. Is there such a thing as a FreeBSD upgrade
"easy button"?

Thank you for all your consideration.

--chris

Am 26.06.2013 18:42, schrieb Chris H:
> Greetings,
>  I haven't upgraded my tree(s) for awhile. My last attempt to rebuild
after an updating
> src && ports, resulted in nearly installing the entire ports tree,
which is why I've
> waited so long. Try as I might, I've had great difficulty finding
something that will
> _only_ upgrade what I already have installed, _and_ respect the
"options" used during the
> original make && make install, or those options expressed in
make.conf.
> As portupgrade(1) && portmaster(8) appear to be the most used in
this scenario,
> I'm soliciting opinions on which of these works best, or if there is
something else to
> better manage this situation. Is there such a thing as a FreeBSD upgrade
"easy button"?
> 
> Thank you for all your consideration.
Chris,

this time around, you will again rebuild almost your entire ports tree
because some basic ports, such as Perl.

Also, you will rarely be able to only upgrade what you already have
installed because sometimes ports grow new requisite other ports you do
not already have.

I haven't used portupgrade in a long time because there was a period
where it had fallen to bit-rot, but both tools are being maintained now.

portupgrade has the decided advantage of being able to continue building
some ports if another port failed as long as the failed port is not
itself a requisite port for one that is yet to be built; portmaster
bails out at the first error.

portmaster, on the other hand, has a "rebuild everything" approach in
the manual page, and can be used to list only leaf ports -- but that
approach will require you to deinstall all ports so that the machine
becomes unusable while it builds.

There are other approaches, like using portmaster just to list this
ports tree, and then use Tinderbox or poudriere to build packages in a
chroot, and then only deinstall and install if you have all packages
built successfully - but I am not familiar with automating this, not
familiar with poudriere, and it requires a bit of work to get your
options transferred to these build systems.  Such a "build all packages
first before you start deinstalling" would reduce the downtime, though.

Hope that helps a little.

Best regards
Matthias Andree

On Wed, Jun 26, 2013 at 09:42:43AM -0700, Chris H wrote:
> Greetings,
>  I haven't upgraded my tree(s) for awhile. My last attempt to rebuild
after an updating
> src && ports, resulted in nearly installing the entire ports tree,
which is why I've
> waited so long. Try as I might, I've had great difficulty finding
something that will
> _only_ upgrade what I already have installed, _and_ respect the
"options" used during the
> original make && make install, or those options expressed in
make.conf.
> As portupgrade(1) && portmaster(8) appear to be the most used in
this scenario,
> I'm soliciting opinions on which of these works best, or if there is
something else to
> better manage this situation. Is there such a thing as a FreeBSD upgrade
"easy button"?
Use portmaster, avoid portupgrade.  And no I will not expand on my
reasoning -- I urge anyone even mentioning the word portupgrade to spend
a few hours of their day reading the horror stories on the mailing lists
over the past 10 years or so (including recently).  Choose wisely.

And before going on any sort of "update crusade", I recommend you
re-examine your make.conf methodologies for options if you haven't
already.  The OPTIONS framework has been revamped and improved many
times over, so you will find things like this on a system whose admin
keeps up with the times (compare this to older ways/methods, which may
break or stop working):

OPTIONS_UNSET+=		X11 IPV6 NLS

php5_SET+=		APACHE
php5_UNSET+=		CGI
postfix_SET+=		PCRE TLS SASL2
samba36_SET+=		AIO_SUPPORT
samba36_UNSET+=		LDAP CUPS ACL_SUPPORT WINBIND POPT
wget_SET+=		OPENSSL
wget_UNSET+=		IDN

When rebuilding everything, I have always resorted to this:

rsync -avH /usr/local/ /usr/local.old/
pkg_delete -a -f
rm -fr /usr/local/*
rm -fr /var/db/ports/*
rm -fr /usr/ports/distfiles/*
cd /usr/ports/whatever
make install clean
{lather rinse repeat until done}

And add some pkg_add -r's in there for large-ish things I don't want to
rebuild from source (I think folks who use X probably do this quite a
bit; I remember hearing how Open/LibreOffice takes something like 3-4
hours to build on some systems).

But that's just how I do things.  My advice on using portmaster,
however, still stands.

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

> Am 26.06.2013 18:42, schrieb Chris H:
>> Greetings,
>>  I haven't upgraded my tree(s) for awhile. My last attempt to
rebuild after an updating
>> src && ports, resulted in nearly installing the entire ports
tree, which is why I've
>> waited so long. Try as I might, I've had great difficulty finding
something that will
>> _only_ upgrade what I already have installed, _and_ respect the
"options" used during the
>> original make && make install, or those options expressed in
make.conf.
>> As portupgrade(1) && portmaster(8) appear to be the most used
in this scenario,
>> I'm soliciting opinions on which of these works best, or if there
is something else to
>> better manage this situation. Is there such a thing as a FreeBSD
upgrade "easy button"?
>>
>> Thank you for all your consideration.
> Chris,
>
> this time around, you will again rebuild almost your entire ports tree
> because some basic ports, such as Perl.
>
> Also, you will rarely be able to only upgrade what you already have
> installed because sometimes ports grow new requisite other ports you do
> not already have.
>
> I haven't used portupgrade in a long time because there was a period
> where it had fallen to bit-rot, but both tools are being maintained now.
>
> portupgrade has the decided advantage of being able to continue building
> some ports if another port failed as long as the failed port is not
> itself a requisite port for one that is yet to be built; portmaster
> bails out at the first error.
>
> portmaster, on the other hand, has a "rebuild everything"
approach in
> the manual page, and can be used to list only leaf ports -- but that
> approach will require you to deinstall all ports so that the machine
> becomes unusable while it builds.
>
> There are other approaches, like using portmaster just to list this
> ports tree, and then use Tinderbox or poudriere to build packages in a
> chroot, and then only deinstall and install if you have all packages
> built successfully - but I am not familiar with automating this, not
> familiar with poudriere, and it requires a bit of work to get your
> options transferred to these build systems.  Such a "build all
packages
> first before you start deinstalling" would reduce the downtime,
though.
>
> Hope that helps a little.
>
> Best regards
> Matthias Andree
Greetings, and thank you for your reply.
I understand that portupgrade _will_ pull in other dependencies _as needed_ -- I
_do_ read
the man(1) pages. :)
But it installed (pulled in) far more than those dependencies actually required.
I believe, due to the fact that it doesn't appear to honor the original
build
options recorded in /var/db/ports/<portname>/options. Nor, do I recall
that it
honored /etc/make.conf -- make.conf(5). Maybe things have changed? I don't
see it.
Oh, and should it not have been clear; I _do_ anticipate the "upgrade"
to re-build
most everything, as that is why I'm trying to find a "mass
upgrader" port, to do the
"dirty work". Also should it not have been clear in the beginning; I
am _not_ doing
anything more than upgrading everything _within_ my current version; eg; no
major point
upgrade, or anything.

Thank you again, for taking the time to respond.

--chris

In article <5e20544e3580a75759c3858f31894dc9.authenticated at
ultimatedns.net>,
bsd-lists at lcommand.com writes:
> I haven't upgraded my tree(s) for awhile. My last attempt to rebuild
>after an updating
>src && ports, resulted in nearly installing the entire ports tree,
which
>is why I've
>waited so long. Try as I might, I've had great difficulty finding
>something that will
>_only_ upgrade what I already have installed, _and_ respect the
>"options" used during the
>original make && make install, or those options expressed in
make.conf.
Having just gone through this in two different environments, I can
very very strongly recommend doing the following.  It's not the "easy
button" of the TV commercials, but it will make things much much
easier in the future.

1) Switch your system to pkgng if you haven't already.  Unfortunately,
this will not result in the right ports being marked as "automatic",
so you'll need to do a bit of post-conversion surgery:

	# pkg set -A 1 -g '*'
	# pkg query -e '%#r==0' '%n-%v: %c'

Then look through the output of "pkg query" to identify the leaf
packages that are the ones you actually wanted explicitly to have
installed.  For each one of those:

	# pkg set -A 0 packagename

Create a list of your desired packages:

	# pkg query -e '%a==0' '%o' > pkg-list

Clean up the unnecessary local packages:

	# pkg autoremove

(You can iterate the last three steps, aborting "pkg autoremove" each
time but the last, until it doesn't offer to remove anything you care
about keeping.)

Repeat this process for each machine, and merge the resulting pkg-list
files using sort -u.  Make sure that pkgng is enabled for ports in
/etc/make.conf.

2) Install and set up poudriere.  Copy /var/db/ports, /etc/src.conf,
and /etc/make.conf to /usr/local/etc (possibly with local variations
as described in poudriere(8) under the heading "CUSTOMISATION").

3) Run "poudriere options" for each jail and setname (if you created
any sets following the customization section referenced above),
providing the package list you constructed, to make sure that any new
options are configured as you require them.

4) Run "poudriere bulk" for each jail and setname (if you created
any), providing the package list as before.  This will create a pkgng
repository for each jail and set, which you can serve by HTTP (using
your choice of Web server) or SSH (with pkgng 1.1+), and all of these
packages will have been built in a clean jail and (if their
dependencies were specified correctly) will have no library
inconsistencies.

5) Configure your client machines to reference the appropriate
repository created in step (4).

6) Run "pkg upgrade -fy" on all of your machines, and resolve any
inconsistencies by "pkg remove"-ing the offending local package.

That seems like a lot of work, and it is, but having done it, there's
a huge benefit the next time you want to do update your systems:

a) Update the ports tree (how you do this depends on how you set up
poudriere -- see the man page).

b) Repeat step (3).

c) Repeat step (4).

d) Check the ports UPDATING file for any warnings about packages you
are about to install.  If it tells you to do "pkg install -fR
somepackage", then do those.

e) Run "pkg upgrade -y" to upgrade any remaining packages.

Even for just three machines it was worth going through this process
-- and worth unifying all of my package sets and options.  Since I now
do one build instead of three, I'm no longer so ocncerned about
minimizing dependencies; it's no big deal if some X libraries get
installed on my server.

-GAWollman