freebsd stable - Nov 2005 - Centralized building

Hi all!

I've spent about a week trying to accomplish a rather simple task: To  
build kernel and world once for each architecture we have, and  
distribute this precompiled src and obj tree via NFS to all the  
systems that need updating. I have combined this with a locally  
maintained CVS tree, in order to assure coherent releases being  
installed on all our systems.

However, I am seeing some peculiar issues that I simply don't manage  
to get around.
Scenario:

I've got one server running 6.0-STABLE-i386. On this host I've  
created a jail for building. We have both i386 and amd64 platforms in- 
house, so I've created a script that build for both:
  make TARGET_ARCH=i386  MAKEOBJDIRPREFIX=/usr/obj.i386  buildworld
  make TARGET_ARCH=amd64 MAKEOBJDIRPREFIX=/usr/obj.amd64 buildworld
And the same for buildkernel.

Starting out trying to upgrade the amd64 hosts, I export the two obj  
directories via NFS, and mount them as /usr/obj on the amd64 hosts  
that need upgrading. This was, at least, my initial approach. I then  
found out that the /usr/src tree in the build jail is somehow tainted  
by the build (and by the options I specified), so I need to export  
that as well (which, I am afraid, means I have to maintain two  
different build jails). Therefore I also export /usr/src and mount it  
on the target hosts.

I then realized that I need to use the same objdir on the target  
hosts as in the build jail, so I try mounting to /usr/obj.<arch> on  
the target hosts. This allows me to get somewhat further.  
Installworld now progresses for a while, until it bombs out with the  
following error:
   ===> sys/boot/i386/boot2 (install)
   dd if=/dev/zero of=boot2.ldr bs=276 count=1
   dd: not found
   *** Error code 127

When looking for dd, I find it in the host PATH, and also in the obj  
dir:
   [root@build] /usr/obj.amd64# find . -name dd -type f
   ./amd64/usr/src/bin/dd/dd

At this point, I get rid of the MAKEOBJDIRPREFIX option and rebuild  
everything with just TARGET_ARCH, only exporting /usr/obj from the  
build jail. I notice that when using TARGET_ARCH with something else  
than the architecture the build is running on (i.e. amd64 on an i386  
host), the resulting build is NOT to be found in /usr/obj, but in / 
usr/obj/amd64. Thus I need to specify MAKEOBJDIRPREFIX=/usr/obj/amd64  
on the target host for installworld to get anything done at all.

I'm still getting the dd: not found error, and I do believe I've  
tried every combination and variation I can think of. Clocks are in  
sync between all the systems, so that is not the problem. Is the  
build system partially broken in 6.0? Have I missed something? Do I  
actually need an amd64 host to be able to build for amd64 systems, or  
are there other ways to accomplish what I'm trying to do? Should I  
prehaps try doing centralized binary upgrades instead?

Any help would be appreciated.
With best regards,
/Eirik

> Starting out trying to upgrade the amd64 hosts, I export the
> two obj directories via NFS, and mount them as /usr/obj on the
> amd64 hosts that need upgrading.
I done upgrades the other way, by having the build machine
mount the clients to-be-root partition and installing to it
using NFS.

--
FreeBSD Volunteer,     http://people.freebsd.org/~jkoshy

On Nov 19, 2005, at 13:28 , Joseph Koshy wrote:
>> Starting out trying to upgrade the amd64 hosts, I export the
>> two obj directories via NFS, and mount them as /usr/obj on the
>> amd64 hosts that need upgrading.
>
> I done upgrades the other way, by having the build machine
> mount the clients to-be-root partition and installing to it
> using NFS.
Would I have to export every filesystem mount point on the hosts  
then? Or does an -alldirs option do the trick (in exports)?

In any case this would not be compatible with our security policy,  
unfortunately.

/Eirik

Eirik ?verby wrote:
> I've spent about a week trying to accomplish a rather simple task: To 
> build kernel and world once for each architecture we have, and 
> distribute this precompiled src and obj tree via NFS to all the  systems
> that need updating. I have combined this with a locally  maintained CVS
> tree, in order to assure coherent releases being  installed on all our
> systems.
AFAICT cross-compiling amd64 on a i386 machine isn't supported yet. I ran
into a similar problem when I upgraded an i386 machine to amd64. I thought I
could just set CPUTYPE=athlon-64 and buildworld would do the right thing.
Apparently not.

I haven't bothered to try building i386 on an amd64 box though. Maybe
you'll have better luck with that.