freebsd stable - Apr 2010 - /libexec/ld-elf.so.1: Cannot execute objects on /

I have large storage partition (/vol0) mounted as noexec and nosuid. 
Then one directory from this partition is mounted by nullfs as "exec and 
suid" so anything on it can be executed.

The directory contains full installation of jail. Jail is running fine, 
but some ports (PHP for example) cannot be compiled inside the jail with 
message:

/libexec/ld-elf.so.1: Cannot execute objects on /

The same apply to executing of apxs

root@rainnew ~/# /usr/local/sbin/apxs -q MPM_NAME
/libexec/ld-elf.so.1: Cannot execute objects on /

apxs:Error: Sorry, no shared object support for Apache.
apxs:Error: available under your platform. Make sure.
apxs:Error: the Apache module mod_so is compiled into.
apxs:Error: your server binary '/usr/local/sbin/httpd'..

(it should return "prefork")

So I think there is some bug in checking the mountpoint options, where 
the check is made on "parent" of the nullfs instead of the nullfs
target
mountpoint.

It is on 6.4-RELEASE i386 GENERIC. I did not test it on another release.

This is list of related mount points:

/dev/mirror/gm0s2d on /vol0 (ufs, local, noexec, nosuid, soft-updates)
/vol0/jail/.nullfs/rain on /vol0/jail/rain_new (nullfs, local)
/usr/ports on /vol0/jail/rain_new/usr/ports (nullfs, local)
devfs on /vol0/jail/rain_new/dev (devfs, local)

If I changed /vol0 options to (ufs, local, soft-updates) the above error 
is gone and apxs / compilation works fine.

Can somebody look at this problem?

Miroslav Lachman

2010/4/20 Miroslav Lachman <000.fbsd@quip.cz>:
> I have large storage partition (/vol0) mounted as noexec and nosuid. Then
> one directory from this partition is mounted by nullfs as "exec and
suid" so
> anything on it can be executed.
>
> The directory contains full installation of jail. Jail is running fine, but
> some ports (PHP for example) cannot be compiled inside the jail with
> message:
>
> /libexec/ld-elf.so.1: Cannot execute objects on /
>
> The same apply to executing of apxs
>
> root@rainnew ~/# /usr/local/sbin/apxs -q MPM_NAME
> /libexec/ld-elf.so.1: Cannot execute objects on /
>
> apxs:Error: Sorry, no shared object support for Apache.
> apxs:Error: available under your platform. Make sure.
> apxs:Error: the Apache module mod_so is compiled into.
> apxs:Error: your server binary '/usr/local/sbin/httpd'..
>
> (it should return "prefork")
>
> So I think there is some bug in checking the mountpoint options, where the
> check is made on "parent" of the nullfs instead of the nullfs
target
> mountpoint.
>
> It is on 6.4-RELEASE i386 GENERIC. I did not test it on another release.
>
> This is list of related mount points:
>
> /dev/mirror/gm0s2d on /vol0 (ufs, local, noexec, nosuid, soft-updates)
> /vol0/jail/.nullfs/rain on /vol0/jail/rain_new (nullfs, local)
> /usr/ports on /vol0/jail/rain_new/usr/ports (nullfs, local)
> devfs on /vol0/jail/rain_new/dev (devfs, local)
>
> If I changed /vol0 options to (ufs, local, soft-updates) the above error is
> gone and apxs / compilation works fine.
>
> Can somebody look at this problem?
    Can you please provide output from ktrace / truss for the issue?
Thanks,
-Garrett