zfs discuss - Oct 2007 - zfs boot doesn''t support /usr on a separate partition.

I would like confirm that Solaris Express Developer Edition 09/07  
b70, you can''t have /usr on a separate zfs filesystem because of  
broken dependencies.

1/ Part of the problem is that /sbin/zpool is linked to /usr/lib/ 
libdiskmgt.so.1

2/ There are probably other broken dependencies since I''ve already  
tried to create /usr/lib and copied the libdiskmgt to it and booting  
still failed.

3/ It is really hard to diagnose the problem because solaris will  
just panic, leaving you with a fraction of a second to read the  
message before it reboots.

4/ Is there a way to add a time out to panic reboots? I guess I''ll  
have to use a video camera to capture the error message.

It would be really nice if we could fix this bug then we could have  
only a minimal root partition on two 1GB flash disk pool (/ takes  
about 500 megs) in mirror mode and have the rest of the system on  
separate raidz2 pool.

Kugutsumen

P.S. Registration on the opensolaris forum seems to be broken... it  
keeps asking me for e-mail confirmation but when I click the link it  
says the token is invalid.

Kugutsumen wrote:
> I would like confirm that Solaris Express Developer Edition 09/07  
> b70, you can''t have /usr on a separate zfs filesystem because of  
> broken dependencies.
> 1/ Part of the problem is that /sbin/zpool is linked to /usr/lib/ 
> libdiskmgt.so.1
> 2/ There are probably other broken dependencies since I''ve already
> tried to create /usr/lib and copied the libdiskmgt to it and booting  
> still failed.

I hestitate to call these "broken dependencies" - deliberate
design decision is how I''d refer to them.

> 3/ It is really hard to diagnose the problem because solaris will  
> just panic, leaving you with a fraction of a second to read the  
> message before it reboots.
> 4/ Is there a way to add a time out to panic reboots? I guess I''ll
> have to use a video camera to capture the error message.
Add "-kv" to the kernel$ line in your grub menu entry. That will
leave the system at kmdb on panic, and you can then run


::status
::msgbuf
*panic_thread::findstack -v


The output will be handy when trying to figure out the root cause.

> It would be really nice if we could fix this bug then we could have  
> only a minimal root partition on two 1GB flash disk pool (/ takes  
> about 500 megs) in mirror mode and have the rest of the system on  
> separate raidz2 pool.
There is an active effort to minimise the requirements for Solaris
in a similar model to what you suggest. However, I think it makes
the assumption that /usr is on /.




James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems

> I would like confirm that Solaris Express Developer Edition 09/07  
> b70, you can''t have /usr on a separate zfs filesystem because of  
> broken dependencies.
> 
> 1/ Part of the problem is that /sbin/zpool is linked to
/usr/lib/libdiskmgt.so.1
Yep, in the past this happened on several occasions for me:  /sbin/zfs,
 /etc/fs/zfs/mount or /lib/libzfs.so.1 depends on libraries that can only be
found
in /usr/lib and are not yet available when you have /usr in a separate zfs
filesystem - the system becomes unbootable.

See also bugs like this:

Bug ID   	 6570056
Synopsis 	/sbin/zpool should not link to files in /usr/lib
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6570056

Bug ID   	 6494840
Synopsis 	libzfs should dlopen libiscsitgt rather than linking to it
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6494840



Workaround was to copy the relevant missing shared libraries into 
the root filesystem.

I currently have snv_72 installed, and bfu''ed to the latest opensolaris
bits.  And I have /usr on a zfs filesystem. This doesn''t need extra
copies of libraries from /usr/lib in the root filesystem, and is able to
mount a separate zfs /usr filesystem.


Note that the system apparently doesn''t need /sbin/zpool for mounting
a zfs /usr filesystem; /etc/fs/zfs/mount or /sbin/zfs should be enough.

Not sure why your system is rebooting, though.  You should boot it
with option "-k", so that you can read the exact panic message.

Note that you need a valid /etc/zfs/zpool.cache file, and for zfs *root*
you also have to make sure that the /etc/zfs/zpool.cache file can be
found in the boot archive.
 
 
This message posted from opensolaris.org

On 01/10/2007, at 7:46 PM, James C. McPherson wrote:
> Kugutsumen wrote:
>> I would like confirm that Solaris Express Developer Edition 09/07   
>> b70, you can''t have /usr on a separate zfs filesystem because
of
>> broken dependencies.
>> 1/ Part of the problem is that /sbin/zpool is linked to /usr/lib/  
>> libdiskmgt.so.1
>> 2/ There are probably other broken dependencies since I''ve  
>> already  tried to create /usr/lib and copied the libdiskmgt to it  
>> and booting  still failed.
>
>
> I hestitate to call these "broken dependencies" - deliberate
> design decision is how I''d refer to them.
zpool and zonename are the only binaries in /sbin which links to /usr/ 
lib, all the others work are linked exclusively to libraries in /lib.

There is a reason we have separate /sbin and /usr/sbin  
directories ;). Historically, we always kept tools like fdisk, mount,  
ifconfig, route, sash, etc... so that in case of a failure you can  
recover the root partition easily and faster. Luckily with zfs, we  
don''t have to wait hours for  and fsck run.
>
>> 3/ It is really hard to diagnose the problem because solaris will   
>> just panic, leaving you with a fraction of a second to read the   
>> message before it reboots.
>> 4/ Is there a way to add a time out to panic reboots? I guess  
>> I''ll  have to use a video camera to capture the error message.
>
> Add "-kv" to the kernel$ line in your grub menu entry. That will
> leave the system at kmdb on panic, and you can then run
>
>
> ::status
> ::msgbuf
> *panic_thread::findstack -v
>
>
> The output will be handy when trying to figure out the root cause.
Thanks for the tip, I am going to try that in a minute and post the  
results. Someone else also suggest to use a serial cable.
>> It would be really nice if we could fix this bug then we could  
>> have  only a minimal root partition on two 1GB flash disk pool (/  
>> takes  about 500 megs) in mirror mode and have the rest of the  
>> system on  separate raidz2 pool.
>
> There is an active effort to minimise the requirements for Solaris
> in a similar model to what you suggest. However, I think it makes
> the assumption that /usr is on /.
It works fine with UFS filesystem and it should work on ZFS...  
especially Sun''s documentation and several sun.com blogs suggests to  
separate / and /usr.

http://docs.sun.com/app/docs/doc/816-5175/6mbba7f16?l=en&a=view
http://docs.sun.com/app/docs/doc/819-6396/6n8dosoq5?a=view

Thanks, you''re my hero... Should I bfu to the latest bits to fix this  
problem or do I also need to install b72?

On 01/10/2007, at 8:22 PM, J?rgen Keil wrote:
>> I would like confirm that Solaris Express Developer Edition 09/07
>> b70, you can''t have /usr on a separate zfs filesystem because
of
>> broken dependencies.
>>
>> 1/ Part of the problem is that /sbin/zpool is linked to /usr/lib/ 
>> libdiskmgt.so.1
>
> Yep, in the past this happened on several occasions for me:  /sbin/ 
> zfs,
>  /etc/fs/zfs/mount or /lib/libzfs.so.1 depends on libraries that  
> can only be found
> in /usr/lib and are not yet available when you have /usr in a  
> separate zfs
> filesystem - the system becomes unbootable.
>
> See also bugs like this:
>
> Bug ID   	 6570056
> Synopsis 	/sbin/zpool should not link to files in /usr/lib
> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6570056
>
> Bug ID   	 6494840
> Synopsis 	libzfs should dlopen libiscsitgt rather than linking to it
> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6494840
>
>
>
> Workaround was to copy the relevant missing shared libraries into
> the root filesystem.
>
> I currently have snv_72 installed, and bfu''ed to the latest  
> opensolaris
> bits.  And I have /usr on a zfs filesystem. This doesn''t need
extra
> copies of libraries from /usr/lib in the root filesystem, and is  
> able to
> mount a separate zfs /usr filesystem.
>
>
> Note that the system apparently doesn''t need /sbin/zpool for
mounting
> a zfs /usr filesystem; /etc/fs/zfs/mount or /sbin/zfs should be  
> enough.
>
> Not sure why your system is rebooting, though.  You should boot it
> with option "-k", so that you can read the exact panic message.
>
> Note that you need a valid /etc/zfs/zpool.cache file, and for zfs  
> *root*
> you also have to make sure that the /etc/zfs/zpool.cache file can be
> found in the boot archive.
>
>
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

> Should I bfu to the latest bits to fix this  
> problem or do I also need to install b72?
bfu to b72 (or newer) should be OK, iff there really is
a difference with shared library dependencies between
b70 and b72. I''m not sure about b70; but b72 with
just an empty /usr directory in the root filesystem,
used as a mount point for for mounting a zfs /usr
works just fine.

Are you trying to setup a system that boots from a 
zfs root filesystem, and has /usr on a separate zfs
filesystem?  What exactly is the panic that you get
when you try to boot with option "-k"?
 
 
This message posted from opensolaris.org