Kugutsumen
2007-Oct-01 12:31 UTC
[zfs-discuss] zfs boot doesn''t support /usr on a separate partition.
I would like confirm that Solaris Express Developer Edition 09/07 b70, you can''t have /usr on a separate zfs filesystem because of broken dependencies. 1/ Part of the problem is that /sbin/zpool is linked to /usr/lib/ libdiskmgt.so.1 2/ There are probably other broken dependencies since I''ve already tried to create /usr/lib and copied the libdiskmgt to it and booting still failed. 3/ It is really hard to diagnose the problem because solaris will just panic, leaving you with a fraction of a second to read the message before it reboots. 4/ Is there a way to add a time out to panic reboots? I guess I''ll have to use a video camera to capture the error message. It would be really nice if we could fix this bug then we could have only a minimal root partition on two 1GB flash disk pool (/ takes about 500 megs) in mirror mode and have the rest of the system on separate raidz2 pool. Kugutsumen P.S. Registration on the opensolaris forum seems to be broken... it keeps asking me for e-mail confirmation but when I click the link it says the token is invalid.
James C. McPherson
2007-Oct-01 12:46 UTC
[zfs-discuss] zfs boot doesn''t support /usr on a separate partition.
Kugutsumen wrote:> I would like confirm that Solaris Express Developer Edition 09/07 > b70, you can''t have /usr on a separate zfs filesystem because of > broken dependencies. > 1/ Part of the problem is that /sbin/zpool is linked to /usr/lib/ > libdiskmgt.so.1 > 2/ There are probably other broken dependencies since I''ve already > tried to create /usr/lib and copied the libdiskmgt to it and booting > still failed.I hestitate to call these "broken dependencies" - deliberate design decision is how I''d refer to them.> 3/ It is really hard to diagnose the problem because solaris will > just panic, leaving you with a fraction of a second to read the > message before it reboots. > 4/ Is there a way to add a time out to panic reboots? I guess I''ll > have to use a video camera to capture the error message.Add "-kv" to the kernel$ line in your grub menu entry. That will leave the system at kmdb on panic, and you can then run ::status ::msgbuf *panic_thread::findstack -v The output will be handy when trying to figure out the root cause.> It would be really nice if we could fix this bug then we could have > only a minimal root partition on two 1GB flash disk pool (/ takes > about 500 megs) in mirror mode and have the rest of the system on > separate raidz2 pool.There is an active effort to minimise the requirements for Solaris in a similar model to what you suggest. However, I think it makes the assumption that /usr is on /. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems
Jürgen Keil
2007-Oct-01 13:22 UTC
[zfs-discuss] zfs boot doesn''t support /usr on a separate partition.
> I would like confirm that Solaris Express Developer Edition 09/07 > b70, you can''t have /usr on a separate zfs filesystem because of > broken dependencies. > > 1/ Part of the problem is that /sbin/zpool is linked to /usr/lib/libdiskmgt.so.1Yep, in the past this happened on several occasions for me: /sbin/zfs, /etc/fs/zfs/mount or /lib/libzfs.so.1 depends on libraries that can only be found in /usr/lib and are not yet available when you have /usr in a separate zfs filesystem - the system becomes unbootable. See also bugs like this: Bug ID 6570056 Synopsis /sbin/zpool should not link to files in /usr/lib http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6570056 Bug ID 6494840 Synopsis libzfs should dlopen libiscsitgt rather than linking to it http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6494840 Workaround was to copy the relevant missing shared libraries into the root filesystem. I currently have snv_72 installed, and bfu''ed to the latest opensolaris bits. And I have /usr on a zfs filesystem. This doesn''t need extra copies of libraries from /usr/lib in the root filesystem, and is able to mount a separate zfs /usr filesystem. Note that the system apparently doesn''t need /sbin/zpool for mounting a zfs /usr filesystem; /etc/fs/zfs/mount or /sbin/zfs should be enough. Not sure why your system is rebooting, though. You should boot it with option "-k", so that you can read the exact panic message. Note that you need a valid /etc/zfs/zpool.cache file, and for zfs *root* you also have to make sure that the /etc/zfs/zpool.cache file can be found in the boot archive. This message posted from opensolaris.org
Kugutsumen
2007-Oct-01 13:30 UTC
[zfs-discuss] zfs boot doesn''t support /usr on a separate partition.
On 01/10/2007, at 7:46 PM, James C. McPherson wrote:> Kugutsumen wrote: >> I would like confirm that Solaris Express Developer Edition 09/07 >> b70, you can''t have /usr on a separate zfs filesystem because of >> broken dependencies. >> 1/ Part of the problem is that /sbin/zpool is linked to /usr/lib/ >> libdiskmgt.so.1 >> 2/ There are probably other broken dependencies since I''ve >> already tried to create /usr/lib and copied the libdiskmgt to it >> and booting still failed. > > > I hestitate to call these "broken dependencies" - deliberate > design decision is how I''d refer to them.zpool and zonename are the only binaries in /sbin which links to /usr/ lib, all the others work are linked exclusively to libraries in /lib. There is a reason we have separate /sbin and /usr/sbin directories ;). Historically, we always kept tools like fdisk, mount, ifconfig, route, sash, etc... so that in case of a failure you can recover the root partition easily and faster. Luckily with zfs, we don''t have to wait hours for and fsck run.> >> 3/ It is really hard to diagnose the problem because solaris will >> just panic, leaving you with a fraction of a second to read the >> message before it reboots. >> 4/ Is there a way to add a time out to panic reboots? I guess >> I''ll have to use a video camera to capture the error message. > > Add "-kv" to the kernel$ line in your grub menu entry. That will > leave the system at kmdb on panic, and you can then run > > > ::status > ::msgbuf > *panic_thread::findstack -v > > > The output will be handy when trying to figure out the root cause.Thanks for the tip, I am going to try that in a minute and post the results. Someone else also suggest to use a serial cable.>> It would be really nice if we could fix this bug then we could >> have only a minimal root partition on two 1GB flash disk pool (/ >> takes about 500 megs) in mirror mode and have the rest of the >> system on separate raidz2 pool. > > There is an active effort to minimise the requirements for Solaris > in a similar model to what you suggest. However, I think it makes > the assumption that /usr is on /.It works fine with UFS filesystem and it should work on ZFS... especially Sun''s documentation and several sun.com blogs suggests to separate / and /usr. http://docs.sun.com/app/docs/doc/816-5175/6mbba7f16?l=en&a=view http://docs.sun.com/app/docs/doc/819-6396/6n8dosoq5?a=view
Kugutsumen
2007-Oct-01 13:43 UTC
[zfs-discuss] zfs boot doesn''t support /usr on a separate partition.
Thanks, you''re my hero... Should I bfu to the latest bits to fix this problem or do I also need to install b72? On 01/10/2007, at 8:22 PM, J?rgen Keil wrote:>> I would like confirm that Solaris Express Developer Edition 09/07 >> b70, you can''t have /usr on a separate zfs filesystem because of >> broken dependencies. >> >> 1/ Part of the problem is that /sbin/zpool is linked to /usr/lib/ >> libdiskmgt.so.1 > > Yep, in the past this happened on several occasions for me: /sbin/ > zfs, > /etc/fs/zfs/mount or /lib/libzfs.so.1 depends on libraries that > can only be found > in /usr/lib and are not yet available when you have /usr in a > separate zfs > filesystem - the system becomes unbootable. > > See also bugs like this: > > Bug ID 6570056 > Synopsis /sbin/zpool should not link to files in /usr/lib > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6570056 > > Bug ID 6494840 > Synopsis libzfs should dlopen libiscsitgt rather than linking to it > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6494840 > > > > Workaround was to copy the relevant missing shared libraries into > the root filesystem. > > I currently have snv_72 installed, and bfu''ed to the latest > opensolaris > bits. And I have /usr on a zfs filesystem. This doesn''t need extra > copies of libraries from /usr/lib in the root filesystem, and is > able to > mount a separate zfs /usr filesystem. > > > Note that the system apparently doesn''t need /sbin/zpool for mounting > a zfs /usr filesystem; /etc/fs/zfs/mount or /sbin/zfs should be > enough. > > Not sure why your system is rebooting, though. You should boot it > with option "-k", so that you can read the exact panic message. > > Note that you need a valid /etc/zfs/zpool.cache file, and for zfs > *root* > you also have to make sure that the /etc/zfs/zpool.cache file can be > found in the boot archive. > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Jürgen Keil
2007-Oct-01 17:32 UTC
[zfs-discuss] zfs boot doesn''t support /usr on a separate partition.
> Should I bfu to the latest bits to fix this > problem or do I also need to install b72?bfu to b72 (or newer) should be OK, iff there really is a difference with shared library dependencies between b70 and b72. I''m not sure about b70; but b72 with just an empty /usr directory in the root filesystem, used as a mount point for for mounting a zfs /usr works just fine. Are you trying to setup a system that boots from a zfs root filesystem, and has /usr on a separate zfs filesystem? What exactly is the panic that you get when you try to boot with option "-k"? This message posted from opensolaris.org