So here's the fun part of what I'm trying to do (and getting frustrated
with)
I have set up a GPT disk with the following setup:
=> 34 625142381 da2 GPT (298G)
34 6 - free - (3.0K)
40 1024 1 freebsd-boot (512K)
1064 4194304 2 freebsd-zfs [bootme] (2.0G)
4195368 134217728 3 freebsd-swap (64G)
138413096 486729312 4 freebsd-zfs (232G)
625142408 7 - free - (3.5K)
Then on freebsd-boot I have written the bootloaders.
The "bootme" filesystem has *only* the /boot directory copied over
from
the rest of the system's root directory (that is, the kernel, loadables,
/boot/loader.conf, etc); that pool is called "zboot"
Partition 4 has the label "root0" on it, and thus shows up in
/dev/gpt.
I have initialized that with geli, set the boot option flag (that is,
prompt on boot) and created a pool called "root" on the resulting .eli
device and then put the system on that. That's all ok.
Finally, I set the bootfs on that latter pool. There is no bootfs set
on /zboot:
# zpool get bootfs zboot
NAME PROPERTY VALUE SOURCE
zboot bootfs - default
It is set on the root pool to the proper filesystem:
# zpool get bootfs root
NAME PROPERTY VALUE SOURCE
root bootfs root/R/10.1-CLEAN local
The problem is that when the system boots geli "finds" the raw device
(in this case /dev/da0p4), prompts for the password and attaches there
instead of in /dev/gpt. The gpt label is missing --- and equally bad
the "root" pool does not appear to import at boot time either.
As a result the system tries to mount root from /zboot (even though it's
not been told to, and HAS been told where to mount off the root pool),
but there's no init in there (or anything else other than the boot
filesystem itself) and as a result I get an immediate panic.
If I boot off a different (working) zfs-based system the probe still
finds the "prompt during boot" flag on that gpt partition and asks for
the password on the device. I can see the pool; zpool import shows it:
pool: root
id: 17719633931604198170
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
root ONLINE
da2p4.eli ONLINE
Not so good.
If I detach that the device reappears in /dev/gpt; I can then attach
geli and import the pool in either location. Putting the cache file
from the previous imported state in the zboot/boot/zfs directory doesn't
help (nor does removing the cache file entirely)
More-interestingly if I reboot the cloned system with the root pool
imported it does come back up, even though the device is the base
(da2p4.eli) rather than in the /dev/gpt directory.
Anyone know what's going on here? And is there a way to have geli
attach during boot-time off the /dev/gpt directory instead of on the
base device partition name?
--
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2711 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20141001/74069f3f/attachment.bin>
On 02/10/2014 00:27, Karl Denninger wrote:> So here's the fun part of what I'm trying to do (and getting frustrated > with) > > I have set up a GPT disk with the following setup: > > => 34 625142381 da2 GPT (298G) > 34 6 - free - (3.0K) > 40 1024 1 freebsd-boot (512K) > 1064 4194304 2 freebsd-zfs [bootme] (2.0G) > 4195368 134217728 3 freebsd-swap (64G) > 138413096 486729312 4 freebsd-zfs (232G) > 625142408 7 - free - (3.5K) > > Then on freebsd-boot I have written the bootloaders. > > The "bootme" filesystem has *only* the /boot directory copied over from > the rest of the system's root directory (that is, the kernel, loadables, > /boot/loader.conf, etc); that pool is called "zboot" > > Partition 4 has the label "root0" on it, and thus shows up in /dev/gpt. > I have initialized that with geli, set the boot option flag (that is, > prompt on boot) and created a pool called "root" on the resulting .eli > device and then put the system on that. That's all ok. > > Finally, I set the bootfs on that latter pool. There is no bootfs set > on /zboot: > > # zpool get bootfs zboot > NAME PROPERTY VALUE SOURCE > zboot bootfs - default > > It is set on the root pool to the proper filesystem: > > # zpool get bootfs root > NAME PROPERTY VALUE SOURCE > root bootfs root/R/10.1-CLEAN local > > The problem is that when the system boots geli "finds" the raw device > (in this case /dev/da0p4), prompts for the password and attaches there > instead of in /dev/gpt. The gpt label is missing --- and equally bad > the "root" pool does not appear to import at boot time either. > > As a result the system tries to mount root from /zboot (even though it's > not been told to, and HAS been told where to mount off the root pool),As far as *I* can see, you have not told the kernel what your root fs should be, so it is using a default root filesystem which the same filesystem from where the kernel itself was loaded.> but there's no init in there (or anything else other than the boot > filesystem itself) and as a result I get an immediate panic.-- Andriy Gapon
Quoth Karl Denninger <karl at denninger.net>:> > The problem is that when the system boots geli "finds" the raw device > (in this case /dev/da0p4), prompts for the password and attaches there > instead of in /dev/gpt.Since you're using ZFS, that doesn't matter (as you noticed below). ZFS treats the names that devices were added to the pool with as hints only, and actually searches through all available disks every time it imports a pool, looking for volumes it recognises.> The gpt label is missing --- and equally bad > the "root" pool does not appear to import at boot time either. > > As a result the system tries to mount root from /zboot (even though it's > not been told to, and HAS been told where to mount off the root pool), > but there's no init in there (or anything else other than the boot > filesystem itself) and as a result I get an immediate panic. > > If I boot off a different (working) zfs-based system the probe still > finds the "prompt during boot" flag on that gpt partition and asks for > the password on the device. I can see the pool; zpool import shows it: > > pool: root > id: 17719633931604198170 > state: ONLINE > action: The pool can be imported using its name or numeric identifier. > config: > > root ONLINE > da2p4.eli ONLINEYou appear to have exported the pool? I don't think you want to that, since, as you've found, the kernel will not (must not) automatically import pools at boot time.> More-interestingly if I reboot the cloned system with the root pool > imported it does come back up, even though the device is the base > (da2p4.eli) rather than in the /dev/gpt directory.Is there any reason not to simply leave the pool imported when you reboot? The geli device will detach when the system shuts down, after ZFS has finished flushing data; when it reattaches at boot time, ZFS will see an imported pool and make it available.> Anyone know what's going on here? And is there a way to have geli > attach during boot-time off the /dev/gpt directory instead of on the > base device partition name?I don't have a definitive answer to that, but I strongly suspect not. The only place for the information to come from would be /boot/loader.conf, and there's no mention of an appropriate tunable in geli(8). But, as you already found out, it doesn't matter. Ben
On 10/01/2014 4:27 pm, Karl Denninger wrote:> So here's the fun part of what I'm trying to do (and getting frustrated > with) > > I have set up a GPT disk with the following setup: > > => 34 625142381 da2 GPT (298G) > 34 6 - free - (3.0K) > 40 1024 1 freebsd-boot (512K) > 1064 4194304 2 freebsd-zfs [bootme] (2.0G) > 4195368 134217728 3 freebsd-swap (64G) > 138413096 486729312 4 freebsd-zfs (232G) > 625142408 7 - free - (3.5K) > > Then on freebsd-boot I have written the bootloaders. > > The "bootme" filesystem has *only* the /boot directory copied over from > the rest of the system's root directory (that is, the kernel, > loadables, > /boot/loader.conf, etc); that pool is called "zboot" > > Partition 4 has the label "root0" on it, and thus shows up in /dev/gpt. > I have initialized that with geli, set the boot option flag (that is, > prompt on boot) and created a pool called "root" on the resulting .eli > device and then put the system on that. That's all ok. > > Finally, I set the bootfs on that latter pool. There is no bootfs set > on /zboot: > > # zpool get bootfs zboot > NAME PROPERTY VALUE SOURCE > zboot bootfs - default > > It is set on the root pool to the proper filesystem: > > # zpool get bootfs root > NAME PROPERTY VALUE SOURCE > root bootfs root/R/10.1-CLEAN local > > The problem is that when the system boots geli "finds" the raw device > (in this case /dev/da0p4), prompts for the password and attaches there > instead of in /dev/gpt. The gpt label is missing --- and equally bad > the "root" pool does not appear to import at boot time either. > > As a result the system tries to mount root from /zboot (even though > it's > not been told to, and HAS been told where to mount off the root pool), > but there's no init in there (or anything else other than the boot > filesystem itself) and as a result I get an immediate panic. > > If I boot off a different (working) zfs-based system the probe still > finds the "prompt during boot" flag on that gpt partition and asks for > the password on the device. I can see the pool; zpool import shows it: > > pool: root > id: 17719633931604198170 > state: ONLINE > action: The pool can be imported using its name or numeric identifier. > config: > > root ONLINE > da2p4.eli ONLINE > > Not so good. > > If I detach that the device reappears in /dev/gpt; I can then attach > geli and import the pool in either location. Putting the cache file > from the previous imported state in the zboot/boot/zfs directory > doesn't > help (nor does removing the cache file entirely) > > More-interestingly if I reboot the cloned system with the root pool > imported it does come back up, even though the device is the base > (da2p4.eli) rather than in the /dev/gpt directory. > > Anyone know what's going on here? And is there a way to have geli > attach during boot-time off the /dev/gpt directory instead of on the > base device partition name?On my work laptop (not turned on so I am going by memory on this), I have a similar setup using a USB thumb drive for the boot volume. My setup is as follows and works quite well, perhaps this will help you. Thumb Drive da0 Disk Drive ada0 da0 has a GPT table of da0 GPT (8G) 1 freebsd-boot (512k) -- /dev/gpt/usbboot 2 freebsd-zfs (8G) -- /dev/gpt/usbzfs gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 ada0 has a GPT table of 1 freebsd-swap (8G) -- /dev/gpt/swap 2 freebsd-zfs (222G) -- /dev/gpt/zroot I used geli init -b /dev/gpt/zroot when attached /dev/gpt/zroot.eli swap is auto encrypted at boot using the fstab line /dev/gpt/swap.eli none none swap sw 0 0 I believe they devices only show up as /dev/gpt/... if the -l ... option is used to set a label on the partition at creation time. 2 configured zpools usbzfs gpt/usbzfs zroot gpt/zroot.eli zpool set bootfs=usbzfs/boot usbzfs zpool set bootfs=zroot/ROOT/installation zroot (not sure if this does anything, I just set it) usbzfs/boot has a mountpoint of /zfsboot loader.conf: zfs_load="YES" vfs.root.mountfrom="zfs:zroot/ROOT/install" copied /boot to /zfsboot/boot zpool export usbzfs It will still boot after the zpool has been exported if the devices is found, just doesn't get mounted, in my case this means I can remove the USB thumb drive as soon as root is remounted from the geli partition, after entering the password without causing any issues. I can send you the full gpt output and zpool status information tomorrow morning when I am back in the office on my laptop if you still need help getting yours working. -- Thanks, Dean E. Weimer http://www.dweimer.net/