thr3ads.net - freebsd stable - 12.1p7 no longer boots after doing zpool upgrade -a [Jul 2020]

If this information is useful, please help other people find it:
Share via:

Kyle Evans

2020-Jul-09 13:24 UTC

12.1p7 no longer boots after doing zpool upgrade -a

On Thu, Jul 9, 2020 at 8:12 AM Guido van Rooij <guido at gvr.org>
wrote:>
> I did a zpool upgrade -a to enable large_dnode and spacemap_v2.
> After that, I did:
> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada0
> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada1
> and:
> gpart bootcode -p /boot/boot1.efifat -i 1 ada0
> gpart bootcode -p /boot/boot1.efifat -i 1 ada1
>
> Now the system no longer boots from either disk and drops to the efi shell.
This method of updating the ESP is no longer recommended for new 12.x
installations -- we now more carefully construct the ESP with an
/EFI/FreeBSD/loader.efi where loader.efi is /boot/loader.efi. You will
want to rebuild this as such, and that may fix part of your problem.
> From memory, the partitions on ada0 and ada1 are as folows:
>
> 1 efi
> 2 freebsd-boot
> 3 freebsd-swap
> 4 freebsd
> p4 is a geli partition with zfs in it.
> ada0p4.eli and ada1p4.eli form a mirrored zpool (called zroot)..
>
> The boot process no longer asks for a GELi password. It just isues
> a no zfs pools found.
>
> When I boot from a stick, it does ask for the GELi password. When
> I set currdev correctly in the loader, I am able to load the kernel,
> opensolaris.ko and zfs.ko. However booting failed because I get an
> Error: Solaris: NOTICE: Cannot find the pool label for 'zroot'
>
Was vfs.root.mountfrom set correctly? My completely uneducated guess
is that it likely wasn't and we couldn't locate the zpool.cache -- the
value of vfs.root.mountfrom should basically match the currdev format
here, except without a trailing colon.

Thanks,

Kyle Evans

Pete French

2020-Jul-09 13:32 UTC

head link

12.1p7 no longer boots after doing zpool upgrade -a

On 09/07/2020 14:24, Kyle Evans wrote:
>> gpart bootcode -p /boot/boot1.efifat -i 1 ada0
>> gpart bootcode -p /boot/boot1.efifat -i 1 ada1
>
> 
> This method of updating the ESP is no longer recommended for new 12.x
> installations -- we now more carefully construct the ESP with an
> /EFI/FreeBSD/loader.efi where loader.efi is /boot/loader.efi. You will
> want to rebuild this as such, and that may fix part of your problem.
Out of interest, how should the ESP partition be upgraded then ? I dont 
have any EFI machines...yet. But one day I will, and I was assuming that 
an upgrade would be done using the above lines too.

-pete.

John Kennedy

2020-Jul-09 21:18 UTC

head link

12.1p7 no longer boots after doing zpool upgrade -a

On Thu, Jul 09, 2020 at 08:24:54AM -0500, Kyle Evans
wrote:> On Thu, Jul 9, 2020 at 8:12 AM Guido van Rooij <guido at gvr.org>
wrote:
> >
> > I did a zpool upgrade -a to enable large_dnode and spacemap_v2.
> > After that, I did:
> > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada0
> > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada1
> > and:
> > gpart bootcode -p /boot/boot1.efifat -i 1 ada0
> > gpart bootcode -p /boot/boot1.efifat -i 1 ada1
> >
> > Now the system no longer boots from either disk and drops to the efi
shell.
> 
> This method of updating the ESP is no longer recommended for new 12.x
> installations -- we now more carefully construct the ESP with an
> /EFI/FreeBSD/loader.efi where loader.efi is /boot/loader.efi. You will
> want to rebuild this as such, and that may fix part of your problem.
  So, I've got a stable box:

	FreeBSD 12.1-STABLE #156 r362983+cd7a9efd329c(stable/12): Tue Jul  7 07:42:17
PDT 2020

  I've got almost the same situation, except I haven't shot myself in
the foot yet.
My partitions:

	# gpart show nvd0 nvd1
	=>       40  976773088  nvd0  GPT  (466G)
	         40     409600     1  efi  (200M)
	     409640       1024     2  freebsd-boot  (512K)
	     410664        984        - free -  (492K)
	     411648   16777216     3  freebsd-swap  (8.0G)
	   17188864  959584256     4  freebsd-zfs  (458G)
	  976773120          8        - free -  (4.0K)

	=>       40  976773088  nvd1  GPT  (466G)
	         40     409600     1  efi  (200M)
	     409640       1024     2  freebsd-boot  (512K)
	     410664        984        - free -  (492K)
	     411648   16777216     3  freebsd-swap  (8.0G)
	   17188864  959584256     4  freebsd-zfs  (458G)
	  976773120          8        - free -  (4.0K)

  So this is my first UEFI system that's hit an upgrade point:

	# zpool status
	  pool: zroot
	 state: ONLINE
	status: Some supported features are not enabled on the pool. The pool can
	        still be used, but some features are unavailable.
	action: Enable all features using 'zpool upgrade'. Once this is done,
	        the pool may no longer be accessible by software that does not support
	        the features. See zpool-features(7) for details.
	  scan: scrub repaired 0 in 0 days 00:02:04 with 0 errors on Wed Jul  1
13:21:57 2020
	config:

	        NAME          STATE     READ WRITE CKSUM
	        zroot         ONLINE       0     0     0
	          nvd1p4.eli  ONLINE       0     0     0
	          nvd0p4.eli  ONLINE       0     0     0

  I usually seed these things off of stick and they evolve from there, but this
one is slightly interesting because the motherboard died repeatedly and I'm
guessing that the M.2 drives got reinstalled in the "wrong" order so
now if I
don't manually select one of the M.2 drives, it won't reboot (acts like
the
encryption password is bad, fails out into a shell where I reboot).

  In any case, I know if I upgrade, it is going to tell me to upgrade the boot
code and I'm wondering what that is these days.

  If this was a Pre-UEFI, I would be expecting to do (only) something like:

	gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nvd0
	gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nvd1

  IE, might hit either drive, update the freebsd-boot partition by # on both,
and special note that the # is now one higher because of the EFI partition.

  The FreeBSD wiki still has the "old" way of doing it here...

	https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot

  ... but your newer way of doing it here:

	https://wiki.freebsd.org/UEFI

  So the newer way of doing it would be something like this, except I'd by
updating
the file rather than looking at it's checksum/version?

	strings -a < /boot/loader.efi | grep -A1 'EFI loader'
		FreeBSD/amd64 EFI loader, Revision 1.1
		(Tue Jul  7 07:33:24 PDT 2020 warlock at ouroboros.phouka.net)

	mount -t msdosfs /dev/nvd0p1 /mnt
	md5 -r /boot/loader.efi /mnt/efi/boot/BOOTx64.efi
		9f64a96f3170e3327ca7dad1d574d852 /boot/loader.efi
		cbff58cc8ee79ce7342f5be23ad14ba3 /mnt/efi/boot/BOOTx64.efi
	strings -a < /mnt/efi/boot/BOOTx64.efi | grep -A1 'EFI loader'
		FreeBSD/amd64 EFI loader, Revision 1.1
		|/-\
	umount -v /mnt

	mount -t msdosfs /dev/nvd1p1 /mnt
	md5 -r /mnt/efi/boot/BOOTx64.efi
		cbff58cc8ee79ce7342f5be23ad14ba3 /mnt/efi/boot/BOOTx64.efi
	umount -v /mnt

  At least some of the checksum difference is because
WITHOUT_REPRODUCIBLE_BUILD=YES.

  (Sorry for extra details, but I know when I was researching this, I ran
   across a *lot* of examples where we didn't have a GPT dump and people
   were just using partition #s and if you stomp on the wrong partition
   you're going to have a bad day, so leaving some specific examples here.)

  So one recipe doesn't even seem to make the freebsd-boot partition, so is
it
optional for a pure UEFI boot?  Should we always gpart-bootcode it if it exists
and upgrade BOOTx64.efi when the EFI partition exists, or is there a few more
wrinkles we need to worry about?

  In any case, is it a logical theory that my possible boot-order problem
is because drive order got swapped and maybe one wasn't properly updated?
They seem to be the same:

	# md5 /dev/nvd[01]p2
		MD5 (/dev/nvd0p2) = 2ded438a2c97ea551446cc2d1d3b498e
		MD5 (/dev/nvd1p2) = 2ded438a2c97ea551446cc2d1d3b498e

  Ideally I'd like to have not boot through the UEFI boot menu every time.

  I'm not sure why the drive order seems to matter right now.

Guido van Rooij

2020-Jul-10 18:29 UTC

head link

12.1p7 no longer boots after doing zpool upgrade -a

On Thu, Jul 09, 2020 at 08:24:54AM -0500, Kyle Evans
wrote:> On Thu, Jul 9, 2020 at 8:12 AM Guido van Rooij <guido at gvr.org>
wrote:
> >
> > I did a zpool upgrade -a to enable large_dnode and spacemap_v2.
> > After that, I did:
> > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada0
> > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada1
> > and:
> > gpart bootcode -p /boot/boot1.efifat -i 1 ada0
> > gpart bootcode -p /boot/boot1.efifat -i 1 ada1
> >
> > Now the system no longer boots from either disk and drops to the efi
shell.
> 
> This method of updating the ESP is no longer recommended for new 12.x
> installations -- we now more carefully construct the ESP with an
> /EFI/FreeBSD/loader.efi where loader.efi is /boot/loader.efi. You will
> want to rebuild this as such, and that may fix part of your problem.
Hi Kyle,

Thnaks for your asnwer. I have not got it to work with that
configuration. What did work was to replace the  /efi/boot/BOOTx64.efi
with loader.efi and and change the content of startup.nsh with
loader.efi. Withoyt the above answer I wouldn't have figure it out
that quickly so thanks!

I will investigate further once I have more time (early next week probably).
> 
> > From memory, the partitions on ada0 and ada1 are as folows:
> >
> > 1 efi
> > 2 freebsd-boot
> > 3 freebsd-swap
> > 4 freebsd
> > p4 is a geli partition with zfs in it.
> > ada0p4.eli and ada1p4.eli form a mirrored zpool (called zroot)..
> >
> > The boot process no longer asks for a GELi password. It just isues
> > a no zfs pools found.
> >
> > When I boot from a stick, it does ask for the GELi password. When
> > I set currdev correctly in the loader, I am able to load the kernel,
> > opensolaris.ko and zfs.ko. However booting failed because I get an
> > Error: Solaris: NOTICE: Cannot find the pool label for 'zroot'
> >
> 
> Was vfs.root.mountfrom set correctly? My completely uneducated guess
> is that it likely wasn't and we couldn't locate the zpool.cache --
the
> value of vfs.root.mountfrom should basically match the currdev format
> here, except without a trailing colon.
I did not set this sysctl so most likely that was the cause. As the system 
is up and running again I didn't test it.
Let me know if you want me to.

-Guido

freebsd stable - Jul 2020 - 12.1p7 no longer boots after doing zpool upgrade -a

12.1p7 no longer boots after doing zpool upgrade -a

12.1p7 no longer boots after doing zpool upgrade -a

12.1p7 no longer boots after doing zpool upgrade -a

12.1p7 no longer boots after doing zpool upgrade -a