freebsd stable - Jul 2020 - 12.1p7 no longer boots after doing zpool upgrade -a

On Thu, Jul 09, 2020 at 08:24:54AM -0500, Kyle Evans
wrote:
> On Thu, Jul 9, 2020 at 8:12 AM Guido van Rooij <guido at gvr.org>
wrote:
> >
> > I did a zpool upgrade -a to enable large_dnode and spacemap_v2.
> > After that, I did:
> > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada0
> > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada1
> > and:
> > gpart bootcode -p /boot/boot1.efifat -i 1 ada0
> > gpart bootcode -p /boot/boot1.efifat -i 1 ada1
> >
> > Now the system no longer boots from either disk and drops to the efi
shell.
> 
> This method of updating the ESP is no longer recommended for new 12.x
> installations -- we now more carefully construct the ESP with an
> /EFI/FreeBSD/loader.efi where loader.efi is /boot/loader.efi. You will
> want to rebuild this as such, and that may fix part of your problem.
Hi Kyle,

Thnaks for your asnwer. I have not got it to work with that
configuration. What did work was to replace the  /efi/boot/BOOTx64.efi
with loader.efi and and change the content of startup.nsh with
loader.efi. Withoyt the above answer I wouldn't have figure it out
that quickly so thanks!

I will investigate further once I have more time (early next week probably).
> 
> > From memory, the partitions on ada0 and ada1 are as folows:
> >
> > 1 efi
> > 2 freebsd-boot
> > 3 freebsd-swap
> > 4 freebsd
> > p4 is a geli partition with zfs in it.
> > ada0p4.eli and ada1p4.eli form a mirrored zpool (called zroot)..
> >
> > The boot process no longer asks for a GELi password. It just isues
> > a no zfs pools found.
> >
> > When I boot from a stick, it does ask for the GELi password. When
> > I set currdev correctly in the loader, I am able to load the kernel,
> > opensolaris.ko and zfs.ko. However booting failed because I get an
> > Error: Solaris: NOTICE: Cannot find the pool label for 'zroot'
> >
> 
> Was vfs.root.mountfrom set correctly? My completely uneducated guess
> is that it likely wasn't and we couldn't locate the zpool.cache --
the
> value of vfs.root.mountfrom should basically match the currdev format
> here, except without a trailing colon.
I did not set this sysctl so most likely that was the cause. As the system 
is up and running again I didn't test it.
Let me know if you want me to.

-Guido

On Fri, Jul 10, 2020 at 08:29:03PM +0200, Guido van Rooij
wrote:
> On Thu, Jul 09, 2020 at 08:24:54AM -0500, Kyle Evans wrote:
> > On Thu, Jul 9, 2020 at 8:12 AM Guido van Rooij <guido at
gvr.org> wrote:
> > >
> > > I did a zpool upgrade -a to enable large_dnode and spacemap_v2.
> > > After that, I did:
> > > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada0
> > > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada1
> > > and:
> > > gpart bootcode -p /boot/boot1.efifat -i 1 ada0
> > > gpart bootcode -p /boot/boot1.efifat -i 1 ada1
> > >
> > > Now the system no longer boots from either disk and drops to the
efi shell.
> > 
> > This method of updating the ESP is no longer recommended for new 12.x
> > installations -- we now more carefully construct the ESP with an
> > /EFI/FreeBSD/loader.efi where loader.efi is /boot/loader.efi. You will
> > want to rebuild this as such, and that may fix part of your problem.
> 
> Hi Kyle,
> 
> Thnaks for your asnwer. I have not got it to work with that
> configuration. What did work was to replace the  /efi/boot/BOOTx64.efi
> with loader.efi and and change the content of startup.nsh with
> loader.efi. Withoyt the above answer I wouldn't have figure it out
> that quickly so thanks!
> 
> I will investigate further once I have more time (early next week
probably).
There was one question I forgot to ask:
Could I have known that my method of updating the ESP was not correct?
If so, where is this documented?

-Guido