freebsd stable - Mar 2021 - FreeBSD 13.0 RC1 UEFI RAID-10 boot problems under VMware Fusion

Hi List,

With FreeBSD 13 getting near release I was trying out a new hardware setup for a
future upgrade, in where a zfs SATA RAID-10 array would be accelated by some
NVME devices for cache, log, and special meta data.

However, booting the setup under VMWare fusion gives me a lot of zio_read error:
5 / ZFS: i/o error - all block copies unavailable whereas in VirtualBox using
the same VMDKs the setup boots without issue, both in UEFI mode

I used the guided ZFS install, GPT UEFI only, and choose RAID-10 and zero swap
as I want to use the NVME devices for that later on.

when on the loader prompt lsdev / lszfs / ls works through latter two throw out
zio_read error: 5 but show recognisable output (zfs filesystems, files)


disk0 through 4 are the SATA disks with only an EFI and ZFS GPT partition each,
disk4/5 is reserved for the special vdevs (but not in use yet) and swap

If I press escape and end up in VMWare?s UEFI setup screen I can boot from any
ada*p1 drive and continue as normal.
Is UEFI with OpenZFS too new, or is this an issue in VMWare?



Also, I?m missing /boot/*efifat* in FreeBSD13. What is the procedure for
updating EFI loaders?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20210309/d330b8a2/attachment.sig>

On Tue, Mar 9, 2021 at 8:56 AM Ruben van Staveren via freebsd-stable <
freebsd-stable at freebsd.org> wrote:
> Hi List,
>
> With FreeBSD 13 getting near release I was trying out a new hardware setup
> for a future upgrade, in where a zfs SATA RAID-10 array would be accelated
> by some NVME devices for cache, log, and special meta data.
>
> However, booting the setup under VMWare fusion gives me a lot of zio_read
> error: 5 / ZFS: i/o error - all block copies unavailable whereas in
> VirtualBox using the same VMDKs the setup boots without issue, both in UEFI
> mode
>
> I used the guided ZFS install, GPT UEFI only, and choose RAID-10 and zero
> swap as I want to use the NVME devices for that later on.
>
> when on the loader prompt lsdev / lszfs / ls works through latter two
> throw out zio_read error: 5 but show recognisable output (zfs filesystems,
> files)
>
>
> disk0 through 4 are the SATA disks with only an EFI and ZFS GPT partition
> each, disk4/5 is reserved for the special vdevs (but not in use yet) and
> swap
>
> If I press escape and end up in VMWare?s UEFI setup screen I can boot from
> any ada*p1 drive and continue as normal.
> Is UEFI with OpenZFS too new, or is this an issue in VMWare?
>
>
>
> Also, I?m missing /boot/*efifat* in FreeBSD13. What is the procedure for
> updating EFI loaders?
>
They have been removed because they are no longer needed (filesystem images
for boot blocks trouble me too).

mount -t msdos /dev/da0pX /mnt
mv /mnt/efi/boot/bootx64.efi /mnt/efi/boot/bootx64-old.efi
cp /boot/loader.efi /mnt/efi/boot/bootx64.efi

The ESP on UEFI systems is just a FAT filesystem.

One issue you may run into is the size of the partition. If it is tiny,
you'll likely have to create a new ESP. Using /boot/boot1.efi may help and
can be used in the last step instead of loader.efi, but it's much less
flexible than loader.efi.

Warner

To continue on the subject of UEFI booting weirdness
> On 9 Mar 2021, at 16:57, Ruben van Staveren <ruben at verweg.com>
wrote:
> 
> If I press escape and end up in VMWare?s UEFI setup screen I can boot from
any ada*p1 drive and continue as normal.
> Is UEFI with OpenZFS too new, or is this an issue in VMWare?
I got an off list tip to see whether this was also the case in bhyve, so I also
created the setup in there, using UEFI boot, and no problems even with the
special/log/cache NVMe vdevs attached to the pool.

So I?m starting to wonder whether the loader  / VMware UEFI firmware (??)
interaction is a bug in VMware or an edge case that needs to be supported too.

Btw, bhyve is looking nice these days!

Cheers,	
    Ruben