freebsd stable - Oct 2016 - Unexpected ahci-hd bytes when running in bhyve(8)

Bez?glich Harry Schmalzbauer's Nachricht vom 27.10.2016 20:05
(localtime):
>  Hello,
>
> I wanted to use a "roaming" ssd with byhve/vmm, which is the home
of a
> GPT based FreeBSD setup.
> I've been using this for years with ESXi and bare-metal-hosts, and
> wanted to try out bhyve.
> Unfortunately this doesn't work the way I'm used to.
> Booting of ufs:/dev/gpt/myROOT fails with error 19, loader does only see
> a diskid/BHYVEDISK, not the GPT partitions.
>
> I guess ahci-hd isn't 1:1 mapping blocks, neither does virtio-blk,
since
> it shows exactly the same result, which is a bit strange to me:
> When I boot a Live-CD in vmm with the physical SSD ahci-hd attached, the
> first 8kByte of /dev/ada0 is 0x0.
>
> The same test on the host ('dd if=/dev/ada4 count=16 | hd') shows
me
> PMBR and GPT content, which I also expected to see in bhyve?
>
> What am I missing?
To verify whether my assumptions might be correct at all, I installed a
bhyve guest on a file backed ahci-hd drive (/usr/local/guest.img).
The first 448 byte of that file are exactly the same when inspected on
the host as the first 448 bytes of /dev/ada4.

Inspecting /dev/ada0 inside the guest vmm, I see again the same 448
bytes when /usr/local/guest.img was attached to ahci-hd (-s
7,ahci-hd,/usr/local/guest.img).
That was what I expected.
What I don't understand is why my expectations are true for
/usr/local/guest.img but not for /dev/ada4.

Like mentioned, while reading the first 448 bytes on the host, I get
identical results from /usr/local/guest.img and /dev/ada4, but when
attaching /dev/ada4 to ahci-hd (-s 7,ahci-hd,/dev/ada4) and inspecting
inside vmm, all I see is 0x0, while ahci-hd attached
/usr/local/guest.img shows the same pmbr as on the host!?

Do I have to exclude /dev/ada4 on the host from geom? As soon as bhyve
opens /dev/ada4, all partitions vanish from the host ? probably ada4
itself gets blocked somehow?

Thanks for any hint in advance,

-harry

Bez?glich Harry Schmalzbauer's Nachricht vom 29.10.2016 17:32 (localtime):

?
> Like mentioned, while reading the first 448 bytes on the host, I get
> identical results from /usr/local/guest.img and /dev/ada4, but when
> attaching /dev/ada4 to ahci-hd (-s 7,ahci-hd,/dev/ada4) and inspecting
> inside vmm, all I see is 0x0, while ahci-hd attached
> /usr/local/guest.img shows the same pmbr as on the host!?
>
> Do I have to exclude /dev/ada4 on the host from geom? As soon as bhyve
> opens /dev/ada4, all partitions vanish from the host ? probably ada4
> itself gets blocked somehow?
Maybe that's related?
https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-April/003509.html
(resulting in https://svnweb.freebsd.org/base?view=revision&revision=281700)

Just another symptom I can only describe, not debug:
Opening /dev/adaX on the host works by 'hd /dev/ada4 | less',
but not inside the guest, where it just leads to endless IO when trying
the same on the ahci-hd attached /dev/ada4.

Of course I found discussion threads about virtio-scsi, which was more
appropriate for my needs, but unfortunately nobody skilled enough had
time to implement yet afaik and it also wouldn't solve my problems while
this ssd is SATA attached (could switch to a SAS port so the HBA would
do SAT which should work then...)
Are there any other ways I'm missing to get mass storage into the guest?
ZVOl is a very good candidate for many setups, but not for all.
RAw-device-mappings to HBA-virtual-drives is doing a great job on ESXi,
but replacing HBA-virt-drive RAW-mappings with ZVOL isn't really the
same and sometimes I need physical devices in the guest.
P(cie)P(ass)T(through) seems to work great in bhyve, but I can't
sacrifice a complete HBA to accomplish.

Thanks,

-harry