freebsd stable - Oct 2016 - Unexpected ahci-hd bytes when running in bhyve(8)

Hello,

I wanted to use a "roaming" ssd with byhve/vmm, which is the home of a
GPT based FreeBSD setup.
I've been using this for years with ESXi and bare-metal-hosts, and
wanted to try out bhyve.
Unfortunately this doesn't work the way I'm used to.
Booting of ufs:/dev/gpt/myROOT fails with error 19, loader does only see
a diskid/BHYVEDISK, not the GPT partitions.

I guess ahci-hd isn't 1:1 mapping blocks, neither does virtio-blk, since
it shows exactly the same result, which is a bit strange to me:
When I boot a Live-CD in vmm with the physical SSD ahci-hd attached, the
first 8kByte of /dev/ada0 is 0x0.

The same test on the host ('dd if=/dev/ada4 count=16 | hd') shows me
PMBR and GPT content, which I also expected to see in bhyve?

What am I missing?

Here's my switches:

bhyve -u -A -H -P \
-S \
-s 0,hostbridge \
-s 6,passthru,6/0/0 \
-s 31,lpc \
-s 1,ahci-cd,releases/ISO-IMAGES/11.0/FreeBSD-11.0-RELEASE-amd64-disc1.iso \
-s 7,ahci-hd,/dev/ada4 \
-l com1,/dev/nmdm0A \
-m 3G -c 4 preed

/dev/ada4 is the "roaming" (hotpuggable) SSD on the host.

Thanks for any hint,

-harry

Bez?glich Harry Schmalzbauer's Nachricht vom 27.10.2016 20:05
(localtime):
>  Hello,
>
> I wanted to use a "roaming" ssd with byhve/vmm, which is the home
of a
> GPT based FreeBSD setup.
> I've been using this for years with ESXi and bare-metal-hosts, and
> wanted to try out bhyve.
> Unfortunately this doesn't work the way I'm used to.
> Booting of ufs:/dev/gpt/myROOT fails with error 19, loader does only see
> a diskid/BHYVEDISK, not the GPT partitions.
>
> I guess ahci-hd isn't 1:1 mapping blocks, neither does virtio-blk,
since
> it shows exactly the same result, which is a bit strange to me:
> When I boot a Live-CD in vmm with the physical SSD ahci-hd attached, the
> first 8kByte of /dev/ada0 is 0x0.
>
> The same test on the host ('dd if=/dev/ada4 count=16 | hd') shows
me
> PMBR and GPT content, which I also expected to see in bhyve?
>
> What am I missing?
To verify whether my assumptions might be correct at all, I installed a
bhyve guest on a file backed ahci-hd drive (/usr/local/guest.img).
The first 448 byte of that file are exactly the same when inspected on
the host as the first 448 bytes of /dev/ada4.

Inspecting /dev/ada0 inside the guest vmm, I see again the same 448
bytes when /usr/local/guest.img was attached to ahci-hd (-s
7,ahci-hd,/usr/local/guest.img).
That was what I expected.
What I don't understand is why my expectations are true for
/usr/local/guest.img but not for /dev/ada4.

Like mentioned, while reading the first 448 bytes on the host, I get
identical results from /usr/local/guest.img and /dev/ada4, but when
attaching /dev/ada4 to ahci-hd (-s 7,ahci-hd,/dev/ada4) and inspecting
inside vmm, all I see is 0x0, while ahci-hd attached
/usr/local/guest.img shows the same pmbr as on the host!?

Do I have to exclude /dev/ada4 on the host from geom? As soon as bhyve
opens /dev/ada4, all partitions vanish from the host ? probably ada4
itself gets blocked somehow?

Thanks for any hint in advance,

-harry