freebsd stable - Mar 2017 - moutnroot failing on zpools in Azure after upgrade from 10 to 11 due to lack of waiting for da0

On 0313T1206, Pete French wrote:
> I have a number of machines in Azure, all booting from ZFS and, until
> the weekend, running 10.3 perfectly happily.
> 
> I started upgrading these to 11. The first went fine, the second would
> not boot. Looking at the boot diagnistics it is having problems finding the
> root pool to mount. I see this is the diagnostic output:
> 
> 	storvsc0: <Hyper-V IDE Storage Interface> on vmbus0
> 	Solaris: NOTICE: Cannot find the pool label for 'rpool'
> 	Mounting from zfs:rpool/ROOT/default failed with error 5.
> 	Root mount waiting for: storvsc
> 	(probe0:blkvsc0:0:storvsc1: 0:<Hyper-V IDE Storage Interface>0):  on
vmbus0
> 	storvsc scsi_status = 2
> 	(da0:blkvsc0:0:0:0): UNMAPPED
> 	(probe1:blkvsc1:0:1:0): storvsc scsi_status = 2
> 	hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
> 	da0 at blkvsc0 bus 0 scbus2 target 0 lun 0
> 
> As you can see, the drive da0 only appears after it has tried, and failed,
> to mount the root pool.
Are you sure the above transcript is right?  There are three reasons
I'm asking.  First, you'll see the "Root mount waiting"
message,
which means the root mount code is, well, waiting for storvsc, exactly
as expected.  Second - there is no "Trying to mount root".  But most
of all - for some reason the "Mounting failed" is shown _before_ the
"Root mount waiting", and I have no idea how this could ever happen.

> Are you sure the above transcript is right?  There are three reasons
> I'm asking.  First, you'll see the "Root mount waiting"
message,
> which means the root mount code is, well, waiting for storvsc, exactly
> as expected.  Second - there is no "Trying to mount root".  But
most
> of all - for some reason the "Mounting failed" is shown _before_
the
> "Root mount waiting", and I have no idea how this could ever
happen.
OK, that's interesting, and kind of worrying! I belive, it's correct - I
have put the full trascript up here for you so you can see all of it:

https://www.twisted.org.uk/~pete/914893a3-249e-4a91-851c-f467fc185eec.txt

I am assuming that Azure's capturing of the outut is correct....

-pete.

On 13/03/2017 21:07, Edward Tomasz Napiera?a wrote:
> Are you sure the above transcript is right?  There are three reasons
> I'm asking.  First, you'll see the "Root mount waiting"
message,
> which means the root mount code is, well, waiting for storvsc, exactly
> as expected.  Second - there is no "Trying to mount root".  But
most
> of all - for some reason the "Mounting failed" is shown _before_
the
> "Root mount waiting", and I have no idea how this could ever
happen.
Edward, your observation is not completely correct.
https://www.twisted.org.uk/~pete/914893a3-249e-4a91-851c-f467fc185eec.txt

We have:

Trying to mount root from zfs:rpool/ROOT/default []... <==vmbus0: version 3.0
...
storvsc0: <Hyper-V IDE Storage Interface> on vmbus0
Solaris: NOTICE: Cannot find the pool label for 'rpool'
Mounting from zfs:rpool/ROOT/default failed with error 5. <==Root mount
waiting for: storvsc <==...

So, the kernel attempted to mount the root even before vmbus was attached and,
thus, before storvsc appeared and informed the kernel that it might be holding
the root.
How ZFS was supposed to know that vmbus is ever going to appear?
To me this sounds more like a problem with the Hyper-V drivers.


-- 
Andriy Gapon