freebsd stable - Mar 2017 - moutnroot failing on zpools in Azure after upgrade from 10 to 11 due to lack of waiting for da0

I have a number of machines in Azure, all booting from ZFS and, until
the weekend, running 10.3 perfectly happily.

I started upgrading these to 11. The first went fine, the second would
not boot. Looking at the boot diagnistics it is having problems finding the
root pool to mount. I see this is the diagnostic output:

	storvsc0: <Hyper-V IDE Storage Interface> on vmbus0
	Solaris: NOTICE: Cannot find the pool label for 'rpool'
	Mounting from zfs:rpool/ROOT/default failed with error 5.
	Root mount waiting for: storvsc
	(probe0:blkvsc0:0:storvsc1: 0:<Hyper-V IDE Storage Interface>0):  on
vmbus0
	storvsc scsi_status = 2
	(da0:blkvsc0:0:0:0): UNMAPPED
	(probe1:blkvsc1:0:1:0): storvsc scsi_status = 2
	hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
	da0 at blkvsc0 bus 0 scbus2 target 0 lun 0

As you can see, the drive da0 only appears after it has tried, and failed,
to mount the root pool.

Normally I would just stick in a big 'vfs.mountroot.timeout' but that
variable doesnt not appear to exist under 11 - or at least it doesnt
show up in sysctl.

I have one machine which boots fine. I can take the drive of this machine,
clone it, and attach to a new VM, and that VM fails to boot! Am now
a bit scared to reboot that virtual machine in case it doesnt come back.

Can anyone offer any suggestions ? Just being able to delay the
mount might be enough if there is a variable which can do that. I do
rather need to get these machines back online....

thanks,

-pete.

One extra datapoint - the machines which do not fail are the small DS1_v2
instances. These seem to boot fine, but if I move to the DS2 size then the
problem shows up.

-pete.

Pete French <petefrench at ingresso.co.uk> wrote:
> I have a number of machines in Azure, all booting from ZFS and, until
> the weekend, running 10.3 perfectly happily.
> 
> I started upgrading these to 11. The first went fine, the second would
> not boot. Looking at the boot diagnistics it is having problems finding
> the root pool to mount. I see this is the diagnostic output:
> 
> 	storvsc0: <Hyper-V IDE Storage Interface> on vmbus0
> 	Solaris: NOTICE: Cannot find the pool label for 'rpool'
> 	Mounting from zfs:rpool/ROOT/default failed with error 5.
> 	Root mount waiting for: storvsc
> 	(probe0:blkvsc0:0:storvsc1: 0:<Hyper-V IDE Storage
> Interface>0):  on vmbus0 storvsc scsi_status = 2
> 	(da0:blkvsc0:0:0:0): UNMAPPED
> 	(probe1:blkvsc1:0:1:0): storvsc scsi_status = 2
> 	hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
> 	da0 at blkvsc0 bus 0 scbus2 target 0 lun 0
> 
> As you can see, the drive da0 only appears after it has tried, and
> failed, to mount the root pool.
> 
> Normally I would just stick in a big 'vfs.mountroot.timeout' but
that
> variable doesnt not appear to exist under 11 - or at least it doesnt
> show up in sysctl.
The variable still exists but is ignored when using ZFS.

It's a known issue. You could try this patch:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=208882#c3

Manually specifying the root pool should workaround the issue.

sysctl(8) does not show the variable as it's only a tunable.
This is unrelated to the update.

Fabian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20170313/2852257a/attachment.sig>

On 0313T1206, Pete French wrote:
> I have a number of machines in Azure, all booting from ZFS and, until
> the weekend, running 10.3 perfectly happily.
> 
> I started upgrading these to 11. The first went fine, the second would
> not boot. Looking at the boot diagnistics it is having problems finding the
> root pool to mount. I see this is the diagnostic output:
> 
> 	storvsc0: <Hyper-V IDE Storage Interface> on vmbus0
> 	Solaris: NOTICE: Cannot find the pool label for 'rpool'
> 	Mounting from zfs:rpool/ROOT/default failed with error 5.
> 	Root mount waiting for: storvsc
> 	(probe0:blkvsc0:0:storvsc1: 0:<Hyper-V IDE Storage Interface>0):  on
vmbus0
> 	storvsc scsi_status = 2
> 	(da0:blkvsc0:0:0:0): UNMAPPED
> 	(probe1:blkvsc1:0:1:0): storvsc scsi_status = 2
> 	hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
> 	da0 at blkvsc0 bus 0 scbus2 target 0 lun 0
> 
> As you can see, the drive da0 only appears after it has tried, and failed,
> to mount the root pool.
Are you sure the above transcript is right?  There are three reasons
I'm asking.  First, you'll see the "Root mount waiting"
message,
which means the root mount code is, well, waiting for storvsc, exactly
as expected.  Second - there is no "Trying to mount root".  But most
of all - for some reason the "Mounting failed" is shown _before_ the
"Root mount waiting", and I have no idea how this could ever happen.

On 0313T1206, Pete French wrote:
> I have a number of machines in Azure, all booting from ZFS and, until
> the weekend, running 10.3 perfectly happily.
> 
> I started upgrading these to 11. The first went fine, the second would
> not boot. Looking at the boot diagnistics it is having problems finding the
> root pool to mount. I see this is the diagnostic output:
> 
> 	storvsc0: <Hyper-V IDE Storage Interface> on vmbus0
> 	Solaris: NOTICE: Cannot find the pool label for 'rpool'
> 	Mounting from zfs:rpool/ROOT/default failed with error 5.
> 	Root mount waiting for: storvsc
> 	(probe0:blkvsc0:0:storvsc1: 0:<Hyper-V IDE Storage Interface>0):  on
vmbus0
> 	storvsc scsi_status = 2
> 	(da0:blkvsc0:0:0:0): UNMAPPED
> 	(probe1:blkvsc1:0:1:0): storvsc scsi_status = 2
> 	hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
> 	da0 at blkvsc0 bus 0 scbus2 target 0 lun 0
> 
> As you can see, the drive da0 only appears after it has tried, and failed,
> to mount the root pool.
Does the same problem still happen with recent 11-STABLE?