freebsd stable - Jul 2013 - [REGRESSION] Root zpool mounting broken between 06/30/2013 and 07/21/2013 when PS/2 support compiled into the kernel

I have a KERNCONF that previously had PS/2 support compiled into the kernel. If
I comment out the following lines like so:

# atkbdc0 controls both the keyboard and the PS/2 mouse
#device         atkbdc          # AT keyboard controller
#device         atkbd           # AT keyboard

then I'm able to mount root again (it was failing with ENOXDEV).

The working kernel was as follows:

$ strings /boot/kernel.WORKING/kernel | grep -B 2 -A 2 BAYONETTA
@(#)FreeBSD 9.1-STABLE #7 r+0304216: Sun Jun 30 15:22:55 PDT 2013
FreeBSD 9.1-STABLE #7 r+0304216: Sun Jun 30 15:22:55 PDT 2013
    gcooper at
bayonetta.local:/usr/obj/scratch/git/github/yaneurabeya-freebsd-stable-9/sys/BAYONETTA
gcc version 4.2.1 20070831 patched [FreeBSD]
FreeBSD
9.1-STABLE
BAYONETTA
$ cd /usr/src; git log 0304216
commit 03042167f73c213732b44218a24d8e1bbea00f8c
Merge: 2edcad2 974abfb
Author: Garrett Cooper <yanegomi at gmail.com>
Date:   Mon Jun 24 19:00:45 2013 -0700

    Merge remote-tracking branch 'upstream/stable/9' into stable/9

The working kernel [with atkbdc] was as follows:

FreeBSD bayonetta.local 9.2-BETA1 FreeBSD 9.2-BETA1 #12 r+c178034: Sun Jul 21
20:19:38 PDT 2013     root at
bayonetta.local:/usr/obj/scratch/git/github/yaneurabeya-freebsd-stable-9/sys/BAYONETTA
amd64
$ git log c178034
commit c17803445f4ffb97e1a46a1be5f7ea04692793f0
Author: avg <avg at FreeBSD.org>
Date:   Tue Jul 9 08:30:31 2013 +0000

    zfsboottest.sh: remove checks for things that are not strictly required
    
    MFC after:  10 days

(Yes, I had to backport some things because they are busted on stable/9 due to
other incomplete/missing MFCs).

I can test out patches, but I don't have time to bisect the actual commit
that caused the failure. That being said my intuition says it's this commit
should be looked at first:

commit 28f961058b0667841d7e9d8639bfd02ed8689faa
Author: jhb <jhb at FreeBSD.org>
Date:   Wed Jul 17 14:04:18 2013 +0000

    MFC 252576:
    Don't perform the acpi_DeviceIsPresent() check for PCI-PCI bridges.  If
    we are probing a PCI-PCI bridge it is because we found one by enumerating
    the devices on a PCI bus, so the bridge is definitely present.  A few
    BIOSes report incorrect status (_STA) for some bridges that claimed they
    were not present when in fact they were.
    
    While here, move this check earlier for Host-PCI bridges so attach fails
    before doing any work that needs to be torn down.
    
    PR:         kern/91594
    Approved by:        re (marius)

More data spew follows.

Thanks,
-Garrett

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: diagnostic-noise.txt
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20130722/2beeb4cf/attachment.txt>

On Monday, July 22, 2013 10:30:32 am Garrett Cooper
wrote:
> I have a KERNCONF that previously had PS/2 support compiled into the
kernel. If I comment out the following lines like so:
> 
> # atkbdc0 controls both the keyboard and the PS/2 mouse
> #device         atkbdc          # AT keyboard controller
> #device         atkbd           # AT keyboard
> 
> then I'm able to mount root again (it was failing with ENOXDEV).
> 
> The working kernel was as follows:
> 
> $ strings /boot/kernel.WORKING/kernel | grep -B 2 -A 2 BAYONETTA
> @(#)FreeBSD 9.1-STABLE #7 r+0304216: Sun Jun 30 15:22:55 PDT 2013
> FreeBSD 9.1-STABLE #7 r+0304216: Sun Jun 30 15:22:55 PDT 2013
>     gcooper at
bayonetta.local:/usr/obj/scratch/git/github/yaneurabeya-freebsd-stable-9/sys/BAYONETTA
> gcc version 4.2.1 20070831 patched [FreeBSD]
> FreeBSD
> 9.1-STABLE
> BAYONETTA
> $ cd /usr/src; git log 0304216
> commit 03042167f73c213732b44218a24d8e1bbea00f8c
> Merge: 2edcad2 974abfb
> Author: Garrett Cooper <yanegomi at gmail.com>
> Date:   Mon Jun 24 19:00:45 2013 -0700
> 
>     Merge remote-tracking branch 'upstream/stable/9' into stable/9
> 
> The working kernel [with atkbdc] was as follows:
> 
> FreeBSD bayonetta.local 9.2-BETA1 FreeBSD 9.2-BETA1 #12 r+c178034: Sun Jul
21 20:19:38 PDT 2013
root at
bayonetta.local:/usr/obj/scratch/git/github/yaneurabeya-freebsd-stable-9/sys/BAYONETTA
amd64
> $ git log c178034
> commit c17803445f4ffb97e1a46a1be5f7ea04692793f0
> Author: avg <avg at FreeBSD.org>
> Date:   Tue Jul 9 08:30:31 2013 +0000
> 
>     zfsboottest.sh: remove checks for things that are not strictly required
>     
>     MFC after:  10 days
> 
> (Yes, I had to backport some things because they are busted on stable/9 due
to other incomplete/missing MFCs).
> 
> I can test out patches, but I don't have time to bisect the actual
commit that caused the failure. That being said my intuition says it's this
commit should be looked at first:
> 
> commit 28f961058b0667841d7e9d8639bfd02ed8689faa
> Author: jhb <jhb at FreeBSD.org>
> Date:   Wed Jul 17 14:04:18 2013 +0000
> 
>     MFC 252576:
>     Don't perform the acpi_DeviceIsPresent() check for PCI-PCI bridges.
If
>     we are probing a PCI-PCI bridge it is because we found one by
enumerating
>     the devices on a PCI bus, so the bridge is definitely present.  A few
>     BIOSes report incorrect status (_STA) for some bridges that claimed
they
>     were not present when in fact they were.
>     
>     While here, move this check earlier for Host-PCI bridges so attach
fails
>     before doing any work that needs to be torn down.
>     
>     PR:         kern/91594
>     Approved by:        re (marius)
I strongly doubt that this is related.  It would be most helpful if you could
obtain a dmesg from the new kernel however (perhaps via a serial console) to
rule it out.  All you would need to see is if the new kernel sees more
"pcib"
devices than the old one to see if this change even has an effect on your
system.

-- 
John Baldwin