Paul Guyot
2012-Mar-14 22:45 UTC
Changes brought to bce(4) disabling ipmi access during boot
Hello, Changes brought to bce(4) prevents booting a R410 Dell server with GELI-encrypted root ZFS partition requiring a passphrase, something that was possible with 9-RELEASE. Using a binary search, the bug comes from the following revision: Updating collection src-all/cvs Edit src/sys/dev/bce/if_bce.c Add delta 1.89.2.4 2012.01.09.19.07.14 yongari Edit src/sys/dev/bce/if_bcereg.h Add delta 1.35.2.3 2012.01.09.19.07.14 yongari Shutting down connection to server RELEASE as well as STABLE with date=2012.01.09.19.00.00 boot properly. The boot fails with date=2012.01.09.19.08.00 For more details: the box is configured to boot from a plain ZFS pool that contains the kernel (zboot) and then to request passphrase for a GELI-encrypted ZFS pool containing everything else (including /etc/rc.d), in a way similar to what is described here: http://www.keltia.net/howtos/freebsd-dedibox The passphrase should be entered from the virtual console (KVM) simulated by the ipmi controller (through Dell's "iDRAC6"). On RELEASE, the boot works properly and can be followed from the KVM console, where the passphrase can be entered. On STABLE, the KVM gets disconnected. Besides, the ipmi is down, and the box is eventually bricked: since plugging a real console is not an option, the only way to get access to the server is to reboot it electrically (and to configure the PXE to perform a netboot in order to switch the kernel). I believe the ipmi controller uses the main ethernet port to simulate a physical console and the change in the bce driver disables the ethernet port. Since the box waits from the passphrase to configure the network, the box gets unreachable. Paul -- Semiocast http://semiocast.com/ +33.183627948 - 20 rue Lacaze, 75014 Paris
YongHyeon PYUN
2012-Mar-15 01:10 UTC
Changes brought to bce(4) disabling ipmi access during boot
On Wed, Mar 14, 2012 at 11:44:37PM +0100, Paul Guyot wrote:> Hello, > > Changes brought to bce(4) prevents booting a R410 Dell server with GELI-encrypted root ZFS partition requiring a passphrase, something that was possible with 9-RELEASE. > > Using a binary search, the bug comes from the following revision: > > Updating collection src-all/cvs > Edit src/sys/dev/bce/if_bce.c > Add delta 1.89.2.4 2012.01.09.19.07.14 yongari > Edit src/sys/dev/bce/if_bcereg.h > Add delta 1.35.2.3 2012.01.09.19.07.14 yongari > Shutting down connection to server >Could you try attach patch and let me know whether it recovers IPMI functionality?> RELEASE as well as STABLE with date=2012.01.09.19.00.00 boot properly. > The boot fails with date=2012.01.09.19.08.00 > > For more details: the box is configured to boot from a plain ZFS pool that contains the kernel (zboot) and then to request passphrase for a GELI-encrypted ZFS pool containing everything else (including /etc/rc.d), in a way similar to what is described here: http://www.keltia.net/howtos/freebsd-dedibox > > The passphrase should be entered from the virtual console (KVM) simulated by the ipmi controller (through Dell's "iDRAC6"). > > On RELEASE, the boot works properly and can be followed from the KVM console, where the passphrase can be entered. On STABLE, the KVM gets disconnected. Besides, the ipmi is down, and the box is eventually bricked: since plugging a real console is not an option, the only way to get access to the server is to reboot it electrically (and to configure the PXE to perform a netboot in order to switch the kernel). > > I believe the ipmi controller uses the main ethernet port to simulate a physical console and the change in the bce driver disables the ethernet port. Since the box waits from the passphrase to configure the network, the box gets unreachable. > > Paul > -- > Semiocast http://semiocast.com/ > +33.183627948 - 20 rue Lacaze, 75014 Paris >-------------- next part -------------- A non-text attachment was scrubbed... Name: bce.ipmi.diff Type: text/x-diff Size: 426 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20120315/48999d19/bce.ipmi.bin
Paul Guyot
2012-Mar-15 08:19 UTC
Changes brought to bce(4) disabling ipmi access during boot
Le 15 mars 2012 ? 18:10, YongHyeon PYUN a ?crit :> On Wed, Mar 14, 2012 at 11:44:37PM +0100, Paul Guyot wrote: >> Hello, >> >> Changes brought to bce(4) prevents booting a R410 Dell server with GELI-encrypted root ZFS partition requiring a passphrase, something that was possible with 9-RELEASE. >> >> Using a binary search, the bug comes from the following revision: >> >> Updating collection src-all/cvs >> Edit src/sys/dev/bce/if_bce.c >> Add delta 1.89.2.4 2012.01.09.19.07.14 yongari >> Edit src/sys/dev/bce/if_bcereg.h >> Add delta 1.35.2.3 2012.01.09.19.07.14 yongari >> Shutting down connection to server >> > > Could you try attach patch and let me know whether it recovers IPMI > functionality?Thank you for your quick patch. Unfortunately, it does not recover IPMI functionality with STABLE@2012.01.09.19.08.00. Paul -- Semiocast http://semiocast.com/ +33.183627948 - 20 rue Lacaze, 75014 Paris
YongHyeon PYUN
2012-Mar-16 01:07 UTC
Changes brought to bce(4) disabling ipmi access during boot
On Thu, Mar 15, 2012 at 09:19:27AM +0100, Paul Guyot wrote:> Le 15 mars 2012 ? 18:10, YongHyeon PYUN a ?crit : > > > On Wed, Mar 14, 2012 at 11:44:37PM +0100, Paul Guyot wrote: > >> Hello, > >> > >> Changes brought to bce(4) prevents booting a R410 Dell server with GELI-encrypted root ZFS partition requiring a passphrase, something that was possible with 9-RELEASE. > >> > >> Using a binary search, the bug comes from the following revision: > >> > >> Updating collection src-all/cvs > >> Edit src/sys/dev/bce/if_bce.c > >> Add delta 1.89.2.4 2012.01.09.19.07.14 yongari > >> Edit src/sys/dev/bce/if_bcereg.h > >> Add delta 1.35.2.3 2012.01.09.19.07.14 yongari > >> Shutting down connection to server > >> > > > > Could you try attach patch and let me know whether it recovers IPMI > > functionality? > > Thank you for your quick patch. Unfortunately, it does not recover IPMI functionality with STABLE@2012.01.09.19.08.00. >Hmm, how about this one? -------------- next part -------------- Index: sys/dev/bce/if_bce.c ==================================================================--- sys/dev/bce/if_bce.c (revision 232950) +++ sys/dev/bce/if_bce.c (working copy) @@ -1992,8 +1992,7 @@ ifp = sc->bce_ifp; mii = device_get_softc(sc->bce_miibus); - if (mii == NULL || ifp == NULL || - (ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) + if (mii == NULL || ifp == NULL) return; sc->bce_link_up = FALSE; @@ -2038,9 +2037,6 @@ } } - if (sc->bce_link_up == FALSE) - return; - /* Set half or full duplex based on PHY settings. */ if ((mii->mii_media_active & IFM_GMASK) == IFM_HDX) { DBPRINT(sc, BCE_INFO_PHY,
Paul Guyot
2012-Mar-16 22:42 UTC
Changes brought to bce(4) disabling ipmi access during boot
Le 16 mars 2012 ? 18:06, YongHyeon PYUN a ?crit :> On Thu, Mar 15, 2012 at 09:19:27AM +0100, Paul Guyot wrote: >> Le 15 mars 2012 ? 18:10, YongHyeon PYUN a ?crit : >> >>> On Wed, Mar 14, 2012 at 11:44:37PM +0100, Paul Guyot wrote: >>>> Hello, >>>> >>>> Changes brought to bce(4) prevents booting a R410 Dell server with GELI-encrypted root ZFS partition requiring a passphrase, something that was possible with 9-RELEASE. >>>> >>>> Using a binary search, the bug comes from the following revision: >>>> >>>> Updating collection src-all/cvs >>>> Edit src/sys/dev/bce/if_bce.c >>>> Add delta 1.89.2.4 2012.01.09.19.07.14 yongari >>>> Edit src/sys/dev/bce/if_bcereg.h >>>> Add delta 1.35.2.3 2012.01.09.19.07.14 yongari >>>> Shutting down connection to server >>>> >>> >>> Could you try attach patch and let me know whether it recovers IPMI >>> functionality? >> >> Thank you for your quick patch. Unfortunately, it does not recover IPMI functionality with STABLE@2012.01.09.19.08.00. >> > > Hmm, how about this one?It did not work either. So I patched the original (RELEASE) driver to print information about the various conditions newly tested by the STABLE driver in bce_miibus_statchg. The result is the following. The box has two bce interfaces, the one connected is bce0. The loader was configured with boot_verbose. Before the passphrase is entered: bce0: <Broadcom NetXtreme II BCM5716 1000Base-T (C0)> mem 0xda000000-0xdbffffff irq 36 at device 0.0 on pci1 bce0: attempting to allocate 1 MSI vectors (16 supported) bce0: using IRQ 256 for MSI miibus0: <MII bus> on bce0 bce0: bpf attached bce0: Ethernet address: 78:2b:cb:18:22:75 bce0: [1998] ifp != NULL bce0: [2000] (ifp->if_drv_flags & IFF_DRV_RUNNING) == 0 bce0: [2008] mii != NULL bce0: [2023] (mii->mii_media_status & IFM_ACTIVE) != IFM_ACTIVE) bce0: [2026] (mii->mii_media_status & IFM_AVALID) == IFM_AVALID) bce0: [2058] Unknown link speed, enabling default GMII interface. bce0: [2082] Disabling RX flow control. bce0: [2095] Disabling TX flow control. bce0: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.11) bce1: <Broadcom NetXtreme II BCM5716 1000Base-T (C0)> mem 0xdc000000-0xddffffff irq 48 at device 0.1 on pci1 bce1: attempting to allocate 1 MSI vectors (16 supported) bce1: using IRQ 257 for MSI miibus1: <MII bus> on bce1 bce1: bpf attached bce1: Ethernet address: 78:2b:cb:18:22:76 bce1: [1998] ifp != NULL bce1: [2000] (ifp->if_drv_flags & IFF_DRV_RUNNING) == 0 bce1: [2008] mii != NULL bce1: [2023] (mii->mii_media_status & IFM_ACTIVE) != IFM_ACTIVE) bce1: [2026] (mii->mii_media_status & IFM_AVALID) == IFM_AVALID) bce1: [2058] Unknown link speed, enabling default GMII interface. bce1: [2082] Disabling RX flow control. bce1: [2095] Disabling TX flow control. bce1: ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (5.2.3); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.11) After the passphrase is entered and network is started: bce0: [1998] ifp != NULL bce0: [2000] (ifp->if_drv_flags & IFF_DRV_RUNNING) == 0 bce0: [2008] mii != NULL bce0: [2023] (mii->mii_media_status & IFM_ACTIVE) != IFM_ACTIVE) bce0: [2026] (mii->mii_media_status & IFM_AVALID) == IFM_AVALID) bce0: [2058] Unknown link speed, enabling default GMII interface. bce0: [2082] Disabling RX flow control. bce0: [2095] Disabling TX flow control. bce0: [1998] ifp != NULL bce0: [2002] (ifp->if_drv_flags & IFF_DRV_RUNNING) != 0 bce0: [2008] mii != NULL bce0: [2018] (mii->mii_media_status & (IFM_ACTIVE | IFM_AVALID)) == (IFM_ACTIVE | IFM_AVALID) bce0: [2053] Enabling GMII interface. bce0: [2082] Disabling RX flow control. bce0: [2095] Disabling TX flow control. bce0: link state changed to UP bce0: Gigabit link up! bce0: Gigabit link up! bce0: Gigabit link up! From what I understand, both new conditions that may return early are true ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0 and later (mii->mii_media_status & IFM_ACTIVE) != IFM_ACTIVE), which yields bce_link_up to be FALSE. Yet I am confused by the role of actually writing to BCE_EMAC_MODE in order to keep the iDRAC link up, and wether the issue would not come from another part of the change. Paul