freebsd stable - Jan 2014 - regression: msk0 watchdog timeout and interrupt storm

I'm getting an interrupt storm from mskc running with the latest
if_msk.c code.  The OS is built from source (259540):

FreeBSD 10.0-PRERELEASE (GENERIC) #0 r259540: Sat Dec 21 00:05:39 EST 2013

While not the latest, the point is that sys/dev/msk is up to date wrt
stable_9 and also wrt head.

The odd thing is that the machine seemed to run fine for a day or two
and then started exhibiting this behaviour and has become useless.

This is now highly reproducible (it happens within seconds when trying
to do a long file transfer between two machines with GbE) so if there
is anything I can do to instrument this, please make suggestions.

What I know so far is:

  1.  When the watchdog occurs, Y2_IS_STAT_BMU is set in the prior
      interrupt mask.

  2.  This would put us in from msk_intr into msk_handle_events, with
      msk_handle_events returning 0.

  3.  msk_handle_events reads in sc->msk_stat_cons.  The last recorded
      value of sc->msk_stat_cons is alway 1024.

  4.  The only way to exit msk_handle_events with sc->msk_stat_cons
      greater than zero yet not do anything is hit the top of loop
      conditional and fall out:

      sd = &sc->msk_stat_ring[cons];
      control = le32toh(sd->msk_control);
      if ((control & HW_OWNER) == 0)
          break;

  5.  The code after the loop can return zero if the ring buffer
      pointer hasn't moved.  That code is:

      sc->msk_stat_cons = cons;
      bus_dmamap_sync(sc->msk_stat_tag, sc->msk_stat_map,
          BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE);

      if (rxput[MSK_PORT_A] > 0)
              msk_rxput(sc->msk_if[MSK_PORT_A]);
      if (rxput[MSK_PORT_B] > 0)
              msk_rxput(sc->msk_if[MSK_PORT_B]);

      return (sc->msk_stat_cons != CSR_READ_2(sc, STAT_PUT_IDX));

  6.  If the return value is zero, the interrupt isn't cleared.  That
      was suspect.  The code in msk_intr is:

      domore = msk_handle_events(sc);
      if ((status & Y2_IS_STAT_BMU) != 0 && domore == 0)
              CSR_WRITE_4(sc, STAT_CTRL, SC_STAT_CLR_IRQ);

  7.  This code before the return in msk_handle_events should force
      the clear but doesn't fix anything.

      if ((control & HW_OWNER) == 0)
              return;

This looks like some sort of fall off the end of a ring buffer type of
problem (since it always points to entry 0x400) but since I haven't
done driver work in ages, that is mostly just a wild guess and I
really have no idea yet at to what is going wrong.

Also please keep me on the Cc since I'm not subscribed to the list,
though I will check the archives from time to time.

Thanks,

Curtis


reference:
http://lists.freebsd.org/pipermail/freebsd-stable/2013-November/075699.html

Replying to self (and top posting).

I'm not sure if the problem is fixed or masked.

The symptom (watchdog and interrupt storm) has gone away with the
following change in if_mskreg.h:

@@ -2329,8 +2329,13 @@
  */
 #if (BUS_SPACE_MAXADDR > 0xFFFFFFFF)
 #define        MSK_64BIT_DMA
+#if 1
+#define MSK_TX_RING_CNT                256
+#define MSK_RX_RING_CNT                256
+#else
 #define MSK_TX_RING_CNT                384
 #define MSK_RX_RING_CNT                512
+#endif
 #else
 #undef MSK_64BIT_DMA
 #define MSK_TX_RING_CNT                256

This backs out a very small part of the change made to if_mskreg.h in
revision 227582.

The following is what I think is affected by this change:

	count = imin(4096, roundup2(count, 1024));
	sc->msk_stat_count = count;
	stat_sz = count * sizeof(struct msk_stat_desc);

The change makes count end up being 1024 (and stat_sz 8192).

For me the problem is fixed/masked but I would also consider putting
the increase to MSK_TX_RING_CNT and MSK_RX_RING_CNT back and forcing
count above to be no greater than 1024 if that would help someone else
debug the problem.  I'm not sure where the 4096 came from but
replacing that with 1024 is equivalent to "count = 1024" with no math
involved.

This does seem to me like a regression in 10.0 caused by the change to
if_mskreg.h (Nov 16).  The workaround so far has been fine for me.

Curtis


In message <201401010153.s011rNcm082703 at maildrop2.v6ds.occnc.com>
Curtis Villamizar writes:
>  
> I'm getting an interrupt storm from mskc running with the latest
> if_msk.c code.  The OS is built from source (259540):
>  
> FreeBSD 10.0-PRERELEASE (GENERIC) #0 r259540: Sat Dec 21 00:05:39 EST 2013
>  
> While not the latest, the point is that sys/dev/msk is up to date wrt
> stable_9 and also wrt head.
>  
> The odd thing is that the machine seemed to run fine for a day or two
> and then started exhibiting this behaviour and has become useless.
>  
> This is now highly reproducible (it happens within seconds when trying
> to do a long file transfer between two machines with GbE) so if there
> is anything I can do to instrument this, please make suggestions.
>  
> What I know so far is:
>  
>   1.  When the watchdog occurs, Y2_IS_STAT_BMU is set in the prior
>       interrupt mask.
>  
>   2.  This would put us in from msk_intr into msk_handle_events, with
>       msk_handle_events returning 0.
>  
>   3.  msk_handle_events reads in sc->msk_stat_cons.  The last recorded
>       value of sc->msk_stat_cons is alway 1024.
>  
>   4.  The only way to exit msk_handle_events with sc->msk_stat_cons
>       greater than zero yet not do anything is hit the top of loop
>       conditional and fall out:
>  
>       sd = &sc->msk_stat_ring[cons];
>       control = le32toh(sd->msk_control);
>       if ((control & HW_OWNER) == 0)
>           break;
>  
>   5.  The code after the loop can return zero if the ring buffer
>       pointer hasn't moved.  That code is:
>  
>       sc->msk_stat_cons = cons;
>       bus_dmamap_sync(sc->msk_stat_tag, sc->msk_stat_map,
>           BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE);
>  
>       if (rxput[MSK_PORT_A] > 0)
>               msk_rxput(sc->msk_if[MSK_PORT_A]);
>       if (rxput[MSK_PORT_B] > 0)
>               msk_rxput(sc->msk_if[MSK_PORT_B]);
>  
>       return (sc->msk_stat_cons != CSR_READ_2(sc, STAT_PUT_IDX));
>  
>   6.  If the return value is zero, the interrupt isn't cleared.  That
>       was suspect.  The code in msk_intr is:
>  
>       domore = msk_handle_events(sc);
>       if ((status & Y2_IS_STAT_BMU) != 0 && domore == 0)
>               CSR_WRITE_4(sc, STAT_CTRL, SC_STAT_CLR_IRQ);
>  
>   7.  This code before the return in msk_handle_events should force
>       the clear but doesn't fix anything.
>  
>       if ((control & HW_OWNER) == 0)
>               return;
>  
> This looks like some sort of fall off the end of a ring buffer type of
> problem (since it always points to entry 0x400) but since I haven't
> done driver work in ages, that is mostly just a wild guess and I
> really have no idea yet at to what is going wrong.
>  
> Also please keep me on the Cc since I'm not subscribed to the list,
> though I will check the archives from time to time.
>  
> Thanks,
>  
> Curtis
>  
>  
> reference:
> http://lists.freebsd.org/pipermail/freebsd-stable/2013-November/075699.html