freebsd stable - Sep 2011 - System freeze: Adaptec (aac) timeouts (releng 8)

Cheers,

we have a reproducible system freeze due to Adaptec driver (aac) timeouts:

Sep  3 05:26:44 foo kernel: aac0: COMMAND 0xffffff80005ae4c0 (TYPE 502) TIMEOUT
AFTER 129 SECONDS
Sep  3 05:26:44 foo kernel: aac0: COMMAND 0xffffff80005ac0e0 (TYPE 502) TIMEOUT
AFTER 129 SECONDS
Sep  3 05:26:44 foo kernel: aac0: COMMAND 0xffffff80005b0fa0 (TYPE 502) TIMEOUT
AFTER 129 SECONDS
<dozens more of these...>

Once this happens, the userland seems to be alive, but the controller is
completely dead. As soon as the disk subsystem is involved, any process
hangs forever (e.g. SSH crypto-exchange still happens, but a shell won't
even start anymore).

We observe the same issue on two systems of (mostly) identical spec, so
it's not a hardware issue.

Apparently this only happens under heavy disk i/o and high cpu load.
Notably high write throughput plus a 'zpool scrub' on a large
GELI-backed zpool usually triggers the problem after a few hours.
Without high activity, they run smooth for weeks.

Both systems are amd64 with an Adaptec 5805 controller and 16 disks (of
which two form a RAID-1 system volume (UFS), and the remaining 14 serve
as JBOD for a large zpool -- a total of 15 "aacd" devices).

Both were running 8.2R originally. I've taken them to 8-STABLE now and
also applied svn r222951 (where the MFC was forgotten, it seems), but
the problem remains.

Any help is greatly appreciated.

Thanks,
- D.

I was having the exact same problem using an Adaptec 52445.  After
downloading and using the latest driver from the adaptec website, the
problems stopped.  I haven't had a single freeze since using the new
code.  The newest driver from the website has source code with it, so
it shouldn't be that big of a deal to incorporate it into the base
system.    I emailed the authors of the aac driver (Mike Smith and
Scott Long), but they have both retired.  So I'm not really sure how
to get this code into the base.  If anyone knows, please take up the
charge.

On Wed, Sep 14, 2011 at 2:08 AM, Dennis Koegel <dk@neveragain.de>
wrote:
> Cheers,
>
> we have a reproducible system freeze due to Adaptec driver (aac) timeouts:
>
> Sep ?3 05:26:44 foo kernel: aac0: COMMAND 0xffffff80005ae4c0 (TYPE 502)
TIMEOUT AFTER 129 SECONDS
> Sep ?3 05:26:44 foo kernel: aac0: COMMAND 0xffffff80005ac0e0 (TYPE 502)
TIMEOUT AFTER 129 SECONDS
> Sep ?3 05:26:44 foo kernel: aac0: COMMAND 0xffffff80005b0fa0 (TYPE 502)
TIMEOUT AFTER 129 SECONDS
> <dozens more of these...>
>
> Once this happens, the userland seems to be alive, but the controller is
> completely dead. As soon as the disk subsystem is involved, any process
> hangs forever (e.g. SSH crypto-exchange still happens, but a shell
won't
> even start anymore).
>
> We observe the same issue on two systems of (mostly) identical spec, so
> it's not a hardware issue.
>
> Apparently this only happens under heavy disk i/o and high cpu load.
> Notably high write throughput plus a 'zpool scrub' on a large
> GELI-backed zpool usually triggers the problem after a few hours.
> Without high activity, they run smooth for weeks.
>
> Both systems are amd64 with an Adaptec 5805 controller and 16 disks (of
> which two form a RAID-1 system volume (UFS), and the remaining 14 serve
> as JBOD for a large zpool -- a total of 15 "aacd" devices).
>
> Both were running 8.2R originally. I've taken them to 8-STABLE now and
> also applied svn r222951 (where the MFC was forgotten, it seems), but
> the problem remains.
>
> Any help is greatly appreciated.
>
> Thanks,
> - D.
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
"freebsd-stable-unsubscribe@freebsd.org"
>