[Lustre-discuss] aacraid kernel panic caused failover

David Noriega
Wed, 06 Apr 2011 14:58:19 -0000

Our adaptec raid card is a Sun StorageTek RAID INT card, made by intel
of all people. So I installed the raid manager software, which of
course doesn''t say anything is wrong, but it does come with a
monitoring daemon and it printed this message after the last aacraid
kernel panic:

Sun StorageTek RAID Manager Agent: [203] The battery-backup cache
device needs a new battery: controller 1.

So could that be the problem?

On Wed, Apr 6, 2011 at 7:52 AM, Jeff Johnson
<jeff.johnson at aeoncomputing.com> wrote:
> I have seen similar behavior on these controllers. On dissimilar configs
and different aged systems. These happened to be non-Lustre standalone nfs and
iscsi target boxes.
>
> Went through controller and drive firmware upgrades, low-level fw dumps
?and analysis from dev engineers.
>
> In the end it was never really explained or resolved. It appears that these
controllers, like small children, have tantrums and fall apart. A power cycle
clears the condition.
>
> Not the best controller for an OSS.
>
> --Jeff
>
> ---mobile signature---
> Jeff Johnson - Aeon Computing
> jeff.johnson at aeoncomputing.com
>
>
> On Apr 6, 2011, at 1:05, Thomas Roth <t.roth at gsi.de> wrote:
>
>> We have ~ 60 servers with these Adaptec controllers, and found this
problem just to happen from time to time.
>> Upgrade of the aacraid module wouldn''t help. We had contacts
to Adaptec, but they had no clue either.
>> Only good thing is it seems that this adapter panic happens in an
instant, halting the machine, but has no prior phase of degradation: the
controller
>> doesn''t start leaving out every second bit or just writing the
''1''s and not the ''0''s or ... - so whatever
data has made it to the disks before the
>> crash seems to be quite sensible. Reboot and never buy Adaptec again.
>>
>> Cheers,
>> Thomas
>>
>> On 04/06/2011 07:03 AM, David Noriega wrote:
>>> Ok I updated the aacraid driver and the raid firmware, yet I still
had
>>> the problem happen, so I did more research and applied the
following
>>> tweaks:
>>>
>>> 1) Rebuilt mkinitrd with the following options:
>>> a) edit /etc/sysconfig/mkinitrid/multipath to contain MULTIPATH=yes
>>> b) mkinitrid initrd-2.6.18-194.3.1.el5_lustre.1.8.4.img
>>> 2.6.18-194.3.1.el5_lustre.1.8.4 --preload=scsi_dh_rdac
>>> 2) Added the local hard disk to the multipath black list
>>> 3) Edited modprobe.conf to have the following aacraid options:
>>> options aacraid firmware_debug=2 startup_timeout=60 #the debug
doesn''t
>>> seem to print anything to dmesg
>>> 4) Added pcie_aspm=off to the kernel boot options
>>>
>>> So things looked good for a while. I did have a problem mounting
the
>>> lustre partitions but this was my fault in misconfiguring some lnet
>>> options I was experimenting with. I fixed that and just as a test,
I
>>> ran ''modprobe lustre'' since I wasn''t
ready to fail back the partitions
>>> just yet(wanted to wait till when activity was the lowest). That
was
>>> earlier today. I was about to fail back tonight, yet when I checked
>>> the server again I saw in dmesg the same aacraid problems from
before.
>>> Is it possible lustre is interfering with aacraid? Its weird since
I
>>> do have a duplicate machine and its not having any of thise
problems.
>>>
>>> On Fri, Mar 25, 2011 at 9:55 AM, Temple ?Jason <jtemple at
cscs.ch> wrote:
>>>> Adaptec should have the firmware and drivers on their site for
your card. ?If not adaptec, then SOracle will have it available somewhere.
>>>>
>>>> The firmware and system drivers usually have a utility that
will check the current version and upgrade it for you.
>>>>
>>>> Hope this helps (I use different cards, so I can''t
tell you exactly).
>>>>
>>>> -Jason
>>>>
>>>> -----Original Message-----
>>>> From: David Noriega [mailto:tsk133 at my.utsa.edu]
>>>> Sent: venerd?, 25. marzo 2011 15:47
>>>> To: Temple Jason
>>>> Subject: Re: [Lustre-discuss] aacraid kernel panic caused
failover
>>>>
>>>> Hmm not sure, whats the best way to find out?
>>>>
>>>> On Fri, Mar 25, 2011 at 9:46 AM, Temple ?Jason <jtemple at
cscs.ch> wrote:
>>>>> Hi,
>>>>>
>>>>> Are you using the latest firmware? ?This sort of thing used
to happen to me, but with different raid cards.
>>>>>
>>>>> -Jason
>>>>>
>>>>> -----Original Message-----
>>>>> From: lustre-discuss-bounces at lists.lustre.org
[mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of David Noriega
>>>>> Sent: venerd?, 25. marzo 2011 15:38
>>>>> To: lustre-discuss at lists.lustre.org
>>>>> Subject: [Lustre-discuss] aacraid kernel panic caused
failover
>>>>>
>>>>> Had some crazyness happen to our lustre system. We have two
OSSs, both
>>>>> identical sun x4140 servers and on only one of them have
I''ve seen
>>>>> this pop up in the kernel messages and then a kernel panic.
The panic
>>>>> seemed to then spread and caused the network to go down and
the second
>>>>> OSS to try to failover(or failback?). Anyways
''splitbrain'' occurred
>>>>> and I was able to get in and set them straight. I
researched this
>>>>> aacraid module messages and so far all I can find says to
increase the
>>>>> timeout, but these are old messages and currently they are
set to 60.
>>>>> Anyone else have any ideas?
>>>>>
>>>>> aacraid: Host adapter abort request (0,0,0,0)
>>>>> aacraid: Host adapter reset request. SCSI hang ?
>>>>> AAC: Host adapter BLINK LED 0xef
>>>>> AAC0: adapter kernel panic''d ef.
>>>>>
>>>>> --
>>>>> Personally, I liked the university. They gave us money and
facilities,
>>>>> we didn''t have to produce anything!
You''ve never been out of college!
>>>>> You don''t know what it''s like out there!
I''ve worked in the private
>>>>> sector. They expect results. -Ray Ghostbusters
>>>>> _______________________________________________
>>>>> Lustre-discuss mailing list
>>>>> Lustre-discuss at lists.lustre.org
>>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Personally, I liked the university. They gave us money and
facilities,
>>>> we didn''t have to produce anything! You''ve
never been out of college!
>>>> You don''t know what it''s like out there!
I''ve worked in the private
>>>> sector. They expect results. -Ray Ghostbusters
>>>>
>>>
>>>
>>>
>>
>> --
>> --------------------------------------------------------------------
>> Thomas Roth
>> Department: Informationstechnologie
>> Location: SB3 1.262
>> Phone: +49-6159-71 1453 ?Fax: +49-6159-71 2986
>>
>> GSI Helmholtzzentrum f?r Schwerionenforschung GmbH
>> Planckstra?e 1
>> 64291 Darmstadt
>> www.gsi.de
>>
>> Gesellschaft mit beschr?nkter Haftung
>> Sitz der Gesellschaft: Darmstadt
>> Handelsregister: Amtsgericht Darmstadt, HRB 1528
>>
>> Gesch?ftsf?hrung: Professor Dr. Dr. h.c. Horst St?cker,
>> Dr. Hartmut Eickhoff
>>
>> Vorsitzende des Aufsichtsrates: Dr. Beatrix Vierkorn-Rudolph
>> Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss