[Lustre-discuss] aacraid kernel panic caused failover

Thomas Roth
Wed, 06 Apr 2011 08:05:07 -0000

We have ~ 60 servers with these Adaptec controllers, and found this problem just
to happen from time to time.
Upgrade of the aacraid module wouldn''t help. We had contacts to
Adaptec, but they had no clue either.
Only good thing is it seems that this adapter panic happens in an instant,
halting the machine, but has no prior phase of degradation: the controller
doesn''t start leaving out every second bit or just writing the
''1''s and not the ''0''s or ... - so whatever
data has made it to the disks before the
crash seems to be quite sensible. Reboot and never buy Adaptec again.

Cheers,
Thomas

On 04/06/2011 07:03 AM, David Noriega wrote:
> Ok I updated the aacraid driver and the raid firmware, yet I still had
> the problem happen, so I did more research and applied the following
> tweaks:
>
> 1) Rebuilt mkinitrd with the following options:
> a) edit /etc/sysconfig/mkinitrid/multipath to contain MULTIPATH=yes
> b) mkinitrid initrd-2.6.18-194.3.1.el5_lustre.1.8.4.img
> 2.6.18-194.3.1.el5_lustre.1.8.4 --preload=scsi_dh_rdac
> 2) Added the local hard disk to the multipath black list
> 3) Edited modprobe.conf to have the following aacraid options:
> options aacraid firmware_debug=2 startup_timeout=60 #the debug
doesn''t
> seem to print anything to dmesg
> 4) Added pcie_aspm=off to the kernel boot options
>
> So things looked good for a while. I did have a problem mounting the
> lustre partitions but this was my fault in misconfiguring some lnet
> options I was experimenting with. I fixed that and just as a test, I
> ran ''modprobe lustre'' since I wasn''t ready to
fail back the partitions
> just yet(wanted to wait till when activity was the lowest). That was
> earlier today. I was about to fail back tonight, yet when I checked
> the server again I saw in dmesg the same aacraid problems from before.
> Is it possible lustre is interfering with aacraid? Its weird since I
> do have a duplicate machine and its not having any of thise problems.
>
> On Fri, Mar 25, 2011 at 9:55 AM, Temple  Jason <jtemple at cscs.ch>
wrote:
>> Adaptec should have the firmware and drivers on their site for your
card.  If not adaptec, then SOracle will have it available somewhere.
>>
>> The firmware and system drivers usually have a utility that will check
the current version and upgrade it for you.
>>
>> Hope this helps (I use different cards, so I can''t tell you
exactly).
>>
>> -Jason
>>
>> -----Original Message-----
>> From: David Noriega [mailto:tsk133 at my.utsa.edu]
>> Sent: venerd?, 25. marzo 2011 15:47
>> To: Temple Jason
>> Subject: Re: [Lustre-discuss] aacraid kernel panic caused failover
>>
>> Hmm not sure, whats the best way to find out?
>>
>> On Fri, Mar 25, 2011 at 9:46 AM, Temple  Jason <jtemple at
cscs.ch> wrote:
>>> Hi,
>>>
>>> Are you using the latest firmware?  This sort of thing used to
happen to me, but with different raid cards.
>>>
>>> -Jason
>>>
>>> -----Original Message-----
>>> From: lustre-discuss-bounces at lists.lustre.org
[mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of David Noriega
>>> Sent: venerd?, 25. marzo 2011 15:38
>>> To: lustre-discuss at lists.lustre.org
>>> Subject: [Lustre-discuss] aacraid kernel panic caused failover
>>>
>>> Had some crazyness happen to our lustre system. We have two OSSs,
both
>>> identical sun x4140 servers and on only one of them have
I''ve seen
>>> this pop up in the kernel messages and then a kernel panic. The
panic
>>> seemed to then spread and caused the network to go down and the
second
>>> OSS to try to failover(or failback?). Anyways
''splitbrain'' occurred
>>> and I was able to get in and set them straight. I researched this
>>> aacraid module messages and so far all I can find says to increase
the
>>> timeout, but these are old messages and currently they are set to
60.
>>> Anyone else have any ideas?
>>>
>>> aacraid: Host adapter abort request (0,0,0,0)
>>> aacraid: Host adapter reset request. SCSI hang ?
>>> AAC: Host adapter BLINK LED 0xef
>>> AAC0: adapter kernel panic''d ef.
>>>
>>> --
>>> Personally, I liked the university. They gave us money and
facilities,
>>> we didn''t have to produce anything! You''ve never
been out of college!
>>> You don''t know what it''s like out there!
I''ve worked in the private
>>> sector. They expect results. -Ray Ghostbusters
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>>
>>
>>
>>
>> --
>> Personally, I liked the university. They gave us money and facilities,
>> we didn''t have to produce anything! You''ve never been
out of college!
>> You don''t know what it''s like out there!
I''ve worked in the private
>> sector. They expect results. -Ray Ghostbusters
>>
>
>
>

--
--------------------------------------------------------------------
Thomas Roth
Department: Informationstechnologie
Location: SB3 1.262
Phone: +49-6159-71 1453  Fax: +49-6159-71 2986

GSI Helmholtzzentrum f?r Schwerionenforschung GmbH
Planckstra?e 1
64291 Darmstadt
www.gsi.de

Gesellschaft mit beschr?nkter Haftung
Sitz der Gesellschaft: Darmstadt
Handelsregister: Amtsgericht Darmstadt, HRB 1528

Gesch?ftsf?hrung: Professor Dr. Dr. h.c. Horst St?cker,
Dr. Hartmut Eickhoff

Vorsitzende des Aufsichtsrates: Dr. Beatrix Vierkorn-Rudolph
Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt