freebsd stable - Jan 2012 - about thumper aka sun fire x4500

I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2
Both fails when i engage > 10 disks, the system craches and messages :
"Hyper transport sync flood" will get into the BIOS errorlog ( but
nothing will
come to syslog since reboot is immediate)

Using a zfs radz of 25 disks and typing "zpool scrub" will bring the
system down in seconds.

Anyone using a x4500 that can comfirm that it works ? Or is this box broken ?
-- 
        Peter H?kanson   

        There's never money to do it right, but always money to do it
        again ... and again ... and again ... and again.
        ( Det ?r billigare att g?ra r?tt. Det ?r dyrt att laga fel. )

Hi, all,

Am 17.01.2012 um 18:59 schrieb peter h <peter@hk.ipsec.se>:
> I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2
> Both fails when i engage > 10 disks, the system craches and messages :
> "Hyper transport sync flood" will get into the BIOS errorlog (
but nothing will
> come to syslog since reboot is immediate)
> 
> Using a zfs radz of 25 disks and typing "zpool scrub" will bring
the system down in seconds.
> 
> Anyone using a x4500 that can comfirm that it works ? Or is this box broken
?
Well, I hate to write that, but ... does it work with the vendor supported [tm]
OS?
If yes, you can rule out a hardware defect. I would at least try Solaris for
this reason.
If no, the HW is broken and there is no need to look for a fault on
FreeBSD's side.

Kind regards,
Patrick

On Tue, Jan 17, 2012 at 06:59:08PM +0100, peter h wrote:
> I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2
> Both fails when i engage > 10 disks, the system craches and messages :
> "Hyper transport sync flood" will get into the BIOS errorlog (
but nothing will
> come to syslog since reboot is immediate)
> 
> Using a zfs radz of 25 disks and typing "zpool scrub" will bring
the system down in seconds.
> 
> Anyone using a x4500 that can comfirm that it works ? Or is this box broken
?
I do not have one of these boxes / am not familiar with them, but
HyperTransport is an AMD thing.  The concept is that it's a bus that
interconnects different pieces of a system to the CPU (and thus the
memory bus).  ASCII diagram coming up:

+-----------------------+
|         RAM           |
+----------+------------+
           |
+----------+------------+
|  CPU (w/ on-die MCH)  |
+----------+------------+
           |
+----------+------------+     +-----------------------------+
| HyperTransport bridge +-----+ PCI Express bus (VGA, etc.) |
+----------+------------+     +-----------------------------+
           |
+----------+---------------+
| Southbridge (SATA, etc.) |
+--------------------------+
           
ZFS is memory I/O intensive.  Your controller, given that it consists of
25 disks, is probably sitting on the PCI Express bus, and thus is
generating an equally high amount of I/O.

Given this above diagram, I'm sure you can figure out how
"flooding"
might occur.  :-)  I'm not sure what "sync flood" means (vs. I/O
flooding).

Googling turns up *tons* of examples of this on the web, except every
time they involve people doing overclocking or having CPU-level problems
pertaining to voltage.

There may be a BIOS option on your system to help curb this behaviour,
or at least try to limit it in some way.  I know on our AMD systems at
work the number of options in the Memory section of the BIOS is quite
large, many of which pertaining to interactivity with HyperTransport.

If you want my advice?  Bring the issue up to Sun.  They will almost
certainly be able to assign the case to an engineer, who although may
not be familiar with FreeBSD, hopefully WILL be familiar with the bus
interconnects described above and might be able to help you out.

-- 
| Jeremy Chadwick                                 jdc@parodius.com |
| Parodius Networking                     http://www.parodius.com/ |
| UNIX Systems Administrator                 Mountain View, CA, US |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

On Tue, 17 Jan 2012 18:59:08 +0100, peter h <peter@hk.ipsec.se> wrote:
> I have been beating on of these a few days, i have udes freebsd 9.0 and  
> 8.2
> Both fails when i engage > 10 disks, the system craches and messages :
> "Hyper transport sync flood" will get into the BIOS errorlog (
but
> nothing will
> come to syslog since reboot is immediate)
>
> Using a zfs radz of 25 disks and typing "zpool scrub" will bring
the
> system down in seconds.
>
> Anyone using a x4500 that can comfirm that it works ? Or is this box  
> broken ?
Does it work if you make 3 raid groups of 8 disks and 1 spare?

Ronald.