freebsd stable - Jun 2013 - Weird I/O hangs (9.1R, arcsas, interrupt spikes on uhci0)

Hi,

very periodically, we see I/O hangs for about 10 seconds, roughly once per
minute.

Each time this happens, the I/O rate simply drops to zero, and all disk access
hangs; this is also very noticeable on the shell, for NFS clients etc.
Everything else (networking, kernel, ?) seems to continue normally.

Environment: FreeBSD 9.1R GENERIC on amd64, using ZFS, on a ARC1320 PCIe with
24x Seagate ST33000650SS (3rd party arcsas.ko driver).

It's easy to observe these hangs under write load, e.g. with 'zpool
iostat 1':

void        22.4T  42.6T     34  2.73K  1.07M   293M
void        22.4T  42.6T     20  2.74K   623K   289M
void        22.4T  42.6T    144  2.62K  4.83M   279M
void        22.4T  42.6T     13  2.60K   437K   283M
void        22.4T  42.6T      0      0      0      0 <-- hang starts
void        22.4T  42.6T      0      0      0      0
void        22.4T  42.6T      0      0      0      0
void        22.4T  42.6T      0      0      0      0
void        22.4T  42.6T      0      0      0      0
void        22.4T  42.6T      0      0      0      0
void        22.4T  42.6T      0      0      0      0
void        22.4T  42.6T      0      0      0      0
void        22.4T  42.6T      0    296  4.00K  34.2M <-- hang ends
void        22.4T  42.6T      2  2.64K  73.8K   288M
void        22.4T  42.6T      8  3.12K   278K   329M

Each time this happens, there is a completely unexplained spike of interrupts on
uhci0: 'systat -vm' then displays numbers around 270k.

# vmstat -i | grep -E '(arcsas|uhci0|Total)'
irq16: uhci0                  1227020890      67708
irq24: arcsas0                  12045211        664
Total                         1266417827      69882

Things to note:

- Booting an USB-less kernel or disabling all USB in the BIOS doesn't change
a thing (no interrupt spikes to be seen, but the hangs remain)
- The hangs / interrupt spikes happen just as often when the system is idle
- Board is a Supermicro x8dth
- There's two igb cards
- Root is ZFS as well (separate pool though)
- BIOS, Areca FW and driver already are latest versions
- Putting the controller to a different slot doesn't change the behaviour
- We have two identical systems and both show the exact same symptoms, so flaky
hardware is probably not the issue

Any ideas would be appreciated.

Thanks,
D.

On Wed, 19 Jun 2013 15:01:14 +0200, Dennis K?gel <dk at neveragain.de>
wrote:
> Hi,
>
> very periodically, we see I/O hangs for about 10 seconds, roughly once  
> per minute.
>
> Each time this happens, the I/O rate simply drops to zero, and all disk  
> access hangs; this is also very noticeable on the shell, for NFS clients  
> etc. Everything else (networking, kernel, ?) seems to continue normally.
>
> Environment: FreeBSD 9.1R GENERIC on amd64, using ZFS, on a ARC1320 PCIe  
> with 24x Seagate ST33000650SS (3rd party arcsas.ko driver).
>
> It's easy to observe these hangs under write load, e.g. with 'zpool
> iostat 1':
>
> void        22.4T  42.6T     34  2.73K  1.07M   293M
> void        22.4T  42.6T     20  2.74K   623K   289M
> void        22.4T  42.6T    144  2.62K  4.83M   279M
> void        22.4T  42.6T     13  2.60K   437K   283M
> void        22.4T  42.6T      0      0      0      0 <-- hang starts
> void        22.4T  42.6T      0      0      0      0
> void        22.4T  42.6T      0      0      0      0
> void        22.4T  42.6T      0      0      0      0
> void        22.4T  42.6T      0      0      0      0
> void        22.4T  42.6T      0      0      0      0
> void        22.4T  42.6T      0      0      0      0
> void        22.4T  42.6T      0      0      0      0
> void        22.4T  42.6T      0    296  4.00K  34.2M <-- hang ends
> void        22.4T  42.6T      2  2.64K  73.8K   288M
> void        22.4T  42.6T      8  3.12K   278K   329M
>
> Each time this happens, there is a completely unexplained spike of  
> interrupts on uhci0: 'systat -vm' then displays numbers around
270k.
>
> # vmstat -i | grep -E '(arcsas|uhci0|Total)'
> irq16: uhci0                  1227020890      67708
> irq24: arcsas0                  12045211        664
> Total                         1266417827      69882
>
> Things to note:
>
> - Booting an USB-less kernel or disabling all USB in the BIOS doesn't  
> change a thing (no interrupt spikes to be seen, but the hangs remain)
> - The hangs / interrupt spikes happen just as often when the system is  
> idle
> - Board is a Supermicro x8dth
> - There's two igb cards
> - Root is ZFS as well (separate pool though)
> - BIOS, Areca FW and driver already are latest versions
> - Putting the controller to a different slot doesn't change the
behaviour
> - We have two identical systems and both show the exact same symptoms,  
> so flaky hardware is probably not the issue
>
> Any ideas would be appreciated.
>
> Thanks,
> D.
First send more information about the system:
- The content of /var/run/dmesg.boot.
- Install /usr/ports/sysutils/zfs-stats and send the output of zfs-stats  
-a.
- Send the output of zpool status + zpool list.
- Did you configure compression or dedup on the pool?
- Do you keep a lot of snapshots?
- Do you run a cronjob every minute which does something with the pool?  
Gathers statistics or something like that.

Ronald.