zfs discuss - Aug 2010 - ZFS p[erformance drop with new Xeon 55xx and 56xx cpus

Greetings,

I am seeing some unexplained performance drop using the above cpus,
using a fairly up-to-date build ( late 145).
Basically, the system seems to be 98% idle, spending most if its time in this
stack:

              unix`i86_mwait+0xd
              unix`cpu_idle_mwait+0xf1
              unix`idle+0x114
              unix`thread_start+0x8
           455645

Most cpus seem to be idling most of the time, sitting on the mwait instruction.
No lock contention, not waiting on io, I am finding myself at a loss explaining
what this system is doing.
(I am monitoring the system w. lockstat, mpstat, prstat).  Despite the
predominantly idle system,
I see some latency reported by prstat microstate accounting on the zfs threads.
 
This is a fairly beefy box, 24G memory,  16 cpus.
Doing a local zfs send | receive, should be getting at least 100MB+, 
and I am only getting  5-10MB.  
I see some Intel errata on the 55xx series xeons, a problem with the 
monitor/mwait instructions, that could conceivably cause missed wake-up or
mis-reported  mwait status.

Anybody else is seeing this?
-- 
This message posted from opensolaris.org

On 08/12/10 04:16, Steve Gonczi wrote:
> Greetings,
>
> I am seeing some unexplained performance drop using the above cpus,
> using a fairly up-to-date build ( late 145).
> Basically, the system seems to be 98% idle, spending most if its time in
this stack:
>
>                unix`i86_mwait+0xd
>                unix`cpu_idle_mwait+0xf1
>                unix`idle+0x114
>                unix`thread_start+0x8
>             455645
>
> Most cpus seem to be idling most of the time, sitting on the mwait
instruction.
> No lock contention, not waiting on io, I am finding myself at a loss
explaining what this system is doing.
> (I am monitoring the system w. lockstat, mpstat, prstat).  Despite the
predominantly idle system,
> I see some latency reported by prstat microstate accounting on the zfs
threads.
>
> This is a fairly beefy box, 24G memory,  16 cpus.
> Doing a local zfs send | receive, should be getting at least 100MB+,
> and I am only getting  5-10MB.
> I see some Intel errata on the 55xx series xeons, a problem with the
> monitor/mwait instructions, that could conceivably cause missed wake-up or
mis-reported  mwait status.
I''d suggest you supply a bit more information (to the list, not to me,
I
don''t know very much about zfs internals):

- zpool/zfs configuration
- history of this issue: has it been like this since you installed the 
machine?
   - if no: what changes were introduced around the time you saw this first?
- does this happen on a busy machine too?
- describe your test in more detail
- provide measurements (lockstat, iostat, maybe some DTrace) before and 
during test, add some timestamps so people can correlate data to events.
- anything else you can think of that might be relevant.

HTH
Michael