zfs discuss - Oct 2009 - Terrible ZFS performance on a Dell 1850 w/ PERC 4e/Si (Sol10U6)

I''ve got a mail machine here that I built using ZFS boot/root. 
It''s been having some major I/O performance problems, which I posted
once before... but that post seems to have disappeared.

Now I''ve managed to obtain another identical machine, and I''ve
built it in the same way as the original.  Running Solaris 10 U6, I''ve
got it fully patched as of 2009/10/06.  It''s using a mirrored disk via
the PERC (LSI Megaraid) controller.

The main problem seems to be ZFS.  If I do the following on a UFS filesystem:

 # /usr/bin/time dd if=/dev/zero of=whee.bin bs=1024000 count=<x>

... then I get "real" times of the following:

 x     time
128  35. 4
256  1:01.8
512  2:19.8

It''s all very linear and fairly decent.  However, if I then destroy
that filesystem and recreate it using ZFS (no special options or kernel
variables set) performance degrades substantially.  With the same dd, I get:

x      time
128  3:45.3
256  6:52.7
512  15:40.4


So basically a 6.5x loss across the board.  I realize that a simple
''dd'' is an extremely weak test, but real-world use on these
machines shows similar problems... long delays logging in, and running a command
that isn''t cached can take 20-30 seconds (even something as simple as
''psrinfo -vp'').

Ironically, the machine works just fine for simple email, because the files are
small and very transient and thus can exist quite easily just in memory.  But
more complex things, like a local copy of our mailmaps, cripples the machine.

I''m about to rebuild the machine with the RAID controller in
passthrough mode, and I''ll see what that accomplishes.  Most of the
machines here are Linux and use the hardware RAID1, so I was/am hesitant to
"break standard" that way.  Does anyone have any experience or
suggestions for trying to make ZFS boot+root work fine on this machine?
-- 
This message posted from opensolaris.org

On 09 October, 2009 - Brandon Hume sent me these 2,0K bytes:
> I''ve got a mail machine here that I built using ZFS boot/root. 
It''s
> been having some major I/O performance problems, which I posted once
> before... but that post seems to have disappeared.
> 
> Now I''ve managed to obtain another identical machine, and
I''ve built
> it in the same way as the original.  Running Solaris 10 U6, I''ve
got
> it fully patched as of 2009/10/06.  It''s using a mirrored disk via
the
> PERC (LSI Megaraid) controller.
> 
> The main problem seems to be ZFS.  If I do the following on a UFS
filesystem:
> 
>  # /usr/bin/time dd if=/dev/zero of=whee.bin bs=1024000 count=<x>
> 
> ... then I get "real" times of the following:
> 
>  x     time
> 128  35. 4
> 256  1:01.8
> 512  2:19.8
Is this minutes:seconds.millisecs ? if so, you''re looking at 3-4MB/s ..
I would say something is wrong.
> It''s all very linear and fairly decent.
Decent?!
> However, if I then destroy that filesystem and recreate it using ZFS
> (no special options or kernel variables set) performance degrades
> substantially.  With the same dd, I get:
> 
> x      time
> 128  3:45.3
> 256  6:52.7
> 512  15:40.4
0.5MB/s .. that''s floppy speed :P
> So basically a 6.5x loss across the board.  I realize that a simple
> ''dd'' is an extremely weak test, but real-world use on
these machines
> shows similar problems... long delays logging in, and running a
> command that isn''t cached can take 20-30 seconds (even something
as
> simple as ''psrinfo -vp'').
> 
> Ironically, the machine works just fine for simple email, because the
> files are small and very transient and thus can exist quite easily
> just in memory.  But more complex things, like a local copy of our
> mailmaps, cripples the machine.
.. because something is messed up, and for some reason ZFS seems to feel
worse than UFS..
> I''m about to rebuild the machine with the RAID controller in
> passthrough mode, and I''ll see what that accomplishes.  Most of
the
> machines here are Linux and use the hardware RAID1, so I was/am
> hesitant to "break standard" that way.  Does anyone have any
> experience or suggestions for trying to make ZFS boot+root work fine
> on this machine?
Check for instance ''iostat -xnzmp 1''  while doing this and see
if any
disk is behaving badly, high service times etc.. Even your speedy
3-4MB/s is nowhere close to what you should be getting, unless you''ve
connected a bunch of floppy drives to your PERC..

/Tomas
-- 
Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Ume?
`- Sysadmin at {cs,acc}.umu.se

I think the raid card is a re-branded LSI SCSI raid. I have LSI 21320-4x and
having same problem with ZFS.

Do you have BBU on the card? You may want to disable cache flush and zil and see
how it works. I tried passthrough and basically the result is same.

I gave up on tuning this card with ZFS, replaced the raid card with plain 21320
SCSI hba and everything works well since. I used ZFS mirror for OS, raidz for
data pool. 2 days ago, one disk in my rpool mirror was dead and I was able to
easily replace it. A few weeks ago, when I ran scrub, zfs found checksum error
on the now dead disk and was able to fix it.

I don''t buy the idea on hardware raid with ZFS anymore, let ZFS handle
it for you.
-- 
This message posted from opensolaris.org

Before you do a dd test try first to do:

echo zfs_vdev_max_pending/W0t1 | mdb -kw

and let us know if it helped or not.

iostat -xnz 1

output while you are doing dd would also help.

-- 
Robert Milkowski
http://milek.blogspot.com

> Is this minutes:seconds.millisecs ? if so, you''re looking at
3-4MB/s ..
> I would say something is wrong.
Ack, you''re right.

I was concentrating so much on the WTFOMG problem that I completely missed the
WTF problem.

In other news, with the Poweredge put into "SCSI" mode instead of
"RAID" mode (which is in the system setup, NOT the Megaraid setup,
which is why I missed it before) and addressing the disks directly, I can do
that same 512M write in 9 seconds.

I''m going to rebuild the box with ZFS boot/root and see how it behaves.

I''m also going to ask the Linux admins and see if they''re
seeing problems.  It''s possible that nobody''s noticed. (I tend
to run the high I/O tasked machines, the rest are typically CPU-bound...)
-- 
This message posted from opensolaris.org

> Before you do a dd test try first to do:
> echo zfs_vdev_max_pending/W0t1 | mdb -kw
I did actually try this about a month ago when I first made an attempt at
figuring this out.  Changing the pending values did make some small difference,
but even the best was far, far short of acceptable performance.

In other news, switching to zfs boot+root, in a ZFS-mirrored configuration, in
passthrough mode, has made a huge difference.  Writing the 512M file took 15
minutes before... I''m down to twelve seconds.
-- 
This message posted from opensolaris.org

I have the same problem you do, ZFS performance under Solaris 10 u8 is horrible.

When you say passthrough mode, do you mean non-RAID configuration?
And if so, could you tell me how you configured it?

The best I can manage is to configure each physical drive as a RAID 0 array then
export that as a logical drive.

All tips/suggestions are appreciated.
-- 
This message posted from opensolaris.org