zfs discuss - Jan 2009 - X4500, snv_101a, hd and zfs

I''ve got a fresh install of 101a on a thumper with 48 disks and zfs
with one large 46 drive raidz2 pool. It has no load at the moment. My problem is
that the SUNWhd tools are excrutiatingly slow, by excrutiating I mean that the
command : "/opt/SUNWhd/hd/bin/hdadm write_cache display all" takes
three hours to iterate over the 48 drives which, I suspect, is not the expected
time for such a command to take.

I''ve hunted around and, while I''ve seen a few pointers to
certain thumpers in a SN range having performance issues, this thumper is not in
that range, nor is it running 10u5 or u6.

I''ve come up empty for an explanation so I thought I''d ask
here since I know there are some issues with zfs and caching, e.g. the ARC
cache, but I''m unclear as to which are currently solved in 101a and
which aren''t. It could be a simple configuration issue, but
I''m unfamiliar with the hardware and would welcome any suggestions or
pointers.
-- 
This message posted from opensolaris.org

Elaine,

Very bizarre problem you''re having. I have no problems on either of my
x4500s. One on 10u6 and one on indiana snv_101b_rc2.

Time on Sol 10u6:
- Just running hd
real    0m25.599s
user    0m0.021s
sys     0m2.376s

Time on OS snv_101b_rc2
- Just running hd
real    0m27.565s
user    0m0.050s
sys     0m1.184s

Only running indiana for the up to date cifs server which does ACLs  
properly. I haven''t done anything but install hd, I did nothing to  
''make them work''.

Regards,
---
Cooper Ry Lees
A boring old UNIX Administrator - Information Management Services (IMS)
Australian Nuclear Science and Technology Organisation

Important: This transmission is intended only for the use of the  
addressee. It is confidential and may contain privileged information  
or copyright material. If you are not the intended recipient, any use  
or further disclosure of this communication is strictly forbidden. If  
you have received this transmission in error, please notify me  
immediately by telephone and delete all copies of this transmission as  
well as any attachments.

On 06/01/2009, at 12:43 PM, Elaine Ashton wrote:
> I''ve got a fresh install of 101a on a thumper with 48 disks and
zfs
> with one large 46 drive raidz2 pool. It has no load at the moment.  
> My problem is that the SUNWhd tools are excrutiatingly slow, by  
> excrutiating I mean that the command : "/opt/SUNWhd/hd/bin/hdadm  
> write_cache display all" takes three hours to iterate over the 48  
> drives which, I suspect, is not the expected time for such a command  
> to take.
>
> I''ve hunted around and, while I''ve seen a few pointers to
certain
> thumpers in a SN range having performance issues, this thumper is  
> not in that range, nor is it running 10u5 or u6.
>
> I''ve come up empty for an explanation so I thought I''d
ask here
> since I know there are some issues with zfs and caching, e.g. the  
> ARC cache, but I''m unclear as to which are currently solved in
101a
> and which aren''t. It could be a simple configuration issue, but
I''m
> unfamiliar with the hardware and would welcome any suggestions or  
> pointers.
> -- 
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090106/b6ad3509/attachment.html>

On Jan 5, 2009, at 9:33 PM, LEES, Cooper wrote:
> Elaine,
>
> Very bizarre problem you''re having. I have no problems on either
of
> my x4500s. One on 10u6 and one on indiana snv_101b_rc2.
I agree, which is why I was hoping someone might know what the deal is.

Just a straight ''hd'' takes over a minute and a half. The real
killer
is "/opt/SUNWhd/hd/bin/hdadm write_cache display all" which displays  
all the write_cache states for each drive. This takes hours. How long  
does that take on your 101b system? I swear, something must be  
terribly amiss with this box, but I''m just not sure where to start  
looking.

e.

On Mon, Jan 5, 2009 at 6:55 PM, Elaine Ashton <Elaine.Ashton at sun.com>
wrote:
>
> On Jan 5, 2009, at 9:33 PM, LEES, Cooper wrote:
>
>> Elaine,
>>
>> Very bizarre problem you''re having. I have no problems on
either of
>> my x4500s. One on 10u6 and one on indiana snv_101b_rc2.
>
> I agree, which is why I was hoping someone might know what the deal is.
>
> Just a straight ''hd'' takes over a minute and a half. The
real killer
> is "/opt/SUNWhd/hd/bin/hdadm write_cache display all" which
displays
> all the write_cache states for each drive. This takes hours. How long
> does that take on your 101b system? I swear, something must be
> terribly amiss with this box, but I''m just not sure where to start
> looking.
>
> e.
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
I''d suggest opening a case with Sun.... but you ARE Sun  ;p
The ''hd'' tools don''t even work on the
X4540''s, and even the ILOM
webgui doesn''t show the drive as even being installed (yet, I had 48,
1TB drives all working fine).
So, at least you''re able to see your drives... sorta.
I -wish- I could see my drives cache status, state, FRU, etc...    :(

-- 
Brent Jones
brent at servuhome.net

On Mon, Jan 05, 2009 at 05:43:23PM -0800, Elaine Ashton
wrote:
> I''ve got a fresh install of 101a on a thumper with 48 disks and
zfs with one large 46 drive raidz2 pool. It has no load at the moment. My
problem is that the SUNWhd tools are excrutiatingly slow, by excrutiating I mean
that the command : "/opt/SUNWhd/hd/bin/hdadm write_cache display all"
takes three hours to iterate over the 48 drives which, I suspect, is not the
expected time for such a command to take.
SunOS isis 5.10 Generic_138889-02 i86pc i386 i86pc
aka S10u6 with latest patches:

+ time -p hdadm display all
...
platform = Sun Fire X4500
...
real 9.90
user 0.04
sys 0.36

+ time -p sh /opt/SUNWhd/hd/bin/write_cache display all
...
real 104.67
user 14.89
sys 6.57

Regards,
jel.
-- 
Otto-von-Guericke University     http://www.cs.uni-magdeburg.de/
Department of Computer Science   Geb. 29 R 027, Universitaetsplatz 2
39106 Magdeburg, Germany         Tel: +49 391 67 12768

Ok, it gets a bit more specific....

hdadm and write_cache run ''format -e -d $disk'' ....

On this system, format will produce the list of devices in short order - format
-e, however, takes much, much longer and would explain why it takes hours to
iterate over 48 drives.

It''s very curious and I''m not sure at this point if
it''s related to ZFS.
-- 
This message posted from opensolaris.org