zfs discuss - Jan 2010 - Large scale ZFS deployments out there (>200 disks)

While thinking about ZFS as the next generation filesystem without limits I am
wondering if the real world is ready for this kind of incredible technology ...

I''m actually speaking of hardware :)

ZFS can handle a lot of devices. Once in the import bug
(http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6761786) is fixed it
should be able to handle a lot of disks.

I want to ask the ZFS community and users what large scale deploments are out
there.  How man disks ? How much capacity ? Single pool or many pools on a
server ? How does resilver work in those environtments ? How to you backup ?
What is the experience so far ? Major headakes ? 

It would be great if large scale users would share their setups and experiences
with ZFS.

Will you ? :)
Thanks, 
Robert
-- 
This message posted from opensolaris.org

On 01/28/10 11:13 PM, Lutz Schumann wrote:
> While thinking about ZFS as the next generation filesystem without
> limits I am wondering if the real world is ready for this kind of
> incredible technology ...
>
> I''m actually speaking of hardware :)
>
> ZFS can handle a lot of devices. Once in the import bug
> (http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6761786)
> is fixed it should be able to handle a lot of disks.
That was fixed in build 125.
> I want to ask the ZFS community and users what large scale deploments
> are out there.  How man disks ? How much capacity ? Single pool or
> many pools on a server ? How does resilver work in those
> environtments ? How to you backup ? What is the experience so far ?
> Major headakes ?
>
> It would be great if large scale users would share their setups and
> experiences with ZFS.
The largest ZFS deployment that we have is currently comprised of 22 
Dell MD1000 enclosures (330 750 GB Nearline SAS disks). We have 3 head 
nodes and use one zpool per node, comprised of rather narrow (5+2) 
RAIDZ2 vdevs. This setup is exclusively used for storing backup data.

Resilver times could be better - I am sure that this will improve once 
we upgrade from S10u9 to 2010.03.

One of the things that I am missing in ZFS is the ability to prioritize 
background operations like scrub and resilver. All our disks are idle 
during daytime and I would love to be able to take advantage of this, 
especially during resilver operations.

This setup has been running for about a year with no major issues so 
far. The only hickups we''ve had were all HW related (no fun in firmware
upgrading 200+ disks).
> Will you ? :) Thanks, Robert

-- 
Med venlig hilsen / Best Regards

Henrik Johansen
henrik at scannet.dk
Tlf. 75 53 35 00

ScanNet Group
A/S ScanNet

+------------------------------------------------------------------------------
| On 2010-01-29 10:36:29, Richard Elling wrote:
| 
| Nit: Solaris 10 u9 is 10/03 or 10/04 or 10/05, depending on what you read.
| Solaris 10 u8 is 11/09.

Nit: S10u8 is 10/09.
 
| Scrub I/O is given the lowest priority and is throttled. However, I am not
| sure that the throttle is in Solaris 10, because that source is not publicly
| available. In general, you will not notice a resource cap until the system
| utilization is high enough that the cap is effective.  In other words, if the
| system is mostly idle, the scrub consumes the bulk of the resources.

Solaris 10 has the scrub throttle; it affects resilver too.. I have often
wished I could turn it off. I''m happy to take a major application
performance
hit during a resilver (especially of a mirror or raidz1) in almost every case.
-- 
bda
cyberpunk is dead. long live cyberpunk.

On Jan 29, 2010, at 12:45 AM, Henrik Johansen wrote:
> On 01/28/10 11:13 PM, Lutz Schumann wrote:
>> While thinking about ZFS as the next generation filesystem without
>> limits I am wondering if the real world is ready for this kind of
>> incredible technology ...
>> 
>> I''m actually speaking of hardware :)
>> 
>> ZFS can handle a lot of devices. Once in the import bug
>> (http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6761786)
>> is fixed it should be able to handle a lot of disks.
> 
> That was fixed in build 125.
> 
>> I want to ask the ZFS community and users what large scale deploments
>> are out there.  How man disks ? How much capacity ? Single pool or
>> many pools on a server ? How does resilver work in those
>> environtments ? How to you backup ? What is the experience so far ?
>> Major headakes ?
>> 
>> It would be great if large scale users would share their setups and
>> experiences with ZFS.
> 
> The largest ZFS deployment that we have is currently comprised of 22 Dell
MD1000 enclosures (330 750 GB Nearline SAS disks). We have 3 head nodes and use
one zpool per node, comprised of rather narrow (5+2) RAIDZ2 vdevs. This setup is
exclusively used for storing backup data.
This is an interesting design.  It looks like a good use of hardware and
redundancy
for backup storage. Would you be able to share more of the details? :-)
> Resilver times could be better - I am sure that this will improve once we
upgrade from S10u9 to 2010.03.
Nit: Solaris 10 u9 is 10/03 or 10/04 or 10/05, depending on what you read.
Solaris 10 u8 is 11/09.
> One of the things that I am missing in ZFS is the ability to prioritize
background operations like scrub and resilver. All our disks are idle during
daytime and I would love to be able to take advantage of this, especially during
resilver operations.
Scrub I/O is given the lowest priority and is throttled. However, I am not
sure that the throttle is in Solaris 10, because that source is not publicly
available. In general, you will not notice a resource cap until the system
utilization is high enough that the cap is effective.  In other words, if the
system is mostly idle, the scrub consumes the bulk of the resources.
> This setup has been running for about a year with no major issues so far.
The only hickups we''ve had were all HW related (no fun in firmware
upgrading 200+ disks).
ugh.
 -- richard

On 01/29/10 07:36 PM, Richard Elling wrote:
> On Jan 29, 2010, at 12:45 AM, Henrik Johansen wrote:
>> On 01/28/10 11:13 PM, Lutz Schumann wrote:
>>> While thinking about ZFS as the next generation filesystem
>>> without limits I am wondering if the real world is ready for this
>>> kind of incredible technology ...
>>>
>>> I''m actually speaking of hardware :)
>>>
>>> ZFS can handle a lot of devices. Once in the import bug
>>>
(http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6761786)
>>>
>>>
is fixed it should be able to handle a lot of disks.
>>
>> That was fixed in build 125.
>>
>>> I want to ask the ZFS community and users what large scale
>>> deploments are out there.  How man disks ? How much capacity ?
>>> Single pool or many pools on a server ? How does resilver work in
>>> those environtments ? How to you backup ? What is the experience
>>> so far ? Major headakes ?
>>>
>>> It would be great if large scale users would share their setups
>>> and experiences with ZFS.
>>
>> The largest ZFS deployment that we have is currently comprised of
>> 22 Dell MD1000 enclosures (330 750 GB Nearline SAS disks). We have
>> 3 head nodes and use one zpool per node, comprised of rather narrow
>> (5+2) RAIDZ2 vdevs. This setup is exclusively used for storing
>> backup data.
>
> This is an interesting design.  It looks like a good use of hardware
> and redundancy for backup storage. Would you be able to share more of
> the details? :-)
Each head node (Dell PE 2900''s) has 3 PERC 6/E controllers (LSI 1078 
based) with 512 MB cache each.

The PERC 6/E supports both load-balancing and path failover so each 
controller has 2 SAS connections to a daisy chained group of 3 MD1000 
enclosures.

The RAIDZ2 vdev layout was chosen because it gives a reasonable 
performance vs space ratio and it maps nicely onto the 15 disk MD1000''s
( 2 x (5+2) +1 ).

There is room for improvement in the design (fewer disks per controller, 
faster PCI Express slots, etc) but performance is good enough for our 
current needs.

>> Resilver times could be better - I am sure that this will improve
>> once we upgrade from S10u9 to 2010.03.
>
> Nit: Solaris 10 u9 is 10/03 or 10/04 or 10/05, depending on what you
> read. Solaris 10 u8 is 11/09.
>
>> One of the things that I am missing in ZFS is the ability to
>> prioritize background operations like scrub and resilver. All our
>> disks are idle during daytime and I would love to be able to take
>> advantage of this, especially during resilver operations.
>
> Scrub I/O is given the lowest priority and is throttled. However, I
> am not sure that the throttle is in Solaris 10, because that source
> is not publicly available. In general, you will not notice a resource
> cap until the system utilization is high enough that the cap is
> effective.  In other words, if the system is mostly idle, the scrub
> consumes the bulk of the resources.
That''s not what I am seeing - resilver operations crawl even when the 
pool is idle.
>> This setup has been running for about a year with no major issues
>> so far. The only hickups we''ve had were all HW related (no fun
in
>> firmware upgrading 200+ disks).
>
> ugh. -- richard
>

-- 
Med venlig hilsen / Best Regards

Henrik Johansen
henrik at scannet.dk
Tlf. 75 53 35 00

ScanNet Group
A/S ScanNet

We got 50+ X4500/X4540''s running in the same DC happiliy with ZFS.
Approximately 2500 drives and growing everyday... 

Br
Mertol 



Mertol Ozyoney 
Storage Practice - Sales Manager

Sun Microsystems, TR
Istanbul TR
Phone +902123352200
Mobile +905339310752
Fax +902123352222
Email mertol.ozyoney at sun.com



-----Original Message-----
From: zfs-discuss-bounces at opensolaris.org
[mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Henrik Johansen
Sent: Friday, January 29, 2010 10:45 AM
To: zfs-discuss at opensolaris.org
Subject: Re: [zfs-discuss] Large scale ZFS deployments out there (>200
disks)

On 01/28/10 11:13 PM, Lutz Schumann wrote:
> While thinking about ZFS as the next generation filesystem without
> limits I am wondering if the real world is ready for this kind of
> incredible technology ...
>
> I''m actually speaking of hardware :)
>
> ZFS can handle a lot of devices. Once in the import bug
> (http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6761786)
> is fixed it should be able to handle a lot of disks.
That was fixed in build 125.
> I want to ask the ZFS community and users what large scale deploments
> are out there.  How man disks ? How much capacity ? Single pool or
> many pools on a server ? How does resilver work in those
> environtments ? How to you backup ? What is the experience so far ?
> Major headakes ?
>
> It would be great if large scale users would share their setups and
> experiences with ZFS.
The largest ZFS deployment that we have is currently comprised of 22 
Dell MD1000 enclosures (330 750 GB Nearline SAS disks). We have 3 head 
nodes and use one zpool per node, comprised of rather narrow (5+2) 
RAIDZ2 vdevs. This setup is exclusively used for storing backup data.

Resilver times could be better - I am sure that this will improve once 
we upgrade from S10u9 to 2010.03.

One of the things that I am missing in ZFS is the ability to prioritize 
background operations like scrub and resilver. All our disks are idle 
during daytime and I would love to be able to take advantage of this, 
especially during resilver operations.

This setup has been running for about a year with no major issues so 
far. The only hickups we''ve had were all HW related (no fun in firmware
upgrading 200+ disks).
> Will you ? :) Thanks, Robert

-- 
Med venlig hilsen / Best Regards

Henrik Johansen
henrik at scannet.dk
Tlf. 75 53 35 00

ScanNet Group
A/S ScanNet
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

I don''t think I have seen this addressed in the follow-ups to your
message.
 One issue we have is with deploying large numbers of files systems per pool
- not necessarily large numbers of disk.  There are major scaling issues
with the sharing of large numbers of file systems, in my configuration I
have about 16K file systems to share and boot times can be several hours.
 There is an open bug

http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6850837

There has been no indication of a horizon for a fix yet as far as I know.


On Thu, Jan 28, 2010 at 5:13 PM, Lutz Schumann
<presales at storageconcepts.de>wrote:
> While thinking about ZFS as the next generation filesystem without limits I
> am wondering if the real world is ready for this kind of incredible
> technology ...
>
> I''m actually speaking of hardware :)
>
> ZFS can handle a lot of devices. Once in the import bug (
> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6761786) is
> fixed it should be able to handle a lot of disks.
>
> I want to ask the ZFS community and users what large scale deploments are
> out there.  How man disks ? How much capacity ? Single pool or many pools
on
> a server ? How does resilver work in those environtments ? How to you
backup
> ?
> What is the experience so far ? Major headakes ?
>
> It would be great if large scale users would share their setups and
> experiences with ZFS.
>
> Will you ? :)
> Thanks,
> Robert
> --
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100225/3327baae/attachment.html>

On Thu, 25 Feb 2010, Alastair Neil wrote:
> I don''t think I have seen this addressed in the follow-ups to your
message. ?One
> issue we have is with deploying large numbers of files systems per pool -
not
> necessarily large numbers of disk. ?There are major scaling issues with the
sharing
> of large numbers of file systems, in my configuration I have about 16K file
systems
> to share and boot times can be several hours. ?There is an open bug
Is boot performance with 16K mounted and exported file systems a whole 
lot better if you use UFS instead?

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

I do not know and I don''t think anyone would deploy a system in that
way
with UFS.  This is the model that is imposed in order to take full advantage
of zfs advanced features such as snapshots, encryption and compression and I
know many universities in particular are eager to adopt it for just that
reason, but are stymied by this problem.

Alastair


On Thu, Feb 25, 2010 at 8:39 PM, Bob Friesenhahn <
bfriesen at simple.dallas.tx.us> wrote:
> On Thu, 25 Feb 2010, Alastair Neil wrote:
>
>  I don''t think I have seen this addressed in the follow-ups to
your
>> message.  One
>> issue we have is with deploying large numbers of files systems per pool
-
>> not
>> necessarily large numbers of disk.  There are major scaling issues with
>> the sharing
>> of large numbers of file systems, in my configuration I have about 16K
>> file systems
>> to share and boot times can be several hours.  There is an open bug
>>
>
> Is boot performance with 16K mounted and exported file systems a whole lot
> better if you use UFS instead?
>
>
> Bob
> --
> Bob Friesenhahn
> bfriesen at simple.dallas.tx.us,
http://www.simplesystems.org/users/bfriesen/
> GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100225/3a1ddf08/attachment.html>

On Thu, 25 Feb 2010, Alastair Neil wrote:
> I do not know and I don''t think anyone would deploy a system in
that way with UFS.?
> This is the model that is imposed in order to take full advantage of zfs
advanced
> features such as snapshots, encryption and compression and I know many
universities
> in particular are eager to adopt it for just that reason, but are stymied
by this
> problem.
It was not really a serious question but it was posed to make a point. 
However, it would be interesting to know if there is another type of 
filesystem (even on Linux or some other OS) which is able to 
reasonably and efficiently support 16K mounted and exported file 
systems.

Eventually Solaris is likely to work much better for this than it does 
today, but most likely there are higher priorities at the moment.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Thu, Feb 25 at 20:21, Bob Friesenhahn wrote:
>On Thu, 25 Feb 2010, Alastair Neil wrote:
>
>>I do not know and I don''t think anyone would deploy a system in
that way with UFS.?
>>This is the model that is imposed in order to take full advantage of zfs
advanced
>>features such as snapshots, encryption and compression and I know many
universities
>>in particular are eager to adopt it for just that reason, but are
stymied by this
>>problem.
>
>It was not really a serious question but it was posed to make a 
>point. However, it would be interesting to know if there is another 
>type of filesystem (even on Linux or some other OS) which is able to 
>reasonably and efficiently support 16K mounted and exported file 
>systems.
>
>Eventually Solaris is likely to work much better for this than it 
>does today, but most likely there are higher priorities at the 
>moment.
I agree with the above, but the best practices guide:

http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#ZFS_file_service_for_SMB_.28CIFS.29_or_SAMBA

states in the SAMBA section that "Beware that mounting 1000s of file
systems, will impact your boot time".  I''d say going from a 2-3
minute
boot time to a 4+ hour boot time is more than just "impact". 
That''s
getting hit by a train.

Might be useful for folks, if the above document listed a few concrete
datapoints of boot time scaling with the number of filesystems or
something similar.

--eric


-- 
Eric D. Mudama
edmudama at mail.bounceswoosh.org

On Feb 26, 2010, at 8:25 PM, Eric D. Mudama wrote:
> On Thu, Feb 25 at 20:21, Bob Friesenhahn wrote:
>> On Thu, 25 Feb 2010, Alastair Neil wrote:
>> 
>>> I do not know and I don''t think anyone would deploy a
system in that way with UFS.
>>> This is the model that is imposed in order to take full advantage
of zfs advanced
>>> features such as snapshots, encryption and compression and I know
many universities
>>> in particular are eager to adopt it for just that reason, but are
stymied by this
>>> problem.
>> 
>> It was not really a serious question but it was posed to make a point.
However, it would be interesting to know if there is another type of filesystem
(even on Linux or some other OS) which is able to reasonably and efficiently
support 16K mounted and exported file systems.
>> 
>> Eventually Solaris is likely to work much better for this than it does
today, but most likely there are higher priorities at the moment.
> 
> I agree with the above, but the best practices guide:
> 
>
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#ZFS_file_service_for_SMB_.28CIFS.29_or_SAMBA
> 
> states in the SAMBA section that "Beware that mounting 1000s of file
> systems, will impact your boot time".  I''d say going from a
2-3 minute
> boot time to a 4+ hour boot time is more than just "impact". 
That''s
> getting hit by a train.
The shares are more troublesome than the mounts.  
> 
> Might be useful for folks, if the above document listed a few concrete
> datapoints of boot time scaling with the number of filesystems or
> something similar.
Gory details and timings are available in the many references to CR 6850837
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6850837
 -- richard

ZFS storage and performance consulting at http://www.RichardElling.com
ZFS training on deduplication, NexentaStor, and NAS performance
http://nexenta-atlanta.eventbrite.com (March 16-18, 2010)

Ironically It''s nfs exporting that is the real hog, cifs shares seem to
come
up pretty fast.  The fact that cifs shares can be fast makes it hard for me
to understand why Sun/Oracle seem to be making such a meal of this bug.
Possibly because it only critically affects poor universities and not
clients with the budget to throw hardware at the problem.

On Fri, Feb 26, 2010 at 11:59 PM, Richard Elling
<richard.elling at gmail.com>wrote:
> On Feb 26, 2010, at 8:25 PM, Eric D. Mudama wrote:
> > On Thu, Feb 25 at 20:21, Bob Friesenhahn wrote:
> >> On Thu, 25 Feb 2010, Alastair Neil wrote:
> >>
> >>> I do not know and I don''t think anyone would deploy a
system in that
> way with UFS.
> >>> This is the model that is imposed in order to take full
advantage of
> zfs advanced
> >>> features such as snapshots, encryption and compression and I
know many
> universities
> >>> in particular are eager to adopt it for just that reason, but
are
> stymied by this
> >>> problem.
> >>
> >> It was not really a serious question but it was posed to make a
point.
> However, it would be interesting to know if there is another type of
> filesystem (even on Linux or some other OS) which is able to reasonably and
> efficiently support 16K mounted and exported file systems.
> >>
> >> Eventually Solaris is likely to work much better for this than it
does
> today, but most likely there are higher priorities at the moment.
> >
> > I agree with the above, but the best practices guide:
> >
> >
>
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#ZFS_file_service_for_SMB_.28CIFS.29_or_SAMBA
> >
> > states in the SAMBA section that "Beware that mounting 1000s of
file
> > systems, will impact your boot time".  I''d say going
from a 2-3 minute
> > boot time to a 4+ hour boot time is more than just "impact".
That''s
> > getting hit by a train.
>
> The shares are more troublesome than the mounts.
>
> >
> > Might be useful for folks, if the above document listed a few concrete
> > datapoints of boot time scaling with the number of filesystems or
> > something similar.
>
> Gory details and timings are available in the many references to CR 6850837
> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6850837
>  -- richard
>
> ZFS storage and performance consulting at http://www.RichardElling.com
> ZFS training on deduplication, NexentaStor, and NAS performance
> http://nexenta-atlanta.eventbrite.com (March 16-18, 2010)
>
>
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100227/b8b49ce3/attachment.html>

On Feb 26, 2010, at 8:59 PM, Richard Elling wrote:
> On Feb 26, 2010, at 8:25 PM, Eric D. Mudama wrote:
>> On Thu, Feb 25 at 20:21, Bob Friesenhahn wrote:
>>> On Thu, 25 Feb 2010, Alastair Neil wrote:
>>> 
>>>> I do not know and I don''t think anyone would deploy a
system in that way with UFS.
>>>> This is the model that is imposed in order to take full
advantage of zfs advanced
>>>> features such as snapshots, encryption and compression and I
know many universities
>>>> in particular are eager to adopt it for just that reason, but
are stymied by this
>>>> problem.
>>> 
>>> It was not really a serious question but it was posed to make a
point. However, it would be interesting to know if there is another type of
filesystem (even on Linux or some other OS) which is able to reasonably and
efficiently support 16K mounted and exported file systems.
>>> 
>>> Eventually Solaris is likely to work much better for this than it
does today, but most likely there are higher priorities at the moment.
>> 
>> I agree with the above, but the best practices guide:
>> 
>>
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#ZFS_file_service_for_SMB_.28CIFS.29_or_SAMBA
>> 
>> states in the SAMBA section that "Beware that mounting 1000s of
file
>> systems, will impact your boot time".  I''d say going from
a 2-3 minute
>> boot time to a 4+ hour boot time is more than just "impact". 
That''s
>> getting hit by a train.
Perhaps someone that has a SAMBA config large enough could make a
test similar to the NFS set described in
http://developers.sun.com/solaris/articles/nfs_zfs.html
(note the date, 2007)
 -- richard

ZFS storage and performance consulting at http://www.RichardElling.com
ZFS training on deduplication, NexentaStor, and NAS performance
http://nexenta-atlanta.eventbrite.com (March 16-18, 2010)

On Fri, Feb 26, 2010 at 09:25:57PM -0700, Eric D. Mudama wrote:
...
> I agree with the above, but the best practices guide:
> 
>
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#ZFS_file_service_for_SMB_.28CIFS.29_or_SAMBA
> 
> states in the SAMBA section that "Beware that mounting 1000s of file
> systems, will impact your boot time".  I''d say going from a
2-3 minute
> boot time to a 4+ hour boot time is more than just "impact". 
That''s
> getting hit by a train.
At least on S10u8 its not that bad. Last time I patched and rebooted 
a X4500 with ~350 ZFS it took about 10min to come up, a X4600 with
a 3510 and ~2350 ZFS took about 20min (almost all are shared via NFS).
Shutting down/unshare them takes roughly the same time ...
On the X4600 creating|destroying a single ZFS (no matter on which pool or
how many ZFS belong to the same pool!) takes about 20 sec, renaming about
40 sec ... - that''s really a pain ...
  
Regards,
jel.
-- 
Otto-von-Guericke University     http://www.cs.uni-magdeburg.de/
Department of Computer Science   Geb. 29 R 027, Universitaetsplatz 2
39106 Magdeburg, Germany         Tel: +49 391 67 12768

Speaking of long boot times, Ive heard that IBM power servers boot in 90 minutes
or more.
-- 
This message posted from opensolaris.org

On Sat, 27 Feb 2010, Jens Elkner wrote:
> At least on S10u8 its not that bad. Last time I patched and rebooted
> a X4500 with ~350 ZFS it took about 10min to come up, a X4600 with
> a 3510 and ~2350 ZFS took about 20min (almost all are shared via NFS).
Our x4500''s with about 8000 filesystems per run about 50-60 minutes to
shut
down, and about the same to boot up, resulting in about a 2 hour boot
cycle, which is kind of annoying. The lack of scalability is in the NFS
sharing, it only takes ~5 minutes to mount all 8000. I hope someday
they''ll
optimize that a bit...


-- 
Paul B. Henson  |  (909) 979-6361  |  http://www.csupomona.edu/~henson/
Operating Systems and Network Analyst  |  henson at csupomona.edu
California State Polytechnic University  |  Pomona CA 91768