Has anyone had issues with creating ZFS pools greater than 1 terabyte (TB)? I''ve created 11 LUNs from a Sun 2540 Disk array (approx 1 TB each). The host system ( SUN Enterprise 5220) reconizes the "disks" as each having 931GB space. So that should be 10+ TB in size total. However when I zpool them together (using raidz) the zpool status reports 9GB instead of 9TB. Does ZFS have problem reporting TB and defaults to GB instead?? Is my pool really TB in size?? I''ve read in the best practice wiki that splitting them into smaller pools. Any recommendation on this?? I''m desperate in keepingas much space useable as possible. Thanks --Kenny This message posted from opensolaris.org
On Wed, Aug 27, 2008 at 1:08 PM, Kenny <knoe at bigfoot.com> wrote:> Has anyone had issues with creating ZFS pools greater than 1 terabyte (TB)? > > I''ve created 11 LUNs from a Sun 2540 Disk array (approx 1 TB each). The > host system ( SUN Enterprise 5220) reconizes the "disks" as each having > 931GB space. So that should be 10+ TB in size total. However when I zpool > them together (using raidz) the zpool status reports 9GB instead of 9TB. > > Does ZFS have problem reporting TB and defaults to GB instead?? Is my pool > really TB in size?? > > I''ve read in the best practice wiki that splitting them into smaller pools. > Any recommendation on this?? I''m desperate in keepingas much space useable > as possible. >OS version and zfs version would be helpful. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080827/e4bcead2/attachment.html>
Couple of questions, What version of Solaris are you using? (cat /etc/release) If you''re exposing each disk individually through a LUN/2540 Volume, you don''t really gain anything by having a spare on the 2540 (which I assume you''re doing by only exposing 11 LUNs instead of 12). Your best bet is to set no spares on the 2540 and then set one of the LUNs as a spare via ZFS. How will you be using the storage? This will help determine how your zpool should be structured. -Aaron On Wed, Aug 27, 2008 at 11:08 AM, Kenny <knoe at bigfoot.com> wrote:> Has anyone had issues with creating ZFS pools greater than 1 terabyte (TB)? > > I''ve created 11 LUNs from a Sun 2540 Disk array (approx 1 TB each). The > host system ( SUN Enterprise 5220) reconizes the "disks" as each having > 931GB space. So that should be 10+ TB in size total. However when I zpool > them together (using raidz) the zpool status reports 9GB instead of 9TB. > > Does ZFS have problem reporting TB and defaults to GB instead?? Is my pool > really TB in size?? > > I''ve read in the best practice wiki that splitting them into smaller pools. > Any recommendation on this?? I''m desperate in keepingas much space useable > as possible. > > Thanks --Kenny > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080827/9625ab17/attachment.html>
> Has anyone had issues with creating ZFS pools greater than 1 terabyte (TB)? > > I''ve created 11 LUNs from a Sun 2540 Disk array (approx 1 TB each). The host system ( SUN Enterprise 5220) reconizes the "disks" as each having 931GB space. So that should be 10+ TB in size total. However when I zpool them together (using raidz) the zpool status reports 9GB instead of 9TB. > > Does ZFS have problem reporting TB and defaults to GB instead?? Is my pool really TB in size?? > > I''ve read in the best practice wiki that splitting them into smaller pools. Any recommendation on this?? I''m desperate in keepingas much space useable as possible.This is from a zpool with three disks at 1 metric TB (= 931 GB) using raidz. claus at malene:~# zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT ef1 2.72T 2.65T 67.0G 97% ONLINE - -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare
Tcook - Sorry bout that... Solaris 10 (8/07 I think) ZFS version 4 How can I upgrade ZFS w/o having to rebuild with Sol 10 5/08? Thanks --Kenny This message posted from opensolaris.org
Claus - Thanks!! At least I know I''m not going crazy!! Yes, I''ve got 11 metric 1 TB disks and would like 10TB useable (end game...) --Kenny This message posted from opensolaris.org
Arron, Thanks... Yes I did reserve one for Hot spare on the hardware side.... Guess I can change that thinking. <grin> Solaris 10 8/07 is my OS. This storage is to become our syslog repository for approx 20 servers. We have approx 3TB of data now and wanted space to grow and keep more online for research before moving items off to tape. Thanks --Kenny This message posted from opensolaris.org
Claus, Thanks for the sanity check... I thought I wasn''t crazy.... Now on to find out why my 9TB turned into 9GB... <grin> Thanks again --Kenny This message posted from opensolaris.org
On Wed, 27 Aug 2008, Kenny wrote:> Tcook - Sorry bout that... > > Solaris 10 (8/07 I think) > ZFS version 4 > > How can I upgrade ZFS w/o having to rebuild with Sol 10 5/08?You can use ''smpatch'' to apply patches to your system so that kernel/zfs wise it is essentially Sol 10 5/08. However, I have never heard of this sort of problem before. Perhaps there is user error. Perhaps you accidentally did something silly like create an 11 disk mirror. Or maybe you thought you configured the StorageTek 2540 to export the entire drive as a volume but got a smaller allocation instead (been there, done that). Using CAM is pretty tedious so you could do the right thing for one disk and accidentally use the minimum default size for the others. You said that ''zpool status'' reported only 9GB but there is no size output produced by ''zpool status''. You can use ''zpool iostat'' to see the space available. With ''zpool iostat -v'' you can see how much space zfs is obtaining from each device. If you can post the output of ''zpool iostat -v'' then people here can help you further. While I don''t have 1TB disks and did not use raidz, I have done much of what you are attempting to do. You can read about what I did at "http://www.simplesystems.org/users/bfriesen/zfs-discuss/2540-zfs-performance.pdf". Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Kenny wrote:> Arron, > > Thanks... Yes I did reserve one for Hot spare on the hardware side.... Guess I can change that thinking. <grin> > > Solaris 10 8/07 is my OS. > > This storage is to become our syslog repository for approx 20 servers. We have approx 3TB of data now and wanted space to grow and keep more online for research before moving items off to tape. >That will compress rather nicely! IMHO, you should enable ZFS compression. -- richard
On Wed, 27 Aug 2008, Kenny wrote:> > Thanks... Yes I did reserve one for Hot spare on the hardware > side.... Guess I can change that thinking. <grin>Disks in the 2540 are expensive. The hot spare does not need to be in the 2540. You also use a suitably large disk (1TB) installed in your server as the hot spare. This assumes that disks in the server are cheaper than in the 2540. With this approach you can then use all 12 disks in your 2540 and configure them as two raidz2 vdevs in one pool. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
On Wed, Aug 27, 2008 at 1:51 PM, Kenny <knoe at bigfoot.com> wrote:> Tcook - Sorry bout that... > > Solaris 10 (8/07 I think) > ZFS version 4 > > How can I upgrade ZFS w/o having to rebuild with Sol 10 5/08? > > Thanks --Kenny > >Please paste the output of df, zpool status, and format so we can verify what you''re seeing. :) --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080827/9d9210f0/attachment.html>
Bob, Thanks for the reply. Yes I did read your white paper and am using it!!
Thanks again!!
I used zpool iostat -v and it did''t give the information as
advertised... see below
bash-3.00# zpool iostat -v
capacity
operations bandwidth
pool used avail read
write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
log_data 147K 9.81G
0 0 0 4
raidz1 147K 9.81G
0 0 0 4
c6t600A0B800049F93C0000030A48B3EA2Cd0 - - 0 0 0
22
c6t600A0B800049F93C0000030D48B3EAB6d0 - - 0 0 0
22
c6t600A0B800049F93C0000031C48B3EC76d0 - - 0 0 0
22
c6t600A0B800049F93C0000031F48B3ECA8d0 - - 0 0 0
22
c6t600A0B800049F93C0000030448B3CDEEd0 - - 0 0 0
22
c6t600A0B800049F93C0000030748B3E9F0d0 - - 0 0 0
22
c6t600A0B800049F93C0000031048B3EB44d0 - - 0 0 0
22
c6t600A0B800049F93C0000031348B3EB94d0 - - 0 0 0
22
c6t600A0B800049F93C0000031648B3EBE4d0 - - 0 0 0
22
c6t600A0B800049F93C0000031948B3EC28d0 - - 0 0 0
22
c6t600A0B800049F93C0000032248B3ECDEd0 - - 0 0 0
22
-------------------------------------- ----- ----- ----- ----- ----- -----
(sorry but I can''t get the horizontal format to set the columns
correctly...)
This message posted from opensolaris.org
Tim,
Per your request...
df -h
bash-3.00# df -h
Filesystem size used avail capacity Mounted on
/dev/md/dsk/d10 98G 4.2G 92G 5% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 32G 1.4M 32G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
/platform/SUNW,SPARC-Enterprise-T5220/lib/libc_psr/libc_psr_hwcap1.so.1
98G 4.2G 92G 5%
/platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,SPARC-Enterprise-T5220/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
98G 4.2G 92G 5%
/platform/sun4v/lib/sparcv9/libc_psr.so.1
fd 0K 0K 0K 0% /dev/fd
/dev/md/dsk/d50 19G 4.3G 15G 23% /var
swap 512M 112K 512M 1% /tmp
swap 32G 40K 32G 1% /var/run
/dev/md/dsk/d30 9.6G 1.5G 8.1G 16% /opt
/dev/md/dsk/d40 1.9G 142M 1.7G 8% /export/home
/vol/dev/dsk/c0t0d0/fm540cd3
591M 591M 0K 100% /cdrom/fm540cd3
log_data 8.8G 44K 8.8G 1% /log_data
bash-3.00# bash-3.00# df -h
v/dsk/c0t0d0/fm540cd3
591M 591M 0K 100% /cdrom/fm540cd3
log_data 8.8G 44K 8.8G 1% /log_data
************************************************************************************
zpool status
bash-3.00# zpool status
pool: log_data
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
log_data ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c6t600A0B800049F93C0000030A48B3EA2Cd0 ONLINE 0 0 0
c6t600A0B800049F93C0000030D48B3EAB6d0 ONLINE 0 0 0
c6t600A0B800049F93C0000031C48B3EC76d0 ONLINE 0 0 0
c6t600A0B800049F93C0000031F48B3ECA8d0 ONLINE 0 0 0
c6t600A0B800049F93C0000030448B3CDEEd0 ONLINE 0 0 0
c6t600A0B800049F93C0000030748B3E9F0d0 ONLINE 0 0 0
c6t600A0B800049F93C0000031048B3EB44d0 ONLINE 0 0 0
c6t600A0B800049F93C0000031348B3EB94d0 ONLINE 0 0 0
c6t600A0B800049F93C0000031648B3EBE4d0 ONLINE 0 0 0
c6t600A0B800049F93C0000031948B3EC28d0 ONLINE 0 0 0
c6t600A0B800049F93C0000032248B3ECDEd0 ONLINE 0 0 0
errors: No known data errors
************************************************************************************
format
bash-3.00# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
/pci at 0/pci at 0/pci at 2/scsi at 0/sd at 0,0
1. c1t1d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
/pci at 0/pci at 0/pci at 2/scsi at 0/sd at 1,0
2. c6t600A0B800049F93C0000030A48B3EA2Cd0
<SUN-LCSM100_F-0670-931.01GB>
/scsi_vhci/ssd at g600a0b800049f93c0000030a48b3ea2c
3. c6t600A0B800049F93C0000030D48B3EAB6d0
<SUN-LCSM100_F-0670-931.01MB>
/scsi_vhci/ssd at g600a0b800049f93c0000030d48b3eab6
4. c6t600A0B800049F93C0000031C48B3EC76d0
<SUN-LCSM100_F-0670-931.01MB>
/scsi_vhci/ssd at g600a0b800049f93c0000031c48b3ec76
5. c6t600A0B800049F93C0000031F48B3ECA8d0
<SUN-LCSM100_F-0670-931.01GB>
/scsi_vhci/ssd at g600a0b800049f93c0000031f48b3eca8
6. c6t600A0B800049F93C0000030448B3CDEEd0
<SUN-LCSM100_F-0670-931.01GB>
/scsi_vhci/ssd at g600a0b800049f93c0000030448b3cdee
7. c6t600A0B800049F93C0000030748B3E9F0d0
<SUN-LCSM100_F-0670-931.01GB>
/scsi_vhci/ssd at g600a0b800049f93c0000030748b3e9f0
8. c6t600A0B800049F93C0000031048B3EB44d0
<SUN-LCSM100_F-0670-931.01MB>
/scsi_vhci/ssd at g600a0b800049f93c0000031048b3eb44
9. c6t600A0B800049F93C0000031348B3EB94d0
<SUN-LCSM100_F-0670-931.01GB>
/scsi_vhci/ssd at g600a0b800049f93c0000031348b3eb94
10. c6t600A0B800049F93C0000031648B3EBE4d0
<SUN-LCSM100_F-0670-931.01GB>
/scsi_vhci/ssd at g600a0b800049f93c0000031648b3ebe4
11. c6t600A0B800049F93C0000031948B3EC28d0
<SUN-LCSM100_F-0670-931.01GB>
/scsi_vhci/ssd at g600a0b800049f93c0000031948b3ec28
12. c6t600A0B800049F93C0000032248B3ECDEd0
<SUN-LCSM100_F-0670-931.01GB>
/scsi_vhci/ssd at g600a0b800049f93c0000032248b3ecde
Specify disk (enter its number):
This message posted from opensolaris.org
Kenny schrieb:> 2. c6t600A0B800049F93C0000030A48B3EA2Cd0 <SUN-LCSM100_F-0670-931.01GB> > /scsi_vhci/ssd at g600a0b800049f93c0000030a48b3ea2c > 3. c6t600A0B800049F93C0000030D48B3EAB6d0 <SUN-LCSM100_F-0670-931.01MB> > /scsi_vhci/ssd at g600a0b800049f93c0000030d48b3eab6Disk 2: 931GB Disk 3: 931MB Do you see the difference? Daniel
Daniel Rock wrote:> > Kenny schrieb: > > 2. c6t600A0B800049F93C0000030A48B3EA2Cd0 > <SUN-LCSM100_F-0670-931.01GB> > > /scsi_vhci/ssd at g600a0b800049f93c0000030a48b3ea2c > > 3. c6t600A0B800049F93C0000030D48B3EAB6d0 > <SUN-LCSM100_F-0670-931.01MB> > > /scsi_vhci/ssd at g600a0b800049f93c0000030d48b3eab6 > > Disk 2: 931GB > Disk 3: 931MB > > Do you see the difference? >Not just disk 3:> AVAILABLE DISK SELECTIONS: > 3. c6t600A0B800049F93C0000030D48B3EAB6d0 <SUN-LCSM100_F-0670-931.01MB> > /scsi_vhci/ssd at g600a0b800049f93c0000030d48b3eab6 > 4. c6t600A0B800049F93C0000031C48B3EC76d0 <SUN-LCSM100_F-0670-931.01MB> > /scsi_vhci/ssd at g600a0b800049f93c0000031c48b3ec76 > 8. c6t600A0B800049F93C0000031048B3EB44d0 <SUN-LCSM100_F-0670-931.01MB> > /scsi_vhci/ssd at g600a0b800049f93c0000031048b3eb44 >This all makes sense now, since a RAIDZ (or RAIDZ2) vdev can only be as big as it''s *smallest* component device. -Kyle> > > Daniel > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
exactly :) On 8/28/08, Kyle McDonald <KMcDonald at egenera.com> wrote:> Daniel Rock wrote: >> >> Kenny schrieb: >> > 2. c6t600A0B800049F93C0000030A48B3EA2Cd0 >> <SUN-LCSM100_F-0670-931.01GB> >> > /scsi_vhci/ssd at g600a0b800049f93c0000030a48b3ea2c >> > 3. c6t600A0B800049F93C0000030D48B3EAB6d0 >> <SUN-LCSM100_F-0670-931.01MB> >> > /scsi_vhci/ssd at g600a0b800049f93c0000030d48b3eab6 >> >> Disk 2: 931GB >> Disk 3: 931MB >> >> Do you see the difference? >> > Not just disk 3: > >> AVAILABLE DISK SELECTIONS: >> 3. c6t600A0B800049F93C0000030D48B3EAB6d0 >> <SUN-LCSM100_F-0670-931.01MB> >> /scsi_vhci/ssd at g600a0b800049f93c0000030d48b3eab6 >> 4. c6t600A0B800049F93C0000031C48B3EC76d0 >> <SUN-LCSM100_F-0670-931.01MB> >> /scsi_vhci/ssd at g600a0b800049f93c0000031c48b3ec76 >> 8. c6t600A0B800049F93C0000031048B3EB44d0 >> <SUN-LCSM100_F-0670-931.01MB> >> /scsi_vhci/ssd at g600a0b800049f93c0000031048b3eb44 >> > This all makes sense now, since a RAIDZ (or RAIDZ2) vdev can only be as > big as it''s *smallest* component device. > > -Kyle > >> >> >> Daniel >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
On Thu, 28 Aug 2008, Kenny wrote:> Bob, Thanks for the reply. Yes I did read your white paper and am using it!! Thanks again!! > > I used zpool iostat -v and it did''t give the information as advertised... see belowThe lack of size information seems quit odd. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
On Thu, 28 Aug 2008, Kenny wrote:> 2. c6t600A0B800049F93C0000030A48B3EA2Cd0 <SUN-LCSM100_F-0670-931.01GB> > /scsi_vhci/ssd at g600a0b800049f93c0000030a48b3ea2cGood.> 3. c6t600A0B800049F93C0000030D48B3EAB6d0 <SUN-LCSM100_F-0670-931.01MB> > /scsi_vhci/ssd at g600a0b800049f93c0000030d48b3eab6Oops! Oops! Oops! It seems that some of your drives have the full 931.01GB exported while others have only 931.01MB exported. The smallest device size will be used to size the vdev in your pool. I sense a user error in the tedious CAM interface. CAM is slow so you need to be patient and take extra care when configuring the 2540 volumes. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Ok so I knew it had to be operator headspace... <grin> I found my error and have fixed it in CAM. Thanks to all for helping my education!! However I do have a question. And pardon if it''s a 101 type... How did you determine from the format output the GB vs MB amount?? Where do you compute 931 GB vs 932 MB from this?? 2. c6t600A0B800049F93C0000030A48B3EA2Cd0 /scsi_vhci/ssd at g600a0b800049f93c0000030a48b3ea2c 3. c6t600A0B800049F93C0000030D48B3EAB6d0 /scsi_vhci/ssd at g600a0b800049f93c0000030d48b3eab6 Please educate me!! <grin> Thanks again! --Kenny -- This message posted from opensolaris.org
Ok so I knew it had to be operator headspace... <grin> I found my error and have fixed it in CAM. Thanks to all for helping my education!! However I do have a question. And pardon if it''s a 101 type... How did you determine from the format output the GB vs MB amount?? Where do you compute 931 GB vs 932 MB from this?? 2. c6t600A0B800049F93C0000030A48B3EA2Cd0 /scsi_vhci/ssd at g600a0b800049f93c0000030a48b3ea2c 3. c6t600A0B800049F93C0000030D48B3EAB6d0 /scsi_vhci/ssd at g600a0b800049f93c0000030d48b3eab6 Please educate me!! <grin> Thanks again! --Kenny -- This message posted from opensolaris.org
Kenny wrote:> > How did you determine from the format output the GB vs MB amount?? > > Where do you compute 931 GB vs 932 MB from this?? > > 2. c6t600A0B800049F93C0000030A48B3EA2Cd0 /scsi_vhci/ssd at g600a0b800049f93c0000030a48b3ea2c > > 3. c6t600A0B800049F93C0000030D48B3EAB6d0 > /scsi_vhci/ssd at g600a0b800049f93c0000030d48b3eab6 >It''s in the part you didn''t cut and paste: AVAILABLE DISK SELECTIONS:> 3. c6t600A0B800049F93C0000030D48B3EAB6d0 <SUN-LCSM100_F-0670-931.01MB> > /scsi_vhci/ssd at g600a0b800049f93c0000030d48b3eab6 > 4. c6t600A0B800049F93C0000031C48B3EC76d0 <SUN-LCSM100_F-0670-931.01MB> > /scsi_vhci/ssd at g600a0b800049f93c0000031c48b3ec76 > 8. c6t600A0B800049F93C0000031048B3EB44d0 <SUN-LCSM100_F-0670-931.01MB> > /scsi_vhci/ssd at g600a0b800049f93c0000031048b3eb44 >Look at the label: <SUN-LCSM100_F-0670-931.01MB> The last field.> Please educate me!! <grin> >No problem. Things like this have happened to me from time to time. -Kyle> Thanks again! > > --Kenny > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >