I hope my terminology is correct....
I have a ZFS array which uses raw devices. I'd rather it use glabel and
supply the GEOM devices to ZFS instead. In addition, I'll also
partition the HDD to avoid using the entire HDD: leave a little bit of
space at the start and end.
Why use glabel?
* So ZFS can find and use the correct HDD should the HDD device ever
get renumbered for whatever reason. e.g. /dev/da0 becomes /dev/da6
when you move it to another controller.
Why use partitions?
* Primarily: two HDD of a given size, say 2TB, do not always provide
the same amount of available space. If you use a slightly smaller
partition instead of the entire physical HDD, you're much more
likely to have a happier experience when it comes time to replace an
HDD.
* There seems to be a consensus amongst some that leaving the start and
and of your HDD empty. Give the rest to ZFS.
Things I've read that led me to the above reasons:
*
http://docs.freebsd.org/cgi/getmsg.cgi?fetch=399538+0+current/freebsd-stable
*
http://lists.freebsd.org/pipermail/freebsd-stable/2010-February/055008.html
* http://lists.freebsd.org/pipermail/freebsd-geom/2009-July/003620.html
The plan for this plan, I'm going to play with just two HDD, because
that's what I have available. Let's assume these two HDD are ad0 and
ad1. I am not planning to boot from these HDD; they are for storage only.
First, create a new GUID Partition Table partition scheme on the HDD:
gpart create -s GPT ad0
Let's see how much space we have. This output will be used to determine
SOMEVALUE in the next command.
gpart show
Create a new partition within that scheme:
gpart add -b 34 -s SOMEVALUE -t freebsd-zfs ad0
Why '-b 34'? Randi pointed me to
http://en.wikipedia.org/wiki/GUID_Partition_Table where it explains what
the first 33 LBA are used for. It's not for us to use here.
Where SOMEVALUE is the number of blocks to use. I plan not to use all
the available blocks but leave a few hundred MB free at the end.
That'll allow for the variance in HDD size.
Now, label the thing:
glabel label -v disk00 /dev/ad0
Repeat the above with ad1 to get disk01. Repeat for all other HDD...
Then create your zpool:
zpool create bigtank disk00 disk01 ... etc
Any suggestions/comments? Is there any advantage to using the -l option
on 'gpart add' instead of the glabel above?
Thanks
--
Dan Langille - http://langille.org/
On 7/21/2010 11:05 PM, Dan Langille wrote (something close to this):> First, create a new GUID Partition Table partition scheme on the HDD: > > gpart create -s GPT ad0 > > > Let's see how much space we have. This output will be used to determine > SOMEVALUE in the next command. > > gpart show > > > Create a new partition within that scheme: > > gpart add -b 34 -s SOMEVALUE -t freebsd-zfs ad0 > > > Now, label the thing: > > glabel label -v disk00 /dev/ad0Or, is this more appropriate? glabel label -v disk00 /dev/ad0s1 -- Dan Langille - http://langille.org/
On Wed, Jul 21, 2010 at 10:05 PM, Dan Langille <dan@langille.org> wrote:> Why '-b 34'? Randi pointed me to > http://en.wikipedia.org/wiki/GUID_Partition_Table where it explains what > the first 33 LBA are used for. It's not for us to use here. > > Where SOMEVALUE is the number of blocks to use. I plan not to use all the > available blocks but leave a few hundred MB free at the end. That'll allow > for the variance in HDD size. > > Any suggestions/comments? Is there any advantage to using the -l option on > 'gpart add' instead of the glabel above? >You'll want to make sure your partitions are aligned, discussion here(says 4k drives, but info pertinent to all): http://lists.freebsd.org/pipermail/freebsd-hackers/2010-March/031154.html My understanding is that you weren't booting from zfs, just using it as an data file system. In that case, you'd want to use "gpart add -b 512 ..." or some other multiple of 16. Even 1024 would be a good safe number. Also GPT creates partitions not slices. Your resulting partitions with be labeled something like ad0p1, ad0p2, etc. -- Adam Vande More
On Wed, Jul 21, 2010 at 10:05 PM, Dan Langille <dan@langille.org> wrote:> I hope my terminology is correct.... > > I have a ZFS array which uses raw devices. ?I'd rather it use glabel and > supply the GEOM devices to ZFS instead. ?In addition, I'll also partition > the HDD to avoid using the entire HDD: leave a little bit of space at the > start and end. > > Why use glabel? > > ?* So ZFS can find and use the correct HDD should the HDD device ever > ? get renumbered for whatever reason. ?e.g. /dev/da0 becomes /dev/da6 > ? when you move it to another controller. > > Why use partitions? > > ?* Primarily: two HDD of a given size, say 2TB, do not always provide > ? the same amount of available space. ?If you use a slightly smaller > ? partition instead of the entire physical HDD, you're much more > ? likely to have a happier experience when it comes time to replace an > ? HDD. > > ?* There seems to be a consensus amongst some that leaving the start and > ? and of your HDD empty. ?Give the rest to ZFS. > > Things I've read that led me to the above reasons: > > * > http://docs.freebsd.org/cgi/getmsg.cgi?fetch=399538+0+current/freebsd-stable > * > http://lists.freebsd.org/pipermail/freebsd-stable/2010-February/055008.html > * http://lists.freebsd.org/pipermail/freebsd-geom/2009-July/003620.html > > The plan for this plan, I'm going to play with just two HDD, because that's > what I have available. ?Let's assume these two HDD are ad0 and ad1. ?I am > not planning to boot from these HDD; they are for storage only. > > First, create a new GUID Partition Table partition scheme on the HDD: > > ?gpart create -s GPT ad0 > > > Let's see how much space we have. ?This output will be used to determine > SOMEVALUE in the next command. > > ?gpart show > > > Create a new partition within that scheme: > > ?gpart add -b 34 -s SOMEVALUE -t freebsd-zfs ad0 >Instead of using glabel to label the partition on a GPT disk, use this command instead: gpart add -b 34 -s SOMEVALUE -t freebsd-zfs -l disk00 ad0 You will then see a /dev/gpt/disk00. Then to create the zpool use: zpool create bigtank gpt/disk00 gpt/disk02 ... Scot
On 22/07/2010, at 12:35, Dan Langille wrote:> Why use glabel? > > * So ZFS can find and use the correct HDD should the HDD device ever > get renumbered for whatever reason. e.g. /dev/da0 becomes /dev/da6 > when you move it to another controller. > > Why use partitions? > > * Primarily: two HDD of a given size, say 2TB, do not always provide > the same amount of available space. If you use a slightly smaller > partition instead of the entire physical HDD, you're much more > likely to have a happier experience when it comes time to replace an > HDD. > > * There seems to be a consensus amongst some that leaving the start and > and of your HDD empty. Give the rest to ZFS.I would combine both! GPT generates a UUID for each partition and glabel presents this so ZFS can use it, eg I have.. [cain 19:45] ~ >sudo zpool status pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2 ONLINE 0 0 0 gptid/d7467802-418f-11df-bcfc-001517e077fb ONLINE 0 0 0 gptid/d7eeeced-418f-11df-bcfc-001517e077fb ONLINE 0 0 0 gptid/d8761aa0-418f-11df-bcfc-001517e077fb ONLINE 0 0 0 gptid/d9083d18-418f-11df-bcfc-001517e077fb ONLINE 0 0 0 gptid/d97203ec-418f-11df-bcfc-001517e077fb ONLINE 0 0 0 and on each disk.. [cain 19:46] ~ >gpart list ada0 Geom name: ada0 fwheads: 16 fwsectors: 63 last: 1953525134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada0p1 Mediasize: 8589934592 (8.0G) Sectorsize: 512 Mode: r0w0e0 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 8589934592 offset: 17408 type: freebsd-swap index: 1 end: 16777249 start: 34 2. Name: ada0p2 Mediasize: 991614917120 (924G) Sectorsize: 512 Mode: r1w1e2 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: (null) length: 991614917120 offset: 8589952000 type: freebsd-zfs index: 2 end: 1953525134 start: 16777250 Consumers: 1. Name: ada0 Mediasize: 1000204886016 (932G) Sectorsize: 512 Mode: r1w1e3 The only tedious part is working out which drive has what UUIDs on it because gpart doesn't list them. The advantage of using the UUIDs is that if you setup another machine the same way you don't have to worry about things when you plug in the disks from it to recover something. Or perhaps you are upgrading at the same time as replacing hardware so you have all the disks in at once.> Create a new partition within that scheme: > > gpart add -b 34 -s SOMEVALUE -t freebsd-zfs ad0 > > Why '-b 34'? Randi pointed me to http://en.wikipedia.org/wiki/GUID_Partition_Table where it explains what the first 33 LBA are used for. It's not for us to use here.If you don't specify -b it will DTRT - that's how I did it. You can also specify the size (and start) in human units (Gb etc). -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C
On Jul 21, 2010, at 11:05 PM, Dan Langille wrote:> I hope my terminology is correct.... > > I have a ZFS array which uses raw devices. I'd rather it use glabel and supply the GEOM devices to ZFS instead. In addition, I'll also partition the HDD to avoid using the entire HDD: leave a little bit of space at the start and end. > > Why use glabel? > > * So ZFS can find and use the correct HDD should the HDD device ever > get renumbered for whatever reason. e.g. /dev/da0 becomes /dev/da6 > when you move it to another controller.I have created ZFS pools using this strategy. However, about a year ago I still fell foul of the drive shuffling problem, when GEOM labels appeared not to be detected properly: http://lists.freebsd.org/pipermail/freebsd-geom/2009-July/003654.html This was using RELENG_7, and the problem was provoked by external USB drives. The same issue might not occur with FreeBSD 8.x, but I thought I'd point out my experience as a possible warning about using glabel. Nowadays, I use GPT labels ("gpart ... -l somelabel", referenced via /dev/gpt/somelabel).> Why use partitions? > > * Primarily: two HDD of a given size, say 2TB, do not always provide > the same amount of available space. If you use a slightly smaller > partition instead of the entire physical HDD, you're much more > likely to have a happier experience when it comes time to replace an > HDD. > > * There seems to be a consensus amongst some that leaving the start and > and of your HDD empty. Give the rest to ZFS.You should also try and accommodate 4K sector size drives these days. Apparently, the performance boosts from hitting 4K-aligned sectors can be very good. Cheers, Paul.
22.07.2010 06:05, Dan Langille wrote:> Create a new partition within that scheme: > > gpart add -b 34 -s SOMEVALUE -t freebsd-zfs ad0 > > Why '-b 34'? Randi pointed me to > http://en.wikipedia.org/wiki/GUID_Partition_Table where it explains what > the first 33 LBA are used for. It's not for us to use here.gpart is not so dumb to not protect this space. If you don't specify -b when creating first partition it automagically defaults to 34. -- Sphinx of black quartz judge my vow.
Thank you to all the helpful discussion. It's been very helpful and educational. Based on the advice and suggestions, I'm going to adjust my original plan as follows. NOTE: glabel will not be used. First, create a new GUID Partition Table partition scheme on the HDD: gpart create -s GPT ad0 Let's see how much space we have. This output will be used to determine SOMEVALUE in the next command. gpart show Create a new partition within that scheme: gpart add -b 1024 -s SOMEVALUE -t freebsd-zfs -l disk00 ad0 The -b 1024 ensures alignment on a 4KB boundary. SOMEVALUE will be set so approximately 200MB is left empty at the end of the HDD. That's part more than necessary to accommodate the different actualy size of 2TB HDD. Repeat the above with ad1 to get disk01. Repeat for all other HDD... Then create your zpool: zpool create bigtank gpt/disk00 gpt/disk02 ... etc This plan will be applied to an existing 5 HDD ZFS pool. I have two new empty HDD which will be added to this new array (giving me 7 x 2TB HDD). The array is raidz1 and I'm wondering if I want to go to raidz2. That would be about 10TB and I'm only using up 3.1TB at present. That represents about 4 months of backups. I do not think I can adjust the existing zpool on the fly. I think I need to copy everything elsewhere (i.e the 2 empty drives). Then start the new zpool from scratch. The risk: when the data is on the 2 spare HDD, there is no redundancy. I wonder if my friend Jerry has a spare 2TB HDD I could borrow for the evening. -- Dan Langille - http://langille.org/