As recently as last October, the best official advice was to make a 64kB boot partition. https://wiki.freebsd.org/action/diff/RootOnZFS/GPTZFSBoot/Mirror?action=diff&rev1=16&rev2=17 Now that turns out to be absolutely terrible advice and some people (like me) have dozens of machines that will never be upgradable to FreeBSD 11 or higher. It looks like there is no reasonable method of upgrade that doesn't involve replacing every hard disk on every machine (that's hundred of disks) with larger models. I use a zvol for swap, so I can't make swap smaller to solve the problem. I started with FreeBSD 4.1 and in 16 years... sigh... The ashift pain some years ago was also caused by FreeBSD default recommendations and settings not anticipating future needs quickly enough. But this mess now is completely self-inflicted foot shooting. 1. Why is the recommendation now 128kB and not much much higher? When that limit is broken in a couple of years, will there be another round of annoyed users? Is someone concerned that ZFS users are running hard disks over under 500Mb and need to save space? Surely the recommendation should be 512kB? 2. Is there any possible short term future where ZFS volumes can be shrunk, or will I be replacing every hard disk (or rebuilding the machine from scratch)? 3. Is there any possibility of getting a gptzfsboot which is 64kB but missing certain features I might not need? eg. a RAIDZ2 version that skips support for RAIDZ3 4. Will support be added to freebsd-update to warn users BEFORE they try to upgrade and kill their system? Please cc me, I'm not subscribed. Ari Maniatis -- --------------------------> Aristedes Maniatis CEO, ish https://www.ish.com.au GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 163 bytes Desc: OpenPGP digital signature URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20170129/6fb80d7b/attachment.sig>
On Sat, Jan 28, 2017 at 9:15 PM, Aristedes Maniatis <ari at ish.com.au> wrote:> As recently as last October, the best official advice was to make a 64kB boot partition. > > https://wiki.freebsd.org/action/diff/RootOnZFS/GPTZFSBoot/Mirror?action=diff&rev1=16&rev2=17 > > > Now that turns out to be absolutely terrible advice and some people (like me) have dozens of machines that will never be upgradable to FreeBSD 11 or higher. It looks like there is no reasonable method of upgrade that doesn't involve replacing every hard disk on every machine (that's hundred of disks) with larger models. I use a zvol for swap, so I can't make swap smaller to solve the problem. > > I started with FreeBSD 4.1 and in 16 years... sigh...If things boot now, you won't have to upgrade the boot blocks. It isn't done by default, and the older boot blocks will boot newer /boot/loader versions.> The ashift pain some years ago was also caused by FreeBSD default recommendations and settings not anticipating future needs quickly enough. But this mess now is completely self-inflicted foot shooting. > > > 1. Why is the recommendation now 128kB and not much much higher? When that limit is broken in a couple of years, will there be another round of annoyed users? Is someone concerned that ZFS users are running hard disks over under 500Mb and need to save space? Surely the recommendation should be 512kB?Unless you are running on tiny disks, you should use 512kB for maximum future proofing. Given the bloat that's happened in boot1/boot2 over the years, this is the only sensible default.> 2. Is there any possible short term future where ZFS volumes can be shrunk, or will I be replacing every hard disk (or rebuilding the machine from scratch)?Not easily. However, there's several options available to you: (1) not upgrading the boot partition (2) shrinking a swap partition to snag some space (3) putting a larger boot partition at the 'end' of the disk where there's usually runt sectors due to how gpart (bogusly imho, but I've not been successful at advocating this viewpoint) rounds. There's up to an entire cylinder at the end (though LBAs make CHS bogus), which might be useful. It's also possible to move the start of the boot partition to a smaller LBA, which gives us more than the 44k we currently have. We may also be able to write a smaller GPT area if we use only a couple of partitions on the disk. The only caveat in doing this is that pmbr reads the entire partition in, and it has to be less than about 534k in size. There's no size header on it.> 3. Is there any possibility of getting a gptzfsboot which is 64kB but missing certain features I might not need? eg. a RAIDZ2 version that skips support for RAIDZ364k is likely an unrealistic goal. We're crowding that limit today when we remove the SKEIN code. Removing it just removes the ability to boot encrypted ZFS volumes. However, the boot blocks on the disks aren't normally upgraded, and if they are booting your system ZFS volumes today, they are necessarily not encrypted. While the boot blocks fit into 64k today with some crazy gymnastics, upgrades in compiler technology likely will make this not a realistic goal (or alternatively, a lot of work).> 4. Will support be added to freebsd-update to warn users BEFORE they try to upgrade and kill their system?Boot partitions usually aren't upgraded, so I'm not sure this is an issue. The project doesn't upgrade boot blocks by default. While there have been some events in the past that necessitated the upgrade, they have been rare and kinda optional. In this case, there's no compelling reason to upgrade the boot blocks that I can see... A quick look at freebsd-update shows no calls to gpart or dd, which is necessary to change them. Warner> > > Please cc me, I'm not subscribed. > > > Ari Maniatis > > > -- > --------------------------> > Aristedes Maniatis > CEO, ish > https://www.ish.com.au > GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A >
On Sun, Jan 29, 2017 at 03:15:19PM +1100, Aristedes Maniatis wrote:> As recently as last October, the best official advice was to make a 64kB boot partition. > > https://wiki.freebsd.org/action/diff/RootOnZFS/GPTZFSBoot/Mirror?action=diff&rev1=16&rev2=17 > > > Now that turns out to be absolutely terrible advice and some people (like me) have dozens of machines that will never be upgradable to FreeBSD 11 or higher. It looks like there is no reasonable method of upgrade that doesn't involve replacing every hard disk on every machine (that's hundred of disks) with larger models. I use a zvol for swap, so I can't make swap smaller to solve the problem. > > I started with FreeBSD 4.1 and in 16 years... sigh... > > The ashift pain some years ago was also caused by FreeBSD default recommendations and settings not anticipating future needs quickly enough. But this mess now is completely self-inflicted foot shooting. > > > 1. Why is the recommendation now 128kB and not much much higher? When that limit is broken in a couple of years, will there be another round of annoyed users? Is someone concerned that ZFS users are running hard disks over under 500Mb and need to save space? Surely the recommendation should be 512kB? > > 2. Is there any possible short term future where ZFS volumes can be shrunk, or will I be replacing every hard disk (or rebuilding the machine from scratch)?It is highly unlikely that ZFS volumes will be able to be reduced in size even in the long term. I believe that requires a piece of work that has been rated as very difficult to do without violating layering policies inside the ZFS code. The alternative is, assuming you have a pool with redundancy (e.g. mirror) is to do a backup, drop one half of the mirror, create a new pool on the now unused disk, zfs send | zfs receive, boot from the new pool and then drop the old pool and add the disk to the mirror It's a pain and a bit of a shuffle but it's possible. I did it on my server once when I found that FreeBSD 9 didn't detect the disks as 4k and the alignment was all wrong. I worked through the procedure in a VM to validate it first, but found that in production I'd managed to hard code the boot pool name in /boot/loader.conf which meant that it didn't reboot and use the bootfs flag on the pool, it just sat at the "Cannot mount root" prompt. Took me a while to find that loader.conf setting and kill it. Regards, Gary> > 3. Is there any possibility of getting a gptzfsboot which is 64kB but missing certain features I might not need? eg. a RAIDZ2 version that skips support for RAIDZ3 > > 4. Will support be added to freebsd-update to warn users BEFORE they try to upgrade and kill their system? > > > > Please cc me, I'm not subscribed. > > > Ari Maniatis > > > -- > --------------------------> > Aristedes Maniatis > CEO, ish > https://www.ish.com.au > GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A >