I've mentioned this problem before but put off doing anything about it
and maybe now someone can suggest the best solution.
I have a 3-member RAID1 set where one of the members is periodically
swapped and rotated offsite. The filesystem contains a backuppc archive
which has millions of hardlinks that make it impractical to copy with a
file-oriented approach. The current filesystem is ext3 with one
partition that uses the entire disk capacity (no lvm). It works as is,
but...
I'd like to use a laptop size drive for the swapped member and the only
ones available that match the size have 4k sectors. I have swappable,
trayless SATA bays available for both drive sizes. The problem is that
with the current partition layout, the drive with 4k sectors takes more
than a day to re-sync even though on read access the speed is a match
for the full sized drives that sync in a few hours.
My questions for any filesystem experts are:
Is there a way to adjust the existing md partitions to get the right
alignment for 4k sectors without having to do a file-oriented copy to
new partitions? A resize + a dd copy to shift the position might be
feasible time-wise if that would work.
Is it worth converting to ext4?
Is there a difference between doing this on 5.6 or 6.x?
If I start over from scratch with 6.x, will the partitioning tools
automatically align for 4k sector drives (with/without lvm?)?
--
Les Mikesell
lesmikesell at gmail.com
On Mon, 25 Jul 2011, Les Mikesell wrote:> My questions for any filesystem experts are: > > Is there a way to adjust the existing md partitions to get the right > alignment for 4k sectors without having to do a file-oriented copy to > new partitions? A resize + a dd copy to shift the position might be > feasible time-wise if that would work.no expert here, but I have the scars across my back from pulling arrows out, as a pioneer We have hit the issue on our storage backend which runs ext4, and on some of our dom0 built before the 4k sector alignment was generally acknowledged and known to be potentially in play We have some non-conformant units, and after seaching, concluded that a 'wipe and rebuild' was the most time efficient process for us -- YMMV> Is it worth converting to ext4?ext4 is pleasant in some large filesystem cases, but probably overkill as a blanket option. Certainly it is 'wayy overkill for domU as a general rule, as it makes for a more fragile image in the sense that generic tools are less likely to work without higher version and skill levels when a filesystem gets horked up and a repair expedition has to be mounted ... we had an issue that a 'dirty' filesystem that would not fsck kept showing up in a nightly backup exception report, and ended up manually repairing what should have been able to be repaired automatically> Is there a difference between doing this on 5.6 or 6.x?in C5, it took extra effort to use the technology preview; in C6 it is natively available> If I start over from scratch with 6.x, will the partitioning tools > automatically align for 4k sector drives (with/without lvm?)?no idea if gparted does this by default -- it does not in all versions; certainly fdisk did not -- 4k alignment is on our deployment checklist, and we are manually checking partitioning to make sure, when we are rebuilding boxes -- Russ herrold
On Mon, Jul 25, 2011 at 1:10 PM, Les Mikesell <lesmikesell at gmail.com> wrote:> I've mentioned this problem before but put off doing anything about it > and maybe now someone can suggest the best solution. > > I have a 3-member RAID1 set where one of the members is periodically > swapped and rotated offsite. ?The filesystem contains a backuppc archive > which has millions of hardlinks that make it impractical to copy with a > file-oriented approach. ?The current filesystem is ext3 with one > partition that uses the entire disk capacity (no lvm). ?It works as is, > but... > > I'd like to use a laptop size drive for the swapped member and the only > ones available that match the size have 4k sectors. ?I have swappable, > trayless SATA bays available for both drive sizes. ?The problem is that > with the current partition layout, the drive with 4k sectors takes more > than a day to re-sync even though on read access the speed is a match > for the full sized drives that sync in a few hours. > > My questions for any filesystem experts are: > > Is there a way to adjust the existing md partitions to get the right > alignment for 4k sectors without having to do a file-oriented copy to > new partitions? ?A resize + a dd copy to shift the position might be > feasible time-wise if that would work. > > Is it worth converting to ext4? > > Is there a difference between doing this on 5.6 or 6.x? > > If I start over from scratch with 6.x, will the partitioning tools > automatically align for 4k sector drives (with/without lvm?)? > > -- > ? Les Mikesell > ? ?lesmikesell at gmail.comI've wondered many times, though haven't tried it, if the issues with hard links and backuppc could be solved by using a container file with a loopback mount, and then that file could be moved around as needed without running into hard-link issues. In this case, you could format the external drive in the optimal mode for 4k sectors, then create a container file and mount it using loopback. Then add the loopback device to the mdraid and have it sync. -? Brian Mathis ?-
On 07/25/2011 10:10 AM, Les Mikesell wrote:> I've mentioned this problem before but put off doing anything about it > and maybe now someone can suggest the best solution. > > I have a 3-member RAID1 set where one of the members is periodically > swapped and rotated offsite. The filesystem contains a backuppc archive > which has millions of hardlinks that make it impractical to copy with a > file-oriented approach. The current filesystem is ext3 with one > partition that uses the entire disk capacity (no lvm). It works as is, > but... > > I'd like to use a laptop size drive for the swapped member and the only > ones available that match the size have 4k sectors. I have swappable, > trayless SATA bays available for both drive sizes. The problem is that > with the current partition layout, the drive with 4k sectors takes more > than a day to re-sync even though on read access the speed is a match > for the full sized drives that sync in a few hours. > > My questions for any filesystem experts are: > > Is there a way to adjust the existing md partitions to get the right > alignment for 4k sectors without having to do a file-oriented copy to > new partitions? A resize + a dd copy to shift the position might be > feasible time-wise if that would work. > > Is it worth converting to ext4? > > Is there a difference between doing this on 5.6 or 6.x? > > If I start over from scratch with 6.x, will the partitioning tools > automatically align for 4k sector drives (with/without lvm?)? >For LVM's see the --dataalignment and --dataalignmentoffset options. For md devices, my understanding is that the raid superblock is at the end of the partition, so the data is aligned with wherever the partition starts. I verified this using: hexdump /dev/md1 | head -6 hexdump /dev/sda4 | head -6 Nataraj
On 2011-07-25 19:10, Les Mikesell wrote:> My questions for any filesystem experts are: > > Is there a way to adjust the existing md partitions to get the right > alignment for 4k sectors without having to do a file-oriented copy to > new partitions? A resize + a dd copy to shift the position might be > feasible time-wise if that would work.I think so. Your partition starts at sector 63, you need anything divisible by 8, so: - 64 is my expectation, - 56 is a fallback solution if the partition does not fit on the disk with 64 sector offset - 2048 would be perfect (1M alignment is currently preferred by Centos 6 and many other OSs) To be on the safe side, take the disk out of the array (mdadm -f /dev/md0 /dev/sdX1 ; mdadm -r /dev/md0 /dev/sdX1) and clear superblock using mdadm --zero-superblock /dev/sdX1. Then repartition the disk using fdisk using the following commands: fdisk /dev/sdX u -- display units are sectors c -- no DOS compatibility (== no cyllinder rounding, you definitely want that) o -- new dos partition table n -- new partition p -- primary 1 -- partition 1 64 -- starting offset 1465144065 -- exact size here, because (just to be on the safe side) you do not want to have a larger partition on a rescue disk than on a base disk. Your partition sdh1 has 732572001 1k-blocks, as you wrote in one e-mail, multiply this by 2 (sectors) add 64, (starting offset) subtract 1 because the offset is inclusive. You get 2*732572001+64-1=1465144065. If fdisk complains that this is too much then offset 64 cannot be used and you need to repeat the procedure using offset 56 (don't forget to recalculate ending sector). t -- type fd -- linux raid autodetect w mdadm -a /dev/mdX /dev/sdX1 And everything should be fine.> Is it worth converting to ext4?I don't know.> Is there a difference between doing this on 5.6 or 6.x?Yes, Centos 6 by default aligns partitions on the disk automatically on 1MiB (2048sectors) boundaries. C6 LVM also aligns lv's on 1MiB boundaries relative to the pv start. Finally md in Centos 6 uses 512KiB chunks and aligns data on this boundary (default md superblock in Centos 6 Installer is 1.1 so it is on beginning of the partition) so it is also OK.> If I start over from scratch with 6.x, will the partitioning tools > automatically align for 4k sector drives (with/without lvm?)? >Yes. But I always check that to be sure :) Andrzej
On Tuesday, July 26, 2011 05:21:58 AM John Doe wrote:> From: John R Pierce <pierce at hogranch.com> > > thats by "cylinder", which is an old MSDOS legacy thing. I believe > > parted and probably some other programs let you partition by sector instead. > In my kickstart pre script, I use: > ... | sfdisk -H $HEADS -S $SECTORS -uS --force -L $DEVICE > For SSDs, I saw the recommended respective values: 224 56 (or 32 32) > fdisk also has -u for sectors unit, and -H/-S to force a fake geometry.> But I must admit that I am still a bit confused with all these alignments...The key thing is to be sector aligned per physical drive; align to eight sector blocks; a starting sector of 56 would work. With RAID and LVM alignment to chunks or stripes is desireable. Forget CHS specifications; they haven't been valid for years anyway; think LBA and only LBA and you'll be fine. No drive made actually has 255 heads anyway.... or a constant 63 sectors per track, either, for that matter. All mechanical hard drives made these days employ ZBR and have a variable number of sectors per track, less than ten (or 12, in the case of some 15K RPM FC and SCSI drives that I know about; have some 15KRPM 36GB SCSI drives with six physical platters, 12 genuine physical heads, all in a half-height 3.5 inch form-factor) heads, and many thousands of cylinders. SSDs don't even have heads or tracks, and thus those specifications are meaningless and need to just go away. It's LBA all the way, and the critical alignment is to erase-block size. See the following articles for more, and better, information that goes into a lot more detail than I have time to do: http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/index.html?ca=dgr-lnxw074KB-Disksdth-LX http://www.ocztechnologyforum.com/forum/showthread.php?48309-Partition-alignment-importance-under-Windows-XP-%2832-bit-and-64-bit%29..why-it-helps-with-stuttering-and-increases-drive-working-life.&p=335049#post335049 (yes, the thread says windows, but the particular post is about Linux) http://www.tcpdump.com/kb/os/windows/disk-alignment/into.html (has some good illustrations that are relevant on Linux, even though the article is about Windows) And there are more; those were all on the first page of a Google search for the terms 'sector alignment linux' (no quotes).