Ext3 users - Jun 2009 - How to optimize the cloning of a disk with a big empty ext3 partition?

Hello,

I need to duplicate a huge number of identical 8GB SD cards on which I
have 1GB of data at the beginning of the disk on various partitions and
then a 7GB ext3 partition which is rather empty (just a few MB of data).

The duplicator device enables me to select which sectors I can
binary-duplicate. I would love to divide by 8 my duplication time, by
duplicating only the first GB and then the beginning of the 7GB ext3
partition.

When I do a mkfs.ext3, I see that the super-block is written to various
locations of the partition which is not good for the optimization I
would like to do.

Anyone can advise how I could optimize my duplication? What are my
options? Would it be the same situation for ext2? Meaning where is the
journal written on the ext3 partition? Is there a single version or
multiple versions?

Thanks in advance for any answer or pointer,

Gregoire

On Fri, Jun 19, 2009 at 04:42:54PM -0700, Gregoire Gentil
wrote:
> Hello,
> 
> I need to duplicate a huge number of identical 8GB SD cards on which I
> have 1GB of data at the beginning of the disk on various partitions and
> then a 7GB ext3 partition which is rather empty (just a few MB of data).
> 
> The duplicator device enables me to select which sectors I can
> binary-duplicate. I would love to divide by 8 my duplication time, by
> duplicating only the first GB and then the beginning of the 7GB ext3
> partition.
> 
> When I do a mkfs.ext3, I see that the super-block is written to various
> locations of the partition which is not good for the optimization I
> would like to do.
The metadata for ext3 is scattered across the disk; it's not just the
superblock, but it's also parts of the inode table, and bitmap
allocation blocks, which must be initialized.

It's possible to determine the list of blocks that need to be
initialized, but it's going to be a rather long list.  If you need to
type the list of sectors into your duplicator, it's probably not
practical.  If on the other hand you can feed it a file with a set of
sector numbers, you could extract that list of blocks in use from the
filesystem (which listed by dumpe2fs, although not in the most
convenient format), and convert it to sector numbers counting from the
beginning of the disk (as opposed to block numbers counting from the
beginning of the partitions), and then feed the whole long list of
sector block ranges to your duplicator device.

						- Ted

A program like partimage will copy only the sectors that are already in use,
but it
requires that the destination partition be at least as big as the original
(otherwise
it would be required to relocate used sectors that are near the end of the
parttition).

The only real way around your problem would be to resize2fs to resize the
filesystem
(not the partition) to  a bit over 1GB At that point you could safely copy
only the first
Gig of the partition (whatever size you changed the filesystem to).  You
could then
(if you wanted to) resize the filesystem back to fill the partition.

If you just need the data from the partition archived, and don't need it to
be live,
then you can just use partimage to dump the used sectors, into a file...
When you
need to reconstitute the filesystem, however, partimage will want to create
a filesystem
the same size as the original.

On Fri, Jun 19, 2009 at 4:42 PM, Gregoire Gentil <gregoire at
gentil.com>wrote:
> Hello,
>
> I need to duplicate a huge number of identical 8GB SD cards on which I
> have 1GB of data at the beginning of the disk on various partitions and
> then a 7GB ext3 partition which is rather empty (just a few MB of data).
>
> The duplicator device enables me to select which sectors I can
> binary-duplicate. I would love to divide by 8 my duplication time, by
> duplicating only the first GB and then the beginning of the 7GB ext3
> partition.
>
> When I do a mkfs.ext3, I see that the super-block is written to various
> locations of the partition which is not good for the optimization I
> would like to do.
>
> Anyone can advise how I could optimize my duplication? What are my
> options? Would it be the same situation for ext2? Meaning where is the
> journal written on the ext3 partition? Is there a single version or
> multiple versions?
>
> Thanks in advance for any answer or pointer,
>
-- 
Stephen Samuel http://www.bcgreen.com  Software, like love,
778-861-7641                              grows when you give it away
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/ext3-users/attachments/20090619/4bbce425/attachment.htm>

--On 19 June 2009 16:42:54 -0700 Gregoire Gentil <gregoire at gentil.com> 
wrote:
> I need to duplicate a huge number of identical 8GB SD cards on which I
> have 1GB of data at the beginning of the disk on various partitions and
> then a 7GB ext3 partition which is rather empty (just a few MB of data).
>
> The duplicator device enables me to select which sectors I can
> binary-duplicate. I would love to divide by 8 my duplication time, by
> duplicating only the first GB and then the beginning of the 7GB ext3
> partition.
I am guessing here about your application, but how about:

1. Leave the 7GB partition unformatted in the SD card image, set the
   partition type in the partition table to a magic value
2. Make a gzip'd version of a disk image of that partition. Do this by
   using a loopback file system on a file the size of the partition
   which is initially full of a single magic byte/word. The resultant
   file should be tiny as the image will be mostly blocks full of the
   magic byte/word. Put this on another partition.
3. In the boot code for the device (I am assuming you can change that
   or better still it is on the CF device), put a little bit of code
   that
   a) checks the partition table for the magic number, and if set,
      do this:
   b) mounts the partition with the gzip image, extracts it,
      writes it (only writing blocks that are not full of the magic
      byte/word, so the process will be quick) to the new partition,
      then unmounts the partition with the gzip image so the boot
      process can continue
   c) change the partition table entry to ext3
   d) continue the boot process
   Of course this only happens on the first boot, and is only writing
   a few Mb of data.
4. Only duplicate the first 1GB of the disk

The program described is a few tens of lines of C linked with zlib.
It also gives you a relatively easy way to restore the partition
if you should ever need to (change partition table magic number back,
reboot). If this is the only r/w partition, that might be useful.

-- 
Alex Bligh