We just upgraded to 1.6.7 and previously we were using LVM snapshots to backup the MDT. Now after the jump to 1.6.7 the tar backups are HUGE. I must be missing something but i can''t find it in the documentation. My backups previously were in the MB''s, now i gave up when the tar file passed 20GB, when the MDT is only using 1.8GB of disk space. Here is what i am running roughly (it''s actually run from another server over ssh): lvcreate -s -L 4G MDS1/lvol0 -n backup mount -t ldiskfs /dev/MDS1/backup /srv/backups/lustre/mds-1 cd /srv/backups/lustre/mds-1 && getfattr -R -d -m ''.*'' -P . | gzip > $backupdir/ea.bak.gz cd /srv/backups/lustre/mds-1 && tar czfS - . | cat > $backupdir/ mdsbak.tgz I have tried it with and without the "S" on the tar, thinking that the sparse file handling was my problem. So can anyone point me in the correct direction for getting these LVM backups under control. Thomas Wakefield Jr. Systems Administrator COLA/IGES 301-902-1268 twake at cola.iges.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090325/54a04a53/attachment.html
On Mar 25, 2009, at 6:53 AM, Thomas Wakefield wrote:> We just upgraded to 1.6.7 and previously we were using LVM snapshots > to backup the MDT. Now after the jump to 1.6.7 the tar backups are > HUGE. I must be missing something but i can''t find it in the > documentation. My backups previously were in the MB''s, now i gave > up when the tar file passed 20GB, when the MDT is only using 1.8GB > of disk space.Johann, is the update of i_size in bz 11063 causing this? We saw the same behavior after upgrading to 1.6.7. j -- Jason Rappleye System Administrator NASA Advanced Supercomputing Division NASA Ames Research Center Moffett Field, CA 94035 (650) 604-2875 jason.rappleye at nasa.gov
The HUGE backup file size increase is due to the huge sparse files in
the MDT. A new future that got slipped into Lustre 1.6.7???
To reduce the size of the backup file use the "--sparse" option in
your
tar command. This option only reduces the backup file size but not the
time taken to tar up your backup as tar keeps reading a whole bunch of
zeros from /dev/zero before it determines that the file is sparse.
To speed up the backup process and to reduce the backup file size, I did
the following:
1. I reduced the size of the MDT partition from 150GB to 40GB.
2. Move the MDT over to the new partition and write ZEROs into all the
unused space on the new MDT partition after mounting the MDT as type
ldiskfs and using a script like:
let i=1
while [ $i -lt 36 ] ;
do
dd if=/dev/zero of=/mnt/new-mdt/tmp/gbfile$i bs=128k count=8192
let i="$i+1"
done
3. Create the backup using the following command after making a LVM
snapshot of the mounted MDT:
dd if=/dev/lustrevol/mdtsnapshot | gzip -c > mdt_backup_file
Hope this helps!
Nirmal
Hi, On Thu, Apr 02, 2009 at 10:57:51AM -0700, Jason Rappleye wrote:> Johann, is the update of i_size in bz 11063 causing this? We saw the > same behavior after upgrading to 1.6.7.Yes, it is likely the cause. i_size is now updated on file close on the MDT (sparse file, no data block allocated). This is done for tools like e2scan. Johann
On Thu, Apr 02, 2009 at 01:27:52PM -0500, Nirmal Seenu wrote:> To reduce the size of the backup file use the "--sparse" option in your > tar command. This option only reduces the backup file size but not the > time taken to tar up your backup as tar keeps reading a whole bunch of > zeros from /dev/zero before it determines that the file is sparse.In this particular case, no blocks are allocated, so tar actually does not need to read the file content. This is something we can implement in our lustre-aware tar (bug 15524). I''m going to file a bug for this. Cheers, Johann
On Apr 02, 2009 21:09 +0200, Johann Lombardi wrote:> On Thu, Apr 02, 2009 at 01:27:52PM -0500, Nirmal Seenu wrote: > > To reduce the size of the backup file use the "--sparse" option in your > > tar command. This option only reduces the backup file size but not the > > time taken to tar up your backup as tar keeps reading a whole bunch of > > zeros from /dev/zero before it determines that the file is sparse. > > In this particular case, no blocks are allocated, so tar actually does not > need to read the file content. This is something we can implement in our > lustre-aware tar (bug 15524). I''m going to file a bug for this.You''re right. A filesystem-aware backup tool like "dump" would hopefully not back up all of this data. Similarly, if tar were to use the "FIEMAP" feature present in ldiskfs for 1.6.7+ and 2.6.28+ upstream kernels it could also avoid reading any of the file blocks. I would recommend that we also make a /proc tunable that can enable or disable the MDS file size data, for sites that are not using e2scan, but are doing MDS backups with tar instead of dump/LVM/raw. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.