Frederik Ferner
2010-Sep-02 13:42 UTC
[Lustre-discuss] MDT backup (using tar) taking very long
Hi List, we are currently reviewing our backup policy for our Lustre file system as backups of the MDT are taking longer and longer. So far we are creating a LVM snapshot of our MDT, mount this via ldiskfs, run getfattr and getfacl followed by tar (RHEL5 version), basically following the instructions from the manual. The tar options include --sparse and --numeric-owner. At the moment I''ve got a backup running where the tar process started on Tuesday, so it has now been running more than 24h. Including the getfattr and the getfacl calls (running in parallel) the whole backup has so far been running for more than 48h to backup the MDT for a 700GB MDT for a 214TB Lustre file system. The tar file created so far is about 2GB compressed with gzip. Tar is currently using anything between 30% and 100% cpu according to top, gzip is below 1% cpu usage, overall the MDS is fairly idle, load is about 1.2 on a 8 core machine, top reports this for the cpus. <snip> Cpu(s): 4.2%us, 4.5%sy, 0.0%ni, 85.8%id, 5.2%wa, 0.0%hi, 0.2%si, 0.0%st </snip> vmstat is not showing any I/O worth mentioning, a few (10-1000) blocks per second. Some file system details for the Lustre file system below. The MDS is running lustre 1.6.7.2.ddn3.5 plus a patch for bz #22820 on RHEL5. [bnh65367 at cs04r-sc-com01-18 ~]$ lfs df -h UUID bytes Used Available Use% Mounted on lustre01-MDT0000_UUID 699.9G 22.1G 677.8G 3% /mnt/lustre01[MDT:0] [snip] filesystem summary: 214.9T 146.6T 68.3T 68% /mnt/lustre01 [bnh65367 at cs04r-sc-com01-18 ~]$ lfs df -ih UUID Inodes IUsed IFree IUse% Mounted on lustre01-MDT0000_UUID 200.0M 71.0M 129.0M 35% /mnt/lustre01[MDT:0] [snip] filesystem summary: 200.0M 71.0M 129.0M 35% /mnt/lustre01 Is this comparable to the backup times other people experience using tar? Could this be because tar has to read the whole file (all zeros) in before deciding that this is a sparse file? For comparison a backup using dd and gzip did ''only'' takes about 8h and gzip was using 100% of one cpu core for all of that time, so using a faster compression algorithm this seems a much better option. Are there any dangerous downsides to this approach that I have missed? Kind regards, Frederik -- Frederik Ferner Computer Systems Administrator phone: +44 1235 77 8624 Diamond Light Source Ltd. mob: +44 7917 08 5110 (Apologies in advance for the lines below. Some bits are a legal requirement and I have no control over them.)
I''d use the dd/gzip option, though you may want to write it to another system, and have that system do the compression. If you''re going that route, you might want to run fsck on the dd''d image before compression, to make sure any errors are fixed. -----Original Message----- From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of Frederik Ferner Sent: Thursday, September 02, 2010 9:42 AM To: lustre-discuss at lists.lustre.org Subject: [Lustre-discuss] MDT backup (using tar) taking very long Hi List, we are currently reviewing our backup policy for our Lustre file system as backups of the MDT are taking longer and longer. So far we are creating a LVM snapshot of our MDT, mount this via ldiskfs, run getfattr and getfacl followed by tar (RHEL5 version), basically following the instructions from the manual. The tar options include --sparse and --numeric-owner. At the moment I''ve got a backup running where the tar process started on Tuesday, so it has now been running more than 24h. Including the getfattr and the getfacl calls (running in parallel) the whole backup has so far been running for more than 48h to backup the MDT for a 700GB MDT for a 214TB Lustre file system. The tar file created so far is about 2GB compressed with gzip. Tar is currently using anything between 30% and 100% cpu according to top, gzip is below 1% cpu usage, overall the MDS is fairly idle, load is about 1.2 on a 8 core machine, top reports this for the cpus. <snip> Cpu(s): 4.2%us, 4.5%sy, 0.0%ni, 85.8%id, 5.2%wa, 0.0%hi, 0.2%si, 0.0%st </snip> vmstat is not showing any I/O worth mentioning, a few (10-1000) blocks per second. Some file system details for the Lustre file system below. The MDS is running lustre 1.6.7.2.ddn3.5 plus a patch for bz #22820 on RHEL5. [bnh65367 at cs04r-sc-com01-18 ~]$ lfs df -h UUID bytes Used Available Use% Mounted on lustre01-MDT0000_UUID 699.9G 22.1G 677.8G 3% /mnt/lustre01[MDT:0] [snip] filesystem summary: 214.9T 146.6T 68.3T 68% /mnt/lustre01 [bnh65367 at cs04r-sc-com01-18 ~]$ lfs df -ih UUID Inodes IUsed IFree IUse% Mounted on lustre01-MDT0000_UUID 200.0M 71.0M 129.0M 35% /mnt/lustre01[MDT:0] [snip] filesystem summary: 200.0M 71.0M 129.0M 35% /mnt/lustre01 Is this comparable to the backup times other people experience using tar? Could this be because tar has to read the whole file (all zeros) in before deciding that this is a sparse file? For comparison a backup using dd and gzip did ''only'' takes about 8h and gzip was using 100% of one cpu core for all of that time, so using a faster compression algorithm this seems a much better option. Are there any dangerous downsides to this approach that I have missed? Kind regards, Frederik -- Frederik Ferner Computer Systems Administrator phone: +44 1235 77 8624 Diamond Light Source Ltd. mob: +44 7917 08 5110 (Apologies in advance for the lines below. Some bits are a legal requirement and I have no control over them.) _______________________________________________ Lustre-discuss mailing list Lustre-discuss at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Bernd Schubert
2010-Sep-02 15:27 UTC
[Lustre-discuss] MDT backup (using tar) taking very long
On Thursday, September 02, 2010, Frederik Ferner wrote:> Hi List, > > we are currently reviewing our backup policy for our Lustre file system > as backups of the MDT are taking longer and longer. >Yes, that is due to the size-on-mds feature, which was introduced into 1.6.7.2. See bug https://bugzilla.lustre.org/show_bug.cgi?id=21376 It has a patch, that also got accepted in upstream tar last week. You may find updated RHEL5 tar packages on my home page: http://www.pci.uni-heidelberg.de/tc/usr/bernd/downloads/ Cheers, Bernd -- Bernd Schubert DataDirect Networks
Frederik Ferner
2010-Sep-02 16:35 UTC
[Lustre-discuss] MDT backup (using tar) taking very long
Bernd Schubert wrote:> On Thursday, September 02, 2010, Frederik Ferner wrote: >> we are currently reviewing our backup policy for our Lustre file system >> as backups of the MDT are taking longer and longer.> Yes, that is due to the size-on-mds feature, which was introduced into > 1.6.7.2. See bug > > https://bugzilla.lustre.org/show_bug.cgi?id=21376 > > It has a patch, that also got accepted in upstream tar last week. You may find > updated RHEL5 tar packages on my home page:Thanks, I''ll give that a go. (Any chance of adding the SRPM to your download page?) Cheers, Frederik -- Frederik Ferner Computer Systems Administrator phone: +44 1235 77 8624 Diamond Light Source Ltd. mob: +44 7917 08 5110 (Apologies in advance for the lines below. Some bits are a legal requirement and I have no control over them.)
Bernd Schubert
2010-Sep-02 17:04 UTC
[Lustre-discuss] MDT backup (using tar) taking very long
On Thursday, September 02, 2010, Frederik Ferner wrote:> Bernd Schubert wrote: > > On Thursday, September 02, 2010, Frederik Ferner wrote: > >> we are currently reviewing our backup policy for our Lustre file system > >> as backups of the MDT are taking longer and longer. > > > > Yes, that is due to the size-on-mds feature, which was introduced into > > 1.6.7.2. See bug > > > > https://bugzilla.lustre.org/show_bug.cgi?id=21376 > > > > It has a patch, that also got accepted in upstream tar last week. You may > > find > > > updated RHEL5 tar packages on my home page: > Thanks, I''ll give that a go. (Any chance of adding the SRPM to your > download page?)I don''t like SRPM too much, I updloaded a tar.bz2 instead. It is a hg repository and mq managed, so patches (including those RedHat added) are in .hg/patches. You will another bugfix there compared to -sun packages. Cheers, Bernd -- Bernd Schubert DataDirect Networks
Frederik Ferner
2010-Sep-08 09:35 UTC
[Lustre-discuss] MDT backup (using tar) taking very long
Hi Bernd, Frederik Ferner wrote:> Bernd Schubert wrote: >> On Thursday, September 02, 2010, Frederik Ferner wrote: >>> we are currently reviewing our backup policy for our Lustre file system >>> as backups of the MDT are taking longer and longer. > >> Yes, that is due to the size-on-mds feature, which was introduced into >> 1.6.7.2. See bug >> >> https://bugzilla.lustre.org/show_bug.cgi?id=21376 >> >> It has a patch, that also got accepted in upstream tar last week. You may find >> updated RHEL5 tar packages on my home page: > > Thanks, I''ll give that a go.I can now report that the backup was faster using this new version of tar. It is still not really fast, though. the backup still takes nearly 24h (that is running getfattr followed by tar...).> (Any chance of adding the SRPM to your download page?)Thanks for the tar file. I seem to remember that the separate getfattr call during the backup was not required anymore with the new version of tar. A quick look in the code seems to confirm this, at least it seems to be able to store the "trusted" extended attributes. Can you confirm if ''tar --xattr --acl'' on the ldiskfs mounted MDT is sufficient to store all required extended attributes (and ACLs)? Thanks, Frederik -- Frederik Ferner Computer Systems Administrator phone: +44 1235 77 8624 Diamond Light Source Ltd. mob: +44 7917 08 5110 (Apologies in advance for the lines below. Some bits are a legal requirement and I have no control over them.)
Andreas Dilger
2010-Sep-09 04:44 UTC
[Lustre-discuss] MDT backup (using tar) taking very long
On 2010-09-08, at 3:35, Frederik Ferner <frederik.ferner at diamond.ac.uk> wrote:>>> >>> >> > I can now report that the backup was faster using this new version of > tar. It is still not really fast, though. the backup still takes nearly > 24h (that is running getfattr followed by tar...).That surprises me a bit, unless you have a huge number of files. You should be able to do backups at least at the stat rate (5000/sec = 18M/hr is not unreasonable for almost any MDS).> I seem to remember that the separate getfattr call during the backup was > not required anymore with the new version of tar. A quick look in the > code seems to confirm this, at least it seems to be able to store the > "trusted" extended attributes. > > Can you confirm if ''tar --xattr --acl'' on the ldiskfs mounted MDT is > sufficient to store all required extended attributes (and ACLs)?Yes, that should work. It always is worthwhile to verify that they are restored correctly, however. You can just start a restore in some temp ext3 filesyste, interrupt it after some files in /ROOT have been restored, and use getfattr to verify the trusted.lov xattrs are restored. Cheers, Andreas