What is a good MGS/MDT backup strategy if there is one? I was thinking of mounting the MGS/MDT partition on the MDS as ext3 and rsync it every 10 mins to another server. Would this work? What would happen in the 9th minute I lose my MDS, would I still be able to have a good copy? Any thoughts or ideas? Any thoughts or ideas? TIA
On Tue, 2008-08-05 at 01:12 -0400, Mag Gam wrote:> What is a good MGS/MDT backup strategy if there is one? > > I was thinking of mounting the MGS/MDT partition on the MDS as ext3 > and rsync it every 10 mins to another server. Would this work? What > would happen in the 9th minute I lose my MDS, would I still be able to > have a good copy? Any thoughts or ideas?Peter Braam answered a similar question and of course, the answer is in the archives. It was the second google hit on a search for "lustre mds backup". The answer is at: http://lists.lustre.org/pipermail/lustre-discuss/2006-June/001655.html Backup of the MDT is also covered in the manual in section 15 at http://manual.lustre.org/manual/LustreManual16_HTML/BackupAndRestore.html#50544703_pgfId-5529 Now, as for mounting the MDT as ext3 (you should actually use ldiskfs, not ext3) every 10 minutes, that means you are going to make your filesystem unavailable every 10 minutes as you CANNOT mount the MDT partition on more than one machine and we have not tested multiple mounting on a single machine with any degree of confidence. Of course Peter''s LVM snapshotting technique will allow you to mount snapshots which you can backup as you describe. But if you are going to have a whole separate machine with enough storage to mirror your MDT why not use something more active like DRBD and have a fully functional active/passive MDT failover strategy? While nobody in the Lustre Group has done any extensive testing of Lustre on DRBD, there have been a number of reports of success with it here on this list. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080805/8f43243c/attachment.bin
Brian: Thanks for the response. I actually seen this response before and was wondering if my technique would simply work. I guess not. I guess another question will be, if I take a snapshot every 10 mins and back it up. If I have a failure at 15th minute. Can I just simply restore my MDS to the previous snapshot and be with it? Ofcourse I will lose my 5 minutes of data, correct? TIA On Tue, Aug 5, 2008 at 12:08 PM, Brian J. Murrell <Brian.Murrell at sun.com> wrote:> On Tue, 2008-08-05 at 01:12 -0400, Mag Gam wrote: >> What is a good MGS/MDT backup strategy if there is one? >> >> I was thinking of mounting the MGS/MDT partition on the MDS as ext3 >> and rsync it every 10 mins to another server. Would this work? What >> would happen in the 9th minute I lose my MDS, would I still be able to >> have a good copy? Any thoughts or ideas? > > Peter Braam answered a similar question and of course, the answer is in > the archives. It was the second google hit on a search for "lustre mds > backup". The answer is at: > > http://lists.lustre.org/pipermail/lustre-discuss/2006-June/001655.html > > Backup of the MDT is also covered in the manual in section 15 at > > http://manual.lustre.org/manual/LustreManual16_HTML/BackupAndRestore.html#50544703_pgfId-5529 > > Now, as for mounting the MDT as ext3 (you should actually use ldiskfs, > not ext3) every 10 minutes, that means you are going to make your > filesystem unavailable every 10 minutes as you CANNOT mount the MDT > partition on more than one machine and we have not tested multiple > mounting on a single machine with any degree of confidence. > > Of course Peter''s LVM snapshotting technique will allow you to mount > snapshots which you can backup as you describe. > > But if you are going to have a whole separate machine with enough > storage to mirror your MDT why not use something more active like DRBD > and have a fully functional active/passive MDT failover strategy? While > nobody in the Lustre Group has done any extensive testing of Lustre on > DRBD, there have been a number of reports of success with it here on > this list. > > b. > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >
Also, what is the best way to test the backup? Other than really remove my MGS and restore it. Is there a better way to test this? TIA On Tue, Aug 5, 2008 at 6:37 PM, Mag Gam <magawake at gmail.com> wrote:> Brian: > > Thanks for the response. I actually seen this response before and was > wondering if my technique would simply work. I guess not. > > I guess another question will be, if I take a snapshot every 10 mins > and back it up. If I have a failure at 15th minute. Can I just simply > restore my MDS to the previous snapshot and be with it? Ofcourse I > will lose my 5 minutes of data, correct? > > TIA > > > On Tue, Aug 5, 2008 at 12:08 PM, Brian J. Murrell <Brian.Murrell at sun.com> wrote: >> On Tue, 2008-08-05 at 01:12 -0400, Mag Gam wrote: >>> What is a good MGS/MDT backup strategy if there is one? >>> >>> I was thinking of mounting the MGS/MDT partition on the MDS as ext3 >>> and rsync it every 10 mins to another server. Would this work? What >>> would happen in the 9th minute I lose my MDS, would I still be able to >>> have a good copy? Any thoughts or ideas? >> >> Peter Braam answered a similar question and of course, the answer is in >> the archives. It was the second google hit on a search for "lustre mds >> backup". The answer is at: >> >> http://lists.lustre.org/pipermail/lustre-discuss/2006-June/001655.html >> >> Backup of the MDT is also covered in the manual in section 15 at >> >> http://manual.lustre.org/manual/LustreManual16_HTML/BackupAndRestore.html#50544703_pgfId-5529 >> >> Now, as for mounting the MDT as ext3 (you should actually use ldiskfs, >> not ext3) every 10 minutes, that means you are going to make your >> filesystem unavailable every 10 minutes as you CANNOT mount the MDT >> partition on more than one machine and we have not tested multiple >> mounting on a single machine with any degree of confidence. >> >> Of course Peter''s LVM snapshotting technique will allow you to mount >> snapshots which you can backup as you describe. >> >> But if you are going to have a whole separate machine with enough >> storage to mirror your MDT why not use something more active like DRBD >> and have a fully functional active/passive MDT failover strategy? While >> nobody in the Lustre Group has done any extensive testing of Lustre on >> DRBD, there have been a number of reports of success with it here on >> this list. >> >> b. >> >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> >
Mag Gam wrote:> Also, what is the best way to test the backup? Other than really > remove my MGS and restore it. Is there a better way to test this?If you really care about the backups, you need to be brave. If you can''t remove the MDS and restore it, then something is wrong with your backup process. Many people seem to focus on the backup part and ignore the ''restore'' bit, so I definately reccomend a live test. That said, if you can bring up your backup MDT image on a separate node, you could configure that node as a failover MDS - this would require you to tunefs.lustre all the servers, and remount all the clients. Then you can test restore using a ''manual failover'' - and once you made the mount changes, you could repeat this test at will, without even halting the filesystem. Also, you would not have to ''remove'' your primary MDS, just stop that node. If your MDS _does_ die, the failover config will cause a slightly longer timeout (everybody will retry the alternate) but otherwise won''t impact you. cliffw> > TIA > > > On Tue, Aug 5, 2008 at 6:37 PM, Mag Gam <magawake at gmail.com> wrote: >> Brian: >> >> Thanks for the response. I actually seen this response before and was >> wondering if my technique would simply work. I guess not. >> >> I guess another question will be, if I take a snapshot every 10 mins >> and back it up. If I have a failure at 15th minute. Can I just simply >> restore my MDS to the previous snapshot and be with it? Ofcourse I >> will lose my 5 minutes of data, correct? >> >> TIA >> >> >> On Tue, Aug 5, 2008 at 12:08 PM, Brian J. Murrell <Brian.Murrell at sun.com> wrote: >>> On Tue, 2008-08-05 at 01:12 -0400, Mag Gam wrote: >>>> What is a good MGS/MDT backup strategy if there is one? >>>> >>>> I was thinking of mounting the MGS/MDT partition on the MDS as ext3 >>>> and rsync it every 10 mins to another server. Would this work? What >>>> would happen in the 9th minute I lose my MDS, would I still be able to >>>> have a good copy? Any thoughts or ideas? >>> Peter Braam answered a similar question and of course, the answer is in >>> the archives. It was the second google hit on a search for "lustre mds >>> backup". The answer is at: >>> >>> http://lists.lustre.org/pipermail/lustre-discuss/2006-June/001655.html >>> >>> Backup of the MDT is also covered in the manual in section 15 at >>> >>> http://manual.lustre.org/manual/LustreManual16_HTML/BackupAndRestore.html#50544703_pgfId-5529 >>> >>> Now, as for mounting the MDT as ext3 (you should actually use ldiskfs, >>> not ext3) every 10 minutes, that means you are going to make your >>> filesystem unavailable every 10 minutes as you CANNOT mount the MDT >>> partition on more than one machine and we have not tested multiple >>> mounting on a single machine with any degree of confidence. >>> >>> Of course Peter''s LVM snapshotting technique will allow you to mount >>> snapshots which you can backup as you describe. >>> >>> But if you are going to have a whole separate machine with enough >>> storage to mirror your MDT why not use something more active like DRBD >>> and have a fully functional active/passive MDT failover strategy? While >>> nobody in the Lustre Group has done any extensive testing of Lustre on >>> DRBD, there have been a number of reports of success with it here on >>> this list. >>> >>> b. >>> >>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>> >>> > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
Cliff White wrote:> Mag Gam wrote: >> Also, what is the best way to test the backup? Other than really >> remove my MGS and restore it. Is there a better way to test this? > > If you really care about the backups, you need to be brave. If you can''t > remove the MDS and restore it, then something is wrong with your backup > process. Many people seem to focus on the backup part and ignore the > ''restore'' bit, so I definately reccomend a live test. > > That said, if you can bring up your backup MDT image on a separate node, > you could configure that node as a failover MDS - this would require you > to tunefs.lustre all the servers, and remount all the clients. Then you > can test restore using a ''manual failover'' - and once you made the mount > changes, you could repeat this test at will, without even halting the > filesystem. Also, you would not have to ''remove'' your primary MDS, just > stop that node. > > If your MDS _does_ die, the failover config will cause a slightly longer > timeout (everybody will retry the alternate) but otherwise won''t impact > you.Just to be clear, there is a potential data loss issue due to the time delta between the backup and the live system. Any transactions in play that miss the snapshot could result in lost data, as the MDS will replay transaction logs and delete orphans on startup. So testing on your live system definately is for the brave. cliffw> > cliffw >> >> TIA >> >> >> On Tue, Aug 5, 2008 at 6:37 PM, Mag Gam <magawake at gmail.com> wrote: >>> Brian: >>> >>> Thanks for the response. I actually seen this response before and was >>> wondering if my technique would simply work. I guess not. >>> >>> I guess another question will be, if I take a snapshot every 10 mins >>> and back it up. If I have a failure at 15th minute. Can I just simply >>> restore my MDS to the previous snapshot and be with it? Ofcourse I >>> will lose my 5 minutes of data, correct? >>> >>> TIA >>> >>> >>> On Tue, Aug 5, 2008 at 12:08 PM, Brian J. Murrell >>> <Brian.Murrell at sun.com> wrote: >>>> On Tue, 2008-08-05 at 01:12 -0400, Mag Gam wrote: >>>>> What is a good MGS/MDT backup strategy if there is one? >>>>> >>>>> I was thinking of mounting the MGS/MDT partition on the MDS as ext3 >>>>> and rsync it every 10 mins to another server. Would this work? What >>>>> would happen in the 9th minute I lose my MDS, would I still be able to >>>>> have a good copy? Any thoughts or ideas? >>>> Peter Braam answered a similar question and of course, the answer is in >>>> the archives. It was the second google hit on a search for "lustre mds >>>> backup". The answer is at: >>>> >>>> http://lists.lustre.org/pipermail/lustre-discuss/2006-June/001655.html >>>> >>>> Backup of the MDT is also covered in the manual in section 15 at >>>> >>>> http://manual.lustre.org/manual/LustreManual16_HTML/BackupAndRestore.html#50544703_pgfId-5529 >>>> >>>> >>>> Now, as for mounting the MDT as ext3 (you should actually use ldiskfs, >>>> not ext3) every 10 minutes, that means you are going to make your >>>> filesystem unavailable every 10 minutes as you CANNOT mount the MDT >>>> partition on more than one machine and we have not tested multiple >>>> mounting on a single machine with any degree of confidence. >>>> >>>> Of course Peter''s LVM snapshotting technique will allow you to mount >>>> snapshots which you can backup as you describe. >>>> >>>> But if you are going to have a whole separate machine with enough >>>> storage to mirror your MDT why not use something more active like DRBD >>>> and have a fully functional active/passive MDT failover strategy? >>>> While >>>> nobody in the Lustre Group has done any extensive testing of Lustre on >>>> DRBD, there have been a number of reports of success with it here on >>>> this list. >>>> >>>> b. >>>> >>>> >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>>> >>>> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss > >
On Thu, 2008-08-07 at 10:51 -0700, Cliff White wrote:> > Just to be clear, there is a potential data loss issue due to the time > delta between the backup and the live system. Any transactions in play > that miss the snapshot could result in lost data, as the MDS will replay > transaction logs and delete orphans on startup. So testing on your live > system definately is for the brave.Indeed. There are a couple of alternatives to consider. I know your production MO will be to take an LVM snapshot of the running MDT and back that up, but if the MDT (i.e. filesystem) were shut down prior to the backup, what you restore should be an identical MDT which you could then start the filesystem against without the risks of in-play transactions and orphan deletion. But indeed it is not a 100% reproduction of what would happen restoring from an in-production backup. Alternatively, rather than trying to start the OSTs against the restored MDT you could simply do a filesystem level (i.e. ldiskfs) comparison of the restored MDT against the production MDT. Indeed, there are other variations that you could use to satisfy yourself that the restore worked. I would highly suggest you do any of this testing either on a testbed (which you could build with a VirtualBox virtual cluster) or on your production system before you put production data on it. It is good system deployment policy to have fully tested backup and restore policies before going live anyway. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20080807/2a42eb2c/attachment.bin
Very nice insight. Thanks Brian, Cliff, and Phil! On Thu, Aug 7, 2008 at 11:14 AM, Brian J. Murrell <Brian.Murrell at sun.com> wrote:> On Thu, 2008-08-07 at 10:51 -0700, Cliff White wrote: >> >> Just to be clear, there is a potential data loss issue due to the time >> delta between the backup and the live system. Any transactions in play >> that miss the snapshot could result in lost data, as the MDS will replay >> transaction logs and delete orphans on startup. So testing on your live >> system definately is for the brave. > > Indeed. There are a couple of alternatives to consider. I know your > production MO will be to take an LVM snapshot of the running MDT and > back that up, but if the MDT (i.e. filesystem) were shut down prior to > the backup, what you restore should be an identical MDT which you could > then start the filesystem against without the risks of in-play > transactions and orphan deletion. But indeed it is not a 100% > reproduction of what would happen restoring from an in-production > backup. > > Alternatively, rather than trying to start the OSTs against the restored > MDT you could simply do a filesystem level (i.e. ldiskfs) comparison of > the restored MDT against the production MDT. > > Indeed, there are other variations that you could use to satisfy > yourself that the restore worked. > > I would highly suggest you do any of this testing either on a testbed > (which you could build with a VirtualBox virtual cluster) or on your > production system before you put production data on it. It is good > system deployment policy to have fully tested backup and restore > policies before going live anyway. > > b. > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >