See comments inline. On Sun, 10 Feb 2019 at 16:33, Rowland Penny via samba <samba at lists.samba.org> wrote:> On Sun, 10 Feb 2019 14:13:27 +0100 > Viktor Trojanovic via samba <samba at lists.samba.org> wrote: > > > I'm currently reviewing my own backup strategy for Samba and I > > realize it is not in line with best practices provided in the Wiki. ( > > https://wiki.samba.org/index.php/Back_up_and_Restoring_a_Samba_AD_DC) > > Said best practices, however, seem a bit like a nightmare to me. > > > > Assuming the AD is gone and you want to restore just one DC, and you > > want things to look just as they did before the crash, the process > > according to the Wiki looks as follows: > > > > 1. Install a Samba DC on a new (!) temporary host and provision the > > domain, just like you would when doing a new install from scratch. > > That task alone is tremendous. > > 2. Stop Samba and restore the AD from backup to this domain not (!) > > into the default Samba folder, advise Samba accordingly when starting > > it. 3. On the original host, set up a Samba DC and join the domain. > > 4. If GPO or scripts exist on sysvol, manually set up sysvol > > replication to get them to the original DC. > > 5. Remove the temporary host. > > > > Just... wow. :) > > Tend to agree with you, the wiki page asks this question 'So which > backup should I use?' It then goes on to enumerate 5 different reasons > why you would need a backup and seems to totally miss the point. Your > domain has gone down and it is headless chicken time ;-) > All you would want to do is to get your domain back up again as quickly > as possible. >Yes. So I'm really glad I haven't encountered that page in a moment of true need! :-)> I think you would only do '1' if you wanted to rename the domain. > > Not sure where you got restoring into a different folder from, I > thought the restore put everything back to where it came from. > >I got both of this from the section "Restoring the backup-file" in the wiki. It says that if you're going to use the command "samba-tool domain backup restore", you *must not* specify a DC that has previously existed. Further, with regards to the files itself, it says that the Samba team recommends "that you restore the domain database into a different targetdir, and then use the '-s' option when running samba".> You shouldn't have to do '4', the backup contains a copy of sysvol and > smb.conf, so you should be able to restore to the DC it came from, it > would just have to be the only DC and all DC's would have to be > stopped, it would probably be better to rename the old DC before > carrying out the restore. > >As mentioned, at least to my understanding of the the wiki a restore of the original DC is not possible using the backup made from it. Are you saying that restoring to a "new DC" is as simple as changing the hostname of "DC1" to "DC1_1"? DNS, GPO, smb.conf would all automatically refer to the new hostname after the restore?> > > > Isn't there a simpler way of doing this? Namely, if all the restore > > operations are done offline anyway, why is it frowned upon to simply > > do everything on the original DC, i.e. forgo the temporary host, > > overwrite the configuration files (/etc/samba) and the local Samba > > folder (e.g. /var/lib/samba) with what's in the backup and be done > > with it? What's the difference between doing this and just restoring > > the whole machine running the DC bit for bit (dd backup and restore)? > > If you are talking about stopping the DC and copying it (somehow), then > this should work, but you would have to be aware that you would have to > stop your DC regularly and that your backup would only be valid for the > time you took it, anything between that backup and the next would be > lost. >Let's assume the DC is in a filesystem that allows snapshots, do I assume correctly that stopping samba would not be required in that case? With regards to information between 2 backups being lost, how is that different with other backup strategies, for example using samba-tool online backup? Viktor
A good practice is to have at least 2 DCs in a domain. If one crashes, you can reinstall one from scratch, join it to the domain and it will synchronize with the safe one. And your services should not stop in the meantime. That doesn’t prevent us from setting up backups, but that reduce risks a lot. :-) G.> Le 10 févr. 2019 à 17:07, Viktor Trojanovic via samba <samba at lists.samba.org> a écrit : > > See comments inline. > > On Sun, 10 Feb 2019 at 16:33, Rowland Penny via samba <samba at lists.samba.org> > wrote: > >> On Sun, 10 Feb 2019 14:13:27 +0100 >> Viktor Trojanovic via samba <samba at lists.samba.org> wrote: >> >>> I'm currently reviewing my own backup strategy for Samba and I >>> realize it is not in line with best practices provided in the Wiki. ( >>> https://wiki.samba.org/index.php/Back_up_and_Restoring_a_Samba_AD_DC) >>> Said best practices, however, seem a bit like a nightmare to me. >>> >>> Assuming the AD is gone and you want to restore just one DC, and you >>> want things to look just as they did before the crash, the process >>> according to the Wiki looks as follows: >>> >>> 1. Install a Samba DC on a new (!) temporary host and provision the >>> domain, just like you would when doing a new install from scratch. >>> That task alone is tremendous. >>> 2. Stop Samba and restore the AD from backup to this domain not (!) >>> into the default Samba folder, advise Samba accordingly when starting >>> it. 3. On the original host, set up a Samba DC and join the domain. >>> 4. If GPO or scripts exist on sysvol, manually set up sysvol >>> replication to get them to the original DC. >>> 5. Remove the temporary host. >>> >>> Just... wow. :) >> >> Tend to agree with you, the wiki page asks this question 'So which >> backup should I use?' It then goes on to enumerate 5 different reasons >> why you would need a backup and seems to totally miss the point. Your >> domain has gone down and it is headless chicken time ;-) >> All you would want to do is to get your domain back up again as quickly >> as possible. > > Yes. So I'm really glad I haven't encountered that page in a moment of true > need! :-) > > >> I think you would only do '1' if you wanted to rename the domain. >> >> Not sure where you got restoring into a different folder from, I >> thought the restore put everything back to where it came from. > I got both of this from the section "Restoring the backup-file" in the > wiki. It says that if you're going to use the command "samba-tool domain > backup restore", you *must not* specify a DC that has previously existed. > Further, with regards to the files itself, it says that the Samba team > recommends "that you restore the domain database into a different > targetdir, and then use the '-s' option when running samba". > > >> You shouldn't have to do '4', the backup contains a copy of sysvol and >> smb.conf, so you should be able to restore to the DC it came from, it >> would just have to be the only DC and all DC's would have to be >> stopped, it would probably be better to rename the old DC before >> carrying out the restore. > As mentioned, at least to my understanding of the the wiki a restore of the > original DC is not possible using the backup made from it. Are you saying > that restoring to a "new DC" is as simple as changing the hostname of "DC1" > to "DC1_1"? DNS, GPO, smb.conf would all automatically refer to the new > hostname after the restore? > > >>> >>> Isn't there a simpler way of doing this? Namely, if all the restore >>> operations are done offline anyway, why is it frowned upon to simply >>> do everything on the original DC, i.e. forgo the temporary host, >>> overwrite the configuration files (/etc/samba) and the local Samba >>> folder (e.g. /var/lib/samba) with what's in the backup and be done >>> with it? What's the difference between doing this and just restoring >>> the whole machine running the DC bit for bit (dd backup and restore)? >> >> If you are talking about stopping the DC and copying it (somehow), then >> this should work, but you would have to be aware that you would have to >> stop your DC regularly and that your backup would only be valid for the >> time you took it, anything between that backup and the next would be >> lost. > > Let's assume the DC is in a filesystem that allows snapshots, do I assume > correctly that stopping samba would not be required in that case? With > regards to information between 2 backups being lost, how is that different > with other backup strategies, for example using samba-tool online backup? > > Viktor > -- > To unsubscribe from this list go to the following URL and read the > instructions: https://lists.samba.org/mailman/options/samba
I am cc'ing Tim Beale on this, he wrote the wikipage and, I think, most of the backup & restore code. On Sun, 10 Feb 2019 17:07:55 +0100 Viktor Trojanovic <viktor at troja.ch> wrote:> See comments inline. > > On Sun, 10 Feb 2019 at 16:33, Rowland Penny via samba > <samba at lists.samba.org> wrote: > > > On Sun, 10 Feb 2019 14:13:27 +0100 > > Viktor Trojanovic via samba <samba at lists.samba.org> wrote: > > > > > I'm currently reviewing my own backup strategy for Samba and I > > > realize it is not in line with best practices provided in the > > > Wiki. > > > ( https://wiki.samba.org/index.php/Back_up_and_Restoring_a_Samba_AD_DC) > > > Said best practices, however, seem a bit like a nightmare to me. > > > > > > Assuming the AD is gone and you want to restore just one DC, and > > > you want things to look just as they did before the crash, the > > > process according to the Wiki looks as follows: > > > > > > 1. Install a Samba DC on a new (!) temporary host and provision > > > the domain, just like you would when doing a new install from > > > scratch. That task alone is tremendous. > > > 2. Stop Samba and restore the AD from backup to this domain not > > > (!) into the default Samba folder, advise Samba accordingly when > > > starting it. 3. On the original host, set up a Samba DC and join > > > the domain. 4. If GPO or scripts exist on sysvol, manually set up > > > sysvol replication to get them to the original DC. > > > 5. Remove the temporary host. > > > > > > Just... wow. :) > > > > Tend to agree with you, the wiki page asks this question 'So which > > backup should I use?' It then goes on to enumerate 5 different > > reasons why you would need a backup and seems to totally miss the > > point. Your domain has gone down and it is headless chicken time ;-) > > All you would want to do is to get your domain back up again as > > quickly as possible. > > > > Yes. So I'm really glad I haven't encountered that page in a moment > of true need! :-) > > > > I think you would only do '1' if you wanted to rename the domain. > > > > Not sure where you got restoring into a different folder from, I > > thought the restore put everything back to where it came from. > > > > > I got both of this from the section "Restoring the backup-file" in the > wiki. It says that if you're going to use the command "samba-tool > domain backup restore", you *must not* specify a DC that has > previously existed. Further, with regards to the files itself, it > says that the Samba team recommends "that you restore the domain > database into a different targetdir, and then use the '-s' option > when running samba".OK, re-read the wiki-page and I do not understand why you have to do this ? Tim, Why do you have to do this ? I would image that, unless the DC is kaput, Samba will be the only thing that needs fixing. Most sysadmins would just clear out the Samba dirs and then carry out the restore. I can understand renaming the DC, but why does it have to be a new install ?> > > > You shouldn't have to do '4', the backup contains a copy of sysvol > > and smb.conf, so you should be able to restore to the DC it came > > from, it would just have to be the only DC and all DC's would have > > to be stopped, it would probably be better to rename the old DC > > before carrying out the restore. > > > > > As mentioned, at least to my understanding of the the wiki a restore > of the original DC is not possible using the backup made from it. Are > you saying that restoring to a "new DC" is as simple as changing the > hostname of "DC1" to "DC1_1"? DNS, GPO, smb.conf would all > automatically refer to the new hostname after the restore?That is my understanding and if not, Tim should be able to explain why not ;-)> > > > > > > > Isn't there a simpler way of doing this? Namely, if all the > > > restore operations are done offline anyway, why is it frowned > > > upon to simply do everything on the original DC, i.e. forgo the > > > temporary host, overwrite the configuration files (/etc/samba) > > > and the local Samba folder (e.g. /var/lib/samba) with what's in > > > the backup and be done with it? What's the difference between > > > doing this and just restoring the whole machine running the DC > > > bit for bit (dd backup and restore)? > > > > If you are talking about stopping the DC and copying it (somehow), > > then this should work, but you would have to be aware that you > > would have to stop your DC regularly and that your backup would > > only be valid for the time you took it, anything between that > > backup and the next would be lost. > > > > Let's assume the DC is in a filesystem that allows snapshots, do I > assume correctly that stopping samba would not be required in that > case?The problem is that a Samba AD DC is constantly in flux, that is, it changes constantly, if your 'snapshot' can guarantee it is correct, then I see no problem, but you would only really know when you tried to restore it.>With regards to information between 2 backups being lost, how > is that different with other backup strategies, for example using > samba-tool online backup?That is the problem with any AD DC backup method, the backups can quickly become out of date. Rowland
On Sun, 10 Feb 2019 17:27:43 +0100 Georges Martin <jrjsmrtn at gmail.com> wrote:> A good practice is to have at least 2 DCs in a domain. > > If one crashes, you can reinstall one from scratch, join it to the > domain and it will synchronize with the safe one. And your services > should not stop in the meantime. > > That doesn’t prevent us from setting up backups, but that reduce > risks a lot. :-)That all depends on the second DC not getting corrupted by the other one ;-) If a DC crashes because of a power supply or other component failure, then yes, you should be able to join a new DC and then clean up the database (remove dead DC etc), if not, this is where a good backup comes in. Rowland
On Sun, 10 Feb 2019 at 17:27, Georges Martin <jrjsmrtn at gmail.com> wrote:> A good practice is to have at least 2 DCs in a domain. > > >Yes, I'm aware of that and I try to adhere where possible. However, I'm supporting many smaller offices where 2 bare metal servers just wouldn't be acceptable in terms of TCO but an AD is still appreciated. So at the most I could virtualize a second DC but that wouldn't help much if the hardware gave up. My clients would accept the downtime in case of a disaster but proper backups are vital. And ideally the recovery shouldn't be as costly as a new server ;-)
On Sun, 10 Feb 2019 at 17:42, Rowland Penny via samba <samba at lists.samba.org> wrote:> > > The problem is that a Samba AD DC is constantly in flux, that is, it > changes constantly, if your 'snapshot' can guarantee it is correct, > then I see no problem, but you would only really know when you tried > to restore it. > > >With regards to information between 2 backups being lost, how > > is that different with other backup strategies, for example using > > samba-tool online backup? > > That is the problem with any AD DC backup method, the backups can > quickly become out of date. > > > You keep saying that but I can't quite wrap my head around it. How exactlyis the DC constantly in flux? Say I set up my small AD, one DC, 10 users, 10 computers, internal DNS and some GPOs and I'm not touching any of that anymore after the initial setup. Yes, users create their files, set permissions etc but that's all done on the filesystem of the member server and not in the AD itself, right? So what will have changed a week later on the DC? Viktor
Hi, Thanks for the feedback. I appreciate the whole backup/restore process is more complicated than it needs to be (especially when you're trying to restore in a panic). I've explained below the technical reasons why the backup/restore has these current limitations. The short answer is: - We spent a huge amount of time implementing the current backup/restore tool. Unfortunately, we just ran out of time to make the backup nicer to use. We figured further improvements could be made in future, as needed. - We're primarily samba developers, rather than samba users, so there's some guesses we made as to how it'd be used/what's acceptable. I think updating the restore tool to avoid these annoyances is not that hard. Part of the hard part is just testing it manually to make sure we get things right. On that note, would any samba users out there be willing to volunteer to help test this? i.e. clone a development git branch, build the source, and try out the backup/restore tool?>> you *must not* specify a DC that has previously existedThe restore process currently works like this: 1. The backed up database (which still contains all the old DCs) is restored to disk. 2. The new DC is added to the database. 3. All the previous DCs are removed from the database. Note that: - #3 has to happen to ensure the restored domain doesn't try to communicate with the old/failed domain (just in case you forgot to switch off a DC). - The DC added in #2 will always be a "new" DC (even if it has the same hostname, it will have different a invocation-ID, etc). - You can't have zero DCs in the database, i.e. the new DC needs to be added before all the old ones are removed. Currently you can't add a new DC with the same name because it already exists in the database. Yes, we could avoid this problem by checking the database, and removing the old DC first. It was just more work than we had time to implement.>> restore the domain database into a different targetdir, >> and then use the '-s' option when running sambaA domain join allows you to overwrite the existing samba database, which is the behaviour you're asking for here. We wanted to avoid that in the restore case because: - If the restore didn't work successfully, you've just obliterated your database. - We want to be really careful that remnants from the old database didn't get left behind, so wanted the restore to always be done into an empty directory. The simple solution would be to rename /usr/local/samba/ to /usr/local/samba-old/ first, then run the restore. The problem here is the samba-tool program that you'll use to do the restore is also in the directory you just renamed. Probably a better solution would be for the restore tool to move the sub-directories it's going to restore first. i.e. automatically move /usr/local/samba/private to /usr/local/samba/private-old.>> Are you saying that restoring to a "new DC" is as simple as changing the >> hostname of "DC1" to "DC1_1"? DNS, GPO, smb.conf would all >> automatically refer to the new hostname after the restore?So, a restore should be similar in functionality to a 'domain join', i.e. the hostname specified in the command should get setup in smb.conf and DNS. Most of the sysvol/GPO backup/restore was implemented by another developer, so I don't know that much about it. Basically all the sysvol files and their NTACLs should be preserved as part of the backup process. So you shouldn't need to do anything to the restored DC. As you rejoin the DCs after the restore, you'd need to rsync the sysvol directory (same as you would if you joined a new DC).>> Let's assume the DC is in a filesystem that allows snapshots, do I >> assume correctly that stopping samba would not be required in that case?If you're foregoing the backup tools and just taking backing up the raw files on disk, then you'd need to stop samba. Otherwise, I think you're at risk of corruption from not holding appropriate DB locks. The offline backup (in 4.10) is essentially just taking a snapshot of the files on disk. However, it does it in a way that's safe WRT DB locks. Samba doesn't need to be stopped in this case. Also note that part of the point of the online backup is to try to protect you from latent/benign database corruptions that might exist on disk. i.e. the raw database on disk has a corruption but samba is still running OK, but then that corrupt object gets modified again and everything dies. If your backup has the same latent corruption, then restoring it won't be much help long-term. Regards, Tim On 11/02/19 5:40 AM, Rowland Penny via samba wrote:> I am cc'ing Tim Beale on this, he wrote the wikipage and, I think, most > of the backup & restore code. > > On Sun, 10 Feb 2019 17:07:55 +0100 > Viktor Trojanovic <viktor at troja.ch> wrote: > >> See comments inline. >> >> On Sun, 10 Feb 2019 at 16:33, Rowland Penny via samba >> <samba at lists.samba.org> wrote: >> >>> On Sun, 10 Feb 2019 14:13:27 +0100 >>> Viktor Trojanovic via samba <samba at lists.samba.org> wrote: >>> >>>> I'm currently reviewing my own backup strategy for Samba and I >>>> realize it is not in line with best practices provided in the >>>> Wiki. >>>> ( https://wiki.samba.org/index.php/Back_up_and_Restoring_a_Samba_AD_DC) >>>> Said best practices, however, seem a bit like a nightmare to me. >>>> >>>> Assuming the AD is gone and you want to restore just one DC, and >>>> you want things to look just as they did before the crash, the >>>> process according to the Wiki looks as follows: >>>> >>>> 1. Install a Samba DC on a new (!) temporary host and provision >>>> the domain, just like you would when doing a new install from >>>> scratch. That task alone is tremendous. >>>> 2. Stop Samba and restore the AD from backup to this domain not >>>> (!) into the default Samba folder, advise Samba accordingly when >>>> starting it. 3. On the original host, set up a Samba DC and join >>>> the domain. 4. If GPO or scripts exist on sysvol, manually set up >>>> sysvol replication to get them to the original DC. >>>> 5. Remove the temporary host. >>>> >>>> Just... wow. :) >>> Tend to agree with you, the wiki page asks this question 'So which >>> backup should I use?' It then goes on to enumerate 5 different >>> reasons why you would need a backup and seems to totally miss the >>> point. Your domain has gone down and it is headless chicken time ;-) >>> All you would want to do is to get your domain back up again as >>> quickly as possible. >>> >> Yes. So I'm really glad I haven't encountered that page in a moment >> of true need! :-) >> >> >>> I think you would only do '1' if you wanted to rename the domain. >>> >>> Not sure where you got restoring into a different folder from, I >>> thought the restore put everything back to where it came from. >>> >>> >> I got both of this from the section "Restoring the backup-file" in the >> wiki. It says that if you're going to use the command "samba-tool >> domain backup restore", you *must not* specify a DC that has >> previously existed. Further, with regards to the files itself, it >> says that the Samba team recommends "that you restore the domain >> database into a different targetdir, and then use the '-s' option >> when running samba". > OK, re-read the wiki-page and I do not understand why you have to do > this ? > > Tim, Why do you have to do this ? I would image that, unless the DC > is kaput, Samba will be the only thing that needs fixing. Most > sysadmins would just clear out the Samba dirs and then carry out the > restore. I can understand renaming the DC, but why does it have to be > a new install ? > >> >>> You shouldn't have to do '4', the backup contains a copy of sysvol >>> and smb.conf, so you should be able to restore to the DC it came >>> from, it would just have to be the only DC and all DC's would have >>> to be stopped, it would probably be better to rename the old DC >>> before carrying out the restore. >>> >>> >> As mentioned, at least to my understanding of the the wiki a restore >> of the original DC is not possible using the backup made from it. Are >> you saying that restoring to a "new DC" is as simple as changing the >> hostname of "DC1" to "DC1_1"? DNS, GPO, smb.conf would all >> automatically refer to the new hostname after the restore? > That is my understanding and if not, Tim should be able to explain why > not ;-) > >> >>>> Isn't there a simpler way of doing this? Namely, if all the >>>> restore operations are done offline anyway, why is it frowned >>>> upon to simply do everything on the original DC, i.e. forgo the >>>> temporary host, overwrite the configuration files (/etc/samba) >>>> and the local Samba folder (e.g. /var/lib/samba) with what's in >>>> the backup and be done with it? What's the difference between >>>> doing this and just restoring the whole machine running the DC >>>> bit for bit (dd backup and restore)? >>> If you are talking about stopping the DC and copying it (somehow), >>> then this should work, but you would have to be aware that you >>> would have to stop your DC regularly and that your backup would >>> only be valid for the time you took it, anything between that >>> backup and the next would be lost. >>> >> Let's assume the DC is in a filesystem that allows snapshots, do I >> assume correctly that stopping samba would not be required in that >> case? > The problem is that a Samba AD DC is constantly in flux, that is, it > changes constantly, if your 'snapshot' can guarantee it is correct, > then I see no problem, but you would only really know when you tried > to restore it. > >> With regards to information between 2 backups being lost, how >> is that different with other backup strategies, for example using >> samba-tool online backup? > That is the problem with any AD DC backup method, the backups can > quickly become out of date. > > Rowland > > >