Il 02/07/20 15:02, Valeri Galtsev ha scritto:> > > On 7/2/20 3:22 AM, Alessandro Baggi wrote: >> Il 01/07/20 17:13, Leroy Tennison ha scritto: >>> I realize this shouldn't happen, the file is a tgz and isn't being >>> modified while being transmitted.? This has happened maybe three >>> times this year and unfortunately I've just had to deal with it >>> rather than invest the time to do the research. >>> >>> >>> Harriscomputer >>> >>> Leroy Tennison >>> Network Information/Cyber Security Sp >> >> Hi Leroy, >> >> I think that in my case I could not use a tgz archive. I'm speaking >> about full backups that reach 600/700GiB, compressing them and then >> rsync them could take so much time that it will be useless. >> > > unless you use tape (of that high capacity), it is advantageous to > restrict volume size to, say, 50GB. Then when you restore, search for > specific files will be faster. And it will help your backup volumes > transfers as well. > > ValeriHi Valeri, thank you for your suggestion. Is bacula the right backup system when I need to replicate data offsite? There are other backup solution that simplify this process? Thank you in advance
On 2020-07-02 08:28, Alessandro Baggi wrote:> > Il 02/07/20 15:02, Valeri Galtsev ha scritto: >> >> >> On 7/2/20 3:22 AM, Alessandro Baggi wrote: >>> Il 01/07/20 17:13, Leroy Tennison ha scritto: >>>> I realize this shouldn't happen, the file is a tgz and isn't being >>>> modified while being transmitted.? This has happened maybe three >>>> times this year and unfortunately I've just had to deal with it >>>> rather than invest the time to do the research. >>>> >>>> >>>> Harriscomputer >>>> >>>> Leroy Tennison >>>> Network Information/Cyber Security Sp >>> >>> Hi Leroy, >>> >>> I think that in my case I could not use a tgz archive. I'm speaking >>> about full backups that reach 600/700GiB, compressing them and then >>> rsync them could take so much time that it will be useless. >>> >> >> unless you use tape (of that high capacity), it is advantageous to >> restrict volume size to, say, 50GB. Then when you restore, search for >> specific files will be faster. And it will help your backup volumes >> transfers as well. >> >> Valeri > > Hi Valeri, > > thank you for your suggestion. > > Is bacula the right backup system when I need to replicate data offsite? > There are other backup solution that simplify this process? >Bacula is great enterprise level open source backup system. I switched to its fork bareos at some point; I use bacula/bareos for at least a decade. And with this your extra requirement I still would stay with bareos (or bacula). If I were to have two sets of backup: on site and off site, I would just set up separate bacula/bareos director and storage daemon(s) off site. Add to FDs (file daemons) extra instances of director - offsite one with different passwords for the sake of security. Then there will be a set of everything off site, not only a set of volumes. Of course, if you only have a set of volumes, but everything else has evaporated, you still will be able to restore everything, including database records by scanning set of volumes. Which will take forever. I would alternate dates of backups in offsite/onsite schedules, or define times of backups so that they do not overlap. Another good news of this vs just rsyncing volumes is: bacula/bareos verifies checksum of every backed up file after receiving it. This will ensure consistency of files in remote volumes, for rsync you will have to at least verify checksum of each volume transferred to destination (unless I miss my wits and rsync does verify checksums of files transferred, I just re-read rsync man and don't see verification - hopefully rsync expert will chime in and correct me if I'm wrong about rsync). Anyway, that is what I would do. Valeri> Thank you in advance > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos-- ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
On 2020-07-02 09:50, Bez Thomas wrote:> Hi Valeri, > > To go off on a tangent, do you use the Bareos community edition, or the paid version? >I use open source - community edition. I work for the department of the university, they do not have much money, thus pretty much everything I set up for them is free as in free bear. I have to invest my time and knowledge - and help of others - instead of paid support, and this way I earn my salary ;-) Suits both myself and my employers. Valeri> Thanks, > Bez Thomas (he/him) > Tech Services, Astronomy/CCAPS > 222 Space Sciences Bldg, Cornell University > 607-255-3434 > > > > > > >> On Jul 2, 2020, at 10:39 AM, Valeri Galtsev <galtsev at kicp.uchicago.edu> wrote: >> >> >> >> On 2020-07-02 08:28, Alessandro Baggi wrote: >>> Il 02/07/20 15:02, Valeri Galtsev ha scritto: >>>> >>>> >>>> On 7/2/20 3:22 AM, Alessandro Baggi wrote: >>>>> Il 01/07/20 17:13, Leroy Tennison ha scritto: >>>>>> I realize this shouldn't happen, the file is a tgz and isn't being modified while being transmitted. This has happened maybe three times this year and unfortunately I've just had to deal with it rather than invest the time to do the research. >>>>>> >>>>>> >>>>>> Harriscomputer >>>>>> >>>>>> Leroy Tennison >>>>>> Network Information/Cyber Security Sp >>>>> >>>>> Hi Leroy, >>>>> >>>>> I think that in my case I could not use a tgz archive. I'm speaking about full backups that reach 600/700GiB, compressing them and then rsync them could take so much time that it will be useless. >>>>> >>>> >>>> unless you use tape (of that high capacity), it is advantageous to restrict volume size to, say, 50GB. Then when you restore, search for specific files will be faster. And it will help your backup volumes transfers as well. >>>> >>>> Valeri >>> Hi Valeri, >>> thank you for your suggestion. >>> Is bacula the right backup system when I need to replicate data offsite? There are other backup solution that simplify this process? >> >> Bacula is great enterprise level open source backup system. I switched to its fork bareos at some point; I use bacula/bareos for at least a decade. And with this your extra requirement I still would stay with bareos (or bacula). >> >> If I were to have two sets of backup: on site and off site, I would just set up separate bacula/bareos director and storage daemon(s) off site. Add to FDs (file daemons) extra instances of director - offsite one with different passwords for the sake of security. Then there will be a set of everything off site, not only a set of volumes. Of course, if you only have a set of volumes, but everything else has evaporated, you still will be able to restore everything, including database records by scanning set of volumes. Which will take forever. I would alternate dates of backups in offsite/onsite schedules, or define times of backups so that they do not overlap. >> >> Another good news of this vs just rsyncing volumes is: bacula/bareos verifies checksum of every backed up file after receiving it. This will ensure consistency of files in remote volumes, for rsync you will have to at least verify checksum of each volume transferred to destination (unless I miss my wits and rsync does verify checksums of files transferred, I just re-read rsync man and don't see verification - hopefully rsync expert will chime in and correct me if I'm wrong about rsync). >> >> Anyway, that is what I would do. >> >> Valeri >> >>> Thank you in advance >>> _______________________________________________ >>> CentOS mailing list >>> CentOS at centos.org >>> https://lists.centos.org/mailman/listinfo/centos >> >> -- >> ++++++++++++++++++++++++++++++++++++++++ >> Valeri Galtsev >> Sr System Administrator >> Department of Astronomy and Astrophysics >> Kavli Institute for Cosmological Physics >> University of Chicago >> Phone: 773-702-4247 >> ++++++++++++++++++++++++++++++++++++++++ >> _______________________________________________ >> CentOS mailing list >> CentOS at centos.org >> https://lists.centos.org/mailman/listinfo/centos >-- ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
Il 02/07/20 16:39, Valeri Galtsev ha scritto:> > > On 2020-07-02 08:28, Alessandro Baggi wrote: >> >> Il 02/07/20 15:02, Valeri Galtsev ha scritto: >>> >>> >>> On 7/2/20 3:22 AM, Alessandro Baggi wrote: >>>> Il 01/07/20 17:13, Leroy Tennison ha scritto: >>>>> I realize this shouldn't happen, the file is a tgz and isn't being >>>>> modified while being transmitted.? This has happened maybe three >>>>> times this year and unfortunately I've just had to deal with it >>>>> rather than invest the time to do the research. >>>>> >>>>> >>>>> Harriscomputer >>>>> >>>>> Leroy Tennison >>>>> Network Information/Cyber Security Sp >>>> >>>> Hi Leroy, >>>> >>>> I think that in my case I could not use a tgz archive. I'm speaking >>>> about full backups that reach 600/700GiB, compressing them and then >>>> rsync them could take so much time that it will be useless. >>>> >>> >>> unless you use tape (of that high capacity), it is advantageous to >>> restrict volume size to, say, 50GB. Then when you restore, search >>> for specific files will be faster. And it will help your backup >>> volumes transfers as well. >>> >>> Valeri >> >> Hi Valeri, >> >> thank you for your suggestion. >> >> Is bacula the right backup system when I need to replicate data >> offsite? There are other backup solution that simplify this process? >> > > Bacula is great enterprise level open source backup system. I switched > to its fork bareos at some point; I use bacula/bareos for at least a > decade. And with this your extra requirement I still would stay with > bareos (or bacula). > > If I were to have two sets of backup: on site and off site, I would > just set up separate bacula/bareos director and storage daemon(s) off > site. Add to FDs (file daemons) extra instances of director - offsite > one with different passwords for the sake of security. Then there will > be a set of everything off site, not only a set of volumes. Of course, > if you only have a set of volumes, but everything else has evaporated, > you still will be able to restore everything, including database > records by scanning set of volumes. Which will take forever. I would > alternate dates of backups in offsite/onsite schedules, or define > times of backups so that they do not overlap. > > Another good news of this vs just rsyncing volumes is: bacula/bareos > verifies checksum of every backed up file after receiving it. This > will ensure consistency of files in remote volumes, for rsync you will > have to at least verify checksum of each volume transferred to > destination (unless I miss my wits and rsync does verify checksums of > files transferred, I just re-read rsync man and don't see verification > - hopefully rsync expert will chime in and correct me if I'm wrong > about rsync). > > Anyway, that is what I would do. > > Valeri >Hi Valeri, I'm in late but thank you for your suggestion.