Hi Leroy, How I can confirm that during rsync transfer corruption are not encountered? Thank you in advance. Il 01/07/20 16:04, Leroy Tennison ha scritto:> I've used rsync (but probably not for the size you're referring to), it works and has enough features to meet most needs. I have had a single situation where corruption occurred during transfer (a few times, have no idea why), might want to independently confirm the integrity of the transfer. > > ________________________________ > From: CentOS <centos-bounces at centos.org> on behalf of Alessandro Baggi <alessandro.baggi at gmail.com> > Sent: Wednesday, July 1, 2020 5:26 AM > To: centos at centos.org <centos at centos.org> > Subject: [EXTERNAL] [CentOS] [OT] Bacula offsite replication > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > > Hi everyone, > > I have updated my backup server to CentOS 8.2. It runs bacula performing > backup on disks. I would like to replicate backups on another offsite > machine. > > I read about the ability to configure a new storage daemon in the > offsite location and create a Migration/Copy Jobs. If I'm not wrong, it > replicates only volumes but not replicate the catalog. I will try this. > > Another way to replicate the volumes on another server is using rsync. > > What is your suggestion about this topic? > > Thank you in advance. > > Alessandro. > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos > > Harriscomputer > > Leroy Tennison > Network Information/Cyber Security Specialist > E: leroy at datavoiceint.com > > > [cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG] > > > 2220 Bush Dr > McKinney, Texas > 75070 > www.datavoiceint.com<http://www..com> > > > This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc. > > If you prefer not to be contacted by Harris Operating Group please notify us<http://subscribe.harriscomputer.com/>. > > > > This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message. > > > > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos
What I did was used cksum to create a checksum of the source file putting it in a separate file, transmitted that via rsync as well and compared that to a cksum computed on the remote end. There are far more accurate alternatives to cksum but I felt cksum was good enough for a basic check. Like most things in the UNIX world, there are probably other ways to do this as well. Interestingly enough, after I sent my previous response I discovered that I had yet another instance of the problem. ________________________________ From: CentOS <centos-bounces at centos.org> on behalf of Alessandro Baggi <alessandro.baggi at gmail.com> Sent: Wednesday, July 1, 2020 9:26 AM To: centos at centos.org <centos at centos.org> Subject: [EXTERNAL] Re: [CentOS] [OT] Bacula offsite replication CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Leroy, How I can confirm that during rsync transfer corruption are not encountered? Thank you in advance. Harriscomputer Leroy Tennison Network Information/Cyber Security Specialist E: leroy at datavoiceint.com [cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG] 2220 Bush Dr McKinney, Texas 75070 www.datavoiceint.com<http://www..com> This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc. If you prefer not to be contacted by Harris Operating Group please notify us<http://subscribe.harriscomputer.com/>. This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message. Il 01/07/20 16:04, Leroy Tennison ha scritto:> I've used rsync (but probably not for the size you're referring to), it works and has enough features to meet most needs. I have had a single situation where corruption occurred during transfer (a few times, have no idea why), might want to independently confirm the integrity of the transfer. > > ________________________________ > From: CentOS <centos-bounces at centos.org> on behalf of Alessandro Baggi <alessandro.baggi at gmail.com> > Sent: Wednesday, July 1, 2020 5:26 AM > To: centos at centos.org <centos at centos.org> > Subject: [EXTERNAL] [CentOS] [OT] Bacula offsite replication > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > > Hi everyone, > > I have updated my backup server to CentOS 8.2. It runs bacula performing > backup on disks. I would like to replicate backups on another offsite > machine. > > I read about the ability to configure a new storage daemon in the > offsite location and create a Migration/Copy Jobs. If I'm not wrong, it > replicates only volumes but not replicate the catalog. I will try this. > > Another way to replicate the volumes on another server is using rsync. > > What is your suggestion about this topic? > > Thank you in advance. > > Alessandro. > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos > > Harriscomputer > > Leroy Tennison > Network Information/Cyber Security Specialist > E: leroy at datavoiceint.com > > > [cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG] > > > 2220 Bush Dr > McKinney, Texas > 75070 > https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.datavoiceint.com&c=E,1,NcuZo7bJlNfnwFN_1U7Eb0PpothLaHlm35UVdQMBqG8TH6hZvo3iMVLxCy4ZNUIAWOktkvYozTYnDWfj0JIvc22V52YUkDfms1NFI7AXHA,,&typo=1<http://www..com> > > > This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc. > > If you prefer not to be contacted by Harris Operating Group please notify us<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fsubscribe.harriscomputer.com%2f&c=E,1,4UMyprULKejN76Lk4p9zM-laz6VtwtLbbjIU8e02p6oWiLS-njfZsTFuXkb0910-WrqQ8x6J4YCieJO5HeN2WGf7pqwFdtVkKJi-m_QGliIsyR6XTAVohBrv&typo=1>. > > > > This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message. > > > > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos_______________________________________________ CentOS mailing list CentOS at centos.org https://lists.centos.org/mailman/listinfo/centos
I realize this shouldn't happen, the file is a tgz and isn't being modified while being transmitted. This has happened maybe three times this year and unfortunately I've just had to deal with it rather than invest the time to do the research. Harriscomputer Leroy Tennison Network Information/Cyber Security Specialist E: leroy at datavoiceint.com [cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG] 2220 Bush Dr McKinney, Texas 75070 www.datavoiceint.com<http://www..com> This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc. If you prefer not to be contacted by Harris Operating Group please notify us<http://subscribe.harriscomputer.com/>. This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message. ________________________________ From: Chris Schanzle <christopher.schanzle at nist.gov> Sent: Wednesday, July 1, 2020 10:03 AM To: CentOS mailing list <centos at centos.org>; Leroy Tennison <leroy at datavoiceint.com> Subject: [EXTERNAL] Re: [CentOS] [OT] Bacula offsite replication CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Unless the file is being modified during rsync, corruption should not happen with good hardware. Consider testing your RAM. Have you noticed any other weird problems with that remote server, like programs crashing / daemons needing restarting? On 7/1/20 10:37 AM, Leroy Tennison wrote:> What I did was used cksum to create a checksum of the source file putting it in a separate file, transmitted that via rsync as well and compared that to a cksum computed on the remote end. There are far more accurate alternatives to cksum but I felt cksum was good enough for a basic check. Like most things in the UNIX world, there are probably other ways to do this as well. > > Interestingly enough, after I sent my previous response I discovered that I had yet another instance of the problem. > > ________________________________ > From: CentOS <centos-bounces at centos.org> on behalf of Alessandro Baggi <alessandro.baggi at gmail.com> > Sent: Wednesday, July 1, 2020 9:26 AM > To: centos at centos.org <centos at centos.org> > Subject: [EXTERNAL] Re: [CentOS] [OT] Bacula offsite replication > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > > Hi Leroy, > > How I can confirm that during rsync transfer corruption are not encountered? > > Thank you in advance. > > > Harriscomputer > > Leroy Tennison > Network Information/Cyber Security Specialist > E: leroy at datavoiceint.com > > > [cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG] > > > 2220 Bush Dr > McKinney, Texas > 75070 > https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fwww.datavoiceint.com%2f&c=E,1,00uoVithcdoZKfdE0IJM8LvvYxwVLFDjVaj-PErs6HOprVgQbdBE1Ev3mXF3w6PUc_C_6eI1odWQtpYUTMU3wRbhn6gDS_pSCiRBZdG-fqPzGNsgh2ZNZFujh1s,&typo=1> > > > This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc. > > If you prefer not to be contacted by Harris Operating Group please notify us<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fsubscribe.harriscomputer.com%2f&c=E,1,ESghWsZAKB3kZUcHUH6MS2ivZGjhaE3linFZeLtQ96hbUtv37Esy1OON4XdoFr1DjlanYK_dt8Kie6diqCOVrkPalJ6KDLXEocN-5BFabl2AiHWvFfo3VvM,&typo=1>. > > > > This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message. > > > > > > Il 01/07/20 16:04, Leroy Tennison ha scritto: >> I've used rsync (but probably not for the size you're referring to), it works and has enough features to meet most needs. I have had a single situation where corruption occurred during transfer (a few times, have no idea why), might want to independently confirm the integrity of the transfer. >> >> ________________________________ >> From: CentOS <centos-bounces at centos.org> on behalf of Alessandro Baggi <alessandro.baggi at gmail.com> >> Sent: Wednesday, July 1, 2020 5:26 AM >> To: centos at centos.org <centos at centos.org> >> Subject: [EXTERNAL] [CentOS] [OT] Bacula offsite replication >> >> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. >> >> >> Hi everyone, >> >> I have updated my backup server to CentOS 8.2. It runs bacula performing >> backup on disks. I would like to replicate backups on another offsite >> machine. >> >> I read about the ability to configure a new storage daemon in the >> offsite location and create a Migration/Copy Jobs. If I'm not wrong, it >> replicates only volumes but not replicate the catalog. I will try this. >> >> Another way to replicate the volumes on another server is using rsync. >> >> What is your suggestion about this topic? >> >> Thank you in advance. >> >> Alessandro. >> >> _______________________________________________ >> CentOS mailing list >> CentOS at centos.org >> https://lists.centos.org/mailman/listinfo/centos >> >> Harriscomputer >> >> Leroy Tennison >> Network Information/Cyber Security Specialist >> E: leroy at datavoiceint.com >> >> >> [cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG] >> >> >> 2220 Bush Dr >> McKinney, Texas >> 75070 >> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.datavoiceint.com&c=E,1,NcuZo7bJlNfnwFN_1U7Eb0PpothLaHlm35UVdQMBqG8TH6hZvo3iMVLxCy4ZNUIAWOktkvYozTYnDWfj0JIvc22V52YUkDfms1NFI7AXHA,,&typo=1> >> >> >> This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc. >> >> If you prefer not to be contacted by Harris Operating Group please notify us<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fsubscribe.harriscomputer.com%2f&c=E,1,4UMyprULKejN76Lk4p9zM-laz6VtwtLbbjIU8e02p6oWiLS-njfZsTFuXkb0910-WrqQ8x6J4YCieJO5HeN2WGf7pqwFdtVkKJi-m_QGliIsyR6XTAVohBrv&typo=1>. >> >> >> >> This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message. >> >> >> >> >> _______________________________________________ >> CentOS mailing list >> CentOS at centos.org >> https://lists.centos.org/mailman/listinfo/centos > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos
Am 01.07.20 um 17:13 schrieb Leroy Tennison:> I realize this shouldn't happen, the file is a tgz and isn't being modified while being transmitted. This has happened maybe three times this year and unfortunately I've just had to deal with it rather than invest the time to do the research. >Maybe a "RunAfterJob" configuration would help to serialize it and prevent this race condition? -- Leon
Il 01/07/20 17:13, Leroy Tennison ha scritto:> I realize this shouldn't happen, the file is a tgz and isn't being modified while being transmitted. This has happened maybe three times this year and unfortunately I've just had to deal with it rather than invest the time to do the research. > > > Harriscomputer > > Leroy Tennison > Network Information/Cyber Security SpHi Leroy, I think that in my case I could not use a tgz archive. I'm speaking about full backups that reach 600/700GiB, compressing them and then rsync them could take so much time that it will be useless.