An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20190712/4cce45c2/attachment.htm>
At 10:56 12.07.2019, Hannes Hutmacher via rsync wrote:>Hi all! :-) > >I have a small rsync script to sync my data to a usb-disk. It works fine, when I start it in console. I get 3 rsync processes (look in top) and the backup takes ~25 min. But, when I add the script in cron to start it at 1am at night it takes 7 - 9 hours and I see up to 180 processes. When I look in top I see a hight load of 60 - 80 and 40 - 60 waits. Why? Can someone explain why it takes so long when it starts with cron? > >This is my rsync command: rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" >This is the entry in cron (crontab -e): * 2 * * * /root/backupscript/backup.shI'd say this starts rsync on every minute between 2:00 and 3:00. You should change that to e.g. 0 2 ...>Data to sync: 18 Gb, 185.000 files. > >When I look in the log files I see errors like this: > >rsync: rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2)That's probably because several rsync instances are stomping on each other's feet. bye Fabi
Hallo Hannes "man 5 crontab" is your friend ;-) Am 12.07.2019 um 10:56 schrieb Hannes Hutmacher via rsync:> But, when I add the script in cron?to start it at 1am at night it > takes 7?- 9?hours and I see up to 180?processes. When I look in top I > see a hight load of 60 - 80 and 40 - 60 waits. Why??Can someone > explain why it takes so long when it starts with cron? > This is my rsync command:??rsync -azc --delete "$QUELLORDNER" > "$ZIELORDNER" > This is the entry in cron (crontab -e):?* 2 * * * > /root/backupscript/backup.sh > Data to sync: 18 Gb, 185.000 files.wrong crontab entry: every minute you start a new rsync process ... crontab syntax:|*1 2 3 4 5 command* ??????? ? ??? field????????? allowed values ????????????? -----????????? -------------- ????????? 1 ? minute???????? 0-59 ????????? 2 ? hour?????????? 0-23 ????????? 3 ? day of month?? 1-31 ????????? 4 ? month????????? 1-12 (or names, see below) ????????? 5 ? day of week??? 0-7 (0 or 7 is Sun, or use names) ????????? command = command to be run Good luck -- Beat | -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20190712/ab3ab1e4/attachment.htm>
On 7/12/19 4:56 AM, Hannes Hutmacher via rsync wrote:> Hi all! :-) > ? > I have a small rsync script to sync my data to a usb-disk. It works > fine, when I start it in console. I get 3 rsync processes (look in > top) and the backup takes ~25 min. But, when I add the script in > cron?to start it at 1am at night it takes 7?- 9?hours and I see up to > 180?processes. When I look in top I see a hight load of 60 - 80 and 40 > - 60 waits. Why??Can someone explain why it takes so long when it > starts with cron? > ? > This is my rsync command:??rsync -azc --delete "$QUELLORDNER" > "$ZIELORDNER"? > This is the entry in cron (crontab -e):?* 2 * * * > /root/backupscript/backup.sh > Data to sync: 18 Gb, 185.000 files. > ? > When I look in the log files I see errors like this:? > ? > rsync: > rename?"/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" > ->?"var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2)? > > rsync error: some files/attrs were not transferred (see previousrsync > error: > > some files/attrs were not transferred (see previous errors) (code 23) at > > main.c(1196) [sender=3.1.2] > > directory (2) > > ? > > Can you help me to solve this problem? > > ? > > regards, > > Hannes Hutmacher > >A couple of questions: (I am not an expert user.) Does your manual job run as the same user (presumably root) as does your cron job? Are you backing up any temporary files that you might be better off excluding? Is anything else running at the same time as your cron job which may be creating and destroying files in the backup source or target? (E.g. Your firewall?) I'm not very familiar with fuzzy searching, but using that in places like /var seems odd. One of the constant refrains on this list is that using checksums is almost never a good idea. In combination with fuzzy, it seems even more tenuous. Joe
In addition to the cron problems that others have pointed out there are rsync problems... #1 rsync's -c is almost always a bad idea and is extremely slow #2 if rsync isn't networking then -z is a waste of time On 7/12/19 4:56 AM, Hannes Hutmacher via rsync wrote:> Hi all! :-) > ? > I have a small rsync script to sync my data to a usb-disk. It works > fine, when I start it in console. I get 3 rsync processes (look in top) > and the backup takes ~25 min. But, when I add the script in cron?to > start it at 1am at night it takes 7?- 9?hours and I see up to > 180?processes. When I look in top I see a hight load of 60 - 80 and 40 - > 60 waits. Why??Can someone explain why it takes so long when it starts > with cron? > ? > This is my rsync command:??rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER"? > This is the entry in cron (crontab -e):?* 2 * * * > /root/backupscript/backup.sh > Data to sync: 18 Gb, 185.000 files. > ? > When I look in the log files I see errors like this:? > ? > rsync: > rename?"/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" > ->?"var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2)? > > rsync error: some files/attrs were not transferred (see previousrsync error: > > some files/attrs were not transferred (see previous errors) (code 23) at > > main.c(1196) [sender=3.1.2] > > directory (2) > > ? > > Can you help me to solve this problem? > > ? > > regards, > > Hannes Hutmacher > >-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: OpenPGP digital signature URL: <http://lists.samba.org/pipermail/rsync/attachments/20190712/8dd24c34/signature.sig>
I think others have answered your questions. You may want to implement some sort of locking system in the script so that you can only run a single instance of your script at. If you are looking for a backup tool which leverages rsync you may want to take a look at LBackup : http://www.lbackup.org There is even a pre-script you can enable (in the example resources) so that the script can run a maximum of once a day / once a month etc (even if you start it many times from cron. Hope you have it all working now :) Henri>> Hi all! :-) >> >> I have a small rsync script to sync my data to a usb-disk. It works >> fine, when I start it in console. I get 3 rsync processes (look in >> top) and the backup takes ~25 min. But, when I add the script in >> cron to start it at 1am at night it takes 7 - 9 hours and I see up to >> 180 processes. When I look in top I see a hight load of 60 - 80 and 40 >> - 60 waits. Why? Can someone explain why it takes so long when it >> starts with cron? >> >> This is my rsync command: rsync -azc --delete "$QUELLORDNER" >> "$ZIELORDNER" >> This is the entry in cron (crontab -e): * 2 * * * >> /root/backupscript/backup.sh >> Data to sync: 18 Gb, 185.000 files. >> >> When I look in the log files I see errors like this: >> >> rsync: >> rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" >> -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) >> >> rsync error: some files/attrs were not transferred (see previousrsync >> error: >> >> some files/attrs were not transferred (see previous errors) (code 23) at >> >> main.c(1196) [sender=3.1.2] >> >> directory (2) >> >> >> >> Can you help me to solve this problem? >> >> >> >> regards, >> >> Hannes Hutmacher >> >> > A couple of questions: (I am not an expert user.) > > Does your manual job run as the same user (presumably root) as does your > cron job? > > Are you backing up any temporary files that you might be better off > excluding? > > Is anything else running at the same time as your cron job which may be > creating and destroying files in the backup source or target? (E.g. Your > firewall?) > > I'm not very familiar with fuzzy searching, but using that in places > like /var seems odd. > > One of the constant refrains on this list is that using checksums is > almost never a good idea. In combination with fuzzy, it seems even more > tenuous. > > Joe > > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reasonably Related Threads
- rsync many processes and slow backup
- rsync many processes and slow backup
- Device not configured (explanation)
- Mac OS X "rsync: unpack_smb_acl: sys_acl_get_info(): Unknown error: 0 (0)"
- Recycling directories and backup performance. Was: Re: rsync --link-dest won't link even if existing file is out of date (fwd)