Lorenz Weber <mail at lenzw.de> wrote:> rsync -avH ${all_gubbins} / user at remote.machine:/dest/ && ssh user at remote.machine touch /etc/donefileNo SSH access between them, only rsync. Besides, it would add the overhead of managing ssh access (users and keys) as well as Rsync.
As an aside to this, part of the problem I've been having is the transfer timing out/getting interrupted during a particular large file (1G, new file, 2-3 hours if it works). So I've been experimenting with --partial and --partial-dir=.rsync-partial which weren't working. It appears to work at first - if the transfer is interrupted, the partial file is correctly saved in the named directory. Then if I run the script again, it deletes the partial file before starting again. I found that I needed to also specify --delete-delay to avoid deleting the partial file before it's used. Is this "known", because it isn't implied (as I read it) by the --partial-dir section in the man page ?
Searching the mail list archives, I found this old link
[https://bugzilla.samba.org/show_bug.cgi?id=2423] with an example of how
to use an external script with filter to select files by date.
It looks like it could do exactly what I need, which is only match files
less than 3 days old, but it doesn't seem to work.
I have added the filter to the rsyncd.conf on the source server, which
receives the native client connection over a network.
filter='-!| newer-filter mm/dd/yy'
This is the script as suggested.
From the cmd line it returns a line of 0's or 1's if I pipe an
'ls' of
the folder to it, so seems ok.
--------------------------------------------------------------
#!/bin/bash
# We convert times to seconds since the Epoch so
# we can compare them with [ -gt ].
cutoff_time=$(date +%s -d "$1")
while IFS='' read fname; do
# Compare file's mtime to cutoff time
if [ $(stat --format=%Y $fname) -gt $cutoff_time ]; then
echo -n 1
else
echo -n 0
fi
done
--------------------------------------------------------------
However, the rsync log output shows the desired files are parsed, but
none are never selected for sending.
Has anyone a working example of using an external script as a Filter
source ?
Thanks in advance,
Mark.
You could try increasing the timeout delay, rather than resume. rsync will tolerate quite long network dropouts and still carry on. I have managed to keep an internet transfer of up to 100Gb alive for two weeks. I didn't find --partial to be much use for very large scale transfers due to the very cpu intensive checksum process. By large scale I have rsync'd several Petabytes of backup files up to 500Gb size over the last five years with good success. On 29/04/2015 2:49 a.m., Simon Hobson wrote:> As an aside to this, part of the problem I've been having is the transfer timing out/getting interrupted during a particular large file (1G, new file, 2-3 hours if it works). > > So I've been experimenting with --partial and --partial-dir=.rsync-partial which weren't working. It appears to work at first - if the transfer is interrupted, the partial file is correctly saved in the named directory. > Then if I run the script again, it deletes the partial file before starting again. > > I found that I needed to also specify --delete-delay to avoid deleting the partial file before it's used. > > Is this "known", because it isn't implied (as I read it) by the --partial-dir section in the man page ? >
On Wed, Apr 29, 2015 at 2:52 AM, Mark <mark0x01 at gmail.com> wrote:> I have added the filter to the rsyncd.conf on the source server, which > receives the native client connection over a network. > > filter='-!| newer-filter mm/dd/yy' >That was a suggested syntax for a feature that nobody implemented. ..wayne.. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20150504/6acc2f08/attachment.html>