similar to: [Bug 9894] New: Rsync can silently zero out chunks in a file

Greetings, I'd like to propose a new option to rsync, which causes it to read device files as if they were regular files. This includes pipes, character devices and block devices (I'm not sure about sockets). The main motivation is cases where you need to synchronize a large amount of data that is not available as regular files, as in the following scenarios: * Keep a copy of a block

From: Sebastian Andrzej Siewior <sebastian at breakpoint.cc> zstd compression was announced as "good compression with high throughput" so I gave it a try. With zlib, on high speed links the CPU is usually the bottle neck. With zstd I'm able to fill a 200Mbit link :) zstd detection happens automatically via pkg-config. No zstd header means no error about missing zstd. So that

Hi, When using rsync-3.0.2 through 3.0.5, I get this error on a large dataset syncing from machine-a to machine-b: $ /bin/rsync -aHSz /local/. machine-b:/local/. invalid len passed to map_ptr: -1737287498 rsync error: error in file IO (code 11) at fileio.c(188) [sender=3.0.5] This happens no matter which side initiates the connection, so this fails in the same way: $ /bin/rsync -aHSz

Hi, I think I found a bug in usage of zlib. rsync 2.5.6 with -z fails like bellow. % cp install-disk2.iso /var/tmp/install-disk2.iso install-disk2.iso 100% |*****************************| 316 MB 00:56 % rsync -vIz install-disk2.iso 127.0.0.1:/var/tmp/install-disk2.iso install-disk2.iso deflate on token returned 0 (16384 bytes left) rsync error: error in rsync protocol

Attached is a patch that makes window strides constant when files are walked with a constant block size. In these cases, it completely avoids all memmoves. In my simple local test of rsyncing 57MB of 10 local files, memmoved bytes went from 18MB to zero. I haven't tested this for a big variety of file cases. I think that this will always reduce the memmoves involved with walking a large

hi, I made some changes to generator.c : - reading data, calculating checksums and sending it to the sender now happens in one loop. - the code has become shorter - it uses less memory - 2 malloc's less that may fail - the line will be used all the time - it should be a bit faster It seems to work for me, please have a look at it. You should run "make proto" after

Mark II of the patch set. The first patch (dynsumlen2.patch) increments the protocol version to support per-file dynamic block checksum sizes. It is a prerequisite for varsumlen2.patch. varsumlen2.patch implements per-file dynamic block and checksum sizes. The current block size calculation only applies to files between 7MB and 160MB setting the block size to 1/10,0000 of the file length for a

Hi, I have compiled the new version 2.5.2 rsync in our servers, then ran rsync last night, there were some files not copied to destination server, both source and destination servers are running 2.5.2, I got the following error messages, can you please let me know what would be caused the errors? I copied back 2.3.2 version on both servers, and rsync went well. Thanks for the help, here is

https://bugzilla.samba.org/show_bug.cgi?id=6293 Summary: rsync crashes when transferring files Product: rsync Version: 3.0.5 Platform: x86 OS/Version: Mac OS X Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: wayned@samba.org ReportedBy: dirk.samba@miriup.de

> Just curious: what about the rsync+ patch? Thanks for the reminder. I've just committed Jos's rsync+ patch onto the "branch_mbp_rsyncplus_merge" branch. If it works OK and nobody screams I will move it across onto the main tree tomorrow or Wednesday. I see the patch doesn't add documentation about the new options to the man page, so we should fix that in the future.

Hi list, Since upgrading rsync to version 2.5.2 on a Solaris 2.5.1 box, I see plenty of Warning: unexpected read size of 0 in map_ptr when doing my nightly backup (rsyncing some directories to an rsyncd on the same LAN). I have never seen this message before, and I only see it on the Solaris machine (I upgraded all my other machines to 2.5.2 as well). I hope this is not a FAQ... I did not find

We're using rsync to mirror a web server. The rsync is initiated using ssh from a Red Hat Linux 7.2 box, the files are located on an Alpha running a version of Digital Unix 4. On the Alpha rsync is 2.5.2, The Linux box used to have a 2.4.6-X version, but with the map_ptr problems I built 2.5.2 on that as well. However, version incompatibility was not the problem, as they persist after the

I am trying to finalize the use of rsync for updatiung a new nfs server before we take the old one offline. I keep getting the following warning during the rsync process: Warning: unexpected rad size of 0 in map_ptr Any ideas where this comes from and how to make it go away? I am using rsync 2.5.2 on Solaris 8 to pull data from rsync 2.5.2 on Solaris 7. Bob roconnor@vectorpartners.com

Currently rsync has a bit of a problem with very large files. Dynamic block sizes were introduced to try handle that automatically if the user didn't specify a block size. Unfortunately that isn't enough and the block size would need to grow faster than the file. Besides, overly large block sizes mean large amounts of data need to be copied even for small changes. The maths indicate

https://bugzilla.samba.org/show_bug.cgi?id=8177 Summary: Problems with big sparsed files Product: rsync Version: 3.0.8 Platform: x64 OS/Version: Linux Status: NEW Severity: normal Priority: P5 Component: core AssignedTo: wayned at samba.org ReportedBy: joluinfante at gmail.com

Rsync has begun failing to replicate a kerberos samba mount point. This mount point has worked previously, but recently an attempt to replicate the mount point returns; lseek failed in map_ptr rsync error: error in file IO (code 11) at fileio.c(228) rsync: writefd_unbuffered failed to write 4092 bytes: phase "unknown": Broken pipe rsync error: error in rsync protocol data stream (code

Warning for kernel folks: I'm not much of an mm person; let me know if I got anything horribly wrong. Many folks use rsync in their nightly backup jobs. In these applications, speed is of minimal concern and should be sacrificed in order to minimize the effect of rsync on the rest of the machine. When rsync is working on a large directory it can quickly fill the page cache with written data,

In short: Platform: linux with 2.4 kernel Version: rsync 2.6.6 Command line: rsync266 -av -W --bwlimit=1 /mnt/somedir/rsync-2.6.6.tar.gz ./ Destination: local disk Source: file on a smbfs mounted filesystem; share is exported on a NT 4.0 workstation over a very slow and unstable link Result: Rsync completes operation with no special message, but the resulting file is damaged, large

When a whole-file checksum is performed, hashing was done in 64 byte blocks, causing overhead and limiting performance. Testing showed the performance improvement to go up quickly going from 64 to 512 bytes, with diminishing returns above, 4096 was where it seemed to plateau for me. Re-used CHUNK_SIZE (32 kB) as it already exists and should be fine to use here anyway. Noticed this because

Maybe security related mails should be sent elsewhere? I didn't notice any so here it goes: sender.c:receive_sums() s->count = read_int(f); .. s->sums = (struct sum_buf *)malloc(sizeof(s->sums[0])*s->count); if (!s->sums) out_of_memory("receive_sums"); for (i=0; i < (int) s->count;i++) { s->sums[i].sum1 = read_int(f);