search for: hash_search

...0 in ?? () #4 0x0805a150 in make_file (fname=0x0, flist=0x0, exclude_level=69) at flist.c:879 #5 0x0805a3bc in send_file_name (f=3, flist=0x41, fname=0x0, recursive=65, base_flags=0) at flist.c:932 #6 0x0805a845 in send_file_list (f=1, argc=1501590384, argv=0x0) at flist.c:1033 #7 0x08053e1a in hash_search (f=1, s=0x0, buf=0x0, len=577936410078638920) at match.c:170 #8 0x08054003 in hash_search (f=1, s=0x0, buf=0x0, len=0) at match.c:202 #9 0x08054726 in match_sums (f=30, s=0x806d638, buf=0x0, len=128983684031) at match.c:319 #10 0x0804d1e4 in recv_files (f_in=30, flist=0x80712b1, local_name=0x6d &...

...ascending order of //tag,index// into the array of checksums. It then accesses the targets in ascending order and writes the index at the tag's location in the tag_table. For any set of targets with identical tags, the highest valued index is the resulting value in that location in the table. hash_search initializes "j" from the tag_table resulting in "j" being the largest index in a group of targets with the same tag value. The for loop that uses "j" as its primary iteration control iterates positively (ie. j++). The termination condition for the loop is any tag value...

...n I try to resync it with: rsync --inplace --ignore-times -vvvv --compress bigfile rsync://xxx at ... The '--ignore-times' is only to force a comparison. The real world scenario would be the sync of a truecrypt container. This transfer aborts with a timeout. The reason is that during the hash_search phase no packets are sent to the peer. If I omit the --compress option, tokens are sent regularly, preventing the timeout, but send_deflated_token() seems to collect an hours worth of responses before sending them out. Flushing them on a regular basis to prevent a timeout would be good. -Arne

...# strace -p 4942 Process 4942 attached - interrupt to quit ^CProcess 4942 detached root at st1:~# gdb /usr/bin/rsync 4942 --snip-- 0x00007f7a75e48914 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0 0x00007f7a75e48914 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x000000000042649f in hash_search (f=1, s=0x192d220, buf=0x192d1e0, len=214748364800) at match.c:236 #2 0x0000000000426cf4 in match_sums (f=1, s=0x192d220, buf=0x192d1e0, len=214748364800) at match.c:398 #3 0x000000000041a175 in send_files (f_in=0, f_out=1) at sender.c:391 #4 0x0000000000423620 in do_server_sender (f_in=0, f_out...

Hi, I recently came across a situation where "rsync --inplace" performs very poorly. If both the source and destination files contain long sequences of identical blocks, but not necessarily in the same location, the sender can spend an inordinate amount of CPU time finding matching blocks. In my case, I came across this problem while backing up multi-hundred-gigabyte MySQL database

...up]# rsync: writefd_unbuffered failed to write 55 bytes to socket [generator]: Broken pipe (32) rsync error: error in rsync protocol data stream (code 12) at io.c(1530) [generator=3.0.8] If I run the same command in gdb: Program received signal SIGSEGV, Segmentation fault. 0x000000000041f562 in hash_search (f=8, s=0x888320, buf=0x888680, len=3906587) at match.c:243 243 if (sum != s- >sums[aligned_i].sum1 Missing separate debuginfos, use: debuginfo-install libattr-2.4.44-6.fc14.x86_64 (gdb) back #0 0x000000000041f562 in hash_search (f=8, s=0x88...

...te /xback2_back1/home_rcr36/20110707-000502/ /mnt/btrfs/home_rcr36/current/ Where both file systems are local. Running strace on the process shows no activity. The hanging process is the rsync with the lowest PID. I attached gdb to the hanging rsync. Here is the back trace: 0x000000000041e5c4 in hash_search (f=4, s=0xcc1ec0, buf=0xcc1ef0, len=2000967340) at match.c:211 211 if (updating_basis_file && s->sums[i].offset < offset (gdb) back #0 0x000000000041e5c4 in hash_search (f=4, s=0xcc1ec0, buf=0xcc1ef0, len=2000967340) at match.c:211 #1 match_sums (f=4, s=...

Attached is a patch that makes window strides constant when files are walked with a constant block size. In these cases, it completely avoids all memmoves. In my simple local test of rsyncing 57MB of 10 local files, memmoved bytes went from 18MB to zero. I haven't tested this for a big variety of file cases. I think that this will always reduce the memmoves involved with walking a large

...%.0f)\n", (double)last_match+j,(double)n1); + map_ptr(buf,last_match+j,n1); + if (buf->m_len==0) break; + sum_update(buf->m_ptr,buf->m_len); } - if (i >= 0) last_match = offset + s->sums[i].len; else last_match = offset; @@ -127,45 +128,41 @@ } static void hash_search(int f,struct sum_struct *s, - struct map_struct *buf,OFF_T len) + struct map_struct *buf) { - OFF_T offset, end; + OFF_T offset; int j,k, last_i; char sum2[SUM_LENGTH]; uint32 s1, s2, sum; - schar *map; /* last_i is used to encourage adjacent matches, allowing the RLL coding of the...

Hello all, Thanks for all the ideas. I liked the remote to remote idea and have told my guide about it. Anyways, can anybody tell me which files to look into, if I want to look at the code for file reconstruction. I have studied Dr. Tridgell's thesis on the Rsync Algorithm and have understood the working of the algorithm. Now, I am trying to read the code to understand it better. Any help

...rg 4) sender.c:112: warning: unsigned int format, uint32_t arg (arg 6) exclude.c: In function `parse_filter_file': exclude.c:1053: warning: unsigned int format, uint32_t arg (arg 5) match.c: In function `matched': match.c:115: warning: int format, int32_t arg (arg 5) match.c: In function `hash_search': match.c:168: warning: unsigned int format, uint32_t arg (arg 3) match.c:185: warning: unsigned int format, long unsigned int arg (arg 4) match.c:185: warning: unsigned int format, long unsigned int arg (arg 5) match.c:219: warning: unsigned int format, uint32_t arg (arg 5) syscall.c: In func...

Hi, When using rsync-3.0.2 through 3.0.5, I get this error on a large dataset syncing from machine-a to machine-b: $ /bin/rsync -aHSz /local/. machine-b:/local/. invalid len passed to map_ptr: -1737287498 rsync error: error in file IO (code 11) at fileio.c(188) [sender=3.0.5] This happens no matter which side initiates the connection, so this fails in the same way: $ /bin/rsync -aHSz

Maybe security related mails should be sent elsewhere? I didn't notice any so here it goes: sender.c:receive_sums() s->count = read_int(f); .. s->sums = (struct sum_buf *)malloc(sizeof(s->sums[0])*s->count); if (!s->sums) out_of_memory("receive_sums"); for (i=0; i < (int) s->count;i++) { s->sums[i].sum1 = read_int(f);

...on `receive_sums': sender.c:50: warning: int format, different type arg (arg 3) sender.c:50: warning: int format, different type arg (arg 4) sender.c:50: warning: int format, different type arg (arg 5) gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -c match.c -o match.o match.c: In function `hash_search': match.c:144: warning: int format, different type arg (arg 3) match.c:146: warning: comparison between signed and unsigned match.c:162: warning: int format, different type arg (arg 3) match.c:162: warning: int format, different type arg (arg 5) match.c:184: warning: comparison between sig...

...xed_bd; With the above patches, I built it on a peecee (i386 architecture, actually a K6-2 - still a 32-bit machine). No warnings. Then I tried it on an alpha (LP64). With the above patches, I got gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -I./popt -c match.c -o match.o match.c: In function `hash_search': "match.c", line 199: warning: comparison between signed and unsigned "match.c", line 235: warning: comparison between signed and unsigned "match.c", line 265: warning: comparison between signed and unsigned These appear to come from applying MIN to a size_t and...

...client': main.c:664: warning: unused variable `whole_file' main.c:665: warning: unused variable `write_batch' gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -I./popt -c checksum.c -o checksum.o gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -I./popt -c match.c -o match.o match.c: In function `hash_search': match.c:144: warning: int format, different type arg (arg 3) match.c:146: warning: comparison between signed and unsigned match.c:146: warning: signed and unsigned type in conditional expression match.c:162: warning: int format, different type arg (arg 3) match.c:162: warning: int format, dif...

...client': main.c:664: warning: unused variable `whole_file' main.c:665: warning: unused variable `write_batch' gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -I./popt -c checksum.c -o checksum.o gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -I./popt -c match.c -o match.o match.c: In function `hash_search': match.c:144: warning: int format, size_t arg (arg 3) match.c:162: warning: int format, size_t arg (arg 3) match.c:162: warning: int format, size_t arg (arg 5) gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -I./popt -c syscall.c -o syscall.o gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -I./popt -c...