search for: sum_buf

Hi all I see the rsync source and rsync makes hashing table and search hashing table tag_table to find the index of array struct sum_buf , which is a element of struct sum_struct. According to the source code, variable last_i is used to encourage adjacent matches allowing the RLL coding of the output to work more efficiently. Why last_i makes more efficiency? I can't understanding what last_i makes and when last_i is used....

...4432 n=131072 rem=0 ERROR: out of memory in receive_sums [sender] [sender] _exit_cleanup(code=22, file=util2.c, line=105): entered rsync error: error allocating core memory buffers (code 22) at util2.c(105) [sender=3.1.3] This is getting called: 92 if (!(s->sums = new_array(struct sum_buf, s->count))) 93 out_of_memory("receive_sums"); And the size of a sum_buf(40 bytes) * the number of sums (33554432) exceeds MALLOC_MAX. How is this supposed to work/why is it breaking here, when I'm pretty sure I've transferred files bigger than this befor...

Maybe security related mails should be sent elsewhere? I didn't notice any so here it goes: sender.c:receive_sums() s->count = read_int(f); .. s->sums = (struct sum_buf *)malloc(sizeof(s->sums[0])*s->count); if (!s->sums) out_of_memory("receive_sums"); for (i=0; i < (int) s->count;i++) { s->sums[i].sum1 = read_int(f); read_buf(f,s->sums[i].sum2,csum_length); .. If I read this right, given high enough s->count makes the mal...

...sum.sums = NULL; */ - if (verbose > 3) + if (sum.count && verbose > 3) rprintf(FINFO,"count=%d rem=%d n=%d flength=%.0f\n", - s->count,s->remainder,s->n,(double)s->flength); + sum.count,sum.remainder,sum.n,(double)sum.flength); - s->sums = (struct sum_buf *)malloc(sizeof(s->sums[0])*s->count); - if (!s->sums) out_of_memory("generate_sums"); - - for (i=0;i<count;i++) { - int n1 = MIN(len,n); - char *map = map_ptr(buf,offset,n1); + write_int(f_out, sum.count); + write_int(f_out, sum.n); + write_int(f_out, sum.remainder); -...

...list (e.g., I don't see a transfer of the total count over the wire used to optimize the allocation on the receiver). 2. The per-block overhead for the checksums for each file as it is processed. This memory exists only for the duration of one file. This is 32 bytes per file (a sum_buf) allocated as on memory chunk. This exists on the receiver as it is computed and transmitted, and on the sender as it receives it and uses it to match against the new file. 3. The match tables built to determine the delta between the original file and the new file. I haven't...

...fer of the total > count over the wire used to optimize the allocation on the > receiver). > > 2. The per-block overhead for the checksums for each file as it is > processed. This memory exists only for the duration of one file. > > This is 32 bytes per file (a sum_buf) allocated as on memory chunk. > This exists on the receiver as it is computed and transmitted, and > on the sender as it receives it and uses it to match against the > new file. > > 3. The match tables built to determine the delta between the original > file and the...

I was planning to use rsync to backup to a second drive, but I ran out of swap space. No problem, I will let rsync do it a few directories at a time overnight when the computer usually sits idle except for incomming email. I have 2 question: 1. How much memory does each file to be copied need. Obvisiouly I have too many files. 2. Why does this command work: rsync -ax /usr/xx /backup/usr/

...on @@ -406,7 +406,8 @@ OFF_T flength; /**< total file length */ size_t count; /**< how many chunks */ size_t remainder; /**< flength % block_length */ - size_t n; /**< block_length */ + size_t blength; /**< block_length */ + size_t s2length; /**< sum2_length */ struct sum_buf *sums; /**< points to info for each chunk */ }; --- proto.h Sat Mar 29 12:18:02 2003 +++ proto.h Sat Mar 29 12:15:38 2003 @@ -91,6 +91,7 @@ struct file_list *flist_new(void); void flist_free(struct file_list *flist); char *f_name(struct file_struct *f); +void write_sum_head(int f, struct...

...on @@ -385,7 +385,8 @@ OFF_T flength; /**< total file length */ size_t count; /**< how many chunks */ size_t remainder; /**< flength % block_length */ - size_t n; /**< block_length */ + size_t blength; /**< block_length */ + size_t s2length; /**< csum_length */ struct sum_buf *sums; /**< points to info for each chunk */ }; --- generator.c Sat Mar 22 03:38:21 2003 +++ generator.c.dynsumlen Sat Mar 22 09:47:43 2003 @@ -109,13 +109,21 @@ /* - send a header that says "we have no checksums" down the f_out fd + * NULL sum_struct means we have no check...

...map_struct *buf, size_t len, int f_out, ! int flist_idx) { ! size_t i, j; struct sum_struct sum; OFF_T offset = 0; sum_sizes_sqroot(&sum, len); write_sum_head(f_out, &sum); + if (write_batch) { /* save all the sums to write out */ + sum.sums = new_array(struct sum_buf, sum.count); + if (!sum.sums) out_of_memory("generate_and_send_sums"); + } for (i = 0; i < sum.count; i++) { unsigned int n1 = MIN(len, sum.blength); *************** *** 256,265 **** --- 267,295 ---- write_buf(f_out, sum2, sum.s2length); len -= n1; offset += n1...

...char *current; - struct string_area *next; -}; +#define FILE_EXTENT (256 * 1024) +#define HLINK_EXTENT (128 * 1024) struct file_list { int count; int malloced; + alloc_pool_t file_pool; + alloc_pool_t hlink_pool; struct file_struct **files; - struct string_area *string_area; }; struct sum_buf { Binary files cvs/t_unsafe and pool2/t_unsafe differ Binary files cvs/tls and pool2/tls differ Binary files cvs/trimslash and pool2/trimslash differ Binary files cvs/wildtest and pool2/wildtest differ

...(f_out, s->sums[i].sum2, csum_length); } } else { /* we don't have checksums */ + rprintf(FINFO, "#ET# send_sums: don't have checksums\n"); write_int(f_out, 0); write_int(f_out, block_size); write_int(f_out, 0); } @@ -193,24 +195,23 @@ s->sums = (struct sum_buf *)malloc(sizeof(s->sums[0])*s->count); if (!s->sums) out_of_memory("generate_sums"); for (i=0;i<count;i++) { - int n1 = MIN(len,n); - char *map = map_ptr(buf,offset,n1); + map_ptr(buf,offset,n); - s->sums[i].sum1 = get_checksum1(map,n1); - get_checksum2(map,n...