On Sunday, March 24, 2013 9:57:12 AM UTC+6, Markiyan Kushnir wrote:> Tested svnup for a while, and I can say it does its job well, and works > basically as I would expect, so thanks for your initiative. Although it > appears to be quite resource greedy. Most of the time it showed > something like: > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 22270 mkushnir 1 102 0 44944K 31804K CPU0 1 6:22 97.56% a.outIt's because of typo in the send_command() procedure. I've placed the patched svnup.c (0.56), the diff and two statically linked binaries on http://ftp.ufanet.ru/pub/boco/freebsd/svnup/ No more CPU eating and/or strange lockups (so far). Tested both against local and remote repository.
On Thursday, April 11, 2013 1:14:57 PM UTC+6, mrb... at gmail.com wrote:> I've placed the patched svnup.c (0.56), the diff and two statically linked binaries on http://ftp.ufanet.ru/pub/boco/freebsd/svnup/I'm sorry, svnup.c.diff is the patch against filtered thru indent svnup.c, with different formatting (see README). The patch against the original svnup.c v0.56 is named svnup.c.diff-original.
On Thu, 11 Apr 2013, mrboco at gmail.com wrote: > On Sunday, March 24, 2013 9:57:12 AM UTC+6, Markiyan Kushnir wrote: > > Tested svnup for a while, and I can say it does its job well, and works > > basically as I would expect, so thanks for your initiative. Although it > > appears to be quite resource greedy. Most of the time it showed > > something like: > > > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > > 22270 mkushnir 1 102 0 44944K 31804K CPU0 1 6:22 97.56% a.out > > It's because of typo in the send_command() procedure. > > I've placed the patched svnup.c (0.56), the diff and two statically > linked binaries on http://ftp.ufanet.ru/pub/boco/freebsd/svnup/ > > No more CPU eating and/or strange lockups (so far). Tested both > against local and remote repository. I'm sorry, but even ignoring all of your whitespace and style(9) differences, your patch appears to go well beyond correcting a typo, which I can't spot anyway, though I'm sure John will know what it is. Care to explain a little more? Also, what advantage, in this particular case, is there in statically linking? Here it turns a 21.5K i386 binary into one of 575K. If this makes it into base, as I hope it may, that seems a little excessive? cheers, Ian
> I'm sorry, but even ignoring all of your whitespace and style(9) > differences, your patch appears to go well beyond correcting a typo, > which I can't spot anyway, though I'm sure John will know what it is. > > Care to explain a little more?Sure. Typo is "strlen(command - total_bytes_written)" instead of "strlen(command) - total_bytes_written". But then I've noticed that John have used non-blocking IO (useless in our case) while not handling IO errors, that lead us to permanent cycling on EAGAIN. So I've replaced John's code with the simpler one that blocks on write() and removed fcntl(..., O_NONBLOCK). Then I've run a lot of tests again my own repository located on the same machine and sometime svnup was locked permanently with send/recieve queues filled up (remote fetching was OK). I've digged a bit in svn code and deceided that it would be helpful to use SO_KEEPALIVE and to set SNDBUF (at least) to the COMMAND_BUFFER value.> Also, what advantage, in this particular case, is there in statically > linking? Here it turns a 21.5K i386 binary into one of 575K. If this > makes it into base, as I hope it may, that seems a little excessive?There is no advantage. I've compiled both binaries for myself to be able to "svnup" hundreds of mahines w/o wondering about exact release =)
On Friday, April 12, 2013 1:09:53 AM UTC+6, Markiyan Kushnir wrote:> > Another thing that might be worth of attention, the patched version has > > been again back to slower checkout time: > > real 91m38.824s > > user 0m26.216s > > sys 0m13.858s > > at 4 Mbit/s link, while the original 0.56 takes ~55min given the same > > load/network conditions.You may just fix typo and not use other fixes. I doubt they actual for remote fetching.
On Friday, April 12, 2013 1:28:43 PM UTC+6, Markiyan Kushnir wrote:> It would be nice to get comparable time from svnup.I think we could get comparable time only with svn. Sorry.
> In the previous version (0.61), the process of checking > file names against the list of known files in the > repository was inefficient and most likely accounts for > the slow down you're seeing.? I've reimplemented it using > a binary search tree and the lookup phase is no longer a > bottleneck.I'm sorry but 0.62 still locks while fetching from a local repository: last pid: 74701; load averages: 2.24, 2.52, 2.56 up 772+03:32:23 13:19:55 96 processes: 2 running, 94 sleeping CPU: 14.8% user, 0.0% nice, 40.3% system, 0.7% interrupt, 44.2% idle Mem: 1191M Active, 436M Inact, 248M Wired, 76M Cache, 112M Buf, 50M Free Swap: 1024M Total, 232M Used, 792M Free, 22% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 30193 root 1 117 0 56220K 9108K CPU1 1 99:16 96.39% svnup The send/receive queues are filled up and not changing over time: root at alpha:~# netstat -an | fgrep -w 3690 tcp4 8192 24576 81.30.199.66.3690 81.30.199.66.44473 ESTABLISHED tcp4 24576 16384 81.30.199.66.44473 81.30.199.66.3690 ESTABLISHED tcp4 0 0 *.3690 *.* LISTEN root at alpha:~# kdump | head -40 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) 30193 svnup RET write -1 errno 35 Resource temporarily unavailable 30193 svnup CALL write(0x3,0x8843a000,0xd91) I think you should either use blocking IO or catch IO errors. And please consider to set the socket options too. Thanks.