Hello all, I posted earlier this message to the Apache User''s list but I haven''t received any response, so I forward this message to this list. Maybe sombeody can help me to solve this problem. Regards Vincent. ---------------------------------------------------- Original Message ----------------------------------------------------- Subject: Apache as a Rsync HTTP proxy From: "Vincent Blondel" <vincent@xtra-net.org> Date: Mon, December 19, 2005 18:46 To: users@httpd.apache.org --------------------------------------------------------------------------------------------------------------------------- Hi all, Recently I had to set-up our external http proxy (apache 1.3.x) in such a way that it accepts rsync traffic coming from our lan. So I decided to add a ''CONNECT'' in my httpd.conf. This is all working now but not so fine as I could expect it. Sometimes the communication is broken and I get this error message coming from rsync : ... rsync: connection unexpectedly closed ... [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(443) rsync: connection unexpectedly closed ... [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(443) ... I give you below my current config. ----- ServerType standalone ServerName sbezoxaa Listen sbezoxaa:8081 User www Group www ServerRoot "/var/www" MinSpareServers 16 StartServers 16 MaxSpareServers 16 MaxClients 100 MaxRequestsPerChild 1000 LoadModule rewrite_module /usr/lib/apache/modules/mod_rewrite.so LoadModule security_module /usr/local/lib/mod_security.so LoadModule proxy_module /usr/lib/apache/modules/libproxy.so LoadModule mime_magic_module /usr/lib/apache/modules/mod_mime_magic.so KeepAlive On MaxKeepAliveRequests 100 KeepAliveTimeout 15 Timeout 300 IdentityCheck Off HostnameLookups Off UseCanonicalName On ServerSignature Off PidFile run/httpd.pid LockFile run/accept.lock ErrorLog "|/usr/local/sbin/cronolog /var/www/logs/%Y/%m/% d/error.log" LogLevel warn LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined CustomLog "|/usr/local/sbin/cronolog /var/www/logs/%Y/%m/% d/access.log" combined TypesConfig etc/apache/mime.types DefaultType text/plain <IfModule mod_mime_magic.c> MIMEMagicFile etc/apache/magic </IfModule> DocumentRoot /tmp RewriteLog /dev/null TransferLog /dev/null AccessConfig /dev/null ResourceConfig /dev/null <Directory /> Options -FollowSymLinks -SymLinksIfOwnerMatch AllowOverride None </Directory> ProxyRequests On ProxyVia On # Note to add these connection authorization for Rsync # 443 : https # 873 : rsync AllowCONNECT 443 873 <Directory proxy:*> Order deny,allow Allow from all </Directory> CacheRoot "/tmp" CacheSize 1048576 CacheGcInterval 1 CacheMaxExpire 24 CacheLastModifiedFactor 0.1 CacheDefaultExpire 1 ProxyReceiveBufferSize 2048 ProxyIOBufferSize 131072 ----- So if somebody gets any experience with this, I would appreciate some help to solve this problem. Regards Vincent -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> Subject: Apache as a Rsync HTTP proxy > From: "Vincent Blondel" <vincent@xtra-net.org> > Date: Mon, December 19, 2005 18:46 > To: users@httpd.apache.org > > Hi all, > > Recently I had to set-up our external http proxy (apache 1.3.x) > in such a way that it accepts rsync traffic coming from our lan. > > So I decided to add a ''CONNECT'' in my httpd.conf. This is all > working now but not so fine as I could expect it. Sometimes > the communication is broken and I get this error message coming > from rsync : > > ... > rsync: connection unexpectedly closed ... [receiver] > rsync error: error in rsync protocol data stream (code 12) at io.c(443) > rsync: connection unexpectedly closed ... [generator] > rsync error: error in rsync protocol data stream (code 12) at io.c(443) > ... > > I give you below my current config.[ snip ] So - you have a server(daemon) being proxied through Apache and you are fetching files from that server? It looks like Apache closes the connections before the rsync client expects it. But only sometimes? Do the files always get transferred reliably and this problem crops up right at the end? This could be the cause: http://httpd.apache.org/docs/1.3/mod/mod_proxy.html#proxyiobuffersize "When a response is received which fits entirely within the IO buffer size, the remote HTTP or FTP server socket will be closed before an attempt is made to write the response to the client. This ensures that the remote server does not remain connected unnecessarily while the response is delivered to a slow client." I think rsync may be picky about when connections get closed. If that doesn''t help - please provide more info, like version of rsync, command used, time it takes after the start of the rsync for the error to appear, etc. John
>> Subject: Apache as a Rsync HTTP proxy >> From: "Vincent Blondel" <vincent@xtra-net.org> >> Date: Mon, December 19, 2005 18:46 >> To: users@httpd.apache.org >> >> Hi all, >> >> Recently I had to set-up our external http proxy (apache 1.3.x) >> in such a way that it accepts rsync traffic coming from our lan. >> >> So I decided to add a ''CONNECT'' in my httpd.conf. This is all >> working now but not so fine as I could expect it. Sometimes >> the communication is broken and I get this error message coming >> from rsync : >> >> ... >> rsync: connection unexpectedly closed ... [receiver] >> rsync error: error in rsync protocol data stream (code 12) at io.c(443) >> rsync: connection unexpectedly closed ... [generator] >> rsync error: error in rsync protocol data stream (code 12) at io.c(443) >> ... >> >> I give you below my current config. > [ snip ] > > So - you have a server(daemon) being proxied through Apache and you > are fetching files from that server? > > It looks like Apache closes the connections before the rsync client > expects it. But only sometimes? > > Do the files always get transferred reliably and this problem crops > up right at the end? This could be the cause: > > http://httpd.apache.org/docs/1.3/mod/mod_proxy.html#proxyiobuffersize > > "When a response is received which fits entirely within the IO > buffer size, the remote HTTP or FTP server socket will be closed > before an attempt is made to write the response to the client. > This ensures that the remote server does not remain connected > unnecessarily while the response is delivered to a slow client." > > I think rsync may be picky about when connections get closed. > > If that doesn''t help - please provide more info, like version of rsync, > command used, time it takes after the start of the rsync for the error > to appear, etc. > > John > >So, to be very clear, I get a server (in my LAN) that synchronizes a website (on the net) twice an hour with the rsync utility. My web server is located in a vlan that doesn''t have any access on the internet. So the server access the internet through an HTTP proxy ( apache 1.3.X ) with rsync 2.6.6. Rsync Server ^ | internet ^ | router ^ | Apache HTTP proxy ^ | Server (Rsync client) When you have a look at my httpd config, I am also using ProxyReceiveBufferSize with value 2048 and ProxyIOBufferSize with value 131072 but the problem is always happening. Each time, rsync starts, the connection breaks a few seconds after it receives the files list and at the end of the day I get 50 rsync idle processes. I get exacly the same situation on the HTTP proxy that gets 50 idle processes waiting an answer from the rsync processes. Do you get some info for me ??? Regards Vincent.
Vincent, Thanks for the additional info. "the problem is always happening" is different from "sometimes" that you used in your original email. The idle rsync processes implies that Apache is closing the channel on its own. So... What is the command that you use to run the rsync client? What environment variables do you set, like RSYNC_PROXY? John On Tue, 20 Dec 2005, Vincent Blondel <vincent@xtra-net.org> wrote:> > So, to be very clear, I get a server (in my LAN) that synchronizes a website (on the net) twice an hour with the rsync > utility. My web server is located in a vlan that doesn''t have any access on the internet. > > So the server access the internet through an HTTP proxy ( apache 1.3.X ) with rsync 2.6.6. > > Rsync Server > ^ > | > internet > ^ > | > router > ^ > | > Apache HTTP proxy > ^ > | > Server (Rsync client) > > When you have a look at my httpd config, I am also using ProxyReceiveBufferSize with value 2048 and ProxyIOBufferSize with > value 131072 but the problem is always happening. > > Each time, rsync starts, the connection breaks a few seconds after it receives the files list and at the end of the day I > get 50 rsync idle processes. I get exacly the same situation on the HTTP proxy that gets 50 idle processes waiting an > answer from the rsync processes.
John, see below .... I hope this can help you.> Vincent, > > Thanks for the additional info. "the problem is always happening" is > different from "sometimes" that you used in your original email. > > The idle rsync processes implies that Apache is closing the channel > on its own. So... > > What is the command that you use to run the rsync client?rsync -clprtzv --delete> > What environment variables do you set, like RSYNC_PROXY?This is the result of "env" command : _=/usr/bin/env BLOCKSIZE=K EDITOR=vi FTP_PASSIVE_MODE=YES HOME=/root MAIL=/var/mail/root PAGER=more PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:/root/bin RSYNC_PROXY=sbezoxaa:8081 PS1=`whoami`@`hostname -s` [$PWD] # SHELL=/bin/ksh TERM=xterm USER=root _AST_FEATURES=UNIVERSE - ucb> > John > > > On Tue, 20 Dec 2005, Vincent Blondel <vincent@xtra-net.org> wrote: >> >> So, to be very clear, I get a server (in my LAN) that synchronizes a website (on the net) twice an hour with the rsync >> utility. My web server is located in a vlan that doesn''t have any access on the internet. >> >> So the server access the internet through an HTTP proxy ( apache 1.3.X ) with rsync 2.6.6. >> >> Rsync Server >> ^ >> | >> internet >> ^ >> | >> router >> ^ >> | >> Apache HTTP proxy >> ^ >> | >> Server (Rsync client) >> >> When you have a look at my httpd config, I am also using ProxyReceiveBufferSize with value 2048 and ProxyIOBufferSize >> with >> value 131072 but the problem is always happening. >> >> Each time, rsync starts, the connection breaks a few seconds after it receives the files list and at the end of the day I >> get 50 rsync idle processes. I get exacly the same situation on the HTTP proxy that gets 50 idle processes waiting an >> answer from the rsync processes. > >
Vincent, I don''t see any problems with what you have. I have no experience with using a proxy like this, and I can find no description of how all this works together, either. So I have to punt. Sorry. Hopefully, someone else on the rsync mailing list can help you with further suggections. John On Wed, 21 Dec 2005, Vincent Blondel <vincent@xtra-net.org> wrote:> > John, > > see below .... > > I hope this can help you. > >> Vincent, >> >> Thanks for the additional info. "the problem is always happening" is >> different from "sometimes" that you used in your original email. >> >> The idle rsync processes implies that Apache is closing the channel >> on its own. So... >> >> What is the command that you use to run the rsync client? > > rsync -clprtzv --delete > >> >> What environment variables do you set, like RSYNC_PROXY? > > This is the result of "env" command : > > _=/usr/bin/env > BLOCKSIZE=K > EDITOR=vi > FTP_PASSIVE_MODE=YES > HOME=/root > MAIL=/var/mail/root > PAGER=more > PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:/root/bin > RSYNC_PROXY=sbezoxaa:8081 > PS1=`whoami`@`hostname -s` [$PWD] # > SHELL=/bin/ksh > TERM=xterm > USER=root > _AST_FEATURES=UNIVERSE - ucb > >> >> John >> >> >> On Tue, 20 Dec 2005, Vincent Blondel <vincent@xtra-net.org> wrote: >>> >>> So, to be very clear, I get a server (in my LAN) that synchronizes a website (on the net) twice an hour with the rsync >>> utility. My web server is located in a vlan that doesn''t have any access on the internet. >>> >>> So the server access the internet through an HTTP proxy ( apache 1.3.X ) with rsync 2.6.6. >>> >>> Rsync Server >>> ^ >>> | >>> internet >>> ^ >>> | >>> router >>> ^ >>> | >>> Apache HTTP proxy >>> ^ >>> | >>> Server (Rsync client) >>> >>> When you have a look at my httpd config, I am also using ProxyReceiveBufferSize with value 2048 and ProxyIOBufferSize >>> with >>> value 131072 but the problem is always happening. >>> >>> Each time, rsync starts, the connection breaks a few seconds after it receives the files list and at the end of the day I >>> get 50 rsync idle processes. I get exacly the same situation on the HTTP proxy that gets 50 idle processes waiting an >>> answer from the rsync processes. >> > > -- > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html