Hello all,
I posted earlier this message to the Apache User''s list but I
haven''t received any response, so I forward this message to
this list.
Maybe sombeody can help me to solve this problem.
Regards
Vincent.
---------------------------------------------------- Original Message
-----------------------------------------------------
Subject: Apache as a Rsync HTTP proxy
From: "Vincent Blondel" <vincent@xtra-net.org>
Date: Mon, December 19, 2005 18:46
To: users@httpd.apache.org
---------------------------------------------------------------------------------------------------------------------------
Hi all,
Recently I had to set-up our external http proxy (apache 1.3.x) in such a way
that it accepts rsync traffic coming from our
lan.
So I decided to add a ''CONNECT'' in my httpd.conf. This is all
working now but not so fine as I could expect it. Sometimes
the communication is broken and I get this error message coming from rsync :
...
rsync: connection unexpectedly closed ... [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(443) rsync:
connection unexpectedly closed ...
[generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(443) ...
I give you below my current config.
-----
ServerType standalone
ServerName sbezoxaa
Listen sbezoxaa:8081
User www
Group www
ServerRoot "/var/www"
MinSpareServers 16
StartServers 16
MaxSpareServers 16
MaxClients 100
MaxRequestsPerChild 1000
LoadModule rewrite_module /usr/lib/apache/modules/mod_rewrite.so LoadModule
security_module
/usr/local/lib/mod_security.so
LoadModule proxy_module /usr/lib/apache/modules/libproxy.so
LoadModule mime_magic_module /usr/lib/apache/modules/mod_mime_magic.so
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
Timeout 300
IdentityCheck Off
HostnameLookups Off
UseCanonicalName On
ServerSignature Off
PidFile run/httpd.pid
LockFile run/accept.lock
ErrorLog "|/usr/local/sbin/cronolog /var/www/logs/%Y/%m/%
d/error.log"
LogLevel warn
LogFormat "%h %l %u %t \"%r\" %>s %b
\"%{Referer}i\"
\"%{User-Agent}i\"" combined
CustomLog "|/usr/local/sbin/cronolog /var/www/logs/%Y/%m/%
d/access.log" combined
TypesConfig etc/apache/mime.types
DefaultType text/plain
<IfModule mod_mime_magic.c>
MIMEMagicFile etc/apache/magic
</IfModule>
DocumentRoot /tmp
RewriteLog /dev/null
TransferLog /dev/null
AccessConfig /dev/null
ResourceConfig /dev/null
<Directory />
Options -FollowSymLinks -SymLinksIfOwnerMatch
AllowOverride None
</Directory>
ProxyRequests On
ProxyVia On
# Note to add these connection authorization for Rsync
# 443 : https
# 873 : rsync
AllowCONNECT 443 873
<Directory proxy:*>
Order deny,allow
Allow from all
</Directory>
CacheRoot "/tmp"
CacheSize 1048576
CacheGcInterval 1
CacheMaxExpire 24
CacheLastModifiedFactor 0.1
CacheDefaultExpire 1
ProxyReceiveBufferSize 2048
ProxyIOBufferSize 131072
-----
So if somebody gets any experience with this, I would appreciate some help to
solve this problem.
Regards
Vincent
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> Subject: Apache as a Rsync HTTP proxy > From: "Vincent Blondel" <vincent@xtra-net.org> > Date: Mon, December 19, 2005 18:46 > To: users@httpd.apache.org > > Hi all, > > Recently I had to set-up our external http proxy (apache 1.3.x) > in such a way that it accepts rsync traffic coming from our lan. > > So I decided to add a ''CONNECT'' in my httpd.conf. This is all > working now but not so fine as I could expect it. Sometimes > the communication is broken and I get this error message coming > from rsync : > > ... > rsync: connection unexpectedly closed ... [receiver] > rsync error: error in rsync protocol data stream (code 12) at io.c(443) > rsync: connection unexpectedly closed ... [generator] > rsync error: error in rsync protocol data stream (code 12) at io.c(443) > ... > > I give you below my current config.[ snip ] So - you have a server(daemon) being proxied through Apache and you are fetching files from that server? It looks like Apache closes the connections before the rsync client expects it. But only sometimes? Do the files always get transferred reliably and this problem crops up right at the end? This could be the cause: http://httpd.apache.org/docs/1.3/mod/mod_proxy.html#proxyiobuffersize "When a response is received which fits entirely within the IO buffer size, the remote HTTP or FTP server socket will be closed before an attempt is made to write the response to the client. This ensures that the remote server does not remain connected unnecessarily while the response is delivered to a slow client." I think rsync may be picky about when connections get closed. If that doesn''t help - please provide more info, like version of rsync, command used, time it takes after the start of the rsync for the error to appear, etc. John
>> Subject: Apache as a Rsync HTTP proxy >> From: "Vincent Blondel" <vincent@xtra-net.org> >> Date: Mon, December 19, 2005 18:46 >> To: users@httpd.apache.org >> >> Hi all, >> >> Recently I had to set-up our external http proxy (apache 1.3.x) >> in such a way that it accepts rsync traffic coming from our lan. >> >> So I decided to add a ''CONNECT'' in my httpd.conf. This is all >> working now but not so fine as I could expect it. Sometimes >> the communication is broken and I get this error message coming >> from rsync : >> >> ... >> rsync: connection unexpectedly closed ... [receiver] >> rsync error: error in rsync protocol data stream (code 12) at io.c(443) >> rsync: connection unexpectedly closed ... [generator] >> rsync error: error in rsync protocol data stream (code 12) at io.c(443) >> ... >> >> I give you below my current config. > [ snip ] > > So - you have a server(daemon) being proxied through Apache and you > are fetching files from that server? > > It looks like Apache closes the connections before the rsync client > expects it. But only sometimes? > > Do the files always get transferred reliably and this problem crops > up right at the end? This could be the cause: > > http://httpd.apache.org/docs/1.3/mod/mod_proxy.html#proxyiobuffersize > > "When a response is received which fits entirely within the IO > buffer size, the remote HTTP or FTP server socket will be closed > before an attempt is made to write the response to the client. > This ensures that the remote server does not remain connected > unnecessarily while the response is delivered to a slow client." > > I think rsync may be picky about when connections get closed. > > If that doesn''t help - please provide more info, like version of rsync, > command used, time it takes after the start of the rsync for the error > to appear, etc. > > John > >So, to be very clear, I get a server (in my LAN) that synchronizes a website (on the net) twice an hour with the rsync utility. My web server is located in a vlan that doesn''t have any access on the internet. So the server access the internet through an HTTP proxy ( apache 1.3.X ) with rsync 2.6.6. Rsync Server ^ | internet ^ | router ^ | Apache HTTP proxy ^ | Server (Rsync client) When you have a look at my httpd config, I am also using ProxyReceiveBufferSize with value 2048 and ProxyIOBufferSize with value 131072 but the problem is always happening. Each time, rsync starts, the connection breaks a few seconds after it receives the files list and at the end of the day I get 50 rsync idle processes. I get exacly the same situation on the HTTP proxy that gets 50 idle processes waiting an answer from the rsync processes. Do you get some info for me ??? Regards Vincent.
Vincent,
Thanks for the additional info. "the problem is always happening" is
different from "sometimes" that you used in your original email.
The idle rsync processes implies that Apache is closing the channel
on its own. So...
What is the command that you use to run the rsync client?
What environment variables do you set, like RSYNC_PROXY?
John
On Tue, 20 Dec 2005, Vincent Blondel <vincent@xtra-net.org>
wrote:>
> So, to be very clear, I get a server (in my LAN) that synchronizes a
website (on the net) twice an hour with the rsync
> utility. My web server is located in a vlan that doesn''t have any
access on the internet.
>
> So the server access the internet through an HTTP proxy ( apache 1.3.X )
with rsync 2.6.6.
>
> Rsync Server
> ^
> |
> internet
> ^
> |
> router
> ^
> |
> Apache HTTP proxy
> ^
> |
> Server (Rsync client)
>
> When you have a look at my httpd config, I am also using
ProxyReceiveBufferSize with value 2048 and ProxyIOBufferSize with
> value 131072 but the problem is always happening.
>
> Each time, rsync starts, the connection breaks a few seconds after it
receives the files list and at the end of the day I
> get 50 rsync idle processes. I get exacly the same situation on the HTTP
proxy that gets 50 idle processes waiting an
> answer from the rsync processes.
John, see below .... I hope this can help you.> Vincent, > > Thanks for the additional info. "the problem is always happening" is > different from "sometimes" that you used in your original email. > > The idle rsync processes implies that Apache is closing the channel > on its own. So... > > What is the command that you use to run the rsync client?rsync -clprtzv --delete> > What environment variables do you set, like RSYNC_PROXY?This is the result of "env" command : _=/usr/bin/env BLOCKSIZE=K EDITOR=vi FTP_PASSIVE_MODE=YES HOME=/root MAIL=/var/mail/root PAGER=more PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:/root/bin RSYNC_PROXY=sbezoxaa:8081 PS1=`whoami`@`hostname -s` [$PWD] # SHELL=/bin/ksh TERM=xterm USER=root _AST_FEATURES=UNIVERSE - ucb> > John > > > On Tue, 20 Dec 2005, Vincent Blondel <vincent@xtra-net.org> wrote: >> >> So, to be very clear, I get a server (in my LAN) that synchronizes a website (on the net) twice an hour with the rsync >> utility. My web server is located in a vlan that doesn''t have any access on the internet. >> >> So the server access the internet through an HTTP proxy ( apache 1.3.X ) with rsync 2.6.6. >> >> Rsync Server >> ^ >> | >> internet >> ^ >> | >> router >> ^ >> | >> Apache HTTP proxy >> ^ >> | >> Server (Rsync client) >> >> When you have a look at my httpd config, I am also using ProxyReceiveBufferSize with value 2048 and ProxyIOBufferSize >> with >> value 131072 but the problem is always happening. >> >> Each time, rsync starts, the connection breaks a few seconds after it receives the files list and at the end of the day I >> get 50 rsync idle processes. I get exacly the same situation on the HTTP proxy that gets 50 idle processes waiting an >> answer from the rsync processes. > >
Vincent,
I don''t see any problems with what you have.
I have no experience with using a proxy like this, and I can find
no description of how all this works together, either.
So I have to punt. Sorry. Hopefully, someone else on the rsync
mailing list can help you with further suggections.
John
On Wed, 21 Dec 2005, Vincent Blondel <vincent@xtra-net.org>
wrote:>
> John,
>
> see below ....
>
> I hope this can help you.
>
>> Vincent,
>>
>> Thanks for the additional info. "the problem is always
happening" is
>> different from "sometimes" that you used in your original
email.
>>
>> The idle rsync processes implies that Apache is closing the channel
>> on its own. So...
>>
>> What is the command that you use to run the rsync client?
>
> rsync -clprtzv --delete
>
>>
>> What environment variables do you set, like RSYNC_PROXY?
>
> This is the result of "env" command :
>
> _=/usr/bin/env
> BLOCKSIZE=K
> EDITOR=vi
> FTP_PASSIVE_MODE=YES
> HOME=/root
> MAIL=/var/mail/root
> PAGER=more
>
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:/root/bin
> RSYNC_PROXY=sbezoxaa:8081
> PS1=`whoami`@`hostname -s` [$PWD] #
> SHELL=/bin/ksh
> TERM=xterm
> USER=root
> _AST_FEATURES=UNIVERSE - ucb
>
>>
>> John
>>
>>
>> On Tue, 20 Dec 2005, Vincent Blondel <vincent@xtra-net.org>
wrote:
>>>
>>> So, to be very clear, I get a server (in my LAN) that synchronizes
a website (on the net) twice an hour with the rsync
>>> utility. My web server is located in a vlan that doesn''t
have any access on the internet.
>>>
>>> So the server access the internet through an HTTP proxy ( apache
1.3.X ) with rsync 2.6.6.
>>>
>>> Rsync Server
>>> ^
>>> |
>>> internet
>>> ^
>>> |
>>> router
>>> ^
>>> |
>>> Apache HTTP proxy
>>> ^
>>> |
>>> Server (Rsync client)
>>>
>>> When you have a look at my httpd config, I am also using
ProxyReceiveBufferSize with value 2048 and ProxyIOBufferSize
>>> with
>>> value 131072 but the problem is always happening.
>>>
>>> Each time, rsync starts, the connection breaks a few seconds after
it receives the files list and at the end of the day I
>>> get 50 rsync idle processes. I get exacly the same situation on the
HTTP proxy that gets 50 idle processes waiting an
>>> answer from the rsync processes.
>>
>
> --
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html