rsync - Sep 2002 - AIX & Large File Support Problem (+ Solution)

Hi,

I just wanted to relate the solution to a problem I was having to hopefully
save someone else a day of frustration.  I'm using rsync-2.5.5 on AIX 4.3,
compiled with gcc 2.95.3.  The file I was sync'ing was very large (>2GB).
Despite being configured with --enable-largefiles (which #defines
_LARGE_FILES properly for AIX), and despite the fact that the initial
transfer of said file worked fine, I would always get the following error
when trying to sync up the file:

  write failed on myfile.txt.3 : Error 0
  rsync error: error in file IO (code 11) at receiver.c(272)
  rsync: connection unexpectedly closed (48 bytes read so far)
  rsync error: error in rsync protocol data stream (code 12) at io.c(150)

This would always happen when rsync had transferred >2GB.  It seems that
AIX's write() doesn't like to set errno until the second time you try to
call write().  So, after adding in an additional call to write(), I saw the
following:

  write failed on myfile.txt.3 : File too large
  rsync error: error in file IO (code 11) at receiver.c(272)
  rsync: connection unexpectedly closed (48 bytes read so far)
  rsync error: error in rsync protocol data stream (code 12) at io.c(150)

After verifying ulimit wasn't the problem, adding a variety of
rprintf()'s,
searching the Web in vain, writing a test program, etc., I finally figured
it out: mkstemp() on AIX 4.3 doesn't appear to support large files.  Since
the call to mkstemp() replaces a call to open(), rsync would fail when the
temporary file it creates for the transfer exceeds 2GB.

My solution to this problem was to simply #undef HAVE_SECURE_MKSTEMP in the
config.h created for AIX.  rsync is much happier with this.  It seems bad to
me that AIX's mkstemp() exhibits this behavior; perhaps someone who knows
how to report such a problem would tell IBM?

Eric

Alternatively, you can re-create your filesystem on mkfs with the big
file option turned on.  (ie -a bf=true) and that should help also. 
We frequently have large files over 2G.

sri

On Wed, Sep 04, 2002 at 08:27:03PM -0700, Eric M. Johnston
wrote:
> Hi,
> 
> I just wanted to relate the solution to a problem I was having to hopefully
> save someone else a day of frustration.  I'm using rsync-2.5.5 on AIX
4.3,
> compiled with gcc 2.95.3.  The file I was sync'ing was very large
(>2GB).
> Despite being configured with --enable-largefiles (which #defines
> _LARGE_FILES properly for AIX), and despite the fact that the initial
> transfer of said file worked fine, I would always get the following error
> when trying to sync up the file:
> 
>   write failed on myfile.txt.3 : Error 0
>   rsync error: error in file IO (code 11) at receiver.c(272)
>   rsync: connection unexpectedly closed (48 bytes read so far)
>   rsync error: error in rsync protocol data stream (code 12) at io.c(150)
> 
> This would always happen when rsync had transferred >2GB.  It seems that
> AIX's write() doesn't like to set errno until the second time you
try to
> call write().  So, after adding in an additional call to write(), I saw the
> following:
> 
>   write failed on myfile.txt.3 : File too large
>   rsync error: error in file IO (code 11) at receiver.c(272)
>   rsync: connection unexpectedly closed (48 bytes read so far)
>   rsync error: error in rsync protocol data stream (code 12) at io.c(150)
> 
> After verifying ulimit wasn't the problem, adding a variety of
rprintf()'s,
> searching the Web in vain, writing a test program, etc., I finally figured
> it out: mkstemp() on AIX 4.3 doesn't appear to support large files. 
Since
> the call to mkstemp() replaces a call to open(), rsync would fail when the
> temporary file it creates for the transfer exceeds 2GB.
> 
> My solution to this problem was to simply #undef HAVE_SECURE_MKSTEMP in the
> config.h created for AIX.  rsync is much happier with this.  It seems bad
to
> me that AIX's mkstemp() exhibits this behavior; perhaps someone who
knows
> how to report such a problem would tell IBM?
> 
> Eric
> -- 
> To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

I haven't worked with AIX for almost exactly 2 years, but I think lsfs, 
with some combination of parameters, will tell you.
Alternately, fsstat, i think it is, can dump the superblock in a 
human-readable format, from which you can get the same info.  Heck, you 
can even turn the support off and on, though that may corrupt the fs in 
some circumstances.

Tim Conway
tim.conway@philips.com
303.682.4917 office, 303.921.0301 cell
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips, caesupport2 on AIM
"There are some who call me.... Tim?"




"Eric M. Johnston" <emj@postal.net>
Sent by: rsync-admin@lists.samba.org
09/05/2002 12:13 PM

 
        To:     rsync@lists.samba.org
        cc:     (bcc: Tim Conway/LMT/SC/PHILIPS)
        Subject:        Re: AIX & Large File Support Problem (+ Solution)
        Classification: 


>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
That said, I have no idea if the filesystems were created with bf=true. 
(Is
there some way to figure that out?)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>