zfs discuss - Nov 2008 - cpio silently not copying files > 2GB -- seriously!?

Hi,

I was just restoring a bunch of files from backup using find|cpio when I 
noticed that cpio does not copy files >2GB properly. The resulting files 
were "oddly" sized (<actual size> % 2GB, perhaps?).

Even more alarming, cpio did not warn in any way about not copying the 
file correctly! The cpio command exited normally ($? was 0). Output from 
"-pmdu" did not indicate any errors!

Fortunately, I happened to run ''du'' and noticed it giving
different sizes
for the restored and "backup" directories, I''ve since
re-restored the
affected files using plain cp.

I guess I was lucky not doing the backup using cpio! :(

Seriously bad behaviour here -- something that really should be fixed for 
e.g "Solaris 11".

FWIW, I know about "largefile(5)", but I don''t think failing
silently like
this (copying "some" data) can be considered proper behaviour in any 
scenario.

(This probably does not strictly belong in zfs-discuss, although for me 
both the backup media and the destination pool to restore to are ZFS.)

/H

On Fri 28/11/08 13:09 , H?kan Olsson ho at rfc.se sent:
> Hi,
> 
> I was just restoring a bunch of files from backup using find|cpio when I
> noticed that cpio does not copy files >2GB properly. The resulting files
> were "oddly" sized (<actual size> % 2GB, perhaps?).
> 
> Even more alarming, cpio did not warn in any way about not copying the 
> file correctly! The cpio command exited normally ($? was 0). Output from
> "-pmdu" did not indicate any errors!
> 
> Seriously bad behaviour here -- something that really should be fixed for
> e.g "Solaris 11".
> 
See the rather confusing comment at the end of the cpio man page:

     The new pax(1) format, with a command that supports it  (for
     example,  pax  ,  tar,  or  cpio),  should be used for large
     files. The cpio command is no longer  part  of  the  current
     POSIX standard and is deprecated in favor of pax.

-- 
Ian.

H?kan Olsson <ho at rfc.se> wrote:
> I was just restoring a bunch of files from backup using find|cpio when I 
> noticed that cpio does not copy files >2GB properly. The resulting files
> were "oddly" sized (<actual size> % 2GB, perhaps?).
>
> Even more alarming, cpio did not warn in any way about not copying the 
> file correctly! The cpio command exited normally ($? was 0). Output from 
> "-pmdu" did not indicate any errors!
What exactly did you do when encountering the problem?

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       schilling at fokus.fraunhofer.de     (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

On Fri, 28 Nov 2008, Joerg Schilling wrote:
> H?kan Olsson <ho at rfc.se> wrote:
>
>> I was just restoring a bunch of files from backup using find|cpio when
I
>> noticed that cpio does not copy files >2GB properly. The resulting
files
>> were "oddly" sized (<actual size> % 2GB, perhaps?).
>>
>> Even more alarming, cpio did not warn in any way about not copying the
>> file correctly! The cpio command exited normally ($? was 0). Output
from
>> "-pmdu" did not indicate any errors!
>
> What exactly did you do when encountering the problem?
cd /backup
find . -depth -print | cpio -pdmvu /pool
<...lots of files listed>
<huge number> blocks copied

I''ve filed a bug report on it now.

The limit is 4GB, not 2 as I believed last night. It''s simple to 
reproduce:

mkdir test
mkfile 4200m bigfile
echo bigfile | cpio -pdmvu test/
echo $?
ls -l bigfile test/bigfile

/H

>cd /backup
>find . -depth -print | cpio -pdmvu /pool
><...lots of files listed>
><huge number> blocks copied
>
>I''ve filed a bug report on it now.
>
>The limit is 4GB, not 2 as I believed last night. It''s simple to=20
>reproduce:
>
>mkdir test
>mkfile 4200m bigfile
>echo bigfile | cpio -pdmvu test/
>echo $?
>ls -l bigfile test/bigfile

Absolutely weird; the "cpio -p" doesn''t use any intermediate
archive
format and it should be able to copy files of any size.

Casper

>Absolutely weird; the "cpio -p" doesn''t use any
intermediate archive
>format and it should be able to copy files of any size.

It works in Solaris 10u5 so it''s broken later (Nevada?) at some point
in
time...

Casper

>
>
>>Absolutely weird; the "cpio -p" doesn''t use any
intermediate archive
>>format and it should be able to copy files of any size.
>
>
>It works in Solaris 10u5 so it''s broken later (Nevada?) at some
point in
>time...

I''ve upped the priority, accepted the bug and moved it to the proper 
category (utility/archiver).

I''ve run cpio from all builds which changed cpio and the first
broken build is snv_81.

Casper

On Fri, 28 Nov 2008 17:36:22 +0100
Casper.Dik at Sun.COM wrote:
> >>Absolutely weird; the "cpio -p" doesn''t use any
intermediate archive
> >>format and it should be able to copy files of any size.
> >
> >It works in Solaris 10u5 so it''s broken later (Nevada?) at
some
> >point in time...
> 
> I''ve upped the priority, accepted the bug and moved it to the
proper
> category (utility/archiver).
> 
> I''ve run cpio from all builds which changed cpio and the first
> broken build is snv_81.
And in the meantime the solution to the broken cpio command like this:

cd /backup
find . -depth -print | cpio -pdmvu /pool

What would be the correct pax syntax?

-- 
Dick Hoogendijk -- PGP/GnuPG key: 01D2433D
+ http://nagual.nl/ | SunOS sxce snv103 ++
+ All that''s really worth doing is what we do for others (Lewis Carrol)

Casper.Dik at Sun.COM wrote:
> I''ve upped the priority, accepted the bug and moved it to the
proper
> category (utility/archiver).
>
> I''ve run cpio from all builds which changed cpio and the first
> broken build is snv_81.
At this time, 30% of the code have been changed. This was a real big change.

The bug is in function data_copy() the 4th arg must be a off_t

Local variables like bytesread, maxwrite, piosize are also wrong and need to be 
off_t.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       schilling at fokus.fraunhofer.de     (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

dick hoogendijk <dick at nagual.nl> wrote:
> And in the meantime the solution to the broken cpio command like this:
>
> cd /backup
> find . -depth -print | cpio -pdmvu /pool
>
> What would be the correct pax syntax?
star fs=128m -copy -p -U -sparse -C /backup . /pool

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       schilling at fokus.fraunhofer.de     (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily