rsync - Sep 2020 - Copying TBs -> error -> work around

People,

When I did:

   rsync -av /home/ /mntb5/  # about 4TB

I got errors like:

   'rsync [sender] expand file_list pointer array to xxx bytes, "did 
move"'

with rsync hanging - after breakout on /home for writing I then get:

   "Read-only file system"

So after unmounting and remounting /home I did:

   cd /home
   find /home/ -type d | sort > ./home_dirs_sorted.txt

delete first line "/home/" of ./home_dirs_sorted.txt then:

   while read dir ; do echo $dir ; rsync -lptgod "$dir"
/mntb5/"$dir" ;
done < ./home_dirs_sorted.txt

and:

   while read dir ; do echo $dir ; rsync -lptgoD "$dir"/\.[a-zA-Z0-9]*
/mntb5/"$dir"/ ; done < ./home_dirs_sorted.txt

and finally with no problems:

   rsync -av --exclude-from=/usr/local/bin/nfb_caches.txt /home/ /mntb5/

If there was a more sensible / efficient way of getting this done I 
would like to know about it!

Thanks,

Phil.

-- 
Philip Rhoades

PO Box 896
Cowra  NSW  2794
Australia
E-mail:  phil at pricom.com.au

>with rsync hanging - after breakout on /home for writing I then get:
 >"Read-only file system"

if your filesystem switches to read-only, you have a serious problem
with your system/storage, not with rsync.

rsync (or the workload) is simply triggering the problem.

regards
roland


Am 10.09.20 um 07:30 schrieb Philip Rhoades via rsync:
> People,
>
> When I did:
>
> ? rsync -av /home/ /mntb5/? # about 4TB
>
> I got errors like:
>
> ? 'rsync [sender] expand file_list pointer array to xxx bytes,
"did
> move"'
>
> with rsync hanging - after breakout on /home for writing I then get:
>
> ? "Read-only file system"
>
> So after unmounting and remounting /home I did:
>
> ? cd /home
> ? find /home/ -type d | sort > ./home_dirs_sorted.txt
>
> delete first line "/home/" of ./home_dirs_sorted.txt then:
>
> ? while read dir ; do echo $dir ; rsync -lptgod "$dir"
/mntb5/"$dir" ;
> done < ./home_dirs_sorted.txt
>
> and:
>
> ? while read dir ; do echo $dir ; rsync -lptgoD
"$dir"/\.[a-zA-Z0-9]*
> /mntb5/"$dir"/ ; done < ./home_dirs_sorted.txt
>
> and finally with no problems:
>
> ? rsync -av --exclude-from=/usr/local/bin/nfb_caches.txt /home/ /mntb5/
>
> If there was a more sensible / efficient way of getting this done I
> would like to know about it!
>
> Thanks,
>
> Phil.
>

Roland,


On 2020-09-10 21:27, Roland wrote:
>> with rsync hanging - after breakout on /home for writing I then get:
>> "Read-only file system"
> 
> if your filesystem switches to read-only, you have a serious problem
> with your system/storage, not with rsync.
> 
> rsync (or the workload) is simply triggering the problem.

Thanks for the response . .

Hmm . . but the drive that goes read-only is being read FROM not TO . . 
it is hard to see how that should be an issue?

The backstory is that a relatively recent internal 8TB Seagate Barracuda 
had its 7.2TB sda5 (home) partition corrupted - which itself was 
suspicious but not impossible of course - so I had to switch temporarily 
to an external USB 4TB drive (which was a backup drive and was already 
up-to-date) for /home.  So now this exercise is rsyncing back to a NEW 
internal 8TB Seagate Barracuda (sda5 again) . .

If you are correct about rsync simply triggering an existing problem on 
the 4TB USB drive, would that problem going to be recognised by a fsck 
(ext4)?  I will check this out after I switch over to the new internal 
sda5 for /home.

Thanks,

Phil.

> regards
> roland
> 
> 
> Am 10.09.20 um 07:30 schrieb Philip Rhoades via rsync:
>> People,
>> 
>> When I did:
>> 
>> ? rsync -av /home/ /mntb5/? # about 4TB
>> 
>> I got errors like:
>> 
>> ? 'rsync [sender] expand file_list pointer array to xxx bytes,
"did
>> move"'
>> 
>> with rsync hanging - after breakout on /home for writing I then get:
>> 
>> ? "Read-only file system"
>> 
>> So after unmounting and remounting /home I did:
>> 
>> ? cd /home
>> ? find /home/ -type d | sort > ./home_dirs_sorted.txt
>> 
>> delete first line "/home/" of ./home_dirs_sorted.txt then:
>> 
>> ? while read dir ; do echo $dir ; rsync -lptgod "$dir"
/mntb5/"$dir" ;
>> done < ./home_dirs_sorted.txt
>> 
>> and:
>> 
>> ? while read dir ; do echo $dir ; rsync -lptgoD
"$dir"/\.[a-zA-Z0-9]*
>> /mntb5/"$dir"/ ; done < ./home_dirs_sorted.txt
>> 
>> and finally with no problems:
>> 
>> ? rsync -av --exclude-from=/usr/local/bin/nfb_caches.txt /home/ 
>> /mntb5/
>> 
>> If there was a more sensible / efficient way of getting this done I
>> would like to know about it!
>> 
>> Thanks,
>> 
>> Phil.
>> 
-- 
Philip Rhoades

PO Box 896
Cowra  NSW  2794
Australia
E-mail:  phil at pricom.com.au