CentOS - Feb 2009 - nightly rsync has started to throw errors

Hi all,

I perform a nightly snapshot of /home to a USB attached drive scheduled
via cron.  The system is CentOS 5.2 and only gets attached to the
internet periodically for updates, otherwise serves as a samba server to
about 20 Windows clients.

The rsync command being used is:
rsync -av --delete /home/ /media/bkup320G/
and has been working well until a few days ago.

Starting with a few days ago, my nightly rsync/cron emails included some
errors as shown here:

rsync: mkstemp
"/media/bkup320G/cprcvs/c/Projects/WindowApps/DLL/HMRControlDLL/.CMDSettingsDialog.cpp,v.I5QnaM"
failed: Read-only file system (30)
rsync: failed to set times on
"/media/bkup320G/cprcvs/c/Projects/WindowApps/DLL/PSIControlDLL":
Read-only file system (30)

There are multiple issuances of these errors for different files....some
files are written just fine.  Where there are errors, its always the
same Read-only file system (30).

Output of df and mount show all filesystems as (rw):
# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     136838136   5518276 124256720   5% /
/dev/sdc1            283783548 167456104 101679380  63% /home
/dev/sdd1            283783548   4648856 264486628   2% /home/905
/dev/sda1               101086     31745     64122  34% /boot
tmpfs                   452084         0    452084   0% /dev/shm
/dev/sdg1            307663800 171859328 120176040  59% /media/bkup320G


# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sdc1 on /home type ext3 (rw)
/dev/sdd1 on /home/905 type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/sdg1 on /media/bkup320G type ext3 (rw)

Googling has not been fruitful in this issue and I''m curious what might
be the issue.  I''d originally thought it might be drive failure on the
external (destination) drive, but changing that produced the same errors.

We typically backup about 165G overnight, so initial write to a new
drive takes a LONG time.

Any pointers or suggestions would be greatly appreciated.

TIA,
-Ray

On Fri, Feb 27, 2009 at 9:40 PM, Ray Leventhal <centos at swhi.net>
wrote:
> rsync: mkstemp
>
"/media/bkup320G/cprcvs/c/Projects/WindowApps/DLL/HMRControlDLL/.CMDSettingsDialog.cpp,v.I5QnaM"
> failed: Read-only file system (30)
> rsync: failed to set times on
> "/media/bkup320G/cprcvs/c/Projects/WindowApps/DLL/PSIControlDLL":
> Read-only file system (30)
> Googling has not been fruitful in this issue and I''m curious what
might
> be the issue. ?I''d originally thought it might be drive failure on
the
> external (destination) drive, but changing that produced the same errors.
Damage usb ports? harddrive casing?

Ray Leventhal wrote:
> Any pointers or suggestions would be greatly appreciated.
Probably a silly question, but does the user running rsync have write 
access to the backup directory?

-- 
Sincerely,
John Thomas

Ray Leventhal wrote:
> Hi all,
> 
> I perform a nightly snapshot of /home to a USB attached drive scheduled
> via cron.  The system is CentOS 5.2 and only gets attached to the
> internet periodically for updates, otherwise serves as a samba server to
> about 20 Windows clients.
> 
> The rsync command being used is:
> rsync -av --delete /home/ /media/bkup320G/
> and has been working well until a few days ago.
> 
> Starting with a few days ago, my nightly rsync/cron emails included some
> errors as shown here:
> 
> rsync: mkstemp
>
"/media/bkup320G/cprcvs/c/Projects/WindowApps/DLL/HMRControlDLL/.CMDSettingsDialog.cpp,v.I5QnaM"
> failed: Read-only file system (30)
> rsync: failed to set times on
> "/media/bkup320G/cprcvs/c/Projects/WindowApps/DLL/PSIControlDLL":
> Read-only file system (30)
> 
Getting any IO errors in the kernel log? Maybe the USB disk is going 
offline occasionally.


-- 
tkb

From: Ray Leventhal <centos at swhi.net>
> rsync: mkstemp
>
"/media/bkup320G/cprcvs/c/Projects/WindowApps/DLL/HMRControlDLL/.CMDSettingsDialog.cpp,v.I5QnaM"
> failed: Read-only file system (30)
> rsync: failed to set times on
> "/media/bkup320G/cprcvs/c/Projects/WindowApps/DLL/PSIControlDLL":
> Read-only file system (30)
Hum... since it works for other files....
Maybe a write protected directory on the backup disk that prevents rsync to
create a temp file, change the date or delete a file...
Login as the "rsync user" and try to:
  touch
/media/bkup320G/cprcvs/c/Projects/WindowApps/DLL/HMRControlDLL/rsync.test
Then try to delete it...

JD

Ray Leventhal wrote:
> Hi all,
> 
> I perform a nightly snapshot of /home to a USB attached drive scheduled
> via cron.  The system is CentOS 5.2 and only gets attached to the
> internet periodically for updates, otherwise serves as a samba server to
> about 20 Windows clients.
> 
> The rsync command being used is:
> rsync -av --delete /home/ /media/bkup320G/
> and has been working well until a few days ago.
> 
<snip>
Thanks to all who replied.  I ran fsck on the disk yesterday and no
errors were reported; had already checked cables, etc.  About an hour
ago I rebooted the machine and let hal mount the attached drive as
opposed to mounting it manually, which I''d been doing via script.

Also, there was nothing unusual in the logs.

I changed my backup script to properly identify the new mount point
(/media/disk vs /media/bkup320G) and ran a test on a few home folders
which had errored earlier - interestingly, no errors now.

So, while my users are offline, I''m rerunning the  whole snapshot.  I
expect it will be fine, though I''m still not sure what got borked up.

When I convince the powers that be to get an internal drive for this
purpose so that we can do the rsync across the backplane vs USB 2.0,
I''ll be doing the dance of joy :)

Again, thanks to all

-Ray