rsync at altfeld-im.de
2023-Dec-14 18:20 UTC
rsync exit code 23 (partial transfer due to errors): List of possible reasons and how to ignore some?
I am trying to find a solution for the open source Linux software
"Back In Time" (https://github.com/bit-team/backintime)
where we evaluate the rsync exit code when taking a backup via rsync
and inform the user that an error has occured.
Questions:
1. Is there full list of possible reasons available that make rsync
exit with the return value 23?
'rsync error: some files/attrs were not transferred (see previous
errors) (code 23) at main.c(1338) [sender=3.2.7]'
I want to decide for each reason if we treat it as an error or warning.
2. I want to ignore some reasons for exit code 23 by only logging it, mainly:
symlink has no referent: "/home/user/Documents/dead-link"
Is there a way to
- either prevent that this error leads to exit code 23 (if no other reason
occurs)
- or to prevent this specific check at all (but eg. copy the symlink "as
is")?
THX a lot!
PS 1: The typcially used rsync command line looks like this:
rsync --recursive --times --devices --specials --hard-links --human-readable -s
--copy-links --acls --perms --executability --group --owner --
info=progress2 --no-inc-recursive -l --delete --delete-excluded -v -i
--out-format=BACKINTIME: %i %n%L --link-dest=../../20231214-143359-584/backup --
chmod=Du+wx --exclude=/home/username/temp/testBAK_profil1
--exclude=/home/username/.local/share/backintime
--exclude=.local/share/backintime/mnt --
include=/home/username/Documents/ --include=/home/username/ --include=/home/
--include=/home/username/temp/deleted_folder/ --
include=/home/username/temp/ --include=/home/username/.mozilla/ --exclude=.gvfs
--exclude=.cache/* --exclude=.thumbnails* --
exclude=.local/share/[Tt]rash* --exclude=*.backup* --exclude=*~
--exclude=.dropbox* --exclude=/proc/* --exclude=/sys/* --exclude=/dev/* --
exclude=/run/* --exclude=/etc/mtab --exclude=/var/cache/apt/archives/*.deb
--exclude=lost+found/* --exclude=/tmp/* --exclude=/var/tmp/* --
exclude=/var/backups/* --exclude=.Private --exclude=lock
--exclude=root_only_file.txt --include=/home/username/Documents/** --
include=/home/username/temp/deleted_folder/**
--include=/home/username/.mozilla/** --exclude=* /
/home/username/temp/testBAK_profil1/backintime/computer1/username/1/new_snapshot/backup
PS 2: You can find more details or even answer in our Github issue too:
https://github.com/bit-team/backintime/issues/1587
Kevin Korb
2023-Dec-14 19:09 UTC
rsync exit code 23 (partial transfer due to errors): List of possible reasons and how to ignore some?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Unfortunately, exit 23 litterally just means something else went wrong and might have scrolled off of the screen if you have rsync listing files (--verbose or --itemize_changes). Essentially, it is anything that doesn't have its own exit code. I just ignore it. The most common reaosn is file vanished. - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., On Thu, 14 Dec 2023, rsync--- via rsync wrote:> Date: Thu, 14 Dec 2023 19:20:15 +0100 > From: rsync--- via rsync <rsync at lists.samba.org> > To: rsync at lists.samba.org > Subject: rsync exit code 23 (partial transfer due to errors): List of possible > reasons and how to ignore some? > > I am trying to find a solution for the open source Linux software > > "Back In Time" (https://github.com/bit-team/backintime) > > where we evaluate the rsync exit code when taking a backup via rsync > and inform the user that an error has occured. > > Questions: > > 1. Is there full list of possible reasons available that make rsync > exit with the return value 23? > > 'rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]' > > I want to decide for each reason if we treat it as an error or warning. > > 2. I want to ignore some reasons for exit code 23 by only logging it, mainly: > > symlink has no referent: "/home/user/Documents/dead-link" > > Is there a way to > - either prevent that this error leads to exit code 23 (if no other reason occurs) > - or to prevent this specific check at all (but eg. copy the symlink "as is")? > > THX a lot! > > > > PS 1: The typcially used rsync command line looks like this: > > rsync --recursive --times --devices --specials --hard-links --human-readable -s --copy-links --acls --perms --executability --group --owner -- > info=progress2 --no-inc-recursive -l --delete --delete-excluded -v -i --out-format=BACKINTIME: %i %n%L --link-dest=../../20231214-143359-584/backup -- > chmod=Du+wx --exclude=/home/username/temp/testBAK_profil1 --exclude=/home/username/.local/share/backintime --exclude=.local/share/backintime/mnt -- > include=/home/username/Documents/ --include=/home/username/ --include=/home/ --include=/home/username/temp/deleted_folder/ -- > include=/home/username/temp/ --include=/home/username/.mozilla/ --exclude=.gvfs --exclude=.cache/* --exclude=.thumbnails* -- > exclude=.local/share/[Tt]rash* --exclude=*.backup* --exclude=*~ --exclude=.dropbox* --exclude=/proc/* --exclude=/sys/* --exclude=/dev/* -- > exclude=/run/* --exclude=/etc/mtab --exclude=/var/cache/apt/archives/*.deb --exclude=lost+found/* --exclude=/tmp/* --exclude=/var/tmp/* -- > exclude=/var/backups/* --exclude=.Private --exclude=lock --exclude=root_only_file.txt --include=/home/username/Documents/** -- > include=/home/username/temp/deleted_folder/** --include=/home/username/.mozilla/** --exclude=* / > /home/username/temp/testBAK_profil1/backintime/computer1/username/1/new_snapshot/backup > > > > > PS 2: You can find more details or even answer in our Github issue too: > > https://github.com/bit-team/backintime/issues/1587 > > >-----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQSHERqysePm7S8yuR9UoLWOVtABBwUCZXtTzQAKCRBUoLWOVtAB B2CnAJ9YGQ/gXTPP2Ntg3arHHDC11cRnfgCeJVpDOnGhTH0OreZfj8pIbnsL3SI=Eqei -----END PGP SIGNATURE-----
rsync at altfeld-im.de
2023-Dec-14 19:19 UTC
rsync exit code 23 (partial transfer due to errors): List of possible reasons and how to ignore some?
On Thu, 2023-12-14 at 14:09 -0500, Kevin Korb wrote:> Unfortunately, exit 23 litterally just means something else went wrong > and might have scrolled off of the screen if you have rsync listing > files (--verbose or --itemize_changes). Essentially, it is anything > that doesn't have its own exit code. I just ignore it. The most common > reaosn is file vanished.THX for sharing your experiences and knowledge :-) I have just tried to "reverse engineer" the possible reasons from the source code and have found 21 reasons that I hope will never happen ;-) So ignoring (or treating as a warning only) sound as best option so far. ----- Looking into the rsync source code I can see only one location where exit code 23 is set: https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/errcode.h#L42 #define RERR_PARTIAL 23 /* partial transfer */ https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/log.c#L97 { RERR_PARTIAL , "some files/attrs were not transferred (see previous errors)" }, https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/cleanup.c#L217-L218 if (io_error & IOERR_GENERAL || got_xfer_error) exit_code = RERR_PARTIAL; So the question is which reasons cause - IOERR_GENERAL - got_xfer_error to be true? IOERR_GENERAL is set for different reasons (first line is the log output format string): 1. receive_sums failed [what is that at all?]: https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L345-L348 2. send_files failed to open %s https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L358-L362 3. fstat failed https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L373C33-L373C45 4. read errors mapping %s https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L433-L436 5. change_dir %s failed https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L366-L369 6. skipping overly long name: %s https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1245-L1247 7. symlink has no referent: %s See the source code comments there when symlinks are checked: https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1271C1-L1282C28 8. readlink_stat(%s) failed https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1293-L1295 9. skipping file with bogus (zero) st_mode: %s https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1298-L1302 10. skipping symlink with 0-length value: %s https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1565-L1566 11. [%s] cannot convert filename: %s (%s) https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1595-L1601 12. [%s] cannot convert symlink data for: %s (%s) https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1609-L1614 13. get_acl(fname, &sx) < 0 // with no explicit error message! https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1628-L1632 14. get_xattr(fname, &sx) < 0 // with no explicit error message! https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1637-L1642 15. link_stat %s failed https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1809-L1811 16. opendir %s failed https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1841-L1843 17. filename overflows max-path len by %u: %s/%s https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1863-L1871 18. cannot send file with empty name in %s https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1873-L1877 19. readdir(%s) https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1886-L1889 20. link_stat %s failed https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L2396-L2399 21. cannot add local filter rules in long-named directory: %s https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/exclude.c#L815-L818