rsync - Sep 2023 - Feature Concept: enable iCloud Drive with rsync

Hi,

I have also posted this on GitHub but it isn?t clear that was the right place:
https://github.com/WayneD/rsync/issues/522

iCloud Drive will evict files that are unused or when additional space is needed
on the local drive. The evicted files are replace by "bookmark" files
that allow MacOS to continue to report the files in the file system as though
they were actually present. The files are downloaded again either on request or
when needed.

rsync, like all similar tools I can find, doesn't have any way of handling
these evicted files. I have been thinking about this and I think I know how to
make it work.

The short explanation is that iCloud Drive preserves access to the required
metadata. Here is an abbreviated output from `mdls` of an evicted file:

```
% mdls .1998-tax-return.pdf.icloud
kMDItemContentCreationDate = 2018-08-28 18:32:24 +0000
kMDItemContentCreationDate_Ranking = 2021-11-05 00:00:00 +0000
kMDItemContentModificationDate = 2018-08-28 18:32:24 +0000
kMDItemDateAdded = 2021-11-05 15:42:15 +0000
kMDItemDisplayName = "1998-tax-return.pdf"
kMDItemFSContentChangeDate = 2018-08-28 18:32:24 +0000
kMDItemFSCreationDate = 2018-08-28 18:32:24 +0000
kMDItemFSSize = 1929932
kMDItemInterestingDate_Ranking = 2018-08-28 00:00:00 +0000
kMDItemLogicalSize = 1929932
kMDItemPhysicalSize = 1929932
```

This metadata, I think, is enough to pass the rsync quick check as described in
the man page. Therefore, I suspect what needs to happen, from a code
perspective, is that rsync needs to be modified to do the following when it
finds an evicted file. All evicted files are named consistently,
`.<filename>.iCloud` so they are easy to spot.

1. Perform the system call equivalent of the mdls above to obtain the
appropriate dates and sizes.
2. If the file cannot be skipped because it either has changed or a checksum is
required, perform the system call equivalent of `brctl download
<filename>` to get the file downloaded.
3. Rsync the file
4. If the file was downloaded by rsync, perform the system call equivalent of
`brctl evict <filename>` to remove the file to leave the system in the
same state.

This simplistic algorithm would leave some open issues/caveats:
- It is likely that the rsync can only be run one-way from iCloud Drive to
non-iCloud Drive data storage. While it is possible it could be run two-way
research is needed on whether you?d have to download the old file before you
replace it.
- Timeouts may happen. iCloud Drive still gets stuck sometimes and there will be
non-zero time during downloads that don?t get stuck.
- There is no guarantee there is ever enough room on disk to hold a specific
file from iCloud Drive. This is likely resolved by MacOS directly, however, this
may take excessive time.
- If you thought running with checksums was slow before, strap in, because those
downloads are going to add up.

All that said, I think this would accomplish a high percentage of what users are
looking for.

Is this kind of a patch welcome? Is the plan sound? I haven't written C
since college (in the 90s!) and would be open to advice/mentoring.

Thank you.

regards,

bex
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20230912/5dc50a2c/attachment.htm>

Is this being accessed via a fuse mount?  If so it seems like that is 
where this kind of feature should be implemented (like a mount option to 
decide how to handle such files).  Rsync shouldn't need special features 
to deal with every kind of file storage.

On 9/12/23 05:22, Brian "bex" Exelbierd via rsync
wrote:
> Hi,
> 
> I have also posted this on GitHub but it isn?t clear that was the right 
> place: https://github.com/WayneD/rsync/issues/522
> 
> iCloud Drive will evict files that are unused or when additional space 
> is needed on the local drive. The evicted files are replace by 
> "bookmark" files that allow MacOS to continue to report the files
in the
> file system as though they were actually present. The files are 
> downloaded again either on request or when needed.
> 
> rsync, like all similar tools I can find, doesn't have any way of 
> handling these evicted files. I have been thinking about this and I 
> think I know how to make it work.
> 
> The short explanation is that iCloud Drive preserves access to the 
> required metadata. Here is an abbreviated output from `mdls` of an 
> evicted file:
> 
> ```
> % mdls .1998-tax-return.pdf.icloud
> kMDItemContentCreationDate = 2018-08-28 18:32:24 +0000
> kMDItemContentCreationDate_Ranking = 2021-11-05 00:00:00 +0000
> kMDItemContentModificationDate = 2018-08-28 18:32:24 +0000
> kMDItemDateAdded = 2021-11-05 15:42:15 +0000
> kMDItemDisplayName = "1998-tax-return.pdf"
> kMDItemFSContentChangeDate = 2018-08-28 18:32:24 +0000
> kMDItemFSCreationDate = 2018-08-28 18:32:24 +0000
> kMDItemFSSize = 1929932
> kMDItemInterestingDate_Ranking = 2018-08-28 00:00:00 +0000
> kMDItemLogicalSize = 1929932
> kMDItemPhysicalSize = 1929932
> ```
> 
> This metadata, I think, is enough to pass the rsync quick check as 
> described in the man page. Therefore, I suspect what needs to happen, 
> from a code perspective, is that rsync needs to be modified to do the 
> following when it finds an evicted file. All evicted files are named 
> consistently, `.<filename>.iCloud` so they are easy to spot.
> 
> 1. Perform the system call equivalent of the mdls above to obtain the 
> appropriate dates and sizes.
> 2. If the file cannot be skipped because it either has changed or a 
> checksum is required, perform the system call equivalent of `brctl 
> download <filename>` to get the file downloaded.
> 3. Rsync the file
> 4. If the file was downloaded by rsync, perform the system call 
> equivalent of `brctl evict <filename>` to remove the file to leave
the
> system in the same state.
> 
> This simplistic algorithm would leave some open issues/caveats:
> - It is likely that the rsync can only be run one-way from iCloud Drive 
> to non-iCloud Drive data storage. While it is possible it could be run 
> two-way research is needed on whether you?d have to download the old 
> file before you replace it.
> - Timeouts may happen. iCloud Drive still gets stuck sometimes and there 
> will be non-zero time during downloads that don?t get stuck.
> - There is no guarantee there is ever enough room on disk to hold a 
> specific file from iCloud Drive. This is likely resolved by MacOS 
> directly, however, this may take excessive time.
> - If you thought running with checksums was slow before, strap in, 
> because those downloads are going to add up.
> 
> All that said, I think this would accomplish a high percentage of what 
> users are looking for.
> 
> Is this kind of a patch welcome? Is the plan sound? I haven't written C
> since college (in the 90s!) and would be open to advice/mentoring.
> 
> Thank you.
> 
> regards,
> 
> bex
> 
-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			https://sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,