Lacey, Nathan
2023-Jun-07 20:17 UTC
[External] Re: ctrl -c while executing --progress --size-only --partial results in unhidden but incomplete file
I'm suggesting a partial file that isn't hidden is worse. Because any other app looking at the file would think the transfer is complete and try to use it. It doesn't need to be a orphaned file, when called next time with --partial rsync could re-opened the incomplete/hidden file, than it could continue putting more bytes to the file until it was really completed. When running in --partial mode, the filename wouldn't have to be random, instead it could be based on the source ip address, so that rsync knows what file to continue processing on. ________________________________ From: rsync <rsync-bounces at lists.samba.org> on behalf of Kevin Korb via rsync <rsync at lists.samba.org> Sent: Wednesday, June 7, 2023 3:26 PM To: rsync at lists.samba.org <rsync at lists.samba.org> Subject: [External] Re: ctrl -c while executing --progress --size-only --partial results in unhidden but incomplete file [You don't often get email from rsync at lists.samba.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] This message is from an EXTERNAL SENDER - be CAUTIOUS of links and attachments. THINK BEFORE YOU CLICK. ________________________________ That is what --partial does. It keeps the partial file. If it kept the temporary file with the random file name that would just be a useless orphaned file. On 6/7/23 15:02, Lacey, Nathan via rsync wrote:> This is a clone from https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FWayneD%2Frsync%2Fissues%2F484&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Au9lcV7aCmwXRKXFT3LA4X5g6QVabasCJy14f01ZMuI%3D&reserved=0 > <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FWayneD%2Frsync%2Fissues%2F484&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Au9lcV7aCmwXRKXFT3LA4X5g6QVabasCJy14f01ZMuI%3D&reserved=0> > > run rsync with --progress --size-only --partial (version 3.1.2) also > tested/built current master ( 3.3.0pre1-5-g6f3c5ecc ) > everything works fine unless I interrupt a transfer ( ctrl c ) > > I would expect to see the incomplete, temporary, hidden file present on > the remote machine. > So that when I call rsync again, it could just pick up where it left off > and continue transferring data. > > But instead there is no hidden file, and the partial/incomplete file > exists, as the actual filename. > > This is bad, because any code that is expecting existing non-hidden > files to be complete will have issues. > > I apologize if I've missed the command line options I'm looking for > Thanks for maintaining a critical piece of software! > > >-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fsanitarium.net%2F&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j42u1PbBcts8QAs7rqJTVuLXLW2mKy6WaifX756cEFI%3D&reserved=0 PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Flists.samba.org%2Fmailman%2Flistinfo%2Frsync&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5p6rzPlJp5WD0dnK9zJUv6sCNpGyXAK9YqSCe%2BMB4Fk%3D&reserved=0 Before posting, read: https://usg02.safelinks.protection.office365.us/?url=http%3A%2F%2Fwww.catb.org%2F~esr%2Ffaqs%2Fsmart-questions.html&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aeRtF6sFctmGy7JNwuYnUhmr45bO0E5PsAmk452Tsvo%3D&reserved=0 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20230607/2be95596/attachment.htm>
Kevin Korb
2023-Jun-07 20:26 UTC
[External] Re: ctrl -c while executing --progress --size-only --partial results in unhidden but incomplete file
Rsync has no memory of what other instances of rsync have done in the past. An existing file with a different name is of no interest to it unless --delete in which case it would be deleted. Maybe what you really want is --partial-dir? On 6/7/23 16:17, Lacey, Nathan wrote:> I'm suggesting a partial file that isn't hidden is worse. > Because any other app looking at the file would think the transfer is > complete and try to use it. > > It doesn't need to be a orphaned file,? when called next time with > --partial rsync could re-opened the incomplete/hidden file, than it > could? continue putting more bytes to the file until it was really > completed. > > When running in --partial mode, the filename wouldn't have to be random, > instead it could be based on the source ip address, so that rsync knows > what file to continue processing on. > ------------------------------------------------------------------------ > *From:* rsync <rsync-bounces at lists.samba.org> on behalf of Kevin Korb > via rsync <rsync at lists.samba.org> > *Sent:* Wednesday, June 7, 2023 3:26 PM > *To:* rsync at lists.samba.org <rsync at lists.samba.org> > *Subject:* [External] Re: ctrl -c while executing --progress --size-only > --partial results in unhidden but incomplete file > [You don't often get email from rsync at lists.samba.org. Learn why this is > important at https://aka.ms/LearnAboutSenderIdentification > <https://aka.ms/LearnAboutSenderIdentification> ] > > This message is from an EXTERNAL SENDER - be CAUTIOUS of links and > attachments. THINK BEFORE YOU CLICK. > ________________________________ > > > That is what --partial does.? It keeps the partial file.? If it kept the > temporary file with the random file name that would just be a useless > orphaned file. > > On 6/7/23 15:02, Lacey, Nathan via rsync wrote: >> This is a clone from https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FWayneD%2Frsync%2Fissues%2F484&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Au9lcV7aCmwXRKXFT3LA4X5g6QVabasCJy14f01ZMuI%3D&reserved=0 > <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FWayneD%2Frsync%2Fissues%2F484&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Au9lcV7aCmwXRKXFT3LA4X5g6QVabasCJy14f01ZMuI%3D&reserved=0> >> <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FWayneD%2Frsync%2Fissues%2F484&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Au9lcV7aCmwXRKXFT3LA4X5g6QVabasCJy14f01ZMuI%3D&reserved=0 > <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FWayneD%2Frsync%2Fissues%2F484&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Au9lcV7aCmwXRKXFT3LA4X5g6QVabasCJy14f01ZMuI%3D&reserved=0>> >> >> run rsync with --progress --size-only --partial (version 3.1.2) also >> tested/built current master ( 3.3.0pre1-5-g6f3c5ecc ) >> everything works fine unless I interrupt a transfer ( ctrl c ) >> >> I would expect to see the incomplete, temporary, hidden file present on >> the remote machine. >> So that when I call rsync again, it could just pick up where it left off >> and continue transferring data. >> >> But instead there is no hidden file, and the partial/incomplete file >> exists, as the actual filename. >> >> This is bad, because any code that is expecting existing non-hidden >> files to be complete will have issues. >> >> I apologize if I've missed the command line options I'm looking for >> Thanks for maintaining a critical piece of software! >> >> >> > > -- > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > ??????? Kevin Korb????????????????????? Phone:??? (407) 252-6853 > ??????? Systems Administrator?????????? Internet: > ??????? FutureQuest, Inc.?????????????? Kevin at FutureQuest.net? (work) > ??????? Orlando, Florida??????????????? kmk at sanitarium.net (personal) > ??????? Web page: > https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fsanitarium.net%2F&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j42u1PbBcts8QAs7rqJTVuLXLW2mKy6WaifX756cEFI%3D&reserved=0 > <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fsanitarium.net%2F&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j42u1PbBcts8QAs7rqJTVuLXLW2mKy6WaifX756cEFI%3D&reserved=0> > ??????? PGP public key available on web site. > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Flists.samba.org%2Fmailman%2Flistinfo%2Frsync&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5p6rzPlJp5WD0dnK9zJUv6sCNpGyXAK9YqSCe%2BMB4Fk%3D&reserved=0 > <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Flists.samba.org%2Fmailman%2Flistinfo%2Frsync&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5p6rzPlJp5WD0dnK9zJUv6sCNpGyXAK9YqSCe%2BMB4Fk%3D&reserved=0> > Before posting, read: > https://usg02.safelinks.protection.office365.us/?url=http%3A%2F%2Fwww.catb.org%2F~esr%2Ffaqs%2Fsmart-questions.html&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aeRtF6sFctmGy7JNwuYnUhmr45bO0E5PsAmk452Tsvo%3D&reserved=0 > <https://usg02.safelinks.protection.office365.us/?url=http%3A%2F%2Fwww.catb.org%2F~esr%2Ffaqs%2Fsmart-questions.html&data=05%7C01%7Cnathan.lacey%40afs.com%7C10b90a1b4f9d4615db4808db678d21e3%7Ca01f407a85cb4a1698bbf28e6384bd28%7C0%7C0%7C638217628080125247%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aeRtF6sFctmGy7JNwuYnUhmr45bO0E5PsAmk452Tsvo%3D&reserved=0>-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,