I've looking for a solution for this and no amount of googling has come up with anything. Is it possible to provide a static listing on a server, say every 24 hours, that a standard end-user rsync can pull and use? I have a lot of files to provide and the idea of every request dynamically providing a file list in real time is killing my server and is simply not needed. I am quite prepared to swap in (atomically) an alternate file tree every 24 hours as long as I can also provide a static file list. I know the files will not change for 24 hours and could easily handle a 10Kb to 100Kb static list being downloaded, plus the actual delta downloads, but adding 100/sec listings of 20,000 files is a killer.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Sounds like a job for a snapshot. If you are on Linux that would be an lvm2 snapshot. Other operating systems with basic volume management usually have an equivalent. On 12/21/11 23:35, Mark Constable wrote:> I've looking for a solution for this and no amount of googling has > come up with anything. > > Is it possible to provide a static listing on a server, say every > 24 hours, that a standard end-user rsync can pull and use? > > I have a lot of files to provide and the idea of every request > dynamically providing a file list in real time is killing my server > and is simply not needed. I am quite prepared to swap in > (atomically) an alternate file tree every 24 hours as long as I can > also provide a static file list. I know the files will not change > for 24 hours and could easily handle a 10Kb to 100Kb static list > being downloaded, plus the actual delta downloads, but adding > 100/sec listings of 20,000 files is a killer.- -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk7ytZMACgkQVKC1jlbQAQeYBQCfetbFyNARMg9oX65c/2MyS5U7 2usAn05WgB8O3G8kcraxF2npiPdRkGej =pdk6 -----END PGP SIGNATURE-----
Mark Constable schrieb:> I've looking for a solution for this and no amount of googling has > come up with anything. > > Is it possible to provide a static listing on a server, say every > 24 hours, that a standard end-user rsync can pull and use? > > I have a lot of files to provide and the idea of every request > dynamically providing a file list in real time is killing my > server and is simply not needed. I am quite prepared to swap in > (atomically) an alternate file tree every 24 hours as long as I > can also provide a static file list. I know the files will not > change for 24 hours and could easily handle a 10Kb to 100Kb static > list being downloaded, plus the actual delta downloads, but adding > 100/sec listings of 20,000 files is a killer.How many are requesting and how often, and does the server do more than providing files? Taking 100 seconds to do a 20000 files list looks slow to me, more like time to upgrade RAM and/or HDD. When it is OK to let the users have an 24h old filelist, is it at the same time OK if the user gets only up to 24h old files? If no then it can't be helped, if yes another solution would be to set up a seperate server for that high number of rsync requests which provides a once-per-day or once-per-hour rsynced copy of the main files. For that separate dedicated server a pre-generated list would be unneeded since the OS would cache the never-changing files.
Joachim Otahal (privat) schrieb:> When it is OK to let the users have an 24h old filelist, is it at the > same time OK if the user gets only up to 24h old files?Whoops, I _hope_ you know I meant "get the files up to 24h late?".