Hi all, Thank you for your very useful product but I have a memory problem. I want use you product to migrate 4TB of data (in 150?millions files) When rsync make the list of files it full my RAM and after my SWAP and my server crash. I have 4GB of RAM and 4GB of SWAP Do you know what can I do?? Can I have an option to use an index on disk or something like this?? Thanks in advance Sylvain?GARGASSON Technicien EUDASYS
On Mon, 2008-01-14 at 17:05 +0100, Sylvain Gargasson wrote:> I want use you product to migrate 4TB of data (in 150 millions files) > > When rsync make the list of files it full my RAM and after my SWAP and my server crash.Use a development version of rsync 3.0.0, which has an incremental recursion mode that avoids keeping a list of all the source files in memory for the entire run. The source code for the latest pre-release of rsync 3.0.0 is at: http://rsync.samba.org/ftp/rsync/rsync-3.0.0pre8.tar.gz Matt
Hi Matt, Thanks a lot, I work in a production and I not very happy from use pre-release version... I have use FAQ and calculate I need 15GB of RAM or SWAP... I can create big swap without problem but my client use Samba for share is 150 Millions files and if I use all RAM I can have performance impact... Do you know if I can use some command to directly use SWAP memory for rsync??? Thanks in advance Best regards Sylvain GARGASSON Technicien EUDASYS 60/62, Rue du Mar?chal Foch 78000 VERSAILLES T?l. : 01 39 25 66 66 Fax : 01 39 25 66 67 C?line Louis Assistante T?l: +33 (0)1.39.25.66.79 Fax: +33 (0)1.39.25.66.67 Ce message contient des informations confidentielles couvertes par le secret professionnel. Si vous n'?tes pas le destinataire d?sign?, nous vous remercions de bien vouloir nous en aviser imm?diatement et de nous retourner ce message ou de le d?truire, sans faire un quelconque usage de son contenu, ni le communiquer ou le diffuser, ni en prendre aucune copie, ?lectronique ou non. La s?curit? des envois de messages ?lectroniques ne peut ?tre assur?e. Ces messages peuvent notamment ?tre intercept?s, modifi?s, alt?r?s, d?truits, perdus, arriver tardivement ou partiellement, ou contenir des virus. L'exp?diteur ne saurait ?tre tenu pour responsable des erreurs ou omissions qui r?sulteraient d'un envoi par message ?lectronique. Si vous souhaitez v?rifier l'authenticit? du message et des fichiers joints, merci d'en solliciter une copie sur papier. -----Message d'origine----- De?: Matt McCutchen [mailto:matt@mattmccutchen.net] Envoy??: mardi 15 janvier 2008 03:16 ??: Sylvain Gargasson Cc?: rsync@lists.samba.org Objet?: Re: 4TB and "150 000 000" files out of memory error On Mon, 2008-01-14 at 17:05 +0100, Sylvain Gargasson wrote:> I want use you product to migrate 4TB of data (in 150 millions files) > > When rsync make the list of files it full my RAM and after my SWAP and my server crash.Use a development version of rsync 3.0.0, which has an incremental recursion mode that avoids keeping a list of all the source files in memory for the entire run. The source code for the latest pre-release of rsync 3.0.0 is at: http://rsync.samba.org/ftp/rsync/rsync-3.0.0pre8.tar.gz Matt
Ok Matt, Thank you very much I have now two solutions: Have bad performance and big SWAP Or Use little spited parts in a script Thank you I always like have two choices. Thanks for all Bye Sylvain GARGASSON Technicien EUDASYS 60/62, Rue du Mar?chal Foch 78000 VERSAILLES T?l. : 01 39 25 66 66 Fax : 01 39 25 66 67 C?line Louis Assistante T?l: +33 (0)1.39.25.66.79 Fax: +33 (0)1.39.25.66.67 Ce message contient des informations confidentielles couvertes par le secret professionnel. Si vous n'?tes pas le destinataire d?sign?, nous vous remercions de bien vouloir nous en aviser imm?diatement et de nous retourner ce message ou de le d?truire, sans faire un quelconque usage de son contenu, ni le communiquer ou le diffuser, ni en prendre aucune copie, ?lectronique ou non. La s?curit? des envois de messages ?lectroniques ne peut ?tre assur?e. Ces messages peuvent notamment ?tre intercept?s, modifi?s, alt?r?s, d?truits, perdus, arriver tardivement ou partiellement, ou contenir des virus. L'exp?diteur ne saurait ?tre tenu pour responsable des erreurs ou omissions qui r?sulteraient d'un envoi par message ?lectronique. Si vous souhaitez v?rifier l'authenticit? du message et des fichiers joints, merci d'en solliciter une copie sur papier. -----Message d'origine----- De?: Matt McCutchen [mailto:matt@mattmccutchen.net] Envoy??: mardi 15 janvier 2008 18:33 ??: Sylvain Gargasson Cc?: rsync@lists.samba.org Objet?: RE: 4TB and "150 000 000" files out of memory error On Tue, 2008-01-15 at 09:55 +0100, Sylvain Gargasson wrote:> I work in a production and I not very happy from use pre-release version... > > I have use FAQ and calculate I need 15GB of RAM or SWAP... > > I can create big swap without problem but my client use Samba for share is 150 Millions files and if I use all RAM I can have performance impact... > > Do you know if I can use some command to directly use SWAP memory for rsync???No. If Linux had a ulimit for the amount of data a process can have in RAM at once, that would be perfect, but unfortunately it doesn't. Another option is to split the copy into smaller portions, each of which can be done within the memory limit. For example, if the size of your source directory is split relatively evenly among a number of immediate subdirectories, you could write a script to iterate over the source subdirectories and copy each to the corresponding destination subdirectory using a separate rsync run (possibly with an additional run to do top-level deletions). Matt