The application I use, StoRM[1] can store checksums on disk in an extended user attribute - and use that to ensure the integrity of files on disk. The algorithm currently used is adler32. The intention is to perform end to end checksumming from file creation through storage, transfer over the WAN and storage at a site. Looking at http://wiki.lustre.org/manual/LustreManual20_HTML/ManagingFileSystemIO.html#50438211_pgfId-1291975 I see that Lustre has some checksum support (though not for checksumming the file on the OST - so we''d still need to use the user attribute for that). http://wiki.lustre.org/manual/LustreManual18_HTML/LustreTuning.html#50651264_pgfId-1291287 Is the value of the checksum user accessible? Or to be more specific, I''d potentially get a big speedup if I were able to ask the diskserver to tell me the checksum of a file without actually transferring it over the network. Is it easy to do this? Chris [1] http://storm.forge.cnaf.infn.it/home This is an SRM implementation we use to give an grid authentication to our storage (we store data for the LHC).
Christopher J.Walker wrote:> The application I use, StoRM[1] can store checksums on disk in an > extended user attribute - and use that to ensure the integrity of files > on disk. The algorithm currently used is adler32. The intention is to > perform end to end checksumming from file creation through storage, > transfer over the WAN and storage at a site. > > Looking at > http://wiki.lustre.org/manual/LustreManual20_HTML/ManagingFileSystemIO.html#50438211_pgfId-1291975 > > I see that Lustre has some checksum support (though not for checksumming > the file on the OST - so we''d still need to use the user attribute for > that). > > > http://wiki.lustre.org/manual/LustreManual18_HTML/LustreTuning.html#50651264_pgfId-1291287 > > > Is the value of the checksum user accessible? Or to be more specific, > I''d potentially get a big speedup if I were able to ask the diskserver > to tell me the checksum of a file without actually transferring it over > the network. Is it easy to do this? >No, the checksum is not currently available, and is not being stored on disk. That being said, feel free to send patches! There were some plans to merge the client-side checksum with the ZFS checksum when the backing store is ZFS, but I have not been following the ZFS status closely enough to know the status of that enhancement. Do note that the Lustre checksums only cover the RPC, so at best each 1MB file chunk would have a separate checksum, generated on the client before doing the RPC (so not quite as end-to-end as an application checksum). Also note that the checksums are not used when using mmap(). See Bug 11742 for the details (it is sent, but failures are ignored). Kevin> Chris > > [1] http://storm.forge.cnaf.infn.it/home This is an SRM implementation > we use to give an grid authentication to our storage (we store data for > the LHC). > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
Hello! On May 25, 2011, at 6:26 AM, Christopher J.Walker wrote:> The application I use, StoRM[1] can store checksums on disk in an > extended user attribute - and use that to ensure the integrity of files > on disk. The algorithm currently used is adler32. The intention is to > perform end to end checksumming from file creation through storage, > transfer over the WAN and storage at a site. > > I see that Lustre has some checksum support (though not for checksumming > the file on the OST - so we''d still need to use the user attribute for > that). > Is the value of the checksum user accessible? Or to be more specific, > I''d potentially get a big speedup if I were able to ask the diskserver > to tell me the checksum of a file without actually transferring it over > the network. Is it easy to do this?Note that Lustre checksumming is on the wire and it''s not the entire file or object checksum, only the currently on the wire data is checksummed. Moreover, the clients have no way to know what the checksum of the entire file would be unless they read the entire thing in to perform the calculation.> [1] http://storm.forge.cnaf.infn.it/home This is an SRM implementation > we use to give an grid authentication to our storage (we store data for > the LHC).I had a cursory look and don''t see much discussion about checksum implementations. I hope you have finer granularity than just "file" or "object". Some arbitrary-size block would have been good I think. (BTW, on the first page there it says "Lustre by Sun Microsystems" which is now somewhat stale.) Bye, Oleg -- Oleg Drokin Senior Software Engineer Whamcloud, Inc.