I've been trying to extract some of my clients' data from a flakey lustre filesystem, and as a result, took some time to write up a page about pulling data out of a semi-crashy filesystem. The page includes a variety of ways of doing so, those ways including but not being limited to rsync and a program I wrote called try-copying-up-to-n-times that was inspired to some extent by rsync. I hope someone will find it useful: http://dcs.nac.uci.edu/~strombrg/recovering-data.html