Dear All,
I am wondering whether it is possible to recover the OST data ?
We have faced the following problem. We installed a new OST server
which is intended to replace an old one. We do the following steps
to migrate the data from old OST to new OST:
1. Figure out the files which located in old OST server:
lfs find --obd {OST_UUID} /mount/point > list
2. For each file in the list, we do the following migration:
cp -a ..../file ..../file.tmp
mv ..../file.tmp ..../file
However, after doing this data transfer for a few days, we suddently
found that the hardware of the new OST is failed, and all the data
migrated on it may not be recovered.
Therefore, I am wondering that whether it is possible to recover
data from the old OST server ? This may sounds quit impossible, but
we would like to try if there is any possibility to do that.
Our Lustre version is 1.8.3.
Thanks very much for your attention in advance.
Cheers,
T.H.Hsieh
On 2011-01-13, at 01:41, thhsieh wrote:> I am wondering whether it is possible to recover the OST data ? > > We have faced the following problem. We installed a new OST server > which is intended to replace an old one. We do the following steps > to migrate the data from old OST to new OST: > > 1. Figure out the files which located in old OST server: > > lfs find --obd {OST_UUID} /mount/point > list > > 2. For each file in the list, we do the following migration: > > cp -a ..../file ..../file.tmp > mv ..../file.tmp ..../file > > > However, after doing this data transfer for a few days, we suddently > found that the hardware of the new OST is failed, and all the data > migrated on it may not be recovered. > > Therefore, I am wondering that whether it is possible to recover > data from the old OST server ? This may sounds quit impossible, but > we would like to try if there is any possibility to do that.This partly boils down to an ext3 data recovery problem, because the files have been deleted from the old OST where you want to recover them. There are a few tools you could try, including "ext3grep". If/when you are able to recover the objects on the old OST, then the secondary problem becomes re-attaching them into the Lustre namespace. If you have 1-stiped files (the default) this is possible, but if you have 2-striped (or wider) files, then it is nearly impossible because the objects on the other OSTs have likely been overwritten by your migration process. Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc.
Dear Andreas, Thanks very much for your response. In any case I am very appriciate. I did not change the strip size of the Lustre defautl value. So I think our Lustre filesystem uses 1-strip. Now the problem is how to re-attach the recovered files to the Lustre name space. In fact, we have a completed list of filenames which are migrated to the new OST. However, only filenames, but without file modified time, size, owner, permission .... etc. But we can still guess about them. So, is there a way to re-attach the files in OST to the Lustre name space? This is the major difficulty we are facing now. Thanks very much. T.H.Hsieh On Thu, Jan 13, 2011 at 11:09:55AM -0700, Andreas Dilger wrote:> On 2011-01-13, at 01:41, thhsieh wrote: > > I am wondering whether it is possible to recover the OST data ? > > > > We have faced the following problem. We installed a new OST server > > which is intended to replace an old one. We do the following steps > > to migrate the data from old OST to new OST: > > > > 1. Figure out the files which located in old OST server: > > > > lfs find --obd {OST_UUID} /mount/point > list > > > > 2. For each file in the list, we do the following migration: > > > > cp -a ..../file ..../file.tmp > > mv ..../file.tmp ..../file > > > > > > However, after doing this data transfer for a few days, we suddently > > found that the hardware of the new OST is failed, and all the data > > migrated on it may not be recovered. > > > > Therefore, I am wondering that whether it is possible to recover > > data from the old OST server ? This may sounds quit impossible, but > > we would like to try if there is any possibility to do that. > > This partly boils down to an ext3 data recovery problem, because the files have been deleted from the old OST where you want to recover them. There are a few tools you could try, including "ext3grep". > > If/when you are able to recover the objects on the old OST, then the secondary problem becomes re-attaching them into the Lustre namespace. If you have 1-stiped files (the default) this is possible, but if you have 2-striped (or wider) files, then it is nearly impossible because the objects on the other OSTs have likely been overwritten by your migration process. > > Cheers, Andreas > -- > Andreas Dilger > Principal Engineer > Whamcloud, Inc. > > >
On 2011-01-13, at 20:33, thhsieh wrote:> Thanks very much for your response. In any case I am very appriciate. > > I did not change the strip size of the Lustre defautl value. So I think > our Lustre filesystem uses 1-strip. Now the problem is how to re-attach > the recovered files to the Lustre name space. > > In fact, we have a completed list of filenames which are migrated to > the new OST. However, only filenames, but without file modified time, > size, owner, permission .... etc. But we can still guess about them. > > So, is there a way to re-attach the files in OST to the Lustre name > space? This is the major difficulty we are facing now.If you are able to recover the object files on the original OST, the easiest way to re-attach them into the Lustre namespace is to just copy them from the ldiskfs-mounted OST into the Lustre filesystem. This can be also be done by running lfsck with the "-l" option.> On Thu, Jan 13, 2011 at 11:09:55AM -0700, Andreas Dilger wrote: >> On 2011-01-13, at 01:41, thhsieh wrote: >>> I am wondering whether it is possible to recover the OST data ? >>> >>> We have faced the following problem. We installed a new OST server >>> which is intended to replace an old one. We do the following steps >>> to migrate the data from old OST to new OST: >>> >>> 1. Figure out the files which located in old OST server: >>> >>> lfs find --obd {OST_UUID} /mount/point > list >>> >>> 2. For each file in the list, we do the following migration: >>> >>> cp -a ..../file ..../file.tmp >>> mv ..../file.tmp ..../file >>> >>> >>> However, after doing this data transfer for a few days, we suddently >>> found that the hardware of the new OST is failed, and all the data >>> migrated on it may not be recovered. >>> >>> Therefore, I am wondering that whether it is possible to recover >>> data from the old OST server ? This may sounds quit impossible, but >>> we would like to try if there is any possibility to do that. >> >> This partly boils down to an ext3 data recovery problem, because the files have been deleted from the old OST where you want to recover them. There are a few tools you could try, including "ext3grep". >> >> If/when you are able to recover the objects on the old OST, then the secondary problem becomes re-attaching them into the Lustre namespace. If you have 1-stiped files (the default) this is possible, but if you have 2-striped (or wider) files, then it is nearly impossible because the objects on the other OSTs have likely been overwritten by your migration process. >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Principal Engineer >> Whamcloud, Inc. >> >> >>Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc.