Dear Chris, James, Valeri and all, Sorry to have not responded as I'm still on struggling with the recovery with no success. I've been trying to set up a new system with the exact same scenario (4 2TB hard drives and remove the 3rd one afterwards). I still cannot recover. We did have a backup system but it went bad for a while and we did not have replacement on time until this happened. From all of your responses, it seems, recovery is almost impossible. I'm now trying to look at the hardware part and get the damaged hard drive to fixed. I appreciate all you helps and still wait and listen to more suggestions. Regards, Khem On 03/01/2015 08:40 AM, Chris Murphy wrote:> On Sat, Feb 28, 2015 at 5:59 PM, James A. Peltier <jpeltier at sfu.ca> wrote: >> There is no difference between a single disk system and a multi-disk system in terms of being able to dynamically resize volumes that reside on a volume group. Having the ability to resize a volume to be either larger or smaller on demand is a really nice feature to have. > I'll better qualify this. For CentOS it's a fine default, as it is for > Fedora Server. For Workstation and Cloud I think LVM overly > complicates things. More non-enterprise users get confused over LVM > than they ever have a need to resize volumes. > >> Did you make / too small and have space on home and you're using ext3/4 then simply resize the home logical volume to be smaller and all the free extents to /. Pretty simple process really and it can be done online. > XFS doesn't support shrink, only grow. XFS is the CentOS 7 default. > The main advantage of LVM for CentOS system disks is ability to use > pvmove to replace a drive online, rather than resize. If Btrfs > stabilizes sufficiently for RHEL/CentOS 8, overall it's a win because > it meets the simple need of mortal users and supports advanced > features for advanced users. (Ergo I think LVM is badass but it's also > the storage equivalent of emacs - managing it is completely crazy.) > >> This is just one example. There are others, but this has nothing to do with the OP. >> >> Getting back to the OP, it would seem that you may be stuck in a position where you need to restore from backup. Without having further details into what exactly is happening I fear you're not going to be able to recover. I'd be available to talk off list if needed. > Yeah my bad for partly derailing this thread. Hopefully the original > poster hasn't been scared off, not least of which may be due to my > bark about cross posting being worse than my bite. >
On Sun, March 1, 2015 9:07 pm, Khemara Lin wrote:> Dear Chris, James, Valeri and all, > > Sorry to have not responded as I'm still on struggling with the recovery > with no success. > > I've been trying to set up a new system with the exact same scenario (4 > 2TB hard drives and remove the 3rd one afterwards). I still cannot > recover. > > We did have a backup system but it went bad for a while and we did not > have replacement on time until this happened. > > From all of your responses, it seems, recovery is almost impossible. > I'm now trying to look at the hardware part and get the damaged hard > drive to fixed. > > I appreciate all you helps and still wait and listen to more suggestions. >There may be a bit expensive route. Depending on how valuable the data are, you may think of contacting professional recovery services. They usually take about a Month, they are expensive. Decent ones will be on the order of $1000 if it is a single drive. Likely more if it is fatally failed RAID. You can do your research and find good ones close to you. The rule of thumb is: if they only charge in case of more or less successful recovery (and sometimes they can recover almost 100%, sometimes 70-80% sometimes nothing - then they will not charge you), then it probably is decent company. They live from results of their work. If they charge for "estimate" even if they tell later they can not recover, this is bad sign. They work with fine equipment to read stuff off the platters of died drives. They work on the level of debugging of filesystems (and RAIDs), so what they charge is usually not that much for the kind of work they do. If you don't feel you are that level of expert as they are, and the data is worth it, I would contact recovery services. I myself usually have good backup (knocking on wood), but I know several people who actually used some of these companies, and their data got recovered. If you come to the point of need some references, contact me off the list, I'll dig up my old emails, and will send you what people (whom I know in person) say about the companies they used successfully. Valeri ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
On Sun, Mar 1, 2015 at 8:07 PM, Khemara Lin <lin.kh at wicam.com.kh> wrote:> Dear Chris, James, Valeri and all, > > Sorry to have not responded as I'm still on struggling with the recovery > with no success. > > I've been trying to set up a new system with the exact same scenario (4 2TB > hard drives and remove the 3rd one afterwards). I still cannot recover.Well, it's effectively a raid0. While it's not block level striping, it's a linear allocation, the way ext4 and XFS write, you're going to get file extents and fs metadata strewn across all four drives. As soon as any one drive is removed the whole thing is sufficiently damaged it can't recover without a lot of work. Imagine a (really bad example physics wise) single drive scenario and magically punching a hole through a drive such that it'll still spin. The fs on that drive is doing to have all sorts of problems because of the hole, even if it can read 3/4 of the drive.> > We did have a backup system but it went bad for a while and we did not have > replacement on time until this happened. > > From all of your responses, it seems, recovery is almost impossible. I'm now > trying to look at the hardware part and get the damaged hard drive to fixed.About the best case scenario with such a situation is literally do nothing with the LVM setup, and send that PV off for block level data recovery (you didn't say how it failed but I'm assuming it's beyond the ability to fix it locally). Then once the recovered replacement PV is back in the setup, things will just work again. *shrug* LVM linear isn't designed to be fail safe in the face of a single device failure. -- Chris Murphy
On Sun, Mar 1, 2015 at 9:03 PM, Valeri Galtsev <galtsev at kicp.uchicago.edu> wrote:> There may be a bit expensive route. Depending on how valuable the data > are, you may think of contacting professional recovery services. They > usually take about a Month, they are expensive. Decent ones will be on the > order of $1000 if it is a single drive. Likely more if it is fatally > failed RAID.Actually, I'm probably wrong in the previous post about sending off the single bad PV for recovery. Your point above made me think, umm yeah no, pretty much any company specializing in data recovery will want the entire array/LV backing drives, even the good ones. Same for RAID, they probably don't want just the dead drive, they want the whole thing. And they charge by the total size. So, yeah probably a lot more than $1K.> If they charge for > "estimate" even if they tell later they can not recover, this is bad sign.Agreed. -- Chris Murphy