I was an idiot. I made a mistake. Hopefully my up front admission of my error will help us get past the blasting I deserve. I updated from xcp 1.1 to 1.6 - upon doing so and rebooting I tried to reattach the HBA storage I was using before - and was told it needed to be reinitialize (no problem I thought - I have good backups!). Regrettably something that was backed up failed to restore... and will have to spend a lot of time recovering. Can I ask what the underlying file system is? Has anyone worked with a recovery company they can recommend? Specifically I only REALLY need a single virtual hard drive - Ideally the whole vm would be nice - which was created late in the host server''s lifetime (so theoretically further into the disk). Failing that, whatever FILES from that vhd that could be pulled would be great. The reinit was fast - so it obviously didn''t WIPE the disk. Sooo... 1) Any information on underlying file system / advice on recovery methods etc. 2) Any recommendations on consultants or firms who have fixed this sort of error for you? 3) Any advice for the future on a way to remount an HBA device after an upgrade or an LVM if that''s preferred (and why does it default to HBA if LVM is better?) I''ve been reading but probably reading the wrong stuff as I don''t see a clear answer yet... Thank you very much for you help in advance! Mitch. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On Mon, 04/15/2013 06:08 PM, "mitch@bitblock.net" <mitch@bitblock.net> wrote:> Specifically I only REALLY need a single virtual hard drive - Ideally the whole vm would be nice - which was created late in the host server''s lifetime (so theoretically further into the disk). > > Failing that, whatever FILES from that vhd that could be pulled would be great. > > The reinit was fast - so it obviously didn''t WIPE the disk. >You are correct. The reinit would have wiped the previous LVM structure and created a new one. Check /etc/lvm/archive or /etc/lvm/backup for a backup vg config. You can then use vgcfgrestore to restore the previous configuration onto the SR.> > 1) Any information on underlying file system / advice on recovery methods etc. > > 2) Any recommendations on consultants or firms who have fixed this sort of error for you? > > 3) Any advice for the future on a way to remount an HBA device after an upgrade or an LVM if that''s preferred (and why does it default to HBA if LVM is better?) I''ve been reading but probably reading the wrong stuff as I don''t see a clear answer yet...HBAoLVM is the default SR type for anything attached via FC through XenCenter.
Hi Errol - thanks for the reply - more below... -----Original Message----- From: Errol Neal [mailto:eneal@businessgrade.com] Sent: April 15, 2013 6:22 PM To: Mitch (Bitblock) Cc: ''Xen-users@lists.xen.org'' Subject: Re: [Xen-users] Storage recovery question / recommendations On Mon, 04/15/2013 06:08 PM, "mitch@bitblock.net" <mitch@bitblock.net> wrote:> Specifically I only REALLY need a single virtual hard drive - Ideally the whole vm would be nice - which was created late in the host server''s lifetime (so theoretically further into the disk). > > Failing that, whatever FILES from that vhd that could be pulled would be great. > > The reinit was fast - so it obviously didn''t WIPE the disk. >You are correct. The reinit would have wiped the previous LVM structure and created a new one. Check /etc/lvm/archive or /etc/lvm/backup for a backup vg config. You can then use vgcfgrestore to restore the previous configuration onto the SR. [Mitch says:] The problem is that the system was reinstalled - so would these contents really be valid? Right now in /etc/lvm/backup I see: VG_XenStorage-1250f2e7-128e-7aff-f0fe-ca03bae1eb56 VG_XenStorage-63aedcf6-e60a-4bc0-935a-7b393a0b3b33 VG_XenStorage-6fb39163-2cb1-9464-5d19-5dc2e951e38a VG_XenStorage-84bae666-255d-196a-54aa-73afc5047d70 How would I tell which is which? Please note, the Type as viewed under storage for this SR was "Hardware HBA" - not LVM - does that matter? I''ll read up on vgcfgrestore Any further pointers appreciated. Thanks!> > 1) Any information on underlying file system / advice on recovery methods etc. > > 2) Any recommendations on consultants or firms who have fixed this sort of error for you? > > 3) Any advice for the future on a way to remount an HBA device after an upgrade or an LVM if that''s preferred (and why does it default to HBA if LVM is better?) I''ve been reading but probably reading the wrong stuff as I don''t see a clear answer yet...HBAoLVM is the default SR type for anything attached via FC through XenCenter. [Mitch says:] These aren''t on FC - it was just an SAS Raid1 SR - so it was set up as "Hardware HBA".
On Mon, 04/15/2013 09:38 PM, "mitch@bitblock.net" <mitch@bitblock.net> wrote:> [Mitch says:] The problem is that the system was reinstalled - so would these contents really be valid? > Right now in /etc/lvm/backup I see: > VG_XenStorage-1250f2e7-128e-7aff-f0fe-ca03bae1eb56 > VG_XenStorage-63aedcf6-e60a-4bc0-935a-7b393a0b3b33 > VG_XenStorage-6fb39163-2cb1-9464-5d19-5dc2e951e38a > VG_XenStorage-84bae666-255d-196a-54aa-73afc5047d70 > How would I tell which is which?Well that is a good question. Check the time stamp. You can also check to see if you have a backup partition on /dev/sda2. Try mounting it.> Please note, the Type as viewed under storage for this SR was "Hardware HBA" - not LVM - does that matter?No.> [Mitch says:] These aren''t on FC - it was just an SAS Raid1 SR - so it was set up as "Hardware HBA".Sorry I''m stuck in FC mode. Yes, no matter, it''s going to be LVM..
I did some reading - on vgcfgbackup / restore / display etc. I think I''m still stuck - or I''m not getting it - how would the backup exist - does xen / xcp automatically back up any existing LVM data on a new SR if it sees one there first when it''s mounting what it see''s as a new drive? Otherwise, the backups I have would only be for the newly re-initialized LVM - meaning the data is still on the drive - or gone. Not in the backups - right? Maybe I should restate what happened in case I was unclear... I think I might have been - sorry about that - in trying to be concise I might have oversimplified. I installed 1.6 on a machine that had 1.1 - it wasn''t an upgrade in the proper sense - it didn''t detect the drives in the same order. So I reinstalled. Plus I wanted to break up a different SR (which was one LVM spanning a couple disks into separate SR''s). When I was finished "upgrading" / reinstalling. I added an SR based on a previously existing raid 1 data set. It said it needed to reinitialize. I let that happen. So the base of that SR was re-written. The data should still be there but any current backup would reflect the newly wiped state of the SR. Is there a tool that can extract the details I need from the drive itself? Like a backup of a partition table or "superblock" etc.? I know the name of the vm I need to recover, the size of the virtual drives, etc. the labels... And if I was going to use such a tool what how should it treat the disk as a corrupted LVM or ? Thank you again for your help - hope I''m making more sense. Mitch. -----Original Message----- From: Errol Neal [mailto:eneal@businessgrade.com] Sent: April 15, 2013 7:33 PM To: Mitch (Bitblock) Cc: ''Xen-users@lists.xen.org'' Subject: Re: RE: [Xen-users] Storage recovery question / recommendations On Mon, 04/15/2013 09:38 PM, "mitch@bitblock.net" <mitch@bitblock.net> wrote:> [Mitch says:] The problem is that the system was reinstalled - so would these contents really be valid? > Right now in /etc/lvm/backup I see: > VG_XenStorage-1250f2e7-128e-7aff-f0fe-ca03bae1eb56 > VG_XenStorage-63aedcf6-e60a-4bc0-935a-7b393a0b3b33 > VG_XenStorage-6fb39163-2cb1-9464-5d19-5dc2e951e38a > VG_XenStorage-84bae666-255d-196a-54aa-73afc5047d70 > How would I tell which is which?Well that is a good question. Check the time stamp. You can also check to see if you have a backup partition on /dev/sda2. Try mounting it.> Please note, the Type as viewed under storage for this SR was "Hardware HBA" - not LVM - does that matter?No.> [Mitch says:] These aren''t on FC - it was just an SAS Raid1 SR - so it was set up as "Hardware HBA".Sorry I''m stuck in FC mode. Yes, no matter, it''s going to be LVM..
On Tue, 04/16/2013 12:46 AM, "mitch@bitblock.net" <mitch@bitblock.net> wrote:> I did some reading - on vgcfgbackup / restore / display etc. > > I think I''m still stuck - or I''m not getting it - how would the backup exist - does xen / xcp automatically back up any existing LVM data on a new SR if it sees one there first when it''s mounting what it see''s as a new drive? Otherwise, the backups I have would only be for the newly re-initialized LVM - meaning the data is still on the drive - or gone. Not in the backups - right?We aren''t talking about backups of data. We are talking about metadata backups of the volume groups(s) themselves.They are created when you make changes to a VG like adding or removing an LV.. So long as actually haven''t written any data to the new volume group, then it should be possible to restore the previous layout and logical volumes. You should review each of the files and see which one matches the characteristics of the VG that you are trying to recover.> Maybe I should restate what happened in case I was unclear... I think I might have been - sorry about that - in trying to be concise I might have oversimplified. > > I installed 1.6 on a machine that had 1.1 - it wasn''t an upgrade in the proper sense - it didn''t detect the drives in the same order. So I reinstalled. Plus I wanted to break up a different SR (which was one LVM spanning a couple disks into separate SR''s).Well if you didn''t upgrade and just reinstalled, also reformatting the root partition, then your are right, you don''t have any backups.> Is there a tool that can extract the details I need from the drive itself? Like a backup of a partition table or "superblock" etc.? >You can try testdisk. I''ve never used it for lvm recovery, but it''s saved my ass for other stuff. Here is an example of someone using it for LVM recovery: http://itknowledgeexchange.techtarget.com/linux-lotus-domino/recovering-files-from-an-lvm-or-ext3-partition-with-testdisk/ Let me know how it works out..
Presuming the files are recoverable, is there likely any way to recognize them? I guess I will be searching for the virtual disk files themselves which I could add as storage to the replacement virtual machine - on vmware I could find files related to a vm under a folder often named for that vm - the structure wasn''t very deep... Is the xen structure similar? I must be using the wrong keywords - but so far I haven''t seen at a command promt what or where xen vm files are stored and what they look like so I''m not sure what to expect. Does anyone have any tips in this regard? Will post my results whether or not I''m successful. Thanks again! M ----- Original Message ----- From: Errol Neal [mailto:eneal@businessgrade.com] Sent: Tuesday, April 16, 2013 04:48 AM To: Mitch (Bitblock) Cc: ''Xen-users@lists.xen.org'' <Xen-users@lists.xen.org> Subject: Re: [Xen-users] Storage recovery question / recommendations On Tue, 04/16/2013 12:46 AM, "mitch@bitblock.net" <mitch@bitblock.net> wrote:> I did some reading - on vgcfgbackup / restore / display etc. > > I think I''m still stuck - or I''m not getting it - how would the backup exist - does xen / xcp automatically back up any existing LVM data on a new SR if it sees one there first when it''s mounting what it see''s as a new drive? Otherwise, the backups I have would only be for the newly re-initialized LVM - meaning the data is still on the drive - or gone. Not in the backups - right?We aren''t talking about backups of data. We are talking about metadata backups of the volume groups(s) themselves.They are created when you make changes to a VG like adding or removing an LV.. So long as actually haven''t written any data to the new volume group, then it should be possible to restore the previous layout and logical volumes. You should review each of the files and see which one matches the characteristics of the VG that you are trying to recover.> Maybe I should restate what happened in case I was unclear... I think I might have been - sorry about that - in trying to be concise I might have oversimplified. > > I installed 1.6 on a machine that had 1.1 - it wasn''t an upgrade in the proper sense - it didn''t detect the drives in the same order. So I reinstalled. Plus I wanted to break up a different SR (which was one LVM spanning a couple disks into separate SR''s).Well if you didn''t upgrade and just reinstalled, also reformatting the root partition, then your are right, you don''t have any backups.> Is there a tool that can extract the details I need from the drive itself? Like a backup of a partition table or "superblock" etc.? >You can try testdisk. I''ve never used it for lvm recovery, but it''s saved my ass for other stuff. Here is an example of someone using it for LVM recovery: http://itknowledgeexchange.techtarget.com/linux-lotus-domino/recovering-files-from-an-lvm-or-ext3-partition-with-testdisk/ Let me know how it works out..
On Tue, 04/16/2013 11:01 AM, "mitch@bitblock.net" <mitch@bitblock.net> wrote:> Presuming the files are recoverable, is there likely any way to recognize them? I guess I will be searching for the virtual disk files themselves which I could add as storage to the replacement virtual machine - on vmware I could find files related to a vm under a folder often named for that vm - the structure wasn''t very deep... > > Is the xen structure similar? I must be using the wrong keywords - but so far I haven''t seen at a command promt what or where xen vm files are stored and what they look like so I''m not sure what to expect. >yes - that is what testdisk is for. it will scan the physical disk and look for recognizable signatures associated with filesystems. so long as you haven''t overwritten or zero''d out any of the data blocks, id'' expect that you''d be able to recover all of your data.
Hi Errol and all :-) Testdisk (photorec actually) was most helpful. I was able to recover a 10GB data partition I think. For some reason (maybe because it was allocated on hardware HBA?) the vhd seemed to be recovered whole... But there is another VHD I need - and this one is harder. It contains a 24GB base part, and was expanded once or twice with a few more GB. I''m not sure how that would work - on NFS when I allocate or increase a VHD it seems to be allocated as required, however when I''ve made these changes on the local storage the allocation seems to occur as needed (resulting in fragmentation). So... 1) am I asking these follow ups in the right place or is this more appropriate for another list? Assuming I''m ok here... 2) Is it likely the additional allocation will be in a single "chunk" - and will there likely be any way to identify it? I recovered the other VHD by adding a signature to photorec for VHD and searching the raw disk for the signature. But I think it only pulled up the first initial allocation - the part that was added on seems to be missing. 3) I have a problem with the partial vhd''s...as a result of the missing pieces, xen, which seems to "validate" the VHD prior to starting the VM. They fail with "The attempt to load the VDI failed". If I could connect them anyways I might be able to scan the contents - looking for another way to do that... I''ve got an active thread there as well on the testdisk site: http://forum.cgsecurity.org/phpBB3/post7921.html#p7921 Thank you for your time. M -----Original Message----- From: Errol Neal [mailto:eneal@businessgrade.com] Sent: April 16, 2013 8:10 AM To: Mitch (Bitblock) Cc: ''Xen-users@lists.xen.org'' Subject: Re: [Xen-users] Storage recovery question / recommendations On Tue, 04/16/2013 11:01 AM, "mitch@bitblock.net" <mitch@bitblock.net> wrote:> Presuming the files are recoverable, is there likely any way to recognize them? I guess I will be searching for the virtual disk files themselves which I could add as storage to the replacement virtual machine - on vmware I could find files related to a vm under a folder often named for that vm - the structure wasn''t very deep... > > Is the xen structure similar? I must be using the wrong keywords - but so far I haven''t seen at a command promt what or where xen vm files are stored and what they look like so I''m not sure what to expect. >yes - that is what testdisk is for. it will scan the physical disk and look for recognizable signatures associated with filesystems. so long as you haven''t overwritten or zero''d out any of the data blocks, id'' expect that you''d be able to recover all of your data.
On Thu, 04/18/2013 04:27 AM, "mitch@bitblock.net" <mitch@bitblock.net> wrote:> Hi Errol and all :-) > > Testdisk (photorec actually) was most helpful. I was able to recover a 10GB data partition I think. For some reason (maybe because it was allocated on hardware HBA?) the vhd seemed to be recovered whole...Good to know things are going in the right direction!> > But there is another VHD I need - and this one is harder. It contains a 24GB base part, and was expanded once or twice with a few more GB. > > I''m not sure how that would work - on NFS when I allocate or increase a VHD it seems to be allocated as required, however when I''ve made these changes on the local storage the allocation seems to occur as needed (resulting in fragmentation).An virtual disk (VHD) on NFS is different than a virtual disk on LVM''d SR. When you extend a VD on such an SR, the blocks may not be allocated sequentially. The latter part of the VD will be allocated as LVM sees fit. Photorec may be your best option in this regard.> So... > 1) am I asking these follow ups in the right place or is this more appropriate for another list? > Assuming I''m ok here...At this point, I''d say (and I''m sure others would agree) that this is less about Xen and more about data recovery. That said, I''m willing to share what I know but your best bet may be the LVM2 forums or the testdisk forums.