Hi all. We are using (or should i say "have been using"...) ocfs2 in conjunction with iscsitaget and open-iscsi. Now yesterday happened what should not have happened: A colleague set up a new subversion server and remembered there was a kind of storage to use for it. He used the open-issi tools to discover the partition and used mkfs on it. So he was just fine and could continue his work. :) But: Our main fileserver lost all its data. Apr 24 18:38:21 rad01 kernel: [4310373.927492] (4211,0):o2hb_do_disk_heartbeat:767 ERROR: Device "sdb": another node is heartbeating in our slot! And some hours later: Apr 25 06:25:19 rad01 kernel: [4351792.508496] (1292,0):ocfs2_check_dir_entry:1778 ERROR: bad entry in directory #197799: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0 When trying to mount our (formerly) ocfs2 partition: root at rad01:/# mount -t ocfs2 /dev/sdb /ocfs2 ocfs2_hb_ctl: Bad magic number in inode while reading uuid mount.ocfs2: Error when attempting to run /sbin/ocfs2_hb_ctl: "Operation not permitted" This is on ubuntu 7.10, Linux rad01 2.6.22-14-server #1 SMP Sun Oct 14 23:34:23 GMT 2007 i686 GNU/Linux Any chance to recover the data? Thanks in advance, Christian
Pull out the tapes? You were using that snapshot feature on your SAN, right? When someone runs mkfs over a block device like that, the results are not likely to be good, and rarely are recoverable without some sort of outside restore. I've found snapshots wonderful for situations like this :-). The larger issue, I've found (from some bad experiences, though not quite with such important data), is that your SAN should only allow certain hosts to connect to those volumes. Most SANs feature access control of some sort, and setting things up so that everyone or even an entire subnet can access a volume is pretty dangerous and will result in something like this. Doesn't help you much now, but in the future make sure to implement (finer) access control on your volumes. -Nick>>> On 2008/04/25 at 02:33, Christian Lox <lox at netzwerkplanet.com> wrote:Hi all. We are using (or should i say "have been using"...) ocfs2 in conjunction with iscsitaget and open-iscsi. Now yesterday happened what should not have happened: A colleague set up a new subversion server and remembered there was a kind of storage to use for it. He used the open-issi tools to discover the partition and used mkfs on it. So he was just fine and could continue his work. :) But: Our main fileserver lost all its data. Apr 24 18:38:21 rad01 kernel: [4310373.927492] (4211,0):o2hb_do_disk_heartbeat:767 ERROR: Device "sdb": another node is heartbeating in our slot! And some hours later: Apr 25 06:25:19 rad01 kernel: [4351792.508496] (1292,0):ocfs2_check_dir_entry:1778 ERROR: bad entry in directory #197799: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0 When trying to mount our (formerly) ocfs2 partition: root at rad01:/# mount -t ocfs2 /dev/sdb /ocfs2 ocfs2_hb_ctl: Bad magic number in inode while reading uuid mount.ocfs2: Error when attempting to run /sbin/ocfs2_hb_ctl: "Operation not permitted" This is on ubuntu 7.10, Linux rad01 2.6.22-14-server #1 SMP Sun Oct 14 23:34:23 GMT 2007 i686 GNU/Linux Any chance to recover the data? Thanks in advance, Christian _______________________________________________ Ocfs2-users mailing list Ocfs2-users at oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20080425/91e5df8b/attachment.html
El vie, 25-04-2008 a las 10:33 +0200, Christian Lox escribi?:> Hi all. > > We are using (or should i say "have been using"...) ocfs2 in > conjunction with iscsitaget and open-iscsi. > > Now yesterday happened what should not have happened: > A colleague set up a new subversion server and remembered there was a > kind of storage to use for it. > He used the open-issi tools to discover the partition and used mkfs on > it. So he was just fine and could continue his work. :) > But: > Our main fileserver lost all its data. > > Apr 24 18:38:21 rad01 kernel: [4310373.927492] > (4211,0):o2hb_do_disk_heartbeat:767 ERROR: Device "sdb": another node > is heartbeating in our slot! > > And some hours later: > Apr 25 06:25:19 rad01 kernel: [4351792.508496] > (1292,0):ocfs2_check_dir_entry:1778 ERROR: bad entry in directory > #197799: rec_len is smaller than minimal - offset=0, inode=0, > rec_len=0, name_len=0 > > When trying to mount our (formerly) ocfs2 partition: > > root at rad01:/# mount -t ocfs2 /dev/sdb /ocfs2 > ocfs2_hb_ctl: Bad magic number in inode while reading uuid > mount.ocfs2: Error when attempting to run /sbin/ocfs2_hb_ctl: > "Operation not permitted" > > This is on ubuntu 7.10, Linux rad01 2.6.22-14-server #1 SMP Sun Oct 14 > 23:34:23 GMT 2007 i686 GNU/Linux > > Any chance to recover the data? > > Thanks in advance, > ChristianIf you don't have any backup, as last change you can try: dd if=/dev/sdb of=/path/to_a_dump_file bs=1024K and try http://jbj.rapanden.dk/magicrescue/ on /path/to_a_dump_file Example of use: http://www.linux.com/feature/126525 Best regards D.
Run fsck.ocfs2 with the backup superblock specified. manpage has the details. Also, look into making use of the (r)dump command in debugfs.ocfs2 to retrieve selected data. Best of luck. Christian Lox wrote:> Hi all. > > We are using (or should i say "have been using"...) ocfs2 in > conjunction with iscsitaget and open-iscsi. > > Now yesterday happened what should not have happened: > A colleague set up a new subversion server and remembered there was a > kind of storage to use for it. > He used the open-issi tools to discover the partition and used mkfs on > it. So he was just fine and could continue his work. :) > But: > Our main fileserver lost all its data. > > Apr 24 18:38:21 rad01 kernel: [4310373.927492] > (4211,0):o2hb_do_disk_heartbeat:767 ERROR: Device "sdb": another node > is heartbeating in our slot! > > And some hours later: > Apr 25 06:25:19 rad01 kernel: [4351792.508496] > (1292,0):ocfs2_check_dir_entry:1778 ERROR: bad entry in directory > #197799: rec_len is smaller than minimal - offset=0, inode=0, > rec_len=0, name_len=0 > > When trying to mount our (formerly) ocfs2 partition: > > root at rad01:/# mount -t ocfs2 /dev/sdb /ocfs2 > ocfs2_hb_ctl: Bad magic number in inode while reading uuid > mount.ocfs2: Error when attempting to run /sbin/ocfs2_hb_ctl: > "Operation not permitted" > > This is on ubuntu 7.10, Linux rad01 2.6.22-14-server #1 SMP Sun Oct 14 > 23:34:23 GMT 2007 i686 GNU/Linux > > Any chance to recover the data? > > Thanks in advance, > Christian > > > _______________________________________________ > Ocfs2-users mailing list > Ocfs2-users at oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users >
Am 25.04.2008 um 17:45 schrieb Sunil Mushran:> Run fsck.ocfs2 with the backup superblock specified. > manpage has the details.Running storage:~ # fsck.ocfs2 -r [1-6] /dev/sdb3 always returns [RECOVER_BACKUP_SUPERBLOCK] Recover superblock information from backup block#1048576? <n> y fsck.ocfs2: Bad magic number in inode while initializing the DLM so I guess we are out of luck. Thanks anyway, Christian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20080425/d76193ec/attachment.html