zfs discuss - Feb 2007 - Re: [osol-help] How to recover from "rm *"?

> On Sun, 18 Feb 2007, Calvin Liu wrote:
>
>> I want to run command "rm Dis*" in a folder but mis-typed a
space in it
>> so it became "rm Dis *". Unfortunately I had pressed the
return button
>> before I noticed the mistake. So you all know what happened... :( :( :(
>
> Ouch!
>
>> How can I get the files back in this case?
>
> You restore them from your backups.
>
>> I haven''t backup them.
This is one ( of many ) reasons why ZFS just rocks.  A snapshot would have
saved you.  I don''t consider a snapshot to be an actual backup however.
I
define a backup as something that you can actually restore to bare metal
when your entire datacenter has vanished into a blackhole.  That means a
tape generally.

In the Lotus Notes/Domino world there is a very nice feature where you can
have soft-deletions.  Essentially you can delete a record from a database
and then still do a recovery if needed within a given retention time period.
 Perhaps a soft-deletion feature to ZFS would be nice.  It would allow a
sysadmin or maybe even a user to delete something and then come back later,
check a deletion log and possibly just unrm the file.

Dennis

Hello Dennis,

Monday, February 19, 2007, 12:20:49 AM, you wrote:
>> On Sun, 18 Feb 2007, Calvin Liu wrote:
>>
>>> I want to run command "rm Dis*" in a folder but mis-typed
a space in it
>>> so it became "rm Dis *". Unfortunately I had pressed the
return button
>>> before I noticed the mistake. So you all know what happened... :(
:( :(
>>
>> Ouch!
>>
>>> How can I get the files back in this case?
>>
>> You restore them from your backups.
>>
>>> I haven''t backup them.
DC> This is one ( of many ) reasons why ZFS just rocks.  A snapshot would
have
DC> saved you.  I don''t consider a snapshot to be an actual backup
however.  I
DC> define a backup as something that you can actually restore to bare metal
DC> when your entire datacenter has vanished into a blackhole.  That means a
DC> tape generally.

DC> In the Lotus Notes/Domino world there is a very nice feature where you
can
DC> have soft-deletions.  Essentially you can delete a record from a database
DC> and then still do a recovery if needed within a given retention time
period.
DC>  Perhaps a soft-deletion feature to ZFS would be nice.  It would allow a
DC> sysadmin or maybe even a user to delete something and then come back
later,
DC> check a deletion log and possibly just unrm the file.

Something similar was proposed here before and IIRC someone even has a
working implementation. I don''t know what happened to it.

Anyone? That someone?

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

> Something similar was proposed here before and IIRC someone even has a
> working implementation. I don''t know what happened to it.
That would be me. AFAIK, no one really wanted it.  The problem that it
solves can be solved by putting snapshots in a cronjob.

-- 
Regards,
Jeremy

Hello Jeremy,

Monday, February 19, 2007, 1:58:18 PM, you wrote:
>> Something similar was proposed here before and IIRC someone even has a
>> working implementation. I don''t know what happened to it.
JT> That would be me. AFAIK, no one really wanted it.  The problem that it
JT> solves can be solved by putting snapshots in a cronjob.

Not exactly the same.

But if people really do not want it...

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

On Tue, Feb 20, 2007 at 02:07:41PM +0100, Robert Milkowski
wrote:
> Hello Jeremy,
> 
> Monday, February 19, 2007, 1:58:18 PM, you wrote:
> 
> >> Something similar was proposed here before and IIRC someone even
has a
> >> working implementation. I don''t know what happened to it.
> 
> JT> That would be me. AFAIK, no one really wanted it.  The problem that
it
> JT> solves can be solved by putting snapshots in a cronjob.
> 
> Not exactly the same.
> 
> But if people really do not want it...
There''s a fundamental problem with an undelete facility.

	$ echo > FILE
	$ undelete FILE
	cannot undelete FILE: file exists

-- 
-Gary Mills-    -Unix Support-    -U of M Academic Computing and Networking-

zfs-discuss-bounces at opensolaris.org wrote on 02/20/2007 08:10:59 AM:
> On Tue, Feb 20, 2007 at 02:07:41PM +0100, Robert Milkowski wrote:
> > Hello Jeremy,
> >
> > Monday, February 19, 2007, 1:58:18 PM, you wrote:
> >
> > >> Something similar was proposed here before and IIRC someone
even has
a
> > >> working implementation. I don''t know what happened
to it.
> >
> > JT> That would be me. AFAIK, no one really wanted it.  The problem
that
it
> > JT> solves can be solved by putting snapshots in a cronjob.
> >
> > Not exactly the same.
> >
> > But if people really do not want it...
>
> There''s a fundamental problem with an undelete facility.
>
>    $ echo > FILE
>    $ undelete FILE
>    cannot undelete FILE: file exists

Why the assumption that an undelete command would be brain dead -- this IS
Unix. =) Seems like a low bar issue,  if file exists and undelete has the
file with the same filename available to restore,  error and have the user
-f (Force), -n <filename>  (reName the restored file) ...
Even this:

$ echo "one" > FILE
$ rm FILE
$ echo "two" > FILE
$ rm FILE
$ echo "three" > FILE
$ undelete -n FILE.restored FILE
ERMVERSIONS:  FILE has multiple deleted versions. Try undelete -l FILE to
get the list of available versions, or -f to Force the operation restoring
the latest version.

Should be doable.  I actually have a perl script that does exactly this for
a given file or path based on available snap data (not per filesystem
change snap) -- it does get a little hairy on a fs with many thousands of
snaps as there is some overhead in stat()ing paths under all of those .zfs
mounts.

-Wade

On Tue, Feb 20, 2007 at 10:14:24AM -0600, Wade.Stuart at fallon.com
wrote:
> 
> zfs-discuss-bounces at opensolaris.org wrote on 02/20/2007 08:10:59 AM:
> 
> > On Tue, Feb 20, 2007 at 02:07:41PM +0100, Robert Milkowski wrote:
> > > Hello Jeremy,
> > >
> > > Monday, February 19, 2007, 1:58:18 PM, you wrote:
> > >
> > > >> Something similar was proposed here before and IIRC
someone even has
> a
> > > >> working implementation. I don''t know what
happened to it.
> > >
> > > JT> That would be me. AFAIK, no one really wanted it.  The
problem that
> it
> > > JT> solves can be solved by putting snapshots in a cronjob.
> > >
> > > Not exactly the same.
> > >
> > > But if people really do not want it...
> >
> > There''s a fundamental problem with an undelete facility.
> >
> >    $ echo > FILE
> >    $ undelete FILE
> >    cannot undelete FILE: file exists
> 
> 
> Why the assumption that an undelete command would be brain dead -- this IS
> Unix. =) Seems like a low bar issue,  if file exists and undelete has the
> file with the same filename available to restore,  error and have the user
> -f (Force), -n <filename>  (reName the restored file) ...
> Even this:
> 
> $ echo "one" > FILE
> $ rm FILE
> $ echo "two" > FILE
> $ rm FILE
> $ echo "three" > FILE
> $ undelete -n FILE.restored FILE
> ERMVERSIONS:  FILE has multiple deleted versions. Try undelete -l FILE to
> get the list of available versions, or -f to Force the operation restoring
> the latest version.
Sorry, the name of the file isn''t the problem.  It''s the
contents of
the file.  Does truncating a file constitute deletion?  How about
replacing the contents?  Or replacing part of the contents?  Perhaps
any change to the file?  How far should this go?  Every new line
appended to a log file?

-- 
-Gary Mills-    -Unix Support-    -U of M Academic Computing and Networking-

> > >
> > > There''s a fundamental problem with an undelete facility.
> > >
> > >    $ echo > FILE
> > >    $ undelete FILE
> > >    cannot undelete FILE: file exists
> >
> >
> > Why the assumption that an undelete command would be brain dead --
this
IS
> > Unix. =) Seems like a low bar issue,  if file exists and undelete has
the
> > file with the same filename available to restore,  error and have the
user
> > -f (Force), -n <filename>  (reName the restored file) ...
> > Even this:
> >
> > $ echo "one" > FILE
> > $ rm FILE
> > $ echo "two" > FILE
> > $ rm FILE
> > $ echo "three" > FILE
> > $ undelete -n FILE.restored FILE
> > ERMVERSIONS:  FILE has multiple deleted versions. Try undelete -l FILE
to
> > get the list of available versions, or -f to Force the operation
restoring
> > the latest version.
>
> Sorry, the name of the file isn''t the problem.  It''s the
contents of
> the file.  Does truncating a file constitute deletion?  How about
> replacing the contents?  Or replacing part of the contents?  Perhaps
> any change to the file?  How far should this go?  Every new line
> appended to a log file?
>
> --
> -Gary Mills-    -Unix Support-    -U of M Academic Computing and
Networking-


Got it, my assumption is undelete would only act on deleted files.
Truncating, changing the files data are not delete operations (unlink).
You are starting to talk about versioning at that point -- in which case
this issue becomes way more complicated.  Applications may do multiple
writes to a file with the intent of one transaction,  there is no way to
tell that the three or four writes leave the file in a "broken" state
until
the fifth write happens,  the app may be watching and handling this
transitory state and there is no shared knowledge of its current state.

>Got it, my assumption is undelete would only act on deleted files.
>Truncating, changing the files data are not delete operations (unlink).
>You are starting to talk about versioning at that point -- in which case
>this issue becomes way more complicated.  Applications may do multiple
>writes to a file with the intent of one transaction,  there is no way to
>tell that the three or four writes leave the file in a "broken"
state until
>the fifth write happens,  the app may be watching and handling this
>transitory state and there is no shared knowledge of its current state.

Quite; of course, ZFS snapshots allow you to capture any type of modification,
including truncation or removal.

I can not think of a clear transaction boundary other than close and fsync.

Casper

<begin crackly, broken record :) >

I, for one, would love to have similar functionality that we had in good 
old netware, where we could ''salvage'' deleted files.

The concept was that when the files were deleted, they were not actually 
removed, nor were the all important references to the files to allow 
undeleting them.

In the event that a user had an oops, they could just run salvage (or 
later, filer) and pick the files from the directory in question, and 
*whammo*, undelete it.

I don''t recall ever having to do whole directories, nor if it was 
actually possible...

IIRC, you could set policy that determined when the deleted files data 
blocks became available for overwriting (and hence permanent deletion of 
the file).

If it happened that there was too much space ''used up'' by the
deleted
files, you could run a purge. (I was not a fan of that part).

I''d have preferred that the deleted files were simply overwritten in a 
fifo manner, and left purge out of it.

Yes - Snapshots are great, but how often do you run a snapshot? Every 60 
seconds?  That''s going to get real ugly if you have a filesystem per 
user...

I once cobbled up a poor man''s version of this sort of thing, aliasing 
rm to a scripted mv, and pushing everything into a /<fs>/deleted/* area 
when someone ran rm (maintaining filesystem directory structure). I then 
had occasional rm''s run through it, once the filesystem reached a 
certain highwater mark.

Something under the covers of ZFS that provided dumb dumb protection 
would be very cool. I was saved a number of times by the hackery above...

cheers!

Nathan.

Robert Milkowski wrote:
> Hello Jeremy,
> 
> Monday, February 19, 2007, 1:58:18 PM, you wrote:
> 
>>> Something similar was proposed here before and IIRC someone even
has a
>>> working implementation. I don''t know what happened to it.
> 
> JT> That would be me. AFAIK, no one really wanted it.  The problem that
it
> JT> solves can be solved by putting snapshots in a cronjob.
> 
> Not exactly the same.
> 
> But if people really do not want it...
>

On 2/20/07, Nathan Kroenert <Nathan.Kroenert at sun.com>
wrote:
>
> <begin crackly, broken record :) >
>
> I, for one, would love to have similar functionality that we had in good
> old netware, where we could ''salvage'' deleted files.
>
> The concept was that when the files were deleted, they were not actually
> removed, nor were the all important references to the files to allow
> undeleting them.
>
> In the event that a user had an oops, they could just run salvage (or
> later, filer) and pick the files from the directory in question, and
> *whammo*, undelete it.
>
> I don''t recall ever having to do whole directories, nor if it was
> actually possible...
>
> IIRC, you could set policy that determined when the deleted files data
> blocks became available for overwriting (and hence permanent deletion of
> the file).
>
> If it happened that there was too much space ''used up'' by
the deleted
> files, you could run a purge. (I was not a fan of that part).
>
> I''d have preferred that the deleted files were simply overwritten
in a
> fifo manner, and left purge out of it.
>
> Yes - Snapshots are great, but how often do you run a snapshot? Every 60
> seconds?  That''s going to get real ugly if you have a filesystem
per
> user...
I''m sure  every 15 minutes is suffient,  if the worker doesn''t
have a slight
penalty he will won''t ever learn to be careful.

James Dickens
uadmin.blogspot.com




I once cobbled up a poor man''s version of this sort of thing,
aliasing
> rm to a scripted mv, and pushing everything into a /<fs>/deleted/*
area
> when someone ran rm (maintaining filesystem directory structure). I then
> had occasional rm''s run through it, once the filesystem reached a
> certain highwater mark.
>
> Something under the covers of ZFS that provided dumb dumb protection
> would be very cool. I was saved a number of times by the hackery above...
>
> cheers!
>
> Nathan.
>
> Robert Milkowski wrote:
> > Hello Jeremy,
> >
> > Monday, February 19, 2007, 1:58:18 PM, you wrote:
> >
> >>> Something similar was proposed here before and IIRC someone
even has a
> >>> working implementation. I don''t know what happened to
it.
> >
> > JT> That would be me. AFAIK, no one really wanted it.  The problem
that
> it
> > JT> solves can be solved by putting snapshots in a cronjob.
> >
> > Not exactly the same.
> >
> > But if people really do not want it...
> >
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070220/30bbeecd/attachment.html>

I''d usually agree with that, but - if we have an opportunity to make 
users love ZFS even more, why not at least investigate it.

A perfect example might be exactly what I did on one occasion, where I 
copied a bunch of photos off a CF card. I then reformatted the CF card, 
and cleaned up the the crappy photos on my hard disk, but unfortunately, 
(and stupidly) removed all of them. My photos were gone forever.  :(

Even a snapshot would not have helped here...

I know; stupid stupid stupid, but it happens, and I would have *really* 
liked to have been able to recover those photos...

A salvage / undelete would have been gold.

Nathan.

James Dickens wrote:
>> Yes - Snapshots are great, but how often do you run a snapshot? Every
60
>> seconds?  That''s going to get real ugly if you have a
filesystem per
>> user...
> 
> I''m sure  every 15 minutes is suffient,  if the worker
doesn''t have a
> slight
> penalty he will won''t ever learn to be careful.
>