Btrfs devel - Dec 2010 - What do the arguments of btrfs filesystem defragment do?

Hi,

I would like to know that the arguments of "btrfs filesystem
defragment"
do. According to the built-in help page, the invocation is as follows:
>         btrfs filesystem defragment [-vcf] [-s start] [-l len] [-t
>                 size] <file>|<dir>
[<file>|<dir>...]
>                 Defragment a file or a directory.
Unfortunately I can''t find any documentation on the meaning of these
arguments. The man page doesn''t list these arguments:
>       filesystem defragment <file>|<dir>
[<file>|<dir>...]
>              Defragment files and/or directories.
Also the online documentation [1] is identical to the man page.

[1] https://btrfs.wiki.kernel.org/index.php/Btrfs(command)

On a somewhat related note I did find another page on the wiki [2] that
explains a bit on the defragmenting subject, more specifically it
mentions this rather important caveat:
> Caveat: Defragmenting a file which has a COW copy (either a snapshot
> copy or one made with bcp or cp --reflinks) will produce two unrelated
> files. If you defragment a subvolume that has a snapshot, you will
> roughly double the disk usage, as the snapshot files are no longer COW
> images of the originals.
[2] https://btrfs.wiki.kernel.org/index.php/Problem_FAQ

From what I''ve heard on IRC this is still the case in current versions,
but the Btrfs(command) documentation contains no mention of this.

I hope someone can shed some light on these subjects.

Kind regards,

Erik.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Excerpts from Erik Logtenberg''s message of 2010-12-15 11:06:03
-0500:
> Hi,
> 
> I would like to know that the arguments of "btrfs filesystem
defragment"
> do. According to the built-in help page, the invocation is as follows:
> 
> >         btrfs filesystem defragment [-vcf] [-s start] [-l len] [-t
> >                 size] <file>|<dir>
[<file>|<dir>...]
> >                 Defragment a file or a directory.
> 
> Unfortunately I can''t find any documentation on the meaning of
these
> arguments. The man page doesn''t list these arguments:
-v is verbose.  It just prints the file name as it defrags
-c forces compression, even if you are mounted without compression on

-s is the starting byte you want to defrag in the file.  This is useful
for very large files that are only compressed in a particular spot

-l is the number of bytes you want to compress, would be paired with -s

-t is a threshold.  If the extent is bigger than this threshold, we
consider it defragged already.

-f starts the IO right away.
> 
> >       filesystem defragment <file>|<dir>
[<file>|<dir>...]
> >              Defragment files and/or directories.
> 
> Also the online documentation [1] is identical to the man page.
> 
> [1] https://btrfs.wiki.kernel.org/index.php/Btrfs(command)
> 
> On a somewhat related note I did find another page on the wiki [2] that
> explains a bit on the defragmenting subject, more specifically it
> mentions this rather important caveat:
> 
> > Caveat: Defragmenting a file which has a COW copy (either a snapshot
> > copy or one made with bcp or cp --reflinks) will produce two unrelated
> > files. If you defragment a subvolume that has a snapshot, you will
> > roughly double the disk usage, as the snapshot files are no longer COW
> > images of the originals.
> 
> [2] https://btrfs.wiki.kernel.org/index.php/Problem_FAQ
> 
> >From what I''ve heard on IRC this is still the case in current
versions,
> but the Btrfs(command) documentation contains no mention of this.
This is still true.
> 
> I hope someone can shed some light on these subjects.
> 
Hope this makes sense ;)

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Chris, thank you very much for your explanation. Indeed this clears
things up a bit.
>>> Caveat: Defragmenting a file which has a COW copy (either a
snapshot
>>> copy or one made with bcp or cp --reflinks) will produce two
unrelated
>>> files. If you defragment a subvolume that has a snapshot, you will
>>> roughly double the disk usage, as the snapshot files are no longer
COW
>>> images of the originals.
>>
>> [2] https://btrfs.wiki.kernel.org/index.php/Problem_FAQ
>>
>> >From what I''ve heard on IRC this is still the case in
current versions,
>> but the Btrfs(command) documentation contains no mention of this.
> 
> This is still true.
Is there a decent way to have btrfs compress already existing files
(that were written before compression was enabled) without hurting any
of the internal structures such as snapshots?
The goal is to increase free disk space and possibly performance, not to
expload disk usage by breaking COW relations. So given your reply, I
assume that defragmenting all files is not the right way (?)

Kind regards,

Erik.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Wednesday, 15 December, 2010, Erik Logtenberg wrote:
> Hi,
> 
> I would like to know that the arguments of "btrfs filesystem
defragment"
> do. According to the built-in help page, the invocation is as follows:
> 
> >         btrfs filesystem defragment [-vcf] [-s start] [-l len] [-t
> >                 size] <file>|<dir>
[<file>|<dir>...]
> >                 Defragment a file or a directory.
> 
> Unfortunately I can''t find any documentation on the meaning of
these
> arguments. The man page doesn''t list these arguments:
I loked at the code and I try to explain. But what I write below may be 
affected by error.

If the parameter is a directory, the subvolume where the dir is placed is 
defragged.

Instead if the parameter is a file, the switches have the following meaning:

-f -> flush after defrag
-s -> start point of defrag
-l -> length of defrag range (if len is <= 0, the rest of file is
considered)
-c -> force compress
-t -> threshold for compression (if the size of file is bigger, compress the 
file ?)


-- 
gpg key@ keyserver.linux.it: Goffredo Baroncelli (ghigo)
<kreijack@inwind.it>
Key fingerprint = 4769 7E51 5293 D36C 814E  C054 BF04 F161 3DC5 0512
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Excerpts from Erik Logtenberg''s message of 2010-12-15 14:14:15
-0500:
> Chris, thank you very much for your explanation. Indeed this clears
> things up a bit.
> 
> >>> Caveat: Defragmenting a file which has a COW copy (either a
snapshot
> >>> copy or one made with bcp or cp --reflinks) will produce two
unrelated
> >>> files. If you defragment a subvolume that has a snapshot, you
will
> >>> roughly double the disk usage, as the snapshot files are no
longer COW
> >>> images of the originals.
> >>
> >> [2] https://btrfs.wiki.kernel.org/index.php/Problem_FAQ
> >>
> >> >From what I''ve heard on IRC this is still the case in
current versions,
> >> but the Btrfs(command) documentation contains no mention of this.
> > 
> > This is still true.
> 
> Is there a decent way to have btrfs compress already existing files
> (that were written before compression was enabled) without hurting any
> of the internal structures such as snapshots?
I''m afraid not yet.  There is code for this in the btrfs balance
routines, but we haven''t yet adapted it to the defragment command.

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

>> Is there a decent way to have btrfs compress already existing files
>> (that were written before compression was enabled) without hurting any
>> of the internal structures such as snapshots?
> 
> I''m afraid not yet.  There is code for this in the btrfs balance
> routines, but we haven''t yet adapted it to the defragment command.
Okay, just to be 100% sure, if I add -o compress to the mount options,
this will cause btrfs to (try to) compress newly written files, right?
Is this fully compatible with existing snapshots, or will compressing
files always (have a chance to) hurt COW relations?

The use case is a filesystem used for backups, which are rsynced
nightly, after which a new snapshot is made. After something like 45
days, the old snapshots are removed. I am assuming that this way, after
45 days all files will be compressed naturally, but this is only
beneficial if snapshots still fully work. If instead it results in
storing the compressed form of every file 45 times on disk, then it
won''t help much.

Kind regards,

Erik.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Excerpts from Erik Logtenberg''s message of 2010-12-15 14:26:49
-0500:
> >> Is there a decent way to have btrfs compress already existing
files
> >> (that were written before compression was enabled) without hurting
any
> >> of the internal structures such as snapshots?
> > 
> > I''m afraid not yet.  There is code for this in the btrfs
balance
> > routines, but we haven''t yet adapted it to the defragment
command.
> 
> Okay, just to be 100% sure, if I add -o compress to the mount options,
> this will cause btrfs to (try to) compress newly written files, right?
Yes
> Is this fully compatible with existing snapshots, or will compressing
> files always (have a chance to) hurt COW relations?
COW references (both via snapshotting and cp --reflink) are broken when
you change the file in either copy.  While the
files are the same, they are shared and the parts that change are
private to each copy.  If you have a reflink copy of a 1TB file and
change only 4K in the middle, the other 1TB-4KB are still shared.

This is true for both the default and with compression on.
> 
> The use case is a filesystem used for backups, which are rsynced
> nightly, after which a new snapshot is made. After something like 45
> days, the old snapshots are removed. I am assuming that this way, after
> 45 days all files will be compressed naturally, but this is only
> beneficial if snapshots still fully work. If instead it results in
> storing the compressed form of every file 45 times on disk, then it
> won''t help much.
Yes, you''ll end up with a fully compressed and fully shared setup after
45 days.

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

* [Chris Mason] 
> Excerpts from Erik Logtenberg''s message of 2010-12-15 14:26:49
-0500:
>> 
>> The use case is a filesystem used for backups, which are rsynced
>> nightly, after which a new snapshot is made. After something like 45
>> days, the old snapshots are removed. I am assuming that this way, after
>> 45 days all files will be compressed naturally, but this is only
>> beneficial if snapshots still fully work. If instead it results in
>> storing the compressed form of every file 45 times on disk, then it
>> won''t help much.
>
> Yes, you''ll end up with a fully compressed and fully shared setup
after
> 45 days.
How would that happen?  Rsync only rewrites files if they have changed.
If compression happens only when a file is written to, any files that
were written uncompressed will remain uncompressed until they change on
the source filesystem (triggering a rewrite on the backup drive).

Unless there''s some magic I''m missing, expiring the old
snapshots from
before -o compress won''t really affect anything.

Øystein
-- 
Outgoing mail is certified Virus Free.
..of course, the virus would tell you the same thing..

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Wed, Dec 15, 2010 at 1:20 PM, Chris Mason <chris.mason@oracle.com>
wrote:
>>
>> Is there a decent way to have btrfs compress already existing files
>> (that were written before compression was enabled) without hurting any
>> of the internal structures such as snapshots?
>
> I''m afraid not yet.  There is code for this in the btrfs balance
> routines, but we haven''t yet adapted it to the defragment command.
>
> -chris
with enough disk space, if the OP adds a compressed device, deletes
the original device, adds it back as compressed, and then deletes the
temporary device, would that work?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html