Btrfs devel - May 2012 - Is it possible to reclaim block groups once they are allocated to data or metadata?

Hello list,

recently reformatted my home partition from XFS to RAID1 btrfs. I used
the default options to mkfs.btrfs except for enabling raid1 for data
as well as metadata. Filesystem is made up of two 1TB drives.

mike@mercury (0) pts/3 ~ $ sudo btrfs filesystem show
Label: none  uuid: f08a8896-e03e-4064-9b94-9342fb547e47
	Total devices 2 FS bytes used 888.06GB
	devid    1 size 931.51GB used 931.51GB path /dev/sdb1
	devid    2 size 931.51GB used 931.49GB path /dev/sdc1

Btrfs Btrfs v0.19

mike@mercury (0) pts/3 ~ $ btrfs filesystem df /home
Data, RAID1: total=893.48GB, used=886.60GB
Data: total=8.00MB, used=0.00
System, RAID1: total=8.00MB, used=136.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=38.00GB, used=2.54GB
Metadata: total=8.00MB, used=0.00

As can be seen I don''t have a lot of free space left and while I am
planning on adding more storage soon I would like to gain a little
breathing room until I can do this. While I don''t have a lot of space
remaining in Data, RAID1 I do have a good chunk in Metadata, RAID1.
2.5GB used out of 38GB. Does this excess become available
automatically to the file system when the block groups in Data, RAID1
are exhausted or, if not, is there a way to manually reallocate them?

Regards,

Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Sat, May 05, 2012 at 10:37:01PM +1000, Mike Sampson
wrote:
> Hello list,
> 
> recently reformatted my home partition from XFS to RAID1 btrfs. I used
> the default options to mkfs.btrfs except for enabling raid1 for data
> as well as metadata. Filesystem is made up of two 1TB drives.
> 
> mike@mercury (0) pts/3 ~ $ sudo btrfs filesystem show
> Label: none  uuid: f08a8896-e03e-4064-9b94-9342fb547e47
> 	Total devices 2 FS bytes used 888.06GB
> 	devid    1 size 931.51GB used 931.51GB path /dev/sdb1
> 	devid    2 size 931.51GB used 931.49GB path /dev/sdc1
> 
> Btrfs Btrfs v0.19
> 
> mike@mercury (0) pts/3 ~ $ btrfs filesystem df /home
> Data, RAID1: total=893.48GB, used=886.60GB
> Data: total=8.00MB, used=0.00
> System, RAID1: total=8.00MB, used=136.00KB
> System: total=4.00MB, used=0.00
> Metadata, RAID1: total=38.00GB, used=2.54GB
> Metadata: total=8.00MB, used=0.00
> 
> As can be seen I don''t have a lot of free space left and while I
am
> planning on adding more storage soon I would like to gain a little
> breathing room until I can do this. While I don''t have a lot of
space
> remaining in Data, RAID1 I do have a good chunk in Metadata, RAID1.
> 2.5GB used out of 38GB. Does this excess become available
> automatically to the file system when the block groups in Data, RAID1
> are exhausted or, if not, is there a way to manually reallocate them?
   Youre best bet at the moment is to try a partial balance of
metadata chunks:

# btrfs balance start -m /home

   That will rewrite all of your metadata, putting it through the
allocator again, and removing the original allocated chunks. This
should have the effect of reducing the allocation of metadata chunks.

   You will need a 3.3 kernel, or later, and an up-to-date userspace
from cmason''s git repository.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ==  PGP
key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
           --- All hope abandon,  Ye who press Enter here. ---

On Sat, May 5, 2012 at 10:52 PM, Hugo Mills <hugo@carfax.org.uk>
wrote:
> On Sat, May 05, 2012 at 10:37:01PM +1000, Mike Sampson wrote:
>> Hello list,
>>
>> recently reformatted my home partition from XFS to RAID1 btrfs. I used
>> the default options to mkfs.btrfs except for enabling raid1 for data
>> as well as metadata. Filesystem is made up of two 1TB drives.
>>
>> mike@mercury (0) pts/3 ~ $ sudo btrfs filesystem show
>> Label: none  uuid: f08a8896-e03e-4064-9b94-9342fb547e47
>>       Total devices 2 FS bytes used 888.06GB
>>       devid    1 size 931.51GB used 931.51GB path /dev/sdb1
>>       devid    2 size 931.51GB used 931.49GB path /dev/sdc1
>>
>> Btrfs Btrfs v0.19
>>
>> mike@mercury (0) pts/3 ~ $ btrfs filesystem df /home
>> Data, RAID1: total=893.48GB, used=886.60GB
>> Data: total=8.00MB, used=0.00
>> System, RAID1: total=8.00MB, used=136.00KB
>> System: total=4.00MB, used=0.00
>> Metadata, RAID1: total=38.00GB, used=2.54GB
>> Metadata: total=8.00MB, used=0.00
>>
>> As can be seen I don''t have a lot of free space left and while
I am
>> planning on adding more storage soon I would like to gain a little
>> breathing room until I can do this. While I don''t have a lot
of space
>> remaining in Data, RAID1 I do have a good chunk in Metadata, RAID1.
>> 2.5GB used out of 38GB. Does this excess become available
>> automatically to the file system when the block groups in Data, RAID1
>> are exhausted or, if not, is there a way to manually reallocate them?
>
>   Youre best bet at the moment is to try a partial balance of
> metadata chunks:
>
> # btrfs balance start -m /home
>
>   That will rewrite all of your metadata, putting it through the
> allocator again, and removing the original allocated chunks. This
> should have the effect of reducing the allocation of metadata chunks.
>
>   You will need a 3.3 kernel, or later, and an up-to-date userspace
> from cmason''s git repository.
Gave this a shot and it did help.

mike@mercury (0) pts/3 ~/.../btrfs-progs git:master $ uname -r
3.3.4-2-ARCH

mike@mercury (0) pts/3 ~/.../btrfs-progs git:master $ sudo ./btrfs
balance start -m /home

Done, had to relocate 40 out of 934 chunks

mike@mercury (0) pts/3 ~/.../btrfs-progs git:master $ ./btrfs
filesystem df /home
Data, RAID1: total=900.97GB, used=880.06GB
Data: total=8.00MB, used=0.00
System, RAID1: total=32.00MB, used=136.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=30.50GB, used=2.54GB

There is now 8GB less in Metadata and I was able to delete some files
as well to free up space. There is still a lot of wasted space in the
metadata block groups. It seems that it allocates more metadata block
groups than required for my filesystem. This will do until I am able
to add a couple of devices to the system. Is there anyway to adjust
the block group allocation strategy at filesystem creation?

Thanks Hugo,

Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Sun, May 06, 2012 at 01:07:06PM +1000, Mike Sampson
wrote:
> On Sat, May 5, 2012 at 10:52 PM, Hugo Mills <hugo@carfax.org.uk>
wrote:
> > On Sat, May 05, 2012 at 10:37:01PM +1000, Mike Sampson wrote:
> >> Hello list,
> >>
> >> recently reformatted my home partition from XFS to RAID1 btrfs. I
used
> >> the default options to mkfs.btrfs except for enabling raid1 for
data
> >> as well as metadata. Filesystem is made up of two 1TB drives.
> >>
> >> mike@mercury (0) pts/3 ~ $ sudo btrfs filesystem show
> >> Label: none  uuid: f08a8896-e03e-4064-9b94-9342fb547e47
> >>       Total devices 2 FS bytes used 888.06GB
> >>       devid    1 size 931.51GB used 931.51GB path /dev/sdb1
> >>       devid    2 size 931.51GB used 931.49GB path /dev/sdc1
> >>
> >> Btrfs Btrfs v0.19
> >>
> >> mike@mercury (0) pts/3 ~ $ btrfs filesystem df /home
> >> Data, RAID1: total=893.48GB, used=886.60GB
> >> Data: total=8.00MB, used=0.00
> >> System, RAID1: total=8.00MB, used=136.00KB
> >> System: total=4.00MB, used=0.00
> >> Metadata, RAID1: total=38.00GB, used=2.54GB
> >> Metadata: total=8.00MB, used=0.00
> >>
> >> As can be seen I don''t have a lot of free space left and
while I am
> >> planning on adding more storage soon I would like to gain a little
> >> breathing room until I can do this. While I don''t have a
lot of space
> >> remaining in Data, RAID1 I do have a good chunk in Metadata,
RAID1.
> >> 2.5GB used out of 38GB. Does this excess become available
> >> automatically to the file system when the block groups in Data,
RAID1
> >> are exhausted or, if not, is there a way to manually reallocate
them?
> >
> >   Youre best bet at the moment is to try a partial balance of
> > metadata chunks:
> >
> > # btrfs balance start -m /home
> >
> >   That will rewrite all of your metadata, putting it through the
> > allocator again, and removing the original allocated chunks. This
> > should have the effect of reducing the allocation of metadata chunks.
> >
> >   You will need a 3.3 kernel, or later, and an up-to-date userspace
> > from cmason''s git repository.
> 
> Gave this a shot and it did help.
> 
> mike@mercury (0) pts/3 ~/.../btrfs-progs git:master $ uname -r
> 3.3.4-2-ARCH
> 
> mike@mercury (0) pts/3 ~/.../btrfs-progs git:master $ sudo ./btrfs
> balance start -m /home
> 
> Done, had to relocate 40 out of 934 chunks
> 
> mike@mercury (0) pts/3 ~/.../btrfs-progs git:master $ ./btrfs
> filesystem df /home
> Data, RAID1: total=900.97GB, used=880.06GB
> Data: total=8.00MB, used=0.00
> System, RAID1: total=32.00MB, used=136.00KB
> System: total=4.00MB, used=0.00
> Metadata, RAID1: total=30.50GB, used=2.54GB
> 
> There is now 8GB less in Metadata and I was able to delete some files
> as well to free up space. There is still a lot of wasted space in the
> metadata block groups. It seems that it allocates more metadata block
> groups than required for my filesystem. This will do until I am able
> to add a couple of devices to the system. Is there anyway to adjust
> the block group allocation strategy at filesystem creation?
No.  Chunk allocator currently allocates a lot more chunks than actually
needed, and it impacts both balancing and normal operation.  Try this:

# btrfs balance start -musage=10 /home

This is suboptimal, but it should get rid of more chunks.

Thanks,

		Ilya
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Sun, May 06, 2012 at 01:26:45PM +0300, Ilya Dryomov
wrote:
> On Sun, May 06, 2012 at 01:07:06PM +1000, Mike Sampson wrote:
> > There is now 8GB less in Metadata and I was able to delete some files
> > as well to free up space. There is still a lot of wasted space in the
> > metadata block groups. It seems that it allocates more metadata block
> > groups than required for my filesystem. This will do until I am able
> > to add a couple of devices to the system. Is there anyway to adjust
> > the block group allocation strategy at filesystem creation?
> 
> No.  Chunk allocator currently allocates a lot more chunks than actually
> needed, and it impacts both balancing and normal operation.  Try this:
> 
> # btrfs balance start -musage=10 /home
> 
> This is suboptimal, but it should get rid of more chunks.
   While we''re talking about it, what is the parameter to the
"usage"
option? I''m assuming it selects chunks which are less than some amount
full -- but is the value a percentage, or a quantity in megabytes
(power-of-10 or power-of-2), or something else?

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ==  PGP
key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
               --- Doughnut furs ache me, Omar Dorlin. ---

On Sun, May 06, 2012 at 11:37:27AM +0100, Hugo Mills
wrote:
> On Sun, May 06, 2012 at 01:26:45PM +0300, Ilya Dryomov wrote:
> > On Sun, May 06, 2012 at 01:07:06PM +1000, Mike Sampson wrote:
> > > There is now 8GB less in Metadata and I was able to delete some
files
> > > as well to free up space. There is still a lot of wasted space in
the
> > > metadata block groups. It seems that it allocates more metadata
block
> > > groups than required for my filesystem. This will do until I am
able
> > > to add a couple of devices to the system. Is there anyway to
adjust
> > > the block group allocation strategy at filesystem creation?
> > 
> > No.  Chunk allocator currently allocates a lot more chunks than
actually
> > needed, and it impacts both balancing and normal operation.  Try this:
> > 
> > # btrfs balance start -musage=10 /home
> > 
> > This is suboptimal, but it should get rid of more chunks.
> 
>    While we''re talking about it, what is the parameter to the
"usage"
> option? I''m assuming it selects chunks which are less than some
amount
> full -- but is the value a percentage, or a quantity in megabytes
> (power-of-10 or power-of-2), or something else?
It''s a percentage, so the command above will balance out chunks that
are
less than 10 percent full.  I''ll update btrfs man page and the wiki
page
you started as soon as I can.

Thanks,

		Ilya
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html