Btrfs devel - Aug 2013 - Ability to free space on a full btrfs filesystem

Hello,

I''m currently experiencing some space issue on btrfs with linux kernel
3.10.
My btrfs partition is full and I fail to remove any data to give some 
free space:

athena:~# rm -f /mnt/storage/current/backup/slash/var/log/syslog
rm: cannot remove `/mnt/storage/current/backup/slash/var/log/syslog'':
No
space left on device

athena:~# true > /mnt/storage/current/backup/slash/var/log/syslog
bash: /mnt/storage/current/backup/slash/var/log/syslog: No space left on 
device

I tried to follow the wiki page 
https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space
but the balance operation fails due to lack of space:
athena:~# btrfs fi balance start -dusage=5 /mnt/storage/
ERROR: error during balancing ''/mnt/storage/'' - No space left
on device

[ Kernel logs ] btrfs: 138 enospc errors during balance

I guess this scenario is not expected, or I am probably missing something.
Is there any way I can free some space and get back a writable filesystem ?

Some more info:
Mount options used for the recovery (from 
https://btrfs.wiki.kernel.org/index.php/Ubuntu_support#ENOSPC.2C_no_space_left_on_device)
/dev/sdb1 on /mnt/storage type btrfs 
(rw,relatime,nodatasum,nodatacow,nospace_cache,clear_cache)

athena:~# btrfs fi show
Label: none  uuid: 8fa5ecc2-fff6-4ad1-a6bb-e384ea2086cb
         Total devices 1 FS bytes used 475.19GB
         devid    1 size 619.33GB used 619.33GB path /dev/sdb1

athena:~# btrfs fi df /mnt/storage/
Data: total=613.30GB, used=472.69GB
System, DUP: total=8.00MB, used=72.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=3.00GB, used=2.50GB
Metadata: total=8.00MB, used=0.00

Thanks in advance
-- 
Matthieu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Sun, Aug 25, 2013 at 06:23:20PM +0200, Matthieu Dalstein
wrote:
> Hello,
> 
> I''m currently experiencing some space issue on btrfs with linux
kernel 3.10.
> My btrfs partition is full and I fail to remove any data to give some 
> free space:
Did you snapshot your filesystem? If so, delete some/all snapshots first.
Then, note that brtfs frees space after you deleted snapshots, it can take
one or several minutes before the space ends up being garbage collected.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" -
A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Aug 25, 2013, at 10:23 AM, Matthieu Dalstein
<matthieu.dalstein@dalmat.net> wrote:
> athena:~# btrfs fi df /mnt/storage/
> Data: total=613.30GB, used=472.69GB
So it seems the issue is that data chunks once allocated aren''t
deallocated, and
> Metadata, DUP: total=3.00GB, used=2.50GB
the file system is thus unable to allocate another metadata chunk. So
it''s stuck. I don''t know if removing a snapshot will
necessarily help but it may be worth a try and see if these numbers change
enough to allow file deletion.

Another way out of this that''s worked for some people in the past is to
btrfs device add another disk to the volume. Maybe just 1GB of USB stick space
is enough so that it can allocate another metadata chunk. But I''d give
it as much of the fastest storage you have: can be any disk partition, or LV.
But I''ve also done it with a USB stick partition.

After it''s added, see if you can do a balance, which should free most
of those 141GB of data chunks. And then you can btrfs device delete the recently
added device, which will move whatever data/metadata chunks are on it, back to
the primary device for the volume.


Chris Murphy--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

> I don''t know if removing a snapshot will necessarily help but it
may be worth a try and see if these numbers change enough to allow file
deletion.
Actually the snapshots could not be removed due to the unsufficient 
remaining space.

> Another way out of this that''s worked for some people in the past
is to btrfs device add another disk to the volume. Maybe just 1GB of USB stick
space is enough so that it can allocate another metadata chunk. But I''d
give it as much of the fastest storage you have: can be any disk partition, or
LV. But I''ve also done it with a USB stick partition.
>
> After it''s added, see if you can do a balance, which should free
most of those 141GB of data chunks. And then you can btrfs device delete the
recently added device, which will move whatever data/metadata chunks are on it,
back to the primary device for the volume.
Thanks for the tip. I would have expected another less intrusive 
recovery but this one worked well. I did not fully rebalance the fs 
(could have lasted days!) but with the d/m usage balance parameters I 
managed to get back to a good state.

I guess this means I should find ways to avoid full fs unless there is a 
plan to recover gracefully from such situations.

Thanks
-- 
Matthieu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Aug 26, 2013, at 3:43 PM, Matthieu Dalstein
<matthieu.dalstein@dalmat.net> wrote:
> Thanks for the tip. I would have expected another less intrusive recovery
but this one worked well. I did not fully rebalance the fs (could have lasted
days!) but with the d/m usage balance parameters I managed to get back to a good
state.
> 
> I guess this means I should find ways to avoid full fs unless there is a
plan to recover gracefully from such situations.
The FAQ describes doing a partial balance to fix this problem as a first step. I
think it''s a good first step for this problem, rather than full
balance, or adding a device. Next step would be add a device and see if a
partial balance works, then delete that device and carry on (with normal usage
or resuming a full balance).

In my few cases I didn''t try partial balances, rather full ones and
they almost immediately failed which is how I decided to add a USB flash stick
to the volume.


Chris Murphy--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html