Linux xev 3.10-2-amd64 #1 SMP Debian 3.10.5-1 (2013-08-07) x86_64 GNU/Linux I''ve attached a file of script output from a system running the above Debian kernel on a system with an Intel 120G SSD. To get this working again I extended the filesystem in question to also use a small USB flash device (4G from memory) and then deleted some old snapshots to free some space. The most intereting part of the script output is pasted below. The system reports that the filesystem is 82% full but that there is no space left. root@xev:~# df -h / Filesystem Size Used Avail Use% Mounted on /dev/disk/by-uuid/586e6f48-2985-4115-9f89-f844b319c7c0 108G 87G 21G 82% / root@xev:~# btrfs filesystem df / Data: total=101.57GB, used=81.50GB System, DUP: total=8.00MB, used=20.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=3.00GB, used=2.50GB Metadata: total=8.00MB, used=0.00 root@xev:~# ls -l > test bash: test: No space left on device root@xev:~# touch test touch: cannot touch ‘test’: No space left on device root@xev:~# -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
You are out of metadata, not "normal space". However, the good question is why 0.5GB of metadata are unused and btrfs reports no space left. I have seen similar behaviour on a machine of mine, with exactly 0.5GB of metadata unused. Regards 2013/8/26 Russell Coker <russell@coker.com.au>:> Linux xev 3.10-2-amd64 #1 SMP Debian 3.10.5-1 (2013-08-07) x86_64 GNU/Linux > > I''ve attached a file of script output from a system running the above Debian > kernel on a system with an Intel 120G SSD. To get this working again I > extended the filesystem in question to also use a small USB flash device (4G > from memory) and then deleted some old snapshots to free some space. > > The most intereting part of the script output is pasted below. The system > reports that the filesystem is 82% full but that there is no space left. > > root@xev:~# df -h / > Filesystem Size Used Avail Use% > Mounted on > /dev/disk/by-uuid/586e6f48-2985-4115-9f89-f844b319c7c0 108G 87G 21G 82% > / > root@xev:~# btrfs filesystem df / > Data: total=101.57GB, used=81.50GB > System, DUP: total=8.00MB, used=20.00KB > System: total=4.00MB, used=0.00 > Metadata, DUP: total=3.00GB, used=2.50GB > Metadata: total=8.00MB, used=0.00 > root@xev:~# ls -l > test > bash: test: No space left on device > root@xev:~# touch test > touch: cannot touch ‘test’: No space left on device > root@xev:~# > > -- > My Main Blog http://etbe.coker.com.au/ > My Documents Blog http://doc.coker.com.au/-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Aug 26, 2013, at 7:35 AM, Russell Coker <russell@coker.com.au> wrote:> > root@xev:~# btrfs filesystem df / > Data: total=101.57GB, used=81.50GB > System, DUP: total=8.00MB, used=20.00KB > System: total=4.00MB, used=0.00 > Metadata, DUP: total=3.00GB, used=2.50GB > Metadata: total=8.00MB, used=0.00 > root@xev:~# ls -l > test > bash: test: No space left on device > root@xev:~# touch test > touch: cannot touch ‘test’: No space left on device > root@xev:~#The issue is data chunks remaining allocated for data, and not enough free space to allocate metadata chunks. Ultimately, the volume needs to be balanced. I''m uncertain if autodefrag avoids this problem. It does seem like in certain instances, like this, the file system needs to be able to prune itself somehow, like a partial balance to consolidate data chunks and then release their space so they can become metadata chunks. Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Aug 26, 2013, at 8:47 AM, Clemens Eisserer <linuxhippy@gmail.com> wrote:> You are out of metadata, not "normal space". > However, the good question is why 0.5GB of metadata are unused and > btrfs reports no space left. > I have seen similar behaviour on a machine of mine, with exactly 0.5GB > of metadata unused.Since metadata is set to DUP, the numbers are actually 6GB allocated and 5GB used, and I''m guessing that at the time it starts to use the last chunk of allocated space, it also wants to allocate a new one. But it can''t allocate more for metadata because all free space in the fs is already allocated for data chunks. Presently btrfs seems unable to unwind itself from this situation without a manual balance, which in many instances also requires more free space than is available once you''re in this situation. So I called it a filesystem wedged into a corner. Fortunately so far I''ve found it to be rather docile in accepting a small device to give it a little extra free space, do a balance, and then delete the device from the volume. I haven''t tried doing a partial balance, deleting the extra device, then finishing the balance. But this suggestion is certainly non-obvious to the mortal and sane. Especially if you do a df, which bases its values off the used portion of allocated chunks, the file system is distinctly not out of free space. So the confusion is understandable. Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Aug 26, 2013, at 9:27 AM, Chris Murphy <lists@colorremedies.com> wrote:> I haven''t tried doing a partial balance, deleting the extra device, then finishing the balance.Btrfs FAQ suggests a partial balance for this situation. So it''s worth a try before adding a device. https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Chris Murphy posted on Mon, 26 Aug 2013 09:15:46 -0600 as excerpted:> I''m uncertain if autodefrag avoids this problem. It does seem like in > certain instances, like this, the file system needs to be able to prune > itself somehow, like a partial balance to consolidate data chunks and > then release their space so they can become metadata chunks.Intriguing question re defrag. I hadn''t thought of the possibility until you suggested it, but indeed, tracking hundreds of extents instead of one or a dozen, does sound like it could reduce metadata usage, reducing the chance of running into the issue in the first place as well as lessening the chance of a "simple" delete temporarily requiring significant new metadata resources in ordered to track all those extent frees before the final atomic root entry update. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
How would I determine when this situation is approaching? On all important systems I have cron jobs and nagios checks to determine via df when thw file system is about to run out of space. How do I do the same for this condition? Chris Murphy <lists@colorremedies.com> wrote:> >On Aug 26, 2013, at 9:27 AM, Chris Murphy <lists@colorremedies.com> >wrote: > >> I haven''t tried doing a partial balance, deleting the extra device, >then finishing the balance. > >Btrfs FAQ suggests a partial balance for this situation. So it''s worth >a try before adding a device. > >https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space > > >Chris Murphy-- Sent from my Samsung Galaxy Note 2 with K-9 Mail. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Aug 31, 2013, at 3:04 AM, Russell Coker <russell@coker.com.au> wrote:> How would I determine when this situation is approaching? On all important systems I have cron jobs and nagios checks to determine via df when thw file system is about to run out of space. How do I do the same for this condition?I don''t think it''s possible to infer this condition from a single command. However, if you look at only ''btrfs fi show'', a small difference between size and used could be used as a trigger for a partial balance operation to avoid the condition. What''s a small difference? I''d say at least 2GB, or the size of the maximum expected file size + 2GB. That way a single write operation doesn''t end up causing all free space to be allocated for data, leaving no more free space to be allocated for metadata. Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html