Hi list, I have a 4-device RAID10 array of 2TB drives on btrfs. It works great. I recently added an additional 4 drives to the array. There is only about 2TB in use across the whole array (which should have an effective capacity of about 8TB). However I have noticed that when I issue btrfs filesystem df against the mountpoint, in the "total" field, I get the same value as the "used" field: root@mckinley:/# btrfs fi df /mnt/shares/btrfsvol0 Data, RAID10: total=2.06TB, used=2.06TB System, RAID10: total=64.00MB, used=188.00KB System: total=4.00MB, used=0.00 Metadata, RAID10: total=3.00GB, used=2.29GB Here''s my btrfs filesystem show: root@mckinley:/# btrfs fi show Label: ''btrfsvol0'' uuid: 1a735971-3ad7-4046-b25b-e834a74f2fbb Total devices 8 FS bytes used 2.06TB devid 7 size 1.82TB used 527.77GB path /dev/sdk1 devid 8 size 1.82TB used 527.77GB path /dev/sdg1 devid 6 size 1.82TB used 527.77GB path /dev/sdi1 devid 5 size 1.82TB used 527.77GB path /dev/sde1 devid 4 size 1.82TB used 527.77GB path /dev/sdj1 devid 2 size 1.82TB used 527.77GB path /dev/sdf1 devid 1 size 1.82TB used 527.77GB path /dev/sdh1 devid 3 size 1.82TB used 527.77GB path /dev/sdc1 This is running the Ubuntu build of kernel 3.9.4 and btrfs-progs from git (v0.20-rc1-324-g650e656). Am I being an idiot and missing something here? I must admit that I still find the df output a bit cryptic (entirely my failure to understand, nothing else), but on another system with only a single device the "total" field returns the capacity of the device. Cheers! ---tim -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, Jun 02, 2013 at 05:17:11PM +0100, Tim Eggleston wrote:> Hi list, > > I have a 4-device RAID10 array of 2TB drives on btrfs. It works > great. I recently added an additional 4 drives to the array. There > is only about 2TB in use across the whole array (which should have > an effective capacity of about 8TB). However I have noticed that > when I issue btrfs filesystem df against the mountpoint, in the > "total" field, I get the same value as the "used" field: > > root@mckinley:/# btrfs fi df /mnt/shares/btrfsvol0 > Data, RAID10: total=2.06TB, used=2.06TB > System, RAID10: total=64.00MB, used=188.00KB > System: total=4.00MB, used=0.00 > Metadata, RAID10: total=3.00GB, used=2.29GB > > Here''s my btrfs filesystem show: > > root@mckinley:/# btrfs fi show > Label: ''btrfsvol0'' uuid: 1a735971-3ad7-4046-b25b-e834a74f2fbb > Total devices 8 FS bytes used 2.06TB > devid 7 size 1.82TB used 527.77GB path /dev/sdk1 > devid 8 size 1.82TB used 527.77GB path /dev/sdg1 > devid 6 size 1.82TB used 527.77GB path /dev/sdi1 > devid 5 size 1.82TB used 527.77GB path /dev/sde1 > devid 4 size 1.82TB used 527.77GB path /dev/sdj1 > devid 2 size 1.82TB used 527.77GB path /dev/sdf1 > devid 1 size 1.82TB used 527.77GB path /dev/sdh1 > devid 3 size 1.82TB used 527.77GB path /dev/sdc1You have 8*527.77 GB = 4222.16 GB of raw space allocated for all purposes. Since RAID-10 takes twice the raw bytes to store data, that gives you 2111.08 GB of usable space so far. From the df output, 2.06 TB ~= 2109.44 GB is allocated as data, and all of that space is used. 3.00 GB is allocated as metadata, and most of that is used. That adds up (within rounding errors) to the 2111.08 GB above. Additional space will be allocated from the available unallocated space as the FS needs it.> This is running the Ubuntu build of kernel 3.9.4 and btrfs-progs > from git (v0.20-rc1-324-g650e656). > > Am I being an idiot and missing something here? I must admit that I > still find the df output a bit cryptic (entirely my failure to > understand, nothing else), but on another system with only a single > device the "total" field returns the capacity of the device.That''s probably already fully-allocated, so used=size in btrfs fi show. If it''s a single device, then you''re probably not using any replication, so the raw storage is equal to the possible storage. HTH, Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk == PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- I can resist everything except temptation ---
Hi Hugo, Thanks for your reply, good to know it''s not an error as such (just me being an idiot!).> Additional space will be allocated from the available unallocated > space as the FS needs it.So I guess my question becomes, how much of that available unallocated space do I have? Instinctively the btrfs df output feels like it''s missing an equivalent to the "size" column from vanilla df. Is there a method of getting this in a RAID situation? I understand that btrfs RAID is more complicated than md RAID, so it''s ok if the answer at this point is "no"... Thanks again, ---tim -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Jun 2, 2013, at 12:17 PM, Tim Eggleston <lists@timeggleston.co.uk> wrote:> > root@mckinley:/# btrfs fi df /mnt/shares/btrfsvol0 > Data, RAID10: total=2.06TB, used=2.06TB > System, RAID10: total=64.00MB, used=188.00KB > System: total=4.00MB, used=0.00 > Metadata, RAID10: total=3.00GB, used=2.29GB > > > Am I being an idiot and missing something here?No, it''s confusing. btrfs fi df doesn''t show free space. The first value is what space the fs has allocated for the data usage type, and the 2nd value is how much of that allocation is actually being used. I personally think the allocated value is useless for mortal users. I''d rather have some idea of what free space I have left, and the regular df command presents this in an annoying way also because it shows the total volume size, not accounting for the double consumption of raid1. So no matter how you slice it, it''s confusing. Chris Murphy-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, Jun 02, 2013 at 05:52:38PM +0100, Tim Eggleston wrote:> Hi Hugo, > > Thanks for your reply, good to know it''s not an error as such (just > me being an idiot!). > > >Additional space will be allocated from the available unallocated > >space as the FS needs it. > > So I guess my question becomes, how much of that available > unallocated space do I have? Instinctively the btrfs df output feels > like it''s missing an equivalent to the "size" column from vanilla > df.Look at btrfs fi show -- you have "size" and "used" there, so the difference there will give you the unallocated space.> Is there a method of getting this in a RAID situation? I understand > that btrfs RAID is more complicated than md RAID, so it''s ok if the > answer at this point is "no"...Not in any obvious (and non-surprising) way. Basically, any way you could work it out is going to give someone a surprise because they were thinking of it some other way around. The problem is that until the space is allocated, the FS can''t know how that space needs to be allocated (to data/metadata, or with what replication type and hence overheads), so we can''t necessarily give a reliable estimate. Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk == PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- If you''re not part of the solution, you''re part --- of the precipiate.
On Sun, Jun 02, 2013 at 12:52:40PM -0400, Chris Murphy wrote:> > On Jun 2, 2013, at 12:17 PM, Tim Eggleston <lists@timeggleston.co.uk> wrote: > > > > root@mckinley:/# btrfs fi df /mnt/shares/btrfsvol0 > > Data, RAID10: total=2.06TB, used=2.06TB > > System, RAID10: total=64.00MB, used=188.00KB > > System: total=4.00MB, used=0.00 > > Metadata, RAID10: total=3.00GB, used=2.29GB > > > > > > Am I being an idiot and missing something here?> No, it''s confusing. btrfs fi df doesn''t show free space. The first > value is what space the fs has allocated for the data usage type, > and the 2nd value is how much of that allocation is actually being > used. I personally think the allocated value is useless for mortal > users. I''d rather have some idea of what free space I have left, and > the regular df command presents this in an annoying way also because > it shows the total volume size, not accounting for the double > consumption of raid1. So no matter how you slice it, it''s confusing.It''s the nature of the beast, unfortunately. So far, nobody''s managed to come up with a simple method of showing free space and space usage that isn''t going to be misleading somehow. Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk == PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- If you''re not part of the solution, you''re part --- of the precipiate.
Hugo Mills posted on Sun, 02 Jun 2013 18:43:59 +0100 as excerpted:> On Sun, Jun 02, 2013 at 12:52:40PM -0400, Chris Murphy wrote: >> >> [I]t''s confusing. btrfs fi df doesn''t show free space. The first >> value is what space the fs has allocated for the data usage type, >> and the 2nd value is how much of that allocation is actually being >> used. I personally think the allocated value is useless for mortal >> users. I''d rather have some idea of what free space I have left, and >> the regular df command presents this in an annoying way also because it >> shows the total volume size, not accounting for the double consumption >> of raid1. So no matter how you slice it, it''s confusing. > > It''s the nature of the beast, unfortunately. So far, nobody''s > managed to come up with a simple method of showing free space and space > usage that isn''t going to be misleading somehow.btrfs.wiki.kernel.org covers this topic as well as I guess it''s possible to be covered at this point, in the FAQ. I definitely recommend reading the user documentation section there, to any btrfs or potential btrfs user who hasn''t done so already, as it really does cover a lot of questions, tho certainly not all (as my posting history here, after reading it, demonstrates). Home page (easiest to remember): https://btrfs.wiki.kernel.org Direct link to the documentation section on that page (perhaps more useful as a bookmark): https://btrfs.wiki.kernel.org/index.php/Main_Page#Documentation The FAQ: https://btrfs.wiki.kernel.org/index.php/FAQ Direct link to FAQ section 4.4, which starts the questions that deal with space (4.4-4.9): https://btrfs.wiki.kernel.org/index.php/ FAQ#Why_does_df_show_incorrect_free_space_for_my_RAID_volume.3F In addition, people using multiple devices should read the sysadmin guide and multiple devices pages (which can be found under the docs link above), tho they don''t really cover space questions. (But the raid-10 diagram in the sysadmin guide may be helpful in visualizing what''s going on.) In particular, see the "Whis is free space so complicated?" question/ answer, which explains the "why" of Hugo''s answer -- I don''t believe it''s yet implemented, but the plan is to allow different subvolumes, which can be created at any time, to have different raid levels. Between differing data and metadata levels and differing subvolume levels, in the general case there''s simply no reasonable way to reliably report on the unallocated space, since there''s no way to know which raid level it''ll be allocated as, until it actually happens. Of course the answer in limited specific cases can be known. Here, I''m just deploying multiple btrfs filesystems across two SSD devices, generally raid1[1] for both data/metadata, with no intention of having differing level subvolumes, so I can simply run regular df and divide the results in half in my head. btrfs filesystem df gives me different, much more technical information, so it''s useful, but not as simply useful as regular df, halving the numbers in my head. Tim (the OP)''s case is similarly knowable since he''s raid10 both data/ metadata across originally four, now eight, similarly sized 2TB devices (unlike me, he''s apparently using the same btrfs across the entire physical device, all now eight devices), assuming he never chooses anything other than raid10 data/metadata for subvolumes, and sticks with two-mirror-copy raid10 once N-way mirroring becomes possible. btrfs raid10, like its raid1, is limited to two mirror-copies, so with eight similarly-sized devices and the caveat that he has already rebalanced across all eight devices since doubled from four, he''s raid10 4-way striping, two-way-mirroring. I''d guess normal df (not btrfs filesystem df) and doing the math in his head will be the simplest for him, as it is for me. But it''s worth noting that normal df with math in your head isn''t /always/ going to be the answer, as things start getting rather more complex as soon as different sized devices get thrown into the mix, or raid1/10 on an /odd/ number of devices (tho there the math simply gets a bit more complex since it''s no longer integers), let alone the case of differing data/metadata allocation mode, without even considering the case of subvolumes having different modes, since I don''t think that''s implemented yet. But in the simple cases of data/metadata of the same raid level on either just one or an even number of devices, regular df, doing the math in your head, should be the simplest and most direct answer. As I said, btrfs filesystem df and btrfs filesystem show are useful, but for more technical purposes or in the complex cases where there''s no easy way to just do the math on normal df. --- [1] My single exception is a separate tiny /boot, one to each device, -- mixed data/metadata DUP mode, as they''re a quarter gig each. I went separate here and separately installed grub2 to each device as well, so I can independently boot from either device. My legacy/backup "spinning rust" device, all reiserfs, is bootable as well, thus giving me three separate boot devices using two different technologies, ssd vs traditional spinning rust and long stable reiserfs vs still under heavy development btrfs, in case of failure of either a physical device or the filesystem. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> I''d guess normal df (not btrfs filesystem df) and doing the math in his > head will be the simplest for him, as it is for me.> But it''s worth noting that normal df with math in your head isn''t > /always/ going to be the answer, as things start getting rather more > complex as soon as different sized devices get thrown into the mix, or > raid1/10 on an /odd/ number of devices (tho there the math simply gets > a > bit more complex since it''s no longer integers), let alone the case of > differing data/metadata allocation mode, without even considering the > case of subvolumes having different modes, since I don''t think that''s > implemented yet.I think you''ve hit the nail on the head here Duncan. You''re absolutely right that given my simple setup (even number of devices, all the same size, on raid10) it''s trivial to do the math in my head. However I don''t think this approach really scales as well as btrfs is intended to scale (up to big numbers, mixed device sizes, mixed raid levels etc etc). Obviously the implementation of a sane df output for all this stuff is non-trivial, but in the long run I don''t think expecting the user to figure it out is the best approach. I speak here as an interested sysadmin who quite enjoys rough edges... but for a lot of users the primary thing they want to know about their storage is "how much space do I have left". The documentation does do a good job of pointing out the problems and explaining why free space is a tricky concept. But does "tricky" /really/ mean "impossible", or is it just "this is a nice to have thing that we''ll figure out once the core filesystem functions are stable"? Cheers, ---tim -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Tim Eggleston posted on Mon, 03 Jun 2013 09:02:26 +0100 as excerpted:>> I''d guess normal df (not btrfs filesystem df) and doing the math in his >> head will be the simplest for him, as it is for me. > >> But it''s worth noting that normal df with math in your head isn''t >> /always/ going to be the answer, as things start getting rather more >> complex as soon as different sized devices get thrown into the mix, or> I think you''ve hit the nail on the head here Duncan. You''re absolutely > right that given my simple setup (even number of devices, all the same > size, on raid10) it''s trivial to do the math in my head. However I don''t > think this approach really scales as well as btrfs is intended to scale > (up to big numbers, mixed device sizes, mixed raid levels etc etc).Indeed.> Obviously the implementation of a sane df output for all this stuff is > non-trivial, but in the long run I don''t think expecting the user to > figure it out is the best approach. I speak here as an interested > sysadmin who quite enjoys rough edges... but for a lot of users the > primary thing they want to know about their storage is "how much space > do I have left".For a lot of sysadmins too, I''d venture. Certainly so if one uses a literal definition of sysadmin, one who administers a system and (possibly in cooperation with others in the family, etc) makes all the decisions not made at the distro level or above, as opposed to the narrower corporate definition. In many cases they take what the distro provides and know little about Linux'' workings, themselves, nor do they care, except that it "just works".> The documentation does do a good job of pointing out the problems and > explaining why free space is a tricky concept. But does "tricky" > /really/ mean "impossible", or is it just "this is a nice to have thing > that we''ll figure out once the core filesystem functions are stable"?In the traditional sense of df, I guess it really /is/ impossible, yes. at least for the as yet unallocated areas of the device, because there really is no way of predicting how that area will be used. However, I''d argue that the current btrfs fi df display can never-the- less be made far more informative than it is. Btrfs knows much more about the current allocation than it lists, and unallocated could be split out as a separate line like so: Unallocated space, future allocation unknown: xxx.xx GB And per-device and total information would be nice... I''d say the btrfs fi df output, perhaps with a --detail option, definitely needs improvement if btrfs is ever to get as widespread utilization as its presumptive position as ext* successor suggests. However, as you said, to a large extent that''s work for down the line, since for instance raid5/6 mode just got mainlined, so big features that would need serious btrfs fi df output changes to account for them if it were much more detailed are still being added. But just the addition of an unallocated space line would be an improvement, and something that could be added immediately. Similarly, at least the per-device output from btrfs fi show could be added as well, displaying the info for both commands together. That would be an improvement and could be done immediately as well. (Of course good sysadmins can easily hack up a script that presents the current information as they''d prefer to see it, and writing this makes me inclined to do just that, but that''s beyond the level of some... Altho arguably the same "some" shouldn''t be using the after all still experimental btrfs at this point anyway, but...) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html