Hi, just a question: Mounting options for file systems are usually given on the command line or in /etc/fstab. Both do not work with mobile storage devices like usb hard disks, since they are either mounted automatically through the desktop, or manually by a user who does not know or remember the mount options for this file system. At least it''s error prone. What happens if a btrfs is mounted sometimes with and sometimes without the compression option? I''d guess it generates a mixture of compressed and uncompressed files, what is not exactly smart. Is there a way to set options like compression on a btrfs permanently to activate them even when mounted automatically by the desktop or manually by a third person? regards Hadmut -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Jan 27, 2012 at 9:03 AM, Hadmut Danisch <hadmut@danisch.de> wrote:> Hi, > > just a question: > > Mounting options for file systems are usually given on the command line > or in /etc/fstab. > > Both do not work with mobile storage devices like usb hard disks, since > they are either mounted automatically through the desktop, or manually > by a user who does not know or remember the mount options for this file > system. At least it''s error prone. > > > > What happens if a btrfs is mounted sometimes with and sometimes without > the compression option? I''d guess it generates a mixture of compressed > and uncompressed files, what is not exactly smart. >It should be okay to mount with compress or without compress. Even if you mount a volume with compressed data without ''-o compress'' you will still be able to correctly read the data (but newly written data will not be compressed)> Is there a way to set options like compression on a btrfs permanently to > activate them even when mounted automatically by the desktop or manually > by a third person? > > > regards > Hadmut > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Am 28.01.2012 00:20, schrieb Chester:> It should be okay to mount with compress or without compress. Even if > you mount a volume with compressed data without ''-o compress'' you will > still be able to correctly read the data (but newly written data will > not be compressed)But having both compressed and uncompressed files in the filesystem is exactly what I want to avoid. Not because of reading problems, but to avoid wasting disk space. I don''t have a reading problem. I have a writing problem. I want to pack some data onto USB hard disks, which exceed the plain disk capacity. With btrfs it works exactly the way as I need it, except for the fact that once the compress option was missed the file system is „tainted” with uncompressed files, thus wasting disk capacity and maybe cause the files to not fit on the disk anymore. I currently don''t see how to repair this afterwards without removing the uncompressed files and writing new ones, which on the other hand spoils hte memory saving effect of using snapshots instead of copies. It would be much better if there was a way to set options like the compression option permanently when initially creating the file system, or later with the btrfs tools. regards Hadmut -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hadmut Danisch posted on Sat, 28 Jan 2012 01:49:26 +0100 as excerpted:> Am 28.01.2012 00:20, schrieb Chester: >> It should be okay to mount with compress or without compress. Even if >> you mount a volume with compressed data without ''-o compress'' you will >> still be able to correctly read the data (but newly written data will >> not be compressed) > > But having both compressed and uncompressed files in the filesystem is > exactly what I want to avoid. Not because of reading problems, but to > avoid wasting disk space. I don''t have a reading problem. I have a > writing problem.Having a compression flag in the superblock (probably more than one, thus allowing indication of compression type, etc), which is I guess what you''re suggesting, sounds useful to me too. Perhaps it''ll get it, once development settles down a bit and they know enough about what''s still missing in the existing superblock to be able to intelligently allocate flags, etc, on a priority basis based on what they then know they need. At this point I can certainly see a reluctance to make that change, tho, since it''s a rather small change to make on its own, and with continuing tweaks to the compression types and etc, it''s worth holding off committing now to what might look like a bad implementation down the line, when it has to continue to be supported.> I want to pack some data onto USB hard disks, which exceed the plain > disk capacity. With btrfs it works exactly the way as I need it, except > for the fact that once the compress option was missed the file system is > „tainted” with uncompressed files, thus wasting disk capacity and maybe > cause the files to not fit on the disk anymore.I''d be a bit leary of your whole idea of using btrfs on external drives attached to multiple systems, at this point. There''s still enough change going into btrfs that there''s a chance that mounting on a system with a newer kernel will prevent mounting on a system with an older kernel. And if you have enough control over kernel versions on all potentially mounting systems to deal with that, then pretty much by definition, you also have enough control over them to hack up a solution that, for instance, looks for a file in specific location on all newly mounted partitions (probably using udev/udisks event hooks), and if found, uses the content of said file as a cue to remount the filesystem with the options found in said file.> I currently don''t see how to repair this afterwards without removing the > uncompressed files and writing new ones, which on the other hand spoils > hte memory saving effect of using snapshots instead of copies.A rebalance is the usual way of handling such things, since that rewrites every chunk, taking account of current mount options, etc. If I''ve been reading correctly, a defrag should handle it now as well, altho that used to break snapshots/cow/reflink-copies, but I /think/ I read that with 3.2 (or was it 3.3-rc1) it doesn''t, any more.> It would be much better if there was a way to set options like the > compression option permanently when initially creating the file system, > or later with the btrfs tools.As hinted above, I''d /guess/ the chances of something like that eventually occurring are reasonably high, but there''s just too much going on still, to make superblock changes for every little new feature. Better to save them up for one final change before the caution about the on-disk format changes comes off, prioritize the changes then, and do them all at once. Then test the new format for a kernel cycle or two just to be sure, and remove the format change boilerplate the cycle after that. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 28 January 2012 14:32, Duncan <1i5t5.duncan@cox.net> wrote:> Hadmut Danisch posted on Sat, 28 Jan 2012 01:49:26 +0100 as excerpted: > >> Am 28.01.2012 00:20, schrieb Chester: >>> It should be okay to mount with compress or without compress. Even if >>> you mount a volume with compressed data without ''-o compress'' you will >>> still be able to correctly read the data (but newly written data will >>> not be compressed) >> >> But having both compressed and uncompressed files in the filesystem is >> exactly what I want to avoid. Not because of reading problems, but to >> avoid wasting disk space. I don''t have a reading problem. I have a >> writing problem. > > Having a compression flag in the superblock (probably more than one, thus > allowing indication of compression type, etc), which is I guess what > you''re suggesting, sounds useful to me too. Perhaps it''ll get it, once > development settles down a bit and they know enough about what''s still > missing in the existing superblock to be able to intelligently allocate > flags, etc, on a priority basis based on what they then know they need.mkfs -O some_feature comes to mind. I know mke2fs has that, but I haven''t yet actually used btrfs; I''m waiting for fsck to be able to repair a damaged filesystem.> At this point I can certainly see a reluctance to make that change, tho, > since it''s a rather small change to make on its own, and with continuing > tweaks to the compression types and etc, it''s worth holding off > committing now to what might look like a bad implementation down the > line, when it has to continue to be supported.This would affect not just a compression-preference option, but existing filesystems with compression. If the compression routine changes in a later kernel/filesystem revision, then you could end up with some files using one routine and others using another. Does btrfs currently have an "algorithm" specifier for compressed files? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sat, 28 Jan 2012 17:05:41 +1100 Ben Klein <shacklein@gmail.com> wrote:> If the compression routine changes in a later kernel/filesystem revision,There are already two different algorithms, zlib and lzo, and a third one - snappy - planned for inclusion.> then you could end up with some files using one routine and others using another.Which is absolutely not a problem even right now. -- With respect, Roman ~~~~~~~~~~~~~~~~~~~~~~~~~~~ "Stallman had a printer, with code he could not see. So he began to tinker, and set the software free."
On 28 January 2012 19:23, Roman Mamedov <rm@romanrm.ru> wrote:> On Sat, 28 Jan 2012 17:05:41 +1100 > Ben Klein <shacklein@gmail.com> wrote: > >> If the compression routine changes in a later kernel/filesystem revision, > > There are already two different algorithms, zlib and lzo, and a third one - > snappy - planned for inclusion. > >> then you could end up with some files using one routine and others using another. > > Which is absolutely not a problem even right now.My bad :)> -- > With respect, > Roman > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > "Stallman had a printer, > with code he could not see. > So he began to tinker, > and set the software free."-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Roman Mamedov posted on Sat, 28 Jan 2012 14:23:44 +0600 as excerpted:> On Sat, 28 Jan 2012 17:05:41 +1100 Ben Klein <shacklein@gmail.com> > wrote: > >> If the compression routine changes in a later kernel/filesystem >> revision, > > There are already two different algorithms, zlib and lzo, and a third > one - snappy - planned for inclusion. > >> then you could end up with some files using one routine and others >> using another. > > Which is absolutely not a problem even right now.Reading them isn''t a problem, sure, but the thread indicates a desire to set the write-compression characteristics. But that''s why I specifically mentioned more than just a compression flag. Presumably it''d be several bits (four bits, 16 values, would leave enough room for additional compression types in the future, tho three or even two would do it with the current three choices plus uncompressed), with one value indicating a no-compression default, and the others various choices as the default. The mount option could then be used to override the default, just as it is now, altho the only choice for default currently is no compression. Actually, given that btrfs is in line to be the successor to the ext* series, I''d say it should be close to a given that many mount options will have settable defaults recorded in the superblocks at mkfs time, and updatable using some btrfs parallel to tune2fs (tunebtrfs? tunebfs?). However, as I said earlier, committing to specifics with incremental on- disk including superblock format changes doesn''t really cut it. If a big format change is needed due to continuing development, so it can be tested and gotchas worked out, yeah, do it now, and hopefully there''s some space in the superblock(s) reserved for more trivial stuff like mount-option defaults already, but especially for mount options since they can be tested without committing to the superblock spec changes necessary to nail it down there, just testing them using the mount options is fine at this point, and the actual recording of the default choices can be nailed down all at once in one final change, just before the "be prepared for on-disk-format-changes" label comes off. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sat, Jan 28, 2012 at 7:49 AM, Hadmut Danisch <hadmut@danisch.de> wrote:> Am 28.01.2012 00:20, schrieb Chester: >> It should be okay to mount with compress or without compress. Even if >> you mount a volume with compressed data without ''-o compress'' you will >> still be able to correctly read the data (but newly written data will >> not be compressed) > > But having both compressed and uncompressed files in the filesystem is > exactly what I want to avoid. Not because of reading problems, but to > avoid wasting disk space. I don''t have a reading problem. I have a > writing problem.If you''ve been using -o compress, then you should know that even then not ALL data is compressed. If btrfs predicts that a data will be unable to benefit frmo compression, it will store it uncompressed. The problem is the prediction is not always right. Which is why there''s -o compress-force. Anyway, for removable media case, there''s a workaround that you can use (at least it works with gnome). Put the entry for the usb block device (e.g. /dev/sdb1) in fstab, with appropriate mount option, and the option will be used when you mount it using nautilus. -- Fajar -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
>> I currently don''t see how to repair this afterwards without removing the >> uncompressed files and writing new ones, which on the other hand spoils >> hte memory saving effect of using snapshots instead of copies. > > A rebalance is the usual way of handling such things, since that rewrites > every chunk, taking account of current mount options, etc. If I''ve been > reading correctly, a defrag should handle it now as well, altho that used > to break snapshots/cow/reflink-copies, but I /think/ I read that with 3.2 > (or was it 3.3-rc1) it doesn''t, any more. >Rebalance won''t do the trick, but defrag can. And defrag still breaks snapshots, which I''m working on. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hadmut Danisch wrote (ao):> I currently don''t see how to repair this afterwards without removing the > uncompressed files and writing new ones, which on the other hand spoils > hte memory saving effect of using snapshots instead of copies.As also mentioned by Li Zefan, you can use defrag. But this will indeed not work nicely with snapshots. And you need more free space than the largest file on the filesystem. find / -xdev -execdir btrfs filesystem defrag -czlib {} + Sander -- Humilis IT Services and Solutions http://www.humilis.net -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> Is there a way to set options like compression on a btrfs permanently to > activate them even when mounted automatically by the desktop or manually > by a third person?Actually there is. Btrfs supports per file compression flag, and if this flag is set to a directory, all files in that directory will inherit the flag. So you can mount the filesystem for the first time and set the flag for the root dir, and that''s it. There''s a flaw that the compression method is zlib, which should be changed to lzo (or lz4, or snappy..) in the future. chattr is used to set file flags, and there was a patchset to enable compression flag for chattr, but was never merged: http://www.spinics.net/lists/linux-btrfs/msg09604.html http://www.spinics.net/lists/linux-btrfs/msg09605.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html