Btrfs devel - Dec 2008 - Selective Compression/Encryption

Currently compression and I assume if encryption is implemented it is
turned on or off during mount. There are however many times when a user may
want to select which files/directories they want to compress or encrypt.
This will also be helpful when implementing btrfs support in grub for
example. We can say the disk can be compressed/encrypted except for /boot so 
compression/encryption doesn''t have to be implemented in grub.

I was thinking of adding this functionality to the userspace application
btrfstune. The way I was thinking of doing this is when btrfstune +c is
applied to a directory or file the directory(and all its contents) or
file will always be compressed reguardless of how the filesystem is
mounted. The opposite would happen when btrfstune -c is used.

Would this be a reasonable thing to implement? Any suggestions before I
start doing this?

Lee
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi there,

On Tue, Dec 9, 2008 at 2:59 PM, Lee Trager <lt73@cs.drexel.edu>
wrote:
> Currently compression and I assume if encryption is implemented it is
> turned on or off during mount. There are however many times when a user may
> want to select which files/directories they want to compress or encrypt.
> This will also be helpful when implementing btrfs support in grub for
> example. We can say the disk can be compressed/encrypted except for /boot
so
> compression/encryption doesn''t have to be implemented in grub.
>
> I was thinking of adding this functionality to the userspace application
> btrfstune. The way I was thinking of doing this is when btrfstune +c is
> applied to a directory or file the directory(and all its contents) or
> file will always be compressed reguardless of how the filesystem is
> mounted. The opposite would happen when btrfstune -c is used.
>
> Would this be a reasonable thing to implement? Any suggestions before I
> start doing this?
>
Things like compression or encription should be used at the "volume"
level.
So.. if a user wants a specific set of files or dirs ..they should
create a mount-point/volume like:

private_vol
bigarchives_vol

and set those volumes as compressed or encripted volumes

Regarding usability, the best would be for the sub-volume creation
tool to optionally allow passing encription/compression arguments.


and then:
 should mount those volumes somewhere like: ~/Confidential or ~/Archives.

Basically, do it at the directory level (which in btrfs is at the
sub-volume level).
File-level granularity is totally unmanageable in the long term.

Kind regards,


-- 
Miguel Sousa Filipe
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, 2008-12-09 at 09:59 -0500, Lee Trager wrote:
> Currently compression and I assume if encryption is implemented it is
> turned on or off during mount. There are however many times when a user may
> want to select which files/directories they want to compress or encrypt.
> This will also be helpful when implementing btrfs support in grub for
> example. We can say the disk can be compressed/encrypted except for /boot
so
> compression/encryption doesn''t have to be implemented in grub.
> 
> I was thinking of adding this functionality to the userspace application
> btrfstune. The way I was thinking of doing this is when btrfstune +c is
> applied to a directory or file the directory(and all its contents) or
> file will always be compressed reguardless of how the filesystem is
> mounted. The opposite would happen when btrfstune -c is used.
> 
This was my plan, but btrfstune probably isn''t the best program to do
it
(the ext2 tune program is mostly aimed at the super block level things).

I think it would be better to make a setattr style program to call the
ioctls.  There is already a per file compression flag, and the code
should already be checking it.

-chris


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Would you suggest using the existing chattr/lsattr commands from
e2fsprogs for userspace control and just add support at the kernel
level?

Miguel suggested this be done at the volume level. Do you have any
thoughts on that?

Thanks,

Lee

On Tue, Dec 09, 2008 at 11:35:16AM -0500, Chris Mason
wrote:
> On Tue, 2008-12-09 at 09:59 -0500, Lee Trager wrote:
> > Currently compression and I assume if encryption is implemented it is
> > turned on or off during mount. There are however many times when a
user may
> > want to select which files/directories they want to compress or
encrypt.
> > This will also be helpful when implementing btrfs support in grub for
> > example. We can say the disk can be compressed/encrypted except for
/boot so
> > compression/encryption doesn''t have to be implemented in
grub.
> > 
> > I was thinking of adding this functionality to the userspace
application
> > btrfstune. The way I was thinking of doing this is when btrfstune +c
is
> > applied to a directory or file the directory(and all its contents) or
> > file will always be compressed reguardless of how the filesystem is
> > mounted. The opposite would happen when btrfstune -c is used.
> > 
> 
> This was my plan, but btrfstune probably isn''t the best program to
do it
> (the ext2 tune program is mostly aimed at the super block level things).
> 
> I think it would be better to make a setattr style program to call the
> ioctls.  There is already a per file compression flag, and the code
> should already be checking it.
> 
> -chris
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tuesday 09 December 2008 08:35:16 Chris Mason wrote:
> On Tue, 2008-12-09 at 09:59 -0500, Lee Trager wrote:
> > Currently compression and I assume if encryption is implemented it is
> > turned on or off during mount. There are however many times when a
user
> > may want to select which files/directories they want to compress or
> > encrypt. This will also be helpful when implementing btrfs support in
> > grub for example. We can say the disk can be compressed/encrypted
except
> > for /boot so compression/encryption doesn''t have to be
implemented in
> > grub.
> >
> > I was thinking of adding this functionality to the userspace
application
> > btrfstune. The way I was thinking of doing this is when btrfstune +c
is
> > applied to a directory or file the directory(and all its contents) or
> > file will always be compressed reguardless of how the filesystem is
> > mounted. The opposite would happen when btrfstune -c is used.
>
> This was my plan, but btrfstune probably isn''t the best program to
do it
> (the ext2 tune program is mostly aimed at the super block level things).
>
> I think it would be better to make a setattr style program to call the
> ioctls.  There is already a per file compression flag, and the code
> should already be checking it.
Is there some reason this can''t be done with the existing extended
attribute
facilities?

It seems like xattrs would be preferable to some btrfs-specific tunable, as 
programs like rsync or backup tools would be able to preserve (and restore) 
these bits with no extra work required.

-- Josh


-- 
Joshua J. Berry

"I haven''t lost my mind -- it''s backed up on tape
somewhere."
    -- /usr/games/fortune

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Joshua J. Berry wrote:
> On Tuesday 09 December 2008 08:35:16 Chris Mason wrote:
>> On Tue, 2008-12-09 at 09:59 -0500, Lee Trager wrote:
>>> Currently compression and I assume if encryption is implemented it
is
>>> turned on or off during mount. There are however many times when a
user
>>> may want to select which files/directories they want to compress or
>>> encrypt. This will also be helpful when implementing btrfs support
in
>>> grub for example. We can say the disk can be compressed/encrypted
except
>>> for /boot so compression/encryption doesn''t have to be
implemented in
>>> grub.
>>>
>>> I was thinking of adding this functionality to the userspace
application
>>> btrfstune. The way I was thinking of doing this is when btrfstune
+c is
>>> applied to a directory or file the directory(and all its contents)
or
>>> file will always be compressed reguardless of how the filesystem is
>>> mounted. The opposite would happen when btrfstune -c is used.
>> This was my plan, but btrfstune probably isn''t the best
program to do it
>> (the ext2 tune program is mostly aimed at the super block level
things).
>>
>> I think it would be better to make a setattr style program to call the
>> ioctls.  There is already a per file compression flag, and the code
>> should already be checking it.
> 
> Is there some reason this can''t be done with the existing extended
attribute
> facilities?
> 
> It seems like xattrs would be preferable to some btrfs-specific tunable, as
> programs like rsync or backup tools would be able to preserve (and restore)
> these bits with no extra work required.
I had the same thought... that many btrfs per-file tunables would
be better implemented in xattrs (I''ve read christoph''s earlier
objections and hope to get around those issues).

I have been working on changing the xattr code with the first
step getting it functioning properly when selinux is enabled
so we can see just how costly btrfs xattrs are in actual use.

jim
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, Dec 9, 2008 at 6:09 PM, Lee Trager <lt73@cs.drexel.edu>
wrote:
> On Tue, Dec 09, 2008 at 05:22:18PM +0100, Christian Hesse wrote:
>> On Tuesday 09 December 2008, Miguel Figueiredo Mascarenhas Sousa Filipe
wrote:
>> > Hi there,
>> >
>> > On Tue, Dec 9, 2008 at 2:59 PM, Lee Trager
<lt73@cs.drexel.edu> wrote:
>> > > Currently compression and I assume if encryption is
implemented it is
>> > > turned on or off during mount. There are however many times
when a user
>> > > may want to select which files/directories they want to
compress or
>> > > encrypt. This will also be helpful when implementing btrfs
support in
>> > > grub for example. We can say the disk can be
compressed/encrypted except
>> > > for /boot so compression/encryption doesn''t have to
be implemented in
>> > > grub.
>>
>> You could just use an additional partition for /boot that has
compression an
>> encryption disabled...
>>
> Yes you could but many distributions are moving away from that such as
> ubuntu. Also while this works well on desktop and server environments
> where space is not an issue on embedded systems where free space is
> measured in kb or mb it starts becomming a huge issue.
>
>> > > I was thinking of adding this functionality to the userspace
application
>> > > btrfstune. The way I was thinking of doing this is when
btrfstune +c is
>> > > applied to a directory or file the directory(and all its
contents) or
>> > > file will always be compressed reguardless of how the
filesystem is
>> > > mounted. The opposite would happen when btrfstune -c is used.
>> > >
>> > > Would this be a reasonable thing to implement? Any
suggestions before I
>> > > start doing this?
>> >
>> > Things like compression or encription should be used at the
"volume" level.
>>
>> That was what I said some time ago when I asked why encryption support
for
>> btrfs is planned. On a volume level you can use dm-crypt and the fs can
>> ignore that part completely.
>>
>> The answer was that different users on a home partition could use their
own
>> encryption key. That sounds like volume level is out of bet. ;)
>>
> It does seem that doing it with volumes would limit user control and add
> lots of complexity to such a simple task.
>
> Miguel, what is your reason for suggesting this be done at the volume
level?
Well, basically, the main issue I have with file level configuration,
is maintenance nightmare in the long term.

Redundancy level settings (raid0,1,5,6..), and compression/encription,
to me, seem like a sub-volume setting.
Its the right granularity to manage and mantain data and interact with it.
If I want this folders compressed.. I can just move them to my
compressed sub-vol.
If I want files: a, b/weird_path/D, and NSA/* confidential, I put them
in my encripted sub-vol.

Imagine working on a confidencial project, where you add new
files/folders practilly everyday, and every single day, you must
remember to (and bother to do extra word so that..) set that file as a
encripted one.

Also, by using sub-vols like this, and if redudancy is applied at the
same level, things just play nice, I can have distinct mount-points
that represent data that I want with a given set of
redundancy/compression and encription rules.

It becomes easier to mantain, and use.

Just imagine having a "confidencial" or a "compressed"
mountpoint/disk
 ICON in you desktop, and just knowing that files in there are
encripted and/or compressed.
Versus, having to browser your file and look for a specific
"encripted" or "compressed" flag, that the file explorer
shows if they
play nice...

Now... if its at the file level, It can be troublesome to look for
allfiles encripted/compressed.
Also, knowing the encription key to each file (if it''s set at the file
level) ... argh!!


Doing this file based, seems to me that will lead to complication the
long term maintenance of things.

In the long term, when someone picks up, or mounts a old, used system,
I can''t imagine the butload of work involved in finding out things
like:
- what in this 4TB fs is compressed, what is encripted?
- (in case of redundancy at the file level): how many redundancy
levels do I have, and which files respectively?
- why is rsync/scp ''ing this folder/BIG_NESTED_TREE_OF_FILES allways
"stop" at folder/**/fileB and folder/z*/* ?
 (might it be that some unsuspecting file in there has a trully beasty
compression ratio, and that 4GB fileB (in the middle of other 4GB
files), is instead a 80Gb file compressed ?


I also don''t like extended attributes, we''re just creating a
whole new
orthogonal dimension/namespace were we store/manipulate information..
it also complicates backups (does the backup system understand and
respects ext. attrs?)

It''s messier doing these things with extended attributes or very
custom tools, instead of levelaging/creating a volume management tool,
for managing this, such tool will allready be there to manage the
fisical storage <-> filesystems/volumes mappings and settings.. might
has well leverage it.



Kind regards,


>
>> > So.. if a user wants a specific set of files or dirs ..they should
>> > create a mount-point/volume like:
>> >
>> > private_vol
>> > bigarchives_vol
>> >
>> > and set those volumes as compressed or encripted volumes
>> >
>> > Regarding usability, the best would be for the sub-volume creation
>> > tool to optionally allow passing encription/compression arguments.
>> >
>> >
>> > and then:
>> >  should mount those volumes somewhere like: ~/Confidential or
~/Archives.
>> >
>> > Basically, do it at the directory level (which in btrfs is at the
>> > sub-volume level).
>> > File-level granularity is totally unmanageable in the long term.
>> >
>> > Kind regards,
>>
>> --
>> Regards,
>> Chris
>
> Lee
>


-- 
Miguel Sousa Filipe
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

El Tue, 9 Dec 2008 13:09:51 -0500, "Lee Trager"
<lt73@cs.drexel.edu> escribió:
> It does seem that doing it with volumes would limit user control and add
> lots of complexity to such a simple task.
IMHO, WRT compression it''s the contrary. Compression on a per-file
basis has
never been very succesful (just look at how many windows users use it)
because it implies taking a decision for every file on the system. OTOH,
volume-level is just a single option to be enabled.

I''m of course not arguing that file-level compression
shouldn''t be possible,
im just saying that is way more difficult to administer and that most people
(including sysadmins) is most likely to use compression in a per-volume basis.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Diego Calleja wrote:
> El Tue, 9 Dec 2008 13:09:51 -0500, "Lee Trager"
<lt73@cs.drexel.edu> escribió:
> 
>> It does seem that doing it with volumes would limit user control and
add
>> lots of complexity to such a simple task.
> 
> IMHO, WRT compression it''s the contrary. Compression on a per-file
basis has
> never been very succesful (just look at how many windows users use it)
> because it implies taking a decision for every file on the system. OTOH,
> volume-level is just a single option to be enabled.
> 
> I''m of course not arguing that file-level compression
shouldn''t be possible,
> im just saying that is way more difficult to administer and that most
people
> (including sysadmins) is most likely to use compression in a per-volume
basis.
While I have not gotten far enough to prove it is feasible...

My idea on controlling features like compression is that
the default mode is inherited from the parent in the
directory tree.  Thus you can turn it on/off at whatever
granularity you want.

jim
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

> While I have not gotten far enough to prove it is feasible...
> 
> My idea on controlling features like compression is that
> the default mode is inherited from the parent in the
> directory tree.  Thus you can turn it on/off at whatever
> granularity you want.
> 
That seems like the ideal solution to me.  Similar to how default ACLs work.

-John
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Having a 3 possible states for each file would seem sensible:

1.  Compression Enabled - this file or folder will be compressed.
2.  Compression Disabled - this file or folder will never be compressed.
3.  Not Specified - This will inherit the compression state from it's
parent.

To keep this system in a consistent state, if a folder is changed from say state
2 to state 1, then all the children of that folder who are in state 3 need to be
compressed - potentially a very time consuming operation.

If this operation is cancelled, or fails, then the compression states in the
filesystem will be inconsistent.  No idea how it's best to solve that -
either just cope with the inconsistency and let a filesystem check tidy it up
(remember this issue would never endanger data, and only occurs in the case of a
system failure while changing the compression flags on a directory).  An
alternative solution could be to make the attribute change aschrynchronous and
have an "operation queue", so in the case of a system failure the
operation can continue where it left off.

Miguel Figueiredo Mascarenhas Sousa Filipe wrote:
> Things like compression or encription should be used at the
"volume"
> level. So.. if a user wants a specific set of files or dirs ..they should
> create a mount-point/volume like:
> 
> private_vol
> bigarchives_vol
Does a normal user have permissions to do this?

I could easily see the case where a normal user has a set of very
compressible files in a directory. They shouldn''t have to get a
sysadmin go
to the trouble of setting this up.

Jeremy


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi there,

On Tue, Dec 9, 2008 at 11:03 PM, Oliver Mattos
<oliver.mattos08@imperial.ac.uk> wrote:
> Hi,
>
> It looks to me like if all the attributes were per file, and files
> automatically inherited settings from the parent, and users
weren''t
> allowed to change inherited permissions (only root could), then you
> could get the same effect as per volume controls.
>
> The above could be enforced at the option of the sysAdmin\distro.
>
> On the other hand if the attributes are per volume then you can''t
> individually set attributes for individual files unless you create one
> volume per file, which doesn''t sound like a clean solution.
>
> As far as I can see, all the use cases that call for per-volume
> configuration are also satisfied by per file configuration with
> inheritance, but not vice versa.
>
You have convinced me.
If this works with default inheritance enabled, so that not only
child/leaf files of a dir tree get the dir setting, but also new
files, then, it fits my usage model and "wanted" features :)

>
> An example where I would want compressed and uncompressed files in the
> same folder is where I have a Virtual Machine with two disks, each
> stored in a disk image on a btrfs partition - disk 1 is the root
> filesystem and compressed to save space, since it contains a lot of
> "zero" bytes, but disk 2 is the virtual swap disk, which
isn''t
> compressed due to needing fast low overhead access.  Since both files
> are associated with the same virtual machine I want to keep them in the
> same folder on the same volume.
>
That''s a good realistic scenario were per-volume settings indeed fail
to behave "properly".

> To resolve the problem with "random" attributes being set on
random
> files, I propose a utility is created to reset all attributes on all
> files in a directory tree.  Another utility could list all non-inherited
> (ie. not the same as the parent) attributes, then you can very quickly
> see which directories are compressed\encrypted.
>
> Thanks
> Oliver
>
> On Tue, 2008-12-09 at 22:14 +0000, Miguel Figueiredo Mascarenhas Sousa
> Filipe wrote:
>> On Tue, Dec 9, 2008 at 6:09 PM, Lee Trager <lt73@cs.drexel.edu>
wrote:
>> > On Tue, Dec 09, 2008 at 05:22:18PM +0100, Christian Hesse wrote:
>> >> On Tuesday 09 December 2008, Miguel Figueiredo Mascarenhas
Sousa Filipe wrote:
>> >> > Hi there,
>> >> >
>> >> > On Tue, Dec 9, 2008 at 2:59 PM, Lee Trager
<lt73@cs.drexel.edu> wrote:
>> >> > > Currently compression and I assume if encryption is
implemented it is
>> >> > > turned on or off during mount. There are however
many times when a user
>> >> > > may want to select which files/directories they want
to compress or
>> >> > > encrypt. This will also be helpful when implementing
btrfs support in
>> >> > > grub for example. We can say the disk can be
compressed/encrypted except
>> >> > > for /boot so compression/encryption doesn''t
have to be implemented in
>> >> > > grub.
>> >>
>> >> You could just use an additional partition for /boot that has
compression an
>> >> encryption disabled...
>> >>
>> > Yes you could but many distributions are moving away from that
such as
>> > ubuntu. Also while this works well on desktop and server
environments
>> > where space is not an issue on embedded systems where free space
is
>> > measured in kb or mb it starts becomming a huge issue.
>> >
>> >> > > I was thinking of adding this functionality to the
userspace application
>> >> > > btrfstune. The way I was thinking of doing this is
when btrfstune +c is
>> >> > > applied to a directory or file the directory(and all
its contents) or
>> >> > > file will always be compressed reguardless of how
the filesystem is
>> >> > > mounted. The opposite would happen when btrfstune -c
is used.
>> >> > >
>> >> > > Would this be a reasonable thing to implement? Any
suggestions before I
>> >> > > start doing this?
>> >> >
>> >> > Things like compression or encription should be used at
the "volume" level.
>> >>
>> >> That was what I said some time ago when I asked why encryption
support for
>> >> btrfs is planned. On a volume level you can use dm-crypt and
the fs can
>> >> ignore that part completely.
>> >>
>> >> The answer was that different users on a home partition could
use their own
>> >> encryption key. That sounds like volume level is out of bet.
;)
>> >>
>> > It does seem that doing it with volumes would limit user control
and add
>> > lots of complexity to such a simple task.
>> >
>> > Miguel, what is your reason for suggesting this be done at the
volume level?
>>
>> Well, basically, the main issue I have with file level configuration,
>> is maintenance nightmare in the long term.
>>
>> Redundancy level settings (raid0,1,5,6..), and compression/encription,
>> to me, seem like a sub-volume setting.
>> Its the right granularity to manage and mantain data and interact with
it.
>> If I want this folders compressed.. I can just move them to my
>> compressed sub-vol.
>> If I want files: a, b/weird_path/D, and NSA/* confidential, I put them
>> in my encripted sub-vol.
>>
>> Imagine working on a confidencial project, where you add new
>> files/folders practilly everyday, and every single day, you must
>> remember to (and bother to do extra word so that..) set that file as a
>> encripted one.
>>
>> Also, by using sub-vols like this, and if redudancy is applied at the
>> same level, things just play nice, I can have distinct mount-points
>> that represent data that I want with a given set of
>> redundancy/compression and encription rules.
>>
>> It becomes easier to mantain, and use.
>>
>> Just imagine having a "confidencial" or a
"compressed" mountpoint/disk
>>  ICON in you desktop, and just knowing that files in there are
>> encripted and/or compressed.
>> Versus, having to browser your file and look for a specific
>> "encripted" or "compressed" flag, that the file
explorer shows if they
>> play nice...
>>
>> Now... if its at the file level, It can be troublesome to look for
>> allfiles encripted/compressed.
>> Also, knowing the encription key to each file (if it''s set at
the file
>> level) ... argh!!
>>
>>
>> Doing this file based, seems to me that will lead to complication the
>> long term maintenance of things.
>>
>> In the long term, when someone picks up, or mounts a old, used system,
>> I can''t imagine the butload of work involved in finding out
things
>> like:
>> - what in this 4TB fs is compressed, what is encripted?
>> - (in case of redundancy at the file level): how many redundancy
>> levels do I have, and which files respectively?
>> - why is rsync/scp ''ing this folder/BIG_NESTED_TREE_OF_FILES
allways
>> "stop" at folder/**/fileB and folder/z*/* ?
>>  (might it be that some unsuspecting file in there has a trully beasty
>> compression ratio, and that 4GB fileB (in the middle of other 4GB
>> files), is instead a 80Gb file compressed ?
>>
>>
>> I also don''t like extended attributes, we''re just
creating a whole new
>> orthogonal dimension/namespace were we store/manipulate information..
>> it also complicates backups (does the backup system understand and
>> respects ext. attrs?)
>>
>> It''s messier doing these things with extended attributes or
very
>> custom tools, instead of levelaging/creating a volume management tool,
>> for managing this, such tool will allready be there to manage the
>> fisical storage <-> filesystems/volumes mappings and settings..
might
>> has well leverage it.
>>
>>
>>
>> Kind regards,
>>
>>
>>
>> >
>> >> > So.. if a user wants a specific set of files or dirs
..they should
>> >> > create a mount-point/volume like:
>> >> >
>> >> > private_vol
>> >> > bigarchives_vol
>> >> >
>> >> > and set those volumes as compressed or encripted volumes
>> >> >
>> >> > Regarding usability, the best would be for the sub-volume
creation
>> >> > tool to optionally allow passing encription/compression
arguments.
>> >> >
>> >> >
>> >> > and then:
>> >> >  should mount those volumes somewhere like:
~/Confidential or ~/Archives.
>> >> >
>> >> > Basically, do it at the directory level (which in btrfs
is at the
>> >> > sub-volume level).
>> >> > File-level granularity is totally unmanageable in the
long term.
>> >> >
>> >> > Kind regards,
>> >>
>> >> --
>> >> Regards,
>> >> Chris
>> >
>> > Lee
>> >
>>
>>
>>
>
>


-- 
Miguel Sousa Filipe
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Wed, Dec 10, 2008 at 9:32 AM, Jeremy Sanders
<jeremy@jeremysanders.net> wrote:
> Miguel Figueiredo Mascarenhas Sousa Filipe wrote:
>
>> Things like compression or encription should be used at the
"volume"
>> level. So.. if a user wants a specific set of files or dirs ..they
should
>> create a mount-point/volume like:
>>
>> private_vol
>> bigarchives_vol
>
> Does a normal user have permissions to do this?
>
> I could easily see the case where a normal user has a set of very
> compressible files in a directory. They shouldn''t have to get a
sysadmin go
> to the trouble of setting this up.
>
that''s a policy decision.
Distro''s will probably allow a admin/wheel user to do such things.

Big deployments, shell servers, or university setups, will all have a
policy decision and usualy hack up costumised tools for managing users
quota, settings, passwds and things like that.

A costumised useradd would probably create such volumes for each user,
for instance, and a costumised passwd, would reencript stored data
with new passwd. (or something similar...)...
I know I did similar things when was a university sysadm, pam_useradd
and pam_mountd were heavily used and costumised.

Kind regards,


-- 
Miguel Sousa Filipe
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, 2008-12-09 at 18:50 -0500, jim owens wrote:
> Diego Calleja wrote:
> > El Tue, 9 Dec 2008 13:09:51 -0500, "Lee Trager"
<lt73@cs.drexel.edu> escribió:
> > 
> >> It does seem that doing it with volumes would limit user control
and add
> >> lots of complexity to such a simple task.
> > 
> > IMHO, WRT compression it''s the contrary. Compression on a
per-file basis has
> > never been very succesful (just look at how many windows users use it)
> > because it implies taking a decision for every file on the system.
OTOH,
> > volume-level is just a single option to be enabled.
> > 
> > I''m of course not arguing that file-level compression
shouldn''t be possible,
> > im just saying that is way more difficult to administer and that most
people
> > (including sysadmins) is most likely to use compression in a
per-volume basis.
> 
> While I have not gotten far enough to prove it is feasible...
> 
> My idea on controlling features like compression is that
> the default mode is inherited from the parent in the
> directory tree.  Thus you can turn it on/off at whatever
> granularity you want.
> 
I had planned to make the bits inheritable from the directory inode
flags.  There are two different discussions around xattrs for this.  One
is using xattrs to store the flag, which I''d would rather avoid because
it is checked in some performance critical places.

The second is using xattr programs to set the flag, which I don''t
really
have an opinion on.  The idea of having the flags backed up by backup
programs or rsync is really nice, but do any of the backup programs
actually copy out all the xattrs?

-chris


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, 2008-12-09 at 15:22 -0500, jim owens wrote:
> Joshua J. Berry wrote:
[ storing compression flags ]
> > 
> > It seems like xattrs would be preferable to some btrfs-specific
tunable, as
> > programs like rsync or backup tools would be able to preserve (and
restore)
> > these bits with no extra work required.
> 
> I had the same thought... that many btrfs per-file tunables would
> be better implemented in xattrs (I''ve read christoph''s
earlier
> objections and hope to get around those issues).
> 
> I have been working on changing the xattr code with the first
> step getting it functioning properly when selinux is enabled
> so we can see just how costly btrfs xattrs are in actual use.
Not really on topic, but how are things broken today with selinux?

-chris


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Chris Mason wrote:
> On Tue, 2008-12-09 at 15:22 -0500, jim owens wrote:
>> I have been working on changing the xattr code with the first
>> step getting it functioning properly when selinux is enabled
>> so we can see just how costly btrfs xattrs are in actual use.
> 
> Not really on topic, but how are things broken today with selinux?
With selinux enabled you can not create any files on
a btrfs filesystem (as of dec9 git tree with fedora 9),
even as root!

There are 2 things needed to make it work:

1) the /etc/selinux load-into-kernel database must be
    patched to recognize btrfs has xattrs. One of our
    security people, Paul Moore, has submitted it to
    the upstream refpolicy.  But it won''t be merged
    until I finish my testing.

After the database is patched, the dec9 git tree
will allow file create on btrfs... but the selinux
xattrs are not set.  Thus "cp -a" will copy the
files but all "selinux context" values are wrong.

2) I have btrfs patches to interface correctly with
    the LSM so we save the selinux context. I''ll be
    sending them up as soon as I have finished testing.

jim

P.S. sane people just disable selinux on install :)
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Wed, Dec 10, 2008 at 08:44:47AM -0500, Chris Mason
wrote:
> I had planned to make the bits inheritable from the directory inode
> flags.  There are two different discussions around xattrs for this.  One
> is using xattrs to store the flag, which I''d would rather avoid
because
> it is checked in some performance critical places.
> 
> The second is using xattr programs to set the flag, which I don''t
really
> have an opinion on.  The idea of having the flags backed up by backup
> programs or rsync is really nice, but do any of the backup programs
> actually copy out all the xattrs?
xfsdump does :)  But I think especially the compressed bit is much
better off in the bit for the set/get flags ioctls used by chattr /
lsattr.  These are implemented by all Linux filesystems, and even have
a compressed bit allocated already.  (and xfsdump of course backs them
up, too ;-))  *run*

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Christoph Hellwig wrote:
> On Wed, Dec 10, 2008 at 08:44:47AM -0500, Chris Mason wrote:
>> I had planned to make the bits inheritable from the directory inode
>> flags.  There are two different discussions around xattrs for this. 
One
>> is using xattrs to store the flag, which I''d would rather
avoid because
>> it is checked in some performance critical places.
>>
>> The second is using xattr programs to set the flag, which I
don''t really
>> have an opinion on.  The idea of having the flags backed up by backup
>> programs or rsync is really nice, but do any of the backup programs
>> actually copy out all the xattrs?
> 
> xfsdump does :)  But I think especially the compressed bit is much
> better off in the bit for the set/get flags ioctls used by chattr /
> lsattr.  These are implemented by all Linux filesystems, and even have
> a compressed bit allocated already.  (and xfsdump of course backs them
> up, too ;-))  *run*
The ''star'' command has an -xattr option.

-- ljk


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Wednesday 10 December 2008 05:44:47 Chris Mason wrote:
[snip]
> The second is using xattr programs to set the flag, which I don''t
really
> have an opinion on.  The idea of having the flags backed up by backup
> programs or rsync is really nice, but do any of the backup programs
> actually copy out all the xattrs?
rsync has the -X option.  rdiff-backup autodetects whether xattrs are 
supported.

Those are the programs I use; dunno about others.

-- Josh


-- 
Joshua J. Berry

"I haven''t lost my mind -- it''s backed up on tape
somewhere."
    -- /usr/games/fortune

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Tue, Dec 09, 2008 at 03:22:29PM -0500, jim owens
wrote:
> Joshua J. Berry wrote:
>> On Tuesday 09 December 2008 08:35:16 Chris Mason wrote:
>>> On Tue, 2008-12-09 at 09:59 -0500, Lee Trager wrote:
>>>> Currently compression and I assume if encryption is implemented
it is
>>>> turned on or off during mount. There are however many times
when a user
>>>> may want to select which files/directories they want to
compress or
>>>> encrypt. This will also be helpful when implementing btrfs
support in
>>>> grub for example. We can say the disk can be
compressed/encrypted except
>>>> for /boot so compression/encryption doesn''t have to be
implemented in
>>>> grub.
>>>>
>>>> I was thinking of adding this functionality to the userspace
application
>>>> btrfstune. The way I was thinking of doing this is when
btrfstune +c is
>>>> applied to a directory or file the directory(and all its
contents) or
>>>> file will always be compressed reguardless of how the
filesystem is
>>>> mounted. The opposite would happen when btrfstune -c is used.
>>> This was my plan, but btrfstune probably isn''t the best
program to do it
>>> (the ext2 tune program is mostly aimed at the super block level
things).
>>>
>>> I think it would be better to make a setattr style program to call
the
>>> ioctls.  There is already a per file compression flag, and the code
>>> should already be checking it.
>>
>> Is there some reason this can''t be done with the existing
extended
>> attribute facilities?
>>
>> It seems like xattrs would be preferable to some btrfs-specific 
>> tunable, as programs like rsync or backup tools would be able to 
>> preserve (and restore) these bits with no extra work required.
>
> I had the same thought... that many btrfs per-file tunables would
> be better implemented in xattrs (I''ve read christoph''s
earlier
> objections and hope to get around those issues).
I agree that implementing it in the xattr would be best but what
userspace tools should be used to control it? lsattr/chattr or our own?
lsattr/chattr does seem to do what we want for compression but it really
has nothing for encryption. Should we also support attribute(a), no
modification(i), etc support in xattrs?
>
> I have been working on changing the xattr code with the first
> step getting it functioning properly when selinux is enabled
> so we can see just how costly btrfs xattrs are in actual use.
Could you send me to xattr code you are working on or will you be
submitting patches soon?

Thanks,

Lee
>
> jim
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Thu, 2008-12-11 at 17:03 -0500, Lee Trager wrote:
> On Tue, Dec 09, 2008 at 03:22:29PM -0500, jim owens wrote:
> > Joshua J. Berry wrote:
> >> On Tuesday 09 December 2008 08:35:16 Chris Mason wrote:
> >>> On Tue, 2008-12-09 at 09:59 -0500, Lee Trager wrote:
> >>>> Currently compression and I assume if encryption is
implemented it is
> >>>> turned on or off during mount. There are however many
times when a user
> >>>> may want to select which files/directories they want to
compress or
> >>>> encrypt. This will also be helpful when implementing btrfs
support in
> >>>> grub for example. We can say the disk can be
compressed/encrypted except
> >>>> for /boot so compression/encryption doesn''t have
to be implemented in
> >>>> grub.
> >>>>
> >>>> I was thinking of adding this functionality to the
userspace application
> >>>> btrfstune. The way I was thinking of doing this is when
btrfstune +c is
> >>>> applied to a directory or file the directory(and all its
contents) or
> >>>> file will always be compressed reguardless of how the
filesystem is
> >>>> mounted. The opposite would happen when btrfstune -c is
used.
> >>> This was my plan, but btrfstune probably isn''t the
best program to do it
> >>> (the ext2 tune program is mostly aimed at the super block
level things).
> >>>
> >>> I think it would be better to make a setattr style program to
call the
> >>> ioctls.  There is already a per file compression flag, and the
code
> >>> should already be checking it.
> >>
> >> Is there some reason this can''t be done with the existing
extended
> >> attribute facilities?
> >>
> >> It seems like xattrs would be preferable to some btrfs-specific 
> >> tunable, as programs like rsync or backup tools would be able to 
> >> preserve (and restore) these bits with no extra work required.
> >
> > I had the same thought... that many btrfs per-file tunables would
> > be better implemented in xattrs (I''ve read
christoph''s earlier
> > objections and hope to get around those issues).
> I agree that implementing it in the xattr would be best but what
> userspace tools should be used to control it? lsattr/chattr or our own?
> lsattr/chattr does seem to do what we want for compression but it really
> has nothing for encryption. Should we also support attribute(a), no
> modification(i), etc support in xattrs?
Just to clarify, I''m fine with xattrs as a user interface to set the
per-inode flags.  The flag for this should be stored in the inode
though, and not as a btrfs xattr.  There is already a flag field in the
inode for this use (see NODATASUM and NODATACOW)

-chris


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html