Lustre devel - Dec 2006 - code drop with kerberos and uid mapping

I know that some of you are keen to try a Lustre code drop with Kerberos 
and uid mapping.  The attached message shows a drop that has this 
feature.  Please be aware that we are not supporting this yet.

Eric, could you post the public wiki page you made with instructions to 
lustre-devel?

- Peter -
-------------- next part --------------
An embedded message was scrubbed...
From: Eric Mei <ericm@clusterfs.com>
Subject: Re: cmd3 code drop for Simms
Date: Wed, 06 Dec 2006 11:43:27 -0700
Size: 1991
Url:
http://mail.clusterfs.com/pipermail/lustre-devel/attachments/20061206/2fb34d72/cmd3codedropforSimms.mht

Hi,

Peter Braam wrote:
> I know that some of you are keen to try a Lustre code drop with Kerberos 
> and uid mapping.  The attached message shows a drop that has this 
> feature.  Please be aware that we are not supporting this yet.
> 
> Eric, could you post the public wiki page you made with instructions to 
> lustre-devel?
The Kerberized Lustre is briefly described at 
https://mail.clusterfs.com/wikis/lustre/KerbLustre

This is still experimental, and only RHEL4 kernel works with it right now.

Thanks
--
Eric

On Wed, 6 Dec 2006, Eric Mei wrote:
> The Kerberized Lustre is briefly described at 
> https://mail.clusterfs.com/wikis/lustre/KerbLustre
Interesting!

Questions (keep in mind I have little knowledge of kerberos):
  - will that stuff stay in 1.8, or could it be backported to 1.6?
  - given the comments on performance overhead, I suspect that krb5i-bulkn 
is what most people will be asking for (good authentication and 
performance, at the expense of data protection): then why not make it a 
basic setting?
  - is it mandatory to deploy a keytab on clients? I don''t remember
having
done that when testing AFS+krb5
  - is it really necessary to create a principal for each server node?
  - why per OST/MDT mount options? does it make sense (at least for the 
common case) to have different options on differents targets?
  - why does a root user *not* need kinit to access the fs?


Cheers,

-- 
Jean-Marc Saffroy - jean-marc.saffroy@ext.bull.net

Hi,

Jean-Marc Saffroy wrote:
> Questions (keep in mind I have little knowledge of kerberos):
>  - will that stuff stay in 1.8, or could it be backported to 1.6?
Only since 1.8, we''v no plan of backporting to 1.6.
>  - given the comments on performance overhead, I suspect that 
> krb5i-bulkn is what most people will be asking for (good authentication 
> and performance, at the expense of data protection): then why not make 
> it a basic setting?
Probably, currently we are just not sure. But we can easily change it 
once we know which is the best. Thanks for the advice.

Speak the performance hit, it''s not seriously tested yet. The impact 
might different from system to system, the main overhead is CPU cycles 
on crypting, and a little bit more network traffic. So on systems with 
powerful CPUs or hardware encryption the situation might not too bad. 
Just a guess though.
>  - is it mandatory to deploy a keytab on clients? I don''t remember
> having done that when testing AFS+krb5
Yes it''s mandatory, this will add some burden to sysadmin. The reason
is
a little bit involved, we hope root could always own valid secure 
contexts, otherwise a failed callback RPC from server to client will 
lead to the client be kicked out off cluster.
>  - is it really necessary to create a principal for each server node?
Maybe I''m wrong: this is required by GSS/Kerberos. But it will be great
if someone confirm that per-server principal is not necessary, that way 
the configuration will be easier and hard to say less secure.
>  - why per OST/MDT mount options? does it make sense (at least for the 
> common case) to have different options on differents targets?
Maybe you are right, same as the above issue of default flavor. Before 
we are sure what kind of interface is better, we just keep the 
flexibility, which may looks ugly.

These code is still in alpha, never have feedback from outside of CFS, 
so any of your opinions are high appreciated!
>  - why does a root user *not* need kinit to access the fs?
Because the keytab installed on each client node, it act like root 
already authenticated.

In real deployment, we suppose on MDTs root users should be mapped to a 
normal user (to prevent a compromised client destroying all data) except 
it''s come from specific trusted clients.

Best Wishes
--
Eric

Hi Eric,

Thanks for your answers!

On Fri, 8 Dec 2006, Eric Mei wrote:
> Speak the performance hit, it''s not seriously tested yet. The
impact
> might different from system to system, the main overhead is CPU cycles 
> on crypting, and a little bit more network traffic. So on systems with 
> powerful CPUs or hardware encryption the situation might not too bad. 
> Just a guess though.
The CPU overhead of bulk data checksumming on servers should be quite 
high; FWIW a simple test of sha1sum gives me 154 MB/s with my fairly 
recent CPU. The additional cost (CPUs or HW crypto engines) required to 
achieve good performance may be too high for most HPC users, at least for 
regular access inside a cluster; it could be interesting to enable or 
disable checksumming for classes of clients.
>>  - is it mandatory to deploy a keytab on clients? I don''t
remember
>> having done that when testing AFS+krb5
>
> Yes it''s mandatory, this will add some burden to sysadmin. The
reason is
> a little bit involved, we hope root could always own valid secure 
> contexts, otherwise a failed callback RPC from server to client will 
> lead to the client be kicked out off cluster.
It sounds like an implementation problem that surfaces to the user. :-/ 
Maybe the operations you mention should be possible with unauthenticated 
clients?
>>  - is it really necessary to create a principal for each server node?
>
> Maybe I''m wrong: this is required by GSS/Kerberos. But it will be
great if
> someone confirm that per-server principal is not necessary, that way the 
> configuration will be easier and hard to say less secure.
If I''m not mistaken, AFS with Kerberos uses one principal for all
servers
in a cell, and a keytab is propagated to all servers. But maybe GSS adds 
specific constraints, I don''t know.


Cheers,

-- 
Jean-Marc Saffroy - jean-marc.saffroy@ext.bull.net

Jean-Marc Saffroy wrote:
> Hi Eric,
>
> Thanks for your answers!
>
> On Fri, 8 Dec 2006, Eric Mei wrote:
>
>> Speak the performance hit, it''s not seriously tested yet. The
impact
>> might different from system to system, the main overhead is CPU 
>> cycles on crypting, and a little bit more network traffic. So on 
>> systems with powerful CPUs or hardware encryption the situation might 
>> not too bad. Just a guess though.
>
> The CPU overhead of bulk data checksumming on servers should be quite 
> high; FWIW a simple test of sha1sum gives me 154 MB/s with my fairly 
> recent CPU. The additional cost (CPUs or HW crypto engines) required 
> to achieve good performance may be too high for most HPC users, at 
> least for regular access inside a cluster; it could be interesting to 
> enable or disable checksumming for classes of clients.
I''ve seen a little laptop running Solaris ZFS do checksumming at over a
1 GByte / sec.  Probably finding the right algorithms is important
here.
>
>>>  - is it mandatory to deploy a keytab on clients? I don''t
remember
>>> having done that when testing AFS+krb5
>>
>> Yes it''s mandatory, this will add some burden to sysadmin. The
reason
>> is a little bit involved, we hope root could always own valid secure 
>> contexts, otherwise a failed callback RPC from server to client will 
>> lead to the client be kicked out off cluster.
>
> It sounds like an implementation problem that surfaces to the user. 
> :-/ Maybe the operations you mention should be possible with 
> unauthenticated clients?
>
This was a choice to force the mount command to be authenticated.  If 
AFS mounts without such a file or keys obtained otherwise, then they 
allow mount without authentication.


- Peter -

On Sat, 9 Dec 2006, Peter Braam wrote:
>> The CPU overhead of bulk data checksumming on servers should be quite
high;
>> FWIW a simple test of sha1sum gives me 154 MB/s with my fairly recent
CPU.
>> The additional cost (CPUs or HW crypto engines) required to achieve
good
>> performance may be too high for most HPC users, at least for regular
access
>> inside a cluster; it could be interesting to enable or disable
checksumming
>> for classes of clients.
> I''ve seen a little laptop running Solaris ZFS do checksumming at
over a 1
> GByte / sec.  Probably finding the right algorithms is important here.
Certainly. SHA1 is a cryptographic hash, and I would not be surprised if 
the ZFS checksum you saw in use were not.

It seems ZFS has 2 checksum algorithms, SHA256 (again a crypto hash) and 
fletcher (3 versions):
   http://opensolaris.org/os/community/zfs/source/#ZIO
   http://cvs.opensolaris.org/source/xref/usr/src/uts/common/fs/zfs/fletcher.c
   http://cvs.opensolaris.org/source/xref/usr/src/uts/common/fs/zfs/sha256.c

It looks like fletcher is terribly simple, and probably not suitable for 
security purposes.
>>>>  - is it mandatory to deploy a keytab on clients? I
don''t remember having
>>>> done that when testing AFS+krb5
>>> 
>>> Yes it''s mandatory, this will add some burden to sysadmin.
The reason is a
>>> little bit involved, we hope root could always own valid secure
contexts,
>>> otherwise a failed callback RPC from server to client will lead to
the
>>> client be kicked out off cluster.
>> 
>> It sounds like an implementation problem that surfaces to the user. :-/
>> Maybe the operations you mention should be possible with
unauthenticated
>> clients?
>> 
> This was a choice to force the mount command to be authenticated.
Ok. Would it be difficult to make this a configuration option?
> If AFS mounts without such a file or keys obtained otherwise, then they 
> allow mount without authentication.
They do: many well-known sites (including MIT, CMU and the US the Naval 
Research Lab) have public AFS cells I can browse from home.


Cheers,

-- 
Jean-Marc Saffroy - jean-marc.saffroy@ext.bull.net

Jean-Marc Saffroy wrote:
>>> It sounds like an implementation problem that surfaces to the user.
>>> :-/ Maybe the operations you mention should be possible with 
>>> unauthenticated clients?
>>>
>> This was a choice to force the mount command to be authenticated.
> 
> Ok. Would it be difficult to make this a configuration option?
This is possible, we''ll think about it.

Thanks
--
Eric

On Sat, 9 Dec 2006, Jean-Marc Saffroy wrote:
>>> The CPU overhead of bulk data checksumming on servers should be
quite
>>> high; FWIW a simple test of sha1sum gives me 154 MB/s with my
fairly
>>> recent CPU. The additional cost (CPUs or HW crypto engines)
required to
>>> achieve good performance may be too high for most HPC users, at
least for
>>> regular access inside a cluster; it could be interesting to enable
or
>>> disable checksumming for classes of clients.
>> I''ve seen a little laptop running Solaris ZFS do checksumming
at over a 1
>> GByte / sec.  Probably finding the right algorithms is important here.
>
> Certainly. SHA1 is a cryptographic hash, and I would not be surprised if
the
> ZFS checksum you saw in use were not.
For the curious, I stumbled upon a performance comparison of various 
related algorithms here:
   http://www.cryptopp.com/benchmarks.html
   http://www.cryptopp.com/amd64-benchmarks.html

It seems that the Panama hash can achieve must better performance than 
SHA1, but it''s not obvious to me if it''s as safe as SHA1. And
I have no
idea if it''s usable for Lustre+Kerberos. :)


-- 
Jean-Marc Saffroy - jean-marc.saffroy@ext.bull.net

Hi,

Just noticed this old message.
> I know that some of you are keen to try a Lustre code drop with Kerberos
> and uid mapping.  The attached message shows a drop that has this feature.
> Please be aware that we are not supporting this yet.
<snip>
> Binaries:
>
ftp://ftp.clusterfs.com/priv/ericm/kernel-lustre-smp-2.6.9-42.EL_lustre_b_new_cmd.200612051940.i686.rpm
>
ftp://ftp.clusterfs.com/priv/ericm/lustre-modules-1.5.95-2.6.9_42.EL_lustre_b_new_cmd.200612051940smp_200612060405.i686.rpm
>
ftp://ftp.clusterfs.com/priv/ericm/lustre-1.5.95-2.6.9_42.EL_lustre_b_new_cmd.200612051940smp_200612060405.i686.rpm
> ftp://ftp.clusterfs.com/priv/ericm/libgssapi-0.10-1.i386.rpm
>
> Source:
>
ftp://ftp.clusterfs.com/priv/ericm/kernel-lustre-source-2.6.9-42.EL_lustre_b_new_cmd.200612051940.i686.rpm
>
ftp://ftp.clusterfs.com/priv/ericm/lustre-source-1.5.95-2.6.9_42.EL_lustre_b_new_cmd.200612051940smp_200612060405.i686.rpm
> ftp://ftp.clusterfs.com/priv/ericm/libgssapi-0.10-1.src.rpm
We have begun looking at Lustre, again, and we use Kerberos alot, so we are
very interested in this version. The mentioned server, ftp.clusterfs.com, seems
to have been phased out since then. Can I find these files somewhere else?

Regards,
Mattias Claesson
PDC/KTH

Hi,

Mattias Claesson wrote:
> Just noticed this old message.
> 
> We have begun looking at Lustre, again, and we use Kerberos alot, so we are
> very interested in this version. The mentioned server, ftp.clusterfs.com,
seems
> to have been phased out since then. Can I find these files somewhere else?
You can find updated packages at:
ftp://ftp.clusterfs.com/priv/ericm/krblustre.070110/

Including both source & i686 binary. And still a brief explanation at:
https://mail.clusterfs.com/wikis/lustre/KerbLustre

Best Regards
--
Eric