Eric,> Here is the user interface change according to previous discussion, > please review: > > - The security flavor of MGS connection is determined by each node, not > controllable by MGS.Is this an unavoidable fact of life or a design decision? See below "XXX"> - By default there''s no protection.See below "XXX"> > - Given the GSS/Kerberos env is ready, mount option "mgssec=flavor" > could be supplied. Pre-configured machine credential will be used, so no > need to supply password or whatsoever. > > - For MDT/OST, the option "mgssec=flavor" could also be written on disk, > like other parameters, but will be override if mount option supplied. > > - The flavor of MGS connection won''t change until umount, no matter how > rest of connection flavors change at runtime.> - MGC->MGS connection is one per node, so only one flavor could be used. > For example, suppose 2 OSTs live in a single node, we do: > # mount -t lustre -o mgssec=krb5p /dev/sda1 /mnt/ost1 > # mount -t lustre -o mgssec=null /dev/sda1 /mnt/ost2 > then only ''mgssec=krb5p'' will take effect, the second ''mgssec=null'' will > be ignored.I don''t think it''s acceptable to allow a previous mount to compromise the security of a later mount. XXX This raises the interesting question of whether servers (MGS included) can demand a minimim level of security from clients connecting to them. Is this normally part of configuring security on a given node (e.g. to set the machine credentials you mentioned above)?> Are these (especially the last one) reasonable? Thanks. > > -- > Eric >
On Jun 4, 2008, at 12:47 PM, Eric Barton wrote:> Eric, > >> Here is the user interface change according to previous discussion, >> please review: >> >> - The security flavor of MGS connection is determined by each >> node, not >> controllable by MGS. > > Is this an unavoidable fact of life or a design decision? See > below "XXX" > >> - By default there''s no protection. > > See below "XXX" > >> >> - Given the GSS/Kerberos env is ready, mount option "mgssec=flavor" >> could be supplied. Pre-configured machine credential will be used, >> so no >> need to supply password or whatsoever. >> >> - For MDT/OST, the option "mgssec=flavor" could also be written on >> disk, >> like other parameters, but will be override if mount option supplied. >> >> - The flavor of MGS connection won''t change until umount, no >> matter how >> rest of connection flavors change at runtime. > >> - MGC->MGS connection is one per node, so only one flavor could be >> used. >> For example, suppose 2 OSTs live in a single node, we do: >> # mount -t lustre -o mgssec=krb5p /dev/sda1 /mnt/ost1 >> # mount -t lustre -o mgssec=null /dev/sda1 /mnt/ost2 >> then only ''mgssec=krb5p'' will take effect, the second >> ''mgssec=null'' will >> be ignored. > > I don''t think it''s acceptable to allow a previous mount to compromise > the security of a later mount. > > XXX > > This raises the interesting question of whether servers (MGS > included) can > demand a minimim level of security from clients connecting to > them. Is this > normally part of configuring security on a given node (e.g. to set the > machine credentials you mentioned above)?As an unsolicited opinion... Conceptually, the MGS does have a choice about security configuration. It can refused to handle incoming requests if they don''t meet a certain level of GSS security. Given my understanding of the types of messages the MGS receives and the importance of those messages in the larger context of Lustre services, it would seem to be an imperative that all MGS contact be krb5p if Kerberos has been specified at all. Given this assumption, there doesn''t seem to be a need for mount time options. The client can attempt to obtain kerberos machine creds and if available, they will be used for interaction with the MGS. If the client receives a GSS error, it can attempt non-GSS/non-kerb. If machine credentials exist, the client goes without and hopefully that is what the admin chose. Note that keytab distribution can be cumbersome for large installations but if the choice is security then there will be some burden. Spencer
Eric Barton wrote:>> Here is the user interface change according to previous discussion, >> please review: >> >> - The security flavor of MGS connection is determined by each node, not >> controllable by MGS. > > Is this an unavoidable fact of life or a design decision?I think it''s not completely unavoidable. For example, MGC can do initial connect without protect, and tell MGS what kind of security mode it support, and MGS replay with its decision, and MGC reconnect with choosed flavor. This way will be much more complicated. And more importantly, what if someone hijack the initial non-protected connection? Things seem not getting any better...>> - By default there''s no protection. > > See below "XXX" > >> - Given the GSS/Kerberos env is ready, mount option "mgssec=flavor" >> could be supplied. Pre-configured machine credential will be used, so no >> need to supply password or whatsoever. >> >> - For MDT/OST, the option "mgssec=flavor" could also be written on disk, >> like other parameters, but will be override if mount option supplied. >> >> - The flavor of MGS connection won''t change until umount, no matter how >> rest of connection flavors change at runtime. > >> - MGC->MGS connection is one per node, so only one flavor could be used. >> For example, suppose 2 OSTs live in a single node, we do: >> # mount -t lustre -o mgssec=krb5p /dev/sda1 /mnt/ost1 >> # mount -t lustre -o mgssec=null /dev/sda1 /mnt/ost2 >> then only ''mgssec=krb5p'' will take effect, the second ''mgssec=null'' will >> be ignored. > > I don''t think it''s acceptable to allow a previous mount to compromise > the security of a later mount.Indeed it looks not so good. But the fact of per-node shared MGS connection means only one flavor could be used. To avoid the confusion, to me the only way is don''t allow the choice via mount option, instead to choose a "proper" one automatically somehow.> XXX > > This raises the interesting question of whether servers (MGS included) can > demand a minimim level of security from clients connecting to them. Is this > normally part of configuring security on a given node (e.g. to set the > machine credentials you mentioned above)?This is the root problem I guess: we can''t assume there''s security environment ready on each nodes. The procedure of setup gss/kerberos is not extremely easy: configure KDC, installing keytabs, configure gssapi, keyring, etc. And for most Luster clusters, strong security are not needed at all, so people most likely choose to skip that. -- Eric
We clearly need to make sure that servers can enforce a level of security for clients - even if it is ugly to do so. Peter On 6/4/08 11:38 AM, "Eric Mei" <Eric.Mei at Sun.COM> wrote:> Eric Barton wrote: >>> Here is the user interface change according to previous discussion, >>> please review: >>> >>> - The security flavor of MGS connection is determined by each node, not >>> controllable by MGS. >> >> Is this an unavoidable fact of life or a design decision? > > I think it''s not completely unavoidable. For example, MGC can do initial > connect without protect, and tell MGS what kind of security mode it > support, and MGS replay with its decision, and MGC reconnect with > choosed flavor. > > This way will be much more complicated. And more importantly, what if > someone hijack the initial non-protected connection? Things seem not > getting any better... > >>> - By default there''s no protection. >> >> See below "XXX" >> >>> - Given the GSS/Kerberos env is ready, mount option "mgssec=flavor" >>> could be supplied. Pre-configured machine credential will be used, so no >>> need to supply password or whatsoever. >>> >>> - For MDT/OST, the option "mgssec=flavor" could also be written on disk, >>> like other parameters, but will be override if mount option supplied. >>> >>> - The flavor of MGS connection won''t change until umount, no matter how >>> rest of connection flavors change at runtime. >> >>> - MGC->MGS connection is one per node, so only one flavor could be used. >>> For example, suppose 2 OSTs live in a single node, we do: >>> # mount -t lustre -o mgssec=krb5p /dev/sda1 /mnt/ost1 >>> # mount -t lustre -o mgssec=null /dev/sda1 /mnt/ost2 >>> then only ''mgssec=krb5p'' will take effect, the second ''mgssec=null'' will >>> be ignored. >> >> I don''t think it''s acceptable to allow a previous mount to compromise >> the security of a later mount. > > Indeed it looks not so good. But the fact of per-node shared MGS > connection means only one flavor could be used. To avoid the confusion, > to me the only way is don''t allow the choice via mount option, instead > to choose a "proper" one automatically somehow. > >> XXX >> >> This raises the interesting question of whether servers (MGS included) can >> demand a minimim level of security from clients connecting to them. Is this >> normally part of configuring security on a given node (e.g. to set the >> machine credentials you mentioned above)? > > This is the root problem I guess: we can''t assume there''s security > environment ready on each nodes. > > The procedure of setup gss/kerberos is not extremely easy: configure > KDC, installing keytabs, configure gssapi, keyring, etc. And for most > Luster clusters, strong security are not needed at all, so people most > likely choose to skip that.
Spencer Shepler wrote:> > On Jun 4, 2008, at 12:47 PM, Eric Barton wrote: > >> Eric, >> >>> Here is the user interface change according to previous discussion, >>> please review: >>> >>> - The security flavor of MGS connection is determined by each node, not >>> controllable by MGS. >> >> Is this an unavoidable fact of life or a design decision? See below >> "XXX" >> >>> - By default there''s no protection. >> >> See below "XXX" >> >>> >>> - Given the GSS/Kerberos env is ready, mount option "mgssec=flavor" >>> could be supplied. Pre-configured machine credential will be used, so no >>> need to supply password or whatsoever. >>> >>> - For MDT/OST, the option "mgssec=flavor" could also be written on disk, >>> like other parameters, but will be override if mount option supplied. >>> >>> - The flavor of MGS connection won''t change until umount, no matter how >>> rest of connection flavors change at runtime. >> >>> - MGC->MGS connection is one per node, so only one flavor could be used. >>> For example, suppose 2 OSTs live in a single node, we do: >>> # mount -t lustre -o mgssec=krb5p /dev/sda1 /mnt/ost1 >>> # mount -t lustre -o mgssec=null /dev/sda1 /mnt/ost2 >>> then only ''mgssec=krb5p'' will take effect, the second ''mgssec=null'' will >>> be ignored. >> >> I don''t think it''s acceptable to allow a previous mount to compromise >> the security of a later mount. >> >> XXX >> >> This raises the interesting question of whether servers (MGS included) >> can >> demand a minimim level of security from clients connecting to them. >> Is this >> normally part of configuring security on a given node (e.g. to set the >> machine credentials you mentioned above)? > > As an unsolicited opinion... > > Conceptually, the MGS does have a choice about security configuration. > It can refused to handle incoming requests if they don''t meet a certain > level of GSS security. Given my understanding of the types of messages > the MGS receives and the importance of those messages in the larger > context of Lustre services, it would seem to be an imperative that > all MGS contact be krb5p if Kerberos has been specified at all. > > Given this assumption, there doesn''t seem to be a need for mount time > options. The client can attempt to obtain kerberos machine creds > and if available, they will be used for interaction with the MGS. > If the client receives a GSS error, it can attempt non-GSS/non-kerb. > If machine credentials exist, the client goes without and hopefully > that is what the admin chose.Maybe there''s practical issues (not 100% sure though): If the env (especially keyring) is not correctly set, kernel may not be notified the exact reason of failure, it could be env problem on client node, transient network partition, server failure, or server rejection.> Note that keytab distribution can be cumbersome for large installations > but if the choice is security then there will be some burden.Yes. And we probably will have tools to ease the procedure a little bit. -- Eric
Comments inline> -----Original Message----- > From: Eric.Mei at Sun.COM [mailto:Eric.Mei at Sun.COM] > Sent: 04 June 2008 7:38 PM > To: Eric Barton > Cc: ''Lustre Development Mailing List''; ''Peter Braam'' > Subject: Re: [Lustre-devel] security: MGS connection > > Eric Barton wrote: > >> Here is the user interface change according to previous discussion, > >> please review: > >> > >> - The security flavor of MGS connection is determined by each node, not > >> controllable by MGS. > > > > Is this an unavoidable fact of life or a design decision? > > I think it''s not completely unavoidable. For example, MGC can do initial > connect without protect, and tell MGS what kind of security mode it > support, and MGS replay with its decision, and MGC reconnect with > choosed flavor. > > This way will be much more complicated. And more importantly, what if > someone hijack the initial non-protected connection? Things seem not > getting any better...I meant some way to ensure the MGS only allows connections with a certain level of security which might depend on who was connecting (of course "who" would have to be authenticated). I would hope this could be configured outside of lustre?> > I don''t think it''s acceptable to allow a previous mount to compromise > > the security of a later mount. > > Indeed it looks not so good. But the fact of per-node shared MGS > connection means only one flavor could be used. To avoid the confusion, > to me the only way is don''t allow the choice via mount option, instead > to choose a "proper" one automatically somehow.It''s OK if the MGS will reject any connection attempt with insufficient security. In that case, you could put the security in the mount option and it should probably be an error to specify different levels of security when talking to the same MGS.> > XXX > > > > This raises the interesting question of whether servers (MGS included) can > > demand a minimim level of security from clients connecting to them. Is this > > normally part of configuring security on a given node (e.g. to set the > > machine credentials you mentioned above)? > > This is the root problem I guess: we can''t assume there''s security > environment ready on each nodes.If the site needs security, it must be set up correctly. Whether this is done within lustre config or outside of it, or both is not the main point - it _has_ to be set up correctly.> The procedure of setup gss/kerberos is not extremely easy: configure > KDC, installing keytabs, configure gssapi, keyring, etc. And for most > Luster clusters, strong security are not needed at all, so people most > likely choose to skip that.1. Making lustre easy to deploy with a null security flavor is a requirement for sites that have no need for security. 2. Making it hard to deploy lustre with non-obvious "holes" in its security is a requirement for sites that need security. 3. Making it easy to test lustre with security enabled anywhere is a "nice to have" but not a requirement and it must certainly not be prioritised over (1) and (2). Cheers, Eric
Eric Barton wrote:> Comments inline > >> -----Original Message----- >> From: Eric.Mei at Sun.COM [mailto:Eric.Mei at Sun.COM] >> Sent: 04 June 2008 7:38 PM >> To: Eric Barton >> Cc: ''Lustre Development Mailing List''; ''Peter Braam'' >> Subject: Re: [Lustre-devel] security: MGS connection >> >> Eric Barton wrote: >>>> Here is the user interface change according to previous discussion, >>>> please review: >>>> >>>> - The security flavor of MGS connection is determined by each node, not >>>> controllable by MGS. >>> Is this an unavoidable fact of life or a design decision? >> I think it''s not completely unavoidable. For example, MGC can do initial >> connect without protect, and tell MGS what kind of security mode it >> support, and MGS replay with its decision, and MGC reconnect with >> choosed flavor. >> >> This way will be much more complicated. And more importantly, what if >> someone hijack the initial non-protected connection? Things seem not >> getting any better... > > I meant some way to ensure the MGS only allows connections with > a certain level of security which might depend on who was connecting > (of course "who" would have to be authenticated). I would hope this > could be configured outside of lustre?MGS can easily enforce that. The "who" actually means "which node".>>> I don''t think it''s acceptable to allow a previous mount to compromise >>> the security of a later mount. >> Indeed it looks not so good. But the fact of per-node shared MGS >> connection means only one flavor could be used. To avoid the confusion, >> to me the only way is don''t allow the choice via mount option, instead >> to choose a "proper" one automatically somehow. > > It''s OK if the MGS will reject any connection attempt with insufficient > security. In that case, you could put the security in the mount option > and it should probably be an error to specify different levels of security > when talking to the same MGS.I guess there''s misunderstanding. The problem is not on MGS, it''s on MGC. Each node have only one MGC, thus only one ptlrpc connection to MGS, which is shared by _all_ Lustre components live on this node. So in case of mounting 2 OSTs on a single node, only one security flavor could be used.>>> XXX >>> >>> This raises the interesting question of whether servers (MGS included) can >>> demand a minimim level of security from clients connecting to them. Is this >>> normally part of configuring security on a given node (e.g. to set the >>> machine credentials you mentioned above)? >> This is the root problem I guess: we can''t assume there''s security >> environment ready on each nodes. > > If the site needs security, it must be set up correctly. Whether this is > done within lustre config or outside of it, or both is not the main point - > it _has_ to be set up correctly. > >> The procedure of setup gss/kerberos is not extremely easy: configure >> KDC, installing keytabs, configure gssapi, keyring, etc. And for most >> Luster clusters, strong security are not needed at all, so people most >> likely choose to skip that. > > 1. Making lustre easy to deploy with a null security flavor is a requirement > for sites that have no need for security. > > 2. Making it hard to deploy lustre with non-obvious "holes" in its security is > a requirement for sites that need security. > > 3. Making it easy to test lustre with security enabled anywhere is a "nice to have" > but not a requirement and it must certainly not be prioritised over (1) and (2).OK I understand. But I didn''t mean people should skip security setting if strong security is needed. I meant in most cases strong security is not needed, thus people _will_ skip security setting, and in any case a proper security to MGS should be chosen. -- Eric
Eric, Here is an updated user interface proposal, please review: - MGS can be configured to "only allow RPC with certain level of security from certain node". The default is ''allow any''. - Each node choose what security flavor to use to connect MGS when mounting target device or client, by mount option "mgssec=flavor". By default ''null'' (no protection) is chosen. - For MDT/OST, the option "mgssec=flavor" could also be written on disk, like other parameters, but will be override if mount option supplied. - If flavor of GSS/Kerberos is specified, some pre-configured machine credential will be used, so no need to supply password or whatsoever. - The flavor of MGS connection won''t change until umount, no matter how rest of connection flavors change at runtime. - If there''s multiple mounts on one node, they must specify the same security flavor. For example, if we do: # mount -t lustre -o mgssec=krb5p /dev/sda1 /mnt/ost1 # mount -t lustre -o mgssec=null /dev/sda1 /mnt/ost2 then the second mount will fail immediately. -- Eric
> > It''s OK if the MGS will reject any connection attempt with insufficient > > security. In that case, you could put the security in the mount option > > and it should probably be an error to specify different levels of security > > when talking to the same MGS. > > I guess there''s misunderstanding. The problem is not on MGS, it''s on > MGC. Each node have only one MGC, thus only one ptlrpc connection to > MGS, which is shared by _all_ Lustre components live on this node. So in > case of mounting 2 OSTs on a single node, only one security flavor could > be used.Yes - I didn''t make myself clear enough. I grok there is only 1 MGC and I suggest we report an error if the MGC has already established a connection with insufficient security for a subsequent mount because this reveals an inconsistency. Cheers, Eric
> Here is an updated user interface proposal, please review: > > - MGS can be configured to "only allow RPC with certain level of > security from certain node". The default is ''allow any''.Fine.> - Each node choose what security flavor to use to connect MGS when > mounting target device or client, by mount option "mgssec=flavor". By > default ''null'' (no protection) is chosen.Fine.> - For MDT/OST, the option "mgssec=flavor" could also be written on disk, > like other parameters, but will be override if mount option supplied.How can "mgssec=flavor" apply to MDT/OST connections? What mount option will override saved MDT/OST parameters? IMHO we have to make an extremely clear separation between MGS connection security (which can only be specified in the mount command) and lustre server connection security (which can be stored on the MGS). Anything that blurs the distinction will be error prone.> - If flavor of GSS/Kerberos is specified, some pre-configured machine > credential will be used, so no need to supply password or whatsoever.Fine.> - The flavor of MGS connection won''t change until umount, no matter how > rest of connection flavors change at runtime.Fine.> - If there''s multiple mounts on one node, they must specify the same > security flavor. For example, if we do: > # mount -t lustre -o mgssec=krb5p /dev/sda1 /mnt/ost1 > # mount -t lustre -o mgssec=null /dev/sda1 /mnt/ost2 > then the second mount will fail immediately.Fine. Cheers, Eric
Eric Barton wrote:>> Here is an updated user interface proposal, please review: >> >> - MGS can be configured to "only allow RPC with certain level of >> security from certain node". The default is ''allow any''. > > Fine. > >> - Each node choose what security flavor to use to connect MGS when >> mounting target device or client, by mount option "mgssec=flavor". By >> default ''null'' (no protection) is chosen. > > Fine. > >> - For MDT/OST, the option "mgssec=flavor" could also be written on disk, >> like other parameters, but will be override if mount option supplied. > > How can "mgssec=flavor" apply to MDT/OST connections? What mount option > will override saved MDT/OST parameters?Sorry I was not clear enough. I meant connection from MDT or OST to MGS. The "mgssec=flavor" could be specified as mount parameter, or stored on disk by mkfs.lustre or tune2fs. If both present, mount option wins. Anyway it''s just some details.> IMHO we have to make an extremely clear separation between MGS connection > security (which can only be specified in the mount command) and lustre server > connection security (which can be stored on the MGS). Anything that blurs the > distinction will be error prone.Yes exactly, they''re completely separated.> >> - If flavor of GSS/Kerberos is specified, some pre-configured machine >> credential will be used, so no need to supply password or whatsoever. > > Fine. > >> - The flavor of MGS connection won''t change until umount, no matter how >> rest of connection flavors change at runtime. > > Fine. > >> - If there''s multiple mounts on one node, they must specify the same >> security flavor. For example, if we do: >> # mount -t lustre -o mgssec=krb5p /dev/sda1 /mnt/ost1 >> # mount -t lustre -o mgssec=null /dev/sda1 /mnt/ost2 >> then the second mount will fail immediately. > > Fine.-- Eric
Out of curiosity - do we secure OSC - OST connections with GSS? We send capabilities over them that are encrypted so perhaps it is not necessary. Peter On 6/5/08 6:39 PM, "Eric Mei" <Eric.Mei at Sun.COM> wrote:> Eric Barton wrote: >>> Here is an updated user interface proposal, please review: >>> >>> - MGS can be configured to "only allow RPC with certain level of >>> security from certain node". The default is ''allow any''. >> >> Fine. >> >>> - Each node choose what security flavor to use to connect MGS when >>> mounting target device or client, by mount option "mgssec=flavor". By >>> default ''null'' (no protection) is chosen. >> >> Fine. >> >>> - For MDT/OST, the option "mgssec=flavor" could also be written on disk, >>> like other parameters, but will be override if mount option supplied. >> >> How can "mgssec=flavor" apply to MDT/OST connections? What mount option >> will override saved MDT/OST parameters? > > Sorry I was not clear enough. I meant connection from MDT or OST to MGS. > The "mgssec=flavor" could be specified as mount parameter, or stored on > disk by mkfs.lustre or tune2fs. If both present, mount option wins. > Anyway it''s just some details. > >> IMHO we have to make an extremely clear separation between MGS connection >> security (which can only be specified in the mount command) and lustre server >> connection security (which can be stored on the MGS). Anything that blurs >> the >> distinction will be error prone. > > Yes exactly, they''re completely separated. > >> >>> - If flavor of GSS/Kerberos is specified, some pre-configured machine >>> credential will be used, so no need to supply password or whatsoever. >> >> Fine. >> >>> - The flavor of MGS connection won''t change until umount, no matter how >>> rest of connection flavors change at runtime. >> >> Fine. >> >>> - If there''s multiple mounts on one node, they must specify the same >>> security flavor. For example, if we do: >>> # mount -t lustre -o mgssec=krb5p /dev/sda1 /mnt/ost1 >>> # mount -t lustre -o mgssec=null /dev/sda1 /mnt/ost2 >>> then the second mount will fail immediately. >> >> Fine.
This is well formulated - make sure we get something like this on the arch wiki and in the manual. Peter On 6/5/08 9:54 AM, "Eric Mei" <Eric.Mei at Sun.COM> wrote:> Eric, > > Here is an updated user interface proposal, please review: > > - MGS can be configured to "only allow RPC with certain level of > security from certain node". The default is ''allow any''. > > - Each node choose what security flavor to use to connect MGS when > mounting target device or client, by mount option "mgssec=flavor". By > default ''null'' (no protection) is chosen. > > - For MDT/OST, the option "mgssec=flavor" could also be written on disk, > like other parameters, but will be override if mount option supplied. > > - If flavor of GSS/Kerberos is specified, some pre-configured machine > credential will be used, so no need to supply password or whatsoever. > > - The flavor of MGS connection won''t change until umount, no matter how > rest of connection flavors change at runtime. > > - If there''s multiple mounts on one node, they must specify the same > security flavor. For example, if we do: > # mount -t lustre -o mgssec=krb5p /dev/sda1 /mnt/ost1 > # mount -t lustre -o mgssec=null /dev/sda1 /mnt/ost2 > then the second mount will fail immediately.
Peter Braam wrote:> Out of curiosity - do we secure OSC - OST connections with GSS? We send > capabilities over them that are encrypted so perhaps it is not necessary.Yes OSC-OST connection could be configured to use any security flavor. -- Eric