Joseph Lorenzini
2017-Mar-19 14:00 UTC
[Gluster-users] Secured mount in GlusterFS using keys
Hi Deepak, Sorta. I think it depends on what we mean by I/O path and performance. If we are referring to disk I/O for gluster servers, then no. If we are referring to the network I/O between a gluster client and server, than yes there will by definition be some additional overhead. That however is true of any security layer one chooses to pick for any application, especially a distributed system. In practice, security of any kind, whether its encryption, ACLs, or even iptables, will degrade the performance of an application. And since distributed systems by definition handle their state through network I/O, that means security + distributed system = network latency. There's a reason people say security is where performance goes to die. :) Now that all said, frequently the issue is not whether there will be network latency, but how much and does it matter? Moreover, what are the specific performance requirements for your gluster pool and have they been weighed against the costs of meeting those requirements? Additionally, how does meeting those performance requirements weigh against all your other requirements like for example having basic network security around a distributed system? I would be quite surprised if openssl MTLS would be any slower compared to some other key-based authentication scheme. Most of the cost of TLS is around the TLS handshake, which is a one-time hit when the gluster client mounts the volume. Since the client is maintaining a persistent TLS connection, most of the overhead is openssl code performing symmetric encryption, which openssl, despite all its warts, is really really good at doing really really fast (understand this all relative to an arbitrary baseline :). Bottom line: with modern hardware, the performance impact of MTLS should be negligible. IMHO, if the performance requirement can't tolerate MTLS, then its in practice preventing you from implementing any reasonable security scheme at all. In that case, you'd be better off just setting up an isolated network and skipping any type of authentication. I'd recommend setting up MTLS with gluster and run your performance tests against it. That will definitively answer your question of whether the performance is acceptable. The MTLS setup is not that hard and the gluster documentation is reasonable though could be improved (I need to submit some PRs against it). if you have any questions about setup and configuration, I am sure I can help. Joe On Sat, Mar 18, 2017 at 2:25 PM, Deepak Naidu <dnaidu at nvidia.com> wrote:> Hi Joe, thanks for taking time for explaining. I am having basic set of > requirements along with IO performance as key factor, my reply below should > justify what I am trying to achieve. > > >>If I am understanding your use case properly, you want to ensure that a > client may only mount a gluster volume if and only if it presents a key or > secret that attests to the client's identity, which the gluster server can > use to verify that client's identity. > > Yes, this is the exact use case for my requirements. > > > > >>That's exactly what gluster MTLS is doing since the gluster server > performs chain-of-trust validation on the client's leaf certificate. > > That's good, but my confusion here is does this MTLS also encrypt's IO > traffic like TLS. If yes, than it's not want I am looking for. The reason > is the IO encryption/decryption is an extra overhead for my use case as > performance of IO is also factor why we're are looking for GlusterFS, > unless my understanding is incorrect that IO encryption has no overhead. > > > > >> I don't understand why I/O path encryption is something you want to > avoid -- seems like an essential part of basic network security that you > get for "free" with the authentication. > > If I understand the term IO path encryption correctly, all the storage IO > will go through extra latency of encryption & decryption which is not > needed for my requirements as this produced extra IO latency which is why I > am trying to avoid IO path encryption & just need basic secret based > authentication. > > > > > -- > Deepak > > > On Mar 18, 2017, at 10:46 AM, Joseph Lorenzini <jaloren at gmail.com> > wrote: > > > > I am little confused about what you are trying to accomplish here. If I > am understanding your use case properly, you want to ensure that a client > may only mount a gluster volume if and only if it presents a key or secret > that attests to the client's identity, which the gluster server can use to > verify that client's identity. That's exactly what gluster MTLS is doing > since the gluster server performs chain-of-trust validation on the client's > leaf certificate. > > > > Of course this will necessarily force encryption of the I/O path since > its TLS. I don't understand why I/O path encryption is something you want > to avoid -- seems like an essential part of basic network security that you > get for "free" with the authentication. It is true that if you want the > key-based authentication of a gluster client, you will need gluster MTLS. > You could treat encryption as the "cost" of getting authentication if you > will. > > > > Now I am personally pretty negative on X.509 and chain-of-trust in > general, since the trust model has been proven to not scale and is > frequently broken by malicious and incompetent CAs. I also think its a > completely inappropriate security model for something like gluster where > all endpoints are known and controlled by a single entity, forcing a > massive amount of unnecessary complexity with certificate management with > no real added security. I have thought about making a feature request that > gluster support a simple public-key encryption that's implemented like SSH. > But all that said, MTLS is a well-tested, well known security protocol and > the gluster team built it on top of openssl so it does get the security job > done in an acceptable way. The fact that the I/O path is encrypted is not > the thing that bothers me about the implementation though. > ------------------------------------------------------------ > ----------------------- > This email message is for the sole use of the intended recipient(s) and > may contain > confidential information. Any unauthorized review, use, disclosure or > distribution > is prohibited. If you are not the intended recipient, please contact the > sender by > reply email and destroy all copies of the original message. > ------------------------------------------------------------ > ----------------------- >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170319/bfafefcc/attachment.html>
Thanks Joe for your inputs. I guess comparing client -- glusterServer IO performance over MTLS and non-MTLS should give me some idea on the client/server IO overhead. Also any URL related to setup & configuring MTLS is appreciated. -- Deepak On Mar 19, 2017, at 7:00 AM, Joseph Lorenzini <jaloren at gmail.com<mailto:jaloren at gmail.com>> wrote: Hi Deepak, Sorta. I think it depends on what we mean by I/O path and performance. If we are referring to disk I/O for gluster servers, then no. If we are referring to the network I/O between a gluster client and server, than yes there will by definition be some additional overhead. That however is true of any security layer one chooses to pick for any application, especially a distributed system. In practice, security of any kind, whether its encryption, ACLs, or even iptables, will degrade the performance of an application. And since distributed systems by definition handle their state through network I/O, that means security + distributed system = network latency. There's a reason people say security is where performance goes to die. :) Now that all said, frequently the issue is not whether there will be network latency, but how much and does it matter? Moreover, what are the specific performance requirements for your gluster pool and have they been weighed against the costs of meeting those requirements? Additionally, how does meeting those performance requirements weigh against all your other requirements like for example having basic network security around a distributed system? I would be quite surprised if openssl MTLS would be any slower compared to some other key-based authentication scheme. Most of the cost of TLS is around the TLS handshake, which is a one-time hit when the gluster client mounts the volume. Since the client is maintaining a persistent TLS connection, most of the overhead is openssl code performing symmetric encryption, which openssl, despite all its warts, is really really good at doing really really fast (understand this all relative to an arbitrary baseline :). Bottom line: with modern hardware, the performance impact of MTLS should be negligible. IMHO, if the performance requirement can't tolerate MTLS, then its in practice preventing you from implementing any reasonable security scheme at all. In that case, you'd be better off just setting up an isolated network and skipping any type of authentication. I'd recommend setting up MTLS with gluster and run your performance tests against it. That will definitively answer your question of whether the performance is acceptable. The MTLS setup is not that hard and the gluster documentation is reasonable though could be improved (I need to submit some PRs against it). if you have any questions about setup and configuration, I am sure I can help. Joe On Sat, Mar 18, 2017 at 2:25 PM, Deepak Naidu <dnaidu at nvidia.com<mailto:dnaidu at nvidia.com>> wrote: Hi Joe, thanks for taking time for explaining. I am having basic set of requirements along with IO performance as key factor, my reply below should justify what I am trying to achieve.>>If I am understanding your use case properly, you want to ensure that a client may only mount a gluster volume if and only if it presents a key or secret that attests to the client's identity, which the gluster server can use to verify that client's identity.Yes, this is the exact use case for my requirements.>>That's exactly what gluster MTLS is doing since the gluster server performs chain-of-trust validation on the client's leaf certificate.That's good, but my confusion here is does this MTLS also encrypt's IO traffic like TLS. If yes, than it's not want I am looking for. The reason is the IO encryption/decryption is an extra overhead for my use case as performance of IO is also factor why we're are looking for GlusterFS, unless my understanding is incorrect that IO encryption has no overhead.>> I don't understand why I/O path encryption is something you want to avoid -- seems like an essential part of basic network security that you get for "free" with the authentication.If I understand the term IO path encryption correctly, all the storage IO will go through extra latency of encryption & decryption which is not needed for my requirements as this produced extra IO latency which is why I am trying to avoid IO path encryption & just need basic secret based authentication. -- Deepak> On Mar 18, 2017, at 10:46 AM, Joseph Lorenzini <jaloren at gmail.com<mailto:jaloren at gmail.com>> wrote: > > I am little confused about what you are trying to accomplish here. If I am understanding your use case properly, you want to ensure that a client may only mount a gluster volume if and only if it presents a key or secret that attests to the client's identity, which the gluster server can use to verify that client's identity. That's exactly what gluster MTLS is doing since the gluster server performs chain-of-trust validation on the client's leaf certificate. > > Of course this will necessarily force encryption of the I/O path since its TLS. I don't understand why I/O path encryption is something you want to avoid -- seems like an essential part of basic network security that you get for "free" with the authentication. It is true that if you want the key-based authentication of a gluster client, you will need gluster MTLS. You could treat encryption as the "cost" of getting authentication if you will. > > Now I am personally pretty negative on X.509 and chain-of-trust in general, since the trust model has been proven to not scale and is frequently broken by malicious and incompetent CAs. I also think its a completely inappropriate security model for something like gluster where all endpoints are known and controlled by a single entity, forcing a massive amount of unnecessary complexity with certificate management with no real added security. I have thought about making a feature request that gluster support a simple public-key encryption that's implemented like SSH. But all that said, MTLS is a well-tested, well known security protocol and the gluster team built it on top of openssl so it does get the security job done in an acceptable way. The fact that the I/O path is encrypted is not the thing that bothers me about the implementation though.----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. ----------------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170319/cc58a269/attachment.html>