Gluster users - Feb 2015 - poor performance with encryption and SSL enabled

Hi gluster folks,

I'm looking for some configuration or debugging advice for a
distributed-replicated volume that uses SSL and at rest encryption.

SSL certs are self-signed and generated on all servers. Combined into a
glusterfs.ca in /etc/ssl. By itself the SSL is working well.

I've also turned on the disk encryption feature. Master key was generated
with 'openssl rand -hex 32' as per the docs and copied to all gluster
servers.

Status of volume: data
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick ip-10-9-0-62.ec2.internal:/export/brick 49152 Y 13393
Brick ip-10-9-0-101.ec2.internal:/export/brick 49152 Y 8412
Brick ip-10-9-0-103.ec2.internal:/export/brick 49152 Y 10125
Brick ip-10-9-0-102.ec2.internal:/export/brick 49152 Y 8266
Brick ip-10-9-0-100.ec2.internal:/export/brick 49152 Y 8263
Brick ip-10-9-0-105.ec2.internal:/export/brick 49152 Y 8277
Brick ip-10-9-0-104.ec2.internal:/export/brick 49152 Y 8261
Brick ip-10-9-0-106.ec2.internal:/export/brick 49152 Y 8272

Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks

Volume Name: data
Type: Distributed-Stripe
Volume ID: afad6283-5bee-42c1-b9e5-c3ed64e04aae
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: ip-10-9-0-62.ec2.internal:/export/brick
Brick2: ip-10-9-0-101.ec2.internal:/export/brick
Brick3: ip-10-9-0-103.ec2.internal:/export/brick
Brick4: ip-10-9-0-102.ec2.internal:/export/brick
Brick5: ip-10-9-0-100.ec2.internal:/export/brick
Brick6: ip-10-9-0-105.ec2.internal:/export/brick
Brick7: ip-10-9-0-104.ec2.internal:/export/brick
Brick8: ip-10-9-0-106.ec2.internal:/export/brick
Options Reconfigured:
server.allow-insecure: on
nfs.ports-insecure: on
auth.allow: *
client.ssl: on
server.ssl: on
auth.ssl-allow: *
features.encryption: on
encryption.master-key: /root/keystore/master.key
performance.quick-read: off
performance.write-behind: off
performance.open-behind: off
nfs.disable: on

If I run dd or any i/o operations I see a flurry of these messages in the
logs.

[2015-02-24 16:58:51.144099] W [stripe.c:5288:stripe_internal_getxattr_cbk]
(--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x3fd0620550]
(-->
/usr/lib64/glusterfs/3.6.2/xlator/cluster/stripe.so(stripe_internal_getxattr_cbk+0x36a)[0x7f6a152a12ba]
(-->
/usr/lib64/glusterfs/3.6.2/xlator/protocol/client.so(client3_3_fgetxattr_cbk+0x174)[0x7f6a154db284]
(--> /usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x3fd0e0ea75]
(--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x142)[0x3fd0e0ff02] )))))
0-data-stripe-3: invalid argument: frame->local

Thanks in advance for any tips/suggestions!

-Adam
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150224/765df563/attachment.html>

> SSL certs are self-signed and generated on all servers. Combined into a
> glusterfs.ca in /etc/ssl. By itself the SSL is working well.
Glad to hear it.  ;)
> If I run dd or any i/o operations I see a flurry of these messages in the
> logs.
>
> [2015-02-24 16:58:51.144099] W [stripe.c:5288:stripe_internal_getxattr_cbk]
> (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x3fd0620550]
> (-->
>
/usr/lib64/glusterfs/3.6.2/xlator/cluster/stripe.so(stripe_internal_getxattr_cbk+0x36a)[0x7f6a152a12ba]
> (-->
>
/usr/lib64/glusterfs/3.6.2/xlator/protocol/client.so(client3_3_fgetxattr_cbk+0x174)[0x7f6a154db284]
> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x3fd0e0ea75]
(-->
> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x142)[0x3fd0e0ff02] )))))
> 0-data-stripe-3: invalid argument: frame->local

Have you tried encryption (at rest) without striping, or vice versa?  I
suspect some kind of bad interaction between the two, but before we go
down that path it would be nice to make sure they're working separately.

Hi Jeff/all,

I took the recommendation of disabled the stripes. Now I just have
encryption (at rest) and SSL enabled. The test I am running is a bwa
indexing. Basic dd read/writes work fine and I don't see any errors in the
gluster logs. Then when I try the bwa index I see the following:

/shared/perftest/bwa/bwa index -a bwtsw hg19.fa
[bwa_index] Pack FASTA... 26.29 sec
[bwa_index] Construct BWT for the packed sequence...
BWTIncConstructFromPacked() : Can't read from hg19.fa.pac : Unexpected end
of file

These are my current volume settings:
glusterfs 3.6.2 built on Jan 22 2015 12:58:11
Volume Name: data
Type: Distribute
Volume ID: 55d1c37b-bfba-47d8-8467-0b28b0e04aa2
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: ip-10-9-0-32.ec2.internal:/export/brick
Brick2: ip-10-9-0-141.ec2.internal:/export/brick
Brick3: ip-10-9-0-142.ec2.internal:/export/brick
Options Reconfigured:
performance.open-behind: off
performance.write-behind: off
performance.quick-read: off
encryption.master-key: /root/keystore/master.key
features.encryption: on
auth.ssl-allow: *
server.ssl: on
client.ssl: on
auth.allow: *

There are no messages in the logs during the job. However there are some
errors from previous lines:
[2015-03-09 15:21:47.868160] E [socket.c:2481:socket_poller]
0-data-client-0: poll error on socket
[2015-03-09 15:21:47.868184] E [socket.c:2481:socket_poller]
0-data-client-1: poll error on socket
[2015-03-09 15:21:47.868288] E [socket.c:2481:socket_poller]
0-data-client-2: poll error on socket

If I take out the encryption and leave just SSL mode on the bwa index is
successful. SSL may be good enough for our needs but I would like to know
if we have the option of at rest encryption. Any ideas? Many thanks in
advance!


On Tue, Feb 24, 2015 at 12:33 PM, Jeff Darcy <jdarcy at redhat.com> wrote:
> > SSL certs are self-signed and generated on all servers. Combined into
a
> > glusterfs.ca in /etc/ssl. By itself the SSL is working well.
>
> Glad to hear it.  ;)
>
> > If I run dd or any i/o operations I see a flurry of these messages in
the
> > logs.
> >
> > [2015-02-24 16:58:51.144099] W
> [stripe.c:5288:stripe_internal_getxattr_cbk]
> > (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x3fd0620550]
> > (-->
> >
>
/usr/lib64/glusterfs/3.6.2/xlator/cluster/stripe.so(stripe_internal_getxattr_cbk+0x36a)[0x7f6a152a12ba]
> > (-->
> >
>
/usr/lib64/glusterfs/3.6.2/xlator/protocol/client.so(client3_3_fgetxattr_cbk+0x174)[0x7f6a154db284]
> > (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x3fd0e0ea75]
> (-->
> > /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x142)[0x3fd0e0ff02] )))))
> > 0-data-stripe-3: invalid argument: frame->local
>
>
> Have you tried encryption (at rest) without striping, or vice versa?  I
> suspect some kind of bad interaction between the two, but before we go
> down that path it would be nice to make sure they're working
separately.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150309/510896e0/attachment.html>