Gluster users - May 2016 - changes to client port range in release 3.1.3

Hi all,

The various port ranges in glusterfs as of now:  (very high level view)


client:
      In case of bind secure:
                        will start from 1023 - 1, In case all these
port exhaust bind to random port (a connect() with out bind() call)
      In case of bind insecure:
                        will start from 65535 all the way down till 1

bricks/server:
      any port starting from 49152 to 65535

glusterd:
      24007


There was a recent bug, In case of bind secure, client see all ports
as exhausted and connect to a random port which was unfortunately in
brick port map range. So client successfully got a connected on a
given port. Now without these information with glusterd (since pmap
alloc done only at start), it passes the same port to brick, where
brick fails to connect on it (also consider the race situation)


To solve this issue we decided to split the client and brick port ranges. [1]

As usual bricks port map range will be IANA  ephemeral port range i.e
49152-65535.
For clients only in-case of secure ports exhaust (which is a rare
case),  we decided to fall back to registered ports i.e 49151 - 1024


If we see the ephemeral port standards
1.  The Internet Assigned Numbers Authority (IANA) suggests the range
49152 to 65535
2.  Many Linux kernels use the port range 32768 to 61000
more at [2]

Some of our thoughts include split the current brick port range ( ~16K
) into two (may be ~8K each or some other ratio) and use them for
client and bricks, which could solve the problem but also  introduce a
 limitation for scalability.

The patch [1] goes in 3.1.3, we wanted know if there are any impacts
caused with these changes.


[1] http://review.gluster.org/#/c/13998/
[2] https://en.wikipedia.org/wiki/Ephemeral_port


Thanks,
--
Prasanna

> The patch [1] goes in 3.1.3, we wanted know if there are any impacts
> caused with these changes.

What is 3.1.3? We are way past that release in GlusterFS. Are you
referring to 3.7.12?

Regards,
Vijay

On Tue, May 3, 2016 at 5:28 AM, Prasanna Kalever <pkalever at redhat.com>
wrote:
> If we see the ephemeral port standards
> 1.  The Internet Assigned Numbers Authority (IANA) suggests the range
> 49152 to 65535
> 2.  Many Linux kernels use the port range 32768 to 61000
> more at [2]
>
> Some of our thoughts include split the current brick port range ( ~16K
> ) into two (may be ~8K each or some other ratio) and use them for
> client and bricks, which could solve the problem but also  introduce a
>  limitation for scalability.
>
I would recommend providing the administrator an ability to override
any logic that we use to implement this behavior.
> The patch [1] goes in 3.1.3, we wanted know if there are any impacts
> caused with these changes.
>
>
> [1] http://review.gluster.org/#/c/13998/

I would have ideally liked the patch to have spent some time in the
review queue after this email was sent. It does look like the patch
got merged within 2 hours after the email was sent. This interval is
grossly inadequate if you are looking to obtain any feedback that can
add value. Nevertheless I have provided my comments on the patchset.
Please incorporate them in a subsequent commit.

Regards,
Vijay