Erik Jacobson
2021-Jan-30 12:40 UTC
[Gluster-users] gnfs exports netmask handling can incorrectly deny access to clients
Hello team -
First, I wish to state that I know we are supposed to move to Ganesha.
We had a lot of trouble with Ganesha in the past with our workload and
we still owe trying the very latest version and working with the
community. Some of our use cases are complicated and require very large
clusters to test. Therefore, switching has remained elusive.
We still rely on Gluster NFS.
Gluster is now used as part of the solution in some of the largest
supercomputers in the world.
We encountered a problem with Gluster NFS handling of the exports file
in relation to how it computes access rights.
We have patched our build of Gluster with this fix. I'm not sure what
the final fix would be like, but I'm hoping what I paste below will
enable us to get a final fix in to the community.
This analysis and patch was developed by Dick Riegner when I asked for
his help on this problem. There were several others involved.
What follows is his analysis. I will then paste the patch we're using
now. We would be happy to test a new version of the fix if you like (so we
can remove our patch when we upgrade). What follows are Dick's words.
ANALYSIS
=============================================================================Here
is my Gluster debug output from its nfs.log file. The working case is
from a compute node client using the IP address 10.31.128.16, and the
failing case is from a client using the IP address 10.31.133.16.
Working case
------------
RJR01: gf_is_ip_in_net() Entered network is 10.31.128.0/18, ip_str is
10.31.128.16
RJR20: gf_is_ip_in_net() subnet is 18, net_str is 10.31.128.0, net_ip is
10.31.128.0
RJR40: gf_is_ip_in_net() Host byte order subnet_mask is 0003ffff, ip_buf is
10801f0a, net_ip_buf is 00801f0a
RJR42: gf_is_ip_in_net() Network byte order subnet_mask is ffff0300, ip_buf is
0a1f8010, net_ip_buf is 0a1f8000
RJR44: gf_is_ip_in_net() Network byte order shifted 14 host bits, ip_buf is
0000287e, net_ip_buf is 0000287e
RJR46: gf_is_ip_in_net() My result is 1
RJR99: gf_is_ip_in_net() Exiting result is 1
Failing Case
------------
RJR01: gf_is_ip_in_net() Entered network is 10.31.128.0/18, ip_str is
10.31.133.16
RJR20: gf_is_ip_in_net() subnet is 18, net_str is 10.31.128.0, net_ip is
10.31.128.0
RJR40: gf_is_ip_in_net() Host byte order subnet_mask is 0003ffff, ip_buf is
10851f0a, net_ip_buf is 00801f0a
RJR42: gf_is_ip_in_net() Network byte order subnet_mask is ffff0300, ip_buf is
0a1f8510, net_ip_buf is 0a1f8000
RJR44: gf_is_ip_in_net() Network byte order shifted 14 host bits, ip_buf is
0000287e, net_ip_buf is 0000287e
RJR46: gf_is_ip_in_net() My result is 1
RJR99: gf_is_ip_in_net() Exiting result is 0
Gluster function gf_is_ip_in_net() verifies a client's authorization to
mount
an export by comparing the subnet address of the client with an allowed
subnet address. The comparison is made by masking the client IP
address and the allowed subnet address and permitting access when the
resulting subnets are equal.
The mask is an all-ones bit-string the length of the subnet. In this case,
the subnet is 18 bits and the subnet mask of 0x3ffff is in Little Endian
ordering used by the Intel x86_64 processor.
1) Analysis of the working case from client IP address 10.31.128.16
These addresses are in Little Endian order on an Intel x86_64 processor.
Client IP Subnet
Address Mask Subnet
0x10801f0a & 0x3ffff => 0x01f0a
Allowed
Subnet Subnet
Address Mask Subnet
0x00801f0a & 0x3ffff => 0x01f0a
The resulting subnets are equal so Gluster allows the client to mount its
exports.
2) Analysis of the failing case from client IP address 10.31.133.16
These addresses are in Little Endian order on an Intel x86_64 processor.
Client IP Subnet
Address Mask Subnet
0x10851f0a & 0x3ffff => 0x11f0a
Allowed
Subnet Subnet
Address Mask Subnet
0x00801f0a & 0x3ffff => 0x01f0a
The resulting subnets are not equal so Gluster does not allow the client to
mount its exports.
The comparison is incorrectly including the two lower-order bits from part
of the host portion of the client IP address (0x85) as part of the subnet.
The subnet comparison fails and the client is incorrectly denied access to
the Gluster exports.
PROPOSED FIX DESCRIPTION
=============================================================================The
fix for the incorrect access denied errors is to convert the client and
allowed subnet IP addresses from Host Byte Order (Little Endian)
format to Network Byte Order (Big Endian) format and then isolate their
subnets. This will ensure that the subnet and host parts of their IP
addresses do not overlap. Once their subnets are properly isolated,
the subnets can be properly compared.
The conversion from Host Byte Order to Network Byte Order is done by
calling the htonl() function.
A subnet mask is no longer used, but the subnet bit length is used to
isolate the subnet address. Once the IP addresses are converted to
Network Byte Order, they are shifted right to eliminate the host portion
of the address. The amount to shift is calculated by:
host_shift = 32 - subnet_bit_length
In this case, subnet_bit_length is 18 so host_shift is 14.
This leaves just the subnet values after the shift. If the subnets are
equal, the client is authorized to mount an export.
1) Fix for working case from client IP address 10.31.128.16
Client IP Shift
Address Right Subnet
0x0a1f8010 >> 14 => 0x0000287e
Allowed
Subnet Shift
Address Right Subnet
0x0a1f8000 >> 14 => 0x0000287e
The resulting subnets are equal so Gluster allows the client to mount
its exports.
2) Fix for failing case from client IP address 10.31.133.16
Client IP Shift
Address Right Subnet
0x0a1f8510 >> 14 => 0x0000287e
Allowed
Subnet Shift
Address Right Subnet
0x0a1f8000 >> 14 => 0x0000287e
The resulting subnets are equal so Gluster allows the client to mount
its exports.
I updated Gluster function gf_is_ip_in_net() to fix the mount access
denied errors seen by some compute node clients with large host
addresses. I also removed my debug code.
diff -ruNp -x '*~' glusterfs-7.2.orig/libglusterfs/src/common-utils.c
glusterfs-7.2.new/libglusterfs/src/common-utils.c
--- glusterfs-7.2.orig/libglusterfs/src/common-utils.c 2020-01-15
11:43:53.748893825 -0600
+++ glusterfs-7.2.new/libglusterfs/src/common-utils.c 2021-01-27
12:24:01.659533006 -0600
@@ -2099,7 +2099,9 @@ gf_is_ip_in_net(const char *network, con
{
unsigned long ip_buf = 0;
unsigned long net_ip_buf = 0;
- unsigned long subnet_mask = 0;
+/* HPE */
+ int host_shift = 0;
+/* HPE */
int ret = -EINVAL;
char *slash = NULL;
char *net_ip = NULL;
@@ -2142,10 +2144,13 @@ gf_is_ip_in_net(const char *network, con
goto out;
}
- /* Converts /x into a mask */
- subnet_mask = (1 << atoi(subnet)) - 1;
-
- result = ((ip_buf & subnet_mask) == (net_ip_buf & subnet_mask));
+/* HPE */
+ /* Calculate host portion bit shift to isolate subnet */
+ host_shift = 32 - atoi(subnet);
+
+ /* Compare subnets in Network Byte Order (Big Endian) */
+ result = ((htonl(ip_buf) >> host_shift) == (htonl(net_ip_buf)
>> host_shift));
+/* HPE */
out:
return result;
}
Strahil Nikolov
2021-Jan-30 17:01 UTC
[Gluster-users] gnfs exports netmask handling can incorrectly deny access to clients
Hi Erik,
let's add devel to the thread.
Best Regards,Strahil Nikolov
Sent from Yahoo Mail on Android
On Sat, Jan 30, 2021 at 14:40, Erik Jacobson<erik.jacobson at hpe.com>
wrote: Hello team -
First, I wish to state that I know we are supposed to move to Ganesha.
We had a lot of trouble with Ganesha in the past with our workload and
we still owe trying the very latest version and working with the
community. Some of our use cases are complicated and require very large
clusters to test. Therefore, switching has remained elusive.
We still rely on Gluster NFS.
Gluster is now used as part of the solution in some of the largest
supercomputers in the world.
We encountered a problem with Gluster NFS handling of the exports file
in relation to how it computes access rights.
We have patched our build of Gluster with this fix. I'm not sure what
the final fix would be like, but I'm hoping what I paste below will
enable us to get a final fix in to the community.
This analysis and patch was developed by Dick Riegner when I asked for
his help on this problem. There were several others involved.
What follows is his analysis. I will then paste the patch we're using
now. We would be happy to test a new version of the fix if you like (so we
can remove our patch when we upgrade). What follows are Dick's words.
ANALYSIS
=============================================================================Here
is my Gluster debug output from its nfs.log file.? The working case is
from a compute node client using the IP address 10.31.128.16, and the
failing case is from a client using the IP address 10.31.133.16.
Working case
------------
RJR01: gf_is_ip_in_net() Entered network is 10.31.128.0/18, ip_str is
10.31.128.16
RJR20: gf_is_ip_in_net() subnet is 18, net_str is 10.31.128.0, net_ip is
10.31.128.0
RJR40: gf_is_ip_in_net() Host byte order subnet_mask is 0003ffff, ip_buf is
10801f0a, net_ip_buf is 00801f0a
RJR42: gf_is_ip_in_net() Network byte order subnet_mask is ffff0300, ip_buf is
0a1f8010, net_ip_buf is 0a1f8000
RJR44: gf_is_ip_in_net() Network byte order shifted 14 host bits, ip_buf is
0000287e, net_ip_buf is 0000287e
RJR46: gf_is_ip_in_net() My result is 1
RJR99: gf_is_ip_in_net() Exiting result is 1
Failing Case
------------
RJR01: gf_is_ip_in_net() Entered network is 10.31.128.0/18, ip_str is
10.31.133.16
RJR20: gf_is_ip_in_net() subnet is 18, net_str is 10.31.128.0, net_ip is
10.31.128.0
RJR40: gf_is_ip_in_net() Host byte order subnet_mask is 0003ffff, ip_buf is
10851f0a, net_ip_buf is 00801f0a
RJR42: gf_is_ip_in_net() Network byte order subnet_mask is ffff0300, ip_buf is
0a1f8510, net_ip_buf is 0a1f8000
RJR44: gf_is_ip_in_net() Network byte order shifted 14 host bits, ip_buf is
0000287e, net_ip_buf is 0000287e
RJR46: gf_is_ip_in_net() My result is 1
RJR99: gf_is_ip_in_net() Exiting result is 0
Gluster function gf_is_ip_in_net() verifies a client's authorization to
mount
an export by comparing the subnet address of the client with an allowed
subnet address.? The comparison is made by masking the client IP
address and the allowed subnet address and permitting access when the
resulting subnets are equal.
The mask is an all-ones bit-string the length of the subnet.? In this case,
the subnet is 18 bits and the subnet mask of 0x3ffff is in Little Endian
ordering used by the Intel x86_64 processor.
1)? Analysis of the working case from client IP address 10.31.128.16
These addresses are in Little Endian order on an Intel x86_64 processor.
Client IP? ? ? Subnet
Address? ? ? ? Mask? ? ? ? Subnet
0x10801f0a? &? 0x3ffff? =>? 0x01f0a
Allowed
Subnet? ? ? ? Subnet
Address? ? ? ? Mask? ? ? ? Subnet
0x00801f0a? &? 0x3ffff? =>? 0x01f0a
The resulting subnets are equal so Gluster allows the client to mount its
exports.
2)? Analysis of the failing case from client IP address 10.31.133.16
These addresses are in Little Endian order on an Intel x86_64 processor.
Client IP? ? ? Subnet
Address? ? ? ? Mask? ? ? ? Subnet
0x10851f0a? &? 0x3ffff? =>? 0x11f0a
Allowed
Subnet? ? ? ? Subnet
Address? ? ? ? Mask? ? ? ? Subnet
0x00801f0a? &? 0x3ffff? =>? 0x01f0a
The resulting subnets are not equal so Gluster does not allow the client to
mount its exports.
The comparison is incorrectly including the two lower-order bits from part
of the host portion of the client IP address (0x85) as part of the subnet.
The subnet comparison fails and the client is incorrectly denied access to
the Gluster exports.
PROPOSED FIX DESCRIPTION
=============================================================================The
fix for the incorrect access denied errors is to convert the client and
allowed subnet IP addresses from Host Byte Order (Little Endian)
format to Network Byte Order (Big Endian) format and then isolate their
subnets.? This will ensure that the subnet and host parts of their IP
addresses do not overlap.? Once their subnets are properly isolated,
the subnets can be properly compared.
The conversion from Host Byte Order to Network Byte Order is done by
calling the htonl() function.
A subnet mask is no longer used, but the subnet bit length is used to
isolate the subnet address.? Once the IP addresses are converted to
Network Byte Order, they are shifted right to eliminate the host portion
of the address.? The amount to shift is calculated by:
host_shift = 32 - subnet_bit_length
In this case, subnet_bit_length is 18 so host_shift is 14.
This leaves just the subnet values after the shift.? If the subnets are
equal, the client is authorized to mount an export.
1)? Fix for working case from client IP address 10.31.128.16
Client IP? ? ? Shift
Address? ? ? ? Right? ? ? Subnet
0x0a1f8010? >>? 14? ? =>? 0x0000287e
Allowed
Subnet? ? ? ? ? Shift
Address? ? ? ? Right? ? ? Subnet
0x0a1f8000? >>? 14? ? =>? 0x0000287e
The resulting subnets are equal so Gluster allows the client to mount
its exports.
2)? Fix for failing case from client IP address 10.31.133.16
Client IP? ? ? Shift
Address? ? ? ? Right? ? ? Subnet
0x0a1f8510? >>? 14? ? =>? 0x0000287e
Allowed
Subnet? ? ? ? ? Shift
Address? ? ? ? Right? ? ? Subnet
0x0a1f8000? >>? 14? ? =>? 0x0000287e
The resulting subnets are equal so Gluster allows the client to mount
its exports.
I updated Gluster function gf_is_ip_in_net() to fix the mount access
denied errors seen by some compute node clients with large host
addresses.? I also removed my debug code.
diff -ruNp -x '*~' glusterfs-7.2.orig/libglusterfs/src/common-utils.c
glusterfs-7.2.new/libglusterfs/src/common-utils.c
--- glusterfs-7.2.orig/libglusterfs/src/common-utils.c??? 2020-01-15
11:43:53.748893825 -0600
+++ glusterfs-7.2.new/libglusterfs/src/common-utils.c??? 2021-01-27
12:24:01.659533006 -0600
@@ -2099,7 +2099,9 @@ gf_is_ip_in_net(const char *network, con
{
? ? unsigned long ip_buf = 0;
? ? unsigned long net_ip_buf = 0;
-? ? unsigned long subnet_mask = 0;
+/* HPE */
+? ? int host_shift = 0;
+/* HPE */
? ? int ret = -EINVAL;
? ? char *slash = NULL;
? ? char *net_ip = NULL;
@@ -2142,10 +2144,13 @@ gf_is_ip_in_net(const char *network, con
? ? ? ? goto out;
? ? }
-? ? /* Converts /x into a mask */
-? ? subnet_mask = (1 << atoi(subnet)) - 1;
-
-? ? result = ((ip_buf & subnet_mask) == (net_ip_buf & subnet_mask));
+/* HPE */
+? ? /* Calculate host portion bit shift to isolate subnet */
+? ? host_shift = 32 - atoi(subnet);
+
+? ? /* Compare subnets in Network Byte Order (Big Endian) */
+? ? result = ((htonl(ip_buf) >> host_shift) == (htonl(net_ip_buf)
>> host_shift));
+/* HPE */
out:
? ? return result;
}
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20210130/9e98c5e2/attachment.html>