bugzilla-daemon at netfilter.org
2024-Apr-03 22:14 UTC
[Bug 1742] New: using nfqueue breaks SCTP connection (tracking)
https://bugzilla.netfilter.org/show_bug.cgi?id=1742
Bug ID: 1742
Summary: using nfqueue breaks SCTP connection (tracking)
Product: libnetfilter_queue
Version: unspecified
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: P5
Component: libnetfilter_queue
Assignee: netfilter-buglog at lists.netfilter.org
Reporter: antonio.ojea.garcia at gmail.com
I'm using a golang library for interacting with nfqueue, it is a very simple
logic, I add the following rule
table inet kube-netpol {
comment "rules for kubernetes NetworkPolicy"
chain forward {
type filter hook forward priority filter - 5; policy accept;
ct state new queue to 100
}
}
and in userspace I process the packet to emit a verdict.
Everything works fine with TCP and UDP, but when using SCTP I can see the
packet its modified and breaks the establishment of the connection, more
details in https://github.com/aojea/kube-netpol/issues/8
Once I remove the `nfqueue` rule the SCTP connection is established correctly.
I triple checked the userspace program accepts the packet and removing the
nfqueue rules makes the connection work.
I've added a trace (by the way kudos for the tracing functionality is really
a
great improvement) and I can see how the packet is dropped in a rule that drops
connections with invalid state
trace id 0329b184 ip filter trace_chain packet: iif "eth0" ether saddr
02:42:c0:a8:08:02 ether daddr 02:42:c0:a8:08:03 ip saddr 10.244.1.47 ip daddr
10.244.2.47 ip dscp cs0 ip ecn ect0 ip ttl 63 ip id 0 ip length 68 sctp sport
47261 sctp dport 8080 sctp vtag 0 @th,96,64 0x10000240486b6e3
trace id 0329b184 ip filter trace_chain rule ip protocol sctp meta nftrace set
1 (verdict continue)
trace id 0329b184 ip filter trace_chain verdict continue
trace id 0329b184 ip filter trace_chain policy accept
trace id 0329b184 inet kube-netpol forward packet: iif "eth0" oif
"vetha2b65671" ether saddr 02:42:c0:a8:08:02 ether daddr
02:42:c0:a8:08:03 ip
saddr 10.244.1.47 ip daddr 10.244.2.47 ip dscp cs0 ip ecn ect0 ip ttl 62 ip id
0 ip protocol sctp ip length 68 sctp sport 47261 sctp dport 8080 sctp vtag 0
@th,96,64 0x10000240486b6e3
trace id 0329b184 inet kube-netpol forward verdict continue
trace id 0329b184 inet kube-netpol forward policy accept
trace id 0329b184 ip filter FORWARD packet: iif "eth0" oif
"vetha2b65671" ether
saddr 02:42:c0:a8:08:02 ether daddr 02:42:c0:a8:08:03 ip saddr 10.244.1.47 ip
daddr 10.244.2.47 ip dscp cs0 ip ecn ect0 ip ttl 62 ip id 0 ip length 68 sctp
sport 47261 sctp dport 8080 sctp vtag 0 @th,96,64 0x10000240486b6e3
trace id 0329b184 ip filter FORWARD rule counter packets 5735 bytes 2667239
jump KUBE-FORWARD (verdict jump KUBE-FORWARD)
trace id 0329b184 ip filter KUBE-FORWARD rule ct state invalid counter packets
8 bytes 544 drop (verdict drop)
if I remove the nfqueue rule the packet goes through
trace id 058bdf29 ip filter trace_chain packet: iif "eth0" ether saddr
02:42:c0:a8:08:02 ether daddr 02:42:c0:a8:08:03 ip saddr 10.244.1.47 ip daddr
10.244.2.47 ip dscp cs0 ip ecn ect0 ip ttl 63 ip id 0 ip length 68 sctp sport
33348 sctp dport 8080 sctp vtag 0 @th,96,64 0x10000244fde5e72
trace id 058bdf29 ip filter trace_chain rule ip protocol sctp meta nftrace set
1 (verdict continue)
trace id 058bdf29 ip filter trace_chain verdict continue
trace id 058bdf29 ip filter trace_chain policy accept
trace id 058bdf29 ip nat PREROUTING packet: iif "eth0" ether saddr
02:42:c0:a8:08:02 ether daddr 02:42:c0:a8:08:03 ip saddr 10.244.1.47 ip daddr
10.244.2.47 ip dscp cs0 ip ecn ect0 ip ttl 63 ip id 0 ip length 68 sctp sport
33348 sctp dport 8080 sctp vtag 0 @th,96,64 0x10000244fde5e72
trace id 058bdf29 ip nat PREROUTING rule counter packets 17924 bytes 1098260
jump KUBE-SERVICES (verdict jump KUBE-SERVICES)
trace id 058bdf29 ip nat KUBE-SERVICES verdict continue
trace id 058bdf29 ip nat PREROUTING verdict continue
trace id 058bdf29 ip nat PREROUTING policy accept
trace id 058bdf29 ip filter FORWARD packet: iif "eth0" oif
"vetha2b65671" ether
saddr 02:42:c0:a8:08:02 ether daddr 02:42:c0:a8:08:03 ip saddr 10.244.1.47 ip
daddr 10.244.2.47 ip dscp cs0 ip ecn ect0 ip ttl 62 ip id 0 ip length 68 sctp
sport 33348 sctp dport 8080 sctp vtag 0 @th,96,64 0x10000244fde5e72
trace id 058bdf29 ip filter FORWARD rule ct state new counter packets 2894
bytes 195836 jump KUBE-PROXY-FIREWALL (verdict jump KUBE-PROXY-FIREWALL)
trace id 058bdf29 ip filter KUBE-PROXY-FIREWALL verdict continue
trace id 058bdf29 ip filter FORWARD rule counter packets 5800 bytes 2671691
jump KUBE-FORWARD (verdict jump KUBE-FORWARD)
trace id 058bdf29 ip filter KUBE-FORWARD verdict continue
trace id 058bdf29 ip filter FORWARD rule ct state new counter packets 2832
bytes 191716 jump KUBE-SERVICES (verdict jump KUBE-SERVICES)
trace id 058bdf29 ip filter KUBE-SERVICES verdict continue
trace id 058bdf29 ip filter FORWARD rule ct state new counter packets 2826
bytes 191324 jump KUBE-EXTERNAL-SERVICES (verdict jump KUBE-EXTERNAL-SERVICES)
trace id 058bdf29 ip filter KUBE-EXTERNAL-SERVICES verdict continue
trace id 058bdf29 ip filter FORWARD verdict continue
trace id 058bdf29 ip filter FORWARD policy accept
trace id 058bdf29 ip nat POSTROUTING packet: iif "eth0" oif
"vetha2b65671"
ether saddr 02:42:c0:a8:08:02 ether daddr 02:42:c0:a8:08:03 ip saddr
10.244.1.47 ip daddr 10.244.2.47 ip dscp cs0 ip ecn ect0 ip ttl 62 ip id 0 ip
length 68 sctp sport 33348 sctp dport 8080 sctp vtag 0 @th,96,64
0x10000244fde5e72
trace id 058bdf29 ip nat POSTROUTING rule counter packets 5868 bytes 374884
jump KUBE-POSTROUTING (verdict jump KUBE-POSTROUTING)
trace id 058bdf29 ip nat KUBE-POSTROUTING verdict return
trace id 058bdf29 ip nat POSTROUTING rule fib daddr type != local counter
packets 2834 bytes 191394 jump KIND-MASQ-AGENT (verdict jump KIND-MASQ-AGENT)
trace id 058bdf29 ip nat KIND-MASQ-AGENT verdict return
trace id 058bdf29 ip nat POSTROUTING verdict continue
trace id 058bdf29 ip nat POSTROUTING policy accept
--
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240403/1ac98ca4/attachment.html>
bugzilla-daemon at netfilter.org
2024-Apr-04 08:42 UTC
[Bug 1742] using nfqueue breaks SCTP connection (tracking)
https://bugzilla.netfilter.org/show_bug.cgi?id=1742
Antonio Ojea <antonio.ojea.garcia at gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Priority|P5 |P2
Component|libnetfilter_queue |netfilter hooks
Product|libnetfilter_queue |netfilter/iptables
--
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240404/369b6da4/attachment.html>
bugzilla-daemon at netfilter.org
2024-Apr-07 20:27 UTC
[Bug 1742] using nfqueue breaks SCTP connection (tracking)
https://bugzilla.netfilter.org/show_bug.cgi?id=1742 --- Comment #1 from Antonio Ojea <antonio.ojea.garcia at gmail.com> --- I?ve used this great tool from the cilium project (pwru) to trace the packet and I can observe that if I don?t use nfqueue I got this trace 0xffff88810983aa00 0 [<empty>(280243)] ip_forward netns=4026533169 mark=0x0 iface=168(eth0) proto=0x0800 mtu=1500 len=84 192.168.8.2:0->10.244.2.47:0() 0xffff88810983aa00 0 [<empty>(280243)] nf_hook_slow netns=4026533169 mark=0x0 iface=168(eth0) proto=0x0800 mtu=1500 len=84 192.168.8.2:0->10.244.2.47:0() 0xffff88810983aa00 0 [<empty>(280243)] ip_forward_finish netns=4026533169 mark=0x0 iface=168(eth0) proto=0x0800 mtu=1500 len=84 192.168.8.2:0->10.244.2.47:0() and when using it there are some functions that operate on the sctp checksum 0xffff88810749bf00 29 [<empty>(274286)] ip_forward netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68 10.244.1.47:0->10.244.2.47:0() 0xffff88810749bf00 29 [<empty>(274286)] nf_hook_slow netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68 10.244.1.47:0->10.244.2.47:0() 0xffff88810749bf00 29 [<empty>(274286)] nf_queue netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68 10.244.1.47:0->10.244.2.47:0() 0xffff88810749bf00 29 [<empty>(274286)] __nf_queue netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68 10.244.1.47:0->10.244.2.47:0() 0xffff88810749bf00 29 [<empty>(274286)] skb_checksum_help netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68 10.244.1.47:0->10.244.2.47:0() 0xffff88810749bf00 29 [<empty>(274286)] skb_ensure_writable netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68 10.244.1.47:0->10.244.2.47:0() 0xffff88810749bf00 26 [<empty>(3319058)] nf_reroute netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68 10.244.1.47:0->10.244.2.47:0() 0xffff88810749bf00 26 [<empty>(3319058)] ip_forward_finish netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68 10.244.1.47:0->10.244.2.47:0() The packet captures confirms that when using nfqueue some packet modification happens https://github.com/aojea/kube-netpol/issues/8#issuecomment-2039184720 As a workaround, if I set the flag NFQA_CFG_F_GSO then the packet is not modified and connection works perfectly (I?m only using nfqueue for dropping) -- You are receiving this mail because: You are watching all bug changes. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240407/08ac02d7/attachment.html>
bugzilla-daemon at netfilter.org
2024-Apr-07 20:34 UTC
[Bug 1742] using nfqueue breaks SCTP connection (tracking)
https://bugzilla.netfilter.org/show_bug.cgi?id=1742
Pablo Neira Ayuso <pablo at netfilter.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |pablo at netfilter.org
--- Comment #2 from Pablo Neira Ayuso <pablo at netfilter.org> ---
(In reply to Antonio Ojea from comment #1)>
> I?ve used this great tool from the cilium project (pwru) to trace the
packet
> and I can observe that if I don?t use nfqueue I got this trace
>
> 0xffff88810983aa00 0 [<empty>(280243)] ip_forward
> netns=4026533169 mark=0x0 iface=168(eth0) proto=0x0800 mtu=1500 len=84
> 192.168.8.2:0->10.244.2.47:0()
> 0xffff88810983aa00 0 [<empty>(280243)] nf_hook_slow
> netns=4026533169 mark=0x0 iface=168(eth0) proto=0x0800 mtu=1500 len=84
> 192.168.8.2:0->10.244.2.47:0()
> 0xffff88810983aa00 0 [<empty>(280243)] ip_forward_finish
> netns=4026533169 mark=0x0 iface=168(eth0) proto=0x0800 mtu=1500 len=84
> 192.168.8.2:0->10.244.2.47:0()
>
> and when using it there are some functions that operate on the sctp
checksum
>
> 0xffff88810749bf00 29 [<empty>(274286)] ip_forward
> netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68
> 10.244.1.47:0->10.244.2.47:0()
> 0xffff88810749bf00 29 [<empty>(274286)] nf_hook_slow
> netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68
> 10.244.1.47:0->10.244.2.47:0()
> 0xffff88810749bf00 29 [<empty>(274286)] nf_queue
> netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68
> 10.244.1.47:0->10.244.2.47:0()
> 0xffff88810749bf00 29 [<empty>(274286)] __nf_queue
> netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68
> 10.244.1.47:0->10.244.2.47:0()
> 0xffff88810749bf00 29 [<empty>(274286)] skb_checksum_help
> netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68
> 10.244.1.47:0->10.244.2.47:0()
> 0xffff88810749bf00 29 [<empty>(274286)] skb_ensure_writable
> netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68
> 10.244.1.47:0->10.244.2.47:0()
> 0xffff88810749bf00 26 [<empty>(3319058)] nf_reroute
> netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68
> 10.244.1.47:0->10.244.2.47:0()
> 0xffff88810749bf00 26 [<empty>(3319058)] ip_forward_finish
> netns=4026532933 mark=0x0 iface=47 proto=0x0800 mtu=1500 len=68
> 10.244.1.47:0->10.244.2.47:0()
>
>
> The packet captures confirms that when using nfqueue some packet
> modification happens
> https://github.com/aojea/kube-netpol/issues/8#issuecomment-2039184720
>
>
> As a workaround, if I set the flag NFQA_CFG_F_GSO then the packet is not
> modified and connection works perfectly (I?m only using nfqueue for
dropping)
SCTP checksum is lacking in nfnetlink_queue.
There is a patch for nftables that could be used for reference:
commit 346e320cb2103edef709c4466a29140c4a8e527a
Author: Davide Caratti <dcaratti at redhat.com>
Date: Thu Oct 15 18:39:27 2020 +0200
netfilter: nftables: allow re-computing sctp CRC-32C in 'payload'
statements
There is a skb_checksum_help() call in nfnetlink_queue that needs to be
replaced by a helper function that run the specific SCTP checksum function.
--
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240407/2b16a70e/attachment-0001.html>
bugzilla-daemon at netfilter.org
2024-Apr-07 20:38 UTC
[Bug 1742] using nfqueue breaks SCTP connection (tracking)
https://bugzilla.netfilter.org/show_bug.cgi?id=1742 --- Comment #3 from Pablo Neira Ayuso <pablo at netfilter.org> --- (In reply to Pablo Neira Ayuso from comment #2)> (In reply to Antonio Ojea from comment #1) > > As a workaround, if I set the flag NFQA_CFG_F_GSO then the packet is not > > modified and connection works perfectly (I?m only using nfqueue for dropping)Workaround makes sense, because branch depends on NFQA_CFG_F_GSO flag: case NFQNL_COPY_PACKET: if (!(queue->flags & NFQA_CFG_F_GSO) && entskb->ip_summed == CHECKSUM_PARTIAL && skb_checksum_help(entskb)) return NULL; if set on, the checksum calculation is skipped. For filtering (only dropping), enabling NFQA_CFG_F_GSO makes perfect sense (unless your heuristic is based on the real packet as seen in the wire, that is, GRO does not interference with the filtering criteria) -- You are receiving this mail because: You are watching all bug changes. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240407/2005a6a7/attachment.html>
bugzilla-daemon at netfilter.org
2024-Apr-07 20:51 UTC
[Bug 1742] using nfqueue breaks SCTP connection (tracking)
https://bugzilla.netfilter.org/show_bug.cgi?id=1742
Antonio Ojea <antonio.ojea.garcia at gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |ASSIGNED
--- Comment #4 from Antonio Ojea <antonio.ojea.garcia at gmail.com>
---> There is a skb_checksum_help() call in nfnetlink_queue that needs to be
replaced by a helper function that run the specific SCTP checksum function.
great, this seems a good opportunity to contribute, let me try to see if I'm
able to submit a patch
--
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240407/ec79ff63/attachment.html>
bugzilla-daemon at netfilter.org
2024-May-02 21:54 UTC
[Bug 1742] using nfqueue breaks SCTP connection (tracking)
https://bugzilla.netfilter.org/show_bug.cgi?id=1742 --- Comment #5 from Antonio Ojea <antonio.ojea.garcia at gmail.com> --- Pablo I need some guidance, I want to add a test and I think the right place is in tools/testing/selftests/netfilter/nft_queue.sh , however, I need some sctp client/server to generate the traffic. The only client/server I could find is in another folder tools/testing/selftests/net/sctp_hello.c Should I duplicate the sctp_hello.c file under tools/testing/selftests/netfilter? -- You are receiving this mail because: You are watching all bug changes. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240502/437f8e77/attachment.html>
bugzilla-daemon at netfilter.org
2024-May-02 22:09 UTC
[Bug 1742] using nfqueue breaks SCTP connection (tracking)
https://bugzilla.netfilter.org/show_bug.cgi?id=1742 --- Comment #6 from Pablo Neira Ayuso <pablo at netfilter.org> --- (In reply to Antonio Ojea from comment #5)> Pablo I need some guidance, I want to add a test and I think the right place > is in tools/testing/selftests/netfilter/nft_queue.sh , however, I need some > sctp client/server to generate the traffic. > > The only client/server I could find is in another folder > tools/testing/selftests/net/sctp_hello.c > > Should I duplicate the sctp_hello.c file under > tools/testing/selftests/netfilter?Are you using the net-next.git tree? Florian has recently moved all netfilter tests to tools/testing/selftests/net/ , that would allow you to reuse the existing file. I can help with rebasing if you have done it already. Thanks a lot for working on a patch for this. -- You are receiving this mail because: You are watching all bug changes. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240502/4498f996/attachment.html>
bugzilla-daemon at netfilter.org
2024-May-02 22:09 UTC
[Bug 1742] using nfqueue breaks SCTP connection (tracking)
https://bugzilla.netfilter.org/show_bug.cgi?id=1742 --- Comment #7 from Pablo Neira Ayuso <pablo at netfilter.org> --- (In reply to Pablo Neira Ayuso from comment #6)> (In reply to Antonio Ojea from comment #5) > > Pablo I need some guidance, I want to add a test and I think the right place > > is in tools/testing/selftests/netfilter/nft_queue.sh , however, I need some > > sctp client/server to generate the traffic. > > > > The only client/server I could find is in another folder > > tools/testing/selftests/net/sctp_hello.c > > > > Should I duplicate the sctp_hello.c file under > > tools/testing/selftests/netfilter? > > Are you using the net-next.git tree?I am referring to this tree: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/ -- You are receiving this mail because: You are watching all bug changes. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240502/f59c77d0/attachment.html>
bugzilla-daemon at netfilter.org
2024-May-03 13:04 UTC
[Bug 1742] using nfqueue breaks SCTP connection (tracking)
https://bugzilla.netfilter.org/show_bug.cgi?id=1742
--- Comment #8 from Antonio Ojea <antonio.ojea.garcia at gmail.com> ---
It is the first time that I send a patch to the kernel so sorry in advance for
all the noise :)
Let me describe the situation, I think I didn't get right the problem, these
are the patches I sent
https://patchwork.ozlabs.org/project/netfilter-devel/list/?series=405380
I have a test in patch 2 that reproduces the problem, that I think is good
If I apply patch 1 IIUIC it skips skb_checksum_help() , this patch solves the
problem but does not seem right, does it?
diff --git a/net/netfilter/nfnetlink_queue.c b/net/netfilter/nfnetlink_queue.c
index 00f4bd21c59b..428014aea396 100644
--- a/net/netfilter/nfnetlink_queue.c
+++ b/net/netfilter/nfnetlink_queue.c
@@ -600,6 +600,7 @@ nfqnl_build_packet_message(struct net *net, struct
nfqnl_instance *queue,
case NFQNL_COPY_PACKET:
if (!(queue->flags & NFQA_CFG_F_GSO) &&
entskb->ip_summed == CHECKSUM_PARTIAL &&
+ (skb_csum_is_sctp(entskb) && skb_crc32c_csum_help(entskb))
&&
skb_checksum_help(entskb))
return NULL;
If instead of an AND I use an OR
- skb_checksum_help(entskb))
+ ((skb_csum_is_sctp(entskb) &&
skb_crc32c_csum_help(entskb)) ||
+ skb_checksum_help(entskb)))
now the test fails ... so it seems the problem is skb_checksum_help() ??
--
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240503/40c64460/attachment.html>
bugzilla-daemon at netfilter.org
2024-Oct-03 10:42 UTC
[Bug 1742] using nfqueue breaks SCTP connection (tracking)
https://bugzilla.netfilter.org/show_bug.cgi?id=1742
Antonio Ojea <antonio.ojea.garcia at gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |FIXED
Status|ASSIGNED |RESOLVED
--- Comment #9 from Antonio Ojea <antonio.ojea.garcia at gmail.com> ---
Fixed by
https://github.com/torvalds/linux/commit/26a77d02891ab62172085a4f94af9b3c90aed387
--
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20241003/527b2905/attachment.html>
Apparently Analagous Threads
- Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them.
- Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them.
- New CentOS Atomic Release and Kubernetes System Containers Now Available
- [Bug 362] Loss of change password functionality
- CentOS-announce Digest, Vol 150, Issue 1