Gluster users - Oct 2015 - MTU size issue?

On 10/14/2015 05:09 PM, Sander Zijlstra wrote:
> LS,
> 
> I recently reconfigured one of my gluster nodes and forgot to update the
MTU size on the switch while I did configure the host with jumbo frames.
> 
> The result was that the complete cluster had communication issues.
> 
> All systems are part of a distributed striped volume with a replica size of
2 but still the cluster was completely unusable until I updated the switch port
to accept jumbo frames rather than to discard them.
This is expected. When enabling the network components to communicate
with TCP jumbo frames in a Gluster Trusted Storage Pool, you'd need to
ensure that all the network components such as switches, nodes are
configured properly. I think with this setting you'd fail to ping the
other nodes in the pool. So that could be a step of verification before
you set the cluster up.
> 
> The symptoms were:
> 
> - Gluster clients had a very hard time reading the volume information and
thus couldn?t do any filesystem ops on them.
> - The glusterfs servers could see each other (peer status) and a volume
info command was ok, but a volume status command would not return or would
return a ?staging failed? error.
> 
> I know MTU size mixing and don?t fragment bit?s can screw up a lot but why
wasn?t that gluster peer just discarded from the cluster so that not all clients
kept on communicating with it and causing all sorts of errors.
To answer this question, peer status & volume info are local operation
and doesn't incur N/W, so in this very same case you might see peer
status showing all the nodes are connected all though there is a
breakage, OTOH in status command originator node communicates with other
peers and hence it fails there.

HTH,
Atin
> 
> I use glusterFS 3.6.2 at the moment?..
> 
> Kind regards
> Sander
> 
> 
> 
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>

Hi Atin,

You?re right in saying if it?s activate then all nodes should have it activated.

What I find strange is that when glusterfsd has problems communicating with the
other peers that that single node with issues isn?t considered ?not connected?
and thus expelled from the cluster somehow; in my case it caused a complete hang
of the trusted storage pool.

And to emphasise this, pinging was no problem as it uses small packets anyway so
jumbo frames were not used at all? enabling jumbo frames on the interface and
switches is only a way to tell the TCP/IP stack that it can send larger packets
but it does?t have to.

Or am I mistaking in that the TCP/IP stack will control wether to send the
bigger packets and that glusterfsd has no control over that?

Met vriendelijke groet / kind regards,

Sander Zijlstra

| Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6
43 99 12 47 | sander.zijlstra at surfsara.nl | www.surfsara.nl |

Regular day off on friday
> On 15 Oct 2015, at 08:24, Atin Mukherjee <amukherj at redhat.com>
wrote:
> 
> 
> 
> On 10/14/2015 05:09 PM, Sander Zijlstra wrote:
>> LS,
>> 
>> I recently reconfigured one of my gluster nodes and forgot to update
the MTU size on the switch while I did configure the host with jumbo frames.
>> 
>> The result was that the complete cluster had communication issues.
>> 
>> All systems are part of a distributed striped volume with a replica
size of 2 but still the cluster was completely unusable until I updated the
switch port to accept jumbo frames rather than to discard them.
> This is expected. When enabling the network components to communicate
> with TCP jumbo frames in a Gluster Trusted Storage Pool, you'd need to
> ensure that all the network components such as switches, nodes are
> configured properly. I think with this setting you'd fail to ping the
> other nodes in the pool. So that could be a step of verification before
> you set the cluster up.
>> 
>> The symptoms were:
>> 
>> - Gluster clients had a very hard time reading the volume information
and thus couldn?t do any filesystem ops on them.
>> - The glusterfs servers could see each other (peer status) and a volume
info command was ok, but a volume status command would not return or would
return a ?staging failed? error.
>> 
>> I know MTU size mixing and don?t fragment bit?s can screw up a lot but
why wasn?t that gluster peer just discarded from the cluster so that not all
clients kept on communicating with it and causing all sorts of errors.
> To answer this question, peer status & volume info are local operation
> and doesn't incur N/W, so in this very same case you might see peer
> status showing all the nodes are connected all though there is a
> breakage, OTOH in status command originator node communicates with other
> peers and hence it fails there.
> 
> HTH,
> Atin
>> 
>> I use glusterFS 3.6.2 at the moment?..
>> 
>> Kind regards
>> Sander
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20151027/76e1a45b/attachment.sig>