> On the other side allow jumbo frames and change mtu on even hundreds on
> nodes is extremely simple,
>
> you can just test it. I don't see "bunch of extra work" here,
just use ssh
> and some scripting or something like ansible...
Our issue is we decided to simplify the configuration in our cluster
manager so that cluster management traffic, NFS, and gluster are
co-mingled. Works great. However, we often need to talk to BMCs on that
same network, and many BMCs don't handle MTU 9K correctly. Often a BMC
will seem to work but if you send something big like firmware flash to
it, it never completes the transfer due to the MTU mismatch. So the
"hard part" is due to our own stuff.
We have a method in the cluster manager to put BMCs in a separate
network but that isn't a common choice.
We are investigating using MTU size-by-path but that gets complicated to
test. Therefore, we are looking to understand the real-world problem with
a 1500 MTU on 2x bonded 10G networks with gluster to decide if we want to
put time and resource to solve the problem. It's been working pretty
well at 1500 MTU so far. If the only issue is less throughput, that may
be a price we can pay since we're not bandwidth bound right now.
Erik