Hi,
I'm deploying a CentOS8 (not stream ?) cluster and I have a question
about MTU on the interfaces. I have a connectX6 Mellanox interface where
I need IBoIP setup.
I've setup this interface via nmcli and set the MTU to 65520 with:
nmcli connection modify ib0 mtu 65520
nmcli connection up ib0
Running "nmcli connection show ib0" report:
infiniband.mtu:???????????????????????? 65520
But "ip addr show ib0" report a mtu of 2044:
6: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> *mtu 2044 *qdisc mq state
UP group default qlen 256
Why ? Who is wrong (possibly me....)?
Thanks
Patrick
what happens when you 'ping -M do -s 65000 -c 4 <other IBoIP system> ? Best Regards, Strahil Nikolov ? ??????????, 14 ???????? 2020 ?., 15:46:50 ???????+2, Patrick B?gou <patrick.begou at legi.grenoble-inp.fr> ??????: Hi, I'm deploying a CentOS8 (not stream ?) cluster and I have a question about MTU on the interfaces. I have a connectX6 Mellanox interface where I need IBoIP setup. I've setup this interface via nmcli and set the MTU to 65520 with: ? ? nmcli connection modify ib0 mtu 65520 ? ? nmcli connection up ib0 Running "nmcli connection show ib0" report: ? ? infiniband.mtu:???????????????????????? 65520 But "ip addr show ib0" report a mtu of 2044: ? ? 6: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> *mtu 2044 *qdisc mq state ? ? UP group default qlen 256 Why ? Who is wrong (possibly me....)? Thanks Patrick _______________________________________________ CentOS mailing list CentOS at centos.org https://lists.centos.org/mailman/listinfo/centos
On Mon, 14 Dec 2020 14:46:30 +0100 Patrick B?gou <Patrick.Begou at legi.grenoble-inp.fr> wrote:> Hi, > > I'm deploying a CentOS8 (not stream ?) cluster and I have a question > about MTU on the interfaces. I have a connectX6 Mellanox interface > where I need IBoIP setup. > I've setup this interface via nmcli and set the MTU to 65520 with: > > nmcli connection modify ib0 mtu 65520 > nmcli connection up ib0 > > Running "nmcli connection show ib0" report: > > infiniband.mtu:???????????????????????? 65520 > > But "ip addr show ib0" report a mtu of 2044: > > 6: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> *mtu 2044 *qdisc mq > state UP group default qlen 2562044 is what you get with DATAGRAM mode and the native IB running with 2K packages. You can get 4092 IPoIB MTU with 4K IB. And you do want to run 4K native IB in general. To run 64K MTU you need to switch from DATAGRAM mode to CONNECTED mode for IPoIB. CONNECTED mode give the highest single stream bandwidth typically but may not give best overall performance (vs 4092 DATAGRAM mode). Cheers, Peter K