Hello all
We have a NFS server witch has three network ports.
We have bonded these interfaces as a lagg interface, but when we use the
server it looks like only two interfaces are used.
This is our rc.conf file
ifconfig_igb0="up"
ifconfig_igb1="up"
ifconfig_igb2="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 laggport igb2
192.168.100.222 netmask 255.255.255.0"
ifconfig tell us the following.
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
ether a0:36:9f:7d:fc:2f
inet 192.168.100.222 netmask 0xffffff00 broadcast 192.168.100.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
laggproto lacp lagghash l2,l3,l4
laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
So all looks fine
But with 4 machines putting a 1GB file on the server one is always using
full wirespeed, the rest is around 30 / 40 MB. It never uses all three
intefaces but two at max.
So we are topped at 200MB/s where we were expecting around 300MB/s
#systat -if shows this also.
It shows two interfaces at work where igb1 is sitting and doing nothing?
/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
Load Average |
Interface Traffic Peak Total
lagg1 in 0.253 KB/s 0.683 KB/s 459.781 KB
out 0.000 KB/s 0.000 KB/s 0.000 KB
lagg0 in 0.289 KB/s 215.439 MB/s 4.113 GB
out 0.091 KB/s 114.269 MB/s 2.061 GB
lo0 in 0.000 KB/s 0.068 KB/s 0.770 KB
out 0.000 KB/s 0.068 KB/s 0.770 KB
igb2 in 0.011 KB/s *98.401 MB/s*
1.039 GB
out 0.022 KB/s 1.474 MB/s 27.311 MB
igb1 in 0.143 KB/s *0.466 KB/s*
192.422 KB
out 0.022 KB/s 1.959 MB/s 27.066 MB
igb0 in 0.135 KB/s *117.340 MB/s *
3.074 GB
out 0.114 KB/s 112.679 MB/s 2.007 GB
Is there something we can do to make sure lagg0 uses all the interfaces.
regards
Johan
On Tue, Nov 17, 2015 at 8:26 AM, Johan Hendriks <joh.hendriks at gmail.com> wrote:> Is there something we can do to make sure lagg0 uses all the interfaces.Nope. LACP doesn't actively load balance its interfaces. Each flow gets assigned to a single interface based on a hash of the source and destination MACs, IP addresses, and TCP/UDP ports. With many clients, all interfaces will probably be used, but with few clients, there's a lot of luck involved. If you want more bandwidth, you can try fiddling with IP addresses and port numbers to influence the hash function, but even if you get it to distribute the way you want, all your work may be undone by a reboot. The best option is to buy a 10Gbps NIC for the server. They aren't too expensive, anymore, though the switches are still pricey. A cheaper option, if you'll only ever have 4 clients, is to discard the lagg and assign a separate IP address to each igb port, then manually distribute those addresses amongst your clients. If you do this, you unfortunately won't gain the reliability features of LACP. -Alan
Hi!> We have a NFS server witch has three network ports. > > We have bonded these interfaces as a lagg interface, but when we use the > server it looks like only two interfaces are used. > > This is our rc.conf file > > ifconfig_igb0="up" > ifconfig_igb1="up" > ifconfig_igb2="up" > cloned_interfaces="lagg0" > ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 laggport igb2 > 192.168.100.222 netmask 255.255.255.0"This says you are lagg'in igb0 to igb2.> ifconfig tell us the following.[...]> laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>This says it's lagg'in igb1 to igb3 ? Why the difference ? -- pi at opsec.eu +49 171 3101372 5 years to go !
On 17/11/2015 15:26, Johan Hendriks wrote:> Hello all > > We have a NFS server witch has three network ports. > > We have bonded these interfaces as a lagg interface, but when we use the > server it looks like only two interfaces are used. > > This is our rc.conf file > > ifconfig_igb0="up" > ifconfig_igb1="up" > ifconfig_igb2="up" > cloned_interfaces="lagg0" > ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 laggport igb2 > 192.168.100.222 netmask 255.255.255.0" > > ifconfig tell us the following. > > lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > > options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO> > ether a0:36:9f:7d:fc:2f > inet 192.168.100.222 netmask 0xffffff00 broadcast 192.168.100.255 > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > media: Ethernet autoselect > status: active > laggproto lacp lagghash l2,l3,l4 > laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> >This shows that your server is using l2,l3,l4 hashing for lacp but what options have you configured on the switch?