I've spent the last couple of days trying to get Multichannel and RSS
working
and it's been hit and miss.? Mostly miss at this point.
I have two servers on the same network, running the same dual Mellanox 100G
card with mostly identical samba configs.? A Win 10 client sees multichannel
and RSS from one server but not the other when running
"Get-SmbMultichannelConnection -IncludeNotSelected".
Servers are running Devuan (Debian 10) with the 5.10 kernel and ZFS.? I
thought my original issue was because the mlx5_core driver in the 5.10 kernel
did not show any queues in /proc/interrupts (grep lan2 /proc/interrupts).?
Installing the driver from Mellanox fixed that problem.? But the odd thing is,
the server that is working, is running the stock kernel driver and and does
not show anything in /proc/interrupts.? I tried both the stock and Mellanox
driver on the server that is not working.
I'm at a loss on how to debug this on why the Win 10 client does not see
multichannel or RSS for one server but not the other..? I upgraded the server
to samba 4.14.6+dfsg-0.1buster1 from 4.9.? The server that is working used
both versions of samba and the Win 10 client saw it with each version.
I'm mostly intested in getting RSS working as I read for multiple nics to
work
I would need to put the IPs on different subnets.
Each server has both of the 100G links bonded into a 802.3ad LAG.
Information:
smb.conf:
[global]
??? netbios name = SB1
??? interfaces = "172.28.202.10;capability=RSS"
"172.28.90.10;capability=RSS"
"172.28.133.10;capability=RSS,speed=100000000000"
??????? server multi channel support = yes
??? workgroup = GIANT
??? wins server = 172.28.202.5
??? os level = 20
??????? log level = 3
??? client min protocol = SMB3
??????? os level = 8
??????? local master = No
??????? domain master = No
??? ntlm auth = yes
??? log file = /var/log/samba/log.%m
??????? max log size = 1000
??? server role = standalone server
??? obey pam restrictions = No
??? unix password sync = no
??? passwd program = /usr/bin/passwd %u
??? passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:*
%n\n *password\supdated\ssuccessfully*
??? invalid users = root
??? usershare path ??? usershare allow guests = yes
??? guest account = nas
??? aio read size = 1
??? aio write size = 1
??? aio max threads = 100
??? vfs objects = catia
??????? mangled names = no
??????? catia:mappings
0x22:0xa8,0x2a:0xa4,0x2f:0xf8,0x3a:0xf7,0x3c:0xab,0x3e:0xbb,0x3f:0xbf,0x5c:0xff,0x7c:0xa6
??? smb2 leases = yes
??? posix locking = no
??? security = user
??? map to guest = bad user???
??? store dos attributes = yes
??? dos filetimes = yes
[plaid]
??? path = /netshares/plaid
??? force user = nas
??? force group = nas
??? create mask = 770
??? force create mode = 770
??? directory mask = 2770
??? force directory mode = 2770
??? guest ok = yes
??? read only = no
??? hosts allow = 172.28.133. 172.28.90.
??? dos filetimes = yes
ethtool -l lan2
Channel parameters for lan2:
Pre-set maximums:
RX:??? ??? 0
TX:??? ??? 0
Other:??? ??? 0
Combined:??? 32
Current hardware settings:
RX:??? ??? 0
TX:??? ??? 0
Other:??? ??? 0
Combined:??? 32
ethtool -g lan2
Ring parameters for lan2:
Pre-set maximums:
RX:??? ??? 8192
RX Mini:??? 0
RX Jumbo:??? 0
TX:??? ??? 8192
Current hardware settings:
RX:??? ??? 1024
RX Mini:??? 0
RX Jumbo:??? 0
TX:??? ??? 1024
ethtool -x lan2
RX flow hash indirection table for lan2 with 32 RX ring(s):
??? 0:????? 0???? 1???? 2???? 3???? 4???? 5???? 6???? 7
??? 8:????? 8???? 9??? 10??? 11??? 12??? 13??? 14??? 15
?? 16:???? 16??? 17??? 18??? 19??? 20??? 21??? 22??? 23
?? 24:???? 24??? 25??? 26??? 27??? 28??? 29??? 30??? 31
?? 32:????? 0???? 1???? 2???? 3???? 4???? 5???? 6???? 7
?? 40:????? 8???? 9??? 10??? 11??? 12??? 13??? 14??? 15
?? 48:???? 16??? 17??? 18??? 19??? 20??? 21??? 22??? 23
?? 56:???? 24??? 25??? 26??? 27??? 28??? 29??? 30??? 31
?? 64:????? 0???? 1???? 2???? 3???? 4???? 5???? 6???? 7
?? 72:????? 8???? 9??? 10??? 11??? 12??? 13??? 14??? 15
?? 80:???? 16??? 17??? 18??? 19??? 20??? 21??? 22??? 23
?? 88:???? 24??? 25??? 26??? 27??? 28??? 29??? 30??? 31
?? 96:????? 0???? 1???? 2???? 3???? 4???? 5???? 6???? 7
? 104:????? 8???? 9??? 10??? 11??? 12??? 13??? 14??? 15
? 112:???? 16??? 17??? 18??? 19??? 20??? 21??? 22??? 23
? 120:???? 24??? 25??? 26??? 27??? 28??? 29??? 30??? 31
RSS hash key:
12:e3:f3:8e:92:fa:db:eb:5b:29:e2:49:cd:39:39:9c:a4:ea:41:e8:c9:ee:66:7f:37:9b:ea:33:99:d9:45:3c:8a:db:bd:a1:bc:66:24:05
RSS hash function:
??? toeplitz: on
??? xor: off
??? crc32: off
grep lan2 /proc/interrupts??
??? http://ix.io/3viC? (note, need very wide window)
ifconfig bond_private
bond_private: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST>? mtu 9000
??????? inet 172.28.133.10? netmask 255.255.255.0? broadcast 172.28.133.255
??????? inet6 fe80::ae1f:6bff:fecf:20e0? prefixlen 64? scopeid 0x20<link>
??????? ether ac:1f:6b:cf:20:e0? txqueuelen 1000? (Ethernet)
??????? RX packets 54063955? bytes 326509018851 (304.0 GiB)
??????? RX errors 0? dropped 5? overruns 0? frame 0
??????? TX packets 34841728? bytes 299047275167 (278.5 GiB)
??????? TX errors 0? dropped 5 overruns 0? carrier 0? collisions 0
PS C:\> Get-SmbClientNetworkInterface
Interface Index RSS Capable RDMA Capable Speed???
IpAddresses???????????????????????????????? Friendly Name
--------------- ----------- ------------ -----???
-----------???????????????????????????????? -------------
14????????????? True??????? False??????? 100 Gbps {fe80::f0b7:6e80:9de4:df83,
172.28.133.101} Ethernet 4
Please let me know if there is any other information I can provide to help me
get this figured out.
??? Chris
Bump.? Any ideas on what the problem be.? I've been working on this and I don't know why it is not working.? We need to get about 15Gb/s to do the project that we are working on and without Multichannel and RSS it is not going to be possible. ??? Chris On 8/7/21 4:43 PM, Chris Dos via samba wrote:> I've spent the last couple of days trying to get Multichannel and RSS working > and it's been hit and miss.? Mostly miss at this point. > > I have two servers on the same network, running the same dual Mellanox 100G > card with mostly identical samba configs.? A Win 10 client sees multichannel > and RSS from one server but not the other when running > "Get-SmbMultichannelConnection -IncludeNotSelected". > > Servers are running Devuan (Debian 10) with the 5.10 kernel and ZFS.? I > thought my original issue was because the mlx5_core driver in the 5.10 kernel > did not show any queues in /proc/interrupts (grep lan2 /proc/interrupts).? > Installing the driver from Mellanox fixed that problem.? But the odd thing is, > the server that is working, is running the stock kernel driver and and does > not show anything in /proc/interrupts.? I tried both the stock and Mellanox > driver on the server that is not working. > > I'm at a loss on how to debug this on why the Win 10 client does not see > multichannel or RSS for one server but not the other..? I upgraded the server > to samba 4.14.6+dfsg-0.1buster1 from 4.9.? The server that is working used > both versions of samba and the Win 10 client saw it with each version. > > I'm mostly intested in getting RSS working as I read for multiple nics to work > I would need to put the IPs on different subnets. > > Each server has both of the 100G links bonded into a 802.3ad LAG. > > Information: > > smb.conf: > [global] > ??? netbios name = SB1 > ??? interfaces = "172.28.202.10;capability=RSS" "172.28.90.10;capability=RSS" > "172.28.133.10;capability=RSS,speed=100000000000" > ??????? server multi channel support = yes > ??? workgroup = GIANT > ??? wins server = 172.28.202.5 > ??? os level = 20 > ??????? log level = 3 > ??? client min protocol = SMB3 > ??????? os level = 8 > ??????? local master = No > ??????? domain master = No > ??? ntlm auth = yes > ??? log file = /var/log/samba/log.%m > ??????? max log size = 1000 > ??? server role = standalone server > ??? obey pam restrictions = No > ??? unix password sync = no > ??? passwd program = /usr/bin/passwd %u > ??? passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* > %n\n *password\supdated\ssuccessfully* > ??? invalid users = root > ??? usershare path > ??? usershare allow guests = yes > ??? guest account = nas > ??? aio read size = 1 > ??? aio write size = 1 > ??? aio max threads = 100 > ??? vfs objects = catia > ??????? mangled names = no > ??????? catia:mappings > 0x22:0xa8,0x2a:0xa4,0x2f:0xf8,0x3a:0xf7,0x3c:0xab,0x3e:0xbb,0x3f:0xbf,0x5c:0xff,0x7c:0xa6 > ??? smb2 leases = yes > ??? posix locking = no > ??? security = user > ??? map to guest = bad user??? > ??? store dos attributes = yes > ??? dos filetimes = yes > > [plaid] > ??? path = /netshares/plaid > ??? force user = nas > ??? force group = nas > ??? create mask = 770 > ??? force create mode = 770 > ??? directory mask = 2770 > ??? force directory mode = 2770 > ??? guest ok = yes > ??? read only = no > ??? hosts allow = 172.28.133. 172.28.90. > ??? dos filetimes = yes > > > > ethtool -l lan2 > Channel parameters for lan2: > Pre-set maximums: > RX:??? ??? 0 > TX:??? ??? 0 > Other:??? ??? 0 > Combined:??? 32 > Current hardware settings: > RX:??? ??? 0 > TX:??? ??? 0 > Other:??? ??? 0 > Combined:??? 32 > > ethtool -g lan2 > Ring parameters for lan2: > Pre-set maximums: > RX:??? ??? 8192 > RX Mini:??? 0 > RX Jumbo:??? 0 > TX:??? ??? 8192 > Current hardware settings: > RX:??? ??? 1024 > RX Mini:??? 0 > RX Jumbo:??? 0 > TX:??? ??? 1024 > > ethtool -x lan2 > RX flow hash indirection table for lan2 with 32 RX ring(s): > ??? 0:????? 0???? 1???? 2???? 3???? 4???? 5???? 6???? 7 > ??? 8:????? 8???? 9??? 10??? 11??? 12??? 13??? 14??? 15 > ?? 16:???? 16??? 17??? 18??? 19??? 20??? 21??? 22??? 23 > ?? 24:???? 24??? 25??? 26??? 27??? 28??? 29??? 30??? 31 > ?? 32:????? 0???? 1???? 2???? 3???? 4???? 5???? 6???? 7 > ?? 40:????? 8???? 9??? 10??? 11??? 12??? 13??? 14??? 15 > ?? 48:???? 16??? 17??? 18??? 19??? 20??? 21??? 22??? 23 > ?? 56:???? 24??? 25??? 26??? 27??? 28??? 29??? 30??? 31 > ?? 64:????? 0???? 1???? 2???? 3???? 4???? 5???? 6???? 7 > ?? 72:????? 8???? 9??? 10??? 11??? 12??? 13??? 14??? 15 > ?? 80:???? 16??? 17??? 18??? 19??? 20??? 21??? 22??? 23 > ?? 88:???? 24??? 25??? 26??? 27??? 28??? 29??? 30??? 31 > ?? 96:????? 0???? 1???? 2???? 3???? 4???? 5???? 6???? 7 > ? 104:????? 8???? 9??? 10??? 11??? 12??? 13??? 14??? 15 > ? 112:???? 16??? 17??? 18??? 19??? 20??? 21??? 22??? 23 > ? 120:???? 24??? 25??? 26??? 27??? 28??? 29??? 30??? 31 > RSS hash key: > 12:e3:f3:8e:92:fa:db:eb:5b:29:e2:49:cd:39:39:9c:a4:ea:41:e8:c9:ee:66:7f:37:9b:ea:33:99:d9:45:3c:8a:db:bd:a1:bc:66:24:05 > RSS hash function: > ??? toeplitz: on > ??? xor: off > ??? crc32: off > > grep lan2 /proc/interrupts?? > ??? http://ix.io/3viC? (note, need very wide window) > > ifconfig bond_private > bond_private: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST>? mtu 9000 > ??????? inet 172.28.133.10? netmask 255.255.255.0? broadcast 172.28.133.255 > ??????? inet6 fe80::ae1f:6bff:fecf:20e0? prefixlen 64? scopeid 0x20<link> > ??????? ether ac:1f:6b:cf:20:e0? txqueuelen 1000? (Ethernet) > ??????? RX packets 54063955? bytes 326509018851 (304.0 GiB) > ??????? RX errors 0? dropped 5? overruns 0? frame 0 > ??????? TX packets 34841728? bytes 299047275167 (278.5 GiB) > ??????? TX errors 0? dropped 5 overruns 0? carrier 0? collisions 0 > > > PS C:\> Get-SmbClientNetworkInterface > > Interface Index RSS Capable RDMA Capable Speed??? > IpAddresses???????????????????????????????? Friendly Name > --------------- ----------- ------------ -----??? > -----------???????????????????????????????? ------------- > 14????????????? True??????? False??????? 100 Gbps {fe80::f0b7:6e80:9de4:df83, > 172.28.133.101} Ethernet 4 > > Please let me know if there is any other information I can provide to help me > get this figured out. > > ??? Chris
Am 08.08.21 um 00:43 schrieb Chris Dos via samba:> Each server has both of the 100G links bonded into a 802.3ad LAG.why? Maybe this is getting in the way. Multichannel is supposed to work with the "raw" interfaces. Sorry, but I don't really have the time to dig into this, but another thing I'd like to point out is that for Multichannel you should use 4.15, in previous versions it was marked experimental for a reason. Cheers! -slow -- Ralph Boehme, Samba Team https://samba.org/ SerNet Samba Team https://www.sernet.de/en/team-samba Samba Development and Support, SerNet Professional Services -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 840 bytes Desc: OpenPGP digital signature URL: <http://lists.samba.org/pipermail/samba/attachments/20210817/7612e980/OpenPGP_signature.sig>