[sorry for some reason, the previous message is showing as empty in archives :(] Hi, I just wanted to try TCP BBR functionality available in FreeBSD 13.0-RC2 and noticed that it?s not usable, if FreeBSD pf is enabled. I?m using following pf rules: ==============block drop all block drop in quick on em0 from <badguys> to any block drop in quick on em0 from <sshguard> to any label "ssh bruteforce" pass in on em0 proto udp from any to any port = domain keep state pass in on em0 proto udp from any to any port = 2015 keep state pass in on em0 proto udp from any to any port = 5001 keep state pass in on em0 proto udp from any to any port = 6881 keep state pass in on em0 proto udp from any to any port = 51234 keep state pass in on em0 proto udp from any to any port 54000:54322 keep state pass in on em0 proto udp from any to any port 60000:61000 keep state pass in on em0 proto tcp from any to any port = ssh flags S/SA keep state pass in on em0 proto tcp from any to any port = xmpp-client flags S/SA keep state pass in on em0 proto tcp from any to any port = xmpp-server flags S/SA keep state pass in on em0 proto tcp from any to any port 54000:54322 flags S/SA keep state pass in on em0 proto icmp all keep state pass in on em0 proto ipv6-icmp all keep state pass out on em0 all flags S/SA keep state ============== Following is how I tried to reproduce (same with IPv6): ==============% wget -4 -O /dev/null https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.3.tar.xz --2021-03-13 09:26:23-- https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.3.tar.xz Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.113.176 Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.113.176|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 117629864 (112M) [application/x-xz] Saving to: ?/dev/null? /dev/null 0%[ ] 137.44K --.-KB/s in 10s 2021-03-13 09:26:33 (13.8 KB/s) - Read error at byte 140737/117629864 (Network is down). Retrying. --2021-03-13 09:26:34-- (try: 2) https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.3.tar.xz Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.113.176|:443... connected. HTTP request sent, awaiting response... 206 Partial Content Length: 117629864 (112M), 117489127 (112M) remaining [application/x-xz] Saving to: ?/dev/null? /dev/null 0%[ ] 243.21K --.-KB/s in 10s 2021-03-13 09:26:44 (10.6 KB/s) - Read error at byte 249046/117629864 (Network is down). Retrying. ============== A copy of tcpdump output (tcpdump -i em0 -nv port 443 and host 151.101.113.176) is uploaded[1]. Since TCP BBR requires custom kernel, following is the diff: ==============--- GENERIC 2021-02-23 10:55:03.397376000 +0000 +++ MYKERNEL 2021-02-23 10:58:39.442981000 +0000 @@ -19,7 +19,7 @@ # $FreeBSD$ cpu HAMMER -ident GENERIC +ident MYKERNEL makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols makeoptions WITH_CTF=1 # Run ctfconvert(1) for DTrace support @@ -382,3 +382,6 @@ options HID_DEBUG # enable debug msgs device hid # Generic HID support options IICHID_SAMPLING # Workaround missing GPIO INTR support + +options ROUTETABLES=12 +options TCPHPTS ============== And following src.conf: ==============WITH_EXTRA_TCP_STACKS= yes ============== sysctl.conf(5) is empty, except following sysctl knob set to switch to BBR stack: ==============net.inet.tcp.functions_default=bbr ============== Following are the contents of loader.conf(5): ==============kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" opensolaris_load="YES" zfs_load="YES" debug.acpi.disabled="thermal" coretemp_load=YES vm.pmap.pti=0 cpu_microcode_load="YES" cpu_microcode_name="/boot/firmware/intel-ucode.bin" aesni_load=YES nullfs_load="YES" vfs.zfs.prefetch_disable=1 vfs.zfs.write_limit_override=1073741824 vfs.zfs.min_auto_ashift=12 vfs.zfs.arc_max="17179869184" vfs.zfs.compressed_arc_enabled=1 ============== Following is the output of kldstat: ==============Id Refs Address Size Name 1 33 0xffffffff80200000 1f12e50 kernel 2 1 0xffffffff82113000 b7b8 opensolaris.ko 3 1 0xffffffff8211f000 9ac0 nullfs.ko 4 1 0xffffffff8249d000 67fbc8 zfs.ko 5 1 0xffffffff82b1d000 4128 coretemp.ko 6 1 0xffffffff82d20000 3378 acpi_wmi.ko 7 1 0xffffffff82d24000 3250 ichsmb.ko 8 1 0xffffffff82d28000 2180 smbus.ko 9 1 0xffffffff82d2b000 2110 pchtherm.ko 10 1 0xffffffff82d2e000 2a08 mac_ntpd.ko 11 1 0xffffffff82d31000 29bc8 tcp_bbr.ko 12 1 0xffffffff82d5b000 36f70 pf.ko ============== Host is running kernel from git revision ?13c22f7495305f5b92874128b088ab47d9512c20?, while rest of the userland is from "freebsd-update upgrade -r 13.0-RC2?. After disabling pf firewall, it becomes usable again: ==============--2021-03-13 09:42:50-- https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.3.tar.xz Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.113.176 Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.113.176|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 117629864 (112M) [application/x-xz] Saving to: ?/dev/null? /dev/null 100%[======================================================================================================>] 112.18M 20.2MB/s in 6.9s 2021-03-13 09:42:57 (16.3 MB/s) - ?/dev/null? saved [117629864/117629864] ============== Please let me know if I overlooked something with TCP BBR configuration. References: [1] https://people.freebsd.org/~ashish/tcpdump-cdn-kernel-org-20210313.txt Thanks! -- Ashish | GPG: F682 CDCC 39DC 0FEA E116 20B6 C746 CFA9 E74F A4B0 ?Sometimes even to live is an act of courage.? (Seneca)