thr3ads.net - freebsd stable - 10.1-BETA2 possible kernel memory leak in routing table [Sep 2014]

If this information is useful, please help other people find it:
Share via:

Rumen Telbizov

2014-Sep-29 18:58 UTC

10.1-BETA2 possible kernel memory leak in routing table

Hello everyone,

I have recently upgraded one of our main firewalls from FreeBSD 9.2
to10.1-BETA2 (r271983). It has been running in production since last
Thursday and overall I am pretty happy with it and seeing significant
improvements in performance.

However I do notice this morning that the *wired* portion of the memory has
been steadily growing over the weekend and is now over a few gigabytes.
Here's a graph:

[image: Inline image 2]

The switchover was done on late 25th when production traffic started
flowing through it. I checked vmstat -m and noticed that the *routetbl* is
pretty large - 2.8GB and keeps on growing. Here's a few minutes worth of
monitoring it:


 # while true; do date; vmstat -m | grep routetbl; sleep 60; done

         Type InUse   MemUse HighUse Requests  Size(s)
Mon Sep 29 18:27:55 UTC 2014
     routetbl 5988792 2888491K       - 14285826  32,64,128,256,512,2048
Mon Sep 29 18:28:55 UTC 2014
     routetbl 5990120 2889131K       - 14288972  32,64,128,256,512,2048
Mon Sep 29 18:29:55 UTC 2014
     routetbl 5991448 2889771K       - 14292352  32,64,128,256,512,2048
Mon Sep 29 18:30:55 UTC 2014
     routetbl 5992776 2890411K       - 14295464  32,64,128,256,512,2048
Mon Sep 29 18:31:55 UTC 2014
     routetbl 5994104 2891051K       - 14298576  32,64,128,256,512,2048
Mon Sep 29 18:32:55 UTC 2014
     routetbl 5995432 2891691K       - 14301904  32,64,128,256,512,2048
Mon Sep 29 18:33:55 UTC 2014
     routetbl 5996096 2892011K       - 14303624  32,64,128,256,512,2048
Mon Sep 29 18:34:55 UTC 2014
     routetbl 5997422 2892650K       - 14306980  32,64,128,256,512,2048
Mon Sep 29 18:35:55 UTC 2014
     routetbl 5998750 2893290K       - 14310092  32,64,128,256,512,2048
Mon Sep 29 18:36:55 UTC 2014
     routetbl 6000078 2893930K       - 14313204  32,64,128,256,512,2048
Mon Sep 29 18:37:55 UTC 2014
     routetbl 6001406 2894570K       - 14316532  32,64,128,256,512,2048
Mon Sep 29 18:38:55 UTC 2014
     routetbl 6002734 2895210K       - 14319644  32,64,128,256,512,2048
Mon Sep 29 18:39:55 UTC 2014
     routetbl 6004062 2895850K       - 14323024  32,64,128,256,512,2048
Mon Sep 29 18:40:56 UTC 2014
     routetbl 6004726 2896170K       - 14324745  32,64,128,256,512,2048
Mon Sep 29 18:41:56 UTC 2014
     routetbl 6006054 2896810K       - 14327857  32,64,128,256,512,2048
Mon Sep 29 18:42:56 UTC 2014
     routetbl 6007382 2897450K       - 14331185  32,64,128,256,512,2048
Mon Sep 29 18:43:56 UTC 2014
     routetbl 6008710 2898090K       - 14334297  32,64,128,256,512,2048

As you can see for this interval it grew 9599K (2898090-2888491). The
routing table does not change at all on this machine. netstat -rn | wc -l
reports 145 entries all the time. As a comparison on the old, inactive
FreeBSD 9.2 firewall I have:

# vmstat -m | grep routetbl
     routetbl   451    98K       -  7336488  32,64,128,256,512,1024

Here are some configuration details of the system.

loader.conf:
autoboot_delay="3"
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot/freebsd10"
vfs.zfs.arc_max="500M"
kern.geom.label.gptid.enable="0"
hw.ix.num_queues="12"
hw.ix.txd="4096"
hw.ix.rxd="4096"

sysctl.conf:
# CARP related
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=2

# Stop sending ICMP redirects
net.inet.ip.redirect=0

# Fast Forwarding
net.inet.ip.fastforwarding=1

# Enable better lacp/lagg transmit distribution
net.link.lagg.default_use_flowid=0


I could provide additional debug information upon request. Your help is
appreciated.

Regards,
-- 
Rumen Telbizov
Unix Systems Administrator <http://telbizov.com>

Alexander V. Chernikov

2014-Sep-30 13:03 UTC

head link

10.1-BETA2 possible kernel memory leak in routing table

On 29.09.2014 22:58, Rumen Telbizov wrote:> Hello everyone,
>
> I have recently upgraded one of our main firewalls from FreeBSD 9.2
> to10.1-BETA2 (r271983). It has been running in production since last
> Thursday and overall I am pretty happy with it and seeing significant
> improvements in performance.
>
> However I do notice this morning that the *wired* portion of the memory has
> been steadily growing over the weekend and is now over a few gigabytes.
> Here's a graph:
>
> [image: Inline image 2]
>
> The switchover was done on late 25th when production traffic started
> flowing through it. I checked vmstat -m and noticed that the *routetbl* is
> pretty large - 2.8GB and keeps on growing. Here's a few minutes worth
of
> monitoring it:
>
>
>   # while true; do date; vmstat -m | grep routetbl; sleep 60; done
>
>           Type InUse   MemUse HighUse Requests  Size(s)
> Mon Sep 29 18:27:55 UTC 2014
>       routetbl 5988792 2888491K       - 14285826  32,64,128,256,512,2048
> Mon Sep 29 18:28:55 UTC 2014
>       routetbl 5990120 2889131K       - 14288972  32,64,128,256,512,2048
> Mon Sep 29 18:29:55 UTC 2014
>       routetbl 5991448 2889771K       - 14292352  32,64,128,256,512,2048
> Mon Sep 29 18:30:55 UTC 2014
>       routetbl 5992776 2890411K       - 14295464  32,64,128,256,512,2048
> Mon Sep 29 18:31:55 UTC 2014
>       routetbl 5994104 2891051K       - 14298576  32,64,128,256,512,2048
> Mon Sep 29 18:32:55 UTC 2014
>       routetbl 5995432 2891691K       - 14301904  32,64,128,256,512,2048
> Mon Sep 29 18:33:55 UTC 2014
>       routetbl 5996096 2892011K       - 14303624  32,64,128,256,512,2048
> Mon Sep 29 18:34:55 UTC 2014
>       routetbl 5997422 2892650K       - 14306980  32,64,128,256,512,2048
> Mon Sep 29 18:35:55 UTC 2014
>       routetbl 5998750 2893290K       - 14310092  32,64,128,256,512,2048
> Mon Sep 29 18:36:55 UTC 2014
>       routetbl 6000078 2893930K       - 14313204  32,64,128,256,512,2048
> Mon Sep 29 18:37:55 UTC 2014
>       routetbl 6001406 2894570K       - 14316532  32,64,128,256,512,2048
> Mon Sep 29 18:38:55 UTC 2014
>       routetbl 6002734 2895210K       - 14319644  32,64,128,256,512,2048
> Mon Sep 29 18:39:55 UTC 2014
>       routetbl 6004062 2895850K       - 14323024  32,64,128,256,512,2048
> Mon Sep 29 18:40:56 UTC 2014
>       routetbl 6004726 2896170K       - 14324745  32,64,128,256,512,2048
> Mon Sep 29 18:41:56 UTC 2014
>       routetbl 6006054 2896810K       - 14327857  32,64,128,256,512,2048
> Mon Sep 29 18:42:56 UTC 2014
>       routetbl 6007382 2897450K       - 14331185  32,64,128,256,512,2048
> Mon Sep 29 18:43:56 UTC 2014
>       routetbl 6008710 2898090K       - 14334297  32,64,128,256,512,2048
>
> As you can see for this interval it grew 9599K (2898090-2888491). The
> routing table does not change at all on this machine. netstat -rn | wc -l
> reports 145 entries all the time. As a comparison on the old, inactive
> FreeBSD 9.2 firewall I have:
>
> # vmstat -m | grep routetbl
>       routetbl   451    98K       -  7336488  32,64,128,256,512,1024Can you show route -Arn output ?

Can you disable carp and see if this changes anything?>
> Here are some configuration details of the system.
>
> loader.conf:
> autoboot_delay="3"
> zfs_load="YES"
> vfs.root.mountfrom="zfs:zroot/freebsd10"
> vfs.zfs.arc_max="500M"
> kern.geom.label.gptid.enable="0"
> hw.ix.num_queues="12"
> hw.ix.txd="4096"
> hw.ix.rxd="4096"
>
> sysctl.conf:
> # CARP related
> net.inet.carp.allow=1
> net.inet.carp.preempt=1
> net.inet.carp.log=2
>
> # Stop sending ICMP redirects
> net.inet.ip.redirect=0
>
> # Fast Forwarding
> net.inet.ip.fastforwarding=1
>
> # Enable better lacp/lagg transmit distribution
> net.link.lagg.default_use_flowid=0
Are you sure this is really "better" ?>
>
> I could provide additional debug information upon request. Your help is
> appreciated.
>
> Regards,

freebsd stable - Sep 2014 - 10.1-BETA2 possible kernel memory leak in routing table

10.1-BETA2 possible kernel memory leak in routing table

10.1-BETA2 possible kernel memory leak in routing table