Andrew McGill
2009-Aug-03 15:48 UTC
[netflow-tools] softflowd -m 512000 ... flow-capture ... 90 % less traffic??
Greetings netflow-tools,
I have softflowd sending information to flow-capture for a network with a few
hundred hosts (don''t ask, the answer is probably "yes").
Softflowd was
configured with the default without a -m parameter, so that softflowd tracked
a maximum of 8192 flows. The primary reason for rolling over flows was
running out of connections - and cpu load was obnoxiously high. So I fixed it
(in the sense of thereifixedit.com, perhaps). I told softflowd that it should
track a maximum of 512000 flows, and it duly did.
The before and after log files for 10 minutes of traffic look like this:
-rw-r--r-- 1 root root 12678211 Jul 26 17:02 ft-v05.2009-07-26.165257+0200
-rw-r--r-- 1 root root 673952 Jul 26 17:32 ft-v05.2009-07-26.172247+0200
... which is great, BUT it seems that most of the traffic is getting lost.
It''s not that this traffic is getting deferred into later stats -- it
simply
never gets reported -- the reported totals dropped to 10% of their previous
values!
before: Average Kbits / second (real) : 49598.9333
after: Average Kbits / second (real) : 3872.6817
The next day it was still roughly 10% of the real amount:
Average Kbits / second (real) : 4617.1089
Is this correct behaviour? Am I doing one or more things wrong?
&:-)
Notes:
Startup parameters:
flow-capture -p /var/run/flow-capture.pid -n 144 -N -1 \
-w /var/log/netflows -S 10 0/0/8828
softflowd -i eth2 -n 127.0.0.1:8828 # BEFORE
softflowd -i eth2 -n 127.0.0.1:8828 -m 512000 # AFTER
In case it''s relevant, this is what flow-stat said about the files:
# --- ---- ---- Report Information --- --- --- (BEFORE)
#
# Fields: Total
# Symbols: Disabled
# Sorting: None
# Name: Overall Summary
#
# Args: flow-stat
#
Total Flows : 723704
Total Octets : 2975935893
Total Packets : 6138299
Total Time (1/1000 secs) (flows): 5790296389
Duration of data (realtime) : 480
Duration of data (1/1000 secs) : 2363291
Average flow time (1/1000 secs) : 8000.9183
Average packet size (octets) : 484.8144
Average flow size (octets) : 4112.0900
Average packets per flow : 8.4818
Average flows / second (flow) : 306.2649
Average flows / second (real) : 1507.7167
Average Kbits / second (flow) : 10075.1113
Average Kbits / second (real) : 49598.9333
IP packet size distribution:
1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480
.000 .379 .286 .090 .082 .044 .024 .021 .013 .007 .005 .003 .003 .001 .002
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
.003 .005 .001 .014 .018 .000 .000 .000 .000 .000 .000
Packets per flow distribution:
1 2 4 8 12 16 20 24 28 32 36 40 44 48 52
.643 .086 .075 .097 .030 .018 .011 .007 .005 .004 .003 .002 .002 .002 .001
60 100 200 300 400 500 600 700 800 900 >900
.002 .005 .003 .001 .001 .000 .000 .000 .000 .000 .001
Octets per flow distribution:
32 64 128 256 512 1280 2048 2816 3584 4352 5120 5888 6656 7424 8192
.000 .241 .298 .191 .104 .082 .022 .011 .007 .004 .004 .002 .002 .002 .002
8960 9728 10496 11264 12032 12800 13568 14336 15104 15872 >15872
.001 .001 .001 .001 .001 .001 .001 .001 .001 .001 .019
Flow time distribution:
10 50 100 200 500 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
.733 .014 .013 .024 .035 .032 .025 .017 .010 .008 .016 .007 .005 .005 .004
12000 14000 16000 18000 20000 22000 24000 26000 28000 30000 >30000
.006 .004 .003 .003 .002 .004 .002 .001 .001 .001 .023
# --- ---- ---- Report Information --- --- --- (AFTER)
#
# Fields: Total
# Symbols: Disabled
# Sorting: None
# Name: Overall Summary
#
# Args: flow-stat
#
Total Flows : 50516
Total Octets : 261406012
Total Packets : 551158
Total Time (1/1000 secs) (flows): 329152148
Duration of data (realtime) : 540
Duration of data (1/1000 secs) : 1366814
Average flow time (1/1000 secs) : 6515.8001
Average packet size (octets) : 474.2851
Average flow size (octets) : 5174.7172
Average packets per flow : 10.9106
Average flows / second (flow) : 36.9810
Average flows / second (real) : 93.5481
Average Kbits / second (flow) : 1530.9283
Average Kbits / second (real) : 3872.6817
IP packet size distribution:
1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480
.000 .205 .364 .125 .116 .053 .028 .020 .019 .009 .005 .004 .003 .002 .002
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
.003 .008 .001 .015 .019 .000 .000 .000 .000 .000 .000
Packets per flow distribution:
1 2 4 8 12 16 20 24 28 32 36 40 44 48 52
.439 .174 .073 .119 .067 .034 .018 .015 .010 .008 .006 .004 .003 .004 .002
60 100 200 300 400 500 600 700 800 900 >900
.004 .010 .005 .001 .001 .000 .000 .000 .000 .000 .001
Octets per flow distribution:
32 64 128 256 512 1280 2048 2816 3584 4352 5120 5888 6656 7424 8192
.000 .059 .297 .200 .153 .154 .042 .022 .011 .006 .004 .003 .002 .004 .003
8960 9728 10496 11264 12032 12800 13568 14336 15104 15872 >15872
.002 .002 .002 .001 .001 .001 .001 .001 .001 .000 .027
Flow time distribution:
10 50 100 200 500 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
.298 .024 .012 .073 .319 .071 .035 .024 .012 .010 .020 .009 .007 .006 .005
12000 14000 16000 18000 20000 22000 24000 26000 28000 30000 >30000
.009 .005 .003 .004 .004 .003 .003 .003 .003 .004 .035
Sebastian Majkowski
2009-Aug-14 08:09 UTC
[netflow-tools] softflowd -m 512000 ... flow-capture ... 90 % less traffic??
Andrew McGill wrote:> Greetings netflow-tools, > > I have softflowd sending information to flow-capture for a network with a few > hundred hosts (don''t ask, the answer is probably "yes"). Softflowd was > configured with the default without a -m parameter, so that softflowd tracked > a maximum of 8192 flows. The primary reason for rolling over flows was > running out of connections - and cpu load was obnoxiously high. So I fixed it > (in the sense of thereifixedit.com, perhaps). I told softflowd that it should > track a maximum of 512000 flows, and it duly did. > > The before and after log files for 10 minutes of traffic look like this: > > -rw-r--r-- 1 root root 12678211 Jul 26 17:02 ft-v05.2009-07-26.165257+0200 > -rw-r--r-- 1 root root 673952 Jul 26 17:32 ft-v05.2009-07-26.172247+0200 > > ... which is great, BUT it seems that most of the traffic is getting lost. > It''s not that this traffic is getting deferred into later stats -- it simply > never gets reported -- the reported totals dropped to 10% of their previous > values! > > before: Average Kbits / second (real) : 49598.9333 > after: Average Kbits / second (real) : 3872.6817 > > The next day it was still roughly 10% of the real amount: > > Average Kbits / second (real) : 4617.1089 > > Is this correct behaviour? Am I doing one or more things wrong? > > &:-) > > > > Notes: > > Startup parameters: > flow-capture -p /var/run/flow-capture.pid -n 144 -N -1 \ > -w /var/log/netflows -S 10 0/0/8828 > > softflowd -i eth2 -n 127.0.0.1:8828 # BEFORE > softflowd -i eth2 -n 127.0.0.1:8828 -m 512000 # AFTER > > > In case it''s relevant, this is what flow-stat said about the files: > > > # --- ---- ---- Report Information --- --- --- (BEFORE) > # > # Fields: Total > # Symbols: Disabled > # Sorting: None > # Name: Overall Summary > # > # Args: flow-stat > # > Total Flows : 723704 > Total Octets : 2975935893 > Total Packets : 6138299 > Total Time (1/1000 secs) (flows): 5790296389 > Duration of data (realtime) : 480 > Duration of data (1/1000 secs) : 2363291 > Average flow time (1/1000 secs) : 8000.9183 > Average packet size (octets) : 484.8144 > Average flow size (octets) : 4112.0900 > Average packets per flow : 8.4818 > Average flows / second (flow) : 306.2649 > Average flows / second (real) : 1507.7167 > Average Kbits / second (flow) : 10075.1113 > Average Kbits / second (real) : 49598.9333 > > > IP packet size distribution: > 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 > .000 .379 .286 .090 .082 .044 .024 .021 .013 .007 .005 .003 .003 .001 .002 > > 512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 > .003 .005 .001 .014 .018 .000 .000 .000 .000 .000 .000 > > Packets per flow distribution: > 1 2 4 8 12 16 20 24 28 32 36 40 44 48 52 > .643 .086 .075 .097 .030 .018 .011 .007 .005 .004 .003 .002 .002 .002 .001 > > 60 100 200 300 400 500 600 700 800 900 >900 > .002 .005 .003 .001 .001 .000 .000 .000 .000 .000 .001 > > Octets per flow distribution: > 32 64 128 256 512 1280 2048 2816 3584 4352 5120 5888 6656 7424 8192 > .000 .241 .298 .191 .104 .082 .022 .011 .007 .004 .004 .002 .002 .002 .002 > > 8960 9728 10496 11264 12032 12800 13568 14336 15104 15872 >15872 > .001 .001 .001 .001 .001 .001 .001 .001 .001 .001 .019 > > Flow time distribution: > 10 50 100 200 500 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 > .733 .014 .013 .024 .035 .032 .025 .017 .010 .008 .016 .007 .005 .005 .004 > > 12000 14000 16000 18000 20000 22000 24000 26000 28000 30000 >30000 > .006 .004 .003 .003 .002 .004 .002 .001 .001 .001 .023 > > # --- ---- ---- Report Information --- --- --- (AFTER) > # > # Fields: Total > # Symbols: Disabled > # Sorting: None > # Name: Overall Summary > # > # Args: flow-stat > # > Total Flows : 50516 > Total Octets : 261406012 > Total Packets : 551158 > Total Time (1/1000 secs) (flows): 329152148 > Duration of data (realtime) : 540 > Duration of data (1/1000 secs) : 1366814 > Average flow time (1/1000 secs) : 6515.8001 > Average packet size (octets) : 474.2851 > Average flow size (octets) : 5174.7172 > Average packets per flow : 10.9106 > Average flows / second (flow) : 36.9810 > Average flows / second (real) : 93.5481 > Average Kbits / second (flow) : 1530.9283 > Average Kbits / second (real) : 3872.6817 > > > IP packet size distribution: > 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 > .000 .205 .364 .125 .116 .053 .028 .020 .019 .009 .005 .004 .003 .002 .002 > > 512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 > .003 .008 .001 .015 .019 .000 .000 .000 .000 .000 .000 > > Packets per flow distribution: > 1 2 4 8 12 16 20 24 28 32 36 40 44 48 52 > .439 .174 .073 .119 .067 .034 .018 .015 .010 .008 .006 .004 .003 .004 .002 > > 60 100 200 300 400 500 600 700 800 900 >900 > .004 .010 .005 .001 .001 .000 .000 .000 .000 .000 .001 > > Octets per flow distribution: > 32 64 128 256 512 1280 2048 2816 3584 4352 5120 5888 6656 7424 8192 > .000 .059 .297 .200 .153 .154 .042 .022 .011 .006 .004 .003 .002 .004 .003 > > 8960 9728 10496 11264 12032 12800 13568 14336 15104 15872 >15872 > .002 .002 .002 .001 .001 .001 .001 .001 .001 .000 .027 > > Flow time distribution: > 10 50 100 200 500 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 > .298 .024 .012 .073 .319 .071 .035 .024 .012 .010 .020 .009 .007 .006 .005 > > 12000 14000 16000 18000 20000 22000 24000 26000 28000 30000 >30000 > .009 .005 .003 .004 .004 .003 .003 .003 .003 .004 .035 > > > > > _______________________________________________ > netflow-tools mailing list > netflow-tools at mindrot.org > https://lists.mindrot.org/mailman/listinfo/netflow-tools > > > >Hi Andrew, I am not an expert in net-flow but I had similar situation. I guess that this is the result of increasing max flows tracked - so less frequently netflow records will be created (and smaller file). When maximum 8192 flows was reached probably softflowd just ends some flows creating records to manage other flows, thats why the file is bigger. The same situation is when manipulating timers - this allows you to decide when (or how long) the flow is tracked before creating netflow record... Maybe it would be good to trace some connections from one of you users to see how they are placed in netflow records - this will prove if the data is tracked or not (as you suspect) Regards, Sebastian
Andrew McGill
2009-Aug-25 10:35 UTC
[netflow-tools] softflowd -m 512000 ... flow-capture ... 90 % less traffic??
On Friday 14 August 2009 10:09:35 Sebastian Majkowski wrote:> Andrew McGill wrote: > > Greetings netflow-tools, > > > > I have softflowd sending information to flow-capture for a network with a > > few hundred hosts (don''t ask, the answer is probably "yes"). Softflowd > > was configured with the default without a -m parameter, so that softflowd > > tracked a maximum of 8192 flows. The primary reason for rolling over > > flows was running out of connections - and cpu load was obnoxiously high. > > So I fixed it (in the sense of thereifixedit.com, perhaps). I told > > softflowd that it should track a maximum of 512000 flows, and it duly > > did. > > > > The before and after log files for 10 minutes of traffic look like this: > > > > -rw-r--r-- 1 root root 12678211 Jul 26 17:02 > > ft-v05.2009-07-26.165257+0200 -rw-r--r-- 1 root root 673952 Jul 26 > > 17:32 ft-v05.2009-07-26.172247+0200 > > > > ... which is great, BUT it seems that most of the traffic is getting > > lost. It''s not that this traffic is getting deferred into later stats -- > > it simply never gets reported -- the reported totals dropped to 10% of > > their previous values! > > > > before: Average Kbits / second (real) : 49598.9333 > > after: Average Kbits / second (real) : 3872.6817 > > > > The next day it was still roughly 10% of the real amount: > > > > Average Kbits / second (real) : 4617.1089 > > > > Is this correct behaviour? Am I doing one or more things wrong? > > > > &:-) > > > > > > > > Notes: > > > > Startup parameters: > > flow-capture -p /var/run/flow-capture.pid -n 144 -N -1 \ > > -w /var/log/netflows -S 10 0/0/8828 > > > > softflowd -i eth2 -n 127.0.0.1:8828 # BEFORE > > softflowd -i eth2 -n 127.0.0.1:8828 -m 512000 # AFTER > > > > > > In case it''s relevant, this is what flow-stat said about the files: > > > > > > # --- ---- ---- Report Information --- --- --- (BEFORE) > > # > > # Fields: Total > > # Symbols: Disabled > > # Sorting: None > > # Name: Overall Summary > > # > > # Args: flow-stat > > # > > Total Flows : 723704 > > Total Octets : 2975935893 > > Total Packets : 6138299 > > Total Time (1/1000 secs) (flows): 5790296389 > > Duration of data (realtime) : 480 > > Duration of data (1/1000 secs) : 2363291 > > Average flow time (1/1000 secs) : 8000.9183 > > Average packet size (octets) : 484.8144 > > Average flow size (octets) : 4112.0900 > > Average packets per flow : 8.4818 > > Average flows / second (flow) : 306.2649 > > Average flows / second (real) : 1507.7167 > > Average Kbits / second (flow) : 10075.1113 > > Average Kbits / second (real) : 49598.9333 > > > > > > IP packet size distribution: > > 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 > > 480 .000 .379 .286 .090 .082 .044 .024 .021 .013 .007 .005 .003 .003 .001 > > .002 > > > > 512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 > > .003 .005 .001 .014 .018 .000 .000 .000 .000 .000 .000 > > > > Packets per flow distribution: > > 1 2 4 8 12 16 20 24 28 32 36 40 44 48 > > 52 .643 .086 .075 .097 .030 .018 .011 .007 .005 .004 .003 .002 .002 .002 > > .001 > > > > 60 100 200 300 400 500 600 700 800 900 >900 > > .002 .005 .003 .001 .001 .000 .000 .000 .000 .000 .001 > > > > Octets per flow distribution: > > 32 64 128 256 512 1280 2048 2816 3584 4352 5120 5888 6656 7424 > > 8192 .000 .241 .298 .191 .104 .082 .022 .011 .007 .004 .004 .002 .002 > > .002 .002 > > > > 8960 9728 10496 11264 12032 12800 13568 14336 15104 15872 >15872 > > .001 .001 .001 .001 .001 .001 .001 .001 .001 .001 .019 > > > > Flow time distribution: > > 10 50 100 200 500 1000 2000 3000 4000 5000 6000 7000 8000 9000 > > 10000 .733 .014 .013 .024 .035 .032 .025 .017 .010 .008 .016 .007 .005 > > .005 .004 > > > > 12000 14000 16000 18000 20000 22000 24000 26000 28000 30000 >30000 > > .006 .004 .003 .003 .002 .004 .002 .001 .001 .001 .023 > > > > # --- ---- ---- Report Information --- --- --- (AFTER) > > # > > # Fields: Total > > # Symbols: Disabled > > # Sorting: None > > # Name: Overall Summary > > # > > # Args: flow-stat > > # > > Total Flows : 50516 > > Total Octets : 261406012 > > Total Packets : 551158 > > Total Time (1/1000 secs) (flows): 329152148 > > Duration of data (realtime) : 540 > > Duration of data (1/1000 secs) : 1366814 > > Average flow time (1/1000 secs) : 6515.8001 > > Average packet size (octets) : 474.2851 > > Average flow size (octets) : 5174.7172 > > Average packets per flow : 10.9106 > > Average flows / second (flow) : 36.9810 > > Average flows / second (real) : 93.5481 > > Average Kbits / second (flow) : 1530.9283 > > Average Kbits / second (real) : 3872.6817 > > > > > > IP packet size distribution: > > 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 > > 480 .000 .205 .364 .125 .116 .053 .028 .020 .019 .009 .005 .004 .003 .002 > > .002 > > > > 512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 > > .003 .008 .001 .015 .019 .000 .000 .000 .000 .000 .000 > > > > Packets per flow distribution: > > 1 2 4 8 12 16 20 24 28 32 36 40 44 48 > > 52 .439 .174 .073 .119 .067 .034 .018 .015 .010 .008 .006 .004 .003 .004 > > .002 > > > > 60 100 200 300 400 500 600 700 800 900 >900 > > .004 .010 .005 .001 .001 .000 .000 .000 .000 .000 .001 > > > > Octets per flow distribution: > > 32 64 128 256 512 1280 2048 2816 3584 4352 5120 5888 6656 7424 > > 8192 .000 .059 .297 .200 .153 .154 .042 .022 .011 .006 .004 .003 .002 > > .004 .003 > > > > 8960 9728 10496 11264 12032 12800 13568 14336 15104 15872 >15872 > > .002 .002 .002 .001 .001 .001 .001 .001 .001 .000 .027 > > > > Flow time distribution: > > 10 50 100 200 500 1000 2000 3000 4000 5000 6000 7000 8000 9000 > > 10000 .298 .024 .012 .073 .319 .071 .035 .024 .012 .010 .020 .009 .007 > > .006 .005 > > > > 12000 14000 16000 18000 20000 22000 24000 26000 28000 30000 >30000 > > .009 .005 .003 .004 .004 .003 .003 .003 .003 .004 .035 > > > > > > > > > > _______________________________________________ > > netflow-tools mailing list > > netflow-tools at mindrot.org > > https://lists.mindrot.org/mailman/listinfo/netflow-tools > > Hi Andrew, > > I am not an expert in net-flow but I had similar situation. I guess that > this is the result of increasing max flows tracked - so less frequently > netflow records will be created (and smaller file). When maximum 8192 > flows was reached probably softflowd just ends some flows creating > records to manage other flows, thats why the file is bigger. The same > situation is when manipulating timers - this allows you to decide when > (or how long) the flow is tracked before creating netflow record...Well, I have worked around the problem by using a smaller connection buffer, and setting a timeout of 59 seconds for anything to make sure that something gets logged: /usr/local/sbin/softflowd -i eth0 -n 127.0.0.1:8818 -m 65536 -t maxlife=59 -t general=59 Flows over 1 minutes -- well, that''s just tough. We''ll count it as multiple flows.> Maybe it would be good to trace some connections from one of you users > to see how they are placed in netflow records - this will prove if the > data is tracked or not (as you suspect)like a regression test ... hmm.. Yes, a good idea .. hmm. I wonder what the correct test is ... something like this ? dd if=/dev/zero bs=1024 count=10 | netcat somewhere.com 80 expect a flow of ... um ? 10 kb plus a bit? (it''s a touchy-feely test ...) &:-)
Damien Miller
2009-Sep-03 08:22 UTC
[netflow-tools] softflowd -m 512000 ... flow-capture ... 90 % less traffic??
On Mon, 3 Aug 2009, Andrew McGill wrote:> ... which is great, BUT it seems that most of the traffic is getting lost. > It''s not that this traffic is getting deferred into later stats -- it simply > never gets reported -- the reported totals dropped to 10% of their previous > values! > > before: Average Kbits / second (real) : 49598.9333 > after: Average Kbits / second (real) : 3872.6817 > > The next day it was still roughly 10% of the real amount: > > Average Kbits / second (real) : 4617.1089 > > Is this correct behaviour? Am I doing one or more things wrong?Long-running flows may live in softflowd''s flow tree indefinitely unless you specify a maxiumum lifetime (-t maxlife=N). If they don''t expire then they will not be exported and therefore won''t appear in your summarisation. 5 minutes is probably a good starting value. -d