netflow tools - Jan 2006 - Up to which amount of traffic does softflowd work properly?

Hello,

I would like to use softflowd for traffic accounting on a Gbit
interface. I already did some testing. I have a AMD Opteron machine with
2 GB RAM that runs Debian Sarge. The machine gets the traffic via a
mirrored port. At 270 Mbit (50,000 packets/sec) softflowd needs 80 % of
the CPU.

Does anyone use softflowd with more Mbit? Are there any possiblities for
a higher performance. I already have a patch for ignoring the ports --
so softflowd will open less flows. This is one possibility to improve
the performace. Are there any other ways to increase performance?

If softflowd isn''t high-performance enough to handle a Gbit interface
on
the abovementioned hardware, are there other tools that are designed for
such an amount of traffic? What are your experiences?

Greetings,
Andreas

On Mon, 30 Jan 2006, Andreas Brillisauer -- Hetzner Online AG wrote:
> Hello,
> 
> I would like to use softflowd for traffic accounting on a Gbit
> interface. I already did some testing. I have a AMD Opteron machine with
> 2 GB RAM that runs Debian Sarge. The machine gets the traffic via a
> mirrored port. At 270 Mbit (50,000 packets/sec) softflowd needs 80 % of
> the CPU.
> 
> Does anyone use softflowd with more Mbit? Are there any possiblities for
> a higher performance. I already have a patch for ignoring the ports --
> so softflowd will open less flows. This is one possibility to improve
> the performace. Are there any other ways to increase performance?
A simple way is to decreate the flow timeouts, so softflowd expires
flows more aggressively. The default ones are quite conservative. 

If there is certain traffic that you are not interested in, then you 
can pre-filter it using a bpf(4) program on softflowd''s command line.
There is no point in letting softflowd see traffic that you don''t care
to report on.

Beyond this, there are several internal optimisations that I have been 
planning on making that will speed up softflowd considerably. The top 
of the list is switching to a pool allocator for flows and associated 
expiry events rather than calling malloc(3) for each one.

If you are familiar with the GNU toolchain, building a profiled version
of softflowd and collecting a good profile would help direct these efforts
to areas that are most likely to be productive.

-d

On Tue, 31 Jan 2006, Damien Miller wrote:
> A simple way is to decreate the flow timeouts, so softflowd expires
"decrease", not "decreate" (a perfectly cromulent word, but
not what I meant).

-d