netflow tools - Jun 2007 - Thanks and request for flowd

First, thanks for a solid and simple flow collector.

I''ve been running it for about 6 months, and it''s been quite
useful.

I''m sure my usage is somewhat out of kilter with some of the intended
functions. Here is what I''ve been doing

    1) I collect everything. I''m not sure what I''ll need to
look at, so
       this seems the best policy.

    2) I use flowd-reader to report. The perl interface indicates it''s
       "just a thin wrapper" and I''ve not really looked at
the Python
       interface. I''ve ended up with shell.

	The details
	    
	    a)	I echo a filter list piped to flowd-reader with a -f flag
		to /dev/stdin. I wish there was a cleaner way to do
		ad-hoc filters.

	    b) I pipe the output to awk to select fields, and then sort
	       etc. to refine the output.


QUESTIONS/REQUESTS

    1) Is there a better way to pipe ad-hoc filters to flowd-reader (or
       another API).

    2) Can tagging improve filtering. It appears that tagging is a way
       to create meta-information for reporting, but I keep wondering if
       I can use it to create positive additive filters ("find me all
       the http traffic, then find me the https")  rather then negative
       filters (discard work fine cumulatively).

Joe
-- 

_______________________________________________________________________

Josef Fortier                             joe.fortier at simondelivers.com
Network Administrator                                    (763) 656-5650	
_______________________________________________________________________

On Fri, 29 Jun 2007, Josef Fortier wrote:
> QUESTIONS/REQUESTS
> 
>     1) Is there a better way to pipe ad-hoc filters to flowd-reader (or
>        another API).
What sort of filters are you after? I wouldn''t oppose allowing some
basic
commandline switches to filter by source/destination address or port.
>     2) Can tagging improve filtering. It appears that tagging is a way
>        to create meta-information for reporting, but I keep wondering if
>        I can use it to create positive additive filters ("find me all
>        the http traffic, then find me the https")  rather then
negative
>        filters (discard work fine cumulatively).
Yes, there have been other requesting that too. I haven''t been able to
figure out a syntax for flowd.conf that works well for additive/cumulative
tagging. The problem is that the filters are now "one match wins", but
cumulative tagging breaks that a little. Suggestions welcome!

-d

Hi Damien:

Thanks for the reply.
>    >     1) Is there a better way to pipe ad-hoc filters to flowd-reader
(or
>    >        another API).
>    
>    What sort of filters are you after? I wouldn''t oppose allowing
some basic
>    commandline switches to filter by source/destination address or port.
First, I should clarify my assumptions....

I''m rotating flowd files once a week. When I run the tools/ scripts
(wormsuspects.pl etc) they take a while. My assumption is that the time
reflects a linear search on the flowd file. Is this correct?

The flowd-reader lets me specify a config file, which in turn can
specify filters. This seems to have much better performance. Looking at
Flowd.pm it looks like there is no facility for filtering. In practice,
this appears to give me a large speed boost.

I''d suspect this because flowd-reader can throw away things much faster
then an external program...

Right now, I''m using short shell script to drive flowd-reader, and then
pump the output into something else (perl, awk) for further processing.

Example:

    echo "
	# Comment regarding filters
	discard quick before date ..........
	discard quick src ! [local-address-range]
	" | flowd-reader /dev/stdin	\
	  | perl -e ''
		# a program to do further processing
	    ''

This works OK, but the syntax is a little gunky (in particular the
/dev/stdin bothers me). 

One options is to attach stdin and read filters from there (a flag
maybe).

For the speed reasons, I''d like to be able to pass ad-hoc filter
statement lists into whatever reads the flowd file. My wish is to get
fast, relatively straightforward scripted reports out. It''s all about
reporting...


>    
>    >     2) Can tagging improve filtering. It appears that tagging is a
way
>    >        to create meta-information for reporting, but I keep
wondering if
>    >        I can use it to create positive additive filters ("find
me all
>    >        the http traffic, then find me the https")  rather then
negative
>    >        filters (discard work fine cumulatively).
>    
>    Yes, there have been other requesting that too. I haven''t been
able to
>    figure out a syntax for flowd.conf that works well for
additive/cumulative
>    tagging. The problem is that the filters are now "one match
wins", but
>    cumulative tagging breaks that a little. Suggestions welcome!

I''ve worked out how to get some of this from the filter statements.
Basically,

    1) Filter out all the negative stuff
    2) use "accept quick" to add to the positive filter
    3) discard everything

Example

    discard quick before date [date_number]
    accept quick src [address_range_ONE]
    accept quick src [address_range_TWO]
    discard any

This is limited in that the accept statements still need to be atomic,
no cumulative statements.


Thanks

Joe
-- 

_______________________________________________________________________

Josef Fortier
Network Administrator
_______________________________________________________________________