netflow tools - Sep 2007 - Flowd-0.9 Python Library Problem

Hi,
So I''m writing a small python program that massages duplicate flows out
of a stream of NetFlow exports and allows one to redirect these flows to
arbitrary locations.
I''m using softflowd on FreeBSD to monitor several links, and export in
v9 format to a different FreeBSD machine.
I''m using the flowd python module to parse the netflow records. They
come in on a UDP port, I pass them to flowd.Flow()...and that''s where
everything explodes.
Softflowd is set to export v9 flows.
Wireshark says these are v9 flows.
flowd.Flow() explodes with:
Traceback (most recent call last):
  File "nfagro.py", line 105, in ?
    main()
  File "nfagro.py", line 84, in main
    msg = NetflowRecord(msg)
  File "nfagro.py", line 24, in __init__
    self.nf = flowd.Flow(blob=msg)
ValueError: Unsupported version

And this is being pulled from (msg, sndaddr) = listensocket.recvfrom(10240).
When I look at the data in msg, too, the first two octets are 0x0009. So...could
this be an endianness issue? Some other crazy thing?

Cheers,
-Jesse Kempf



------------------------------------------------------------------------
The information contained in this communication is intended
only for the use of the recipient(s) named above. It may
contain information that is privileged or confidential, and
may be protected by State and/or Federal Regulations. If
the reader of this message is not the intended recipient,
you are hereby notified that any dissemination,
distribution, or copying of this communication, or any of
its contents, is strictly prohibited. If you have received
this communication in error, please return it to the sender
immediately and delete the original message and any copy
of it from your computer system. If you have any questions
concerning this message, please contact the sender.
------------------------------------------------------------------------

On Sun, 16 Sep 2007, Jesse Kempf wrote:
> Hi,
> So I''m writing a small python program that massages duplicate
flows
> out of a stream of NetFlow exports and allows one to redirect these
> flows to arbitrary locations. I''m using softflowd on FreeBSD to
> monitor several links, and export in v9 format to a different FreeBSD
> machine. I''m using the flowd python module to parse the netflow
> records. They come in on a UDP port, I pass them to flowd.Flow()...and
> that''s where everything explodes.
>
> Softflowd is set to export v9 flows.
> Wireshark says these are v9 flows.
> flowd.Flow() explodes with:
> Traceback (most recent call last):
>   File "nfagro.py", line 105, in ?
>     main()
>   File "nfagro.py", line 84, in main
>     msg = NetflowRecord(msg)
>   File "nfagro.py", line 24, in __init__
>     self.nf = flowd.Flow(blob=msg)
> ValueError: Unsupported version
>
> And this is being pulled from (msg, sndaddr) >
listensocket.recvfrom(10240). When I look at the data in msg, too, the
> first two octets are 0x0009. So...could this be an endianness issue?
> Some other crazy thing?
Are you trying to parse netflow records directly with the flowd Python
module? That won''t work - the Python module is to read logs written by
flowd. flowd writes its own NetFlow version independant log format.

I agree that a making a lightweight NetFlow parser library out of
flowd''s
guts would be a good thing though :)

-d

On Sep 17, 2007, at 2:18 AM, Damien Miller wrote:
> Are you trying to parse netflow records directly with the flowd Python
> module?
Well, yes.
> That won''t work - the Python module is to read logs written by
> flowd. flowd writes its own NetFlow version independant log format.
That would certainly explain why things weren''t working properly.
> I agree that a making a lightweight NetFlow parser library out of  
> flowd''s
> guts would be a good thing though :)
I find myself wondering whether it would be better from a system  
building perspective to build this thing as a backend for flowd by  
way of the Unix domain socket export, or actually break down and make  
a libnetflow and then create a python wrapper around it.

Cheers,
-Jesse Kempf



------------------------------------------------------------------------
The information contained in this communication is intended
only for the use of the recipient(s) named above. It may
contain information that is privileged or confidential, and
may be protected by State and/or Federal Regulations. If
the reader of this message is not the intended recipient,
you are hereby notified that any dissemination,
distribution, or copying of this communication, or any of
its contents, is strictly prohibited. If you have received
this communication in error, please return it to the sender
immediately and delete the original message and any copy
of it from your computer system. If you have any questions
concerning this message, please contact the sender.
------------------------------------------------------------------------

On Mon, 17 Sep 2007 10:34:16 -0400
Jesse Kempf <jkempf at davisvision.com> wrote:
> > I agree that a making a lightweight NetFlow parser library out of  
> > flowd''s
> > guts would be a good thing though :)
> 
> I find myself wondering whether it would be better from a system  
> building perspective to build this thing as a backend for flowd by  
> way of the Unix domain socket export, or actually break down and make  
> a libnetflow and then create a python wrapper around it.
Having given this more than three or four seconds of thought now, the insanity
of this suggestion is obvious to me.

I''m looking at extracting the individual netflow parsers from flowd.c,
but it seems like each of the process_netflow_v?() functions are tightly bound
to flowd''s tracking of the sanity of the sending agents.

It shouldn''t be too unreasonable to generalize this with a struct of
function pointers for handlers for each of the error conditions checked for in
process_netflow_v?(). Then the caller could have his functions handled however
he may please, as in the incrementing of peer->ninvalid in flowd.c, or
throwing exceptions in the case of a Python library.

Cheers,
-Jesse Kempf



------------------------------------------------------------------------
The information contained in this communication is intended
only for the use of the recipient(s) named above. It may
contain information that is privileged or confidential, and
may be protected by State and/or Federal Regulations. If
the reader of this message is not the intended recipient,
you are hereby notified that any dissemination,
distribution, or copying of this communication, or any of
its contents, is strictly prohibited. If you have received
this communication in error, please return it to the sender
immediately and delete the original message and any copy
of it from your computer system. If you have any questions
concerning this message, please contact the sender.
------------------------------------------------------------------------

On Mon, 17 Sep 2007, Jesse Kempf wrote:
> I''m looking at extracting the individual netflow parsers from
flowd.c,
> but it seems like each of the process_netflow_v?() functions are
> tightly bound to flowd''s tracking of the sanity of the sending
agents.
Yes, Netflow v.9 requires collectors to hold state.
> It shouldn''t be too unreasonable to generalize this with a struct
> of function pointers for handlers for each of the error conditions
> checked for in process_netflow_v?(). Then the caller could have his
> functions handled however he may please, as in the incrementing of
> peer->ninvalid in flowd.c, or throwing exceptions in the case of a
> Python library.
It might be cleaner to make these function return an integer error code
and propogate these back up to process_packet() (or higher).

-d