Is there any documentation on the export as generated by flowd-reader? For example, what are the possible values and meanings for proto (I know 6 is TCP)? What is the most accurate way of matching bi-directional packets (is it simply a specific port number range)? Can I simply assume that the LOWER port number is the port, and the higher is for matching? I have tried all of the README files, installed documentation and Googled, but can find nothing on this. I have also grepped a downloaded copy of the mailing list archive. Can anyone help? Thanks. Murray.
Murray Shields schreef:> Is there any documentation on the export as generated by flowd-reader? > For example, what are the possible values and meanings for proto (I know > 6 is TCP)?http://www.iana.org/assignments/protocol-numbers googlin for ''ip protocol numbers'' was quite usefull.> What is the most accurate way of matching bi-directional > packets (is it simply a specific port number range)? > Can I simply assume that the LOWER port number is the port, and the > higher is for matching? >By my knowledge flows are uni-directional. So if you have a TCP session, 2 flows are created. There is a source and destination port, but now lower and higher. But maybe I''m wrong...
Hello, On Fri, Mar 24, 2006 at 12:47:23PM +1000, Murray Shields wrote:> > Is there any documentation on the export as generated by flowd-reader? > For example, what are the possible values and meanings for proto (I know > 6 is TCP)? What is the most accurate way of matching bi-directional > packets (is it simply a specific port number range)?About protocols : less /etc/protocols (Unix) or "www.iana.org" and for "bi-directional matching": on the server''s side, there''s one defined port but from client''s point of vue, it''s not true.> Can I simply assume that the LOWER port number is the port, and the > higher is for matching?I''m not sure to understand what do you want to say ? Best regards.
* Murray Shields:> Is there any documentation on the export as generated by flowd-reader? > For example, what are the possible values and meanings for proto (I know > 6 is TCP)? What is the most accurate way of matching bi-directional > packets (is it simply a specific port number range)?You can match the connection quadruple (twice IP address and port). They are the same for both directions, except that sender and receiver are swapped.
Along the same lines of this question; when NetFlow lists something as being the "Source", for TCP connections, does this mean the full connection source (within the context of a TCP connection, three-way-handshake etc), or just where that specific set of packets is going to/coming from? i.e. if I''m looking at web traffic, will it look like this Source Dest SrcPort DstPort Prot A B 1064 80 6 Or this: Source Dest SrcPort DstPort Prot A B 1064 80 6 B A 80 1064 6 ? Thanks for everyone''s assistance in clarifying this. Yours truly, Nathan -----Original Message----- From: netflow-tools-bounces+nathan=inorb.com at mindrot.org [mailto:netflow-tools-bounces+nathan=inorb.com at mindrot.org] On Behalf Of Murray Shields Sent: March 23, 2006 9:47 PM To: netflow-tools at mindrot.org Subject: [netflow-tools] flowd-reader export Is there any documentation on the export as generated by flowd-reader? For example, what are the possible values and meanings for proto (I know 6 is TCP)? What is the most accurate way of matching bi-directional packets (is it simply a specific port number range)? Can I simply assume that the LOWER port number is the port, and the higher is for matching? I have tried all of the README files, installed documentation and Googled, but can find nothing on this. I have also grepped a downloaded copy of the mailing list archive. Can anyone help? Thanks. Murray. _______________________________________________ netflow-tools mailing list netflow-tools at mindrot.org http://www.mindrot.org/mailman/listinfo/netflow-tools
On Fri, 24 Mar 2006, Nathan Einwechter wrote:> > Along the same lines of this question; when NetFlow lists something as > being the "Source", for TCP connections, does this mean the full > connection source (within the context of a TCP connection, > three-way-handshake etc), or just where that specific set of packets is > going to/coming from?The latter, unfortunately. NetFlow''s design shows its lineage as part of Cisco''s old forwarding cache - it doesn''t have any conceptions of bidirectionality. Even NetFlow v.9 has not addressed this problem. Maybe IPFIX (IETF flow export) will, but I haven''t looked at the drafts for a while. -d
Gijs Molenaar wrote:> Murray Shields schreef: >> Is there any documentation on the export as generated by >> flowd-reader? For example, what are the possible values and meanings >> for proto (I know 6 is TCP)? > http://www.iana.org/assignments/protocol-numbers > > googlin for ''ip protocol numbers'' was quite usefull.Excellent, thank you.>> What is the most accurate way of matching bi-directional packets (is >> it simply a specific port number range)? >> Can I simply assume that the LOWER port number is the port, and the >> higher is for matching? >> > By my knowledge flows are uni-directional. So if you have a TCP > session, 2 flows are > created. There is a source and destination port, but now lower and > higher. But maybe > I''m wrong... >
Jean-Philippe Luiggi wrote:> Hello, > > On Fri, Mar 24, 2006 at 12:47:23PM +1000, Murray Shields wrote: > >> Is there any documentation on the export as generated by flowd-reader? >> For example, what are the possible values and meanings for proto (I know >> 6 is TCP)? What is the most accurate way of matching bi-directional >> packets (is it simply a specific port number range)? >> > > About protocols : less /etc/protocols (Unix) or "www.iana.org" > and for "bi-directional matching": on the server''s side, there''s one defined > port but from client''s point of vue, it''s not true. > > >> Can I simply assume that the LOWER port number is the port, and the >> higher is for matching? >> > > I''m not sure to understand what do you want to say ? >UDP packets are easy: in the test feed I am currently looking at the source has a port of zero (0) and the destination 771. For example: [192.168.1.1]:0 => [192.168.2.254]:771 But for unidirectional we are effectively getting two port numbers via two matching flow records. For example: [192.168.2.1]:45223 => [192.168.1.1]:80 [192.168.1.1]:80 => [192.168.2.1]:45223 For my purposes I do not need to match these two lines as this is for an ISP billing system (and they only bill for outgoing, not incoming). I also need to allocate the traffic to services via the ip address (eg, group port 80 web traffic on the bill). Looking at a single line in isolation the other port (45223) is significantly higher that the port for web traffic (80) that I am interested in. So to rephrase the question: can I assume that the port number that I need to look at is ALWAYS the lower of the two port numbers? That is, discard the higher number (45223) and assume that the traffic relates to the lower port (in this instance, port 80)? Is there a circumstance where this would fail me? Regards, Murray.> Best regards. > >
Florian Weimer wrote:> * Murray Shields: > > >> Is there any documentation on the export as generated by flowd-reader? >> For example, what are the possible values and meanings for proto (I know >> 6 is TCP)? What is the most accurate way of matching bi-directional >> packets (is it simply a specific port number range)? >> > > You can match the connection quadruple (twice IP address and port). > They are the same for both directions, except that sender and receiver > are swapped. >When you perform this match (I will have to add the received time into this equation as I am getting repeats at different times) this will give me a bi-directional pair for a request/response flow of traffic. THEREFORE can I use the destination port from the FIRST of these two records, and use it as the port identifying the type of traffic? For instance, the following matched pair: [192.168.2.1]:45223 => [192.168.1.1]:80 [192.168.1.1]:80 => [192.168.2.1]:45223 means: 192.168.2.1 used port 54223 to send a packet request a web server on 192.168.1.1 using port 80. 192.168.1.1 used port 80 to send a response to 192.168.2.1 using port 54223. Therefore the port indicating the traffic type is 80 (the first destination). Makes sense to me. Any holes in this logic? Murray.
* Murray Shields:> Makes sense to me. Any holes in this logic?It might be a very long connection which results in multiple flows. In this case, the first packet in the two flows is not sent by the client. In general, it is quite difficult to reconstruct the roles without TCP flags export (and the way it is done by some vendors is not really helpful, either).
On Tue, 28 Mar 2006, at 14:00, Florian Weimer wrote:> * Murray Shields: > > > Makes sense to me. Any holes in this logic? > > It might be a very long connection which results in multiple flows. > In this case, the first packet in the two flows is not sent by the > client. > > > In general, it is quite difficult to reconstruct the roles without TCP > flags export (and the way it is done by some vendors is not really > helpful, either).Even when you are lucky enough to have the flags, it not that helpful: as flags are ORed, you end up for a ''complete'' tcp ''session'' with both uni-directional flows having at least SAF set - no way to distinguish the client (in an ip sense) from the server Or do i minsunderstand you ?
On Sat, 25 Mar 2006, at 11:12, Damien Miller wrote:> On Fri, 24 Mar 2006, Nathan Einwechter wrote: > > > > > Along the same lines of this question; when NetFlow lists something as > > being the "Source", for TCP connections, does this mean the full > > connection source (within the context of a TCP connection, > > three-way-handshake etc), or just where that specific set of packets is > > going to/coming from? > > The latter, unfortunately. > > NetFlow''s design shows its lineage as part of Cisco''s old forwarding > cache - it doesn''t have any conceptions of bidirectionality. Even > NetFlow v.9 has not addressed this problem. > > Maybe IPFIX (IETF flow export) will, but I haven''t looked at the > drafts for a while.There is a biflow draft btw: http://www.ietf.org/internet-drafts/draft-boschi-ipfix-biflow-01.txt there is discussions regarding this draft on the ipfix list
On Tue, 28 Mar 2006, at 15:56, Yann Berthier wrote:> There is a biflow draft btw: > http://www.ietf.org/internet-drafts/draft-boschi-ipfix-biflow-01.txtSorry, should have been http://www.ietf.org/internet-drafts/draft-trammell-ipfix-biflow-00.txt
Yann Said: Even when you are lucky enough to have the flags, it not that helpful: as flags are ORed, you end up for a ''complete'' tcp ''session'' with both uni-directional flows having at least SAF set - no way to distinguish the client (in an ip sense) from the server Or do i minsunderstand you ---------------------- Okay - here''s what I''m doing now as a test and want to see if this will work as I anticipate. For TCP connections, I''m filtering only those that are active connections (in my case, I don''t care about those that aren''t full fledged connections) using the flags ala: tcp_flags mask 0x12 equals 0x12 This creates a situation where the true connection source and destinations are reversed in the log, due to the stage of communications that these flags are set. So, when I export them using my perl exporter, I simply invert them once again to get the true source and destination for my final processing. Does this work as I anticipate? Would this give me the actual source and destinations? From what I''ve seen it does, but there may be exceptions. Also, you mention, in a later message, that connections separated by significant time will not be aggregated into a single entry. Any idea how long this is etc? That becomes important. I have a long and memory intensive process to remove these duplicates, but if I could have a timeframe after which duplicate entries are not inserted, then I could reduce the inefficiency of this process. Thanks again. -- Nathan
* Yann Berthier:> On Tue, 28 Mar 2006, at 14:00, Florian Weimer wrote: > >> * Murray Shields: >> >> > Makes sense to me. Any holes in this logic? >> >> It might be a very long connection which results in multiple flows. >> In this case, the first packet in the two flows is not sent by the >> client. >> >> >> In general, it is quite difficult to reconstruct the roles without TCP >> flags export (and the way it is done by some vendors is not really >> helpful, either). > > Even when you are lucky enough to have the flags, it not that > helpful: as flags are ORed, you end up for a ''complete'' tcp > ''session'' with both uni-directional flows having at least SAF set - > no way to distinguish the client (in an ip sense) from the server > > Or do i minsunderstand you ?I would expect a configuration tweak to break the netflow spec and transmit the flag of the *first* segment. 8-) The SAF combo is not very useful indeed.