I''m working on a script which uses Flowd::read_flow (0.8.5 w/FIFO patches) to read in and then dump everything to a database. Everything looks fine, except I noticed a duplication of one entry during testing. The flow was a 10MB zeroes file scp''d from my laptop (192.168.0.14) to a server (10.0.0.104) binatted behind a PF box (192.168.0.22). You can see the duplication of this flow on lines 13-16 of the ascii table at: http://www.dixongroup.net/netmon.txt Any idea what might have caused this duplication? I see no other signs of duplication in the database. Thanks, -- Jason Dixon DixonGroup Consulting http://www.dixongroup.net
On May 20, 2005, at 2:02 PM, Jason Dixon wrote:> I''m working on a script which uses Flowd::read_flow (0.8.5 w/FIFO > patches) to read in and then dump everything to a database. > Everything looks fine, except I noticed a duplication of one entry > during testing. The flow was a 10MB zeroes file scp''d from my laptop > (192.168.0.14) to a server (10.0.0.104) binatted behind a PF box > (192.168.0.22). You can see the duplication of this flow on lines > 13-16 of the ascii table at: > > http://www.dixongroup.net/netmon.txt > > Any idea what might have caused this duplication? I see no other > signs of duplication in the database.I''ve updated the page to reflect my more recent findings. It appears that this behavior has something to do with state being created on both interfaces. That is to say, for connections that do NOT get routed through the firewall (in this case, binat), I am only seeing one set of flows (in/out) for each connection. However, if the connection is passing from one network to the other, I see duplicate entries for each flow. Obviously, a "SELECT DISTINCT" is a sufficient workaround, but I would like to understand why this is happening. http://www.dixongroup.net/netmon.txt (updated) P.S. DJM is probably en route to the hackathon, so I''d be curious if anyone else in the community has any ideas. Thanks, -- Jason Dixon DixonGroup Consulting http://www.dixongroup.net
Jason Dixon wrote:> I''ve updated the page to reflect my more recent findings. It appears > that this behavior has something to do with state being created on both > interfaces. That is to say, for connections that do NOT get routed > through the firewall (in this case, binat), I am only seeing one set of > flows (in/out) for each connection. However, if the connection is > passing from one network to the other, I see duplicate entries for each > flow. Obviously, a "SELECT DISTINCT" is a sufficient workaround, but I > would like to understand why this is happening.You are correct: pfflowd is reporting what pfsync gives it, so if pf creates state in two places, then pfflowd will see two state entries You can work around this by: 1) not creating state on certain traffic 2) fiddling with "set state-policy" (maybe) 3) using pfflowd''s -S option Eliminating this dupes within flowd might be tricky, because they might not always be obvious. Consider the case of a firewall running a dynamic routing protocol that changes the outbound interface of a flow part way through its lifetime. Try running tcpdump on the pfsync interface and seeing if there is any other distinguishing features in the raw pfsync frames. pfflowd could probably be taught to filter on these (and it does need filtering support),> P.S. DJM is probably en route to the hackathon, so I''d be curious if > anyone else in the community has any ideas.Unfortunately I can''t make it this year :( However, I will be badgering the pf guys about making life easier for pfflowd. Starting with 64-bit packet and octet counters in pfsync... -d
On May 20, 2005, at 6:26 PM, Damien Miller wrote:> Jason Dixon wrote: >> I''ve updated the page to reflect my more recent findings. It appears >> that this behavior has something to do with state being created on >> both interfaces. That is to say, for connections that do NOT get >> routed through the firewall (in this case, binat), I am only seeing >> one set of flows (in/out) for each connection. However, if the >> connection is passing from one network to the other, I see duplicate >> entries for each flow. Obviously, a "SELECT DISTINCT" is a >> sufficient workaround, but I would like to understand why this is >> happening. > > You are correct: pfflowd is reporting what pfsync gives it, so if pf > creates state in two places, then pfflowd will see two state entries > > You can work around this by: > > 1) not creating state on certain trafficMessy, doesn''t scale well for a 3rd party application. I don''t want to have to force design requirements on my users.> 2) fiddling with "set state-policy" (maybe)No good, tried it already with if-bound.> 3) using pfflowd''s -S optionNot a solution, we definitely want to track both inbound and outbound traffic.> Eliminating this dupes within flowd might be tricky, because they might > not always be obvious. Consider the case of a firewall running a > dynamic > routing protocol that changes the outbound interface of a flow part way > through its lifetime.Agreed.> Try running tcpdump on the pfsync interface and seeing if there is any > other distinguishing features in the raw pfsync frames. pfflowd could > probably be taught to filter on these (and it does need filtering > support),What tcpdump flags do you suggest for this? I''ve tried "tcpdump -vvvnettti" on the physical interface, but I don''t see anything revealing: <snip> ay 20 23:27:05.468242 0:c0:4f:46:94:48 1:0:5e:0:0:f0 0800 262: 10.255.255.2 > 224.0.0.240: PFSYNCv2 count 1: INS ST: (DF) [tos 0x10] (ttl 255, id 57925) May 20 23:27:06.467300 0:c0:4f:46:94:48 1:0:5e:0:0:f0 0800 214: 10.255.255.2 > 224.0.0.240: PFSYNCv2 count 2: UPD ST COMP: (DF) [tos 0x10] (ttl 255, id 35988) May 20 23:27:07.467299 0:c0:4f:46:94:48 1:0:5e:0:0:f0 0800 214: 10.255.255.2 > 224.0.0.240: PFSYNCv2 count 2: UPD ST COMP: (DF) [tos 0x10] (ttl 255, id 62175) </snip>> Unfortunately I can''t make it this year :(Very sorry to hear it. Wish we''d know beforehand, perhaps we could''ve organized a DJM hackathon drive. :-P> However, I will be badgering the pf guys about making life easier for > pfflowd. Starting with 64-bit packet and octet counters in pfsync...For the time being, "SELECT DISTINCT" will be fine. But in the long run, it would be nice if PF didn''t have to create dedicated state entries for each interface. Sounds good from a dreamer''s perspective, but not very practical in real life. Thanks, -- Jason Dixon DixonGroup Consulting http://www.dixongroup.net