Hi. I''m using Flowd 0.9 built on Linux (CentOS 5) and I''m using an adapted version of your sockclient.py to inject qualified info into MySQL on another server. flowd is using the ''logsock'' logging mechanism After about 20 seconds of running, I get the following: output_flow_enqueue: output queue full output_flow_enqueue: output queue full process_flow: enqueue failed after flush privsep_master: child exited child exited with status 1 Here are a few samples right before this happens: output_flow_enqueue: offset 523972 alloc 524288 process_flow: ACCEPT flow FLOW recv_time 2008-05-16T17:26:46.771122 proto 6 tcpflags 11 tos 00 agent [208.78.27.14] src [88.238.206.102]:1187 dst [69.9.45.36]:80 gateway [69.9.45.36] packets 1 octets 40 in_if 268 out_if 256 sys_uptime_ms 6w1d14h39m12s.099 time_sec 2008-05-19T21:18:47 time_nanosec 0 netflow ver 5 flow_start 6w1d14h38m20s.364 flow_finish 6w1d14h38m20s.364 src_AS 9121 src_masklen 17 dst_AS 29838 dst_masklen 28 engine_type 0 engine_id 0 seq 79937 source 0 crc32 00000000 output_flow_enqueue: offset 524088 alloc 524288 process_flow: ACCEPT flow FLOW recv_time 2008-05-16T17:26:46.771122 proto 6 tcpflags 18 tos 00 agent [208.78.27.14] src [69.9.40.103]:80 dst [82.225.26.147]:3925 gateway [66.216.8.41] packets 1 octets 1500 in_if 102 out_if 268 sys_uptime_ms 6w1d14h39m12s.099 time_sec 2008-05-19T21:18:47 time_nanosec 0 netflow ver 5 flow_start 6w1d14h38m4s.241 flow_finish 6w1d14h38m4s.241 src_AS 29838 src_masklen 24 dst_AS 12322 dst_masklen 11 engine_type 0 engine_id 0 seq 79937 source 0 crc32 00000000 output_flow_enqueue: offset 524204 alloc 524288 process_flow: ACCEPT flow FLOW recv_time 2008-05-16T17:26:46.771122 proto 6 tcpflags 10 tos 00 agent [208.78.27.14] src [69.9.40.103]:80 dst [82.225.26.147]:3931 gateway [66.216.8.41] packets 2 octets 3000 in_if 102 out_if 268 sys_uptime_ms 6w1d14h39m12s.099 time_sec 2008-05-19T21:18:47 time_nanosec 0 netflow ver 5 flow_start 6w1d14h38m6s.068 flow_finish 6w1d14h38m6s.500 src_AS 29838 src_masklen 24 dst_AS 12322 dst_masklen 11 engine_type 0 engine_id 0 seq 79937 source 0 crc32 00000000 I''m not exactly sure what the problem may be but I''m wondering what any limitations may be with overall processing power (Not sending a tremendous amount of flows at it) and i''m also wondering how ''blocking'' my python script could be, with all the mysql inserts and what not. Anyone have any thoughts? Thanks -- James Cornman Chief Technical Officer Atlantic Metro Communications e: james at atlanticmetro.net w: http://www.atlanticmetro.net v: 212-792-9950 f: 718-559-4862 CONFIDENTIALITY NOTICE: This communication and any documents, files or previous e-mail messages attached to it, constitute an electronic communication within the scope of the Electronic Communication Privacy Act, 18 USCA 2510. This communication may contain non-public, confidential, or legally privileged information intended for the sole use of the designated recipient(s). The unlawful interception, use or disclosure of such information is strictly prohibited under 18 USCA 2511 and any applicable laws. If you are not the intended recipient, or have received this communication in error, please notify the sender immediately by reply email at support at atlanticmetro.net or by telephone at 212-792-9950 and delete all copies of this communication, including attachments, without reading them or saving them to disk.
James Cornman wrote:> Hi. > > I''m using Flowd 0.9 built on Linux (CentOS 5) and I''m using an adapted > version of your sockclient.py to inject qualified info into MySQL on > another server. flowd is using the ''logsock'' logging mechanism > > After about 20 seconds of running, I get the following: > > output_flow_enqueue: output queue full > output_flow_enqueue: output queue full > process_flow: enqueue failed after flush > privsep_master: child exited > child exited with status 1 > ><snip>> I''m not exactly sure what the problem may be but I''m wondering what > any limitations may be with overall processing power (Not sending a > tremendous amount of flows at it) and i''m also wondering how > ''blocking'' my python script could be, with all the mysql inserts and > what not. > > Anyone have any thoughts? >What you''re seeing is a bug in 0.9 which got fixed back in the fall -- flowd wasn''t emptying its file output queue and would explode after a certain, rather small (several hundred, I think?) number of records was received. I have a setup that does 200 records/sec up to several thousand records/sec, and that shoves everything into a PostgreSQL database. Flowd is decidedly not the bottleneck. Try the latest snapshot and see if that works for you. Cheers, -Jesse ------------------------------------------------------------------------ The information contained in this communication is intended only for the use of the recipient(s) named above. It may contain information that is privileged or confidential, and may be protected by State and/or Federal Regulations. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. ------------------------------------------------------------------------
Sweet. Its working like a champ now. That was quick. Thanks On Mon, May 19, 2008 at 5:47 PM, Jesse Kempf <jkempf at davisvision.com> wrote:> James Cornman wrote: >> >> Hi. >> >> I''m using Flowd 0.9 built on Linux (CentOS 5) and I''m using an adapted >> version of your sockclient.py to inject qualified info into MySQL on >> another server. flowd is using the ''logsock'' logging mechanism >> >> After about 20 seconds of running, I get the following: >> >> output_flow_enqueue: output queue full >> output_flow_enqueue: output queue full >> process_flow: enqueue failed after flush >> privsep_master: child exited >> child exited with status 1 >> >> > > <snip> >> >> I''m not exactly sure what the problem may be but I''m wondering what >> any limitations may be with overall processing power (Not sending a >> tremendous amount of flows at it) and i''m also wondering how >> ''blocking'' my python script could be, with all the mysql inserts and >> what not. >> >> Anyone have any thoughts? >> > > What you''re seeing is a bug in 0.9 which got fixed back in the fall -- flowd > wasn''t emptying its file output queue and would explode after a certain, > rather small (several hundred, I think?) number of records was received. I > have a setup that does 200 records/sec up to several thousand records/sec, > and that shoves everything into a PostgreSQL database. Flowd is decidedly > not the bottleneck. > Try the latest snapshot and see if that works for you. > > Cheers, > -Jesse > > > > ------------------------------------------------------------------------ > The information contained in this communication is intended > only for the use of the recipient(s) named above. It may > contain information that is privileged or confidential, and > may be protected by State and/or Federal Regulations. If > the reader of this message is not the intended recipient, > you are hereby notified that any dissemination, > distribution, or copying of this communication, or any of > its contents, is strictly prohibited. If you have received > this communication in error, please return it to the sender > immediately and delete the original message and any copy > of it from your computer system. If you have any questions > concerning this message, please contact the sender. > ------------------------------------------------------------------------ > >-- James Cornman Chief Technical Officer Atlantic Metro Communications e: james at atlanticmetro.net w: http://www.atlanticmetro.net v: 212-792-9950 f: 718-559-4862 CONFIDENTIALITY NOTICE: This communication and any documents, files or previous e-mail messages attached to it, constitute an electronic communication within the scope of the Electronic Communication Privacy Act, 18 USCA 2510. This communication may contain non-public, confidential, or legally privileged information intended for the sole use of the designated recipient(s). The unlawful interception, use or disclosure of such information is strictly prohibited under 18 USCA 2511 and any applicable laws. If you are not the intended recipient, or have received this communication in error, please notify the sender immediately by reply email at support at atlanticmetro.net or by telephone at 212-792-9950 and delete all copies of this communication, including attachments, without reading them or saving them to disk.