Forwarding to netflow-tools where it belongs... I''m trying to get flowd to write to FIFO so I can read it in with a perl script and output to my choice of storage (file, db, etc). The patch below from djm allows flowd to write to fifo, but Flowd.pm fails on init() with a "bad magic" error. Thanks, Jason> From: Damien Miller <djm at mindrot.org> > Date: May 16, 2005 2:42:59 AM EDT > To: ports at openbsd.org > Subject: Re: Netflow collector on OpenBSD > > Jason Dixon wrote: >> I''m looking for a Netflow collector on OpenBSD that can dump to >> database. I would prefer to use djm''s flowd, but it doesn''t support >> database output. I''ve tried having it log to fifo for reading in >> with a Perl script, but it dies trying to perform a seek on startup: > > This is probably best taken to the flowd mailing list[1], but > interested > users can try the attached patch. > > -d > > [1] http://www.mindrot.org/mailman/listinfo/netflow-tools > ? build > ? buildit.sh > ? filter.day > ? flowd.conf.test > ? flowd.log > ? flowd.log.v46 > ? flowd.pyc > ? netflow-v9 > ? testwrite.py > ? x.conf > ? xxx.conf > ? tools/stats.py > ? tools/stats.pyc > Index: flowd.c > ==================================================================> RCS file: /var/cvs/flowd/flowd.c,v > retrieving revision 1.56 > diff -u -p -r1.56 flowd.c > --- flowd.c 28 Apr 2005 09:02:58 -0000 1.56 > +++ flowd.c 16 May 2005 04:16:30 -0000 > @@ -121,9 +121,19 @@ start_log(int monitor_fd) > int fd; > off_t pos; > char ebuf[512]; > + struct stat sb; > > if ((fd = client_open_log(monitor_fd)) == -1) > logerrx("Logfile open failed, exiting"); > + > + if (fstat(fd, &sb) == -1) > + logerr("log fstat"); > + > + /* Don''t bother writing header to FIFOs */ > + if (S_ISFIFO(sb.st_mode)) { > + logit(LOG_DEBUG, "logfile is FIFO, skipping header write"); > + return (fd); > + } > > /* Only write out the header if we are at the start of the file */ > switch ((pos = lseek(fd, 0, SEEK_END))) { > Index: privsep.c > ==================================================================> RCS file: /var/cvs/flowd/privsep.c,v > retrieving revision 1.26 > diff -u -p -r1.26 privsep.c > --- privsep.c 14 May 2005 06:04:18 -0000 1.26 > +++ privsep.c 16 May 2005 04:16:30 -0000 > @@ -712,12 +712,22 @@ static int > answer_open_log(struct flowd_config *conf, int client_fd) > { > int fd; > + struct stat sb; > > logit(LOG_DEBUG, "%s: entering", __func__); > > fd = open(conf->log_file, O_RDWR|O_APPEND|O_CREAT, 0600); > if (fd == -1) { > - logitm(LOG_ERR, "%s: open", __func__); > + logitm(LOG_ERR, "%s: open(%.100s)", __func__, conf->log_file); > + return (-1); > + } > + if (fstat(fd, &sb) == -1) { > + logitm(LOG_ERR, "%s: fstat(%.100s)", __func__, conf->log_file); > + return (-1); > + } > + if (!S_ISREG(sb.st_mode) && !S_ISFIFO(sb.st_mode)) { > + logit(LOG_ERR, "%s: log file \"%.100s\" is neither a regular " > + "file nor a FIFO", __func__, conf->log_file); > return (-1); > } > if (send_fd(client_fd, fd) == -1) >-- Jason Dixon DixonGroup Consulting http://www.dixongroup.net
Jason Dixon wrote:> Forwarding to netflow-tools where it belongs... > > I''m trying to get flowd to write to FIFO so I can read it in with a perl > script and output to my choice of storage (file, db, etc). The patch > below from djm allows flowd to write to fifo, but Flowd.pm fails on > init() with a "bad magic" error.Please try the attached (completely untested) patch. flowd (with the FIFO patch) won''t write a header when it sends records to a FIFO, so the Perl and Python APIs have to be adjusted to cope. Maybe it should just send a header after all, but I need to think about it some more. -d -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: flowd-logfifo-pp.diff Url: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050516/5b49d8f6/attachment.ksh
On May 16, 2005, at 9:36 AM, Damien Miller wrote:> Please try the attached (completely untested) patch. > > flowd (with the FIFO patch) won''t write a header when it sends records > to a FIFO, so the Perl and Python APIs have to be adjusted to cope. > Maybe it should just send a header after all, but I need to think about > it some more.<snip> Missing semicolon. --- Flowd-perl/lib/Flowd.pm.old Mon May 16 10:12:38 2005 +++ Flowd-perl/lib/Flowd.pm Mon May 16 10:16:50 2005 @@ -144,7 +144,7 @@ open($fhandle, "<$filename") or die "open($filename): $!"; $self->{handle} = $fhandle; - @st = stat($self->{handle}) + @st = stat($self->{handle}); if (($st[2] & 0170000) == 0010000) { # If reading from FIFO, assume that version matches $self->{version} = 0x00000002; Looks like it''s working now! Thanks! -- Jason Dixon DixonGroup Consulting http://www.dixongroup.net
Jason Dixon wrote:> Forwarding to netflow-tools where it belongs... > > I''m trying to get flowd to write to FIFO so I can read it in with a perl > script and output to my choice of storage (file, db, etc). The patch > below from djm allows flowd to write to fifo, but Flowd.pm fails on > init() with a "bad magic" error.Hi, I have thought about this some more - rather than skipping log headers based on what type of file your are reading from / writing to, I think it is better that users a choice. This patch does this - it adds a "skip header" option to flowd, flowd-reader and the perl/python APIs. So you should be able to put: logfile "/path/to/fifo" noheader in your flowd.conf and directly listen to it with "flowd-reader -S /path/to/fifo". You can also use flowd-reader to write to a fifo Read the diff to see the equivalent options for perl and python. Does this work for you? If so, I''ll tidy it up, document and commit it. -d
It would help if I actually attached the patch. -d Damien Miller wrote:> Jason Dixon wrote: > >>Forwarding to netflow-tools where it belongs... >> >>I''m trying to get flowd to write to FIFO so I can read it in with a perl >>script and output to my choice of storage (file, db, etc). The patch >>below from djm allows flowd to write to fifo, but Flowd.pm fails on >>init() with a "bad magic" error. > > > Hi, > > I have thought about this some more - rather than skipping log headers > based on what type of file your are reading from / writing to, I think > it is better that users a choice. > > This patch does this - it adds a "skip header" option to flowd, > flowd-reader and the perl/python APIs. So you should be able to put: > > logfile "/path/to/fifo" noheader > > in your flowd.conf and directly listen to it with "flowd-reader -S > /path/to/fifo". You can also use flowd-reader to write to a fifo > > Read the diff to see the equivalent options for perl and python. > > Does this work for you? If so, I''ll tidy it up, document and commit it. > > -d > > _______________________________________________ > netflow-tools mailing list > netflow-tools at mindrot.org > http://www.mindrot.org/mailman/listinfo/netflow-tools-------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: flowd-noheader.diff Url: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050709/1cdfb5e3/attachment.ksh