Shorewall users - May 2011 - How to do per interface accounting with NFLOG in shorewall?

Hi, I need to accurately account for all traffic through several
interfaces on a box running Shorewall.  I will be using NFLOG with a
custom logger because I actually need to log traffic "per user" also
(and so there is also a captive portal type arrangement tagging packets
with an fwmark.)

I guess I just need a simple rule at the head of the INPUT and OUTPUT
chains to grab this logging - however, what is the best way to manage
this rule and make sure it''s there whenever shorewall is restarted,
etc?

(Basically I don''t want to have to ensure every single entry in
"rules"
has a correct log entry against it.  I require to log every single
packet in/out)

Thanks for any thoughts

Ed W

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay

Anyone?

To rephrase the question - I need to maintain a separate iptables rule
which has to match (and nf_log) ALL traffic. How to best maintain such
an additional iptables line to exist past restarts, etc? (probably
externally, but how?)

Alternatively phrased question - how difficult would it be to support
NFLOG accounting in the accounting table?

Thanks

Ed W


On 15/05/2011 10:36, Ed W wrote:
> Hi, I need to accurately account for all traffic through several
> interfaces on a box running Shorewall.  I will be using NFLOG with a
> custom logger because I actually need to log traffic "per user"
also
> (and so there is also a captive portal type arrangement tagging packets
> with an fwmark.)
> 
> I guess I just need a simple rule at the head of the INPUT and OUTPUT
> chains to grab this logging - however, what is the best way to manage
> this rule and make sure it''s there whenever shorewall is
restarted, etc?
> 
> (Basically I don''t want to have to ensure every single entry in
"rules"
> has a correct log entry against it.  I require to log every single
> packet in/out)
> 
> Thanks for any thoughts
> 
> Ed W
------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay

On May 16, 2011, at 10:51 AM, Ed W wrote:
> Anyone?
> 
> To rephrase the question - I need to maintain a separate iptables rule
> which has to match (and nf_log) ALL traffic. How to best maintain such
> an additional iptables line to exist past restarts, etc? (probably
> externally, but how?)
No single rule can do what you want because there is no single chain through
which all traffic flows so you need more than one. But you can add them in the
''start'' extension script.
> Alternatively phrased question - how difficult would it be to support
> NFLOG accounting in the accounting table?
It''s not difficult but I''m not sure I want to try to support
such a feature.

-Tom

Tom Eastep        \ When I die, I want to go like my Grandfather who
Shoreline,         \ died peacefully in his sleep. Not screaming like
Washington, USA     \ all of the passengers in his car
http://shorewall.net \________________________________________________




------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay

On May 16, 2011, at 4:23 PM, Tom Eastep wrote:
> 
> On May 16, 2011, at 10:51 AM, Ed W wrote:
> 
>> Anyone?
>> 
>> To rephrase the question - I need to maintain a separate iptables rule
>> which has to match (and nf_log) ALL traffic. How to best maintain such
>> an additional iptables line to exist past restarts, etc? (probably
>> externally, but how?)
> 
> No single rule can do what you want because there is no single chain
through which all traffic flows so you need more than one. But you can add them
in the ''start'' extension script.
> 
>> Alternatively phrased question - how difficult would it be to support
>> NFLOG accounting in the accounting table?
> 
> It''s not difficult but I''m not sure I want to try to
support such a feature.
I''ve looked at this and it''s not as straight-forward as I had
assumed. So it isn''t something that I can whip up in a day or two.

-Tom

Tom Eastep        \ When I die, I want to go like my Grandfather who
Shoreline,         \ died peacefully in his sleep. Not screaming like
Washington, USA     \ all of the passengers in his car
http://shorewall.net \________________________________________________




------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay

On 17/05/2011 00:23, Tom Eastep wrote:
> 
> On May 16, 2011, at 10:51 AM, Ed W wrote:
> 
>> Anyone?
>>
>> To rephrase the question - I need to maintain a separate iptables rule
>> which has to match (and nf_log) ALL traffic. How to best maintain such
>> an additional iptables line to exist past restarts, etc? (probably
>> externally, but how?)
> 
> No single rule can do what you want because there is no single chain
through which all traffic flows so you need more than one. But you can add them
in the ''start'' extension script.
Thanks Tom

Looking at the accounting man pages, it seems like the "feature" would
be to allow DONE:NFLOG and COUNT:NFLOG in the accounting rules? I
haven''t yet pulled out of the code to understand how tricky this is,
but
I accept your reply that it''s not straightforward for now!

The only thing I haven''t considered properly is whether I have all the
info at this chain to decide how the packet will be routed (which
interface), but I *think* I do since the current design should use
firewall marks to choose routing options...

I guess logging a bonded VPN over some advanced split routing with
probabilistic routing isn''t going to be correctly logged this way...

Any other tips for best location (chains) to place these logging rules?
 To recap we have several internet connections which have variable cost
and we will use NFLOG to examine every packet to read it''s mark and
bill
the sender appropriately.

Cheers

Ed W

------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay

On 05/17/2011 02:27 AM, Ed W wrote:
> On 17/05/2011 00:23, Tom Eastep wrote:
>>
>> On May 16, 2011, at 10:51 AM, Ed W wrote:
>>
>>> Anyone?
>>>
>>> To rephrase the question - I need to maintain a separate iptables
rule
>>> which has to match (and nf_log) ALL traffic. How to best maintain
such
>>> an additional iptables line to exist past restarts, etc? (probably
>>> externally, but how?)
>>
>> No single rule can do what you want because there is no single chain
through which all traffic flows so you need more than one. But you can add them
in the ''start'' extension script.
> 
> Thanks Tom
> 
> Looking at the accounting man pages, it seems like the "feature"
would
> be to allow DONE:NFLOG and COUNT:NFLOG in the accounting rules? I
> haven''t yet pulled out of the code to understand how tricky this
is, but
> I accept your reply that it''s not straightforward for now!
The problem is that all of Shorewall''s logging infrastructure assumes
that you want to pass each log message through the rate-limiting
restrictions imposed by LOGBURST and LOGLIMIT. That clearly doesn''t
work
when you are trying to use the log messages for accounting purposes.
> 
> The only thing I haven''t considered properly is whether I have all
the
> info at this chain to decide how the packet will be routed (which
> interface), but I *think* I do since the current design should use
> firewall marks to choose routing options...
> 
> I guess logging a bonded VPN over some advanced split routing with
> probabilistic routing isn''t going to be correctly logged this
way...
The difficulty of correct VPN accounting depends on the VPN technology
you are using.
> 
> Any other tips for best location (chains) to place these logging rules?
>  To recap we have several internet connections which have variable cost
> and we will use NFLOG to examine every packet to read it''s mark
and bill
> the sender appropriately.
If you only want to log messages leaving the firewall (download traffic
would be accounted as it leaves the local firewall interfaces), then you
could do it in the POSTROUTING mangle chain.

-Tom
-- 
Tom Eastep        \ When I die, I want to go like my Grandfather who
Shoreline,         \ died peacefully in his sleep. Not screaming like
Washington, USA     \ all of the passengers in his car
http://shorewall.net \________________________________________________



------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay

On 05/17/2011 02:27 AM, Ed W wrote:
> I accept your reply that it''s not straightforward for now!
As I stated, to do as you suggest (COUNT:NFLOG... or DONE:...) would be
considerable work. To simple add an NFLOG ''ACTION'' to the
accounting
file is trivial. The attached patch does it.

	NFLOG[(...)]	chain	source	...

This will be in 4.4.20 along with the ability to add your accounting
rules in the mangle table rather than the filter table. The latter will
allow accounting rules in PREROUTING and POSTROUTING.

-Tom
-- 
Tom Eastep        \ When I die, I want to go like my Grandfather who
Shoreline,         \ died peacefully in his sleep. Not screaming like
Washington, USA     \ all of the passengers in his car
http://shorewall.net \________________________________________________



------------------------------------------------------------------------------
What Every C/C++ and Fortran developer Should Know!
Read this article and learn how Intel has extended the reach of its 
next-generation tools to help Windows* and Linux* C/C++ and Fortran 
developers boost performance applications - including clusters. 
http://p.sf.net/sfu/intel-dev2devmay

On 17/05/2011 21:18, Tom Eastep wrote:
> On 05/17/2011 02:27 AM, Ed W wrote:
> 
>> I accept your reply that it''s not straightforward for now!
> 
> As I stated, to do as you suggest (COUNT:NFLOG... or DONE:...) would be
> considerable work. To simple add an NFLOG ''ACTION'' to the
accounting
> file is trivial. The attached patch does it.
> 
> 	NFLOG[(...)]	chain	source	...
> 
> This will be in 4.4.20 along with the ability to add your accounting
> rules in the mangle table rather than the filter table. The latter will
> allow accounting rules in PREROUTING and POSTROUTING.
Thanks Tom,

I need to study this, but it looks exactly spot on?  My goal is simply
to get a simple entry that logs every single packet that goes in/out of
the internet route. I think the limitations you list are pretty much as
expected because NFLOG is really intended for more course logging and
you can rely on your log demon to help with aggregation and filtering. I
don''t particularly expect to filter the input to nflog, although
perhaps
some folks might use some very course filtering (interface, protocol)?

In fact I think it''s more the reverse - if you are NFLOGing then
probably you are trying to centralise some fairly course grained
logging. If you want to trace very specific packets, then probably
adding the entry to the rule table makes more sense?

Still learning my way around shorewall, so might be completely missing
the point.  Many thanks

Ed W

P.S.  Just checking that this can''t be done through the rules table -
the docs suggest that it''s not possible to have a "log" only
rule there,
ie something which matches but doesn''t influence packet destination or
stop the flow of processing?

------------------------------------------------------------------------------
What Every C/C++ and Fortran developer Should Know!
Read this article and learn how Intel has extended the reach of its 
next-generation tools to help Windows* and Linux* C/C++ and Fortran 
developers boost performance applications - including clusters. 
http://p.sf.net/sfu/intel-dev2devmay

On 05/18/2011 11:27 AM, Ed W wrote:
> On 17/05/2011 21:18, Tom Eastep wrote:
>> On 05/17/2011 02:27 AM, Ed W wrote:
>>
>>> I accept your reply that it''s not straightforward for now!
>>
>> As I stated, to do as you suggest (COUNT:NFLOG... or DONE:...) would be
>> considerable work. To simple add an NFLOG ''ACTION'' to
the accounting
>> file is trivial. The attached patch does it.
>>
>> 	NFLOG[(...)]	chain	source	...
>>
>> This will be in 4.4.20 along with the ability to add your accounting
>> rules in the mangle table rather than the filter table. The latter will
>> allow accounting rules in PREROUTING and POSTROUTING.
> 
> Thanks Tom,
> 
> I need to study this, but it looks exactly spot on?  My goal is simply
> to get a simple entry that logs every single packet that goes in/out of
> the internet route. I think the limitations you list are pretty much as
> expected because NFLOG is really intended for more course logging and
> you can rely on your log demon to help with aggregation and filtering. I
> don''t particularly expect to filter the input to nflog, although
perhaps
> some folks might use some very course filtering (interface, protocol)?
Possibly.
> 
> In fact I think it''s more the reverse - if you are NFLOGing then
> probably you are trying to centralise some fairly course grained
> logging. If you want to trace very specific packets, then probably
> adding the entry to the rule table makes more sense?
Yes, I think so. For accounting, you need the packet length and the
protocol headers and not the payload. So you should only capture the
first 40 bytes or so of each packet (second NFLOG parameter). Also, for
decreased overhead, the third argument should be set > 1 (I would start
at 20 or so which would be 800 bytes if you are capturing 40 bytes/packet).
> 
> Still learning my way around shorewall, so might be completely missing
> the point.  Many thanks
> 
> Ed W
> 
> P.S.  Just checking that this can''t be done through the rules
table -
> the docs suggest that it''s not possible to have a "log"
only rule there,
> ie something which matches but doesn''t influence packet
destination or
> stop the flow of processing?
You could use LOG:NFLOG(...) in the rules file, but you would need 3
times as many rules as in the accounting file (your LOG:NFLOG rules
would need to be replicated in each of the three sections of the file).
Additionally, each logged packet would include the prefix (default is
''Shorewall:<chain>:<disposition>:'') which you
could care less about.

-Tom
-- 
Tom Eastep        \ When I die, I want to go like my Grandfather who
Shoreline,         \ died peacefully in his sleep. Not screaming like
Washington, USA     \ all of the passengers in his car
http://shorewall.net \________________________________________________



------------------------------------------------------------------------------
What Every C/C++ and Fortran developer Should Know!
Read this article and learn how Intel has extended the reach of its 
next-generation tools to help Windows* and Linux* C/C++ and Fortran 
developers boost performance applications - including clusters. 
http://p.sf.net/sfu/intel-dev2devmay