libvirt users - Dec 2016 - nwfilters seem fundamentally unusable or unfinished

Hello!

I just spent the last four days working with nwfilters only to decide 
that they are apparently unusable. I've come to the mailing list seeking 
input on this subject.

First off, please forgive my offensiveness. I'm sure people worked hard 
on nwfilters and it looks like a lot of effort went into providing this 
functionality. This is also an extremely difficult subject to get right 
in the many possible use cases, so I'm very sympathetic to how difficult 
it would be to try and implement this. However, the existing system 
didn't work out for me, I've found a number of other people who are 
saying the same thing (it didn't work out for them), and I don't see any
hope continuing down the path of trying to make it work.

For now, I've given up on nwfilters and I created a hook script that 
works with my existing iptables rules and applies network filter 
policies on specific VM/guests where needed.

If you are doing extensive VM network filtering in your environment, how 
did you do it?

I've listed a bunch of my gripes below. Please correct me if I've gotten
anything wrong here. I'm new to nwfilters so maybe I overlooked 
something or I might just misunderstand the whole thing and could be 
totally wrong.



The first and primary problem that I have with nwfilters are that the 
documentation is poor. There is very little documentation which exists, 
and that which does exist seems like it was spat out just to fulfill 
business requirements that some documentation be produced, rather than 
an effort into creating good usable documentation. I've run into large 
amounts of undocumented behavior and I don't feel like reading the 
source code any further to figure out what the intent of these tools were.

My second big issue, and a clue very few people actually use nwfilters 
in the wild, is the low quantity of examples and how-to docs I found 
while googling. Complex examples just don't seem to exist. Further, of 
those complex examples I did find, people were often going down the 
route of creating their own hook script programs to replace nwfilters, 
indicating that this isn't just me.

Additionally...

I discovered that nwfilters do not play well with existing system 
iptables/ebtables rules. There is some good examples on this regarding 
Red Hat's firewalld and how libvirt's nwfilters does not play well 
together if you google around a little. It seems like this was just not 
considered in scope, or the assumption was that the local host would not 
have any existing iptables/ebtables rules and that libvirt would have 
complete control over the hypervisor host. There is no documented means 
of controlling where libvirt inserts it's rules into an existing set of 
rules, and libvirt creates numerous rules in both ebtables and iptables, 
making the problem even more complex.

nwfilter seems to have been designed with a bias towards 
user-networking. I am using bridged interfaces, and some features and 
virsh commands don't apply to this mode of operation.

I've been able to produce scenarios where nwfilter would abandon rules 
after changes had been made to running guests, and the only way I could 
get rid of them was manual intervention (iptables/ebtables -F -X).

There is no command/control to apply an existing nwfilter to a running 
guest, or to remove/clear the existing nwfilters on a running guest. 
This item is a huge indication that this isn't a production-ready 
feature set.

I think the worst problem I've run into, however, is that I was able to 
create very simple nwfilters that either broke networking of the 
hypervisor system (stopped all traffic), or failed to drop traffic which 
should have been dropped. I still don't understand why nwfilter is often 
creating rules in the ebtables "nat" table instead of the
"filter"
table, where they belong. That one right there is a huge WTF -- packets 
never get inspected because the rules are in the wrong table!

In general, I found the output iptables/ebables rules that nwfilter 
generated often did not reflect the obvious intent of the rules that 
went into the nwfilter xml configuration. This abstraction layer 
produces unreliable and/or confusing results. I put a series of rules 
into a nwfilter xml file and the iptables/ebtables rules that I get out 
are insane. Nwfilter rules in = mystery meat out.

Priorities are a huge WTF that caused me a lot of grief. Are rules going 
to be assembled in iptables/ebtables in the order which they are 
declared in XML? (this is undocumented) If so, why do priorities exist? 
(undocumented). What is the default priority? Is it zero? (undocumented).

Want to create a filter rule that will log certain packets? Apparently 
there is no logging functionality at all. Can't be done. Anything beyond 
the most basic packet allow/drop (even reject was an afterthought) isn't 
supported by nwfilter rules.

Thanks for reading