libvirt users - Aug 2022 - Libvirt virsh : Error starting network, cannot execute binary /usr/sbin/iptables

Hi,

I am a bit lost and hope someone can help me. I am running Debian 
bookworm (testing) with last updates.

$ sudo apt policy libvirt-daemon
libvirt-daemon:
 ? Install??: 8.5.0-1
 ? Candidat?: 8.5.0-1
 ?Table de version?:
 ?*** 8.5.0-1 100
 ??????? 100 /var/lib/dpkg/status


I am unable to start default network , and get an error related to 
iptables :

$ sudo virsh net-start default
erreur?:Impossible de d?marrer le r?seau default
erreur?:internal error: Failed to apply firewall rules 
/usr/sbin/iptables -w --table filter --list-rules: libvirt:? erreur : 
cannot execute binary /usr/sbin/iptables: Aucun fichier ou dossier de ce 
type

Sorry for the french, it says "impossible to start default network"
and
"no such file or folder" at the end.

It is true I removed iptables because I want to use only nftables (I 
removed both ufw and iptables packages (apt remove), and enabled the 
nftables service before error raises). Before this, all was fine, but 
when I enabled nftables, all VMs disapeared? from virt-manager).

I uninstalled KVM related packages and reinstalled, still the same.

I also installed back iptables, but strangely I still get the same 
error, although binary /usr/sbin/iptables is there.

I tried many things with no luck, restarted libvirtd service, recreated 
the network, etc...

Has anyone some idea about what is happening here ? is there some 
incompatibility with nftables (firewalld service is disabled) and libvirt ?

Thank you,
Pascal

On 8/15/22 1:00 PM, Pascal wrote:
> Hi,
> 
> I am a bit lost and hope someone can help me. I am running Debian 
> bookworm (testing) with last updates.
> 
> $ sudo apt policy libvirt-daemon
> libvirt-daemon:
>  ? Install??: 8.5.0-1
>  ? Candidat?: 8.5.0-1
>  ?Table de version?:
>  ?*** 8.5.0-1 100
>  ??????? 100 /var/lib/dpkg/status
> 
> 
> I am unable to start default network , and get an error related to 
> iptables :
> 
> $ sudo virsh net-start default
> erreur?:Impossible de d?marrer le r?seau default
> erreur?:internal error: Failed to apply firewall rules 
> /usr/sbin/iptables -w --table filter --list-rules: libvirt:? erreur : 
> cannot execute binary /usr/sbin/iptables: Aucun fichier ou dossier de ce 
> type
> 
> Sorry for the french, it says "impossible to start default
network" and
> "no such file or folder" at the end.
libvirt execs the iptables command to install packet filtering rules 
that are part of the setup of virtual networks with forward mode of 
"nat", "route", and for those virtual networks that have no
forward mode
at all (aka "isolated"). libvirt's default network uses NAT to
make all
virtual machines appear to be at the IP address of the host's external 
ethernet interface, so it requires the iptables command

Although I've been working on patches to make it possible for libvirt to 
use /usr/sbin/nft to add & remove packet filter rules, libvirt still 
uses iptables to add its rules, so if have a virtual network that needs 
to add packet filtering rules (or a guest that has nwfilter rules), 
you'll need the iptables command, and if it's not found then libvirt 
will fail at runtime (as you've seen).

Beyond that, if you install the libvirt-daemon-driver-nwfilter or 
libvirt-daemon-driver-network packages, then iptables *should* be a hard 
prerequisite listed in the package manifest; if your distro's packaging 
system is allowing you to remove the iptables package without also 
removing those libvirt packages (which is what you say you were able to 
do), or allowing you to install those libvirt packages without 
(semi)automatically pulling in iptables (which you also say you've 
done), then that is a bug in the distro's packaging files for libvirt.

It's unclear from your original message if you really require libvirt's 
default network in any of your guests. If you don't, then you can just 
not install or start the libvirt-daemon-driver-network package, and 
don't try to start the default network, and you should be okay without 
the iptables package on your system. It's not really gaining you 
anything (other than a miniscule amount of disk space) to do that though.

These days /usr/sbin/iptables is a symbolic link to either iptables-nft 
(which uses the nftables API to put all the rules into special nft 
tables named "filter" and "nat", so you actually see the
iptables rules
as nft rules when you run "nft list ruleset") or to 
/usr/sbin/iptables-legacy (which uses the old iptables API to install 
rules that *don't* show up in "nft list ruleset"). In either case 
though, packets are processed using the same nft packet matching code in 
the kernel. There is some amount of info about these two variants here:

 
https://developers.redhat.com/blog/2020/08/18/iptables-the-two-variants-and-their-relationship-with-nftables

The end of it all though is that even if you have some userspace 
programs that "use iptables", they are in the end using nftables in
the
kernel.
> It is true I removed iptables because I want to use only nftables (I 
> removed both ufw and iptables packages (apt remove), and enabled the 
> nftables service before error raises). Before this, all was fine, but 
> when I enabled nftables, all VMs disapeared? from virt-manager).
Are the guests completely disappearing from the list? Or is it that 
you're unable to start the guests? A guest would only completely 
disappear spontaneously from virt-manager (i.e. be completely removed 
from the list of all guests) if something went wrong while parsing the 
guest config during libvirtd (or virtqemud) startup. Look in the system 
logs for any errors during libvirtd startup (or virtqemud startup if 
your debian has switched to modular daemons). I can't think of anything 
that could/should change in the parsing of the config due to enabling 
the nftables servive (which libvirt knows nothing about).
> I uninstalled KVM related packages and reinstalled, still the same.
> 
> I also installed back iptables, but strangely I still get the same 
> error, although binary /usr/sbin/iptables is there.
Now that certainly makes no sense! :-/ Possibly /usr/sbin/iptables is a 
symbolic link to a file that doesn't exist (because there was an 
additional iptables package, e.g. iptables-nft or iptables-legacy, that 
you didn't reinstall?)
> I tried many things with no luck, restarted libvirtd service, recreated 
> the network, etc...
> 
> Has anyone some idea about what is happening here ? is there some 
> incompatibility with nftables (firewalld service is disabled) and libvirt ?
As I've said above, libvirt is still using iptables to add its rules, 
but there is no incompatibility with nftables, and in fact the rules 
under the hood end up being nftables rules.