libvirt users - Nov 2019 - Re: It takes long time to start kvm virtual machine with nwfilter in docker container.

1.  It takes minutes to start the virtual machine when I add
"filterref" to
libvirt.xml and run command "virsh start  vm1".
It also takes minutes to destroy the virtual machine.

<interface type="bridge">
  <mac address="fa:16:3e:fa:f7:94"/>
  <target dev="tap69e948b0-bf"/>
  <source bridge="br02"/>
  <model type="virtio"/>
  <filterref filter="no-arp-spoofing">
    <parameter name="IP" value="192.168.2.2"/>
    <parameter name="MAC" value="fa:16:3e:fa:f7:94"/>
  </filterref>
</interface>

2. I found some logs in /var/log/libvirt/libvirtd.log


2019-11-04 03:46:18.495+0000: 15257: info : virFirewallApplyRule:815 :
Applying rule '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i
tap69e948b0-bf -j libvirt-J-tap69e948b0-bf'
2019-11-04 03:46:18.495+0000: 15257: debug : virCommandRunAsync:2499 :
About to run /usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i
tap69e948b0-bf -j libvirt-J-tap69e948b0-bf
2019-11-04 03:46:18.496+0000: 15257: debug : virFileClose:111 : Closed fd 24
2019-11-04 03:46:18.496+0000: 15257: debug : virFileClose:111 : Closed fd 26
2019-11-04 03:46:18.496+0000: 15257: debug : virFileClose:111 : Closed fd 29
2019-11-04 03:46:18.496+0000: 15257: debug : virCommandRunAsync:2502 :
Command result 0, with PID 16291
2019-11-04 03:46:19.242+0000: 15257: debug : virCommandRun:2350 : Result
invalid value 255, stdout: '' stderr: '2019-11-04 03:46:19.239+0000:
16291:
debug : virFileClose:111 : Closed fd 26
2019-11-04 03:46:19.239+0000: 16291: debug : virFileClose:111 : Closed fd 29
2019-11-04 03:46:19.239+0000: 16291: debug : virFileClose:111 : Closed fd 24
Illegal target name 'libvirt-J-tap69e948b0-bf'.
'
2019-11-04 03:46:19.242+0000: 15257: debug : virFirewallApplyRuleDirect:704
: Ignoring error running command
2019-11-04 03:46:19.242+0000: 15257: debug : virFileClose:111 : Closed fd 25
2019-11-04 03:46:19.243+0000: 15257: debug : virFileClose:111 : Closed fd 28
2019-11-04 03:46:19.243+0000: 15257: info : virFirewallApplyRule:815 :
Applying rule '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o
tap69e948b0-bf -j libvirt-P-tap69e948b0-bf'
2019-11-04 03:46:19.243+0000: 15257: debug : virCommandRunAsync:2499 :
About to run /usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o
tap69e948b0-bf -j libvirt-P-tap69e948b0-bf
2019-11-04 03:46:19.243+0000: 15257: debug : virFileClose:111 : Closed fd 24
2019-11-04 03:46:19.243+0000: 15257: debug : virFileClose:111 : Closed fd 26
2019-11-04 03:46:19.244+0000: 15257: debug : virFileClose:111 : Closed fd 29
2019-11-04 03:46:19.244+0000: 15257: debug : virCommandRunAsync:2502 :
Command result 0, with PID 16292
2019-11-04 03:46:19.990+0000: 15257: debug : virCommandRun:2350 : Result
invalid value 255, stdout: '' stderr: '2019-11-04 03:46:19.986+0000:
16292:
debug : virFileClose:111 : Closed fd 26
2019-11-04 03:46:19.986+0000: 16292: debug : virFileClose:111 : Closed fd 29
2019-11-04 03:46:19.986+0000: 16292: debug : virFileClose:111 : Closed fd 24
Illegal target name 'libvirt-P-tap69e948b0-bf'.
'
2019-11-04 03:46:19.990+0000: 15257: debug : virFirewallApplyRuleDirect:704
: Ignoring error running command
2019-11-04 03:46:19.990+0000: 15257: debug : virFileClose:111 : Closed fd 25
2019-11-04 03:46:19.990+0000: 15257: debug : virFileClose:111 : Closed fd 28

3. It works fine on hosts.
4. It works fine in other host's libvirt container.


How can I solve this problem?

Best regards,
John
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20191104/cfeef1e0/attachment.htm>

On Mon, Nov 04, 2019 at 04:55:07PM +0800, John Y. wrote:
> 1.  It takes minutes to start the virtual machine when I add
"filterref" to
> libvirt.xml and run command "virsh start  vm1".
> It also takes minutes to destroy the virtual machine.
You don't mention which version of libvirt you have...

My guess is that your docker container has set an enourmous ulimit
for max files. On a normal host it is 1024, but on docker I've seen
it default to 1 million.

This impacts libvirt when it spawns processes, which is common with
nwfilter in particular, because we must close all open file handles.
In v5.6.0 we  added code to let libvirt use /proc/$PID/fd to close
file handles, which is massively faster when ulimits are high.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Thank you for your reply.
According to what you said, I set open files to 1024 and solved the problem.
Thanks!

-------
Sorry for not mentioning the version information.
The version of libvirt is 4.5.0.

Regards,
John

Daniel P. Berrangé <berrange@redhat.com> 于2019年11月4日周一 下午6:04写道：
> On Mon, Nov 04, 2019 at 04:55:07PM +0800, John Y. wrote:
> > 1.  It takes minutes to start the virtual machine when I add
"filterref"
> to
> > libvirt.xml and run command "virsh start  vm1".
> > It also takes minutes to destroy the virtual machine.
>
> You don't mention which version of libvirt you have...
>
> My guess is that your docker container has set an enourmous ulimit
> for max files. On a normal host it is 1024, but on docker I've seen
> it default to 1 million.
>
> This impacts libvirt when it spawns processes, which is common with
> nwfilter in particular, because we must close all open file handles.
> In v5.6.0 we  added code to let libvirt use /proc/$PID/fd to close
> file handles, which is massively faster when ulimits are high.
>
>
> Regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>