libvirt users - Jun 2011 - libvirt-bin MOSTLY fails to create default virtual network

I have observed 3 scenarios after I reboot the host with 2 guests (which are
supposed to autostart after boot/reboot of the host):
1. Default network is created but I can not ping guests. Gests don't have
assigned IP address or they both have IP address 192.168.122.100.
2. Default network is not created.
3. Default netork is properly created and everything works fine.
In each case guests are started and they work properly. Cases 1. and 2.
happen a lot more often then case 3.

My solutions:
- When I restart libvirt-bin then everything is fine.
- When I set 'sleep 40' in 'pre-start' section of
/etc/init/libvirt-bin.conf
and modify 'start on' section in the following way:

start on (runlevel [2345] and net-device-up IFACE=br0)

then everything works fine.


However, none of the above solutions indeed solves the problem. Please, let
me know if there is solution to this problem or what are your suggestions
regarding this issue.

****
HOST$ uname -a
Linux ras 2.6.32-32-server #62-Ubuntu SMP Wed Apr 20 22:07:43 UTC 2011
x86_64 GNU/Linux

GUEST1 and GUEST2
$ uname -a
Linux server1-desktop 2.6.32-32-generic #62-Ubuntu SMP Wed Apr 20 21:54:21
UTC 2011 i686 GNU/Linux

HOST$ libvirtd --version
libvirtd (libvirt) 0.7.5

My idea was that some kernel modules sometimes are loaded before libvirt-bin
starts, and sometimes libvirt-bin starts earlier, or in the middle of
loading of these modules, that is why 3 scenarios can be observed.
That is why I have added:
 cp /proc/modules /home/wsadmin2/modules1
 cp /proc/modules /home/wsadmin2/modules2
to 'pre-start' and 'post-start' parts respectively of
/etc/init/libvirt-bin.conf.

Of course, beween execution of
- pre-start part
- exec /usr/sbin/libvirtd -d
- post-start part
other kernel modules can be loaded, so we can not be sure which additional
modules in file 'modules2' were indeed loaded before libvirtd was
actually
started.

Here are my observations:
Scenatio 1.
33 (fixed) modules were logged - call them BASE MODULES.

Scenatio 2.
Beside BASE MODULES + the following modules were loaded:
> iptable_filter 1841 0 - Live 0xffffffffa007c000
> ip_tables 18201 1 iptable_filter, Live 0xffffffffa0172000
> x_tables 22361 1 ip_tables, Live 0xffffffffa0164000
Scenatio 3.
Beside BASE MODULES + the following modules were loaded:

$ diff modules1 modules2
1c1,6
< iptable_filter 1841 0 - Live 0xffffffffa008b000
---
> xt_state 1490 14 - Live 0xffffffffa01ab000
> xt_tcpudp 2667 12 - Live 0xffffffffa01a5000
> nf_conntrack_ipv4 12742 14 - Live 0xffffffffa019b000
> nf_conntrack 73326 2 xt_state,nf_conntrack_ipv4, Live 0xffffffffa017d000
> nf_defrag_ipv4 1481 1 nf_conntrack_ipv4, Live 0xffffffffa0177000
> iptable_filter 1841 1 - Live 0xffffffffa008b000
3c8
< x_tables 22361 1 ip_tables, Live 0xffffffffa0162000
---
> x_tables 22361 3 xt_state,xt_tcpudp,ip_tables, Live 0xffffffffa0162000
Other observations:
Scenatio 1.
No errors in libvirtd.log
Scenario 2.

10:06:06.980: error : networkAddIptablesRules:750 : failed to add iptables
rule to allow DHCP requests from 'virbr0': Invalid argument
(...)
10:06:08.630: error : qemudNetworkIfaceConnect:1445 : Failed to add tap
interface to bridge 'virbr0': No such device
10:06:08.630: error : qemuAutostartDomain:584 : Failed to autostart VM
'cluster_server1': Failed to add tap interface to bridge
'virbr0': No such
device

10:06:08.921: info : qemudDispatchSignalEvent:390 : Received unexpected
signal 17
10:06:08.927: debug : virExecWithHook:617 : LC_ALL=C /usr/bin/kvm -help
10:06:08.938: info : qemudDispatchSignalEvent:390 : Received unexpected
signal 17
10:06:09.000: error : qemudNetworkIfaceConnect:1445 : Failed to add tap
interface to bridge 'virbr0': No such device
10:06:09.000: error : qemuAutostartDomain:584 : Failed to autostart VM
'cluster_server2': Failed to add tap interface to bridge
'virbr0': No such
device
(...)
Scenatio 3.
No errors in libvirtd.log


I didn't see any valuable logs in /var/log/boot.log, /var/log/messages,
/var/log/syslog. Please let me know if you want to see them anyway or have
any suggestions right away.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20110606/54a07ab3/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: scenario1
Type: application/octet-stream
Size: 19318 bytes
Desc: not available
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20110606/54a07ab3/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: scenario2
Type: application/octet-stream
Size: 8295 bytes
Desc: not available
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20110606/54a07ab3/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: scenario3
Type: application/octet-stream
Size: 2367 bytes
Desc: not available
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20110606/54a07ab3/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: libvirt-bin.conf
Type: application/octet-stream
Size: 925 bytes
Desc: not available
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20110606/54a07ab3/attachment-0003.obj>

On 06/06/2011 03:24 PM, m m wrote:
> I have observed 3 scenarios after I reboot the host with 2 guests 
> (which are supposed to autostart after boot/reboot of the host):
> 1. Default network is created but I can not ping guests. Gests don't 
> have assigned IP address or they both have IP address 192.168.122.100.
> 2. Default network is not created.
> 3. Default netork is properly created and everything works fine.
> In each case guests are started and they work properly. Cases 1. and 
> 2. happen a lot more often then case 3.
>
> My solutions:
> - When I restart libvirt-bin then everything is fine.
> - When I set 'sleep 40' in 'pre-start' section of 
> /etc/init/libvirt-bin.conf and modify 'start on' section in the 
> following way:
>
> start on (runlevel [2345] and net-device-up IFACE=br0)
>
> then everything works fine.
>
>
> However, none of the above solutions indeed solves the problem. 
> Please, let me know if there is solution to this problem or what are 
> your suggestions regarding this issue.
>
> ****
> HOST$ uname -a
> Linux ras 2.6.32-32-server #62-Ubuntu SMP Wed Apr 20 22:07:43 UTC 2011 
> x86_64 GNU/Linux
>
> GUEST1 and GUEST2
> $ uname -a
> Linux server1-desktop 2.6.32-32-generic #62-Ubuntu SMP Wed Apr 20 
> 21:54:21 UTC 2011 i686 GNU/Linux
>
> HOST$ libvirtd --version
> libvirtd (libvirt) 0.7.5
>
> My idea was that some kernel modules sometimes are loaded before 
> libvirt-bin starts, and sometimes libvirt-bin starts earlier, or in 
> the middle of loading of these modules, that is why 3 scenarios can be 
> observed.
> That is why I have added:
>  cp /proc/modules /home/wsadmin2/modules1
>  cp /proc/modules /home/wsadmin2/modules2
> to 'pre-start' and 'post-start' parts respectively of 
> /etc/init/libvirt-bin.conf.
>
> Of course, beween execution of
> - pre-start part
> - exec /usr/sbin/libvirtd -d
> - post-start part
> other kernel modules can be loaded, so we can not be sure which 
> additional modules in file 'modules2' were indeed loaded before 
> libvirtd was actually started.
>
> Here are my observations:
> Scenatio 1.
> 33 (fixed) modules were logged - call them BASE MODULES.
>
> Scenatio 2.
> Beside BASE MODULES + the following modules were loaded:
>
> > iptable_filter 1841 0 - Live 0xffffffffa007c000
> > ip_tables 18201 1 iptable_filter, Live 0xffffffffa0172000
> > x_tables 22361 1 ip_tables, Live 0xffffffffa0164000
In your "scenario 1" the modules necessary for iptables to function 
haven't been loaded. Each virtual network requires a set of iptables 
rules to function properly, and if there is no iptables, the rules can't 
be added and network creation fails. Your libvirtd.log confirms that 
problem:

   10:06:06.980: error : networkAddIptablesRules:750 : failed to add 
iptables rule to allow DHCP requests from 'virbr0': Invalid argument


0.7.5 is quite old (> 1 year), is there perhaps a newer pre-built 
version of libvirt available for your release of Ubuntu?

It seems that Ubuntu's method of starting up libvirtd (at least on your 
machine) is missing a dependency to force iptables to be loaded before 
libvirtd starts. I'm unfamiliar with Ubuntu/debian startup stuff 
(upstart, I guess), but that's where you need to look for the solution.

I found the following in the ubuntu bug database, which seems similar, 
but not identical (basically the system networking isn't completely 
ready by the time the autostart domains are brought up):

https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/495394
>
> Scenatio 3.
> Beside BASE MODULES + the following modules were loaded:
>
> $ diff modules1 modules2
> 1c1,6
> < iptable_filter 1841 0 - Live 0xffffffffa008b000
> ---
> > xt_state 1490 14 - Live 0xffffffffa01ab000
> > xt_tcpudp 2667 12 - Live 0xffffffffa01a5000
> > nf_conntrack_ipv4 12742 14 - Live 0xffffffffa019b000
> > nf_conntrack 73326 2 xt_state,nf_conntrack_ipv4, Live
0xffffffffa017d000
conntrack is used by iptables. I haven't tried it, but wouldn't be 
surprised if adding rules that required conntrack failed if it hadn't 
been loaded, or something like that.
> > nf_defrag_ipv4 1481 1 nf_conntrack_ipv4, Live 0xffffffffa0177000
> > iptable_filter 1841 1 - Live 0xffffffffa008b000
> 3c8
> < x_tables 22361 1 ip_tables, Live 0xffffffffa0162000
> ---
> > x_tables 22361 3 xt_state,xt_tcpudp,ip_tables, Live 0xffffffffa0162000
>
> Other observations:
> Scenatio 1.
> No errors in libvirtd.log
> Scenario 2.
>
> 10:06:06.980: error : networkAddIptablesRules:750 : failed to add 
> iptables rule to allow DHCP requests from 'virbr0': Invalid
argument
> (...)
> 10:06:08.630: error : qemudNetworkIfaceConnect:1445 : Failed to add 
> tap interface to bridge 'virbr0': No such device
> 10:06:08.630: error : qemuAutostartDomain:584 : Failed to autostart VM 
> 'cluster_server1': Failed to add tap interface to bridge
'virbr0': No
> such device
>
> 10:06:08.921: info : qemudDispatchSignalEvent:390 : Received 
> unexpected signal 17
> 10:06:08.927: debug : virExecWithHook:617 : LC_ALL=C /usr/bin/kvm -help
> 10:06:08.938: info : qemudDispatchSignalEvent:390 : Received 
> unexpected signal 17
> 10:06:09.000: error : qemudNetworkIfaceConnect:1445 : Failed to add 
> tap interface to bridge 'virbr0': No such device
> 10:06:09.000: error : qemuAutostartDomain:584 : Failed to autostart VM 
> 'cluster_server2': Failed to add tap interface to bridge
'virbr0': No
> such device
> (...)
> Scenatio 3.
> No errors in libvirtd.log
>
>
> I didn't see any valuable logs in /var/log/boot.log, 
> /var/log/messages, /var/log/syslog. Please let me know if you want to 
> see them anyway or have any suggestions right away.
>
>
> _______________________________________________
> libvirt-users mailing list
> libvirt-users at redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users