libvirt users - Oct 2016 - Libvirt newer than 2.1.0 doesnt start up

Hi all,

I have some problems with libvirt >2.1.0 (so 2.2.0 and 2.3.0): I have a
working environment using libvirt 2.1.0 with 1 domain (called "mail")
and one network (called "default6"), both are autostarted. I have no
problems with this environment using libvirt 2.1.0, everything works great.

But after (testwise) updating to libvirt 2.2.0 or 2.3.0, libvirt doesn't
start up anymore.

Some investigations showed that apparently my "default6" network is
the
problem. Libvirt is starting correctly after disabling all autostarts.
When I then virsh' into the system daemon and type "net-start
default6",
then nothing happens and that command just hangs infinitely.

Further information, relevant log entries and e.g. the used default6.xml
configuration file can be found here, as I first thought it would be a
configuration error from my side:
https://bbs.archlinux.org/viewtopic.php?pid=1653221

Do you maybe have an idea what could be going on? Or do you maybe need
more information, and if yes, which exactly?

Thanks in advance
Regards
Rene

On 10/17/2016 10:32 AM, Rene Pasing wrote:
> Hi all,
>
> I have some problems with libvirt >2.1.0 (so 2.2.0 and 2.3.0): I have a
> working environment using libvirt 2.1.0 with 1 domain (called
"mail")
> and one network (called "default6"), both are autostarted. I have
no
> problems with this environment using libvirt 2.1.0, everything works great.
>
> But after (testwise) updating to libvirt 2.2.0 or 2.3.0, libvirt
doesn't
> start up anymore.
>
> Some investigations showed that apparently my "default6" network
is the
> problem. Libvirt is starting correctly after disabling all autostarts.
> When I then virsh' into the system daemon and type "net-start
default6",
> then nothing happens and that command just hangs infinitely.
>
> Further information, relevant log entries and e.g. the used default6.xml
> configuration file can be found here, as I first thought it would be a
> configuration error from my side:
> https://bbs.archlinux.org/viewtopic.php?pid=1653221
>
> Do you maybe have an idea what could be going on? Or do you maybe need
> more information, and if yes, which exactly?
Nothing seems unusual about your config, and your troubleshooting so far 
eliminates some common problems. I've just tried starting an IPv6 
network with a libvirtd built from the latest sources (but on Fedora 24) 
and don't experience your problem.

I guess at this point what's most likely to get closer to the root of 
the problem would be to attach gdb to the hung libvirtd process and run 
"thread apply all bt". Presumably one of the threads will be waiting
on
a system call to return and that may give a further clue.

On 17.10.2016 22:32, Rene Pasing wrote:
> Hi all,
> 
> I have some problems with libvirt >2.1.0 (so 2.2.0 and 2.3.0): I have a
> working environment using libvirt 2.1.0 with 1 domain (called
"mail")
> and one network (called "default6"), both are autostarted. I have
no
> problems with this environment using libvirt 2.1.0, everything works great.
> 
> But after (testwise) updating to libvirt 2.2.0 or 2.3.0, libvirt
doesn't
> start up anymore.
> 
> Some investigations showed that apparently my "default6" network
is the
> problem. Libvirt is starting correctly after disabling all autostarts.
> When I then virsh' into the system daemon and type "net-start
default6",
> then nothing happens and that command just hangs infinitely.
> 
> Further information, relevant log entries and e.g. the used default6.xml
> configuration file can be found here, as I first thought it would be a
> configuration error from my side:
> https://bbs.archlinux.org/viewtopic.php?pid=1653221
> 
> Do you maybe have an idea what could be going on? Or do you maybe need
> more information, and if yes, which exactly?
Those "Network is already in use by interface virbr1" error messages
worry me. Is the bridge active before you start libvirtd? If so we might
be losing netowrk state XML (or fail to parse it) and thus think network
is not started and because of autostart we try to start it again (which
fails).
Also, can you please share debug logs (apart from what Laine already
requested)

http://wiki.libvirt.org/page/DebugLogs

Before you start libvirt, are there any files under
/var/run/libvirt/network/ ? If so, what's their content?

Michal

On 10/19/2016 01:58 AM, Michal Privoznik wrote:
> On 17.10.2016 22:32, Rene Pasing wrote:
>> Hi all,
>>
>> I have some problems with libvirt >2.1.0 (so 2.2.0 and 2.3.0): I
have a
>> working environment using libvirt 2.1.0 with 1 domain (called
"mail")
>> and one network (called "default6"), both are autostarted. I
have no
>> problems with this environment using libvirt 2.1.0, everything works
great.
>>
>> But after (testwise) updating to libvirt 2.2.0 or 2.3.0, libvirt
doesn't
>> start up anymore.
>>
>> Some investigations showed that apparently my "default6"
network is the
>> problem. Libvirt is starting correctly after disabling all autostarts.
>> When I then virsh' into the system daemon and type "net-start
default6",
>> then nothing happens and that command just hangs infinitely.
>>
>> Further information, relevant log entries and e.g. the used
default6.xml
>> configuration file can be found here, as I first thought it would be a
>> configuration error from my side:
>> https://bbs.archlinux.org/viewtopic.php?pid=1653221
>>
>> Do you maybe have an idea what could be going on? Or do you maybe need
>> more information, and if yes, which exactly?
> Those "Network is already in use by interface virbr1" error
messages
> worry me. Is the bridge active before you start libvirtd? If so we might
> be losing netowrk state XML (or fail to parse it) and thus think network
> is not started and because of autostart we try to start it again (which
> fails).
My suspicion was that error happens during a 2nd attempt to start the 
network when libvirtd has already hung once while trying to start the 
network, and was killed, thus preventing proper cleanup of a 
half-started network.