libvirt users - Mar 2015 - Re: [virt-tools-list] libvirt/virt-manager source integration with xen

On 03/02/2015 10:55 AM, Wayne Mills wrote:
> Hi,
> 
> I built and installed Xen 4.5.0 from source, on top of Ubuntu 14.04.2,
using
> "make world" and "make install" targets.  I then
installed latest virt-manager
> from pre-built packages.  After bringing up virt-manager I attempt to
connect
> to Xen hypervisor and got:
> 
>      unable to connect to libvirt
>      Failed to connect socket to /var/run/libvirt/libvirt-sock' No such
file
> or directory
> 
> I then noticed libvirt-bin isn't running.  If I try to issue
'service
> libvirt-bin start' it gives me back a process number, but it apparently
dies
> quickly because the service still shows as down.  Three log files are
touched
> during my start attempt:
> 
>      -rw-r--r-- 1 root      root  39757 Feb 27 06:00
/var/log/xenstored-access.log
>      -rw-r--r-- 1 root      root  44908 Feb 27 06:00
> /var/log/libvirt/libxl/libxl-driver.log
>      -rw------- 1 root      root  46396 Feb 27 06:00
/var/log/libvirt/libvirtd.log
> 
> * xenstored-access.log has 22 new entries, grouped into pairs that
increment
> an  "Axx" identifier and go from A63 to A73.  Here is the A63
logs:
> 
>      [20150227T11:00:29.478Z]  A63          newconn
>      [20150227T11:00:29.479Z]  A63          endconn
> 
> * libxl-driver.log also has 11 log groupings that are just the same set of
> logs repeated 11 times.  Here is the first group:
> 
>      xc: detail: sysctl operation failed -- need to rebuild the user-space
> tool set?
>      libxl: error: libxl.c:4320:libxl_get_physinfo: getting physinfo:
> Permission denied
>      xc: debug: hypercall buffer: total allocations:7 total releases:7
>      xc: debug: hypercall buffer: current allocations:0 maximum
allocations:1
>      xc: debug: hypercall buffer: cache current size:1
>      xc: debug: hypercall buffer: cache hits:6 misses:1 toobig:0
> 
> * libvirtd.log also has 11 log groupings, that are just the same set of
logs
> repeated 11 times.  Here is that group:
> 
>      2015-02-27 11:00:29.479+0000: 4842: info : libvirt version: 1.2.2
>      2015-02-27 11:00:29.479+0000: 4842: error : libxlDriverConfigNew:1131
:
> Unable to configure libxl's memory management parameters
>      2015-02-27 11:00:29.479+0000: 4842: error : virStateInitialize:749 :
> Initialization of LIBXL state driver failed: Unknown problem
>      2015-02-27 11:00:29.479+0000: 4842: error : daemonRunStateInit:920 :
> Driver state initialization failed
> 
> When I google for libxl_get_physinfo I see other reported errors during
"xl
> info" and other "xl" commands.    I tried "xl
info" and that works for me and
> does not alter the log files:
> 
> root@<server>:~# xl info
> host                   : server
> release                : 3.13.0-46-generic
> version                : #75-Ubuntu SMP Tue Feb 10 15:24:04 UTC 2015
> machine                : x86_64
> nr_cpus                : 8
> max_cpu_id             : 7
> nr_nodes               : 2
> cores_per_socket       : 4
> threads_per_core       : 1
> cpu_mhz                : 2400
> hw_caps                :
> bfebfbff:2c100800:00000000:00003f00:17bee3ff:00000000:00000001:00000000
> virt_caps              : hvm hvm_directio
> total_memory           : 98168
> free_memory            : 128
> sharing_freed_memory   : 0
> sharing_used_memory    : 0
> outstanding_claims     : 0
> free_cpus              : 0
> xen_major              : 4
> xen_minor              : 5
> xen_extra              : .0
> xen_version            : 4.5.0
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : Mon Jan 12 11:30:05 2015 -0500 git:a8ac229
> xen_commandline        : placeholder
> cc_compiler            : gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2
> cc_compile_by          : root
> cc_compile_domain      : cisco.com <http://cisco.com>
> cc_compile_date        : Thu Feb 19 07:26:00 EST 2015
> xend_config_format     : 4
> root@<server>:~#
> 
> After consulting with xen-users mailer, I was given this response by
"Ian":
> 
> "Did you install libvirt from source or from packages? If the latter
then you
> may have a disconnect between the packaged version and your source-built
Xen.
> You'll probably need to rebuild libvirt against your Xen
libraries."
> 
> What are the virt-related steps to take to test out Ian's idea?  I have
root
> access to one machine that serves both as a host to VM's as well as a
build
> environment for source builds such as this.  I already built xen 4.5.0 and
> installed it.  Do I now need to separately download and build libvirt, then
> download and build virt-manager?  Is it important to place the virt* code
> within the xen source tree, or do something else so that the virt* code
> compiles "against" an appropriate xen environment? 
> 
> In summary, I would like some guidance on how to build libvirt, and if
needed,
> virt-manager, source against a specific Xen version.
> 
Distro packages of virt-manager should work fine, once you get libvirt building.

I recommend you download the ubuntu .deb config files for libxl, and
rebuild+install libxl with the exact same build flags your distro packages
use. They've already figured out the necessary flags to make things work
together with the other distro packages. Then do the same with libvirt.

- Cole

Thanks Cole, unfortunately I'm a n00b in this area so your response just
raised many more questions for me :/

* what are the config file names for libxl, libvirt and virt-manager
* where are they located in the respective source trees
* are the build flags defined in those config files?
* libxl is part of xen 4.5.0 distro (xen/tools/libxl after untarring). But
no config file is in xen/tools/libxl directory
* I do see CFLAGS definitions in the various makefiles within each
directory.  Are those the build flags you are referring to?  here's an
example from libxl/Makefile:

CFLAGS += -Werror -Wno-format-zero-length -Wmissing-declarations \
        -Wno-declaration-after-statement -Wformat-nonliteral
CFLAGS += -I. -fPIC

* by distro I assume you mean binary package, and that I can install
virt-manager binary package on top of libxl/libvirt/xen that I built and
installed locally from source.  if I have compiled libxl/libvirt/xen with
virt-manager compile flags.  Does that summarize what you are
recommending?  If so then how do I know virt-manager flags?  That was a
binary.  Do I have to download virt-manager source to find those?

* libxl was part of xen source tarball, and therefore has a source tree
position within xen (xen/tools/libxl as mentioned before).  Should I place
libvirt and virt-manager in some specific source tree spot relative to
xen?  Should libvirt and virt-manager be aware of libxl source location?
Is there any makefile engineering involved?

Trying to work with your answer but I have a serious background deficit to
deal with.  For this reason I love technical 'cookbooks' that provide
all
missing details :)

Thanks,
Wayne

On Mon, Mar 2, 2015 at 12:12 PM, Cole Robinson <crobinso@redhat.com>
wrote:
> On 03/02/2015 10:55 AM, Wayne Mills wrote:
> > Hi,
> >
> > I built and installed Xen 4.5.0 from source, on top of Ubuntu 14.04.2,
> using
> > "make world" and "make install" targets.  I then
installed latest
> virt-manager
> > from pre-built packages.  After bringing up virt-manager I attempt to
> connect
> > to Xen hypervisor and got:
> >
> >      unable to connect to libvirt
> >      Failed to connect socket to /var/run/libvirt/libvirt-sock' No
such
> file
> > or directory
> >
> > I then noticed libvirt-bin isn't running.  If I try to issue
'service
> > libvirt-bin start' it gives me back a process number, but it
apparently
> dies
> > quickly because the service still shows as down.  Three log files are
> touched
> > during my start attempt:
> >
> >      -rw-r--r-- 1 root      root  39757 Feb 27 06:00
> /var/log/xenstored-access.log
> >      -rw-r--r-- 1 root      root  44908 Feb 27 06:00
> > /var/log/libvirt/libxl/libxl-driver.log
> >      -rw------- 1 root      root  46396 Feb 27 06:00
> /var/log/libvirt/libvirtd.log
> >
> > * xenstored-access.log has 22 new entries, grouped into pairs that
> increment
> > an  "Axx" identifier and go from A63 to A73.  Here is the
A63 logs:
> >
> >      [20150227T11:00:29.478Z]  A63          newconn
> >      [20150227T11:00:29.479Z]  A63          endconn
> >
> > * libxl-driver.log also has 11 log groupings that are just the same
set
> of
> > logs repeated 11 times.  Here is the first group:
> >
> >      xc: detail: sysctl operation failed -- need to rebuild the
> user-space
> > tool set?
> >      libxl: error: libxl.c:4320:libxl_get_physinfo: getting physinfo:
> > Permission denied
> >      xc: debug: hypercall buffer: total allocations:7 total releases:7
> >      xc: debug: hypercall buffer: current allocations:0 maximum
> allocations:1
> >      xc: debug: hypercall buffer: cache current size:1
> >      xc: debug: hypercall buffer: cache hits:6 misses:1 toobig:0
> >
> > * libvirtd.log also has 11 log groupings, that are just the same set
of
> logs
> > repeated 11 times.  Here is that group:
> >
> >      2015-02-27 11:00:29.479+0000: 4842: info : libvirt version: 1.2.2
> >      2015-02-27 11:00:29.479+0000: 4842: error :
> libxlDriverConfigNew:1131 :
> > Unable to configure libxl's memory management parameters
> >      2015-02-27 11:00:29.479+0000: 4842: error :
virStateInitialize:749 :
> > Initialization of LIBXL state driver failed: Unknown problem
> >      2015-02-27 11:00:29.479+0000: 4842: error :
daemonRunStateInit:920 :
> > Driver state initialization failed
> >
> > When I google for libxl_get_physinfo I see other reported errors
during
> "xl
> > info" and other "xl" commands.    I tried "xl
info" and that works for
> me and
> > does not alter the log files:
> >
> > root@<server>:~# xl info
> > host                   : server
> > release                : 3.13.0-46-generic
> > version                : #75-Ubuntu SMP Tue Feb 10 15:24:04 UTC 2015
> > machine                : x86_64
> > nr_cpus                : 8
> > max_cpu_id             : 7
> > nr_nodes               : 2
> > cores_per_socket       : 4
> > threads_per_core       : 1
> > cpu_mhz                : 2400
> > hw_caps                :
> >
bfebfbff:2c100800:00000000:00003f00:17bee3ff:00000000:00000001:00000000
> > virt_caps              : hvm hvm_directio
> > total_memory           : 98168
> > free_memory            : 128
> > sharing_freed_memory   : 0
> > sharing_used_memory    : 0
> > outstanding_claims     : 0
> > free_cpus              : 0
> > xen_major              : 4
> > xen_minor              : 5
> > xen_extra              : .0
> > xen_version            : 4.5.0
> > xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> > hvm-3.0-x86_32p hvm-3.0-x86_64
> > xen_scheduler          : credit
> > xen_pagesize           : 4096
> > platform_params        : virt_start=0xffff800000000000
> > xen_changeset          : Mon Jan 12 11:30:05 2015 -0500 git:a8ac229
> > xen_commandline        : placeholder
> > cc_compiler            : gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2
> > cc_compile_by          : root
> > cc_compile_domain      : cisco.com <http://cisco.com>
> > cc_compile_date        : Thu Feb 19 07:26:00 EST 2015
> > xend_config_format     : 4
> > root@<server>:~#
> >
> > After consulting with xen-users mailer, I was given this response by
> "Ian":
> >
> > "Did you install libvirt from source or from packages? If the
latter
> then you
> > may have a disconnect between the packaged version and your
source-built
> Xen.
> > You'll probably need to rebuild libvirt against your Xen
libraries."
> >
> > What are the virt-related steps to take to test out Ian's idea?  I
have
> root
> > access to one machine that serves both as a host to VM's as well
as a
> build
> > environment for source builds such as this.  I already built xen 4.5.0
> and
> > installed it.  Do I now need to separately download and build libvirt,
> then
> > download and build virt-manager?  Is it important to place the virt*
code
> > within the xen source tree, or do something else so that the virt*
code
> > compiles "against" an appropriate xen environment?
> >
> > In summary, I would like some guidance on how to build libvirt, and if
> needed,
> > virt-manager, source against a specific Xen version.
> >
>
> Distro packages of virt-manager should work fine, once you get libvirt
> building.
>
> I recommend you download the ubuntu .deb config files for libxl, and
> rebuild+install libxl with the exact same build flags your distro packages
> use. They've already figured out the necessary flags to make things
work
> together with the other distro packages. Then do the same with libvirt.
>
> - Cole
>

On 03/02/2015 01:39 PM, Wayne Mills wrote:
> Thanks Cole, unfortunately I'm a n00b in this area so your response
just
> raised many more questions for me :/
> 
> * what are the config file names for libxl, libvirt and virt-manager
> * where are they located in the respective source trees
> * are the build flags defined in those config files?
> * libxl is part of xen 4.5.0 distro (xen/tools/libxl after untarring). But
no
> config file is in xen/tools/libxl directory
> * I do see CFLAGS definitions in the various makefiles within each
directory.
> Are those the build flags you are referring to?  here's an example from
> libxl/Makefile:
> 
> CFLAGS += -Werror -Wno-format-zero-length -Wmissing-declarations \
>         -Wno-declaration-after-statement -Wformat-nonliteral
> CFLAGS += -I. -fPIC
> 
> * by distro I assume you mean binary package, and that I can install
> virt-manager binary package on top of libxl/libvirt/xen that I built and
> installed locally from source.  if I have compiled libxl/libvirt/xen with
> virt-manager compile flags.  Does that summarize what you are recommending?
> If so then how do I know virt-manager flags?  That was a binary.  Do I have
to
> download virt-manager source to find those?
> 
I meant the files that are used to generate your distro's binary packages.
The
bits that determine how software is compiled before wrapping it up in a
distribution package.

For example, I googled 'libvirt ubuntu package' and got to this:

https://launchpad.net/ubuntu/+source/libvirt

I clicked the top link for 1.2.12-0ubuntu7:

https://launchpad.net/ubuntu/+source/libvirt/1.2.12-0ubuntu7

On this page is a link to the .deb packaging files:

https://launchpad.net/ubuntu/+archive/primary/+files/libvirt_1.2.12-0ubuntu7.debian.tar.xz

Somewhere in there ./configure command line that generates the build for
ubuntu's libvirt package. You'll want to pull that out for libvirt and
for
whatever provides libxl on ubuntu.

At least that's the process I follow on Fedora when I want to compile some
random software package from source.
> * libxl was part of xen source tarball, and therefore has a source tree
> position within xen (xen/tools/libxl as mentioned before).  Should I place
> libvirt and virt-manager in some specific source tree spot relative to xen?
> Should libvirt and virt-manager be aware of libxl source location?  Is
there
> any makefile engineering involved?
> 
If you copy the distribution ./configure lines, you shouldn't have to deal
with specifying any explicit paths.
> Trying to work with your answer but I have a serious background deficit to
> deal with.  For this reason I love technical 'cookbooks' that
provide all
> missing details :)
> 
I don't know of any explicit instructions for what you want, since these
types
of things vary a lot distro to distro and between software. Building multiple
dependent packages from source is often a non-trivial task. And if you are
'make install'ing various different build configurations and things
don't
'just work', it can be very difficult to debug things if they go wrong.

- Cole

> On Mon, Mar 2, 2015 at 12:12 PM, Cole Robinson <crobinso@redhat.com
> <mailto:crobinso@redhat.com>> wrote:
> 
>     On 03/02/2015 10:55 AM, Wayne Mills wrote:
>     > Hi,
>     >
>     > I built and installed Xen 4.5.0 from source, on top of Ubuntu
14.04.2, using
>     > "make world" and "make install" targets.  I
then installed latest
>     virt-manager
>     > from pre-built packages.  After bringing up virt-manager I attempt
to
>     connect
>     > to Xen hypervisor and got:
>     >
>     >      unable to connect to libvirt
>     >      Failed to connect socket to
/var/run/libvirt/libvirt-sock' No such file
>     > or directory
>     >
>     > I then noticed libvirt-bin isn't running.  If I try to issue
'service
>     > libvirt-bin start' it gives me back a process number, but it
apparently dies
>     > quickly because the service still shows as down.  Three log files
are
>     touched
>     > during my start attempt:
>     >
>     >      -rw-r--r-- 1 root      root  39757 Feb 27 06:00
>     /var/log/xenstored-access.log
>     >      -rw-r--r-- 1 root      root  44908 Feb 27 06:00
>     > /var/log/libvirt/libxl/libxl-driver.log
>     >      -rw------- 1 root      root  46396 Feb 27 06:00
>     /var/log/libvirt/libvirtd.log
>     >
>     > * xenstored-access.log has 22 new entries, grouped into pairs that
increment
>     > an  "Axx" identifier and go from A63 to A73.  Here is
the A63 logs:
>     >
>     >      [20150227T11:00:29.478Z]  A63          newconn
>     >      [20150227T11:00:29.479Z]  A63          endconn
>     >
>     > * libxl-driver.log also has 11 log groupings that are just the
same set of
>     > logs repeated 11 times.  Here is the first group:
>     >
>     >      xc: detail: sysctl operation failed -- need to rebuild the
user-space
>     > tool set?
>     >      libxl: error: libxl.c:4320:libxl_get_physinfo: getting
physinfo:
>     > Permission denied
>     >      xc: debug: hypercall buffer: total allocations:7 total
releases:7
>     >      xc: debug: hypercall buffer: current allocations:0 maximum
>     allocations:1
>     >      xc: debug: hypercall buffer: cache current size:1
>     >      xc: debug: hypercall buffer: cache hits:6 misses:1 toobig:0
>     >
>     > * libvirtd.log also has 11 log groupings, that are just the same
set of logs
>     > repeated 11 times.  Here is that group:
>     >
>     >      2015-02-27 11:00:29.479+0000: 4842: info : libvirt version:
1.2.2
>     >      2015-02-27 11:00:29.479+0000: 4842: error :
libxlDriverConfigNew:1131 :
>     > Unable to configure libxl's memory management parameters
>     >      2015-02-27 11:00:29.479+0000: 4842: error :
virStateInitialize:749 :
>     > Initialization of LIBXL state driver failed: Unknown problem
>     >      2015-02-27 11:00:29.479+0000: 4842: error :
daemonRunStateInit:920 :
>     > Driver state initialization failed
>     >
>     > When I google for libxl_get_physinfo I see other reported errors
during "xl
>     > info" and other "xl" commands.    I tried "xl
info" and that works for
>     me and
>     > does not alter the log files:
>     >
>     > root@<server>:~# xl info
>     > host                   : server
>     > release                : 3.13.0-46-generic
>     > version                : #75-Ubuntu SMP Tue Feb 10 15:24:04 UTC
2015
>     > machine                : x86_64
>     > nr_cpus                : 8
>     > max_cpu_id             : 7
>     > nr_nodes               : 2
>     > cores_per_socket       : 4
>     > threads_per_core       : 1
>     > cpu_mhz                : 2400
>     > hw_caps                :
>     >
bfebfbff:2c100800:00000000:00003f00:17bee3ff:00000000:00000001:00000000
>     > virt_caps              : hvm hvm_directio
>     > total_memory           : 98168
>     > free_memory            : 128
>     > sharing_freed_memory   : 0
>     > sharing_used_memory    : 0
>     > outstanding_claims     : 0
>     > free_cpus              : 0
>     > xen_major              : 4
>     > xen_minor              : 5
>     > xen_extra              : .0
>     > xen_version            : 4.5.0
>     > xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p
hvm-3.0-x86_32
>     > hvm-3.0-x86_32p hvm-3.0-x86_64
>     > xen_scheduler          : credit
>     > xen_pagesize           : 4096
>     > platform_params        : virt_start=0xffff800000000000
>     > xen_changeset          : Mon Jan 12 11:30:05 2015 -0500
git:a8ac229
>     > xen_commandline        : placeholder
>     > cc_compiler            : gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2
>     > cc_compile_by          : root
>     > cc_compile_domain      : cisco.com <http://cisco.com>
<http://cisco.com>
>     > cc_compile_date        : Thu Feb 19 07:26:00 EST 2015
>     > xend_config_format     : 4
>     > root@<server>:~#
>     >
>     > After consulting with xen-users mailer, I was given this response
by "Ian":
>     >
>     > "Did you install libvirt from source or from packages? If the
latter then you
>     > may have a disconnect between the packaged version and your
source-built Xen.
>     > You'll probably need to rebuild libvirt against your Xen
libraries."
>     >
>     > What are the virt-related steps to take to test out Ian's
idea?  I have root
>     > access to one machine that serves both as a host to VM's as
well as a build
>     > environment for source builds such as this.  I already built xen
4.5.0 and
>     > installed it.  Do I now need to separately download and build
libvirt, then
>     > download and build virt-manager?  Is it important to place the
virt* code
>     > within the xen source tree, or do something else so that the virt*
code
>     > compiles "against" an appropriate xen environment?
>     >
>     > In summary, I would like some guidance on how to build libvirt,
and if needed,
>     > virt-manager, source against a specific Xen version.
>     >
> 
>     Distro packages of virt-manager should work fine, once you get libvirt
>     building.
> 
>     I recommend you download the ubuntu .deb config files for libxl, and
>     rebuild+install libxl with the exact same build flags your distro
packages
>     use. They've already figured out the necessary flags to make things
work
>     together with the other distro packages. Then do the same with libvirt.
> 
>     - Cole
> 
>