Hi, I am trying to preload two shared libraries while running Qemu using libvirt. To preload the shared libraries I have added an environment variable in the $domain.xml file with the following command: ``` virt-xml generic --edit --confirm --qemu-commandline 'env=LD_PRELOAD=/home/probir/Downloads/kpv_temp/temp/kvmTest/build_kvm/libperform.so:/home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/libmonitor/installDir/lib/libmonitor.so' ``` However, while running the domain, I get the following message of failure to preload the shared files in the log. ``` 2019-07-17 17:35:22.939+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.27 (Marc Deslauriers <marc.deslauriers@ubuntu.com> Tue, 02 Jul 2019 09:22:37 -0400), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.39), hostname: probir-XPS-13-9343 LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=spice LD_PRELOAD=/home/probir/Downloads/kpv_temp/temp/kvmTest/build_kvm/libperform.so:/home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/libmonitor/installDir/lib/libmonitor.so /usr/bin/kvm-spice -name generic -S -machine pc-i440fx-xenial,accel=kvm,usb=off -cpu Broadwell-noTSX-IBRS -m 4096 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 8f755c9c-adf8-4baf-8d6a-947ebdf61ff8 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-generic/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/generic.img,format=qcow2,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,id=drive-ide0-0-1,readonly=on -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev tap,fd=26,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:82:62:ba,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on Domain id=9 is tainted: custom-argv ERROR: ld.so: object '/home/probir/Downloads/kpv_temp/temp/kvmTest/build_kvm/libperform.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored. ERROR: ld.so: object '/home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/libmonitor/installDir/lib/libmonitor.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored. ERROR: ld.so: object '/home/probir/Downloads/kpv_temp/temp/kvmTest/build_kvm/libperform.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored. ERROR: ld.so: object '/home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/libmonitor/installDir/lib/libmonitor.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored. char device redirected to /dev/pts/21 (label charserial0) ``` I further went ahead to verify the shared library with ldd and ls -l. Following results came out: ``` probir@probir-XPS-13-9343:~/Downloads/kpv_temp/temp/kvmTest/test/kvm$ ls -l /home/probir/Downloads/kpv_temp/temp/kvmTest/build_kvm/libperform.so -rwxrwxr-x 1 probir probir 834240 Jun 13 17:53 /home/probir/Downloads/kpv_temp/temp/kvmTest/build_kvm/libperform.so probir@probir-XPS-13-9343:~/Downloads/kpv_temp/temp/kvmTest/test/kvm$ ldd /home/probir/Downloads/kpv_temp/temp/kvmTest/build_kvm/libperform.so linux-vdso.so.1 => (0x00007ffdd97d5000) libxed.so => /home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/xed/kits/xed-install-base/lib/libxed.so (0x00007f706442e000) libmonitor.so.0 => /home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/libmonitor/installDir/lib/libmonitor.so.0 (0x00007f706420a000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7063fed000) libbfd-2.26.1-system.so => /usr/lib/x86_64-linux-gnu/libbfd-2.26.1-system.so (0x00007f7063ca8000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f7063926000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7063710000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7063346000) /lib64/ld-linux-x86-64.so.2 (0x00007f7064f48000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7063142000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f7062f28000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7062c1f000) ``` and ``` probir@probir-XPS-13-9343:~/Downloads/kpv_temp/temp/kvmTest/test/kvm$ ls -l /home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/libmonitor/installDir/lib/libmonitor.so lrwxrwxrwx 1 probir probir 19 Mar 28 13:45 /home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/libmonitor/installDir/lib/libmonitor.so -> libmonitor.so.0.0.0 probir@probir-XPS-13-9343:~/Downloads/kpv_temp/temp/kvmTest/test/kvm$ ldd /home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/libmonitor/installDir/lib/libmonitor.so linux-vdso.so.1 => (0x00007ffc5d12b000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff8ae23e000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff8ade74000) /lib64/ld-linux-x86-64.so.2 (0x00007ff8ae666000) ``` I cannot figure out what is causing the failure of preload. Previously, I manually preloaded the same shared libraries for other application. Probir
Peter Krempa
2019-Jul-18 08:25 UTC
Re: [libvirt-users] Preloading shared library with libVirt
On Wed, Jul 17, 2019 at 13:33:08 -0500, Probir Roy wrote:> Hi, > > I am trying to preload two shared libraries while running Qemu using > libvirt. To preload the shared libraries I have added an environment > variable in the $domain.xml file with the following command: > > ``` > virt-xml generic --edit --confirm --qemu-commandline > 'env=LD_PRELOAD=/home/probir/Downloads/kpv_temp/temp/kvmTest/build_kvm/libperform.so:/home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/libmonitor/installDir/lib/libmonitor.so' > > ``` > However, while running the domain, I get the following message of failure > to preload the shared files in the log. > > ``` > 2019-07-17 17:35:22.939+0000: starting up libvirt version: 1.3.1, package: > 1ubuntu10.27 (Marc Deslauriers <marc.deslauriers@ubuntu.com> Tue, 02 Jul > 2019 09:22:37 -0400), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.39), > hostname: probir-XPS-13-9343 > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin > QEMU_AUDIO_DRV=spice > LD_PRELOAD=/home/probir/Downloads/kpv_temp/temp/kvmTest/build_kvm/libperform.so:/home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/libmonitor/installDir/lib/libmonitor.so > /usr/bin/kvm-spice -name generic -S -machine[...]> ERROR: ld.so: object > '/home/probir/Downloads/kpv_temp/temp/kvmTest/build_kvm/libperform.so' from > LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored. > ERROR: ld.so: object > '/home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/libmonitor/installDir/lib/libmonitor.so' > from LD_PRELOAD cannot be preloaded (cannot open shared object file): > ignored.My rough guess would be that the process we forked off for qemu has security confinment applied (apparmor and unix permissions changed in case of ubuntu) and ld.so is thus unable to open the libraries you want it to preload. For any other resource libvirt configures the confinment such as that the process has access, but in this case this is out of our scope and you have to make sure that the libs are accessible yourself.
Daniel P. Berrangé
2019-Jul-18 09:01 UTC
Re: [libvirt-users] Preloading shared library with libVirt
On Thu, Jul 18, 2019 at 10:25:57AM +0200, Peter Krempa wrote:> On Wed, Jul 17, 2019 at 13:33:08 -0500, Probir Roy wrote: > > Hi, > > > > I am trying to preload two shared libraries while running Qemu using > > libvirt. To preload the shared libraries I have added an environment > > variable in the $domain.xml file with the following command: > > > > ``` > > virt-xml generic --edit --confirm --qemu-commandline > > 'env=LD_PRELOAD=/home/probir/Downloads/kpv_temp/temp/kvmTest/build_kvm/libperform.so:/home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/libmonitor/installDir/lib/libmonitor.so' > > > > ``` > > However, while running the domain, I get the following message of failure > > to preload the shared files in the log. > > > > ``` > > 2019-07-17 17:35:22.939+0000: starting up libvirt version: 1.3.1, package: > > 1ubuntu10.27 (Marc Deslauriers <marc.deslauriers@ubuntu.com> Tue, 02 Jul > > 2019 09:22:37 -0400), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.39), > > hostname: probir-XPS-13-9343 > > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin > > QEMU_AUDIO_DRV=spice > > LD_PRELOAD=/home/probir/Downloads/kpv_temp/temp/kvmTest/build_kvm/libperform.so:/home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/libmonitor/installDir/lib/libmonitor.so > > /usr/bin/kvm-spice -name generic -S -machine > > [...] > > > ERROR: ld.so: object > > '/home/probir/Downloads/kpv_temp/temp/kvmTest/build_kvm/libperform.so' from > > LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored. > > ERROR: ld.so: object > > '/home/probir/Downloads/kpv_temp/temp/kvmTest/dependencies/libmonitor/installDir/lib/libmonitor.so' > > from LD_PRELOAD cannot be preloaded (cannot open shared object file): > > ignored. > > My rough guess would be that the process we forked off for qemu has > security confinment applied (apparmor and unix permissions changed in > case of ubuntu) and ld.so is thus unable to open the libraries you want > it to preload.Yep, $HOME permissions would be the most likely explanation as $HOME is usually locked down by default which would prevent the qemu user from loading anything from under it. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|