Konrad Rzeszutek Wilk
2010-Jan-26 18:36 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
> speculation of what may be the case isn''t helping _me_ find the > problem, so i''ll try to debug this a step at a time. any suggestions > as to *specific* steps to take in debugging this would be great. > here''s all the info i can muster atm ...You are missing this one: [2010-01-26 10:09:10 4764] WARNING (image:561) domain test: device model failure: pid 5379: malfunctioning (closed sentinel), killed; see /var/log/xen/qemu-dm-test.log Can you see what it has? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
0bo0
2010-Jan-26 18:36 UTC
[Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
in another thread @xen-devel, a crash @ DomU init under Xen 4.0.x, i was advised, On Mon, Jan 25, 2010 at 11:34 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:> If qemu-dm died without logging anything, it probably crashed again. If it > produced a core file you could load that in gdb and get a symbolic > backtrace.trying to turn on core dumps in xend-config, unfortunately, Xen seems to fail @ core dump ... in addition to failing to launch a DomU which works fine under Xen3, ... [2010-01-26 09:44:36 8552] ERROR (XendDomainInfo:2327) core dump failed: id = 6 name = test: (1, ''Internal error'', ''p2m_size < nr_pages -1 (0 < fffff'') ...> My guess, since qemu-dm does seem to be working in most people''s tests, is > that either you have managed to install and not replace a mismatching > version, somehow, or the set of options getting passed to it from you domain > configuration have not been tested and do not work.speculation of what may be the case isn''t helping _me_ find the problem, so i''ll try to debug this a step at a time. any suggestions as to *specific* steps to take in debugging this would be great. here''s all the info i can muster atm ... starting with a clean opensuse 11.2 system install uname -a Linux server 2.6.31.8-0.1-default #1 SMP 2009-12-15 23:55:40 +0100 x86_64 x86_64 x86_64 GNU/Linux zypper in xen xen-devel kernel-xen kernel-xen-devel xen-tools rpm -qa | grep -i xen xen-libs-4.0.0_20809_01-28.2.x86_64 xen-tools-4.0.0_20809_01-28.2.x86_64 kernel-xen-2.6.31.8-0.1.1.x86_64 xen-devel-4.0.0_20809_01-28.2.x86_64 xen-4.0.0_20809_01-28.2.x86_64 kernel-xen-devel-2.6.31.8-0.1.1.x86_64 updatedb ls -al `locate qemu-dm | grep lib` lrwxrwxrwx 1 root root 7 2010-01-26 09:28 /usr/lib64/xen/bin/qemu-dm -> qemu-dm -rwxr-xr-x 1 root root 1287042 2010-01-19 17:13 /usr/lib/xen/bin/qemu-dm /usr/lib/xen/bin/qemu-dm --help | grep "emulator version" QEMU PC emulator version 0.10.2, Copyright (c) 2003-2008 Fabrice Bellard cat /boot/grub/menu.lst --------------------------------- ... title Xen (symlink) root (hd0,0) kernel /xen.gz loglvl=all loglvl_guest=all apic_verbosity=debug dom0_mem=1024M vga=gfx-1280x1024x32 console=vga,com1 com1=57600,8n1 iommu=1 dom0_max_vcpus=4 dom0_vcpus_pin=true sched=credit cpuidle=1 cpufreq=xen clocksource=acpi numa=on module /vmlinuz-xen root=/dev/VG0/ROOT rootfstype=ext4 rootflags=journal_checksum resume=/dev/VG0/SWAP showopts splash=silent selinux=0 SELINUX_INIT=NO elevator=cfq vga=0x31a console=tty0 console=xvc0,57600 guestdev=0000:04:07.0,0000:02:00.0,0000:04:06.0 reassign_resources iommu=1 clocksource=xen mce=off module /initrd-xen ... --------------------------------- cat /etc/xen/xend-config.sxp --------------------------------- (logfile /var/log/xen/xend.log) (loglevel DEBUG) (console-limit 1024) (keymap ''en-us'') (qemu-dm-logrotate-count 10) (dom0-min-mem 1024) (enable-dom0-ballooning no) (total_available_memory 0) (dom0-cpus 1) (enable-dump yes) (xend-unix-server yes) (xend-unix-path /var/lib/xend/xend-socket) (xend-http-server no) (xend-relocation-server no) (external-migration-tool '''') (xen-api-server ((unix))) (xend-unix-xmlrpc-server yes) (xend-tcp-xmlrpc-server no) (vnc-listen ''172.30.11.2'') (vncpasswd '''') (network-script ) (vif-script vif-bridge) (pci-passthrough-strict-check no) --------------------------------- cat /etc/xen/vm/test.cfg --------------------------------- name = ''test'' builder = ''linux'' bootloader = ''/usr/lib/xen/boot/domUloader.py'' bootargs = ''--entry=xvda1:vmlinuz-xen,/boot/initrd-xen'' disk = [ ''phy:/dev/VG0/boot,xvda,w'',''phy:/dev/VG0/swap,xvdb,w'',''phy:/dev/VG0/root,xvdc,w'',] boot = ''c'' #d is cdrom boot, c is disk boot root = ''/dev/xvdc1'' vif = [''''] vfb = [ ''type=vnc, vncdisplay=102, vnclisten=127.0.0.1'' ] extra = ''textmode=1 xencons=xvc0 elevator=noop'' maxmem = 4096 memory = 4096 vcpus = 4 cpu_cap = 100 cpu_weight = 256 localtime = 0 netif = ''yes'' on_shutdown = ''destroy'' on_reboot = ''restart'' on_crash = ''destroy'' --------------------------------- reboot ... uname -a Linux server 2.6.31.8-0.1-xen #1 SMP 2009-12-15 23:55:40 +0100 x86_64 x86_64 x86_64 GNU/Linux xm dmesg | grep "Xen version" -A2 (XEN) Xen version 4.0.0_20809_01-28.2 (abuild@) (gcc version 4.4.1 [gcc-4_4-branch revision 150839] (SUSE Linux) ) Wed Jan 20 01:09:45 UTC 2010 (XEN) Latest ChangeSet: 20809 (XEN) Command line: loglvl=all loglvl_guest=all apic_verbosity=debug dom0_mem=1024M vga=gfx-1280x1024x32 console=vga,com1 com1=57600,8n1 iommu=1 dom0_max_vcpus=4 dom0_vcpus_pin=true sched=credit cpuidle=0 cpufreq=xen clocksource=acpi numa=on xm create -c /etc/xen/vm/test.cfg Using config file "/etc/xen/vm/test.cfg". Error: Domain ''test'' does not exist. tail -f /var/log/xen/* ------------------------------------ ==> /var/log/xen/xend.log <=[2010-01-26 10:09:07 4764] WARNING (XendConfig:876) Unconverted key: Description [2010-01-26 10:09:07 4764] DEBUG (XendDomainInfo:102) XendDomainInfo.create([''vm'', [''name'', ''test''], [''memory'', 4096], [''maxmem'', 4096], [''on_reboot'', ''restart''], [''on_crash'', ''destroy''], [''on_xend_start'', ''ignore''], [''on_xend_stop'', ''ignore''], [''vcpus'', 4], [''cpu_cap'', 100], [''cpu_weight'', 256], [''backend'', [''netif'']], [''oos'', 1], [''bootloader'', ''/usr/lib/xen/boot/domUloader.py''], [''bootloader_args'', ''--entry=xvda1:vmlinuz-xen,/boot/initrd-xen''], [''image'', [''linux'', [''root'', ''/dev/xvdc1''], [''videoram'', 4], [''args'', ''textmode=1 xencons=xvc0''], [''tsc_mode'', 0], [''nomigrate'', 0]]], [''s3_integrity'', 1], [''device'', [''vbd'', [''uname'', ''phy:/dev/VG0/boot''], [''dev'', ''xvda''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', ''phy:/dev/VG0/swap''], [''dev'', ''xvdb''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', ''phy:/dev/VG0/root''], [''dev'', ''xvdc''], [''mode'', ''w'']]], [''device'', [''vif'']], [''device'', [''vkbd'']], [''device'', [''vfb'', [''vnc'', ''1''], [''vnclisten'', ''127.0.0.1''], [''vncdisplay'', ''102''], [''display'', ''localhost:11.0''], [''xauthority'', ''/root/.Xauthority'']]]]) [2010-01-26 10:09:07 4764] DEBUG (XendDomainInfo:2513) XendDomainInfo.constructDomain [2010-01-26 10:09:07 4764] DEBUG (balloon:239) Balloon: 6946324 KiB free; need 4096; done. [2010-01-26 10:09:07 4764] DEBUG (XendDomain:465) Adding Domain: 1 [2010-01-26 10:09:07 4764] DEBUG (XendDomainInfo:2821) XendDomainInfo.initDomain: 1 256 [2010-01-26 10:09:07 5315] DEBUG (XendBootloader:130) Launching bootloader as [''/usr/lib/xen/boot/domUloader.py'', ''--args=root=/dev/xvdc1 textmode=1 xencons=xvc0'', ''--output=/var/run/xend/boot/xenbl.24317'', ''--entry=xvda1:vmlinuz-xen,/boot/initrd-xen'', ''/dev/VG0/boot'']. ==> /var/log/xen/domain-builder-ng.log <=### ----- xc domain builder logfile opened ----- xc_dom_allocate: cmdline="root=/dev/xvdc1 textmode=1 xencons=xvc0", features="" xc_dom_kernel_file: filename="/var/lib/xen/tmp/kernel.GF9ro1" xc_dom_malloc_filemap : 3297 kB xc_dom_malloc : 12278 kB xc_dom_do_gunzip: unzip ok, 0x338703 -> 0xbfd9c8 xc_dom_ramdisk_file: filename="/var/lib/xen/tmp/ramdisk.usyS9V" xc_dom_malloc_filemap : 8150 kB xc_dom_boot_xen_init: ver 4.0, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xc_dom_parse_image: called xc_dom_find_loader: trying ELF-generic loader ... OK elf_parse_binary: phdr: paddr=0x2000 memsz=0x733000 elf_parse_binary: phdr: paddr=0x735000 memsz=0x51dd0 elf_parse_binary: phdr: paddr=0x787000 memsz=0x8c8 elf_parse_binary: phdr: paddr=0x788000 memsz=0xaca0 elf_parse_binary: phdr: paddr=0x792ca0 memsz=0x14e360 elf_parse_binary: memory: 0x2000 -> 0x8e1000 elf_xen_parse_note: GUEST_OS = "linux" elf_xen_parse_note: GUEST_VERSION = "2.6" elf_xen_parse_note: XEN_VERSION = "xen-3.0" elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000 elf_xen_parse_note: PADDR_OFFSET = 0x0 elf_xen_parse_note: ENTRY = 0xffffffff80002000 elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff80008000 elf_xen_parse_note: unknown xen elf note (0xd) elf_xen_parse_note: INIT_P2M = 0xffffea0000000000 elf_xen_parse_note: FEATURES "writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel" elf_xen_parse_note: LOADER = "generic" elf_xen_parse_note: SUSPEND_CANCEL = 0x1 elf_xen_addr_calc_check: addresses: virt_base = 0xffffffff80000000 elf_paddr_offset = 0x0 virt_offset = 0xffffffff80000000 virt_kstart = 0xffffffff80002000 virt_kend = 0xffffffff808e1000 virt_entry = 0xffffffff80002000 p2m_base = 0xffffea0000000000 xc_dom_parse_elf_kernel: xen-3.0-x86_64: 0xffffffff80002000 -> 0xffffffff808e1000 xc_dom_mem_init: mem 4096 MB, pages 0x100000 pages, 4k each xc_dom_mem_init: 0x100000 pages xc_dom_boot_mem_init: called x86_compat: guest xen-3.0-x86_64, address size 64 xc_dom_malloc : 8192 kB xc_dom_build_image: called xc_dom_alloc_segment: kernel : 0xffffffff80002000 -> 0xffffffff808e1000 (pfn 0x2 + 0x8df pages) xc_dom_pfn_to_ptr: domU mapping: pfn 0x2+0x8df at 0x7fc2e4726000 elf_load_binary: phdr 0 at 0x0x7fc2e4726000 -> 0x0x7fc2e4e59000 elf_load_binary: phdr 1 at 0x0x7fc2e4e59000 -> 0x0x7fc2e4eaadd0 elf_load_binary: phdr 2 at 0x0x7fc2e4eab000 -> 0x0x7fc2e4eab8c8 elf_load_binary: phdr 3 at 0x0x7fc2e4eac000 -> 0x0x7fc2e4eb6ca0 elf_load_binary: phdr 4 at 0x0x7fc2e4eb6ca0 -> 0x0x7fc2e4f11000 xc_dom_alloc_segment: ramdisk : 0xffffffff808e1000 -> 0xffffffff81cde000 (pfn 0x8e1 + 0x13fd pages) xc_dom_malloc : 119 kB xc_dom_pfn_to_ptr: domU mapping: pfn 0x8e1+0x13fd at 0x7fc2e3329000 xc_dom_do_gunzip: unzip ok, 0x7f5ba8 -> 0x13fcc10 xc_dom_alloc_segment: phys2mach : 0xffffffff81cde000 -> 0xffffffff824de000 (pfn 0x1cde + 0x800 pages) xc_dom_pfn_to_ptr: domU mapping: pfn 0x1cde+0x800 at 0x7fc2e2b29000 xc_dom_alloc_page : start info : 0xffffffff824de000 (pfn 0x24de) xc_dom_alloc_page : xenstore : 0xffffffff824df000 (pfn 0x24df) xc_dom_alloc_page : console : 0xffffffff824e0000 (pfn 0x24e0) nr_page_tables: 0x0000ffffffffffff/48: 0xffff000000000000 -> 0xffffffffffffffff, 1 table(s) nr_page_tables: 0x0000007fffffffff/39: 0xffffff8000000000 -> 0xffffffffffffffff, 1 table(s) nr_page_tables: 0x000000003fffffff/30: 0xffffffff80000000 -> 0xffffffffbfffffff, 1 table(s) nr_page_tables: 0x00000000001fffff/21: 0xffffffff80000000 -> 0xffffffff827fffff, 20 table(s) xc_dom_alloc_segment: page tables : 0xffffffff824e1000 -> 0xffffffff824f8000 (pfn 0x24e1 + 0x17 pages) xc_dom_pfn_to_ptr: domU mapping: pfn 0x24e1+0x17 at 0x7fc2f5336000 xc_dom_alloc_page : boot stack : 0xffffffff824f8000 (pfn 0x24f8) xc_dom_build_image : virt_alloc_end : 0xffffffff824f9000 xc_dom_build_image : virt_pgtab_end : 0xffffffff82800000 xc_dom_boot_image: called arch_setup_bootearly: doing nothing xc_dom_compat_check: supported guest type: xen-3.0-x86_64 <= matches xc_dom_compat_check: supported guest type: xen-3.0-x86_32p xc_dom_compat_check: supported guest type: hvm-3.0-x86_32 xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p xc_dom_compat_check: supported guest type: hvm-3.0-x86_64 xc_dom_update_guest_p2m: dst 64bit, pages 0x100000 clear_page: pfn 0x24e0, mfn 0x215140 clear_page: pfn 0x24df, mfn 0x215141 xc_dom_pfn_to_ptr: domU mapping: pfn 0x24de+0x1 at 0x7fc2f5335000 start_info_x86_64: called setup_hypercall_page: vaddr=0xffffffff80008000 pfn=0x8 domain builder memory footprint allocated malloc : 20694 kB anon mmap : 0 bytes mapped file mmap : 11448 kB domU mmap : 36 MB arch_setup_bootlate: shared_info: pfn 0x0, mfn 0xcfa87 shared_info_x86_64: called vcpu_x86_64: called vcpu_x86_64: cr3: pfn 0x24e1 mfn 0x21513f launch_vm: called, ctxt=0x7fc2e67f73e0 xc_dom_release: called ==> /var/log/xen/xend.log <=[2010-01-26 10:09:08 4764] DEBUG (image:339) No VNC passwd configured for vfb access [2010-01-26 10:09:08 4764] DEBUG (XendDomainInfo:2848) _initDomain:shadow_memory=0x0, memory_static_max=0x100000000, memory_static_min=0x0. [2010-01-26 10:09:08 4764] INFO (image:182) buildDomain os=linux dom=1 vcpus=4 [2010-01-26 10:09:08 4764] DEBUG (image:734) domid = 1 [2010-01-26 10:09:08 4764] DEBUG (image:735) memsize = 4096 [2010-01-26 10:09:08 4764] DEBUG (image:736) image /var/lib/xen/tmp/kernel.GF9ro1 [2010-01-26 10:09:08 4764] DEBUG (image:737) store_evtchn = 1 [2010-01-26 10:09:08 4764] DEBUG (image:738) console_evtchn = 2 [2010-01-26 10:09:08 4764] DEBUG (image:739) cmdline root=/dev/xvdc1 textmode=1 xencons=xvc0 [2010-01-26 10:09:08 4764] DEBUG (image:740) ramdisk /var/lib/xen/tmp/ramdisk.usyS9V [2010-01-26 10:09:08 4764] DEBUG (image:741) vcpus = 4 [2010-01-26 10:09:08 4764] DEBUG (image:742) features [2010-01-26 10:09:08 4764] DEBUG (image:743) flags = 0 [2010-01-26 10:09:08 4764] DEBUG (image:744) superpages = 0 [2010-01-26 10:09:08 4764] INFO (XendDomainInfo:2372) createDevice: vkbd : {''uuid'': ''3e629ba2-c496-f073-e797-f6317e3bdcd4''} [2010-01-26 10:09:08 4764] DEBUG (DevController:95) DevController: writing {''protocol'': ''x86_64-abi'', ''state'': ''1'', ''backend-id'': ''0'', ''backend'': ''/local/domain/0/backend/vkbd/1/0''} to /local/domain/1/device/vkbd/0. [2010-01-26 10:09:08 4764] DEBUG (DevController:97) DevController: writing {''frontend-id'': ''1'', ''domain'': ''test'', ''frontend'': ''/local/domain/1/device/vkbd/0'', ''state'': ''1'', ''online'': ''1''} to /local/domain/0/backend/vkbd/1/0. [2010-01-26 10:09:08 4764] INFO (XendDomainInfo:2372) createDevice: vfb : {''other_config'': {''display'': ''localhost:11.0'', ''vnclisten'': ''127.0.0.1'', ''vnc'': ''1'', ''xauthority'': ''/root/.Xauthority'', ''vncdisplay'': ''102''}, ''vnc'': ''1'', ''xauthority'': ''/root/.Xauthority'', ''vnclisten'': ''127.0.0.1'', ''vncdisplay'': ''102'', ''keymap'': ''en-us'', ''display'': ''localhost:11.0'', ''uuid'': ''219090ab-f3d2-f998-46f3-efbfa6221f7f''} [2010-01-26 10:09:08 4764] DEBUG (DevController:95) DevController: writing {''protocol'': ''x86_64-abi'', ''state'': ''1'', ''backend-id'': ''0'', ''backend'': ''/local/domain/0/backend/vfb/1/0''} to /local/domain/1/device/vfb/0. [2010-01-26 10:09:08 4764] DEBUG (DevController:97) DevController: writing {''domain'': ''test'', ''frontend'': ''/local/domain/1/device/vfb/0'', ''xauthority'': ''/root/.Xauthority'', ''frontend-id'': ''1'', ''vnclisten'': ''127.0.0.1'', ''vncdisplay'': ''102'', ''state'': ''1'', ''keymap'': ''en-us'', ''online'': ''1'', ''vnc'': ''1'', ''display'': ''localhost:11.0'', ''uuid'': ''219090ab-f3d2-f998-46f3-efbfa6221f7f''} to /local/domain/0/backend/vfb/1/0. [2010-01-26 10:09:08 4764] INFO (XendDomainInfo:2372) createDevice: vbd : {''uuid'': ''41bdfd95-b845-e04e-0643-7803fcb5645b'', ''bootable'': 1, ''driver'': ''paravirtualised'', ''dev'': ''xvda'', ''uname'': ''phy:/dev/VG0/boot'', ''mode'': ''w''} [2010-01-26 10:09:08 4764] DEBUG (DevController:95) DevController: writing {''virtual-device'': ''51712'', ''device-type'': ''disk'', ''protocol'': ''x86_64-abi'', ''backend-id'': ''0'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vbd/1/51712''} to /local/domain/1/device/vbd/51712. [2010-01-26 10:09:08 4764] DEBUG (DevController:97) DevController: writing {''domain'': ''test'', ''frontend'': ''/local/domain/1/device/vbd/51712'', ''uuid'': ''41bdfd95-b845-e04e-0643-7803fcb5645b'', ''bootable'': ''1'', ''dev'': ''xvda'', ''state'': ''1'', ''params'': ''/dev/VG0/boot'', ''mode'': ''w'', ''online'': ''1'', ''frontend-id'': ''1'', ''type'': ''phy''} to /local/domain/0/backend/vbd/1/51712. [2010-01-26 10:09:08 4764] INFO (XendDomainInfo:2372) createDevice: vbd : {''uuid'': ''6ee5a108-2bef-fa26-5d3d-a6be7d088ea3'', ''bootable'': 0, ''driver'': ''paravirtualised'', ''dev'': ''xvdb'', ''uname'': ''phy:/dev/VG0/swap'', ''mode'': ''w''} [2010-01-26 10:09:08 4764] DEBUG (DevController:95) DevController: writing {''virtual-device'': ''51728'', ''device-type'': ''disk'', ''protocol'': ''x86_64-abi'', ''backend-id'': ''0'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vbd/1/51728''} to /local/domain/1/device/vbd/51728. [2010-01-26 10:09:08 4764] DEBUG (DevController:97) DevController: writing {''domain'': ''test'', ''frontend'': ''/local/domain/1/device/vbd/51728'', ''uuid'': ''6ee5a108-2bef-fa26-5d3d-a6be7d088ea3'', ''bootable'': ''0'', ''dev'': ''xvdb'', ''state'': ''1'', ''params'': ''/dev/VG0/swap'', ''mode'': ''w'', ''online'': ''1'', ''frontend-id'': ''1'', ''type'': ''phy''} to /local/domain/0/backend/vbd/1/51728. [2010-01-26 10:09:08 4764] INFO (XendDomainInfo:2372) createDevice: vbd : {''uuid'': ''e7a397a0-903b-eed9-e1c4-a9639b2416e7'', ''bootable'': 0, ''driver'': ''paravirtualised'', ''dev'': ''xvdc'', ''uname'': ''phy:/dev/VG0/root'', ''mode'': ''w''} [2010-01-26 10:09:08 4764] DEBUG (DevController:95) DevController: writing {''virtual-device'': ''51744'', ''device-type'': ''disk'', ''protocol'': ''x86_64-abi'', ''backend-id'': ''0'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vbd/1/51744''} to /local/domain/1/device/vbd/51744. [2010-01-26 10:09:08 4764] DEBUG (DevController:97) DevController: writing {''domain'': ''test'', ''frontend'': ''/local/domain/1/device/vbd/51744'', ''uuid'': ''e7a397a0-903b-eed9-e1c4-a9639b2416e7'', ''bootable'': ''0'', ''dev'': ''xvdc'', ''state'': ''1'', ''params'': ''/dev/VG0/root'', ''mode'': ''w'', ''online'': ''1'', ''frontend-id'': ''1'', ''type'': ''phy''} to /local/domain/0/backend/vbd/1/51744. [2010-01-26 10:09:08 4764] INFO (XendDomainInfo:2372) createDevice: vif : {''mac'': ''00:16:3e:4f:32:d7'', ''uuid'': ''e54d312a-774d-0d65-e250-0b3c1f42e6bb''} [2010-01-26 10:09:08 4764] DEBUG (DevController:95) DevController: writing {''mac'': ''00:16:3e:4f:32:d7'', ''handle'': ''0'', ''protocol'': ''x86_64-abi'', ''backend-id'': ''0'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vif/1/0''} to /local/domain/1/device/vif/0. [2010-01-26 10:09:08 4764] DEBUG (DevController:97) DevController: writing {''domain'': ''test'', ''frontend'': ''/local/domain/1/device/vif/0'', ''uuid'': ''e54d312a-774d-0d65-e250-0b3c1f42e6bb'', ''script'': ''/etc/xen/scripts/vif-bridge'', ''mac'': ''00:16:3e:4f:32:d7'', ''frontend-id'': ''1'', ''state'': ''1'', ''online'': ''1'', ''handle'': ''0''} to /local/domain/0/backend/vif/1/0. [2010-01-26 10:09:09 4764] INFO (image:420) spawning device models: /usr/lib/xen/bin/qemu-dm [''/usr/lib/xen/bin/qemu-dm'', ''-d'', ''1'', ''-serial'', ''pty'', ''-domain-name'', ''test'', ''-videoram'', ''4'', ''-k'', ''en-us'', ''-vnc'', ''127.0.0.1:102'', ''-vncunused'', ''-monitor'', ''null'', ''-M'', ''xenpv''] [2010-01-26 10:09:09 4764] INFO (image:469) device model pid: 5379 [2010-01-26 10:09:09 4764] INFO (image:603) waiting for sentinel_fifo [2010-01-26 10:09:09 4764] DEBUG (XendDomainInfo:3420) Storing VM details: {''on_xend_stop'': ''ignore'', ''shadow_memory'': ''0'', ''uuid'': ''730a466e-80a3-4aa3-79f8-7eb6a6f3abc8'', ''on_reboot'': ''restart'', ''start_time'': ''1264529349.19'', ''on_poweroff'': ''destroy'', ''bootloader_args'': ''--entry=xvda1:vmlinuz-xen,/boot/initrd-xen'', ''on_xend_start'': ''ignore'', ''on_crash'': ''destroy'', ''xend/restart_count'': ''0'', ''vcpus'': ''4'', ''vcpu_avail'': ''15'', ''bootloader'': ''/usr/lib/xen/boot/domUloader.py'', ''image'': "(linux (kernel ) (args ''root=/dev/xvdc1 textmode=1 xencons=xvc0'') (superpages 0) (videoram 4) (pci ()) (nomigrate 0) (tsc_mode 0) (device_model /usr/lib/xen/bin/qemu-dm) (keymap en-us) (notes (FEATURES ''writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel'') (VIRT_BASE 18446744071562067968) (GUEST_VERSION 2.6) (PADDR_OFFSET 0) (GUEST_OS linux) (HYPERCALL_PAGE 18446744071562100736) (LOADER generic) (INIT_P2M 18446719884453740544) (SUSPEND_CANCEL 1) (ENTRY 18446744071562076160) (XEN_VERSION xen-3.0)))", ''name'': ''test''} [2010-01-26 10:09:09 4764] DEBUG (XendDomainInfo:1805) Storing domain details: {''console/ring-ref'': ''2183488'', ''image/entry'': ''18446744071562076160'', ''console/port'': ''2'', ''cpu/3/availability'': ''online'', ''store/ring-ref'': ''2183489'', ''image/loader'': ''generic'', ''vm'': ''/vm/730a466e-80a3-4aa3-79f8-7eb6a6f3abc8'', ''control/platform-feature-multiprocessor-suspend'': ''1'', ''description'': '''', ''cpu/2/availability'': ''online'', ''cpu/1/availability'': ''online'', ''image/features/writable-descriptor-tables'': ''1'', ''image/virt-base'': ''18446744071562067968'', ''memory/target'': ''4194304'', ''image/guest-version'': ''2.6'', ''image/features/supervisor-mode-kernel'': ''1'', ''image/guest-os'': ''linux'', ''console/limit'': ''1048576'', ''image/paddr-offset'': ''0'', ''image/hypercall-page'': ''18446744071562100736'', ''image/suspend-cancel'': ''1'', ''cpu/0/availability'': ''online'', ''image/features/pae-pgdir-above-4gb'': ''1'', ''image/features/writable-page-tables'': ''1'', ''console/type'': ''ioemu'', ''image/features/auto-translated-physmap'': ''1'', ''name'': ''test'', ''domid'': ''1'', ''image/xen-version'': ''xen-3.0'', ''image/init-p2m'': ''18446719884453740544'', ''store/port'': ''1''} [2010-01-26 10:09:10 4764] WARNING (image:561) domain test: device model failure: pid 5379: malfunctioning (closed sentinel), killed; see /var/log/xen/qemu-dm-test.log [2010-01-26 10:09:10 4764] DEBUG (DevController:95) DevController: writing {''protocol'': ''x86_64-abi'', ''state'': ''1'', ''backend-id'': ''0'', ''backend'': ''/local/domain/0/backend/console/1/0''} to /local/domain/1/device/console/0. [2010-01-26 10:09:10 4764] DEBUG (DevController:97) DevController: writing {''domain'': ''test'', ''frontend'': ''/local/domain/1/device/console/0'', ''uuid'': ''164d06b3-1ce7-ccfc-4258-72184fd3757a'', ''frontend-id'': ''1'', ''state'': ''1'', ''location'': ''2'', ''online'': ''1'', ''protocol'': ''vt100''} to /local/domain/0/backend/console/1/0. [2010-01-26 10:09:10 4764] WARNING (XendDomainInfo:2072) Domain has crashed: name=test id=1. ==> /var/log/xen/xend-debug.log <=ERROR Internal error: p2m_size < nr_pages -1 (0 < fffff ==> /var/log/xen/xend.log <=[2010-01-26 10:09:10 4764] DEBUG (DevController:139) Waiting for devices tap2. [2010-01-26 10:09:10 4764] DEBUG (DevController:139) Waiting for devices vif. [2010-01-26 10:09:10 4764] DEBUG (DevController:144) Waiting for 0. [2010-01-26 10:09:11 4764] ERROR (XendDomainInfo:2327) core dump failed: id = 1 name = test: (1, ''Internal error'', ''p2m_size < nr_pages -1 (0 < fffff'') [2010-01-26 10:09:11 4764] DEBUG (XendDomainInfo:3061) XendDomainInfo.destroy: domid=1 [2010-01-26 10:09:12 4764] DEBUG (XendDomainInfo:2416) Destroying device model [2010-01-26 10:09:12 4764] DEBUG (DevController:653) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status. [2010-01-26 10:09:12 4764] DEBUG (DevController:667) hotplugStatusCallback 1. [2010-01-26 10:09:12 4764] DEBUG (DevController:139) Waiting for devices vkbd. [2010-01-26 10:09:13 4764] DEBUG (DevController:139) Waiting for devices ioports. [2010-01-26 10:09:13 4764] DEBUG (DevController:139) Waiting for devices tap. [2010-01-26 10:09:13 4764] DEBUG (DevController:139) Waiting for devices vif2. [2010-01-26 10:09:13 4764] DEBUG (DevController:139) Waiting for devices console. [2010-01-26 10:09:13 4764] DEBUG (DevController:144) Waiting for 0. [2010-01-26 10:09:13 4764] DEBUG (DevController:139) Waiting for devices vscsi. [2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:2423) Releasing devices [2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:2429) Removing vif/0 [2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0 [2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:2429) Removing vkbd/0 [2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vkbd, device = vkbd/0 [2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:2429) Removing console/0 [2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = console, device console/0 [2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:2429) Removing vbd/51712 [2010-01-26 10:09:13 4764] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51712 [2010-01-26 10:09:14 4764] DEBUG (XendDomainInfo:2429) Removing vbd/51728 [2010-01-26 10:09:14 4764] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51728 [2010-01-26 10:09:14 4764] DEBUG (XendDomainInfo:2429) Removing vbd/51744 [2010-01-26 10:09:14 4764] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51744 [2010-01-26 10:09:14 4764] DEBUG (DevController:139) Waiting for devices vbd. [2010-01-26 10:09:14 4764] DEBUG (DevController:144) Waiting for 51712. [2010-01-26 10:09:14 4764] DEBUG (XendDomainInfo:2429) Removing vfb/0 [2010-01-26 10:09:14 4764] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0 [2010-01-26 10:09:15 4764] DEBUG (DevController:653) hotplugStatusCallback /local/domain/0/backend/vbd/1/51712/hotplug-status. [2010-01-26 10:09:15 4764] DEBUG (DevController:667) hotplugStatusCallback 1. [2010-01-26 10:09:15 4764] DEBUG (DevController:144) Waiting for 51728. ------------------------------------ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
0bo0
2010-Jan-26 18:55 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
hi, On Tue, Jan 26, 2010 at 10:36 AM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> You are missing this one: > > [2010-01-26 10:09:10 4764] WARNING (image:561) domain test: device > model failure: pid 5379: malfunctioning (closed sentinel), killed; see > /var/log/xen/qemu-dm-test.log > > Can you see what it has?yep, it was created after I launched the tail ... checking, not much there: cat qemu-dm-test.log domid: 1 Warning: vlan 0 is not connected to host network Using xvda for guest''s hda _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Jan-26 21:10 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On 26/01/2010 18:36, "0bo0" <0.bugs.only.0@gmail.com> wrote:>> If qemu-dm died without logging anything, it probably crashed again. If it >> produced a core file you could load that in gdb and get a symbolic >> backtrace. > > trying to turn on core dumps in xend-config, unfortunately, Xen seems > to fail @ core dump ... in addition to failing to launch a DomU which > works fine under Xen3,Ah, sorry, I meant a core file produced when qemu-dm crashes (that is, assuming that qemu-dm does crash, which appears likely). All the log files tell us is that xend thinks that qemu-dm process simply disappeared without warning. The question is obviously: why did it disappear, and only for you in your setup. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
0bo0
2010-Jan-26 21:25 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On Tue, Jan 26, 2010 at 1:10 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:> On 26/01/2010 18:36, "0bo0" <0.bugs.only.0@gmail.com> wrote: > >>> If qemu-dm died without logging anything, it probably crashed again. If it >>> produced a core file you could load that in gdb and get a symbolic >>> backtrace. >> >> trying to turn on core dumps in xend-config, unfortunately, Xen seems >> to fail @ core dump ... in addition to failing to launch a DomU which >> works fine under Xen3, > > Ah, sorry, I meant a core file produced when qemu-dm crashes (that is, > assuming that qemu-dm does crash, which appears likely). All the log files > tell us is that xend thinks that qemu-dm process simply disappeared without > warning. The question is obviously: why did it disappear, and only for you > in your setup.np. i''m happy to provide the info. so since i''m not launching qemu-dm directly, how -- specifically -- do I grab that core dump for you? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Jan-26 21:38 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On 26/01/2010 21:25, "0bo0" <0.bugs.only.0@gmail.com> wrote:>> Ah, sorry, I meant a core file produced when qemu-dm crashes (that is, >> assuming that qemu-dm does crash, which appears likely). All the log files >> tell us is that xend thinks that qemu-dm process simply disappeared without >> warning. The question is obviously: why did it disappear, and only for you >> in your setup. > > np. i''m happy to provide the info. > > so since i''m not launching qemu-dm directly, how -- specifically -- > do I grab that core dump for you?As root: ulimit -c unlimited echo ''/var/core/core.%e.%u.%p'' > /proc/sys/kernel/core_pattern Now try starting your guest. It fails, and hopefully you have a core file in /var/core/. You can load it into gdb as ''gdb /usr/lib/xen/qemu-dm /var/core/name-of-core-file'' and then execute bt command at the gdb prompt. That hopefully tells us where qemu-dm crashed. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
0bo0
2010-Jan-26 22:12 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On Tue, Jan 26, 2010 at 1:38 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:> As root: > echo ''/var/core/core.%e.%u.%p'' > /proc/sys/kernel/core_pattern > > Now try starting your guest. It fails, and hopefully you have a core file in > /var/core/. You can load it into gdb as ''gdb /usr/lib/xen/qemu-dm > /var/core/name-of-core-file'' and then execute bt command at the gdb prompt. > That hopefully tells us where qemu-dm crashed.bash ulimit -c unlimited echo ''/var/core/core.%e.%u.%p'' > /proc/sys/kernel/core_pattern mkdir -p /var/core cat /proc/sys/kernel/core_pattern /var/core/core.%e.%u.%p xm create /etc/xen/vm/test.cfg Using config file "/etc/xen/vm/test.cfg". Error: Domain ''test'' does not exist. ls -al /var/core total 8 drwxr-xr-x 2 root root 4096 2010-01-26 13:48 ./ drwxr-xr-x 18 root root 4096 2010-01-26 13:48 ../ @ syslog, though, i do see .... Jan 26 14:09:17 server kernel: [ 547.023464] kjournald starting. Commit interval 15 seconds Jan 26 14:09:17 server kernel: [ 547.118565] EXT3-fs: mounted filesystem with ordered data mode. Jan 26 14:09:18 server logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/3/51712 Jan 26 14:09:18 server logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/3/51744 Jan 26 14:09:18 server logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/3/51728 Jan 26 14:09:18 server kernel: [ 548.456142] qemu-dm[6791]: segfault at 7fff7ee4169a ip 000000000047b974 sp 00007fff7ee40210 error 4 in qemu-dm[400000+10b000] Jan 26 14:09:18 server logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/3/0 Jan 26 14:09:18 server kernel: [ 548.697863] device vif3.0 entered promiscuous mode Jan 26 14:09:19 server kernel: [ 548.828028] br02: port 1(vif3.0) entering forwarding state Jan 26 14:09:19 server kernel: [ 548.984448] physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. Jan 26 14:09:19 server logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif3.0, bridge br02. Jan 26 14:09:19 server logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/3/0/hotplug-status connected to xenstore. Jan 26 14:09:19 server logger: /etc/xen/scripts/block: Writing backend/vbd/3/51712/physical-device fd:d to xenstore. Jan 26 14:09:19 server kernel: [ 549.267962] (cdrom_add_media_watch() file=/usr/src/packages/BUILD/kernel-xen-2.6.31.8/linux-2.6.31/drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/3/51712 Jan 26 14:09:19 server kernel: [ 549.474577] (cdrom_is_type() file=/usr/src/packages/BUILD/kernel-xen-2.6.31.8/linux-2.6.31/drivers/xen/blkback/cdrom.c, line=95) type:0 Jan 26 14:09:19 server logger: /etc/xen/scripts/block: Writing backend/vbd/3/51712/hotplug-status connected to xenstore. Jan 26 14:09:20 server logger: /etc/xen/scripts/block: Writing backend/vbd/3/51728/physical-device fd:e to xenstore. Jan 26 14:09:20 server kernel: [ 550.443509] (cdrom_add_media_watch() file=/usr/src/packages/BUILD/kernel-xen-2.6.31.8/linux-2.6.31/drivers/xen/blkback/cdrom.c, line=108) nodename:backend/vbd/3/51728 Jan 26 14:09:20 server kernel: [ 550.657671] (cdrom_is_type() file=/usr/src/packages/BUILD/kernel-xen-2.6.31.8/linux-2.6.31/drivers/xen/blkback/cdrom.c, line=95) type:0 Jan 26 14:09:20 server logger: /etc/xen/scripts/block: Writing backend/vbd/3/51728/hotplug-status connected to xenstore. Jan 26 14:09:21 server logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vkbd/3/0 Jan 26 14:09:21 server logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/console/3/0 Jan 26 14:09:22 server kernel: [ 552.224799] br02: port 1(vif3.0) entering disabled state Jan 26 14:09:22 server kernel: [ 552.335801] br02: port 1(vif3.0) entering disabled state Jan 26 14:09:22 server logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vfb/3/0 Jan 26 14:09:22 server logger: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/3/51728 Jan 26 14:09:22 server logger: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/3/51712 Jan 26 14:09:22 server logger: /etc/xen/scripts/vif-bridge: offline XENBUS_PATH=backend/vif/3/0 Jan 26 14:09:22 server logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vbd/3/51728 Jan 26 14:09:22 server logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vbd/3/51712 Jan 26 14:09:22 server logger: /etc/xen/scripts/vif-bridge: brctl delif br02 vif3.0 failed Jan 26 14:09:22 server logger: /etc/xen/scripts/vif-bridge: ifconfig vif3.0 down failed Jan 26 14:09:22 server kernel: [ 552.662465] physdev match: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore. Jan 26 14:09:22 server logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge offline for vif3.0, bridge br02. Jan 26 14:09:22 server logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vif/3/0 Jan 26 14:09:23 server logger: /etc/xen/scripts/block: Writing backend/vbd/3/51744/physical-device fd:f to xenstore. Jan 26 14:09:23 server logger: /etc/xen/scripts/block: Writing backend/vbd/3/51744/hotplug-status connected to xenstore. Jan 26 14:09:23 server logger: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/3/51744 Jan 26 14:09:23 server logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vbd/3/51744 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Jan-26 22:30 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On 26/01/2010 22:12, "0bo0" <0.bugs.only.0@gmail.com> wrote:> On Tue, Jan 26, 2010 at 1:38 PM, Keir Fraser <keir.fraser@eu.citrix.com> > wrote: >> As root: >> echo ''/var/core/core.%e.%u.%p'' > /proc/sys/kernel/core_pattern >> >> Now try starting your guest. It fails, and hopefully you have a core file in >> /var/core/. You can load it into gdb as ''gdb /usr/lib/xen/qemu-dm >> /var/core/name-of-core-file'' and then execute bt command at the gdb prompt. >> That hopefully tells us where qemu-dm crashed. > > bash > ulimit -c unlimitedActually that probably only enable core dumps for the shell you are running. Qemu-dm will inherit its ulimits from xend. So you need to enable core dumps by running ''ulimit -c unlimited'' before you start xend, or put it in the startup script that starts xend. You could also script the echo to /proc/sys/kernel/core_pattern, but it''s definitely safe to do that later from a shall, as it''s a global value, rather than per-process. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
0bo0
2010-Jan-26 22:47 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On Tue, Jan 26, 2010 at 2:30 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:>>> Now try starting your guest. It fails, and hopefully you have a core file in >>> /var/core/. You can load it into gdb as ''gdb /usr/lib/xen/qemu-dm >>> /var/core/name-of-core-file'' and then execute bt command at the gdb prompt. >>> That hopefully tells us where qemu-dm crashed. >> >> bash >> ulimit -c unlimited > > Actually that probably only enable core dumps for the shell you are running. > Qemu-dm will inherit its ulimits from xend. So you need to enable core dumps > by running ''ulimit -c unlimited'' before you start xend, or put it in the > startup script that starts xend. You could also script the echo to > /proc/sys/kernel/core_pattern, but it''s definitely safe to do that later > from a shall, as it''s a global value, rather than per-process.bash ulimit -c unlimited echo ''/var/core/core.%e.%u.%p'' > /proc/sys/kernel/core_pattern mkdir -p /var/core cat /proc/sys/kernel/core_pattern /var/core/core.%e.%u.%p service xend restart xm create /etc/xen/vm/test.cfg ls -al /var/core/ | grep core -rw------- 1 root root 4542464 2010-01-26 14:36 core.qemu-dm.0.7772 gdb /usr/lib64/xen/bin/qemu-dm /var/core/core.qemu-dm.0.7772 ... /usr/lib64/xen/bin/qemu-dm: Too many levels of symbolic links. Missing separate debuginfo for the main executable file Try: zypper install -C "debuginfo(build-id)=9326caa0db4accd719f10b02a9a41a40c10b662a" Core was generated by `/usr/lib/xen/bin/qemu-dm -d 4 -serial pty -domain-name test -videoram 4 -k en-u''. Program terminated with signal 11, Segmentation fault. #0 0x000000000047b974 in ?? () (gdb) quit zypper install -C "debuginfo(build-id)=9326caa0db4accd719f10b02a9a41a40c10b662a" Loading repository data... Reading installed packages... ''debuginfo(build-id)=9326caa0db4accd719f10b02a9a41a40c10b662a'' not found. Resolving package dependencies... so where/how do I get the debug symbols? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
0bo0
2010-Jan-27 00:25 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
apparently, the install debuginfo by buildid has -- challenges. installing as much as I can find by hand, xm create -c /etc/xen/vm/test.cfg gdb /usr/lib/xen/bin/qemu-dm /var/core/core.qemu-dm.0.17726 Using config file "/etc/xen/vm/test.cfg". xenconsole: Could not read tty from store: No such file or directory GNU gdb (GDB) SUSE (6.8.91.20090930-2.4) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/lib/xen/bin/qemu-dm...(no debugging symbols found)...done. warning: Can''t read pathname for load map: Input/output error. Missing separate debuginfo for /lib64/libz.so.1 Try: zypper install -C "debuginfo(build-id)=4a569edbadd351f13aab9ec81e31a7f31d76a80f" Missing separate debuginfo for /usr/lib64/libxenctrl.so.4.0 Try: zypper install -C "debuginfo(build-id)=219d2e00aba1b2bfd538cb1ad88d3eddd6db682b" Missing separate debuginfo for /usr/lib64/libxenstore.so.3.0 Try: zypper install -C "debuginfo(build-id)=fcf7d0904b10edcaf468e44ba4e9964478083f29" Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/libpthread-2.10.1.so...Reading symbols from /usr/lib/debug/lib64/libpthread-2.10.1.so.debug...done. done. Loaded symbols for /lib64/libpthread-2.10.1.so Reading symbols from /lib64/librt-2.10.1.so...Reading symbols from /usr/lib/debug/lib64/librt-2.10.1.so.debug...done. done. Loaded symbols for /lib64/librt-2.10.1.so Reading symbols from /lib64/libutil-2.10.1.so...Reading symbols from /usr/lib/debug/lib64/libutil-2.10.1.so.debug...done. done. Loaded symbols for /lib64/libutil-2.10.1.so Reading symbols from /usr/lib64/libxenctrl.so.4.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libxenctrl.so.4.0 Reading symbols from /usr/lib64/libxenstore.so.3.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libxenstore.so.3.0 Reading symbols from /lib64/libpci.so.3.1.3...Reading symbols from /usr/lib/debug/lib64/libpci.so.3.1.3.debug...done. done. Loaded symbols for /lib64/libpci.so.3.1.3 Reading symbols from /usr/lib64/libSDL-1.2.so.0.11.2...Reading symbols from /usr/lib/debug/usr/lib64/libSDL-1.2.so.0.11.2.debug...done. done. Loaded symbols for /usr/lib64/libSDL-1.2.so.0.11.2 Reading symbols from /usr/lib64/libX11.so.6.3.0...Reading symbols from /usr/lib/debug/usr/lib64/libX11.so.6.3.0.debug...done. done. Loaded symbols for /usr/lib64/libX11.so.6.3.0 Reading symbols from /usr/lib64/libGL.so.1.2...Reading symbols from /usr/lib/debug/usr/lib64/libGL.so.1.2.debug...done. done. Loaded symbols for /usr/lib64/libGL.so.1.2 Reading symbols from /lib64/libc-2.10.1.so...Reading symbols from /usr/lib/debug/lib64/libc-2.10.1.so.debug...done. done. Loaded symbols for /lib64/libc-2.10.1.so Reading symbols from /lib64/ld-2.10.1.so...Reading symbols from /usr/lib/debug/lib64/ld-2.10.1.so.debug...done. done. Loaded symbols for /lib64/ld-2.10.1.so Reading symbols from /lib64/libresolv-2.10.1.so...Reading symbols from /usr/lib/debug/lib64/libresolv-2.10.1.so.debug...done. done. Loaded symbols for /lib64/libresolv-2.10.1.so Reading symbols from /lib64/libm-2.10.1.so...Reading symbols from /usr/lib/debug/lib64/libm-2.10.1.so.debug...done. done. Loaded symbols for /lib64/libm-2.10.1.so Reading symbols from /lib64/libdl-2.10.1.so...Reading symbols from /usr/lib/debug/lib64/libdl-2.10.1.so.debug...done. done. Loaded symbols for /lib64/libdl-2.10.1.so Reading symbols from /usr/lib64/libXext.so.6.4.0...Reading symbols from /usr/lib/debug/usr/lib64/libXext.so.6.4.0.debug...done. done. Loaded symbols for /usr/lib64/libXext.so.6.4.0 Reading symbols from /usr/lib64/libXrandr.so.2.2.0...Reading symbols from /usr/lib/debug/usr/lib64/libXrandr.so.2.2.0.debug...done. done. Loaded symbols for /usr/lib64/libXrandr.so.2.2.0 Reading symbols from /usr/lib64/libxcb.so.1.1.0...Reading symbols from /usr/lib/debug/usr/lib64/libxcb.so.1.1.0.debug...done. done. Loaded symbols for /usr/lib64/libxcb.so.1.1.0 Reading symbols from /usr/lib64/libXxf86vm.so.1.0.0...Reading symbols from /usr/lib/debug/usr/lib64/libXxf86vm.so.1.0.0.debug...done. done. Loaded symbols for /usr/lib64/libXxf86vm.so.1.0.0 Reading symbols from /usr/lib64/libXdamage.so.1.1.0...Reading symbols from /usr/lib/debug/usr/lib64/libXdamage.so.1.1.0.debug...done. done. Loaded symbols for /usr/lib64/libXdamage.so.1.1.0 Reading symbols from /usr/lib64/libXfixes.so.3.1.0...Reading symbols from /usr/lib/debug/usr/lib64/libXfixes.so.3.1.0.debug...done. done. Loaded symbols for /usr/lib64/libXfixes.so.3.1.0 Reading symbols from /usr/lib64/libdrm.so.2.4.0...Reading symbols from /usr/lib/debug/usr/lib64/libdrm.so.2.4.0.debug...done. done. Loaded symbols for /usr/lib64/libdrm.so.2.4.0 Reading symbols from /usr/lib64/libXrender.so.1.3.0...Reading symbols from /usr/lib/debug/usr/lib64/libXrender.so.1.3.0.debug...done. done. Loaded symbols for /usr/lib64/libXrender.so.1.3.0 Reading symbols from /usr/lib64/libXau.so.6.0.0...Reading symbols from /usr/lib/debug/usr/lib64/libXau.so.6.0.0.debug...done. done. Loaded symbols for /usr/lib64/libXau.so.6.0.0 Core was generated by `/usr/lib/xen/bin/qemu-dm -d 11 -serial pty -domain-name test -videoram 4 -k en-''. Program terminated with signal 11, Segmentation fault. #0 0x000000000047b974 in bdrv_get_type_hint () (gdb) Error: Domain ''test'' does not exist.\ (gbd) bt #0 0x000000000047b974 in bdrv_get_type_hint () #1 0x0000000000464082 in xenstore_parse_domain_config () #2 0x000000000040c099 in main () (gbd) quit _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
0bo0
2010-Jan-27 00:59 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On Tue, Jan 26, 2010 at 4:25 PM, 0bo0 <0.bugs.only.0@gmail.com> wrote:> #1 0x0000000000464082 in xenstore_parse_domain_config ()making note of the ''domain_config'' ... if in test.cfg i change, - vfb = [ ''type=vnc, vncdisplay=102, vnclisten=127.0.0.1'' ] + #vfb = [ ''type=vnc, vncdisplay=102, vnclisten=127.0.0.1'' ] then the DomU launches OK, xm list Name ID Mem VCPUs State Time(s) Domain-0 0 1022 1 r----- 119.5 test 8 4096 4 -b---- 17.1 @ DomU ... uname -a Linux test 2.6.31.8-0.1-xen #1 SMP 2009-12-15 23:55:40 +0100 x86_64 x86_64 x86_64 GNU/Linux so, is the parser broken, or has something changed in allowed vfb specification? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Jan-27 08:27 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On 27/01/2010 00:25, "0bo0" <0.bugs.only.0@gmail.com> wrote:> (gbd) bt > #0 0x000000000047b974 in bdrv_get_type_hint () > #1 0x0000000000464082 in xenstore_parse_domain_config () > #2 0x000000000040c099 in main ()This is a weird backtrace since bdrv_get_type_hint() is not called from xenstore_parse_domain_config(). Further, I wouldn''t expect bdrv_get_type_hint() (a function specific to block devices) to be called for parsing of a vfb configuration, which is apparently the domain-config line causing the problems. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Jan-27 08:27 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On 27/01/2010 00:59, "0bo0" <0.bugs.only.0@gmail.com> wrote:> On Tue, Jan 26, 2010 at 4:25 PM, 0bo0 <0.bugs.only.0@gmail.com> wrote: >> #1 0x0000000000464082 in xenstore_parse_domain_config () > > making note of the ''domain_config'' ... > > if in test.cfg i change, > > - vfb = [ ''type=vnc, vncdisplay=102, vnclisten=127.0.0.1'' ] > + #vfb = [ ''type=vnc, vncdisplay=102, vnclisten=127.0.0.1'' ]Maybe I can repro the crash. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Jan-27 09:10 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On 27/01/2010 08:27, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:>> On Tue, Jan 26, 2010 at 4:25 PM, 0bo0 <0.bugs.only.0@gmail.com> wrote: >>> #1 0x0000000000464082 in xenstore_parse_domain_config () >> >> making note of the ''domain_config'' ... >> >> if in test.cfg i change, >> >> - vfb = [ ''type=vnc, vncdisplay=102, vnclisten=127.0.0.1'' ] >> + #vfb = [ ''type=vnc, vncdisplay=102, vnclisten=127.0.0.1'' ] > > Maybe I can repro the crash.I had a go but couldn''t reproduce the qemu crash. I did note however that your PV guest only gets a qemu-dm process when you include the vfb config line above. Hence the crash is not really related to vfb directly, but simply the fact that qemu starts up and then crashes for some other bizarre reason. I tried reproducing a vbd config similar to yours (since your crash backtrace indicated crashing in blockdevice-parsing code), but that caused no problems for me. I don''t know whether it could be related to the domUloader.py bootloader stuff. I''ve never used that and I think it''s suse specific? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
0bo0
2010-Jan-27 15:41 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
hi On Wed, Jan 27, 2010 at 1:10 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:> I don''t know whether it could be related to the domUloader.py bootloader > stuff. I''ve never used that and I think it''s suse specific?yes, it''s suse-specific (it''s already been requested that they support pvgrub ...) but, doesn''t seem like the issue :-/ removing the use of the domUloader.py bootloader builder = ''linux'' - bootloader = ''/usr/lib/xen/boot/domUloader.py'' - bootargs = ''--entry=xvda1:vmlinuz-xen,/boot/initrd-xen'' + kernel = ''/home/xen_store/distros/opensuse112/xenboot/vmlinuz-xen'' + ramdisk = ''/home/xen_store/distros/opensuse112/xenboot/initrd-xen'' still results on the same problem/crash -- as lang as vfb = [''''] exists. again, simply removing the vfb spec''n allows the DomU to sanely boot. thanks. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Jan-27 15:46 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On 27/01/2010 15:41, "0bo0" <0.bugs.only.0@gmail.com> wrote:> - bootloader = ''/usr/lib/xen/boot/domUloader.py'' > - bootargs = ''--entry=xvda1:vmlinuz-xen,/boot/initrd-xen'' > + kernel = ''/home/xen_store/distros/opensuse112/xenboot/vmlinuz-xen'' > + ramdisk = ''/home/xen_store/distros/opensuse112/xenboot/initrd-xen'' > > still results on the same problem/crash -- as lang as > > vfb = [''''] > > exists. again, simply removing the vfb spec''n allows the DomU to sanely boot.Hm, well, can you try stripping your config file right back (but still specifying vfb=['''']) -- like removing all vbds for example? Obviously your guest will not successfully boot, but qemu-dm crashing is no doubt preventing your guest from even starting in the first place. So you can strip back and see whether at any point behaviour changes from ''qemu-dm crashes'' to ''guest partially boots but then fails to find a rootfs''. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Sander Eikelenboom
2010-Jan-27 16:19 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
Hmm I could suspect parsing errors for the last comma in the disk option, or the empty vif ? cat /etc/xen/vm/test.cfg --------------------------------- name = ''test'' builder = ''linux'' bootloader = ''/usr/lib/xen/boot/domUloader.py'' bootargs = ''--entry=xvda1:vmlinuz-xen,/boot/initrd-xen'' disk = [ ''phy:/dev/VG0/boot,xvda,w'',''phy:/dev/VG0/swap,xvdb,w'',''phy:/dev/VG0/root,xvdc,w'',] boot = ''c'' #d is cdrom boot, c is disk boot root = ''/dev/xvdc1'' vif = [''''] vfb = [ ''type=vnc, vncdisplay=102, vnclisten=127.0.0.1'' ] extra = ''textmode=1 xencons=xvc0 elevator=noop'' maxmem = 4096 memory = 4096 vcpus = 4 cpu_cap = 100 cpu_weight = 256 localtime = 0 netif = ''yes'' on_shutdown = ''destroy'' on_reboot = ''restart'' on_crash = ''destroy'' --------------------------------- Wednesday, January 27, 2010, 4:46:40 PM, you wrote:> On 27/01/2010 15:41, "0bo0" <0.bugs.only.0@gmail.com> wrote:>> - bootloader = ''/usr/lib/xen/boot/domUloader.py'' >> - bootargs = ''--entry=xvda1:vmlinuz-xen,/boot/initrd-xen'' >> + kernel = ''/home/xen_store/distros/opensuse112/xenboot/vmlinuz-xen'' >> + ramdisk = ''/home/xen_store/distros/opensuse112/xenboot/initrd-xen'' >> >> still results on the same problem/crash -- as lang as >> >> vfb = [''''] >> >> exists. again, simply removing the vfb spec''n allows the DomU to sanely boot.> Hm, well, can you try stripping your config file right back (but still > specifying vfb=['''']) -- like removing all vbds for example? Obviously your > guest will not successfully boot, but qemu-dm crashing is no doubt > preventing your guest from even starting in the first place. So you can > strip back and see whether at any point behaviour changes from ''qemu-dm > crashes'' to ''guest partially boots but then fails to find a rootfs''.> -- Keir-- Best regards, Sander mailto:linux@eikelenboom.it _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
0bo0
2010-Jan-27 16:20 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On Wed, Jan 27, 2010 at 7:46 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:> Hm, well, can you try stripping your config file right back (but still > specifying vfb=['''']) -- like removing all vbds for example? Obviously your > guest will not successfully boot, but qemu-dm crashing is no doubt > preventing your guest from even starting in the first place. So you can > strip back and see whether at any point behaviour changes from ''qemu-dm > crashes'' to ''guest partially boots but then fails to find a rootfs''.i''ve been stripping back as suggested, and trying different config option combos. nothing seems to make any difference to ability to launch, so far, except removal of ''vfb=['''']'' :-/ i''ve switched to booting from a different local vmlinuz & initrd, instead of either the domUbootloader.py or the distro release -xen imgs, and have noticed one change in logs: ... [2010-01-27 08:16:51 2217] WARNING (XendDomainInfo:2072) Domain has crashed: name=nas id=30. [2010-01-27 08:16:51 2217] DEBUG (XendDomainInfo:3061) XendDomainInfo.destroy: domid=30 [2010-01-27 08:16:52 2217] DEBUG (pciif:460) XendDomainInfo.handleAerStateWatch [2010-01-27 08:16:52 2217] DEBUG (XendDomainInfo:1892) XendDomainInfo.handleShutdownWatch [2010-01-27 08:16:52 2217] DEBUG (XendDomainInfo:2416) Destroying device model [2010-01-27 08:16:53 2217] DEBUG (XendDomainInfo:2423) Releasing devices [2010-01-27 08:16:53 2217] DEBUG (XendDomainInfo:2429) Removing vif/0 [2010-01-27 08:16:53 2217] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0 [2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:2429) Removing vif/1 [2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/1 [2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:2429) Removing vif/2 [2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/2 [2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:2429) Removing vif/3 [2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/3 [2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:2429) Removing vkbd/0 [2010-01-27 08:16:54 2217] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vkbd, device = vkbd/0 [2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:2429) Removing vbd/51712 [2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51712 [2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:2429) Removing vbd/51728 [2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51728 [2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:2429) Removing vbd/51744 [2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51744 [2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:2429) Removing vfb/0 [2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0 [2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:2429) Removing pci/0 [2010-01-27 08:16:55 2217] DEBUG (XendDomainInfo:1271) XendDomainInfo.destroyDevice: deviceClass = pci, device = pci/0 [2010-01-27 08:16:55 2217] DEBUG (pciif:578) pci: unregister aer watch [2010-01-27 08:16:55 2217] ERROR (XendConfig:1175) dumping sxp from device controllers Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/xen/xend/XendConfig.py", line 1161, in to_sxp configs = controller.configurations(txn) File "/usr/lib64/python2.6/site-packages/xen/xend/server/DevController.py", line 242, in configurations return map(lambda x: self.configuration(x, transaction), self.deviceIDs(transaction)) File "/usr/lib64/python2.6/site-packages/xen/xend/server/DevController.py", line 242, in <lambda> return map(lambda x: self.configuration(x, transaction), self.deviceIDs(transaction)) File "/usr/lib64/python2.6/site-packages/xen/xend/server/DevController.py", line 249, in configuration configDict = self.getDeviceConfiguration(devid, transaction) File "/usr/lib64/python2.6/site-packages/xen/xend/server/netif.py", line 194, in getDeviceConfiguration y = self.readBackendTxn(transaction, devid, x) File "/usr/lib64/python2.6/site-packages/xen/xend/server/DevController.py", line 447, in readBackendTxn raise VmError("Device %s not connected" % devid) VmError: Device 2 not connected the [2010-01-27 08:16:55 2217] ERROR (XendConfig:1175) dumping sxp from device controllers Traceback (most recent call last): ... are not there in other circumstances ... _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
0bo0
2010-Jan-27 16:22 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On Wed, Jan 27, 2010 at 8:19 AM, Sander Eikelenboom <linux@eikelenboom.it> wrote:> Hmm I could suspect parsing errors for the last comma in the disk option, or the empty vif ?although neither was a problem under Xen3x, never hurts to try. i removed the trailing comma, and replaced the populated vif. no luck -- same inability to boot, until/unless ''vfb=['''']'' is removed. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
0bo0
2010-Jan-27 16:40 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
fyi, BOOTS OK name = ''test'' builder = ''linux'' kernel = ''/boot/vmlinuz-2.6.31.12-2-xen'' ramdisk = ''/boot/initrd-2.6.31.12-2-xen'' disk = [ ''phy:/dev/VG0/boot,xvda,w'',''phy:/dev/VG0/swap,xvdb,w'',''phy:/dev/VG0/root,xvdc,w'',] root = ''/dev/xvdc1'' #vfb = [''''] CRASHES @ qemu-dm, AS REPORTED name = ''test'' builder = ''linux'' kernel = ''/boot/vmlinuz-2.6.31.12-2-xen'' ramdisk = ''/boot/initrd-2.6.31.12-2-xen'' disk = [ ''phy:/dev/VG0/boot,xvda,w'',''phy:/dev/VG0/swap,xvdb,w'',''phy:/dev/VG0/root,xvdc,w'',] root = ''/dev/xvdc1'' vfb = [''''] NO CRASH, JUST WAITS FOR NON-EXSITENT DISK name = ''test'' builder = ''linux'' kernel = ''/boot/vmlinuz-2.6.31.12-2-xen'' ramdisk = ''/boot/initrd-2.6.31.12-2-xen'' #disk = [ ''phy:/dev/VG0/boot,xvda,w'',''phy:/dev/VG0/swap,xvdb,w'',''phy:/dev/VG0/root,xvdc,w'',] root = ''/dev/xvdc1'' vfb = [''''] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
0bo0
2010-Jan-28 22:08 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
just fyi, i did a quick check & reproduced this behavior on a colleague''s box. not configured/installed by me, but admittedly still opensuse. afaict, with vfb=[''''] removed from the config, there are no suspicious effects i see at the DomU -- yet. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Justin T. Gibbs
2010-Jan-29 00:21 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On 1/26/2010 11:36 AM, 0bo0 wrote:> in another thread @xen-devel, a crash @ DomU init under Xen 4.0.x, i > was advised, > > On Mon, Jan 25, 2010 at 11:34 PM, Keir Fraser<keir.fraser@eu.citrix.com> wrote:> > If qemu-dm died without logging anything, it probably crashed again.If it> > produced a core file you could load that in gdb and get a symbolic > > backtrace. > > trying to turn on core dumps in xend-config, unfortunately, Xen seems > to fail @ core dump ... in addition to failing to launch a DomU which > works fine under Xen3,... I encountered this same problem. I debugged it by pulling the OpenSuSE source package for xen-4.0.0 and then manually rebuilding qemu with symbols. The problem is in a supplemental patch that OpenSuSE is adding for CDROM media change support: cdrom-removable.patch. This patched code accesses an uninitialized variable, causing the segfault. If you want to rebuild the package from source, I''ve attached a corrected version of the cdrom-removable.patch file. Just install the source rpm (zypper si xen), replace the /usr/src/packages/SOURCES/cdrom-removable.patch file with what is below, rebuild your packages (rpmbuild -ba /usr/src/packages/SPECS/xen.spec), and install the packages found in /usr/src/packages/RPMS/<arch>/. You only need to reinstall xen-tools to get the fix. I''ve also posted information about this problem to the OpenSuSE-virtual list. Hopefully they will quickly incorporate a fix. -- Justin Index: xen-4.0.0-testing/tools/python/xen/xend/server/HalDaemon.py ==================================================================--- /dev/null +++ xen-4.0.0-testing/tools/python/xen/xend/server/HalDaemon.py @@ -0,0 +1,243 @@ +#!/usr/bin/env python +# -*- mode: python; -*- +#===========================================================================+# This library is free software; you can redistribute it and/or +# modify it under the terms of version 2.1 of the GNU Lesser General Public +# License as published by the Free Software Foundation. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +#===========================================================================+# Copyright (C) 2007 Pat Campbell <plc@novell.com> +# Copyright (C) 2007 Novell Inc. +#===========================================================================+ +"""hald (Hardware Abstraction Layer Daemon) watcher for Xen management + of removable block device media. + +""" + +import gobject +import dbus +import dbus.glib +import os +import types +import sys +import signal +import traceback +from xen.xend.xenstore.xstransact import xstransact, complete +from xen.xend.xenstore.xsutil import xshandle +from xen.xend import PrettyPrint +from xen.xend import XendLogging +from xen.xend.XendLogging import log + +DEVICE_TYPES = [''vbd'', ''tap''] + +class HalDaemon: + """The Hald block device watcher for XEN + """ + + """Default path to the log file. """ + logfile_default = "/var/log/xen/hald.log" + + """Default level of information to be logged.""" + loglevel_default = ''INFO'' + + + def __init__(self): + + XendLogging.init(self.logfile_default, self.loglevel_default) + log.debug( "%s", "__init__") + + self.udi_dict = {} + self.debug = 0 + self.dbpath = "/local/domain/0/backend" + self.bus = dbus.SystemBus() + self.hal_manager_obj = self.bus.get_object(''org.freedesktop.Hal'', ''/org/freedesktop/Hal/Manager'') + self.hal_manager = dbus.Interface( self.hal_manager_obj, ''org.freedesktop.Hal.Manager'') + self.gatherBlockDevices() + self.registerDeviceCallbacks() + + def run(self): + log.debug( "%s", "In new run" ); + try: + self.mainloop = gobject.MainLoop() + self.mainloop.run() + except KeyboardInterrupt, ex: + log.debug(''Keyboard exception handler: %s'', ex ) + self.mainloop.quit() + except Exception, ex: + log.debug(''Generic exception handler: %s'', ex ) + self.mainloop.quit() + + def __del__(self): + log.debug( "%s", "In del " ); + self.unRegisterDeviceCallbacks() + self.mainloop.quit() + + def shutdown(self): + log.debug( "%s", "In shutdown now " ); + self.unRegisterDeviceCallbacks() + self.mainloop.quit() + + def stop(self): + log.debug( "%s", "In stop now " ); + self.unRegisterDeviceCallbacks() + self.mainloop.quit() + + def gatherBlockDevices(self): + + # Get all the current devices from hal and save in a dictionary + try: + device_names = self.hal_manager.GetAllDevices() + i = 0; + for name in device_names: + #log.debug("device name, device=%s",name) + dev_obj = self.bus.get_object (''org.freedesktop.Hal'', name) + dev = dbus.Interface (dev_obj, ''org.freedesktop.Hal.Device'') + dev_properties = dev_obj.GetAllProperties(dbus_interface="org.freedesktop.Hal.Device") + if dev_properties.has_key(''block.device''): + dev_str = dev_properties[''block.device''] + dev_major = dev_properties[''block.major''] + dev_minor = dev_properties[''block.minor''] + udi_info = {} + udi_info[''device''] = dev_str + udi_info[''major''] = dev_major + udi_info[''minor''] = dev_minor + udi_info[''udi''] = name + self.udi_dict[i] = udi_info + i = i + 1 + except Exception, ex: + print >>sys.stderr, ''Exception gathering block devices:'', ex + log.warn("Exception gathering block devices (%s)",ex) + + # + def registerDeviceCallbacks(self): + # setup the callbacks for when the gdl changes + self.hal_manager.connect_to_signal(''DeviceAdded'', self.device_added_callback) + self.hal_manager.connect_to_signal(''DeviceRemoved'', self.device_removed_callback) + + # + def unRegisterDeviceCallbacks(self): + # setup the callbacks for when the gdl changes + self.hal_manager.remove_signal_receiver(self.device_added_callback,''DeviceAdded'') + self.hal_manager.remove_signal_receiver(self.device_removed_callback,''DeviceRemoved'') + + # + def device_removed_callback(self,udi): + log.debug(''UDI %s was removed'',udi) + self.show_dict(self.udi_dict) + for key in self.udi_dict: + udi_info = self.udi_dict[key] + if udi_info[''udi''] == udi: + device = udi_info[''device''] + major = udi_info[''major''] + minor = udi_info[''minor''] + self.change_xenstore( "remove", device, major, minor) + + # Adds device to dictionary if not already there + def device_added_callback(self,udi): + log.debug(''UDI %s was added'', udi) + self.show_dict(self.udi_dict) + dev_obj = self.bus.get_object (''org.freedesktop.Hal'', udi) + dev = dbus.Interface (dev_obj, ''org.freedesktop.Hal.Device'') + device = dev.GetProperty (''block.device'') + major = dev.GetProperty (''block.major'') + minor = dev.GetProperty (''block.minor'') + udi_info = {} + udi_info[''device''] = device + udi_info[''major''] = major + udi_info[''minor''] = minor + udi_info[''udi''] = udi + already = 0 + cnt = 0; + for key in self.udi_dict: + info = self.udi_dict[key] + if info[''udi''] == udi: + already = 1 + break + cnt = cnt + 1 + if already == 0: + self.udi_dict[cnt] = udi_info; + log.debug(''UDI %s was added, device:%s major:%s minor:%s index:%d\n'', udi, device, major, minor, cnt) + self.change_xenstore( "add", device, major, minor) + + # Debug helper, shows dictionary contents + def show_dict(self,dict=None): + if self.debug == 0 : + return + if dict == None : + dict = self.udi_dict + for key in dict: + log.debug(''udi_info %s udi_info:%s'',key,dict[key]) + + # Set or clear xenstore media-present depending on the action argument + # for every vbd that has this block device + def change_xenstore(self,action, device, major, minor): + for type in DEVICE_TYPES: + path = self.dbpath + ''/'' + type + domains = xstransact.List(path) + log.debug(''domains: %s'', domains) + for domain in domains: # for each domain + devices = xstransact.List( path + ''/'' + domain) + log.debug(''devices: %s'',devices) + for device in devices: # for each vbd device + str = device.split(''/'') + vbd_type = None; + vbd_physical_device = None + vbd_media = None + vbd_device_path = path + ''/'' + domain + ''/'' + device + listing = xstransact.List(vbd_device_path) + for entry in listing: # for each entry + item = path + ''/'' + entry + value = xstransact.Read( vbd_device_path + ''/'' + entry) + log.debug(''%s=%s'',item,value) + if item.find(''media-present'') != -1: + vbd_media = item; + vbd_media_path = item + if item.find(''physical-device'') != -1: + vbd_physical_device = value; + if item.find(''type'') != -1: + vbd_type = value; + if vbd_type is not None and vbd_physical_device is not None and vbd_media is not None : + inode = vbd_physical_device.split('':'') + imajor = parse_hex(inode[0]) + iminor = parse_hex(inode[1]) + log.debug("action:%s major:%s- minor:%s- imajor:%s- iminor:%s- inode: %s", + action,major,minor, imajor, iminor, inode) + if int(imajor) == int(major) and int(iminor) == int(minor): + if action == "add": + xs_dict = {''media'': "1"} + xstransact.Write(vbd_device_path, ''media-present'', "1" ) + log.debug("wrote xenstore media-present 1 path:%s",vbd_media_path) + else: + xstransact.Write(vbd_device_path, ''media-present'', "0" ) + log.debug("wrote xenstore media 0 path:%s",vbd_media_path) + +def mylog( fmt, *args): + f = open(''/tmp/haldaemon.log'', ''a'') + print >>f, "HalDaemon ", fmt % args + f.close() + + +def parse_hex(val): + try: + if isinstance(val, types.StringTypes): + return int(val, 16) + else: + return val + except ValueError: + return None + +if __name__ == "__main__": + watcher = HalDaemon() + watcher.run() + print ''Falling off end'' + + Index: xen-4.0.0-testing/tools/python/xen/xend/server/Hald.py ==================================================================--- /dev/null +++ xen-4.0.0-testing/tools/python/xen/xend/server/Hald.py @@ -0,0 +1,125 @@ +#===========================================================================+# This library is free software; you can redistribute it and/or +# modify it under the terms of version 2.1 of the GNU Lesser General Public +# License as published by the Free Software Foundation. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +#===========================================================================+# Copyright (C) 2007 Pat Campbell <plc@novell.com> +# Copyright (C) 2007 Novell Inc. +#===========================================================================+ +import errno +import types +import os +import sys +import time +import signal +from traceback import print_exc + +from xen.xend.XendLogging import log + +class Hald: + def __init__(self): + self.ready = False + self.running = True + + def run(self): + """Starts the HalDaemon process + """ + self.ready = True + try: + myfile = self.find("xen/xend/server/HalDaemon.py") + args = (["python", myfile ]) + self.pid = self.daemonize("python", args ) + #log.debug( "%s %s pid:%d", "Hald.py starting ", args, self.pid ) + except: + self.pid = -1 + log.debug("Unable to start HalDaemon process") + + def shutdown(self): + """Shutdown the HalDaemon process + """ + log.debug("%s pid:%d", "Hald.shutdown()", self.pid) + self.running = False + self.ready = False + if self.pid != -1: + try: + os.kill(self.pid, signal.SIGINT) + except: + print_exc() + + def daemonize(self,prog, args): + """Runs a program as a daemon with the list of arguments. Returns the PID + of the daemonized program, or returns 0 on error. + Copied from xm/create.py instead of importing to reduce coupling + """ + r, w = os.pipe() + pid = os.fork() + + if pid == 0: + os.close(r) + w = os.fdopen(w, ''w'') + os.setsid() + try: + pid2 = os.fork() + except: + pid2 = None + if pid2 == 0: + os.chdir("/") + env = os.environ.copy() + env[''PYTHONPATH''] = self.getpythonpath() + for fd in range(0, 256): + try: + os.close(fd) + except: + pass + os.open("/dev/null", os.O_RDWR) + os.dup2(0, 1) + os.dup2(0, 2) + os.execvpe(prog, args, env) + os._exit(1) + else: + w.write(str(pid2 or 0)) + w.close() + os._exit(0) + os.close(w) + r = os.fdopen(r) + daemon_pid = int(r.read()) + r.close() + os.waitpid(pid, 0) + #log.debug( "daemon_pid: %d", daemon_pid ) + return daemon_pid + + def getpythonpath(self): + str = " " + for p in sys.path: + if str != " ": + str = str + ":" + p + else: + if str != "": + str = p + return str + + def find(self,path, matchFunc=os.path.isfile): + """Find a module in the sys.path + From web page: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52224 + """ + for dirname in sys.path: + candidate = os.path.join(dirname, path) + if matchFunc(candidate): + return candidate + raise Error("Can''t find file %s" % path) + +if __name__ == "__main__": + watcher = Hald() + watcher.run() + time.sleep(10) + watcher.shutdown() Index: xen-4.0.0-testing/tools/python/xen/xend/server/SrvServer.py ==================================================================--- xen-4.0.0-testing.orig/tools/python/xen/xend/server/SrvServer.py +++ xen-4.0.0-testing/tools/python/xen/xend/server/SrvServer.py @@ -56,6 +56,7 @@ from xen.web.SrvDir import SrvDir from SrvRoot import SrvRoot from XMLRPCServer import XMLRPCServer +from xen.xend.server.Hald import Hald xoptions = XendOptions.instance() @@ -245,6 +246,8 @@ def _loadConfig(servers, root, reload): if xoptions.get_xend_unix_xmlrpc_server(): servers.add(XMLRPCServer(XendAPI.AUTH_PAM, False)) + servers.add(Hald()) + def create(): root = SrvDir() Index: xen-4.0.0-testing/tools/ioemu-remote/xenstore.c ==================================================================--- xen-4.0.0-testing.orig/tools/ioemu-remote/xenstore.c +++ xen-4.0.0-testing/tools/ioemu-remote/xenstore.c @@ -594,6 +594,20 @@ #endif bs = bdrv_new(dev); + /* if cdrom pyhsical put a watch on media-present */ + if (bdrv_get_type_hint(bs) == BDRV_TYPE_CDROM) { + if (drv && !strcmp(drv, "phy")) { + if (pasprintf(&buf, "%s/media-present", bpath) != -1) { + if (bdrv_is_inserted(bs)) + xs_write(xsh, XBT_NULL, buf, "1", strlen("1")); + else { + xs_write(xsh, XBT_NULL, buf, "0", strlen("0")); + } + xs_watch(xsh, buf, "media-present"); + } + } + } + /* check if it is a cdrom */ if (danger_type && !strcmp(danger_type, "cdrom")) { bdrv_set_type_hint(bs, BDRV_TYPE_CDROM); @@ -938,6 +951,50 @@ void xenstore_record_dm_state(const char xenstore_record_dm("state", state); } +void xenstore_process_media_change_event(char **vec) +{ + char *media_present = NULL; + unsigned int len; + + media_present = xs_read(xsh, XBT_NULL, vec[XS_WATCH_PATH], &len); + + if (media_present) { + BlockDriverState *bs; + char *buf = NULL, *cp = NULL, *path = NULL, *dev = NULL; + + path = strdup(vec[XS_WATCH_PATH]); + cp = strstr(path, "media-present"); + if (cp){ + *(cp-1) = ''\0''; + pasprintf(&buf, "%s/dev", path); + dev = xs_read(xsh, XBT_NULL, buf, &len); + if (dev) { + if ( !strncmp(dev, "xvd", 3)) { + memmove(dev, dev+1, strlen(dev)); + dev[0] = ''h''; + dev[1] = ''d''; + } + bs = bdrv_find(dev); + if (!bs) { + term_printf("device not found\n"); + return; + } + if (strcmp(media_present, "0") == 0 && bs) { + bdrv_close(bs); + } + else if (strcmp(media_present, "1") == 0 && + bs != NULL && bs->drv == NULL) { + if (bdrv_open(bs, bs->filename, 0 /* snapshot */) < 0) { + fprintf(logfile, "%s() qemu: could not open cdrom disk ''%s''\n", + __func__, bs->filename); + } + bs->media_changed = 1; + } + } + } + } +} + void xenstore_process_event(void *opaque) { char **vec, *offset, *bpath = NULL, *buf = NULL, *drv = NULL, *image = NULL; @@ -968,6 +1025,11 @@ void xenstore_process_event(void *opaque xenstore_watch_callbacks[i].cb(vec[XS_WATCH_TOKEN], xenstore_watch_callbacks[i].opaque); + if (!strcmp(vec[XS_WATCH_TOKEN], "media-present")) { + xenstore_process_media_change_event(vec); + goto out; + } + if (strncmp(vec[XS_WATCH_TOKEN], "hd", 2) || strlen(vec[XS_WATCH_TOKEN]) != 3) goto out; _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
0bo0
2010-Jan-29 00:42 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
hi On Thu, Jan 28, 2010 at 4:21 PM, Justin T. Gibbs <gibbs@scsiguy.com> wrote:> I encountered this same problem._which_ problem did you encounter? getting hard to keep track of them all :-) does your patch address (1) the crash on DomU launch in the presence of ''vfb=['''']''? or (2) the failure to core dump with dumps enabled in xend-config.sxp ? thanks. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Justin T. Gibbs
2010-Jan-29 00:44 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On 1/28/2010 5:42 PM, 0bo0 wrote:> hi > > On Thu, Jan 28, 2010 at 4:21 PM, Justin T. Gibbs<gibbs@scsiguy.com> wrote: > >> I encountered this same problem. >> > _which_ problem did you encounter? getting hard to keep track of them all :-) > > does your patch address > > (1) the crash on DomU launch in the presence of ''vfb=['''']''? >Just 1. -- Justin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
0bo0
2010-Jan-29 00:54 UTC
Re: [Xen-devel] trying to debug xen4 & qemu-dm, Xen reports "core dump failed"
On Thu, Jan 28, 2010 at 4:44 PM, Justin T. Gibbs <gibbs@scsiguy.com> wrote:>> (1) the crash on DomU launch in the presence of ''vfb=['''']''? >> > > Just 1.thanks. in that case, your patch is relevant here https://bugzilla.novell.com/show_bug.cgi?id=574046#c4, despite the protestations that Xen4 on OpenSuse 11.2 is something they neither support nor test :-S it''s already referenced there, added by @novell ... now, perhaps, they''ll include the fix. thanks again. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel