Xen devel - Oct 2013 - xl console regression on xen-unstable

I''m unable to start xl console on latest xen-unstable (build with
commit
8e0da8c07f4f80e14314977a11f738bd74a5b62b).
> xl -vvv console sid
> xenconsole: Could not read tty from store: No such file or directory
I also tried "xl create -c", and it shows console from pygrub but when
the "xl create" reaches the end I got the same
error:
> xenconsole: Could not read tty from store: No such file or directory
I tried on Sid minimal (debian unstable) and Precise desktop (Ubuntu 
12.04) pv domUs.
I haven''t use it for a long time (probably some months), I
don''t
remember what is the last test build where was working.

If you need more details tell me and I''ll post them.

Thanks for any reply.

On Tue, Oct 08, 2013 at 04:47:17PM +0200, Fabio Fantoni
wrote:
> I''m unable to start xl console on latest xen-unstable (build with
> commit 8e0da8c07f4f80e14314977a11f738bd74a5b62b).
> >xl -vvv console sid
> >xenconsole: Could not read tty from store: No such file or directory
> 
It would be good if you can paste the output here.
> I also tried "xl create -c", and it shows console from pygrub but
> when the "xl create" reaches the end I got the same error:
> >xenconsole: Could not read tty from store: No such file or directory
> 
Please run xl -vvv create -c, paste output here.
> I tried on Sid minimal (debian unstable) and Precise desktop (Ubuntu
> 12.04) pv domUs.
> I haven''t use it for a long time (probably some months), I
don''t
> remember what is the last test build where was working.
> 
> If you need more details tell me and I''ll post them.
> 
> Thanks for any reply.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

On Mon, Oct 14, 2013 at 04:21:52PM +0200, Fabio Fantoni
wrote:
> Il 14/10/2013 16:02, Wei Liu ha scritto:
> >On Tue, Oct 08, 2013 at 04:47:17PM +0200, Fabio Fantoni wrote:
> >>I''m unable to start xl console on latest xen-unstable
(build with
> >>commit 8e0da8c07f4f80e14314977a11f738bd74a5b62b).
> >>>xl -vvv console sid
> >>>xenconsole: Could not read tty from store: No such file or
directory
> >It would be good if you can paste the output here.
> 
> Thanks for reply.
> The "xenconsole: ..." line is the only output of "xl -vvv
console
> sid" command, before that it shows nothing for some seconds, then
> return to command line.
> 
> >>I also tried "xl create -c", and it shows console from
pygrub but
> >>when the "xl create" reaches the end I got the same
error:
> >>>xenconsole: Could not read tty from store: No such file or
directory
> >Please run xl -vvv create -c, paste output here.
> 
> See on attachment.
> 
Could you also attach the output of "xenstore-ls" when your DomU is up
and running.
> Thanks for any reply.
> 
> >>I tried on Sid minimal (debian unstable) and Precise desktop
(Ubuntu
> >>12.04) pv domUs.
> >>I haven''t use it for a long time (probably some months), I
don''t
> >>remember what is the last test build where was working.
> >>
> >>If you need more details tell me and I''ll post them.
> >>
> >>Thanks for any reply.
> >>
> >>_______________________________________________
> >>Xen-devel mailing list
> >>Xen-devel@lists.xen.org
> >>http://lists.xen.org/xen-devel
> 
> xl -vvv create /etc/xen/sid.cfg -c
> Parsing config from /etc/xen/sid.cfg
> libxl: debug: libxl_create.c:1263:do_domain_create: ao 0x8540d0: create:
how=(nil) callback=(nil) poller=0x853f90
> libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk
vdev=xvda spec.backend=unknown
> libxl: debug: libxl_device.c:197:disk_try_backend: Disk vdev=xvda, backend
phy unsuitable as phys path not a block device
> libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk
vdev=xvda, using backend qdisk
> libxl: debug: libxl_create.c:707:initiate_domain_create: running bootloader
> libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk
vdev=(null) spec.backend=qdisk
> libxl: debug: libxl.c:2654:libxl__device_disk_local_initiate_attach:
locally attaching qdisk /mnt/vm/disks/sid.disk1.xm
> libxl: debug: libxl_bootloader.c:409:bootloader_disk_attached_cb: Config
bootloader value: pygrub
> libxl: debug: libxl_bootloader.c:425:bootloader_disk_attached_cb: Checking
for bootloader in libexec path: /usr/lib/xen/bin/pygrub
> libxl: debug: libxl_create.c:1277:do_domain_create: ao 0x8540d0:
inprogress: poller=0x853f90, flags=i
> libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch
w=0x854458 wpath=/local/domain/6 token=3/0: register slotnum=3
> libxl: debug: libxl_event.c:1748:libxl__ao_progress_report: ao 0x8540d0:
progress report: callback queued aop=0x8553f0
> libxl: debug: libxl_bootloader.c:535:bootloader_gotptys: executing
bootloader: /usr/lib/xen/bin/pygrub
> libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader arg:
/usr/lib/xen/bin/pygrub
> libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader arg:
--output=/var/run/xen/bootloader.6.out
> libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader arg:
--output-format=simple0
> libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader arg:
--output-directory=/var/run/xen/bootloader.6.d
> libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader arg:
/mnt/vm/disks/sid.disk1.xm
> libxl: debug: libxl_event.c:1160:egc_run_callbacks: ao 0x8540d0: progress
report: callback aop=0x8553f0
> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x854458
wpath=/local/domain/6 token=3/0: event epath=/local/domain/6
> Using <class ''grub.GrubConf.Grub2ConfigFile''> to
parse /boot/grub/grub.cfg
>                                                                          
WARNING:root:Unknown directive menuentry_id_option
>                                                                            
WARNING:root:Unknown directive else
>                                                                            
WARNING:root:Unknown directive menuentry_id_option
>                                WARNING:root:Unknown directive export
>                                                                    
WARNING:root:Unknown directive font
>                                                                            
WARNING:root:Unknown directive else
>                                                                            
WARNING:root:Unknown directive else
>                                                                            
WARNING:root:Unknown directive font
>                               WARNING:root:Unknown directive load_video
>                                                                       
WARNING:root:Unknown directive terminal_output
>                                                                            
WARNING:root:Unknown image directive load_video
>                                                                            
WARNING:root:Unknown image directive if
>                          WARNING:root:Unknown image directive else
>                                                                  
WARNING:root:Unknown image directive fi
>                                                                            
WARNING:root:Unknown image directive load_video
>                                                                            
WARNING:root:Unknown image directive if
>              WARNING:root:Unknown image directive else
>                                                       WARNING:root:Unknown
image directive fi
>                                                                            
WARNING:root:Unknown image directive load_video
>                                                                            
WARNING:root:Unknown image directive if
>  WARNING:root:Unknown image directive else
>                                           WARNING:root:Unknown image
directive fi
>                                                                            
WARNING:root:Unknown image directive load_video
>                                                                            
WARNING:root:Unknown image directive if
>                                                                            
WARNING:root:Unknown image directive else
>                               WARNING:root:Unknown image directive fi
>                                                                     
WARNING:root:Unknown image directive load_video
>                                                                            
WARNING:root:Unknown image directive if
>                                                                            
WARNING:root:Unknown image directive else
>                   WARNING:root:Unknown image directive fi
>                                                         
WARNING:root:Unknown image directive load_video
>                                                                            
WARNING:root:Unknown image directive if
>                                                                            
WARNING:root:Unknown image directive else
>       WARNING:root:Unknown image directive fi
>                                              WARNING:root:Unknown image
directive load_video
>                                                                            
WARNING:root:Unknown image directive if
>                                                                            
WARNING:root:Unknown image directive else
>                                                                            
WARNING:root:Unknown image directive fi
>                                  WARNING:root:Unknown directive source
>                                                                      
WARNING:root:Unknown directive elif
>                                                                            
WARNING:root:Unknown directive source
> 
>     pyGRUB  version 0.6
>  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
>  x Debian GNU/Linux'' --class debian --class gnu-linux --class gnu
--class o
>  s Debian GNU/Linux, con Linux 3.10-3-amd64'' --class debian
--class gnu-lin
>  uxDebian GNU/Linux, con Linux 3.10-3-amd64 (modalità
ripristino)'' --class
>   dDebian GNU/Linux, con Linux 3.10-2-amd64'' --class debian
--class gnu-lin
>  uxDebian GNU/Linux, con Linux 3.10-2-amd64 (modalità
ripristino)'' --class
>   dDebian GNU/Linux, con Linux 3.2.0-4-amd64'' --class debian
--class gnu-li
>  nuDebian GNU/Linux, con Linux 3.2.0-4-amd64 (modalità
ripristino)'' --clas
>  s debian --class gnu-linux --class gnu --class os $menuentry_id_option
''gn
>  ulinux-3.2.0-4-amd64-recovery-3ab55964-09d1-4853-be38-661b5a476a14qqqqqqqj
>      Use the ^ and v keys to select which entry is highlighted.
>      Press enter to boot the selected OS, ''e'' to edit the
>      commands before booting, ''a'' to modify the kernel
arguments
>      before booting, or ''c'' for a command line.
> 
> 
> 
> 
>      Will boot selected entry in  1 seconds
> 
> 
> libxl: debug: libxl_bootloader.c:634:bootloader_finished: bootloader
completed
>                                                                            
libxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader
output contained kernel /var/run/xen/bootloader.6.d/boot_kernel.8w74ac
>                                              libxl: debug:
libxl_bootloader.c:136:bootloader_result_command: bootloader output contained
ramdisk /var/run/xen/bootloader.6.d/boot_ramdisk.CnDxeo
>               libxl: debug:
libxl_bootloader.c:136:bootloader_result_command: bootloader output contained
args root=UUID=3ab55964-09d1-4853-be38-661b5a476a14 ro console=tty0 quiet
>  libxl: debug: libxl_bootloader.c:647:bootloader_finished: bootloader
execution successful
>                                                                            
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x854458
wpath=/local/domain/6 token=3/0: deregister slotnum=3
>                                             libxl: debug:
libxl_numa.c:475:libxl__get_numa_candidate: New best NUMA placement candidate
found: nr_nodes=1, nr_cpus=8, nr_vcpus=10, free_memkb=9051
>                 libxl: detail: libxl_dom.c:195:numa_place_domain: NUMA
placement candidate with 1 nodes, 8 cpus and 9051 KB free selected
>                                                                            
domainbuilder: detail: xc_dom_allocate:
cmdline="root=UUID=3ab55964-09d1-4853-be38-661b5a476a14 ro console=tty0
quiet", features="(null)"
>                                                                            
libxl: debug: libxl_dom.c:341:libxl__build_pv: pv kernel mapped 1 path
/var/run/xen/bootloader.6.d/boot_kernel.8w74ac
> 
>                                    domainbuilder: detail:
xc_dom_kernel_mem: called
>                                                                            
domainbuilder: detail: xc_dom_ramdisk_mem: called
>                                                                            
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.4, caps xen-3.0-x86_64
xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
>                                                                            
domainbuilder: detail: xc_dom_parse_image: called
>                                                                            
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
>                                        domainbuilder: detail: loader probe
failed
>                                                                            
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...
>                                                                            
domainbuilder: detail: XZ: Saw data stream end
>                         domainbuilder: detail: _xc_try_lzma_decode: XZ
decompress OK, 0x250b14 -> 0xb88960
>                                                                            
domainbuilder: detail: loader probe OK
>                                                                            
xc: detail: elf_parse_binary: phdr: paddr=0x1000000 memsz=0x585000
>                                 xc: detail: elf_parse_binary: phdr:
paddr=0x1600000 memsz=0x9d0f0
>                                                                            
xc: detail: elf_parse_binary: phdr: paddr=0x169e000 memsz=0x14bc0
>                                                                            
xc: detail: elf_parse_binary: phdr: paddr=0x16b3000 memsz=0x5dd000
>                                                   xc: detail:
elf_parse_binary: memory: 0x1000000 -> 0x1c90000
>                                                                            
xc: detail: elf_xen_parse_note: GUEST_OS = "linux"
>                                                                            
xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6"
>                                    xc: detail: elf_xen_parse_note:
XEN_VERSION = "xen-3.0"
>                                                                            
xc: detail: elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
>                                                                            
xc: detail: elf_xen_parse_note: ENTRY = 0xffffffff816b31e0
>                                 xc: detail: elf_xen_parse_note:
HYPERCALL_PAGE = 0xffffffff81001000
>                                                                            
xc: detail: elf_xen_parse_note: FEATURES =
"!writable_page_tables|pae_pgdir_above_4gb"
>        xc: detail: elf_xen_parse_note: PAE_MODE = "yes"
>                                                        xc: detail:
elf_xen_parse_note: LOADER = "generic"
>                                                                            
xc: detail: elf_xen_parse_note: unknown xen elf note (0xd)
>                                                                            
xc: detail: elf_xen_parse_note: SUSPEND_CANCEL = 0x1
>                                      xc: detail: elf_xen_parse_note:
HV_START_LOW = 0xffff800000000000
>                                                                            
xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0x0
>                                                                            
xc: detail: elf_xen_addr_calc_check: addresses:
>                      xc: detail:     virt_base        = 0xffffffff80000000
>                                                                          
xc: detail:     elf_paddr_offset = 0x0
>                                                                            
xc: detail:     virt_offset      = 0xffffffff80000000
>                                                                            
xc: detail:     virt_kstart      = 0xffffffff81000000
>                                         xc: detail:     virt_kend        =
0xffffffff81c90000
>                                                                            
xc: detail:     virt_entry       = 0xffffffff816b31e0
>                                                                            
xc: detail:     p2m_base         = 0xffffffffffffffff
>                      domainbuilder: detail: xc_dom_parse_elf_kernel:
xen-3.0-x86_64: 0xffffffff81000000 -> 0xffffffff81c90000
>                                                                            
domainbuilder: detail: xc_dom_mem_init: mem 1024 MB, pages 0x40000 pages, 4k
each
>                             domainbuilder: detail: xc_dom_mem_init: 0x40000
pages
>                                                                            
domainbuilder: detail: xc_dom_boot_mem_init: called
>                                                                            
domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64
>                           domainbuilder: detail: xc_dom_malloc            :
2048 kB
>                                                                            
domainbuilder: detail: xc_dom_build_image: called
>                                                                            
domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 0xffffffff81000000
-> 0xffffffff81c90000  (pfn 0x1000 + 0xc90 pages)
>                                                                            
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x1000+0xc90 at 0x7ffd49b8b000
>      xc: detail: elf_load_binary: phdr 0 at 0x7ffd49b8b000 ->
0x7ffd4a110000
>                                                                            
xc: detail: elf_load_binary: phdr 1 at 0x7ffd4a18b000 -> 0x7ffd4a2280f0
>                                                                            
xc: detail: elf_load_binary: phdr 2 at 0x7ffd4a229000 -> 0x7ffd4a23dbc0
>                                         xc: detail: elf_load_binary: phdr 3
at 0x7ffd4a23e000 -> 0x7ffd4a313000
>                                                                            
domainbuilder: detail: xc_dom_alloc_segment:   ramdisk      : 0xffffffff81c90000
-> 0xffffffff83d8f000  (pfn 0x1c90 + 0x20ff pages)
>                                                                
domainbuilder: detail: xc_dom_malloc            : 197 kB
>                                                                            
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x1c90+0x20ff at 0x7ffd47a8c000
>                                           domainbuilder: detail:
xc_dom_do_gunzip: unzip ok, 0xb0660a -> 0x20fe210
>                                                                            
domainbuilder: detail: xc_dom_alloc_segment:   phys2mach    : 0xffffffff83d8f000
-> 0xffffffff83f8f000  (pfn 0x3d8f + 0x200 pages)
>                                                                  
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x3d8f+0x200 at 0x7ffd4655f000
>                                                                            
domainbuilder: detail: xc_dom_alloc_page   :   start info   : 0xffffffff83f8f000
(pfn 0x3f8f)
>                                                                            
domainbuilder: detail: xc_dom_alloc_page   :   xenstore     : 0xffffffff83f90000
(pfn 0x3f90)
>                                                                            
domainbuilder: detail: xc_dom_alloc_page   :   console      : 0xffffffff83f91000
(pfn 0x3f91)
>                                                                            
domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48: 0xffff000000000000
-> 0xffffffffffffffff, 1 table(s)
>                         domainbuilder: detail: nr_page_tables:
0x0000007fffffffff/39: 0xffffff8000000000 -> 0xffffffffffffffff, 1 table(s)
>                                                                            
domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30: 0xffffffff80000000
-> 0xffffffffbfffffff, 1 table(s)
>                                                                          
domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21: 0xffffffff80000000
-> 0xffffffff843fffff, 34 table(s)
>            domainbuilder: detail: xc_dom_alloc_segment:   page tables  :
0xffffffff83f92000 -> 0xffffffff83fb7000  (pfn 0x3f92 + 0x25 pages)
>                                                                            
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x3f92+0x25
at 0x7ffd4d9fc000
>                                                             domainbuilder:
detail: xc_dom_alloc_page   :   boot stack   : 0xffffffff83fb7000 (pfn 0x3fb7)
>                                                                            
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0xffffffff83fb8000
>                                                        domainbuilder:
detail: xc_dom_build_image  : virt_pgtab_end : 0xffffffff84400000
>                                                                            
domainbuilder: detail: xc_dom_boot_image: called
>      domainbuilder: detail: arch_setup_bootearly: doing nothing
>                                                               
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64
<= matches
>                                                                            
domainbuilder: detail: xc_dom_compat_check: supported guest type:
xen-3.0-x86_32p
>                                                          domainbuilder:
detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32
>                                                                            
domainbuilder: detail: xc_dom_compat_check: supported guest type:
hvm-3.0-x86_32p
>                                         domainbuilder: detail:
xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
>                                                                            
domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x40000
>               domainbuilder: detail: clear_page: pfn 0x3f91, mfn 0x33d99b
>                                                                         
domainbuilder: detail: clear_page: pfn 0x3f90, mfn 0x33d99c
>                                                                            
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x3f8f+0x1
at 0x7ffd4d9f9000
>                                                    domainbuilder: detail:
start_info_x86_64: called
>                                                                            
domainbuilder: detail: setup_hypercall_page: vaddr=0xffffffff81001000 pfn=0x1001
>  domainbuilder: detail: domain builder memory footprint
>                                                        domainbuilder:
detail:    allocated
>                                                                            
domainbuilder: detail:       malloc             : 2336 kB
>                                                                            
domainbuilder: detail:       anon mmap          : 0 bytes
>                           domainbuilder: detail:    mapped
>                                                           domainbuilder:
detail:       file mmap          : 0 bytes
>                                                                            
domainbuilder: detail:       domU mmap          : 47 MB
>                                                                            
domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0xbf2f4
>                                                                     
domainbuilder: detail: shared_info_x86_64: called
>                                                                            
domainbuilder: detail: vcpu_x86_64: called
>                                                                            
domainbuilder: detail: vcpu_x86_64: cr3: pfn 0x3f92 mfn 0x33d99a
>                                               domainbuilder: detail:
launch_vm: called, ctxt=0x7ffd4d9fa004
>                                                                            
domainbuilder: detail: xc_dom_release: called
>                                                                            
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda
spec.backend=qdisk
>                                                                        
libxl: debug: libxl_dm.c:1280:libxl__spawn_local_dm: Spawning device-model
/usr/lib/xen/bin/qemu-system-i386 with arguments:
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:  
/usr/lib/xen/bin/qemu-system-i386
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   -xen-domid
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   6
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   -chardev
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:  
socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-6,server,nowait
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   -mon
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:  
chardev=libxl-cmd,mode=control
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   -nodefaults
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   -xen-attach
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   -name
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   sid
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   -vnc
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   0.0.0.0:0,to=99
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   -k
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   it
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   -M
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   xenpv
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   -m
> libxl: debug: libxl_dm.c:1282:libxl__spawn_local_dm:   1025
> libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch
w=0x854690 wpath=/local/domain/0/device-model/6/state token=3/1: register
slotnum=3
> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x854690
wpath=/local/domain/0/device-model/6/state token=3/1: event
epath=/local/domain/0/device-model/6/state
> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x854690
wpath=/local/domain/0/device-model/6/state token=3/1: event
epath=/local/domain/0/device-model/6/state
> libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch
w=0x854690 wpath=/local/domain/0/device-model/6/state token=3/1: deregister
slotnum=3
> libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch
w=0x854690: deregister unregistered
> libxl: debug: libxl_qmp.c:707:libxl__qmp_initialize: connected to
/var/run/xen/qmp-libxl-6
> libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: qmp
> libxl: debug: libxl_qmp.c:555:qmp_send_prepare: next qmp command:
''{
>     "execute": "qmp_capabilities",
>     "id": 1
> }
> ''
> libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
> libxl: debug: libxl_qmp.c:555:qmp_send_prepare: next qmp command:
''{
>     "execute": "query-chardev",
>     "id": 2
> }
> ''
> libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
> libxl: debug: libxl_qmp.c:555:qmp_send_prepare: next qmp command:
''{
>     "execute": "query-vnc",
>     "id": 3
> }
> ''
> libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
> libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch
w=0x85ad98 wpath=/local/domain/0/backend/vif/6/0/state token=3/2: register
slotnum=3
> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x85ad98
wpath=/local/domain/0/backend/vif/6/0/state token=3/2: event
epath=/local/domain/0/backend/vif/6/0/state
> libxl: debug: libxl_event.c:647:devstate_watch_callback: backend
/local/domain/0/backend/vif/6/0/state wanted state 2 still waiting state 1
> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x85ad98
wpath=/local/domain/0/backend/vif/6/0/state token=3/2: event
epath=/local/domain/0/backend/vif/6/0/state
> libxl: debug: libxl_event.c:643:devstate_watch_callback: backend
/local/domain/0/backend/vif/6/0/state wanted state 2 ok
> libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch
w=0x85ad98 wpath=/local/domain/0/backend/vif/6/0/state token=3/2: deregister
slotnum=3
> libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch
w=0x85ad98: deregister unregistered
> libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script:
/etc/xen/scripts/vif-bridge online
> libxl: debug: libxl_event.c:1748:libxl__ao_progress_report: ao 0x8540d0:
progress report: callback queued aop=0x859680
> libxl: debug: libxl_event.c:1570:libxl__ao_complete: ao 0x8540d0: complete,
rc=0
> libxl: debug: libxl_event.c:1160:egc_run_callbacks: ao 0x8540d0: progress
report: callback aop=0x859680
> libxl: debug: libxl_event.c:1542:libxl__ao__destroy: ao 0x8540d0: destroy
> Daemon running with PID 6784
> xenconsole: Could not read tty from store: No such file or directory
> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: console child
[0] exited with error status 2
> xc: debug: hypercall buffer: total allocations:241 total releases:241
> xc: debug: hypercall buffer: current allocations:0 maximum allocations:4
> xc: debug: hypercall buffer: cache current size:4
> xc: debug: hypercall buffer: cache hits:230 misses:4 toobig:7

On Mon, Oct 14, 2013 at 04:57:48PM +0200, Fabio Fantoni
wrote:
> Il 14/10/2013 16:29, Wei Liu ha scritto:
> >On Mon, Oct 14, 2013 at 04:21:52PM +0200, Fabio Fantoni wrote:
> >>Il 14/10/2013 16:02, Wei Liu ha scritto:
> >>>On Tue, Oct 08, 2013 at 04:47:17PM +0200, Fabio Fantoni wrote:
> >>>>I''m unable to start xl console on latest
xen-unstable (build with
> >>>>commit 8e0da8c07f4f80e14314977a11f738bd74a5b62b).
> >>>>>xl -vvv console sid
> >>>>>xenconsole: Could not read tty from store: No such file
or directory
> >>>It would be good if you can paste the output here.
> >>Thanks for reply.
> >>The "xenconsole: ..." line is the only output of "xl
-vvv console
> >>sid" command, before that it shows nothing for some seconds,
then
> >>return to command line.
> >>
> >>>>I also tried "xl create -c", and it shows console
from pygrub but
> >>>>when the "xl create" reaches the end I got the
same error:
> >>>>>xenconsole: Could not read tty from store: No such file
or directory
> >>>Please run xl -vvv create -c, paste output here.
> >>See on attachment.
> >>
> >Could you also attach the output of "xenstore-ls" when your
DomU is up
> >and running.
> 
> See attachment.
> 
> Thanks for any reply.
> 
That''s weird. Xenstore-ls log''s tty= field is empty while
there''s no
complain about writing console node in xl -vvv create log.

Also I don''t see any significant change in xenconsole code. I also
tested latest tip for this everything worked. Probably you had some
stale libraries somewhere? Could you use ldd to confirm that the
libraries are the latest version you installed?

Wei.

Il 14/10/2013 17:26, Wei Liu ha scritto:
> On Mon, Oct 14, 2013 at 04:57:48PM +0200, Fabio Fantoni wrote:
>> Il 14/10/2013 16:29, Wei Liu ha scritto:
>>> On Mon, Oct 14, 2013 at 04:21:52PM +0200, Fabio Fantoni wrote:
>>>> Il 14/10/2013 16:02, Wei Liu ha scritto:
>>>>> On Tue, Oct 08, 2013 at 04:47:17PM +0200, Fabio Fantoni
wrote:
>>>>>> I''m unable to start xl console on latest
xen-unstable (build with
>>>>>> commit 8e0da8c07f4f80e14314977a11f738bd74a5b62b).
>>>>>>> xl -vvv console sid
>>>>>>> xenconsole: Could not read tty from store: No such
file or directory
>>>>> It would be good if you can paste the output here.
>>>> Thanks for reply.
>>>> The "xenconsole: ..." line is the only output of
"xl -vvv console
>>>> sid" command, before that it shows nothing for some
seconds, then
>>>> return to command line.
>>>>
>>>>>> I also tried "xl create -c", and it shows
console from pygrub but
>>>>>> when the "xl create" reaches the end I got
the same error:
>>>>>>> xenconsole: Could not read tty from store: No such
file or directory
>>>>> Please run xl -vvv create -c, paste output here.
>>>> See on attachment.
>>>>
>>> Could you also attach the output of "xenstore-ls" when
your DomU is up
>>> and running.
>> See attachment.
>>
>> Thanks for any reply.
>>
> That''s weird. Xenstore-ls log''s tty= field is empty while
there''s no
> complain about writing console node in xl -vvv create log.
>
> Also I don''t see any significant change in xenconsole code. I also
> tested latest tip for this everything worked. Probably you had some
> stale libraries somewhere? Could you use ldd to confirm that the
> libraries are the latest version you installed?
>
> Wei.
Thanks for reply.
I checked and seems that all xen libraries is installed and to latest 
version.
The only library missed is linux-vdso that I not found on debian 7 
repository.
Below there are the ldd command and ls of libxen* on my last xen 
compilation.

ldd /usr/sbin/xl
         linux-vdso.so.1 =>  (0x00007fff87bff000)
         libxlutil.so.4.3 => /usr/lib/libxlutil.so.4.3 (0x00007fa4fb195000)
         libxenlight.so.4.3 => /usr/lib/libxenlight.so.4.3 
(0x00007fa4faf35000)
         libxenctrl.so.4.3 => /usr/lib/libxenctrl.so.4.3 
(0x00007fa4fad10000)
         libyajl.so.2 => /usr/lib/x86_64-linux-gnu/libyajl.so.2 
(0x00007fa4fab06000)
         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x00007fa4fa8ea000)
         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa4fa55f000)
         libxenguest.so.4.3 => /usr/lib/libxenguest.so.4.3 
(0x00007fa4fa335000)
         libxenstore.so.3.0 => /usr/lib/libxenstore.so.3.0 
(0x00007fa4fa12c000)
         libblktapctl.so.1.0 => /usr/lib/libblktapctl.so.1.0 
(0x00007fa4f9f24000)
         libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 
(0x00007fa4f9d21000)
         libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 
(0x00007fa4f9b1c000)
         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa4f9917000)
         /lib64/ld-linux-x86-64.so.2 (0x00007fa4fb3b2000)
         libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 
(0x00007fa4f9707000)
         liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 
(0x00007fa4f94e4000)
         libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fa4f92cc000)

ls -alh /mnt/vm/xen/Xen/dist/install/usr/lib/libxen*
-rw-r--r-- 1 root root 1,6M ott 14 10:47 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenctrl.a
lrwxrwxrwx 1 root root   17 ott 14 12:06 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenctrl.so -> libxenctrl.so.4.3
lrwxrwxrwx 1 root root   19 ott 14 12:06 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenctrl.so.4.3 -> 
libxenctrl.so.4.3.0
-rwxr-xr-x 1 root root 822K ott 14 10:47 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenctrl.so.4.3.0
-rw-r--r-- 1 root root 1,3M ott 14 10:47 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenguest.a
lrwxrwxrwx 1 root root   18 ott 14 12:06 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenguest.so -> libxenguest.so.4.3
lrwxrwxrwx 1 root root   20 ott 14 12:06 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenguest.so.4.3 -> 
libxenguest.so.4.3.0
-rwxr-xr-x 1 root root 717K ott 14 10:47 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenguest.so.4.3.0
-rw-r--r-- 1 root root 3,2M ott 14 10:50 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenlight.a
lrwxrwxrwx 1 root root   18 ott 14 12:10 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenlight.so -> libxenlight.so.4.3
lrwxrwxrwx 1 root root   20 ott 14 12:10 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenlight.so.4.3 -> 
libxenlight.so.4.3.0
-rwxr-xr-x 1 root root 1,7M ott 14 10:50 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenlight.so.4.3.0
-rw-r--r-- 1 root root 110K ott 14 10:49 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenstat.a
lrwxrwxrwx 1 root root   15 ott 14 12:06 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenstat.so -> libxenstat.so.0
lrwxrwxrwx 1 root root   17 ott 14 12:06 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenstat.so.0 -> libxenstat.so.0.0
-rwxr-xr-x 1 root root  77K ott 14 10:49 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenstat.so.0.0
-rw-r--r-- 1 root root 107K ott 14 10:47 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenstore.a
lrwxrwxrwx 1 root root   18 ott 14 12:06 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenstore.so -> libxenstore.so.3.0
lrwxrwxrwx 1 root root   20 ott 14 12:06 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenstore.so.3.0 -> 
libxenstore.so.3.0.3
-rwxr-xr-x 1 root root  89K ott 14 10:47 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenstore.so.3.0.3
-rw-r--r-- 1 root root  71K ott 14 10:49 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenvchan.a
lrwxrwxrwx 1 root root   18 ott 14 12:06 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenvchan.so -> libxenvchan.so.1.0
lrwxrwxrwx 1 root root   20 ott 14 12:06 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenvchan.so.1.0 -> 
libxenvchan.so.1.0.0
-rwxr-xr-x 1 root root  50K ott 14 10:49 
/mnt/vm/xen/Xen/dist/install/usr/lib/libxenvchan.so.1.0.0

On Mon, 2013-10-14 at 16:26 +0100, Wei Liu wrote:
> On Mon, Oct 14, 2013 at 04:57:48PM +0200, Fabio Fantoni wrote:
> > Il 14/10/2013 16:29, Wei Liu ha scritto:
> > >On Mon, Oct 14, 2013 at 04:21:52PM +0200, Fabio Fantoni wrote:
> > >>Il 14/10/2013 16:02, Wei Liu ha scritto:
> > >>>On Tue, Oct 08, 2013 at 04:47:17PM +0200, Fabio Fantoni
wrote:
> > >>>>I''m unable to start xl console on latest
xen-unstable (build with
> > >>>>commit 8e0da8c07f4f80e14314977a11f738bd74a5b62b).
> > >>>>>xl -vvv console sid
> > >>>>>xenconsole: Could not read tty from store: No such
file or directory
> > >>>It would be good if you can paste the output here.
> > >>Thanks for reply.
> > >>The "xenconsole: ..." line is the only output of
"xl -vvv console
> > >>sid" command, before that it shows nothing for some
seconds, then
> > >>return to command line.
> > >>
> > >>>>I also tried "xl create -c", and it shows
console from pygrub but
> > >>>>when the "xl create" reaches the end I got
the same error:
> > >>>>>xenconsole: Could not read tty from store: No such
file or directory
> > >>>Please run xl -vvv create -c, paste output here.
> > >>See on attachment.
> > >>
> > >Could you also attach the output of "xenstore-ls" when
your DomU is up
> > >and running.
> > 
> > See attachment.
> > 
> > Thanks for any reply.
> > 
> 
> That''s weird. Xenstore-ls log''s tty= field is empty while
there''s no
> complain about writing console node in xl -vvv create log.
xenconsoled writes that node IIRC.

It''s possible that this is a kernel side bug? In that the console is
not
created because of something going on on the guest side.

earlyprintk=xen on the guest console + a debug version of Xen +
loglvl_guest=all should cause the guest to print to the xen console,
which might give a clue.

It might also be useful to enable guest console logging and see if
anything comes out there
http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen#Guest_console_logs

Lastly, does "xl console <guest>" work once the guest has been
created?
> Also I don''t see any significant change in xenconsole code. I also
> tested latest tip for this everything worked. Probably you had some
> stale libraries somewhere? Could you use ldd to confirm that the
> libraries are the latest version you installed?
> 
> Wei.

On Tue, Oct 15, 2013 at 10:16:09AM +0100, Ian Campbell
wrote:
> On Mon, 2013-10-14 at 16:26 +0100, Wei Liu wrote:
> > On Mon, Oct 14, 2013 at 04:57:48PM +0200, Fabio Fantoni wrote:
> > > Il 14/10/2013 16:29, Wei Liu ha scritto:
> > > >On Mon, Oct 14, 2013 at 04:21:52PM +0200, Fabio Fantoni
wrote:
> > > >>Il 14/10/2013 16:02, Wei Liu ha scritto:
> > > >>>On Tue, Oct 08, 2013 at 04:47:17PM +0200, Fabio
Fantoni wrote:
> > > >>>>I''m unable to start xl console on latest
xen-unstable (build with
> > > >>>>commit 8e0da8c07f4f80e14314977a11f738bd74a5b62b).
> > > >>>>>xl -vvv console sid
> > > >>>>>xenconsole: Could not read tty from store: No
such file or directory
> > > >>>It would be good if you can paste the output here.
> > > >>Thanks for reply.
> > > >>The "xenconsole: ..." line is the only output
of "xl -vvv console
> > > >>sid" command, before that it shows nothing for some
seconds, then
> > > >>return to command line.
> > > >>
> > > >>>>I also tried "xl create -c", and it
shows console from pygrub but
> > > >>>>when the "xl create" reaches the end I
got the same error:
> > > >>>>>xenconsole: Could not read tty from store: No
such file or directory
> > > >>>Please run xl -vvv create -c, paste output here.
> > > >>See on attachment.
> > > >>
> > > >Could you also attach the output of "xenstore-ls"
when your DomU is up
> > > >and running.
> > > 
> > > See attachment.
> > > 
> > > Thanks for any reply.
> > > 
> > 
> > That''s weird. Xenstore-ls log''s tty= field is empty
while there''s no
> > complain about writing console node in xl -vvv create log.
> 
> xenconsoled writes that node IIRC.
> 
Right, then there should be some info in xenconsoled''s log. I
wasn''t
paying attention to that part because there doesn''t seem to be
significant change.
> It''s possible that this is a kernel side bug? In that the console
is not
> created because of something going on on the guest side.
> 
Fabio, can you check the log of xenconsoled? Is it running normally?

It''s a bit hard for me to imagine a guest kernel side bug causing
xenconsoled to misbehave, but that''s also possible. If xenconsoled is
OK
please follow Ian''s suggestion to provide more info.

Wei.

Il 15/10/2013 11:45, Wei Liu ha scritto:
> On Tue, Oct 15, 2013 at 10:16:09AM +0100, Ian Campbell wrote:
>> On Mon, 2013-10-14 at 16:26 +0100, Wei Liu wrote:
>>> On Mon, Oct 14, 2013 at 04:57:48PM +0200, Fabio Fantoni wrote:
>>>> Il 14/10/2013 16:29, Wei Liu ha scritto:
>>>>> On Mon, Oct 14, 2013 at 04:21:52PM +0200, Fabio Fantoni
wrote:
>>>>>> Il 14/10/2013 16:02, Wei Liu ha scritto:
>>>>>>> On Tue, Oct 08, 2013 at 04:47:17PM +0200, Fabio
Fantoni wrote:
>>>>>>>> I''m unable to start xl console on
latest xen-unstable (build with
>>>>>>>> commit
8e0da8c07f4f80e14314977a11f738bd74a5b62b).
>>>>>>>>> xl -vvv console sid
>>>>>>>>> xenconsole: Could not read tty from store:
No such file or directory
>>>>>>> It would be good if you can paste the output here.
>>>>>> Thanks for reply.
>>>>>> The "xenconsole: ..." line is the only output
of "xl -vvv console
>>>>>> sid" command, before that it shows nothing for
some seconds, then
>>>>>> return to command line.
>>>>>>
>>>>>>>> I also tried "xl create -c", and it
shows console from pygrub but
>>>>>>>> when the "xl create" reaches the end
I got the same error:
>>>>>>>>> xenconsole: Could not read tty from store:
No such file or directory
>>>>>>> Please run xl -vvv create -c, paste output here.
>>>>>> See on attachment.
>>>>>>
>>>>> Could you also attach the output of "xenstore-ls"
when your DomU is up
>>>>> and running.
>>>> See attachment.
>>>>
>>>> Thanks for any reply.
>>>>
>>> That''s weird. Xenstore-ls log''s tty= field is
empty while there''s no
>>> complain about writing console node in xl -vvv create log.
>> xenconsoled writes that node IIRC.
>>
> Right, then there should be some info in xenconsoled''s log. I
wasn''t
> paying attention to that part because there doesn''t seem to be
> significant change.
>
>> It''s possible that this is a kernel side bug? In that the
console is not
>> created because of something going on on the guest side.
>>
> Fabio, can you check the log of xenconsoled? Is it running normally?
>
> It''s a bit hard for me to imagine a guest kernel side bug causing
> xenconsoled to misbehave, but that''s also possible. If xenconsoled
is OK
> please follow Ian''s suggestion to provide more info.
>
> Wei.
I not found xenconsoled log, it seems not running: "ps ax | grep 
xenconsoled" returns nothing and the /var/run/xenconsoled.pid has stored 
pid of died process.
I tried to restart xenconsoled manually and after that xl console 
returns working.
I''ll do a completly fresh system on the next days and I will let you
know.

Il 15/10/2013 16:18, Fabio Fantoni ha scritto:
> Il 15/10/2013 11:45, Wei Liu ha scritto:
>> On Tue, Oct 15, 2013 at 10:16:09AM +0100, Ian Campbell wrote:
>>> On Mon, 2013-10-14 at 16:26 +0100, Wei Liu wrote:
>>>> On Mon, Oct 14, 2013 at 04:57:48PM +0200, Fabio Fantoni wrote:
>>>>> Il 14/10/2013 16:29, Wei Liu ha scritto:
>>>>>> On Mon, Oct 14, 2013 at 04:21:52PM +0200, Fabio Fantoni
wrote:
>>>>>>> Il 14/10/2013 16:02, Wei Liu ha scritto:
>>>>>>>> On Tue, Oct 08, 2013 at 04:47:17PM +0200, Fabio
Fantoni wrote:
>>>>>>>>> I''m unable to start xl console on
latest xen-unstable (build with
>>>>>>>>> commit
8e0da8c07f4f80e14314977a11f738bd74a5b62b).
>>>>>>>>>> xl -vvv console sid
>>>>>>>>>> xenconsole: Could not read tty from
store: No such file or
>>>>>>>>>> directory
>>>>>>>> It would be good if you can paste the output
here.
>>>>>>> Thanks for reply.
>>>>>>> The "xenconsole: ..." line is the only
output of "xl -vvv console
>>>>>>> sid" command, before that it shows nothing for
some seconds, then
>>>>>>> return to command line.
>>>>>>>
>>>>>>>>> I also tried "xl create -c", and
it shows console from pygrub but
>>>>>>>>> when the "xl create" reaches the
end I got the same error:
>>>>>>>>>> xenconsole: Could not read tty from
store: No such file or
>>>>>>>>>> directory
>>>>>>>> Please run xl -vvv create -c, paste output
here.
>>>>>>> See on attachment.
>>>>>>>
>>>>>> Could you also attach the output of
"xenstore-ls" when your DomU
>>>>>> is up
>>>>>> and running.
>>>>> See attachment.
>>>>>
>>>>> Thanks for any reply.
>>>>>
>>>> That''s weird. Xenstore-ls log''s tty= field is
empty while there''s no
>>>> complain about writing console node in xl -vvv create log.
>>> xenconsoled writes that node IIRC.
>>>
>> Right, then there should be some info in xenconsoled''s log. I
wasn''t
>> paying attention to that part because there doesn''t seem to be
>> significant change.
>>
>>> It''s possible that this is a kernel side bug? In that the
console is
>>> not
>>> created because of something going on on the guest side.
>>>
>> Fabio, can you check the log of xenconsoled? Is it running normally?
>>
>> It''s a bit hard for me to imagine a guest kernel side bug
causing
>> xenconsoled to misbehave, but that''s also possible. If
xenconsoled is OK
>> please follow Ian''s suggestion to provide more info.
>>
>> Wei.
>
> I not found xenconsoled log, it seems not running: "ps ax | grep 
> xenconsoled" returns nothing and the /var/run/xenconsoled.pid has 
> stored pid of died process.
> I tried to restart xenconsoled manually and after that xl console 
> returns working.
> I''ll do a completly fresh system on the next days and I will let
you
> know.
I did a full clean test system and for now xenconsoled is not crashed.
I will keep xl console tested more often to find the cause if re-happen.