xen discuss - Sep 2007 - xenstored won''t start - "process dumped core" message in logfile

Hi,

Am trying to start Hypervisor Services via svcadm as per the Sys Admin Guide,
but continually getting repeated:

[ Sep 17 17:14:58 Executing start method ("/lib/svc/method/xenstored
start") ]
[ Sep 17 17:14:58 Method "start" exited with status 0 ]
[ Sep 17 17:15:02 Stopping because process dumped core. ]

entries in the /var/svc/log/system-xctl-store:default.log after executing the
command:

bash-3.00#svcadm enable xctl/store

and as such cannot start xctl/xend or xctl/console either.  Output from uname
is:

bash-3.00# uname -i
i86xpv

any ideas why I cannot get xenstored to start?
 
 
This message posted from opensolaris.org

On Mon, Sep 17, 2007 at 01:02:04AM -0700, Gareth wrote:
> Am trying to start Hypervisor Services via svcadm as per the Sys Admin
Guide, but continually getting repeated:
> 
> [ Sep 17 17:14:58 Executing start method ("/lib/svc/method/xenstored
start") ]
> [ Sep 17 17:14:58 Method "start" exited with status 0 ]
> [ Sep 17 17:15:02 Stopping because process dumped core. ]
> 
> entries in the /var/svc/log/system-xctl-store:default.log after executing
the command:
We recently fixed some nasty corruption issues from upstream. It might be due
to that. Can you do a ''pstaack'' or xenstored''s core
file?

cheers
john

Hi John,

Thanks for your reply.  The core files generated are quite large, so
I''ve posted a pstack of the start and the end of one of them with the
similar bits in the middle skipped:

bash-3.00# pstack core.xenstored.1815 | less
core ''core.xenstored.1815'' of 1815:     /usr/lib/xenstored
--pid-file=/var/run/xenstore.pid
 cf2ed450 s_fcntl  (8, 23, 7800070) + 9
 cf300910 fcntl    (8, 23, 7800070, 23, 78000a0, 23) + b0
 0805aad3 tdb_brlock (1, 23, 0, cf35c468, 1078, 1079) + 73
 0805ac72 tdb_lock (1, cf3567c8, 0, cf355000, 10b8, 2f) + a2
 0805c4e8 tdb_find_lock_hash (8b8ef10, 1, 308f2203, 1, 7800150, 2f) + 28
 0805c5d0 tdb_fetch (78001b0, 8072118, 8b8ef10, 1, 8b8ef10, 8b8ef10) + 50
 08054cbd read_node (0, 0, 78001f8, 0, 0, 8b8eed0) + 3d
 080552c2 construct_node (0, 0, 7800228, 0, 0, 8b8ee90) + 22
 0805541e construct_node (0, 0, 7800258, 0, 0, 8b8ee50) + 17e
 0805541e construct_node (0, 0, 7800288, 0, 0, 8b8ee10) + 17e
 0805541e construct_node (0, 0, 78002b8, 0, 0, 8b8edd0) + 17e
 0805541e construct_node (0, 0, 78002e8, 0, 0, 8b8ed90) + 17e
 0805541e construct_node (0, 0, 7800318, 0, 0, 8b8ed50) + 17e
 0805541e construct_node (0, 0, 7800348, 0, 0, 8b8ed10) + 17e
 0805541e construct_node (0, 0, 7800378, 0, 0, 8b8ecd0) + 17e
 0805541e construct_node (0, 0, 78003a8, 0, 0, 8b8ec90) + 17e
 0805541e construct_node (0, 0, 78003d8, 0, 0, 8b8ec50) + 17e
 0805541e construct_node (0, 0, 7800408, 0, 0, 8b8ec10) + 17e
 0805541e construct_node (0, 0, 7800438, 0, 0, 8b8ebd0) + 17e
 0805541e construct_node (0, 0, 7800468, 0, 0, 8b8eb90) + 17e
 0805541e construct_node (0, 0, 7800498, 0, 0, 8b8eb50) + 17e
 0805541e construct_node (0, 0, 78004c8, 0, 0, 8b8eb10) + 17e
 0805541e construct_node (0, 0, 78004f8, 0, 0, 8b8ead0) + 17e
 0805541e construct_node (0, 0, 7800528, 0, 0, 8b8ea90) + 17e
 0805541e construct_node (0, 0, 7800558, 0, 0, 8b8ea50) + 17e
 0805541e construct_node (0, 0, 7800588, 0, 0, 8b8ea10) + 17e
 0805541e construct_node (0, 0, 78005b8, 0, 0, 8b8e9d0) + 17e
 0805541e construct_node (0, 0, 78005e8, 0, 0, 8b8e990) + 17e
 0805541e construct_node (0, 0, 7800618, 0, 0, 8b8e950) + 17e
 0805541e construct_node (0, 0, 7800648, 0, 0, 8b8e910) + 17e
 0805541e construct_node (0, 0, 7800678, 0, 0, 8b8e8d0) + 17e
 0805541e construct_node (0, 0, 78006a8, 0, 0, 8b8e890) + 17e
 0805541e construct_node (0, 0, 78006d8, 0, 0, 8b8e850) + 17e
 0805541e construct_node (0, 0, 7800708, 0, 0, 8b8e810) + 17e
 0805541e construct_node (0, 0, 7800738, 0, 0, 8b8e7d0) + 17e
 0805541e construct_node (0, 0, 7800768, 0, 0, 8b8e790) + 17e
 0805541e construct_node (0, 0, 7800798, 0, 0, 8b8e750) + 17e
 0805541e construct_node (0, 0, 78007c8, 0, 0, 8b8e710) + 17e
 0805541e construct_node (0, 0, 78007f8, 0, 0, 8b8e6d0) + 17e
 0805541e construct_node (0, 0, 7800828, 0, 0, 8b8e690) + 17e
 0805541e construct_node (0, 0, 7800858, 0, 0, 8b8e650) + 17e
 0805541e construct_node (0, 0, 7800888, 0, 0, 8b8e610) + 17e
 0805541e construct_node (0, 0, 78008b8, 0, 0, 8b8e5d0) + 17e
...skipping...
 0805541e construct_node (0, 0, 80473c8, 0, 0, 8085708) + 17e
 0805541e construct_node (0, 0, 80473f8, 0, 0, 80856c8) + 17e
 0805541e construct_node (0, 0, 8047428, 0, 0, 8085688) + 17e
 0805541e construct_node (0, 0, 8047458, 0, 0, 8085648) + 17e
 0805541e construct_node (0, 0, 8047488, 0, 0, 8085608) + 17e
 0805541e construct_node (0, 0, 80474b8, 0, 0, 80855c8) + 17e
 0805541e construct_node (0, 0, 80474e8, 0, 0, 8085588) + 17e
 0805541e construct_node (0, 0, 8047518, 0, 0, 8085548) + 17e
 0805541e construct_node (0, 0, 8047548, 0, 0, 8085508) + 17e
 0805541e construct_node (0, 0, 8047578, 0, 0, 80854c8) + 17e
 0805541e construct_node (0, 0, 80475a8, 0, 0, 8085488) + 17e
 0805541e construct_node (0, 0, 80475d8, 0, 0, 8085448) + 17e
 0805541e construct_node (0, 0, 8047608, 0, 0, 8085408) + 17e
 0805541e construct_node (0, 0, 8047638, 0, 0, 80853c8) + 17e
 0805541e construct_node (0, 0, 8047668, 0, 0, 8085388) + 17e
 0805541e construct_node (0, 0, 8047698, 0, 0, 8085348) + 17e
 0805541e construct_node (0, 0, 80476c8, 0, 0, 8085308) + 17e
 0805541e construct_node (0, 0, 80476f8, 0, 0, 80852c8) + 17e
 0805541e construct_node (0, 0, 8047728, 0, 0, 8085288) + 17e
 0805541e construct_node (0, 0, 8047758, 0, 0, 8085248) + 17e
 0805541e construct_node (0, 0, 8047788, 0, 0, 8085208) + 17e
 0805541e construct_node (0, 0, 80477b8, 0, 0, 80851c8) + 17e
 0805541e construct_node (0, 0, 80477e8, 0, 0, 8085188) + 17e
 0805541e construct_node (0, 0, 8047818, 0, 0, 8085148) + 17e
 0805541e construct_node (0, 0, 8047848, 0, 0, 8085108) + 17e
 0805541e construct_node (0, 0, 8047878, 0, 0, 80850c8) + 17e
 0805541e construct_node (0, 0, 80478a8, 0, 0, 8085088) + 17e
 0805541e construct_node (0, 0, 80478d8, 0, 0, 8085048) + 17e
 0805541e construct_node (0, 0, 8047908, 0, 0, 8085008) + 17e
 0805541e construct_node (0, 0, 8047938, 0, 0, 8084fc8) + 17e
 0805541e construct_node (0, 0, 8047968, 0, 0, 8084f88) + 17e
 0805541e construct_node (0, 0, 8047998, 0, 0, 8084f48) + 17e
 0805541e construct_node (0, 0, 80479c8, 0, 0, 8084f08) + 17e
 0805541e construct_node (0, 0, 80479f8, 0, 0, 8084ec8) + 17e
 0805541e construct_node (0, 0, 8047a28, 0, 0, 8082590) + 17e
 0805541e construct_node (0, 0, 8047a58, 0, 0, 8082550) + 17e
 0805541e construct_node (0, 0, 8047a88, 0, 0, 8082510) + 17e
 0805541e construct_node (0, 0, 8047ab8, 0, 0, 80824d0) + 17e
 0805541e construct_node (0, 0, 8047ae8, 0, 0, 8081a88) + 17e
 0805541e construct_node (8072118, 8072118, cf153640, 8047b18, cf2b9603,
8081a48) + 17e
 0805541e construct_node (8081a58, 0, 7, 0, 8081a88, 8072088) + 17e
 08055da0 create_node (0, 0, 0, 2, 0, cf3c5968) + 10
 080576b9 main     (2, 8047dac, 8047db8) + a39
 08054220 _start   (2, 8047e54, 8047e67, 0, 8047e88, 8047ea3) + 80
 
 
This message posted from opensolaris.org

On Tue, Sep 18, 2007 at 12:35:08AM -0700, Gareth wrote:
> Thanks for your reply.  The core files generated are quite large, so
I''ve posted a pstack of the start and the end of one of them with the
similar bits in the middle skipped:
Interesting, looks like you have a stack overflow. I suspect strongly that you
have a corrupt tdb as mentioned. Try:

svcadm disable -t xctl/store
rm /var/lib/xenstored/tdb (I think - check this. the directory should be empty)
reboot

And see if the problem persists

regards
john