Hello Everyone,
We have noticed that one of our servers xen Dom0 is restarting
idiopathically. Atleast, we are not able to analyze the cause. I have
checked the logs but couldn''t find anything relating to it.
Are there any tools such as kdump (which doesn''t seem to work on
xenified
kernels) for catching the kernel crash snapshot.
Regards,
Sree Harsha Totakura
Member, Open Source Platform - TEG
Tata Consultancy Services
Ph:- +91-22-67782038
Mailto: sreeharsha.totakura@tcs.com
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services
Business Solutions
Outsourcing
____________________________________________
=====-----=====-----====Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
On Fri, Jul 17, 2009 at 3:07 PM, Sreeharsha Totakura<sreeharsha.totakura@tcs.com> wrote:> > Hello Everyone, > > We have noticed that one of our servers xen Dom0 is restarting > idiopathically. Atleast, we are not able to analyze the cause. I have > checked the logs but couldn''t find anything relating to it.what xen version, Can crash be reproduced????> > Are there any tools such as kdump (which doesn''t seem to work on xenified > kernels) for catching the kernel crash snapshot.netconsole, collects log.> > Regards, > Sree Harsha Totakura > Member, Open Source Platform - TEG > Tata Consultancy Services > Ph:- +91-22-67782038 > Mailto: sreeharsha.totakura@tcs.com > Website: http://www.tcs.com > ____________________________________________ > Experience certainty. IT Services > Business Solutions > Outsourcing > ____________________________________________ > > =====-----=====-----====> Notice: The information contained in this e-mail > message and/or attachments to it may contain > confidential or privileged information. If you are > not the intended recipient, any dissemination, use, > review, distribution, printing or copying of the > information contained in this e-mail message > and/or attachments to it are strictly prohibited. If > you have received this communication in error, > please notify us by reply e-mail or telephone and > immediately and permanently delete the message > and any attachments. Thank you > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sreeharsha Totakura
2009-Jul-17 11:12 UTC
Re: [Xen-users] Tools to take a kernel crash dump.
Hi Tej,
The restart (crash) cannot be reproduced but its intermittent. It happens
idiopathic, we cannot predict when it will next restart (crash). We have
been observing this for a month. It ran perfectly last week and it
restarted (crashed) today. Within minutes of restart it again took a
restart. Now it appears to be running fine, the next restart can be in
minutes, days or weeks.
The Dom0 is running on PV Xen 3.1 kernel.
I observed the system logs and there was nothing that hinted a cause for
the system restart.
If I should be looking at any other log, please let me know.
Regards,
Sree Harsha Totakura
Member, Open Source Platform - TEG
Tata Consultancy Services
Ph:- +91-22-67782038
Mailto: sreeharsha.totakura@tcs.com
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services
Business Solutions
Outsourcing
____________________________________________
Tej
<bewith.tej@gmail
.com> To
Sreeharsha Totakura
07/17/2009 03:27 <sreeharsha.totakura@tcs.com>
PM cc
Xen User-List
<xen-users@lists.xensource.com>
Subject
Re: [Xen-users] Tools to take a
kernel crash dump.
On Fri, Jul 17, 2009 at 3:07 PM, Sreeharsha
Totakura<sreeharsha.totakura@tcs.com> wrote:>
> Hello Everyone,
>
> We have noticed that one of our servers xen Dom0 is restarting
> idiopathically. Atleast, we are not able to analyze the cause. I have
> checked the logs but couldn''t find anything relating to it.
what xen version, Can crash be reproduced????
>
> Are there any tools such as kdump (which doesn''t seem to work on
xenified
> kernels) for catching the kernel crash snapshot.
netconsole, collects log.
>
> Regards,
> Sree Harsha Totakura
> Member, Open Source Platform - TEG
> Tata Consultancy Services
> Ph:- +91-22-67782038
> Mailto: sreeharsha.totakura@tcs.com
> Website: http://www.tcs.com
> ____________________________________________
> Experience certainty. IT Services
> Business Solutions
> Outsourcing
> ____________________________________________
>
> =====-----=====-----====> Notice: The information contained in this
e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
ForwardSourceID:NT0000710A
=====-----=====-----====Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
On Fri, Jul 17, 2009 at 4:42 PM, Sreeharsha Totakura<sreeharsha.totakura@tcs.com> wrote:> Hi Tej, > > The restart (crash) cannot be reproduced but its intermittent. It happens > idiopathic, we cannot predict when it will next restart (crash). We have > been observing this for a month. It ran perfectly last week and it > restarted (crashed) today. Within minutes of restart it again took a > restart. Now it appears to be running fine, the next restart can be in > minutes, days or weeks. > > The Dom0 is running on PV Xen 3.1 kernel. > > I observed the system logs and there was nothing that hinted a cause for > the system restart. > If I should be looking at any other log, please let me know.Collect the logs when system restart/panic using netconsole. Can you try some newer xen version like 3.4.. HTH> > Regards, > Sree Harsha Totakura > Member, Open Source Platform - TEG > Tata Consultancy Services > Ph:- +91-22-67782038 > Mailto: sreeharsha.totakura@tcs.com > Website: http://www.tcs.com > ____________________________________________ > Experience certainty. IT Services > Business Solutions > Outsourcing > ____________________________________________ > > > > Tej > <bewith.tej@gmail > .com> To > Sreeharsha Totakura > 07/17/2009 03:27 <sreeharsha.totakura@tcs.com> > PM cc > Xen User-List > <xen-users@lists.xensource.com> > Subject > Re: [Xen-users] Tools to take a > kernel crash dump. > > > > > > > > > > > On Fri, Jul 17, 2009 at 3:07 PM, Sreeharsha > Totakura<sreeharsha.totakura@tcs.com> wrote: >> >> Hello Everyone, >> >> We have noticed that one of our servers xen Dom0 is restarting >> idiopathically. Atleast, we are not able to analyze the cause. I have >> checked the logs but couldn''t find anything relating to it. > > what xen version, Can crash be reproduced???? > >> >> Are there any tools such as kdump (which doesn''t seem to work on xenified >> kernels) for catching the kernel crash snapshot. > > netconsole, collects log. > >> >> Regards, >> Sree Harsha Totakura >> Member, Open Source Platform - TEG >> Tata Consultancy Services >> Ph:- +91-22-67782038 >> Mailto: sreeharsha.totakura@tcs.com >> Website: http://www.tcs.com >> ____________________________________________ >> Experience certainty. IT Services >> Business Solutions >> Outsourcing >> ____________________________________________ >> >> =====-----=====-----====>> Notice: The information contained in this e-mail >> message and/or attachments to it may contain >> confidential or privileged information. If you are >> not the intended recipient, any dissemination, use, >> review, distribution, printing or copying of the >> information contained in this e-mail message >> and/or attachments to it are strictly prohibited. If >> you have received this communication in error, >> please notify us by reply e-mail or telephone and >> immediately and permanently delete the message >> and any attachments. Thank you >> >> >> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> > > > ForwardSourceID:NT0000710A > > =====-----=====-----====> Notice: The information contained in this e-mail > message and/or attachments to it may contain > confidential or privileged information. If you are > not the intended recipient, any dissemination, use, > review, distribution, printing or copying of the > information contained in this e-mail message > and/or attachments to it are strictly prohibited. If > you have received this communication in error, > please notify us by reply e-mail or telephone and > immediately and permanently delete the message > and any attachments. Thank you > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sreeharsha Totakura <sreeharsha.totakura@tcs.com> writes:> We have noticed that one of our servers xen Dom0 is restarting > idiopathically. Atleast, we are not able to analyze the cause. I have > checked the logs but couldn''t find anything relating to it. > > Are there any tools such as kdump (which doesn''t seem to work on xenified > kernels) for catching the kernel crash snapshot.a console usually gets you most of the required information, and is easier to setup than a kernel dump. Personally, I run a serial console on all boxes, and log it. This has the added bonus that you can get into the system if you, for example, lock yourself out using the firewall. If, for whatever reason, you can''t use a serial console, another user suggested netconsole. (I haven''t gotten netconsole working on a xenified kernel, but that is not to say it can''t be done; I like serial consoles so I didn''t spend much time on netconsole) - the other downside of netconsole is that it''s read only; fine for looking at your panics, but it won''t help you in the aforementioned ''I changed my firewall and now I can''t get in'' case. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Luke S Crawford wrote:> If, for whatever reason, you can''t use a serial console, another user > suggested netconsole. (I haven''t gotten netconsole working on a xenified > kernel, but that is not to say it can''t be done; I like serial consoles > so I didn''t spend much time on netconsole) - the other downside of > netconsole is that it''s read only; fine for looking at your panics, but > it won''t help you in the aforementioned ''I changed my firewall and now I can''t > get in'' case. >I have had netconsole working with a debian host. I never did get it working in a domU though (I dont know if it would work the way networking is done). Netconsole isnt too hard to setup but there is one key point. You have to specify the mac address of the destination host. Now if your destination host is not on the same subnet as you you need to specify the mac of your gateway. http://www.cmdln.org/2009/01/21/remote-kernel-logging-with-netconsole-for-fun-and-profit/ _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sreeharsha Totakura
2009-Jul-18 10:25 UTC
Re: [Xen-users] Tools to take a kernel crash dump.
Thanks Guys,
I have installed netconsole of the server and waiting for the next restart
(crash).
A few things I noted while installing netconsole are:
1. The netconsole kernel module doesn''t work on virtual interfaces.
2. The xend service when started will alter the network interfaces. In our
setup, the real eth0 is changes as peth0 and the eth0 is made virtual. I
assume this is done to pacify complaining programs whose setting are set to
use eth0 by default.
3. For installing netconsole, use:
/sbin/modprobe netconsole
netconsole=6666@172.17.122.21/peth0,6666@172.17.122.174/00:1e:37:42:23:22
This setting is required because the xen created virtual interface
eth0 doesn''t support pooling. Only the real physical interface (peth0)
supports it.
Thanks again for your responses. Will post the kernel crash errors if we
observe any.
Regards,
Sree Harsha Totakura
Member, Open Source Platform - TEG
Tata Consultancy Services
Ph:- +91-22-67782038
Mailto: sreeharsha.totakura@tcs.com
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services
Business Solutions
Outsourcing
____________________________________________
Nick Anderson <nick@anders0n.net> wrote on 07/18/2009 12:47:41 AM:
> Nick Anderson <nick@anders0n.net>
> 07/18/2009 12:47 AM
>
> To
>
> Luke S Crawford <lsc@prgmr.com>
>
> cc
>
> Sreeharsha Totakura <sreeharsha.totakura@tcs.com>, Xen User-List
> <xen-users@lists.xensource.com>
>
> Subject
>
> Re: [Xen-users] Tools to take a kernel crash dump.
>
> Luke S Crawford wrote:
> > If, for whatever reason, you can''t use a serial console,
another user
> > suggested netconsole. (I haven''t gotten netconsole working
on a
xenified> > kernel, but that is not to say it can''t be done; I like
serial
consoles> > so I didn''t spend much time on netconsole) - the other
downside of
> > netconsole is that it''s read only; fine for looking at your
panics,
but> > it won''t help you in the aforementioned ''I changed
my firewall and
> now I can''t
> > get in'' case.
> >
> I have had netconsole working with a debian host. I never did get it
> working in a domU though (I dont know if it would work the way
> networking is done). Netconsole isnt too hard to setup but there is one
> key point. You have to specify the mac address of the destination host.
> Now if your destination host is not on the same subnet as you you need
> to specify the mac of your gateway.
>
> http://www.cmdln.org/2009/01/21/remote-kernel-logging-with-
> netconsole-for-fun-and-profit/
>
>
>
> ForwardSourceID:NT000071EA
=====-----=====-----====Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
On Sat, Jul 18, 2009 at 3:55 PM, Sreeharsha Totakura<sreeharsha.totakura@tcs.com> wrote:> Thanks Guys, > > I have installed netconsole of the server and waiting for the next restart > (crash). > > A few things I noted while installing netconsole are: > 1. The netconsole kernel module doesn''t work on virtual interfaces. > 2. The xend service when started will alter the network interfaces. In our > setup, the real eth0 is changes as peth0 and the eth0 is made virtual. I > assume this is done to pacify complaining programs whose setting are set to > use eth0 by default. > 3. For installing netconsole, use: > /sbin/modprobe netconsole > netconsole=6666@172.17.122.21/peth0,6666@172.17.122.174/00:1e:37:42:23:22 > This setting is required because the xen created virtual interface > eth0 doesn''t support pooling. Only the real physical interface (peth0) > supports it.It will be better to increase console log level also.> > Thanks again for your responses. Will post the kernel crash errors if we > observe any. > > Regards, > Sree Harsha Totakura > Member, Open Source Platform - TEG > Tata Consultancy Services > Ph:- +91-22-67782038 > Mailto: sreeharsha.totakura@tcs.com > Website: http://www.tcs.com > ____________________________________________ > Experience certainty. IT Services > Business Solutions > Outsourcing > ____________________________________________ > > Nick Anderson <nick@anders0n.net> wrote on 07/18/2009 12:47:41 AM: > >> Nick Anderson <nick@anders0n.net> >> 07/18/2009 12:47 AM >> >> To >> >> Luke S Crawford <lsc@prgmr.com> >> >> cc >> >> Sreeharsha Totakura <sreeharsha.totakura@tcs.com>, Xen User-List >> <xen-users@lists.xensource.com> >> >> Subject >> >> Re: [Xen-users] Tools to take a kernel crash dump. >> >> Luke S Crawford wrote: >> > If, for whatever reason, you can''t use a serial console, another user >> > suggested netconsole. (I haven''t gotten netconsole working on a > xenified >> > kernel, but that is not to say it can''t be done; I like serial > consoles >> > so I didn''t spend much time on netconsole) - the other downside of >> > netconsole is that it''s read only; fine for looking at your panics, > but >> > it won''t help you in the aforementioned ''I changed my firewall and >> now I can''t >> > get in'' case. >> > >> I have had netconsole working with a debian host. I never did get it >> working in a domU though (I dont know if it would work the way >> networking is done). Netconsole isnt too hard to setup but there is one >> key point. You have to specify the mac address of the destination host. >> Now if your destination host is not on the same subnet as you you need >> to specify the mac of your gateway. >> >> http://www.cmdln.org/2009/01/21/remote-kernel-logging-with- >> netconsole-for-fun-and-profit/ >> >> >> > >> ForwardSourceID:NT000071EA > > =====-----=====-----====> Notice: The information contained in this e-mail > message and/or attachments to it may contain > confidential or privileged information. If you are > not the intended recipient, any dissemination, use, > review, distribution, printing or copying of the > information contained in this e-mail message > and/or attachments to it are strictly prohibited. If > you have received this communication in error, > please notify us by reply e-mail or telephone and > immediately and permanently delete the message > and any attachments. Thank you > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sreeharsha Totakura
2009-Jul-20 06:08 UTC
Re: [Xen-users] Tools to take a kernel crash dump.
Tej <bewith.tej@gmail.com> wrote on 07/20/2009 11:24:29 AM:> Tej <bewith.tej@gmail.com> > 07/20/2009 11:24 AM > > To > > Sreeharsha Totakura <sreeharsha.totakura@tcs.com> > > cc > > Nick Anderson <nick@anders0n.net>, Luke S Crawford <lsc@prgmr.com>, > Xen User-List <xen-users@lists.xensource.com> > > Subject > > Re: [Xen-users] Tools to take a kernel crash dump. > > On Sat, Jul 18, 2009 at 3:55 PM, Sreeharsha > Totakura<sreeharsha.totakura@tcs.com> wrote: > > Thanks Guys, > > > > I have installed netconsole of the server and waiting for the nextrestart> > (crash). > > > > A few things I noted while installing netconsole are: > > 1. The netconsole kernel module doesn''t work on virtual interfaces. > > 2. The xend service when started will alter the network interfaces. Inour> > setup, the real eth0 is changes as peth0 and the eth0 is made virtual.I> > assume this is done to pacify complaining programs whose setting areset to> > use eth0 by default. > > 3. For installing netconsole, use: > > /sbin/modprobe netconsole > >netconsole=6666@172.17.122.21/peth0,6666@172.17.122.174/00:1e:37:42:23:22> > This setting is required because the xen created virtual interface > > eth0 doesn''t support pooling. Only the real physical interface (peth0) > > supports it. > > It will be better to increase console log level also.We have increased the log level to 7 anticipating to get more detailed messages. The server is stable till now and didn''t have any crash. Will update you with the messages once it crashes. Thanks, Sree Harsha. =====-----=====-----====Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users