Sander Eikelenboom
2010-Nov-12 17:30 UTC
[Xen-devel] No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping
I''m encountering the following problem: When trying to shutdown a PV domU with more than 1 vpcu, the shutdown fails. The domain keeps running with 100% cpu, and it''s still possible to get the console of this domU with xm console. When only 1 vcpu is assigned the domain does shutdown. Last lines of the PV domU console: Debian GNU/Linux 5.0 tv hvc0 INIT: Switching to runlevel: 0 INIT: Sending processes the TERM signal Stopping web server: apache2 ... waiting . Stopping MythTV server: mythbackend No /usr/bin/mythbackend found running; none killed. . Stopping MTA: exim4_listener. Stopping rsync daemon: rsync. Stopping MySQL database server: mysqld. Saving the system clock. Cannot access the Hardware Clock via any known method. Use the --debug option to see the details of our search for an access method. Stopping enhanced syslogd: rsyslogd. Asking all remaining processes to terminate...done. All processes ended within 2 seconds....done. Deconfiguring network interfaces...done. Cleaning up ifupdown.... Deactivating swap...done. Unmounting local filesystems...done. Will now halt. [ 4336.046876] md: stopping all md devices. [ 4337.047171] xenbus_dev_shutdown: trying shutdown of device/console/0: Initialising [ 4337.047194] xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping [ 4337.047200] xenbus_dev_shutdown: result of shutdown of device/console/0: Initialising [ 4337.047205] xenbus_dev_shutdown: trying shutdown of device/vif/0: Connected [ 4337.110869] xenbus_dev_shutdown: result of shutdown of device/vif/0: Closed [ 4337.110883] xenbus_dev_shutdown: trying shutdown of device/vbd/51714: Connected [ 4337.161975] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: Closed [ 4337.161989] xenbus_dev_shutdown: trying shutdown of device/vbd/51713: Connected [ 4337.217136] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: Closed [ 4337.217167] System halted. But when using xenstore-ls .. i see that for every domain (but 1 and multiple vcpu''s): - All devices have state=4 - Except all backend = "/local/domain/0/backend/console/*/0" entries, those have state=1 - Although xenstore is saying the state is initializing .. xm console works perfectly for all domains. - Perhaps this also explains the high event/0 load in dom0, related to tty and xenconsoled ? DomU kernels vary from debian 2.6.26-xen kernels, to 2.6.37-rc1 mainline. Xen-unstable-tip and xen-next-2.6.32 dom0 -- Sander _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Sander Eikelenboom
2010-Nov-17 11:58 UTC
[Xen-devel] Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping
Hmm .. i haven''t received any response, is there anyone who could point me to the functions involved in communicating the state of the console from "initializing" to "connected" ? That way i could at some additional printk''s to find out why the state of domU consoles stays "1" instead of "4" in xenstore. -- Sander Friday, November 12, 2010, 6:30:10 PM, you wrote:> I''m encountering the following problem:> When trying to shutdown a PV domU with more than 1 vpcu, the shutdown fails. > The domain keeps running with 100% cpu, and it''s still possible to get the console of this domU with xm console. > When only 1 vcpu is assigned the domain does shutdown.> Last lines of the PV domU console:> Debian GNU/Linux 5.0 tv hvc0> INIT: Switching to runlevel: 0 > INIT: Sending processes the TERM signal > Stopping web server: apache2 ... waiting . > Stopping MythTV server: mythbackend No /usr/bin/mythbackend found running; none killed. > . > Stopping MTA: exim4_listener. > Stopping rsync daemon: rsync. > Stopping MySQL database server: mysqld. > Saving the system clock. > Cannot access the Hardware Clock via any known method. > Use the --debug option to see the details of our search for an access method. > Stopping enhanced syslogd: rsyslogd. > Asking all remaining processes to terminate...done. > All processes ended within 2 seconds....done. > Deconfiguring network interfaces...done. > Cleaning up ifupdown.... > Deactivating swap...done. > Unmounting local filesystems...done. > Will now halt. > [ 4336.046876] md: stopping all md devices. > [ 4337.047171] xenbus_dev_shutdown: trying shutdown of device/console/0: Initialising > [ 4337.047194] xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping > [ 4337.047200] xenbus_dev_shutdown: result of shutdown of device/console/0: Initialising > [ 4337.047205] xenbus_dev_shutdown: trying shutdown of device/vif/0: Connected > [ 4337.110869] xenbus_dev_shutdown: result of shutdown of device/vif/0: Closed > [ 4337.110883] xenbus_dev_shutdown: trying shutdown of device/vbd/51714: Connected > [ 4337.161975] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: Closed > [ 4337.161989] xenbus_dev_shutdown: trying shutdown of device/vbd/51713: Connected > [ 4337.217136] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: Closed > [ 4337.217167] System halted.> But when using xenstore-ls .. i see that for every domain (but 1 and multiple vcpu''s): > - All devices have state=4 > - Except all backend = "/local/domain/0/backend/console/*/0" entries, those have state=1 > - Although xenstore is saying the state is initializing .. xm console works perfectly for all domains. > - Perhaps this also explains the high event/0 load in dom0, related to tty and xenconsoled ?> DomU kernels vary from debian 2.6.26-xen kernels, to 2.6.37-rc1 mainline. > Xen-unstable-tip and xen-next-2.6.32 dom0> -- > Sander-- Best regards, Sander mailto:linux@eikelenboom.it _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Nov-17 12:06 UTC
[Xen-devel] Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping
Consoles do not have a connection handshake. If there is a state field in xenstore, it is only unused detritus written by the toolstack (xend/libxl/whatever). -- Keir On 17/11/2010 11:58, "Sander Eikelenboom" <linux@eikelenboom.it> wrote:> Hmm .. i haven''t received any response, is there anyone who could point me to > the functions involved in communicating the state of the console from > "initializing" to "connected" ? > That way i could at some additional printk''s to find out why the state of domU > consoles stays "1" instead of "4" in xenstore. > > -- > Sander > > Friday, November 12, 2010, 6:30:10 PM, you wrote: > >> I''m encountering the following problem: > >> When trying to shutdown a PV domU with more than 1 vpcu, the shutdown fails. >> The domain keeps running with 100% cpu, and it''s still possible to get the >> console of this domU with xm console. >> When only 1 vcpu is assigned the domain does shutdown. > >> Last lines of the PV domU console: > >> Debian GNU/Linux 5.0 tv hvc0 > >> INIT: Switching to runlevel: 0 >> INIT: Sending processes the TERM signal >> Stopping web server: apache2 ... waiting . >> Stopping MythTV server: mythbackend No /usr/bin/mythbackend found running; >> none killed. >> . >> Stopping MTA: exim4_listener. >> Stopping rsync daemon: rsync. >> Stopping MySQL database server: mysqld. >> Saving the system clock. >> Cannot access the Hardware Clock via any known method. >> Use the --debug option to see the details of our search for an access method. >> Stopping enhanced syslogd: rsyslogd. >> Asking all remaining processes to terminate...done. >> All processes ended within 2 seconds....done. >> Deconfiguring network interfaces...done. >> Cleaning up ifupdown.... >> Deactivating swap...done. >> Unmounting local filesystems...done. >> Will now halt. >> [ 4336.046876] md: stopping all md devices. >> [ 4337.047171] xenbus_dev_shutdown: trying shutdown of device/console/0: >> Initialising >> [ 4337.047194] xenbus_dev_shutdown: device/console/0: Initialising !>> Connected, skipping >> [ 4337.047200] xenbus_dev_shutdown: result of shutdown of device/console/0: >> Initialising >> [ 4337.047205] xenbus_dev_shutdown: trying shutdown of device/vif/0: >> Connected >> [ 4337.110869] xenbus_dev_shutdown: result of shutdown of device/vif/0: >> Closed >> [ 4337.110883] xenbus_dev_shutdown: trying shutdown of device/vbd/51714: >> Connected >> [ 4337.161975] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: >> Closed >> [ 4337.161989] xenbus_dev_shutdown: trying shutdown of device/vbd/51713: >> Connected >> [ 4337.217136] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: >> Closed >> [ 4337.217167] System halted. > > >> But when using xenstore-ls .. i see that for every domain (but 1 and multiple >> vcpu''s): >> - All devices have state=4 >> - Except all backend = "/local/domain/0/backend/console/*/0" entries, >> those have state=1 >> - Although xenstore is saying the state is initializing .. xm console >> works perfectly for all domains. >> - Perhaps this also explains the high event/0 load in dom0, related to >> tty and xenconsoled ? > >> DomU kernels vary from debian 2.6.26-xen kernels, to 2.6.37-rc1 mainline. >> Xen-unstable-tip and xen-next-2.6.32 dom0 > > >> -- >> Sander > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Sander Eikelenboom
2010-Nov-17 12:17 UTC
[Xen-devel] Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping
OK ... so when a pv domU shutdown, in xenbus_dev_shutdown, what does the call dev->state return for a console ? and why does it only seems to cause a problem for domains with multiple vcpu''s assigned ? -- Sander Wednesday, November 17, 2010, 1:06:04 PM, you wrote:> Consoles do not have a connection handshake. If there is a state field in > xenstore, it is only unused detritus written by the toolstack > (xend/libxl/whatever).> -- Keir> On 17/11/2010 11:58, "Sander Eikelenboom" <linux@eikelenboom.it> wrote:>> Hmm .. i haven''t received any response, is there anyone who could point me to >> the functions involved in communicating the state of the console from >> "initializing" to "connected" ? >> That way i could at some additional printk''s to find out why the state of domU >> consoles stays "1" instead of "4" in xenstore. >> >> -- >> Sander >> >> Friday, November 12, 2010, 6:30:10 PM, you wrote: >> >>> I''m encountering the following problem: >> >>> When trying to shutdown a PV domU with more than 1 vpcu, the shutdown fails. >>> The domain keeps running with 100% cpu, and it''s still possible to get the >>> console of this domU with xm console. >>> When only 1 vcpu is assigned the domain does shutdown. >> >>> Last lines of the PV domU console: >> >>> Debian GNU/Linux 5.0 tv hvc0 >> >>> INIT: Switching to runlevel: 0 >>> INIT: Sending processes the TERM signal >>> Stopping web server: apache2 ... waiting . >>> Stopping MythTV server: mythbackend No /usr/bin/mythbackend found running; >>> none killed. >>> . >>> Stopping MTA: exim4_listener. >>> Stopping rsync daemon: rsync. >>> Stopping MySQL database server: mysqld. >>> Saving the system clock. >>> Cannot access the Hardware Clock via any known method. >>> Use the --debug option to see the details of our search for an access method. >>> Stopping enhanced syslogd: rsyslogd. >>> Asking all remaining processes to terminate...done. >>> All processes ended within 2 seconds....done. >>> Deconfiguring network interfaces...done. >>> Cleaning up ifupdown.... >>> Deactivating swap...done. >>> Unmounting local filesystems...done. >>> Will now halt. >>> [ 4336.046876] md: stopping all md devices. >>> [ 4337.047171] xenbus_dev_shutdown: trying shutdown of device/console/0: >>> Initialising >>> [ 4337.047194] xenbus_dev_shutdown: device/console/0: Initialising !>>> Connected, skipping >>> [ 4337.047200] xenbus_dev_shutdown: result of shutdown of device/console/0: >>> Initialising >>> [ 4337.047205] xenbus_dev_shutdown: trying shutdown of device/vif/0: >>> Connected >>> [ 4337.110869] xenbus_dev_shutdown: result of shutdown of device/vif/0: >>> Closed >>> [ 4337.110883] xenbus_dev_shutdown: trying shutdown of device/vbd/51714: >>> Connected >>> [ 4337.161975] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: >>> Closed >>> [ 4337.161989] xenbus_dev_shutdown: trying shutdown of device/vbd/51713: >>> Connected >>> [ 4337.217136] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: >>> Closed >>> [ 4337.217167] System halted. >> >> >>> But when using xenstore-ls .. i see that for every domain (but 1 and multiple >>> vcpu''s): >>> - All devices have state=4 >>> - Except all backend = "/local/domain/0/backend/console/*/0" entries, >>> those have state=1 >>> - Although xenstore is saying the state is initializing .. xm console >>> works perfectly for all domains. >>> - Perhaps this also explains the high event/0 load in dom0, related to >>> tty and xenconsoled ? >> >>> DomU kernels vary from debian 2.6.26-xen kernels, to 2.6.37-rc1 mainline. >>> Xen-unstable-tip and xen-next-2.6.32 dom0 >> >> >>> -- >>> Sander >> >>-- Best regards, Sander mailto:linux@eikelenboom.it _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Sander Eikelenboom
2010-Nov-17 12:28 UTC
[Xen-devel] Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping
Ahh the difference between console on the other devices seems to be "XENBUS: Device with no driver: device/console/0" That makes it states "initializing" for ever ... It then probably goes wrong with the "out:" path in xenbus_dev_shutdown() doing the put_device ? I only fail to see why it seems to cause a problem with multiple vpcu''s assigned and not with just one. -- Sander Wednesday, November 17, 2010, 1:06:04 PM, you wrote:> Consoles do not have a connection handshake. If there is a state field in > xenstore, it is only unused detritus written by the toolstack > (xend/libxl/whatever).> -- Keir> On 17/11/2010 11:58, "Sander Eikelenboom" <linux@eikelenboom.it> wrote:>> Hmm .. i haven''t received any response, is there anyone who could point me to >> the functions involved in communicating the state of the console from >> "initializing" to "connected" ? >> That way i could at some additional printk''s to find out why the state of domU >> consoles stays "1" instead of "4" in xenstore. >> >> -- >> Sander >> >> Friday, November 12, 2010, 6:30:10 PM, you wrote: >> >>> I''m encountering the following problem: >> >>> When trying to shutdown a PV domU with more than 1 vpcu, the shutdown fails. >>> The domain keeps running with 100% cpu, and it''s still possible to get the >>> console of this domU with xm console. >>> When only 1 vcpu is assigned the domain does shutdown. >> >>> Last lines of the PV domU console: >> >>> Debian GNU/Linux 5.0 tv hvc0 >> >>> INIT: Switching to runlevel: 0 >>> INIT: Sending processes the TERM signal >>> Stopping web server: apache2 ... waiting . >>> Stopping MythTV server: mythbackend No /usr/bin/mythbackend found running; >>> none killed. >>> . >>> Stopping MTA: exim4_listener. >>> Stopping rsync daemon: rsync. >>> Stopping MySQL database server: mysqld. >>> Saving the system clock. >>> Cannot access the Hardware Clock via any known method. >>> Use the --debug option to see the details of our search for an access method. >>> Stopping enhanced syslogd: rsyslogd. >>> Asking all remaining processes to terminate...done. >>> All processes ended within 2 seconds....done. >>> Deconfiguring network interfaces...done. >>> Cleaning up ifupdown.... >>> Deactivating swap...done. >>> Unmounting local filesystems...done. >>> Will now halt. >>> [ 4336.046876] md: stopping all md devices. >>> [ 4337.047171] xenbus_dev_shutdown: trying shutdown of device/console/0: >>> Initialising >>> [ 4337.047194] xenbus_dev_shutdown: device/console/0: Initialising !>>> Connected, skipping >>> [ 4337.047200] xenbus_dev_shutdown: result of shutdown of device/console/0: >>> Initialising >>> [ 4337.047205] xenbus_dev_shutdown: trying shutdown of device/vif/0: >>> Connected >>> [ 4337.110869] xenbus_dev_shutdown: result of shutdown of device/vif/0: >>> Closed >>> [ 4337.110883] xenbus_dev_shutdown: trying shutdown of device/vbd/51714: >>> Connected >>> [ 4337.161975] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: >>> Closed >>> [ 4337.161989] xenbus_dev_shutdown: trying shutdown of device/vbd/51713: >>> Connected >>> [ 4337.217136] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: >>> Closed >>> [ 4337.217167] System halted. >> >> >>> But when using xenstore-ls .. i see that for every domain (but 1 and multiple >>> vcpu''s): >>> - All devices have state=4 >>> - Except all backend = "/local/domain/0/backend/console/*/0" entries, >>> those have state=1 >>> - Although xenstore is saying the state is initializing .. xm console >>> works perfectly for all domains. >>> - Perhaps this also explains the high event/0 load in dom0, related to >>> tty and xenconsoled ? >> >>> DomU kernels vary from debian 2.6.26-xen kernels, to 2.6.37-rc1 mainline. >>> Xen-unstable-tip and xen-next-2.6.32 dom0 >> >> >>> -- >>> Sander >> >>-- Best regards, Sander mailto:linux@eikelenboom.it _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Nov-17 15:36 UTC
[Xen-devel] Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping
Our 2.6.18 tree explicitly ignores console devices in drivers/xen/xenbus/xenbus_probe.c:xenbus_probe_frontend(). More modern kernels should be doing similar -- I think it''s a bug for guest''s xenbus driver to attempt to manage console devices. One of our kernel guys will be able to say more, no doubt. K. On 17/11/2010 12:28, "Sander Eikelenboom" <linux@eikelenboom.it> wrote:> Ahh the difference between console on the other devices seems to be "XENBUS: > Device with no driver: device/console/0" > That makes it states "initializing" for ever ... > It then probably goes wrong with the "out:" path in xenbus_dev_shutdown() > doing the put_device ? > > I only fail to see why it seems to cause a problem with multiple vpcu''s > assigned and not with just one. > > -- > Sander > > > Wednesday, November 17, 2010, 1:06:04 PM, you wrote: > >> Consoles do not have a connection handshake. If there is a state field in >> xenstore, it is only unused detritus written by the toolstack >> (xend/libxl/whatever). > >> -- Keir > >> On 17/11/2010 11:58, "Sander Eikelenboom" <linux@eikelenboom.it> wrote: > >>> Hmm .. i haven''t received any response, is there anyone who could point me >>> to >>> the functions involved in communicating the state of the console from >>> "initializing" to "connected" ? >>> That way i could at some additional printk''s to find out why the state of >>> domU >>> consoles stays "1" instead of "4" in xenstore. >>> >>> -- >>> Sander >>> >>> Friday, November 12, 2010, 6:30:10 PM, you wrote: >>> >>>> I''m encountering the following problem: >>> >>>> When trying to shutdown a PV domU with more than 1 vpcu, the shutdown >>>> fails. >>>> The domain keeps running with 100% cpu, and it''s still possible to get the >>>> console of this domU with xm console. >>>> When only 1 vcpu is assigned the domain does shutdown. >>> >>>> Last lines of the PV domU console: >>> >>>> Debian GNU/Linux 5.0 tv hvc0 >>> >>>> INIT: Switching to runlevel: 0 >>>> INIT: Sending processes the TERM signal >>>> Stopping web server: apache2 ... waiting . >>>> Stopping MythTV server: mythbackend No /usr/bin/mythbackend found running; >>>> none killed. >>>> . >>>> Stopping MTA: exim4_listener. >>>> Stopping rsync daemon: rsync. >>>> Stopping MySQL database server: mysqld. >>>> Saving the system clock. >>>> Cannot access the Hardware Clock via any known method. >>>> Use the --debug option to see the details of our search for an access >>>> method. >>>> Stopping enhanced syslogd: rsyslogd. >>>> Asking all remaining processes to terminate...done. >>>> All processes ended within 2 seconds....done. >>>> Deconfiguring network interfaces...done. >>>> Cleaning up ifupdown.... >>>> Deactivating swap...done. >>>> Unmounting local filesystems...done. >>>> Will now halt. >>>> [ 4336.046876] md: stopping all md devices. >>>> [ 4337.047171] xenbus_dev_shutdown: trying shutdown of device/console/0: >>>> Initialising >>>> [ 4337.047194] xenbus_dev_shutdown: device/console/0: Initialising !>>>> Connected, skipping >>>> [ 4337.047200] xenbus_dev_shutdown: result of shutdown of device/console/0: >>>> Initialising >>>> [ 4337.047205] xenbus_dev_shutdown: trying shutdown of device/vif/0: >>>> Connected >>>> [ 4337.110869] xenbus_dev_shutdown: result of shutdown of device/vif/0: >>>> Closed >>>> [ 4337.110883] xenbus_dev_shutdown: trying shutdown of device/vbd/51714: >>>> Connected >>>> [ 4337.161975] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: >>>> Closed >>>> [ 4337.161989] xenbus_dev_shutdown: trying shutdown of device/vbd/51713: >>>> Connected >>>> [ 4337.217136] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: >>>> Closed >>>> [ 4337.217167] System halted. >>> >>> >>>> But when using xenstore-ls .. i see that for every domain (but 1 and >>>> multiple >>>> vcpu''s): >>>> - All devices have state=4 >>>> - Except all backend = "/local/domain/0/backend/console/*/0" entries, >>>> those have state=1 >>>> - Although xenstore is saying the state is initializing .. xm console >>>> works perfectly for all domains. >>>> - Perhaps this also explains the high event/0 load in dom0, related to >>>> tty and xenconsoled ? >>> >>>> DomU kernels vary from debian 2.6.26-xen kernels, to 2.6.37-rc1 mainline. >>>> Xen-unstable-tip and xen-next-2.6.32 dom0 >>> >>> >>>> -- >>>> Sander >>> >>> > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Sander Eikelenboom
2010-Nov-17 20:22 UTC
[Xen-devel] Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping
Ah yes, 2.6.18 does contain the following line in xenbus_probe.c and 2.6.37-rc2 not: - if (!strcmp(type, "console")) - return 0; It seems to be changed, There seems to have been a patch http://lists.xensource.com/archives/html/xen-devel/2009-11/msg01072.html but it seems it''s not applied to xenbus in linux upstream. -- Sander Wednesday, November 17, 2010, 4:36:11 PM, you wrote:> Our 2.6.18 tree explicitly ignores console devices in > drivers/xen/xenbus/xenbus_probe.c:xenbus_probe_frontend(). More modern > kernels should be doing similar -- I think it''s a bug for guest''s xenbus > driver to attempt to manage console devices. One of our kernel guys will be > able to say more, no doubt.> K.> On 17/11/2010 12:28, "Sander Eikelenboom" <linux@eikelenboom.it> wrote:>> Ahh the difference between console on the other devices seems to be "XENBUS: >> Device with no driver: device/console/0" >> That makes it states "initializing" for ever ... >> It then probably goes wrong with the "out:" path in xenbus_dev_shutdown() >> doing the put_device ? >> >> I only fail to see why it seems to cause a problem with multiple vpcu''s >> assigned and not with just one. >> >> -- >> Sander >> >> >> Wednesday, November 17, 2010, 1:06:04 PM, you wrote: >> >>> Consoles do not have a connection handshake. If there is a state field in >>> xenstore, it is only unused detritus written by the toolstack >>> (xend/libxl/whatever). >> >>> -- Keir >> >>> On 17/11/2010 11:58, "Sander Eikelenboom" <linux@eikelenboom.it> wrote: >> >>>> Hmm .. i haven''t received any response, is there anyone who could point me >>>> to >>>> the functions involved in communicating the state of the console from >>>> "initializing" to "connected" ? >>>> That way i could at some additional printk''s to find out why the state of >>>> domU >>>> consoles stays "1" instead of "4" in xenstore. >>>> >>>> -- >>>> Sander >>>> >>>> Friday, November 12, 2010, 6:30:10 PM, you wrote: >>>> >>>>> I''m encountering the following problem: >>>> >>>>> When trying to shutdown a PV domU with more than 1 vpcu, the shutdown >>>>> fails. >>>>> The domain keeps running with 100% cpu, and it''s still possible to get the >>>>> console of this domU with xm console. >>>>> When only 1 vcpu is assigned the domain does shutdown. >>>> >>>>> Last lines of the PV domU console: >>>> >>>>> Debian GNU/Linux 5.0 tv hvc0 >>>> >>>>> INIT: Switching to runlevel: 0 >>>>> INIT: Sending processes the TERM signal >>>>> Stopping web server: apache2 ... waiting . >>>>> Stopping MythTV server: mythbackend No /usr/bin/mythbackend found running; >>>>> none killed. >>>>> . >>>>> Stopping MTA: exim4_listener. >>>>> Stopping rsync daemon: rsync. >>>>> Stopping MySQL database server: mysqld. >>>>> Saving the system clock. >>>>> Cannot access the Hardware Clock via any known method. >>>>> Use the --debug option to see the details of our search for an access >>>>> method. >>>>> Stopping enhanced syslogd: rsyslogd. >>>>> Asking all remaining processes to terminate...done. >>>>> All processes ended within 2 seconds....done. >>>>> Deconfiguring network interfaces...done. >>>>> Cleaning up ifupdown.... >>>>> Deactivating swap...done. >>>>> Unmounting local filesystems...done. >>>>> Will now halt. >>>>> [ 4336.046876] md: stopping all md devices. >>>>> [ 4337.047171] xenbus_dev_shutdown: trying shutdown of device/console/0: >>>>> Initialising >>>>> [ 4337.047194] xenbus_dev_shutdown: device/console/0: Initialising !>>>>> Connected, skipping >>>>> [ 4337.047200] xenbus_dev_shutdown: result of shutdown of device/console/0: >>>>> Initialising >>>>> [ 4337.047205] xenbus_dev_shutdown: trying shutdown of device/vif/0: >>>>> Connected >>>>> [ 4337.110869] xenbus_dev_shutdown: result of shutdown of device/vif/0: >>>>> Closed >>>>> [ 4337.110883] xenbus_dev_shutdown: trying shutdown of device/vbd/51714: >>>>> Connected >>>>> [ 4337.161975] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: >>>>> Closed >>>>> [ 4337.161989] xenbus_dev_shutdown: trying shutdown of device/vbd/51713: >>>>> Connected >>>>> [ 4337.217136] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: >>>>> Closed >>>>> [ 4337.217167] System halted. >>>> >>>> >>>>> But when using xenstore-ls .. i see that for every domain (but 1 and >>>>> multiple >>>>> vcpu''s): >>>>> - All devices have state=4 >>>>> - Except all backend = "/local/domain/0/backend/console/*/0" entries, >>>>> those have state=1 >>>>> - Although xenstore is saying the state is initializing .. xm console >>>>> works perfectly for all domains. >>>>> - Perhaps this also explains the high event/0 load in dom0, related to >>>>> tty and xenconsoled ? >>>> >>>>> DomU kernels vary from debian 2.6.26-xen kernels, to 2.6.37-rc1 mainline. >>>>> Xen-unstable-tip and xen-next-2.6.32 dom0 >>>> >>>> >>>>> -- >>>>> Sander >>>> >>>> >> >> >> >>-- Best regards, Sander mailto:linux@eikelenboom.it _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Nov-17 21:57 UTC
Re: [Xen-devel] Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping
On 11/17/2010 12:22 PM, Sander Eikelenboom wrote:> Ah yes, 2.6.18 does contain the following line in xenbus_probe.c and 2.6.37-rc2 not: > > - if (!strcmp(type, "console")) > - return 0; > It seems to be changed, > > > There seems to have been a patch http://lists.xensource.com/archives/html/xen-devel/2009-11/msg01072.html but it seems it''s not applied to xenbus in linux upstream.And does this patch fix things for you? But I have to say that''s pretty gross. Is it really the right thing to do? J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Sander Eikelenboom
2010-Nov-18 19:57 UTC
Re: [Xen-devel] Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping
Hi Jeremy/Keir, I have tried to add the following lines: if (!strcmp(type, "console")) return 0; But it seems to be a red herring .. although the "the xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping" warning is gone. The PV domU guest still doesn''t get completely halted: - It stays in a running state - It uses 100% cpu according to xentop - I can still connect to it''s console So probably something is stuck in a waiting loop, which doesn''t sleep in between and consumes 100% cpu. Last lines of domU''s console(with some additional printk''s in xenbus_dev_shutdown): Cannot access the Hardware Clock via any known method. Use the --debug option to see the details of our search for an access method. Stopping enhanced syslogd: rsyslogd. Asking all remaining processes to terminate...done. All processes ended within 2 seconds....done. Deconfiguring network interfaces...done. Cleaning up ifupdown.... Deactivating swap...done. Unmounting local filesystems...done. Will now halt. [ 46.643035] md: stopping all md devices. [ 47.643320] xenbus_dev_shutdown: trying shutdown of device/vif/0: Connected [ 47.716448] xenbus_dev_shutdown: result of shutdown of device/vif/0: Closed [ 47.716461] xenbus_dev_shutdown: trying shutdown of device/vbd/51714: Connected [ 47.772293] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: Closed [ 47.772306] xenbus_dev_shutdown: trying shutdown of device/vbd/51713: Connected [ 47.829384] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: Closed [ 47.829415] System halted. it''s a domU using a 2.6.37-rc2 kernel, file: based disk access, nothing very special. the only difference that triggers it is: vcpus=1 works fine .. vcpus>1 symptoms above .. I have also added xend.log, with first a boot and shutdown with vpcus=1 and then a boot and shutdown with vpcus=4 I left the domain(with 4vcpus) running with 100% cpu for 10 minutes, but doesn''t seem to hit any timeout, and keeps running. On 20:40 it ends up stuck, on 20:51 i''m shooting the domain off with xm destroy. Any pointers for adding extra debug info ? -- Sander Wednesday, November 17, 2010, 10:57:47 PM, you wrote:> On 11/17/2010 12:22 PM, Sander Eikelenboom wrote: >> Ah yes, 2.6.18 does contain the following line in xenbus_probe.c and 2.6.37-rc2 not: >> >> - if (!strcmp(type, "console")) >> - return 0; >> It seems to be changed, >> >> >> There seems to have been a patch http://lists.xensource.com/archives/html/xen-devel/2009-11/msg01072.html but it seems it''s not applied to xenbus in linux upstream.> And does this patch fix things for you?> But I have to say that''s pretty gross. Is it really the right thing to do?> J_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Nov-19 17:07 UTC
Re: [Xen-devel] Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping
On 11/18/2010 11:57 AM, Sander Eikelenboom wrote:> Hi Jeremy/Keir, > > I have tried to add the following lines: > if (!strcmp(type, "console")) > return 0; > > But it seems to be a red herring .. although the "the xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping" warning is gone. > The PV domU guest still doesn''t get completely halted: > - It stays in a running state > - It uses 100% cpu according to xentop > - I can still connect to it''s console > > So probably something is stuck in a waiting loop, which doesn''t sleep in between and consumes 100% cpu.Hm, I generally see clean shutdowns of my multi-VCPU domains. Can you do # gdbsx -a <domid> <arch-size> 9999 (arch-size is 32 or 64) then $ gdb vmlinux (gdb) tar rem :9999 (gdb) thread apply all bt to see what''s going on? (You may need to enable CONFIG_DEBUG_INFO to get useful info if you haven''t already.) J> Last lines of domU''s console(with some additional printk''s in xenbus_dev_shutdown): > > Cannot access the Hardware Clock via any known method. > Use the --debug option to see the details of our search for an access method. > Stopping enhanced syslogd: rsyslogd. > Asking all remaining processes to terminate...done. > All processes ended within 2 seconds....done. > Deconfiguring network interfaces...done. > Cleaning up ifupdown.... > Deactivating swap...done. > Unmounting local filesystems...done. > Will now halt. > [ 46.643035] md: stopping all md devices. > [ 47.643320] xenbus_dev_shutdown: trying shutdown of device/vif/0: Connected > [ 47.716448] xenbus_dev_shutdown: result of shutdown of device/vif/0: Closed > [ 47.716461] xenbus_dev_shutdown: trying shutdown of device/vbd/51714: Connected > [ 47.772293] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: Closed > [ 47.772306] xenbus_dev_shutdown: trying shutdown of device/vbd/51713: Connected > [ 47.829384] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: Closed > [ 47.829415] System halted. > > > > it''s a domU using a 2.6.37-rc2 kernel, file: based disk access, nothing very special. > > the only difference that triggers it is: > vcpus=1 works fine .. > vcpus>1 symptoms above .. > > I have also added xend.log, with first a boot and shutdown with vpcus=1 and then a boot and shutdown with vpcus=4 > I left the domain(with 4vcpus) running with 100% cpu for 10 minutes, but doesn''t seem to hit any timeout, and keeps running. > On 20:40 it ends up stuck, on 20:51 i''m shooting the domain off with xm destroy. > > > Any pointers for adding extra debug info ? > > -- > Sander > > > > > Wednesday, November 17, 2010, 10:57:47 PM, you wrote: > >> On 11/17/2010 12:22 PM, Sander Eikelenboom wrote: >>> Ah yes, 2.6.18 does contain the following line in xenbus_probe.c and 2.6.37-rc2 not: >>> >>> - if (!strcmp(type, "console")) >>> - return 0; >>> It seems to be changed, >>> >>> >>> There seems to have been a patch http://lists.xensource.com/archives/html/xen-devel/2009-11/msg01072.html but it seems it''s not applied to xenbus in linux upstream. >> And does this patch fix things for you? >> But I have to say that''s pretty gross. Is it really the right thing to do? >> J >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Sander Eikelenboom
2010-Nov-19 18:39 UTC
Re: [Xen-devel] Re: No shutdown of domU: xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping
Sorry, something went wrong with copy and pasting, here is the complete output: (gdb) tar rem :9999 Remote debugging using :9999 [New Thread 0] [Switching to Thread 0] smp_call_function_many (mask=0xffffffff81ce8e70, func=0xffffffff81007c4d <stop_self>, info=0x0, wait=true) at kernel/smp.c:104 104 while (data->flags & CSD_FLAG_LOCK) (gdb) thread apply all bt [New Thread 1] [New Thread 2] [New Thread 3] Thread 4 (Thread 3): #0 0xffffffff8100130a in hypercall_page () #1 0xffffffffffff02dd in ?? () #2 0x0000000000000080 in irq_stack_union () #3 0xffffffff81007c9a in stop_self (v=<value optimized out>) at /usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/xen/hypercall.h:413 #4 0xffffffff81077adf in generic_smp_call_function_interrupt () at kernel/smp.c:200 #5 0xffffffff81007eb2 in xen_call_function_interrupt (irq=<value optimized out>, dev_id=<value optimized out>) at arch/x86/xen/smp.c:472 #6 0xffffffff8109907b in handle_IRQ_event (irq=286, action=0xffff88001e8257e0) at kernel/irq/handle.c:68 #7 0xffffffff8109ae9c in handle_percpu_irq (irq=286, desc=0xffff88001ea05500) at kernel/irq/chip.c:684 #8 0xffffffff8127a3b0 in __xen_evtchn_do_upcall () at include/linux/irqdesc.h:119 #9 0xffffffff8127ab80 in xen_evtchn_do_upcall (regs=<value optimized out>) at drivers/xen/events.c:1143 #10 0xffffffff8100bc2e in xen_do_hypervisor_callback () at arch/x86/kernel/entry_64.S:1233 #11 0xffff88001e9e3e28 in ?? () #12 0x0000000000000000 in ?? () Thread 3 (Thread 2): #0 0xffffffff8100130a in hypercall_page () #1 0xffffffff81006fbd in xen_force_evtchn_callback () at /usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/xen/hypercall.h:362 #2 0xffffffff81077adf in generic_smp_call_function_interrupt () at kernel/smp.c:200 #3 0xffffffff81007eb2 in xen_call_function_interrupt (irq=<value optimized out>, dev_id=<value optimized out>) at arch/x86/xen/smp.c:472 #4 0xffffffff8109907b in handle_IRQ_event (irq=291, action=0xffff88001e825600) at kernel/irq/handle.c:68 #5 0xffffffff8109ae9c in handle_percpu_irq (irq=291, desc=0xffff88001ea05000) at kernel/irq/chip.c:684 #6 0xffffffff8127a3b0 in __xen_evtchn_do_upcall () at include/linux/irqdesc.h:119 #7 0xffffffff8127ab80 in xen_evtchn_do_upcall (regs=<value optimized out>) at drivers/xen/events.c:1143 #8 0xffffffff8100bc2e in xen_do_hypervisor_callback () at arch/x86/kernel/entry_64.S:1233 #9 0xffff88001e9e1e28 in ?? () #10 0x0000000000000000 in ?? () Current language: auto; currently asm Thread 2 (Thread 1): #0 0xffffffff8100130a in hypercall_page () #1 0xffffffff81ce8e70 in cpu_possible_bits () #2 0x0000000000000080 in irq_stack_union () #3 0xffffffff81007c9a in stop_self (v=<value optimized out>) at /usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/xen/hypercall.h:413 #4 0xffffffff81077adf in generic_smp_call_function_interrupt () at kernel/smp.c:200 #5 0xffffffff81007eb2 in xen_call_function_interrupt (irq=<value optimized out>, dev_id=<value optimized out>) at arch/x86/xen/smp.c:472 #6 0xffffffff8109907b in handle_IRQ_event (irq=296, action=0xffff88001e825420) at kernel/irq/handle.c:68 #7 0xffffffff8109ae9c in handle_percpu_irq (irq=296, desc=0xffff88001e824b00) at kernel/irq/chip.c:684 #8 0xffffffff8127a3b0 in __xen_evtchn_do_upcall () at include/linux/irqdesc.h:119 #9 0xffffffff8127ab80 in xen_evtchn_do_upcall (regs=<value optimized out>) at drivers/xen/events.c:1143 #10 0xffffffff8100bc2e in xen_do_hypervisor_callback () at arch/x86/kernel/entry_64.S:1233 #11 0xffff88001e9d7e28 in ?? () #12 0x0000000000000000 in ?? () Thread 1 (Thread 0): #0 smp_call_function_many (mask=0xffffffff81ce8e70, func=0xffffffff81007c4d <stop_self>, info=0x0, wait=true) at kernel/smp.c:104 #1 0xffffffff81077a5a in smp_call_function (func=0x40 <irq_stack_union+64>, info=<value optimized out>, wait=<value optimized out>) at kernel/smp.c:506 #2 0xffffffff81007eda in xen_stop_other_cpus (wait=<value optimized out>) at arch/x86/xen/smp.c:431 #3 0xffffffff81003172 in xen_reboot () at /usr/src/new/linux-2.6-usb-msi/arch/x86/include/asm/smp.h:81 #4 0xffffffff810031b5 in xen_machine_halt () at arch/x86/xen/enlighten.c:1039 #5 0xffffffff81023795 in machine_halt () at arch/x86/kernel/reboot.c:726 #6 0xffffffff8105e9b3 in kernel_halt () at kernel/sys.c:336 #7 0xffffffff8105eb03 in sys_reboot (magic1=-18751827, magic2=672274793, cmd=3454992675, arg=0x0) at kernel/sys.c:407 #8 0xffffffff8100acc2 in system_call_fastpath () at arch/x86/kernel/entry_64.S:479 #9 0x0000000000000202 in irq_stack_union () #10 0x0000000000000000 in ?? () Current language: auto; currently c Friday, November 19, 2010, 6:07:43 PM, you wrote:> On 11/18/2010 11:57 AM, Sander Eikelenboom wrote: >> Hi Jeremy/Keir, >> >> I have tried to add the following lines: >> if (!strcmp(type, "console")) >> return 0; >> >> But it seems to be a red herring .. although the "the xenbus_dev_shutdown: device/console/0: Initialising != Connected, skipping" warning is gone. >> The PV domU guest still doesn''t get completely halted: >> - It stays in a running state >> - It uses 100% cpu according to xentop >> - I can still connect to it''s console >> >> So probably something is stuck in a waiting loop, which doesn''t sleep in between and consumes 100% cpu.> Hm, I generally see clean shutdowns of my multi-VCPU domains.> Can you do> # gdbsx -a <domid> <arch-size> 9999 (arch-size is 32 or 64)> then> $ gdb vmlinux > (gdb) tar rem :9999 > (gdb) thread apply all bt> to see what''s going on? (You may need to enable CONFIG_DEBUG_INFO to > get useful info if you haven''t already.)> J>> Last lines of domU''s console(with some additional printk''s in xenbus_dev_shutdown): >> >> Cannot access the Hardware Clock via any known method. >> Use the --debug option to see the details of our search for an access method. >> Stopping enhanced syslogd: rsyslogd. >> Asking all remaining processes to terminate...done. >> All processes ended within 2 seconds....done. >> Deconfiguring network interfaces...done. >> Cleaning up ifupdown.... >> Deactivating swap...done. >> Unmounting local filesystems...done. >> Will now halt. >> [ 46.643035] md: stopping all md devices. >> [ 47.643320] xenbus_dev_shutdown: trying shutdown of device/vif/0: Connected >> [ 47.716448] xenbus_dev_shutdown: result of shutdown of device/vif/0: Closed >> [ 47.716461] xenbus_dev_shutdown: trying shutdown of device/vbd/51714: Connected >> [ 47.772293] xenbus_dev_shutdown: result of shutdown of device/vbd/51714: Closed >> [ 47.772306] xenbus_dev_shutdown: trying shutdown of device/vbd/51713: Connected >> [ 47.829384] xenbus_dev_shutdown: result of shutdown of device/vbd/51713: Closed >> [ 47.829415] System halted. >> >> >> >> it''s a domU using a 2.6.37-rc2 kernel, file: based disk access, nothing very special. >> >> the only difference that triggers it is: >> vcpus=1 works fine .. >> vcpus>1 symptoms above .. >> >> I have also added xend.log, with first a boot and shutdown with vpcus=1 and then a boot and shutdown with vpcus=4 >> I left the domain(with 4vcpus) running with 100% cpu for 10 minutes, but doesn''t seem to hit any timeout, and keeps running. >> On 20:40 it ends up stuck, on 20:51 i''m shooting the domain off with xm destroy. >> >> >> Any pointers for adding extra debug info ? >> >> -- >> Sander >> >> >> >> >> Wednesday, November 17, 2010, 10:57:47 PM, you wrote: >> >>> On 11/17/2010 12:22 PM, Sander Eikelenboom wrote: >>>> Ah yes, 2.6.18 does contain the following line in xenbus_probe.c and 2.6.37-rc2 not: >>>> >>>> - if (!strcmp(type, "console")) >>>> - return 0; >>>> It seems to be changed, >>>> >>>> >>>> There seems to have been a patch http://lists.xensource.com/archives/html/xen-devel/2009-11/msg01072.html but it seems it''s not applied to xenbus in linux upstream. >>> And does this patch fix things for you? >>> But I have to say that''s pretty gross. Is it really the right thing to do? >>> J >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel-- Best regards, Sander mailto:linux@eikelenboom.it _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel