I understand the trade-offs that xen is currently optimized for. However, I
don''t think that HZ=100 is right for all places and all times. I also
think that
users would appreciate it if they didn''t have to go edit config.h and
re-compile
if they wanted to change HZ. For those using xen as a platform for development
it probably makes sense if HZ is the same as the kernel they intend to ship. For
example, very early on in testing with HZ=1000 I hit an interrupt race condition
in XenFreeBSD that I have never seen before.
# This is a BitKeeper generated diff -Nru style patch.
#
# ChangeSet
# 2005/05/08 13:12:30-07:00 kmacy@curly.lab.netapp.com
# Make HZ a boot-time configurable
# Signed-off-by: Kip Macy <kmacy@netapp.com>
#
# BitKeeper/etc/logging_ok
# 2005/05/08 13:12:30-07:00 kmacy@curly.lab.netapp.com +1 -0
# Logging to logging@openlogging.org accepted
#
# xen/include/asm-x86/config.h
# 2005/05/08 13:12:15-07:00 kmacy@curly.lab.netapp.com +3 -1
# make HZ a tunable
#
# xen/arch/x86/nmi.c
# 2005/05/08 13:12:15-07:00 kmacy@curly.lab.netapp.com +2 -1
# make HZ tunable
#
# xen/arch/x86/apic.c
# 2005/05/08 13:12:15-07:00 kmacy@curly.lab.netapp.com +3 -0
# make HZ tunable
#
diff -Nru a/xen/arch/x86/apic.c b/xen/arch/x86/apic.c
--- a/xen/arch/x86/apic.c 2005-05-07 13:19:11 -07:00
+++ b/xen/arch/x86/apic.c 2005-05-07 13:19:11 -07:00
@@ -42,6 +42,9 @@
static int enabled_via_apicbase;
+unsigned int HZ = 100;
+integer_param("hz", HZ);
+
int get_maxlvt(void)
{
unsigned int v, ver, maxlvt;
diff -Nru a/xen/arch/x86/nmi.c b/xen/arch/x86/nmi.c
--- a/xen/arch/x86/nmi.c 2005-05-07 13:19:11 -07:00
+++ b/xen/arch/x86/nmi.c 2005-05-07 13:19:11 -07:00
@@ -29,7 +29,7 @@
unsigned int nmi_watchdog = NMI_NONE;
unsigned int watchdog_on = 0;
-static unsigned int nmi_hz = HZ;
+static unsigned int nmi_hz;
unsigned int nmi_perfctr_msr; /* the MSR to reset in NMI handler */
extern int logical_proc_id[];
@@ -227,6 +227,7 @@
void __pminit setup_apic_nmi_watchdog(void)
{
+ nmi_hz = HZ;
if (!nmi_watchdog)
return;
diff -Nru a/xen/include/asm-x86/config.h b/xen/include/asm-x86/config.h
--- a/xen/include/asm-x86/config.h 2005-05-07 13:19:11 -07:00
+++ b/xen/include/asm-x86/config.h 2005-05-07 13:19:11 -07:00
@@ -23,7 +23,9 @@
#define CONFIG_ACPI 1
#define CONFIG_ACPI_BOOT 1
-#define HZ 100
+#ifndef __ASSEMBLY__
+extern unsigned int HZ;
+#endif
#define OPT_CONSOLE_STR "com1,vga"
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
> I understand the trade-offs that xen is currently optimized > for. However, I don''t think that HZ=100 is right for all > places and all times. I also think that users would > appreciate it if they didn''t have to go edit config.h and > re-compile if they wanted to change HZ. For those using xen > as a platform for development it probably makes sense if HZ > is the same as the kernel they intend to ship. For example, > very early on in testing with HZ=1000 I hit an interrupt race > condition in XenFreeBSD that I have never seen before.I''d really prefer to kill the periodic ticker in Xen altogether. If domains want a periodic ticker they should just set their alarm timer appropriately. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
That would be ideal, I wasn''t aware that that was up for discussion. -Kip On Mon, 9 May 2005, Ian Pratt wrote:> > > I understand the trade-offs that xen is currently optimized > > for. However, I don''t think that HZ=100 is right for all > > places and all times. I also think that users would > > appreciate it if they didn''t have to go edit config.h and > > re-compile if they wanted to change HZ. For those using xen > > as a platform for development it probably makes sense if HZ > > is the same as the kernel they intend to ship. For example, > > very early on in testing with HZ=1000 I hit an interrupt race > > condition in XenFreeBSD that I have never seen before. > > I''d really prefer to kill the periodic ticker in Xen altogether. If > domains want a periodic ticker they should just set their alarm timer > appropriately. > > Ian > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >-- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Before I go file this, I want to ask if this is a known problem that may already be fixed. The problem is that destroying a domain that is incompletely started often fails, and sometimes locks up domain0. Repeat by using ssh to log in in two terminal sessions at once to your domain 0. In the first terminal, do "xm create" on a convenient domain. In the second, immediately do "xm destroy". I''m seeing this on the FC4-pre code, which I know is old, so I don''t really want to file a bug if this is known. shap _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 9 May 2005, at 07:22, Jonathan S. Shapiro wrote:> > The problem is that destroying a domain that is incompletely started > often fails, and sometimes locks up domain0. > > Repeat by using ssh to log in in two terminal sessions at once to your > domain 0. > > In the first terminal, do "xm create" on a convenient domain. In the > second, immediately do "xm destroy".Failure to destroy an incompletely started domain is not a bug imo, but locking up obviously is. Certainly worth a bug report. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Neugebauer, Rolf
2005-May-09 09:58 UTC
RE: [Xen-devel] [PATCH] Make HZ a boot-time configurable
I think the periodic ticker in *Xen* is only used for time keeping and is not reflected to guests. We basically only use the PIT as a stable time source (see xen/arch/x86/time.c). HZ=100 should be sufficient for this. All other timers should be run off the APIC timer which is used in one shot mode. Guests can set a one shot timer/alarm timer using the set_timer hypercall. In Linux guests this is currently only used when the guest is blocking to set the value to the next scheduled function. In addition the scheduler is set up to send the *current* guest a periodic ticker at 100HZ. This last value is hardcoded in xen/common/schedule.c. again this timer is run off the local APIC. Arguably, the frequency for this ticker should be settable per guest to reflect its HZ value (or equivalent). rolf> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > bounces@lists.xensource.com] On Behalf Of Kip Macy > Sent: 09 May 2005 06:27 > To: Ian Pratt > Cc: xen-devel@lists.xensource.com > Subject: RE: [Xen-devel] [PATCH] Make HZ a boot-time configurable > > That would be ideal, I wasn''t aware that that was up for discussion. > > > -Kip > > On Mon, 9 May 2005, Ian Pratt wrote: > > > > > > I understand the trade-offs that xen is currently optimized > > > for. However, I don''t think that HZ=100 is right for all > > > places and all times. I also think that users would > > > appreciate it if they didn''t have to go edit config.h and > > > re-compile if they wanted to change HZ. For those using xen > > > as a platform for development it probably makes sense if HZ > > > is the same as the kernel they intend to ship. For example, > > > very early on in testing with HZ=1000 I hit an interrupt race > > > condition in XenFreeBSD that I have never seen before. > > > > I''d really prefer to kill the periodic ticker in Xen altogether. If > > domains want a periodic ticker they should just set their alarmtimer> > appropriately. > > > > Ian > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > > -- > "Debugging is twice as hard as writing the code in the first place. > Therefore, if you write the code as cleverly as possible, you are, > by definition, not smart enough to debug it." - Brian W. Kernighan > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> In addition the scheduler is set up to send the *current* > guest a periodic ticker at 100HZ. This last value is > hardcoded in xen/common/schedule.c. again this timer is run > off the local APIC. > Arguably, the frequency for this ticker should be settable > per guest to reflect its HZ value (or equivalent).Or done away with altogether... As I recall, the complication with doing this is that we don''t want periodic timer interrupts to wake the domain up when its otherwise not running (whether due to having been preempted or being blocked). At a minimum, we need to make the ticker freq programmable on a per-domain basis (including ''0Hz''). Alternatively, we introduce a new event notification function that only actually ''kicks'' the domain if its already runnning. Which option is preferred? I''d like to see this make the 3.0-testing cut, though I guess option 1 could be done in a hypervisor-API backward compatible fashion. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Neugebauer, Rolf
2005-May-09 13:54 UTC
RE: [Xen-devel] [PATCH] Make HZ a boot-time configurable
> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > bounces@lists.xensource.com] On Behalf Of Ian Pratt > Sent: 09 May 2005 14:21 > To: Neugebauer, Rolf; Kip Macy > Cc: xen-devel@lists.xensource.com > Subject: RE: [Xen-devel] [PATCH] Make HZ a boot-time configurable > > > > In addition the scheduler is set up to send the *current* > > guest a periodic ticker at 100HZ. This last value is > > hardcoded in xen/common/schedule.c. again this timer is run > > off the local APIC. > > Arguably, the frequency for this ticker should be settable > > per guest to reflect its HZ value (or equivalent). > > Or done away with altogether... > > As I recall, the complication with doing this is that we don''t want > periodic timer interrupts to wake the domain up when its otherwise not > running (whether due to having been preempted or being blocked).Correct.> At a minimum, we need to make the ticker freq programmable on a > per-domain basis (including ''0Hz''). > > Alternatively, we introduce a new event notification function thatonly> actually ''kicks'' the domain if its already runnning. > > Which option is preferred? I''d like to see this make the 3.0-testing > cut, though I guess option 1 could be done in a hypervisor-APIbackward> compatible fashion.I think option 1 (per domain HZ) is better. Most OSes seem to rely on a periodic ticker for some stats collection, scheduling etc. Changing this to only have a reprogrammable timer looks quite tedious. We could have the guest OS manage this themselves, ie emulate a periodic ticker when running (reprogramming the alarm timer periodically) and use the alarm more in one-shot mode when idle but that would require the guest to know when actually running.... I''m not sure if I understand your option 2? Rolf> > Ian > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2005-May-09 14:34 UTC
[Xen-devel] Re: [PATCH] Make HZ a boot-time configurable
For the record, in Xen/ia64 each domain has complete control over when timer ticks are delivered; each domain can use a different frequency. If the domain is inactive when the timer tick would have happened, it is delivered as soon as the domain is active again. (Time marches on in Xen/ia64... every now and then each domain experiences a _very long_ interrupt.) Xen(/ia64) itself also has a periodic timer tick to ensure periodic scheduling activities. This frequency is independent of the domains'' frequency. There was some talk recently on xen-ia64-devel about switching over the Xen/ia64 timer implementation to be more like Xen/x86. Sounds like the opposite is more likely. Dan P.S. If you want more detail, please also respond to me directly as I am receiving xen-devel in digest form. Is there a convenient way to respond to digested messages to ensure the threading continues to work properly? (I''m using gmane for this...) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ronald G. Minnich
2005-May-09 15:49 UTC
RE: [Xen-devel] [PATCH] Make HZ a boot-time configurable
On Mon, 9 May 2005, Ian Pratt wrote:> I''d really prefer to kill the periodic ticker in Xen altogether. If > domains want a periodic ticker they should just set their alarm timer > appropriately.That''s a GREAT idea. I would like to see that. ron _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel