Xen devel - Feb 2012 - Xen domU Timekeeping

Howdy, all.

Is there definitive documentation about accurate timekeeping on Linux PV domUs
(Xen-4.1.2, Linux-3.1-pvops)?

Specifically:

	* Is there a way to keep good time (i.e., bare-metal accuracy) on domU?

	* What''s happened to /proc/sys/xen/independent_wallclock?

	* What should be done with /sbin/hwclock (if copied from a dom0)?

	* Does NTP on domU "work"?  Does adjtimex do anything?

	* Are there "bad side-effects" to "bad time" on domUs (see
below)...?

I''d be happy for an: "RTFM @ http://...," response, if the
docs were definitive.

==============================================================================
In addition, I''m having hard-to-track issues using ext4 on a domU,
which I suspect may be related to the timekeeping issue(s).  When I boot this
domU the first time, everything comes up nicely.  The kernel has ext2, ext3, and
ext4 drivers built in.  I see this on the console:

	[ 0.283049] EXT3-fs (xvda1): error: couldn''t
mount...unsupported...features
	[ 0.288476] EXT2-fs (xvda1): error: couldn''t
mount...unsupported...features

which is expected--and makes perfect sense--because I expect the next line to be
this:

	[ 0.318273] EXT4-fs (xvda1): mounted filesystem with ordered data mode...

And, that is what I observe.  At least, on the first boot...

But, upon reboot of the domU, I get stuck at the ext3/ext2 errors.  Guessing, I
destroyed the nonfunctional domU, and reformatted its drive as ext3.  This
worked.  I was able to reboot that domU without a problem.  Google
didn''t find too much information except this:

	http://lists.openwall.net/linux-ext4/2009/10/12/12

But this is from 2009.  And, I''m not sure how relevant it is, directly,
but it did make me wonder...Does not having "good time" on domUs
affect the ability of the kernel to mount filesystems?  Could that be breaking
ext4 on a domU with NTP?  And, despite the article''s age, is using ext4
with "barriers=0" still valid advice...?

Then...Accidentally, when I had the domU disk device mounted on dom0 (debugging,
and forgot to umount before xl create), the domU came up fine--albeit slightly
pissed off because it thought that the filesystem had errors.  But, it
"repaired" the "errors" just fine (I assume they were
related to the double-rw-mount), and booted.

I''m assuming all of this is documented somewhere; I just need a pointer
to where to find this info.

==============================================================================
Thanks,
	Q

On Mon, Feb 13, 2012 at 06:18:07PM -0800, Qrux wrote:
> Howdy, all.
> 
> Is there definitive documentation about accurate timekeeping on Linux PV
domUs (Xen-4.1.2, Linux-3.1-pvops)?
> 
> Specifically:
> 
> 	* Is there a way to keep good time (i.e., bare-metal accuracy) on domU?
It does that now. It uses the same clock as the hypervisor does so there
is no "lost ticks" or such.
> 
> 	* What''s happened to /proc/sys/xen/independent_wallclock?
No idea. What did that do?
> 
> 	* What should be done with /sbin/hwclock (if copied from a dom0)?
Well, nothing. Having a domU change the hardware time is a security
violation. It should not be able to change the hardware
time.
> 
> 	* Does NTP on domU "work"?  Does adjtimex do anything?
It will fix whatever time issues (if any) of the guest. But it won''t
adjust the hardware clock. That can only be done in dom0. So if you run
NTP in dom0 it will do it.
> 
> 	* Are there "bad side-effects" to "bad time" on domUs
(see below)...?
Sure. If the time is skewed or off there are scheduling issues. Meaning
some applications will run longer (or shorter) than they are suppose to.
> 
> I''d be happy for an: "RTFM @ http://...," response, if
the docs were definitive.
> 
>
==============================================================================>
> In addition, I''m having hard-to-track issues using ext4 on a domU,
which I suspect may be related to the timekeeping issue(s).  When I boot this
domU the first time, everything comes up nicely.  The kernel has ext2, ext3, and
ext4 drivers built in.  I see this on the console:
> 
> 	[ 0.283049] EXT3-fs (xvda1): error: couldn''t
mount...unsupported...features
> 	[ 0.288476] EXT2-fs (xvda1): error: couldn''t
mount...unsupported...features
> 
> which is expected--and makes perfect sense--because I expect the next line
to be this:
> 
> 	[ 0.318273] EXT4-fs (xvda1): mounted filesystem with ordered data mode...
> 
> And, that is what I observe.  At least, on the first boot...
> 
> But, upon reboot of the domU, I get stuck at the ext3/ext2 errors. 
Guessing, I destroyed the nonfunctional domU, and reformatted its drive as ext3.
This worked.  I was able to reboot that domU without a problem.  Google
didn''t find too much information except this:
> 
> 	http://lists.openwall.net/linux-ext4/2009/10/12/12
> 
> But this is from 2009.  And, I''m not sure how relevant it is,
directly, but it did make me wonder...Does not having "good time" on
domUs affect the ability of the kernel to mount filesystems?  Could that be
breaking ext4 on a domU with NTP?  And, despite the article''s age, is
using ext4 with "barriers=0" still valid advice...?
The issue you are hitting is probably based on what version of backend
you are using. Meaning what version of dom0 you have? It might be that
it needs :
http://old-list-archives.xen.org/archives/html/xen-devel/2011-05/msg01784.html
> 
> Then...Accidentally, when I had the domU disk device mounted on dom0
(debugging, and forgot to umount before xl create), the domU came up
fine--albeit slightly pissed off because it thought that the filesystem had
errors.  But, it "repaired" the "errors" just fine (I assume
they were related to the double-rw-mount), and booted.
> 
> I''m assuming all of this is documented somewhere; I just need a
pointer to where to find this info.
> 
>
==============================================================================>
> Thanks,
> 	Q
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Mr Qrux, please do not cross post. I have moved xen-devel to BCC since
the bit I''m replying too seems more appropriate to the xen-users list.

On Tue, 2012-02-14 at 15:57 +0000, Konrad Rzeszutek Wilk
wrote:
> On Mon, Feb 13, 2012 at 06:18:07PM -0800, Qrux wrote:
> > Howdy, all.
> > 
> > Is there definitive documentation about accurate timekeeping on Linux
PV domUs (Xen-4.1.2, Linux-3.1-pvops)?
> > 
> > Specifically:
> > 
> > 	* Is there a way to keep good time (i.e., bare-metal accuracy) on
domU?
> 
> It does that now. It uses the same clock as the hypervisor does so there
> is no "lost ticks" or such.
> 
> > 
> > 	* What''s happened to /proc/sys/xen/independent_wallclock?
> 
> No idea. What did that do?
There was a feature of the classic-Xen Linux kernels called dependent
wallclock (it was the default for those kernels). In this mode each call
to gettimeofday would return the time direct from the wallclock time
provided by the hypervisor in the shared info (wc_*). This means that
guest userspace would always get the wallclock time from the hypervisor.
dom0 would keep the hypervisor up to date by running ntp and pushing the
results down and therefore keep all guests in sync automatically.

Setting independent_wallclock would configure a guest to not use the
shared wallclock time but instead to grab the time once from the shared
info at boot and thereafter maintain its own idea of time based on its
timer ticks. This is analogous to how things happen on native (i.e. read
the RTC on boot and then user the ticks to keep in sync).

A pvops kernel has no concept of dependent_wallclock and is effectively
always in independent_wallclock mode. Jeremy made this call IIRC because
it matches how native works which reduces the special casing needed for
VMs.

This does however mean that you need to run NTP in a guest which runs a
pvops kernel.
> > 
> > 	* Does NTP on domU "work"?  Does adjtimex do anything?
For the reasons above running NTP is highly recommended in any domain
running a pvops kernel.

Ian.

Mr Qrux, please do not cross post. I have moved xen-devel to BCC since
the bit I''m replying too seems more appropriate to the xen-users list.

On Tue, 2012-02-14 at 15:57 +0000, Konrad Rzeszutek Wilk
wrote:
> On Mon, Feb 13, 2012 at 06:18:07PM -0800, Qrux wrote:
> > Howdy, all.
> > 
> > Is there definitive documentation about accurate timekeeping on Linux
PV domUs (Xen-4.1.2, Linux-3.1-pvops)?
> > 
> > Specifically:
> > 
> > 	* Is there a way to keep good time (i.e., bare-metal accuracy) on
domU?
> 
> It does that now. It uses the same clock as the hypervisor does so there
> is no "lost ticks" or such.
> 
> > 
> > 	* What''s happened to /proc/sys/xen/independent_wallclock?
> 
> No idea. What did that do?
There was a feature of the classic-Xen Linux kernels called dependent
wallclock (it was the default for those kernels). In this mode each call
to gettimeofday would return the time direct from the wallclock time
provided by the hypervisor in the shared info (wc_*). This means that
guest userspace would always get the wallclock time from the hypervisor.
dom0 would keep the hypervisor up to date by running ntp and pushing the
results down and therefore keep all guests in sync automatically.

Setting independent_wallclock would configure a guest to not use the
shared wallclock time but instead to grab the time once from the shared
info at boot and thereafter maintain its own idea of time based on its
timer ticks. This is analogous to how things happen on native (i.e. read
the RTC on boot and then user the ticks to keep in sync).

A pvops kernel has no concept of dependent_wallclock and is effectively
always in independent_wallclock mode. Jeremy made this call IIRC because
it matches how native works which reduces the special casing needed for
VMs.

This does however mean that you need to run NTP in a guest which runs a
pvops kernel.
> > 
> > 	* Does NTP on domU "work"?  Does adjtimex do anything?
For the reasons above running NTP is highly recommended in any domain
running a pvops kernel.

Ian.

Thanks, Ian and Konrad, for your responses.  More questions inline...

On Feb 14, 2012, at 8:19 AM, Ian Campbell wrote:
>>> 	* Is there a way to keep good time (i.e., bare-metal accuracy) on
domU?
>> 
>> It does that now. It uses the same clock as the hypervisor does so
there
>> is no "lost ticks" or such.
Awesome.

Are pvops kernels (dom0 and domU) using tickless timekeeping when configured
with NO_HZ?  If not, what is it doing?  If so, which clocksource is it using? 
TSC?  HPET?  What are the proper options to set in the kernel?

If one doesn''t use NO_HZ, are pvops kernels counting ticks to keep
time?  If so, which clocksource does that use?  PIT?  And, what are the proper
options for that?

	Q: Which configuration is "preferred" in the interest of good
timekeeping?

Obviously, a tick-counting kernel will generate far more interrupts.  OTOH,
one-shot tickless timekeeping has higher latencies, AFAIK.  So, if there *is* a
preferred configuration for good timekeeping...

	Q: Does "good timekeeping" trade-off other benefits?
> A pvops kernel has no concept of dependent_wallclock and is effectively
> always in independent_wallclock mode. Jeremy made this call IIRC because
> it matches how native works which reduces the special casing needed for
> VMs.
> 
> This does however mean that you need to run NTP in a guest which runs a
> pvops kernel.
I''m thoroughly baffled.  Konrad basically said that domUs uses the same
time as the hypervisor.  That sort of implies dependent_wallclock (at least in
operation, if not explicitly as a kernel feature).  But, then, Ian said that we
need to run NTP in a pvops guest.  Which sort of implies independent_wallclock.

	Q: Which is it?

===I think I get the picture; please correct me if this is a misunderstanding:

Only dom0 has access to the RTC (via hwclock or the ioctl interface).  But, domU
doesn''t (which I verified, at least with hwclock--it complained about
no way to access any hardware clock).  Since domU doesn''t have a
working hwclock, someone has to set "wall time" (the way that dom0 can
with hwclock, or via the kernel during boot).

So, we use NTP on domU to retrieve a time.  But, I assume that *any* time source
can be used for this purpose, even old port-13-daytime.  Basically, we just need
something as roughly accurate as RTC to give a "reference start time"
to the kernel/software/system clock--which I believe are all the same, according
to the usage in time(7) and rtc(4).

In addition, I believe NTP uses adjtimex (on Linux) to discipline the kernel
clock--but  not RTC (see usage).  Which would imply that it''s safe to
run on domU (though, how accurate it can be...is unclear).
===
*Whew*

So, if that *is* the picture...we can move on to the meat of it!

If a pvops domU cannot see the RTC, it has no fracking idea what time it is
(calendar/wall time).  It only knows what *relative* time it is, w.r.t. when it
was created.  Now, on most bare-metal setups, NTP comes into the process way
late.  Like, rc3.d-late.  Which means if we''re doing disk-mounts in
rcS.d, fsck will have no idea what time it is.

Are there bad interactions if fsck doesn''t have any clue what time it
is?  Does this also mean that NTP should get started earlier?  And, if so, does
that imply networking would need to be moved way up in rcS.d?  I assume there
*must* be guidelines about what *needs* to go into pvops domU /etc/{rc,init}.d/,
and suggested orderings.

	Q: What is the correct sequence of events on pvops domU boot?

And, finally, for my specific situation, does this help explain why my ext4 domU
(which uses an LVM volume formatted with ext4 on dom0) won''t reboot,
but a ext3 one will?  Is ext4 pickier about times during the loading of its
driver?

	Q

Dear Abby,

I''ve been having intermittent problems booting an ext4 domU--LFS,
Linux-3.1 pvops, Xen-4.1.2, xl (not xm).  This thread was started under a
different guise, which Ian and Konrad were helpful; I thought the issue was
related to NTP.  After more research, with their input, I was able to answer
many of my own questions.  Then, after even more research, it seems clear the
issue may be far deeper down the abyss...

To start, here''s the domU config:

	kernel = "/boot/vmlinuz-3.1-lfs-7.0-DomU"
	memory = 1024
	name = "e4no23-ntp"
	vif = [ ''mac=00:16:3e:00:00:01, bridge=br0'' ]
	disk = [ ''phy:/dev/vg-xen/e4no23-ntp,xvda1,w'' ]
	root = "/dev/xvda1"
	on_crash = "preserve"

/dev/vg-xen/e4no23-ntp, as the name suggests, is an LVM volume.  When an
otherwise identical volume is formatted as ext3, this problem cannot be
reproduced.  When this volume is formatted at ext4, the problem below is
intermittent, though once it appears, it seems to happen quite often.

In this domU, I boot it and watch the console.  It works, and I copied the
output.  Then, I SSH-in and reboot it, again watching the console.  It fails,
and I copy the output.  I''ve included a diff of the good boot vs the
bad one, which has 3 chunks.  I''m happy to throw more info into a
pastebin if it will help someone trying to eyeball this, but the summary is that
in both boots, we see these 5 lines:

  bio: create slab <bio-0> at 0
  xen/balloon: Initialising balloon driver.
  last_pfn = 0x40100 max_arch_pfn = 0x400000000
  Switching to clocksource xen
  Switched to NOHz mode on CPU #0

as the last 5 lines of common output before things get weird in the bad boot:

+ CE: xen increased min_delta_ns to 150000 nsec
+ CE: xen increased min_delta_ns to 225000 nsec
+ CE: xen increased min_delta_ns to 337500 nsec
+ CE: xen increased min_delta_ns to 506250 nsec
+ CE: xen increased min_delta_ns to 759375 nsec
+ CE: xen increased min_delta_ns to 1139062 nsec
+ CE: xen increased min_delta_ns to 1708593 nsec
+ CE: xen increased min_delta_ns to 2562889 nsec
+ CE: xen increased min_delta_ns to 3844333 nsec
+ CE: xen increased min_delta_ns to 4000000 nsec
+ CE: Reprogramming failure. Giving up
+ CE: Reprogramming failure. Giving up
+ hrtimer: interrupt took 9750 ns
+ CE: Reprogramming failure. Giving up
  FS-Cache: Loaded
  CacheFiles: Loaded
  NET: Registered protocol family 2
  IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
  TCP established hash table entries: 131072 (order: 9, 2097152 bytes)

From my research, it seems as if this issue has come up before, in perhaps 3
manifestations:

	* Jul 2010
	* [Xen-devel] xen tsc problems?
	* http://tinyurl.com/7dqf5qx

	* Jan 2011
	* [Xen-users] CentOS 5.5 x86_64 - XEN DomU LVM ext4 partition support
	* http://tinyurl.com/842rx8b

	* Aug 2011
	* [Xen-devel] xen-4.1: PV domain hanging at startup, jiffies stopped
	* http://tinyurl.com/7f9osav

Of course, I''m not absolutely certain that they are related, but the
symptoms that all 3 people described fit my current observations.  In the first
one, it seemed like the resolution was about timer_mode being changed from its
default of 0 (zero) to 1 (one) in xl.  The post suggests that the TSC can
operate in multiple "modes", though I have no idea what those are, and
what they do.  That seemed to conclude the issue in mid 2010...

In Jan 2011, it manifests (in my interpretation) as an EXT4 failure.  And, in my
particular case, I believe these issues are related, because I cannot repro this
problem using ext3.

Then, in Aug 2011, it comes back.  (Interestingly, this issue appears alongside
a nearly identical issue captured in a thread in Feb 2011...more on this later.)
The OP does an epic amount of debugging, during which, someone suggest using
xenctx to look at the crashed VM.  Here''s what I get:

===xlapp [~] # /usr/lib/xen/bin/xenctx -s /boot/System.map-3.1-DomU 3 0
rip: ffffffff810013aa hypercall_page+0x3aa 
flags: 00001246 i z p
rsp: ffffffff81709ee0
rax: 0000000000000000	rcx: ffffffff810013aa	rdx: 0000000000000000
rbx: ffffffff81709fd8	rsi: 0000000000000000	rdi: 0000000000000001
rbp: ffffffff81709ef8	 r8: 0000000000000000	 r9: 0000000000000000
r10: 0000000000000400	r11: 0000000000000246	r12: 0000000000000000
r13: ffff88003fff3b80	r14: ffffffffffffffff	r15: 0000000000000000
 cs: e033	 ss: e02b	 ds: 0000	 es: 0000
 fs: 0000 @ 0000000000000000
 gs: 0000 @ ffff88003ffd3000/0000000000000000
Code (instr addr ffffffff810013aa)
cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59
c3 cc cc cc cc cc cc cc


Stack:
 0000000000000000 00000000ffffffff ffffffff81009f60 ffffffff81709f28
 ffffffff8101a2c0 ffffffff81709fd8 ffffffff817784c0 ffff88003fff3b80
 ffffffffffffffff ffffffff81709f48 ffffffff810121b6 ffffffff817c6e40
 ffffffff817ca5e0 ffffffff81709f58 ffffffff81526549 ffffffff81709f98

Call Trace:
  [<ffffffff810013aa>] hypercall_page+0x3aa  <--
  [<ffffffff81009f60>] xen_safe_halt+0x10 
  [<ffffffff8101a2c0>] default_idle+0x60 
  [<ffffffff810121b6>] cpu_idle+0x66 
  [<ffffffff81526549>] rest_init+0x6d 
  [<ffffffff81796b87>] start_kernel+0x332 
  [<ffffffff81796347>] x86_64_start_reservations+0x132 
  [<ffffffff817994a9>] xen_start_kernel+0x50c 
===
This is the same "useless" output that the OP of thread 3 saw: a trace
ending in the same "hypercall_page+0x3aa" line.  Later on in the
thread, having received very little feedback, OP makes this claim (responding to
his own email):

===> I''ve collected few more messages from successful and failed
domU starts.
> The only difference is the place where "Switched to NOHz mode on CPU
#0"
> appears and existence of "CE: xen increased min_delta_ns to ..."
and
> "CE: Reprogramming failure. Giving up" messages.
> 
> I think it can be related to:
> http://lists.xensource.com/archives/html/xen-devel/2010-07/msg00649.html
> (this was on HVM not PV, but looks similar)
> 
> I''ve tried also xenpm set-max-cstate 0 and tsc_mode=1 in domU
config,
> but it doesn''t help. Also pinning vcpu doesn''t help (this
domUs have
> only 1 vcpu). Is ''xenpm set-max-cstate 0'' the same as
booting xen with
> max_cstate=0?
Looks like tsc_mode=2 solves the problem.
===
Other people go on to say that setting tsc_mode=2 is a work-around, and only
indicative of a deeper problem, though no one seems to know what it is.  Someone
else suggests that tsc_mode=0 is (was?) the default and should be equivalent to
tsc_mode=2, making the OP''s change an effective no-op.  Later, the OP
references a document:

	http://lxr.xensource.com/lxr/source/docs/misc/tscmode.txt

which doesn''t exist, but I assume is this:

	xen-4.1.2/docs/misc/tscmode.txt

I''ve run:

	xl debug-key s; xl dmesg | tail

per its suggestion:

===xlapp [~] # xl dmesg | tail
(XEN) PCI add device 00:1d.2
(XEN) PCI add device 00:1d.3
(XEN) PCI add device 00:1d.7
(XEN) PCI add device 00:1e.0
(XEN) PCI add device 00:1f.0
(XEN) PCI add device 00:1f.1
(XEN) PCI add device 00:1f.2
(XEN) PCI add device 01:00.0
(XEN) TSC has constant rate, no deep Cstates, passed warp test, deemed reliable,
warp=0 (count=1)
(XEN) dom3: mode=0,ofs=0xf193f5ae955,khz=2628837,inc=1,vtsc count: 516338
kernel, 0 user
===
It appears as if the system is TSC-safe (in the lexicon of
docs/misc/tscmode.txt).  So, it looks like it''s going to use mode=0,
and essentially use the native rdtsc family of instructions.

But, this doesn''t appear to be the case.  Like the OP from Aug 11,
explicitly setting tsc_mode=1 breaks domU (repro''s the hang), whereas
setting tsc_mode=2 unbreaks it (I have not, over several restarts, seen the
problem reappear.

* * *

After doing more research, it turns out there was a parallel discussion to the
one in Aug 2011 which started 6 months earlier, but continued through Sept 2011:

	* Feb 2011 - Sept 2011
	* [Xen-devel] Xen 4 TSC problems
	* http://xen.1045712.n5.nabble.com/Xen-4-TSC-problems-td3396848.html

Lots of issues were discussed, including changing the Xen platform timer to PIT
instead of TSC/HPET, setting cpuidle=0, disabling HPET from the BIOS, turning
deep C states off, and so on.  I hate to complain about a product I''ve
so enjoyed using, now in version 4.x.y, but that''s--

	A LOT of trial-and-error...

...with A LOT of wait-and-see if it solves the issue.  The main question, IMO,
is:

	Is there a definitive configuration which is safe?

The academic-leaning "correctness" issue seems to be:

	Why isn''t tsc_mode=1 working, when it''s claimed to be
"slow-but-correct"?

But, practically, is there a safe CPU configuration?  Putting aside
everyone''s concerns about saving the planet, (or budgetary concerns, if
your colo bills by the Watt or if you run your own data center and pay for the
electricity), is there a "safe" server config in terms of C-states,
ACPI, etc?  And, by "safe", I mean, a statement like this:

==="We know this BIOS config will very likely work if the CPU
doesn''t try to do a bunch of fancy speed-stepping or turbo-boosting
(who are we, Knight Rider?) or deep-sleeping or blah-blah-blah.  Turn X on, and
Y off, and set Z to this in your BIOS, start the hypervisor (kernel=xen line)
with options A=a, B=b, C=c, run dom0 (module=vmlinuz-dom0 line) with options
L=l, M=m, and N=n, and try to get your machine to state EFGHIJK so we know the
machine will roughly have behaviors P, Q, R, and S, which should work with
tsc_mode=N."
===
Since September, I can''t find any further information about this issue.
What is the state of this issue?  The inconsistency I see right now is this: in
the July 2010 TSC discussion, a "Stefano Stabellini" posted this:

===> /me wonders if timer_mode=1 is the default for
xl?
> Or only for xm?
no, it is not.
Xl defaults to 0 [zero], I am going to change it right now.
===
So, it seems like (at least as of July 2010), xl is defaulting to
"timer_mode=1".  That is, assuming that the then-current timer_mode is
the same as present-day tsc_mode.  In addition, I''m assuming he was
changing it from 0 (zero) to 1 (one)--and not some other mode.  But,

	xen-4.1.2/docs/misc/tscmode.txt

says:

	"The default mode (tsc_mode==0) checks TSC-safeness of the underlying
	hardware on which the virtual machine is launched.  If it is
	TSC-safe, rdtsc will execute at hardware speed; if it is not, rdtsc
	will be emulated."

Which implies the default is always 0 (zero).  Which is it?  More importantly,
is the solution to force tsc_mode=2?  If so, under what
BIOS/xen-boot-params/dom0-boot-params conditions?  And--please excuse my
exasperation--but WTH does this have to do with ext3 versus ext4?  Is ext4
exquisitely sensitive to TSC/HPET "jumpiness" (if that''s even
what''s happening)?

Sincerely,
  Deeply Concerned & Slightly Frustrated




p.s. I''ve attached my dom0 ''xl dmesg'' output, in case
that helps:

======== START xl dmesg ======= __  __            _  _    _   ____  
 \ \/ /___ _ __   | || |  / | |___ \ 
  \  // _ \ ''_ \  | || |_ | |   __) |
  /  \  __/ | | | |__   _|| |_ / __/ 
 /_/\_\___|_| |_|    |_|(_)_(_)_____|
                                     
(XEN) Xen version 4.1.2 (blfs@site) (gcc version 4.6.1 (GCC) ) Tue Feb 14
21:47:22 PST 2012
(XEN) Latest ChangeSet: unavailable
(XEN) Console output is synchronous.
(XEN) Bootloader: GNU GRUB 0.97
(XEN) Command line: dom0_mem=1024M loglvl=all guest_loglvl=all console_to_ring
sync_console
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 3 MBR signatures
(XEN)  Found 3 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009fc00 (usable)
(XEN)  000000000009fc00 - 00000000000a0000 (reserved)
(XEN)  00000000000e4000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000ddd80000 (usable)
(XEN)  00000000ddd80000 - 00000000ddd8e000 (ACPI data)
(XEN)  00000000ddd8e000 - 00000000dddd0000 (ACPI NVS)
(XEN)  00000000dddd0000 - 00000000e0000000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000fff00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000220000000 (usable)
(XEN) ACPI: RSDP 000FB760, 0024 (r2 ACPIAM)
(XEN) ACPI: XSDT DDD80100, 0054 (r1 A_M_I_ OEMXSDT  10001121 MSFT       97)
(XEN) ACPI: FACP DDD80290, 00F4 (r3 A_M_I_ OEMFACP  10001121 MSFT       97)
(XEN) ACPI: DSDT DDD80440, 8794 (r1  A1745 A1745000        0 INTL 20060113)
(XEN) ACPI: FACS DDD8E000, 0040
(XEN) ACPI: APIC DDD80390, 006C (r1 A_M_I_ OEMAPIC  10001121 MSFT       97)
(XEN) ACPI: MCFG DDD80400, 003C (r1 A_M_I_ OEMMCFG  10001121 MSFT       97)
(XEN) ACPI: OEMB DDD8E040, 0089 (r1 A_M_I_ AMI_OEM  10001121 MSFT       97)
(XEN) ACPI: HPET DDD88BE0, 0038 (r1 A_M_I_ OEMHPET  10001121 MSFT       97)
(XEN) ACPI: GSCI DDD8E0D0, 2024 (r1 A_M_I_ GMCHSCI  10001121 MSFT       97)
(XEN) System RAM: 8157MB (8352892kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-0000000220000000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000ff780
(XEN) DMI present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x808
(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[804,0], pm1x_evt[800,0]
(XEN) ACPI:                  wakeup_vec[ddd8e00c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 6:15 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
(XEN) Processor #1 6:15 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
(XEN) Processor #2 6:15 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
(XEN) Processor #3 6:15 APIC version 20
(XEN) ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 4, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) PCI: MCFG configuration 0: base f0000000 segment 0 buses 0 - 63
(XEN) PCI: Not using MMCONFIG.
(XEN) Table is not found!
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) IRQ limits: 24 GSI, 760 MSI/MSI-X
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2628.837 MHz processor.
(XEN) Initing memory sharing.
(XEN) mce_intel.c:1162: MCA Capability: BCAST 1 SER 0 CMCI 0 firstbank 1
extended MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) I/O virtualisation disabled
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 32 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC TPR shadow
(XEN)  - MSR direct-access bitmap
(XEN) HVM: ASIDs disabled.
(XEN) HVM: VMX enabled
(XEN) Brought up 4 CPUs
(XEN) HPET: 3 timers in total, 0 timers will be used for broadcast
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x23cf000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000214000000->0000000218000000 (245760 pages to be
allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff823cf000
(XEN)  Init. ramdisk: ffffffff823cf000->ffffffff823cf000
(XEN)  Phys-Mach map: ffffffff823cf000->ffffffff825cf000
(XEN)  Start info:    ffffffff825cf000->ffffffff825cf4b4
(XEN)  Page tables:   ffffffff825d0000->ffffffff825e7000
(XEN)  Boot stack:    ffffffff825e7000->ffffffff825e8000
(XEN)  TOTAL:         ffffffff80000000->ffffffff82800000
(XEN)  ENTRY ADDRESS: ffffffff820e3200
(XEN) Dom0 has maximum 4 VCPUs
(XEN) Scrubbing Free RAM:
......................................................................done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) **********************************************
(XEN) ******* WARNING: CONSOLE OUTPUT IS SYNCHRONOUS
(XEN) ******* This option is intended to aid debugging of Xen by ensuring
(XEN) ******* that all output is synchronously delivered on the serial line.
(XEN) ******* However it can introduce SIGNIFICANT latencies and affect
(XEN) ******* timekeeping. It is NOT recommended for production use!
(XEN) **********************************************
(XEN) 3... 2... 1... 
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type ''CTRL-a'' three times
to switch input to Xen)
(XEN) Freed 216kB init memory.
mapping kernel into physical memory
Xen: setup ISA identity maps
about to get started...
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.1.0-xlapp-dom0-debug (root@xlapp) (gcc version
4.6.1 (GCC) ) #1 SMP Wed Feb 15 19:22:24 PST 2012
[    0.000000] Command line: root=/dev/sda5 raid=noautodetect console=tty0
earlyprintk=xen nomodeset initcall_debug debug loglevel=10
[    0.000000] released 0 pages of unused memory
[    0.000000] Set 140000 page(s) to 1-1 mapping.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 000000000009f000 (usable)
[    0.000000]  Xen: 000000000009fc00 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000040000000 (usable)
[    0.000000]  Xen: 0000000040000000 - 00000000ddd80000 (unusable)
[    0.000000]  Xen: 00000000ddd80000 - 00000000ddd8e000 (ACPI data)
[    0.000000]  Xen: 00000000ddd8e000 - 00000000dddd0000 (ACPI NVS)
[    0.000000]  Xen: 00000000dddd0000 - 00000000e0000000 (reserved)
[    0.000000]  Xen: 00000000fec00000 - 00000000fec01000 (reserved)
[    0.000000]  Xen: 00000000fee00000 - 00000000fee01000 (reserved)
[    0.000000]  Xen: 00000000fff00000 - 0000000100000000 (reserved)
[    0.000000]  Xen: 0000000100000000 - 00000002bdd80000 (usable)
[    0.000000] bootconsole [xenboot0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI present.
[    0.000000] DMI: System manufacturer System Product Name/P5G41T-M LX PLUS,
BIOS 0502    10/21/2011
[    0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable)
==> (reserved)
[    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x2bdd80 max_arch_pfn = 0x400000000
[    0.000000] last_pfn = 0x40000 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [ffff8800000ff780] ff780
[    0.000000] initial memory mapped : 0 - 023cf000
[    0.000000] Base memory trampoline at [ffff88000009d000] 9d000 size 8192
[    0.000000] init_memory_mapping: 0000000000000000-0000000040000000
[    0.000000]  0000000000 - 0040000000 page 4k
[    0.000000] kernel direct mapping tables up to 40000000 @ dfe000-1000000
[    0.000000] xen: setting RW the range fe6000 - 1000000
[    0.000000] init_memory_mapping: 0000000100000000-00000002bdd80000
[    0.000000]  0100000000 - 02bdd80000 page 4k
[    0.000000] kernel direct mapping tables up to 2bdd80000 @ 3ea05000-40000000
[    0.000000] xen: setting RW the range 3f7fb000 - 40000000
[    0.000000] ACPI: RSDP 00000000000fb760 00024 (v02 ACPIAM)
[    0.000000] ACPI: XSDT 00000000ddd80100 00054 (v01 A_M_I_ OEMXSDT  10001121
MSFT 00000097)
[    0.000000] ACPI: FACP 00000000ddd80290 000F4 (v03 A_M_I_ OEMFACP  10001121
MSFT 00000097)
[    0.000000] ACPI: DSDT 00000000ddd80440 08794 (v01  A1745 A1745000 00000000
INTL 20060113)
[    0.000000] ACPI: FACS 00000000ddd8e000 00040
[    0.000000] ACPI: APIC 00000000ddd80390 0006C (v01 A_M_I_ OEMAPIC  10001121
MSFT 00000097)
[    0.000000] ACPI: MCFG 00000000ddd80400 0003C (v01 A_M_I_ OEMMCFG  10001121
MSFT 00000097)
[    0.000000] ACPI: OEMB 00000000ddd8e040 00089 (v01 A_M_I_ AMI_OEM  10001121
MSFT 00000097)
[    0.000000] ACPI: HPET 00000000ddd88be0 00038 (v01 A_M_I_ OEMHPET  10001121
MSFT 00000097)
[    0.000000] ACPI: GSCI 00000000ddd8e0d0 02024 (v01 A_M_I_ GMCHSCI  10001121
MSFT 00000097)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at 0000000000000000-00000002bdd80000
[    0.000000] Initmem setup node 0 0000000000000000-00000002bdd80000
[    0.000000]   NODE_DATA [000000003fffb000 - 000000003fffffff]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x002bdd80
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[3] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x0000009f
[    0.000000]     0: 0x00000100 -> 0x00040000
[    0.000000]     0: 0x00100000 -> 0x002bdd80
[    0.000000] On node 0 totalpages: 2088207
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 490 pages reserved
[    0.000000]   DMA zone: 3429 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 16320 pages used for memmap
[    0.000000]   DMA32 zone: 241728 pages, LIFO batch:31
[    0.000000]   Normal zone: 28534 pages used for memmap
[    0.000000]   Normal zone: 1797642 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0x808
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] BIOS bug: APIC version is 0 for CPU 0/0x0, fixing up to 0x10
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
[    0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 4, version 255, address 0xfec00000, GSI 0-255
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] nr_irqs_gsi: 272
[    0.000000] Allocating PCI resources starting at e0000000 (gap:
e0000000:1ec00000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.1.2 (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:4
nr_node_ids:1
[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff88003ff57000 s78912 r8192
d23488 u110592
[    0.000000] pcpu-alloc: s78912 r8192 d23488 u110592 alloc=27*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total
pages: 2042799
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: root=/dev/sda5 raid=noautodetect
console=tty0 earlyprintk=xen nomodeset initcall_debug debug loglevel=10
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Placing 64MB software IO TLB between ffff880032600000 -
ffff880036600000
[    0.000000] software IO TLB at phys 0x32600000 - 0x36600000
[    0.000000] Memory: 812604k/11499008k available (11712k kernel code, 3146180k
absent, 7540224k reserved, 5491k data, 828k init)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=4,
Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] NR_IRQS:4352 nr_irqs:1024 16
[    0.000000] xen: sci override: global_irq=9 trigger=0 polarity=0
[    0.000000] xen: registering gsi 9 triggering 0 polarity 0
[    0.000000] xen: --> pirq=9 -> irq=9 (gsi=9)
[    0.000000] xen: acpi sci 9
[    0.000000] xen: --> pirq=1 -> irq=1 (gsi=1)
[    0.000000] xen: --> pirq=2 -> irq=2 (gsi=2)
[    0.000000] xen: --> pirq=3 -> irq=3 (gsi=3)
[    0.000000] xen: --> pirq=4 -> irq=4 (gsi=4)
[    0.000000] xen: --> pirq=5 -> irq=5 (gsi=5)
[    0.000000] xen: --> pirq=6 -> irq=6 (gsi=6)
[    0.000000] xen: --> pirq=7 -> irq=7 (gsi=7)
[    0.000000] xen: --> pirq=8 -> irq=8 (gsi=8)
[    0.000000] xen_map_pirq_gsi: returning irq 9 for gsi 9
[    0.000000] xen: --> pirq=9 -> irq=9 (gsi=9)
[    0.000000] xen: --> pirq=10 -> irq=10 (gsi=10)
[    0.000000] xen: --> pirq=11 -> irq=11 (gsi=11)
[    0.000000] xen: --> pirq=12 -> irq=12 (gsi=12)
[    0.000000] xen: --> pirq=13 -> irq=13 (gsi=13)
[    0.000000] xen: --> pirq=14 -> irq=14 (gsi=14)
[    0.000000] xen: --> pirq=15 -> irq=15 (gsi=15)
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [tty0] enabled, bootconsole disabled
(XEN) PCI add device 00:00.0
(XEN) PCI add device 00:02.0
(XEN) PCI add device 00:1b.0
(XEN) PCI add device 00:1c.0
(XEN) PCI add device 00:1c.1
(XEN) PCI add device 00:1d.0
(XEN) PCI add device 00:1d.1
(XEN) PCI add device 00:1d.2
(XEN) PCI add device 00:1d.3
(XEN) PCI add device 00:1d.7
(XEN) PCI add device 00:1e.0
(XEN) PCI add device 00:1f.0
(XEN) PCI add device 00:1f.1
(XEN) PCI add device 00:1f.2
(XEN) PCI add device 01:00.0
(XEN) TSC has constant rate, no deep Cstates, passed warp test, deemed reliable,
warp=0 (count=1)
(XEN) dom3: mode=0,ofs=0xf193f5ae955,khz=2628837,inc=1,vtsc count: 516338
kernel, 0 user
========  END  xl dmesg ========

2012/2/16 Qrux <qrux.qed@gmail.com>:
> Thanks, Ian and Konrad, for your responses.  More questions inline...

This is really interesting. Unfortunately I still don''t fully
understand how things are supposed to work in a pvops domU.
Currently we have an old PV kernel setup that relies on not having an
independent wallclock. Having all domUs running the same time on a
sub-millisecond level is, err, was, one of the killer features of Xen.
On the other hand half of all linux distro kernels had it broken :/
> Obviously, a tick-counting kernel will generate far more interrupts.  OTOH,
one-shot tickless timekeeping has higher latencies, AFAIK.  So, if there *is* a
preferred configuration for good timekeeping...
We''re being far too vague there. Thats why there was stuff like
max_clock_jitter in Xen in the first place.

Could one of you explain how different the timekeeping overhead is in
a pvops domU over the old method (let me add I''ve already fired up
something like 200 alpinelinux domUs that run pvops and still saw
typical Xen-like low overhead.
And I think Qrux'' question regarding NO_HZ is also important. There
are cases where clock jitter is really intolerable and NTP alone is
NOT suitable to fight clocks suddenly going faster. It can very well
deal with a certain, constant drift, but not with a clock that goes
nuts like one would see when running old CentOS on VMWare or
VirtualBox.
> If one doesn''t use NO_HZ, are pvops kernels counting ticks to keep
time?  If so, which clocksource does that use?  PIT?  And, what are the proper
options for that?
Specifically: Which is the right clocksource for pvops?
> I''m thoroughly baffled.  Konrad basically said that domUs uses the
same time as the hypervisor.  That sort of implies dependent_wallclock (at least
in operation, if not explicitly as a kernel feature).  But, then, Ian said that
we need to run NTP in a pvops guest.  Which sort of implies
independent_wallclock.
I think they mean the same thing:
Konrad saying all domUs will start out with the RTC time. Which, btw,
is not transparent and still leaves the special casing by now showing
special hardware instead of a special case in the kernel. like i.e.
hwclock failing which gives false alerts at shutdown time.

And I think Ian is saying you need to run NTP as linux doesn''t care
about the RTC (at least, not about the TIME it holds) after it booted
and something needs to keep an eye on the kernel clock to avoid drift.
> In addition, I believe NTP uses adjtimex (on Linux) to discipline the
kernel clock--but  not RTC (see usage).  Which would imply that it''s
safe to run on domU (though, how accurate it can be...is unclear).
Yes, it tries. And it is not good fighting unstable erratic drift.
A recent batch of Fujitsu workstations had a bios with funny timer
issues. the bios time was super stable, but the kernel wallclock was
doing 2-3s per sec. NTP is totally unable to correct something like
that.
> If a pvops domU cannot see the RTC, it has no fracking idea what time it is
(calendar/wall time).  It only knows what *relative* time it is, w.r.t. when it
was created.  Now, on most bare-metal setups, NTP comes into the process way
late.  Like, rc3.d-late.  Which means if we''re doing disk-mounts in
rcS.d, fsck will have no idea what time it is.
I think that''s not true, it should have a reasonably correct idea of
"time".
Maybe 5 seconds off in a troublesome world, but not much more.
> Are there bad interactions if fsck doesn''t have any clue what time
it is?  Does this also mean that NTP should get started earlier?  And, if so,
does that imply networking would need to be moved way up in rcS.d?  I assume
there *must* be guidelines about what *needs* to go into pvops domU
/etc/{rc,init}.d/, and suggested orderings.
You can try to use the _netdev flag with all but / and /usr, that way
they''ll be mounted after networking is up. (The HP-UX admin in me is
going all ROFLMAO at this)
> And, finally, for my specific situation, does this help explain why my ext4
domU (which uses an LVM volume formatted with ext4 on dom0) won''t
reboot, but a ext3 one will?  Is ext4 pickier about times during the loading of
its driver?
Maybe yes. ext4 is less rubbish that ext3 and can finally detect some
internal errors.




Florian

On Feb 16, 2012, at 10:06 AM, Florian Heigl wrote:
> 2012/2/16 Qrux <qrux.qed@gmail.com>:
>> Thanks, Ian and Konrad, for your responses.  More questions inline...
> 
> This is really interesting. Unfortunately I still don''t fully
> understand how things are supposed to work in a pvops domU.
Florian, thanks for the response and interesting new questions.  I was able to
answer a lot of my own questions regarding these specific issues (and, ATM,
I''ve stopped caring about time accuracy, especially via NTP on domUs). 
I''ve pinpointed my "ext4 problem" to a problem that occurs
deeper in the system (that is related to time, but not at the userland level).

The message id is:

	A4D2CFC5-7B9D-4EBD-BDBC-58D6B35FCF6C@gmail.com

and the new subject is:

	Re: [Xen-devel] Xen domU Timekeeping (a.k.a TSC/HPET issues) 

and I posted to xen-devel, sometime this morning (around 10am UTC).  It
seems...more complex now.  While I am interested in the interaction of NTP with
a pvops domU/dom0, I care much more that my domU won''t even boot under
certain circumstances.  :)

	Q

I''m afraid I don''t know the answer to most of your questions
(hence I''m
afraid I''ve trimmed the quotes rather aggressively) but here''s
some of
what I do know.
> But, practically, is there a safe CPU configuration? 
I think that part of the problem here is that it is very hard to
determine this at the hardware level. There are at least 3 (if not more)
CPUID feature bits which say "no really, the TSC is good and safe to use
this time, you can rely on that" because they keep inventing new ways to
get it wrong.

[...]
> 
> Since September, I can''t find any further information about this
> issue. What is the state of this issue?  The inconsistency I see right
> now is this: in the July 2010 TSC discussion, a "Stefano
Stabellini"
> posted this:
> 
> ===> > /me wonders if timer_mode=1 is the default for xl?
> > Or only for xm?
> 
> no, it is not.
> Xl defaults to 0 [zero], I am going to change it right now.
> ===> 
> So, it seems like (at least as of July 2010), xl is defaulting to
> "timer_mode=1".  That is, assuming that the then-current
timer_mode is
> the same as present-day tsc_mode.
No, I believe they are different things.

tsc_mode is to do with the TSC, emulation vs direct exposure etc. Per
xen/include/asm-x86/time.h and (in recent xen-unstable) xl.cfg(5)

timer_mode is to do with the the way that timer interrupts are injected
into the guest. This is described in xen/include/public/hvm/params.h.
This isn''t documented in xl.cfg(5) because I couldn''t make
head nor tail
of the meaning of that header :-(
>   In addition, I''m assuming he was changing it from 0 (zero) to 1
> (one)--and not some other mode.  But,
> 
>         xen-4.1.2/docs/misc/tscmode.txt
Remember that he was referring to timer_mode not tsc_mode...
> says:
> 
>         "The default mode (tsc_mode==0) checks TSC-safeness of the
underlying
>         hardware on which the virtual machine is launched.  If it is
>         TSC-safe, rdtsc will execute at hardware speed; if it is not, rdtsc
>         will be emulated."
> 
> Which implies the default is always 0 (zero).  Which is it?
It seems that xl, in xen-unstable, defaults to:
	timer_mode = 1
	tsc_mode = 0
as does 4.1 as far as I can tell via code inspection.
> More importantly, is the solution to force tsc_mode=2?
IMHO this is safe in most situations unless you are running some sort of
workload (e.g. a well known database) which has stringent requirements
regarding the TSC for transactional consistency (hence the conservative
default).
>   If so, under what BIOS/xen-boot-params/dom0-boot-params conditions?
> And--please excuse my exasperation--but WTH does this have to do with
> ext3 versus ext4?  Is ext4 exquisitely sensitive to TSC/HPET
> "jumpiness" (if that''s even what''s happening)?
Sorry, I have no idea how/why the filesystem would be related to the
TSC.

It is possible you are actually seeing two bugs I suppose -- there have
been issues relating to ext4 and barriers in some kernel versions (I''m
afraid I don''t recall the details, the list archives ought to contain
something).

Ian.