I''m continuing to investigate the usage of rdtsc in userspace and whether there are programs "out there" that use it "unsafely" that might randomly break under Xen if rdtsc is not emulated, e.g. across a migration. Some have argued that nobody should use rdtsc and any programs that use rdtsc directly are "fundamentally broken" so the default for rdtsc emulation should be off. In measuring rdtsc usage in the kernel, both Jeremy and I noticed that compiling the kernel causes a large number of userland rdtsc''s. At first I thought that this meant that gcc was using rdtsc, but gcc sources do not show any use of rdtsc. Next I suspected bash, but it doesn''t either. Finally, it appears that the calls are occurring from glibc, and searching for uses of rdtsc, I found that glibc has its own implementation of clock_gettime that uses rdtsc directly! The manpage for clock_gettime ("man 3 clock_gettime") has a lengthy caveat about using clock_gettime on an SMP system BUT provides a mechanism to test to see if your SMP system is safe, clock_getcpuclockid(0). See for example (near the end): http://linux.die.net/man/3/clock_gettime But testing this on a system I know is unsafe, the test says my system is safe! Looking at the source code, I can see why... the test is a compile-time test, not a run-time test, and on x86 it appears that it will always pass. So I wrote a simple test program that uses clock_gettime(). It does indeed do userland rdtsc''s. So... any program that uses clock_gettime to approximate the passage of time is subject to break on Xen across a migration. Even if the programmer uses the prescribed method for testing "safeness", it is subject to break. True I still can''t point to a specific example of breakage, but this appears to be a much broader problem than code where the programmer has explicitly used asm-like constructs to directly utilize the rdtsc instruction, in spite of dire warnings to not use it. Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 10/23/09 15:51, Dan Magenheimer wrote:> In measuring rdtsc usage in the kernel, both Jeremy > and I noticed that compiling the kernel causes a > large number of userland rdtsc''s. At first I thought > that this meant that gcc was using rdtsc, but gcc > sources do not show any use of rdtsc. Next I suspected > bash, but it doesn''t either.I think the dynamic linker may use rdtsc as a piece of randomness for randomizing the addresses of libraries and other mappings.> Finally, it appears that > the calls are occurring from glibc, and searching for > uses of rdtsc, I found that glibc has its > own implementation of clock_gettime that uses rdtsc > directly! >When I run clock_gettime on my system it uses the vsyscall version (which is either a syscall or, with my recent patches, all in usermode). Using strace on your test program should show whether its really bypassing the syscall or not.> The manpage for clock_gettime ("man 3 clock_gettime") > has a lengthy caveat about using clock_gettime on > an SMP system BUT provides a mechanism to test to > see if your SMP system is safe, clock_getcpuclockid(0). >My manpages don''t have anything like that for clock_gettime().> See for example (near the end): > > http://linux.die.net/man/3/clock_gettime >That caveat is for CLOCK_PROCESS_CPUTIME_ID which doesn''t seem reasonable to count on for monotonicity (my manpages CLOCK_PROCESS_CPUTIME_ID just refer to as "High resolution per-process timer."). CLOCK_REALTIME or CLOCK_MONOTONIC is a different matter.> But testing this on a system I know is unsafe, the > test says my system is safe! Looking at the source > code, I can see why... the test is a compile-time > test, not a run-time test, and on x86 it appears that > it will always pass. > > So I wrote a simple test program that uses clock_gettime(). > It does indeed do userland rdtsc''s. >Using what CLOCK_?> So... any program that uses clock_gettime to approximate > the passage of time is subject to break on Xen across > a migration. Even if the programmer uses the prescribed > method for testing "safeness", it is subject to break. >I always see clock_gettime(CLOCK_MONOTONIC or CLOCK_REALTIME, ...) make a syscall or vsyscall. If you''re testing any of the thread-local times then I think its less interesting. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] > On 10/23/09 15:51, Dan Magenheimer wrote: > > In measuring rdtsc usage in the kernel, both Jeremy > > and I noticed that compiling the kernel causes a > > large number of userland rdtsc''s. At first I thought > > that this meant that gcc was using rdtsc, but gcc > > sources do not show any use of rdtsc. Next I suspected > > bash, but it doesn''t either. > > I think the dynamic linker may use rdtsc as a piece of randomness for > randomizing the addresses of libraries and other mappings.Interesting. I''ll look into that. I''ve collected some other data that might appear somewhat contradictory: I record the page offset of eip for each emulated rdtsc and there are about 28 different values collected, of which about 20 of them increment for each exec''ed process. One would think/hope that there would only be one place where randomness is generated for this kind of purpose, but I suppose if it is inlined, it''s hard to say. Anyway, thanks for the tip, I will dig further.> > Finally, it appears that > > the calls are occurring from glibc, and searching for > > uses of rdtsc, I found that glibc has its > > own implementation of clock_gettime that uses rdtsc > > directly! > > > > When I run clock_gettime on my system it uses the vsyscall version > (which is either a syscall or, with my recent patches, all in > usermode).I could be mistaken but I don''t think my test guest, which is EL5u2-32bit (2.6.18-based kernel) has vsyscall. At least the detection mechanisms that I am aware of ("sysctl -a | grep vsyscall" and "ls /proc/sys/kernel/vsys*" don''t work. Is there another way on older kernels?> Using strace on your test program should show whether its really > bypassing the syscall or not.You''re right! Strace does show syscalls for each clock_gettime(), and I was mistaken... these rdtsc''s are in the kernel, not userland.> > The manpage for clock_gettime ("man 3 clock_gettime") > > has a lengthy caveat about using clock_gettime on > > an SMP system BUT provides a mechanism to test to > > see if your SMP system is safe, clock_getcpuclockid(0). > > My manpages don''t have anything like that for clock_gettime().RHEL5 does. Though now I am not sure how to use the section 3 calls.> That caveat is for CLOCK_PROCESS_CPUTIME_ID which doesn''t seem > reasonable to count on for monotonicity (my manpages > CLOCK_PROCESS_CPUTIME_ID just refer to as "High resolution per-process > timer."). CLOCK_REALTIME or CLOCK_MONOTONIC is a different matter. > > Using what CLOCK_? > > I always see clock_gettime(CLOCK_MONOTONIC or CLOCK_REALTIME, > ...) make > a syscall or vsyscall. If you''re testing any of the > thread-local times > then I think its less interesting. > > JI''ll dig further next week. Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Just a followup on my attempt to track down existing uses of rdtsc in userspace in booting an OS (for at least one PV guest, 32-bit EL5u2). See below for previous status. I removed all the uses of rdtsc in glibc, which were mostly in the dynamic linker (used to gather stats for LD_DEBUG=statistics output). After that, there remains 5 different "offsets" collected for userland rdtsc (see below) and only one increments at every process launch. Three others have small totals (102, 8, 14) that don''t change after boot. The final/fifth is also small but seems to randomly increment by one about once every 20-40 seconds. While there remains a non-zero probability that any one of these remaining rdtsc''s could "break" on an ill-timed migration, I don''t think there''s any low-hanging fruit here to prove my point so am abandoning this investigation.> -----Original Message----- > From: Dan Magenheimer > Sent: Saturday, October 24, 2009 7:21 PM > To: Jeremy Fitzhardinge > Cc: Xen-Devel (E-mail); Keir Fraser; Tim Deegan; Avi Kivity; > Kurt Hackel > Subject: RE: rdtsc in userspace > > > > From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] > > On 10/23/09 15:51, Dan Magenheimer wrote: > > > In measuring rdtsc usage in the kernel, both Jeremy > > > and I noticed that compiling the kernel causes a > > > large number of userland rdtsc''s. At first I thought > > > that this meant that gcc was using rdtsc, but gcc > > > sources do not show any use of rdtsc. Next I suspected > > > bash, but it doesn''t either. > > > > I think the dynamic linker may use rdtsc as a piece of > randomness for > > randomizing the addresses of libraries and other mappings. > > Interesting. I''ll look into that. I''ve collected > some other data that might appear somewhat contradictory: > I record the page offset of eip for each emulated rdtsc > and there are about 28 different values collected, > of which about 20 of them increment for each > exec''ed process. One would think/hope that there > would only be one place where randomness is generated > for this kind of purpose, but I suppose if it is > inlined, it''s hard to say. > > Anyway, thanks for the tip, I will dig further. > > > > Finally, it appears that > > > the calls are occurring from glibc, and searching for > > > uses of rdtsc, I found that glibc has its > > > own implementation of clock_gettime that uses rdtsc > > > directly! > > > > > > > When I run clock_gettime on my system it uses the vsyscall version > > (which is either a syscall or, with my recent patches, all in > > usermode). > > I could be mistaken but I don''t think my test guest, > which is EL5u2-32bit (2.6.18-based kernel) has vsyscall. > At least the detection mechanisms that I am aware > of ("sysctl -a | grep vsyscall" and "ls /proc/sys/kernel/vsys*" > don''t work. Is there another way on older kernels? > > > Using strace on your test program should show whether its really > > bypassing the syscall or not. > > You''re right! Strace does show syscalls for > each clock_gettime(), and I was mistaken... these > rdtsc''s are in the kernel, not userland. > > > > The manpage for clock_gettime ("man 3 clock_gettime") > > > has a lengthy caveat about using clock_gettime on > > > an SMP system BUT provides a mechanism to test to > > > see if your SMP system is safe, clock_getcpuclockid(0). > > > > My manpages don''t have anything like that for clock_gettime(). > > RHEL5 does. Though now I am not sure how to use the > section 3 calls. > > > That caveat is for CLOCK_PROCESS_CPUTIME_ID which doesn''t seem > > reasonable to count on for monotonicity (my manpages > > CLOCK_PROCESS_CPUTIME_ID just refer to as "High resolution > per-process > > timer."). CLOCK_REALTIME or CLOCK_MONOTONIC is a different matter. > > > > Using what CLOCK_? > > > > I always see clock_gettime(CLOCK_MONOTONIC or CLOCK_REALTIME, > > ...) make > > a syscall or vsyscall. If you''re testing any of the > > thread-local times > > then I think its less interesting. > > > > J > > I''ll dig further next week. > > Dan > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Apparently Analagous Threads
- [patch V2 00/11] x86/vdso: Cleanups, simmplifications and CLOCK_TAI support
- [patch 00/11] x86/vdso: Cleanups, simmplifications and CLOCK_TAI support
- [patch 00/11] x86/vdso: Cleanups, simmplifications and CLOCK_TAI support
- "Failed to start domain..."
- [PATCH] Fix compilation on OS/2