Andrew Theurer
2005-Nov-02 19:10 UTC
[Xen-devel] 46% performance drop with change in glibc
I have been trying to track down a xen performance regression from Fedora core3 to core4 i386 and I think I might have narrowed it down. First some background info: I ran the SDET benchmark [1] and discovered that we have about a 45% regression after upgrading to Fedora core 4. I noticed xenoprofile showed ~20% in error_code on FC4, but only ~3.5% in error_code (should we even have 3%?!?) on FC3. At first I thought the change to gcc-4 might have something to do with it, so I tried gcc4 on FC3 -no regression. I noticed glibc was also different, so I upgraded FC3 from 2.3.3 to 2.3.5 -bingo. I am not sure what about glibc is causing this, but here are the two profiles: fc3 with glic-2.3.3: samples % app name symbol name 56097 22.8770 cc1 (no symbols) 22780 9.2900 troff (no symbols) 8646 3.5259 xen-unstable-syms error_code 7822 3.1899 grotty (no symbols) 5283 2.1545 libc-2.3.3.so _int_malloc 4914 2.0040 vmlinux-2.6.12-xen0-up buffered_rmqueue 4453 1.8160 bash (no symbols) 4372 1.7830 xen-unstable-syms get_page_from_l1e 3492 1.4241 xen-unstable-syms put_page_from_l1e 3338 1.3613 vmlinux-2.6.12-xen0-up system_call 3040 1.2397 xen-unstable-syms hypercall 2801 1.1423 xen-unstable-syms alloc_page_type 2799 1.1415 vmlinux-2.6.12-xen0-up zap_pte_range 2798 1.1411 libstdc++.so.6.0.3 (no symbols) 2761 1.1260 libc-2.3.3.so vfprintf 2637 1.0754 vmlinux-2.6.12-xen0-up do_no_page 2555 1.0420 libc-2.3.3.so __i686.get_pc_thunk.bx 2548 1.0391 vmlinux-2.6.12-xen0-up page_fault fc3 with glibc-2.3.5: samples % app name symbol name 133404 22.7469 xen-unstable-syms error_code 31411 5.3559 libc-2.3.5.so malloc 30316 5.1692 troff (no symbols) 20632 3.5180 libc-2.3.5.so vfprintf 16626 2.8349 xen-unstable-syms gpf_emulate_4gb 10639 1.8141 xen-unstable-syms do_general_protection 10105 1.7230 grotty (no symbols) 9837 1.6773 libc-2.3.5.so getwchar 8057 1.3738 xen-unstable-syms decode_register 7347 1.2527 libc-2.3.5.so iswgraph 7114 1.2130 vmlinux-2.6.12-xen0-up buffered_rmqueue 6926 1.1810 cc1 is_attribute_p 6679 1.1388 libc-2.3.5.so _int_malloc 6350 1.0827 xen-unstable-syms fixup_seg 5673 0.9673 xen-unstable-syms get_baselimit 5039 0.8592 bash (no symbols) 4957 0.8452 xen-unstable-syms get_page_from_l1e 4908 0.8369 xen-unstable-syms put_page_from_l1e 4576 0.7803 ld-2.3.5.so do_lookup_x 4322 0.7370 xen-unstable-syms hypercall 3915 0.6676 vmlinux-2.6.12-xen0-up system_call I assume something related to gpf_emulate_4gb... Anyone have an idea why this would cause such a regression? -Andrew [1] SDET described here: http://www.spec.org/osg/sdm91/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
The segment offset handling emulation is expensive. You need to either rename your TLS libraries away or use a glibc with the xen TLS patch. -Kip On 11/2/05, Andrew Theurer <habanero@us.ibm.com> wrote:> > I have been trying to track down a xen performance regression from Fedora > core3 to core4 i386 and I think I might have narrowed it down. First some > background info: I ran the SDET benchmark [1] and discovered that we have > about a 45% regression after upgrading to Fedora core 4. I noticed > xenoprofile showed ~20% in error_code on FC4, but only ~3.5% in error_code > (should we even have 3%?!?) on FC3. At first I thought the change to gcc-4 > might have something to do with it, so I tried gcc4 on FC3 -no regression. > I noticed glibc was also different, so I upgraded FC3 from 2.3.3 to 2.3.5 > -bingo. I am not sure what about glibc is causing this, but here are the > two profiles: > > fc3 with glic-2.3.3: > > samples % app name symbol name > 56097 22.8770 cc1 (no symbols) > 22780 9.2900 troff (no symbols) > 8646 3.5259 xen-unstable-syms error_code > 7822 3.1899 grotty (no symbols) > 5283 2.1545 libc-2.3.3.so <http://libc-2.3.3.so> _int_malloc > 4914 2.0040 vmlinux-2.6.12-xen0-up buffered_rmqueue > 4453 1.8160 bash (no symbols) > 4372 1.7830 xen-unstable-syms get_page_from_l1e > 3492 1.4241 xen-unstable-syms put_page_from_l1e > 3338 1.3613 vmlinux-2.6.12-xen0-up system_call > 3040 1.2397 xen-unstable-syms hypercall > 2801 1.1423 xen-unstable-syms alloc_page_type > 2799 1.1415 vmlinux-2.6.12-xen0-up zap_pte_range > 2798 1.1411 libstdc++.so.6.0.3 (no symbols) > 2761 1.1260 libc-2.3.3.so <http://libc-2.3.3.so> vfprintf > 2637 1.0754 vmlinux-2.6.12-xen0-up do_no_page > 2555 1.0420 libc-2.3.3.so <http://libc-2.3.3.so> __i686.get_pc_thunk.bx > 2548 1.0391 vmlinux-2.6.12-xen0-up page_fault > > fc3 with glibc-2.3.5: > > samples % app name symbol name > 133404 22.7469 xen-unstable-syms error_code > 31411 5.3559 libc-2.3.5.so <http://libc-2.3.5.so> malloc > 30316 5.1692 troff (no symbols) > 20632 3.5180 libc-2.3.5.so <http://libc-2.3.5.so> vfprintf > 16626 2.8349 xen-unstable-syms gpf_emulate_4gb > 10639 1.8141 xen-unstable-syms do_general_protection > 10105 1.7230 grotty (no symbols) > 9837 1.6773 libc-2.3.5.so <http://libc-2.3.5.so> getwchar > 8057 1.3738 xen-unstable-syms decode_register > 7347 1.2527 libc-2.3.5.so <http://libc-2.3.5.so> iswgraph > 7114 1.2130 vmlinux-2.6.12-xen0-up buffered_rmqueue > 6926 1.1810 cc1 is_attribute_p > 6679 1.1388 libc-2.3.5.so <http://libc-2.3.5.so> _int_malloc > 6350 1.0827 xen-unstable-syms fixup_seg > 5673 0.9673 xen-unstable-syms get_baselimit > 5039 0.8592 bash (no symbols) > 4957 0.8452 xen-unstable-syms get_page_from_l1e > 4908 0.8369 xen-unstable-syms put_page_from_l1e > 4576 0.7803 ld-2.3.5.so <http://ld-2.3.5.so> do_lookup_x > 4322 0.7370 xen-unstable-syms hypercall > 3915 0.6676 vmlinux-2.6.12-xen0-up system_call > > I assume something related to gpf_emulate_4gb... > Anyone have an idea why this would cause such a regression? > > -Andrew > > > [1] SDET described here: > http://www.spec.org/osg/sdm91/ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andrew Theurer
2005-Nov-02 19:23 UTC
Re: [Xen-devel] 46% performance drop with change in glibc
I had thought I moved tls to tls.disabled, but I''ll check again. -Andrew On Wednesday 02 November 2005 13:17, Kip Macy wrote:> The segment offset handling emulation is expensive. You need to either > rename your TLS libraries away or use a glibc with the xen TLS patch. > > -Kip > > On 11/2/05, Andrew Theurer <habanero@us.ibm.com> wrote: > > I have been trying to track down a xen performance regression from > > Fedora core3 to core4 i386 and I think I might have narrowed it down. > > First some background info: I ran the SDET benchmark [1] and discovered > > that we have about a 45% regression after upgrading to Fedora core 4. I > > noticed xenoprofile showed ~20% in error_code on FC4, but only ~3.5% in > > error_code (should we even have 3%?!?) on FC3. At first I thought the > > change to gcc-4 might have something to do with it, so I tried gcc4 on > > FC3 -no regression. I noticed glibc was also different, so I upgraded > > FC3 from 2.3.3 to 2.3.5 -bingo. I am not sure what about glibc is > > causing this, but here are the two profiles: > > > > fc3 with glic-2.3.3: > > > > samples % app name symbol name > > 56097 22.8770 cc1 (no symbols) > > 22780 9.2900 troff (no symbols) > > 8646 3.5259 xen-unstable-syms error_code > > 7822 3.1899 grotty (no symbols) > > 5283 2.1545 libc-2.3.3.so <http://libc-2.3.3.so> _int_malloc > > 4914 2.0040 vmlinux-2.6.12-xen0-up buffered_rmqueue > > 4453 1.8160 bash (no symbols) > > 4372 1.7830 xen-unstable-syms get_page_from_l1e > > 3492 1.4241 xen-unstable-syms put_page_from_l1e > > 3338 1.3613 vmlinux-2.6.12-xen0-up system_call > > 3040 1.2397 xen-unstable-syms hypercall > > 2801 1.1423 xen-unstable-syms alloc_page_type > > 2799 1.1415 vmlinux-2.6.12-xen0-up zap_pte_range > > 2798 1.1411 libstdc++.so.6.0.3 (no symbols) > > 2761 1.1260 libc-2.3.3.so <http://libc-2.3.3.so> vfprintf > > 2637 1.0754 vmlinux-2.6.12-xen0-up do_no_page > > 2555 1.0420 libc-2.3.3.so <http://libc-2.3.3.so> __i686.get_pc_thunk.bx > > 2548 1.0391 vmlinux-2.6.12-xen0-up page_fault > > > > fc3 with glibc-2.3.5: > > > > samples % app name symbol name > > 133404 22.7469 xen-unstable-syms error_code > > 31411 5.3559 libc-2.3.5.so <http://libc-2.3.5.so> malloc > > 30316 5.1692 troff (no symbols) > > 20632 3.5180 libc-2.3.5.so <http://libc-2.3.5.so> vfprintf > > 16626 2.8349 xen-unstable-syms gpf_emulate_4gb > > 10639 1.8141 xen-unstable-syms do_general_protection > > 10105 1.7230 grotty (no symbols) > > 9837 1.6773 libc-2.3.5.so <http://libc-2.3.5.so> getwchar > > 8057 1.3738 xen-unstable-syms decode_register > > 7347 1.2527 libc-2.3.5.so <http://libc-2.3.5.so> iswgraph > > 7114 1.2130 vmlinux-2.6.12-xen0-up buffered_rmqueue > > 6926 1.1810 cc1 is_attribute_p > > 6679 1.1388 libc-2.3.5.so <http://libc-2.3.5.so> _int_malloc > > 6350 1.0827 xen-unstable-syms fixup_seg > > 5673 0.9673 xen-unstable-syms get_baselimit > > 5039 0.8592 bash (no symbols) > > 4957 0.8452 xen-unstable-syms get_page_from_l1e > > 4908 0.8369 xen-unstable-syms put_page_from_l1e > > 4576 0.7803 ld-2.3.5.so <http://ld-2.3.5.so> do_lookup_x > > 4322 0.7370 xen-unstable-syms hypercall > > 3915 0.6676 vmlinux-2.6.12-xen0-up system_call > > > > I assume something related to gpf_emulate_4gb... > > Anyone have an idea why this would cause such a regression? > > > > -Andrew > > > > > > [1] SDET described here: > > http://www.spec.org/osg/sdm91/ > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andrew Theurer
2005-Nov-02 20:14 UTC
Re: [Xen-devel] 46% performance drop with change in glibc
Well, I thought I was losing my mind, but hopefully not. When I upgraded glibc on FC3, I did forget to move tls to tls.disabled. So, I moved it to tls.disabled, and the problem goes away. HOWEVER, on FC4, I always had tls moved to tls.disabled, and I am still getting this problem. So, on FC4, why would these libs not be in /lib/tls in the first place? /lib/tls[.disabled] on fc3: i486 i686 libc.so.6 libm.so.6 libpthread.so.0 librt.so.1 libthread_db.so.1 i586 libc-2.3.3.so libm-2.3.3.so libpthread-2.3.3.so librt-2.3.3.so libthread_db-1.0.so /lib/tls[.disabled] on fc4: i486 i586 i686 -Andrew On Wednesday 02 November 2005 13:23, Andrew Theurer wrote:> I had thought I moved tls to tls.disabled, but I''ll check again. > > -Andrew > > On Wednesday 02 November 2005 13:17, Kip Macy wrote: > > The segment offset handling emulation is expensive. You need to either > > rename your TLS libraries away or use a glibc with the xen TLS patch. > > > > -Kip > > > > On 11/2/05, Andrew Theurer <habanero@us.ibm.com> wrote: > > > I have been trying to track down a xen performance regression from > > > Fedora core3 to core4 i386 and I think I might have narrowed it down. > > > First some background info: I ran the SDET benchmark [1] and > > > discovered that we have about a 45% regression after upgrading to > > > Fedora core 4. I noticed xenoprofile showed ~20% in error_code on FC4, > > > but only ~3.5% in error_code (should we even have 3%?!?) on FC3. At > > > first I thought the change to gcc-4 might have something to do with > > > it, so I tried gcc4 on FC3 -no regression. I noticed glibc was also > > > different, so I upgraded FC3 from 2.3.3 to 2.3.5 -bingo. I am not sure > > > what about glibc is causing this, but here are the two profiles: > > > > > > fc3 with glic-2.3.3: > > > > > > samples % app name symbol name > > > 56097 22.8770 cc1 (no symbols) > > > 22780 9.2900 troff (no symbols) > > > 8646 3.5259 xen-unstable-syms error_code > > > 7822 3.1899 grotty (no symbols) > > > 5283 2.1545 libc-2.3.3.so <http://libc-2.3.3.so> _int_malloc > > > 4914 2.0040 vmlinux-2.6.12-xen0-up buffered_rmqueue > > > 4453 1.8160 bash (no symbols) > > > 4372 1.7830 xen-unstable-syms get_page_from_l1e > > > 3492 1.4241 xen-unstable-syms put_page_from_l1e > > > 3338 1.3613 vmlinux-2.6.12-xen0-up system_call > > > 3040 1.2397 xen-unstable-syms hypercall > > > 2801 1.1423 xen-unstable-syms alloc_page_type > > > 2799 1.1415 vmlinux-2.6.12-xen0-up zap_pte_range > > > 2798 1.1411 libstdc++.so.6.0.3 (no symbols) > > > 2761 1.1260 libc-2.3.3.so <http://libc-2.3.3.so> vfprintf > > > 2637 1.0754 vmlinux-2.6.12-xen0-up do_no_page > > > 2555 1.0420 libc-2.3.3.so <http://libc-2.3.3.so> > > > __i686.get_pc_thunk.bx 2548 1.0391 vmlinux-2.6.12-xen0-up page_fault > > > > > > fc3 with glibc-2.3.5: > > > > > > samples % app name symbol name > > > 133404 22.7469 xen-unstable-syms error_code > > > 31411 5.3559 libc-2.3.5.so <http://libc-2.3.5.so> malloc > > > 30316 5.1692 troff (no symbols) > > > 20632 3.5180 libc-2.3.5.so <http://libc-2.3.5.so> vfprintf > > > 16626 2.8349 xen-unstable-syms gpf_emulate_4gb > > > 10639 1.8141 xen-unstable-syms do_general_protection > > > 10105 1.7230 grotty (no symbols) > > > 9837 1.6773 libc-2.3.5.so <http://libc-2.3.5.so> getwchar > > > 8057 1.3738 xen-unstable-syms decode_register > > > 7347 1.2527 libc-2.3.5.so <http://libc-2.3.5.so> iswgraph > > > 7114 1.2130 vmlinux-2.6.12-xen0-up buffered_rmqueue > > > 6926 1.1810 cc1 is_attribute_p > > > 6679 1.1388 libc-2.3.5.so <http://libc-2.3.5.so> _int_malloc > > > 6350 1.0827 xen-unstable-syms fixup_seg > > > 5673 0.9673 xen-unstable-syms get_baselimit > > > 5039 0.8592 bash (no symbols) > > > 4957 0.8452 xen-unstable-syms get_page_from_l1e > > > 4908 0.8369 xen-unstable-syms put_page_from_l1e > > > 4576 0.7803 ld-2.3.5.so <http://ld-2.3.5.so> do_lookup_x > > > 4322 0.7370 xen-unstable-syms hypercall > > > 3915 0.6676 vmlinux-2.6.12-xen0-up system_call > > > > > > I assume something related to gpf_emulate_4gb... > > > Anyone have an idea why this would cause such a regression? > > > > > > -Andrew > > > > > > > > > [1] SDET described here: > > > http://www.spec.org/osg/sdm91/ > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xensource.com > > > http://lists.xensource.com/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony Liguori
2005-Nov-02 20:37 UTC
Re: [Xen-devel] 46% performance drop with change in glibc
Hi Andrew, My immediate guess would be TLS emulation. Do you not disable TLS on FC4? Regards, Anthony Liguori Andrew Theurer wrote:>I have been trying to track down a xen performance regression from Fedora >core3 to core4 i386 and I think I might have narrowed it down. First some >background info: I ran the SDET benchmark [1] and discovered that we have >about a 45% regression after upgrading to Fedora core 4. I noticed >xenoprofile showed ~20% in error_code on FC4, but only ~3.5% in error_code >(should we even have 3%?!?) on FC3. At first I thought the change to gcc-4 >might have something to do with it, so I tried gcc4 on FC3 -no regression. >I noticed glibc was also different, so I upgraded FC3 from 2.3.3 to 2.3.5 >-bingo. I am not sure what about glibc is causing this, but here are the >two profiles: > >fc3 with glic-2.3.3: > >samples % app name symbol name >56097 22.8770 cc1 (no symbols) >22780 9.2900 troff (no symbols) >8646 3.5259 xen-unstable-syms error_code >7822 3.1899 grotty (no symbols) >5283 2.1545 libc-2.3.3.so _int_malloc >4914 2.0040 vmlinux-2.6.12-xen0-up buffered_rmqueue >4453 1.8160 bash (no symbols) >4372 1.7830 xen-unstable-syms get_page_from_l1e >3492 1.4241 xen-unstable-syms put_page_from_l1e >3338 1.3613 vmlinux-2.6.12-xen0-up system_call >3040 1.2397 xen-unstable-syms hypercall >2801 1.1423 xen-unstable-syms alloc_page_type >2799 1.1415 vmlinux-2.6.12-xen0-up zap_pte_range >2798 1.1411 libstdc++.so.6.0.3 (no symbols) >2761 1.1260 libc-2.3.3.so vfprintf >2637 1.0754 vmlinux-2.6.12-xen0-up do_no_page >2555 1.0420 libc-2.3.3.so __i686.get_pc_thunk.bx >2548 1.0391 vmlinux-2.6.12-xen0-up page_fault > >fc3 with glibc-2.3.5: > >samples % app name symbol name >133404 22.7469 xen-unstable-syms error_code >31411 5.3559 libc-2.3.5.so malloc >30316 5.1692 troff (no symbols) >20632 3.5180 libc-2.3.5.so vfprintf >16626 2.8349 xen-unstable-syms gpf_emulate_4gb >10639 1.8141 xen-unstable-syms do_general_protection >10105 1.7230 grotty (no symbols) >9837 1.6773 libc-2.3.5.so getwchar >8057 1.3738 xen-unstable-syms decode_register >7347 1.2527 libc-2.3.5.so iswgraph >7114 1.2130 vmlinux-2.6.12-xen0-up buffered_rmqueue >6926 1.1810 cc1 is_attribute_p >6679 1.1388 libc-2.3.5.so _int_malloc >6350 1.0827 xen-unstable-syms fixup_seg >5673 0.9673 xen-unstable-syms get_baselimit >5039 0.8592 bash (no symbols) >4957 0.8452 xen-unstable-syms get_page_from_l1e >4908 0.8369 xen-unstable-syms put_page_from_l1e >4576 0.7803 ld-2.3.5.so do_lookup_x >4322 0.7370 xen-unstable-syms hypercall >3915 0.6676 vmlinux-2.6.12-xen0-up system_call > >I assume something related to gpf_emulate_4gb... >Anyone have an idea why this would cause such a regression? > >-Andrew > > >[1] SDET described here: >http://www.spec.org/osg/sdm91/ > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ulrich Drepper
2005-Nov-02 21:31 UTC
Re: [Xen-devel] 46% performance drop with change in glibc
Andrew Theurer wrote:> So, on FC4, why > would these libs not be in /lib/tls in the first place?Too l33t to read release notes? There is no tls subdir since NPTL is the default (and LT is completely gone in FC5). The code in /lib is the TLS code using the correct ABI. The crippled ABI is supported by the code in /lib/i686/nosegneg. This code is *automatically* used if the kernel is compiled to match the glibc version. This happens using the linux-2.6-xen-vdso-note.patch patch in Red Hat''s kernel RPMs. -- ➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> There is no tls subdir since NPTL is the default (and LT is > completely gone in FC5). The code in /lib is the TLS code > using the correct ABI. > > The crippled ABI is supported by the code in > /lib/i686/nosegneg. This code is *automatically* used if the > kernel is compiled to match the glibc version. This happens using the > > linux-2.6-xen-vdso-note.patchI believe we have this patch in the xen repo, but I haven''t ever verified whether the right thing happens under FC4. Andrew: please could you check you have the xen glibc rpm installed and verify that the glibc selection is working using one of our kernels. Thanks, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andrew Theurer
2005-Nov-07 21:09 UTC
Re: [Xen-devel] 46% performance drop with change in glibc
On Friday 04 November 2005 10:08, you wrote:> > There is no tls subdir since NPTL is the default (and LT is > > completely gone in FC5). The code in /lib is the TLS code > > using the correct ABI. > > > > The crippled ABI is supported by the code in > > /lib/i686/nosegneg. This code is *automatically* used if the > > kernel is compiled to match the glibc version. This happens using the > > > > linux-2.6-xen-vdso-note.patch > > I believe we have this patch in the xen repo, but I haven''t ever > verified whether the right thing happens under FC4.I am not seeing this patch in ./patches/linux-2.6.12 directory. Is it somewhere else?> Andrew: please could you check you have the xen glibc rpm installed and > verify that the glibc selection is working using one of our kernels.I am pretty sure I have the xen glibc, but a I can''t verify it''s working until I get a clean build of a patched xen kernel. I suspect we need a little more than just the linux-2.6-xen-vdso-note patch to build cleanly: kernel/built-in.o(.text+0x2671f): In function `kallsyms_lookup'': : undefined reference to `in_gate_area_no_task'' mm/built-in.o(.text+0x12f5f): In function `get_user_pages'': : undefined reference to `in_gate_area'' mm/built-in.o(.text+0x12f7b): In function `get_user_pages'': : undefined reference to `get_gate_vma'' fs/built-in.o(.text+0x34608): In function `show_map'': task_mmu.c: undefined reference to `get_gate_vma'' fs/built-in.o(.text+0x34785): In function `m_start'': task_mmu.c: undefined reference to `get_gate_vma'' fs/built-in.o(.text+0x34887): In function `m_stop'': task_mmu.c: undefined reference to `get_gate_vma'' fs/built-in.o(.text+0x348e6): In function `m_next'': task_mmu.c: undefined reference to `get_gate_vma'' _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > > The crippled ABI is supported by the code in /lib/i686/nosegneg. > > > This code is *automatically* used if the kernel is > compiled to match > > > the glibc version. This happens using the > > > > > > linux-2.6-xen-vdso-note.patch > > > > I believe we have this patch in the xen repo, but I haven''t ever > > verified whether the right thing happens under FC4. > > I am not seeing this patch in ./patches/linux-2.6.12 > directory. Is it somewhere else?It should be in the main tree. The appended patch was included some time ago, but I''ve never tested it to see whether it works since I don''t generally run bleeding edge Fedora. Is there some piece of the patch missing? Thanks, Ian --- linux-2.6/arch/xen/i386/kernel/vsyscall-note.S +++ linux-2.6/arch/xen/i386/kernel/vsyscall-note.S @@ -0,0 +1,32 @@ +/* + * This supplies .note.* sections to go into the PT_NOTE inside the vDSO text. + * Here we can supply some information useful to userland. + * First we get the vanilla i386 note that supplies the kernel version info. + */ + +#include "../../../i386/kernel/vsyscall-note.S" + +/* + * Now we add a special note telling glibc''s dynamic linker a fake +hardware + * flavor that it will use to choose the search path for libraries in +the + * same way it uses real hardware capabilities like "mmx". + * We supply "nosegneg" as the fake capability, to indicate that we + * do not like negative offsets in instructions using segment +overrides, + * since we implement those inefficiently. This makes it possible to + * install libraries optimized to avoid those access patterns in +someplace + * like /lib/i686/tls/nosegneg. Note that an /etc/ld.so.conf.d/file + * corresponding to the bits here is needed to make ldconfig work right. + * It should contain: + * hwcap 0 nosegneg + * to match the mapping of bit to name that we give here. + */ +#define NOTE_KERNELCAP_BEGIN(ncaps, mask) \ + ASM_ELF_NOTE_BEGIN(".note.kernelcap", "a", "GNU", 2) \ + .long ncaps, mask +#define NOTE_KERNELCAP(bit, name) \ + .byte bit; .asciz name +#define NOTE_KERNELCAP_END ASM_ELF_NOTE_END + +NOTE_KERNELCAP_BEGIN(1, 1) +NOTE_KERNELCAP(1, "nosegneg") /* Change 1 back to 0 when glibc is +fixed! */ NOTE_KERNELCAP_END _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andrew Theurer
2005-Nov-08 13:05 UTC
Re: [Xen-devel] 46% performance drop with change in glibc
Ian Pratt wrote:>>>>The crippled ABI is supported by the code in /lib/i686/nosegneg. >>>>This code is *automatically* used if the kernel is >>>> >>compiled to match >> >>>>the glibc version. This happens using the >>>> >>>>linux-2.6-xen-vdso-note.patch >>>> >>>I believe we have this patch in the xen repo, but I haven''t ever >>>verified whether the right thing happens under FC4. >>> >>I am not seeing this patch in ./patches/linux-2.6.12 >>directory. Is it somewhere else? >> > >It should be in the main tree. The appended patch was included some time >ago, but I''ve never tested it to see whether it works since I don''t >generally run bleeding edge Fedora. > >Is there some piece of the patch missing? >No, it''s all there. Only part I needed was the change to ld.so.conf. So, anyone with FC4, you might want to make sure you have the entry "hwcap 0 nosegneg". It looks like the performance regression has disappeared now. Thanks all for you help. -Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> >It should be in the main tree. The appended patch was included some > >time ago, but I''ve never tested it to see whether it works since I > >don''t generally run bleeding edge Fedora. > > > >Is there some piece of the patch missing? > > > No, it''s all there. Only part I needed was the change to > ld.so.conf. > So, anyone with FC4, you might want to make sure you have the > entry "hwcap 0 nosegneg". It looks like the performance > regression has disappeared now. Thanks all for you help.Perhaps we should have the install script insert a file called /etc/ld.so.conf.d/xen containing "hwcap 0 nosegneg" ? Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andrew Theurer
2005-Nov-08 14:57 UTC
Re: [Xen-devel] 46% performance drop with change in glibc
On Tuesday 08 November 2005 07:17, Ian Pratt wrote:> > >It should be in the main tree. The appended patch was included some > > > > > >time ago, but I''ve never tested it to see whether it works since I > > >don''t generally run bleeding edge Fedora. > > > > > >Is there some piece of the patch missing? > > > > No, it''s all there. Only part I needed was the change to > > ld.so.conf. > > So, anyone with FC4, you might want to make sure you have the > > entry "hwcap 0 nosegneg". It looks like the performance > > regression has disappeared now. Thanks all for you help. > > Perhaps we should have the install script insert a file called > /etc/ld.so.conf.d/xen containing "hwcap 0 nosegneg" ?Assuming most/all distros are doing it this way. I do recall a beta version of openSUSE just moved the tls dir in an rc script, but maybe they are doing something different now. Anyone involved with Linux distributions care to comment? -Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ulrich Drepper
2005-Nov-08 16:53 UTC
Re: [Xen-devel] 46% performance drop with change in glibc
Andrew Theurer wrote:>> Perhaps we should have the install script insert a file called >> /etc/ld.so.conf.d/xen containing "hwcap 0 nosegneg" ? > > Anyone involved with Linux distributions care to comment?Installing the xen kernel RPM from FC automatically adds the appropriate line to ld.so.conf. -- ➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kurt Garloff
2005-Nov-08 19:30 UTC
Re: [Xen-devel] 46% performance drop with change in glibc
Hi, On Tue, Nov 08, 2005 at 09:57:05AM -0500, Andrew Theurer wrote:> Assuming most/all distros are doing it this way. I do recall a beta version > of openSUSE just moved the tls dir in an rc script,Yes, indeed, that was done prior to SUSE Linux 10.0: just detect whether you''re running under Xen and if so move away /lib/tls.> but maybe they are doing something different now.In openSUSE Linux 10.0, we just yanked the negative segment references out of gcc + glibc. The effort to have two versions just does not justify the non-measurable performance / code-density gains. My recommendation would be to do that everywhere. Best, -- Kurt Garloff, Director SUSE Labs, Novell Inc. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Katz
2005-Nov-09 02:02 UTC
RE: [Xen-devel] 46% performance drop with change in glibc
On Tue, 2005-11-08 at 13:17 +0000, Ian Pratt wrote:> > >It should be in the main tree. The appended patch was included some > > >time ago, but I''ve never tested it to see whether it works since I > > >don''t generally run bleeding edge Fedora. > > > > > >Is there some piece of the patch missing? > > > > > No, it''s all there. Only part I needed was the change to > > ld.so.conf. > > So, anyone with FC4, you might want to make sure you have the > > entry "hwcap 0 nosegneg". It looks like the performance > > regression has disappeared now. Thanks all for you help. > > Perhaps we should have the install script insert a file called > /etc/ld.so.conf.d/xen containing "hwcap 0 nosegneg" ?Sounds sane. As Uli said, this mirrors what happens when our xen kernel package is installed. Jeremy _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel