freebsd stable - Oct 2007 - PAE Slowdown

Hey all,

I know that AMD64's the preferred way to run >4 gig systems, but I'm 
having a weird situation with 6.2-RELEASE-p8 and 6-STABLE as of last 
night.  When I compile the PAE kernel, my system performance drops 
like a rock.  It still boots and everything still runs, but for 
instance, running the Flops port my megaflops drop from the 950 
MFLOPS range to 4 MFLOPS.  It feels about as fast as a 486.

I'm not sure what I should try disabling.  I tried nodevice usb, but 
that didn't seem to change anything.  SMP and GENERIC kernels work 
fine.

CPU: Intel Core Duo 2 Quad 2.4ghz
Memory: 8 gig (4 2 gig dimms)
Swap: 16 gig partition

If I try to boot without ACPI disabled the kernel doesn't finish 
booting, it stops after ata7.



Full kernel message listing:

Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE-p8 #0: Mon Oct  8 11:04:03 CDT 2007
     root@ein.polycot.com:/usr/src/sys/i386/compile/PAE
ACPI APIC Table: <INTEL  DP965LT >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Quad CPU           @ 2.40GHz (2412.00-MHz 
686-class CPU)
   Origin = "GenuineIntel"  Id = 0x6f7  Stepping = 7
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  
Features2=0xe3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,<b9>,CX16,<b14>,<b15>>
   AMD Features=0x20100000<NX,LM>
   AMD Features2=0x1<LAHF>
   Cores per package: 4
real memory  = 9059696640 (8640 MB)
avail memory = 8340590592 (7954 MB)
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <INTEL DP965LT> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_perf0: <ACPI CPU Frequency Control> on cpu0
acpi_perf0: failed in PERF_STATUS attach
device_attach: acpi_perf0 attach returned 6
acpi_throttle0: <ACPI CPU Throttling> on cpu0
acpi_perf0: <ACPI CPU Frequency Control> on cpu0
acpi_perf0: failed in PERF_STATUS attach
device_attach: acpi_perf0 attach returned 6
acpi_button0: <Sleep Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
3ware device driver for 9000 series storage controllers, version: 3.60.03.006
twa0: <3ware 9000 series Storage Controller> port 0x3000-0x30ff mem 
0xe8000000-0xe9ffffff,0xea200000-0xea200fff irq 16 at device 0.0 on 
pci1
twa0: [GIANT-LOCKED]
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-4LPML, 
4 ports, Firmware FE9X 3.08.02.005, BIOS BE9X 3.08.00.002
pci0: <simple comms> at device 3.0 (no driver attached)
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 
0x40c0-0x40df mem 0xea300000-0xea31ffff,0xea320000-0xea320fff irq 20 
at device 25.0 on pci0
em0: Ethernet address: 00:19:d1:b0:d5:d0
pci0: <serial bus, USB> at device 26.0 (no driver attached)
pci0: <serial bus, USB> at device 26.1 (no driver attached)
pci0: <serial bus, USB> at device 26.7 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 28.1 on pci0
pci3: <ACPI PCI bus> on pcib3
atapci0: <GENERIC ATA controller> port 
0x2018-0x201f,0x2024-0x2027,0x2010-0x2017,0x2020-0x2023,0x2000-0x200f 
mem 0xea100000-0xea1001ff irq 17 at device 0.0 on pci3
ata2: <ATA channel 0> on atapci0
ata3: <ATA channel 1> on atapci0
pcib4: <ACPI PCI-PCI bridge> at device 28.2 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 28.3 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 28.4 on pci0
pci6: <ACPI PCI bus> on pcib6
pci0: <serial bus, USB> at device 29.0 (no driver attached)
pci0: <serial bus, USB> at device 29.1 (no driver attached)
pci0: <serial bus, USB> at device 29.2 (no driver attached)
pci0: <serial bus, USB> at device 29.7 (no driver attached)
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci7: <ACPI PCI bus> on pcib7
pci7: <display, VGA> at device 0.0 (no driver attached)
fwohci0: <Texas Instruments TSB43AB22/A> mem 
0xea084000-0xea0847ff,0xea080000-0xea083fff irq 19 at device 3.0 on 
pci7
fwohci0: OHCI version 1.10 (ROM=0)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:90:27:00:01:e7:67:5e
fwohci0: Phy 1394a available S400, 2 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 02:90:27:e7:67:5e
fwe0: Ethernet address: 02:90:27:e7:67:5e
fwe0: if_start running deferred for Giant
sbp0: <SBP-2/SCSI over FireWire> on firewire0
fwohci0: Initiate bus reset
fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
firewire0: bus manager 0 (me)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <Intel ICH8 SATA300 controller> port 
0x4438-0x443f,0x444c-0x444f,0x4430-0x4437,0x4448-0x444b,0x4410-0x441f,0x4400-0x440f
irq 19 at device 31.2 on pci0
ata4: <ATA channel 0> on atapci1
ata5: <ATA channel 1> on atapci1
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atapci2: <Intel ICH8 SATA300 controller> port 
0x4428-0x442f,0x4444-0x4447,0x4420-0x4427,0x4440-0x4443,0x40f0-0x40ff,0x40e0-0x40ef
irq 19 at device 31.5 on pci0
ata6: <ATA channel 0> on atapci2
ata7: <ATA channel 1> on atapci2
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcb000-0xccfff on isa0
ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
ata1 at port 0x170-0x177,0x376 irq 15 on isa0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250 or not responding
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2411997363 Hz quality 800
Timecounters tick every 1.000 msec
da0 at twa0 bus 0 target 0 lun 0
da0: <AMCC 9650SE-4LP DISK 3.08> Fixed Direct Access SCSI-5 device
da0: 100.000MB/s transfers
da0: 610330MB (1249955840 512 byte sectors: 255H 63S/T 77806C)
Trying to mount root from ufs:/dev/da0s1a


-- 

Jeff Kramer
jeffk@well.com
http://www.jeffkramer.org/

Jeff Kramer wrote:
> Hey all,
> 
> I know that AMD64's the preferred way to run >4 gig systems, but
I'm
> having a weird situation with 6.2-RELEASE-p8 and 6-STABLE as of last
> night.  When I compile the PAE kernel, my system performance drops like
> a rock.  It still boots and everything still runs, but for instance,
> running the Flops port my megaflops drop from the 950 MFLOPS range to 4
> MFLOPS.  It feels about as fast as a 486.
> 
> I'm not sure what I should try disabling.  I tried nodevice usb, but
> that didn't seem to change anything.  SMP and GENERIC kernels work
fine.
> 
> CPU: Intel Core Duo 2 Quad 2.4ghz
> Memory: 8 gig (4 2 gig dimms)
> Swap: 16 gig partition
> 
> If I try to boot without ACPI disabled the kernel doesn't finish
> booting, it stops after ata7.
Perhaps unrelated, but why don't you run amd64 version?  I think PAE is
a hack, for instance it does not allow processes to use more than 2GB
memory, while AMD64 (called EM64T by Intel implementation) provides much
more...

Cheers,
-- 
Xin LI <delphij@delphij.net>	http://www.delphij.net/
FreeBSD - The Power to Serve!

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 249 bytes
Desc: OpenPGP digital signature
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20071008/1b2e977c/signature.pgp

Jeff Kramer wrote:
> Hey all,
> 
> I know that AMD64's the preferred way to run >4 gig systems, but
I'm
> having a weird situation with 6.2-RELEASE-p8 and 6-STABLE as of last
> night.  When I compile the PAE kernel, my system performance drops like
> a rock.  It still boots and everything still runs, but for instance,
> running the Flops port my megaflops drop from the 950 MFLOPS range to 4
> MFLOPS.  It feels about as fast as a 486.
Does vmstat -i show unusually high interrupt rates?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 250 bytes
Desc: OpenPGP digital signature
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20071008/03776ed2/signature.pgp

More weirdness, if I take out 4 gig of ram and only run with 4 total, 
the PAE kernel works fine.


At 11:23 AM -0500 10/8/07, Jeff Kramer wrote:
>Hey all,
>
>I know that AMD64's the preferred way to run >4 gig systems, but
I'm
>having a weird situation with 6.2-RELEASE-p8 and 6-STABLE as of last 
>night.  When I compile the PAE kernel, my system performance drops 
>like a rock.  It still boots and everything still runs, but for 
>instance, running the Flops port my megaflops drop from the 950 
>MFLOPS range to 4 MFLOPS.  It feels about as fast as a 486.
>
>I'm not sure what I should try disabling.  I tried nodevice usb, but 
>that didn't seem to change anything.  SMP and GENERIC kernels work 
>fine.
>
>CPU: Intel Core Duo 2 Quad 2.4ghz
>Memory: 8 gig (4 2 gig dimms)
>Swap: 16 gig partition
-- 

Jeff Kramer
jeffk@well.com
http://www.jeffkramer.org/

Hello,



On 10/8/07, Jeff Kramer <jeffk@well.com> wrote:
> More weirdness, if I take out 4 gig of ram and only run with 4 total,
> the PAE kernel works fine.
Please don't top post, so we could track the thread :)

As Li said, you better for for AMD64 arch to enjoy the speed of your box.

-- 
Regards,

-Abdullah Ibn Hamad Al-Marri
Arab Portal
http://www.WeArab.Net/

Jeff Kramer <jeffk@well.com> writes:
> Hey all,
>
> I know that AMD64's the preferred way to run >4 gig systems, but
I'm
> having a weird situation with 6.2-RELEASE-p8 and 6-STABLE as of last
> night.  When I compile the PAE kernel, my system performance drops
> like a rock.  It still boots and everything still runs, but for
> instance, running the Flops port my megaflops drop from the 950 MFLOPS
> range to 4 MFLOPS.  It feels about as fast as a 486.
This may not be a PAE-related problem.  I say this because I noticed you
have the same MB I have:
> ACPI APIC Table: <INTEL  DP965LT >
Several Intel MBs, including the DP965LT, have a BIOS bug that rears
its head when you have 4G (or more) of memory installed, where the
BIOS sets the cache control registers incorrectly.  This cause a chunk
of your main memory (on my system, the chunk between 448MB and 512MB)
to be labeled uncachable, with the result being random slowdowns
whenever the kernel or user processes happen to touch memory in that
chunk.  This problem drove me crazy trying to figure out what the problem
was until I stumbled on this report on a Linux users' forum explaining 
the situation.

  http://forums.fedoraforum.org/showthread.php?t=157232

Fortunately, the workaround is fairly straightforward, adding an rc.d script
to twiddle the MTRRs.  Assuming this is your problem, if you could post the 
output of "memcontrol list" it should be possible to id which of the
entires
is bogus and needs to be removed.