Jay Libove
2007-Apr-18 12:36 UTC
[Bridge] recent crashes? Linux kernel 2.6.18-1.2239.fc5 (Linux Fedora Core 5)
Recently, possibly since the most recent one or two kernel upgrades in
Linux Fedora Core 5, my system has been crashing with a kernel panic.
The panic prevents anything useful from being written to any log files,
but on the crash trace left on the screen each time I always see
something which looks like it is in the bridge code.
This system is based on Linux Fedora Core 5, and currently has the
following showing up with "kernel" in the name in the output of
"rpm -q
-a":
kernel-2.6.18-1.2200.fc5
kernel-2.6.18-1.2239.fc5
I'm a bit concerned about the fact that more than one single kernel
instance appears in this output! Not sure that I've ever seen that
before... "dmesg -a" shows that it is the .2239 version of the kernel
which is booting.
The system has 576MB of RAM (dmesg -a shows 575MB, I think it shares 1MB
with an on-board video module; this thing never runs in graphics mode
anyway). It has a Celeron 766MHz CPU.
I've pasted the output of "dmesg -a" at the bottom of this email,
in
case it provides any useful diagnostic information.
I know this is very little information. I'm hoping that what is
happening here is an incompatibility between a recent Fedora Core 5
kernel upgrade and the bridge code, and that others have seen it, and
someone knows a fix / workaround; or that there is some kind of attack
going on which is randomly hitting my system and triggering the panic,
and, again someone has seen it and has a fix / workaround.
Thanks very much!
-Jay
Linux version 2.6.18-1.2239.fc5
(brewbuilder@hs20-bc1-7.build.redhat.com) (gcc version 4.1.1 20060525
(Red Hat 4.1.1-1)) #1 Fri Nov 10 13:04:06 EST 2006
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 0000000023ff0000 (usable)
BIOS-e820: 0000000023ff0000 - 0000000023ff3000 (ACPI NVS)
BIOS-e820: 0000000023ff3000 - 0000000024000000 (ACPI data)
BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
0MB HIGHMEM available.
575MB LOWMEM available.
Using x86 segment limits to approximate NX protection
On node 0 totalpages: 147440
DMA zone: 4096 pages, LIFO batch:0
Normal zone: 143344 pages, LIFO batch:31
DMI 2.3 present.
IO/L-APIC disabled because your old system seems to be old
overwrite with "apic"
ACPI: RSDP (v000 ABIT ) @ 0x000f6fd0
ACPI: RSDT (v001 ABIT AWRDACPI 0x42302e31 AWRD 0x00000000) @
0x23ff3000
ACPI: FADT (v001 ABIT AWRDACPI 0x42302e31 AWRD 0x00000000) @
0x23ff3040
ACPI: DSDT (v001 ABIT AWRDACPI 0x00001000 MSFT 0x0100000c) @
0x00000000
ACPI: PM-Timer IO Port: 0x4008
Allocating PCI resources starting at 30000000 (gap: 24000000:dbff0000)
Detected 768.433 MHz processor.
Built 1 zonelists. Total pages: 147440
Kernel command line: ro root=LABEL=/1
mapped APIC to ffffd000 (01487000)
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c075a000 soft=c0759000
PID hash table entries: 4096 (order: 12, 16384 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 578960k/589760k available (2040k kernel code, 10240k reserved,
798k data, 220k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode...
Ok.
Calibrating delay using timer specific routine.. 1537.46 BogoMIPS
(lpj=768733)
Security Framework v1.0.0 initialized
SELinux: Initializing.
SELinux: Starting in permissive mode
selinux_register_security: Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 512
CPU: After generic identify, caps: 0383f9ff 00000000 00000000 00000000
00000000 00000000 00000000
CPU: After vendor identify, caps: 0383f9ff 00000000 00000000 00000000
00000000 00000000 00000000
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 128K
CPU: After all inits, caps: 0383f1ff 00000000 00000000 00000040 00000000
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU: Intel Celeron (Coppermine) stepping 06
Checking 'hlt' instruction... OK.
ACPI: Core revision 20060707
ACPI: setting ELCR to 0200 (from 0e20)
checking if image is initramfs... it is
Freeing initrd memory: 918k freed
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: PCI BIOS revision 2.10 entry at 0xfb3b0, last bus=2
PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: Interpreter enabled
ACPI: Using PIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Probing PCI hardware (bus 00)
ACPI: Assume root bridge [\_SB_.PCI0] bus is 0
* Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
* this clock source is slow. Consider trying other clock sources
PCI quirk: region 4000-403f claimed by PIIX4 ACPI
PCI quirk: region 5000-500f claimed by PIIX4 SMB
Boot video device is 0000:00:0f.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 10 11 12 14 15) *9
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 *10 11 12 14 15)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 13 devices
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq". If it helps,
post a
report
PCI: Bridge: 0000:00:01.0
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Bridge: 0000:00:0d.0
IO window: 9000-9fff
MEM window: d4000000-d5ffffff
PREFETCH window: 30000000-300fffff
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 8, 1310720 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
apm: BIOS version 1.2 Flags 0x07 (Driver version 1.16ac)
apm: overridden by ACPI.
audit: initializing netlink socket (disabled)
audit(1164826911.852:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
SELinux: Registering netfilter hooks
Initializing Cryptographic API
ksign: Installing public key data
Loading keyring
- Added public key 481F17AAEB289250
- User ID: Red Hat, Inc. (Kernel Module GPG key)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Limiting direct PCI/PCI transfers.
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
ACPI: Fan [FAN0] (on)
ACPI: CPU0 (power states: C1[C1] C2[C2])
ACPI: Processor [CPU0] (supports 2 throttling states)
isapnp: Scanning for PnP cards...
isapnp: No Plug & Play device found
Real Time Clock Driver v1.12ac
Non-volatile memory driver v1.2
Linux agpgart interface v0.101 (c) Dave Jones
agpgart: Detected an Intel 440BX Chipset.
agpgart: AGP aperture is 64M @ 0xd0000000
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with
idebus=xx
PIIX4: IDE controller at PCI slot 0000:00:07.1
PIIX4: chipset revision 1
PIIX4: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:pio
ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:DMA, hdd:pio
Probing IDE interface ide0...
hda: SAMSUNG SP1213N, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: DVDROM 12X, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
HPT374: IDE controller at PCI slot 0000:00:0b.0
ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 5
PCI: setting IRQ 5 as level-triggered
ACPI: PCI Interrupt 0000:00:0b.0[A] -> Link [LNKC] -> GSI 5 (level, low)
-> IRQ 5
HPT374: chipset revision 7
HPT374: 100% native mode on irq 5
HPT37X: using 33MHz PCI clock
ide2: BM-DMA at 0xb000-0xb007, BIOS settings: hde:pio, hdf:pio
HPT37X: using 33MHz PCI clock
ide3: BM-DMA at 0xb008-0xb00f, BIOS settings: hdg:pio, hdh:pio
ACPI: PCI Interrupt 0000:00:0b.1[A] -> Link [LNKC] -> GSI 5 (level, low)
-> IRQ 5
HPT37X: using 33MHz PCI clock
ide4: BM-DMA at 0xc400-0xc407, BIOS settings: hdi:pio, hdj:pio
HPT37X: using 33MHz PCI clock
ide5: BM-DMA at 0xc408-0xc40f, BIOS settings: hdk:pio, hdl:pio
Probing IDE interface ide2...
Probing IDE interface ide3...
Probing IDE interface ide4...
Probing IDE interface ide5...
Probing IDE interface ide2...
Probing IDE interface ide3...
Probing IDE interface ide4...
Probing IDE interface ide5...
hda: max request size: 512KiB
hda: 234493056 sectors (120060 MB) w/8192KiB Cache, CHS=16383/255/63,
UDMA(33)
hda: cache flushes supported
hda: hda1 hda2 hda3
ide-floppy driver 0.99.newide
usbcore: registered new driver libusual
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
TCP bic registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
Using IPI Shortcut mode
Time: tsc clocksource has been installed.
ACPI: (supports S0 S1 S4 S5)
Time: acpi_pm clocksource has been installed.
Freeing unused kernel memory: 220k freed
Write protecting the kernel read-only data: 377k
input: AT Translated Set 2 keyboard as /class/input/input0
input: ImExPS/2 Generic Explorer Mouse as /class/input/input1
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting. Commit interval 5 seconds
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
security: 3 users, 6 roles, 1481 types, 152 bools, 1 sens, 256 cats
security: 58 classes, 43474 rules
SELinux: Completing initialization.
SELinux: Setting up existing superblocks.
SELinux: initialized (dev hda2, type ext3), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev debugfs, type debugfs), uses genfs_contexts
SELinux: initialized (dev selinuxfs, type selinuxfs), uses
genfs_contexts
SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs
SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses
genfs_contexts
SELinux: initialized (dev devpts, type devpts), uses transition SIDs
SELinux: initialized (dev eventpollfs, type eventpollfs), uses task SIDs
SELinux: initialized (dev inotifyfs, type inotifyfs), uses
genfs_contexts
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev futexfs, type futexfs), uses genfs_contexts
SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
SELinux: initialized (dev sockfs, type sockfs), uses task SIDs
SELinux: initialized (dev proc, type proc), uses genfs_contexts
SELinux: initialized (dev bdev, type bdev), uses genfs_contexts
SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts
SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
audit(1164826922.708:2): policy loaded auid=4294967295
SELinux: initialized (dev usbfs, type usbfs), uses genfs_contexts
hdc: ATAPI DVD-ROM drive, 512kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
input: PC Speaker as /class/input/input2
piix4_smbus 0000:00:07.3: Found 0000:00:07.3 device
Linux Tulip driver version 1.1.13 (May 11, 2002)
ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
PCI: setting IRQ 11 as level-triggered
ACPI: PCI Interrupt 0000:02:04.0[A] -> Link [LNKB] -> GSI 11 (level,
low) -> IRQ 11
tulip0: EEPROM default media type Autosense.
tulip0: Index #0 - Media 10baseT (#0) described by a 21140 non-MII (0)
block.
tulip0: Index #1 - Media 100baseTx (#3) described by a 21140 non-MII
(0) block.
tulip0: Index #2 - Media 10baseT-FDX (#4) described by a 21140 non-MII
(0) block.
tulip0: Index #3 - Media 100baseTx-FDX (#5) described by a 21140
non-MII (0) block.
eth0: Digital DS21140 Tulip rev 34 at e481a000, 00:00:BC:11:56:D7, IRQ
11.
ACPI: PCI Interrupt 0000:02:05.0[A] -> Link [LNKC] -> GSI 5 (level, low)
-> IRQ 5
tulip1: EEPROM default media type Autosense.
tulip1: Index #0 - Media 10baseT (#0) described by a 21140 non-MII (0)
block.
tulip1: Index #1 - Media 100baseTx (#3) described by a 21140 non-MII
(0) block.
tulip1: Index #2 - Media 10baseT-FDX (#4) described by a 21140 non-MII
(0) block.
tulip1: Index #3 - Media 100baseTx-FDX (#5) described by a 21140
non-MII (0) block.
eth1: Digital DS21140 Tulip rev 34 at e483c000, 00:00:BC:11:56:D6, IRQ
5.
ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10
PCI: setting IRQ 10 as level-triggered
ACPI: PCI Interrupt 0000:02:06.0[A] -> Link [LNKD] -> GSI 10 (level,
low) -> IRQ 10
tulip2: EEPROM default media type Autosense.
tulip2: Index #0 - Media 10baseT (#0) described by a 21140 non-MII (0)
block.
tulip2: Index #1 - Media 100baseTx (#3) described by a 21140 non-MII
(0) block.
tulip2: Index #2 - Media 10baseT-FDX (#4) described by a 21140 non-MII
(0) block.
tulip2: Index #3 - Media 100baseTx-FDX (#5) described by a 21140
non-MII (0) block.
eth2: Digital DS21140 Tulip rev 34 at e483e000, 00:00:BC:11:56:D5, IRQ
10.
ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11
ACPI: PCI Interrupt 0000:02:07.0[A] -> Link [LNKA] -> GSI 11 (level,
low) -> IRQ 11
tulip3: EEPROM default media type Autosense.
tulip3: Index #0 - Media 10baseT (#0) described by a 21140 non-MII (0)
block.
tulip3: Index #1 - Media 100baseTx (#3) described by a 21140 non-MII
(0) block.
tulip3: Index #2 - Media 10baseT-FDX (#4) described by a 21140 non-MII
(0) block.
tulip3: Index #3 - Media 100baseTx-FDX (#5) described by a 21140
non-MII (0) block.
eth3: Digital DS21140 Tulip rev 34 at e4840000, 00:00:BC:11:56:D4, IRQ
11.
Floppy drive(s): fd0 is 1.44M
USB Universal Host Controller Interface driver v3.0
ACPI: PCI Interrupt 0000:00:07.2[D] -> Link [LNKD] -> GSI 10 (level,
low) -> IRQ 10
uhci_hcd 0000:00:07.2: UHCI Host Controller
uhci_hcd 0000:00:07.2: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:07.2: irq 10, io base 0x0000c800
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
FDC 0 is a post-1991 82077
parport: PnPBIOS parport detected.
parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
lp0: using parport0 (interrupt-driven).
lp0: console ready
ACPI: Power Button (FF) [PWRF]
ACPI: Power Button (CM) [PWRB]
ibm_acpi: ec object not found
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
device-mapper: ioctl: 4.7.0-ioctl (2006-06-24) initialised:
dm-devel@redhat.com
EXT3 FS on hda2, internal journal
kjournald starting. Commit interval 5 seconds
EXT3 FS on hda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev hda1, type ext3), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
Adding 1124540k swap on /dev/hda3. Priority:-1 extents:1
across:1124540k
SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses
genfs_contexts
audit(1164844962.313:3): avc: denied { append } for pid=1557
comm="syslogd" name="messages" dev=hda2 ino=163850
scontext=system_u:system_r:syslogd_t:s0
tcontext=system_u:object_r:root_t:s0 tclass=file
audit(1164844962.313:4): avc: denied { ioctl } for pid=1557
comm="syslogd" name="messages" dev=hda2 ino=163850
scontext=system_u:system_r:syslogd_t:s0
tcontext=system_u:object_r:root_t:s0 tclass=file
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
ADDRCONF(NETDEV_UP): eth2: link is not ready
IPv6 over IPv4 tunneling driver
SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses
genfs_contexts
SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
ip_tables: (C) 2000-2006 Netfilter Core Team
arp_tables: (C) 2002 David S. Miller
Ebtables v2.0 registered
Netfilter messages via NETLINK v0.30.
ip_conntrack version 2.4 (4607 buckets, 36856 max) - 228 bytes per
conntrack
Bridge firewalling registered
eth3: Promiscuous mode enabled.
device eth3 entered promiscuous mode
audit(1164844969.791:5): dev=eth3 prom=256 old_prom=0 auid=4294967295
br0: port 1(eth3) entering learning state
audit(1164844970.479:6): avc: denied { append } for pid=1557
comm="syslogd" name="messages" dev=hda2 ino=163850
scontext=system_u:system_r:syslogd_t:s0
tcontext=system_u:object_r:root_t:s0 tclass=file
audit(1164844973.658:7): avc: denied { name_bind } for pid=2142
comm="httpd" src=563 scontext=system_u:system_r:httpd_t:s0
tcontext=system_u:object_r:reserved_port_t:s0 tclass=tcp_socket
audit(1164844973.660:8): avc: denied { name_bind } for pid=2142
comm="httpd" src=143 scontext=system_u:system_r:httpd_t:s0
tcontext=system_u:object_r:pop_port_t:s0 tclass=tcp_socket
eth3: no IPv6 routers present
eth1: no IPv6 routers present
eth0: no IPv6 routers present
audit(1164844976.694:9): avc: denied { write } for pid=2172
comm="smbd" name="krb5.conf" dev=hda2 ino=14877755
scontext=system_u:system_r:smbd_t:s0 tcontext=root:object_r:etc_t:s0
tclass=file
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
br0: no IPv6 routers present
audit(1164844980.421:10): avc: denied { append } for pid=1557
comm="syslogd" name="messages" dev=hda2 ino=163850
scontext=system_u:system_r:syslogd_t:s0
tcontext=system_u:object_r:root_t:s0 tclass=file
audit(1164844980.543:11): avc: denied { write } for pid=2287
comm="ifconfig" name="openvpn-status1.log" dev=hda2
ino=16089668
scontext=system_u:system_r:ifconfig_t:s0 tcontext=root:object_r:etc_t:s0
tclass=file
audit(1164844980.543:12): avc: denied { read write } for pid=2287
comm="ifconfig" name="[6565]" dev=sockfs ino=6565
scontext=system_u:system_r:ifconfig_t:s0
tcontext=system_u:system_r:initrc_t:s0 tclass=udp_socket
audit(1164844981.105:13): avc: denied { read write } for pid=2333
comm="ifconfig" name="[6688]" dev=sockfs ino=6688
scontext=system_u:system_r:ifconfig_t:s0
tcontext=system_u:system_r:initrc_t:s0 tclass=tcp_socket
br0: topology change detected, propagating
br0: port 1(eth3) entering forwarding state
audit(1164845166.512:14): avc: denied { append } for pid=1557
comm="syslogd" name="messages" dev=hda2 ino=163850
scontext=system_u:system_r:syslogd_t:s0
tcontext=system_u:object_r:root_t:s0 tclass=file
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.linux-foundation.org/pipermail/bridge/attachments/20061129/8fda1b4f/attachment-0001.htm
Jay Libove
2007-Apr-18 17:23 UTC
[Bridge] recent crashes? Linux kernel 2.6.18-1.2239.fc5 (Linux Fedora Core 5)
At the risk of angering the crash Gods, my sustem has NOT crashed again since I downgraded the kernel from 2.6.18-1.2239.fc5 to 2.6.18-1.2200.fc5. Given that newfound stability, and my lack of time, I'm going to put on hold any further diagnostics, until the next kernel revision is released. I have submitted a report at bugzilla.redhat.com (bug 218128). (Ah, nuts; accidentally created a duplicate at 218129 also. Sorry about that). Uh, does anyone have a Fedora Core 5 kernel Changelog from 2200 to 2239, so I can figure out what fixes I'm *not* running right now, as a result of my backing off from 2239 to 2200? Heh. Thanks! -Jay