freebsd stable - Oct 2012 - stable/9 @r241776 panic: REDZONE: Buffer underflow detected...

This seems ... fairly weird to me.

Yesterday, I built & booted:

FreeBSD g1-227.catwhisker.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #274
241726M: Fri Oct 19 05:40:05 PDT 2012     root at
g1-227.catwhisker.org:/usr/obj/usr/src/sys/CANARY  i386

and used the machine all day; nothing unusual (including various
reboots (e.g. when I disembarked the train for the final leg of my
commute home, so I powered the laptop off).

This morning, I built:

FreeBSD g1-227.catwhisker.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #275
241776M: Sat Oct 20 04:34:45 PDT 2012     root at
g1-227.catwhisker.org:/usr/obj/usr/src/sys/CANARY  i386

and on first reboot, I got a panic.

After a bit of experimentation, it appears that I get a panic @r241776
if I attempt a normal boot into multi-user mode, but if I first boot to
single-user mode, then exit single-user mode, it comes up without a
problem.

I don't have a serial console, so I started to write down some of the
panic information, but my patience ran a bit short.  Here's whet I
recorded (warning: hand-transcripted -- twice!):

...
Starting devd.
REDZONE: Buffer underflow detected.  1 byte corrupted before 0xced40080
(4294966796 bytes allocated).
Allocation backtrace:
#0 0xc0ceac8f at redzone_setup+0xcf
#1 0xc0a5d5c9 at malloc+0x1d9
...[about 20 more such lines I didn't record]...
> bt
Tracing pid 901 tid 100106 td 0xd2b99000
kdb_enter(...)
panic(...)
free(...)
devread(ce8c2d00,f7274c0c,0,c0b1e4f0,d279e380,...) at devread+0x1a6
giant_read(...) at giant_read+0x87
devfs_read(...) at devfs_read+0xc6
dofileread(...) at dofileread+0x99
sys_read(...) at sys_read+0x98
syscall(f7274d08) at syscall+0x387

Within the bounds described above, this appears to be quite reproducible
-- on my laptop.  My build machine (updated in parallel, at the same
GRNs) does not exhibit the panic.

I was unable to get a crash dump; I have

dumpdev="AUTO"

in /etc/rc.conf, and the panic was occurring well after swap was
enabled.  (Yes, I know I have swap over-allocated.  I plan to do
something about it at some point.)

I've attached a copy of dmesg.boot.

Anyone else seeing this?  Any ideas how to diagnose it?

Thanks!

Peace,
david
-- 
David H. Wolfskill				david at catwhisker.org
Taliban: Evil men with guns afraid of truth from a 14-year old girl.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.
-------------- next part --------------
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.1-PRERELEASE #275 241776M: Sat Oct 20 04:34:45 PDT 2012
    root at g1-227.catwhisker.org:/usr/obj/usr/src/sys/CANARY i386
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
MEMGUARD DEBUGGING ALLOCATOR INITIALIZED:
	MEMGUARD map base: 0xc8000000
	MEMGUARD map limit: 0xce681000
	MEMGUARD map size: 104964 KBytes
CPU: Intel(R) Core(TM)2 Duo CPU     T9600  @ 2.80GHz (2793.06-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x10676  Family = 0x6  Model = 0x17 
Stepping = 6
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 
Features2=0x8e3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1>
  AMD Features=0x20100000<NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 3643670528 (3474 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <DELL   M09    >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <DELL M09    > on motherboard
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
acpi0: reservation of 0, 9f000 (3) failed
acpi0: reservation of 100000, df351c00 (3) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 2 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
acpi_ec0: <Embedded Controller: GPE 0x11> port 0x930,0x934 on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xdf00-0xdf7f mem
0xf5000000-0xf5ffffff,0xe0000000-0xefffffff,0xf2000000-0xf3ffffff irq 16 at
device 0.0 on pci1
nvidia0: <Quadro FX 770M> on vgapci0
vgapci0: child nvidia0 requested pci_enable_io
vgapci0: child nvidia0 requested pci_enable_io
pci0: <simple comms> at device 3.0 (no driver attached)
atapci0: <Intel ATA controller> port
0xef78-0xef7f,0xef70-0xef73,0xef80-0xef87,0xef74-0xef77,0xef90-0xef9f irq 18 at
device 3.2 on pci0
ata2: <ATA channel> at channel 0 on atapci0
ata3: <ATA channel> at channel 1 on atapci0
pci0: <simple comms, UART> at device 3.3 (no driver attached)
em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xefe0-0xefff mem
0xf6fe0000-0xf6ffffff,0xf6fdb000-0xf6fdbfff irq 22 at device 25.0 on pci0
em0: Using an MSI interrupt
em0: Ethernet address: 00:24:e8:9c:11:0f
uhci0: <Intel 82801I (ICH9) USB controller> port 0x6f60-0x6f7f irq 20 at
device 26.0 on pci0
uhci0: LegSup = 0x2f00
usbus0 on uhci0
uhci1: <Intel 82801I (ICH9) USB controller> port 0x6f80-0x6f9f irq 21 at
device 26.1 on pci0
uhci1: LegSup = 0x2f00
usbus1 on uhci1
uhci2: <Intel 82801I (ICH9) USB controller> port 0x6fa0-0x6fbf irq 22 at
device 26.2 on pci0
uhci2: LegSup = 0x2f00
usbus2 on uhci2
ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xfed1c400-0xfed1c7ff
irq 22 at device 26.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
hdac0: <Intel 82801I HDA Controller> mem 0xf6fdc000-0xf6fdffff irq 21 at
device 27.0 on pci0
pcib2: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci11: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 28.1 on pci0
pci12: <ACPI PCI bus> on pcib3
iwn0: <Intel Ultimate N WiFi Link 5300> mem 0xf1ffe000-0xf1ffffff irq 17
at device 0.0 on pci12
pcib4: <ACPI PCI-PCI bridge> at device 28.2 on pci0
pci13: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 28.3 on pci0
pci14: <ACPI PCI bus> on pcib5
uhci3: <Intel 82801I (ICH9) USB controller> port 0x6f00-0x6f1f irq 20 at
device 29.0 on pci0
uhci3: LegSup = 0x2f00
usbus4 on uhci3
uhci4: <Intel 82801I (ICH9) USB controller> port 0x6f20-0x6f3f irq 21 at
device 29.1 on pci0
uhci4: LegSup = 0x2f00
usbus5 on uhci4
uhci5: <Intel 82801I (ICH9) USB controller> port 0x6f40-0x6f5f irq 22 at
device 29.2 on pci0
uhci5: LegSup = 0x2f00
usbus6 on uhci5
ehci1: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xfed1c000-0xfed1c3ff
irq 20 at device 29.7 on pci0
usbus7: EHCI version 1.0
usbus7 on ehci1
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci3: <ACPI PCI bus> on pcib6
cbb0: <RF5C476 PCI-CardBus Bridge> irq 19 at device 1.0 on pci3
cardbus0: <CardBus bus> on cbb0
pccard0: <16-bit PCCard bus> on cbb0
fwohci0: <1394 Open Host Controller Interface> mem 0xf1bff800-0xf1bfffff
irq 17 at device 1.1 on pci3
fwohci0: OHCI version 1.10 (ROM=0)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 4a:4f:c0:00:10:37:06:01
fwohci0: Phy 1394a available S400, 1 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 4a:4f:c0:37:06:01
fwe0: Ethernet address: 4a:4f:c0:37:06:01
fwip0: <IP over FireWire> on firewire0
fwip0: Firewire address: 4a:4f:c0:00:10:37:06:01 @ 0xfffe00000000, S400, maxrec
2048
dcons_crom0: <dcons configuration ROM> on firewire0
dcons_crom0: bus_addr 0x20e8000
fwohci0: Initiate bus reset
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode
sdhci0: <RICOH R5C822 SD> mem 0xf1bff600-0xf1bff6ff irq 18 at device 1.2
on pci3
sdhci0: 1 slot(s) allocated
pci3: <base peripheral> at device 1.3 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel ICH9M AHCI SATA controller> port
0x6e70-0x6e77,0x6e78-0x6e7b,0x6e80-0x6e87,0x6e88-0x6e8b,0x6ea0-0x6ebf mem
0xfed1c800-0xfed1cfff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.20 with 4 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ichsmb0: <Intel 82801I (ICH9) SMBus controller> port 0x1100-0x111f mem
0xf6fd9f00-0xf6fd9fff irq 19 at device 31.3 on pci0
smbus0: <System Management Bus> on ichsmb0
smb0: <SMBus generic I/O> on smbus0
acpi_lid0: <Control Method Lid Switch> on acpi0
acpi_button0: <Power Button> on acpi0
acpi_button1: <Sleep Button> on acpi0
acpi_acad0: <AC Adapter> on acpi0
battery0: <ACPI Control Method Battery> on acpi0
battery1: <ACPI Control Method Battery> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64,0x62,0x66 irq 1 on
acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model GlidePoint, device ID 0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xce7ff,0xce800-0xcffff pnpid
ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ata0: <ATA channel> at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
ata1: <ATA channel> at port 0x170-0x177,0x376 irq 15 on isa0
ppc0: parallel port not found.
ctl: CAM Target Layer loaded
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
Timecounters tick every 1.000 msec
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0)  (me) 
firewire0: bus manager 0 
ipfw2 (+ipv6) initialized, divert enabled, nat loadable, rule-based forwarding
enabled, default to deny, logging disabled
DUMMYNET 0 with IPv6 initialized (100409)
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
hdacc0: <IDT 92HD71B7 HDA CODEC> at cad 0 on hdac0
hdaa0: <IDT 92HD71B7 Audio Function Group> at nid 1 on hdacc0
pcm0: <IDT 92HD71B7 (Analog 2.0+HP/2.0)> at nid 13,10 and 11,14 on hdaa0
pcm1: <IDT 92HD71B7 (Analog)> at nid 15 and 24 on hdaa0
pcm2: <IDT 92HD71B7 (Rear Digital)> at nid 30 on hdaa0
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
Expensive timeout(9) function: 0xc0ab2680(0xcefa12f0) 0.002124711 s
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ugen7.1: <Intel> at usbus7
uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
Expensive timeout(9) function: 0xc055ae80(0xcf307e80) 0.365924998 s
uhub3: 6 ports with 6 removable, self powered
uhub7: 6 ports with 6 removable, self powered
ada0 at ahcich0 bus 0 scbus2 target 0 lun 0
ada0: <ST95005620AS SD23> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad8
cd0 at ahcich1 bus 0 scbus3 target 0 lun 0
cd0: <TSSTcorp DVD+-RW TS-U633A D200> Removable CD-ROM SCSI-0 device 
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray
closed
SMP: AP CPU #1 Launched!
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
GEOM: ada0s1: geometry does not match label (255h,63s != 16h,63s).
GEOM: ada0s2: geometry does not match label (255h,63s != 16h,63s).
GEOM: ada0s3: geometry does not match label (255h,63s != 16h,63s).
GEOM: ada0s4: geometry does not match label (255h,63s != 16h,63s).
Trying to mount root from ufs:/dev/ada0s1a [rw]...
WARNING: / was not properly dismounted
warning: total configured swap (5242880 pages) exceeds maximum recommended
amount (2097312 pages).
warning: increase kern.maxswzone or reduce amount of swap.
wlan0: Ethernet address: 00:21:6a:26:34:c0
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20121020/6dc3cb9f/attachment.sig>

On Sat, Oct 20, 2012 at 07:10:19AM -0700, David Wolfskill
wrote:
> This seems ... fairly weird to me.
> 
> Yesterday, I built & booted:
> 
> FreeBSD g1-227.catwhisker.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #274
241726M: Fri Oct 19 05:40:05 PDT 2012     root at
g1-227.catwhisker.org:/usr/obj/usr/src/sys/CANARY  i386
> 
> and used the machine all day; nothing unusual (including various
> reboots (e.g. when I disembarked the train for the final leg of my
> commute home, so I powered the laptop off).
> 
> This morning, I built:
> 
> FreeBSD g1-227.catwhisker.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #275
241776M: Sat Oct 20 04:34:45 PDT 2012     root at
g1-227.catwhisker.org:/usr/obj/usr/src/sys/CANARY  i386
> 
> and on first reboot, I got a panic.
> 
> After a bit of experimentation, it appears that I get a panic @r241776
> if I attempt a normal boot into multi-user mode, but if I first boot to
> single-user mode, then exit single-user mode, it comes up without a
> problem.
> 
> I don't have a serial console, so I started to write down some of the
> panic information, but my patience ran a bit short.  Here's whet I
> recorded (warning: hand-transcripted -- twice!):
> 
> ...
> Starting devd.
> REDZONE: Buffer underflow detected.  1 byte corrupted before 0xced40080
(4294966796 bytes allocated).
> Allocation backtrace:
> #0 0xc0ceac8f at redzone_setup+0xcf
> #1 0xc0a5d5c9 at malloc+0x1d9
> ...[about 20 more such lines I didn't record]...
> 
> > bt
> Tracing pid 901 tid 100106 td 0xd2b99000
> kdb_enter(...)
> panic(...)
> free(...)
> devread(ce8c2d00,f7274c0c,0,c0b1e4f0,d279e380,...) at devread+0x1a6
> giant_read(...) at giant_read+0x87
> devfs_read(...) at devfs_read+0xc6
> dofileread(...) at dofileread+0x99
> sys_read(...) at sys_read+0x98
> syscall(f7274d08) at syscall+0x387
> 
> Within the bounds described above, this appears to be quite reproducible
> -- on my laptop.  My build machine (updated in parallel, at the same
> GRNs) does not exhibit the panic.
> 
> I was unable to get a crash dump; I have
> 
> dumpdev="AUTO"
> 
> in /etc/rc.conf, and the panic was occurring well after swap was
> enabled.  (Yes, I know I have swap over-allocated.  I plan to do
> something about it at some point.)
> 
> I've attached a copy of dmesg.boot.
> 
> Anyone else seeing this?  Any ideas how to diagnose it?
devread is the method of devctl(4) which passes devd notifications from
the kernel to userland (to devd, specifically). There were no changes to
devctl(4) for quite a time.

The corruption is, most likely, in some unrelated piece of code. Could
you try to bisect the stable to catch the offender ? The bisect is not
guaranteed to work, obviously, since the random corruption effects are
unpredictable.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20121021/eeb3368b/attachment.sig>

On Sat, Oct 20, 2012 at 07:10:19AM -0700, David Wolfskill
wrote:
> This seems ... fairly weird to me.
> 
> Yesterday, I built & booted:
> 
> FreeBSD g1-227.catwhisker.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #274
241726M: Fri Oct 19 05:40:05 PDT 2012     root at
g1-227.catwhisker.org:/usr/obj/usr/src/sys/CANARY  i386
> 
> and used the machine all day; nothing unusual (including various
> reboots (e.g. when I disembarked the train for the final leg of my
> commute home, so I powered the laptop off).
> 
> This morning, I built:
> 
> FreeBSD g1-227.catwhisker.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #275
241776M: Sat Oct 20 04:34:45 PDT 2012     root at
g1-227.catwhisker.org:/usr/obj/usr/src/sys/CANARY  i386
> 
> and on first reboot, I got a panic.
> 
[..]
> 
> ...
> Starting devd.
> REDZONE: Buffer underflow detected.  1 byte corrupted before 0xced40080
(4294966796 bytes allocated).
> Allocation backtrace:
> #0 0xc0ceac8f at redzone_setup+0xcf
> #1 0xc0a5d5c9 at malloc+0x1d9
> ...[about 20 more such lines I didn't record]...
> 
> > bt
> Tracing pid 901 tid 100106 td 0xd2b99000
> kdb_enter(...)
> panic(...)
> free(...)
> devread(ce8c2d00,f7274c0c,0,c0b1e4f0,d279e380,...) at devread+0x1a6
> giant_read(...) at giant_read+0x87
> devfs_read(...) at devfs_read+0xc6
> dofileread(...) at dofileread+0x99
> sys_read(...) at sys_read+0x98
> syscall(f7274d08) at syscall+0x387
> 
This looks a lot like issue you reported a couple of months earlier,
even affected buffer address matches.

At least part of REDZONE metadata placed directly before the buffer is
corrupted. So the idea is to set a watchpoint at a place that is known
to contain wrong data (in this case allocation size) and wait for some
code to try to modify it.

I hacked up the following (really ugly, but should do the job):
http://people.freebsd.org/~mjg/patches/watchpoint-hack.diff

Note: this assumes that address of affected buffer is always the same.

Assuming I didn't mess anything up, instructions are simple:
Just try to reproduce the issue, at some point you should be dropped to
the debugger. If that happens when dumpdevice is configured, please get a
core. Otherwise just a backtrace ("bt" command).

Note 2: this code does no clear the watchpoint, so if it fails to catch
the offending case, it may catch completely legitimate code later.