Hi,
I've just upgraded a 8-STABLE box to 9-STABLE (well, just few commits
before it has been tagged as STABLE), which runs from NFS (pxebooted).
It has some IPSec config in ipsec.conf, like this for several boxes:
add 172.28.16.4 172.16.248.2 ah 15704 -A hmac-md5 "asdfgh";
add 172.16.248.2 172.28.16.4 ah 24504 -A hmac-md5 "asdfgh";
add 172.28.16.4 172.16.248.2 esp 15705 -E blowfish-cbc "hgfdsa";
add 172.16.248.2 172.28.16.4 esp 24505 -E blowfish-cbc "hgfdsa";
spdadd 172.28.16.4 172.16.248.2 any -P out ipsec
esp/transport/172.28.16.4-172.16.248.2/default
ah/transport/172.28.16.4-172.16.248.2/default;
Running /etc/rc.d/ipsec start instantly panics it with:
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0xa0
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff809bf779
stack pointer = 0x28:0xffffff80002cd350
frame pointer = 0x28:0xffffff80002cd390
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (swi1: netisr 0)
trap number = 12
panic: page fault
cpuid = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
kdb_backtrace() at kdb_backtrace+0x37
panic() at panic+0x187
trap_fatal() at trap_fatal+0x290
trap_pfault() at trap_pfault+0x1f9
trap() at trap+0x3df
calltrap() at calltrap+0x8
--- trap 0xc, rip = 0xffffffff809bf779, rsp = 0xffffff80002cd350, rbp
= 0xffffff80002cd390 ---
ipsec_process_done() at ipsec_process_done+0x119
esp_output_cb() at esp_output_cb+0x1a1
crypto_done() at crypto_done+0x102
swcr_process() at swcr_process+0x1d7
crypto_invoke() at crypto_invoke+0x6b
crypto_dispatch() at crypto_dispatch+0xfb
esp_output() at esp_output+0x5a2
ipsec4_process_packet() at ipsec4_process_packet+0x1f8
ip_ipsec_output() at ip_ipsec_output+0x16a
ip_output() at ip_output+0x526
icmp_reflect() at icmp_reflect+0x339
icmp_input() at icmp_input+0x257
ip_input() at ip_input+0x1de
swi_net() at swi_net+0x14d
intr_event_execute_handlers() at intr_event_execute_handlers+0x104
ithread_loop() at ithread_loop+0xa4
fork_exit() at fork_exit+0x11f
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80002cdd00, rbp = 0 ---
Uptime: 3m31s
The machine in question is a VMWare virtual environment (if it
counts), the kernel config is GENERIC plus this:
options BOOTP
options BOOTP_NFSV3
options BOOTP_NFSROOT
options NFSCLIENT
device carp
device crypto
device cryptodev
options IPSEC
options MAC
options ROUTETABLES=1
options KDB
options DDB
options KDB_UNATTENDED
kgdb says:
(kgdb) bt
#0 doadump (textdump=1) at /data/usr/src/sys/kern/kern_shutdown.c:260
#1 0xffffffff80845705 in kern_reboot (howto=260)
at /data/usr/src/sys/kern/kern_shutdown.c:442
#2 0xffffffff80845bb1 in panic (fmt=Variable "fmt" is not
available.
)
at /data/usr/src/sys/kern/kern_shutdown.c:607
#3 0xffffffff80b167a0 in trap_fatal (frame=0xc, eva=Variable "eva"
is
not available.
)
at /data/usr/src/sys/amd64/amd64/trap.c:819
#4 0xffffffff80b16ae9 in trap_pfault (frame=0xffffff80002cd2a0,
usermode=0)
at /data/usr/src/sys/amd64/amd64/trap.c:735
#5 0xffffffff80b16faf in trap (frame=0xffffff80002cd2a0)
at /data/usr/src/sys/amd64/amd64/trap.c:474
#6 0xffffffff80b012ef in calltrap ()
at /data/usr/src/sys/amd64/amd64/exception.S:228
#7 0xffffffff809bf779 in ipsec_process_done (m=0xfffffe000c7c7a00,
isr=0xfffffe001bf54380) at
/data/usr/src/sys/netipsec/ipsec_output.c:170
#8 0xffffffff809ce931 in esp_output_cb (crp=0xfffffe011103c058)
at /data/usr/src/sys/netipsec/xform_esp.c:1007
#9 0xffffffff809f4e12 in crypto_done (crp=0xfffffe011103c058)
at /data/usr/src/sys/opencrypto/crypto.c:1156
#10 0xffffffff809f89c7 in swcr_process (dev=Variable "dev" is not
available.
)
at /data/usr/src/sys/opencrypto/cryptosoft.c:1054
#11 0xffffffff809f5c9b in crypto_invoke (cap=0xfffffe000c12f700,
crp=0xfffffe011103c058, hint=0) at cryptodev_if.h:53
#12 0xffffffff809f6acb in crypto_dispatch (crp=0xfffffe011103c058)
at /data/usr/src/sys/opencrypto/crypto.c:806
#13 0xffffffff809cef82 in esp_output (m=0xfffffe000c7c7a00,
isr=0xfffffe001bf54380, mp=Variable "mp" is not available.
) at /data/usr/src/sys/netipsec/xform_esp.c:907
#14 0xffffffff809bfa98 in ipsec4_process_packet (m=0xfffffe000c7c7a00,
isr=0xfffffe001bf54380, flags=Variable "flags" is not
available.
)
at /data/usr/src/sys/netipsec/ipsec_output.c:580
#15 0xffffffff8096f5da in ip_ipsec_output (m=0xffffff80002cd898,
inp=0x0,
flags=0xffffff80002cd894, error=0xffffff80002cd8ac)
at /data/usr/src/sys/netinet/ip_ipsec.c:353
#16 0xffffffff80971be6 in ip_output (m=0xfffffe000c7c7a00,
opt=Variable "opt" is not available.
)
at /data/usr/src/sys/netinet/ip_output.c:480
#17 0xffffffff8096c519 in icmp_reflect (m=0xfffffe000c7c7a00)
at /data/usr/src/sys/netinet/ip_icmp.c:886
#18 0xffffffff8096caa7 in icmp_input (m=0xfffffe000c7c7a00, off=20)
at /data/usr/src/sys/netinet/ip_icmp.c:571
#19 0xffffffff8096eede in ip_input (m=0xfffffe000c7c7a00)
at /data/usr/src/sys/netinet/ip_input.c:760
#20 0xffffffff80909bdd in swi_net (arg=Variable "arg" is not
available.
) at /data/usr/src/sys/net/netisr.c:806
#21 0xffffffff8081b804 in intr_event_execute_handlers (p=Variable
"p"
is not available.
)
at /data/usr/src/sys/kern/kern_intr.c:1257
#22 0xffffffff8081cfc4 in ithread_loop (arg=0xfffffe000c1133a0)
at /data/usr/src/sys/kern/kern_intr.c:1270
#23 0xffffffff8081845f in fork_exit (
callout=0xffffffff8081cf20 <ithread_loop>, arg=0xfffffe000c1133a0,
frame=0xffffff80002cdc50) at
/data/usr/src/sys/kern/kern_fork.c:995
#24 0xffffffff80b0181e in fork_trampoline ()
at /data/usr/src/sys/amd64/amd64/exception.S:602
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000001 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x0000000000000000 in ?? ()
#41 0x0000000000000000 in ?? ()
#42 0x0000000000000000 in ?? ()
#43 0x0000000000000000 in ?? ()
#44 0x0000000000000000 in ?? ()
#45 0x0000000000000000 in ?? ()
#46 0x0000000000000000 in ?? ()
#47 0x0000000000000000 in ?? ()
#48 0x0000000000000000 in ?? ()
#49 0xffffffff81136080 in affinity ()
#50 0x0000000000000000 in ?? ()
#51 0x0000000000000000 in ?? ()
#52 0xfffffe000c134000 in ?? ()
#53 0xffffff80002ccef0 in ?? ()
#54 0xffffff80002cce98 in ?? ()
#55 0xfffffe000c118000 in ?? ()
#56 0xffffffff8086de12 in sched_switch (td=0xffffffff8081cf20,
newtd=dwarf2_read_address: Corrupted DWARF expression.
)
at /data/usr/src/sys/kern/sched_ule.c:1848
Previous frame inner to this frame (corrupt stack?)
8-STABLE runs fine with the same config.
VANHULLEBUS Yvan
2012-Jan-04 15:10 UTC
Enabling IPSec panics stable/9 (runs OK on stable/8)
On Wed, Jan 04, 2012 at 01:46:03PM +0100, Attila Nagy wrote:> Hi,Hi.> I've just upgraded a 8-STABLE box to 9-STABLE (well, just few commits > before it has been tagged as STABLE), which runs from NFS (pxebooted). > It has some IPSec config in ipsec.conf, like this for several boxes: > add 172.28.16.4 172.16.248.2 ah 15704 -A hmac-md5 "asdfgh"; > add 172.16.248.2 172.28.16.4 ah 24504 -A hmac-md5 "asdfgh"; > add 172.28.16.4 172.16.248.2 esp 15705 -E blowfish-cbc "hgfdsa"; > add 172.16.248.2 172.28.16.4 esp 24505 -E blowfish-cbc "hgfdsa"; > spdadd 172.28.16.4 172.16.248.2 any -P out ipsec > esp/transport/172.28.16.4-172.16.248.2/default > ah/transport/172.28.16.4-172.16.248.2/default;There is probably nothing related to the crash, but do you really use static IPsec without IKE keying ???? [....]> kgdb says: > (kgdb) bt > #0 doadump (textdump=1) at /data/usr/src/sys/kern/kern_shutdown.c:260 > #1 0xffffffff80845705 in kern_reboot (howto=260) > at /data/usr/src/sys/kern/kern_shutdown.c:442 > #2 0xffffffff80845bb1 in panic (fmt=Variable "fmt" is not available. > ) > at /data/usr/src/sys/kern/kern_shutdown.c:607 > #3 0xffffffff80b167a0 in trap_fatal (frame=0xc, eva=Variable "eva" is > not available. > ) > at /data/usr/src/sys/amd64/amd64/trap.c:819 > #4 0xffffffff80b16ae9 in trap_pfault (frame=0xffffff80002cd2a0, > usermode=0) > at /data/usr/src/sys/amd64/amd64/trap.c:735 > #5 0xffffffff80b16faf in trap (frame=0xffffff80002cd2a0) > at /data/usr/src/sys/amd64/amd64/trap.c:474 > #6 0xffffffff80b012ef in calltrap () > at /data/usr/src/sys/amd64/amd64/exception.S:228 > #7 0xffffffff809bf779 in ipsec_process_done (m=0xfffffe000c7c7a00, > isr=0xfffffe001bf54380) at > /data/usr/src/sys/netipsec/ipsec_output.c:170Here seems to be the problem.... Can you do the following (in this order) in kgdb: frame 7 p saidx p *saidx The latest will probably generate an error, as (if you have the exact same ipsec_output.c as I have from HEAD) saidx will probably have an invalid adress. [...]> 8-STABLE runs fine with the same config.Strange.... I'll review changes in IPsec stack which have been done in STABLE/9 and not backported to STABLE/8..... Yvan.