Mike McCormack
2004-Jan-22 09:38 UTC
PROBLEM: Filesystem became readonly, then kernel crashes on shutdown
Hi All,
I just had "an issue" with my Linux 2.6.1 kernel. While working, I
tried to rf -rf a directory, and was told my filesystem was mounted
readonly (which it wasn't). I then shut the machine down, and on
shutdown, I got a "kernel BUG at fs/ext3/super.c: 411". Full report
below:
Mike
[1.] One line summary of the problem:
Filesystem became readonly, then kernel crashed on shutdown
[2.] Full description of the problem/report:
While working, I tried to rf -rf a directory, and was told my filesystem
was mounted readonly (which it wasn't). I then shut the machine down,
and on shutdown, I got a "kernel BUG at fs/ext3/super.c: 411".
bash-2.05b$ rm -rf ~/cxoffice/
rm: cannot remove `/home/mike/cxoffice//.manifest/cxoffice.xml':
Read-only file system
<many files listed, all with //>
bash-2.05b$ ls -l /home/mike/cxoffice//.manifest/cxoffice.xml
-rw-r--r-- 1 mike users 48062 Jan 22 01:57
/home/mike/cxoffice//.manifest/cxoffice.xml
[3.] Keywords (i.e., modules, networking, kernel):
ext3, slackware 9.1
[4.] Kernel version (from /proc/version):
bash-2.05b$ cat /proc/version
Linux version 2.6.1 (mike@kimchi) (gcc version 3.2.3) #11 Sun Jan 11
15:57:17 KST 2004
[5.] Output of Oops.. message (if applicable) with symbolic information
resolved (see Documentation/oops-tracing.txt)
(as copied from the screen - i have it written on paper too)
kernel BUG at fs/etx3/super.c: 411
invalid operand: 0000 [#1]
CPU: 0
EIP: 0060:[<c0196745>] Not tainted
EFLAGS: 00010282
EIP is at ext_put_super+0x135/0x1a0
eax:0000005e ebx:dff6ee84 ecx:c02bbd50 edx:00000286
esi:dff6ee00 edi:dfe07400 ebp:dda93f10 esp:dda93ef0
Process umount(pid: 4168, threadinfo=dda92000 task=dc858120)
Stack: c0295680 c0284f2b c02939b9 0000019b
c029399e dfe0744c dfe07400 c02c1960
dda93f2c c0159a7d dfe07400 0000077
dfe07400 dffe1040 dda92000 dda93f44
c015a6fd dfe07400 0804df58 dfe07440
dfe07400 dda93f64 c015979e dfe07400
Call Trace:
[<c0159a7d>] generic_shutdown_super+0x18d/0x1a0
[<c015a6fd>] kill_block_super+0x1d/0x50
[<c015979e>] deactivate_super+0x7e/0xe0
[<c017021c>] sys_umount+0x3c/0xa0
[<c0170299>] sys_oldmount+0x19/0x20
[<c01094bb>] syscall_call+0x07/0x0b
code: 0f 0b 9b 01 b9 39 29 c0 e9 62 ff ff ff 89 74 24 04 89 3c 24
/etc/rc.d/rc.6: line 141: 4168 Segmentation fault umount -a -r -t nonfs
Remount root filesystem read-only.
(then the system froze)
[6.] A small shell script or example program which triggers the
problem (if possible)
I've removed the same directory many times, and the problem didn't
happen before....
[7.] Environment
bash-2.05b$ set
BASH=/usr/bin/bash
BASH_VERSINFO=([0]="2" [1]="05b" [2]="0"
[3]="1" [4]="release"
[5]="i486-slackwa
re-linux-gnu")
BASH_VERSION='2.05b.0(1)-release'
COLUMNS=80
CVSROOT=:ext:mike@localhost:/cvstrees/crossover
CVS_RSH=ssh
CXOFFICE_DRIVE_TYPE_HACK=hd
DIRSTACK=()
DISPLAY=:0.0
EUID=1000
GDK_USE_XFT=1
GROUPS=()
HISTFILE=/home/mike/.bash_history
HISTFILESIZE=500
HISTSIZE=500
HOME=/home/mike
HOSTNAME=kimchi.codeweavers.com
HOSTTYPE=i486
HUSHLOGIN=FALSE
HZ=100
IFS=$' \t\n'
INPUTRC=/etc/inputrc
JAVA_HOME=/usr/lib/java
LANG=C
LD_LIBRARY_PATH=:/home/mike/codeweavers/office/wine/libs/unicode:/home/mike/code
weavers/office/wine/dlls:/home/mike/codeweavers/office/wine/tsx11:/home/mike/cod
eweavers/office/wine/libs/wine:/home/mike/codeweavers/office/freetype/objs/.libs
LESS=-M
LESSOPEN='|lesspipe.sh %s'
LINES=24
LOGNAME=mike
LS_COLORS='no=00:fi=00:di=01;36:ln=01;34:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;
01:or=40;31;01:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01
;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.bz
2=01;31:*.rpm=01;31:*.deb=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.jpg=01;35:*.gif
=01;35:*.bmp=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*
.mpg=01;35:*.avi=01;35:*.mov=01;35:'
LS_OPTIONS=' --color=always -F -b -T 0'
MACHTYPE=i486-slackware-linux-gnu
MAIL=/var/spool/mail/mike
MAILCHECK=60
MANPATH=/usr/local/man:/usr/man:/usr/X11R6/man:/usr/lib/java/man:/usr/share/texm
f/man
MINICOM='-c on'
OLDPATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games:/opt/www/htdig/bi
n:/usr/lib/java/bin:/usr/lib/java/jre/bin:/usr/share/texmf/bin:.:/home/mike/bin
OLD_LD_LIBRARY_PATHOPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games:/opt/www/htdig/bin:/
usr/lib/java/bin:/usr/lib/java/jre/bin:/usr/share/texmf/bin:.:/home/mike/bin:/ho
me/mike/codeweavers/office/wine:/home/mike/codeweavers/office/wine/tools/winegcc
PIPESTATUS=([0]="0")
PPID=430
PS1='\s-\v\$ '
PS2='> '
PS4='+ '
PWD=/home/mike
SHELL=/bin/bash
SHELLOPTS=braceexpand:emacs:hashall:histexpand:history:interactive-comments:moni
tor
SHLVL=3
SSH_AUTH_SOCK=/home/mike/.ssh-agent-mike
T1LIB_CONFIG=/usr/share/t1lib/t1lib.config
TERM=xterm
UID=1000
USER=mike
USERNAME=Administrator
USERPROFILE='C:\Windows\Profiles'
WINDOWID=18874382
WINDOW_MANAGER=sawfish
WINEDBG=/home/mike/codeweavers/office/wine/debugger/winedbg
WINEDLLPATH=/home/mike/codeweavers/office/wine/dlls:/home/mike/codeweavers/offic
e/wine/programs
WINEPREFIX=/home/mike/.cxoffice/dotwine
WINESERVER=/home/mike/codeweavers/office/wine/server/wineserver
_=/home/mike/bin/cxwine_envsetup
[7.1.] Software (add the output of the ver_linux script here)
bash-2.05b$ scripts/ver_linux
If some fields are empty or look unusual you may have an old version.
Compare to the current minimal requirements in Documentation/Changes.
Linux kimchi 2.6.1 #11 Sun Jan 11 15:57:17 KST 2004 i686 unknown unknown
GNU/Linux
Gnu C 3.2.3
Gnu make 3.80
util-linux 2.12
mount 2.12
module-init-tools 0.9.14
e2fsprogs 1.34
jfsutils 1.1.3
xfsprogs 2.5.6
pcmcia-cs 3.2.5
quota-tools 3.09.
PPP 2.4.1
nfs-utils 1.0.6
Linux C Library 2.3.2
Dynamic linker (ldd) 2.3.2
Linux C++ Library 5.0.3
Procps 2.0.16
Net-tools 1.60
Kbd 1.08
Sh-utils 5.0
Modules Loaded iptable_filter ip_tables loop snd_pcm_oss
snd_mixer_oss snd_cmipci snd_pcm snd_page_alloc snd_opl3_lib snd_timer
snd_hwdep snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore pppoe
pppox ppp_generic slhc floppy sg sd_mod isofs 8139too mii crc32
usb_storage scsi_mod ov511 videodev uhci_hcd usbcore agpgart
[7.2.] Processor information (from /proc/cpuinfo):
bash-2.05b$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 8
model name : Pentium III (Coppermine)
stepping : 6
cpu MHz : 1004.258
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca
cmov pat pse36 mmx fxsr sse
bogomips : 1986.56
[7.3.] Module information (from /proc/modules):
bash-2.05b$ cat /proc/modules
iptable_filter 2080 1 - Live 0xe089a000
ip_tables 16400 1 iptable_filter, Live 0xe0935000
loop 14184 0 - Live 0xe093b000
snd_pcm_oss 49316 0 - Live 0xe0977000
snd_mixer_oss 17440 1 snd_pcm_oss, Live 0xe0949000
snd_cmipci 32568 0 - Live 0xe0940000
snd_pcm 89216 2 snd_pcm_oss,snd_cmipci, Live 0xe0950000
snd_page_alloc 9220 1 snd_pcm, Live 0xe08c6000
snd_opl3_lib 9344 1 snd_cmipci, Live 0xe091f000
snd_timer 22240 2 snd_pcm,snd_opl3_lib, Live 0xe092e000
snd_hwdep 7680 1 snd_opl3_lib, Live 0xe08dc000
snd_mpu401_uart 6016 1 snd_cmipci, Live 0xe08cd000
snd_rawmidi 20768 1 snd_mpu401_uart, Live 0xe0927000
snd_seq_device 6692 2 snd_opl3_lib,snd_rawmidi, Live 0xe08ca000
snd 45188 10
snd_pcm_oss,snd_mixer_oss,snd_cmipci,snd_pcm,snd_opl3_lib,snd_timer,snd_hwdep,snd_mpu401_uart,snd_raw
midi,snd_seq_device, Live 0xe0903000
soundcore 6912 1 snd, Live 0xe0894000
pppoe 11264 0 - Live 0xe08a9000
pppox 2728 1 pppoe, Live 0xe0858000
ppp_generic 22888 2 pppoe,pppox, Live 0xe08fc000
slhc 6016 1 ppp_generic, Live 0xe0897000
floppy 57332 0 - Live 0xe0910000
sg 34700 0 - Live 0xe08d2000
sd_mod 14496 0 - Live 0xe08c1000
isofs 22628 0 - Live 0xe08a2000
8139too 19488 0 - Live 0xe089c000
mii 4064 1 8139too, Live 0xe0876000
crc32 3840 1 8139too, Live 0xe0874000
usb_storage 101136 0 - Live 0xe08e2000
scsi_mod 74820 3 sg,sd_mod,usb_storage, Live 0xe08ad000
ov511 81788 0 - Live 0xe085f000
videodev 7488 1 ov511, Live 0xe085c000
uhci_hcd 30760 0 - Live 0xe0846000
usbcore 107604 5 usb_storage,ov511,uhci_hcd, Live 0xe0878000
agpgart 27112 0 - Live 0xe0850000
[7.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem)
bash-2.05b$ cat /proc/ioports
0000-001f : dma1
0020-0021 : pic1
0040-005f : timer
0060-006f : keyboard
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
0170-0177 : ide1
01f0-01f7 : ide0
0330-0331 : MPU401 UART
0376-0376 : ide1
0388-0389 : OPL2/3 (left)
038a-038b : OPL2/3 (right)
03c0-03df : vga+
03f6-03f6 : ide0
0cf8-0cff : PCI conf1
5000-500f : 0000:00:1f.3
c000-c0ff : 0000:02:01.0
c000-c0ff : 8139too
c400-c4ff : 0000:02:02.0
c400-c4ff : CMI8738
d000-d01f : 0000:00:1f.2
d000-d01f : uhci_hcd
d800-d81f : 0000:00:1f.4
d800-d81f : uhci_hcd
f000-f00f : 0000:00:1f.1
f000-f007 : ide0
f008-f00f : ide1
bash-2.05b$ cat /proc/iomem
00000000-0009fbff : System RAM
0009fc00-0009ffff : reserved
000a0000-000bffff : Video RAM area
000c0000-000c7fff : Video ROM
000f0000-000fffff : System ROM
00100000-1ffeffff : System RAM
00100000-00282ac4 : Kernel code
00282ac5-00318ebf : Kernel data
1fff0000-1fff2fff : ACPI Non-volatile Storage
1fff3000-1fffffff : ACPI Tables
d0000000-d7ffffff : PCI Bus #01
d0000000-d7ffffff : 0000:01:00.0
d8000000-dbffffff : 0000:00:00.0
dc000000-ddffffff : PCI Bus #01
dc000000-dcffffff : 0000:01:00.0
de000000-de0000ff : 0000:02:01.0
de000000-de0000ff : 8139too
ffb00000-ffffffff : reserved
[7.5.] PCI information ('lspci -vvv' as root)
bash-2.05b# lspci -vvv
00:00.0 Host bridge: Intel Corp. 82815 815 Chipset Host Bridge and
Memory Controller Hub (rev 02)
Subsystem: Intel Corp. 82815 815 Chipset Host Bridge and Memory
Controller Hub
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort+ >SERR- <PERR-
Latency: 0
Region 0: Memory at d8000000 (32-bit, prefetchable) [size=64M]
Capabilities: [88] #09 [f104]
Capabilities: [a0] AGP version 2.0
Status: RQ=32 Iso- ArqSz=0 Cal=0 SBA+ ITACoh- GART64-
HTrans- 64bit- FW- AGP3- Rate=x1,x2,x4
Command: RQ=1 ArqSz=0 Cal=0 SBA- AGP- GART64- 64bit-
FW- Rate=<none>
00:01.0 PCI bridge: Intel Corp. 82815 815 Chipset AGP Bridge (rev 02)
(prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 32
Bus: primary=00, secondary=01, subordinate=01, sec-latency=32
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: dc000000-ddffffff
Prefetchable memory behind bridge: d0000000-d7ffffff
BridgeCtl: Parity- SERR- NoISA+ VGA+ MAbort- >Reset- FastB2B-
00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB PCI Bridge (rev 02)
(prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Bus: primary=00, secondary=02, subordinate=02, sec-latency=32
I/O behind bridge: 0000c000-0000cfff
Memory behind bridge: de000000-de0fffff
Prefetchable memory behind bridge: fff00000-000fffff
BridgeCtl: Parity- SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
00:1f.0 ISA bridge: Intel Corp. 82801BA ISA Bridge (LPC) (rev 02)
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
00:1f.1 IDE interface: Intel Corp. 82801BA IDE U100 (rev 02) (prog-if 80
[Master])
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Region 4: I/O ports at f000 [size=16]
00:1f.2 USB Controller: Intel Corp. 82801BA/BAM USB (Hub #1) (rev 02)
(prog-if 00 [UHCI])
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Interrupt: pin D routed to IRQ 9
Region 4: I/O ports at d000 [size=32]
00:1f.3 SMBus: Intel Corp. 82801BA/BAM SMBus (rev 02)
Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR-
Interrupt: pin B routed to IRQ 9
Region 4: I/O ports at 5000 [size=16]
00:1f.4 USB Controller: Intel Corp. 82801BA/BAM USB (Hub #2) (rev 02)
(prog-if 00 [UHCI])
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Interrupt: pin C routed to IRQ 9
Region 4: I/O ports at d800 [size=32]
01:00.0 VGA compatible controller: nVidia Corporation NV11DDR [GeForce2
MX 100 DDR/200 DDR] (rev b2) (prog-if 00 [VGA])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (1250ns min, 250ns max)
Interrupt: pin A routed to IRQ 10
Region 0: Memory at dc000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at d0000000 (32-bit, prefetchable) [size=128M]
Expansion ROM at <unassigned> [disabled] [size=64K]
Capabilities: [60] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [44] AGP version 2.0
Status: RQ=32 Iso- ArqSz=0 Cal=0 SBA- ITACoh- GART64-
HTrans- 64bit- FW+ AGP3- Rate=x1,x2,x4
Command: RQ=1 ArqSz=0 Cal=0 SBA- AGP- GART64- 64bit-
FW- Rate=<none>
02:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL-8139/8139C/8139C+ (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RT8139
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (8000ns min, 16000ns max)
Interrupt: pin A routed to IRQ 11
Region 0: I/O ports at c000 [size=256]
Region 1: Memory at de000000 (32-bit, non-prefetchable) [size=256]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0-,D1+,D2+,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
02:02.0 Multimedia audio controller: C-Media Electronics Inc CM8738 (rev 10)
Subsystem: C-Media Electronics Inc CMI8738/C3DX PCI Audio Device
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium
>TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (500ns min, 6000ns max)
Interrupt: pin A routed to IRQ 5
Region 0: I/O ports at c400 [size=256]
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
[7.6.] SCSI information (from /proc/scsi/scsi)
bash-2.05b# cat /proc/scsi/scsi
Attached devices:
[7.7.] Other information that might be relevant to the problem
(please look in /proc and include all information that you
think to be relevant):
My distro is slackware 9.1. I use the machine for work and play almost
every day, so it had a few days uptime when the crash occured.
Looks as though the // in the path might be relevant... after rebooting
the machine, I successfully removed the same directory.
I am a Wine developer and have done some limitted development with the
Linux kernel before.
After a hard reset of the machine (after the crash), my /dev/hda2
partition that contains my home directory was full of errors. I wrote
some of these down after running e2fsck manually.
This is the first time the problem has happened, so I'm dutifully
reporting it :)
[X.] Other notes, patches, fixes, workarounds:
None known
Mike McCormack
2004-Jan-22 15:21 UTC
Re: PROBLEM: Filesystem became readonly, then kernel crashes on shutdown
Hi Bryan, Bryan Kadzban wrote:> Are there any other oops reports in your kernel log? One of the > possible error condition behaviors of ext3 is to remount the FS readonly > in case of a problem detected in the code, so that might be what made it > read-only in the first place. > > Are you using Alsa by any chance? I've heard vague rumors of Alsa 0.9.7 > (the one that comes in kernel 2.6.something) causing FS corruption > issues, though I hadn't seen it happen with 2.6.1-rc1.Yeah, it looks like I missed a bunch of messages in /var/log/syslog... see the attached trace. The trouble appears to have started with this message: Jan 22 01:52:42 kimchi kernel: EXT3-fs warning (device hda2): ext3_unlink: Deleting nonexistent file (2874981), 0 and goes until I shut the machine down: Jan 22 17:03:42 kimchi dhcpcd[106]: terminating on signal 15 Jan 22 17:53:57 kimchi kernel: Linux version 2.6.1 (mike@kimchi) (gcc version 3.2.3) #11 Sun Jan 11 15:57:17 KST 2 Full log is attached. Mike