Alessio Cecchi
2006-Jul-03 08:15 UTC
[CentOS] Problem with CentOS 4.3 on kernel and ipvsadm
I have installed two CentOS 4.3 boxes with LVS (from http://mirror.centos.org/centos/4/csgfs/ ) but all boxes died withe this error frequently: kernel panic - not syncing: fs/block_dev.c:396: spin_lock (fs/block_dev.c:c0361c0) already locked by fs/block_dev.c/287. I have read from this thread http://threebit.net/mail-archive/centos/msg00243.html that this is an unsolved problem. So i have installed the kernel-2.6.9-39.EL from testing repo, but the box diedi with another error: Jul 2 16:40:40 lvs2 kernel: Bad page state at free_hot_cold_page (in process 'smtp', page c10a2140) Jul 2 16:40:40 lvs2 kernel: flags:0x20000014 mapping:00000000 mapcount:256 count:0 Jul 2 16:40:40 lvs2 kernel: Backtrace: Jul 2 16:40:40 lvs2 kernel: [<c014eaad>] bad_page+0x58/0x89 Jul 2 16:40:40 lvs2 kernel: [<c014f2e1>] free_hot_cold_page+0x5f/0xc8 Jul 2 16:40:40 lvs2 kernel: [<c0159b6a>] zap_pte_range+0x1d9/0x226 Jul 2 16:40:40 lvs2 kernel: [<c0159bf9>] zap_pmd_range+0x42/0x68 Jul 2 16:40:40 lvs2 kernel: [<c0159c58>] unmap_page_range+0x39/0x5f Jul 2 16:40:40 lvs2 kernel: [<c0159d7f>] unmap_vmas+0x101/0x1f8 Jul 2 16:40:40 lvs2 kernel: [<c015ed4a>] exit_mmap+0xb8/0x1d0 Jul 2 16:40:40 lvs2 kernel: [<c0121bb4>] mmput+0xb3/0xd6 Jul 2 16:40:40 lvs2 kernel: [<c0127248>] do_exit+0x255/0x59c Jul 2 16:40:40 lvs2 kernel: [<c0127727>] sys_exit_group+0x0/0xd Jul 2 16:40:40 lvs2 kernel: [<c0317637>] syscall_call+0x7/0xb Jul 2 16:40:40 lvs2 kernel: Trying to fix it up, but a reboot is needed Jul 3 04:02:02 lvs2 kernel: ------------[ cut here ]------------ Jul 3 04:02:02 lvs2 kernel: kernel BUG at mm/rmap.c:479! Jul 3 04:02:02 lvs2 kernel: invalid operand: 0000 [#1] Jul 3 04:02:02 lvs2 kernel: Modules linked in: ip_vs_wlc ip_vs md5 ipv6 iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack ipt_MARK iptable_mangle ip_tables button battery ac uhci_hcd parport_pc parport snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore 8139too mii floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod Jul 3 04:02:02 lvs2 kernel: CPU: 0 Jul 3 04:02:02 lvs2 kernel: EIP: 0060:[<c0160ee9>] Tainted: G B VLI Jul 3 04:02:02 lvs2 kernel: EFLAGS: 00010296 (2.6.9-39.EL) Jul 3 04:02:02 lvs2 kernel: EIP is at page_remove_rmap+0x22/0x36 Jul 3 04:02:02 lvs2 kernel: eax: fffffe00 ebx: c1082140 ecx: c1082140 edx: c1082140 Jul 3 04:02:02 lvs2 kernel: esi: 00000000 edi: 00003000 ebp: c42fd384 esp: c4241dec Jul 3 04:02:02 lvs2 kernel: ds: 007b es: 007b ss: 0068 Jul 3 04:02:02 lvs2 kernel: Process smtp (pid: 5596, threadinfo=c4241000 task=cc920840) Jul 3 04:02:02 lvs2 kernel: Stack: c0159b58 0410a067 00005000 080de000 c0405094 080de000 080e3000 c474d080 Jul 3 04:02:02 lvs2 kernel: c0405094 c0159bf9 00005000 00000000 080de000 c474d080 080e3000 c0405094 Jul 3 04:02:02 lvs2 kernel: c0159c58 00005000 00000000 c4241e9c 080de000 cc5cc5b8 080e3000 c0159d7f Jul 3 04:02:02 lvs2 kernel: Call Trace: Jul 3 04:02:02 lvs2 kernel: [<c0159b58>] zap_pte_range+0x1c7/0x226 Jul 3 04:02:02 lvs2 kernel: [<c0159bf9>] zap_pmd_range+0x42/0x68 Jul 3 04:02:02 lvs2 kernel: [<c0159c58>] unmap_page_range+0x39/0x5f Jul 3 04:02:02 lvs2 kernel: [<c0159d7f>] unmap_vmas+0x101/0x1f8 Jul 3 04:02:03 lvs2 kernel: [<c015ed4a>] exit_mmap+0xb8/0x1d0 Jul 3 04:02:03 lvs2 kernel: [<c0121bb4>] mmput+0xb3/0xd6 Jul 3 04:02:03 lvs2 kernel: [<c0177e51>] exec_mmap+0x2cd/0x2e7 Jul 3 04:02:03 lvs2 kernel: [<c01780d6>] flush_old_exec+0x43/0x367 Jul 3 04:02:03 lvs2 kernel: [<c0177b7a>] kernel_read+0x31/0x3b Jul 3 04:02:03 lvs2 kernel: [<c019cbeb>] load_elf_binary+0x520/0xbe0 Jul 3 04:02:03 lvs2 kernel: [<c01776c1>] copy_strings+0x22b/0x235 Jul 3 04:02:03 lvs2 kernel: [<c01791b1>] search_binary_handler+0x81/0x1cc Jul 3 04:02:03 lvs2 kernel: [<c0179464>] do_execve+0x168/0x1f6 Jul 3 04:02:03 lvs2 kernel: [<c010496c>] sys_execve+0x2e/0x185 Jul 3 04:02:03 lvs2 kernel: [<c0317637>] syscall_call+0x7/0xb Jul 3 04:02:03 lvs2 kernel: Code: ff 05 90 af 42 c0 50 9d c3 89 c2 8b 00 f6 c4 08 74 08 0f 0b dc 01 7c bd 32 c0 83 42 08 ff 0f 98 c0 84 c0 74 19 8b 42 08 40 79 08 <0f> 0b df 01 7c bd 32 c0 9c 58 fa ff 0d 90 af 42 c0 50 9d c3 55 Jul 3 04:02:03 lvs2 kernel: <0>Fatal exception: panic in 5 seconds Jul 3 04:02:03 lvs2 kernel: bad: scheduling while atomic! Jul 3 04:02:03 lvs2 kernel: [<c0314921>] schedule+0x2d/0x5f1 Jul 3 04:02:03 lvs2 kernel: [<c023a692>] poke_blanked_console+0x8f/0x9a Jul 3 04:02:03 lvs2 kernel: [<c0315fde>] schedule_timeout+0x15a/0x179 Jul 3 04:02:03 lvs2 kernel: [<c012e79e>] process_timeout+0x0/0x5 Jul 3 04:02:03 lvs2 kernel: [<c0124919>] printk+0xe/0x11 Jul 3 04:02:03 lvs2 kernel: [<c0106924>] die+0x210/0x221 Jul 3 04:02:03 lvs2 kernel: [<c0139d5f>] search_exception_tables+0x1f/0x21 Jul 3 04:02:03 lvs2 kernel: [<c0106c87>] do_invalid_op+0xcf/0xf2 Jul 3 04:02:03 lvs2 kernel: [<c01299bb>] current_fs_time+0x44/0x4c Jul 3 04:02:03 lvs2 kernel: [<c0160ee9>] page_remove_rmap+0x22/0x36 Jul 3 04:02:03 lvs2 kernel: [<c014bd63>] do_generic_mapping_read+0x3fb/0x403 Jul 3 04:02:03 lvs2 kernel: [<c0106bb8>] do_invalid_op+0x0/0xf2 Jul 3 04:02:03 lvs2 kernel: [<c03186df>] error_code+0x2f/0x40 Jul 3 04:02:03 lvs2 kernel: [<c0160ee9>] page_remove_rmap+0x22/0x36 Jul 3 04:02:03 lvs2 kernel: [<c0159b58>] zap_pte_range+0x1c7/0x226 Jul 3 04:02:03 lvs2 kernel: [<c0159bf9>] zap_pmd_range+0x42/0x68 Jul 3 04:02:03 lvs2 kernel: [<c0159c58>] unmap_page_range+0x39/0x5f Jul 3 04:02:03 lvs2 kernel: [<c0159d7f>] unmap_vmas+0x101/0x1f8 Jul 3 04:02:03 lvs2 kernel: [<c015ed4a>] exit_mmap+0xb8/0x1d0 Jul 3 04:02:03 lvs2 kernel: [<c0121bb4>] mmput+0xb3/0xd6 Jul 3 04:02:03 lvs2 kernel: [<c0177e51>] exec_mmap+0x2cd/0x2e7 Jul 3 04:02:03 lvs2 kernel: [<c01780d6>] flush_old_exec+0x43/0x367 Jul 3 04:02:03 lvs2 kernel: [<c0177b7a>] kernel_read+0x31/0x3b Jul 3 04:02:03 lvs2 kernel: [<c019cbeb>] load_elf_binary+0x520/0xbe0 Jul 3 04:02:03 lvs2 kernel: [<c01776c1>] copy_strings+0x22b/0x235 Jul 3 04:02:03 lvs2 kernel: [<c01791b1>] search_binary_handler+0x81/0x1cc Jul 3 04:02:03 lvs2 kernel: [<c0179464>] do_execve+0x168/0x1f6 Jul 3 04:02:03 lvs2 kernel: [<c010496c>] sys_execve+0x2e/0x185 Jul 3 04:02:03 lvs2 kernel: [<c0317637>] syscall_call+0x7/0xb Jul 3 04:02:03 lvs2 kernel: Bad page state at prep_new_page (in process 'smtp', page c1082400) Jul 3 04:02:03 lvs2 kernel: flags:0x20000000 mapping:00000300 mapcount:0 count:0 Jul 3 04:02:03 lvs2 kernel: Backtrace: Jul 3 04:02:03 lvs2 kernel: [<c014eaad>] bad_page+0x58/0x89 Jul 3 04:02:03 lvs2 kernel: [<c014eeba>] prep_new_page+0x24/0x3a Jul 3 04:02:03 lvs2 kernel: [<c014f51f>] buffered_rmqueue+0x1c4/0x1e7 Jul 3 04:02:03 lvs2 kernel: [<c014f5f5>] __alloc_pages+0xb3/0x29a Jul 3 04:02:03 lvs2 kernel: [<c01604df>] anon_vma_prepare+0x1f/0x1bf Jul 3 04:02:03 lvs2 kernel: [<c015ad3a>] do_wp_page+0x1d4/0x422 Jul 3 04:02:03 lvs2 kernel: [<c015bf5c>] handle_mm_fault+0x136/0x212 Jul 3 04:02:03 lvs2 kernel: [<c011d967>] do_page_fault+0x1ac/0x50c Jul 3 04:02:03 lvs2 kernel: [<c01345fe>] sys_rt_sigaction+0xdc/0xf0 Jul 3 04:02:03 lvs2 kernel: [<c016b77f>] filp_close+0x5a/0x60 Jul 3 04:02:03 lvs2 kernel: [<c011d7bb>] do_page_fault+0x0/0x50c Jul 3 04:02:03 lvs2 kernel: [<c03186df>] error_code+0x2f/0x40 Jul 3 04:02:03 lvs2 kernel: Trying to fix it up, but a reboot is needed Jul 3 04:02:03 lvs2 kernel: Bad page state at prep_new_page (in process 'syslogd', page c1082440) Jul 3 04:02:03 lvs2 kernel: flags:0x20000000 mapping:00000100 mapcount:0 count:0 Jul 3 04:02:03 lvs2 kernel: Backtrace: Jul 3 04:02:03 lvs2 kernel: [<c014eaad>] bad_page+0x58/0x89 Jul 3 04:02:03 lvs2 kernel: [<c014eeba>] prep_new_page+0x24/0x3a Jul 3 04:02:03 lvs2 kernel: [<c014f51f>] buffered_rmqueue+0x1c4/0x1e7 Jul 3 04:02:03 lvs2 kernel: [<c014f5f5>] __alloc_pages+0xb3/0x29a Jul 3 04:02:03 lvs2 kernel: [<c014f7f4>] __get_free_pages+0x18/0x24 Jul 3 04:02:03 lvs2 kernel: [<c018083e>] __pollwait+0x2d/0x94 Jul 3 04:02:03 lvs2 kernel: [<c02b8ca0>] datagram_poll+0x25/0xc5 Jul 3 04:02:03 lvs2 kernel: [<c02b2d28>] sock_poll+0x12/0x14 Jul 3 04:02:03 lvs2 kernel: [<c0180ba5>] do_select+0x25b/0x377 Jul 3 04:02:03 lvs2 kernel: [<c0180811>] __pollwait+0x0/0x94 Jul 3 04:02:03 lvs2 kernel: [<c0180fb1>] sys_select+0x2dd/0x437 Jul 3 04:02:03 lvs2 kernel: [<c016db16>] sys_fsync+0xcd/0xd5 Jul 3 04:02:03 lvs2 kernel: [<c0317637>] syscall_call+0x7/0xb Jul 3 04:02:03 lvs2 kernel: Trying to fix it up, but a reboot is needed Jul 3 04:02:03 lvs2 kernel: Bad page state at prep_new_page (in process 'http', page c1082460) Jul 3 04:02:03 lvs2 kernel: flags:0x20000000 mapping:00000300 mapcount:0 count:0 [...] Jul 3 04:02:07 lvs2 kernel: Trying to fix it up, but a reboot is needed Jul 3 04:02:07 lvs2 kernel: Bad page state at prep_new_page (in process 'pop3', page c10825a0) Jul 3 04:02:07 lvs2 kernel: flags:0x20000000 mapping:00000100 mapcount:0 count:0 Jul 3 04:02:07 lvs2 kernel: Backtrace: Jul 3 04:02:07 lvs2 kernel: [<c014eaad>] bad_page+0x58/0x89 Jul 3 04:02:07 lvs2 kernel: [<c014eeba>] prep_new_page+0x24/0x3a Jul 3 04:02:07 lvs2 kernel: [<c014f51f>] buffered_rmqueue+0x1c4/0x1e7 Jul 3 04:02:07 lvs2 kernel: [<c014f5f5>] __alloc_pages+0xb3/0x29a Jul 3 04:02:07 lvs2 kernel: [<c011d21e>] pte_alloc_one+0x18/0x49 Jul 3 04:02:07 lvs2 kernel: [<c0159465>] pte_alloc_map+0x66/0x12d Jul 3 04:02:07 lvs2 kernel: [<c0159737>] copy_page_range+0xfe/0x358 Jul 3 04:02:07 lvs2 kernel: [<c0122380>] dup_mmap+0x3de/0x4a6 Jul 3 04:02:07 lvs2 kernel: [<c0121f61>] copy_mm+0x10e/0x14f Jul 3 04:02:07 lvs2 kernel: [<c0123172>] copy_process+0x709/0xd52 Jul 3 04:02:07 lvs2 kernel: [<c0186d5e>] d_alloc+0xc2/0x284 Jul 3 04:02:07 lvs2 kernel: [<c01238b1>] do_fork+0x98/0x1a2 Jul 3 04:02:07 lvs2 kernel: [<c0187057>] d_instantiate+0x137/0x13a So LVS is unusable because after of no more hourse of running the kernel go to panic. There is a solutions to this problem? I can help to test some patch or kernel. Have somebody solved this situations? Thanks
Will McDonald
2006-Jul-03 09:10 UTC
[CentOS] Problem with CentOS 4.3 on kernel and ipvsadm
On 03/07/06, Alessio Cecchi <acecchi at telemaco.it> wrote:> I have installed two CentOS 4.3 boxes with LVS (from > http://mirror.centos.org/centos/4/csgfs/ ) but all boxes died withe this > error frequently: > > kernel panic - not syncing: fs/block_dev.c:396: spin_lock > (fs/block_dev.c:c0361c0) already locked by fs/block_dev.c/287. > > I have read from this thread http://threebit.net/mail-archive/centos/msg00243.html that this is an unsolved problem. > > So i have installed the kernel-2.6.9-39.EL from testing repo, but the box diedi with another error: >Johnny recommended installing kernel-2.6.9-34.19 from the CentOS Testing repository, it resolves a bunch of the spinlock errors. NB: it's a testing kernel and I was warned accordingly. :)> Jul 2 16:40:40 lvs2 kernel: Bad page state at free_hot_cold_page (in process 'smtp', page c10a2140) > Jul 2 16:40:40 lvs2 kernel: flags:0x20000014 mapping:00000000 mapcount:256 count:0 > Jul 2 16:40:40 lvs2 kernel: Backtrace: > > So LVS is unusable because after of no more hourse of running the kernel go to panic. >If you're already running kernel-2.6.9-34.19 and not kernel-2.6.9-39.EL as you've stated above, I had problems getting kernel-2.6.9-34.19 to boot on certain hardware. IBM eServer e300s to be specific. http://lists.centos.org/pipermail/centos/2006-May/064305.html We switched all LVS boxes to Dell 1650s running kernel-2.6.9-34.19 a few months back and haven't had any panics since (touch wood). Will.