Hello, We have 50+ servers with the common configuration (CentOS 5.x 64bit + Xen 3.4.3) in different DC. Recently some of these servers have started to reboot with kernel panic. Please check the dmesg output after the panic below. Looking forward to get any ideas how to solve this issue. Thank you. # xm info : release : 2.6.18-274.3.1.el5xen version : #1 SMP Tue Sep 6 20:57:11 EDT 2011 machine : x86_64 nr_cpus : 8 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2533 hw_caps : bfebfbff:28100800:00000000:00000340:0098e3fd:00000000:00000001:00000000 virt_caps : hvm total_memory : 32759 free_memory : 16822 node_to_cpu : node0:0-7 node_to_memory : node0:16822 xen_major : 3 xen_minor : 4 xen_extra : .3 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-50) cc_compile_by : root cc_compile_domain : gitco.tld cc_compile_date : Sun Jun 19 13:52:00 CEST 2011 xend_config_format : 4 Вот пример вывода ядра перед паднием: Oct 4 16:46:00 server Unable to handle kernel NULL pointer dereference Oct 4 16:46:00 server at 00000000000003c8 RIP: Oct 4 16:46:00 server [<ffffffff88652383>] :bridge:__br_deliver+0xcd/0xfb Oct 4 16:46:00 server PGD 1a8bd067 Oct 4 16:46:00 server PUD 132e9067 Oct 4 16:46:00 server PMD 0 Oct 4 16:46:00 server Oct 4 16:46:00 server Oops: 0000 [1] Oct 4 16:46:00 server SMP Oct 4 16:46:00 server Oct 4 16:46:00 server last sysfs file: /devices/xen-backend/vbd-2-768/statistics/wr_sect Oct 4 16:46:00 server CPU 1 Oct 4 16:46:00 server Oct 4 16:46:00 server Modules linked in: Oct 4 16:46:00 server netconsole Oct 4 16:46:00 server ebtable_filter Oct 4 16:46:00 server ebtables Oct 4 16:46:00 server tun Oct 4 16:46:00 server bridge Oct 4 16:46:00 server netloop Oct 4 16:46:00 server netbk Oct 4 16:46:00 server blktap Oct 4 16:46:00 server blkbk Oct 4 16:46:00 server ipt_REDIRECT Oct 4 16:46:00 server ipt_owner Oct 4 16:46:00 server ip_nat_ftp Oct 4 16:46:00 server ip_conntrack_ftp Oct 4 16:46:00 server xt_state Oct 4 16:46:00 server xt_length Oct 4 16:46:00 server ipt_ttl Oct 4 16:46:00 server xt_tcpmss Oct 4 16:46:00 server ipt_TCPMSS Oct 4 16:46:00 server xt_multiport Oct 4 16:46:00 server xt_limit Oct 4 16:46:00 server ipt_LOG Oct 4 16:46:00 server ipt_TOS Oct 4 16:46:00 server ipt_tos Oct 4 16:46:00 server ipt_REJECT Oct 4 16:46:00 server iptable_filter Oct 4 16:46:00 server iptable_nat Oct 4 16:46:00 server ip_nat Oct 4 16:46:00 server ip_conntrack Oct 4 16:46:00 server nfnetlink Oct 4 16:46:00 server ip_tables Oct 4 16:46:00 server x_tables Oct 4 16:46:00 server loop Oct 4 16:46:00 server dm_mirror Oct 4 16:46:00 server dm_multipath Oct 4 16:46:00 server scsi_dh Oct 4 16:46:00 server video Oct 4 16:46:00 server backlight Oct 4 16:46:00 server sbs Oct 4 16:46:01 server power_meter Oct 4 16:46:01 server hwmon Oct 4 16:46:01 server i2c_ec Oct 4 16:46:01 server dell_wmi Oct 4 16:46:01 server wmi Oct 4 16:46:01 server button Oct 4 16:46:01 server battery Oct 4 16:46:01 server asus_acpi Oct 4 16:46:01 server ac Oct 4 16:46:01 server parport_pc Oct 4 16:46:01 server lp Oct 4 16:46:01 server parport Oct 4 16:46:01 server sr_mod Oct 4 16:46:01 server cdrom Oct 4 16:46:01 server sg Oct 4 16:46:01 server snd_hda_intel Oct 4 16:46:01 server snd_seq_dummy Oct 4 16:46:01 server snd_seq_oss Oct 4 16:46:01 server snd_seq_midi_event Oct 4 16:46:01 server snd_seq Oct 4 16:46:01 server snd_seq_device Oct 4 16:46:01 server snd_pcm_oss Oct 4 16:46:01 server snd_mixer_oss Oct 4 16:46:01 server snd_pcm Oct 4 16:46:01 server snd_timer Oct 4 16:46:01 server snd_page_alloc Oct 4 16:46:01 server tpm_tis Oct 4 16:46:01 server snd_hwdep Oct 4 16:46:01 server i2c_i801 Oct 4 16:46:01 server snd Oct 4 16:46:01 server e1000e Oct 4 16:46:01 server tpm Oct 4 16:46:01 server serial_core Oct 4 16:46:01 server soundcore Oct 4 16:46:01 server i2c_core Oct 4 16:46:01 server pcspkr Oct 4 16:46:01 server serio_raw Oct 4 16:46:01 server tpm_bios Oct 4 16:46:01 server dm_raid45 Oct 4 16:46:01 server dm_message Oct 4 16:46:01 server dm_region_hash Oct 4 16:46:01 server dm_log Oct 4 16:46:01 server dm_mod Oct 4 16:46:01 server dm_mem_cache Oct 4 16:46:01 server ahci Oct 4 16:46:01 server libata Oct 4 16:46:01 server shpchp Oct 4 16:46:01 server megaraid_sas Oct 4 16:46:01 server sd_mod Oct 4 16:46:01 server scsi_mod Oct 4 16:46:01 server ext3 Oct 4 16:46:01 server jbd Oct 4 16:46:01 server uhci_hcd Oct 4 16:46:01 server ohci_hcd Oct 4 16:46:01 server ehci_hcd Oct 4 16:46:01 server Oct 4 16:46:01 server Pid: 26388, comm: qemu-dm Not tainted 2.6.18-274.3.1.el5xen #1 Oct 4 16:46:01 server RIP: e030:[<ffffffff88652383>] Oct 4 16:46:01 server [<ffffffff88652383>] :bridge:__br_deliver+0xcd/0xfb Oct 4 16:46:01 server RSP: e02b:ffff8800065d3b18 EFLAGS: 00010296 Oct 4 16:46:01 server RAX: 0000000000000000 RBX: ffff880034831000 RCX: ffff880009b95bc0 Oct 4 16:46:01 server RDX: ffffffff80000000 RSI: ffff880009b95bc0 RDI: ffff88003846be00 Oct 4 16:46:01 server RBP: ffff880037dfa500 R08: 0000000002000000 R09: ffffffff88652195 Oct 4 16:46:01 server R10: 0000000080000000 R11: ffffffff8853f9c0 R12: ffff880009b95c18 Oct 4 16:46:01 server R13: ffffffff80567f00 R14: ffff880037dfa000 R15: 0000000000000000 Oct 4 16:46:01 server FS: 00002ba775bc9bf0(0000) GS:ffffffff8062e080(0000) knlGS:0000000000000000 Oct 4 16:46:01 server CS: e033 DS: 0000 ES: 0000 Oct 4 16:46:01 server Process qemu-dm (pid: 26388, threadinfo ffff88003e064000, task ffff88001b614080) Oct 4 16:46:01 server Stack: Oct 4 16:46:01 server ffff880009b95bc0 Oct 4 16:46:01 server ffff880009b95bc0 Oct 4 16:46:01 server ffff880037dfa500 Oct 4 16:46:01 server ffffffff88651294 Oct 4 16:46:01 server Oct 4 16:46:01 server ffff88001df131c0 Oct 4 16:46:01 server ffff880009b95bc0 Oct 4 16:46:01 server ffffffff808d3870 Oct 4 16:46:01 server ffffffff80424b93 Oct 4 16:46:01 server Oct 4 16:46:01 server ffff880033474460 Oct 4 16:46:01 server ffff880009b95bc0 Oct 4 16:46:01 server Oct 4 16:46:01 server Call Trace: Oct 4 16:46:01 server <IRQ> Oct 4 16:46:01 server [<ffffffff88651294>] :bridge:br_dev_xmit+0xc7/0xdb Oct 4 16:46:01 server [<ffffffff80424b93>] dev_hard_start_xmit+0x1b7/0x28a Oct 4 16:46:01 server [<ffffffff80230b81>] dev_queue_xmit+0x31f/0x3ef Oct 4 16:46:01 server [<ffffffff80233073>] ip_output+0x29a/0x2dd Oct 4 16:46:01 server [<ffffffff80441cb7>] ip_forward+0x24f/0x2bd Oct 4 16:46:01 server [<ffffffff80236cdc>] ip_rcv+0x539/0x57c Oct 4 16:46:01 server [<ffffffff80221590>] netif_receive_skb+0x495/0x4c4 Oct 4 16:46:01 server [<ffffffff88652efd>] :bridge:br_handle_frame_finish+0x1bc/0x1d3 Oct 4 16:46:01 server [<ffffffff88657123>] :bridge:br_nf_pre_routing_finish+0x2e9/0x2f8 Oct 4 16:46:02 server [<ffffffff88656e3a>] :bridge:br_nf_pre_routing_finish+0x0/0x2f8 Oct 4 16:46:02 server [<ffffffff80258542>] nf_hook_slow+0x58/0xbc Oct 4 16:46:02 server [<ffffffff88656e3a>] :bridge:br_nf_pre_routing_finish+0x0/0x2f8 Oct 4 16:46:02 server [<ffffffff8028829d>] find_busiest_group+0x1db/0x44a Oct 4 16:46:02 server [<ffffffff88657d20>] :bridge:br_nf_pre_routing+0x600/0x61c Oct 4 16:46:02 server [<ffffffff802352d4>] nf_iterate+0x41/0x7d Oct 4 16:46:02 server [<ffffffff88652d41>] :bridge:br_handle_frame_finish+0x0/0x1d3 Oct 4 16:46:02 server [<ffffffff80258542>] nf_hook_slow+0x58/0xbc Oct 4 16:46:02 server [<ffffffff88652d41>] :bridge:br_handle_frame_finish+0x0/0x1d3 Oct 4 16:46:02 server [<ffffffff88653082>] :bridge:br_handle_frame+0x16e/0x1a4 Oct 4 16:46:02 server [<ffffffff802214a3>] netif_receive_skb+0x3a8/0x4c4 Oct 4 16:46:02 server [<ffffffff802488d3>] try_to_wake_up+0x392/0x3a4 Oct 4 16:46:02 server [<ffffffff802319a4>] process_backlog+0x9b/0x104 Oct 4 16:46:02 server [<ffffffff8020d0a1>] net_rx_action+0xb4/0x1c6 Oct 4 16:46:02 server [<ffffffff80212f06>] __do_softirq+0x8d/0x13b Oct 4 16:46:02 server [<ffffffff8025fda4>] call_softirq+0x1c/0x278 Oct 4 16:46:02 server <EOI> Oct 4 16:46:02 server [<ffffffff8026db69>] do_softirq+0x31/0x90 Oct 4 16:46:02 server [<ffffffff8024fc30>] netif_rx_ni+0x19/0x1d Oct 4 16:46:02 server [<ffffffff8866d51d>] :tun:tun_chr_writev+0x3b4/0x402 Oct 4 16:46:02 server [<ffffffff8020bf28>] free_hot_cold_page+0x133/0x175 Oct 4 16:46:02 server [<ffffffff8866d585>] :tun:tun_chr_write+0x1a/0x1f Oct 4 16:46:02 server [<ffffffff8021747d>] vfs_write+0xce/0x174 Oct 4 16:46:02 server [<ffffffff80217cc6>] sys_write+0x45/0x6e Oct 4 16:46:02 server [<ffffffff8025f106>] system_call+0x86/0x8b Oct 4 16:46:02 server [<ffffffff8025f080>] system_call+0x0/0x8b Oct 4 16:46:02 server Oct 4 16:46:02 server Oct 4 16:46:02 server Code: Oct 4 16:46:02 server 48 Oct 4 16:46:02 server 8b Oct 4 16:46:02 server 80 Oct 4 16:46:02 server c8 Oct 4 16:46:02 server 03 Oct 4 16:46:02 server 00 Oct 4 16:46:02 server 00 Oct 4 16:46:02 server 48 Oct 4 16:46:02 server 85 Oct 4 16:46:02 server c0 Oct 4 16:46:02 server 74 Oct 4 16:46:02 server 1e Oct 4 16:46:02 server 48 Oct 4 16:46:02 server 8b Oct 4 16:46:02 server 50 Oct 4 16:46:02 server 38 Oct 4 16:46:02 server 48 Oct 4 16:46:02 server 8b Oct 4 16:46:02 server 45 Oct 4 16:46:02 server 18 Oct 4 16:46:02 server Oct 4 16:46:02 server RIP Oct 4 16:46:02 server [<ffffffff88652383>] :bridge:__br_deliver+0xcd/0xfb Oct 4 16:46:02 server RSP <ffff8800065d3b18> Oct 4 16:46:02 server CR2: 00000000000003c8 Oct 4 16:46:02 server Oct 4 16:46:02 server Kernel panic - not syncing: Fatal exception Oct 4 16:46:02 server _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users