Hi,
I have a problematic CentOS XEN server and hope someone could point me
in the right direction to optimize it a bit.
The server runs on a Core2Quad 9300, with 8GB RAM (max motherboard can
take, 1U chassis) on an Intel motherboard with a 1TB SATA HDD.
dom0 is set to 512MB limit with a few small XEM VM''s running:
root@zaxen01:[~]$ xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 512 4 r----- 96.5
actionco.vm 3 1519 1 -b---- 14.8
byracers.vm 4 511 1 -b---- 85.7
ns1 5 511 1 -b---- 22.3
picturestravel 6 255 1 -b---- 13.3
rafttheworld 7 255 1 -b---- 11.3
zafepres.vm 8 511 1 -b---- 19.0
the server itself seems to eat up a lot of resources:
root@zaxen01:[~]$ free -m
total used free shared buffers cached
Mem: 512 472 39 0 13 215
-/+ buffers/cache: 244 268
Swap: 4095 0 4095[/CODE]
Yet, it only has XEN, Webmin (since it''s a CloudMin XEN server), Exim,
Apache and a few other services running:
root@zaxen01:[~]$ chkconfig --list |grep "3:on" |awk ''{print
$1}'' |sort
acpid
auditd
crond
csf
dhcpd
exim
haldaemon
httpd
iptables
iscsi
iscsid
kudzu
lfd
lvm2-monitor
mdmonitor
network
qemu
restorecond
setroubleshoot
smartd
snmpd
sshd
syslog
sysstat
webmin
xend
xendomains
Is there anything I can optimize on such a server?
The server runs CentOS 5.5 x64:
root@zaxen01:[~]$ cat /etc/redhat-release
CentOS release 5.5 (Final)
root@zaxen01:[~]$ uname -a
Linux zaxen01.softdux.com 2.6.18-194.32.1.el5xen #1 SMP Wed Jan 5
18:44:24 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
with Xen version 3.1.2-194.32.1.el5
And there''s the xm dmesg output:
Xen version 3.1.2-194.32.1.el5 (mockbuild@centos.org) (gcc version
4.1.2 20080704 (Red Hat 4.1.2-48)) Wed Jan 5 17:43:03 EST 2011
Latest ChangeSet: unavailable
(XEN) Command line: dom0_mem=512M
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN) Found 1 MBR signatures
(XEN) Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 000000000008f000 (usable)
(XEN) 000000000008f000 - 00000000000a0000 (reserved)
(XEN) 00000000000e0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 00000000cf53f000 (usable)
(XEN) 00000000cf53f000 - 00000000cf54b000 (reserved)
(XEN) 00000000cf54b000 - 00000000cf620000 (usable)
(XEN) 00000000cf620000 - 00000000cf6e8000 (ACPI NVS)
(XEN) 00000000cf6e8000 - 00000000cf6ec000 (usable)
(XEN) 00000000cf6ec000 - 00000000cf6f1000 (ACPI data)
(XEN) 00000000cf6f1000 - 00000000cf6f2000 (usable)
(XEN) 00000000cf6f2000 - 00000000cf6ff000 (ACPI data)
(XEN) 00000000cf6ff000 - 00000000cf700000 (usable)
(XEN) 00000000cf700000 - 00000000d0000000 (reserved)
(XEN) 00000000fff00000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 0000000230000000 (usable)
(XEN) System RAM: 8181MB (8378020kB)
(XEN) Xen heap: 13MB (13720kB)
(XEN) Domain heap initialised: DMA width 32 bits
(XEN) Processor #0 7:7 APIC version 20
(XEN) Processor #2 7:7 APIC version 20
(XEN) Processor #1 7:7 APIC version 20
(XEN) Processor #3 7:7 APIC version 20
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2485.797 MHz processor.
(XEN) HVM: VMX enabled
(XEN) VMX: MSR intercept bitmap enabled
(XEN) I/O virtualisation disabled
(XEN) CPU0: Intel(R) Core(TM)2 Quad CPU Q9300 @ 2.50GHz stepping 07
(XEN) Booting processor 1/2 eip 90000
(XEN) CPU1: Intel(R) Core(TM)2 Quad CPU Q9300 @ 2.50GHz stepping 07
(XEN) Booting processor 2/1 eip 90000
(XEN) CPU2: Intel(R) Core(TM)2 Quad CPU Q9300 @ 2.50GHz stepping 07
(XEN) Booting processor 3/3 eip 90000
(XEN) CPU3: Intel(R) Core(TM)2 Quad CPU Q9300 @ 2.50GHz stepping 07
(XEN) Total of 4 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) Platform timer overflows in 2 jiffies.
(XEN) Platform timer is 1.193MHz PIT
(XEN) Brought up 4 CPUs
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0xffffffff80200000 memsz=0x2f4d70
(XEN) elf_parse_binary: phdr: paddr=0xffffffff804f4d80 memsz=0x14c510
(XEN) elf_parse_binary: phdr: paddr=0xffffffff80642000 memsz=0xc08
(XEN) elf_parse_binary: phdr: paddr=0xffffffff80644000 memsz=0x11be8c
(XEN) elf_parse_binary: memory: 0xffffffff80200000 -> 0xffffffff8075fe8c
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff80200000
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff80206000
(XEN) elf_xen_parse_note: FEATURES
"writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_k
ernel"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_addr_calc_check: addresses:
(XEN) virt_base = 0xffffffff80000000
(XEN) elf_paddr_offset = 0xffffffff80000000
(XEN) virt_offset = 0x0
(XEN) virt_kstart = 0xffffffff80200000
(XEN) virt_kend = 0xffffffff8075fe8c
(XEN) virt_entry = 0xffffffff80200000
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 ->
0xffffffff8075fe8c
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000222000000->0000000224000000 (122880 pages
to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
--
Kind Regards
Rudi Ahlers
SoftDux
Website: http://www.SoftDux.com
Technical Blog: http://Blog.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532
--
Kind Regards
Rudi Ahlers
SoftDux
Website: http://www.SoftDux.com
Technical Blog: http://Blog.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
On Wed, Feb 23, 2011 at 01:29:17AM +0200, Rudi Ahlers wrote:> Hi, > > I have a problematic CentOS XEN server and hope someone could point me > in the right direction to optimize it a bit. > > The server runs on a Core2Quad 9300, with 8GB RAM (max motherboard can > take, 1U chassis) on an Intel motherboard with a 1TB SATA HDD. > > dom0 is set to 512MB limit with a few small XEM VM''s running: > > > > the server itself seems to eat up a lot of resources: >use "ps wfaux" and check what dom0 processes are using the memory. You could also run "top" and sort by memory usage. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of Rudi Ahlers > Subject: [Xen-users] how to optimize CentOS XEN dom0? > > the server itself seems to eat up a lot of resources: > > root@zaxen01:[~]$ free -m > total used free shared bufferscached> Mem: 512 472 39 0 13215> -/+ buffers/cache: 244 268 > Swap: 4095 0 4095[/CODE]Really? 244MB memory committed, it looks like, aside from buffer cache. Do you think that''s excessive? I''ve commonly heard that 256MB is the bare minimum to run a CentOS 5.x image. Nothing has swapped yet, so you should have ample headroom. If you were to shrink memory a little more, it would begin to swap seldom-used pages. Since you''re running Apache, see if you can use the worker MPM. It tends to spawn fewer processes (thus eating less RAM) than prefork. If you have to stick with prefork, turn down StartServers, MinSpareServers and MaxSpareServers to the minimum you can comfortably handle. -Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Feb 23, 2011 at 3:57 PM, Jeff Sturm <jeff.sturm@eprize.com> wrote:>> -----Original Message----- >> From: xen-users-bounces@lists.xensource.com [mailto:xen-users- >> bounces@lists.xensource.com] On Behalf Of Rudi Ahlers >> Subject: [Xen-users] how to optimize CentOS XEN dom0? >> >> the server itself seems to eat up a lot of resources: >> >> root@zaxen01:[~]$ free -m >> total used free shared buffers > cached >> Mem: 512 472 39 0 13 > 215 >> -/+ buffers/cache: 244 268 >> Swap: 4095 0 4095[/CODE] > > Really? 244MB memory committed, it looks like, aside from buffer cache. > Do you think that''s excessive? I''ve commonly heard that 256MB is the > bare minimum to run a CentOS 5.x image.These stats were taken a few minutes after the server was hard rebooted cause it didn''t respond anything on the network, internet or even console.> > Nothing has swapped yet, so you should have ample headroom. If you were > to shrink memory a little more, it would begin to swap seldom-used > pages. > > Since you''re running Apache, see if you can use the worker MPM. It > tends to spawn fewer processes (thus eating less RAM) than prefork. If > you have to stick with prefork, turn down StartServers, MinSpareServers > and MaxSpareServers to the minimum you can comfortably handle. >I already tuned the Apache as follows: StartServers 4 MinSpareServers 5 MaxSpareServers 10 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 Apache is used for CloudMin, so there''s no actual websites running on the server.> -Jeff > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: Rudi Ahlers [mailto:Rudi@SoftDux.com] > Sent: Wednesday, February 23, 2011 9:17 AM > To: Jeff Sturm > Cc: xen-users > Subject: Re: [Xen-users] how to optimize CentOS XEN dom0? > > These stats were taken a few minutes after the server was hardrebooted cause it> didn''t respond anything on the network, internet or even console.I see. Did the dom0 log anything (e.g. /var/log/messages) before it died? Might be helpful to install sysstat. Using sar, I can see performance trends right up to the point where a server died, including load average and memory growth. -Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Feb 23, 2011 at 6:13 PM, Jeff Sturm <jeff.sturm@eprize.com> wrote:>> -----Original Message----- >> From: Rudi Ahlers [mailto:Rudi@SoftDux.com] >> Sent: Wednesday, February 23, 2011 9:17 AM >> To: Jeff Sturm >> Cc: xen-users >> Subject: Re: [Xen-users] how to optimize CentOS XEN dom0? >> >> These stats were taken a few minutes after the server was hard > rebooted cause it >> didn''t respond anything on the network, internet or even console. > > I see. Did the dom0 log anything (e.g. /var/log/messages) before it > died? > > Might be helpful to install sysstat. Using sar, I can see performance > trends right up to the point where a server died, including load average > and memory growth. > > -Jeff > > > > _______________________________________________I see a lot of these errors in /var/log/messages shortly before it crashed: Feb 22 15:27:14 zaxen01 kernel: HighMem: empty Feb 22 15:27:14 zaxen01 kernel: 918 pagecache pages Feb 22 15:27:14 zaxen01 kernel: Swap cache: add 2248198, delete 2248009, find 160685591/160898897, race 0+453 Feb 22 15:27:14 zaxen01 kernel: Free swap = 0kB Feb 22 15:27:14 zaxen01 kernel: Total swap = 4194296kB Feb 22 15:27:14 zaxen01 kernel: Free swap: 0kB Feb 22 15:27:14 zaxen01 kernel: 133120 pages of RAM Feb 22 15:27:14 zaxen01 kernel: 22818 reserved pages Feb 22 15:27:16 zaxen01 kernel: 105840 pages shared Feb 22 15:27:16 zaxen01 kernel: 189 pages swap cached Feb 22 15:27:17 zaxen01 kernel: Out of memory: Killed process 17464, UID 99, (sendmail). Feb 23 00:35:38 zaxen01 syslogd 1.4.1: restart. Feb 23 00:35:38 zaxen01 kernel: klogd 1.4.1, log source = /proc/kmsg started. Feb 23 00:35:38 zaxen01 kernel: Bootdata ok (command line is ro root=/dev/System/root rhgb quiet xencons=tty6) Feb 23 00:35:38 zaxen01 kernel: Linux version 2.6.18-194.32.1.el5xen (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Jan 5 18:44:24 EST 2011 Feb 23 00:35:38 zaxen01 kernel: BIOS-provided physical RAM map: Feb 23 00:35:38 zaxen01 kernel: Xen: 0000000000000000 - 0000000020800000 (usable) Feb 23 00:35:38 zaxen01 kernel: DMI 2.4 present. Feb 23 00:35:38 zaxen01 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) Feb 23 00:35:38 zaxen01 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled) Feb 23 00:35:38 zaxen01 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled) Feb 23 00:35:38 zaxen01 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled) Feb 23 00:35:38 zaxen01 kernel: ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1]) Feb 23 00:35:38 zaxen01 kernel: ACPI: LAPIC_NMI (acpi_id[0x02] dfl dfl lint[0x1]) Feb 23 00:35:38 zaxen01 kernel: ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0]) Feb 23 00:35:38 zaxen01 kernel: IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23 Feb 23 00:35:38 zaxen01 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) Feb 23 00:35:38 zaxen01 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) Feb 23 00:35:38 zaxen01 kernel: Setting APIC routing to xen Feb 23 00:35:38 zaxen01 kernel: Using ACPI (MADT) for SMP configuration information Feb 23 00:35:38 zaxen01 kernel: Allocating PCI resources starting at d4000000 (gap: d0000000:2ff00000) Feb 23 00:35:38 zaxen01 kernel: Built 1 zonelists. Total pages: 133120 Feb 23 00:35:38 zaxen01 kernel: Kernel command line: ro root=/dev/System/root rhgb quiet xencons=tty6 Feb 23 00:35:38 zaxen01 kernel: Initializing CPU#0 Feb 23 00:35:38 zaxen01 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes) -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Juergen Gotteswinter
2011-Feb-25 09:41 UTC
Re: [Xen-users] how to optimize CentOS XEN dom0?
that means that your host is running out of memory... try to let more ram left for the dom0 Am 25.02.11 10:27, schrieb Rudi Ahlers:> On Wed, Feb 23, 2011 at 6:13 PM, Jeff Sturm <jeff.sturm@eprize.com> wrote: >>> -----Original Message----- >>> From: Rudi Ahlers [mailto:Rudi@SoftDux.com] >>> Sent: Wednesday, February 23, 2011 9:17 AM >>> To: Jeff Sturm >>> Cc: xen-users >>> Subject: Re: [Xen-users] how to optimize CentOS XEN dom0? >>> >>> These stats were taken a few minutes after the server was hard >> rebooted cause it >>> didn''t respond anything on the network, internet or even console. >> >> I see. Did the dom0 log anything (e.g. /var/log/messages) before it >> died? >> >> Might be helpful to install sysstat. Using sar, I can see performance >> trends right up to the point where a server died, including load average >> and memory growth. >> >> -Jeff >> >> >> >> _______________________________________________ > > > > I see a lot of these errors in /var/log/messages shortly before it crashed: > > > > Feb 22 15:27:14 zaxen01 kernel: HighMem: empty > Feb 22 15:27:14 zaxen01 kernel: 918 pagecache pages > Feb 22 15:27:14 zaxen01 kernel: Swap cache: add 2248198, delete > 2248009, find 160685591/160898897, race 0+453 > Feb 22 15:27:14 zaxen01 kernel: Free swap = 0kB > Feb 22 15:27:14 zaxen01 kernel: Total swap = 4194296kB > Feb 22 15:27:14 zaxen01 kernel: Free swap: 0kB > Feb 22 15:27:14 zaxen01 kernel: 133120 pages of RAM > Feb 22 15:27:14 zaxen01 kernel: 22818 reserved pages > Feb 22 15:27:16 zaxen01 kernel: 105840 pages shared > Feb 22 15:27:16 zaxen01 kernel: 189 pages swap cached > Feb 22 15:27:17 zaxen01 kernel: Out of memory: Killed process 17464, > UID 99, (sendmail). > Feb 23 00:35:38 zaxen01 syslogd 1.4.1: restart. > Feb 23 00:35:38 zaxen01 kernel: klogd 1.4.1, log source = /proc/kmsg started. > Feb 23 00:35:38 zaxen01 kernel: Bootdata ok (command line is ro > root=/dev/System/root rhgb quiet xencons=tty6) > Feb 23 00:35:38 zaxen01 kernel: Linux version 2.6.18-194.32.1.el5xen > (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat > 4.1.2-48)) #1 SMP > Wed Jan 5 18:44:24 EST 2011 > Feb 23 00:35:38 zaxen01 kernel: BIOS-provided physical RAM map: > Feb 23 00:35:38 zaxen01 kernel: Xen: 0000000000000000 - > 0000000020800000 (usable) > Feb 23 00:35:38 zaxen01 kernel: DMI 2.4 present. > Feb 23 00:35:38 zaxen01 kernel: ACPI: LAPIC (acpi_id[0x01] > lapic_id[0x00] enabled) > Feb 23 00:35:38 zaxen01 kernel: ACPI: LAPIC (acpi_id[0x03] > lapic_id[0x02] enabled) > Feb 23 00:35:38 zaxen01 kernel: ACPI: LAPIC (acpi_id[0x02] > lapic_id[0x01] enabled) > Feb 23 00:35:38 zaxen01 kernel: ACPI: LAPIC (acpi_id[0x04] > lapic_id[0x03] enabled) > Feb 23 00:35:38 zaxen01 kernel: ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl > lint[0x1]) > Feb 23 00:35:38 zaxen01 kernel: ACPI: LAPIC_NMI (acpi_id[0x02] dfl dfl > lint[0x1]) > Feb 23 00:35:38 zaxen01 kernel: ACPI: IOAPIC (id[0x02] > address[0xfec00000] gsi_base[0]) > Feb 23 00:35:38 zaxen01 kernel: IOAPIC[0]: apic_id 2, version 32, > address 0xfec00000, GSI 0-23 > Feb 23 00:35:38 zaxen01 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 > global_irq 2 dfl dfl) > Feb 23 00:35:38 zaxen01 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 > global_irq 9 high level) > Feb 23 00:35:38 zaxen01 kernel: Setting APIC routing to xen > Feb 23 00:35:38 zaxen01 kernel: Using ACPI (MADT) for SMP > configuration information > Feb 23 00:35:38 zaxen01 kernel: Allocating PCI resources starting at > d4000000 (gap: d0000000:2ff00000) > Feb 23 00:35:38 zaxen01 kernel: Built 1 zonelists. Total pages: 133120 > Feb 23 00:35:38 zaxen01 kernel: Kernel command line: ro > root=/dev/System/root rhgb quiet xencons=tty6 > Feb 23 00:35:38 zaxen01 kernel: Initializing CPU#0 > Feb 23 00:35:38 zaxen01 kernel: PID hash table entries: 4096 (order: > 12, 32768 bytes) > > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users