Shvayakov A.
2010-May-23 18:30 UTC
[Xen-users] Test - how many XEN VPS can run on my server
Hello, I want to know how many XEN VPS can run on my server: CPU: 2 x Xeon X5650 2.67GHz MEMORY: 36Gb HDD: Adaptec 5405Z MaxIQ (SSD Cache) RAID10 2TB Debian GNU/Linux 5.0.4 (lenny), linux-image-2.6.26-2-xen-amd64, xen-hypervisor-3.2-1, xen-utils-3.2-1 I gradually add 5 VPS and get parameters of system and test results. On each VPS (1 CPU 512Mb ) run autorepeat the test apache benchmark and generator a small load on the hdd (rsync) Up to 30 VPS, I observe a smooth change of parameters, but after 30 VPS the tests results are beginning to jump and I can see that much of the VPS is blocked. I have not much experience with XEN This is normal, 30 VPS is my limit, or I have a bug? See attached screenshot _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2010-May-24 03:44 UTC
Re: [Xen-users] Test - how many XEN VPS can run on my server
On Mon, May 24, 2010 at 1:30 AM, Shvayakov A. <a.shvayakov@gmail.com> wrote:> I want to know how many XEN VPS can run on my server: > CPU: 2 x Xeon X5650 2.67GHz > MEMORY: 36Gb > HDD: Adaptec 5405Z MaxIQ (SSD Cache) RAID10 2TB> Up to 30 VPS, I observe a smooth change of parameters, but after 30 VPS > the tests results are beginning to jumpHow did you determine that? apache benchmark results?> and I can see that much of the > VPS is blocked."r" is shown when a domU actually uses a CPU. Since you have two 6 (?) core CPUs, if you enabled hypterhread, it might show up to 24 domUs in "r" state. It''s normal that other domUs will be in "b" state.> I have not much experience with XEN > This is normal, 30 VPS is my limit, or I have a bug?I''d start by looking at disk statistics. It''s possible that you''re running out of disk IOPS. Try "iostat -x 3" on dom0, and see if disk utilization is at 100%. After that, look at dom0''s memory and CPU usage. Is dom0 CPU usage at 100%? is dom0 using swap? -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-May-24 07:30 UTC
Re: [Xen-users] Test - how many XEN VPS can run on my server
On Mon, May 24, 2010 at 10:44:40AM +0700, Fajar A. Nugraha wrote:> On Mon, May 24, 2010 at 1:30 AM, Shvayakov A. <a.shvayakov@gmail.com> wrote: > > I want to know how many XEN VPS can run on my server: > > CPU: 2 x Xeon X5650 2.67GHz > > MEMORY: 36Gb > > HDD: Adaptec 5405Z MaxIQ (SSD Cache) RAID10 2TB > > > Up to 30 VPS, I observe a smooth change of parameters, but after 30 VPS > > the tests results are beginning to jump > > How did you determine that? apache benchmark results? > > > and I can see that much of the > > VPS is blocked. > > "r" is shown when a domU actually uses a CPU. Since you have two 6 (?) > core CPUs, if you enabled hypterhread, it might show up to 24 domUs in > "r" state. It''s normal that other domUs will be in "b" state. > > > I have not much experience with XEN > > This is normal, 30 VPS is my limit, or I have a bug? > > I''d start by looking at disk statistics. It''s possible that you''re > running out of disk IOPS. Try "iostat -x 3" on dom0, and see if disk > utilization is at 100%. > After that, look at dom0''s memory and CPU usage. Is dom0 CPU usage at > 100%? is dom0 using swap? >Also: http://wiki.xensource.com/xenwiki/XenBestPractices -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Shvayakov A.
2010-May-24 08:12 UTC
Re: [Xen-users] Test - how many XEN VPS can run on my server
В Пнд, 24/05/2010 в 10:44 +0700, Fajar A. Nugraha пишет:> On Mon, May 24, 2010 at 1:30 AM, Shvayakov A. <a.shvayakov@gmail.com> wrote: > > I want to know how many XEN VPS can run on my server: > > CPU: 2 x Xeon X5650 2.67GHz > > MEMORY: 36Gb > > HDD: Adaptec 5405Z MaxIQ (SSD Cache) RAID10 2TB > > > Up to 30 VPS, I observe a smooth change of parameters, but after 30 VPS > > the tests results are beginning to jump > > How did you determine that? apache benchmark results?Yes. I observe the apache benchmark results, rsync results, CPU%, ioutil% in Dom0 and DomU Each DomU send reports to Dom0 by scp where I collect them for calculate the average value by script. I hope to calculate the critical number of UPU and was looking for a point nonlinear parameter changes, but I get unstable situation Sometime I get: BUG: soft lockup - CPU#1 stuck for 61s! [sshd:769] BUG: soft lockup - CPU#8 stuck for 61s! [swapper:0] Kernel parameter kernel.softlockup_thresh = 60 and unavailable for change. Not sure but I think that locked DomU not create new reports. Because I get improved average results under the conditions of blocking VPS. In this case, my script uses the previously created reports to calculate. I do not know insofar understandable my stats, see attachment> > and I can see that much of the > > VPS is blocked. > > "r" is shown when a domU actually uses a CPU. Since you have two 6 (?) > core CPUs, if you enabled hypterhread, it might show up to 24 domUs in > "r" state. It''s normal that other domUs will be in "b" state. > > > I have not much experience with XEN > > This is normal, 30 VPS is my limit, or I have a bug? > > I''d start by looking at disk statistics. It''s possible that you''re > running out of disk IOPS. Try "iostat -x 3" on dom0, and see if disk > utilization is at 100%.Yes - ioutil% is growing very rapidly and without any apparent reason> After that, look at dom0''s memory and CPU usage. Is dom0 CPU usage at > 100%? is dom0 using swap?Thank you for your reply, Fajar>_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2010-May-24 08:23 UTC
Re: [Xen-users] Test - how many XEN VPS can run on my server
On Mon, May 24, 2010 at 3:12 PM, Shvayakov A. <a.shvayakov@gmail.com> wrote:> I hope to calculate the critical number of UPU and was looking for a > point nonlinear parameter changes, but I get unstable situation > > Sometime I get: > BUG: soft lockup - CPU#1 stuck for 61s! [sshd:769] > BUG: soft lockup - CPU#8 stuck for 61s! [swapper:0] > > Kernel parameter kernel.softlockup_thresh = 60 and unavailable for > change.linux-image-2.6.26-2-xen-amd64 is known to have some problems, and this might be one of them. I''m not sure if Debian guys even maintain it anymore. If it were me, I''d go with known-to-work configuration with well-maintained kernel. Centos 5.4/RHEL 5.5 is a good start. Since you''re basically only use apache and rsync, it should be sufficient for your test. Another option would be to try newer kernels and hope the problem was fixed there. This might be a little hard though, and does not guarantee to fix your problem.>> I''d start by looking at disk statistics. It''s possible that you''re >> running out of disk IOPS. Try "iostat -x 3" on dom0, and see if disk >> utilization is at 100%. > > Yes - ioutil% is growing very rapidly and without any apparent reasonWhy did you say "without any apparent reason"? AFAIK by default every reads to a file will result in a write to update atime. That''s why I usually set mount option "noatime". Your test should be I/O heavy. In that case the bottleneck would be your disk, not CPU or memory. Having lots of properly-configured disk might help. If the disk is only 4 x 1 TB (for example), then you definitely need to increase the number of disks to support that many I/O-intensive domUs. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users