I have a kvm virtual host running on what will become CentOS 6 with 12GB
of memory and a Quad Xeon X5560 2.8Ghz . The store for virtual
machines will be a software raid 6 array of 6 disks with an LVM layered
on top. I'm not initially planning any major overcommitment of
resources, though there could be a need for some overcommitment with a
light workload on the guests.
In recent years people seem to configure a wide range of different swap
allocations. I was thinking initially to spread swap across seperate
non-raid partitions on 4 of these disks, but the downside of that is if
I put 2gb on each disk, then I can only swap processes that will fit in
2gb swap space. Also, if one of the disks fails, I have to reboot if
anything was swapped to that drive.
My questions are as follows:
1. What experience are others having with putting swap space on raid
partitions? I was thinking about maybe swapping on a raid10
device, otherwise an LVM spanning multiple drives.
2. In practice, what kinds of swap allocation are people finding
useful for a kvm virtual host of this size?
I definitely don't want a system that is so overcommited that
performance is impacted, but if some overcommitment is reasonable for
VM's that have light workload, then I consider that. I can increase
system resources when that becomes necessary.
Nataraj
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos/attachments/20110314/4cc70b73/attachment-0001.html>
On Mon, Mar 14, 2011 at 5:24 PM, Nataraj <incoming-centos at rjl.com> wrote: [snip]> ?In recent years people seem to configure a wide range of different swap > allocations.? I was thinking initially to spread swap across seperate > non-raid partitions on 4 of these disks, but the downside of that is if I > put 2gb on each disk, then I can only swap processes that will fit in 2gb > swap space.? Also, if one of the disks fails, I have to reboot if anything > was swapped to that drive.I'm not certain that's entirely accurate... It's not so much "swap space" as "page space". I.e., there are pages being swapped out rather than entire processes.
on 14:24 Mon 14 Mar, Nataraj (incoming-centos at rjl.com) wrote:> I have a kvm virtual host running on what will become CentOS 6 with 12GB > of memory and a Quad Xeon X5560 2.8Ghz . The store for virtual > machines will be a software raid 6 array of 6 disks with an LVM layered > on top. I'm not initially planning any major overcommitment of > resources, though there could be a need for some overcommitment with a > light workload on the guests. > > In recent years people seem to configure a wide range of different swap > allocations. I was thinking initially to spread swap across seperate > non-raid partitions on 4 of these disks, but the downside of that is if > I put 2gb on each disk, then I can only swap processes that will fit in > 2gb swap space.Incorrect. Linux processes aren't "swapped" to disk (the entire process memory space), but are paged (given memory blocks are swapped out individually). Swap allocated over multiple spindles is effectively striped (treated as one large RAID 0 partition). If you've got SSD, you'll get even better swap performance. For an excellent explanation of how Linux pages / handles memory: http://sourcefrog.net/weblog/software/linux-kernel/swap.html> Also, if one of the disks fails, I have to reboot if > anything was swapped to that drive. > > My questions are as follows: > > 1. What experience are others having with putting swap space on raid > partitions? I was thinking about maybe swapping on a raid10 > device, otherwise an LVM spanning multiple drives.I'd allocate swap to the raw devices rather than the RAID devices, particularly if using SW RAID. In the case of HW RAID, it's a bit of a toss-up. Whatever's easier to manage.> 2. In practice, what kinds of swap allocation are people finding > useful for a kvm virtual host of this size?1-3x RAM is still my rule of thumb.> I definitely don't want a system that is so overcommited that > performance is impacted, but if some overcommitment is reasonable for > VM's that have light workload, then I consider that. I can increase > system resources when that becomes necessary.For this, you'll want to set the overcommit and swappiness kernel parameters. Amount of swap space is a secondary consideration. How much swap you /have/ and how much swap you're /doing/ are two different things. -- Dr. Ed Morbius, Chief Scientist / | Robot Wrangler / Staff Psychologist | When you seek unlimited power Krell Power Systems Unlimited | Go to Krell!
On Mon, Mar 14, 2011 at 5:24 PM, Nataraj <incoming-centos at rjl.com> wrote:> I have a kvm virtual host running on what will become CentOS 6 with 12GB of > memory and a Quad Xeon? X5560 2.8Ghz .? The store for virtual machines will > be a software raid 6 array of 6 disks with an LVM layered on top.? I'm not > initially planning any major overcommitment of resources, though there could > be a need for some overcommitment with a light workload on the guests. > > ?In recent years people seem to configure a wide range of different swap > allocations.? I was thinking initially to spread swap across seperate > non-raid partitions on 4 of these disks, but the downside of that is if I > put 2gb on each disk, then I can only swap processes that will fit in 2gb > swap space.? Also, if one of the disks fails, I have to reboot if anything > was swapped to that drive. > > My questions are as follows: > > What experience are others having with putting swap space on raid > partitions?? I was thinking about maybe swapping on a raid10 device, > otherwise an LVM spanning multiple drives. > In practice, what kinds of swap allocation are people finding useful for a > kvm virtual host of this size? > > I definitely don't want a system that is so overcommited that performance is > impacted, but if some overcommitment is reasonable for VM's that have light > workload, then I consider that.? I can increase system resources when that > becomes necessary. > > NatarajI recommend against using any type of parity-based RAID (5 or 6) for any workload, period. RAID10 is the only way to go for everything these days. VMs often run into IO issues, and RAID[56] is going to make that problem worse. Also, RAID10 in software shouldn't have much performance impact at all, as opposed to RAID[56] which will have a big load in software. I wouldn't worry as much about swap, as others have said. Money spent adding another device for swap would be better spent on more RAM. However, I usually put swap on the same device as everything else. It might not strictly be best practice, but again, if you're hitting swap, you have bigger problems.