CentOS - Jun 2015 - LVM hatred, was Re: /boot on a separate partition?

On 06/23/2015 08:10 PM, Marko Vojinovic wrote:
> Ok, you made me curious. Just how dramatic can it be? From where I'm
> sitting, a read/write to a disk takes the amount of time it takes, the
> hardware has a certain physical speed, regardless of the presence of
> LVM. What am I missing?
Well, there's best and worst case scenarios.  Best case for file-backed 
VMs is pre-allocated files.  It takes up more space, and takes a while 
to set up initially, but it skips block allocation and probably some 
fragmentation performance hits later.

Worst case, though, is sparse files.  In such a setup, when you write a 
new file in a guest, the kernel writes the metadata to the journal, then 
writes the file's data block, then flushes the journal to the 
filesystem.  Every one of those writes goes through the host filesystem 
layer, often allocating new blocks, which goes through the host's 
filesystem journal.  If each of those three writes hit blocks not 
previously used, then the host may do three writes for each of them.  In 
that case, one write() in an application in a VM becomes nine disk 
writes in the VM host.

The first time I benchmarked a sparse-file-backed guest vs an LVM backed 
guest, bonnie++ measured block write bandwidth at about 12.5% (1/8) 
native disk write performance.

Yesterday I moved a bunch of VMs from a file-backed virt server (set up 
by someone else) to one that used logical volumes.  Block write speed on 
the old server, measured with bonnie++, was about 21.6MB/s in the guest 
and about 39MB/s on the host.  So, less bad than a few years prior, but 
still bad.  (And yes, all of those numbers are bad.  It's a 3ware 
controller, what do you expect?)

LVM backed guests measure very nearly the same as bare metal 
performance.  After migration, bonnie++ reports about 180MB/s block 
write speed.
> For concreteness, let's say I have a guest machine, with a
> dedicated physical partition for it, on a single drive. Or, I have the
> same thing, only the dedicated partition is inside LVM. Why is there a
> performance difference, and how dramatic is it?
Well, I said that there's a big performance hit to file-backed guests, 
not partition backed guests.  You should see exactly the same disk 
performance on partition backed guests as LV backed guests.

However, partitions have other penalties relative to LVM.

1) If you have a system with a single disk, you have to reboot to add 
partitions for new guests.  Linux won't refresh the partition table on 
the disk it boots from.
2) If you have two disks you can allocate new partitions on the second 
disk without a reboot.  However, your partition has to be contiguous, 
which may be a problem, especially over time if you allocate VMs of 
different sizes.
3) If you want redundancy, partitions on top of RAID is more complex 
than LVM on top of RAID.  As far as I know, partitions on top of RAID 
are subject to the same limitation as in #1.
4) As far as I know, Anaconda can't set up a logical volume that's a 
redundant type, so LVM on top of RAID is the only practical way to 
support redundant storage of your host filesystems.

If you use LVM, you don't have to remember any oddball rules.  You don't
have to reboot to set up new VMs when you have one disk.  You don't have 
to manage partition fragmentation.  Every system, whether it's one disk 
or a RAID set behaves the same way.

Gordon Messmer wrote:
> On 06/23/2015 08:10 PM, Marko Vojinovic wrote:
>> Ok, you made me curious. Just how dramatic can it be? From where
I'm
>> sitting, a read/write to a disk takes the amount of time it takes, the
>> hardware has a certain physical speed, regardless of the presence of
>> LVM. What am I missing?
>
> Well, there's best and worst case scenarios.  Best case for file-backed
> VMs is pre-allocated files.  It takes up more space, and takes a while
> to set up initially, but it skips block allocation and probably some
> fragmentation performance hits later.
>
> Worst case, though, is sparse files.  In such a setup, when you write a
> new file in a guest, the kernel writes the metadata to the journal, then
<MVNCH>

Here's a question: all of the arguments you're giving have to do with
VMs.
Do you have some for straight-on-the-server, non-VM cases?

       mark

On 06/24/2015 11:06 AM, m.roth at 5-cent.us wrote:
> Here's a question: all of the arguments you're giving have to do
with VMs.
> Do you have some for straight-on-the-server, non-VM cases?
Marko sent two messages and suggested that we keep the VM performance 
question as a reply to that one.  My reply to his other message is not 
specific to VMs.

Once upon a time, m.roth at 5-cent.us <m.roth at 5-cent.us>
said:
> Here's a question: all of the arguments you're giving have to do
with VMs.
> Do you have some for straight-on-the-server, non-VM cases?
I've used LVM on servers with hot-swap drives to migrate to new storage
without downtime a number of times.  Add new drives to the system,
configure RAID (software or hardware), pvcreate, vgextend, pvmove,
vgreduce, and pvremove (and maybe a lvextend and resize2fs/xfs_growfs).
Never unmounted a filesystem, just some extra disk I/O.

Even in cases where I had to shutdown or reboot a server to get drives
added, moving data could take a long downtime, but with LVM I can
live-migrate from place to place.

LVM snapshots make it easy to get point-in-time consistent backups,
including databases.  For example, with MySQL, you can freeze and flush
all the databases, snapshot the LV, and release the freeze.  MySQL takes
a brief pause (few seconds), and then you mount and back up the snapshot
for a fully consistent database (only way to do that other than freezing
all writes during a mysqldump, which can take a long time for larger
DBs).  That also avoids the access-time churn (for backup programs that
don't know O_NOATIME, like any that use rsync).

That's server stuff.  On a desktop with a combination of SSD and
"spinning rust" drives, LVM can give you transparent SSD caching of
"hot" data (rather than you having to put some filesystems on SSD and
some on hard drive).

Now, if btrfs ever gets all the kinks worked out (and has a stable
"fsck" for the corner cases), it integrates volume management into the
filesystem, which makes some of the management easier.  I used AdvFS on
DEC/Compaq/HP Tru64 Unix, which had some of that, and it made some of
this easier/faster/smoother.  Btrfs may eventually obsolete a lot of
uses of LVM, but that's down the road.
-- 
Chris Adams <linux at cmadams.net>

On 6/24/2015 1:06 PM, m.roth at 5-cent.us wrote:
> Gordon Messmer wrote:
>> On 06/23/2015 08:10 PM, Marko Vojinovic wrote:
>>> Ok, you made me curious. Just how dramatic can it be? From where
I'm
>>> sitting, a read/write to a disk takes the amount of time it takes,
the
>>> hardware has a certain physical speed, regardless of the presence
of
>>> LVM. What am I missing?
>> Well, there's best and worst case scenarios.  Best case for
file-backed
>> VMs is pre-allocated files.  It takes up more space, and takes a while
>> to set up initially, but it skips block allocation and probably some
>> fragmentation performance hits later.
>>
>> Worst case, though, is sparse files.  In such a setup, when you write a
>> new file in a guest, the kernel writes the metadata to the journal,
then
> <MVNCH>
>
> Here's a question: all of the arguments you're giving have to do
with VMs.
> Do you have some for straight-on-the-server, non-VM cases?
>
>        mark
>
>
Is there an easy to follow "howto" for normal LVM administration
tasks. I get
tired of googling every-time I have to do something I don't remember how to
do
regarding LVM, so I usually just don't bother with it at all.

I believe it has some benefit for my use cases, but I've been reticent to
use
it, since the last time I got LVM problems, I lost everything on the volume, and
had to restore from backups anyway. I suspect I shot myself in the foot, but I
still don't know for sure.

thanks,
-chuck

--

At Wed, 24 Jun 2015 14:06:30 -0400 CentOS mailing list <centos at
centos.org> wrote:
> 
> Gordon Messmer wrote:
> > On 06/23/2015 08:10 PM, Marko Vojinovic wrote:
> >> Ok, you made me curious. Just how dramatic can it be? From where
I'm
> >> sitting, a read/write to a disk takes the amount of time it takes,
the
> >> hardware has a certain physical speed, regardless of the presence
of
> >> LVM. What am I missing?
> >
> > Well, there's best and worst case scenarios.  Best case for
file-backed
> > VMs is pre-allocated files.  It takes up more space, and takes a while
> > to set up initially, but it skips block allocation and probably some
> > fragmentation performance hits later.
> >
> > Worst case, though, is sparse files.  In such a setup, when you write
a
> > new file in a guest, the kernel writes the metadata to the journal,
then
> <MVNCH>
> 
> Here's a question: all of the arguments you're giving have to do
with VMs.
> Do you have some for straight-on-the-server, non-VM cases?
In the most *common* case the straight-on-the-server, non-VM case are the VM 
themselves.  Basically, in the vast number of servers you  most commonly have 
a host with a number of VMs.  The VMs are the publicly visible servers and the 
host is pretty much invisible.  The VMs themselves won't be using LVM, but
the
host server will be.

Otherwise...

I recently upgraded to a newer laptop and put a 128G SSD disk in it.  My 
previous laptop had a 60gig IDE disk.  Since I didn't have any need for more
files (at this time!) I set the laptop with LVM.  Because of how I do backups 
and because of the kinds of things I have on my laptop, I have multiple 
logical volumes:

newgollum.deepsoft.com% df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_newgollum-lv_root
                      9.8G  5.7G  3.6G  62% /
tmpfs                 1.9G  8.2M  1.9G   1% /dev/shm
/dev/sda1             477M   86M  367M  19% /boot
/dev/mapper/vg_newgollum-lv_home
                      4.8G  4.0G  602M  88% /home
/dev/mapper/vg_newgollum-scratch
                       30G   10G   18G  36% /scratch
/dev/mapper/vg_newgollum-mp3s
                      9.8G  5.1G  4.2G  55% /mp3s

I only have about 60gig presently allocated (there is about 60gig
'free').
And yes, this is a laptop with a single physical disk.  Some day I might 
create additional LVs and/or grow the existing LVs.  I *might* even install a 
VM or two on this laptop.

My disktop machine is also a host to a number of VMs (mostly used for build 
environments for different versions / flavors of Linux). Here LVM is pretty 
much a requirement, esp. since its disks are RAID'ed.

I also manage a server for the local public library. The host runs CentOS 6 on
the bare metal. It also provides DHCP, DNS, Firewall, and IP routing. The
library's workstations (for staff and patrons) are diskless and boot using
tftp, but they actually run Ubuntu 14.04 (since it is more 'user
friendly'),
so I have a Ubuntu 14.04 (server) VM providing tftp boot for Ubuntu 14.04's
kernel and NFS for Ubuntu 14.04's root and /usr file systems. (The CentOS
host
provides the /home file system.) And just as an extra 'benefit' (?) I
have a
VM running a 32-bit version of MS-Windows 8 (this is needed to talk to the
library's heating system). This is a basic server, but uses virtualization
for
selected services. Except for 'appliance' servers, I see things being
more and
more common that pure 'bare metal' servers becoming the exception rather
than
the rule. For all sorts of reasons (including security), servers will commonly
be using virtualization for many purposes. And LVM makes things really easy to
deal with disk space for VMs.
> 
>        mark
> 
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
> 
>                                                
-- 
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller at deepsoft.com       -- Webhosting Services

On Wed, 24 Jun 2015 10:40:59 -0700
Gordon Messmer <gordon.messmer at gmail.com> wrote:
> On 06/23/2015 08:10 PM, Marko Vojinovic wrote:
> 
> > For concreteness, let's say I have a guest machine, with a
> > dedicated physical partition for it, on a single drive. Or, I have
> > the same thing, only the dedicated partition is inside LVM. Why is
> > there a performance difference, and how dramatic is it?
> 
> Well, I said that there's a big performance hit to file-backed
> guests, not partition backed guests.  You should see exactly the same
> disk performance on partition backed guests as LV backed guests.
Oh, I see, I missed the detail about the guest being file-backed when I
read your previous reply. Of course, I'm fully familiar with the
drawbacks of file-backed virtual drives, as opposed to physical (or LVM)
partitions. I was (mistakenly) under the impression that you were
talking about the performance difference between a bare partition and a
LVM partition that the guest lives on.
> However, partitions have other penalties relative to LVM.
Ok, so basically what you're saying is that in the usecase when one is
spinning VM's on a daily basis, LVM is more flexible than dedicating
hardware partitions for each new VM. I can understand that. Although, I
could guess that if one is spinning VM's on a daily basis, their
performance probably isn't an issue, so that a file-backed VM would do
the job... It depends on what you use them for, in the end.

It's true I never came across such a scenario. In my experience so far,
spinning a new VM is a rare process, which includes planning,
designing, estimating resource usage, etc... And then, once the VM is
put in place, it is intended to work long-term (usually until its OS
reaches EOL or the hardware breaks).

But I get your point, with LVM you have additional flexibility to spin
test-VM's basically every day if you need to, keeping the benefit of
performance level of partition-backed virtual drives.

Ok, you have me convinced! :-) Next time I get my hands on a new
harddrive, I'll put LVM on it, and see if it helps me manage VM's more
efficiently. Doing this on a single drive doesn't run the risk of
losing more than one drive's worth of data if it fails, so I'll play
with it a little more in the context of VM's, and we'll see if it
improves my workflow.

Maybe I'll have a change of heart over LVM after all. ;-)

Best, :-)
Marko