Hi,
I have a new fast computer to run many virtual machines. Everything
looks very fast, except the installation of new operating systems in
KVM. The installation is very fast until it begins to write on disk. It
looks like it writes slower and slower. I tried Debian, FreeBSD,
OpenIndiana and OpenBSD: same problem. The FreeBSD installer displays
the speed: it starts at 780 KB/sec (which it already very slow) to
finish between 1 and 8 KB/sec.
darksatanic suggested me to use nodatacow mount flag: it is not faster,
and it looks even slower (fewer wsec/s in iostat output, see below).
The computer is an Intel i7 2600 (4 cores with hyper threading: 8
threads), 12 GB or RAM, 2 hard drives of 1 TB (Western Digital Caviar
Blue 1 To 7200 RPM 32 Mo Serial ATA 6Gb/s - WD10EALX). Both disks are
connected to SATA 6 GB/sec connectors using a P67 chipset. I''m using
RAID 0 with Linux software (MD) RAID, and I have one unique btrfs
partition of 2 TB. The host OS is Fedora 15 (Linux kernel 2.6.38).
I''m using hardware virtualisation with KVM. Debian is installed using
virtio, so it should not be an issue with the hard drive driver of KVM.
I''m watching iostat during the Debian installation. With the default
mount option, wsec/s starts at 49000 to finish near 42000 (on the MD
device), %usage is greater than 50% of both disks (near 80% for sda,
near 60% for sdb). Using nodatacow option, wsec/s starts at 12000
(%usage > 75%) to finish near 10000 (%usage always > 75%). It is slower,
right? A sector is 512 bytes. The Debian image size is 40 GB, its type
is "raw". The system load is greater than 2 (or maybe 3) during the
installation of the VM, while CPU usage is under 8% and wa% is also low
(maybe 10% or lower, I don''t remember).
bonnie++ output (on the Fedora host, not in a VM):
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
ned 24048M 346 98 220839 24 98489 19 245 84 251547 18
199.2 259
Latency 37256us 326ms 943ms 251ms 197ms
151ms
Version 1.96 ------Sequential Create------ --------Random
Create--------
ned -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 11128 34 +++++ +++ 16558 45 14006 40 +++++ +++
17226 48
Latency 14997us 663us 11401us 8115us 282us
10105us
1.96,1.96,ned,1,1311365747,24048M,,346,98,220839,24,98489,19,245,84,251547,18,199.2,259,16,,,,,11128,34,+++++,+++,16558,45,14006,40,+++++,+++,17226,48,37256us,326ms,943ms,251ms,197ms,151ms,14997us,663us,11401us,8115us,282us,10105us
Do you have any idea why the %usage is so high (in iostat), while the
speed looks so low? The disk speed during the installation is between 5
MB/sec and 23 MB/sec, whereas the raw speed is greater than 200 MB/sec
on the host (234 MB/sec according to hdparm -t /dev/md127, 220 MB/sec
according to bonnie++ on sequential output).
It''s difficult to read bonnie++ output: random create is near 14 MB/sec
if I read correctly. btrfs behaves maybe very badly with a raw image of
40 GB, especially on RAID 0 with 2 disks?
Should I try other KVM option (e.g. use another type of image)? Try
btrfs RAID instead of Linux MD RAID? Try to disable some CPU cores? Or
maybe not using btrfs for KVM images? :-)
Victor
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Jul 22, 2011 at 2:44 PM, Victor Stinner <victor.stinner@haypocalc.com> wrote:> Hi, > > I have a new fast computer to run many virtual machines. Everything looks > very fast, except the installation of new operating systems in KVM. The > installation is very fast until it begins to write on disk. It looks like it > writes slower and slower. I tried Debian, FreeBSD, OpenIndiana and OpenBSD: > same problem. The FreeBSD installer displays the speed: it starts at 780 > KB/sec (which it already very slow) to finish between 1 and 8 KB/sec.) is the host FS btrfs? ) are virtio modules in the initramfs (or kernel probably)? ) are you sure virtio is being used (eg. are the disks called vdX vs sdX)? ) is the disk bus set to virtio (virtmanager)? ) is the disk''s cache mode set to none [or maybe writeback] (virtmanager)? ) is the disk''s storage format set to "raw", should be (virtmanager)? ) is caching enabled on the image? () probably need to change the cache mode on the disk, or if the host is btrfs you need to flag the image with whetever is needed to prevent continuous COWing. C Anthony -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Jul 22, 2011 at 2:59 PM, C Anthony Risinger <anthony@xtfx.me> wrote:> On Fri, Jul 22, 2011 at 2:44 PM, Victor Stinner > <victor.stinner@haypocalc.com> wrote: > > ) is caching enabled on the image? ()oops, disregard that ... remainder left over from editing copy/paste :-) C Anthony -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 07/22/2011 03:44 PM, Victor Stinner wrote:> Hi, > > I have a new fast computer to run many virtual machines. Everything > looks very fast, except the installation of new operating systems in > KVM. The installation is very fast until it begins to write on disk. It > looks like it writes slower and slower. I tried Debian, FreeBSD, > OpenIndiana and OpenBSD: same problem. The FreeBSD installer displays > the speed: it starts at 780 KB/sec (which it already very slow) to > finish between 1 and 8 KB/sec. > > darksatanic suggested me to use nodatacow mount flag: it is not faster, > and it looks even slower (fewer wsec/s in iostat output, see below). > > The computer is an Intel i7 2600 (4 cores with hyper threading: 8 > threads), 12 GB or RAM, 2 hard drives of 1 TB (Western Digital Caviar > Blue 1 To 7200 RPM 32 Mo Serial ATA 6Gb/s - WD10EALX). Both disks are > connected to SATA 6 GB/sec connectors using a P67 chipset. I''m using > RAID 0 with Linux software (MD) RAID, and I have one unique btrfs > partition of 2 TB. The host OS is Fedora 15 (Linux kernel 2.6.38). > > I''m using hardware virtualisation with KVM. Debian is installed using > virtio, so it should not be an issue with the hard drive driver of KVM. > > I''m watching iostat during the Debian installation. With the default > mount option, wsec/s starts at 49000 to finish near 42000 (on the MD > device), %usage is greater than 50% of both disks (near 80% for sda, > near 60% for sdb). Using nodatacow option, wsec/s starts at 12000 > (%usage > 75%) to finish near 10000 (%usage always > 75%). It is slower, > right? A sector is 512 bytes. The Debian image size is 40 GB, its type > is "raw". The system load is greater than 2 (or maybe 3) during the > installation of the VM, while CPU usage is under 8% and wa% is also low > (maybe 10% or lower, I don''t remember). > > bonnie++ output (on the Fedora host, not in a VM): > > Version 1.96 ------Sequential Output------ --Sequential Input- > --Random- > Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP > /sec %CP > ned 24048M 346 98 220839 24 98489 19 245 84 251547 18 > 199.2 259 > Latency 37256us 326ms 943ms 251ms 197ms 151ms > Version 1.96 ------Sequential Create------ --------Random > Create-------- > ned -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > /sec %CP > 16 11128 34 +++++ +++ 16558 45 14006 40 +++++ +++ > 17226 48 > Latency 14997us 663us 11401us 8115us 282us 10105us > 1.96,1.96,ned,1,1311365747,24048M,,346,98,220839,24,98489,19,245,84,251547,18,199.2,259,16,,,,,11128,34,+++++,+++,16558,45,14006,40,+++++,+++,17226,48,37256us,326ms,943ms,251ms,197ms,151ms,14997us,663us,11401us,8115us,282us,10105us > > > Do you have any idea why the %usage is so high (in iostat), while the > speed looks so low? The disk speed during the installation is between 5 > MB/sec and 23 MB/sec, whereas the raw speed is greater than 200 MB/sec > on the host (234 MB/sec according to hdparm -t /dev/md127, 220 MB/sec > according to bonnie++ on sequential output). > > It''s difficult to read bonnie++ output: random create is near 14 MB/sec > if I read correctly. btrfs behaves maybe very badly with a raw image of > 40 GB, especially on RAID 0 with 2 disks? > > Should I try other KVM option (e.g. use another type of image)? Try > btrfs RAID instead of Linux MD RAID? Try to disable some CPU cores? Or > maybe not using btrfs for KVM images? :-) >Use the kvm option of cache=none for your device. Granted its still going to be slow, but it should be a bit faster. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, 22 Jul 2011 21:44:24 +0200, Victor Stinner wrote:> Should I try other KVM option (e.g. use another type of image)? Try > btrfs RAID instead of Linux MD RAID? Try to disable some CPU cores? > Or > maybe not using btrfs for KVM images? :-)Hi, I would suggest you the following points: - qemu-img create -f qcow2 -o size=40000000000,preallocation=metadata vdisk.img - disk: cache=none - controller: virtio Best regards, Morten -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi, Le 22/07/2011 21:59, C Anthony Risinger a écrit :> ) is the host FS btrfs?The host (Fedora) FS is btrfs, the VM (Debian) uses ext4 (or maybe ext3, I will have to check).> ) are virtio modules in the initramfs (or kernel probably)? > ) are you sure virtio is being used (eg. are the disks called vdX vs sdX)? > ) is the disk bus set to virtio (virtmanager)?I''m not sure. I will have to check.> ) is the disk''s cache mode set to none [or maybe writeback] (virtmanager)?I don''t remember, I will check. What is the effect of this option? (What is the effect on the host FS (btrfs)?)> ) is the disk''s storage format set to "raw", should be (virtmanager)?Yes, the image type is "raw".> ) is caching enabled on the image? ()What do you mean? Is it possible to enable or disable the cache on a specific file on the host (Fedora, btrfs)? Or you mean set cache to none in the configuration of the VM?> probably need to change the cache mode on the disk, or if the host is > btrfs you need to flag the image with whetever is needed to prevent > continuous COWing.Can nodatacow option of btrfs have an effect on this issue? Is it possible to set this option for a specific file or directory? Victor PS: I have subscribed to the mailing list. PS2: I''m not at home this week-end, I will check monday. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 22/07/2011 21:59, C Anthony Risinger wrote: > is the disk''s cache mode set to none [or maybe writeback] > (virtmanager)? I tested various options: the most important is the cache mode. cache=none is a little bit better than cache=default (writethrough), but there is an huge difference using cache=unsafe. This option is not supported by virt-manager, but by kvm. Using cache=unsafe, the installation of FreeBSD is ~100x faster: - cache=default: write at 1 to 8 KB/sec - cache=none: write at 40 KB/sec - cache=unsafe: write at 1200 KB/sec According to agraf__ on IRC (#kvm on FreeNode), the cache mode has the following effect: - cache=writethrough calls fsync() after every write() - cache=none uses O_DIRECT - cache=writeback calls fsync() when the guest issues a barrier() (don''t use O_DIRECT) - cache=unsafe doesn''t do any fsync() (but don''t use O_DIRECT) I still have to test writeback ;-) I found another mail thread, of last summer (July 2010), which is exactly the same problem that I had: "BTRFS: Unbelievably slow with kvm/qemu" on this mailing list. It contains another advice: "Make sure you build your file system with "mkfs.btrfs -m single -d single /dev/whatever". You may well be writing duplicate copies of everything." http://lkml.org/lkml/2010/7/12/5 -- FreeBSD installation in VirtualBox is as fast (or maybe a little bit slower) than the installation in kvm using cache=unsafe. I suppose that VirtualBox uses something like cache=unsafe or cache=writeback. Victor -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Jul 25, 2011 at 04:04:49PM +0200, Victor Stinner wrote:> According to agraf__ on IRC (#kvm on FreeNode), the cache mode has > the following effect: > > - cache=writethrough calls fsync() after every write()It uses O_DSYNC, which an be appromiately described as an fdatasync after every write. Not that it makes a different for most workloads on btrfs.> - cache=none uses O_DIRECT > - cache=writeback calls fsync() when the guest issues a barrier() > (don''t use O_DIRECT)Issue a cache flush. Barriers were an Linux-internal concept that is now gone. And there never was a barriere() call.> FreeBSD installation in VirtualBox is as fast (or maybe a little bit > slower) than the installation in kvm using cache=unsafe. I suppose > that VirtualBox uses something like cache=unsafe or cache=writeback.What filesystem do you use in FreeBSD? FFS traditionally never issued cache flushes, so cache=writeback is equal to cache=unsafe for it. ZFS can issue cache flushes on Solaris, but I''m not sure if this was ported correctly to FreeBSD. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html