Hi, I''m using KVM, and the virtual disk (a 20 GB file using the "raw" qemu format according to virt-manager and, of course, placed on a btrfs filesystem, running the latest mainline git) is awfully slow, no matter what OS is running inside the VM. The PCBSD installer says it''s copying data at a 40-50 KB/s rate. Is someone using KVM and having better numbers than me? How can I help to debug this workload? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
>Hi, I''m using KVM, and the virtual disk (a 20 GB file using the "raw" >qemu format according to virt-manager and, of course, placed on a btrfs >filesystem, running the latest mainline git) is awfully slow, no matter >what OS is running inside the VM. The PCBSD installer says it''s copying >data at a 40-50 KB/s rate. Is someone using KVM and having better numbers >than me? How can I help to debug this workload?Maybe cache=writeback helps? -drive file=image.raw,cache=writeback -Markus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, Mar 28, 2010 at 05:18:03PM +0200, Diego Calleja wrote:> Hi, I''m using KVM, and the virtual disk (a 20 GB file using the "raw" > qemu format according to virt-manager and, of course, placed on a btrfs > filesystem, running the latest mainline git) is awfully slow, no matter > what OS is running inside the VM. The PCBSD installer says it''s copying > data at a 40-50 KB/s rate. Is someone using KVM and having better numbers > than me? How can I help to debug this workload?The problem is that qemu uses O_SYNC by default, which makes btrfs do log commits for every write. Once the O_DIRECT read patch is in, you can switch to that, or tell qemu to use a writeback cache instead. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 03/30/2010 03:56 PM, Chris Mason wrote:> On Sun, Mar 28, 2010 at 05:18:03PM +0200, Diego Calleja wrote: > >> Hi, I''m using KVM, and the virtual disk (a 20 GB file using the "raw" >> qemu format according to virt-manager and, of course, placed on a btrfs >> filesystem, running the latest mainline git) is awfully slow, no matter >> what OS is running inside the VM. The PCBSD installer says it''s copying >> data at a 40-50 KB/s rate. Is someone using KVM and having better numbers >> than me? How can I help to debug this workload? >> > The problem is that qemu uses O_SYNC by default, which makes btrfs do > log commits for every write. >Problem is, btrfs takes the 50 KB/s guest rate and inflates it to something much larger (megabytes/sec). Are there plans to reduce the amount of O_SYNC overhead writes? I saw this too, but with 2.6.31 or 2.6.32 IIRC.> Once the O_DIRECT read patch is in, you can switch to that, or tell qemu > to use a writeback cache instead. >Even with writeback qemu will issue a lot of fsyncs. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Avi Kivity wrote:> On 03/30/2010 03:56 PM, Chris Mason wrote: >> On Sun, Mar 28, 2010 at 05:18:03PM +0200, Diego Calleja wrote: >> >>> Hi, I''m using KVM, and the virtual disk (a 20 GB file using the "raw" >>> qemu format according to virt-manager and, of course, placed on a btrfs >>> filesystem, running the latest mainline git) is awfully slow, no matter >>> what OS is running inside the VM. The PCBSD installer says it''s copying >>> data at a 40-50 KB/s rate. Is someone using KVM and having better >>> numbers >>> than me? How can I help to debug this workload? >>> >> The problem is that qemu uses O_SYNC by default, which makes btrfs do >> log commits for every write. >> > > Problem is, btrfs takes the 50 KB/s guest rate and inflates it to > something much larger (megabytes/sec). Are there plans to reduce the > amount of O_SYNC overhead writes? > > I saw this too, but with 2.6.31 or 2.6.32 IIRC. > >> Once the O_DIRECT read patch is in, you can switch to that, or tell qemu >> to use a writeback cache instead. >> > > Even with writeback qemu will issue a lot of fsyncs. >My understanding was that with cache=writeback qemu shouldn''t issue any fsyncs at all, but I could be wrong. Gordan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Apr 08, 2010 at 05:58:17PM +0300, Avi Kivity wrote:> On 03/30/2010 03:56 PM, Chris Mason wrote: > >On Sun, Mar 28, 2010 at 05:18:03PM +0200, Diego Calleja wrote: > >>Hi, I''m using KVM, and the virtual disk (a 20 GB file using the "raw" > >>qemu format according to virt-manager and, of course, placed on a btrfs > >>filesystem, running the latest mainline git) is awfully slow, no matter > >>what OS is running inside the VM. The PCBSD installer says it''s copying > >>data at a 40-50 KB/s rate. Is someone using KVM and having better numbers > >>than me? How can I help to debug this workload? > >The problem is that qemu uses O_SYNC by default, which makes btrfs do > >log commits for every write. > > Problem is, btrfs takes the 50 KB/s guest rate and inflates it to > something much larger (megabytes/sec). Are there plans to reduce > the amount of O_SYNC overhead writes? > > I saw this too, but with 2.6.31 or 2.6.32 IIRC.With O_DIRECT the writeback rates are very reasonable. I''ll work up a way to pass the barrier down from the guest to btrfs to force logging of updated metadata when required.> > >Once the O_DIRECT read patch is in, you can switch to that, or tell qemu > >to use a writeback cache instead. > > Even with writeback qemu will issue a lot of fsyncs.Oh, I didn''t see that when I was testing, when does it fsync? -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 04/08/2010 06:26 PM, Chris Mason wrote:>> >>> Once the O_DIRECT read patch is in, you can switch to that, or tell qemu >>> to use a writeback cache instead. >>> >> Even with writeback qemu will issue a lot of fsyncs. >> > Oh, I didn''t see that when I was testing, when does it fsync? >When it updates qcow2 metadata or when the guest issues a barrier. It''s relatively new. I have a patch that introduces cache=volatile somewhere. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Apr 08, 2010 at 11:26:15AM -0400, Chris Mason wrote:> With O_DIRECT the writeback rates are very reasonable. I''ll work up a > way to pass the barrier down from the guest to btrfs to force logging of > updated metadata when required.Barriers are implemented in the guest kernel using queue drains and cache flush commands. Qemu maps the cache flush to fdatasync.> > >Once the O_DIRECT read patch is in, you can switch to that, or tell qemu > > >to use a writeback cache instead. > > > > Even with writeback qemu will issue a lot of fsyncs. > > Oh, I didn''t see that when I was testing, when does it fsync?When the guest issues a barrier (and it''s actually a fdatasync) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Apr 08, 2010 at 06:28:54PM +0300, Avi Kivity wrote:> On 04/08/2010 06:26 PM, Chris Mason wrote: > >> > >>>Once the O_DIRECT read patch is in, you can switch to that, or tell qemu > >>>to use a writeback cache instead. > >>Even with writeback qemu will issue a lot of fsyncs. > >Oh, I didn''t see that when I was testing, when does it fsync? > > When it updates qcow2 metadata or when the guest issues a barrier. > It''s relatively new. I have a patch that introduces cache=volatile > somewhere.Ok, that''s actually perfect. I''ll retest with that if I can get qemu-git to work here. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Apr 08, 2010 at 06:28:54PM +0300, Avi Kivity wrote:> When it updates qcow2 metadata or when the guest issues a barrier. It''s > relatively new. I have a patch that introduces cache=volatile somewhere.qcow2 does not issues any fsyncs by itself, it only passes throught the guests ones. The only other placess issueing fsyncs is commit a COW image back to the base image, and on migreation. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 04/08/2010 06:34 PM, Christoph Hellwig wrote:> On Thu, Apr 08, 2010 at 06:28:54PM +0300, Avi Kivity wrote: > >> When it updates qcow2 metadata or when the guest issues a barrier. It''s >> relatively new. I have a patch that introduces cache=volatile somewhere. >> > qcow2 does not issues any fsyncs by itself, it only passes throught the > guests ones. The only other placess issueing fsyncs is commit a COW > image back to the base image, and on migreation. >Shouldn''t it do that then? What''s the point of fsyncing guest data if qcow2 metadata is volatile? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Apr 08, 2010 at 06:36:15PM +0300, Avi Kivity wrote:> Shouldn''t it do that then? What''s the point of fsyncing guest data if > qcow2 metadata is volatile?Not my territory - but in the end getting qcow2 as-is solid in face of crashes will be an uphill battel - I''d rather recommend not using it. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html