Tobias Geiger
2012-Apr-23 10:02 UTC
Degregated I/O Performance since 3.4 - Regression in 3.4?
Hello! i noticed a considerable drop in I/O Performance when using 3.4 (rc3 and rc4 tested) as Dom0 Kernel; With 3.3 i get over 100mb/s in a HVM DomU (win64) with PV Drivers (gplpv_Vista2008x64_0.11.0.357.msi); With 3.4 it drops to about a third of that. Xen Version is xen-unstable: xen_changeset : Tue Apr 17 19:13:52 2012 +0100 25209:e6b20ec1824c Disk config line is: disk = [ ''/dev/vg_ssd/win7system,,hda'' ] - it uses blkback. Greetings Tobias
Stefano Stabellini
2012-Apr-23 11:53 UTC
Re: Degregated I/O Performance since 3.4 - Regression in 3.4?
On Mon, 23 Apr 2012, Tobias Geiger wrote:> Hello! > > i noticed a considerable drop in I/O Performance when using 3.4 (rc3 and rc4 > tested) as Dom0 Kernel; > > With 3.3 i get over 100mb/s in a HVM DomU (win64) with PV Drivers > (gplpv_Vista2008x64_0.11.0.357.msi); > With 3.4 it drops to about a third of that. > > Xen Version is xen-unstable: > xen_changeset : Tue Apr 17 19:13:52 2012 +0100 25209:e6b20ec1824c > > Disk config line is: > disk = [ ''/dev/vg_ssd/win7system,,hda'' ] > - it uses blkback.I fail to see what could be the cause of the issue: nothing on the blkback side should affect performances significantly. You could try reverting the four patches to blkback that were applied between 3.3 and 3.4-rc3 just to make sure it is not a blkback regression: $ git shortlog v3.3..v3.4-rc3 drivers/block/xen-blkback Daniel De Graaf (2): xen/blkback: use grant-table.c hypercall wrappers xen/blkback: Enable blkback on HVM guests Konrad Rzeszutek Wilk (2): xen/blkback: Squash the discard support for ''file'' and ''phy'' type. xen/blkback: Make optional features be really optional.
Konrad Rzeszutek Wilk
2012-Apr-23 15:24 UTC
Re: Degregated I/O Performance since 3.4 - Regression in 3.4?
On Mon, Apr 23, 2012 at 12:53:03PM +0100, Stefano Stabellini wrote:> On Mon, 23 Apr 2012, Tobias Geiger wrote: > > Hello! > > > > i noticed a considerable drop in I/O Performance when using 3.4 (rc3 and rc4 > > tested) as Dom0 Kernel; > > > > With 3.3 i get over 100mb/s in a HVM DomU (win64) with PV Drivers > > (gplpv_Vista2008x64_0.11.0.357.msi); > > With 3.4 it drops to about a third of that. > > > > Xen Version is xen-unstable: > > xen_changeset : Tue Apr 17 19:13:52 2012 +0100 25209:e6b20ec1824c > > > > Disk config line is: > > disk = [ ''/dev/vg_ssd/win7system,,hda'' ] > > - it uses blkback. > > I fail to see what could be the cause of the issue: nothing on the > blkback side should affect performances significantly. > You could try reverting the four patches to blkback that were applied > between 3.3 and 3.4-rc3 just to make sure it is not a blkback > regression: > > $ git shortlog v3.3..v3.4-rc3 drivers/block/xen-blkback > Daniel De Graaf (2): > xen/blkback: use grant-table.c hypercall wrappersHm.. Perhaps this patch fixes it a possible perf (I would think that the compiler would have kept the result of the first call to vaddr(req, i) somewhere.. but not sure) lost with the mentioned patch: diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c index 73f196c..65dbadc 100644 --- a/drivers/block/xen-blkback/blkback.c +++ b/drivers/block/xen-blkback/blkback.c @@ -327,13 +327,15 @@ static void xen_blkbk_unmap(struct pending_req *req) int ret; for (i = 0; i < req->nr_pages; i++) { + unsigned long addr; handle = pending_handle(req, i); if (handle == BLKBACK_INVALID_HANDLE) continue; - gnttab_set_unmap_op(&unmap[invcount], vaddr(req, i), + addr = vaddr(req, i); + gnttab_set_unmap_op(&unmap[invcount], addr, GNTMAP_host_map, handle); pending_handle(req, i) = BLKBACK_INVALID_HANDLE; - pages[invcount] = virt_to_page(vaddr(req, i)); + pages[invcount] = virt_to_page(addr); invcount++; }> xen/blkback: Enable blkback on HVM guests > > Konrad Rzeszutek Wilk (2): > xen/blkback: Squash the discard support for ''file'' and ''phy'' type. > xen/blkback: Make optional features be really optional. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Tobias Geiger
2012-Apr-23 20:53 UTC
Re: Degregated I/O Performance since 3.4 - Regression in 3.4?
Am 23.04.2012 17:24, schrieb Konrad Rzeszutek Wilk:> On Mon, Apr 23, 2012 at 12:53:03PM +0100, Stefano Stabellini wrote: >> On Mon, 23 Apr 2012, Tobias Geiger wrote: >>> Hello! >>> >>> i noticed a considerable drop in I/O Performance when using 3.4 (rc3 and rc4 >>> tested) as Dom0 Kernel; >>> >>> With 3.3 i get over 100mb/s in a HVM DomU (win64) with PV Drivers >>> (gplpv_Vista2008x64_0.11.0.357.msi); >>> With 3.4 it drops to about a third of that. >>> >>> Xen Version is xen-unstable: >>> xen_changeset : Tue Apr 17 19:13:52 2012 +0100 25209:e6b20ec1824c >>> >>> Disk config line is: >>> disk = [ ''/dev/vg_ssd/win7system,,hda'' ] >>> - it uses blkback. >> I fail to see what could be the cause of the issue: nothing on the >> blkback side should affect performances significantly. >> You could try reverting the four patches to blkback that were applied >> between 3.3 and 3.4-rc3 just to make sure it is not a blkback >> regression: >> >> $ git shortlog v3.3..v3.4-rc3 drivers/block/xen-blkback >> Daniel De Graaf (2): >> xen/blkback: use grant-table.c hypercall wrappers > Hm.. Perhaps this patch fixes it a possible perf (I would think that > the compiler would have kept the result of the first call to vaddr(req, i) > somewhere.. but not sure) lost with the mentioned patch: > > diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c > index 73f196c..65dbadc 100644 > --- a/drivers/block/xen-blkback/blkback.c > +++ b/drivers/block/xen-blkback/blkback.c > @@ -327,13 +327,15 @@ static void xen_blkbk_unmap(struct pending_req *req) > int ret; > > for (i = 0; i< req->nr_pages; i++) { > + unsigned long addr; > handle = pending_handle(req, i); > if (handle == BLKBACK_INVALID_HANDLE) > continue; > - gnttab_set_unmap_op(&unmap[invcount], vaddr(req, i), > + addr = vaddr(req, i); > + gnttab_set_unmap_op(&unmap[invcount], addr, > GNTMAP_host_map, handle); > pending_handle(req, i) = BLKBACK_INVALID_HANDLE; > - pages[invcount] = virt_to_page(vaddr(req, i)); > + pages[invcount] = virt_to_page(addr); > invcount++; > } > >> xen/blkback: Enable blkback on HVM guests >> >> Konrad Rzeszutek Wilk (2): >> xen/blkback: Squash the discard support for ''file'' and ''phy'' type. >> xen/blkback: Make optional features be really optional. >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-develthat made it even worse :) Write Performance is down to about 7mb/s (with 3.3: ~130mb/s) Read "only" down to 40mb/s (with 3.3: ~140mb/s) Greetings Tobias
Jan Beulich
2012-Apr-24 07:27 UTC
Re: Degregated I/O Performance since 3.4 - Regression in 3.4?
>>> On 23.04.12 at 22:53, Tobias Geiger <tobias.geiger@vido.info> wrote: > Am 23.04.2012 17:24, schrieb Konrad Rzeszutek Wilk: >> On Mon, Apr 23, 2012 at 12:53:03PM +0100, Stefano Stabellini wrote: >>> On Mon, 23 Apr 2012, Tobias Geiger wrote: >>>> Hello! >>>> >>>> i noticed a considerable drop in I/O Performance when using 3.4 (rc3 and rc4 >>>> tested) as Dom0 Kernel; >>>> >>>> With 3.3 i get over 100mb/s in a HVM DomU (win64) with PV Drivers >>>> (gplpv_Vista2008x64_0.11.0.357.msi); >>>> With 3.4 it drops to about a third of that. >>>> >>>> Xen Version is xen-unstable: >>>> xen_changeset : Tue Apr 17 19:13:52 2012 +0100 25209:e6b20ec1824c >>>> >>>> Disk config line is: >>>> disk = [ ''/dev/vg_ssd/win7system,,hda'' ] >>>> - it uses blkback. >>> I fail to see what could be the cause of the issue: nothing on the >>> blkback side should affect performances significantly. >>> You could try reverting the four patches to blkback that were applied >>> between 3.3 and 3.4-rc3 just to make sure it is not a blkback >>> regression: >>> >>> $ git shortlog v3.3..v3.4-rc3 drivers/block/xen-blkback >>> Daniel De Graaf (2): >>> xen/blkback: use grant-table.c hypercall wrappers >> Hm.. Perhaps this patch fixes it a possible perf (I would think that >> the compiler would have kept the result of the first call to vaddr(req, i) >> somewhere.. but not sure) lost with the mentioned patch: >> >> diff --git a/drivers/block/xen-blkback/blkback.c > b/drivers/block/xen-blkback/blkback.c >> index 73f196c..65dbadc 100644 >> --- a/drivers/block/xen-blkback/blkback.c >> +++ b/drivers/block/xen-blkback/blkback.c >> @@ -327,13 +327,15 @@ static void xen_blkbk_unmap(struct pending_req *req) >> int ret; >> >> for (i = 0; i< req->nr_pages; i++) { >> + unsigned long addr; >> handle = pending_handle(req, i); >> if (handle == BLKBACK_INVALID_HANDLE) >> continue; >> - gnttab_set_unmap_op(&unmap[invcount], vaddr(req, i), >> + addr = vaddr(req, i); >> + gnttab_set_unmap_op(&unmap[invcount], addr, >> GNTMAP_host_map, handle); >> pending_handle(req, i) = BLKBACK_INVALID_HANDLE; >> - pages[invcount] = virt_to_page(vaddr(req, i)); >> + pages[invcount] = virt_to_page(addr); >> invcount++; >> } >> >>> xen/blkback: Enable blkback on HVM guests >>> >>> Konrad Rzeszutek Wilk (2): >>> xen/blkback: Squash the discard support for ''file'' and ''phy'' type. >>> xen/blkback: Make optional features be really optional. >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xen.org >>> http://lists.xen.org/xen-devel > that made it even worse :) > Write Performance is down to about 7mb/s (with 3.3: ~130mb/s) > Read "only" down to 40mb/s (with 3.3: ~140mb/s)I doubt this patch can have any meaningful positive or negative performance effect at all - are you sure you''re doing comparable runs? After all this is all just about a few arithmetic operations and an array access, which I''d expect to hide in the noise. Jan
Tobias Geiger
2012-Apr-24 12:09 UTC
Re: Degregated I/O Performance since 3.4 - Regression in 3.4?
Am Dienstag, 24. April 2012, 09:27:42 schrieb Jan Beulich:> >>> On 23.04.12 at 22:53, Tobias Geiger <tobias.geiger@vido.info> wrote: > > Am 23.04.2012 17:24, schrieb Konrad Rzeszutek Wilk: > >> On Mon, Apr 23, 2012 at 12:53:03PM +0100, Stefano Stabellini wrote: > >>> On Mon, 23 Apr 2012, Tobias Geiger wrote: > >>>> Hello! > >>>> > >>>> i noticed a considerable drop in I/O Performance when using 3.4 (rc3 > >>>> and rc4 tested) as Dom0 Kernel; > >>>> > >>>> With 3.3 i get over 100mb/s in a HVM DomU (win64) with PV Drivers > >>>> (gplpv_Vista2008x64_0.11.0.357.msi); > >>>> With 3.4 it drops to about a third of that. > >>>> > >>>> Xen Version is xen-unstable: > >>>> xen_changeset : Tue Apr 17 19:13:52 2012 +0100 > >>>> 25209:e6b20ec1824c > >>>> > >>>> Disk config line is: > >>>> disk = [ ''/dev/vg_ssd/win7system,,hda'' ] > >>>> - it uses blkback. > >>> > >>> I fail to see what could be the cause of the issue: nothing on the > >>> blkback side should affect performances significantly. > >>> You could try reverting the four patches to blkback that were applied > >>> between 3.3 and 3.4-rc3 just to make sure it is not a blkback > >>> regression: > >>> > >>> $ git shortlog v3.3..v3.4-rc3 drivers/block/xen-blkback > >>> > >>> Daniel De Graaf (2): > >>> xen/blkback: use grant-table.c hypercall wrappers > >> > >> Hm.. Perhaps this patch fixes it a possible perf (I would think that > >> the compiler would have kept the result of the first call to vaddr(req, > >> i) somewhere.. but not sure) lost with the mentioned patch: > >> > >> diff --git a/drivers/block/xen-blkback/blkback.c > > > > b/drivers/block/xen-blkback/blkback.c > > > >> index 73f196c..65dbadc 100644 > >> --- a/drivers/block/xen-blkback/blkback.c > >> +++ b/drivers/block/xen-blkback/blkback.c > >> @@ -327,13 +327,15 @@ static void xen_blkbk_unmap(struct pending_req > >> *req) > >> > >> int ret; > >> > >> for (i = 0; i< req->nr_pages; i++) { > >> > >> + unsigned long addr; > >> > >> handle = pending_handle(req, i); > >> if (handle == BLKBACK_INVALID_HANDLE) > >> > >> continue; > >> > >> - gnttab_set_unmap_op(&unmap[invcount], vaddr(req, i), > >> + addr = vaddr(req, i); > >> + gnttab_set_unmap_op(&unmap[invcount], addr, > >> > >> GNTMAP_host_map, handle); > >> > >> pending_handle(req, i) = BLKBACK_INVALID_HANDLE; > >> > >> - pages[invcount] = virt_to_page(vaddr(req, i)); > >> + pages[invcount] = virt_to_page(addr); > >> > >> invcount++; > >> > >> } > >> > >>> xen/blkback: Enable blkback on HVM guests > >>> > >>> Konrad Rzeszutek Wilk (2): > >>> xen/blkback: Squash the discard support for ''file'' and ''phy'' > >>> type. xen/blkback: Make optional features be really optional. > >>> > >>> _______________________________________________ > >>> Xen-devel mailing list > >>> Xen-devel@lists.xen.org > >>> http://lists.xen.org/xen-devel > > > > that made it even worse :) > > Write Performance is down to about 7mb/s (with 3.3: ~130mb/s) > > Read "only" down to 40mb/s (with 3.3: ~140mb/s) > > I doubt this patch can have any meaningful positive or negative > performance effect at all - are you sure you''re doing comparable > runs? After all this is all just about a few arithmetic operations > and an array access, which I''d expect to hide in the noise. > > JanI redid the test; a) with 3.3.0 kernel b) with 3.4.0-rc4 c) with 3.40-rc4 and above patch everything else remained the same, i.e. test-program and test-scenario was not changed and started after about 5min of domu bootup (so that no strange bootup-effects become relevant); same phy-backend (lvm on ssd), same everything else; so i cant see what else except the used dom0 kernel is causing this issue; but here are the numbers: a) read: 135mb/s write: 142mb/s b) read: 39mb/s write: 39mb/s c) read: 40mb/s write: 40mb/s Only thing that may become relevant is the difference in kernel-config betwen 3.3 and 3.4 - here''s the diff : http://pastebin.com/raw.php?i=Dy71Fegq Jan, it seems you''re right: The patch doesn''t add extra performance regression - i guess i had an i/o intensive task running in dom0 while doing the benchmark yesterday, so that the write performance got so bad. sorry for that. Still there''s a significant performance penalty from 3.3 to 3.4 Greetings Tobias
Stefano Stabellini
2012-Apr-24 12:52 UTC
Re: Degregated I/O Performance since 3.4 - Regression in 3.4?
On Tue, 24 Apr 2012, Tobias Geiger wrote:> Am Dienstag, 24. April 2012, 09:27:42 schrieb Jan Beulich: > > >>> On 23.04.12 at 22:53, Tobias Geiger <tobias.geiger@vido.info> wrote: > > > Am 23.04.2012 17:24, schrieb Konrad Rzeszutek Wilk: > > >> On Mon, Apr 23, 2012 at 12:53:03PM +0100, Stefano Stabellini wrote: > > >>> On Mon, 23 Apr 2012, Tobias Geiger wrote: > > >>>> Hello! > > >>>> > > >>>> i noticed a considerable drop in I/O Performance when using 3.4 (rc3 > > >>>> and rc4 tested) as Dom0 Kernel; > > >>>> > > >>>> With 3.3 i get over 100mb/s in a HVM DomU (win64) with PV Drivers > > >>>> (gplpv_Vista2008x64_0.11.0.357.msi); > > >>>> With 3.4 it drops to about a third of that. > > >>>> > > >>>> Xen Version is xen-unstable: > > >>>> xen_changeset : Tue Apr 17 19:13:52 2012 +0100 > > >>>> 25209:e6b20ec1824c > > >>>> > > >>>> Disk config line is: > > >>>> disk = [ ''/dev/vg_ssd/win7system,,hda'' ] > > >>>> - it uses blkback. > > >>> > > >>> I fail to see what could be the cause of the issue: nothing on the > > >>> blkback side should affect performances significantly. > > >>> You could try reverting the four patches to blkback that were applied > > >>> between 3.3 and 3.4-rc3 just to make sure it is not a blkback > > >>> regression: > > >>> > > >>> $ git shortlog v3.3..v3.4-rc3 drivers/block/xen-blkback > > >>> > > >>> Daniel De Graaf (2): > > >>> xen/blkback: use grant-table.c hypercall wrappers > > >> > > >> Hm.. Perhaps this patch fixes it a possible perf (I would think that > > >> the compiler would have kept the result of the first call to vaddr(req, > > >> i) somewhere.. but not sure) lost with the mentioned patch: > > >> > > >> diff --git a/drivers/block/xen-blkback/blkback.c > > > > > > b/drivers/block/xen-blkback/blkback.c > > > > > >> index 73f196c..65dbadc 100644 > > >> --- a/drivers/block/xen-blkback/blkback.c > > >> +++ b/drivers/block/xen-blkback/blkback.c > > >> @@ -327,13 +327,15 @@ static void xen_blkbk_unmap(struct pending_req > > >> *req) > > >> > > >> int ret; > > >> > > >> for (i = 0; i< req->nr_pages; i++) { > > >> > > >> + unsigned long addr; > > >> > > >> handle = pending_handle(req, i); > > >> if (handle == BLKBACK_INVALID_HANDLE) > > >> > > >> continue; > > >> > > >> - gnttab_set_unmap_op(&unmap[invcount], vaddr(req, i), > > >> + addr = vaddr(req, i); > > >> + gnttab_set_unmap_op(&unmap[invcount], addr, > > >> > > >> GNTMAP_host_map, handle); > > >> > > >> pending_handle(req, i) = BLKBACK_INVALID_HANDLE; > > >> > > >> - pages[invcount] = virt_to_page(vaddr(req, i)); > > >> + pages[invcount] = virt_to_page(addr); > > >> > > >> invcount++; > > >> > > >> } > > >> > > >>> xen/blkback: Enable blkback on HVM guests > > >>> > > >>> Konrad Rzeszutek Wilk (2): > > >>> xen/blkback: Squash the discard support for ''file'' and ''phy'' > > >>> type. xen/blkback: Make optional features be really optional. > > >>> > > >>> _______________________________________________ > > >>> Xen-devel mailing list > > >>> Xen-devel@lists.xen.org > > >>> http://lists.xen.org/xen-devel > > > > > > that made it even worse :) > > > Write Performance is down to about 7mb/s (with 3.3: ~130mb/s) > > > Read "only" down to 40mb/s (with 3.3: ~140mb/s) > > > > I doubt this patch can have any meaningful positive or negative > > performance effect at all - are you sure you''re doing comparable > > runs? After all this is all just about a few arithmetic operations > > and an array access, which I''d expect to hide in the noise. > > > > Jan > > I redid the test; > > a) with 3.3.0 kernel > b) with 3.4.0-rc4 > c) with 3.40-rc4 and above patch > > everything else remained the same, i.e. test-program and test-scenario was not > changed and started after about 5min of domu bootup (so that no strange > bootup-effects become relevant); same phy-backend (lvm on ssd), same everything > else; so i cant see what else except the used dom0 kernel is causing this > issue; but here are the numbers: > > a) read: 135mb/s write: 142mb/s > b) read: 39mb/s write: 39mb/s > c) read: 40mb/s write: 40mb/s > > Only thing that may become relevant is the difference in kernel-config betwen > 3.3 and 3.4 - here''s the diff : > http://pastebin.com/raw.php?i=Dy71Fegq > > Jan, it seems you''re right: The patch doesn''t add extra performance regression > - i guess i had an i/o intensive task running in dom0 while doing the > benchmark yesterday, so that the write performance got so bad. sorry for that. > > Still there''s a significant performance penalty from 3.3 to 3.4Could you please try to revert the following commits? git revert -n a71e23d9925517e609dfcb72b5874f33cdb0d2ad git revert -n 3389bb8bf76180eecaffdfa7dd5b35fa4a2ce9b5 git revert -n 4dae76705fc8f9854bb732f9944e7ff9ba7a8e9f git revert -n b2167ba6dd89d55ced26a867fad8f0fe388fd595 git revert -n 4f14faaab4ee46a046b6baff85644be199de718c git revert -n 9846ff10af12f9e7caac696737db6c990592a74a
Tobias Geiger
2012-Apr-24 14:07 UTC
Re: Degregated I/O Performance since 3.4 - Regression in 3.4?
Am Dienstag, 24. April 2012, 14:52:31 schrieb Stefano Stabellini:> On Tue, 24 Apr 2012, Tobias Geiger wrote: > > Am Dienstag, 24. April 2012, 09:27:42 schrieb Jan Beulich: > > > >>> On 23.04.12 at 22:53, Tobias Geiger <tobias.geiger@vido.info> wrote: > > > > Am 23.04.2012 17:24, schrieb Konrad Rzeszutek Wilk: > > > >> On Mon, Apr 23, 2012 at 12:53:03PM +0100, Stefano Stabellini wrote: > > > >>> On Mon, 23 Apr 2012, Tobias Geiger wrote: > > > >>>> Hello! > > > >>>> > > > >>>> i noticed a considerable drop in I/O Performance when using 3.4 > > > >>>> (rc3 and rc4 tested) as Dom0 Kernel; > > > >>>> > > > >>>> With 3.3 i get over 100mb/s in a HVM DomU (win64) with PV Drivers > > > >>>> (gplpv_Vista2008x64_0.11.0.357.msi); > > > >>>> With 3.4 it drops to about a third of that. > > > >>>> > > > >>>> Xen Version is xen-unstable: > > > >>>> xen_changeset : Tue Apr 17 19:13:52 2012 +0100 > > > >>>> 25209:e6b20ec1824c > > > >>>> > > > >>>> Disk config line is: > > > >>>> disk = [ ''/dev/vg_ssd/win7system,,hda'' ] > > > >>>> - it uses blkback. > > > >>> > > > >>> I fail to see what could be the cause of the issue: nothing on the > > > >>> blkback side should affect performances significantly. > > > >>> You could try reverting the four patches to blkback that were > > > >>> applied between 3.3 and 3.4-rc3 just to make sure it is not a > > > >>> blkback regression: > > > >>> > > > >>> $ git shortlog v3.3..v3.4-rc3 drivers/block/xen-blkback > > > >>> > > > >>> Daniel De Graaf (2): > > > >>> xen/blkback: use grant-table.c hypercall wrappers > > > >> > > > >> Hm.. Perhaps this patch fixes it a possible perf (I would think that > > > >> the compiler would have kept the result of the first call to > > > >> vaddr(req, i) somewhere.. but not sure) lost with the mentioned > > > >> patch: > > > >> > > > >> diff --git a/drivers/block/xen-blkback/blkback.c > > > > > > > > b/drivers/block/xen-blkback/blkback.c > > > > > > > >> index 73f196c..65dbadc 100644 > > > >> --- a/drivers/block/xen-blkback/blkback.c > > > >> +++ b/drivers/block/xen-blkback/blkback.c > > > >> @@ -327,13 +327,15 @@ static void xen_blkbk_unmap(struct pending_req > > > >> *req) > > > >> > > > >> int ret; > > > >> > > > >> for (i = 0; i< req->nr_pages; i++) { > > > >> > > > >> + unsigned long addr; > > > >> > > > >> handle = pending_handle(req, i); > > > >> if (handle == BLKBACK_INVALID_HANDLE) > > > >> > > > >> continue; > > > >> > > > >> - gnttab_set_unmap_op(&unmap[invcount], vaddr(req, i), > > > >> + addr = vaddr(req, i); > > > >> + gnttab_set_unmap_op(&unmap[invcount], addr, > > > >> > > > >> GNTMAP_host_map, handle); > > > >> > > > >> pending_handle(req, i) = BLKBACK_INVALID_HANDLE; > > > >> > > > >> - pages[invcount] = virt_to_page(vaddr(req, i)); > > > >> + pages[invcount] = virt_to_page(addr); > > > >> > > > >> invcount++; > > > >> > > > >> } > > > >> > > > >>> xen/blkback: Enable blkback on HVM guests > > > >>> > > > >>> Konrad Rzeszutek Wilk (2): > > > >>> xen/blkback: Squash the discard support for ''file'' and ''phy'' > > > >>> type. xen/blkback: Make optional features be really > > > >>> optional. > > > >>> > > > >>> _______________________________________________ > > > >>> Xen-devel mailing list > > > >>> Xen-devel@lists.xen.org > > > >>> http://lists.xen.org/xen-devel > > > > > > > > that made it even worse :) > > > > Write Performance is down to about 7mb/s (with 3.3: ~130mb/s) > > > > Read "only" down to 40mb/s (with 3.3: ~140mb/s) > > > > > > I doubt this patch can have any meaningful positive or negative > > > performance effect at all - are you sure you''re doing comparable > > > runs? After all this is all just about a few arithmetic operations > > > and an array access, which I''d expect to hide in the noise. > > > > > > Jan > > > > I redid the test; > > > > a) with 3.3.0 kernel > > b) with 3.4.0-rc4 > > c) with 3.40-rc4 and above patch > > > > everything else remained the same, i.e. test-program and test-scenario > > was not changed and started after about 5min of domu bootup (so that no > > strange bootup-effects become relevant); same phy-backend (lvm on ssd), > > same everything else; so i cant see what else except the used dom0 > > kernel is causing this issue; but here are the numbers: > > > > a) read: 135mb/s write: 142mb/s > > b) read: 39mb/s write: 39mb/s > > c) read: 40mb/s write: 40mb/s > > > > Only thing that may become relevant is the difference in kernel-config > > betwen 3.3 and 3.4 - here''s the diff : > > http://pastebin.com/raw.php?i=Dy71Fegq > > > > Jan, it seems you''re right: The patch doesn''t add extra performance > > regression - i guess i had an i/o intensive task running in dom0 while > > doing the benchmark yesterday, so that the write performance got so bad. > > sorry for that. > > > > Still there''s a significant performance penalty from 3.3 to 3.4 > > Could you please try to revert the following commits? > > git revert -n a71e23d9925517e609dfcb72b5874f33cdb0d2ad > git revert -n 3389bb8bf76180eecaffdfa7dd5b35fa4a2ce9b5 > git revert -n 4dae76705fc8f9854bb732f9944e7ff9ba7a8e9f > git revert -n b2167ba6dd89d55ced26a867fad8f0fe388fd595 > git revert -n 4f14faaab4ee46a046b6baff85644be199de718c > git revert -n 9846ff10af12f9e7caac696737db6c990592a74aafter reverting said 6 commits (thanks for the ids of these - had difficulties to find them), the performance is back to normal. should i try to circle it down to one of this 6, or do you have a hint on which it might be? Greetings Tobias
Stefano Stabellini
2012-Apr-24 14:16 UTC
Re: Degregated I/O Performance since 3.4 - Regression in 3.4?
On Tue, 24 Apr 2012, Tobias Geiger wrote:> Am Dienstag, 24. April 2012, 14:52:31 schrieb Stefano Stabellini: > > On Tue, 24 Apr 2012, Tobias Geiger wrote: > > > Am Dienstag, 24. April 2012, 09:27:42 schrieb Jan Beulich: > > > > >>> On 23.04.12 at 22:53, Tobias Geiger <tobias.geiger@vido.info> wrote: > > > > > Am 23.04.2012 17:24, schrieb Konrad Rzeszutek Wilk: > > > > >> On Mon, Apr 23, 2012 at 12:53:03PM +0100, Stefano Stabellini wrote: > > > > >>> On Mon, 23 Apr 2012, Tobias Geiger wrote: > > > > >>>> Hello! > > > > >>>> > > > > >>>> i noticed a considerable drop in I/O Performance when using 3.4 > > > > >>>> (rc3 and rc4 tested) as Dom0 Kernel; > > > > >>>> > > > > >>>> With 3.3 i get over 100mb/s in a HVM DomU (win64) with PV Drivers > > > > >>>> (gplpv_Vista2008x64_0.11.0.357.msi); > > > > >>>> With 3.4 it drops to about a third of that. > > > > >>>> > > > > >>>> Xen Version is xen-unstable: > > > > >>>> xen_changeset : Tue Apr 17 19:13:52 2012 +0100 > > > > >>>> 25209:e6b20ec1824c > > > > >>>> > > > > >>>> Disk config line is: > > > > >>>> disk = [ ''/dev/vg_ssd/win7system,,hda'' ] > > > > >>>> - it uses blkback. > > > > >>> > > > > >>> I fail to see what could be the cause of the issue: nothing on the > > > > >>> blkback side should affect performances significantly. > > > > >>> You could try reverting the four patches to blkback that were > > > > >>> applied between 3.3 and 3.4-rc3 just to make sure it is not a > > > > >>> blkback regression: > > > > >>> > > > > >>> $ git shortlog v3.3..v3.4-rc3 drivers/block/xen-blkback > > > > >>> > > > > >>> Daniel De Graaf (2): > > > > >>> xen/blkback: use grant-table.c hypercall wrappers > > > > >> > > > > >> Hm.. Perhaps this patch fixes it a possible perf (I would think that > > > > >> the compiler would have kept the result of the first call to > > > > >> vaddr(req, i) somewhere.. but not sure) lost with the mentioned > > > > >> patch: > > > > >> > > > > >> diff --git a/drivers/block/xen-blkback/blkback.c > > > > > > > > > > b/drivers/block/xen-blkback/blkback.c > > > > > > > > > >> index 73f196c..65dbadc 100644 > > > > >> --- a/drivers/block/xen-blkback/blkback.c > > > > >> +++ b/drivers/block/xen-blkback/blkback.c > > > > >> @@ -327,13 +327,15 @@ static void xen_blkbk_unmap(struct pending_req > > > > >> *req) > > > > >> > > > > >> int ret; > > > > >> > > > > >> for (i = 0; i< req->nr_pages; i++) { > > > > >> > > > > >> + unsigned long addr; > > > > >> > > > > >> handle = pending_handle(req, i); > > > > >> if (handle == BLKBACK_INVALID_HANDLE) > > > > >> > > > > >> continue; > > > > >> > > > > >> - gnttab_set_unmap_op(&unmap[invcount], vaddr(req, i), > > > > >> + addr = vaddr(req, i); > > > > >> + gnttab_set_unmap_op(&unmap[invcount], addr, > > > > >> > > > > >> GNTMAP_host_map, handle); > > > > >> > > > > >> pending_handle(req, i) = BLKBACK_INVALID_HANDLE; > > > > >> > > > > >> - pages[invcount] = virt_to_page(vaddr(req, i)); > > > > >> + pages[invcount] = virt_to_page(addr); > > > > >> > > > > >> invcount++; > > > > >> > > > > >> } > > > > >> > > > > >>> xen/blkback: Enable blkback on HVM guests > > > > >>> > > > > >>> Konrad Rzeszutek Wilk (2): > > > > >>> xen/blkback: Squash the discard support for ''file'' and ''phy'' > > > > >>> type. xen/blkback: Make optional features be really > > > > >>> optional. > > > > >>> > > > > >>> _______________________________________________ > > > > >>> Xen-devel mailing list > > > > >>> Xen-devel@lists.xen.org > > > > >>> http://lists.xen.org/xen-devel > > > > > > > > > > that made it even worse :) > > > > > Write Performance is down to about 7mb/s (with 3.3: ~130mb/s) > > > > > Read "only" down to 40mb/s (with 3.3: ~140mb/s) > > > > > > > > I doubt this patch can have any meaningful positive or negative > > > > performance effect at all - are you sure you''re doing comparable > > > > runs? After all this is all just about a few arithmetic operations > > > > and an array access, which I''d expect to hide in the noise. > > > > > > > > Jan > > > > > > I redid the test; > > > > > > a) with 3.3.0 kernel > > > b) with 3.4.0-rc4 > > > c) with 3.40-rc4 and above patch > > > > > > everything else remained the same, i.e. test-program and test-scenario > > > was not changed and started after about 5min of domu bootup (so that no > > > strange bootup-effects become relevant); same phy-backend (lvm on ssd), > > > same everything else; so i cant see what else except the used dom0 > > > kernel is causing this issue; but here are the numbers: > > > > > > a) read: 135mb/s write: 142mb/s > > > b) read: 39mb/s write: 39mb/s > > > c) read: 40mb/s write: 40mb/s > > > > > > Only thing that may become relevant is the difference in kernel-config > > > betwen 3.3 and 3.4 - here''s the diff : > > > http://pastebin.com/raw.php?i=Dy71Fegq > > > > > > Jan, it seems you''re right: The patch doesn''t add extra performance > > > regression - i guess i had an i/o intensive task running in dom0 while > > > doing the benchmark yesterday, so that the write performance got so bad. > > > sorry for that. > > > > > > Still there''s a significant performance penalty from 3.3 to 3.4 > > > > Could you please try to revert the following commits? > > > > git revert -n a71e23d9925517e609dfcb72b5874f33cdb0d2ad > > git revert -n 3389bb8bf76180eecaffdfa7dd5b35fa4a2ce9b5 > > git revert -n 4dae76705fc8f9854bb732f9944e7ff9ba7a8e9f > > git revert -n b2167ba6dd89d55ced26a867fad8f0fe388fd595 > > git revert -n 4f14faaab4ee46a046b6baff85644be199de718c > > git revert -n 9846ff10af12f9e7caac696737db6c990592a74a > > after reverting said 6 commits (thanks for the ids of these - had difficulties > to find them), the performance is back to normal. > > should i try to circle it down to one of this 6, or do you have a hint on > which it might be?that would be great :)
Konrad Rzeszutek Wilk
2012-Apr-24 16:30 UTC
Re: Degregated I/O Performance since 3.4 - Regression in 3.4?
> > > I redid the test; > > > > > > a) with 3.3.0 kernel > > > b) with 3.4.0-rc4 > > > c) with 3.40-rc4 and above patch > > > > > > everything else remained the same, i.e. test-program and test-scenario > > > was not changed and started after about 5min of domu bootup (so that no > > > strange bootup-effects become relevant); same phy-backend (lvm on ssd), > > > same everything else; so i cant see what else except the used dom0 > > > kernel is causing this issue; but here are the numbers: > > > > > > a) read: 135mb/s write: 142mb/s > > > b) read: 39mb/s write: 39mb/s > > > c) read: 40mb/s write: 40mb/s > > > > > > Only thing that may become relevant is the difference in kernel-config > > > betwen 3.3 and 3.4 - here''s the diff : > > > http://pastebin.com/raw.php?i=Dy71Fegq > > > > > > Jan, it seems you''re right: The patch doesn''t add extra performance > > > regression - i guess i had an i/o intensive task running in dom0 while > > > doing the benchmark yesterday, so that the write performance got so bad. > > > sorry for that. > > > > > > Still there''s a significant performance penalty from 3.3 to 3.4 > > > > Could you please try to revert the following commits? > > > > git revert -n a71e23d9925517e609dfcb72b5874f33cdb0d2adNo way> > git revert -n 3389bb8bf76180eecaffdfa7dd5b35fa4a2ce9b5Startup.> > git revert -n 4dae76705fc8f9854bb732f9944e7ff9ba7a8e9fHm, this is just during startup.> > git revert -n b2167ba6dd89d55ced26a867fad8f0fe388fd595No way.> > git revert -n 4f14faaab4ee46a046b6baff85644be199de718cPerhaps? But I am not seeing it.> > git revert -n 9846ff10af12f9e7caac696737db6c990592a74aPerhaps?> > after reverting said 6 commits (thanks for the ids of these - had difficulties > to find them), the performance is back to normal. > > should i try to circle it down to one of this 6, or do you have a hint on > which it might be?I think either off these: 4f14faaab4ee46a046b6baff85644be199de718c 9846ff10af12f9e7caac696737db6c990592a74a might be the culprit. Try the 9846ff10 first.> > Greetings > Tobias
Tobias Geiger
2012-Apr-24 23:21 UTC
Re: Degregated I/O Performance since 3.4 - Regression in 3.4?
Am 24.04.2012 18:30, schrieb Konrad Rzeszutek Wilk:>>>> I redid the test; >>>> >>>> a) with 3.3.0 kernel >>>> b) with 3.4.0-rc4 >>>> c) with 3.40-rc4 and above patch >>>> >>>> everything else remained the same, i.e. test-program and test-scenario >>>> was not changed and started after about 5min of domu bootup (so that no >>>> strange bootup-effects become relevant); same phy-backend (lvm on ssd), >>>> same everything else; so i cant see what else except the used dom0 >>>> kernel is causing this issue; but here are the numbers: >>>> >>>> a) read: 135mb/s write: 142mb/s >>>> b) read: 39mb/s write: 39mb/s >>>> c) read: 40mb/s write: 40mb/s >>>> >>>> Only thing that may become relevant is the difference in kernel-config >>>> betwen 3.3 and 3.4 - here''s the diff : >>>> http://pastebin.com/raw.php?i=Dy71Fegq >>>> >>>> Jan, it seems you''re right: The patch doesn''t add extra performance >>>> regression - i guess i had an i/o intensive task running in dom0 while >>>> doing the benchmark yesterday, so that the write performance got so bad. >>>> sorry for that. >>>> >>>> Still there''s a significant performance penalty from 3.3 to 3.4 >>> Could you please try to revert the following commits? >>> >>> git revert -n a71e23d9925517e609dfcb72b5874f33cdb0d2ad > No way >>> git revert -n 3389bb8bf76180eecaffdfa7dd5b35fa4a2ce9b5 > Startup. >>> git revert -n 4dae76705fc8f9854bb732f9944e7ff9ba7a8e9f > Hm, this is just during startup. >>> git revert -n b2167ba6dd89d55ced26a867fad8f0fe388fd595 > No way. > > >>> git revert -n 4f14faaab4ee46a046b6baff85644be199de718c > Perhaps? But I am not seeing it. > >>> git revert -n 9846ff10af12f9e7caac696737db6c990592a74a > Perhaps? >> after reverting said 6 commits (thanks for the ids of these - had difficulties >> to find them), the performance is back to normal. >> >> should i try to circle it down to one of this 6, or do you have a hint on >> which it might be? > I think either off these: 4f14faaab4ee46a046b6baff85644be199de718c > 9846ff10af12f9e7caac696737db6c990592a74a might be the culprit. > > Try the 9846ff10 first. > >> Greetings >> TobiasHi, 9846ff10 was it! after reverting it, performance returned to normal. Thanks! Tobias
Stefano Stabellini
2012-Apr-25 13:44 UTC
Re: Degregated I/O Performance since 3.4 - Regression in 3.4?
On Wed, 25 Apr 2012, Tobias Geiger wrote:> Hi, > > 9846ff10 was it! > after reverting it, performance returned to normal.Argh, that is one of my commits, and is supposed to be a performance improvement! I couldn''t reproduce the regression you are seeing but the patch is pretty simple, so just by reading it again I think I manage to spot a possible error. Could you please try again with the appended patch? diff --git a/drivers/xen/events.c b/drivers/xen/events.c index 4b33acd..0a8a17c 100644 --- a/drivers/xen/events.c +++ b/drivers/xen/events.c @@ -274,7 +274,7 @@ static unsigned int cpu_from_evtchn(unsigned int evtchn) static bool pirq_check_eoi_map(unsigned irq) { - return test_bit(irq, pirq_eoi_map); + return test_bit(pirq_from_irq(irq), pirq_eoi_map); } static bool pirq_needs_eoi_flag(unsigned irq)
Tobias Geiger
2012-Apr-25 13:57 UTC
Re: Degregated I/O Performance since 3.4 - Regression in 3.4?
Am Mittwoch, 25. April 2012, 15:44:44 schrieb Stefano Stabellini:> On Wed, 25 Apr 2012, Tobias Geiger wrote: > > Hi, > > > > 9846ff10 was it! > > after reverting it, performance returned to normal. > > Argh, that is one of my commits, and is supposed to be a performance > improvement! > > I couldn''t reproduce the regression you are seeing but the patch is > pretty simple, so just by reading it again I think I manage to spot a > possible error. Could you please try again with the appended patch? > > > > diff --git a/drivers/xen/events.c b/drivers/xen/events.c > index 4b33acd..0a8a17c 100644 > --- a/drivers/xen/events.c > +++ b/drivers/xen/events.c > @@ -274,7 +274,7 @@ static unsigned int cpu_from_evtchn(unsigned int > evtchn) > > static bool pirq_check_eoi_map(unsigned irq) > { > - return test_bit(irq, pirq_eoi_map); > + return test_bit(pirq_from_irq(irq), pirq_eoi_map); > } > > static bool pirq_needs_eoi_flag(unsigned irq)Looks good; I applied it to 3.4.0-rc4 and the performance went to normal again. Thanks! Tobias
Tobias Geiger
2012-Apr-25 15:03 UTC
Re: Degregated I/O Performance since 3.4 - Regression in 3.4?
Am Mittwoch, 25. April 2012, 17:05:26 schrieb Stefano Stabellini:> On Wed, 25 Apr 2012, Tobias Geiger wrote: > > Am Mittwoch, 25. April 2012, 15:44:44 schrieb Stefano Stabellini: > > > On Wed, 25 Apr 2012, Tobias Geiger wrote: > > > > Hi, > > > > > > > > 9846ff10 was it! > > > > after reverting it, performance returned to normal. > > > > > > Argh, that is one of my commits, and is supposed to be a performance > > > improvement! > > > > > > I couldn''t reproduce the regression you are seeing but the patch is > > > pretty simple, so just by reading it again I think I manage to spot a > > > possible error. Could you please try again with the appended patch? > > > > > > > > > > > > diff --git a/drivers/xen/events.c b/drivers/xen/events.c > > > index 4b33acd..0a8a17c 100644 > > > --- a/drivers/xen/events.c > > > +++ b/drivers/xen/events.c > > > @@ -274,7 +274,7 @@ static unsigned int cpu_from_evtchn(unsigned int > > > evtchn) > > > > > > static bool pirq_check_eoi_map(unsigned irq) > > > { > > > > > > - return test_bit(irq, pirq_eoi_map); > > > + return test_bit(pirq_from_irq(irq), pirq_eoi_map); > > > > > > } > > > > > > static bool pirq_needs_eoi_flag(unsigned irq) > > > > Looks good; > > > > I applied it to 3.4.0-rc4 and the performance went to normal again. > > I''ll add your tested-by if it is OK for youof course - thanks!
Stefano Stabellini
2012-Apr-25 15:05 UTC
Re: Degregated I/O Performance since 3.4 - Regression in 3.4?
On Wed, 25 Apr 2012, Tobias Geiger wrote:> Am Mittwoch, 25. April 2012, 15:44:44 schrieb Stefano Stabellini: > > On Wed, 25 Apr 2012, Tobias Geiger wrote: > > > Hi, > > > > > > 9846ff10 was it! > > > after reverting it, performance returned to normal. > > > > Argh, that is one of my commits, and is supposed to be a performance > > improvement! > > > > I couldn''t reproduce the regression you are seeing but the patch is > > pretty simple, so just by reading it again I think I manage to spot a > > possible error. Could you please try again with the appended patch? > > > > > > > > diff --git a/drivers/xen/events.c b/drivers/xen/events.c > > index 4b33acd..0a8a17c 100644 > > --- a/drivers/xen/events.c > > +++ b/drivers/xen/events.c > > @@ -274,7 +274,7 @@ static unsigned int cpu_from_evtchn(unsigned int > > evtchn) > > > > static bool pirq_check_eoi_map(unsigned irq) > > { > > - return test_bit(irq, pirq_eoi_map); > > + return test_bit(pirq_from_irq(irq), pirq_eoi_map); > > } > > > > static bool pirq_needs_eoi_flag(unsigned irq) > > Looks good; > > I applied it to 3.4.0-rc4 and the performance went to normal again.I''ll add your tested-by if it is OK for you