Florian Manschwetus wrote:> Today a hvm guest on my xvm-3.4 testserver crashed. > It is a win2003R2 x64 (no PV-Drivers) running an exchange 2007> The domain is viridian enabled.I assume you have a H/W which supports HAP?> the corresponding qemu-dm process was around 11G Ram according to top.OK, thanks.. I see if I can reproduce... can you check to see if you are running the latest qemu bits... i.e. does a hg qpop -a;hg pull -uv in your qemu.hg dir pull anything? thanks, MRJ> florian > > xm li -l of domain after reboot > ---------------------------------------------------------------- > xm li -l tmpexch > (domain > > (domid 7) > > (on_crash restart) > > (uuid c764f542-de73-5037-5d48-db8da222bc39) > > (bootloader_args ) > > (vcpus 2) > > (name tmpexch) > > (on_poweroff destroy) > > (on_reboot restart) > > (cpus (() ())) > > (bootloader ) > > (maxmem 4096) > > (memory 4096) > > (shadow_memory 34) > > (features ) > > (on_xend_start ignore) > > (on_xend_stop shutdown) > > (start_time 1254813502.58) > > (cpu_time 215.303626766) > > (online_vcpus 2) > > (image > > (hvm > > (kernel ) > > (videoram 4) > > (hpet 1) > > (stdvga 0) > > (loader /usr/lib/xen/boot/hvmloader) > > (serial pty) > > (vncunused 1) > > (xen_platform_pci 1) > > (boot c) > > (rtc_timeoffset 32) > > (pci ()) > > (pae 1) > > (vpt_align 1) > > (hap 1) > > (viridian 1) > > (acpi 1) > > (localtime 1) > > (timer_mode 1) > > (nographic 0) > > (guest_os_type windows) > > (pci_msitranslate 1) > > (apic 1) > > (monitor 0) > > (usbdevice tablet) > > (device_model /usr/lib/xen/bin/amd64/qemu-dm) > > (keymap en-us) > > (pci_power_mgmt 0) > > (usb 1) > > (xauthority /export/home/tkadmin/.Xauthority) > > (isa 0) > > (notes (SUSPEND_CANCEL 1)) > > ) > > ) > > (status 2) > > (state -b----) > > (store_mfn 1044476) > > (device > > (vif > > (bridge e1000g0) > > (mac 00:16:3e:00:00:A1) > > (script /usr/lib/xen/scripts/vif-vnic) > > (uuid 9301430c-3cab-dcf4-5511-5d7310e54adc) > > (backend 0) > > ) > > ) > > (device > > (vbd > > (uuid 511c91e4-e9b0-7165-c278-e8b06df5290d) > > (bootable 1) > > (dev hda:disk) > > (uname phy:/dev/dsk/c0t600A0B800049E902000008E64A967E8Ed0p0) > > (mode w) > > (backend 0) > > (bootable 1) > > (VDI ) > > ) > > ) > > (device > > (vbd > > (uuid 8e39ff23-9e26-d7f2-b519-50eb09aa4497) > > (bootable 0) > > (dev hdb:disk) > > (uname phy:/dev/dsk/c0t600A0B800049E902000008E74A967ECCd0p0) > > (mode w) > > (backend 0) > > (bootable 0) > > (VDI ) > > ) > > ) > > (device > > (vbd > > (uuid 027a5704-eb60-c0f8-aaab-826ae13e9119) > > (bootable 0) > > (dev hdc:disk) > (uname phy:/dev/dsk/c0t600A0B800049E9020000090A4AA199D6d0p0) > (mode w) > (backend 0) > (bootable 0) > (VDI ) > ) > ) > (device (vkbd (backend 0))) > (device > (vfb > (vncunused 1) > (vnc 1) > (xauthority /export/home/tkadmin/.Xauthority) > (keymap de) > (location 127.0.0.1:5900) > (uuid c0bf43d4-cb09-9f0d-878a-843fdbd1c404) > ) > ) > (device > (console > (protocol vt100) > (location 4) > (uuid e2d0abfe-ea79-2c56-ac99-fb005c81e57d) > ) > ) > ) > > > > ------------------------------------------------------------------------ > > _______________________________________________ > xen-discuss mailing list > xen-discuss@opensolaris.org
Today a hvm guest on my xvm-3.4 testserver crashed.
It is a win2003R2 x64 (no PV-Drivers) running an exchange 2007
The domain is viridian enabled.
the corresponding qemu-dm process was around 11G Ram according to top.
florian
xm li -l of domain after reboot
----------------------------------------------------------------
xm li -l tmpexch
(domain
(domid 7)
(on_crash restart)
(uuid c764f542-de73-5037-5d48-db8da222bc39)
(bootloader_args )
(vcpus 2)
(name tmpexch)
(on_poweroff destroy)
(on_reboot restart)
(cpus (() ()))
(bootloader )
(maxmem 4096)
(memory 4096)
(shadow_memory 34)
(features )
(on_xend_start ignore)
(on_xend_stop shutdown)
(start_time 1254813502.58)
(cpu_time 215.303626766)
(online_vcpus 2)
(image
(hvm
(kernel )
(videoram 4)
(hpet 1)
(stdvga 0)
(loader /usr/lib/xen/boot/hvmloader)
(serial pty)
(vncunused 1)
(xen_platform_pci 1)
(boot c)
(rtc_timeoffset 32)
(pci ())
(pae 1)
(vpt_align 1)
(hap 1)
(viridian 1)
(acpi 1)
(localtime 1)
(timer_mode 1)
(nographic 0)
(guest_os_type windows)
(pci_msitranslate 1)
(apic 1)
(monitor 0)
(usbdevice tablet)
(device_model /usr/lib/xen/bin/amd64/qemu-dm)
(keymap en-us)
(pci_power_mgmt 0)
(usb 1)
(xauthority /export/home/tkadmin/.Xauthority)
(isa 0)
(notes (SUSPEND_CANCEL 1))
)
)
(status 2)
(state -b----)
(store_mfn 1044476)
(device
(vif
(bridge e1000g0)
(mac 00:16:3e:00:00:A1)
(script /usr/lib/xen/scripts/vif-vnic)
(uuid 9301430c-3cab-dcf4-5511-5d7310e54adc)
(backend 0)
)
)
(device
(vbd
(uuid 511c91e4-e9b0-7165-c278-e8b06df5290d)
(bootable 1)
(dev hda:disk)
(uname phy:/dev/dsk/c0t600A0B800049E902000008E64A967E8Ed0p0)
(mode w)
(backend 0)
(bootable 1)
(VDI )
)
)
(device
(vbd
(uuid 8e39ff23-9e26-d7f2-b519-50eb09aa4497)
(bootable 0)
(dev hdb:disk)
(uname phy:/dev/dsk/c0t600A0B800049E902000008E74A967ECCd0p0)
(mode w)
(backend 0)
(bootable 0)
(VDI )
)
)
(device
(vbd
(uuid 027a5704-eb60-c0f8-aaab-826ae13e9119)
(bootable 0)
(dev hdc:disk)
(uname phy:/dev/dsk/c0t600A0B800049E9020000090A4AA199D6d0p0)
(mode w)
(backend 0)
(bootable 0)
(VDI )
)
)
(device (vkbd (backend 0)))
(device
(vfb
(vncunused 1)
(vnc 1)
(xauthority /export/home/tkadmin/.Xauthority)
(keymap de)
(location 127.0.0.1:5900)
(uuid c0bf43d4-cb09-9f0d-878a-843fdbd1c404)
)
)
(device
(console
(protocol vt100)
(location 4)
(uuid e2d0abfe-ea79-2c56-ac99-fb005c81e57d)
)
)
)
Am 06.10.2009 00:37, schrieb Mark Johnson:> > > Florian Manschwetus wrote: >> Today a hvm guest on my xvm-3.4 testserver crashed. >> It is a win2003R2 x64 (no PV-Drivers) running an exchange 2007 > > >> The domain is viridian enabled. > > I assume you have a H/W which supports HAP?X4150> > >> the corresponding qemu-dm process was around 11G Ram according to top. > > OK, thanks.. I see if I can reproduce... can you check to see if you > are running > the latest qemu bits... i.e. does a hg qpop -a;hg pull -uv in your > qemu.hg dir pull > anything? > > > > thanks, > > > MRJ > > >> florian >> >> xm li -l of domain after reboot >> ---------------------------------------------------------------- >> xm li -l tmpexch >> (domain >> >> (domid 7) >> >> (on_crash restart) >> >> (uuid c764f542-de73-5037-5d48-db8da222bc39) >> >> (bootloader_args ) >> >> (vcpus 2) >> >> (name tmpexch) >> >> (on_poweroff destroy) >> >> (on_reboot restart) >> >> (cpus (() ())) >> >> (bootloader ) >> >> (maxmem 4096) >> >> (memory 4096) >> >> (shadow_memory 34) >> >> (features ) >> >> (on_xend_start ignore) >> >> (on_xend_stop shutdown) >> >> (start_time 1254813502.58) >> >> (cpu_time 215.303626766) >> >> (online_vcpus 2) >> >> (image >> >> (hvm >> >> (kernel ) >> >> (videoram 4) >> >> (hpet 1) >> >> (stdvga 0) >> >> (loader /usr/lib/xen/boot/hvmloader) >> >> (serial pty) >> >> (vncunused 1) >> >> (xen_platform_pci 1) >> >> (boot c) >> >> (rtc_timeoffset 32) >> >> (pci ()) >> >> (pae 1) >> >> (vpt_align 1) >> >> (hap 1) >> >> (viridian 1) >> >> (acpi 1) >> >> (localtime 1) >> >> (timer_mode 1) >> >> (nographic 0) >> >> (guest_os_type windows) >> >> (pci_msitranslate 1) >> >> (apic 1) >> >> (monitor 0) >> >> (usbdevice tablet) >> >> (device_model /usr/lib/xen/bin/amd64/qemu-dm) >> >> (keymap en-us) >> >> (pci_power_mgmt 0) >> >> (usb 1) >> >> (xauthority /export/home/tkadmin/.Xauthority) >> >> (isa 0) >> >> (notes (SUSPEND_CANCEL 1)) >> >> ) >> >> ) >> >> (status 2) >> >> (state -b----) >> >> (store_mfn 1044476) >> >> (device >> >> (vif >> >> (bridge e1000g0) >> >> (mac 00:16:3e:00:00:A1) >> >> (script /usr/lib/xen/scripts/vif-vnic) >> >> (uuid 9301430c-3cab-dcf4-5511-5d7310e54adc) >> >> (backend 0) >> >> ) >> >> ) >> >> (device >> >> (vbd >> >> (uuid 511c91e4-e9b0-7165-c278-e8b06df5290d) >> >> (bootable 1) >> >> (dev hda:disk) >> >> (uname phy:/dev/dsk/c0t600A0B800049E902000008E64A967E8Ed0p0) >> >> (mode w) >> >> (backend 0) >> >> (bootable 1) >> >> (VDI ) >> >> ) >> >> ) >> >> (device >> >> (vbd >> >> (uuid 8e39ff23-9e26-d7f2-b519-50eb09aa4497) >> >> (bootable 0) >> >> (dev hdb:disk) >> >> (uname phy:/dev/dsk/c0t600A0B800049E902000008E74A967ECCd0p0) >> >> (mode w) >> >> (backend 0) >> >> (bootable 0) >> >> (VDI ) >> >> ) >> >> ) >> >> (device >> >> (vbd >> >> (uuid 027a5704-eb60-c0f8-aaab-826ae13e9119) >> >> (bootable 0) >> >> (dev hdc:disk) >> (uname phy:/dev/dsk/c0t600A0B800049E9020000090A4AA199D6d0p0) >> (mode w) >> (backend 0) >> (bootable 0) >> (VDI ) >> ) >> ) >> (device (vkbd (backend 0))) >> (device >> (vfb >> (vncunused 1) >> (vnc 1) >> (xauthority /export/home/tkadmin/.Xauthority) >> (keymap de) >> (location 127.0.0.1:5900) >> (uuid c0bf43d4-cb09-9f0d-878a-843fdbd1c404) >> ) >> ) >> (device >> (console >> (protocol vt100) >> (location 4) >> (uuid e2d0abfe-ea79-2c56-ac99-fb005c81e57d) >> ) >> ) >> ) >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> xen-discuss mailing list >> xen-discuss@opensolaris.org >
Florian Manschwetus wrote:> Today a hvm guest on my xvm-3.4 testserver crashed. > It is a win2003R2 x64 (no PV-Drivers) running an exchange 2007 > The domain is viridian enabled. > > the corresponding qemu-dm process was around 11G Ram according to top.I believe you are running into the bug the following patch fixed... Ian just pushed a bunch of fixes to qemu-xen-unstable... I pinged him to see if he plans on pushing them to 3.4 too? If so, I''ll sync up and push.. If not, I should be able to get the fixes backported this week. MRJ commit 9c58cf9321f746b223bdd778c49c31ff756b6b1b Author: Ian Jackson <ian.jackson@eu.citrix.com> Date: Fri Sep 4 16:12:28 2009 +0100 fix qemu memory leak in block interface the qemu block interface leaks memory every time a read or write request is issued, this patch fixes it. This is also the bug that is causing stubdomains to crash under high disk IO. Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Mark, could you also make a short info howto update, a system I build just yesterday, when the patch is inplace. The older Machine I would have to re-bfu due to changed userspace, but the machine installed yesterday should be fixed with rebuilding and reinstalling xvm packages? Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build b123 with b124, due to missing, with b124 obsolete, headerfiles. Florian Am 07.10.2009 17:46, schrieb Mark Johnson:> > > Florian Manschwetus wrote: >> Today a hvm guest on my xvm-3.4 testserver crashed. >> It is a win2003R2 x64 (no PV-Drivers) running an exchange 2007 >> The domain is viridian enabled. >> >> the corresponding qemu-dm process was around 11G Ram according to top. > > I believe you are running into the bug the following patch fixed... > Ian just pushed a bunch of fixes to qemu-xen-unstable... I pinged him to > see if he plans on pushing them to 3.4 too? If so, I''ll sync up and > push.. > > If not, I should be able to get the fixes backported this week. > > > MRJ > > > commit 9c58cf9321f746b223bdd778c49c31ff756b6b1b > Author: Ian Jackson <ian.jackson@eu.citrix.com> > Date: Fri Sep 4 16:12:28 2009 +0100 > > fix qemu memory leak in block interface > > the qemu block interface leaks memory every time a read or write > request > is issued, this patch fixes it. > This is also the bug that is causing stubdomains to crash under high > disk IO. > > Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> > > > > >
Florian Manschwetus wrote:> Mark, > could you also make a short info howto update, a system I build just > yesterday, when the patch is inplace. The older Machine I would have to > re-bfu due to changed userspace, but the machine installed yesterday > should be fixed with rebuilding and reinstalling xvm packages?If I understand what your asking, yes you only need to update the xvm-packages...> Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build > b123 with b124, due to missing, with b124 obsolete, headerfiles.I just sync''d onnv up to b125 and pushed it a couple of hours ago... The e-mail notification is blocked waiting for approval.. Should be out soon. MRJ> Florian > > Am 07.10.2009 17:46, schrieb Mark Johnson: >> >> Florian Manschwetus wrote: >>> Today a hvm guest on my xvm-3.4 testserver crashed. >>> It is a win2003R2 x64 (no PV-Drivers) running an exchange 2007 >>> The domain is viridian enabled. >>> >>> the corresponding qemu-dm process was around 11G Ram according to top. >> I believe you are running into the bug the following patch fixed... >> Ian just pushed a bunch of fixes to qemu-xen-unstable... I pinged him to >> see if he plans on pushing them to 3.4 too? If so, I''ll sync up and >> push.. >> >> If not, I should be able to get the fixes backported this week. >> >> >> MRJ >> >> >> commit 9c58cf9321f746b223bdd778c49c31ff756b6b1b >> Author: Ian Jackson <ian.jackson@eu.citrix.com> >> Date: Fri Sep 4 16:12:28 2009 +0100 >> >> fix qemu memory leak in block interface >> >> the qemu block interface leaks memory every time a read or write >> request >> is issued, this patch fixes it. >> This is also the bug that is causing stubdomains to crash under high >> disk IO. >> >> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> >> >> >> >> >> > >
Has some one news on what the outcome will be? Florian Am 07.10.2009 19:40, schrieb Mark Johnson:> > > Florian Manschwetus wrote: >> Mark, >> could you also make a short info howto update, a system I build just >> yesterday, when the patch is inplace. The older Machine I would have to >> re-bfu due to changed userspace, but the machine installed yesterday >> should be fixed with rebuilding and reinstalling xvm packages? > > If I understand what your asking, yes you only need to update the > xvm-packages... > > > >> Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build >> b123 with b124, due to missing, with b124 obsolete, headerfiles. > > I just sync''d onnv up to b125 and pushed it a couple of hours ago... > The e-mail notification is blocked waiting for approval.. Should be out > soon. > > > > > MRJ > > > > > > >> Florian >> >> Am 07.10.2009 17:46, schrieb Mark Johnson: >>> >>> Florian Manschwetus wrote: >>>> Today a hvm guest on my xvm-3.4 testserver crashed. >>>> It is a win2003R2 x64 (no PV-Drivers) running an exchange 2007 >>>> The domain is viridian enabled. >>>> >>>> the corresponding qemu-dm process was around 11G Ram according to top. >>> I believe you are running into the bug the following patch fixed... >>> Ian just pushed a bunch of fixes to qemu-xen-unstable... I pinged him to >>> see if he plans on pushing them to 3.4 too? If so, I''ll sync up and >>> push.. >>> >>> If not, I should be able to get the fixes backported this week. >>> >>> >>> MRJ >>> >>> >>> commit 9c58cf9321f746b223bdd778c49c31ff756b6b1b >>> Author: Ian Jackson <ian.jackson@eu.citrix.com> >>> Date: Fri Sep 4 16:12:28 2009 +0100 >>> >>> fix qemu memory leak in block interface >>> >>> the qemu block interface leaks memory every time a read or write >>> request >>> is issued, this patch fixes it. >>> This is also the bug that is causing stubdomains to crash under high >>> disk IO. >>> >>> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> >>> >>> >>> >>> >>> >> >> >
Florian Manschwetus wrote:> Has some one news on what the outcome will be?Ian said he would push the fixed to the upstream 3.4 qemu-xen gate after a couple of more days of soak time in the unstable gate. MRJ> Florian > > Am 07.10.2009 19:40, schrieb Mark Johnson: >> >> Florian Manschwetus wrote: >>> Mark, >>> could you also make a short info howto update, a system I build just >>> yesterday, when the patch is inplace. The older Machine I would have to >>> re-bfu due to changed userspace, but the machine installed yesterday >>> should be fixed with rebuilding and reinstalling xvm packages? >> If I understand what your asking, yes you only need to update the >> xvm-packages... >> >> >> >>> Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build >>> b123 with b124, due to missing, with b124 obsolete, headerfiles. >> I just sync''d onnv up to b125 and pushed it a couple of hours ago... >> The e-mail notification is blocked waiting for approval.. Should be out >> soon. >> >> >> >> >> MRJ >> >> >> >> >> >> >>> Florian >>> >>> Am 07.10.2009 17:46, schrieb Mark Johnson: >>>> Florian Manschwetus wrote: >>>>> Today a hvm guest on my xvm-3.4 testserver crashed. >>>>> It is a win2003R2 x64 (no PV-Drivers) running an exchange 2007 >>>>> The domain is viridian enabled. >>>>> >>>>> the corresponding qemu-dm process was around 11G Ram according to top. >>>> I believe you are running into the bug the following patch fixed... >>>> Ian just pushed a bunch of fixes to qemu-xen-unstable... I pinged him to >>>> see if he plans on pushing them to 3.4 too? If so, I''ll sync up and >>>> push.. >>>> >>>> If not, I should be able to get the fixes backported this week. >>>> >>>> >>>> MRJ >>>> >>>> >>>> commit 9c58cf9321f746b223bdd778c49c31ff756b6b1b >>>> Author: Ian Jackson <ian.jackson@eu.citrix.com> >>>> Date: Fri Sep 4 16:12:28 2009 +0100 >>>> >>>> fix qemu memory leak in block interface >>>> >>>> the qemu block interface leaks memory every time a read or write >>>> request >>>> is issued, this patch fixes it. >>>> This is also the bug that is causing stubdomains to crash under high >>>> disk IO. >>>> >>>> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> >>>> >>>> >>>> >>>> >>>> >>> > >
Hi Florian, Florian Manschwetus wrote:> Mark, > could you also make a short info howto update, a system I build just > yesterday, when the patch is inplace. The older Machine I would have to > re-bfu due to changed userspace, but the machine installed yesterday > should be fixed with rebuilding and reinstalling xvm packages? > > Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build > b123 with b124, due to missing, with b124 obsolete, headerfiles.I''ve sync''d up with the changes in qemu-unstable, and qemu-dm is still leaking a ton of memory for HVM guests which don''t have PV drivers. So it looks like they don''t have a correct fix upstream for this. MRJ
Florian Manschwetus
2009-Oct-14 21:58 UTC
Re: [xen-discuss] possible memory-leak in qemu-dm
Am 14.10.2009 21:46, schrieb Mark Johnson:> > Hi Florian, > > Florian Manschwetus wrote: >> Mark, >> could you also make a short info howto update, a system I build just >> yesterday, when the patch is inplace. The older Machine I would have to >> re-bfu due to changed userspace, but the machine installed yesterday >> should be fixed with rebuilding and reinstalling xvm packages? >> >> Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build >> b123 with b124, due to missing, with b124 obsolete, headerfiles. > > I''ve sync''d up with the changes in qemu-unstable, and qemu-dm is > still leaking a ton of memory for HVM guests which don''t have PV > drivers. So it looks like they don''t have a correct fix upstream > for this.Uh, if this could be confirmed it is really a huge and ultimately ugly problem. If I could do anything to help you to fix it, give me some detail. Florian> > > > > MRJ >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2009-Oct-15 10:26 UTC
Re: Re: [xen-discuss] possible memory-leak in qemu-dm
On Wed, Oct 14, 2009 at 11:58:14PM +0200, Florian Manschwetus wrote:> Am 14.10.2009 21:46, schrieb Mark Johnson: > > > > Hi Florian, > > > > Florian Manschwetus wrote: > >> Mark, > >> could you also make a short info howto update, a system I build just > >> yesterday, when the patch is inplace. The older Machine I would have to > >> re-bfu due to changed userspace, but the machine installed yesterday > >> should be fixed with rebuilding and reinstalling xvm packages? > >> > >> Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build > >> b123 with b124, due to missing, with b124 obsolete, headerfiles. > > > > I''ve sync''d up with the changes in qemu-unstable, and qemu-dm is > > still leaking a ton of memory for HVM guests which don''t have PV > > drivers. So it looks like they don''t have a correct fix upstream > > for this. > Uh, if this could be confirmed it is really a huge and ultimately ugly > problem. If I could do anything to help you to fix it, give me some detail. >Yeah.. what kind of HVM guest is triggering this bug? How fast it is leaking mem? -- Pasi
Florian Manschwetus
2009-Oct-15 10:41 UTC
Re: Re: [xen-discuss] possible memory-leak in qemu-dm
Am 15.10.2009 12:26, schrieb Pasi Kärkkäinen:> On Wed, Oct 14, 2009 at 11:58:14PM +0200, Florian Manschwetus wrote: >> Am 14.10.2009 21:46, schrieb Mark Johnson: >>> >>> Hi Florian, >>> >>> Florian Manschwetus wrote: >>>> Mark, >>>> could you also make a short info howto update, a system I build just >>>> yesterday, when the patch is inplace. The older Machine I would have to >>>> re-bfu due to changed userspace, but the machine installed yesterday >>>> should be fixed with rebuilding and reinstalling xvm packages? >>>> >>>> Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build >>>> b123 with b124, due to missing, with b124 obsolete, headerfiles. >>> >>> I''ve sync''d up with the changes in qemu-unstable, and qemu-dm is >>> still leaking a ton of memory for HVM guests which don''t have PV >>> drivers. So it looks like they don''t have a correct fix upstream >>> for this. >> Uh, if this could be confirmed it is really a huge and ultimately ugly >> problem. If I could do anything to help you to fix it, give me some detail. >> > > Yeah.. what kind of HVM guest is triggering this bug? > How fast it is leaking mem?It is a win2003R2 with exchange2007 xm uptime tmpexch Name ID Uptime tmpexch 7 9 days, 3:21:39 PID USERNAME NLWP PRI NICE SIZE RES STATE TIME CPU COMMAND 2106 xvm 3 13 0 7340M 3807M cpu/4 31.4H 6.57% qemu-dm Florian> > -- Pasi >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2009-Oct-15 11:04 UTC
Re: Re: [xen-discuss] possible memory-leak in qemu-dm
On Thu, 15 Oct 2009, Florian Manschwetus wrote:> Am 15.10.2009 12:26, schrieb Pasi Kärkkäinen: > > On Wed, Oct 14, 2009 at 11:58:14PM +0200, Florian Manschwetus wrote: > >> Am 14.10.2009 21:46, schrieb Mark Johnson: > >>> > >>> Hi Florian, > >>> > >>> Florian Manschwetus wrote: > >>>> Mark, > >>>> could you also make a short info howto update, a system I build just > >>>> yesterday, when the patch is inplace. The older Machine I would have to > >>>> re-bfu due to changed userspace, but the machine installed yesterday > >>>> should be fixed with rebuilding and reinstalling xvm packages? > >>>> > >>>> Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build > >>>> b123 with b124, due to missing, with b124 obsolete, headerfiles. > >>> > >>> I''ve sync''d up with the changes in qemu-unstable, and qemu-dm is > >>> still leaking a ton of memory for HVM guests which don''t have PV > >>> drivers. So it looks like they don''t have a correct fix upstream > >>> for this. > >> Uh, if this could be confirmed it is really a huge and ultimately ugly > >> problem. If I could do anything to help you to fix it, give me some detail. > >> > > > > Yeah.. what kind of HVM guest is triggering this bug? > > How fast it is leaking mem? > It is a win2003R2 with exchange2007 > > xm uptime tmpexch > Name ID Uptime > tmpexch 7 9 days, 3:21:39 > > PID USERNAME NLWP PRI NICE SIZE RES STATE TIME CPU COMMAND > 2106 xvm 3 13 0 7340M 3807M cpu/4 31.4H 6.57% qemu-dm > > Florian >Was this happening before you applied my patches, but after "fix qemu memory leak in block interface"? I am trying to understand if this is a recent regression. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Florian Manschwetus
2009-Oct-15 11:24 UTC
Re: Re: [xen-discuss] possible memory-leak in qemu-dm
Am 15.10.2009 13:04, schrieb Stefano Stabellini:> On Thu, 15 Oct 2009, Florian Manschwetus wrote: >> Am 15.10.2009 12:26, schrieb Pasi Kärkkäinen: >>> On Wed, Oct 14, 2009 at 11:58:14PM +0200, Florian Manschwetus wrote: >>>> Am 14.10.2009 21:46, schrieb Mark Johnson: >>>>> >>>>> Hi Florian, >>>>> >>>>> Florian Manschwetus wrote: >>>>>> Mark, >>>>>> could you also make a short info howto update, a system I build just >>>>>> yesterday, when the patch is inplace. The older Machine I would have to >>>>>> re-bfu due to changed userspace, but the machine installed yesterday >>>>>> should be fixed with rebuilding and reinstalling xvm packages? >>>>>> >>>>>> Btw, could you sync-up the onnv-3.4 with b124? It is difficult to build >>>>>> b123 with b124, due to missing, with b124 obsolete, headerfiles. >>>>> >>>>> I''ve sync''d up with the changes in qemu-unstable, and qemu-dm is >>>>> still leaking a ton of memory for HVM guests which don''t have PV >>>>> drivers. So it looks like they don''t have a correct fix upstream >>>>> for this. >>>> Uh, if this could be confirmed it is really a huge and ultimately ugly >>>> problem. If I could do anything to help you to fix it, give me some detail. >>>> >>> >>> Yeah.. what kind of HVM guest is triggering this bug? >>> How fast it is leaking mem? >> It is a win2003R2 with exchange2007 >> >> xm uptime tmpexch >> Name ID Uptime >> tmpexch 7 9 days, 3:21:39 >> >> PID USERNAME NLWP PRI NICE SIZE RES STATE TIME CPU COMMAND >> 2106 xvm 3 13 0 7340M 3807M cpu/4 31.4H 6.57% qemu-dm >> >> Florian >> > > Was this happening before you applied my patches, but after "fix qemu > memory leak in block interface"? > I am trying to understand if this is a recent regression.It is an older build of xvm-3.4 gate I have to review current patch state and will update accordingly if possible. It was Mark, who told that the problem isn''t fixed by the Patch. So I not rushed to update. Florian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Oct 15, 2009 at 7:04 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> Was this happening before you applied my patches, but after "fix qemu > memory leak in block interface"? > I am trying to understand if this is a recent regression.I''m not sure when it first showed up... It exists in current qemu-xen-3.4 bits. I''ve updated the qemu bits to the latest qemu-xen-unstable bits (basically running qemu-xen-unstable with xen 3.4) including your patches, and qemu-dm is still leaking memory with a HVM guest with no PV drivers.. Tried a couple of different HVM guests with the same results.. MRJ
Pasi Kärkkäinen
2009-Oct-21 11:24 UTC
Re: Re: [xen-discuss] possible memory-leak in qemu-dm
On Thu, Oct 15, 2009 at 01:24:29PM +0200, Florian Manschwetus wrote:> >>> > >>> Yeah.. what kind of HVM guest is triggering this bug? > >>> How fast it is leaking mem? > >> It is a win2003R2 with exchange2007 > >> > >> xm uptime tmpexch > >> Name ID Uptime > >> tmpexch 7 9 days, 3:21:39 > >> > >> PID USERNAME NLWP PRI NICE SIZE RES STATE TIME CPU COMMAND > >> 2106 xvm 3 13 0 7340M 3807M cpu/4 31.4H 6.57% qemu-dm > >> > >> Florian > >> > > > > Was this happening before you applied my patches, but after "fix qemu > > memory leak in block interface"? > > I am trying to understand if this is a recent regression. > It is an older build of xvm-3.4 gate > I have to review current patch state and will update accordingly if > possible. > > It was Mark, who told that the problem isn''t fixed by the Patch. So I > not rushed to update. >Hmm.. wondering if this is opensolaris specific, or if it happens also with Linux dom0s. -- Pasi
On Wed, Oct 21, 2009 at 7:24 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:> On Thu, Oct 15, 2009 at 01:24:29PM +0200, Florian Manschwetus wrote: >> > >> > Was this happening before you applied my patches, but after "fix qemu >> > memory leak in block interface"? >> > I am trying to understand if this is a recent regression. >> It is an older build of xvm-3.4 gate >> I have to review current patch state and will update accordingly if >> possible. >> >> It was Mark, who told that the problem isn''t fixed by the Patch. So I >> not rushed to update. >> > > Hmm.. wondering if this is opensolaris specific,It''s possible... I looked through our qemu patches and we don''t have any changes in that code path.. So I would be surprised..> or if it happens also with Linux dom0s.Has anyone else seen this on 3.4 or unstable? e.g. boot a rhel3 guest and watch the memory usage of qemu-dm... You can''t miss it if its leaking... MRJ
Stefano Stabellini
2009-Oct-21 15:00 UTC
Re: Re: [xen-discuss] possible memory-leak in qemu-dm
On Wed, 21 Oct 2009, Mark Johnson wrote:> On Wed, Oct 21, 2009 at 7:24 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote: > > On Thu, Oct 15, 2009 at 01:24:29PM +0200, Florian Manschwetus wrote: > >> > > >> > Was this happening before you applied my patches, but after "fix qemu > >> > memory leak in block interface"? > >> > I am trying to understand if this is a recent regression. > >> It is an older build of xvm-3.4 gate > >> I have to review current patch state and will update accordingly if > >> possible. > >> > >> It was Mark, who told that the problem isn''t fixed by the Patch. So I > >> not rushed to update. > >> > > > > Hmm.. wondering if this is opensolaris specific, > > It''s possible... I looked through our qemu patches and we don''t have > any changes in that code path.. So I would be surprised.. > > > > or if it happens also with Linux dom0s. > > Has anyone else seen this on 3.4 or unstable? e.g. boot a rhel3 guest and watch > the memory usage of qemu-dm... You can''t miss it if its leaking... >What kind of operations are you doing in the guest? Is there a simple benchmark that triggers this behaviour (for example a disk or network benchmark)? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Oct 21, 2009 at 9:54 AM, Mark Johnson <johnson.nh@gmail.com> wrote:> Has anyone else seen this on 3.4 or unstable? e.g. boot a rhel3 guest and watch > the memory usage of qemu-dm... You can''t miss it if its leaking...I just wanted to report that I also saw this memory leak problem on 3.4.1. I applied the patch mentioned before: http://lists.xensource.com/archives/html/xen-changelog/2009-10/msg00132.html After applying the patch, the memory leak seems to be fixed. This is on a Linux 64-bit dom0 and freebsd HVM guest. Memory leak of qemu-dm was apparent just by extracting a ~5MB tarball of various text files.