Anthony PERARD
2011-Aug-01 19:26 UTC
[Xen-devel] [PATCH] xen-mapcache: Fix rlimit set size.
Previously, the address space soft limit was set mcache_max_size. So, before the mcache_max_size was reached by the mapcache, QEMU was killed for overuse of the virtual address space. This patch fix that by setting the soft limit to mcache_max_size + 80MB. I observed that QEMU use 75MB more than max_mcache_size after several empirical tests. Signed-off-by: Anthony PERARD <anthony.perard@citrix.com> --- xen-mapcache.c | 13 ++++++++++--- 1 files changed, 10 insertions(+), 3 deletions(-) diff --git a/xen-mapcache.c b/xen-mapcache.c index 007136a..40212f7 100644 --- a/xen-mapcache.c +++ b/xen-mapcache.c @@ -40,6 +40,13 @@ #endif #define MCACHE_BUCKET_SIZE (1UL << MCACHE_BUCKET_SHIFT) +/* This is the size of the virtual address space reserve to QEMU that will not + * be use by MapCache. + * From empirical tests I observed that qemu use 75MB more than the + * max_mcache_size. + */ +#define NON_MCACHE_MEMORY_SIZE (80 * 1024 * 1024) + #define mapcache_lock() ((void)0) #define mapcache_unlock() ((void)0) @@ -93,14 +100,14 @@ void xen_map_cache_init(void) mapcache->last_address_index = -1; getrlimit(RLIMIT_AS, &rlimit_as); - if (rlimit_as.rlim_max < MCACHE_MAX_SIZE) { + if (rlimit_as.rlim_max < MCACHE_MAX_SIZE + NON_MCACHE_MEMORY_SIZE) { rlimit_as.rlim_cur = rlimit_as.rlim_max; } else { - rlimit_as.rlim_cur = MCACHE_MAX_SIZE; + rlimit_as.rlim_cur = MCACHE_MAX_SIZE + NON_MCACHE_MEMORY_SIZE; } setrlimit(RLIMIT_AS, &rlimit_as); - mapcache->max_mcache_size = rlimit_as.rlim_cur; + mapcache->max_mcache_size = rlimit_as.rlim_cur - NON_MCACHE_MEMORY_SIZE; mapcache->nr_buckets (((mapcache->max_mcache_size >> XC_PAGE_SHIFT) + -- Anthony PERARD _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2011-Aug-02 00:53 UTC
Re: [Xen-devel] [PATCH] xen-mapcache: Fix rlimit set size.
>>> Anthony PERARD 08/01/11 9:27 PM >>> >Previously, the address space soft limit was set mcache_max_size. So, >before the mcache_max_size was reached by the mapcache, QEMU was killed >for overuse of the virtual address space. > >This patch fix that by setting the soft limit to mcache_max_size + 80MB. >I observed that QEMU use 75MB more than max_mcache_size after several >empirical tests.Rather fragile going forward I would say. I there any reason not to set the limit to infinity? Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Alexander Graf
2011-Aug-02 14:50 UTC
[Xen-devel] Re: [PATCH] xen-mapcache: Fix rlimit set size.
On 08/01/2011 09:26 PM, Anthony PERARD wrote:> Previously, the address space soft limit was set mcache_max_size. So, > before the mcache_max_size was reached by the mapcache, QEMU was killed > for overuse of the virtual address space. > > This patch fix that by setting the soft limit to mcache_max_size + 80MB. > I observed that QEMU use 75MB more than max_mcache_size after several > empirical tests.Not sure I understand. What are those 75MB? Alex> Signed-off-by: Anthony PERARD<anthony.perard@citrix.com> > --- > xen-mapcache.c | 13 ++++++++++--- > 1 files changed, 10 insertions(+), 3 deletions(-) > > diff --git a/xen-mapcache.c b/xen-mapcache.c > index 007136a..40212f7 100644 > --- a/xen-mapcache.c > +++ b/xen-mapcache.c > @@ -40,6 +40,13 @@ > #endif > #define MCACHE_BUCKET_SIZE (1UL<< MCACHE_BUCKET_SHIFT) > > +/* This is the size of the virtual address space reserve to QEMU that will not > + * be use by MapCache. > + * From empirical tests I observed that qemu use 75MB more than the > + * max_mcache_size. > + */ > +#define NON_MCACHE_MEMORY_SIZE (80 * 1024 * 1024) > + > #define mapcache_lock() ((void)0) > #define mapcache_unlock() ((void)0) > > @@ -93,14 +100,14 @@ void xen_map_cache_init(void) > mapcache->last_address_index = -1; > > getrlimit(RLIMIT_AS,&rlimit_as); > - if (rlimit_as.rlim_max< MCACHE_MAX_SIZE) { > + if (rlimit_as.rlim_max< MCACHE_MAX_SIZE + NON_MCACHE_MEMORY_SIZE) { > rlimit_as.rlim_cur = rlimit_as.rlim_max; > } else { > - rlimit_as.rlim_cur = MCACHE_MAX_SIZE; > + rlimit_as.rlim_cur = MCACHE_MAX_SIZE + NON_MCACHE_MEMORY_SIZE; > } > > setrlimit(RLIMIT_AS,&rlimit_as); > - mapcache->max_mcache_size = rlimit_as.rlim_cur; > + mapcache->max_mcache_size = rlimit_as.rlim_cur - NON_MCACHE_MEMORY_SIZE; > > mapcache->nr_buckets > (((mapcache->max_mcache_size>> XC_PAGE_SHIFT) +_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Aug-02 22:01 UTC
[Xen-devel] Re: [PATCH] xen-mapcache: Fix rlimit set size.
On Mon, 1 Aug 2011, Anthony PERARD wrote:> Previously, the address space soft limit was set mcache_max_size. So, > before the mcache_max_size was reached by the mapcache, QEMU was killed > for overuse of the virtual address space. > > This patch fix that by setting the soft limit to mcache_max_size + 80MB. > I observed that QEMU use 75MB more than max_mcache_size after several > empirical tests. > > Signed-off-by: Anthony PERARD <anthony.perard@citrix.com> > --- > xen-mapcache.c | 13 ++++++++++--- > 1 files changed, 10 insertions(+), 3 deletions(-) > > diff --git a/xen-mapcache.c b/xen-mapcache.c > index 007136a..40212f7 100644 > --- a/xen-mapcache.c > +++ b/xen-mapcache.c > @@ -40,6 +40,13 @@ > #endif > #define MCACHE_BUCKET_SIZE (1UL << MCACHE_BUCKET_SHIFT) > > +/* This is the size of the virtual address space reserve to QEMU that will not > + * be use by MapCache. > + * From empirical tests I observed that qemu use 75MB more than the > + * max_mcache_size. > + */ > +#define NON_MCACHE_MEMORY_SIZE (80 * 1024 * 1024) > + > #define mapcache_lock() ((void)0) > #define mapcache_unlock() ((void)0) > > @@ -93,14 +100,14 @@ void xen_map_cache_init(void) > mapcache->last_address_index = -1; > > getrlimit(RLIMIT_AS, &rlimit_as); > - if (rlimit_as.rlim_max < MCACHE_MAX_SIZE) { > + if (rlimit_as.rlim_max < MCACHE_MAX_SIZE + NON_MCACHE_MEMORY_SIZE) { > rlimit_as.rlim_cur = rlimit_as.rlim_max; > } else { > - rlimit_as.rlim_cur = MCACHE_MAX_SIZE; > + rlimit_as.rlim_cur = MCACHE_MAX_SIZE + NON_MCACHE_MEMORY_SIZE; > }In the ''else'' case we should just set rlimit_as.rlim_cur to rlimit_as.rlim_max. Also we should detect if we are running as a privileged process and if that is the case just modify rlimit_as.rlim_max to RLIM_INFINITY. If we are not running as a privileged process and if rlim_max is not high enough we should print a warning because trying to guess how much virtual memory qemu is going to use is obviously error prone. So this is how I think it should work: if (rlimit_as.rlim_max < MCACHE_MAX_SIZE) { if (qemu is priviledged) { rlimit_as.rlim_max = RLIM_INFINITY; rlimit_as.rlim_cur = RLIM_INFINITY; } else { print a warning mapcache->max_mcache_size = rlimit_as.rlim_cur - NON_MCACHE_MEMORY_SIZE; rlimit_as.rlim_cur = rlimit_as.rlim_max; } } else { rlimit_as.rlim_cur = rlimit_as.rlim_max; } setrlimit> setrlimit(RLIMIT_AS, &rlimit_as); > - mapcache->max_mcache_size = rlimit_as.rlim_cur; > + mapcache->max_mcache_size = rlimit_as.rlim_cur - NON_MCACHE_MEMORY_SIZE; > > mapcache->nr_buckets > (((mapcache->max_mcache_size >> XC_PAGE_SHIFT) + > -- > Anthony PERARD >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Aug-02 22:03 UTC
Re: [Xen-devel] [PATCH] xen-mapcache: Fix rlimit set size.
On Tue, 2 Aug 2011, Jan Beulich wrote:> >>> Anthony PERARD 08/01/11 9:27 PM >>> > >Previously, the address space soft limit was set mcache_max_size. So, > >before the mcache_max_size was reached by the mapcache, QEMU was killed > >for overuse of the virtual address space. > > > >This patch fix that by setting the soft limit to mcache_max_size + 80MB. > >I observed that QEMU use 75MB more than max_mcache_size after several > >empirical tests. > > Rather fragile going forward I would say.Yes, I agree. At the very least we should print a warning if rlimit_max is too low so that the admin can fix it.> I there any reason not to set > the limit to infinity?We shouldn''t assume that qemu is running as a privileged process, but if it is we should definitely do it. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2011-Aug-03 14:42 UTC
Re: [Xen-devel] Re: [PATCH] xen-mapcache: Fix rlimit set size.
>>> Stefano Stabellini 08/03/11 12:01 AM >>> >if (rlimit_as.rlim_max < MCACHE_MAX_SIZE) {This condition minimally needs to include what Anthony''s original version had.>if (qemu is priviledged) { >rlimit_as.rlim_max = RLIM_INFINITY; >rlimit_as.rlim_cur = RLIM_INFINITY; >} else { >print a warning >mapcache->max_mcache_size = rlimit_as.rlim_cur - NON_MCACHE_MEMORY_SIZE; >rlimit_as.rlim_cur = rlimit_as.rlim_max; >} >} else {Not printing a warning here means that there are still cases where the fuzzy upper bound may not be precise enough anymore (hence causing silent failure). I think the privileged case needs to be handled without any other surrounding condition, and the warning ought to be prinited in any case in the non-privileged case.>rlimit_as.rlim_cur = rlimit_as.rlim_max; >}Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Aug-03 20:57 UTC
Re: [Xen-devel] Re: [PATCH] xen-mapcache: Fix rlimit set size.
On Wed, 3 Aug 2011, Jan Beulich wrote:> >>> Stefano Stabellini 08/03/11 12:01 AM >>> > >if (rlimit_as.rlim_max < MCACHE_MAX_SIZE) { > > This condition minimally needs to include what Anthony''s original version had.right> >if (qemu is priviledged) { > >rlimit_as.rlim_max = RLIM_INFINITY; > >rlimit_as.rlim_cur = RLIM_INFINITY; > >} else { > >print a warning > >mapcache->max_mcache_size = rlimit_as.rlim_cur - NON_MCACHE_MEMORY_SIZE; > >rlimit_as.rlim_cur = rlimit_as.rlim_max; > >} > >} else { > > Not printing a warning here means that there are still cases where the > fuzzy upper bound may not be precise enough anymore (hence causing > silent failure). I think the privileged case needs to be handled without > any other surrounding condition, and the warning ought to be prinited > in any case in the non-privileged case.good idea> > >rlimit_as.rlim_cur = rlimit_as.rlim_max; > >}_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony PERARD
2011-Aug-04 13:33 UTC
[Xen-devel] Re: [Qemu-devel] [PATCH] xen-mapcache: Fix rlimit set size.
On Tue, Aug 2, 2011 at 15:50, Alexander Graf <agraf@suse.de> wrote:> On 08/01/2011 09:26 PM, Anthony PERARD wrote: >> >> Previously, the address space soft limit was set mcache_max_size. So, >> before the mcache_max_size was reached by the mapcache, QEMU was killed >> for overuse of the virtual address space. >> >> This patch fix that by setting the soft limit to mcache_max_size + 80MB. >> I observed that QEMU use 75MB more than max_mcache_size after several >> empirical tests. > > Not sure I understand. What are those 75MB?I did the calculation like that: - I make sure that the MapCache was full. - I did the difference between the value in mapcache->max_mcache_size and the size of the used address space by the QEMU process. -- Anthony PERARD _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Wei Huang
2011-Aug-04 18:44 UTC
[Xen-devel] hvm_domain_use_pirq() breaks device passthru on xen-4.1
Hi Stefano, The recent patch you put into xen-4.1-testing.hg broke device passthru on Windows. I bisected the changesets and it turned out to be 23092. The NIC can''t get the IP address correctly. Changset 23091 worked fine. Regarding your patch, do we need to differentiate pv-on-hvm guests from others? The Windows VM I used is unmodified. I think hvm_domain_use_pirq() returned an incorrect answer for such guests. Thanks, -Wei _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2011-Aug-08 07:36 UTC
Re: [Xen-devel] hvm_domain_use_pirq() breaks device passthru on xen-4.1
>>> On 04.08.11 at 20:44, Wei Huang <wei.huang2@amd.com> wrote: > Hi Stefano, > > The recent patch you put into xen-4.1-testing.hg broke device passthru > on Windows. I bisected the changesets and it turned out to be 23092. The > NIC can''t get the IP address correctly. Changset 23091 worked fine.With that I''m not really certain whether I should push the merge fix for my mis-merge of what became 23573:584c2e5e03d9 and the prior 23550:fb5f0febeddc. Thoughts? Jan> Regarding your patch, do we need to differentiate pv-on-hvm guests from > others? The Windows VM I used is unmodified. I think > hvm_domain_use_pirq() returned an incorrect answer for such guests. > > Thanks, > -Wei > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Aug-16 18:15 UTC
[Xen-devel] Re: hvm_domain_use_pirq() breaks device passthru on xen-4.1
On Thu, 4 Aug 2011, Wei Huang wrote:> Hi Stefano, > > The recent patch you put into xen-4.1-testing.hg broke device passthru > on Windows. I bisected the changesets and it turned out to be 23092. The > NIC can''t get the IP address correctly. Changset 23091 worked fine. > > Regarding your patch, do we need to differentiate pv-on-hvm guests from > others? The Windows VM I used is unmodified. I think > hvm_domain_use_pirq() returned an incorrect answer for such guests.The problem is that when qemu enables MSIs for a pt device on behalf of the guest physdev_map_pirq is called and pirq_emuirq is set to IRQ_PT.>From that point on hvm_domain_use_pirq is going to return trueincorrectly. Before 23092 we would also check for the event channel to be bound before returning true in hvm_domain_use_pirq, but that turns out to be a problem if a PV on HVM guest disables and unbinds the event channel. I think that reverting 23092 is the simplest thing to do here, then we need to find a different way to distinguish MSIs that have to be injected using the emulated path and the ones that have to be injected using the PV path. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel