Xen devel - Oct 2006 - [PATCH - resend] addition of loglevel for printf in Xen HV

In the current implementation, the logging in the Hypervisor is 
basically at two levels.  The developer can print "always" or
"verbose".
  The always print is when the developer uses "printk/printf" in the 
hypervisor.  The "verbose" is when the developer uses DPRINTK, since
the
DPRINTK macro is only defined when the "verbose" flag is set.

Now the problem we are having, is that when we send out a distribution, 
we want enough information to be printed if one of our customers has a 
problem.  But we also don''t want to have so much printed that it
affects
the performance of the system.  Or worst yet, if the system has a serial 
console attached, one of the DomU''s could in essence cause a DoS attack
on the whole system by causing the DomU to do certain errors that causes 
a thousand prints to the serial console.  These prints are done with 
interrupts disabled, so the system suffers large latencies for each print.

Since it would be helpful to have prints for different environments, 
I''ve introduced with this patch a Linux like loglevel from 0-7. When
the
user sets the loglevel to 0, only emergency prints are preformed, and 
when 7 is selected, all prints are done.

Here''s the list of levels that I have added:

#define XEN_EMERG   "<0>"
#define XEN_ALERT   "<1>"
#define XEN_CRIT    "<2>"
#define XEN_ERR     "<3>"
#define XEN_WARNING "<4>"
#define XEN_GUEST   "<5>"
#define XEN_INFO    "<6>"
#define XEN_DEBUG   "<7>"

Basically, these correspond to their Linux equivalents. The one 
difference, is that the KERN_NOTICE was replaced by XEN_GUEST.  The 
XEN_GUEST is the level that a guest can "at will" cause this print. 
So
a distribution can set the print log level to 4 (XEN_WARNING) and all 
prints that are of errors and can be used for debugging are now 
available, without the prints that allow a Guest to cause a DoS.

I also added the keyhandlers ''0'' to ''7'' to
allow the user to change the
loglevel on a running system to the corresponding key. This is similar 
to the way Linux uses sysrq to change its log levels.

Two things that are not implemented with this patch but should be.

1. How to default the log level on build. Should the verbose now be a 
number, and not a config?  i.e. VERBOSE=4

2. How the user in Linux can change the log level (from dom0).  Should 
we add a hypercall or some other method to allow the user to change the 
logging?

Also, I made a pass though to assign all DPRINTK a log level.  I may not 
have done this correctly.  But a loglevel of 7 is now the same a 
VERBOSE=y  and a loglevel of 0 is almost the same of a VERBOSE=n. There 
was a couple of places where XEN_EMERG was used.

This patch really does need to go in.  Whether in its current form, or 
modified greatly.  The ability to give the developer a level of printing 
is a great asset.  That is why Linux has this ability.

-- Steve

Signed-off-by: Steven Rostedt <srostedt@redhat.com>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 20/10/06 20:23, "Steven Rostedt" <srostedt@redhat.com> wrote:
> This patch really does need to go in.  Whether in its current form, or
> modified greatly.  The ability to give the developer a level of printing
> is a great asset.  That is why Linux has this ability.
I agree. Here are some comments:

 * We don''t need 8 log levels. Noone knows the difference between
EMERG/CRIT/ALERT, or at least I really doubt that they can be used
consistently in a large code base. I suggest:
    XENLOG_ERR    -- Bad errors, likely fatal (to a guest or the host)
    XENLOG_WARN   -- Weird stuff happening, recoverable (maybe)
    XENLOG_INFO   -- Interesting info, not too noisy
    XENLOG_DEBUG  -- Noisy as you like

 * I think your ratelimit stuff could be usefully integrated. Instead of
having a single threshold, have two: one above which everything is printed,
one below which nothing is printed. In between is rate limited.

 * I think the guest/not-guest printing is orthogonal to the log-level
scale. Perhaps we should have another <> specifier (<G>?) that can
be used
alongside a log level specifier. This would give another two threshold
values (two for guest, two for non-guest) which would maybe be overkill but
would be reasonable if we gave a dom0 tool to control it. Actually this
could be extended to other subsystems (shadow code for example). Then we
would have a control per subsystem falling back to default thresholds if not
specified. That probably really is overkill!

 * Likely defaults:
      Non-guest prints ERR/WARN always, no INFO/DEBUG
      Guest prints ERR/WARN rate-limited, no INFO/DEBUG
      (Like Linux, we''d probably have slacker default controls until
initial
bootstrap has completed, so you get useful info out).

What do you think?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> 
> 
> On 20/10/06 20:23, "Steven Rostedt" <srostedt@redhat.com>
wrote:
> 
>> This patch really does need to go in.  Whether in its current form, or
>> modified greatly.  The ability to give the developer a level of
printing
>> is a great asset.  That is why Linux has this ability.
> 
> I agree. Here are some comments:
Keir, Thanks for the comments, they are very useful.
> 
>  * We don''t need 8 log levels. Noone knows the difference between
> EMERG/CRIT/ALERT, or at least I really doubt that they can be used
> consistently in a large code base. I suggest:
>     XENLOG_ERR    -- Bad errors, likely fatal (to a guest or the host)
>     XENLOG_WARN   -- Weird stuff happening, recoverable (maybe)
>     XENLOG_INFO   -- Interesting info, not too noisy
>     XENLOG_DEBUG  -- Noisy as you like
Perfectly agree. Four is fine, especially if we implement a <G> option 
for guest.
> 
>  * I think your ratelimit stuff could be usefully integrated. Instead of
> having a single threshold, have two: one above which everything is printed,
> one below which nothing is printed. In between is rate limited.
I''m not quite sure I understand this fully.

So what you are saying is instead of having the print_log_level have a 
printk_upper_log_level and a printk_lower_log_level where, as you say, 
everything below the printk_lower_log_level is printed, and the 
printk_upper_log_level where everyting above is not printed, and thus 
everything in between is rate limited? (remember 0 is ERR and 3 is DEBUG)

> 
>  * I think the guest/not-guest printing is orthogonal to the log-level
> scale. Perhaps we should have another <> specifier (<G>?) that
can be used
> alongside a log level specifier. This would give another two threshold
> values (two for guest, two for non-guest) which would maybe be overkill but
> would be reasonable if we gave a dom0 tool to control it. Actually this
> could be extended to other subsystems (shadow code for example). Then we
> would have a control per subsystem falling back to default thresholds if
not
> specified. That probably really is overkill!
> 
>  * Likely defaults:
>       Non-guest prints ERR/WARN always, no INFO/DEBUG
>       Guest prints ERR/WARN rate-limited, no INFO/DEBUG
>       (Like Linux, we''d probably have slacker default controls
until initial
> bootstrap has completed, so you get useful info out).
> 
> What do you think?
>
I have to admit I like the ideas you have thrown to me.

So let me put this is my own words/code so that you can see what my view 
of this is.

Have 4 thresholds.

printk_dom0_upper_threshold
printk_dom0_lower_threshold
printk_guest_upper_threshold
printk_guest_lower_threshold

Have 4 log levels:

XENLOG_ERR
XENLOG_WARN
XENLOG_INFO
XENLOG_DEBUG

Add a XENLOG_GUEST (defined as "<G>")

And for less typing add

XENLOG_G_ERR
XENLOG_G_WARN
XENLOG_G_INFO
XENLOG_G_DEBUG

which would just append the XENLOG_GUEST in front of the associated warning.

example;

    printk(XENLOG_G_ERR "fatal error in guest\n");

    printk(XENLOG_INFO "dom0 is running happily\n");


Then in printk itself, we would have

   upper_thresh = printk_dom0_upper_threshold;
   lower_thresh = printk_dom0_lower_threshold;
   level = some_default;

   if (strncmp("<G>",fmt,3)==0) {
	upper_thresh = printk_guest_upper_threshold;
	lower_thresh = printk_guest_lower_threshold;
	level = some_guest_default;
	fmt += 3;
   }
   if (strncmp("<[0-3]>",fmt,3)==0) {  /* using regexpr and not
C to make
                                          this readable */
	level =~ s/<(0-3)>/$1/;  /* perl format :-) */
   }

   if (level > upper_thresh)
	return; /* do nothing */

   if (level >= lower_thresh)
	if (rate_limit())
		return;

   print as normal.


So anything in between the upper and lower thresholds inclusive would be 
rate limited.  Also I''m not sure of the best way dom0 can communicate 
threshold changes to the hypervisor.  As well as if there should be a 
config to set the defaults.

So what are your thoughts?

-- Steve

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26/10/06 2:30 pm, "Steven Rostedt" <srostedt@redhat.com>
wrote:
> So anything in between the upper and lower thresholds inclusive would be
> rate limited.  Also I''m not sure of the best way dom0 can
communicate
> threshold changes to the hypervisor.  As well as if there should be a
> config to set the defaults.
I think yet-another-little-tool in dom0 is the right way to go. We can add a
sysctl to get/set the parameters. It can pretty-print the current parameters
and also provide a way of changing them. I don''t think there''s
any benefit
to integration with xm -- it''s not like this needs to be integrated
with
xend or anything like that.

Yes, we should also have a way to set them as a boot parameter. I suggest we
should only enact those explicit parameters when dom0 starts to boot though
(no need to stop Xen being noisy in initial boot). We could have an extra
parameter (e.g., ''quiet'') to override that if someone really
cares.

It''d be nice to have debug-key support for changing it too, but
actually
that''s probably not so important if we have the dom0 tool.

Everything you said I agree with, so please do go hack it up. :-)

Another thought... By pushing the rate-limiting into printk() itself, we
lose the ability to print all-or-nothing for related blocks of printk (e.g.,
register dumps). We might want a ''same as previous line''
specifier, which
would print/not-print same as previous output line?

 -- Keir
> So what are your thoughts?


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> 
> I think yet-another-little-tool in dom0 is the right way to go. We can add
a
> sysctl to get/set the parameters. It can pretty-print the current
parameters
> and also provide a way of changing them. I don''t think
there''s any benefit
> to integration with xm -- it''s not like this needs to be
integrated with
> xend or anything like that.
That should be easy enough. What''s the best way to communicate between 
dom0 and the HV?  Add a hypercall for this?
> 
> Yes, we should also have a way to set them as a boot parameter. I suggest
we
> should only enact those explicit parameters when dom0 starts to boot though
> (no need to stop Xen being noisy in initial boot). We could have an extra
> parameter (e.g., ''quiet'') to override that if someone
really cares.
Yeah, but this can be handled after the initial work is done.
> 
> It''d be nice to have debug-key support for changing it too, but
actually
> that''s probably not so important if we have the dom0 tool.
My original patch included a debug key for log level.  I''m sure it
won''t
be too hard to add something to change the thresholds too.  (for those 
that only communicate to the machine via a serial console).
> 
> Everything you said I agree with, so please do go hack it up. :-)
Thanks!  Will start hacking!
> 
> Another thought... By pushing the rate-limiting into printk() itself, we
> lose the ability to print all-or-nothing for related blocks of printk
(e.g.,
> register dumps). We might want a ''same as previous line''
specifier, which
> would print/not-print same as previous output line?
> 
Hmm, I wonder if we should have a XENLOG_CRASH level which is above 
XENLOG_ERR.  This will be when the system is actually taking a dump, and 
we want to show everything.  Or just simply force the thresholds to max 
just before showing the crash print outputs.  Or add a variable to know 
we are in the process of crashing, and have printk ignore the print rate 
limit if that''s the case, similar to the oops_in_progress of Linux.

-- Steve



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26/10/06 4:16 pm, "Steven Rostedt" <srostedt@redhat.com>
wrote:
>> Another thought... By pushing the rate-limiting into printk() itself,
we
>> lose the ability to print all-or-nothing for related blocks of printk
(e.g.,
>> register dumps). We might want a ''same as previous
line'' specifier, which
>> would print/not-print same as previous output line?
>> 
> 
> Hmm, I wonder if we should have a XENLOG_CRASH level which is above
> XENLOG_ERR.  This will be when the system is actually taking a dump, and
> we want to show everything.  Or just simply force the thresholds to max
> just before showing the crash print outputs.  Or add a variable to know
> we are in the process of crashing, and have printk ignore the print rate
> limit if that''s the case, similar to the oops_in_progress of
Linux.
We dump registers for guest crashes too, and it''d be nice to have those
disable-able. I think going for a first cut without worrying about this is
fine. We can finesse things later. It''ll be a mega-big patch as it is!

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel