Objective: Reduce the amount of writing done to the managed node's filesystem. Since the managed node is run on an in-memory file system, excessive writes to such things as log files will eventually cause an out-of-memory condition. Our goal is to limit the amount of writes. Discussion: Initially, rotating logs every hour and deleting all old logs is the likely first pass. A usable feature would be for the WUI to push down (during the identify phase, maybe?) a threshold for when logs should be aggressively deleted. Then the admin can decide to hold onto logs until disk space hits some threshold and then start throwing away old logs. That way some history can be viewed if needed. In future, a more preferable solution would be to have the oVirt server collect logging information for all managed nodes centrally. However, this would need to use an encrypted pipe to send the data. Thoughts? -- Darryl L. Pierce, Sr. Software Engineer Red Hat, Inc. - http://www.redhat.com/ oVirt - Virtual Machine Management - http://www.ovirt.org/ "What do you care what other people think, Mr. Feynman?"
Darryl L. Pierce wrote:> Objective: > Reduce the amount of writing done to the managed node's filesystem. > Since the managed node is run on an in-memory file system, excessive > writes to such things as log files will eventually cause an > out-of-memory condition. Our goal is to limit the amount of writes. > > Discussion: > Initially, rotating logs every hour and deleting all old logs is the > likely first pass.Yes, this is the minimum way to ensure that we don't cause OOM on the managed nodes. It's what we should implement for now (and it should be easy enough just to modify some cron scripts in the %post section of the ovirt-host-creator).> > A usable feature would be for the WUI to push down (during the identify > phase, maybe?) a threshold for when logs should be aggressively deleted. > Then the admin can decide to hold onto logs until disk space hits some > threshold and then start throwing away old logs. That way some history > can be viewed if needed.Actually, I'm not sure this is really useful. Administrators really shouldn't need to be aware of the threshold, and ideally they wouldn't have to log into the individual nodes at all.> > In future, a more preferable solution would be to have the oVirt server > collect logging information for all managed nodes centrally. However, > this would need to use an encrypted pipe to send the data.This is the right way to do things. When the managed node comes up, it could pull some configuration data down from the WUI, and then it would know where to send the log files, and how to do so (encrypted, not-encrypted, etc. etc.). This way the administrator just has to go to one place (of his choosing) to look at the logs from all of his machines. That being said, I think the minimal "delete logs every hour" should prevent the worst disasters for now, and we can visit this again in the future. Chris Lalancette
On Wed, Jun 25, 2008 at 5:40 AM, Darryl L. Pierce <dpierce at redhat.com> wrote:> Objective: > Reduce the amount of writing done to the managed node's filesystem. Since > the managed node is run on an in-memory file system, excessive writes to > such things as log files will eventually cause an out-of-memory condition. > Our goal is to limit the amount of writes. > > Discussion: > Initially, rotating logs every hour and deleting all old logs is the likely > first pass. > > A usable feature would be for the WUI to push down (during the identify > phase, maybe?) a threshold for when logs should be aggressively deleted. > Then the admin can decide to hold onto logs until disk space hits some > threshold and then start throwing away old logs. That way some history can > be viewed if needed. > > In future, a more preferable solution would be to have the oVirt server > collect logging information for all managed nodes centrally. However, this > would need to use an encrypted pipe to send the data.Why not use a standard mechanism such as syslog(3)? Fedora already ships rsyslog since F8 and it supports encrypted connections via tls or ssl. Would adding rsyslog be too much bloat? -- Jeff Schroeder Don't drink and derive, alcohol and analysis don't mix. http://www.digitalprognosis.com