Greetings list, Yesterday the following scenario took place. In short, I had a HVM host with a poorly configured firewall. It logged way more than it should, and it had been doing that for days. IPTables entries with -j LOG show up in the console. I told Konrad about this but I said it was a PV host while in fact it is concerning a HVM host. Anyway, when I saw that this was the case I corrected iptables and then I removed those faulty log entries from the logfiles in /var/log. Finally I issued /etc/init.d/rsyslogd reload. Suddenly the server stopped responding at all, not even arp requests were answered. I opened up a console via xl (none was open before this) and I was greeted with a crazy amount of log entries - it took about five or six minutes before the whole buffer was flushed to my screen. Then the server responded again. Today I viewed syslog''s messages in detail and found this and a lot of similar entries: http://pastebin.com/Ga7aE7hb So the questions that I have now are - Is this a bug? How is console history handled and where is the console''s unread history stored? Somewhere in the hypervisor or in dom0 or domU memory space? Is there a limit to how much info can be stored? The behaviour I encountered i.e. a system lock seems to suggest that there was some kind of buffer overflow. And this can be achieved by something as simple as ''iptables -I INPUT -j LOG''. Perhaps it is a wise idea to consider xl console -c to clear the console''s history, then at least I could exit the console and clear unsent messages... I am not receiving xen-devel messages, so please CC me on replies. Thank you. -- Stay in touch, Mark van Dijk. ,----------------------------------- -------------------------------'' Wed Aug 08 15:29 UTC 2012 Today is Setting Orange, the 1st day of Bureaucracy in the YOLD 3178
On 08/08/2012 16:43, "Mark van Dijk" <lists+xen@internecto.net> wrote:> Greetings list, > > Yesterday the following scenario took place. In short, I had a HVM host > with a poorly configured firewall. It logged way more than it should, > and it had been doing that for days. IPTables entries with -j LOG > show up in the console. I told Konrad about this but I said it was a PV > host while in fact it is concerning a HVM host. > > Anyway, when I saw that this was the case I corrected iptables and then > I removed those faulty log entries from the logfiles in /var/log. > Finally I issued /etc/init.d/rsyslogd reload. > > Suddenly the server stopped responding at all, not even arp requests > were answered. I opened up a console via xl (none was open before > this) and I was greeted with a crazy amount of log entries - it took > about five or six minutes before the whole buffer was flushed to my > screen. Then the server responded again. > > Today I viewed syslog''s messages in detail and found this and a lot of > similar entries: > > http://pastebin.com/Ga7aE7hb > > So the questions that I have now are - Is this a bug? How is console > history handled and where is the console''s unread history stored? > Somewhere in the hypervisor or in dom0 or domU memory space? Is there a > limit to how much info can be stored? > > The behaviour I encountered i.e. a system lock seems to suggest that > there was some kind of buffer overflow. And this can be achieved by > something as simple as ''iptables -I INPUT -j LOG''. Perhaps it is a wise > idea to consider xl console -c to clear the console''s history, then at > least I could exit the console and clear unsent messages...Is your VM logging to its virtual serial line? You could just not do that... I think this is due to the guest''s qemu-dm process stuffing its transmitted serial data into a pty, to be consumed by ''xl console'', and if that doesn''t happen the buffers fill up, serial processing stops, and we back up all the way to the guest. -- Keir> I am not receiving xen-devel messages, so please CC me on replies. > Thank you.
Greetings,> > The behaviour I encountered i.e. a system lock seems to suggest that > > there was some kind of buffer overflow. And this can be achieved by > > something as simple as ''iptables -I INPUT -j LOG''. Perhaps it is a > > wise idea to consider xl console -c to clear the console''s history, > > then at least I could exit the console and clear unsent messages... > > Is your VM logging to its virtual serial line? You could just not do > that...Correct, it is logging to the virtual serial line and you''re right, I could just not do that. But isn''t that just a workaround? I just think that if you emulate something like a serial line, it should be emulated properly. Why don''t we just discard the data that is sent to a detached serial line? That''s what bare metal servers do too, right?> I think this is due to the guest''s qemu-dm process stuffing its > transmitted serial data into a pty, to be consumed by ''xl console'', > and if that doesn''t happen the buffers fill up, serial processing > stops, and we back up all the way to the guest.Right. So the issue is only with the guest, not with the hypervisor. That''s a relief. :) -- Stay in touch, Mark van Dijk. ,-------------------------------- ----------------------------'' Fri Aug 10 09:11 UTC 2012 Today is Boomtime, the 3rd day of Bureaucracy in the YOLD 3178