Hi, I tried asking this question on xen-users, but didn''t get a response so I''m now asking on xen-devel instead.... Is netconsole supposed to work under pvops? When I try to enable it I get the messages below. I''m wondering if netconsole just doesn''t work at all with pvops or whether it''s dependent on the NIC used by the dom0. Initialising Xen virtual ethernet driver. blkfront: xvda1: barriers enabled netconsole: local port 11111 netconsole: local IP 10.20.40.2 netconsole: interface eth1 netconsole: remote port 9514 netconsole: remote IP 10.20.50.2 netconsole: remote ethernet address 00:16:3e:00:50:02 netconsole: eth1 doesn''t support polling, aborting. netconsole: cleaning up Cheers, Brad _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Feb 05, 2010 at 11:15:07AM +1100, Brad Plant wrote:> Hi, > > I tried asking this question on xen-users, but didn''t get a response so I''m now asking on xen-devel instead.... > > Is netconsole supposed to work under pvops? When I try to enable it I get the messages below. I''m wondering if netconsole just doesn''t work at all with pvops or whether it''s dependent on the NIC used by the dom0.Hmmm, I _think_ I had it working at some point, but this was with dom0, not with domUs, which is what you are looking to be doing?> Initialising Xen virtual ethernet driver._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 5 Feb 2010 11:34:30 -0500 Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> > Is netconsole supposed to work under pvops? When I try to enable it I get the messages below. I''m wondering if netconsole just doesn''t work at all with pvops or whether it''s dependent on the NIC used by the dom0. > > Hmmm, I _think_ I had it working at some point, but this was with dom0, > not with domUs, which is what you are looking to be doing?That''s correct - I''m trying to use netconsole on a domU. Cheers, Brad _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 02/04/2010 04:15 PM, Brad Plant wrote:> Initialising Xen virtual ethernet driver. > blkfront: xvda1: barriers enabled > netconsole: local port 11111 > netconsole: local IP 10.20.40.2 > netconsole: interface eth1 > netconsole: remote port 9514 > netconsole: remote IP 10.20.50.2 > netconsole: remote ethernet address 00:16:3e:00:50:02 > netconsole: eth1 doesn''t support polling, aborting. > netconsole: cleaning up >The xen-netfront device doesn''t support the polling interface that netconsole relies on. It probably wouldn''t be too hard to add that capability if you want to. It''s independent of the backend and/or the actual underlying hardware device. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 12 Feb 2010 15:09:12 -0800 Jeremy Fitzhardinge <jeremy@goop.org> wrote:> On 02/04/2010 04:15 PM, Brad Plant wrote: > > Initialising Xen virtual ethernet driver. > > blkfront: xvda1: barriers enabled > > netconsole: local port 11111 > > netconsole: local IP 10.20.40.2 > > netconsole: interface eth1 > > netconsole: remote port 9514 > > netconsole: remote IP 10.20.50.2 > > netconsole: remote ethernet address 00:16:3e:00:50:02 > > netconsole: eth1 doesn''t support polling, aborting. > > netconsole: cleaning up > > > > The xen-netfront device doesn''t support the polling interface that > netconsole relies on. It probably wouldn''t be too hard to add that > capability if you want to. It''s independent of the backend and/or the > actual underlying hardware device.I like the sound of that. I''ll give it a crack :-) Cheers, Brad _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 12 Feb 2010 15:09:12 -0800 Jeremy Fitzhardinge <jeremy@goop.org> wrote:> On 02/04/2010 04:15 PM, Brad Plant wrote: > > Initialising Xen virtual ethernet driver. > > blkfront: xvda1: barriers enabled > > netconsole: local port 11111 > > netconsole: local IP 10.20.40.2 > > netconsole: interface eth1 > > netconsole: remote port 9514 > > netconsole: remote IP 10.20.50.2 > > netconsole: remote ethernet address 00:16:3e:00:50:02 > > netconsole: eth1 doesn''t support polling, aborting. > > netconsole: cleaning up > > > > The xen-netfront device doesn''t support the polling interface that > netconsole relies on. It probably wouldn''t be too hard to add that > capability if you want to. It''s independent of the backend and/or the > actual underlying hardware device.Ok, I''ve come up with something that "works". Patch against 2.6.32.7 is attached. While it works, I''ve got no idea if what I''ve done is the "right way". I used virtio_net.c as a guide. I''ve got some issues that I''m stuck on and would like some guidance: 1. When the first message is sent over the net console, the WARN_ONCE() at net/core/netpoll.c:327 triggers. I''m not sure what to do about this. 2. When either netconsole is setup or the first message is sent, messages already logged to the main console get logged again. Maybe this will go away when the above is fixed? 3. Initially netconsole would not initialise because the interface didn''t yet have a mac address as setup_netfront hadn''t yet been called. I changed the module_init() in netconsole.c to late_initcall() if !CONFIG_MODULES && CONFIG_XEN. Is this allowed and sane? Cheers, Brad _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 02/13/2010 06:28 PM, Brad Plant wrote:> On Fri, 12 Feb 2010 15:09:12 -0800 > Jeremy Fitzhardinge<jeremy@goop.org> wrote: > > >> On 02/04/2010 04:15 PM, Brad Plant wrote: >> >>> Initialising Xen virtual ethernet driver. >>> blkfront: xvda1: barriers enabled >>> netconsole: local port 11111 >>> netconsole: local IP 10.20.40.2 >>> netconsole: interface eth1 >>> netconsole: remote port 9514 >>> netconsole: remote IP 10.20.50.2 >>> netconsole: remote ethernet address 00:16:3e:00:50:02 >>> netconsole: eth1 doesn''t support polling, aborting. >>> netconsole: cleaning up >>> >>> >> The xen-netfront device doesn''t support the polling interface that >> netconsole relies on. It probably wouldn''t be too hard to add that >> capability if you want to. It''s independent of the backend and/or the >> actual underlying hardware device. >> > Ok, I''ve come up with something that "works". Patch against 2.6.32.7 is attached. While it works, I''ve got no idea if what I''ve done is the "right way". I used virtio_net.c as a guide. > > I''ve got some issues that I''m stuck on and would like some guidance: > > 1. When the first message is sent over the net console, the WARN_ONCE() at net/core/netpoll.c:327 triggers. I''m not sure what to do about this. > >What''s the backtrace in the warning? I had a quick look at e1000''s and it calls the interrupt handler; the netfront equivalent would be to call xennet_interrupt() with interrupts disabled. I''m not sure virtio_net is a good model because it doesn''t seem to have any interrupt-related stuff in it; I guess that happens elsewhere in virtio.> 2. When either netconsole is setup or the first message is sent, messages already logged to the main console get logged again. Maybe this will go away when the above is fixed? >Probably not. It sounds like a feature.> 3. Initially netconsole would not initialise because the interface didn''t yet have a mac address as setup_netfront hadn''t yet been called. I changed the module_init() in netconsole.c to late_initcall() if !CONFIG_MODULES&& CONFIG_XEN. Is this allowed and sane? >Probably not. In practice every kernel will have modules and Xen enabled, and it will change netconsole for all users. I guess we probably need to find some way to init the MAC address earlier, or trigger netconsole init once the mac address is known.> Cheers, > > Brad >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sat, 13 Feb 2010 19:56:51 -0800 Jeremy Fitzhardinge <jeremy@goop.org> wrote:> > 1. When the first message is sent over the net console, the WARN_ONCE() at net/core/netpoll.c:327 triggers. I''m not sure what to do about this. > > What''s the backtrace in the warning?------------[ cut here ]------------ WARNING: at net/core/netpoll.c:329 netpoll_send_skb+0x145/0x1db() netpoll_send_skb(): eth1 enabled interrupts in poll (xennet_start_xmit+0x0/0x6bb) Pid: 1, comm: swapper Not tainted 2.6.32.7 #36 Call Trace: [<ffffffff81039100>] warn_slowpath_common+0x72/0x8a [<ffffffff81039165>] warn_slowpath_fmt+0x3c/0x3e [<ffffffff8100dd7f>] ? xen_restore_fl_direct_end+0x0/0x1 [<ffffffff8124e2a0>] ? xennet_start_xmit+0x0/0x6bb [<ffffffff81294e48>] netpoll_send_skb+0x145/0x1db [<ffffffff813604d4>] ? _spin_unlock_irqrestore+0x19/0x1c [<ffffffff812950dc>] netpoll_send_udp+0x1fe/0x20e [<ffffffff8124004d>] ? firmware_loading_store+0x120/0x1b5 [<ffffffff8124f9d6>] write_msg+0x89/0xca [<ffffffff8124004d>] ? firmware_loading_store+0x120/0x1b5 [<ffffffff81039281>] __call_console_drivers+0x67/0x79 [<ffffffff810392ec>] _call_console_drivers+0x59/0x5d [<ffffffff81039613>] release_console_sem+0x120/0x1c7 [<ffffffff81039e73>] register_console+0x22e/0x2ae [<ffffffff81503380>] init_netconsole+0x15d/0x1c8 [<ffffffff81049033>] ? schedule_delayed_work+0x16/0x1b [<ffffffff81503223>] ? init_netconsole+0x0/0x1c8 [<ffffffff8100a05f>] do_one_initcall+0x59/0x154 [<ffffffff814e56a9>] kernel_init+0x152/0x1a8 [<ffffffff810112aa>] child_rip+0xa/0x20 [<ffffffff810104a1>] ? int_ret_from_sys_call+0x7/0x1b [<ffffffff81010c5d>] ? retint_restore_args+0x5/0x6 [<ffffffff810112a0>] ? child_rip+0x0/0x20 ---[ end trace 0f898deb8e1e2914 ]---> I had a quick look at e1000''s and it calls the interrupt handler; the > netfront equivalent would be to call xennet_interrupt() with interrupts > disabled. I''m not sure virtio_net is a good model because it doesn''t > seem to have any interrupt-related stuff in it; I guess that happens > elsewhere in virtio.Actually, I tried something like this first because I first looked at pcnet32.c and e100.c: disable_irq(dev->irq); xennet_interrupt(dev->irq, dev); enable_irq(dev->irq); The same WARN_ONCE() was triggered.> > 2. When either netconsole is setup or the first message is sent, messages already logged to the main console get logged again. Maybe this will go away when the above is fixed? > > Probably not. It sounds like a feature.Ok. This one was the least of my worries anyway.> > 3. Initially netconsole would not initialise because the interface didn''t yet have a mac address as setup_netfront hadn''t yet been called. I changed the module_init() in netconsole.c to late_initcall() if !CONFIG_MODULES&& CONFIG_XEN. Is this allowed and sane? > > Probably not. In practice every kernel will have modules and Xen > enabled, and it will change netconsole for all users. I guess we > probably need to find some way to init the MAC address earlier, or > trigger netconsole init once the mac address is known.I figured it was dodgy. How would you init the MAC address earlier though? I''ve observed the xen devices being initialised at slightly different times with respect to other devices which suggests to me that they (and possibly all devices) are being initialised in separate threads. It would mean setting the MAC address before device initialisation yes? Not sure how correct this is either, but another solution might be to set the MAC address to say 00:00:00:00:00:01 when the netdev structure is first created. This lets netconsole initialise since the MAC address isn''t all zeros or all 0xFF. By the time any net console logs are transmitted, setup_netfront will have already been called and the correct MAC address set so no packets will actually be sent with the above MAC address. I have attached a patch that does this. Cheers, Brad _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 02/13/2010 08:56 PM, Brad Plant wrote:> Actually, I tried something like this first because I first looked at pcnet32.c and e100.c: > > disable_irq(dev->irq); > xennet_interrupt(dev->irq, dev); > enable_irq(dev->irq); > > The same WARN_ONCE() was triggered. >Given that netconsole works with e1000 (I presume; haven''t tested it lately), I guess it needs a close look to see what differs from the netfront case.>>> 3. Initially netconsole would not initialise because the interface didn''t yet have a mac address as setup_netfront hadn''t yet been called. I changed the module_init() in netconsole.c to late_initcall() if !CONFIG_MODULES&& CONFIG_XEN. Is this allowed and sane? >>> >> Probably not. In practice every kernel will have modules and Xen >> enabled, and it will change netconsole for all users. I guess we >> probably need to find some way to init the MAC address earlier, or >> trigger netconsole init once the mac address is known. >> > I figured it was dodgy. How would you init the MAC address earlier though? I''ve observed the xen devices being initialised at slightly different times with respect to other devices which suggests to me that they (and possibly all devices) are being initialised in separate threads. It would mean setting the MAC address before device initialisation yes? > > Not sure how correct this is either, but another solution might be to set the MAC address to say 00:00:00:00:00:01 when the netdev structure is first created. This lets netconsole initialise since the MAC address isn''t all zeros or all 0xFF. By the time any net console logs are transmitted, setup_netfront will have already been called and the correct MAC address set so no packets will actually be sent with the above MAC address. I have attached a patch that does this. >That seems a bit sketchy. netfront is like a lot of network devices; it needs to query some external source to find its own mac address. Mostly this means querying some eeprom over i2c or something, but in our case it means talking to xenstore to find the configured mac address. I think the problem here is that this happens asynchronously, rather than synchronously in the net device''s init routine. One fix might be to synchronously wait for the mac address to be set before returning from the init function. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel