Alexander Dahl
2019-Dec-18 20:24 UTC
[Pkg-xen-devel] system still crashes with bullseye and kernel v5.3
Hello there, meanwhile I'm running bullseye with kernel v5.3, but the problem persists and my Xen system is annoyingly unstable due to this bug. I attach some more logs from the last days and add the debian xen devel list in Cc. Maybe someone over there has an idea how to fix this. After all the log shows plenty of hints it could have something to do with Xen. Anyone care to help debug this? I have no idea where to start. Can kernel or xen generate coredumps one could analyze? Or is the log output the only thing? (If you look at the logs, the strange thing is the system does not crash and reboot immediately, but later after lots of errors with storage, but comes back fine after reboot.) Greets Alex -------------- next part -------------- A non-text attachment was scrubbed... Name: 2019-12-03_console.log Type: text/x-log Size: 3197 bytes Desc: not available URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20191218/9876a9ab/attachment-0005.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: 2019-12-04_console.log Type: text/x-log Size: 5554 bytes Desc: not available URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20191218/9876a9ab/attachment-0006.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: 2019-12-07_console.log Type: text/x-log Size: 3800 bytes Desc: not available URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20191218/9876a9ab/attachment-0007.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: 2019-12-15_console.log Type: text/x-log Size: 4766 bytes Desc: not available URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20191218/9876a9ab/attachment-0008.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: 2019-12-18_console.log Type: text/x-log Size: 4119 bytes Desc: not available URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20191218/9876a9ab/attachment-0009.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20191218/9876a9ab/attachment-0001.sig>
Hans van Kranenburg
2019-Dec-18 21:37 UTC
[Pkg-xen-devel] system still crashes with bullseye and kernel v5.3
Hi Alexander, On 12/18/19 9:24 PM, Alexander Dahl wrote:> > meanwhile I'm running bullseye with kernel v5.3, but the problem > persists and my Xen system is annoyingly unstable due to this bug. I > attach some more logs from the last days and add the debian xen devel > list in Cc. Maybe someone over there has an idea how to fix this. After > all the log shows plenty of hints it could have something to do with Xen.I think the xen parts you see in the stack trace listings are usual calls that show that a domU is asking dom0 via the hypervisor to do some disk read/writes or send data over the network (the 'upcall'). https://wiki.xen.org/wiki/Event_Channel_Internals So, after getting that request, the dom0 Linux kernel tries to execute it, which is e.g. the enqueue function to throw a network packet at the physical network interface. The first error we see is the "transmit queue 0 timed out". This looks like the Linux kernel is looking at the network port hardware, and expects it to accept the packet, deal with it and put it on the wire. When this does not happen, and the network port hardware seems frozen and timeouts, it's forcibly reset (I don't know if the thing is resetting itself because it crashed, or if the Linux kernel does something to reset it). "Reset adapter unexpectedly" gives me the feeling that the firmware inside the network card crashed and something inside there also reset it.> Anyone care to help debug this? I have no idea where to start. Can > kernel or xen generate coredumps one could analyze? Or is the log output > the only thing? > > (If you look at the logs, the strange thing is the system does not crash > and reboot immediately, but later after lots of errors with storage, but > comes back fine after reboot.)The ata errors (disk fails to process a command) happen after all of the above happens. Usually disk errors that look like this point at broken disk hardware or bugs in the firmware in the disk. However, if it consistently happens 6 to 7 seconds after the network card disaster, it might be a symptom of the former. The first thing I would recommend is disabling transmit segmentation offloading to the network card in dom0 (ethtool enp1s0 tso off) and see if it prevents the network card from choking on some kind of input. If not, play with more settings like transmit checksum offloading (tx off). If this does not help, we can start asking some Xen developers if they have an idea how we can help with debugging and what we should do. (I help maintaining the Xen packages in Debian, my knowledge about internals of it is mostly limited to all the been-there-done-thats during the years of using it as a user.) I expect the problem to be related to Linux and the hardware, and not specifically Xen. Knowing if the same happens when just booting Linux without Xen is valuable debugging info. However, I realize that it's likely a bit complicated to, in that case, try triggering the problem by generate the same workload that's now coming from the domUs. Curious to hear what happens, Thanks, Hans van Kranenburg