George Styles
2005-May-17 13:18 UTC
[Xen-users] Crashing mythtv, mysqld and NAN errors all over the place in a domain
Hi, Im running Suse 9.3 pro, with the supplied kernel (Linux version 2.6.11.4-20a-xen (geeko@buildhost) (gcc version 3.3.5 20050117 (prerelease) (SUSE Linux)) #1 Wed Mar 23 21:52:37 UTC 2005 ) Im also using the version of Xen supplied with Suse 9.3 Ive got 4 or 5 domains, all running perfectly, including a samba one with heavy network traffic. All works perfectly... Im trying to build a mythtv domain - its almost working - it can see thje DVB cards, and can record and so on, but occasionally (usually when MythTV is hitting the mysql database on the same domain) im getting some wierd crashes in MythTV. When it crashes, the process (mythbackend) says ''killed'' - 1. is is possible the kernel is killing the process because its misbehaving? dmesg doesnt say anything like this 2. The domain is allocated 256meg of ram - just before it crashes, it seems to have around 4 megs of ram left (using top), and loads of swap space (but its used NO swapspace which seems suspect). If I decrease the ram to 32meg, it does use the swapspace, but still crashes. 3. When its gone wrong, there seems to be a problem with other services as well - sshd wont let me connect (but recovers after 5 mins or so) and mysqld also crashes (with errors in the log that dont tell me much - see below). Ive tried building and installing the non-suse kernel - same thing. 4. The way I built the system is as follows (is this correct)? a) installed Suse 9.3 onto a 3 gig root partition, with Xen installed b) Disabled tls (mv tls tls.disabled) c) dd''ed the partition into a disk file d) Booted Xen pointing it as that disk file Like I said, I created 4 other domains that run fine. 5. Are there any Xen logs or debug stuff I can look at for clues? dmesg doesnt reveal anything I can see is related. 6. It crashes even when I dont have the DVB cards ''passed through'' to the domain, so its not a driver issue. 7. The other thing thats happening in this domain, is that im getting NaN (not a number) reported all over the place (eg, in top it reports NaN% processor usage) and mythtv keeps building queries that say to insert NaN into a table. This seems associated with times more than anything. 8. It looks to me a lot like the domain is running out of resources of some kind - perhaps memory and its refusing to use the swap space for some reason? 9. I am getting errors about the hardware clock when the domain boots, but I expect this as I have not passed through Help! 10. Also, sleep has bombed a couple of times, with a ASSERT error (Assert(time > 0)) - could this be related to the ''time is going backwards'' issue? 11. Would I benefit from upgrading Xen to the latest testing version? is this a known issue? thanks George The MySQL errors are: 050516 22:43:13 mysqld restarted 050516 22:43:14 InnoDB: Started; log sequence number 0 43634 /usr/sbin/mysqld: ready for connections. Version: ''4.1.10a'' socket: ''/var/lib/mysql/mysql.sock'' port: 3306 SUSE MySQL RPM mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=16777216 read_buffer_size=258048 max_used_connections=1 max_connections=100 threads_connected=1 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 92783 K bytes of memory Hope that''s ok; if not, decrease some variables in the equation. thd=0x89f3ab8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x42ee4c38, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x8168e0b 0x4005d105 0x81b01e8 0x81b0a09 0x81b3940 0x81b7e7f 0x8183306 0x8184fd1 0x8186be4 0x818753b 0x81880a7 0x40056e31 0x4028d6fa New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x8a17428 = SELECT data FROM settings WHERE value "RecordChanged" thd->thread_id=1 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. Number of processes running now: 1 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
George Styles
2005-May-17 14:29 UTC
[Xen-users] Crashing mythtv, mysqld and NAN errors all over the place in a domain
Hi, Im running Suse 9.3 pro, with the supplied kernel (Linux version 2.6.11.4-20a-xen (geeko@buildhost) (gcc version 3.3.5 20050117 (prerelease) (SUSE Linux)) #1 Wed Mar 23 21:52:37 UTC 2005 ) Im also using the version of Xen supplied with Suse 9.3 Ive got 4 or 5 domains, all running perfectly, including a samba one with heavy network traffic. All works perfectly... Im trying to build a mythtv domain - its almost working - it can see thje DVB cards, and can record and so on, but occasionally (usually when MythTV is hitting the mysql database on the same domain) im getting some wierd crashes in MythTV. When it crashes, the process (mythbackend) says ''killed'' - 1. is is possible the kernel is killing the process because its misbehaving? dmesg doesnt say anything like this 2. The domain is allocated 256meg of ram - just before it crashes, it seems to have around 4 megs of ram left (using top), and loads of swap space (but its used NO swapspace which seems suspect). If I decrease the ram to 32meg, it does use the swapspace, but still crashes. 3. When its gone wrong, there seems to be a problem with other services as well - sshd wont let me connect (but recovers after 5 mins or so) and mysqld also crashes (with errors in the log that dont tell me much - see below). Ive tried building and installing the non-suse kernel - same thing. 4. The way I built the system is as follows (is this correct)? a) installed Suse 9.3 onto a 3 gig root partition, with Xen installed b) Disabled tls (mv tls tls.disabled) c) dd''ed the partition into a disk file d) Booted Xen pointing it as that disk file Like I said, I created 4 other domains that run fine. 5. Are there any Xen logs or debug stuff I can look at for clues? dmesg doesnt reveal anything I can see is related. 6. It crashes even when I dont have the DVB cards ''passed through'' to the domain, so its not a driver issue. 7. The other thing thats happening in this domain, is that im getting NaN (not a number) reported all over the place (eg, in top it reports NaN% processor usage) and mythtv keeps building queries that say to insert NaN into a table. This seems associated with times more than anything. 8. It looks to me a lot like the domain is running out of resources of some kind - perhaps memory and its refusing to use the swap space for some reason? 9. I am getting errors about the hardware clock when the domain boots, but I expect this as I have not passed through Help! 10. Also, sleep has bombed a couple of times, with a ASSERT error (Assert(time > 0)) - could this be related to the ''time is going backwards'' issue? 11. Would I benefit from upgrading Xen to the latest testing version? is this a known issue? thanks George The MySQL errors are: 050516 22:43:13 mysqld restarted 050516 22:43:14 InnoDB: Started; log sequence number 0 43634 /usr/sbin/mysqld: ready for connections. Version: ''4.1.10a'' socket: ''/var/lib/mysql/mysql.sock'' port: 3306 SUSE MySQL RPM mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=16777216 read_buffer_size=258048 max_used_connections=1 max_connections=100 threads_connected=1 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 92783 K bytes of memory Hope that''s ok; if not, decrease some variables in the equation. thd=0x89f3ab8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x42ee4c38, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x8168e0b 0x4005d105 0x81b01e8 0x81b0a09 0x81b3940 0x81b7e7f 0x8183306 0x8184fd1 0x8186be4 0x818753b 0x81880a7 0x40056e31 0x4028d6fa New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x8a17428 = SELECT data FROM settings WHERE value "RecordChanged" thd->thread_id=1 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. Number of processes running now: 1 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Williamson
2005-May-18 14:05 UTC
Re: [Xen-users] Crashing mythtv, mysqld and NAN errors all over the place in a domain
> Im trying to build a mythtv domain - its almost working - it can see > thje DVB cards, and can record and so on, but occasionally (usually > when MythTV is hitting the mysql database on the same domain) im > getting some wierd crashes in MythTV. > When it crashes, the process (mythbackend) says ''killed'' -Weird. So you''re assigning the DVB cards to this domain... Have you remembered to hide the card from dom0? Does Xen confirm that this hiding has occurred when it boots up? Can you post menu.lst, the domain''s config, the output of lspci (under vanilla linux) and the output of xm dmesg?> 1. is is possible the kernel is killing the process because its > misbehaving? dmesg doesnt say anything like thisIt could be... But a kernel error could be occurring whilst this process is running.> 2. The domain is allocated 256meg of ram - just before it crashes, it > seems to have around 4 megs of ram left (using top), and loads of swap > space (but its used NO swapspace which seems suspect). If I decrease > the ram to 32meg, it does use the swapspace, but still crashes.It''s very strange for Linux not to use any swap space! If you don''t try assigning a device to the domain and use it as a fully unprivileged domain, does it behave more normally? I can''t see why it would make a difference though...> 3. When its gone wrong, there seems to be a problem with other > services as well - sshd wont let me connect (but recovers after 5 mins > or so) and mysqld also crashes (with errors in the log that dont tell > me much - see below).Ouch!> 6. It crashes even when I dont have the DVB cards ''passed through'' to > the domain, so its not a driver issue.Ah OK, that''s very weird. Is it just MythTV that breaks this? What happens if you try running it in another domain?> 7. The other thing thats happening in this domain, is that im getting > NaN (not a number) reported all over the place (eg, in top it reports > NaN% processor usage) and mythtv keeps building queries that say to > insert NaN into a table. This seems associated with times more than > anything.That''s probably a clue as to the source of the error but it''s not obvious to me what it could be.> 8. It looks to me a lot like the domain is running out of resources of > some kind - perhaps memory and its refusing to use the swap space for > some reason?Are you sure swap space is configured OK?> 9. I am getting errors about the hardware clock when the domain boots, > but I expect this as I have not passed throughYeah, that''s normal. Domains can''t program the hardware clock directly, nor do they need to.> 10. Also, sleep has bombed a couple of times, with a ASSERT error > (Assert(time > 0)) - could this be related to the ''time is going > backwards'' issue?"Time going backwards" is a Xen error and probably not exactly the problem here. Nonetheless, something weird does seem to be happening to time on your system.> 11. Would I benefit from upgrading Xen to the latest testing version? > is this a known issue?You could try it. There are a few bug fixes in -testing but I don''t know of anything in your version that''d cause this to happen... Cheers, Mark> > thanks > > George > > The MySQL errors are: > > 050516 22:43:13 mysqld restarted > 050516 22:43:14 InnoDB: Started; log sequence number 0 43634 > /usr/sbin/mysqld: ready for connections. > Version: ''4.1.10a'' socket: ''/var/lib/mysql/mysql.sock'' port: 3306 > SUSE MySQL RPM > mysqld got signal 11; > This could be because you hit a bug. It is also possible that this binary > or one of the libraries it was linked against is corrupt, improperly built, > or misconfigured. This error can also be caused by malfunctioning hardware. > We will try our best to scrape up some info that will hopefully help > diagnose the problem, but since we have already crashed, something is > definitely wrong and this may fail. > > key_buffer_size=16777216 > read_buffer_size=258048 > max_used_connections=1 > max_connections=100 > threads_connected=1 > It is possible that mysqld could use up to > key_buffer_size + (read_buffer_size + > sort_buffer_size)*max_connections = 92783 K > bytes of memory > Hope that''s ok; if not, decrease some variables in the equation. > > thd=0x89f3ab8 > Attempting backtrace. You can use the following information to find out > where mysqld died. If you see no messages after this, something went > terribly wrong... > Cannot determine thread, fp=0x42ee4c38, backtrace may not be correct. > Stack range sanity check OK, backtrace follows: > 0x8168e0b > 0x4005d105 > 0x81b01e8 > 0x81b0a09 > 0x81b3940 > 0x81b7e7f > 0x8183306 > 0x8184fd1 > 0x8186be4 > 0x818753b > 0x81880a7 > 0x40056e31 > 0x4028d6fa > New value of fp=(nil) failed sanity check, terminating stack trace! > Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html > and follow instructions on how to resolve the stack trace. Resolved > stack trace is much more helpful in diagnosing the problem, so please do > resolve it > Trying to get some variables. > Some pointers may be invalid and cause the dump to abort... > thd->query at 0x8a17428 = SELECT data FROM settings WHERE value > "RecordChanged" > thd->thread_id=1 > The manual page at http://www.mysql.com/doc/en/Crashing.html contains > information that should help you find out what is causing the crash. > > Number of processes running now: 1 > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users