Wayne Sierke
2008-Jan-08 09:26 UTC
6.3-PRERELEASE desktop system periodically freezes momentarily
FreeBSD 6.3-PRERELEASE #5: Sat Dec 29 19:25:43 CST 2007 i386 I've noticed that this system is freezing periodically, for anywhere from around a fraction of a second up to perhaps 1.5 seconds, and occurring on average about twice a minute, but with no particular pattern - i.e. it could happen twice in fairly quick succession, or not for 40-50 seconds. By running glxgears the problem is easily witnessed, also by maintaining the mouse in constant motion, and it was originally noticed during normal use when, e.g. the system momentarily becomes unresponsive to keyboard input. I've now also witnessed it by running top with a delay setting of zero and displaying system processes. The shorter delays are not easily spotted but the longer delays are. This was seen both in a terminal session and at a console. Using top also revealed another interesting phenomenon - the 'top' display freezes for very brief periods, probably less than 0.1s, but quite regularly, at around once per second. Seeing this prompted me to realise that glxgears stutters in what appears to be a corresponding way. This behaviour (the freezing itself) is not entirely consistent, either. Just while I've been writing these paragraphs I've noticed glxgears' behaviour range from stuttering at around once per second, to around twice per second, to not doing it (perceptibly) at all. I now realise that I've actually witnessed this behaviour in glxgears for quite some time, extending back into the distant history of this system, which may include 6.0-RELEASE but would certainly include a few different revisions in the 6.x series. It has been always source-upgraded since the original install. I'd always just thought that it was some hiccough in the graphics system. Ports were recently upgraded so it is now running xorg-7.3_1, gnome-2.20.2, nvidia-driver-96.43.01. The system has been up for 5 days now and is quite loaded up with applications at the moment and is normally left running. Currently it has running (on gnome): evolution, firefox-devel (with some dozens of windows and tabs open), xmms, gimp, xchat, eclipse-devel, gedit, a handful of terminal sessions. It's running a custom kernel - mainly raid and unused network drivers removed - mods listed below. The hardware is a Gigabyte GA-8SQ800 motherboard (SiS), with a 2.4GHz P4, 1.5GB DDR, nVidia MX440 AGP graphics, RTL-8139 network, AHA-2940 SCSI controller (for a scanner), 2 x 80GB ATA disks. This system has also had a tendency at times to crash when switching VTs between console and xorg. This problem has varied in consistency with the different incarnations of OS and nvidia-driver versions and hasn't yet occurred with this latest update. I'd really appreciate any suggestions to help identify what's going on here. Thanks, Wayne diff of kernel config with GENERIC: -# cpu I486_CPU -# cpu I586_CPU -# device eisa -# device ataraid # ATA RAID drives -# device atapifd # ATAPI floppy drives -# device amr # AMI MegaRAID -# device arcmsr # Areca SATA II RAID -# device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID -# device ciss # Compaq Smart RAID 5* -# device dpt # DPT Smartcache III, IV - See NOTES for options -# device hptmv # Highpoint RocketRAID 182x -# device hptrr # Highpoint RocketRAID 17xx, 22xx, 23xx, 25xx -# device rr232x # Highpoint RocketRAID 232x -# device iir # Intel Integrated RAID -# device ips # IBM (Adaptec) ServeRAID -# device mly # Mylex AcceleRAID/eXtremeRAID -# device twa # 3ware 9000 series PATA/SATA RAID -# device aac # Adaptec FSA RAID -# device aacp # SCSI passthrough for aac (requires CAM) -# device ida # Compaq Smart RAID -# device mfi # LSI MegaRAID SAS -# device mlx # Mylex DAC960 family -# device pst # Promise Supertrak SX6000 -# device twe # 3ware ATA RAID -# device plip # TCP/IP over parallel -# device cs # Crystal Semiconductor CS89x0 NIC -# device ed # NE[12]000, SMC Ultra, 3c503, DS8390 cards -# device ex # Intel EtherExpress Pro/10 and Pro/10+ -# device ep # Etherlink III based cards -# device fe # Fujitsu MB8696x based cards -# device ie # EtherExpress 8/16, 3C507, StarLAN 10 etc. -# device lnc # NE2100, NE32-VL Lance Ethernet cards -# device sn # SMC's 9000 series of Ethernet chips -# device xe # Xircom pccard Ethernet -# device ural # Ralink Technology RT2500USB wireless NICs -# device firewire # FireWire bus code -# device sbp # SCSI over FireWire (Requires scbus and da) -# device fwe # Ethernet over FireWire (non-standard!) # vmstat -i interrupt total rate irq1: atkbd0 256273 0 irq6: fdc0 8 0 irq14: ata0 4009537 8 irq15: ata1 2778519 6 irq16: nvidia0 18466724 41 irq18: pcm0 ahc0 325275270 727 irq19: rl0 4942307 11 irq20: ohci0 69879564 156 irq21: ohci1 993537 2 irq23: ehci0 1 0 cpu0: timer 894649288 2000 Total 1321251028 2953
Julian H. Stacey
2008-Jan-08 10:39 UTC
6.3-PRERELEASE desktop system periodically freezes momentarily
> By running glxgears the problem is easily witnessed, also by maintainingTo save others hunting sources which have migrated between FreeBSD releasess: FreeBSD/releases/4.11-RELEASE/ports x11/XFree86-4-clients/pkg-plist:bin/glxgears x11/xorg-clients/pkg-plist:bin/glxgears FreeBSD/releases/6.2-RELEASE/ports x11/XFree86-4-clients/files/manpages: glxgears.1 \ x11/XFree86-4-clients/pkg-plist:bin/glxgears x11/xorg-clients/files/manpages: glxgears.1 \ x11/xorg-clients/pkg-plist:bin/glxgears FreeBSD/branches/-current/ports graphics/mesa-demos/Makefile:XDEMO_PROGS= glthreads glxcontexts glxdemo glxgears glxgears_fbconfig \ builds OK on 7-Stable x11/XFree86-4-clients/files/manpages: glxgears.1 \ x11/XFree86-4-clients/pkg-plist:bin/glxgears XFree86-clients-4.5.0_4 is part of XFree86 and you have xorg set for X11 distribution. -- Julian Stacey. Munich Computer Consultant, BSD Unix C Linux. http://berklix.com
Toomas Aas
2008-Jan-08 11:29 UTC
6.3-PRERELEASE desktop system periodically freezes momentarily
Wayne Sierke wrote:> FreeBSD 6.3-PRERELEASE #5: Sat Dec 29 19:25:43 CST 2007 i386 > > I've noticed that this system is freezing periodically, for anywhere > from around a fraction of a second up to perhaps 1.5 seconds, and > occurring on average about twice a minute, but with no particular > pattern - i.e. it could happen twice in fairly quick succession, or not > for 40-50 seconds.I'm very surprised to find out that maybe I'm not alone in the universe. I have a 6.3-PRERELEASE system (last cvsupped Dec 12), which always freezes *once* a few minutes after booting up. This happens typically at the time when I've just logged in using xdm, fluxbox has started up and I'm opening my first xterm. The freeze lasts approximately 1-3 seconds. Once this freeze is over, it doesn't happen again until I shut down the machine for the night. But it does happen every time after starting up.> Ports were recently upgraded so it is now running xorg-7.3_1, > gnome-2.20.2, nvidia-driver-96.43.01.I have xorg 7.3. I don't have the full Gnome installed, but I do have gtk 2.12 and lots of other gtk-based "stuff" from gnome 2.20> The system has been up for 5 days now and is quite loaded up with > applications at the moment and is normally left running.Mine gets shut down every night.> The hardware is a Gigabyte GA-8SQ800 motherboard (SiS), with a 2.4GHz > P4, 1.5GB DDR, nVidia MX440 AGP graphics, RTL-8139 network, AHA-2940 > SCSI controller (for a scanner), 2 x 80GB ATA disks.Intel D865GLC mobo (i865), 2.8 GHz P4, 768 MB DDR, integrated i865 graphics, integrated Intel Pro/100 Ethernet, additional VIA Fire II (VT6306) PCI FireWire adapter, 1x80 GB ATA disk. Arbitrary PCI SoundBlaster workalike which calls itself "AudioPCI ES1373-B".> I'd really appreciate any suggestions to help identify what's going on > here.I've always just thought it to be some random hardware glitch that is hopeless to track down, but maybe I'm wrong. This machine was installed with FreeBSD just last autumn and initially it didn't have this problem running 6.2-STABLE. Unfortunately I can't pinpoint which source update or ports update or any other change it was that triggered this behaviour. My kernel config diff from GENERIC: -cpu I486_CPU -cpu I586_CPU -makeoptions DEBUG=-g # (<- the irony) -options INET6 -options NFSCLIENT -options NFSSERVER -options NFS_ROOT -options COMPAT_FREEBSD4 -options COMPAT_FREEBSD5 -device ataraid -device atapifd -device atapist -device ahb -device ahc -options AHC_REG_PRETTY_PRINT -device ahd -options AHD_REG_PRETTY_PRINT -device amd -device isp -device mpt -device sym -device trm -device adv -device adw -device aha -device aic -device bt -device ncv -device nsp -device stg -device ch -device sa -device ses -device amr -device arcmsr -device asr -device ciss -device dpt -device hptmv -device rr232x -device iir -device ips -device mly -device twa -device aac -device aacp -device ida -device mfi -device mlx -device pst -device twe -device cbb -device pccard -device cardbus -device ppc -device ppbus -device lpt -device plip -device ppi -device de -device em -device ixgb -device txp -device vx -device bce -device bfe -device bge -device dc -device lge -device msk -device nge -device nve -device pcn -device re -device rl -device sf -device sis -device sk -device ste -device stge -device ti -device tl -device tx -device vge -device vr -device wb -device xl -device cs -device ed -device ex -device ep -device fe -device ie -device lnc -device sn -device xe -device wlan -device wlan_wep -device wlan_ccmp -device wlan_tkip -device an -device ath -device ath_hal -device ath_rate_sample -device awi -device ral -device wi -device sl -device ppp -device gif -device faith -device ohci -device ural -device urio -device uscanner -device aue -device axe -device cdce -device cue -device kue -device rue -device fwe +device sound +device snd_es137x +device smb +device smbus +device ichsmb +device atapicam +options COMPAT_LINUX +device pf +device pflog +device drm +device i915drm # vmstat -i interrupt total rate irq1: atkbd0 3 0 irq6: fdc0 8 0 irq12: psm0 122250 36 irq14: ata0 8244 2 irq15: ata1 1392 0 irq16: uhci0 uhci+ 23973 7 irq20: fxp0 2337 0 irq21: fwohci0 1 0 irq22: pcm0 254359 76 irq23: ehci0 1 0 cpu0: timer 6650935 1999 Total 7063503 2123 # cat /etc/make.conf CPUTYPE=pentium4 CFLAGS=-O2 -pipe NO_ATM=true NO_BLUETOOTH=true NO_I4B=true NO_INET6=true NO_NIS=true NO_PROFILE=true PERL_VER=5.8.8 PERL_VERSION=5.8.8> > # vmstat -i > interrupt total rate > irq1: atkbd0 256273 0 > irq6: fdc0 8 0 > irq14: ata0 4009537 8 > irq15: ata1 2778519 6 > irq16: nvidia0 18466724 41 > irq18: pcm0 ahc0 325275270 727 > irq19: rl0 4942307 11 > irq20: ohci0 69879564 156 > irq21: ohci1 993537 2 > irq23: ehci0 1 0 > cpu0: timer 894649288 2000 > Total 1321251028 2953 > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
Kris Kennaway
2008-Jan-13 12:00 UTC
RELENG_7 2008/01/10 desktop system also periodically freezes
J.R. Oldroyd wrote:> On Sat, 12 Jan 2008 13:40:34 -0500, I wrote: >> Ah! Just experienced a short freeze ... > > And another! This one about 3 or 4 seconds. > > The profile from the last minute is here: > http://opal.com/jr/freebsd/releng_7-freeze/200801121342-unknown.txt > > This one shows a long hold_avg (I meant hold_avg in the previous > message, too) for: > /usr/src/sys/kern/vfs_vnops.c:515 (lockmgr:ufs) > > -jrThanks. Those particular things you mention are both normal, I will look at the traces in more detail later. Kris
Kris Kennaway
2008-Jan-13 12:10 UTC
6.3-PRERELEASE desktop system periodically freezes momentarily
Wayne Sierke wrote:> On Sun, 2008-01-13 at 12:39 +0100, Kris Kennaway wrote: >> MUTEX_PROFILING changes the kernel ABI so modules that are not compiled >> with that option will not work. If you use make buildkernel to build >> your kernel + modules together then it uses the kernel config file for >> both so they are compatible, otherwise your modules only are built with >> default options. So, if you have any other modules apart from nvidia >> then use make buildkernel for those, and add -DMUTEX_PROFILING to the >> CFLAGS of the nvidia build and try that. It may still not be enough >> since nvidia is a wrapper around a binary module, so you may also need >> to revert to nv. >> >> Kris > > Kris, > > Success, I hope. I noticed that the duration of the pause when the > stuttering occurs is obviously magnified, I'd guess probably by approx a > factor of 5. e.g. normally stuttering interval is quite short, perhaps > 50-100ms at a guess, with debug.mutex.prof.enable=1 duration of stutter > might be 200ms or so. I understand that this is to be expected, just > want to let you know. > > Could you please confirm that I'm following correct procedure to obtain > results, then I'll endeavour to capture logs for the other freezes that > I witness. And of course let me know if you want anything specific from > me.Yeah, this shows things like contention between the mouse device and other parts of the kernel that still require the Giant lock in 6.x. It is not likely that these will be fixed in 6.x but most of them are in 7.0, so you should obtain better performance by upgrading to 7.0. Kris
Kris Kennaway
2008-Jan-13 12:11 UTC
6.3-PRERELEASE desktop system periodically freezes momentarily
Toomas Aas wrote:> Kris Kennaway wrote: >> >> sysctl debug.mutex.prof.enable=1 >> ... trigger hang ... >> sysctl debug.mutex.prof.enable=0 >> >> and send me the output of >> >> sysctl debug.mutex.prof.stats > > The output is here: > http://www.raad.tartu.ee/~toomas/mutex_stats.txtThis one also shows giant contention from the syncer and your mouse pointer that should also be fixed by updating to 7.0. Kris
Kostik Belousov
2008-Jan-13 21:32 UTC
RELENG_7 2008/01/10 desktop system also periodically freezes
On Mon, Jan 14, 2008 at 12:03:54AM -0500, J.R. Oldroyd wrote:> Yet another: > http://opal.com/jr/freebsd/releng_7-freeze/200801132359-ktr.out > > Shows just the same as the first, just: > CPU 0 > irq 17: pcm0 ath0 > ath0 taskqBTW, I am experiencing hard hang on my laptop when ath0 is brought up and powerd is running. It takes several seconds when this happens with no AP nearby. It may take up to the hour when machine is able associate to the AP. Reliable workaround for me was to stop powerd before turning on ath0 on the laptop. On the other hand, one of my desktop workstation has atheros card, and runs powerd without problem, lowering the frequency of the CPU. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20080114/bc4e4612/attachment.pgp
J.R. Oldroyd
2008-Jan-14 10:14 UTC
RELENG_7 2008/01/10 desktop system also periodically freezes
Here is another ktr dump. This freeze was a longer one, getting on for two minutes: http://opal.com/jr/freebsd/releng_7-freeze/200801141259-ktr.out This one again shows some post-freeze activity, and yet again the only activity during the freeze is that shared ath0/pcm irq and the ath0 taskq. -jr -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20080114/71f166d7/signature.pgp
In an attempt to track down this mouse freezing/stuttering (i.e. "jerky mouse movement) behavior in FreeBSD 7.0-RC1, I have come up with a reliable way to cause it to happen, and I have created a longer trace showing the results. Note that I am using the ULE scheduler. In general, it becomes easier to see the effect if there is CPU activity. I have noticed it during kernel compiles, while at the same time loading web pages in firefox that contain images (and moving the mouse while this is happening). But a more controlled way to see it is to run something that uses some CPU and then generating lots of X events. In my case, I start "xtrs" (TRS-80 emulator) in Model IV mode, which happens to poll for input, using the CPU. Then I move the mouse back and forth quickly between windows in "focus under mouse" mode (in my case, a KDE focus mode), which causes many focus events quickly. In about 15 or 20 seconds, the mouse reliably starts to show erratic movement, not moving smoothly. I really hope this can shed more light on what might be going on. Here is the trace: http://www.skyrush.com/downloads/ktr_ule_4.out Thanks, Joe
Julian H. Stacey
2008-Jan-24 02:30 UTC
New KTR trace for mouse freezing/stuttering in 7.0-RC1
Joe Peterson wrote:> In an attempt to track down this mouse freezing/stuttering (i.e. "jerky > mouse movement) behavior in FreeBSD 7.0-RC1, I have come up with a > reliable way to cause it to happen, and I have created a longer trace > showing the results. Note that I am using the ULE scheduler.Have you tried not touching the mouse & playing mp3 music do you also hear sound interruptions after a few minutes ? As well as the people talking re. mouse (BTW Kris asked current@ recently if anyone had time to look at Giant lock); Others hear sound interruptions: I wonder if there may be some commonality as well as/beyond mouse locking, in scheduling/locking etc. I've heard sound interuptions on kernels with both schedulers, (but my hosts are sensitive as slow & with DMA off (else wont boot), & I must review my configs some more). -- Julian Stacey. Munich Computer Consultant, BSD Unix C Linux. http://berklix.com