Yesterday I was running star | star on another machine reading 2 ZFS filesystems from this machine over NFS. Data in the beginning appeared to be moving quite quickly. This morning I came in to find that it had come all but a screeching halt. The client wasn''t complaining (that I could find) about the NFS server not responding, but the disk usage on the destination disk wasn''t increasing at all that I could see. I had a window on this server machine, and I tried running some commands, like dmesg, vmstat, iostat. They all hung with out running. I''m thinking resource shortage. I went to the machine to try the console, and I couldn''t get the monitor to even wake up. I hit Stop-A and sync''d it, and booted it back up. It had been running NV_32. For some reason that wouldn''t boot. But I had live upgrade the other partition to NV_35 so I booted that instead. The NFS picked up where it had left off (I love NFS!) and I started a ''vmstat 1'' (not the best choice I know.) to watch the machine. After a few minutes though I started seeing numbers so huge in the vmstat output I couldn''t believe it. I''m no expert on what this stuff means but a change like that seems unusual to me. What better tools should I use? What other data should I capture? Is this normal? -Kyle vmstat output: bell# vmstat 1 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 m0 in sy cs us sy id 15 0 0 1727480 760360 79 361 1995 3 3 0 97 77 77 0 64 1875 4884 1999 16 48 36 0 0 0 2093976 54216 1 46 69 7 7 0 0 6 5 0 3 5024 255 1499 1 61 38 0 0 0 2091584 51832 0 1 0 0 0 0 0 0 0 0 0 3595 60 1114 1 54 45 0 0 0 2090224 50472 0 55 38 0 0 0 0 2 3 0 5 3404 86 1105 1 53 46 0 0 0 2088264 48480 0 0 0 0 0 0 0 0 0 0 0 3498 58 1082 0 52 48 0 0 0 2085920 46136 0 0 0 0 0 0 0 0 0 0 0 3517 67 1077 1 54 45 9 0 0 2081336 41552 0 3 0 0 0 0 0 1 1 0 0 5356 52 1661 1 70 29 0 0 0 2078176 38392 0 0 0 0 0 0 0 0 0 0 0 3470 54 1103 0 50 50 0 0 0 2076624 36840 0 0 0 0 0 0 0 0 0 0 0 3775 60 997 1 66 33 0 0 0 2075456 35672 0 0 0 0 0 0 0 23 22 0 19 3350 56 646 1 46 53 0 0 0 2075344 35560 0 0 0 0 0 0 0 0 0 0 0 3259 65 792 0 42 58 0 0 0 2074216 34432 0 0 0 0 0 0 0 0 0 0 0 3601 56 1075 1 55 44 0 0 0 2073416 33632 0 0 0 0 0 0 0 0 0 0 0 3455 58 1083 0 46 54 13 0 0 2072264 32480 0 0 0 0 0 0 0 0 0 0 0 3729 54 1033 1 59 40 0 0 0 2069448 29664 0 0 0 0 0 0 0 0 0 0 0 3549 51 962 1 58 41 0 0 0 2064544 24760 0 0 0 0 0 0 0 2 2 0 0 3893 59 1123 0 69 31 0 0 0 2118200 78416 0 0 0 0 0 0 0 0 0 0 0 4545 50 1191 1 57 42 2 0 0 2115968 76184 0 0 0 0 0 0 0 0 0 0 0 3611 55 1024 0 63 37 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 m0 in sy cs us sy id 0 0 0 2113400 73616 0 0 0 0 0 0 0 0 0 0 0 3519 56 788 1 44 55 0 0 0 2112952 73168 0 0 0 0 0 0 0 0 0 0 0 3733 66 832 1 56 43 0 0 0 2111824 72040 0 0 0 0 0 0 0 0 0 0 0 3435 59 745 0 48 52 9 0 0 2110392 70608 0 0 0 0 0 0 0 0 0 0 0 3529 56 942 1 50 49 2 0 0 2108312 68528 0 0 0 0 0 0 0 0 0 0 0 3723 55 836 0 63 37 0 0 0 2108304 68520 0 0 0 0 0 0 0 0 0 0 0 3178 58 643 1 40 59 0 0 0 2106936 67152 0 0 0 0 0 0 0 1 1 0 0 3509 65 883 1 49 50 0 0 0 2106280 66496 0 0 0 0 0 0 0 0 0 0 0 3666 52 1088 0 54 46 0 0 0 2104968 65184 0 0 0 0 0 0 0 0 0 0 0 3601 54 920 1 52 47 11 0 0 2104960 65176 0 0 0 0 0 0 0 0 0 0 0 3557 60 834 1 48 51 11 0 0 2103912 64128 0 0 0 0 0 0 0 0 0 0 0 3668 56 751 0 59 41 0 0 0 2103888 64104 0 0 0 0 0 0 0 0 0 0 0 3323 64 608 1 52 47 0 0 0 2103880 64096 0 0 0 0 0 0 0 0 0 0 0 3256 57 654 0 42 58 2 0 0 2103712 63928 0 0 0 0 0 0 0 0 0 0 0 3451 58 868 1 42 57 0 0 0 2103016 63232 0 0 0 0 0 0 0 0 0 0 0 3713 56 910 0 57 43 1 0 0 2102624 62840 0 0 0 0 0 0 0 0 0 0 0 3348 56 830 1 44 55 1 0 0 2101136 61352 0 0 0 0 0 0 0 0 0 0 0 3512 64 1094 1 51 48 0 0 0 2099552 59768 0 0 0 0 0 0 0 0 0 0 0 3893 51 903 0 62 38 9 0 0 2099152 59368 0 0 0 0 0 0 0 0 0 0 0 3197 50 679 1 40 59 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 m0 in sy cs us sy id 8 0 0 2098472 58688 0 0 0 0 0 0 0 0 0 0 0 3933 54 820 1 58 41 0 0 0 2097728 57968 0 9 0 0 0 0 0 4 4 0 1 3374 133 733 0 50 50 0 0 0 2097656 57896 0 0 0 0 0 0 0 0 0 0 0 3488 56 715 1 52 47 8 0 0 2095720 55960 0 0 0 0 0 0 0 0 0 0 0 3272 53 881 1 44 55 0 0 0 2094592 54832 0 0 0 0 0 0 0 0 0 0 0 3615 55 881 0 49 51 2 0 0 2093176 53416 0 0 0 0 0 0 0 0 0 0 0 3833 59 989 1 55 44 11 0 0 2092200 52440 0 0 0 0 0 0 0 0 0 0 0 3967 65 1034 0 55 45 9 0 0 2091104 51344 0 0 0 0 0 0 0 0 0 0 0 3485 54 900 1 47 52 1 0 0 2089952 50192 0 0 0 0 0 0 0 0 0 0 0 3759 55 918 1 57 42 1 0 0 2089560 49800 0 0 0 0 0 0 0 7 7 0 0 3647 61 945 0 45 55 0 0 0 2088824 49064 0 0 0 0 0 0 0 73 73 0 67 4123 55 1044 1 60 39 9 0 0 2088368 48608 0 0 0 0 0 0 0 13 14 0 16 4015 63 1023 0 54 46 1 0 0 2087576 47816 0 0 0 0 0 0 0 0 0 0 0 3542 55 890 1 46 53 0 0 0 2087176 47416 0 0 0 0 0 0 0 0 0 0 0 3715 57 885 1 54 45 0 0 0 2087104 47344 0 0 0 0 0 0 0 0 0 0 0 3606 53 794 0 55 45 0 0 0 2086440 46680 0 4 0 0 0 0 0 0 0 0 0 3515 70 865 1 44 55 0 0 0 2086048 46288 0 0 0 0 0 0 0 2 2 0 0 3539 66 978 0 57 43 0 0 0 2085712 45952 0 0 0 0 0 0 0 0 0 0 0 3546 58 828 1 52 47 8 0 0 2085312 45552 0 0 0 0 0 0 0 0 0 0 0 3637 55 1028 1 50 49 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 m0 in sy cs us sy id 0 0 0 2084456 44696 0 0 0 0 0 0 0 0 0 0 0 3643 60 967 0 50 50 8 0 0 2083096 43336 0 0 0 0 0 0 0 0 0 0 0 4049 64 1114 1 58 41 1 0 0 2081472 41712 0 0 0 0 0 0 0 2 2 0 0 3653 56 1056 0 55 45 0 0 0 2080200 40440 0 0 0 0 0 0 0 0 0 0 0 3838 60 1179 1 52 47 0 0 0 2079488 39728 0 0 0 0 0 0 0 0 0 0 0 3820 56 1083 1 51 48 11 0 0 2078880 39120 0 0 0 0 0 0 0 0 0 0 0 3779 59 1144 0 54 46 0 0 0 2078456 38696 0 0 0 0 0 0 0 2 2 0 0 3810 70 1030 1 59 40 6 0 0 2078128 38368 0 0 0 0 0 0 0 0 0 0 0 3573 55 937 1 46 53 4 0 0 2077064 37304 0 0 0 0 0 0 0 0 0 0 0 3822 55 1098 0 56 44 0 0 0 2074632 34872 0 0 0 0 0 0 0 0 0 0 0 3749 60 1089 1 56 43 0 0 0 2072080 32320 0 0 0 0 0 0 0 0 0 0 0 3702 115 1116 1 56 43 0 0 0 2071032 31272 0 0 0 0 0 0 0 2 2 0 0 3498 63 1092 0 51 49 0 0 0 2068888 29128 0 0 0 0 0 0 0 0 0 0 0 3513 56 1063 1 51 48 0 0 0 2067200 27440 0 0 0 0 0 0 0 0 0 0 0 3450 58 1127 0 52 48 2 0 0 2065352 25592 0 0 0 0 0 0 0 0 0 0 0 3618 54 1190 1 56 43 0 0 0 2064056 24296 0 0 0 0 0 0 0 0 0 0 0 3599 56 877 1 56 43 0 0 0 2063984 24224 0 0 0 0 0 0 0 1 1 0 0 3572 67 821 0 48 52 8 0 0 2076400 36640 0 0 0 0 0 0 0 0 0 0 0 3613 56 872 1 52 47 0 0 0 2076072 36312 0 0 0 0 0 0 0 0 0 0 0 3534 55 885 0 52 48 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 m0 in sy cs us sy id 13 0 0 2075064 35304 0 0 0 0 0 0 0 0 0 0 0 3581 58 1071 1 49 50 0 0 0 2072256 32496 0 0 0 0 0 0 0 0 0 0 0 3659 57 1186 1 56 43 9 0 0 2068864 29104 0 0 0 0 0 0 0 0 0 0 0 5050 67 1647 0 70 30 0 0 0 2067768 28008 0 0 0 0 0 0 0 0 0 0 0 3836 55 1176 1 66 33 0 0 0 2064080 24344 0 0 0 0 0 0 0 0 0 0 0 3271 55 1262 0 52 48 5 0 0 2059136 20696 0 5 6 0 0 0 0 1 0 0 1 3420 50 1988 1 69 31 0 0 0 2053920 17128 0 10 31 0 0 0 0 3 4 0 5 3438 77 1960 1 61 38 6 0 0 2059016 22184 0 2 15 0 0 0 0 1 0 0 1 5015 54 3867 0 66 34 0 0 0 2054768 17936 0 0 0 0 2000 0 420 0 0 0 0 3226 50 2124 1 72 27 2 0 0 2074960 41160 0 8 8 0 0 0 0 0 1 0 1 3692 58 869 1 54 45 10 0 0 2073656 39848 0 9 0 0 0 0 0 0 0 0 0 3897 56 1052 0 50 50 0 0 0 2069112 35304 0 1 0 0 0 0 0 0 0 0 0 4212 59 1014 1 60 39 0 0 0 2068768 34960 0 6 0 0 0 0 0 0 0 0 0 4506 55 1091 1 51 48 3 0 0 2067472 33664 0 0 0 0 0 0 0 0 0 0 0 3795 58 926 0 58 42 0 0 0 2065720 31904 0 5 7 0 0 0 0 1 0 0 1 3643 54 825 1 51 48 8 0 0 2063912 30096 0 0 0 0 0 0 0 0 0 0 0 3605 51 910 0 49 51 0 0 0 2062448 28632 0 0 0 0 0 0 0 0 0 0 0 3726 65 1098 1 55 44 0 0 0 2060264 26448 0 0 0 0 0 0 0 0 0 0 0 3723 56 1096 1 49 50 0 0 0 2060600 26712 0 15 71 0 0 0 0 1 2 0 3 3740 60 998 0 55 45 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 m0 in sy cs us sy id 1 0 0 2057344 23456 0 0 0 0 0 0 0 2 1 0 3 3425 54 1580 1 58 41 0 0 0 2053888 19952 0 6 38 0 0 0 0 4 4 0 4 3752 68 2767 0 70 30 3 0 0 2047424 16840 0 4 23 0 3936 0 741 2 2 0 4 4258 42 3383 1 79 20 0 0 0 2042232 15848 0 1 8 0 1313 0 245 0 0 0 0 3165 54 2058 1 59 40 1 0 0 2040792 16544 0 3 8 0 1177 0 222 0 1 0 1 3362 56 2499 0 60 40 3 0 0 2037408 15344 0 1 0 0 2720 0 503 0 0 0 0 3282 50 2229 1 70 29 2 0 0 2035952 13112 0 8 30 0 2264 0 437 2 2 0 4 4662 63 3999 1 70 29 0 0 0 2035952 17168 0 0 0 0 1359 0 264 0 0 0 0 3278 55 2214 0 59 41 2 0 0 2035952 15904 0 42 15 0 513 0 88 6 5 0 3 3439 81 2512 1 63 36 9 0 0 2035952 16360 2 32 89 0 4296 0 769 5 6 0 10 3065 56 2000 1 69 31 8 0 0 2035952 21432 0 8 23 0 3508 0 655 1 0 0 1 3039 64 1817 0 60 40 0 0 0 2035952 23144 0 5 0 0 0 0 0 0 0 0 0 4260 55 3631 1 72 27 0 0 0 2035952 19632 0 0 0 0 0 0 0 0 0 0 0 3423 58 2507 0 62 38 13 0 0 2035952 13976 0 2 15 0 0 0 0 1 1 0 2 3038 52 2239 1 63 36 9 0 0 2035952 15064 0 38 81 0 3247 0 606 6 6 0 12 3234 61 1777 1 67 32 13 0 0 2035952 12912 0 9 30 0 0 0 0 3 2 0 2 4637 63 4075 0 78 22 0 0 0 2035952 16560 0 2 0 0 2902 0 549 0 1 0 1 3082 48 1988 1 74 25 1 0 0 2035952 16792 0 43 61 0 594 0 97 3 3 0 6 3204 107 2068 1 59 40 1 0 0 2035952 15920 0 13 15 0 667 0 110 3 1 0 3 3356 105 2080 0 75 25 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 m0 in sy cs us sy id 0 0 0 2035952 17216 0 7 23 0 536 0 86 1 2 0 3 3261 59 2149 1 55 44 0 0 0 2035952 18000 0 71 69 0 2888 0 476 4 4 0 8 3100 68 1810 1 68 31 0 0 0 2035952 21264 0 1 8 0 1723 0 348 0 0 0 0 3188 55 2319 0 60 40 1 0 0 2035952 13312 0 0 0 0 1163 0 221 0 0 0 0 3140 48 1894 1 66 34 0 0 0 2035952 20784 0 2 0 0 0 0 0 0 0 0 0 3419 58 1989 1 60 39 0 0 0 2035952 17024 0 4 15 0 0 0 36430 2 2 0 2 3810 64 3234 0 66 34 0 0 0 2035952 14376 0 1 6 0 0 0 16974 1 1 0 2 3784 43 2200 1 75 25 0 0 0 2035952 19408 2 19 76 3469 5303 0 56063 44 44 0 22 3292 53 1822 1 67 32 2 0 0 2035952 20464 7 38 231 0 0 0 0 6 6 0 11 3377 59 2721 0 60 40 1 0 0 2035952 17896 5 28 219 0 0 0 0 10 11 0 20 3252 60 2663 1 63 36 14 0 0 2035952 12800 9 29 108 2146 2292 0 111878 36 36 0 13 4288 65 2926 1 84 15 1 0 0 2035952 14720 3 5 7 3875 4308 0 63733 40 39 0 1 3476 55 2452 0 70 30 0 0 0 2035952 16240 1 2 8 3985 3985 0 4656 43 43 0 3 3281 55 2426 1 64 35 0 0 0 2035952 15120 6 10 94 2979 2990 0 43561 39 40 0 9 3281 40 2490 0 73 27 1 0 0 2035952 12664 2 16 200 3700 3714 0 99670 42 42 0 11 3361 51 2727 1 73 26 11 0 0 2035952 13936 13 34 172 2855 3102 0 149021 51 51 0 24 3072 55 1885 1 69 30 4 0 0 2035952 14248 28 59 243 3541 3805 0 207660 57 57 0 14 2964 40 1752 1 73 26 0 0 0 2035952 13712 53 79 173 2662 2662 0 101122 33 33 0 1 3514 50 2351 0 70 30 3 0 0 2035952 19072 2 3 7 1536 1629 0 105140 28 28 0 1 4480 51 3559 1 81 18 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 m0 in sy cs us sy id 1 0 0 2035952 10216 28 68 247 2296 2668 0 345773 45 46 0 16 2675 47 1374 1 84 15 3 0 0 2035952 9592 52 70 167 1617 1944 0 493507 50 50 0 8 2513 44 1473 0 76 24 3 0 0 2035952 9344 91 120 261 1948 2545 0 458570 49 49 0 18 2344 51 1685 1 85 14 5 0 0 2035952 7088 92 125 302 2535 3188 0 524223 47 46 0 16 2919 35 2562 0 85 14 3 0 0 2035952 8488 61 89 249 1442 1862 0 575243 54 55 0 15 2225 27 1549 0 84 16 10 0 0 2035952 11144 82 114 264 594 1019 0 572697 23 23 0 7 2922 30 2114 1 84 15 5 0 0 2035952 7408 199 239 558 921 2098 0 645309 46 46 0 15 2267 60 1190 1 80 19 0 0 0 2035952 6808 142 155 246 632 1396 0 607211 30 31 0 10 2296 57 1741 1 84 15 7 0 0 2035952 6544 152 159 103 587 1284 0 754783 32 32 0 3 1966 48 1100 1 81 19 16 0 74 2035952 4144 138 146 54 1078 2366 0 568403 13 13 0 2 2771 53 1759 0 93 7 2 0 80 2035896 26168 42 64 125 126 351 0 1339510 48 47 0 39 1033 10 519 0 98 2 5 0 80 2035880 19672 44 48 50 0 0 0 0 0 1 0 1 3562 51 2487 1 62 38 1 0 80 2035880 17688 0 1 8 0 0 0 0 4 3 0 1 3371 57 2338 1 59 40 11 0 80 2035880 14168 32 33 0 941 1725 0 140260 12 12 0 0 3295 74 2351 0 72 28 4 0 80 2035880 14656 118 120 0 975 1410 0 39007 18 18 0 0 4877 64 3842 1 76 23 7 0 80 2035880 10472 63 72 41 461 1150 0 236936 13 16 0 1 2936 45 1859 1 72 27 0 0 80 2035880 40144 111 106 0 442 933 0 174258 11 11 0 0 2910 50 2044 0 74 26 1 0 80 2035880 39144 84 82 21 269 262 0 0 8 8 0 2 3384 50 2121 1 60 39 13 0 80 2035880 34592 105 194 517 0 0 0 0 32 32 0 12 3434 57 2122 1 64 35 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 m0 in sy cs us sy id 0 0 80 2035880 34904 50 62 0 0 0 0 0 0 0 0 0 3133 64 2537 1 66 33 11 0 80 2035880 29272 0 0 0 0 0 0 0 0 0 0 0 3215 57 1703 0 60 40 3 0 80 2035880 32688 0 0 0 0 0 0 0 2 2 0 0 3140 56 1991 1 58 41 2 0 80 2035864 32792 7 50 287 0 0 96 0 18 17 0 14 3582 65 2350 0 56 44 6 0 78 2035864 27704 73 84 169 0 0 0 0 3 2 0 2 4407 83 3117 1 71 28 11 0 78 2035864 22968 0 0 0 0 0 0 0 0 0 0 0 3163 47 1588 1 65 34 2 0 78 2035864 25856 0 0 0 0 0 0 0 0 0 0 0 3126 54 1853 0 59 41 12 0 78 2035864 26616 0 0 0 0 0 0 0 0 0 0 0 3516 58 2600 1 61 38 12 0 78 2035864 22320 0 0 0 0 0 0 0 0 0 0 0 3276 54 2546 1 62 37 0 0 78 2035864 28912 0 0 0 0 0 0 0 0 0 0 0 2950 69 4489 0 61 39 1 0 78 2035864 25832 0 0 0 0 0 0 0 0 0 0 0 3276 54 2261 1 66 33 0 0 78 2035864 19056 0 0 0 0 0 0 19420 1 1 0 0 3244 52 1989 1 66 34 12 0 78 2035864 39960 1 1 0 0 0 0 0 0 0 0 0 3587 55 2520 0 62 38 6 0 78 2035864 39024 0 0 0 0 0 0 0 25 26 0 19 4579 51 3447 1 76 23 0 0 78 2035864 39752 0 2 15 0 0 0 0 1 1 0 2 3864 62 2718 0 65 35 1 0 78 2035864 37704 0 0 0 0 0 0 0 0 0 0 0 3469 57 2002 1 67 32 0 0 78 2035864 37488 0 0 0 0 0 0 0 0 0 0 0 3228 55 1603 1 53 46 14 0 78 2035864 29552 0 0 0 0 0 0 0 0 0 0 0 3552 53 2575 0 65 35 5 0 78 2035864 29072 13 39 164 0 0 0 0 12 13 0 9 4299 104 3151 1 74 26 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 m0 in sy cs us sy id 0 0 78 2035864 27792 0 0 0 0 0 0 0 0 0 0 0 3295 54 2168 1 54 45 0 0 78 2035864 25944 0 0 0 0 0 0 0 0 0 0 0 3132 45 1933 0 68 32 0 0 78 2035864 24352 0 0 0 0 0 0 0 0 0 0 0 3184 54 1978 1 60 39 0 0 78 2035864 23608 0 0 0 0 0 0 0 0 0 0 0 2879 47 2261 1 55 44 2 0 78 2035864 22896 0 0 0 0 0 0 0 0 0 0 0 3341 64 2346 0 56 44 1 0 78 2035864 21240 0 0 0 0 0 0 25599 0 0 0 0 3208 55 1803 1 66 33 0 0 78 2035864 18272 0 0 0 0 0 0 0 0 0 0 0 3483 58 3621 0 64 36 0 0 78 2035864 36256 0 0 0 0 0 0 12799 0 0 0 0 3717 55 3262 1 68 31 13 0 78 2035864 21760 0 0 0 0 0 0 0 0 0 0 0 4553 47 3561 1 81 19 0 0 78 2035864 19528 0 0 0 0 0 0 0 0 0 0 0 3788 54 3645 0 68 32 13 0 78 2035864 11704 0 0 0 244 404 0 27305 4 4 0 0 3351 57 1993 1 66 33 1 0 78 2035864 15528 12 19 0 410 1242 0 89291 21 21 0 0 3214 46 2224 1 68 31 0 0 78 2035864 15264 159 153 44 684 793 0 29787 8 7 0 1 5018 121 3971 1 76 23 0 0 78 2035864 15024 12 12 0 269 723 0 93535 5 5 0 0 3276 58 2154 0 62 38 0 0 78 2035864 17016 37 37 0 885 1923 0 180183 16 16 0 1 2815 55 1711 1 66 33 1 0 78 2035864 14648 93 89 0 400 775 0 225114 13 13 0 0 3076 45 1998 1 75 24 0 0 78 2035864 13688 174 182 53 465 853 0 161887 16 15 0 0 4311 60 3241 0 77 23 2 0 78 2035864 11896 182 183 8 891 1669 0 336453 31 32 0 0 2870 69 1956 1 72 27 1 0 78 2035864 10632 153 154 0 604 1425 0 476964 6 6 0 0 2696 50 1692 1 79 20 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 m0 in sy cs us sy id 2 0 78 2035864 9480 115 110 0 483 921 0 571891 12 13 0 1 2449 54 1644 1 80 19 5 0 78 2035864 16184 99 95 26 410 793 0 461213 9 9 0 0 2587 48 937 0 75 25 1 0 78 2035864 28952 86 81 0 136 136 0 0 3 3 0 0 4628 51 3575 1 66 33 0 0 78 2035864 28232 57 63 0 0 0 0 0 0 0 0 0 3686 62 2245 0 57 43 0 0 78 2035864 28048 0 0 0 0 0 0 0 4 4 0 0 3527 55 1840 1 62 38 0 0 78 2035864 26584 0 0 0 0 0 0 0 0 0 0 0 3423 57 1753 1 61 38 0 0 78 2035864 26544 0 0 0 0 0 0 0 23 22 0 17 3837 51 2022 0 59 41 0 0 78 2035864 25232 156 188 115 0 0 0 0 7 8 0 7 3756 85 2214 1 58 41 0 0 78 2035864 25072 0 0 0 0 0 0 0 0 0 0 0 3636 53 2474 1 62 37 0 0 78 2035864 22960 0 0 0 0 0 0 0 3 3 0 0 3528 58 2477 0 58 42 9 0 78 2035864 22112 0 0 0 0 0 0 0 0 0 0 0 3657 53 2178 1 67 32 0 0 78 2035864 21704 0 0 0 0 0 0 0 0 0 0 0 4883 54 4258 1 70 30 0 0 78 2035864 20320 4 18 78 0 0 0 0 7 5 0 3 3442 62 2609 0 66 34 0 0 78 2035864 18624 7 23 108 69 85 144 71876 7 7 0 2 3629 83 2248 1 65 34 3 0 77 2035864 17688 3 3 0 0 0 0 0 5 4 0 0 3646 53 2063 0 56 44 9 0 77 2035864 20776 55 88 228 744 1382 0 225907 35 36 0 19 3413 40 2061 0 84 16 0 0 77 2035864 20048 153 224 509 425 403 0 0 31 30 0 8 3717 52 2331 1 65 34 0 0 77 2035864 18744 19 20 8 0 0 0 0 3 2 0 1 3767 54 2793 1 60 39 2 0 77 2035864 18632 0 0 0 0 0 0 5795 0 0 0 0 3714 54 2574 1 66 33 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 m0 in sy cs us sy id 1 0 77 2035864 18192 37 61 117 0 0 0 0 8 8 0 9 5085 71 3994 0 77 23 0 0 77 2035864 16504 23 42 151 259 577 0 73801 15 14 0 15 3333 64 2167 1 67 32 0 0 77 2035864 14328 29 28 0 1064 2068 0 113023 12 12 0 0 3316 54 2427 1 69 30 1 0 77 2035864 13712 88 83 0 658 1264 0 285184 10 10 0 0 2840 56 2612 0 68 32 0 0 77 2035864 12520 114 109 0 495 967 0 292302 13 14 0 1 3010 52 2660 1 74 25 11 0 77 2035864 9864 121 117 7 434 841 0 341908 11 11 0 0 3818 49 3079 1 81 19 0 0 77 2035864 9160 226 234 0 912 1951 0 507211 23 22 0 1 2277 66 1932 1 78 21 2 0 77 2035864 6920 113 109 6 460 926 0 653905 17 18 0 1 2196 47 1335 1 85 14 2 0 77 2035864 5504 113 109 7 438 883 0 882417 13 12 0 1 1844 50 877 0 90 10 0 0 77 2035864 4472 135 132 84 467 1093 0 777219 14 14 0 1 2677 35 2116 1 91 8 0 0 77 2035864 19072 148 159 30 359 359 0 29666 17 17 0 2 3371 66 1971 1 62 37 0 0 77 2035864 17656 42 43 8 179 334 0 75556 10 9 0 1 3212 55 2200 0 69 31 1 0 77 2035864 15248 27 27 0 85 177 0 38397 1 1 0 0 3419 55 2157 1 64 35 12 0 77 2035864 11288 29 43 62 103 379 0 110342 20 22 0 4 4070 58 3315 1 76 23 1 0 77 2035864 12456 202 278 262 1211 2145 0 374254 42 41 0 15 2973 67 1787 1 73 26 3 0 77 2035864 11648 164 167 0 769 1638 0 376119 18 18 0 0 2988 64 1949 0 78 22 1 0 77 2035864 9232 119 115 0 467 941 0 445626 19 19 0 0 2531 53 1605 1 70 29 3 0 77 2035864 7776 131 127 0 452 1014 0 514341 23 23 0 0 2564 46 1630 1 78 21 2 0 77 2035864 5992 114 124 80 413 901 0 810573 32 32 0 7 1881 31 931 1 88 12 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 m0 in sy cs us sy id 4 0 77 2035864 4360 159 183 127 494 1327 0 875620 18 18 0 4 1843 35 986 0 87 13 3 0 79 2035864 4144 82 84 34 318 732 0 915286 13 11 0 1 2142 28 1816 0 90 10 2 0 79 2035864 18472 140 225 580 91 349 0 302388 37 41 0 18 2908 44 1895 1 70 29 14 0 79 2035864 15040 53 68 23 0 0 0 38397 3 2 0 2 3346 67 1774 1 60 39 8 0 79 2035864 13408 8 8 0 290 731 0 84842 11 11 0 0 3465 54 2123 0 65 35 6 0 79 2035864 9752 85 95 79 1030 1685 0 179196 18 18 0 2 3205 49 1658 1 73 26 2 0 79 2035864 14504 140 164 251 709 1539 0 233078 26 27 0 16 2782 64 1434 1 70 30 5 0 79 2035864 13480 149 150 0 493 1047 0 308049 16 16 0 1 2665 56 1228 1 72 28 0 0 79 2035864 13952 113 110 0 576 1301 0 178957 12 12 0 0 3118 53 2016 0 64 36 0 0 79 2035864 12568 127 170 325 406 856 0 273596 30 30 0 5 2875 65 1668 2 73 26 0 0 79 2035864 11288 138 166 136 864 1486 0 374853 29 29 0 2 2495 54 2004 1 65 34 6 0 79 2035864 7904 219 275 274 1007 2267 48 427611 32 31 0 19 2202 78 4038 1 75 24 7 0 77 2035864 5088 102 133 212 384 910 0 884450 24 24 0 5 1798 42 1133 0 87 13 0 0 77 2035864 15528 75 96 170 310 680 96 821516 34 35 0 16 2319 25 1471 1 90 9 4 0 76 2035864 13424 169 261 568 785 1607 0 228721 58 58 0 19 3260 93 2407 1 69 30 5 0 76 2034576 12112 207 280 644 995 1750 0 283885 57 58 0 13 4096 143 3668 1 85 15 12 0 76 2033320 9144 199 316 492 1464 2949 0 498610 81 81 0 26 2709 88 1725 1 88 11 9 0 76 2033320 7752 418 511 612 1159 3671 0 719054 72 70 0 28 2125 64 1504 1 87 12 1 0 76 2033320 6200 545 641 721 1067 4085 0 693523 71 73 0 24 2393 69 1493 2 86 13 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 m0 in sy cs us sy id 10 0 76 2035840 6424 399 404 245 1155 4032 12904 951325 39 40 0 10 2107 97 1131 1 94 6 5 0 76 2035768 7904 202 206 176 509 1733 10456 1227418 29 27 0 9 1605 39 919 0 97 2 8 0 76 2035664 6624 381 397 134 808 2857 8480 1195503 39 40 0 9 1275 57 570 0 97 3 13 0 78 2035632 6120 326 344 101 890 2988 6872 1198958 29 29 0 4 1279 62 780 1 99 1 0 0 78 2035592 17824 307 404 475 556 1641 6480 723306 83 85 0 31 2142 81 1328 1 88 11 10 0 75 2035624 15280 521 691 1177 1947 5291 5928 450919 84 83 0 30 2945 189 1876 2 83 15 ^Cbell#
Kyle McDonald <Kyle.McDonald at Sun.COM> wrote:> Yesterday I was running star | star on another machine reading 2 ZFS > filesystems from this machine over NFS. Data in the beginning appeared > to be moving quite quickly. > > This morning I came in to find that it had come all but a screeching > halt. The client wasn''t complaining (that I could find) about the NFS > server not responding, but the disk usage on the destination disk wasn''t > increasing at all that I could see. > > I had a window on this server machine, and I tried running some > commands, like dmesg, vmstat, iostat. They all hung with out running. > I''m thinking resource shortage. I went to the machine to try the > console, and I couldn''t get the monitor to even wake up. > > I hit Stop-A and sync''d it, and booted it back up. > > It had been running NV_32. For some reason that wouldn''t boot. But I had > live upgrade the other partition to NV_35 so I booted that instead. > > The NFS picked up where it had left off (I love NFS!) and I started a > ''vmstat 1'' (not the best choice I know.) to watch the machine. > > After a few minutes though I started seeing numbers so huge in the > vmstat output I couldn''t believe it. I''m no expert on what this stuff > means but a change like that seems unusual to me. > > What better tools should I use? > What other data should I capture?I would recommend to use "star -copy ..." But note that I recently fixed a FIFO deadlock problem that caused star to starve after a few GBs in case you run it on a fast enough multi-processor machine. Fixed in star-1.5a73 Intersting to see that I has no problems during more than 15 years, found myself the problen in early January and did receive a bug report one day before I did find a fix for the problem. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily