Mike.Li@schenker.ca
2002-Feb-19 07:22 UTC
Dump Analysis -- when NCR server frozen by rsync
Hi, rsync was running against filesystems /disk5 and /disk7 to back them onto remote server (172.16.101.4) using the following script: if [ `ps -ef | grep -v grep | grep ::d5 | /usr/bin/wc -l` -eq 0 ] then rm -f /etc/rsync5.log echo " --- Disk5 --- starts `date`" > /etc/rsync5.log /usr/local/bin/rsync -a --recursive --compress /disk5/ 172.16.101.4::d5 >> /etc/rsync.log echo "`cat /etc/rsync5.log` --- Disk5 --- Finishes `date`" >> /etc/rsync.log else echo "-- Disk5 rsync running `date`" >> /etc/rsync.log fi Here is the analysis of the freezing from the NCR engineer: Pls advise if any other's experienced this problem before. Thanks. SCL3550:/usr/local/bin>rsync --version rsync version 2.5.1 protocol version 25 Copyright (C) 1996-2001 by Andrew Tridgell and others <http://rsync.samba.org/> Capabilities: 32-bit files, socketpairs, hard links, symlinks, batchfiles, no I6 WARNING: no 64-bit integers on this platform! --- My dump analysis shows that the reason for the system hang is the 227 processes sleeping on locked ( vxfs ) inodes ( priority 94 ). Of these, 179 were sleeping on the vxfs inode c3fb4ac0, 37 on vxfs inode c43ba400, 3 on the vxfs inode c379eb81, 1 on the vxfs inode c368e280 , 3 on the vxfs inode c50add80, and 1 on the vxfs inode c3c06570. If we examine the vxfs inode c3bf4ac0, we note that the rowner is rsync. When we examine the stack, I noted that we are in DAP code. I suggest that we engage the third party vendor for rsync support. If you have any questions or concerns, please contact me. Best regards, Bea Beatrice Lyons-Daniels Solution Engineer 3325 Platt Springs Road West Columbia, SC 29170> *(803)-939-7845 V+ 633-7845 FAX: (803)-939-7707 > * beatrice.lyons-daniels@ncr.com >If I have helped somebody as I walked this land ... then my living ... then my living would not have been in vain. Dump Analysis --- 0> p ! nawk '{ if ($9==94) print }' 13 c4732800 s 1868 1792 1792 1792 2063 94 0 c3fb4ac1 cron load nwak 27 c3bd6000 s 1858 1850 1792 1792 0 94 0 c3fb4ac1 sh load nwak 28 c49de000 s 1845 1792 1792 1792 1577 94 0 c3fb4ac1 cron load nwak 32 c3bdc000 s 3501 1 692 692 1577 94 0 c3fb4ac1 plb load nwak 34 c47c5a00 s 1842 1841 1792 1792 0 94 0 c3fb4ac1 sh load nwak 40 c53df800 s 1881 1880 1792 1792 0 94 0 c3fb4ac1 sh load nwak 42 c38fc200 s 1838 1837 1792 1792 0 94 0 c3fb4ac1 sh load nwak 60 c3951800 s 738 1 737 737 0 94 0 c5b476c1 dcclpdser load nwak 64 c3257e00 s 1867 1866 1792 1792 0 94 0 c3fb4ac1 sh load nwak 68 c315ce00 s 3505 1 692 692 1577 94 0 c3fb4ac1 plb load nwak 71 c603ba00 s 1879 1877 1792 1792 0 94 0 c3fb4ac1 sh load nwak 76 c3baf600 s 3503 1 692 692 1577 94 0 c3fb4ac1 plb load nwak 79 c3153e00 s 3497 1 692 692 1577 94 0 c3fb4ac1 plb load nwak 85 c3c70a00 s 1876 1792 1792 1792 2063 94 0 c3fb4ac1 cron load nwak 87 c42c3e00 s 1884 1883 1792 1792 0 94 0 c3fb4ac1 sh load nwak 88 c4329c00 s 2131 2129 1792 1792 0 94 0 c3fb4ac1 sh load nwak 90 c30fd000 s 3495 1 692 692 1577 94 0 c3fb4ac1 plb load nwak 91 c5288e00 s 1844 1843 1792 1792 0 94 0 c3fb4ac1 sh load nwak 97 c3111e00 s 5114 1 5111 5111 0 94 0 c50add80 nfsd load nwak 99 c49b1000 s 1836 1833 1792 1792 0 94 0 c3fb4ac1 sh load nwak 101 c4022400 s 1898 1890 1792 1792 0 94 0 c3fb4ac1 sh load nwak 103 c401ac00 s 1864 1863 1792 1792 0 94 0 c3fb4ac1 sh load nwak 107 c3107200 s 3493 1 692 692 1577 94 0 c3fb4ac1 plb load nwak 108 c315cc00 s 3499 1 692 692 1577 94 0 c3fb4ac1 plb load nawk 109 c4563000 s 1887 1886 1792 1792 0 94 0 c3fb4ac1 sh load nwak 111 c4417600 s 28877 28872 692 692 0 94 0 c43ba401 sh load nwak 114 c3264400 s 5120 5114 5111 5111 0 94 0 c3c06570 nfsd load nwak 116 c30f2200 s 5123 5114 5111 5111 0 94 0 c50add80 nfsd load nwak 119 c3ba9000 s 5126 5114 5111 5111 0 94 0 c50add80 nfsd load nwak 120 c4102400 s 1937 1792 1792 1792 2063 94 0 c3fb4ac1 cron load nwak 122 c387c600 s 1895 1892 1792 1792 0 94 0 c3fb4ac1 sh load nwak 125 c446c200 s 1897 1894 1792 1792 0 94 0 c3fb4ac1 sh load nwak 129 c527e400 s 2000 1994 1792 1792 0 94 0 c3fb4ac1 sh load nwak 132 c4ec3600 s 1861 1860 1792 1792 0 94 0 c3fb4ac1 sh load nwak 138 c311f000 s 1854 1848 1792 1792 0 94 0 c3fb4ac1 sh load nwak 150 c42bd200 s 28879 28876 692 692 0 94 0 c43ba401 sh load nwak 167 c488fe00 s 1933 1932 1792 1792 0 94 0 c3fb4ac1 sh load nwak 169 c49e4200 s 2175 1792 1792 1792 2063 94 0 c3fb4ac1 cron load nwak 177 c2e00e00 s 28916 28910 692 692 0 94 0 c379eb81 sh load nwak 188 c39b3c00 s 20117 1 692 692 0 94 0 c43ba401 q-nite-jb load nwak 207 c60e5400 s 188 1 187 171 2108 94 0 c3fb4ac1 ksh load nwak 236 c3cec400 s 2145 2143 1792 1792 0 94 0 c3fb4ac1 sh load nwak 240 c4db2400 s 28878 28874 692 692 0 94 0 c43ba401 sh load nwak 250 c5d2e400 s 2053 1792 1792 1792 2063 94 0 c3fb4ac1 cron load nwak 262 c4e03000 s 29438 29437 692 692 0 94 0 c3fb4ac1 sh load nwak 275 c330d200 s 29918 1 692 692 1577 94 0 c3fb4ac1 cron load nwak 281 c5082000 s 1950 1948 1792 1792 0 94 0 c3fb4ac1 sh load nwak 313 c49c0800 s 1899 1896 1792 1792 0 94 0 c3fb4ac1 sh load nwak 339 c5f54400 s 2110 2106 1792 1792 0 94 0 c3fb4ac1 sh load nwak 340 c46aaa00 s 29924 29920 692 692 0 94 0 c3fb4ac1 sh load nwak 347 c5082400 s 29707 29706 692 692 0 94 0 c3fb4ac1 sh load nwak 348 c4852200 s 2036 1792 1792 1792 2063 94 0 c3fb4ac1 cron load nwak 354 c32da000 s 29478 29476 692 692 0 94 0 c3fb4ac1 sh load nwak 359 c4204800 s 29258 29256 692 692 0 94 0 c3fb4ac1 sh load nwak 375 c3cfe000 s 28362 28361 692 692 0 94 0 c43ba401 sh load nwak 394 c3908a00 s 28832 1 28832 28832 2108 94 0 c379eb81 ksh load nwak 413 c5f2d600 s 28563 28562 692 692 0 94 0 c43ba401 sh load nwak 428 c3207000 s 28847 28846 692 692 0 94 0 c43ba401 sh load nwak 429 c4c4ca00 s 29916 29910 692 692 0 94 0 c3fb4ac1 sh load nwak 449 c5eed000 s 28926 28921 692 692 0 94 0 c43ba401 sh load nwak 456 c4d80c00 s 2117 2116 1792 1792 0 94 0 c3fb4ac1 sh load nwak 457 c523da00 s 1905 1904 1792 1792 0 94 0 c3fb4ac1 sh load nwak 469 c53a7200 s 184 182 692 692 0 94 0 c3fb4ac1 sh load nwak 473 c414c400 s 29773 29767 692 692 0 94 0 c3fb4ac1 sh load nwak 486 c473fe00 s 2030 2027 1792 1792 0 94 0 c3fb4ac1 sh load nwak 501 c533b600 s 29518 29514 692 692 0 94 0 c3fb4ac1 sh load nwak 562 c4813c00 s 29818 29817 633 633 114 94 0 c368e281 rcp load nwak 569 c4401e00 s 1922 1916 1792 1792 0 94 0 c3fb4ac1 sh load nwak 579 c53aac00 s 2126 1792 1792 1792 2063 94 0 c3fb4ac1 cron load nwak 603 c41e6c00 s 1408 1 1408 1408 2114 94 0 c3fb4ac1 ksh load nwak 622 c475a800 s 29807 29806 692 692 0 94 0 c3fb4ac1 sh load nwak 642 c55fd200 s 2093 2092 1792 1792 0 94 0 c3fb4ac1 sh load nwak 650 c41cfe00 s 28350 28349 692 692 0 94 0 c43ba401 sh load nwak 660 c3a81800 s 2099 2098 1792 1792 0 94 0 c3fb4ac1 sh load nwak 670 c524f000 s 1945 1792 1792 1792 2063 94 0 c3fb4ac1 cron load nwak 698 c48b3c00 s 2101 2100 1792 1792 0 94 0 c3fb4ac1 sh load nwak 704 c4157200 s 29954 29951 692 692 0 94 0 c3fb4ac1 sh load nwak 711 c3bd4400 s 28165 28164 692 692 0 94 0 c43ba401 sh load nwak 713 c4301000 s 28917 28912 692 692 0 94 0 c43ba401 sh load nwak 751 c5316200 s 28330 28329 692 692 0 94 0 c43ba401 sh load nwak 791 c4901800 s 1969 1965 1792 1792 0 94 0 c3fb4ac1 sh load nwak 794 c649ae00 s 2163 2158 1792 1792 0 94 0 c3fb4ac1 sh load nwak 796 c49ad200 s 20119 1 692 692 0 94 0 c43ba401 e-nite-jb load nwak 833 c5f32800 s 1686 1 1686 1664 2114 94 0 c3fb4ac1 tail load nwak 845 c4286400 s 1823 1822 1792 1792 0 94 0 c3fb4ac1 sh load nwak 860 c491a200 s 29830 29829 692 692 0 94 0 c3fb4ac1 sh load nwak 894 c49f4000 s 28346 28341 692 692 0 94 0 c43ba401 sh load nwak 901 c317ca00 s 28932 28929 692 692 0 94 0 c43ba401 sh load nwak 911 c5d8bc00 s 29858 29855 692 692 0 94 0 c3fb4ac1 sh load nwak 914 c42f6600 s 1104 1 1104 1104 111 94 0 c3fb4ac1 ksh load nwak 916 c370dc00 s 1990 1985 1792 1792 0 94 0 c3fb4ac1 sh load nwak 922 c4247a00 s 1641 1640 692 692 0 94 0 c3fb4ac1 sh load nwak 930 c3c92200 s 148 145 692 692 0 94 0 c3fb4ac1 sh load nwak 948 c4bf2e00 s 29860 29859 692 692 0 94 0 c3fb4ac1 sh load nwak 951 c482e600 s 2002 1998 1792 1792 0 94 0 c3fb4ac1 sh load nwak 955 c5a4c000 s 53 1 692 692 2063 94 0 c3fb4ac1 cron load nwak 957 c3ff0600 s 2095 2094 1792 1792 0 94 0 c3fb4ac1 sh load nwak 969 c48ea200 s 1936 1935 1792 1792 0 94 0 c3fb4ac1 sh load nwak 978 c48f8a00 s 28316 28315 692 692 0 94 0 c43ba401 sh load nwak 982 c6287e00 s 28686 28679 692 692 0 94 0 c43ba401 sh load nwak 1009 c488ca00 s 29775 29774 692 692 0 94 0 c3fb4ac1 sh load nwak 1021 c4e12400 s 29913 29905 692 692 0 94 0 c3fb4ac1 sh load nwak 1024 c3c29800 s 29459 29454 692 692 0 94 0 c3fb4ac1 sh load nwak 1026 c3c68400 s 2059 2054 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1034 c3ecf000 s 2039 2038 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1051 c4b04c00 s 2124 2123 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1057 c56cae00 s 29997 1 692 692 2063 94 0 c3fb4ac1 cron load nwak 1065 c4105600 s 2168 2167 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1069 c5035c00 s 150 140 692 692 0 94 0 c3fb4ac1 sh load nwak 1077 c5673e00 s 115 114 692 692 0 94 0 c3fb4ac1 sh load nwak 1088 c40a5c00 s 1909 1792 1792 1792 1577 94 0 c3fb4ac1 cron load nwak 1098 c5dc5000 s 29677 29676 692 692 0 94 0 c3fb4ac1 sh load nwak 1105 c51e5000 s 1556 1554 692 692 0 94 0 c3fb4ac1 sh load nwak 1119 c5c6b600 s 28881 28880 692 692 0 94 0 c43ba401 sh load nwak 1130 c48de600 s 1617 1616 692 692 0 94 0 c3fb4ac1 sh load nwak 1146 c360d600 s 1022 1 1022 1022 111 94 0 c3fb4ac1 ksh load nwak 1151 c4060200 s 29417 1 29417 29417 2108 94 0 c3fb4ac1 ksh load nwak 1153 c3958c00 s 94 93 692 692 0 94 0 c3fb4ac1 sh load nwak 1156 c5829a00 s 29856 29851 692 692 0 94 0 c3fb4ac1 sh load nwak 1174 c3c41000 s 28671 28670 28669 28669 114 94 0 c43ba401 sh load nwak 1195 c4622400 s 2153 2152 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1201 c49ad000 s 1153 1 1153 1153 111 94 0 c3fb4ac1 ksh load nwak 1206 c426a000 s 1925 1920 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1209 c4bf1800 s 2142 2140 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1215 c3a92a00 s 224 1 692 692 1577 94 0 c3fb4ac1 cron load nwak 1239 c5847a00 s 20116 1 692 692 0 94 0 c43ba401 a-nite-jb load nwak 1266 c3ae9800 s 2080 2079 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1294 c32f3400 s 2035 2034 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1307 c4508800 s 28895 28894 692 692 0 94 0 c43ba401 sh load nwak 1312 c445e200 s 1052 1 1052 1052 111 94 0 c3fb4ac1 ksh load nwak 1323 c3e97c00 s 142 138 692 692 0 94 0 c3fb4ac1 sh load nwak 1374 c5b29e00 s 29180 29179 692 692 0 94 0 c43ba401 sh load nwak 1396 c4c4a800 s 2147 2146 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1397 c3139a00 s 1940 1939 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1411 c4cdb400 s 28348 28345 692 692 0 94 0 c43ba401 sh load nwak 1432 c3c98400 s 2083 2082 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1453 c3653800 s 1977 1975 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1468 c4e1e000 s 34 33 692 692 0 94 0 c3fb4ac1 sh load nwak 1480 c5982a00 s 1912 1908 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1500 c44b0200 s 29765 1 692 692 2063 94 0 c3fb4ac1 cron load nwak 1503 c616a800 s 200 1 200 200 2108 94 0 c3fb4ac1 ksh load nwak 1509 c5675600 s 29440 29439 633 633 114 94 0 c3fb4ac1 rcp load nwak 1531 c3f83800 s 2149 2148 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1533 c3366000 s 2137 2136 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1535 c325b000 s 20118 1 692 692 0 94 0 c43ba401 b-nite-jb load nwak 1544 c5e24400 s 28347 28343 692 692 0 94 0 c43ba401 sh load nwak 1561 c4808200 s 28589 28586 692 692 0 94 0 c43ba401 sh load nwak 1583 c48a7a00 s 28398 28396 692 692 0 94 0 c43ba401 sh load nwak 1599 c422b600 s 29275 1 692 692 2063 94 0 c3fb4ac1 cron load nwak 1611 c3ab5e00 s 29509 1 692 692 1577 94 0 c3fb4ac1 cron load nwak 1644 c5f94a00 s 2165 2162 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1650 c42be400 s 1871 1870 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1660 c3586c00 s 1927 1926 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1673 c4669c00 s 29513 29508 692 692 0 94 0 c3fb4ac1 sh load nwak 1680 c4286e00 s 28739 1 28739 28739 2108 94 0 c5a885c0 ksh load nwak 1704 c3857e00 s 29463 29462 692 692 0 94 0 c3fb4ac1 sh load nwak 1712 c45dd800 s 2155 2154 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1721 c4eb2800 s 29857 29853 692 692 0 94 0 c3fb4ac1 sh load nwak 1730 c47f1a00 s 1834 1829 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1776 c4591200 s 29210 29200 692 692 0 94 0 c43ba401 sh load nwak 1782 c3bdf200 s 28574 28573 692 692 0 94 0 c43ba401 sh load nwak 1793 c4835600 s 1775 1 1765 1732 0 94 0 fe7fdce0 reboot load nwak 1797 c5c6b000 s 2089 2086 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1800 c359e200 s 222 220 692 692 0 94 0 c3fb4ac1 sh load nwak 1801 c48c5a00 s 2108 2104 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1805 c39e3200 s 2171 2170 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1808 c40d5200 s 1957 1956 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1824 c46e6200 s 149 147 692 692 0 94 0 c3fb4ac1 sh load nwak 1825 c46d2800 s 230 229 692 692 0 94 0 c3fb4ac1 sh load nwak 1828 c3023200 s 29414 29413 692 692 0 94 0 c3fb4ac1 sh load nwak 1840 c4246600 s 1960 1959 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1841 c4ebfa00 s 29280 29272 692 692 0 94 0 c3fb4ac1 sh load nwak 1845 c3307600 s 1978 1976 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1849 c475c800 s 29884 29882 692 692 0 94 0 c3fb4ac1 sh load nwak 1851 c4da0600 s 1949 1946 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1867 c4951200 s 29517 29511 692 692 0 94 0 c3fb4ac1 sh load nwak 1872 c3161600 s 1943 1942 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1879 c40eb000 s 29129 29128 692 692 0 94 0 c43ba401 sh load nwak 1900 c32ddc00 s 2121 2120 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1914 c3299200 s 1835 1831 1792 1792 0 94 0 c3fb4ac1 sh load nwak 1923 c4cf1000 s 29921 29917 692 692 0 94 0 c3fb4ac1 sh load nwak 1933 c38d2200 s 29978 29977 692 692 0 94 0 c3fb4ac1 sh load nwak 1982 c5fa3e00 s 2118 1792 1792 1792 2063 94 0 c3fb4ac1 cron load nwak 2020 c3964400 s 29477 29475 692 692 0 94 0 c3fb4ac1 sh load nwak 2027 c4945600 s 29259 29257 692 692 0 94 0 c3fb4ac1 sh load nwak 2031 c4c88800 s 1972 1971 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2034 c3aa6800 s 2004 2003 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2037 c585e400 s 2111 2109 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2043 c365de00 s 2134 2133 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2055 c3cfc600 s 1970 1967 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2058 c4171800 s 1874 1873 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2060 c4505e00 s 28927 28923 692 692 0 94 0 c379eb81 sh load nwak 2066 c4137600 s 13 29999 692 692 0 94 0 c3fb4ac1 sh load nwak 2067 c3f8b600 s 2090 2088 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2089 c4ac9000 s 29704 1 692 692 2063 94 0 c3fb4ac1 cron load nwak 2095 c54c1c00 s 2044 2042 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2119 c606dc00 s 29648 29647 692 692 0 94 0 c3fb4ac1 sh load nwak 2128 c486a200 s 29736 29735 692 692 0 94 0 c3fb4ac1 sh load nwak 2131 c48e8400 s 228 225 692 692 0 94 0 c3fb4ac1 sh load nwak 2141 c416ee00 s 1562 1560 692 692 0 94 0 c3fb4ac1 sh load nwak 2148 c4758c00 s 28387 28384 692 692 0 94 0 c43ba401 sh load nwak 2161 c3ffec00 s 2164 2160 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2167 c48ba600 s 2174 2173 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2169 c3a74c00 s 28364 28363 692 692 0 94 0 c43ba401 sh load nwak 2177 c4e6e600 s 1988 1983 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2178 c3165a00 s 2105 1792 1792 1792 1577 94 0 c3fb4ac1 cron load nwak 2189 c4b06200 s 1903 1902 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2192 c48de400 s 29460 29456 692 692 0 94 0 c3fb4ac1 sh load nwak 2202 c400f400 s 28861 28860 692 692 0 94 0 c43ba401 sh load nwak 2203 c60e0a00 s 227 223 692 692 0 94 0 c3fb4ac1 sh load nwak 2213 c59e2e00 s 29461 29458 692 692 0 94 0 c3fb4ac1 sh load nwak 2224 c3d58c00 s 56 55 692 692 0 94 0 c3fb4ac1 sh load nwak 2240 c39b3e00 s 29125 29123 692 692 0 94 0 c43ba401 sh load nwak 2252 c4c75a00 s 28893 28892 692 692 0 94 0 c43ba401 sh load nwak 2268 c3fcd000 s 28957 28933 692 692 0 94 0 c43ba401 sh load nwak 2275 c3c09000 s 28651 28650 692 692 0 94 0 c43ba401 sh load nwak 2287 c3a39000 s 2045 2043 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2289 c4d71c00 s 29883 29881 692 692 0 94 0 c3fb4ac1 sh load nwak 2290 c49a6000 s 2130 2127 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2308 c367d200 s 185 183 692 692 0 94 0 c3fb4ac1 sh load nwak 2314 c37e6400 s 29287 29284 692 692 0 94 0 c3fb4ac1 sh load nwak 2322 c5910200 s 2114 2113 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2332 c486e600 s 2161 1792 1792 1792 1577 94 0 c3fb4ac1 cron load nwak 2346 c3632a00 s 2061 2058 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2386 c48d2800 s 1968 1963 1792 1792 0 94 0 c3fb4ac1 sh load nwak 2396 c3bdfa00 s 1999 1792 1792 1792 1577 94 0 c3fb4ac1 cron load nwak 2397 c45d4000 s 2032 2031 1792 1792 0 94 0 c3fb4ac1 sh load nwak 0> p ! nawk '{ if ( $9==94 ) print }' | wc -l 227 <<< Total on the system 0> p ! nawk '{ if ( $9==94 ) print }' | grep c3fb4ac1 | wc -l 179 0> t 2397 STACK TRACE FOR PROCESS 2397: STKADDR FRAMEPTR FUNCTION POSSIBLE ARGUMENTS fe000ad0 fe000b24 00000000 resume (c3fb4ac1,a,c3fb4c68,0) fe000b2c fe000b84 00000000 vx_irwlock+0x55 (c3fb4c68,fe000bf0,fe000cf4,fe000da4) fe000b8c fe000bb0 00000000 vop_lookup+0x4a (c3fb4c68,fe000bf0,fe000cf4,fe000da4) fe000bb8 fe000d28 00000000 lookuppn+0x1ac (fe000da4,1,0,fe000dc0) fe000d30 fe000dd0 00000000 exece+0xbd (fe0010ac,fe000dfc,3,fe6cb8ae) fe000dd8 fe000e24 00000000 systrap+0x116 (fe000e34,0) fe000e34 sys_call+0x3f from 08060314 ax: 3b cx: 0 dx: 806a06e bx: 806a058 fl: 246 ds: 1f fs: 0 sp:fe000e64 bp: 0 si: 8069fc4 di: 1 err: 0 es: 1f gs: 0 0> vxi -f c3fb4ac0 VXFS INODE TABLE SIZE = 5396 SLOT MAJ/MIN INUMB RCNT LINK UID GID SIZE MODE FLAGS 2220 65,7 2 187 29 0 0 2048 d---777 rw FORW BACK AFOR ABCK MAPSZ OLDSZ - 3732 - - 0 OWNER RWOWNER COUNT NEXTR IWANT IWANTRW IWANTG IWANTP 306 297 0 0 0 179 0 0 VNODE : VCNT VFSMNTED VFSP STREAMP VTYPE RDEV VDATA VFILOCKS VFLAG 187 0 c3cf7900 0 d - c3fb4ac0 0 root v_lck mutex lock: 0 type: excl sleep recur mtx sleep_on_busy: 0 flags: 0 db_flags: inited spinners: 0 processor: 0xffffffff use_count: 0 wake_one: 0 wake_all: 0 mpdebug: 0 locker: 0 unlocker: 0 v_piocount: 0 0> p 297 PROC TABLE SIZE = 3072 SLOT ADDR ST PID PPID PGID SID UID PRI CPU EVENT NAME FLAGS 297 c46ab400 s 29228 1 692 692 0 84 0 fe97da80 rsync load nwak 0> tr 297 STACK TRACE FOR PROCESS 297: STKADDR FRAMEPTR FUNCTION POSSIBLE ARGUMENTS fe0009ec fe000a40 00000000 resume (fe97da80,14,fe964628,c3afae74) fe000a48 fe000a84 00000000 vdstrategy2+0x26f (c3afae74,800,0,c3afae74) fe000a8c fe000ab8 00000000 vdstrategy+0xbe (c3c06400,c3afae74,800,0) fe000ac0 fe000b44 00000000 vx_breada+0x56 (c3c06400,3,fe000b84,0) fe000b4c fe000b94 00000000 vx_dirlook+0x184 (c3fb4ac0,fe000c44,fe000bc4,c4e5d300) fe000b9c fe000bd8 00000000 vx_lookup+0x18d (c3fb4c68,fe000c44,fe000d48,fe000d9c) fe000be0 fe000c04 00000000 vop_lookup+0x4a (c3fb4c68,fe000c44,fe000d48,fe000d9c) fe000c0c fe000d7c 00000000 lookuppn+0x1ac (fe000d9c,0,0,fe000dcc) fe000d84 fe000da8 00000000 lookupname+0x46 (8045c78,0,0,0) fe000db0 fe000dd0 00000000 lxstat+0x2c (fe0010ac,fe000dfc,3,c26fed00) fe000dd8 fe000e24 00000000 systrap+0x116 (fe000e34,0) fe000e34 sys_call+0x3f from bffb528c ax: 7c cx: 0 dx: 807d6cc bx: 807e8f0 fl: 246 ds: 1f fs: 0 sp:fe000e64 bp: 8045830 si: 8045c78 di: 807d6cc err: 0 es: 1f gs: 0 0 0> p ! nawk '{ if ( $9==94 ) print }' | grep c43ba401 | wc -l 37 0> t 2252 STACK TRACE FOR PROCESS 2252: STKADDR FRAMEPTR FUNCTION POSSIBLE ARGUMENTS fe000ad0 fe000b24 00000000 resume (c43ba401,a,c43ba5a8,0) fe000b2c fe000b84 00000000 vx_irwlock+0x55 (c43ba5a8,fe000bf0,fe000cf4,fe000da4) fe000b8c fe000bb0 00000000 vop_lookup+0x4a (c43ba5a8,fe000bf0,fe000cf4,fe000da4) fe000bb8 fe000d28 00000000 lookuppn+0x1ac (fe000da4,1,0,fe000dc0) fe000d30 fe000dd0 00000000 exece+0xbd (fe0010ac,fe000dfc,3,c26fed00) fe000dd8 fe000e24 00000000 systrap+0x116 (fe000e34,0) fe000e34 sys_call+0x3f from 08060314 ax: 3b cx: 0 dx: 806a0a0 bx: 806a088 fl: 246 ds: 1f fs: 0 sp:fe000e64 bp: 0 si: 8069ff0 di: 1 err: 0 es: 1f gs: 0 0> vxi -f c43ba400 VXFS INODE TABLE SIZE = 5396 SLOT MAJ/MIN INUMB RCNT LINK UID GID SIZE MODE FLAGS 2643 65,7 1973373 42 41 108 1 45056 d---777 rw FORW BACK AFOR ABCK MAPSZ OLDSZ - 543 - - 0 OWNER RWOWNER COUNT NEXTR IWANT IWANTRW IWANTG IWANTP 1776 1330 0 0 0 37 0 0 VNODE : VCNT VFSMNTED VFSP STREAMP VTYPE RDEV VDATA VFILOCKS VFLAG 42 0 c3cf7900 0 d - c43ba400 0 v_lck mutex lock: 0 type: excl sleep recur mtx sleep_on_busy: 0 flags: 0 db_flags: inited spinners: 0 processor: 0xffffffff use_count: 0 wake_one: 0 wake_all: 0 mpdebug: 0 locker: 0 unlocker: 0 v_piocount: 0 0> p 1330 PROC TABLE SIZE = 3072 SLOT ADDR ST PID PPID PGID SID UID PRI CPU EVENT NAME FLAGS 1330 c4418e00 s 28162 28160 692 692 0 84 0 fe97da80 sh load nwak -------------- next part -------------- HTML attachment scrubbed and removed
You're saying the whole system froze? I don't see how that can be caused by an application process. It sounds like from the report of the NCR engineer that a lot of applications were waiting on a lock to be released by rsync, but why would they be? If you had killed the rsync process, would it have allowed the system to free up? - Dave Dykstra On Mon, Feb 18, 2002 at 03:22:41PM -0500, Mike.Li@schenker.ca wrote:> Hi, > rsync was running against filesystems /disk5 and /disk7 to back them onto > remote server (172.16.101.4) using the following script: > if [ `ps -ef | grep -v grep | grep ::d5 | /usr/bin/wc -l` -eq 0 ] > then > rm -f /etc/rsync5.log > echo " --- Disk5 --- starts `date`" > /etc/rsync5.log > /usr/local/bin/rsync -a --recursive --compress /disk5/ 172.16.101.4::d5 >> > /etc/rsync.log > echo "`cat /etc/rsync5.log` --- Disk5 --- Finishes `date`" >> > /etc/rsync.log > else > echo "-- Disk5 rsync running `date`" >> /etc/rsync.log > fi > > Here is the analysis of the freezing from the NCR engineer: > Pls advise if any other's experienced this problem before. Thanks. > > SCL3550:/usr/local/bin>rsync --version > rsync version 2.5.1 protocol version 25 > Copyright (C) 1996-2001 by Andrew Tridgell and others > <http://rsync.samba.org/> > Capabilities: 32-bit files, socketpairs, hard links, symlinks, batchfiles, > no I6 > > WARNING: no 64-bit integers on this platform! > > --- > My dump analysis shows that the reason for the system hang is the 227 > processes sleeping on locked ( vxfs ) inodes ( priority 94 ). Of these, > 179 > were sleeping on the vxfs inode c3fb4ac0, 37 on vxfs inode c43ba400, 3 on > the vxfs inode c379eb81, 1 on the vxfs inode c368e280 , 3 on the vxfs > inode > c50add80, and 1 on the vxfs inode c3c06570. If we examine the vxfs inode > c3bf4ac0, we note that the rowner is rsync. When we examine the stack, I > noted that we are in DAP code. I suggest that we engage the third party > vendor for rsync support. If you have any questions or concerns, please > contact me. > Best regards, > Bea > Beatrice Lyons-Daniels > Solution Engineer > 3325 Platt Springs Road > West Columbia, SC 29170 > > *(803)-939-7845 V+ 633-7845 FAX: (803)-939-7707 > > * beatrice.lyons-daniels@ncr.com > >
Mike.Li@schenker.ca
2002-Mar-05 01:26 UTC
Dump Analysis -- when NCR server frozen by rsync
Hi,
I've recently experienced some griefs with rsync for svr4 running on NCR
MP-RAS. It locks files (cobol files), corrupting them and interfere with
Netvault backups. Has anyone experience this before?
Best regards,
Mike Li
Dave Dykstra <dwd@bell-labs.com>
02/21/02 12:00 PM
To: Mike.Li@schenker.ca
cc: rsync@lists.samba.org
Subject: Re: Dump Analysis -- when NCR server frozen by rsync
You're saying the whole system froze? I don't see how that can be
caused
by an application process. It sounds like from the report of the NCR
engineer that a lot of applications were waiting on a lock to be released
by rsync, but why would they be? If you had killed the rsync process,
would it have allowed the system to free up?
- Dave Dykstra
On Mon, Feb 18, 2002 at 03:22:41PM -0500, Mike.Li@schenker.ca
wrote:> Hi,
> rsync was running against filesystems /disk5 and /disk7 to back them
onto> remote server (172.16.101.4) using the following script:
> if [ `ps -ef | grep -v grep | grep ::d5 | /usr/bin/wc -l` -eq 0 ]
> then
> rm -f /etc/rsync5.log
> echo " --- Disk5 --- starts `date`" > /etc/rsync5.log
> /usr/local/bin/rsync -a --recursive --compress /disk5/ 172.16.101.4::d5
>>
> /etc/rsync.log
> echo "`cat /etc/rsync5.log` --- Disk5 --- Finishes `date`"
>>
> /etc/rsync.log
> else
> echo "-- Disk5 rsync running `date`" >> /etc/rsync.log
> fi
>
> Here is the analysis of the freezing from the NCR engineer:
> Pls advise if any other's experienced this problem before. Thanks.
>
> SCL3550:/usr/local/bin>rsync --version
> rsync version 2.5.1 protocol version 25
> Copyright (C) 1996-2001 by Andrew Tridgell and others
> <http://rsync.samba.org/>
> Capabilities: 32-bit files, socketpairs, hard links, symlinks,
batchfiles,> no I6
>
> WARNING: no 64-bit integers on this platform!
>
> ---
> My dump analysis shows that the reason for the system hang is the 227
> processes sleeping on locked ( vxfs ) inodes ( priority 94 ). Of these,
> 179
> were sleeping on the vxfs inode c3fb4ac0, 37 on vxfs inode c43ba400, 3
on> the vxfs inode c379eb81, 1 on the vxfs inode c368e280 , 3 on the vxfs
> inode
> c50add80, and 1 on the vxfs inode c3c06570. If we examine the vxfs
inode> c3bf4ac0, we note that the rowner is rsync. When we examine the stack,
I> noted that we are in DAP code. I suggest that we engage the third party
> vendor for rsync support. If you have any questions or concerns, please
> contact me.
> Best regards,
> Bea
> Beatrice Lyons-Daniels
> Solution Engineer
> 3325 Platt Springs Road
> West Columbia, SC 29170
> > *(803)-939-7845 V+ 633-7845 FAX: (803)-939-7707
> > * beatrice.lyons-daniels@ncr.com
> >
-------------- next part --------------
HTML attachment scrubbed and removed
Apparently Analagous Threads
- : use of Error() for repeated measures with more than 2 factors
- : Bug in Error() and the use of Error() for repeated measures with more than 2 fa ctors
- [Gluster-devel] gfid and volume-id extended attributes lost
- Reg: Porting UFS/VxFs to ext2 (fwd)
- [Gluster-devel] gfid and volume-id extended attributes lost