Re: Hot rb_next, setup_cluster_no_bitmap

Simon Kirby
2011-08-04T01:32:10Z

Perhaps as a further clue as to what is going on, on this same backup box
after all of the rsyncs are finished/killed and a good amount of time has
passed (no cleaner processes running in the background or anything),
"sync" is still consistently takes ~4 minutes to run, and pushes out a
lot to disk every time it is run. Example:

echo 3 > /proc/sys/vm/drop_caches
sync
echo 3 > /proc/sys/vm/drop_caches
sync
vmstat 1 &
time sync

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0     68 15656528   3660 258232    0    0   783   673   72   33  3 30 13 54
 0  0     68 15656612   3660 258232    0    0     0     0  439  152  0  0 100  0
 0  0     68 15656488   3660 258232    0    0     0     0  395   87  0  0 100  0
 0  0     68 15656488   3660 258232    0    0     0     0  309   89  0  0 100  0
 0  0     68 15655992   3660 258232    0    0     0     0  450  128  0  0 100  0
 0  0     68 15655984   3660 258232    0    0     0     0  446  159  0  1 99  0
 0  0     68 15655860   3660 258232    0    0     0     0  448  105  0  0 100  0
 0  0     68 15655720   3660 258276    0    0     0     0  824  238  1  2 98  0
 1  0     68 15651388   3660 258276    0    0     0     0  621  150  2  0 98  0
 0  0     68 15655860   3660 258276    0    0     0    44  886  236  2  1 98  0
 0  0     68 15656604   3660 258276    0    0     0     0  544  161  0  1 99  0
 0  0     68 15656612   3660 258276    0    0     0     0  616  328  0  1 99  0
<sync started here>
 0  2     68 15655764   3660 258420    0    0   328   607 1777 1450  0  1 74 25
 0  1     68 15654648   3660 259968    0    0   752     0 1978 1519  0  1 66 33
 0  1     68 15654028   3660 260520    0    0   616     0 1498 1186  0  0 75 25
 0  1     68 15653556   3660 260992    0    0   220  1545 1878 1937  0  1 75 24
 1  0     68 15652288   3660 262540    0    0   392  1976 2990 3072  0  4 75 21
 1  0     68 15652252   3660 262496    0    0     0     0 2848  164  0 27 73  0
 1  0     68 15652260   3660 262496    0    0     0     0 2586   86  0 26 74  0
 1  0     68 15652260   3660 262496    0    0     0     0 2591  148  0 25 75  0
 1  0     68 15652136   3660 262496    0    0     0     0 2544   98  0 24 76  0
 1  0     68 15652136   3660 262496    0    0     0     0 2518   75  0 26 74  0
 1  0     68 15652136   3660 262496    0    0     0     0 2676  105  0 25 75  0
 1  0     68 15652136   3660 262496    0    0     0     0 2531   83  0 25 75  0
 1  0     68 15652136   3660 262496    0    0     0     0 2595   81  0 25 75  0
 1  0     68 15652136   3660 262496    0    0     0     0 2570   89  0 25 75  0
 1  0     68 15652136   3660 262496    0    0     0     0 2539   76  0 25 75  0
 1  0     68 15652004   3660 262540    0    0     0     0 2914  166  0 25 74  0
 1  0     68 15652012   3660 262540    0    0     0     0 2596   87  0 25 75  0
 1  0     68 15652012   3660 262540    0    0     0     0 2591   82  0 25 75  0
 1  0     68 15652012   3660 262540    0    0     0     0 2607   90  0 25 75  0
 1  0     68 15652012   3660 262540    0    0     0     0 2535   89  0 25 75  0
 1  0     68 15652012   3660 262540    0    0     0     0 2629  109  0 26 74  0
 1  0     68 15652012   3660 262540    0    0     0     0 2549   81  0 25 75  0
 1  0     68 15652012   3660 262540    0    0     0     0 2757  230  0 26 73  0
 1  0     68 15652012   3660 262540    0    0     0     0 2571  105  0 24 76  0
 1  0     68 15652012   3660 262540    0    0     0     0 2568   96  0 26 74  0
 1  0     68 15651880   3660 262584    0    0     0     0 2930  173  0 28 72  0
 1  0     68 15651888   3660 262584    0    0     0     0 2564   79  0 26 74  0
 1  0     68 15651888   3660 262584    0    0     0     0 2594   84  0 22 78  0
 1  0     68 15651888   3660 262584    0    0     0     0 2568   96  0 25 75  0
 1  0     68 15651888   3660 262584    0    0     0     0 2578   91  0 25 75  0
 1  0     68 15651888   3660 262584    0    0     0     0 2660  104  0 26 74  0
 1  0     68 15651888   3660 262584    0    0     0     0 2537   84  0 25 75  0
 1  0     68 15651888   3660 262584    0    0     0     0 2553   82  0 25 75  0
 1  0     68 15651888   3660 262584    0    0     0    39 2808  204  0 26 74  0
 1  0     68 15651888   3660 262584    0    0     0     0 2573   91  0 25 75  0
 1  0     68 15651756   3660 262628    0    0     0    44 2868  153  1 29 71  0
 1  0     68 15651764   3660 262628    0    0     0     0 2569   79  0 23 76  0
 1  0     68 15651764   3660 262628    0    0     0     0 2587   79  0 27 73  0
 1  0     68 15651764   3660 262628    0    0     0     0 2509   73  0 23 77  0
 1  0     68 15651764   3660 262628    0    0     0     0 2520   81  0 25 75  0
 1  0     68 15651764   3660 262628    0    0     0     0 2664   97  0 25 75  0
 1  0     68 15651740   3660 262628    0    0   112     0 2680  146  0 25 75  0
 1  0     68 15651640   3660 262740    0    0     0     0 2627   79  0 25 75  0
 1  0     68 15651640   3660 262740    0    0     0     0 2534   83  0 25 75  0
 1  0     68 15651640   3660 262740    0    0     0     0 2525   85  0 25 75  0
 1  0     68 15651756   3660 262784    0    0     0    44 3059  307  0 27 72  0
 1  0     68 15651764   3660 262784    0    0     0     0 2593   73  0 29 71  0
 1  0     68 15651764   3660 262784    0    0     0     0 2573   82  0 22 78  0
 1  0     68 15651764   3660 262784    0    0     0     0 2567   84  0 26 74  0
 1  0     68 15651764   3660 262784    0    0     0     0 2577   90  0 25 75  0
 1  0     68 15651764   3660 262784    0    0     0     0 2675   86  0 25 75  0
 1  0     68 15651764   3660 262784    0    0     0     0 2577   91  0 25 75  0
 1  0     68 15651764   3660 262784    0    0     0     0 2610   86  0 25 75  0
 1  0     68 15651764   3660 262784    0    0     0     0 2558   88  0 25 75  0
 1  0     68 15651764   3660 262784    0    0     0     0 2537   83  0 25 75  0
 1  0     68 15651252   3660 262832    0    0     0    44 2959  181  0 26 74  0
 1  0     68 15651268   3660 262832    0    0     0     0 2572   88  0 25 75  0
 1  0     68 15651764   3660 262832    0    0     0     0 3136  138  3 25 72  0
 1  0     68 15651764   3660 262832    0    0     0     0 2525   70  0 29 71  0
 1  0     68 15651764   3660 262832    0    0     0     0 2584   96  0 22 78  0
 1  0     68 15651640   3660 262832    0    0     0     0 2703  106  0 25 75  0
 1  0     68 15651640   3660 262832    0    0     0     0 2572  101  0 25 75  0
 1  0     68 15651516   3660 262832    0    0     0     0 2588   75  0 25 75  0
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  0     68 15651516   3660 262832    0    0     0    79 2624  110  0 25 75  0
 1  0     68 15651516   3660 262832    0    0     0     0 2555   85  0 25 75  0
 1  0     68 15651136   3660 262876    0    0     0    44 2858  159  0 25 74  0
 1  0     68 15651144   3660 262876    0    0     0     0 2560   86  0 25 75  0
 1  0     68 15651144   3660 262876    0    0     0     0 2643  138  0 25 75  0
 1  0     68 15651020   3660 262876    0    0     0     0 2818  291  0 26 74  0
 1  0     68 15651020   3660 262876    0    0     0     0 2542   89  0 25 75  0
 1  0     68 15650896   3660 262876    0    0     0     0 2645  111  0 25 75  0
 1  0     68 15650896   3660 262876    0    0     0     0 2535   87  0 25 75  0
 1  0     68 15650896   3660 262876    0    0     0     0 2562   84  0 25 75  0
 1  0     68 15650896   3660 262876    0    0     0     0 2533   85  0 25 75  0
 1  0     68 15650896   3660 262876    0    0     0     0 2534   92  0 25 75  0
 1  0     68 15650888   3660 262920    0    0     0    44 2881  158  0 26 74  0
 1  0     68 15650772   3660 262920    0    0     0     0 2585   84  0 25 75  0
 1  0     68 15650772   3660 262920    0    0     0     0 2590   82  0 25 75  0
 1  0     68 15650772   3660 262920    0    0     0     0 2565   73  0 25 75  0
 1  0     68 15650772   3660 262920    0    0     0     0 2577   82  0 25 75  0
 1  0     68 15650772   3660 262920    0    0     0     0 2655  102  0 25 75  0
 1  0     68 15650772   3660 262920    0    0     0     0 2548   87  0 25 75  0
 1  0     68 15650772   3660 262920    0    0     0     0 2569   80  0 29 71  0
 1  0     68 15650648   3660 262920    0    0     0     0 2569   83  0 22 78  0
 1  0     68 15650648   3660 262920    0    0     0     0 2574   92  0 25 75  0
 1  0     68 15650516   3660 262964    0    0     0     0 2944  155  0 26 73  0
 1  0     68 15650524   3660 262964    0    0     0     0 2555   98  0 25 75  0
 1  0     68 15650524   3660 262964    0    0     0     0 2584   98  0 25 75  0
 1  0     68 15650524   3660 262964    0    0     0     0 2549   84  0 25 75  0
 1  0     68 15650524   3660 262964    0    0     0     0 2549   85  0 25 75  0
 1  0     68 15650400   3660 262964    0    0     0    56 2694  102  0 25 75  0
 2  0     68 15650400   3660 262964    0    0     0     0 2650  157  0 25 75  0
 1  0     68 15650400   3660 262964    0    0     0     0 2717  220  0 26 74  0
 1  0     68 15650400   3660 262964    0    0     0    53 2588  106  0 25 75  0
 1  0     68 15650400   3660 262964    0    0     0     0 2574   75  0 25 75  0
 1  0     68 15650516   3660 263008    0    0     0    44 3056  247  1 27 73  0
 1  0     68 15650524   3660 263008    0    0     0     0 2530   95  0 26 74  0
 1  0     68 15650400   3660 263008    0    0     0     0 2570   83  0 24 76  0
 1  0     68 15650400   3660 263008    0    0     0     0 2538   81  0 27 73  0
 1  0     68 15650400   3660 263008    0    0     0     0 2545   87  0 23 77  0
 1  0     68 15650400   3660 263008    0    0     0     0 2661   99  0 26 74  0
 1  0     68 15650400   3660 263008    0    0     0     0 2568   79  0 25 75  0
 1  0     68 15650400   3660 263008    0    0     0     0 2551   72  0 25 75  0
 1  0     68 15650400   3660 263008    0    0     0     0 2537   83  0 25 75  0
 1  0     68 15650276   3660 263008    0    0     0     0 2539   98  0 25 75  0
 1  0     68 15650144   3660 263052    0    0     0    44 2868  163  0 25 74  0
 1  0     68 15650152   3660 263052    0    0     0     0 2572   82  0 25 75  0
 1  0     68 15650152   3660 263052    0    0     0     0 2579   85  0 25 75  0
 0  2     68 15648672   3660 263052    0    0     0   776 2510  359  0 21 69 10
 0  2     68 15648680   3660 263052    0    0     0  2728 3182 3256  0  1 62 37
 0  2     68 15648804   3660 263052    0    0     0  2608 3274 3240  0  0 64 36
 0  2     68 15648804   3660 263052    0    0     0  4140 4235 4838  0  1 63 36
 0  2     68 15648804   3660 263052    0    0     0  3056 3958 3869  0  1 51 48
 0  2     68 15648804   3660 263052    0    0     0  3792 4189 4663  0  0 50 50
 0  2     68 15648804   3660 263052    0    0     0  4880 5483 6341  0  1 52 47
 0  2     68 15649028   3668 263100    0    0     8  4300 5806 5987  2  6 51 42
 0  2     68 15648648   3668 263100    0    0     0  5016 6278 6481  4  1 51 43
 0  2     68 15648680   3668 263100    0    0     0  4712 5037 5670  0  1 50 50
 1  2     68 15648804   3668 263100    0    0     0  8432 7785 10086  0  1 57 42
 0  2     68 15649052   3668 263100    0    0     0 10256 8857 11715  0  2 54 44
 1  0     68 15650416   3684 263092    0    0     0  9344 9150 11329  0 11 64 25
 1  0     68 15650556   3684 263100    0    0     0     0 2597   87  0 28 72  0
 1  0     68 15650556   3684 263100    0    0     0     0 2584   85  0 26 74  0
 1  0     68 15650556   3684 263100    0    0     0   105 2627  178  0 23 77  0
 1  0     68 15650556   3684 263100    0    0     0     0 2573  107  0 24 76  0
 1  0     68 15650548   3684 263144    0    0     0    44 2856  164  0 25 74  0
 1  0     68 15650556   3684 263144    0    0     0     0 2538   85  0 27 73  0
 1  0     68 15650556   3684 263144    0    0     0     0 2650   90  0 27 73  0
 1  0     68 15650680   3684 263144    0    0     0     0 2682  186  0 22 78  0
 1  0     68 15650080   3684 263144    0    0     0     0 2794  193  0 25 75  0
 1  0     68 15649664   3684 263144    0    0     0     0 3080  253  1 25 74  0
 1  0     68 15649804   3684 263144    0    0     0     0 2529   84  0 25 75  0
 1  0     68 15649804   3684 263144    0    0     0     0 2558   79  0 25 75  0
 1  0     68 15649804   3684 263144    0    0     0     0 2548   80  0 25 75  0
 1  0     68 15649740   3684 263144    0    0     0     0 2648  141  0 25 75  0
 1  0     68 15649548   3684 263188    0    0     0    44 3198  507  0 28 72  0
 1  0     68 15649132   3688 263184    0    0   124     0 2966  291  0 25 74  0
 1  0     68 15649292   3688 263308    0    0     0     0 2984  144  1 26 74  0
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 3  0     68 15649184   3688 263308    0    0     0     0 3593  237  0 35 65  0
 2  0     68 15649184   3688 263308    0    0     0    16 5052   83  0 50 50  0
 2  0     68 15649152   3688 263308    0    0     0     0 5313  159  1 51 48  0
 2  0     68 15649028   3688 263308    0    0     0     0 5908  702  2 52 47  0
 4  0     68 15648948   3696 263300    0    0     0    32 5836  861  2 53 45  0
 2  0     68 15643968   3696 263308    0    0     0     0 5901  293  6 52 42  0
 2  0     68 15643976   3696 263308    0    0     0     0 5016   87  0 50 50  0
 2  0     68 15643968   3696 263352    0    0     0    48 5285  145  0 51 49  0
 2  0     68 15643976   3696 263352    0    0     0     0 4956   79  0 50 50  0
 2  0     68 15643976   3696 263352    0    0     0     0 4994   85  0 50 50  0
 2  0     68 15643976   3704 263344    0    0     0    32 5014   91  0 50 50  0
 2  0     68 15643976   3704 263352    0    0     0     0 5016   77  0 50 50  0
 2  0     68 15643976   3704 263352    0    0     0     0 5070   90  0 51 49  0
 2  0     68 15643976   3704 263352    0    0     0     0 5011   83  0 50 50  0
 2  0     68 15643976   3704 263352    0    0     0     0 4961   84  0 49 51  0
 2  0     68 15650176   3704 263352    0    0     0    62 5005  120  0 50 50  0
 2  0     68 15650184   3704 263352    0    0     0     0 4949   85  0 51 49  0
 2  0     68 15650052   3704 263396    0    0     0    60 5238  142  0 51 48  0
 2  0     68 15649812   3704 263396    0    0     0     0 5086  175  0 50 50  0
 2  0     68 15649812   3704 263396    0    0     0     0 4961   75  0 51 49  0
 2  0     68 15649812   3704 263396    0    0     0     0 4995   75  0 50 50  0
 2  0     68 15649812   3704 263396    0    0     0     0 4880   79  0 51 49  0
 2  0     68 15649812   3704 263396    0    0     0     4 5054   96  0 50 50  0
 2  0     68 15649812   3704 263396    0    0     0     0 5102  202  0 53 47  0
 2  0     68 15649812   3704 263396    0    0     0     0 5002   74  0 50 50  0
 2  0     68 15649812   3704 263396    0    0     0   208 5023   91  0 50 50  0
 2  0     68 15649688   3704 263396    0    0     0     0 4991   85  0 50 50  0
 2  0     68 15649400   3704 263440    0    0     0     0 5304  209  0 51 49  0
 2  0     68 15649440   3704 263440    0    0     0     0 5002   70  0 50 50  0
 2  0     68 15649440   3704 263440    0    0     0     0 4956   78  0 50 50  0
 2  0     68 15649440   3704 263440    0    0     0     0 5012   82  0 50 50  0
 2  0     68 15649440   3704 263440    0    0     0    16 4926   86  0 50 50  0
 2  0     68 15649316   3704 263440    0    0     0    44 5082   94  0 50 50  0
 2  0     68 15649316   3704 263440    0    0     0     0 4935   81  0 50 50  0
 2  0     68 15649316   3704 263440    0    0     0     0 5055   90  0 50 50  0
 2  0     68 15649316   3704 263440    0    0     0     0 4931   80  0 50 50  0
 2  0     68 15649316   3704 263440    0    0     0     0 5078   89  0 50 50  0
 2  0     68 15649052   3704 263488    0    0     0    68 5291  168  0 51 49  0
 2  0     68 15648920   3704 263488    0    0     0     0 5404  126  3 50 47  0
 2  0     68 15649068   3704 263488    0    0     0     0 4903   81  0 50 50  0
 2  0     68 15649068   3704 263488    0    0     0     8 5024   81  0 50 50  0
 2  0     68 15649068   3704 263488    0    0     0     0 5029   81  0 50 50  0
 2  0     68 15649068   3704 263488    0    0     0     0 5130  118  0 50 50  0
 2  0     68 15649068   3704 263488    0    0     0     0 5062   79  0 50 50  0
 2  2     68 15648512   3704 263488    0    0     0   656 5137  114  0 50 47  2
 2  2     68 15648896   3704 263488    0    0     0  4782 6934 5075  0 26 40 34
 1  2     68 15648928   3704 263488    0    0     0  5580 7916 6711  0 27 39 34
 1  2     68 15648920   3704 263532    0    0     0  6880 8815 7614  0 27 33 39
 1  2     68 15648928   3704 263532    0    0     0  6252 8454 7160  0 26 32 43
 1  2     68 15648928   3704 263532    0    0     0  4136 6695 4881  0 25 34 41
 1  2     68 15648928   3704 263532    0    0     0  5136 7219 5559  0 26 31 43
 1  2     68 15648928   3704 263532    0    0     0  5572 7200 5775  0 26 30 44
 2  0     68 15648844   3704 263884    0    0    92  1944 5819 3102  0 34 48 18
 2  0     68 15649068   3704 263828    0    0     0     0 5013   77  0 50 50  0
 2  0     68 15649068   3704 263828    0    0     0     0 4954   92  0 50 50  0
 2  0     68 15649068   3704 263828    0    0     0     0 5015   85  0 50 50  0
 2  0     68 15649068   3704 263828    0    0     0     0 5008   72  0 50 50  0
 2  0     68 15649184   3704 263872    0    0     0    44 5233  153  1 51 49  0
 2  0     68 15649068   3704 263872    0    0     0     0 5010   93  0 50 50  0
 2  0     68 15649068   3704 263872    0    0     0     0 5065   83  0 50 50  0
 2  0     68 15648944   3704 263872    0    0     0     0 4985   76  0 50 50  0
 2  0     68 15648820   3704 263872    0    0     0     0 4964   80  0 50 50  0
 2  0     68 15648820   3704 263872    0    0     0     0 5067  112  0 50 50  0
 2  0     68 15648572   3704 263872    0    0     0     0 4999   85  0 50 50  0
 2  0     68 15648572   3704 263872    0    0     0     0 4989   71  0 50 50  0
 2  0     68 15648448   3704 263872    0    0     0     0 4926   82  0 50 50  0
 2  0     68 15648324   3704 263872    0    0     0     0 4972   87  0 50 50  0
 2  2     68 15646872   3704 263916    0    0     0  3148 5919 2947  0 30 32 37
 1  2     68 15646944   3704 263916    0    0     0  3548 6195 4149  0 26 39 35
 1  2     68 15646944   3704 263916    0    0     0  2244 5206 2890  0 27 34 39
 1  2     68 15646944   3704 263916    0    0     0  3024 5736 3625  0 26 33 42
 1  2     68 15646944   3704 263916    0    0     0  3144 5638 3633  0 26 29 45
 1  2     68 15646944   3704 263916    0    0     0  2600 5400 3222  0 25 29 46
 1  2     68 15646944   3704 263916    0    0     0  3264 5927 3960  0 26 36 38
 3  2     68 15646944   3704 263916    0    0     0  5904 8307 7033  0 28 28 44
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 3  2     68 15646944   3704 263916    0    0     0  4945 8941 4388  0 50 13 37
 4  2     68 15646944   3704 263916    0    0     0  4976 8981 4674  0 51 12 37
 2  2     68 15646564   3704 263960    0    0     0  4756 9027 4543  0 51 19 30
 3  2     68 15646816   3704 263960    0    0     0 10248 11107 6925  0 51 12 36
 2  2     68 15647068   3704 263960    0    0     0  5480 9019 4345  0 51 11 37
 3  0     68 15647944   3704 263960    0    0     0   952 7274 1747  0 61 22 17
 3  0     68 15647952   3704 264004    0    0     0     0 7166   75  0 75 25  0
 3  0     68 15647952   3704 264004    0    0     0     0 7185   98  0 75 25  0
 3  0     68 15647952   3704 264004    0    0     0     0 7118   72  0 75 25  0
 3  0     68 15647952   3704 264004    0    0     0     0 7144   74  0 75 25  0
 3  0     68 15647952   3704 264004    0    0     0     0 7124   78  0 75 25  0
 3  0     68 15647952   3704 264004    0    0     0     0 7118   84  0 75 25  0
 3  0     68 15647944   3704 264048    0    0     0    44 7249  140  0 76 24  0
 3  0     68 15647952   3704 264048    0    0     0     0 7149   76  0 75 25  0
 3  0     68 15647952   3704 264048    0    0     0     0 7144   77  0 75 25  0
 3  0     68 15647952   3704 264048    0    0     0    16 7125   88  0 75 25  0
 3  0     68 15647952   3704 264048    0    0     0     0 7153   74  0 75 25  0
 3  0     68 15647828   3704 264048    0    0     0     0 7288  199  0 76 24  0
 3  0     68 15647828   3704 264048    0    0     0     0 7169   89  0 75 25  0
 3  0     68 15647580   3704 264048    0    0     0     0 7168   82  0 75 25  0
 2  2     68 15647572   3704 264048    0    0     0  4240 8458 3963  0 53 19 27
 2  2     68 15647696   3704 264048    0    0     0  3784 8568 4128  0 51 14 36
 2  0     68 15647680   3704 264096    0    0     0  4432 8868 4449  0 55 15 29
 2  0     68 15647936   3704 264096    0    0     0     0 5447  130  3 50 47  0
 2  0     68 15647960   3704 264096    0    0     0     0 4989   79  0 53 47  0
 2  0     68 15647960   3704 264096    0    0     0     0 5003   81  0 48 52  0
 2  0     68 15647960   3704 264096    0    0     0     0 5015  181  0 53 47  0
 2  0     68 15647960   3704 264096    0    0     0     8 5107  139  0 47 53  0
 2  0     68 15647960   3704 264096    0    0     0     0 5046  178  0 50 50  0
 2  0     68 15647712   3704 264096    0    0     0     0 4997   84  0 50 50  0
 2  0     68 15647712   3704 264096    0    0     0    54 5098  163  0 52 48  0
 2  0     68 15647712   3704 264096    0    0     0     0 4978  104  0 50 50  0
 2  0     68 15648076   3704 264140    0    0     0    44 5260  141  1 50 49  0
 2  0     68 15648084   3704 264140    0    0     0     0 4957   84  0 51 50  0
 2  0     68 15648084   3704 264140    0    0     0     0 5052  181  0 50 50  0
 2  0     68 15647836   3704 264140    0    0     0     0 5063  133  0 50 50  0
 1  2     68 15646464   3704 264140    0    0     0  3992 6813 3727  0 33 34 33
 1  2     68 15646340   3704 264140    0    0     0  5668 8204 6632  0 26 36 38
 1  2     68 15646332   3704 264140    0    0     0  7140 8842 7701  0 26 31 43
 2  2     68 15646332   3704 264140    0    0     0  7972 9638 9240  0 26 34 41
 1  2     68 15646332   3704 264140    0    0     0  9224 10442 9945  0 29 29 42
 1  2     68 15646332   3704 264140    0    0     0  4684 6481 4572  0 26 43 31
 3  2     68 15646448   3704 264184    0    0     0  2136 5114 2539  0 26 37 36
 1  2     68 15646456   3704 264184    0    0     0  3292 5654 3349  0 26 43 31
 1  2     68 15646456   3704 264184    0    0     0  2848 5365 3155  0 25 29 46
 1  2     68 15646456   3704 264184    0    0     0  3936 6192 4294  0 26 30 45
 2  2     68 15646952   3704 264184    0    0     0  5784 7514 6012  0 26 28 46
 1  2     68 15647076   3704 264184    0    0     0  3900 6474 4544  0 26 36 39
<sync finishes around here>
 1  0     68 15647836   3704 264184    0    0     0  3424 6108 4457  0 27 45 27
 1  0     68 15647960   3704 264184    0    0     0     0 2560   76  0 29 71  0
 1  0     68 15647960   3704 264184    0    0     0     0 2607  107  0 22 78  0
 1  0     68 15647960   3704 264184    0    0     0     0 2571   90  0 28 72  0
 1  0     68 15647952   3704 264228    0    0     0    48 2943  175  0 23 76  0
 1  0     68 15647960   3704 264228    0    0     0     0 2552   73  0 29 71  0
 1  0     68 15647960   3704 264228    0    0     0     0 2591   95  0 22 78  0
 1  0     68 15647960   3704 264228    0    0     0     0 2570   84  0 25 75  0
 1  0     68 15647960   3704 264228    0    0     0     0 2602   98  0 25 75  0
 1  0     68 15647960   3704 264228    0    0     0     0 2672   83  0 25 75  0
 1  0     68 15647960   3704 264228    0    0     0     0 2815  113  0 25 75  0
 1  0     68 15647960   3704 264228    0    0     0     0 2623   93  0 25 75  0
 1  0     68 15647960   3704 264228    0    0     0    53 2628  121  0 25 75  0
 1  0     68 15648084   3704 264228    0    0     0     0 2604   89  0 26 74  0
 1  0     68 15648200   3704 264272    0    0     0    44 2868  154  0 26 74  0
 1  0     68 15648208   3704 264272    0    0     0     0 2737  215  0 26 74  0
 0  2     68 15648128   3704 264272    0    0     0  2084 2682 1967  0  5 55 40
 0  2     68 15648128   3704 264272    0    0     0  3936 4635 4782  0  2 49 50
 0  2     68 15648128   3704 264272    0    0     0  3552 4006 4310  0  1 49 50
 0  2     68 15648128   3704 264272    0    0     0  4244 4798 5309  0  1 50 50
 0  2     68 15648128   3704 264272    0    0     0  5988 6143 7430  0  1 50 49
 0  2     68 15648128   3704 264272    0    0     0  7480 7222 8853  0  2 61 37
 0  2     68 15648128   3704 264272    0    0     0  6756 6671 8141  0  2 60 39
 0  0     68 15649128   3704 264272    0    0     0  6356 6545 8209  0  1 61 38
 0  0     68 15649128   3704 264316    0    0     0    44  645  161  0  1 99  0
 0  0     68 15649136   3704 264316    0    0     0     0  334   71  0  0 100  0
 0  0     68 15649136   3704 264316    0    0     0     0  367   83  0  0 100  0
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0     68 15649136   3704 264316    0    0     0    16  319   87  0  0 100  0
 0  0     68 15649136   3704 264316    0    0     0     0  382  100  0  0 100  0
 0  0     68 15649136   3704 264316    0    0     0     0  449   76  0  0 100  0
 0  0     68 15649136   3704 264316    0    0     0     0  348   90  0  0 100  0
 0  0     68 15649136   3704 264316    0    0     0     0  411   87  0  0 100  0
 0  0     68 15649136   3704 264316    0    0     0     0  357   91  0  0 100  0
 1  0     68 15649384   3704 264316    0    0     0     0  401   97  0  0 100  0
 0  0     68 15649136   3704 264368    0    0     0     0  311   79  0  0 100  0
 0  0     68 15649128   3704 264368    0    0     0     0  410   88  0  0 100  0

Nothing (other than "sync") was writing during this time, all cleaner
processes were idle. I tried to umount -t btrfs -a and after "umount"
finally finishes, mounting them again is normal and umounts immediately
after are instantaneous.