Hello, I have found, that ocfs2 is very slow when doing I/O operation without cache. See a simple test: ng-vvv1:~# dd if=/data/verejna/dd-1G bs=1k | dd of=/dev/null 1048576+0 records in 1048576+0 records out 1073741824 bytes (1.1 GB) copied, 395.183 s, 2.7 MB/s 2097152+0 records in 2097152+0 records out 1073741824 bytes (1.1 GB) copied, 395.184 s, 2.7 MB/s The underlying block device is quite quick: ng-vvv1:~# dd if=/dev/mapper/3600a0b800067df83000003ba4ba0f5ef bs=1k count=1M | dd of=/dev/null 1048576+0 records in 1048576+0 records out 2097152+0 records in 2097152+0 records out 1073741824 bytes (1.1 GB) copied, 6.32306 s, 170 MB/s 1073741824 bytes (1.1 GB) copied, 6.32293 s, 170 MB/s Here is the environment: * Infrastructure: two nodes, each with two port FC card conected to a disk array * OS: Debian GNU/Linux 5.0 * Kernel: 2.6.26-2-amd64 * Ocfs2: 1.4.1 * Multipath: # multipath -ll 3600a0b800067df83000003ba4ba0f5ef 3600a0b800067df83000003ba4ba0f5efdm-7 SUN ,LCSM100_F [size=100G][features=1 queue_if_no_path][hwhandler=1 rdac] \_ round-robin 0 [prio=3][active] \_ 7:0:0:0 sdb 8:16 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 8:0:0:0 sde 8:64 [active][ghost] * Creation of the FS: # mkfs.ocfs2 -T mail /dev/mapper/mpath0 * Mount parameters: # mount | grep verejna /dev/mapper/3600a0b800067df83000003ba4ba0f5ef on /data/verejna type ocfs2 (rw,_netdev,heartbeat=local) # mounted.ocfs2 -d Device FS UUID Label /dev/sdb ocfs2 91ae6ead-eee5-42fe-957c-9392e02b5d90 /dev/dm-7 ocfs2 91ae6ead-eee5-42fe-957c-9392e02b5d90 Please, could I ask, what could be wrong? Thanks. Best regards, Michal Vyoral
Can you email me the following: debugfs.ocfs2 -R "stats" /dev/mapper/3600a0b800067df83000003ba4ba0f5ef debugfs.ocfs2 -R "stat dd-1G" /dev/mapper/3600a0b800067df83000003ba4ba0f5ef This will tell us the fs params and the file layout. Next run the same with different blocksizes. Run the same with bs=4K and bs=1M. iflag=direct will bypass the cache. echo 3 >/proc/sys/vm/drop_caches strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=4K echo 3 >/proc/sys/vm/drop_caches strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=1M strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=4K iflag=direct strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=1M iflag=direct echo 3 >/proc/sys/vm/drop_caches strace -c dd if=/dev/... of=/dev/null of=/dev/null bs=4K echo 3 >/proc/sys/vm/drop_caches strace -c dd if=/dev/... of=/dev/null of=/dev/null bs=1M strace -c dd if=/dev/... of=/dev/null of=/dev/null bs=4K iflag=direct strace -c dd if=/dev/... of=/dev/null of=/dev/null bs=1M iflag=direct Sunil On 12/13/2010 04:18 AM, Michal Vyoral wrote:> Hello, > > I have found, that ocfs2 is very slow when doing I/O operation without > cache. See a simple test: > > ng-vvv1:~# dd if=/data/verejna/dd-1G bs=1k | dd of=/dev/null > 1048576+0 records in > 1048576+0 records out > 1073741824 bytes (1.1 GB) copied, 395.183 s, 2.7 MB/s > 2097152+0 records in > 2097152+0 records out > 1073741824 bytes (1.1 GB) copied, 395.184 s, 2.7 MB/s > > The underlying block device is quite quick: > > ng-vvv1:~# dd if=/dev/mapper/3600a0b800067df83000003ba4ba0f5ef bs=1k > count=1M | dd of=/dev/null > 1048576+0 records in > 1048576+0 records out > 2097152+0 records in > 2097152+0 records out > 1073741824 bytes (1.1 GB) copied, 6.32306 s, 170 MB/s > 1073741824 bytes (1.1 GB) copied, 6.32293 s, 170 MB/s > > Here is the environment: > > * Infrastructure: two nodes, each with two port FC card conected to a > disk array > > * OS: Debian GNU/Linux 5.0 > > * Kernel: 2.6.26-2-amd64 > > * Ocfs2: 1.4.1 > > * Multipath: > # multipath -ll 3600a0b800067df83000003ba4ba0f5ef > 3600a0b800067df83000003ba4ba0f5efdm-7 SUN ,LCSM100_F > [size=100G][features=1 queue_if_no_path][hwhandler=1 rdac] > \_ round-robin 0 [prio=3][active] > \_ 7:0:0:0 sdb 8:16 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 8:0:0:0 sde 8:64 [active][ghost] > > > * Creation of the FS: > # mkfs.ocfs2 -T mail /dev/mapper/mpath0 > > * Mount parameters: > # mount | grep verejna > /dev/mapper/3600a0b800067df83000003ba4ba0f5ef on /data/verejna type > ocfs2 (rw,_netdev,heartbeat=local) > # mounted.ocfs2 -d > Device FS UUID > Label > /dev/sdb ocfs2 91ae6ead-eee5-42fe-957c-9392e02b5d90 > /dev/dm-7 ocfs2 91ae6ead-eee5-42fe-957c-9392e02b5d90 > > Please, could I ask, what could be wrong? Thanks. > > Best regards, > Michal Vyoral > > > _______________________________________________ > Ocfs2-users mailing list > Ocfs2-users at oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users
Hello Sunil, I am sending the required statistics: Sunil Mushran p??e v Po 13. 12. 2010 v 10:40 -0800:> Can you email me the following: > > debugfs.ocfs2 -R "stats" /dev/mapper/3600a0b800067df83000003ba4ba0f5ef > debugfs.ocfs2 -R "statdd-1G" /dev/mapper/3600a0b800067df83000003ba4ba0f5ef # debugfs.ocfs2 -R "stats" /dev/mapper/3600a0b800067df83000003ba4ba0f5ef Revision: 0.90 Mount Count: 0 Max Mount Count: 20 State: 0 Errors: 0 Check Interval: 0 Last Check: Mon Apr 12 12:56:37 2010 Creator OS: 0 Feature Compat: 1 BackupSuper Feature Incompat: 16 Sparse Tunefs Incomplete: 0 None Feature RO compat: 1 Unwritten Root Blknum: 5 System Dir Blknum: 6 First Cluster Group Blknum: 3 Block Size Bits: 12 Cluster Size Bits: 12 Max Node Slots: 4 Label: UUID: 91AE6EADEEE542FE957C9392E02B5D90 Cluster stack: classic o2cb Inode: 2 Mode: 00 Generation: 3493325634 (0xd037eb42) FS Generation: 3493325634 (0xd037eb42) Type: Unknown Attr: 0x0 Flags: Valid System Superblock User: 0 (root) Group: 0 (root) Size: 0 Links: 0 Clusters: 26214400 ctime: 0x4bc31885 -- Mon Apr 12 12:56:37 2010 atime: 0x0 -- Thu Jan 1 00:00:00 1970 mtime: 0x4bc31885 -- Mon Apr 12 12:56:37 2010 dtime: 0x0 -- Thu Jan 1 00:00:00 1970 ctime_nsec: 0x00000000 -- 0 atime_nsec: 0x00000000 -- 0 mtime_nsec: 0x00000000 -- 0 Last Extblk: 0 Sub Alloc Slot: Global Sub Alloc Bit: 65535 # debugfs.ocfs2 -R "stat dd-1G" /dev/mapper/3600a0b800067df83000003ba4ba0f5ef Inode: 2677885 Mode: 0644 Generation: 508664856 (0x1e519c18) FS Generation: 3493325634 (0xd037eb42) Type: Regular Attr: 0x0 Flags: Valid User: 0 (root) Group: 0 (root) Size: 1073741824 Links: 1 Clusters: 262144 ctime: 0x4cd40119 -- Fri Nov 5 13:05:29 2010 atime: 0x4cf63a8c -- Wed Dec 1 12:07:40 2010 mtime: 0x4cd40119 -- Fri Nov 5 13:05:29 2010 dtime: 0x0 -- Thu Jan 1 00:00:00 1970 ctime_nsec: 0x2938e3a0 -- 691594144 atime_nsec: 0x02d717ad -- 47650733 mtime_nsec: 0x2938e3a0 -- 691594144 Last Extblk: 1375775 Sub Alloc Slot: 0 Sub Alloc Bit: 312 Tree Depth: 2 Count: 243 Next Free Rec: 5 ## Offset Clusters Block# 0 0 63504 1375374 1 63504 63504 1375380 2 127008 63504 7630229 3 190512 63504 1375633 4 254016 8128 7630482 SubAlloc Bit: 141 SubAlloc Slot: 0 Blknum: 1375374 Next Leaf: 0 Tree Depth: 1 Count: 252 Next Free Rec: 252 ## Offset Clusters Block# 0 0 252 7629957 1 252 252 7629958 2 504 252 7629959 3 756 252 7629960 4 1008 252 7629961 5 1260 252 7629962 6 1512 252 7629963 7 1764 252 7629964 8 2016 252 7629965 9 2268 252 7629966 10 2520 252 7629967 .... (truncated)> This will tell us the fs params and the file layout. > > Next run the same with different blocksizes. Run the same with bs=4Kand> bs=1M. iflag=direct will bypass the cache. > > echo 3 >/proc/sys/vm/drop_caches > strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=4K > echo 3 >/proc/sys/vm/drop_caches > strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=1M > strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=4K iflag=direct > strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=1M iflag=direct > > echo 3 >/proc/sys/vm/drop_caches > strace -c dd if=/dev/... of=/dev/null of=/dev/null bs=4K > echo 3 >/proc/sys/vm/drop_caches > strace -c dd if=/dev/... of=/dev/null of=/dev/null bs=1M > strace -c dd if=/dev/... of=/dev/null of=/dev/null bs=4K iflag=direct > strace -c dd if=/dev/... of=/dev/null of=/dev/null bs=1M iflag=direct ># sync; echo 3 >/proc/sys/vm/drop_caches # strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=4K 262144+0 records in 262144+0 records out 1073741824 bytes (1.1 GB) copied, 391.75 s, 2.7 MB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 98.55 0.885950 3 262150 read 1.45 0.013016 0 262147 write 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 15 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 0.898966 524374 12 total # echo 3 >/proc/sys/vm/drop_caches # strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=1M 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 413.33 s, 2.6 MB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 1.944121 1887 1030 read 0.00 0.000025 0 1027 write 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 16 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 1.944146 2135 12 total ng-vvv1:/data/verejna# strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=4K iflag=direct 262144+0 records in 262144+0 records out 1073741824 bytes (1.1 GB) copied, 2289.85 s, 469 kB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 99.86 10.222044 39 262150 read 0.14 0.014632 0 262147 write 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 15 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 10.236676 524374 12 total ng-vvv1:/data/verejna# strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=1M iflag=direct 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 361.797 s, 3.0 MB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 1.364085 1324 1030 read 0.00 0.000028 0 1027 write 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 16 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 1.364113 2135 12 total ng-vvv1:/data/verejna# echo 3 >/proc/sys/vm/drop_caches ng-vvv1:/data/verejna# strace -c dd if=/dev/mapper/3600a0b800067df83000003ba4ba0f5ef of=/dev/null of=/dev/null bs=4K count=262144 262144+0 records in 262144+0 records out 1073741824 bytes (1.1 GB) copied, 15.5014 s, 69.3 MB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 81.93 0.033628 0 262149 read 18.01 0.007391 0 262147 write 0.07 0.000028 2 15 mmap 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 0.041047 524373 12 total ng-vvv1:/data/verejna# echo 3 >/proc/sys/vm/drop_caches ng-vvv1:/data/verejna# strace -c dd if=/dev/mapper/3600a0b800067df83000003ba4ba0f5ef of=/dev/null of=/dev/null bs=1M count=1k 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 9.24075 s, 116 MB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 99.58 0.945906 919 1029 read 0.42 0.004000 4000 1 execve 0.00 0.000000 0 1027 write 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 16 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 0.949906 2134 12 total ng-vvv1:/data/verejna# strace -c dd if=/dev/mapper/3600a0b800067df83000003ba4ba0f5ef of=/dev/null of=/dev/null bs=4K count=262144 iflag=direct 262144+0 records in 262144+0 records out 1073741824 bytes (1.1 GB) copied, 76.6662 s, 14.0 MB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 94.79 0.238143 1 262149 read 5.21 0.013084 0 262147 write 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 15 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 0.251227 524373 12 total ng-vvv1:/data/verejna# strace -c dd if=/dev/mapper/3600a0b800067df83000003ba4ba0f5ef of=/dev/null of=/dev/null bs=1M count=1k iflag=direct 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 4.75594 s, 226 MB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 99.91 0.128550 125 1029 read 0.09 0.000111 0 1027 write 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 16 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 0.128661 2134 12 total Michal
Hello Sunil, I apologize for the chaos in my responding. So once again: Sunil Mushran p??e v Po 13. 12. 2010 v 10:40 -0800:> Can you email me the following: > > debugfs.ocfs2 -R "stats" /dev/mapper/3600a0b800067df83000003ba4ba0f5ef > debugfs.ocfs2 -R "stat dd-1G" /dev/mapper/3600a0b800067df83000003ba4ba0f5ef > > This will tell us the fs params and the file layout. ># debugfs.ocfs2 -R "stats" /dev/mapper/3600a0b800067df83000003ba4ba0f5ef Revision: 0.90 Mount Count: 0 Max Mount Count: 20 State: 0 Errors: 0 Check Interval: 0 Last Check: Mon Apr 12 12:56:37 2010 Creator OS: 0 Feature Compat: 1 BackupSuper Feature Incompat: 16 Sparse Tunefs Incomplete: 0 None Feature RO compat: 1 Unwritten Root Blknum: 5 System Dir Blknum: 6 First Cluster Group Blknum: 3 Block Size Bits: 12 Cluster Size Bits: 12 Max Node Slots: 4 Label: UUID: 91AE6EADEEE542FE957C9392E02B5D90 Cluster stack: classic o2cb Inode: 2 Mode: 00 Generation: 3493325634 (0xd037eb42) FS Generation: 3493325634 (0xd037eb42) Type: Unknown Attr: 0x0 Flags: Valid System Superblock User: 0 (root) Group: 0 (root) Size: 0 Links: 0 Clusters: 26214400 ctime: 0x4bc31885 -- Mon Apr 12 12:56:37 2010 atime: 0x0 -- Thu Jan 1 00:00:00 1970 mtime: 0x4bc31885 -- Mon Apr 12 12:56:37 2010 dtime: 0x0 -- Thu Jan 1 00:00:00 1970 ctime_nsec: 0x00000000 -- 0 atime_nsec: 0x00000000 -- 0 mtime_nsec: 0x00000000 -- 0 Last Extblk: 0 Sub Alloc Slot: Global Sub Alloc Bit: 65535 # debugfs.ocfs2 -R "stat dd-1G" /dev/mapper/3600a0b800067df83000003ba4ba0f5ef Inode: 2677885 Mode: 0644 Generation: 508664856 (0x1e519c18) FS Generation: 3493325634 (0xd037eb42) Type: Regular Attr: 0x0 Flags: Valid User: 0 (root) Group: 0 (root) Size: 1073741824 Links: 1 Clusters: 262144 ctime: 0x4cd40119 -- Fri Nov 5 13:05:29 2010 atime: 0x4cf63a8c -- Wed Dec 1 12:07:40 2010 mtime: 0x4cd40119 -- Fri Nov 5 13:05:29 2010 dtime: 0x0 -- Thu Jan 1 00:00:00 1970 ctime_nsec: 0x2938e3a0 -- 691594144 atime_nsec: 0x02d717ad -- 47650733 mtime_nsec: 0x2938e3a0 -- 691594144 Last Extblk: 1375775 Sub Alloc Slot: 0 Sub Alloc Bit: 312 Tree Depth: 2 Count: 243 Next Free Rec: 5 ## Offset Clusters Block# 0 0 63504 1375374 1 63504 63504 1375380 2 127008 63504 7630229 3 190512 63504 1375633 4 254016 8128 7630482 SubAlloc Bit: 141 SubAlloc Slot: 0 Blknum: 1375374 Next Leaf: 0 Tree Depth: 1 Count: 252 Next Free Rec: 252 ## Offset Clusters Block# 0 0 252 7629957 1 252 252 7629958 2 504 252 7629959 3 756 252 7629960 4 1008 252 7629961 5 1260 252 7629962 6 1512 252 7629963 7 1764 252 7629964 8 2016 252 7629965 9 2268 252 7629966 10 2520 252 7629967 .... (truncated)> Next run the same with different blocksizes. Run the same with bs=4K and > bs=1M. iflag=direct will bypass the cache. > > echo 3 >/proc/sys/vm/drop_caches > strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=4K > echo 3 >/proc/sys/vm/drop_caches > strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=1M > strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=4K iflag=direct > strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=1M iflag=direct > > echo 3 >/proc/sys/vm/drop_caches > strace -c dd if=/dev/... of=/dev/null of=/dev/null bs=4K > echo 3 >/proc/sys/vm/drop_caches > strace -c dd if=/dev/... of=/dev/null of=/dev/null bs=1M > strace -c dd if=/dev/... of=/dev/null of=/dev/null bs=4K iflag=direct > strace -c dd if=/dev/... of=/dev/null of=/dev/null bs=1M iflag=direct ># sync; echo 3 >/proc/sys/vm/drop_caches # strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=4K 262144+0 records in 262144+0 records out 1073741824 bytes (1.1 GB) copied, 391.75 s, 2.7 MB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 98.55 0.885950 3 262150 read 1.45 0.013016 0 262147 write 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 15 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 0.898966 524374 12 total # echo 3 >/proc/sys/vm/drop_caches # strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=1M 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 413.33 s, 2.6 MB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 1.944121 1887 1030 read 0.00 0.000025 0 1027 write 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 16 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 1.944146 2135 12 total ng-vvv1:/data/verejna# strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=4K iflag=direct 262144+0 records in 262144+0 records out 1073741824 bytes (1.1 GB) copied, 2289.85 s, 469 kB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 99.86 10.222044 39 262150 read 0.14 0.014632 0 262147 write 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 15 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 10.236676 524374 12 total ng-vvv1:/data/verejna# strace -c dd if=/data/verejna/dd-1G of=/dev/null bs=1M iflag=direct 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 361.797 s, 3.0 MB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00 1.364085 1324 1030 read 0.00 0.000028 0 1027 write 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 16 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 1.364113 2135 12 total ng-vvv1:/data/verejna# echo 3 >/proc/sys/vm/drop_caches ng-vvv1:/data/verejna# strace -c dd if=/dev/mapper/3600a0b800067df83000003ba4ba0f5ef of=/dev/null of=/dev/null bs=4K count=262144 262144+0 records in 262144+0 records out 1073741824 bytes (1.1 GB) copied, 15.5014 s, 69.3 MB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 81.93 0.033628 0 262149 read 18.01 0.007391 0 262147 write 0.07 0.000028 2 15 mmap 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 0.041047 524373 12 total ng-vvv1:/data/verejna# echo 3 >/proc/sys/vm/drop_caches ng-vvv1:/data/verejna# strace -c dd if=/dev/mapper/3600a0b800067df83000003ba4ba0f5ef of=/dev/null of=/dev/null bs=1M count=1k 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 9.24075 s, 116 MB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 99.58 0.945906 919 1029 read 0.42 0.004000 4000 1 execve 0.00 0.000000 0 1027 write 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 16 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 0.949906 2134 12 total ng-vvv1:/data/verejna# strace -c dd if=/dev/mapper/3600a0b800067df83000003ba4ba0f5ef of=/dev/null of=/dev/null bs=4K count=262144 iflag=direct 262144+0 records in 262144+0 records out 1073741824 bytes (1.1 GB) copied, 76.6662 s, 14.0 MB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 94.79 0.238143 1 262149 read 5.21 0.013084 0 262147 write 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 15 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 0.251227 524373 12 total ng-vvv1:/data/verejna# strace -c dd if=/dev/mapper/3600a0b800067df83000003ba4ba0f5ef of=/dev/null of=/dev/null bs=1M count=1k iflag=direct 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 4.75594 s, 226 MB/s % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 99.91 0.128550 125 1029 read 0.09 0.000111 0 1027 write 0.00 0.000000 0 14 6 open 0.00 0.000000 0 12 1 close 0.00 0.000000 0 6 fstat 0.00 0.000000 0 1 lseek 0.00 0.000000 0 16 mmap 0.00 0.000000 0 4 mprotect 0.00 0.000000 0 2 munmap 0.00 0.000000 0 3 brk 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 1 rt_sigprocmask 0.00 0.000000 0 5 5 access 0.00 0.000000 0 2 dup2 0.00 0.000000 0 1 execve 0.00 0.000000 0 1 getrlimit 0.00 0.000000 0 1 arch_prctl 0.00 0.000000 0 1 futex 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list ------ ----------- ----------- --------- --------- ---------------- 100.00 0.128661 2134 12 total Michal
Maybe Matching Threads
- Confusing lstat() performance
- Performance optimization tips Gluster 3.3? (small files / directory listings)
- [LLVMdev] Symlink to llvm-config returns wrong results on Linux
- Staged installation fail on some file systems
- Staged installation fail on some file systems