With a large (12 tbyte) ocfs2 file system mounted, all I need to do is a
"find ." and ocfs2 will slowly consume all ram until oom-killer is
invoked.
Previously this was thought to be related to the file system being
exported via nfs but this has now been ruled out.
The following are some stats; first section is after the "find ." had
been running for about 10 minutes. It is far from out of memory at this
point but I didn't have time this afternoon to run it until it died.
Second section is right after issuing a
sync ; echo 3 > /proc/sys/vm/drop_caches
so you can see the memory get freed up.
This is running a recent SUSE SP1 BRANCH kernel
2.6.16.37-SLES10_SP1_BRANCH_20070213192756-smp
I have already been told on the bug tracker that this is not an ocfs2
issue. I'm not trying to be a pain, I just wanted to repeat my findings
so that ocfs2 list is aware that while it may not be an ocfs2 code
problem it certainly does prevent you from using ocfs2 in production.
If someone has a better place where these finding could be reported
please let me know.
======= section 1 ========
Mon Mar 5 15:15:01 CST 2007
# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 1 240 1563768 241264 47076 0 0 1 4 11 6 0 1 94 4
0
# vmstat -s
2075084 total memory
511456 used memory
182952 active memory
223820 inactive memory
1563628 free memory
241264 buffer memory
47076 swap cache
2056304 total swap
240 used swap
2056064 free swap
612937 non-nice user cpu ticks
7186 nice user cpu ticks
2272575 system cpu ticks
293987671 idle cpu ticks
13392542 IO-wait cpu ticks
50629 IRQ cpu ticks
1667207 softirq cpu ticks
0 steal cpu ticks
88657366 pages paged in
1816763473 pages paged out
0 pages swapped in
61 pages swapped out
2268215696 interrupts
2382140383 CPU context switches
1171569943 boot time
109385 forks
# vmstat -m
Cache Num Total Size Pages
rpc_buffers 8 8 2048 2
rpc_tasks 8 15 256 15
rpc_inode_cache 0 0 512 7
ocfs2_lock 152 203 16 203
ocfs2_inode_cache 29868 29868 896 4
ocfs2_uptodate 6655 6780 32 113
ocfs2_em_ent 29854 29854 64 59
dlmfs_inode_cache 1 6 640 6
dlm_mle_cache 10 10 384 10
configfs_dir_cache 33 78 48 78
fib6_nodes 7 113 32 113
ip6_dst_cache 7 15 256 15
ndisc_cache 1 15 256 15
RAWv6 5 6 640 6
UDPv6 3 6 640 6
tw_sock_TCPv6 0 0 128 30
request_sock_TCPv6 0 0 128 30
TCPv6 10 12 1280 3
ip_fib_alias 16 113 32 113
ip_fib_hash 16 113 32 113
dm_events 16 169 20 169
Cache Num Total Size Pages
dm_tio 3676 3857 16 203
dm_io 3695 3887 20 169
uhci_urb_priv 0 0 40 92
ext3_inode_cache 950 1912 512 8
ext3_xattr 0 0 48 78
journal_handle 2 169 20 169
journal_head 462 648 52 72
revoke_table 6 254 12 254
revoke_record 0 0 16 203
qla2xxx_srbs 242 270 128 30
scsi_cmd_cache 59 90 384 10
sgpool-256 32 32 4096 1
sgpool-128 32 32 2048 2
sgpool-64 32 32 1024 4
sgpool-32 32 32 512 8
sgpool-16 32 45 256 15
sgpool-8 156 210 128 30
scsi_io_context 0 0 104 37
UNIX 410 427 512 7
ip_mrt_cache 0 0 128 30
tcp_bind_bucket 12 203 16 203
Cache Num Total Size Pages
inet_peer_cache 89 118 64 59
secpath_cache 0 0 128 30
xfrm_dst_cache 0 0 384 10
ip_dst_cache 178 285 256 15
arp_cache 5 15 256 15
RAW 3 7 512 7
UDP 32 56 512 7
tw_sock_TCP 0 0 128 30
request_sock_TCP 0 0 64 59
TCP 16 35 1152 7
flow_cache 0 0 128 30
cfq_ioc_pool 184 720 96 40
cfq_pool 166 600 96 40
crq_pool 181 312 48 78
deadline_drq 0 0 52 72
as_arq 0 0 64 59
mqueue_inode_cache 1 6 640 6
isofs_inode_cache 0 0 384 10
minix_inode_cache 0 0 420 9
hugetlbfs_inode_cache 1 11 356 11
ext2_inode_cache 0 0 492 8
Cache Num Total Size Pages
ext2_xattr 0 0 48 78
dnotify_cache 1 169 20 169
dquot 0 0 128 30
eventpoll_pwq 17 101 36 101
eventpoll_epi 17 30 128 30
inotify_event_cache 0 0 28 127
inotify_watch_cache 40 184 40 92
kioctx 0 0 256 15
kiocb 0 0 128 30
fasync_cache 1 203 16 203
shmem_inode_cache 618 624 460 8
posix_timers_cache 0 0 100 39
uid_cache 7 59 64 59
blkdev_ioc 88 254 28 127
blkdev_queue 58 60 960 4
blkdev_requests 204 264 176 22
biovec-(256) 312 312 3072 2
biovec-128 368 370 1536 5
biovec-64 480 485 768 5
biovec-16 480 495 256 15
biovec-4 480 531 64 59
Cache Num Total Size Pages
biovec-1 609 5684 16 203
bio 592 720 128 30
sock_inode_cache 505 553 512 7
skbuff_fclone_cache 73 80 384 10
skbuff_head_cache 673 870 256 15
file_lock_cache 8 126 92 42
acpi_operand 634 828 40 92
acpi_parse_ext 0 0 44 84
acpi_parse 0 0 28 127
acpi_state 0 0 48 78
delayacct_cache 191 312 48 78
taskstats_cache 32 32 236 16
proc_inode_cache 59 140 372 10
sigqueue 97 135 144 27
radix_tree_node 87360 87360 276 14
bdev_cache 56 56 512 7
sysfs_dir_cache 4887 4968 40 92
mnt_cache 29 60 128 30
inode_cache 1071 1309 356 11
dentry_cache 34133 34133 132 29
filp 3048 4180 192 20
Cache Num Total Size Pages
names_cache 26 26 4096 1
idr_layer_cache 203 232 136 29
buffer_head 60840 60840 52 72
mm_struct 120 171 448 9
vm_area_struct 5831 8888 88 44
fs_cache 120 236 64 59
files_cache 121 198 448 9
signal_cache 166 200 384 10
sighand_cache 157 177 1344 3
task_struct 182 230 1376 5
anon_vma 2781 3048 12 254
pgd 97 97 4096 1
size-131072(DMA) 0 0 131072 1
size-131072 0 0 131072 1
size-65536(DMA) 0 0 65536 1
size-65536 0 0 65536 1
size-32768(DMA) 0 0 32768 1
size-32768 3 3 32768 1
size-16384(DMA) 0 0 16384 1
size-16384 21 21 16384 1
size-8192(DMA) 0 0 8192 1
Cache Num Total Size Pages
size-8192 175 175 8192 1
size-4096(DMA) 0 0 4096 1
size-4096 111 111 4096 1
size-2048(DMA) 0 0 2048 2
size-2048 682 708 2048 2
size-1024(DMA) 0 0 1024 4
size-1024 373 404 1024 4
size-512(DMA) 1 8 512 8
size-512 529 568 512 8
size-256(DMA) 0 0 256 15
size-256 30690 30690 256 15
size-128(DMA) 0 0 128 30
size-128 33360 33360 128 30
size-64(DMA) 0 0 64 59
size-32(DMA) 0 0 32 113
size-64 4166 7080 64 59
size-32 35030 35030 32 113
kmem_cache 150 150 256 15
======== section 2 =======
Mon Mar 5 15:15:28 CST 2007
# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 240 1873872 5460 42120 0 0 1 4 11 6 0 1 94 4
0
# vmstat -s
2075084 total memory
201212 used memory
161028 active memory
4184 inactive memory
1873872 free memory
5460 buffer memory
42120 swap cache
2056304 total swap
240 used swap
2056064 free swap
612951 non-nice user cpu ticks
7186 nice user cpu ticks
2272690 system cpu ticks
293990233 idle cpu ticks
13395382 IO-wait cpu ticks
50630 IRQ cpu ticks
1667219 softirq cpu ticks
0 steal cpu ticks
88673230 pages paged in
1816764408 pages paged out
0 pages swapped in
61 pages swapped out
2268233528 interrupts
2382233674 CPU context switches
1171569943 boot time
109411 forks
# vmstat -m
Cache Num Total Size Pages
rpc_buffers 8 8 2048 2
rpc_tasks 8 15 256 15
rpc_inode_cache 0 0 512 7
ocfs2_lock 152 203 16 203
ocfs2_inode_cache 452 452 896 4
ocfs2_uptodate 6496 6780 32 113
ocfs2_em_ent 523 1357 64 59
dlmfs_inode_cache 1 6 640 6
dlm_mle_cache 10 20 384 10
configfs_dir_cache 33 78 48 78
fib6_nodes 7 113 32 113
ip6_dst_cache 7 15 256 15
ndisc_cache 1 15 256 15
RAWv6 5 6 640 6
UDPv6 3 6 640 6
tw_sock_TCPv6 0 0 128 30
request_sock_TCPv6 0 0 128 30
TCPv6 10 12 1280 3
ip_fib_alias 16 113 32 113
ip_fib_hash 16 113 32 113
dm_events 16 169 20 169
Cache Num Total Size Pages
dm_tio 3718 3857 16 203
dm_io 3663 3887 20 169
uhci_urb_priv 0 0 40 92
ext3_inode_cache 859 1904 512 8
ext3_xattr 0 0 48 78
journal_handle 18 169 20 169
journal_head 127 648 52 72
revoke_table 6 254 12 254
revoke_record 0 0 16 203
qla2xxx_srbs 211 270 128 30
scsi_cmd_cache 86 90 384 10
sgpool-256 32 32 4096 1
sgpool-128 32 32 2048 2
sgpool-64 32 32 1024 4
sgpool-32 40 40 512 8
sgpool-16 49 60 256 15
sgpool-8 159 210 128 30
scsi_io_context 0 0 104 37
UNIX 394 427 512 7
ip_mrt_cache 0 0 128 30
tcp_bind_bucket 12 203 16 203
Cache Num Total Size Pages
inet_peer_cache 89 118 64 59
secpath_cache 0 0 128 30
xfrm_dst_cache 0 0 384 10
ip_dst_cache 169 285 256 15
arp_cache 6 15 256 15
RAW 3 7 512 7
UDP 32 56 512 7
tw_sock_TCP 0 0 128 30
request_sock_TCP 0 0 64 59
TCP 16 35 1152 7
flow_cache 0 0 128 30
cfq_ioc_pool 261 720 96 40
cfq_pool 243 600 96 40
crq_pool 241 312 48 78
deadline_drq 0 0 52 72
as_arq 0 0 64 59
mqueue_inode_cache 1 6 640 6
isofs_inode_cache 0 0 384 10
minix_inode_cache 0 0 420 9
hugetlbfs_inode_cache 1 11 356 11
ext2_inode_cache 0 0 492 8
Cache Num Total Size Pages
ext2_xattr 0 0 48 78
dnotify_cache 1 169 20 169
dquot 0 0 128 30
eventpoll_pwq 1 101 36 101
eventpoll_epi 1 30 128 30
inotify_event_cache 0 127 28 127
inotify_watch_cache 40 184 40 92
kioctx 0 0 256 15
kiocb 0 0 128 30
fasync_cache 1 203 16 203
shmem_inode_cache 618 624 460 8
posix_timers_cache 0 0 100 39
uid_cache 7 59 64 59
blkdev_ioc 150 254 28 127
blkdev_queue 58 60 960 4
blkdev_requests 211 264 176 22
biovec-(256) 312 312 3072 2
biovec-128 368 370 1536 5
biovec-64 485 485 768 5
biovec-16 495 495 256 15
biovec-4 531 531 64 59
Cache Num Total Size Pages
biovec-1 723 5684 16 203
bio 654 750 128 30
sock_inode_cache 473 553 512 7
skbuff_fclone_cache 78 80 384 10
skbuff_head_cache 673 870 256 15
file_lock_cache 12 126 92 42
acpi_operand 634 828 40 92
acpi_parse_ext 0 0 44 84
acpi_parse 0 0 28 127
acpi_state 0 0 48 78
delayacct_cache 236 312 48 78
taskstats_cache 12 32 236 16
proc_inode_cache 21 140 372 10
sigqueue 93 135 144 27
radix_tree_node 3225 9464 276 14
bdev_cache 56 56 512 7
sysfs_dir_cache 4887 4968 40 92
mnt_cache 29 60 128 30
inode_cache 1055 1309 356 11
dentry_cache 3351 9541 132 29
filp 3096 4180 192 20
Cache Num Total Size Pages
names_cache 26 26 4096 1
idr_layer_cache 203 232 136 29
buffer_head 1577 11592 52 72
mm_struct 120 171 448 9
vm_area_struct 5748 8888 88 44
fs_cache 210 236 64 59
files_cache 121 198 448 9
signal_cache 150 200 384 10
sighand_cache 157 177 1344 3
task_struct 182 230 1376 5
anon_vma 2751 3048 12 254
pgd 96 96 4096 1
size-131072(DMA) 0 0 131072 1
size-131072 0 0 131072 1
size-65536(DMA) 0 0 65536 1
size-65536 0 0 65536 1
size-32768(DMA) 0 0 32768 1
size-32768 3 3 32768 1
size-16384(DMA) 0 0 16384 1
size-16384 21 21 16384 1
size-8192(DMA) 0 0 8192 1
Cache Num Total Size Pages
size-8192 169 169 8192 1
size-4096(DMA) 0 0 4096 1
size-4096 109 109 4096 1
size-2048(DMA) 0 0 2048 2
size-2048 682 708 2048 2
size-1024(DMA) 0 0 1024 4
size-1024 373 404 1024 4
size-512(DMA) 1 8 512 8
size-512 491 560 512 8
size-256(DMA) 0 0 256 15
size-256 1560 2295 256 15
size-128(DMA) 0 0 128 30
size-128 3870 17580 128 30
size-64(DMA) 0 0 64 59
size-32(DMA) 0 0 32 113
size-64 4118 7080 64 59
size-32 5920 19097 32 113
kmem_cache 150 150 256 15
================
If this is not an OCFS2 issue, then what is it? I have recently been told by
Oracle support that the supported method for RAC on Oracle 10G is OCFS2. These
systems are going into production very shortly ...
________________________________
From: ocfs2-users-bounces@oss.oracle.com on behalf of John Lange
Sent: Mon 3/5/2007 4:25 PM
To: ocfs2-users
Subject: [Ocfs2-users] ocfs2 is still eating memory
With a large (12 tbyte) ocfs2 file system mounted, all I need to do is a
"find ." and ocfs2 will slowly consume all ram until oom-killer is
invoked.
Previously this was thought to be related to the file system being
exported via nfs but this has now been ruled out.
The following are some stats; first section is after the "find ." had
been running for about 10 minutes. It is far from out of memory at this
point but I didn't have time this afternoon to run it until it died.
Second section is right after issuing a
sync ; echo 3 > /proc/sys/vm/drop_caches
so you can see the memory get freed up.
This is running a recent SUSE SP1 BRANCH kernel
2.6.16.37-SLES10_SP1_BRANCH_20070213192756-smp
I have already been told on the bug tracker that this is not an ocfs2
issue. I'm not trying to be a pain, I just wanted to repeat my findings
so that ocfs2 list is aware that while it may not be an ocfs2 code
problem it certainly does prevent you from using ocfs2 in production.
If someone has a better place where these finding could be reported
please let me know.
======= section 1 ========
Mon Mar 5 15:15:01 CST 2007
# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 1 240 1563768 241264 47076 0 0 1 4 11 6 0 1 94 4
0
# vmstat -s
2075084 total memory
511456 used memory
182952 active memory
223820 inactive memory
1563628 free memory
241264 buffer memory
47076 swap cache
2056304 total swap
240 used swap
2056064 free swap
612937 non-nice user cpu ticks
7186 nice user cpu ticks
2272575 system cpu ticks
293987671 idle cpu ticks
13392542 IO-wait cpu ticks
50629 IRQ cpu ticks
1667207 softirq cpu ticks
0 steal cpu ticks
88657366 pages paged in
1816763473 pages paged out
0 pages swapped in
61 pages swapped out
2268215696 interrupts
2382140383 CPU context switches
1171569943 boot time
109385 forks
# vmstat -m
Cache Num Total Size Pages
rpc_buffers 8 8 2048 2
rpc_tasks 8 15 256 15
rpc_inode_cache 0 0 512 7
ocfs2_lock 152 203 16 203
ocfs2_inode_cache 29868 29868 896 4
ocfs2_uptodate 6655 6780 32 113
ocfs2_em_ent 29854 29854 64 59
dlmfs_inode_cache 1 6 640 6
dlm_mle_cache 10 10 384 10
configfs_dir_cache 33 78 48 78
fib6_nodes 7 113 32 113
ip6_dst_cache 7 15 256 15
ndisc_cache 1 15 256 15
RAWv6 5 6 640 6
UDPv6 3 6 640 6
tw_sock_TCPv6 0 0 128 30
request_sock_TCPv6 0 0 128 30
TCPv6 10 12 1280 3
ip_fib_alias 16 113 32 113
ip_fib_hash 16 113 32 113
dm_events 16 169 20 169
Cache Num Total Size Pages
dm_tio 3676 3857 16 203
dm_io 3695 3887 20 169
uhci_urb_priv 0 0 40 92
ext3_inode_cache 950 1912 512 8
ext3_xattr 0 0 48 78
journal_handle 2 169 20 169
journal_head 462 648 52 72
revoke_table 6 254 12 254
revoke_record 0 0 16 203
qla2xxx_srbs 242 270 128 30
scsi_cmd_cache 59 90 384 10
sgpool-256 32 32 4096 1
sgpool-128 32 32 2048 2
sgpool-64 32 32 1024 4
sgpool-32 32 32 512 8
sgpool-16 32 45 256 15
sgpool-8 156 210 128 30
scsi_io_context 0 0 104 37
UNIX 410 427 512 7
ip_mrt_cache 0 0 128 30
tcp_bind_bucket 12 203 16 203
Cache Num Total Size Pages
inet_peer_cache 89 118 64 59
secpath_cache 0 0 128 30
xfrm_dst_cache 0 0 384 10
ip_dst_cache 178 285 256 15
arp_cache 5 15 256 15
RAW 3 7 512 7
UDP 32 56 512 7
tw_sock_TCP 0 0 128 30
request_sock_TCP 0 0 64 59
TCP 16 35 1152 7
flow_cache 0 0 128 30
cfq_ioc_pool 184 720 96 40
cfq_pool 166 600 96 40
crq_pool 181 312 48 78
deadline_drq 0 0 52 72
as_arq 0 0 64 59
mqueue_inode_cache 1 6 640 6
isofs_inode_cache 0 0 384 10
minix_inode_cache 0 0 420 9
hugetlbfs_inode_cache 1 11 356 11
ext2_inode_cache 0 0 492 8
Cache Num Total Size Pages
ext2_xattr 0 0 48 78
dnotify_cache 1 169 20 169
dquot 0 0 128 30
eventpoll_pwq 17 101 36 101
eventpoll_epi 17 30 128 30
inotify_event_cache 0 0 28 127
inotify_watch_cache 40 184 40 92
kioctx 0 0 256 15
kiocb 0 0 128 30
fasync_cache 1 203 16 203
shmem_inode_cache 618 624 460 8
posix_timers_cache 0 0 100 39
uid_cache 7 59 64 59
blkdev_ioc 88 254 28 127
blkdev_queue 58 60 960 4
blkdev_requests 204 264 176 22
biovec-(256) 312 312 3072 2
biovec-128 368 370 1536 5
biovec-64 480 485 768 5
biovec-16 480 495 256 15
biovec-4 480 531 64 59
Cache Num Total Size Pages
biovec-1 609 5684 16 203
bio 592 720 128 30
sock_inode_cache 505 553 512 7
skbuff_fclone_cache 73 80 384 10
skbuff_head_cache 673 870 256 15
file_lock_cache 8 126 92 42
acpi_operand 634 828 40 92
acpi_parse_ext 0 0 44 84
acpi_parse 0 0 28 127
acpi_state 0 0 48 78
delayacct_cache 191 312 48 78
taskstats_cache 32 32 236 16
proc_inode_cache 59 140 372 10
sigqueue 97 135 144 27
radix_tree_node 87360 87360 276 14
bdev_cache 56 56 512 7
sysfs_dir_cache 4887 4968 40 92
mnt_cache 29 60 128 30
inode_cache 1071 1309 356 11
dentry_cache 34133 34133 132 29
filp 3048 4180 192 20
Cache Num Total Size Pages
names_cache 26 26 4096 1
idr_layer_cache 203 232 136 29
buffer_head 60840 60840 52 72
mm_struct 120 171 448 9
vm_area_struct 5831 8888 88 44
fs_cache 120 236 64 59
files_cache 121 198 448 9
signal_cache 166 200 384 10
sighand_cache 157 177 1344 3
task_struct 182 230 1376 5
anon_vma 2781 3048 12 254
pgd 97 97 4096 1
size-131072(DMA) 0 0 131072 1
size-131072 0 0 131072 1
size-65536(DMA) 0 0 65536 1
size-65536 0 0 65536 1
size-32768(DMA) 0 0 32768 1
size-32768 3 3 32768 1
size-16384(DMA) 0 0 16384 1
size-16384 21 21 16384 1
size-8192(DMA) 0 0 8192 1
Cache Num Total Size Pages
size-8192 175 175 8192 1
size-4096(DMA) 0 0 4096 1
size-4096 111 111 4096 1
size-2048(DMA) 0 0 2048 2
size-2048 682 708 2048 2
size-1024(DMA) 0 0 1024 4
size-1024 373 404 1024 4
size-512(DMA) 1 8 512 8
size-512 529 568 512 8
size-256(DMA) 0 0 256 15
size-256 30690 30690 256 15
size-128(DMA) 0 0 128 30
size-128 33360 33360 128 30
size-64(DMA) 0 0 64 59
size-32(DMA) 0 0 32 113
size-64 4166 7080 64 59
size-32 35030 35030 32 113
kmem_cache 150 150 256 15
======== section 2 =======
Mon Mar 5 15:15:28 CST 2007
# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 240 1873872 5460 42120 0 0 1 4 11 6 0 1 94 4
0
# vmstat -s
2075084 total memory
201212 used memory
161028 active memory
4184 inactive memory
1873872 free memory
5460 buffer memory
42120 swap cache
2056304 total swap
240 used swap
2056064 free swap
612951 non-nice user cpu ticks
7186 nice user cpu ticks
2272690 system cpu ticks
293990233 idle cpu ticks
13395382 IO-wait cpu ticks
50630 IRQ cpu ticks
1667219 softirq cpu ticks
0 steal cpu ticks
88673230 pages paged in
1816764408 pages paged out
0 pages swapped in
61 pages swapped out
2268233528 interrupts
2382233674 CPU context switches
1171569943 boot time
109411 forks
# vmstat -m
Cache Num Total Size Pages
rpc_buffers 8 8 2048 2
rpc_tasks 8 15 256 15
rpc_inode_cache 0 0 512 7
ocfs2_lock 152 203 16 203
ocfs2_inode_cache 452 452 896 4
ocfs2_uptodate 6496 6780 32 113
ocfs2_em_ent 523 1357 64 59
dlmfs_inode_cache 1 6 640 6
dlm_mle_cache 10 20 384 10
configfs_dir_cache 33 78 48 78
fib6_nodes 7 113 32 113
ip6_dst_cache 7 15 256 15
ndisc_cache 1 15 256 15
RAWv6 5 6 640 6
UDPv6 3 6 640 6
tw_sock_TCPv6 0 0 128 30
request_sock_TCPv6 0 0 128 30
TCPv6 10 12 1280 3
ip_fib_alias 16 113 32 113
ip_fib_hash 16 113 32 113
dm_events 16 169 20 169
Cache Num Total Size Pages
dm_tio 3718 3857 16 203
dm_io 3663 3887 20 169
uhci_urb_priv 0 0 40 92
ext3_inode_cache 859 1904 512 8
ext3_xattr 0 0 48 78
journal_handle 18 169 20 169
journal_head 127 648 52 72
revoke_table 6 254 12 254
revoke_record 0 0 16 203
qla2xxx_srbs 211 270 128 30
scsi_cmd_cache 86 90 384 10
sgpool-256 32 32 4096 1
sgpool-128 32 32 2048 2
sgpool-64 32 32 1024 4
sgpool-32 40 40 512 8
sgpool-16 49 60 256 15
sgpool-8 159 210 128 30
scsi_io_context 0 0 104 37
UNIX 394 427 512 7
ip_mrt_cache 0 0 128 30
tcp_bind_bucket 12 203 16 203
Cache Num Total Size Pages
inet_peer_cache 89 118 64 59
secpath_cache 0 0 128 30
xfrm_dst_cache 0 0 384 10
ip_dst_cache 169 285 256 15
arp_cache 6 15 256 15
RAW 3 7 512 7
UDP 32 56 512 7
tw_sock_TCP 0 0 128 30
request_sock_TCP 0 0 64 59
TCP 16 35 1152 7
flow_cache 0 0 128 30
cfq_ioc_pool 261 720 96 40
cfq_pool 243 600 96 40
crq_pool 241 312 48 78
deadline_drq 0 0 52 72
as_arq 0 0 64 59
mqueue_inode_cache 1 6 640 6
isofs_inode_cache 0 0 384 10
minix_inode_cache 0 0 420 9
hugetlbfs_inode_cache 1 11 356 11
ext2_inode_cache 0 0 492 8
Cache Num Total Size Pages
ext2_xattr 0 0 48 78
dnotify_cache 1 169 20 169
dquot 0 0 128 30
eventpoll_pwq 1 101 36 101
eventpoll_epi 1 30 128 30
inotify_event_cache 0 127 28 127
inotify_watch_cache 40 184 40 92
kioctx 0 0 256 15
kiocb 0 0 128 30
fasync_cache 1 203 16 203
shmem_inode_cache 618 624 460 8
posix_timers_cache 0 0 100 39
uid_cache 7 59 64 59
blkdev_ioc 150 254 28 127
blkdev_queue 58 60 960 4
blkdev_requests 211 264 176 22
biovec-(256) 312 312 3072 2
biovec-128 368 370 1536 5
biovec-64 485 485 768 5
biovec-16 495 495 256 15
biovec-4 531 531 64 59
Cache Num Total Size Pages
biovec-1 723 5684 16 203
bio 654 750 128 30
sock_inode_cache 473 553 512 7
skbuff_fclone_cache 78 80 384 10
skbuff_head_cache 673 870 256 15
file_lock_cache 12 126 92 42
acpi_operand 634 828 40 92
acpi_parse_ext 0 0 44 84
acpi_parse 0 0 28 127
acpi_state 0 0 48 78
delayacct_cache 236 312 48 78
taskstats_cache 12 32 236 16
proc_inode_cache 21 140 372 10
sigqueue 93 135 144 27
radix_tree_node 3225 9464 276 14
bdev_cache 56 56 512 7
sysfs_dir_cache 4887 4968 40 92
mnt_cache 29 60 128 30
inode_cache 1055 1309 356 11
dentry_cache 3351 9541 132 29
filp 3096 4180 192 20
Cache Num Total Size Pages
names_cache 26 26 4096 1
idr_layer_cache 203 232 136 29
buffer_head 1577 11592 52 72
mm_struct 120 171 448 9
vm_area_struct 5748 8888 88 44
fs_cache 210 236 64 59
files_cache 121 198 448 9
signal_cache 150 200 384 10
sighand_cache 157 177 1344 3
task_struct 182 230 1376 5
anon_vma 2751 3048 12 254
pgd 96 96 4096 1
size-131072(DMA) 0 0 131072 1
size-131072 0 0 131072 1
size-65536(DMA) 0 0 65536 1
size-65536 0 0 65536 1
size-32768(DMA) 0 0 32768 1
size-32768 3 3 32768 1
size-16384(DMA) 0 0 16384 1
size-16384 21 21 16384 1
size-8192(DMA) 0 0 8192 1
Cache Num Total Size Pages
size-8192 169 169 8192 1
size-4096(DMA) 0 0 4096 1
size-4096 109 109 4096 1
size-2048(DMA) 0 0 2048 2
size-2048 682 708 2048 2
size-1024(DMA) 0 0 1024 4
size-1024 373 404 1024 4
size-512(DMA) 1 8 512 8
size-512 491 560 512 8
size-256(DMA) 0 0 256 15
size-256 1560 2295 256 15
size-128(DMA) 0 0 128 30
size-128 3870 17580 128 30
size-64(DMA) 0 0 64 59
size-32(DMA) 0 0 32 113
size-64 4118 7080 64 59
size-32 5920 19097 32 113
kmem_cache 150 150 256 15
===============
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20070305/ca14681b/attachment-0001.html
Well, kswapd is supposed to flush the caches. As in, the vm controls the lifetime of the inodes in the inode_cache not ocfs2. All ocfs2 can do is free the memory associated with the inode when asked to. And it does that when you manually flush the cache. Qs is why the vm is not doing it on its own. (fwiw, you are on a beta kernel.) John Lange wrote:> With a large (12 tbyte) ocfs2 file system mounted, all I need to do is a > "find ." and ocfs2 will slowly consume all ram until oom-killer is > invoked. > > Previously this was thought to be related to the file system being > exported via nfs but this has now been ruled out. > > The following are some stats; first section is after the "find ." had > been running for about 10 minutes. It is far from out of memory at this > point but I didn't have time this afternoon to run it until it died. > > Second section is right after issuing a > > sync ; echo 3 > /proc/sys/vm/drop_caches > > so you can see the memory get freed up. > > This is running a recent SUSE SP1 BRANCH kernel > > 2.6.16.37-SLES10_SP1_BRANCH_20070213192756-smp > > I have already been told on the bug tracker that this is not an ocfs2 > issue. I'm not trying to be a pain, I just wanted to repeat my findings > so that ocfs2 list is aware that while it may not be an ocfs2 code > problem it certainly does prevent you from using ocfs2 in production. > > If someone has a better place where these finding could be reported > please let me know. > > > > > ======= section 1 ========> > Mon Mar 5 15:15:01 CST 2007 > # vmstat > procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------ > r b swpd free buff cache si so bi bo in cs us sy id wa st > 2 1 240 1563768 241264 47076 0 0 1 4 11 6 0 1 94 4 0 > # vmstat -s > 2075084 total memory > 511456 used memory > 182952 active memory > 223820 inactive memory > 1563628 free memory > 241264 buffer memory > 47076 swap cache > 2056304 total swap > 240 used swap > 2056064 free swap > 612937 non-nice user cpu ticks > 7186 nice user cpu ticks > 2272575 system cpu ticks > 293987671 idle cpu ticks > 13392542 IO-wait cpu ticks > 50629 IRQ cpu ticks > 1667207 softirq cpu ticks > 0 steal cpu ticks > 88657366 pages paged in > 1816763473 pages paged out > 0 pages swapped in > 61 pages swapped out > 2268215696 interrupts > 2382140383 CPU context switches > 1171569943 boot time > 109385 forks > # vmstat -m > Cache Num Total Size Pages > rpc_buffers 8 8 2048 2 > rpc_tasks 8 15 256 15 > rpc_inode_cache 0 0 512 7 > ocfs2_lock 152 203 16 203 > ocfs2_inode_cache 29868 29868 896 4 > ocfs2_uptodate 6655 6780 32 113 > ocfs2_em_ent 29854 29854 64 59 > dlmfs_inode_cache 1 6 640 6 > dlm_mle_cache 10 10 384 10 > configfs_dir_cache 33 78 48 78 > fib6_nodes 7 113 32 113 > ip6_dst_cache 7 15 256 15 > ndisc_cache 1 15 256 15 > RAWv6 5 6 640 6 > UDPv6 3 6 640 6 > tw_sock_TCPv6 0 0 128 30 > request_sock_TCPv6 0 0 128 30 > TCPv6 10 12 1280 3 > ip_fib_alias 16 113 32 113 > ip_fib_hash 16 113 32 113 > dm_events 16 169 20 169 > Cache Num Total Size Pages > dm_tio 3676 3857 16 203 > dm_io 3695 3887 20 169 > uhci_urb_priv 0 0 40 92 > ext3_inode_cache 950 1912 512 8 > ext3_xattr 0 0 48 78 > journal_handle 2 169 20 169 > journal_head 462 648 52 72 > revoke_table 6 254 12 254 > revoke_record 0 0 16 203 > qla2xxx_srbs 242 270 128 30 > scsi_cmd_cache 59 90 384 10 > sgpool-256 32 32 4096 1 > sgpool-128 32 32 2048 2 > sgpool-64 32 32 1024 4 > sgpool-32 32 32 512 8 > sgpool-16 32 45 256 15 > sgpool-8 156 210 128 30 > scsi_io_context 0 0 104 37 > UNIX 410 427 512 7 > ip_mrt_cache 0 0 128 30 > tcp_bind_bucket 12 203 16 203 > Cache Num Total Size Pages > inet_peer_cache 89 118 64 59 > secpath_cache 0 0 128 30 > xfrm_dst_cache 0 0 384 10 > ip_dst_cache 178 285 256 15 > arp_cache 5 15 256 15 > RAW 3 7 512 7 > UDP 32 56 512 7 > tw_sock_TCP 0 0 128 30 > request_sock_TCP 0 0 64 59 > TCP 16 35 1152 7 > flow_cache 0 0 128 30 > cfq_ioc_pool 184 720 96 40 > cfq_pool 166 600 96 40 > crq_pool 181 312 48 78 > deadline_drq 0 0 52 72 > as_arq 0 0 64 59 > mqueue_inode_cache 1 6 640 6 > isofs_inode_cache 0 0 384 10 > minix_inode_cache 0 0 420 9 > hugetlbfs_inode_cache 1 11 356 11 > ext2_inode_cache 0 0 492 8 > Cache Num Total Size Pages > ext2_xattr 0 0 48 78 > dnotify_cache 1 169 20 169 > dquot 0 0 128 30 > eventpoll_pwq 17 101 36 101 > eventpoll_epi 17 30 128 30 > inotify_event_cache 0 0 28 127 > inotify_watch_cache 40 184 40 92 > kioctx 0 0 256 15 > kiocb 0 0 128 30 > fasync_cache 1 203 16 203 > shmem_inode_cache 618 624 460 8 > posix_timers_cache 0 0 100 39 > uid_cache 7 59 64 59 > blkdev_ioc 88 254 28 127 > blkdev_queue 58 60 960 4 > blkdev_requests 204 264 176 22 > biovec-(256) 312 312 3072 2 > biovec-128 368 370 1536 5 > biovec-64 480 485 768 5 > biovec-16 480 495 256 15 > biovec-4 480 531 64 59 > Cache Num Total Size Pages > biovec-1 609 5684 16 203 > bio 592 720 128 30 > sock_inode_cache 505 553 512 7 > skbuff_fclone_cache 73 80 384 10 > skbuff_head_cache 673 870 256 15 > file_lock_cache 8 126 92 42 > acpi_operand 634 828 40 92 > acpi_parse_ext 0 0 44 84 > acpi_parse 0 0 28 127 > acpi_state 0 0 48 78 > delayacct_cache 191 312 48 78 > taskstats_cache 32 32 236 16 > proc_inode_cache 59 140 372 10 > sigqueue 97 135 144 27 > radix_tree_node 87360 87360 276 14 > bdev_cache 56 56 512 7 > sysfs_dir_cache 4887 4968 40 92 > mnt_cache 29 60 128 30 > inode_cache 1071 1309 356 11 > dentry_cache 34133 34133 132 29 > filp 3048 4180 192 20 > Cache Num Total Size Pages > names_cache 26 26 4096 1 > idr_layer_cache 203 232 136 29 > buffer_head 60840 60840 52 72 > mm_struct 120 171 448 9 > vm_area_struct 5831 8888 88 44 > fs_cache 120 236 64 59 > files_cache 121 198 448 9 > signal_cache 166 200 384 10 > sighand_cache 157 177 1344 3 > task_struct 182 230 1376 5 > anon_vma 2781 3048 12 254 > pgd 97 97 4096 1 > size-131072(DMA) 0 0 131072 1 > size-131072 0 0 131072 1 > size-65536(DMA) 0 0 65536 1 > size-65536 0 0 65536 1 > size-32768(DMA) 0 0 32768 1 > size-32768 3 3 32768 1 > size-16384(DMA) 0 0 16384 1 > size-16384 21 21 16384 1 > size-8192(DMA) 0 0 8192 1 > Cache Num Total Size Pages > size-8192 175 175 8192 1 > size-4096(DMA) 0 0 4096 1 > size-4096 111 111 4096 1 > size-2048(DMA) 0 0 2048 2 > size-2048 682 708 2048 2 > size-1024(DMA) 0 0 1024 4 > size-1024 373 404 1024 4 > size-512(DMA) 1 8 512 8 > size-512 529 568 512 8 > size-256(DMA) 0 0 256 15 > size-256 30690 30690 256 15 > size-128(DMA) 0 0 128 30 > size-128 33360 33360 128 30 > size-64(DMA) 0 0 64 59 > size-32(DMA) 0 0 32 113 > size-64 4166 7080 64 59 > size-32 35030 35030 32 113 > kmem_cache 150 150 256 15 > > > ======== section 2 =======> > Mon Mar 5 15:15:28 CST 2007 > # vmstat > procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------ > r b swpd free buff cache si so bi bo in cs us sy id wa st > 0 1 240 1873872 5460 42120 0 0 1 4 11 6 0 1 94 4 0 > # vmstat -s > 2075084 total memory > 201212 used memory > 161028 active memory > 4184 inactive memory > 1873872 free memory > 5460 buffer memory > 42120 swap cache > 2056304 total swap > 240 used swap > 2056064 free swap > 612951 non-nice user cpu ticks > 7186 nice user cpu ticks > 2272690 system cpu ticks > 293990233 idle cpu ticks > 13395382 IO-wait cpu ticks > 50630 IRQ cpu ticks > 1667219 softirq cpu ticks > 0 steal cpu ticks > 88673230 pages paged in > 1816764408 pages paged out > 0 pages swapped in > 61 pages swapped out > 2268233528 interrupts > 2382233674 CPU context switches > 1171569943 boot time > 109411 forks > # vmstat -m > Cache Num Total Size Pages > rpc_buffers 8 8 2048 2 > rpc_tasks 8 15 256 15 > rpc_inode_cache 0 0 512 7 > ocfs2_lock 152 203 16 203 > ocfs2_inode_cache 452 452 896 4 > ocfs2_uptodate 6496 6780 32 113 > ocfs2_em_ent 523 1357 64 59 > dlmfs_inode_cache 1 6 640 6 > dlm_mle_cache 10 20 384 10 > configfs_dir_cache 33 78 48 78 > fib6_nodes 7 113 32 113 > ip6_dst_cache 7 15 256 15 > ndisc_cache 1 15 256 15 > RAWv6 5 6 640 6 > UDPv6 3 6 640 6 > tw_sock_TCPv6 0 0 128 30 > request_sock_TCPv6 0 0 128 30 > TCPv6 10 12 1280 3 > ip_fib_alias 16 113 32 113 > ip_fib_hash 16 113 32 113 > dm_events 16 169 20 169 > Cache Num Total Size Pages > dm_tio 3718 3857 16 203 > dm_io 3663 3887 20 169 > uhci_urb_priv 0 0 40 92 > ext3_inode_cache 859 1904 512 8 > ext3_xattr 0 0 48 78 > journal_handle 18 169 20 169 > journal_head 127 648 52 72 > revoke_table 6 254 12 254 > revoke_record 0 0 16 203 > qla2xxx_srbs 211 270 128 30 > scsi_cmd_cache 86 90 384 10 > sgpool-256 32 32 4096 1 > sgpool-128 32 32 2048 2 > sgpool-64 32 32 1024 4 > sgpool-32 40 40 512 8 > sgpool-16 49 60 256 15 > sgpool-8 159 210 128 30 > scsi_io_context 0 0 104 37 > UNIX 394 427 512 7 > ip_mrt_cache 0 0 128 30 > tcp_bind_bucket 12 203 16 203 > Cache Num Total Size Pages > inet_peer_cache 89 118 64 59 > secpath_cache 0 0 128 30 > xfrm_dst_cache 0 0 384 10 > ip_dst_cache 169 285 256 15 > arp_cache 6 15 256 15 > RAW 3 7 512 7 > UDP 32 56 512 7 > tw_sock_TCP 0 0 128 30 > request_sock_TCP 0 0 64 59 > TCP 16 35 1152 7 > flow_cache 0 0 128 30 > cfq_ioc_pool 261 720 96 40 > cfq_pool 243 600 96 40 > crq_pool 241 312 48 78 > deadline_drq 0 0 52 72 > as_arq 0 0 64 59 > mqueue_inode_cache 1 6 640 6 > isofs_inode_cache 0 0 384 10 > minix_inode_cache 0 0 420 9 > hugetlbfs_inode_cache 1 11 356 11 > ext2_inode_cache 0 0 492 8 > Cache Num Total Size Pages > ext2_xattr 0 0 48 78 > dnotify_cache 1 169 20 169 > dquot 0 0 128 30 > eventpoll_pwq 1 101 36 101 > eventpoll_epi 1 30 128 30 > inotify_event_cache 0 127 28 127 > inotify_watch_cache 40 184 40 92 > kioctx 0 0 256 15 > kiocb 0 0 128 30 > fasync_cache 1 203 16 203 > shmem_inode_cache 618 624 460 8 > posix_timers_cache 0 0 100 39 > uid_cache 7 59 64 59 > blkdev_ioc 150 254 28 127 > blkdev_queue 58 60 960 4 > blkdev_requests 211 264 176 22 > biovec-(256) 312 312 3072 2 > biovec-128 368 370 1536 5 > biovec-64 485 485 768 5 > biovec-16 495 495 256 15 > biovec-4 531 531 64 59 > Cache Num Total Size Pages > biovec-1 723 5684 16 203 > bio 654 750 128 30 > sock_inode_cache 473 553 512 7 > skbuff_fclone_cache 78 80 384 10 > skbuff_head_cache 673 870 256 15 > file_lock_cache 12 126 92 42 > acpi_operand 634 828 40 92 > acpi_parse_ext 0 0 44 84 > acpi_parse 0 0 28 127 > acpi_state 0 0 48 78 > delayacct_cache 236 312 48 78 > taskstats_cache 12 32 236 16 > proc_inode_cache 21 140 372 10 > sigqueue 93 135 144 27 > radix_tree_node 3225 9464 276 14 > bdev_cache 56 56 512 7 > sysfs_dir_cache 4887 4968 40 92 > mnt_cache 29 60 128 30 > inode_cache 1055 1309 356 11 > dentry_cache 3351 9541 132 29 > filp 3096 4180 192 20 > Cache Num Total Size Pages > names_cache 26 26 4096 1 > idr_layer_cache 203 232 136 29 > buffer_head 1577 11592 52 72 > mm_struct 120 171 448 9 > vm_area_struct 5748 8888 88 44 > fs_cache 210 236 64 59 > files_cache 121 198 448 9 > signal_cache 150 200 384 10 > sighand_cache 157 177 1344 3 > task_struct 182 230 1376 5 > anon_vma 2751 3048 12 254 > pgd 96 96 4096 1 > size-131072(DMA) 0 0 131072 1 > size-131072 0 0 131072 1 > size-65536(DMA) 0 0 65536 1 > size-65536 0 0 65536 1 > size-32768(DMA) 0 0 32768 1 > size-32768 3 3 32768 1 > size-16384(DMA) 0 0 16384 1 > size-16384 21 21 16384 1 > size-8192(DMA) 0 0 8192 1 > Cache Num Total Size Pages > size-8192 169 169 8192 1 > size-4096(DMA) 0 0 4096 1 > size-4096 109 109 4096 1 > size-2048(DMA) 0 0 2048 2 > size-2048 682 708 2048 2 > size-1024(DMA) 0 0 1024 4 > size-1024 373 404 1024 4 > size-512(DMA) 1 8 512 8 > size-512 491 560 512 8 > size-256(DMA) 0 0 256 15 > size-256 1560 2295 256 15 > size-128(DMA) 0 0 128 30 > size-128 3870 17580 128 30 > size-64(DMA) 0 0 64 59 > size-32(DMA) 0 0 32 113 > size-64 4118 7080 64 59 > size-32 5920 19097 32 113 > kmem_cache 150 150 256 15 > > > ===============> > > > _______________________________________________ > Ocfs2-users mailing list > Ocfs2-users@oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users >
I think sunil just wants to ensure that if something is urgent and production its good to get formal support and filing bugs for stuff in bugzilla is looked at for sure but might not get the same priority . Nothing more. Note also that the guys work hard on making this product good. And they do their very best to do it the right way. I must say that sometimes language is a hit harsh and disrespectful to say the least. Which is never really appreciated. A little bit of respect and more constructive feedback usually goes a very long way. Everyone is tryong their best. -----Original Message----- From: "Alexei_Roudnev" <Alexei_Roudnev@exigengroup.com> To: "Sunil Mushran" <Sunil.Mushran@oracle.com>; "John Lange" <john.lange@open-it.ca> Cc: "ocfs2-users" <ocfs2-users@oss.oracle.com>; "Lars Marowsky-Bree" <lmb@suse.de> Sent: 3/8/07 5:36 PM Subject: Re: [Ocfs2-users] ocfs2 is still eating memory Sunil, you DONT UNDERSTAND. They DONT ASK for the SUPPORT. They ask, "HOW WE CAN REPORT A BUG?" I had the same problem many times - there IS NOT a simple way to report a bug to the Novell. Not a surprise that systems are so buggy. For the beta versions, if you are not signed as a beta tester, then there is not easy way to report bug, too. (Signing as a beta tester means many obligations, and what if I am going to test a few components only?) For now, I am testing SP1 Beta4 (I have access as a partner so that we can test new software before it is released). I never asked for suppirt, but I saw a bugs many many times, and each time when I tried to report it (possible or absolute bug), it was a headache. Let's take open-iscsi - it require few small improvements for 100% sure, I can test them in our case, many other users can test them too, but we CAN;T REPORT them. (Just to have a list: - lvm is not called after the iscsi, so it don't see open-iscsi. On the other hand, having multiport support dropped in new iSCSI means that you can't use human readable names from disk/by-path but must use multipath disk ID's, so the only way to do it is lvm - but lvm is not called after iSCSI; - documentation have numerous bugs and don't explain how to mount iSCSI disks (SuSe dropped netfs and did not add anything instead); - few actions require timeouts; for example, you should wait 5 - 10 seconds after discovery and before conenction. ) They have the same problem. tested version, something don't work, call support _We see a problem_ , and got response _you have not premium support so we wil not talk_ (next time when I see your home burning, I call you and you say _dont telemarket me, hang on_ instead of _what's the matter? OI, it's serious. let's look together). The sad story is that many of these bugs are easy to fix, and that system itself is excellent... but quality is far from production grade, and the futher the worst. ----- Original Message ----- From: "Sunil Mushran" <Sunil.Mushran@oracle.com> To: "John Lange" <john.lange@open-it.ca> Cc: "ocfs2-users" <ocfs2-users@oss.oracle.com>; "Lars Marowsky-Bree" <lmb@suse.de> Sent: Thursday, March 08, 2007 4:37 PM Subject: Re: [Ocfs2-users] ocfs2 is still eating memory> If you are running a prod shop, you should looking into buying support. > > John Lange wrote: > > On Mon, 2007-03-05 at 13:46 -0800, Sunil Mushran wrote: > > > >> Well, kswapd is supposed to flush the caches. As in, the vm > >> controls the lifetime of the inodes in the inode_cache not ocfs2. > >> > >> All ocfs2 can do is free the memory associated with the inode when > >> asked to. And it does that when you manually flush the cache. Qs is > >> why the vm is not doing it on its own. (fwiw, you are on a betakernel.)> >> > > > > We are using beta kernels in an attempt to solve this problem. As > > everyone knows, the most recent official SUSE kernel (2.6.16.21-0.25 i > > believe?) completely broke ocfs2. Downgrading to 2.6.16.21-0.15 solves > > that problem but the memory issue remains. > > > > So as far as I am aware, there is no SUSE kernel that works with ocfs2 > > which is where we find ourselves today. > > > > I just upgraded to the latest KOTD: > > > > 2.6.16.42-SLES10_SP1_BRANCH_20070307114604-smp > > > > And still, when running ocfs2, all ram gets consumed. > > > > Right now Novell is playing the "you don't have premium support" game so > > where should I report this bug? > > > > Regards, > > > > John Lange > > > > > > > > _______________________________________________ > Ocfs2-users mailing list > Ocfs2-users@oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users >_______________________________________________ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi John -
I'm taking a look at the memory consumption issue you reported, and I
just can't seem to reproduce it in the manner you've described. I'm
running our CVS kernel, which at this point is really the same thing as
the KOTD with two OCFS2 DLM fixes I added today that should be entirely
unrelated to this bug.
I created a file system with about 890,000 files, rebooted with mem=512M
and did a find -exec stat {}\; on the file system. I can see it sucking
down all the memory as you described, but it's not OOM killing or even
going into swap.
I've been seeing this for about the last 15 minutes, give or take the
usual fluctuations:
r b swpd free buff cache si so bi bo in cs us sy
id wa st
6 1 168 5028 257956 15388 0 0 1045 0 1466 17997 3 9
71 17
Running a quick and dirty vmpressure program that allocates a chunk of
RAM and then writes to each page caused the system to properly drop
inodes from cache to honor the allocation request.
When I created the files, the inodes were using about 700 MB of RAM (8
GB machine), but I had run the vmpressure program earlier to see if they
would actually get dropped.
Out of curiosity, can you reproduce this with a vanilla kernel?
- -Jeff
- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org
iD8DBQFF+GxWLPWxlyuTD7IRAuGrAKCQ8xWlhCHPKBMpD3lcj3HAP1m/cgCdEJtY
8FNUIo/DRbbDFSGY9fVIprc=v4eq
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 John Lange wrote:> On Wed, 2007-03-14 at 17:42 -0400, Jeff Mahoney wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> >> Hi John - >> >> I'm taking a look at the memory consumption issue you reported, and I >> just can't seem to reproduce it in the manner you've described. I'm >> running our CVS kernel, which at this point is really the same thing as >> the KOTD with two OCFS2 DLM fixes I added today that should be entirely >> unrelated to this bug. >> >> I created a file system with about 890,000 files, rebooted with mem=512M >> and did a find -exec stat {}\; on the file system. I can see it sucking >> down all the memory as you described, but it's not OOM killing or even >> going into swap. > > Agreed, it does not go into swap. It would not make any sense for the > kernel to swap cache pages since it can just flush them (if they are > clean) or write them to disk (if they are dirty). > > They question is, why is it not flushing the cache unless you force it > to do so? > > I only ran the test environment just long enough to see that it was > exhibiting similar behavior. Even still I did see processes killed > though I didn't see them in the logs. However, that is might be because > syslog was one of the processes killed. > > My suspicion is disk activity alone might not cause oom-killer since the > kernel doesn't consider cache pages as normal memory usage. Once it > fills up the ram it might be forcing a flush on some pages just to keep > its head above water. > > However, if you start firing up other applications and consuming ram in > the normal way that might trigger it. I really can't say because I've > never done that much extensive testing. > > In any case, having the file system use up all the RAM that way is not > normal.If there is no other demand for memory, why shouldn't caches stay in memory? That's the entire point of caches. If we start OOM killing processes due to the caches taking all the memory, that's absolutely a bug.> Here is what Sunil Mushran from Oracle had to say about the issue: > >> Well, kswapd is supposed to flush the caches. As in, the vm >> controls the lifetime of the inodes in the inode_cache not ocfs2. >> >> All ocfs2 can do is free the memory associated with the inode when >> asked to. And it does that when you manually flush the cache. Qs is >> why the vm is not doing it on its own. > > So he is saying there is a problem with the way the kernel is handling > virtual memory. Why it only happens with ocfs2 on SUSE is unknown.Now that's the interesting part. Might you be willing to try a mainline kernel with OCFS2 1.2? If this is a SUSE problem, I'd like to isolate it. - -Jeff - -- Jeff Mahoney SUSE Labs -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFF+fxrLPWxlyuTD7IRAjPOAJ0WtR+NvKDxzzaYPVrlSlVKx0kF4wCgmzHL IQZGgPj1K9V+hGqoS2gx+ro=U1LW -----END PGP SIGNATURE-----