Hi Guys, My users are reporting some issues with memory on our lustre 1.8.1 clients. It looks like when they submit a single job at a time the run time was about 4.5 minutes. However, when they ran multiple jobs (10 or less) on a client with 192GB of memory on a single node the run time for each job was exceeding 3-4X the run time for the single process. They also noticed that the swap space kept climbing even though there was plenty of free memory on the system. Could this possibly be related to the lustre client? Does it reserve any memory that is not accessible by any other process even though it might not be in use? Thanks much, -J -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100419/10ddd4cf/attachment-0001.html
There is a known problem with the DLM LRU size that may be affecting
you. It may be something else too. Please check /proc/
{slabinfo,meminfo} to see what is using the memory on the client.
Cheers, Andreas
On 2010-04-19, at 10:43, Jagga Soorma <jagga13 at gmail.com> wrote:
> Hi Guys,
>
> My users are reporting some issues with memory on our lustre 1.8.1
> clients. It looks like when they submit a single job at a time the
> run time was about 4.5 minutes. However, when they ran multiple
> jobs (10 or less) on a client with 192GB of memory on a single node
> the run time for each job was exceeding 3-4X the run time for the
> single process. They also noticed that the swap space kept climbing
> even though there was plenty of free memory on the system. Could
> this possibly be related to the lustre client? Does it reserve any
> memory that is not accessible by any other process even though it
> might not be in use?
>
> Thanks much,
> -J
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
Thanks for the response Andreas. What is the known problem with the DLM LRU size? Here is what my slabinfo/meminfo look like on one of the clients. I don''t see anything out of the ordinary: (then again there are no jobs currently running on this system) Thanks -J -- slabinfo: .. slabinfo - version: 2.1 # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> nfs_direct_cache 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 nfs_write_data 36 44 704 11 2 : tunables 54 27 8 : slabdata 4 4 0 nfs_read_data 32 33 704 11 2 : tunables 54 27 8 : slabdata 3 3 0 nfs_inode_cache 0 0 984 4 1 : tunables 54 27 8 : slabdata 0 0 0 nfs_page 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 rpc_buffers 8 8 2048 2 1 : tunables 24 12 8 : slabdata 4 4 0 rpc_tasks 8 12 320 12 1 : tunables 54 27 8 : slabdata 1 1 0 rpc_inode_cache 0 0 832 4 1 : tunables 54 27 8 : slabdata 0 0 0 ll_async_page 326589 328572 320 12 1 : tunables 54 27 8 : slabdata 27381 27381 0 ll_file_data 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0 lustre_inode_cache 769 772 896 4 1 : tunables 54 27 8 : slabdata 193 193 0 lov_oinfo 1322 1392 320 12 1 : tunables 54 27 8 : slabdata 116 116 0 osc_quota_info 0 0 32 112 1 : tunables 120 60 8 : slabdata 0 0 0 ll_qunit_cache 0 0 112 34 1 : tunables 120 60 8 : slabdata 0 0 0 llcd_cache 0 0 3952 1 1 : tunables 24 12 8 : slabdata 0 0 0 ptlrpc_cbdatas 0 0 32 112 1 : tunables 120 60 8 : slabdata 0 0 0 interval_node 1166 3240 128 30 1 : tunables 120 60 8 : slabdata 108 108 0 ldlm_locks 2624 3688 512 8 1 : tunables 54 27 8 : slabdata 461 461 0 ldlm_resources 2002 3340 384 10 1 : tunables 54 27 8 : slabdata 334 334 0 ll_import_cache 0 0 1248 3 1 : tunables 24 12 8 : slabdata 0 0 0 ll_obdo_cache 0 452282156 208 19 1 : tunables 120 60 8 : slabdata 0 23804324 0 ll_obd_dev_cache 13 13 5672 1 2 : tunables 8 4 0 : slabdata 13 13 0 obd_lvfs_ctxt_cache 0 0 96 40 1 : tunables 120 60 8 : slabdata 0 0 0 SDP 0 0 1728 4 2 : tunables 24 12 8 : slabdata 0 0 0 fib6_nodes 7 118 64 59 1 : tunables 120 60 8 : slabdata 2 2 0 ip6_dst_cache 14 36 320 12 1 : tunables 54 27 8 : slabdata 3 3 0 ndisc_cache 4 30 256 15 1 : tunables 120 60 8 : slabdata 2 2 0 RAWv6 35 36 960 4 1 : tunables 54 27 8 : slabdata 9 9 0 UDPLITEv6 0 0 960 4 1 : tunables 54 27 8 : slabdata 0 0 0 UDPv6 7 12 960 4 1 : tunables 54 27 8 : slabdata 3 3 0 tw_sock_TCPv6 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0 request_sock_TCPv6 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0 TCPv6 2 4 1792 2 1 : tunables 24 12 8 : slabdata 2 2 0 ib_mad 2069 2160 448 8 1 : tunables 54 27 8 : slabdata 270 270 6 fuse_request 0 0 608 6 1 : tunables 54 27 8 : slabdata 0 0 0 fuse_inode 0 0 704 11 2 : tunables 54 27 8 : slabdata 0 0 0 kcopyd_job 0 0 360 11 1 : tunables 54 27 8 : slabdata 0 0 0 dm_uevent 0 0 2608 3 2 : tunables 24 12 8 : slabdata 0 0 0 dm_clone_bio_info 0 0 16 202 1 : tunables 120 60 8 : slabdata 0 0 0 dm_rq_target_io 0 0 408 9 1 : tunables 54 27 8 : slabdata 0 0 0 dm_target_io 0 0 24 144 1 : tunables 120 60 8 : slabdata 0 0 0 dm_io 0 0 32 112 1 : tunables 120 60 8 : slabdata 0 0 0 uhci_urb_priv 1 67 56 67 1 : tunables 120 60 8 : slabdata 1 1 0 ext3_inode_cache 224598 224625 768 5 1 : tunables 54 27 8 : slabdata 44925 44925 0 ext3_xattr 0 0 88 44 1 : tunables 120 60 8 : slabdata 0 0 0 journal_handle 9 288 24 144 1 : tunables 120 60 8 : slabdata 2 2 0 journal_head 76 120 96 40 1 : tunables 120 60 8 : slabdata 3 3 3 revoke_table 4 202 16 202 1 : tunables 120 60 8 : slabdata 1 1 0 revoke_record 0 0 32 112 1 : tunables 120 60 8 : slabdata 0 0 0 sgpool-128 2 2 4096 1 1 : tunables 24 12 8 : slabdata 2 2 0 sgpool-64 2 2 2048 2 1 : tunables 24 12 8 : slabdata 1 1 0 sgpool-32 2 4 1024 4 1 : tunables 54 27 8 : slabdata 1 1 0 sgpool-16 2 8 512 8 1 : tunables 54 27 8 : slabdata 1 1 0 sgpool-8 2 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0 scsi_data_buffer 0 0 24 144 1 : tunables 120 60 8 : slabdata 0 0 0 scsi_io_context 0 0 112 34 1 : tunables 120 60 8 : slabdata 0 0 0 flow_cache 0 0 96 40 1 : tunables 120 60 8 : slabdata 0 0 0 cfq_io_context 46 207 168 23 1 : tunables 120 60 8 : slabdata 9 9 0 cfq_queue 42 224 136 28 1 : tunables 120 60 8 : slabdata 8 8 0 bsg_cmd 0 0 312 12 1 : tunables 54 27 8 : slabdata 0 0 0 mqueue_inode_cache 1 4 896 4 1 : tunables 54 27 8 : slabdata 1 1 0 isofs_inode_cache 0 0 608 6 1 : tunables 54 27 8 : slabdata 0 0 0 minix_inode_cache 0 0 624 6 1 : tunables 54 27 8 : slabdata 0 0 0 hugetlbfs_inode_cache 1 7 576 7 1 : tunables 54 27 8 : slabdata 1 1 0 dnotify_cache 0 0 40 92 1 : tunables 120 60 8 : slabdata 0 0 0 dquot 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 inotify_event_cache 3 92 40 92 1 : tunables 120 60 8 : slabdata 1 1 0 inotify_watch_cache 93 212 72 53 1 : tunables 120 60 8 : slabdata 4 4 0 kioctx 0 0 384 10 1 : tunables 54 27 8 : slabdata 0 0 0 kiocb 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 fasync_cache 0 0 24 144 1 : tunables 120 60 8 : slabdata 0 0 0 shmem_inode_cache 870 960 784 5 1 : tunables 54 27 8 : slabdata 192 192 0 pid_namespace 0 0 2112 3 2 : tunables 24 12 8 : slabdata 0 0 0 nsproxy 0 0 56 67 1 : tunables 120 60 8 : slabdata 0 0 0 posix_timers_cache 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0 uid_cache 5 30 128 30 1 : tunables 120 60 8 : slabdata 1 1 0 UNIX 125 330 704 11 2 : tunables 54 27 8 : slabdata 30 30 0 ip_mrt_cache 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 UDP-Lite 0 0 832 9 2 : tunables 54 27 8 : slabdata 0 0 0 tcp_bind_bucket 8 118 64 59 1 : tunables 120 60 8 : slabdata 2 2 0 inet_peer_cache 1 59 64 59 1 : tunables 120 60 8 : slabdata 1 1 0 secpath_cache 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 xfrm_dst_cache 0 0 384 10 1 : tunables 54 27 8 : slabdata 0 0 0 ip_fib_alias 0 0 32 112 1 : tunables 120 60 8 : slabdata 0 0 0 ip_fib_hash 15 106 72 53 1 : tunables 120 60 8 : slabdata 2 2 0 ip_dst_cache 24 72 320 12 1 : tunables 54 27 8 : slabdata 6 6 2 arp_cache 3 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0 RAW 33 35 768 5 1 : tunables 54 27 8 : slabdata 7 7 0 UDP 9 18 832 9 2 : tunables 54 27 8 : slabdata 2 2 0 tw_sock_TCP 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0 request_sock_TCP 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 TCP 11 16 1664 4 2 : tunables 24 12 8 : slabdata 4 4 0 eventpoll_pwq 69 265 72 53 1 : tunables 120 60 8 : slabdata 5 5 0 eventpoll_epi 69 210 128 30 1 : tunables 120 60 8 : slabdata 7 7 0 pfm_event_set 0 0 57344 1 16 : tunables 8 4 0 : slabdata 0 0 0 pfm_context 0 0 8192 1 2 : tunables 8 4 0 : slabdata 0 0 0 blkdev_integrity 0 0 112 34 1 : tunables 120 60 8 : slabdata 0 0 0 blkdev_queue 10 12 2264 3 2 : tunables 24 12 8 : slabdata 4 4 0 blkdev_requests 13 20 368 10 1 : tunables 54 27 8 : slabdata 2 2 0 blkdev_ioc 44 371 72 53 1 : tunables 120 60 8 : slabdata 7 7 0 biovec-256 2 2 4096 1 1 : tunables 24 12 8 : slabdata 2 2 0 biovec-128 2 4 2048 2 1 : tunables 24 12 8 : slabdata 2 2 0 biovec-64 2 8 1024 4 1 : tunables 54 27 8 : slabdata 2 2 0 biovec-16 2 30 256 15 1 : tunables 120 60 8 : slabdata 2 2 0 biovec-4 2 118 64 59 1 : tunables 120 60 8 : slabdata 2 2 0 biovec-1 42 404 16 202 1 : tunables 120 60 8 : slabdata 2 2 3 bio_integrity_payload 2 60 128 30 1 : tunables 120 60 8 : slabdata 2 2 0 bio 8 60 128 30 1 : tunables 120 60 8 : slabdata 2 2 1 sock_inode_cache 232 372 640 6 1 : tunables 54 27 8 : slabdata 62 62 0 skbuff_fclone_cache 7 7 512 7 1 : tunables 54 27 8 : slabdata 1 1 0 skbuff_head_cache 5028 6210 256 15 1 : tunables 120 60 8 : slabdata 414 414 35 file_lock_cache 4 66 176 22 1 : tunables 120 60 8 : slabdata 3 3 0 Acpi-Operand 889 1802 72 53 1 : tunables 120 60 8 : slabdata 34 34 0 Acpi-ParseExt 0 0 72 53 1 : tunables 120 60 8 : slabdata 0 0 0 Acpi-Parse 0 0 48 77 1 : tunables 120 60 8 : slabdata 0 0 0 Acpi-State 0 0 80 48 1 : tunables 120 60 8 : slabdata 0 0 0 Acpi-Namespace 617 672 32 112 1 : tunables 120 60 8 : slabdata 6 6 0 task_delay_info 354 918 112 34 1 : tunables 120 60 8 : slabdata 27 27 0 taskstats 0 0 328 12 1 : tunables 54 27 8 : slabdata 0 0 0 page_cgroup 0 0 40 92 1 : tunables 120 60 8 : slabdata 0 0 0 proc_inode_cache 1431 1458 608 6 1 : tunables 54 27 8 : slabdata 243 243 0 sigqueue 8 96 160 24 1 : tunables 120 60 8 : slabdata 4 4 0 radix_tree_node 14146 15386 552 7 1 : tunables 54 27 8 : slabdata 2198 2198 0 bdev_cache 5 20 768 5 1 : tunables 54 27 8 : slabdata 4 4 0 sysfs_dir_cache 19120 19296 80 48 1 : tunables 120 60 8 : slabdata 402 402 0 mnt_cache 30 60 256 15 1 : tunables 120 60 8 : slabdata 4 4 0 inode_cache 1327 1344 560 7 1 : tunables 54 27 8 : slabdata 192 192 0 dentry 276001 276203 208 19 1 : tunables 120 60 8 : slabdata 14537 14537 0 filp 1054 2760 192 20 1 : tunables 120 60 8 : slabdata 138 138 86 names_cache 18 18 4096 1 1 : tunables 24 12 8 : slabdata 18 18 1 key_jar 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0 buffer_head 73846 73889 104 37 1 : tunables 120 60 8 : slabdata 1997 1997 1 mm_struct 80 136 896 4 1 : tunables 54 27 8 : slabdata 34 34 1 vm_area_struct 2311 3784 176 22 1 : tunables 120 60 8 : slabdata 172 172 29 fs_cache 75 590 64 59 1 : tunables 120 60 8 : slabdata 10 10 1 files_cache 63 165 768 5 1 : tunables 54 27 8 : slabdata 33 33 1 signal_cache 297 420 960 4 1 : tunables 54 27 8 : slabdata 105 105 0 sighand_cache 295 381 2112 3 2 : tunables 24 12 8 : slabdata 127 127 0 task_xstate 105 256 512 8 1 : tunables 54 27 8 : slabdata 32 32 0 task_struct 349 350 5872 1 2 : tunables 8 4 0 : slabdata 349 350 0 anon_vma 777 1584 24 144 1 : tunables 120 60 8 : slabdata 11 11 0 pid 342 870 128 30 1 : tunables 120 60 8 : slabdata 29 29 0 shared_policy_node 0 0 48 77 1 : tunables 120 60 8 : slabdata 0 0 0 numa_policy 15 112 136 28 1 : tunables 120 60 8 : slabdata 4 4 0 idr_layer_cache 282 315 544 7 1 : tunables 54 27 8 : slabdata 45 45 0 size-4194304(DMA) 0 0 4194304 1 1024 : tunables 1 1 0 : slabdata 0 0 0 size-4194304 0 0 4194304 1 1024 : tunables 1 1 0 : slabdata 0 0 0 size-2097152(DMA) 0 0 2097152 1 512 : tunables 1 1 0 : slabdata 0 0 0 size-2097152 0 0 2097152 1 512 : tunables 1 1 0 : slabdata 0 0 0 size-1048576(DMA) 0 0 1048576 1 256 : tunables 1 1 0 : slabdata 0 0 0 size-1048576 0 0 1048576 1 256 : tunables 1 1 0 : slabdata 0 0 0 size-524288(DMA) 0 0 524288 1 128 : tunables 1 1 0 : slabdata 0 0 0 size-524288 0 0 524288 1 128 : tunables 1 1 0 : slabdata 0 0 0 size-262144(DMA) 0 0 262144 1 64 : tunables 1 1 0 : slabdata 0 0 0 size-262144 0 0 262144 1 64 : tunables 1 1 0 : slabdata 0 0 0 size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0 size-131072 3 3 131072 1 32 : tunables 8 4 0 : slabdata 3 3 0 size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0 size-65536 6 6 65536 1 16 : tunables 8 4 0 : slabdata 6 6 0 size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0 size-32768 8 8 32768 1 8 : tunables 8 4 0 : slabdata 8 8 0 size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0 size-16384 43 43 16384 1 4 : tunables 8 4 0 : slabdata 43 43 0 size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 : slabdata 0 0 0 size-8192 3610 3610 8192 1 2 : tunables 8 4 0 : slabdata 3610 3610 0 size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8 : slabdata 0 0 0 size-4096 1769 1769 4096 1 1 : tunables 24 12 8 : slabdata 1769 1769 0 size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8 : slabdata 0 0 0 size-2048 4598 4630 2048 2 1 : tunables 24 12 8 : slabdata 2315 2315 1 size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8 : slabdata 0 0 0 size-1024 4749 4784 1024 4 1 : tunables 54 27 8 : slabdata 1196 1196 0 size-512(DMA) 0 0 512 8 1 : tunables 54 27 8 : slabdata 0 0 0 size-512 1406 1440 512 8 1 : tunables 54 27 8 : slabdata 180 180 29 size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 size-256 5428 5670 256 15 1 : tunables 120 60 8 : slabdata 378 378 2 size-128(DMA) 0 0 128 30 1 : tunables 120 60 8 : slabdata 0 0 0 size-64(DMA) 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 size-64 21391 43306 64 59 1 : tunables 120 60 8 : slabdata 734 734 0 size-32(DMA) 0 0 32 112 1 : tunables 120 60 8 : slabdata 0 0 0 size-128 10539 31650 128 30 1 : tunables 120 60 8 : slabdata 1055 1055 0 size-32 11992 13552 32 112 1 : tunables 120 60 8 : slabdata 121 121 6 kmem_cache 181 181 4224 1 2 : tunables 8 4 0 : slabdata 181 181 0 .. -- -- meminfo .. MemTotal: 198091444 kB MemFree: 99978176 kB Buffers: 268288 kB Cached: 1457808 kB SwapCached: 23672 kB Active: 1667172 kB Inactive: 114552 kB SwapTotal: 75505460 kB SwapFree: 75461372 kB Dirty: 116 kB Writeback: 0 kB AnonPages: 53284 kB Mapped: 8884 kB Slab: 95664132 kB SReclaimable: 256656 kB SUnreclaim: 95407476 kB PageTables: 2368 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 174551180 kB Committed_AS: 137540 kB VmallocTotal: 34359738367 kB VmallocUsed: 588416 kB VmallocChunk: 34359149923 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 8432 kB DirectMap2M: 201308160 kB .. -- On Mon, Apr 19, 2010 at 10:07 AM, Andreas Dilger <andreas.dilger at oracle.com>wrote:> There is a known problem with the DLM LRU size that may be affecting you. > It may be something else too. Please check /proc/{slabinfo,meminfo} to see > what is using the memory on the client. > > Cheers, Andreas > > > On 2010-04-19, at 10:43, Jagga Soorma <jagga13 at gmail.com> wrote: > > Hi Guys, >> >> My users are reporting some issues with memory on our lustre 1.8.1 >> clients. It looks like when they submit a single job at a time the run time >> was about 4.5 minutes. However, when they ran multiple jobs (10 or less) on >> a client with 192GB of memory on a single node the run time for each job was >> exceeding 3-4X the run time for the single process. They also noticed that >> the swap space kept climbing even though there was plenty of free memory on >> the system. Could this possibly be related to the lustre client? Does it >> reserve any memory that is not accessible by any other process even though >> it might not be in use? >> >> Thanks much, >> -J >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100419/c2bda61b/attachment-0001.html
Actually this does not seem correct: SUnreclaim: 95407476 kB Shouldn''t this be a lot smaller? -Simran On Mon, Apr 19, 2010 at 10:16 AM, Jagga Soorma <jagga13 at gmail.com> wrote:> Thanks for the response Andreas. > > What is the known problem with the DLM LRU size? Here is what my > slabinfo/meminfo look like on one of the clients. I don''t see anything out > of the ordinary: > > (then again there are no jobs currently running on this system) > > Thanks > -J > > -- > slabinfo: > .. > slabinfo - version: 2.1 > # name <active_objs> <num_objs> <objsize> <objperslab> > <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata > <active_slabs> <num_slabs> <sharedavail> > nfs_direct_cache 0 0 128 30 1 : tunables 120 60 8 > : slabdata 0 0 0 > nfs_write_data 36 44 704 11 2 : tunables 54 27 8 > : slabdata 4 4 0 > nfs_read_data 32 33 704 11 2 : tunables 54 27 8 > : slabdata 3 3 0 > nfs_inode_cache 0 0 984 4 1 : tunables 54 27 8 > : slabdata 0 0 0 > nfs_page 0 0 128 30 1 : tunables 120 60 8 > : slabdata 0 0 0 > rpc_buffers 8 8 2048 2 1 : tunables 24 12 8 > : slabdata 4 4 0 > rpc_tasks 8 12 320 12 1 : tunables 54 27 8 > : slabdata 1 1 0 > rpc_inode_cache 0 0 832 4 1 : tunables 54 27 8 > : slabdata 0 0 0 > ll_async_page 326589 328572 320 12 1 : tunables 54 27 8 > : slabdata 27381 27381 0 > ll_file_data 0 0 192 20 1 : tunables 120 60 8 > : slabdata 0 0 0 > lustre_inode_cache 769 772 896 4 1 : tunables 54 27 8 > : slabdata 193 193 0 > lov_oinfo 1322 1392 320 12 1 : tunables 54 27 8 > : slabdata 116 116 0 > osc_quota_info 0 0 32 112 1 : tunables 120 60 8 > : slabdata 0 0 0 > ll_qunit_cache 0 0 112 34 1 : tunables 120 60 8 > : slabdata 0 0 0 > llcd_cache 0 0 3952 1 1 : tunables 24 12 8 > : slabdata 0 0 0 > ptlrpc_cbdatas 0 0 32 112 1 : tunables 120 60 8 > : slabdata 0 0 0 > interval_node 1166 3240 128 30 1 : tunables 120 60 8 > : slabdata 108 108 0 > ldlm_locks 2624 3688 512 8 1 : tunables 54 27 8 > : slabdata 461 461 0 > ldlm_resources 2002 3340 384 10 1 : tunables 54 27 8 > : slabdata 334 334 0 > ll_import_cache 0 0 1248 3 1 : tunables 24 12 8 > : slabdata 0 0 0 > ll_obdo_cache 0 452282156 208 19 1 : tunables 120 60 > 8 : slabdata 0 23804324 0 > ll_obd_dev_cache 13 13 5672 1 2 : tunables 8 4 0 > : slabdata 13 13 0 > obd_lvfs_ctxt_cache 0 0 96 40 1 : tunables 120 60 > 8 : slabdata 0 0 0 > SDP 0 0 1728 4 2 : tunables 24 12 8 > : slabdata 0 0 0 > fib6_nodes 7 118 64 59 1 : tunables 120 60 8 > : slabdata 2 2 0 > ip6_dst_cache 14 36 320 12 1 : tunables 54 27 8 > : slabdata 3 3 0 > ndisc_cache 4 30 256 15 1 : tunables 120 60 8 > : slabdata 2 2 0 > RAWv6 35 36 960 4 1 : tunables 54 27 8 > : slabdata 9 9 0 > UDPLITEv6 0 0 960 4 1 : tunables 54 27 8 > : slabdata 0 0 0 > UDPv6 7 12 960 4 1 : tunables 54 27 8 > : slabdata 3 3 0 > tw_sock_TCPv6 0 0 192 20 1 : tunables 120 60 8 > : slabdata 0 0 0 > request_sock_TCPv6 0 0 192 20 1 : tunables 120 60 8 > : slabdata 0 0 0 > TCPv6 2 4 1792 2 1 : tunables 24 12 8 > : slabdata 2 2 0 > ib_mad 2069 2160 448 8 1 : tunables 54 27 8 > : slabdata 270 270 6 > fuse_request 0 0 608 6 1 : tunables 54 27 8 > : slabdata 0 0 0 > fuse_inode 0 0 704 11 2 : tunables 54 27 8 > : slabdata 0 0 0 > kcopyd_job 0 0 360 11 1 : tunables 54 27 8 > : slabdata 0 0 0 > dm_uevent 0 0 2608 3 2 : tunables 24 12 8 > : slabdata 0 0 0 > dm_clone_bio_info 0 0 16 202 1 : tunables 120 60 8 > : slabdata 0 0 0 > dm_rq_target_io 0 0 408 9 1 : tunables 54 27 8 > : slabdata 0 0 0 > dm_target_io 0 0 24 144 1 : tunables 120 60 8 > : slabdata 0 0 0 > dm_io 0 0 32 112 1 : tunables 120 60 8 > : slabdata 0 0 0 > uhci_urb_priv 1 67 56 67 1 : tunables 120 60 8 > : slabdata 1 1 0 > ext3_inode_cache 224598 224625 768 5 1 : tunables 54 27 8 > : slabdata 44925 44925 0 > ext3_xattr 0 0 88 44 1 : tunables 120 60 8 > : slabdata 0 0 0 > journal_handle 9 288 24 144 1 : tunables 120 60 8 > : slabdata 2 2 0 > journal_head 76 120 96 40 1 : tunables 120 60 8 > : slabdata 3 3 3 > revoke_table 4 202 16 202 1 : tunables 120 60 8 > : slabdata 1 1 0 > revoke_record 0 0 32 112 1 : tunables 120 60 8 > : slabdata 0 0 0 > sgpool-128 2 2 4096 1 1 : tunables 24 12 8 > : slabdata 2 2 0 > sgpool-64 2 2 2048 2 1 : tunables 24 12 8 > : slabdata 1 1 0 > sgpool-32 2 4 1024 4 1 : tunables 54 27 8 > : slabdata 1 1 0 > sgpool-16 2 8 512 8 1 : tunables 54 27 8 > : slabdata 1 1 0 > sgpool-8 2 15 256 15 1 : tunables 120 60 8 > : slabdata 1 1 0 > scsi_data_buffer 0 0 24 144 1 : tunables 120 60 8 > : slabdata 0 0 0 > scsi_io_context 0 0 112 34 1 : tunables 120 60 8 > : slabdata 0 0 0 > flow_cache 0 0 96 40 1 : tunables 120 60 8 > : slabdata 0 0 0 > cfq_io_context 46 207 168 23 1 : tunables 120 60 8 > : slabdata 9 9 0 > cfq_queue 42 224 136 28 1 : tunables 120 60 8 > : slabdata 8 8 0 > bsg_cmd 0 0 312 12 1 : tunables 54 27 8 > : slabdata 0 0 0 > mqueue_inode_cache 1 4 896 4 1 : tunables 54 27 8 > : slabdata 1 1 0 > isofs_inode_cache 0 0 608 6 1 : tunables 54 27 8 > : slabdata 0 0 0 > minix_inode_cache 0 0 624 6 1 : tunables 54 27 8 > : slabdata 0 0 0 > hugetlbfs_inode_cache 1 7 576 7 1 : tunables 54 > 27 8 : slabdata 1 1 0 > dnotify_cache 0 0 40 92 1 : tunables 120 60 8 > : slabdata 0 0 0 > dquot 0 0 256 15 1 : tunables 120 60 8 > : slabdata 0 0 0 > inotify_event_cache 3 92 40 92 1 : tunables 120 60 > 8 : slabdata 1 1 0 > inotify_watch_cache 93 212 72 53 1 : tunables 120 60 > 8 : slabdata 4 4 0 > kioctx 0 0 384 10 1 : tunables 54 27 8 > : slabdata 0 0 0 > kiocb 0 0 256 15 1 : tunables 120 60 8 > : slabdata 0 0 0 > fasync_cache 0 0 24 144 1 : tunables 120 60 8 > : slabdata 0 0 0 > shmem_inode_cache 870 960 784 5 1 : tunables 54 27 8 > : slabdata 192 192 0 > pid_namespace 0 0 2112 3 2 : tunables 24 12 8 > : slabdata 0 0 0 > nsproxy 0 0 56 67 1 : tunables 120 60 8 > : slabdata 0 0 0 > posix_timers_cache 0 0 192 20 1 : tunables 120 60 8 > : slabdata 0 0 0 > uid_cache 5 30 128 30 1 : tunables 120 60 8 > : slabdata 1 1 0 > UNIX 125 330 704 11 2 : tunables 54 27 8 > : slabdata 30 30 0 > ip_mrt_cache 0 0 128 30 1 : tunables 120 60 8 > : slabdata 0 0 0 > UDP-Lite 0 0 832 9 2 : tunables 54 27 8 > : slabdata 0 0 0 > tcp_bind_bucket 8 118 64 59 1 : tunables 120 60 8 > : slabdata 2 2 0 > inet_peer_cache 1 59 64 59 1 : tunables 120 60 8 > : slabdata 1 1 0 > secpath_cache 0 0 64 59 1 : tunables 120 60 8 > : slabdata 0 0 0 > xfrm_dst_cache 0 0 384 10 1 : tunables 54 27 8 > : slabdata 0 0 0 > ip_fib_alias 0 0 32 112 1 : tunables 120 60 8 > : slabdata 0 0 0 > ip_fib_hash 15 106 72 53 1 : tunables 120 60 8 > : slabdata 2 2 0 > ip_dst_cache 24 72 320 12 1 : tunables 54 27 8 > : slabdata 6 6 2 > arp_cache 3 15 256 15 1 : tunables 120 60 8 > : slabdata 1 1 0 > RAW 33 35 768 5 1 : tunables 54 27 8 > : slabdata 7 7 0 > UDP 9 18 832 9 2 : tunables 54 27 8 > : slabdata 2 2 0 > tw_sock_TCP 0 0 192 20 1 : tunables 120 60 8 > : slabdata 0 0 0 > request_sock_TCP 0 0 128 30 1 : tunables 120 60 8 > : slabdata 0 0 0 > TCP 11 16 1664 4 2 : tunables 24 12 8 > : slabdata 4 4 0 > eventpoll_pwq 69 265 72 53 1 : tunables 120 60 8 > : slabdata 5 5 0 > eventpoll_epi 69 210 128 30 1 : tunables 120 60 8 > : slabdata 7 7 0 > pfm_event_set 0 0 57344 1 16 : tunables 8 4 0 > : slabdata 0 0 0 > pfm_context 0 0 8192 1 2 : tunables 8 4 0 > : slabdata 0 0 0 > blkdev_integrity 0 0 112 34 1 : tunables 120 60 8 > : slabdata 0 0 0 > blkdev_queue 10 12 2264 3 2 : tunables 24 12 8 > : slabdata 4 4 0 > blkdev_requests 13 20 368 10 1 : tunables 54 27 8 > : slabdata 2 2 0 > blkdev_ioc 44 371 72 53 1 : tunables 120 60 8 > : slabdata 7 7 0 > biovec-256 2 2 4096 1 1 : tunables 24 12 8 > : slabdata 2 2 0 > biovec-128 2 4 2048 2 1 : tunables 24 12 8 > : slabdata 2 2 0 > biovec-64 2 8 1024 4 1 : tunables 54 27 8 > : slabdata 2 2 0 > biovec-16 2 30 256 15 1 : tunables 120 60 8 > : slabdata 2 2 0 > biovec-4 2 118 64 59 1 : tunables 120 60 8 > : slabdata 2 2 0 > biovec-1 42 404 16 202 1 : tunables 120 60 8 > : slabdata 2 2 3 > bio_integrity_payload 2 60 128 30 1 : tunables 120 > 60 8 : slabdata 2 2 0 > bio 8 60 128 30 1 : tunables 120 60 8 > : slabdata 2 2 1 > sock_inode_cache 232 372 640 6 1 : tunables 54 27 8 > : slabdata 62 62 0 > skbuff_fclone_cache 7 7 512 7 1 : tunables 54 27 > 8 : slabdata 1 1 0 > skbuff_head_cache 5028 6210 256 15 1 : tunables 120 60 8 > : slabdata 414 414 35 > file_lock_cache 4 66 176 22 1 : tunables 120 60 8 > : slabdata 3 3 0 > Acpi-Operand 889 1802 72 53 1 : tunables 120 60 8 > : slabdata 34 34 0 > Acpi-ParseExt 0 0 72 53 1 : tunables 120 60 8 > : slabdata 0 0 0 > Acpi-Parse 0 0 48 77 1 : tunables 120 60 8 > : slabdata 0 0 0 > Acpi-State 0 0 80 48 1 : tunables 120 60 8 > : slabdata 0 0 0 > Acpi-Namespace 617 672 32 112 1 : tunables 120 60 8 > : slabdata 6 6 0 > task_delay_info 354 918 112 34 1 : tunables 120 60 8 > : slabdata 27 27 0 > taskstats 0 0 328 12 1 : tunables 54 27 8 > : slabdata 0 0 0 > page_cgroup 0 0 40 92 1 : tunables 120 60 8 > : slabdata 0 0 0 > proc_inode_cache 1431 1458 608 6 1 : tunables 54 27 8 > : slabdata 243 243 0 > sigqueue 8 96 160 24 1 : tunables 120 60 8 > : slabdata 4 4 0 > radix_tree_node 14146 15386 552 7 1 : tunables 54 27 8 > : slabdata 2198 2198 0 > bdev_cache 5 20 768 5 1 : tunables 54 27 8 > : slabdata 4 4 0 > sysfs_dir_cache 19120 19296 80 48 1 : tunables 120 60 8 > : slabdata 402 402 0 > mnt_cache 30 60 256 15 1 : tunables 120 60 8 > : slabdata 4 4 0 > inode_cache 1327 1344 560 7 1 : tunables 54 27 8 > : slabdata 192 192 0 > dentry 276001 276203 208 19 1 : tunables 120 60 8 > : slabdata 14537 14537 0 > filp 1054 2760 192 20 1 : tunables 120 60 8 > : slabdata 138 138 86 > names_cache 18 18 4096 1 1 : tunables 24 12 8 > : slabdata 18 18 1 > key_jar 0 0 192 20 1 : tunables 120 60 8 > : slabdata 0 0 0 > buffer_head 73846 73889 104 37 1 : tunables 120 60 8 > : slabdata 1997 1997 1 > mm_struct 80 136 896 4 1 : tunables 54 27 8 > : slabdata 34 34 1 > vm_area_struct 2311 3784 176 22 1 : tunables 120 60 8 > : slabdata 172 172 29 > fs_cache 75 590 64 59 1 : tunables 120 60 8 > : slabdata 10 10 1 > files_cache 63 165 768 5 1 : tunables 54 27 8 > : slabdata 33 33 1 > signal_cache 297 420 960 4 1 : tunables 54 27 8 > : slabdata 105 105 0 > sighand_cache 295 381 2112 3 2 : tunables 24 12 8 > : slabdata 127 127 0 > task_xstate 105 256 512 8 1 : tunables 54 27 8 > : slabdata 32 32 0 > task_struct 349 350 5872 1 2 : tunables 8 4 0 > : slabdata 349 350 0 > anon_vma 777 1584 24 144 1 : tunables 120 60 8 > : slabdata 11 11 0 > pid 342 870 128 30 1 : tunables 120 60 8 > : slabdata 29 29 0 > shared_policy_node 0 0 48 77 1 : tunables 120 60 8 > : slabdata 0 0 0 > numa_policy 15 112 136 28 1 : tunables 120 60 8 > : slabdata 4 4 0 > idr_layer_cache 282 315 544 7 1 : tunables 54 27 8 > : slabdata 45 45 0 > size-4194304(DMA) 0 0 4194304 1 1024 : tunables 1 1 0 > : slabdata 0 0 0 > size-4194304 0 0 4194304 1 1024 : tunables 1 1 0 > : slabdata 0 0 0 > size-2097152(DMA) 0 0 2097152 1 512 : tunables 1 1 0 > : slabdata 0 0 0 > size-2097152 0 0 2097152 1 512 : tunables 1 1 0 > : slabdata 0 0 0 > size-1048576(DMA) 0 0 1048576 1 256 : tunables 1 1 0 > : slabdata 0 0 0 > size-1048576 0 0 1048576 1 256 : tunables 1 1 0 > : slabdata 0 0 0 > size-524288(DMA) 0 0 524288 1 128 : tunables 1 1 0 > : slabdata 0 0 0 > size-524288 0 0 524288 1 128 : tunables 1 1 0 > : slabdata 0 0 0 > size-262144(DMA) 0 0 262144 1 64 : tunables 1 1 0 > : slabdata 0 0 0 > size-262144 0 0 262144 1 64 : tunables 1 1 0 > : slabdata 0 0 0 > size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 > : slabdata 0 0 0 > size-131072 3 3 131072 1 32 : tunables 8 4 0 > : slabdata 3 3 0 > size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 > : slabdata 0 0 0 > size-65536 6 6 65536 1 16 : tunables 8 4 0 > : slabdata 6 6 0 > size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 > : slabdata 0 0 0 > size-32768 8 8 32768 1 8 : tunables 8 4 0 > : slabdata 8 8 0 > size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 > : slabdata 0 0 0 > size-16384 43 43 16384 1 4 : tunables 8 4 0 > : slabdata 43 43 0 > size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 > : slabdata 0 0 0 > size-8192 3610 3610 8192 1 2 : tunables 8 4 0 > : slabdata 3610 3610 0 > size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8 > : slabdata 0 0 0 > size-4096 1769 1769 4096 1 1 : tunables 24 12 8 > : slabdata 1769 1769 0 > size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8 > : slabdata 0 0 0 > size-2048 4598 4630 2048 2 1 : tunables 24 12 8 > : slabdata 2315 2315 1 > size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8 > : slabdata 0 0 0 > size-1024 4749 4784 1024 4 1 : tunables 54 27 8 > : slabdata 1196 1196 0 > size-512(DMA) 0 0 512 8 1 : tunables 54 27 8 > : slabdata 0 0 0 > size-512 1406 1440 512 8 1 : tunables 54 27 8 > : slabdata 180 180 29 > size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 > : slabdata 0 0 0 > size-256 5428 5670 256 15 1 : tunables 120 60 8 > : slabdata 378 378 2 > size-128(DMA) 0 0 128 30 1 : tunables 120 60 8 > : slabdata 0 0 0 > size-64(DMA) 0 0 64 59 1 : tunables 120 60 8 > : slabdata 0 0 0 > size-64 21391 43306 64 59 1 : tunables 120 60 8 > : slabdata 734 734 0 > size-32(DMA) 0 0 32 112 1 : tunables 120 60 8 > : slabdata 0 0 0 > size-128 10539 31650 128 30 1 : tunables 120 60 8 > : slabdata 1055 1055 0 > size-32 11992 13552 32 112 1 : tunables 120 60 8 > : slabdata 121 121 6 > kmem_cache 181 181 4224 1 2 : tunables 8 4 0 > : slabdata 181 181 0 > .. > -- > > -- > meminfo > .. > MemTotal: 198091444 kB > MemFree: 99978176 kB > Buffers: 268288 kB > Cached: 1457808 kB > SwapCached: 23672 kB > Active: 1667172 kB > Inactive: 114552 kB > SwapTotal: 75505460 kB > SwapFree: 75461372 kB > Dirty: 116 kB > Writeback: 0 kB > AnonPages: 53284 kB > Mapped: 8884 kB > Slab: 95664132 kB > SReclaimable: 256656 kB > SUnreclaim: 95407476 kB > PageTables: 2368 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 174551180 kB > Committed_AS: 137540 kB > VmallocTotal: 34359738367 kB > VmallocUsed: 588416 kB > VmallocChunk: 34359149923 kB > HugePages_Total: 0 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > DirectMap4k: 8432 kB > DirectMap2M: 201308160 kB > .. > -- > > > On Mon, Apr 19, 2010 at 10:07 AM, Andreas Dilger < > andreas.dilger at oracle.com> wrote: > >> There is a known problem with the DLM LRU size that may be affecting you. >> It may be something else too. Please check /proc/{slabinfo,meminfo} to see >> what is using the memory on the client. >> >> Cheers, Andreas >> >> >> On 2010-04-19, at 10:43, Jagga Soorma <jagga13 at gmail.com> wrote: >> >> Hi Guys, >>> >>> My users are reporting some issues with memory on our lustre 1.8.1 >>> clients. It looks like when they submit a single job at a time the run time >>> was about 4.5 minutes. However, when they ran multiple jobs (10 or less) on >>> a client with 192GB of memory on a single node the run time for each job was >>> exceeding 3-4X the run time for the single process. They also noticed that >>> the swap space kept climbing even though there was plenty of free memory on >>> the system. Could this possibly be related to the lustre client? Does it >>> reserve any memory that is not accessible by any other process even though >>> it might not be in use? >>> >>> Thanks much, >>> -J >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100419/9e818385/attachment-0001.html
Andreas,
I am seeing the problem again on one of my hosts and here is a live capture
of the data. Can you assist with this?
--
# free
total used free shared buffers cached
Mem: 198091444 197636852 454592 0 4260 34251452
-/+ buffers/cache: 163381140 34710304
Swap: 75505460 10281796 65223664
# cat /proc/meminfo
MemTotal: 198091444 kB
MemFree: 458048 kB
Buffers: 4268 kB
Cached: 34099372 kB
SwapCached: 7730744 kB
Active: 62919152 kB
Inactive: 34107188 kB
SwapTotal: 75505460 kB
SwapFree: 65220676 kB
Dirty: 444 kB
Writeback: 0 kB
AnonPages: 58704728 kB
Mapped: 12036 kB
Slab: 99806476 kB
SReclaimable: 118532 kB
SUnreclaim: 99687944 kB
PageTables: 131200 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 174551180 kB
Committed_AS: 65739660 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 588416 kB
VmallocChunk: 34359149923 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 8432 kB
DirectMap2M: 201308160 kB
# cat /proc/slabinfo
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize>
<objperslab>
<pagesperslab> : tunables <limit> <batchcount>
<sharedfactor> : slabdata
<active_slabs> <num_slabs> <sharedavail>
nfs_direct_cache 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
nfs_write_data 36 44 704 11 2 : tunables 54 27 8 :
slabdata 4 4 0
nfs_read_data 32 33 704 11 2 : tunables 54 27 8 :
slabdata 3 3 0
nfs_inode_cache 0 0 984 4 1 : tunables 54 27 8 :
slabdata 0 0 0
nfs_page 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
rpc_buffers 8 8 2048 2 1 : tunables 24 12 8 :
slabdata 4 4 0
rpc_tasks 8 12 320 12 1 : tunables 54 27 8 :
slabdata 1 1 0
rpc_inode_cache 0 0 832 4 1 : tunables 54 27 8 :
slabdata 0 0 0
ll_async_page 8494811 8507076 320 12 1 : tunables 54 27 8
: slabdata 708923 708923 216
ll_file_data 16 40 192 20 1 : tunables 120 60 8 :
slabdata 2 2 0
lustre_inode_cache 95 184 896 4 1 : tunables 54 27 8
: slabdata 46 46 0
lov_oinfo 56 180 320 12 1 : tunables 54 27 8 :
slabdata 15 15 0
osc_quota_info 0 0 32 112 1 : tunables 120 60 8 :
slabdata 0 0 0
ll_qunit_cache 0 0 112 34 1 : tunables 120 60 8 :
slabdata 0 0 0
llcd_cache 0 0 3952 1 1 : tunables 24 12 8 :
slabdata 0 0 0
ptlrpc_cbdatas 0 0 32 112 1 : tunables 120 60 8 :
slabdata 0 0 0
interval_node 1680 5730 128 30 1 : tunables 120 60 8 :
slabdata 191 191 0
ldlm_locks 2255 6232 512 8 1 : tunables 54 27 8 :
slabdata 779 779 0
ldlm_resources 2227 5570 384 10 1 : tunables 54 27 8 :
slabdata 557 557 0
ll_import_cache 0 0 1248 3 1 : tunables 24 12 8 :
slabdata 0 0 0
ll_obdo_cache 0 459630919 208 19 1 : tunables 120 60
8 : slabdata 0 24191101 0
ll_obd_dev_cache 13 13 5672 1 2 : tunables 8 4 0 :
slabdata 13 13 0
obd_lvfs_ctxt_cache 0 0 96 40 1 : tunables 120 60 8
: slabdata 0 0 0
SDP 0 0 1728 4 2 : tunables 24 12 8 :
slabdata 0 0 0
fib6_nodes 7 59 64 59 1 : tunables 120 60 8 :
slabdata 1 1 0
ip6_dst_cache 10 24 320 12 1 : tunables 54 27 8 :
slabdata 2 2 0
ndisc_cache 3 30 256 15 1 : tunables 120 60 8 :
slabdata 2 2 0
RAWv6 35 36 960 4 1 : tunables 54 27 8 :
slabdata 9 9 0
UDPLITEv6 0 0 960 4 1 : tunables 54 27 8 :
slabdata 0 0 0
UDPv6 7 12 960 4 1 : tunables 54 27 8 :
slabdata 3 3 0
tw_sock_TCPv6 0 0 192 20 1 : tunables 120 60 8 :
slabdata 0 0 0
request_sock_TCPv6 0 0 192 20 1 : tunables 120 60 8
: slabdata 0 0 0
TCPv6 3 4 1792 2 1 : tunables 24 12 8 :
slabdata 2 2 0
ib_mad 2051 2096 448 8 1 : tunables 54 27 8 :
slabdata 262 262 0
fuse_request 0 0 608 6 1 : tunables 54 27 8 :
slabdata 0 0 0
fuse_inode 0 0 704 11 2 : tunables 54 27 8 :
slabdata 0 0 0
kcopyd_job 0 0 360 11 1 : tunables 54 27 8 :
slabdata 0 0 0
dm_uevent 0 0 2608 3 2 : tunables 24 12 8 :
slabdata 0 0 0
dm_clone_bio_info 0 0 16 202 1 : tunables 120 60 8 :
slabdata 0 0 0
dm_rq_target_io 0 0 408 9 1 : tunables 54 27 8 :
slabdata 0 0 0
dm_target_io 0 0 24 144 1 : tunables 120 60 8 :
slabdata 0 0 0
dm_io 0 0 32 112 1 : tunables 120 60 8 :
slabdata 0 0 0
uhci_urb_priv 1 67 56 67 1 : tunables 120 60 8 :
slabdata 1 1 0
ext3_inode_cache 2472 2610 768 5 1 : tunables 54 27 8 :
slabdata 522 522 0
ext3_xattr 0 0 88 44 1 : tunables 120 60 8 :
slabdata 0 0 0
journal_handle 56 288 24 144 1 : tunables 120 60 8 :
slabdata 2 2 0
journal_head 216 240 96 40 1 : tunables 120 60 8 :
slabdata 6 6 0
revoke_table 4 202 16 202 1 : tunables 120 60 8 :
slabdata 1 1 0
revoke_record 0 0 32 112 1 : tunables 120 60 8 :
slabdata 0 0 0
sgpool-128 2 2 4096 1 1 : tunables 24 12 8 :
slabdata 2 2 0
sgpool-64 2 2 2048 2 1 : tunables 24 12 8 :
slabdata 1 1 0
sgpool-32 2 4 1024 4 1 : tunables 54 27 8 :
slabdata 1 1 0
sgpool-16 2 8 512 8 1 : tunables 54 27 8 :
slabdata 1 1 0
sgpool-8 2 15 256 15 1 : tunables 120 60 8 :
slabdata 1 1 0
scsi_data_buffer 0 0 24 144 1 : tunables 120 60 8 :
slabdata 0 0 0
scsi_io_context 0 0 112 34 1 : tunables 120 60 8 :
slabdata 0 0 0
flow_cache 0 0 96 40 1 : tunables 120 60 8 :
slabdata 0 0 0
cfq_io_context 58 207 168 23 1 : tunables 120 60 8 :
slabdata 9 9 0
cfq_queue 56 308 136 28 1 : tunables 120 60 8 :
slabdata 11 11 0
bsg_cmd 0 0 312 12 1 : tunables 54 27 8 :
slabdata 0 0 0
mqueue_inode_cache 1 4 896 4 1 : tunables 54 27 8
: slabdata 1 1 0
isofs_inode_cache 0 0 608 6 1 : tunables 54 27 8 :
slabdata 0 0 0
minix_inode_cache 0 0 624 6 1 : tunables 54 27 8 :
slabdata 0 0 0
hugetlbfs_inode_cache 1 7 576 7 1 : tunables 54 27
8 : slabdata 1 1 0
dnotify_cache 0 0 40 92 1 : tunables 120 60 8 :
slabdata 0 0 0
dquot 0 0 256 15 1 : tunables 120 60 8 :
slabdata 0 0 0
inotify_event_cache 0 0 40 92 1 : tunables 120 60 8
: slabdata 0 0 0
inotify_watch_cache 94 159 72 53 1 : tunables 120 60 8
: slabdata 3 3 0
kioctx 0 0 384 10 1 : tunables 54 27 8 :
slabdata 0 0 0
kiocb 0 0 256 15 1 : tunables 120 60 8 :
slabdata 0 0 0
fasync_cache 0 0 24 144 1 : tunables 120 60 8 :
slabdata 0 0 0
shmem_inode_cache 878 1040 784 5 1 : tunables 54 27 8 :
slabdata 208 208 0
pid_namespace 0 0 2112 3 2 : tunables 24 12 8 :
slabdata 0 0 0
nsproxy 0 0 56 67 1 : tunables 120 60 8 :
slabdata 0 0 0
posix_timers_cache 0 0 192 20 1 : tunables 120 60 8
: slabdata 0 0 0
uid_cache 7 60 128 30 1 : tunables 120 60 8 :
slabdata 2 2 0
UNIX 128 220 704 11 2 : tunables 54 27 8 :
slabdata 20 20 0
ip_mrt_cache 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
UDP-Lite 0 0 832 9 2 : tunables 54 27 8 :
slabdata 0 0 0
tcp_bind_bucket 15 118 64 59 1 : tunables 120 60 8 :
slabdata 2 2 0
inet_peer_cache 1 59 64 59 1 : tunables 120 60 8 :
slabdata 1 1 0
secpath_cache 0 0 64 59 1 : tunables 120 60 8 :
slabdata 0 0 0
xfrm_dst_cache 0 0 384 10 1 : tunables 54 27 8 :
slabdata 0 0 0
ip_fib_alias 0 0 32 112 1 : tunables 120 60 8 :
slabdata 0 0 0
ip_fib_hash 15 106 72 53 1 : tunables 120 60 8 :
slabdata 2 2 0
ip_dst_cache 40 84 320 12 1 : tunables 54 27 8 :
slabdata 7 7 0
arp_cache 8 15 256 15 1 : tunables 120 60 8 :
slabdata 1 1 0
RAW 33 35 768 5 1 : tunables 54 27 8 :
slabdata 7 7 0
UDP 11 36 832 9 2 : tunables 54 27 8 :
slabdata 4 4 0
tw_sock_TCP 4 20 192 20 1 : tunables 120 60 8 :
slabdata 1 1 0
request_sock_TCP 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
TCP 16 24 1664 4 2 : tunables 24 12 8 :
slabdata 6 6 0
eventpoll_pwq 69 159 72 53 1 : tunables 120 60 8 :
slabdata 3 3 0
eventpoll_epi 69 150 128 30 1 : tunables 120 60 8 :
slabdata 5 5 0
pfm_event_set 0 0 57344 1 16 : tunables 8 4 0 :
slabdata 0 0 0
pfm_context 0 0 8192 1 2 : tunables 8 4 0 :
slabdata 0 0 0
blkdev_integrity 0 0 112 34 1 : tunables 120 60 8 :
slabdata 0 0 0
blkdev_queue 10 12 2264 3 2 : tunables 24 12 8 :
slabdata 4 4 0
blkdev_requests 91 130 368 10 1 : tunables 54 27 8 :
slabdata 13 13 27
blkdev_ioc 56 371 72 53 1 : tunables 120 60 8 :
slabdata 7 7 0
biovec-256 2 2 4096 1 1 : tunables 24 12 8 :
slabdata 2 2 0
biovec-128 2 4 2048 2 1 : tunables 24 12 8 :
slabdata 2 2 0
biovec-64 2 8 1024 4 1 : tunables 54 27 8 :
slabdata 2 2 0
biovec-16 2 30 256 15 1 : tunables 120 60 8 :
slabdata 2 2 0
biovec-4 2 118 64 59 1 : tunables 120 60 8 :
slabdata 2 2 0
biovec-1 223 606 16 202 1 : tunables 120 60 8 :
slabdata 3 3 70
bio_integrity_payload 2 60 128 30 1 : tunables 120 60
8 : slabdata 2 2 0
bio 205 330 128 30 1 : tunables 120 60 8 :
slabdata 11 11 70
sock_inode_cache 245 300 640 6 1 : tunables 54 27 8 :
slabdata 50 50 0
skbuff_fclone_cache 14 14 512 7 1 : tunables 54 27 8
: slabdata 2 2 0
skbuff_head_cache 5121 5985 256 15 1 : tunables 120 60 8 :
slabdata 399 399 68
file_lock_cache 4 22 176 22 1 : tunables 120 60 8 :
slabdata 1 1 0
Acpi-Operand 889 1749 72 53 1 : tunables 120 60 8 :
slabdata 33 33 0
Acpi-ParseExt 0 0 72 53 1 : tunables 120 60 8 :
slabdata 0 0 0
Acpi-Parse 0 0 48 77 1 : tunables 120 60 8 :
slabdata 0 0 0
Acpi-State 0 0 80 48 1 : tunables 120 60 8 :
slabdata 0 0 0
Acpi-Namespace 617 672 32 112 1 : tunables 120 60 8 :
slabdata 6 6 0
task_delay_info 389 884 112 34 1 : tunables 120 60 8 :
slabdata 26 26 0
taskstats 0 0 328 12 1 : tunables 54 27 8 :
slabdata 0 0 0
page_cgroup 0 0 40 92 1 : tunables 120 60 8 :
slabdata 0 0 0
proc_inode_cache 1397 1446 608 6 1 : tunables 54 27 8 :
slabdata 240 241 190
sigqueue 29 96 160 24 1 : tunables 120 60 8 :
slabdata 4 4 1
radix_tree_node 193120 196672 552 7 1 : tunables 54 27 8 :
slabdata 28096 28096 216
bdev_cache 5 15 768 5 1 : tunables 54 27 8 :
slabdata 3 3 0
sysfs_dir_cache 19120 19296 80 48 1 : tunables 120 60 8 :
slabdata 402 402 0
mnt_cache 30 105 256 15 1 : tunables 120 60 8 :
slabdata 7 7 0
inode_cache 1128 1176 560 7 1 : tunables 54 27 8 :
slabdata 166 168 24
dentry 4651 8189 208 19 1 : tunables 120 60 8 :
slabdata 431 431 0
filp 1563 2720 192 20 1 : tunables 120 60 8 :
slabdata 136 136 242
names_cache 142 142 4096 1 1 : tunables 24 12 8 :
slabdata 142 142 96
key_jar 0 0 192 20 1 : tunables 120 60 8 :
slabdata 0 0 0
buffer_head 1129 3071 104 37 1 : tunables 120 60 8 :
slabdata 83 83 0
mm_struct 86 136 896 4 1 : tunables 54 27 8 :
slabdata 34 34 1
vm_area_struct 3406 4136 176 22 1 : tunables 120 60 8 :
slabdata 188 188 26
fs_cache 140 531 64 59 1 : tunables 120 60 8 :
slabdata 9 9 1
files_cache 83 150 768 5 1 : tunables 54 27 8 :
slabdata 30 30 1
signal_cache 325 388 960 4 1 : tunables 54 27 8 :
slabdata 97 97 0
sighand_cache 317 369 2112 3 2 : tunables 24 12 8 :
slabdata 123 123 0
task_xstate 155 256 512 8 1 : tunables 54 27 8 :
slabdata 32 32 2
task_struct 368 372 5872 1 2 : tunables 8 4 0 :
slabdata 368 372 0
anon_vma 966 1728 24 144 1 : tunables 120 60 8 :
slabdata 12 12 0
pid 377 960 128 30 1 : tunables 120 60 8 :
slabdata 32 32 0
shared_policy_node 0 0 48 77 1 : tunables 120 60 8
: slabdata 0 0 0
numa_policy 15 112 136 28 1 : tunables 120 60 8 :
slabdata 4 4 0
idr_layer_cache 284 322 544 7 1 : tunables 54 27 8 :
slabdata 46 46 0
size-4194304(DMA) 0 0 4194304 1 1024 : tunables 1 1 0
: slabdata 0 0 0
size-4194304 0 0 4194304 1 1024 : tunables 1 1 0
: slabdata 0 0 0
size-2097152(DMA) 0 0 2097152 1 512 : tunables 1 1 0
: slabdata 0 0 0
size-2097152 0 0 2097152 1 512 : tunables 1 1 0
: slabdata 0 0 0
size-1048576(DMA) 0 0 1048576 1 256 : tunables 1 1 0
: slabdata 0 0 0
size-1048576 0 0 1048576 1 256 : tunables 1 1 0
: slabdata 0 0 0
size-524288(DMA) 0 0 524288 1 128 : tunables 1 1 0 :
slabdata 0 0 0
size-524288 0 0 524288 1 128 : tunables 1 1 0 :
slabdata 0 0 0
size-262144(DMA) 0 0 262144 1 64 : tunables 1 1 0 :
slabdata 0 0 0
size-262144 0 0 262144 1 64 : tunables 1 1 0 :
slabdata 0 0 0
size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 :
slabdata 0 0 0
size-131072 3 3 131072 1 32 : tunables 8 4 0 :
slabdata 3 3 0
size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 :
slabdata 0 0 0
size-65536 6 6 65536 1 16 : tunables 8 4 0 :
slabdata 6 6 0
size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 :
slabdata 0 0 0
size-32768 10 10 32768 1 8 : tunables 8 4 0 :
slabdata 10 10 0
size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 :
slabdata 0 0 0
size-16384 44 44 16384 1 4 : tunables 8 4 0 :
slabdata 44 44 0
size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 :
slabdata 0 0 0
size-8192 3611 3611 8192 1 2 : tunables 8 4 0 :
slabdata 3611 3611 0
size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8 :
slabdata 0 0 0
size-4096 1771 1771 4096 1 1 : tunables 24 12 8 :
slabdata 1771 1771 0
size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8 :
slabdata 0 0 0
size-2048 4609 4714 2048 2 1 : tunables 24 12 8 :
slabdata 2357 2357 0
size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8 :
slabdata 0 0 0
size-1024 4829 4900 1024 4 1 : tunables 54 27 8 :
slabdata 1225 1225 0
size-512(DMA) 0 0 512 8 1 : tunables 54 27 8 :
slabdata 0 0 0
size-512 1478 1520 512 8 1 : tunables 54 27 8 :
slabdata 190 190 39
size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 :
slabdata 0 0 0
size-256 4662 5550 256 15 1 : tunables 120 60 8 :
slabdata 370 370 1
size-128(DMA) 0 0 128 30 1 : tunables 120 60 8 :
slabdata 0 0 0
size-64(DMA) 0 0 64 59 1 : tunables 120 60 8 :
slabdata 0 0 0
size-64 17232 29382 64 59 1 : tunables 120 60 8 :
slabdata 498 498 0
size-32(DMA) 0 0 32 112 1 : tunables 120 60 8 :
slabdata 0 0 0
size-128 9907 16140 128 30 1 : tunables 120 60 8 :
slabdata 538 538 0
size-32 12487 13104 32 112 1 : tunables 120 60 8 :
slabdata 117 117 0
kmem_cache 181 181 4224 1 2 : tunables 8 4 0 :
slabdata 181 181 0
Tasks: 278 total, 1 running, 276 sleeping, 0 stopped, 1 zombie
Cpu(s): 3.8%us, 0.1%sy, 0.0%ni, 96.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 198091444k total, 197636988k used, 454456k free, 4544k buffers
Swap: 75505460k total, 8567448k used, 66938012k free, 29144008k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
107 root 15 -5 0 0 0 D 10 0.0 5:06.43
kswapd1
19328 user1 20 0 66.5g 60g 2268 D 4 32.0 31:48.49
R
1 root 20 0 1064 64 32 S 0 0.0 0:21.20
init
2 root 15 -5 0 0 0 S 0 0.0 0:00.06
kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.24
migration/0
4 root 15 -5 0 0 0 S 0 0.0 1:01.12
ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.30
migration/1
6 root 15 -5 0 0 0 S 0 0.0 0:00.50
ksoftirqd/1
7 root RT -5 0 0 0 S 0 0.0 0:00.22
migration/2
8 root 15 -5 0 0 0 S 0 0.0 0:00.36
ksoftirqd/2
9 root RT -5 0 0 0 S 0 0.0 0:00.28
migration/3
10 root 15 -5 0 0 0 S 0 0.0 0:00.60
ksoftirqd/3
11 root RT -5 0 0 0 S 0 0.0 0:00.18
migration/4
12 root 15 -5 0 0 0 S 0 0.0 0:00.40
ksoftirqd/4
13 root RT -5 0 0 0 S 0 0.0 0:00.26
migration/5
14 root 15 -5 0 0 0 S 0 0.0 0:00.76
ksoftirqd/5
15 root RT -5 0 0 0 S 0 0.0 0:00.20
migration/6
16 root 15 -5 0 0 0 S 0 0.0 0:00.36
ksoftirqd/6
17 root RT -5 0 0 0 S 0 0.0 0:00.26
migration/7
18 root 15 -5 0 0 0 S 0 0.0 0:00.68
ksoftirqd/7
19 root RT -5 0 0 0 S 0 0.0 0:00.88
migration/8
20 root 15 -5 0 0 0 S 0 0.0 0:07.70
ksoftirqd/8
21 root RT -5 0 0 0 S 0 0.0 0:01.12
migration/9
22 root 15 -5 0 0 0 S 0 0.0 0:01.20
ksoftirqd/9
23 root RT -5 0 0 0 S 0 0.0 0:03.50
migration/10
24 root 15 -5 0 0 0 S 0 0.0 0:01.22
ksoftirqd/10
25 root RT -5 0 0 0 S 0 0.0 0:04.84
migration/11
26 root 15 -5 0 0 0 S 0 0.0 0:01.90
ksoftirqd/11
27 root RT -5 0 0 0 S 0 0.0 0:01.46
migration/12
28 root 15 -5 0 0 0 S 0 0.0 0:01.42
ksoftirqd/12
29 root RT -5 0 0 0 S 0 0.0 0:01.62
migration/13
30 root 15 -5 0 0 0 S 0 0.0 0:01.84
ksoftirqd/13
31 root RT -5 0 0 0 S 0 0.0 0:01.90
migration/14
32 root 15 -5 0 0 0 S 0 0.0 0:01.18
ksoftirqd/14
--
Thanks,
-J
On Mon, Apr 19, 2010 at 10:07 AM, Andreas Dilger
<andreas.dilger at oracle.com>wrote:
> There is a known problem with the DLM LRU size that may be affecting you.
> It may be something else too. Please check /proc/{slabinfo,meminfo} to see
> what is using the memory on the client.
>
> Cheers, Andreas
>
>
> On 2010-04-19, at 10:43, Jagga Soorma <jagga13 at gmail.com> wrote:
>
> Hi Guys,
>>
>> My users are reporting some issues with memory on our lustre 1.8.1
>> clients. It looks like when they submit a single job at a time the run
time
>> was about 4.5 minutes. However, when they ran multiple jobs (10 or
less) on
>> a client with 192GB of memory on a single node the run time for each
job was
>> exceeding 3-4X the run time for the single process. They also noticed
that
>> the swap space kept climbing even though there was plenty of free
memory on
>> the system. Could this possibly be related to the lustre client? Does
it
>> reserve any memory that is not accessible by any other process even
though
>> it might not be in use?
>>
>> Thanks much,
>> -J
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100419/f710c37f/attachment-0001.html
Here is something from April 12 that I see in the client logs. Not sure if this is related: -- Apr 12 14:51:16 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) ino 424411146 page 0 (0) not covered by a lock (mmap?). check debug logs. Apr 12 14:51:16 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) ino 424411146 page 1480 (6062080) not covered by a lock (mmap?). check debug logs. Apr 12 14:51:16 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) Skipped 1479 previous similar messages Apr 12 14:51:17 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) ino 424411146 page 273025 (1118310400) not covered by a lock (mmap?). check debug logs. Apr 12 14:51:17 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) Skipped 271544 previous similar messages -- -J On Mon, Apr 19, 2010 at 11:02 AM, Jagga Soorma <jagga13 at gmail.com> wrote:> Andreas, > > I am seeing the problem again on one of my hosts and here is a live capture > of the data. Can you assist with this? > > -- > # free > total used free shared buffers cached > Mem: 198091444 197636852 454592 0 4260 34251452 > -/+ buffers/cache: 163381140 34710304 > Swap: 75505460 10281796 65223664 > > # cat /proc/meminfo > MemTotal: 198091444 kB > MemFree: 458048 kB > Buffers: 4268 kB > Cached: 34099372 kB > SwapCached: 7730744 kB > Active: 62919152 kB > Inactive: 34107188 kB > SwapTotal: 75505460 kB > SwapFree: 65220676 kB > Dirty: 444 kB > Writeback: 0 kB > AnonPages: 58704728 kB > Mapped: 12036 kB > Slab: 99806476 kB > SReclaimable: 118532 kB > SUnreclaim: 99687944 kB > PageTables: 131200 kB > > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 174551180 kB > Committed_AS: 65739660 kB > > VmallocTotal: 34359738367 kB > VmallocUsed: 588416 kB > VmallocChunk: 34359149923 kB > HugePages_Total: 0 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > DirectMap4k: 8432 kB > DirectMap2M: 201308160 kB > > # cat /proc/slabinfo > slabinfo - version: 2.1 > # name <active_objs> <num_objs> <objsize> <objperslab> > <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata > <active_slabs> <num_slabs> <sharedavail> > nfs_direct_cache 0 0 128 30 1 : tunables 120 60 8 > : slabdata 0 0 0 > nfs_write_data 36 44 704 11 2 : tunables 54 27 8 > : slabdata 4 4 0 > nfs_read_data 32 33 704 11 2 : tunables 54 27 8 > : slabdata 3 3 0 > nfs_inode_cache 0 0 984 4 1 : tunables 54 27 8 > : slabdata 0 0 0 > nfs_page 0 0 128 30 1 : tunables 120 60 8 > : slabdata 0 0 0 > rpc_buffers 8 8 2048 2 1 : tunables 24 12 8 > : slabdata 4 4 0 > rpc_tasks 8 12 320 12 1 : tunables 54 27 8 > : slabdata 1 1 0 > rpc_inode_cache 0 0 832 4 1 : tunables 54 27 8 > : slabdata 0 0 0 > ll_async_page 8494811 8507076 320 12 1 : tunables 54 27 > 8 : slabdata 708923 708923 216 > ll_file_data 16 40 192 20 1 : tunables 120 60 8 > : slabdata 2 2 0 > lustre_inode_cache 95 184 896 4 1 : tunables 54 27 8 > : slabdata 46 46 0 > lov_oinfo 56 180 320 12 1 : tunables 54 27 8 > : slabdata 15 15 0 > > osc_quota_info 0 0 32 112 1 : tunables 120 60 8 > : slabdata 0 0 0 > ll_qunit_cache 0 0 112 34 1 : tunables 120 60 8 > : slabdata 0 0 0 > llcd_cache 0 0 3952 1 1 : tunables 24 12 8 > : slabdata 0 0 0 > ptlrpc_cbdatas 0 0 32 112 1 : tunables 120 60 8 > : slabdata 0 0 0 > interval_node 1680 5730 128 30 1 : tunables 120 60 8 > : slabdata 191 191 0 > ldlm_locks 2255 6232 512 8 1 : tunables 54 27 8 > : slabdata 779 779 0 > ldlm_resources 2227 5570 384 10 1 : tunables 54 27 8 > : slabdata 557 557 0 > > ll_import_cache 0 0 1248 3 1 : tunables 24 12 8 > : slabdata 0 0 0 > ll_obdo_cache 0 459630919 208 19 1 : tunables 120 60 > 8 : slabdata 0 24191101 0 > > ll_obd_dev_cache 13 13 5672 1 2 : tunables 8 4 0 > : slabdata 13 13 0 > obd_lvfs_ctxt_cache 0 0 96 40 1 : tunables 120 60 > 8 : slabdata 0 0 0 > SDP 0 0 1728 4 2 : tunables 24 12 8 > : slabdata 0 0 0 > fib6_nodes 7 59 64 59 1 : tunables 120 60 8 > : slabdata 1 1 0 > ip6_dst_cache 10 24 320 12 1 : tunables 54 27 8 > : slabdata 2 2 0 > > ndisc_cache 3 30 256 15 1 : tunables 120 60 8 > : slabdata 2 2 0 > RAWv6 35 36 960 4 1 : tunables 54 27 8 > : slabdata 9 9 0 > UDPLITEv6 0 0 960 4 1 : tunables 54 27 8 > : slabdata 0 0 0 > UDPv6 7 12 960 4 1 : tunables 54 27 8 > : slabdata 3 3 0 > tw_sock_TCPv6 0 0 192 20 1 : tunables 120 60 8 > : slabdata 0 0 0 > request_sock_TCPv6 0 0 192 20 1 : tunables 120 60 8 > : slabdata 0 0 0 > TCPv6 3 4 1792 2 1 : tunables 24 12 8 > : slabdata 2 2 0 > ib_mad 2051 2096 448 8 1 : tunables 54 27 8 > : slabdata 262 262 0 > > fuse_request 0 0 608 6 1 : tunables 54 27 8 > : slabdata 0 0 0 > fuse_inode 0 0 704 11 2 : tunables 54 27 8 > : slabdata 0 0 0 > kcopyd_job 0 0 360 11 1 : tunables 54 27 8 > : slabdata 0 0 0 > dm_uevent 0 0 2608 3 2 : tunables 24 12 8 > : slabdata 0 0 0 > dm_clone_bio_info 0 0 16 202 1 : tunables 120 60 8 > : slabdata 0 0 0 > dm_rq_target_io 0 0 408 9 1 : tunables 54 27 8 > : slabdata 0 0 0 > dm_target_io 0 0 24 144 1 : tunables 120 60 8 > : slabdata 0 0 0 > dm_io 0 0 32 112 1 : tunables 120 60 8 > : slabdata 0 0 0 > uhci_urb_priv 1 67 56 67 1 : tunables 120 60 8 > : slabdata 1 1 0 > ext3_inode_cache 2472 2610 768 5 1 : tunables 54 27 8 > : slabdata 522 522 0 > > ext3_xattr 0 0 88 44 1 : tunables 120 60 8 > : slabdata 0 0 0 > journal_handle 56 288 24 144 1 : tunables 120 60 8 > : slabdata 2 2 0 > journal_head 216 240 96 40 1 : tunables 120 60 8 > : slabdata 6 6 0 > > revoke_table 4 202 16 202 1 : tunables 120 60 8 > : slabdata 1 1 0 > revoke_record 0 0 32 112 1 : tunables 120 60 8 > : slabdata 0 0 0 > sgpool-128 2 2 4096 1 1 : tunables 24 12 8 > : slabdata 2 2 0 > sgpool-64 2 2 2048 2 1 : tunables 24 12 8 > : slabdata 1 1 0 > sgpool-32 2 4 1024 4 1 : tunables 54 27 8 > : slabdata 1 1 0 > sgpool-16 2 8 512 8 1 : tunables 54 27 8 > : slabdata 1 1 0 > sgpool-8 2 15 256 15 1 : tunables 120 60 8 > : slabdata 1 1 0 > scsi_data_buffer 0 0 24 144 1 : tunables 120 60 8 > : slabdata 0 0 0 > scsi_io_context 0 0 112 34 1 : tunables 120 60 8 > : slabdata 0 0 0 > flow_cache 0 0 96 40 1 : tunables 120 60 8 > : slabdata 0 0 0 > cfq_io_context 58 207 168 23 1 : tunables 120 60 8 > : slabdata 9 9 0 > cfq_queue 56 308 136 28 1 : tunables 120 60 8 > : slabdata 11 11 0 > > bsg_cmd 0 0 312 12 1 : tunables 54 27 8 > : slabdata 0 0 0 > mqueue_inode_cache 1 4 896 4 1 : tunables 54 27 8 > : slabdata 1 1 0 > isofs_inode_cache 0 0 608 6 1 : tunables 54 27 8 > : slabdata 0 0 0 > minix_inode_cache 0 0 624 6 1 : tunables 54 27 8 > : slabdata 0 0 0 > hugetlbfs_inode_cache 1 7 576 7 1 : tunables 54 > 27 8 : slabdata 1 1 0 > dnotify_cache 0 0 40 92 1 : tunables 120 60 8 > : slabdata 0 0 0 > dquot 0 0 256 15 1 : tunables 120 60 8 > : slabdata 0 0 0 > inotify_event_cache 0 0 40 92 1 : tunables 120 60 > 8 : slabdata 0 0 0 > inotify_watch_cache 94 159 72 53 1 : tunables 120 60 > 8 : slabdata 3 3 0 > > kioctx 0 0 384 10 1 : tunables 54 27 8 > : slabdata 0 0 0 > kiocb 0 0 256 15 1 : tunables 120 60 8 > : slabdata 0 0 0 > fasync_cache 0 0 24 144 1 : tunables 120 60 8 > : slabdata 0 0 0 > shmem_inode_cache 878 1040 784 5 1 : tunables 54 27 8 > : slabdata 208 208 0 > > pid_namespace 0 0 2112 3 2 : tunables 24 12 8 > : slabdata 0 0 0 > nsproxy 0 0 56 67 1 : tunables 120 60 8 > : slabdata 0 0 0 > posix_timers_cache 0 0 192 20 1 : tunables 120 60 8 > : slabdata 0 0 0 > uid_cache 7 60 128 30 1 : tunables 120 60 8 > : slabdata 2 2 0 > UNIX 128 220 704 11 2 : tunables 54 27 8 > : slabdata 20 20 0 > > ip_mrt_cache 0 0 128 30 1 : tunables 120 60 8 > : slabdata 0 0 0 > UDP-Lite 0 0 832 9 2 : tunables 54 27 8 > : slabdata 0 0 0 > tcp_bind_bucket 15 118 64 59 1 : tunables 120 60 8 > : slabdata 2 2 0 > > inet_peer_cache 1 59 64 59 1 : tunables 120 60 8 > : slabdata 1 1 0 > secpath_cache 0 0 64 59 1 : tunables 120 60 8 > : slabdata 0 0 0 > xfrm_dst_cache 0 0 384 10 1 : tunables 54 27 8 > : slabdata 0 0 0 > ip_fib_alias 0 0 32 112 1 : tunables 120 60 8 > : slabdata 0 0 0 > ip_fib_hash 15 106 72 53 1 : tunables 120 60 8 > : slabdata 2 2 0 > ip_dst_cache 40 84 320 12 1 : tunables 54 27 8 > : slabdata 7 7 0 > > arp_cache 8 15 256 15 1 : tunables 120 60 8 > : slabdata 1 1 0 > RAW 33 35 768 5 1 : tunables 54 27 8 > : slabdata 7 7 0 > UDP 11 36 832 9 2 : tunables 54 27 8 > : slabdata 4 4 0 > tw_sock_TCP 4 20 192 20 1 : tunables 120 60 8 > : slabdata 1 1 0 > > request_sock_TCP 0 0 128 30 1 : tunables 120 60 8 > : slabdata 0 0 0 > TCP 16 24 1664 4 2 : tunables 24 12 8 > : slabdata 6 6 0 > eventpoll_pwq 69 159 72 53 1 : tunables 120 60 8 > : slabdata 3 3 0 > eventpoll_epi 69 150 128 30 1 : tunables 120 60 8 > : slabdata 5 5 0 > > pfm_event_set 0 0 57344 1 16 : tunables 8 4 0 > : slabdata 0 0 0 > pfm_context 0 0 8192 1 2 : tunables 8 4 0 > : slabdata 0 0 0 > blkdev_integrity 0 0 112 34 1 : tunables 120 60 8 > : slabdata 0 0 0 > blkdev_queue 10 12 2264 3 2 : tunables 24 12 8 > : slabdata 4 4 0 > blkdev_requests 91 130 368 10 1 : tunables 54 27 8 > : slabdata 13 13 27 > blkdev_ioc 56 371 72 53 1 : tunables 120 60 8 > : slabdata 7 7 0 > > biovec-256 2 2 4096 1 1 : tunables 24 12 8 > : slabdata 2 2 0 > biovec-128 2 4 2048 2 1 : tunables 24 12 8 > : slabdata 2 2 0 > biovec-64 2 8 1024 4 1 : tunables 54 27 8 > : slabdata 2 2 0 > biovec-16 2 30 256 15 1 : tunables 120 60 8 > : slabdata 2 2 0 > biovec-4 2 118 64 59 1 : tunables 120 60 8 > : slabdata 2 2 0 > biovec-1 223 606 16 202 1 : tunables 120 60 8 > : slabdata 3 3 70 > > bio_integrity_payload 2 60 128 30 1 : tunables 120 > 60 8 : slabdata 2 2 0 > bio 205 330 128 30 1 : tunables 120 60 8 > : slabdata 11 11 70 > sock_inode_cache 245 300 640 6 1 : tunables 54 27 8 > : slabdata 50 50 0 > skbuff_fclone_cache 14 14 512 7 1 : tunables 54 27 > 8 : slabdata 2 2 0 > skbuff_head_cache 5121 5985 256 15 1 : tunables 120 60 8 > : slabdata 399 399 68 > file_lock_cache 4 22 176 22 1 : tunables 120 60 8 > : slabdata 1 1 0 > Acpi-Operand 889 1749 72 53 1 : tunables 120 60 8 > : slabdata 33 33 0 > > Acpi-ParseExt 0 0 72 53 1 : tunables 120 60 8 > : slabdata 0 0 0 > Acpi-Parse 0 0 48 77 1 : tunables 120 60 8 > : slabdata 0 0 0 > Acpi-State 0 0 80 48 1 : tunables 120 60 8 > : slabdata 0 0 0 > Acpi-Namespace 617 672 32 112 1 : tunables 120 60 8 > : slabdata 6 6 0 > task_delay_info 389 884 112 34 1 : tunables 120 60 8 > : slabdata 26 26 0 > > taskstats 0 0 328 12 1 : tunables 54 27 8 > : slabdata 0 0 0 > page_cgroup 0 0 40 92 1 : tunables 120 60 8 > : slabdata 0 0 0 > proc_inode_cache 1397 1446 608 6 1 : tunables 54 27 8 > : slabdata 240 241 190 > sigqueue 29 96 160 24 1 : tunables 120 60 8 > : slabdata 4 4 1 > radix_tree_node 193120 196672 552 7 1 : tunables 54 27 8 > : slabdata 28096 28096 216 > bdev_cache 5 15 768 5 1 : tunables 54 27 8 > : slabdata 3 3 0 > > sysfs_dir_cache 19120 19296 80 48 1 : tunables 120 60 8 > : slabdata 402 402 0 > mnt_cache 30 105 256 15 1 : tunables 120 60 8 > : slabdata 7 7 0 > inode_cache 1128 1176 560 7 1 : tunables 54 27 8 > : slabdata 166 168 24 > dentry 4651 8189 208 19 1 : tunables 120 60 8 > : slabdata 431 431 0 > filp 1563 2720 192 20 1 : tunables 120 60 8 > : slabdata 136 136 242 > names_cache 142 142 4096 1 1 : tunables 24 12 8 > : slabdata 142 142 96 > > key_jar 0 0 192 20 1 : tunables 120 60 8 > : slabdata 0 0 0 > buffer_head 1129 3071 104 37 1 : tunables 120 60 8 > : slabdata 83 83 0 > mm_struct 86 136 896 4 1 : tunables 54 27 8 > : slabdata 34 34 1 > vm_area_struct 3406 4136 176 22 1 : tunables 120 60 8 > : slabdata 188 188 26 > fs_cache 140 531 64 59 1 : tunables 120 60 8 > : slabdata 9 9 1 > files_cache 83 150 768 5 1 : tunables 54 27 8 > : slabdata 30 30 1 > signal_cache 325 388 960 4 1 : tunables 54 27 8 > : slabdata 97 97 0 > sighand_cache 317 369 2112 3 2 : tunables 24 12 8 > : slabdata 123 123 0 > task_xstate 155 256 512 8 1 : tunables 54 27 8 > : slabdata 32 32 2 > task_struct 368 372 5872 1 2 : tunables 8 4 0 > : slabdata 368 372 0 > anon_vma 966 1728 24 144 1 : tunables 120 60 8 > : slabdata 12 12 0 > pid 377 960 128 30 1 : tunables 120 60 8 > : slabdata 32 32 0 > > shared_policy_node 0 0 48 77 1 : tunables 120 60 8 > : slabdata 0 0 0 > numa_policy 15 112 136 28 1 : tunables 120 60 8 > : slabdata 4 4 0 > idr_layer_cache 284 322 544 7 1 : tunables 54 27 8 > : slabdata 46 46 0 > > size-4194304(DMA) 0 0 4194304 1 1024 : tunables 1 1 0 > : slabdata 0 0 0 > size-4194304 0 0 4194304 1 1024 : tunables 1 1 0 > : slabdata 0 0 0 > size-2097152(DMA) 0 0 2097152 1 512 : tunables 1 1 0 > : slabdata 0 0 0 > size-2097152 0 0 2097152 1 512 : tunables 1 1 0 > : slabdata 0 0 0 > size-1048576(DMA) 0 0 1048576 1 256 : tunables 1 1 0 > : slabdata 0 0 0 > size-1048576 0 0 1048576 1 256 : tunables 1 1 0 > : slabdata 0 0 0 > size-524288(DMA) 0 0 524288 1 128 : tunables 1 1 0 > : slabdata 0 0 0 > size-524288 0 0 524288 1 128 : tunables 1 1 0 > : slabdata 0 0 0 > size-262144(DMA) 0 0 262144 1 64 : tunables 1 1 0 > : slabdata 0 0 0 > size-262144 0 0 262144 1 64 : tunables 1 1 0 > : slabdata 0 0 0 > size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 > : slabdata 0 0 0 > size-131072 3 3 131072 1 32 : tunables 8 4 0 > : slabdata 3 3 0 > size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 > : slabdata 0 0 0 > size-65536 6 6 65536 1 16 : tunables 8 4 0 > : slabdata 6 6 0 > size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 > : slabdata 0 0 0 > size-32768 10 10 32768 1 8 : tunables 8 4 0 > : slabdata 10 10 0 > > size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 > : slabdata 0 0 0 > size-16384 44 44 16384 1 4 : tunables 8 4 0 > : slabdata 44 44 0 > > size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 > : slabdata 0 0 0 > size-8192 3611 3611 8192 1 2 : tunables 8 4 0 > : slabdata 3611 3611 0 > > size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8 > : slabdata 0 0 0 > size-4096 1771 1771 4096 1 1 : tunables 24 12 8 > : slabdata 1771 1771 0 > > size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8 > : slabdata 0 0 0 > size-2048 4609 4714 2048 2 1 : tunables 24 12 8 > : slabdata 2357 2357 0 > > size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8 > : slabdata 0 0 0 > size-1024 4829 4900 1024 4 1 : tunables 54 27 8 > : slabdata 1225 1225 0 > > size-512(DMA) 0 0 512 8 1 : tunables 54 27 8 > : slabdata 0 0 0 > size-512 1478 1520 512 8 1 : tunables 54 27 8 > : slabdata 190 190 39 > > size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 > : slabdata 0 0 0 > size-256 4662 5550 256 15 1 : tunables 120 60 8 > : slabdata 370 370 1 > > size-128(DMA) 0 0 128 30 1 : tunables 120 60 8 > : slabdata 0 0 0 > size-64(DMA) 0 0 64 59 1 : tunables 120 60 8 > : slabdata 0 0 0 > size-64 17232 29382 64 59 1 : tunables 120 60 8 > : slabdata 498 498 0 > > size-32(DMA) 0 0 32 112 1 : tunables 120 60 8 > : slabdata 0 0 0 > size-128 9907 16140 128 30 1 : tunables 120 60 8 > : slabdata 538 538 0 > size-32 12487 13104 32 112 1 : tunables 120 60 8 > : slabdata 117 117 0 > > kmem_cache 181 181 4224 1 2 : tunables 8 4 0 > : slabdata 181 181 0 > > > Tasks: 278 total, 1 running, 276 sleeping, 0 stopped, 1 zombie > Cpu(s): 3.8%us, 0.1%sy, 0.0%ni, 96.0%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Mem: 198091444k total, 197636988k used, 454456k free, 4544k buffers > Swap: 75505460k total, 8567448k used, 66938012k free, 29144008k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > > 107 root 15 -5 0 0 0 D 10 0.0 5:06.43 > kswapd1 > > 19328 user1 20 0 66.5g 60g 2268 D 4 32.0 31:48.49 > R > > 1 root 20 0 1064 64 32 S 0 0.0 0:21.20 > init > > 2 root 15 -5 0 0 0 S 0 0.0 0:00.06 > kthreadd > > 3 root RT -5 0 0 0 S 0 0.0 0:00.24 > migration/0 > > 4 root 15 -5 0 0 0 S 0 0.0 1:01.12 > ksoftirqd/0 > > 5 root RT -5 0 0 0 S 0 0.0 0:00.30 > migration/1 > > 6 root 15 -5 0 0 0 S 0 0.0 0:00.50 > ksoftirqd/1 > > 7 root RT -5 0 0 0 S 0 0.0 0:00.22 > migration/2 > > 8 root 15 -5 0 0 0 S 0 0.0 0:00.36 > ksoftirqd/2 > > 9 root RT -5 0 0 0 S 0 0.0 0:00.28 > migration/3 > > 10 root 15 -5 0 0 0 S 0 0.0 0:00.60 > ksoftirqd/3 > > 11 root RT -5 0 0 0 S 0 0.0 0:00.18 > migration/4 > > 12 root 15 -5 0 0 0 S 0 0.0 0:00.40 > ksoftirqd/4 > > 13 root RT -5 0 0 0 S 0 0.0 0:00.26 > migration/5 > > 14 root 15 -5 0 0 0 S 0 0.0 0:00.76 > ksoftirqd/5 > > 15 root RT -5 0 0 0 S 0 0.0 0:00.20 > migration/6 > > 16 root 15 -5 0 0 0 S 0 0.0 0:00.36 > ksoftirqd/6 > > 17 root RT -5 0 0 0 S 0 0.0 0:00.26 > migration/7 > > 18 root 15 -5 0 0 0 S 0 0.0 0:00.68 > ksoftirqd/7 > > 19 root RT -5 0 0 0 S 0 0.0 0:00.88 > migration/8 > > 20 root 15 -5 0 0 0 S 0 0.0 0:07.70 > ksoftirqd/8 > > 21 root RT -5 0 0 0 S 0 0.0 0:01.12 > migration/9 > > 22 root 15 -5 0 0 0 S 0 0.0 0:01.20 > ksoftirqd/9 > > 23 root RT -5 0 0 0 S 0 0.0 0:03.50 > migration/10 > > 24 root 15 -5 0 0 0 S 0 0.0 0:01.22 > ksoftirqd/10 > > 25 root RT -5 0 0 0 S 0 0.0 0:04.84 > migration/11 > > 26 root 15 -5 0 0 0 S 0 0.0 0:01.90 > ksoftirqd/11 > > 27 root RT -5 0 0 0 S 0 0.0 0:01.46 > migration/12 > > 28 root 15 -5 0 0 0 S 0 0.0 0:01.42 > ksoftirqd/12 > > 29 root RT -5 0 0 0 S 0 0.0 0:01.62 > migration/13 > > 30 root 15 -5 0 0 0 S 0 0.0 0:01.84 > ksoftirqd/13 > > 31 root RT -5 0 0 0 S 0 0.0 0:01.90 > migration/14 > > 32 root 15 -5 0 0 0 S 0 0.0 0:01.18 > ksoftirqd/14 > -- > > Thanks, > -J > > On Mon, Apr 19, 2010 at 10:07 AM, Andreas Dilger < > andreas.dilger at oracle.com> wrote: > >> There is a known problem with the DLM LRU size that may be affecting you. >> It may be something else too. Please check /proc/{slabinfo,meminfo} to see >> what is using the memory on the client. >> >> Cheers, Andreas >> >> >> On 2010-04-19, at 10:43, Jagga Soorma <jagga13 at gmail.com> wrote: >> >> Hi Guys, >>> >>> My users are reporting some issues with memory on our lustre 1.8.1 >>> clients. It looks like when they submit a single job at a time the run time >>> was about 4.5 minutes. However, when they ran multiple jobs (10 or less) on >>> a client with 192GB of memory on a single node the run time for each job was >>> exceeding 3-4X the run time for the single process. They also noticed that >>> the swap space kept climbing even though there was plenty of free memory on >>> the system. Could this possibly be related to the lustre client? Does it >>> reserve any memory that is not accessible by any other process even though >>> it might not be in use? >>> >>> Thanks much, >>> -J >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100419/8000e20d/attachment-0001.html
Could it be locking? I do have the flock option enabled. -- lustre_inode_cache 123 192 896 4 1 : tunables 54 27 8 : slabdata 48 48 0 lov_oinfo 128 228 320 12 1 : tunables 54 27 8 : slabdata 19 19 0 ldlm_locks 1550 3992 512 8 1 : tunables 54 27 8 : slabdata 499 499 0 ldlm_resources 1449 3600 384 10 1 : tunables 54 27 8 : slabdata 360 360 0 -- Thanks, -J On Mon, Apr 19, 2010 at 11:26 AM, Jagga Soorma <jagga13 at gmail.com> wrote:> Here is something from April 12 that I see in the client logs. Not sure if > this is related: > > -- > Apr 12 14:51:16 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) ino > 424411146 page 0 (0) not covered by a lock (mmap?). check debug logs. > Apr 12 14:51:16 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) ino > 424411146 page 1480 (6062080) not covered by a lock (mmap?). check debug > logs. > Apr 12 14:51:16 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) > Skipped 1479 previous similar messages > Apr 12 14:51:17 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) ino > 424411146 page 273025 (1118310400) not covered by a lock (mmap?). check > debug logs. > Apr 12 14:51:17 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) > Skipped 271544 previous similar messages > -- > > -J > > > On Mon, Apr 19, 2010 at 11:02 AM, Jagga Soorma <jagga13 at gmail.com> wrote: > >> Andreas, >> >> I am seeing the problem again on one of my hosts and here is a live >> capture of the data. Can you assist with this? >> >> -- >> # free >> total used free shared buffers cached >> Mem: 198091444 197636852 454592 0 4260 34251452 >> -/+ buffers/cache: 163381140 34710304 >> Swap: 75505460 10281796 65223664 >> >> # cat /proc/meminfo >> MemTotal: 198091444 kB >> MemFree: 458048 kB >> Buffers: 4268 kB >> Cached: 34099372 kB >> SwapCached: 7730744 kB >> Active: 62919152 kB >> Inactive: 34107188 kB >> SwapTotal: 75505460 kB >> SwapFree: 65220676 kB >> Dirty: 444 kB >> Writeback: 0 kB >> AnonPages: 58704728 kB >> Mapped: 12036 kB >> Slab: 99806476 kB >> SReclaimable: 118532 kB >> SUnreclaim: 99687944 kB >> PageTables: 131200 kB >> >> NFS_Unstable: 0 kB >> Bounce: 0 kB >> WritebackTmp: 0 kB >> CommitLimit: 174551180 kB >> Committed_AS: 65739660 kB >> >> VmallocTotal: 34359738367 kB >> VmallocUsed: 588416 kB >> VmallocChunk: 34359149923 kB >> HugePages_Total: 0 >> HugePages_Free: 0 >> HugePages_Rsvd: 0 >> HugePages_Surp: 0 >> Hugepagesize: 2048 kB >> DirectMap4k: 8432 kB >> DirectMap2M: 201308160 kB >> >> # cat /proc/slabinfo >> slabinfo - version: 2.1 >> # name <active_objs> <num_objs> <objsize> <objperslab> >> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata >> <active_slabs> <num_slabs> <sharedavail> >> nfs_direct_cache 0 0 128 30 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> nfs_write_data 36 44 704 11 2 : tunables 54 27 8 >> : slabdata 4 4 0 >> nfs_read_data 32 33 704 11 2 : tunables 54 27 8 >> : slabdata 3 3 0 >> nfs_inode_cache 0 0 984 4 1 : tunables 54 27 8 >> : slabdata 0 0 0 >> nfs_page 0 0 128 30 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> rpc_buffers 8 8 2048 2 1 : tunables 24 12 8 >> : slabdata 4 4 0 >> rpc_tasks 8 12 320 12 1 : tunables 54 27 8 >> : slabdata 1 1 0 >> rpc_inode_cache 0 0 832 4 1 : tunables 54 27 8 >> : slabdata 0 0 0 >> ll_async_page 8494811 8507076 320 12 1 : tunables 54 27 >> 8 : slabdata 708923 708923 216 >> ll_file_data 16 40 192 20 1 : tunables 120 60 8 >> : slabdata 2 2 0 >> lustre_inode_cache 95 184 896 4 1 : tunables 54 27 >> 8 : slabdata 46 46 0 >> lov_oinfo 56 180 320 12 1 : tunables 54 27 8 >> : slabdata 15 15 0 >> >> osc_quota_info 0 0 32 112 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> ll_qunit_cache 0 0 112 34 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> llcd_cache 0 0 3952 1 1 : tunables 24 12 8 >> : slabdata 0 0 0 >> ptlrpc_cbdatas 0 0 32 112 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> interval_node 1680 5730 128 30 1 : tunables 120 60 8 >> : slabdata 191 191 0 >> ldlm_locks 2255 6232 512 8 1 : tunables 54 27 8 >> : slabdata 779 779 0 >> ldlm_resources 2227 5570 384 10 1 : tunables 54 27 8 >> : slabdata 557 557 0 >> >> ll_import_cache 0 0 1248 3 1 : tunables 24 12 8 >> : slabdata 0 0 0 >> ll_obdo_cache 0 459630919 208 19 1 : tunables 120 >> 60 8 : slabdata 0 24191101 0 >> >> ll_obd_dev_cache 13 13 5672 1 2 : tunables 8 4 0 >> : slabdata 13 13 0 >> obd_lvfs_ctxt_cache 0 0 96 40 1 : tunables 120 60 >> 8 : slabdata 0 0 0 >> SDP 0 0 1728 4 2 : tunables 24 12 8 >> : slabdata 0 0 0 >> fib6_nodes 7 59 64 59 1 : tunables 120 60 >> 8 : slabdata 1 1 0 >> ip6_dst_cache 10 24 320 12 1 : tunables 54 27 8 >> : slabdata 2 2 0 >> >> ndisc_cache 3 30 256 15 1 : tunables 120 60 8 >> : slabdata 2 2 0 >> RAWv6 35 36 960 4 1 : tunables 54 27 8 >> : slabdata 9 9 0 >> UDPLITEv6 0 0 960 4 1 : tunables 54 27 8 >> : slabdata 0 0 0 >> UDPv6 7 12 960 4 1 : tunables 54 27 8 >> : slabdata 3 3 0 >> tw_sock_TCPv6 0 0 192 20 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> request_sock_TCPv6 0 0 192 20 1 : tunables 120 60 >> 8 : slabdata 0 0 0 >> TCPv6 3 4 1792 2 1 : tunables 24 12 8 >> : slabdata 2 2 0 >> ib_mad 2051 2096 448 8 1 : tunables 54 27 8 >> : slabdata 262 262 0 >> >> fuse_request 0 0 608 6 1 : tunables 54 27 8 >> : slabdata 0 0 0 >> fuse_inode 0 0 704 11 2 : tunables 54 27 8 >> : slabdata 0 0 0 >> kcopyd_job 0 0 360 11 1 : tunables 54 27 8 >> : slabdata 0 0 0 >> dm_uevent 0 0 2608 3 2 : tunables 24 12 8 >> : slabdata 0 0 0 >> dm_clone_bio_info 0 0 16 202 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> dm_rq_target_io 0 0 408 9 1 : tunables 54 27 8 >> : slabdata 0 0 0 >> dm_target_io 0 0 24 144 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> dm_io 0 0 32 112 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> uhci_urb_priv 1 67 56 67 1 : tunables 120 60 8 >> : slabdata 1 1 0 >> ext3_inode_cache 2472 2610 768 5 1 : tunables 54 27 8 >> : slabdata 522 522 0 >> >> ext3_xattr 0 0 88 44 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> journal_handle 56 288 24 144 1 : tunables 120 60 8 >> : slabdata 2 2 0 >> journal_head 216 240 96 40 1 : tunables 120 60 8 >> : slabdata 6 6 0 >> >> revoke_table 4 202 16 202 1 : tunables 120 60 8 >> : slabdata 1 1 0 >> revoke_record 0 0 32 112 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> sgpool-128 2 2 4096 1 1 : tunables 24 12 8 >> : slabdata 2 2 0 >> sgpool-64 2 2 2048 2 1 : tunables 24 12 8 >> : slabdata 1 1 0 >> sgpool-32 2 4 1024 4 1 : tunables 54 27 8 >> : slabdata 1 1 0 >> sgpool-16 2 8 512 8 1 : tunables 54 27 8 >> : slabdata 1 1 0 >> sgpool-8 2 15 256 15 1 : tunables 120 60 8 >> : slabdata 1 1 0 >> scsi_data_buffer 0 0 24 144 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> scsi_io_context 0 0 112 34 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> flow_cache 0 0 96 40 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> cfq_io_context 58 207 168 23 1 : tunables 120 60 8 >> : slabdata 9 9 0 >> cfq_queue 56 308 136 28 1 : tunables 120 60 8 >> : slabdata 11 11 0 >> >> bsg_cmd 0 0 312 12 1 : tunables 54 27 8 >> : slabdata 0 0 0 >> mqueue_inode_cache 1 4 896 4 1 : tunables 54 27 >> 8 : slabdata 1 1 0 >> isofs_inode_cache 0 0 608 6 1 : tunables 54 27 8 >> : slabdata 0 0 0 >> minix_inode_cache 0 0 624 6 1 : tunables 54 27 8 >> : slabdata 0 0 0 >> hugetlbfs_inode_cache 1 7 576 7 1 : tunables 54 >> 27 8 : slabdata 1 1 0 >> dnotify_cache 0 0 40 92 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> dquot 0 0 256 15 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> inotify_event_cache 0 0 40 92 1 : tunables 120 60 >> 8 : slabdata 0 0 0 >> inotify_watch_cache 94 159 72 53 1 : tunables 120 60 >> 8 : slabdata 3 3 0 >> >> kioctx 0 0 384 10 1 : tunables 54 27 8 >> : slabdata 0 0 0 >> kiocb 0 0 256 15 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> fasync_cache 0 0 24 144 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> shmem_inode_cache 878 1040 784 5 1 : tunables 54 27 8 >> : slabdata 208 208 0 >> >> pid_namespace 0 0 2112 3 2 : tunables 24 12 8 >> : slabdata 0 0 0 >> nsproxy 0 0 56 67 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> posix_timers_cache 0 0 192 20 1 : tunables 120 60 >> 8 : slabdata 0 0 0 >> uid_cache 7 60 128 30 1 : tunables 120 60 8 >> : slabdata 2 2 0 >> UNIX 128 220 704 11 2 : tunables 54 27 8 >> : slabdata 20 20 0 >> >> ip_mrt_cache 0 0 128 30 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> UDP-Lite 0 0 832 9 2 : tunables 54 27 8 >> : slabdata 0 0 0 >> tcp_bind_bucket 15 118 64 59 1 : tunables 120 60 8 >> : slabdata 2 2 0 >> >> inet_peer_cache 1 59 64 59 1 : tunables 120 60 8 >> : slabdata 1 1 0 >> secpath_cache 0 0 64 59 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> xfrm_dst_cache 0 0 384 10 1 : tunables 54 27 8 >> : slabdata 0 0 0 >> ip_fib_alias 0 0 32 112 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> ip_fib_hash 15 106 72 53 1 : tunables 120 60 8 >> : slabdata 2 2 0 >> ip_dst_cache 40 84 320 12 1 : tunables 54 27 8 >> : slabdata 7 7 0 >> >> arp_cache 8 15 256 15 1 : tunables 120 60 8 >> : slabdata 1 1 0 >> RAW 33 35 768 5 1 : tunables 54 27 8 >> : slabdata 7 7 0 >> UDP 11 36 832 9 2 : tunables 54 27 8 >> : slabdata 4 4 0 >> tw_sock_TCP 4 20 192 20 1 : tunables 120 60 8 >> : slabdata 1 1 0 >> >> request_sock_TCP 0 0 128 30 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> TCP 16 24 1664 4 2 : tunables 24 12 8 >> : slabdata 6 6 0 >> eventpoll_pwq 69 159 72 53 1 : tunables 120 60 8 >> : slabdata 3 3 0 >> eventpoll_epi 69 150 128 30 1 : tunables 120 60 8 >> : slabdata 5 5 0 >> >> pfm_event_set 0 0 57344 1 16 : tunables 8 4 0 >> : slabdata 0 0 0 >> pfm_context 0 0 8192 1 2 : tunables 8 4 0 >> : slabdata 0 0 0 >> blkdev_integrity 0 0 112 34 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> blkdev_queue 10 12 2264 3 2 : tunables 24 12 8 >> : slabdata 4 4 0 >> blkdev_requests 91 130 368 10 1 : tunables 54 27 8 >> : slabdata 13 13 27 >> blkdev_ioc 56 371 72 53 1 : tunables 120 60 8 >> : slabdata 7 7 0 >> >> biovec-256 2 2 4096 1 1 : tunables 24 12 8 >> : slabdata 2 2 0 >> biovec-128 2 4 2048 2 1 : tunables 24 12 8 >> : slabdata 2 2 0 >> biovec-64 2 8 1024 4 1 : tunables 54 27 8 >> : slabdata 2 2 0 >> biovec-16 2 30 256 15 1 : tunables 120 60 8 >> : slabdata 2 2 0 >> biovec-4 2 118 64 59 1 : tunables 120 60 8 >> : slabdata 2 2 0 >> biovec-1 223 606 16 202 1 : tunables 120 60 8 >> : slabdata 3 3 70 >> >> bio_integrity_payload 2 60 128 30 1 : tunables 120 >> 60 8 : slabdata 2 2 0 >> bio 205 330 128 30 1 : tunables 120 60 >> 8 : slabdata 11 11 70 >> sock_inode_cache 245 300 640 6 1 : tunables 54 27 8 >> : slabdata 50 50 0 >> skbuff_fclone_cache 14 14 512 7 1 : tunables 54 27 >> 8 : slabdata 2 2 0 >> skbuff_head_cache 5121 5985 256 15 1 : tunables 120 60 8 >> : slabdata 399 399 68 >> file_lock_cache 4 22 176 22 1 : tunables 120 60 8 >> : slabdata 1 1 0 >> Acpi-Operand 889 1749 72 53 1 : tunables 120 60 8 >> : slabdata 33 33 0 >> >> Acpi-ParseExt 0 0 72 53 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> Acpi-Parse 0 0 48 77 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> Acpi-State 0 0 80 48 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> Acpi-Namespace 617 672 32 112 1 : tunables 120 60 8 >> : slabdata 6 6 0 >> task_delay_info 389 884 112 34 1 : tunables 120 60 8 >> : slabdata 26 26 0 >> >> taskstats 0 0 328 12 1 : tunables 54 27 8 >> : slabdata 0 0 0 >> page_cgroup 0 0 40 92 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> proc_inode_cache 1397 1446 608 6 1 : tunables 54 27 8 >> : slabdata 240 241 190 >> sigqueue 29 96 160 24 1 : tunables 120 60 8 >> : slabdata 4 4 1 >> radix_tree_node 193120 196672 552 7 1 : tunables 54 27 8 >> : slabdata 28096 28096 216 >> bdev_cache 5 15 768 5 1 : tunables 54 27 8 >> : slabdata 3 3 0 >> >> sysfs_dir_cache 19120 19296 80 48 1 : tunables 120 60 8 >> : slabdata 402 402 0 >> mnt_cache 30 105 256 15 1 : tunables 120 60 8 >> : slabdata 7 7 0 >> inode_cache 1128 1176 560 7 1 : tunables 54 27 8 >> : slabdata 166 168 24 >> dentry 4651 8189 208 19 1 : tunables 120 60 8 >> : slabdata 431 431 0 >> filp 1563 2720 192 20 1 : tunables 120 60 8 >> : slabdata 136 136 242 >> names_cache 142 142 4096 1 1 : tunables 24 12 8 >> : slabdata 142 142 96 >> >> key_jar 0 0 192 20 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> buffer_head 1129 3071 104 37 1 : tunables 120 60 8 >> : slabdata 83 83 0 >> mm_struct 86 136 896 4 1 : tunables 54 27 8 >> : slabdata 34 34 1 >> vm_area_struct 3406 4136 176 22 1 : tunables 120 60 8 >> : slabdata 188 188 26 >> fs_cache 140 531 64 59 1 : tunables 120 60 8 >> : slabdata 9 9 1 >> files_cache 83 150 768 5 1 : tunables 54 27 8 >> : slabdata 30 30 1 >> signal_cache 325 388 960 4 1 : tunables 54 27 8 >> : slabdata 97 97 0 >> sighand_cache 317 369 2112 3 2 : tunables 24 12 8 >> : slabdata 123 123 0 >> task_xstate 155 256 512 8 1 : tunables 54 27 8 >> : slabdata 32 32 2 >> task_struct 368 372 5872 1 2 : tunables 8 4 0 >> : slabdata 368 372 0 >> anon_vma 966 1728 24 144 1 : tunables 120 60 8 >> : slabdata 12 12 0 >> pid 377 960 128 30 1 : tunables 120 60 8 >> : slabdata 32 32 0 >> >> shared_policy_node 0 0 48 77 1 : tunables 120 60 >> 8 : slabdata 0 0 0 >> numa_policy 15 112 136 28 1 : tunables 120 60 8 >> : slabdata 4 4 0 >> idr_layer_cache 284 322 544 7 1 : tunables 54 27 8 >> : slabdata 46 46 0 >> >> size-4194304(DMA) 0 0 4194304 1 1024 : tunables 1 1 >> 0 : slabdata 0 0 0 >> size-4194304 0 0 4194304 1 1024 : tunables 1 1 >> 0 : slabdata 0 0 0 >> size-2097152(DMA) 0 0 2097152 1 512 : tunables 1 1 >> 0 : slabdata 0 0 0 >> size-2097152 0 0 2097152 1 512 : tunables 1 1 >> 0 : slabdata 0 0 0 >> size-1048576(DMA) 0 0 1048576 1 256 : tunables 1 1 >> 0 : slabdata 0 0 0 >> size-1048576 0 0 1048576 1 256 : tunables 1 1 >> 0 : slabdata 0 0 0 >> size-524288(DMA) 0 0 524288 1 128 : tunables 1 1 0 >> : slabdata 0 0 0 >> size-524288 0 0 524288 1 128 : tunables 1 1 0 >> : slabdata 0 0 0 >> size-262144(DMA) 0 0 262144 1 64 : tunables 1 1 0 >> : slabdata 0 0 0 >> size-262144 0 0 262144 1 64 : tunables 1 1 0 >> : slabdata 0 0 0 >> size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 >> : slabdata 0 0 0 >> size-131072 3 3 131072 1 32 : tunables 8 4 0 >> : slabdata 3 3 0 >> size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 >> : slabdata 0 0 0 >> size-65536 6 6 65536 1 16 : tunables 8 4 0 >> : slabdata 6 6 0 >> size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 >> : slabdata 0 0 0 >> size-32768 10 10 32768 1 8 : tunables 8 4 0 >> : slabdata 10 10 0 >> >> size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 >> : slabdata 0 0 0 >> size-16384 44 44 16384 1 4 : tunables 8 4 0 >> : slabdata 44 44 0 >> >> size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 >> : slabdata 0 0 0 >> size-8192 3611 3611 8192 1 2 : tunables 8 4 0 >> : slabdata 3611 3611 0 >> >> size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8 >> : slabdata 0 0 0 >> size-4096 1771 1771 4096 1 1 : tunables 24 12 8 >> : slabdata 1771 1771 0 >> >> size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8 >> : slabdata 0 0 0 >> size-2048 4609 4714 2048 2 1 : tunables 24 12 8 >> : slabdata 2357 2357 0 >> >> size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8 >> : slabdata 0 0 0 >> size-1024 4829 4900 1024 4 1 : tunables 54 27 8 >> : slabdata 1225 1225 0 >> >> size-512(DMA) 0 0 512 8 1 : tunables 54 27 8 >> : slabdata 0 0 0 >> size-512 1478 1520 512 8 1 : tunables 54 27 8 >> : slabdata 190 190 39 >> >> size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> size-256 4662 5550 256 15 1 : tunables 120 60 8 >> : slabdata 370 370 1 >> >> size-128(DMA) 0 0 128 30 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> size-64(DMA) 0 0 64 59 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> size-64 17232 29382 64 59 1 : tunables 120 60 8 >> : slabdata 498 498 0 >> >> size-32(DMA) 0 0 32 112 1 : tunables 120 60 8 >> : slabdata 0 0 0 >> size-128 9907 16140 128 30 1 : tunables 120 60 8 >> : slabdata 538 538 0 >> size-32 12487 13104 32 112 1 : tunables 120 60 8 >> : slabdata 117 117 0 >> >> kmem_cache 181 181 4224 1 2 : tunables 8 4 0 >> : slabdata 181 181 0 >> >> >> Tasks: 278 total, 1 running, 276 sleeping, 0 stopped, 1 zombie >> Cpu(s): 3.8%us, 0.1%sy, 0.0%ni, 96.0%id, 0.0%wa, 0.0%hi, 0.0%si, >> 0.0%st >> Mem: 198091444k total, 197636988k used, 454456k free, 4544k buffers >> Swap: 75505460k total, 8567448k used, 66938012k free, 29144008k cached >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> >> 107 root 15 -5 0 0 0 D 10 0.0 5:06.43 >> kswapd1 >> >> 19328 user1 20 0 66.5g 60g 2268 D 4 32.0 31:48.49 >> R >> >> 1 root 20 0 1064 64 32 S 0 0.0 0:21.20 >> init >> >> 2 root 15 -5 0 0 0 S 0 0.0 0:00.06 >> kthreadd >> >> 3 root RT -5 0 0 0 S 0 0.0 0:00.24 >> migration/0 >> >> 4 root 15 -5 0 0 0 S 0 0.0 1:01.12 >> ksoftirqd/0 >> >> 5 root RT -5 0 0 0 S 0 0.0 0:00.30 >> migration/1 >> >> 6 root 15 -5 0 0 0 S 0 0.0 0:00.50 >> ksoftirqd/1 >> >> 7 root RT -5 0 0 0 S 0 0.0 0:00.22 >> migration/2 >> >> 8 root 15 -5 0 0 0 S 0 0.0 0:00.36 >> ksoftirqd/2 >> >> 9 root RT -5 0 0 0 S 0 0.0 0:00.28 >> migration/3 >> >> 10 root 15 -5 0 0 0 S 0 0.0 0:00.60 >> ksoftirqd/3 >> >> 11 root RT -5 0 0 0 S 0 0.0 0:00.18 >> migration/4 >> >> 12 root 15 -5 0 0 0 S 0 0.0 0:00.40 >> ksoftirqd/4 >> >> 13 root RT -5 0 0 0 S 0 0.0 0:00.26 >> migration/5 >> >> 14 root 15 -5 0 0 0 S 0 0.0 0:00.76 >> ksoftirqd/5 >> >> 15 root RT -5 0 0 0 S 0 0.0 0:00.20 >> migration/6 >> >> 16 root 15 -5 0 0 0 S 0 0.0 0:00.36 >> ksoftirqd/6 >> >> 17 root RT -5 0 0 0 S 0 0.0 0:00.26 >> migration/7 >> >> 18 root 15 -5 0 0 0 S 0 0.0 0:00.68 >> ksoftirqd/7 >> >> 19 root RT -5 0 0 0 S 0 0.0 0:00.88 >> migration/8 >> >> 20 root 15 -5 0 0 0 S 0 0.0 0:07.70 >> ksoftirqd/8 >> >> 21 root RT -5 0 0 0 S 0 0.0 0:01.12 >> migration/9 >> >> 22 root 15 -5 0 0 0 S 0 0.0 0:01.20 >> ksoftirqd/9 >> >> 23 root RT -5 0 0 0 S 0 0.0 0:03.50 >> migration/10 >> >> 24 root 15 -5 0 0 0 S 0 0.0 0:01.22 >> ksoftirqd/10 >> >> 25 root RT -5 0 0 0 S 0 0.0 0:04.84 >> migration/11 >> >> 26 root 15 -5 0 0 0 S 0 0.0 0:01.90 >> ksoftirqd/11 >> >> 27 root RT -5 0 0 0 S 0 0.0 0:01.46 >> migration/12 >> >> 28 root 15 -5 0 0 0 S 0 0.0 0:01.42 >> ksoftirqd/12 >> >> 29 root RT -5 0 0 0 S 0 0.0 0:01.62 >> migration/13 >> >> 30 root 15 -5 0 0 0 S 0 0.0 0:01.84 >> ksoftirqd/13 >> >> 31 root RT -5 0 0 0 S 0 0.0 0:01.90 >> migration/14 >> >> 32 root 15 -5 0 0 0 S 0 0.0 0:01.18 >> ksoftirqd/14 >> -- >> >> Thanks, >> -J >> >> On Mon, Apr 19, 2010 at 10:07 AM, Andreas Dilger < >> andreas.dilger at oracle.com> wrote: >> >>> There is a known problem with the DLM LRU size that may be affecting you. >>> It may be something else too. Please check /proc/{slabinfo,meminfo} to see >>> what is using the memory on the client. >>> >>> Cheers, Andreas >>> >>> >>> On 2010-04-19, at 10:43, Jagga Soorma <jagga13 at gmail.com> wrote: >>> >>> Hi Guys, >>>> >>>> My users are reporting some issues with memory on our lustre 1.8.1 >>>> clients. It looks like when they submit a single job at a time the run time >>>> was about 4.5 minutes. However, when they ran multiple jobs (10 or less) on >>>> a client with 192GB of memory on a single node the run time for each job was >>>> exceeding 3-4X the run time for the single process. They also noticed that >>>> the swap space kept climbing even though there was plenty of free memory on >>>> the system. Could this possibly be related to the lustre client? Does it >>>> reserve any memory that is not accessible by any other process even though >>>> it might not be in use? >>>> >>>> Thanks much, >>>> -J >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>>> >>> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100419/140b8af6/attachment-0001.html
I have tried: echo 1 > /proc/sys/vm/drop_caches & echo 3 > /proc/sys/vm/drop_caches However the free memory does not change at all. Any ideas what might be going on? -Simran On Mon, Apr 19, 2010 at 11:37 AM, Jagga Soorma <jagga13 at gmail.com> wrote:> Could it be locking? I do have the flock option enabled. > > -- > lustre_inode_cache 123 192 896 4 1 : tunables 54 27 8 > : slabdata 48 48 0 > lov_oinfo 128 228 320 12 1 : tunables 54 27 8 > : slabdata 19 19 0 > ldlm_locks 1550 3992 512 8 1 : tunables 54 27 8 > : slabdata 499 499 0 > ldlm_resources 1449 3600 384 10 1 : tunables 54 27 8 > : slabdata 360 360 0 > -- > > Thanks, > -J > > > On Mon, Apr 19, 2010 at 11:26 AM, Jagga Soorma <jagga13 at gmail.com> wrote: > >> Here is something from April 12 that I see in the client logs. Not sure >> if this is related: >> >> -- >> Apr 12 14:51:16 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) ino >> 424411146 page 0 (0) not covered by a lock (mmap?). check debug logs. >> Apr 12 14:51:16 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) ino >> 424411146 page 1480 (6062080) not covered by a lock (mmap?). check debug >> logs. >> Apr 12 14:51:16 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) >> Skipped 1479 previous similar messages >> Apr 12 14:51:17 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) ino >> 424411146 page 273025 (1118310400) not covered by a lock (mmap?). check >> debug logs. >> Apr 12 14:51:17 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) >> Skipped 271544 previous similar messages >> -- >> >> -J >> >> >> On Mon, Apr 19, 2010 at 11:02 AM, Jagga Soorma <jagga13 at gmail.com> wrote: >> >>> Andreas, >>> >>> I am seeing the problem again on one of my hosts and here is a live >>> capture of the data. Can you assist with this? >>> >>> -- >>> # free >>> total used free shared buffers cached >>> Mem: 198091444 197636852 454592 0 4260 34251452 >>> -/+ buffers/cache: 163381140 34710304 >>> Swap: 75505460 10281796 65223664 >>> >>> # cat /proc/meminfo >>> MemTotal: 198091444 kB >>> MemFree: 458048 kB >>> Buffers: 4268 kB >>> Cached: 34099372 kB >>> SwapCached: 7730744 kB >>> Active: 62919152 kB >>> Inactive: 34107188 kB >>> SwapTotal: 75505460 kB >>> SwapFree: 65220676 kB >>> Dirty: 444 kB >>> Writeback: 0 kB >>> AnonPages: 58704728 kB >>> Mapped: 12036 kB >>> Slab: 99806476 kB >>> SReclaimable: 118532 kB >>> SUnreclaim: 99687944 kB >>> PageTables: 131200 kB >>> >>> NFS_Unstable: 0 kB >>> Bounce: 0 kB >>> WritebackTmp: 0 kB >>> CommitLimit: 174551180 kB >>> Committed_AS: 65739660 kB >>> >>> VmallocTotal: 34359738367 kB >>> VmallocUsed: 588416 kB >>> VmallocChunk: 34359149923 kB >>> HugePages_Total: 0 >>> HugePages_Free: 0 >>> HugePages_Rsvd: 0 >>> HugePages_Surp: 0 >>> Hugepagesize: 2048 kB >>> DirectMap4k: 8432 kB >>> DirectMap2M: 201308160 kB >>> >>> # cat /proc/slabinfo >>> slabinfo - version: 2.1 >>> # name <active_objs> <num_objs> <objsize> <objperslab> >>> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata >>> <active_slabs> <num_slabs> <sharedavail> >>> nfs_direct_cache 0 0 128 30 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> nfs_write_data 36 44 704 11 2 : tunables 54 27 >>> 8 : slabdata 4 4 0 >>> nfs_read_data 32 33 704 11 2 : tunables 54 27 >>> 8 : slabdata 3 3 0 >>> nfs_inode_cache 0 0 984 4 1 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> nfs_page 0 0 128 30 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> rpc_buffers 8 8 2048 2 1 : tunables 24 12 >>> 8 : slabdata 4 4 0 >>> rpc_tasks 8 12 320 12 1 : tunables 54 27 >>> 8 : slabdata 1 1 0 >>> rpc_inode_cache 0 0 832 4 1 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> ll_async_page 8494811 8507076 320 12 1 : tunables 54 >>> 27 8 : slabdata 708923 708923 216 >>> ll_file_data 16 40 192 20 1 : tunables 120 60 >>> 8 : slabdata 2 2 0 >>> lustre_inode_cache 95 184 896 4 1 : tunables 54 27 >>> 8 : slabdata 46 46 0 >>> lov_oinfo 56 180 320 12 1 : tunables 54 27 >>> 8 : slabdata 15 15 0 >>> >>> osc_quota_info 0 0 32 112 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> ll_qunit_cache 0 0 112 34 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> llcd_cache 0 0 3952 1 1 : tunables 24 12 >>> 8 : slabdata 0 0 0 >>> ptlrpc_cbdatas 0 0 32 112 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> interval_node 1680 5730 128 30 1 : tunables 120 60 >>> 8 : slabdata 191 191 0 >>> ldlm_locks 2255 6232 512 8 1 : tunables 54 27 >>> 8 : slabdata 779 779 0 >>> ldlm_resources 2227 5570 384 10 1 : tunables 54 27 >>> 8 : slabdata 557 557 0 >>> >>> ll_import_cache 0 0 1248 3 1 : tunables 24 12 >>> 8 : slabdata 0 0 0 >>> ll_obdo_cache 0 459630919 208 19 1 : tunables 120 >>> 60 8 : slabdata 0 24191101 0 >>> >>> ll_obd_dev_cache 13 13 5672 1 2 : tunables 8 4 >>> 0 : slabdata 13 13 0 >>> obd_lvfs_ctxt_cache 0 0 96 40 1 : tunables 120 >>> 60 8 : slabdata 0 0 0 >>> SDP 0 0 1728 4 2 : tunables 24 12 >>> 8 : slabdata 0 0 0 >>> fib6_nodes 7 59 64 59 1 : tunables 120 60 >>> 8 : slabdata 1 1 0 >>> ip6_dst_cache 10 24 320 12 1 : tunables 54 27 >>> 8 : slabdata 2 2 0 >>> >>> ndisc_cache 3 30 256 15 1 : tunables 120 60 >>> 8 : slabdata 2 2 0 >>> RAWv6 35 36 960 4 1 : tunables 54 27 >>> 8 : slabdata 9 9 0 >>> UDPLITEv6 0 0 960 4 1 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> UDPv6 7 12 960 4 1 : tunables 54 27 >>> 8 : slabdata 3 3 0 >>> tw_sock_TCPv6 0 0 192 20 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> request_sock_TCPv6 0 0 192 20 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> TCPv6 3 4 1792 2 1 : tunables 24 12 >>> 8 : slabdata 2 2 0 >>> ib_mad 2051 2096 448 8 1 : tunables 54 27 >>> 8 : slabdata 262 262 0 >>> >>> fuse_request 0 0 608 6 1 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> fuse_inode 0 0 704 11 2 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> kcopyd_job 0 0 360 11 1 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> dm_uevent 0 0 2608 3 2 : tunables 24 12 >>> 8 : slabdata 0 0 0 >>> dm_clone_bio_info 0 0 16 202 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> dm_rq_target_io 0 0 408 9 1 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> dm_target_io 0 0 24 144 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> dm_io 0 0 32 112 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> uhci_urb_priv 1 67 56 67 1 : tunables 120 60 >>> 8 : slabdata 1 1 0 >>> ext3_inode_cache 2472 2610 768 5 1 : tunables 54 27 >>> 8 : slabdata 522 522 0 >>> >>> ext3_xattr 0 0 88 44 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> journal_handle 56 288 24 144 1 : tunables 120 60 >>> 8 : slabdata 2 2 0 >>> journal_head 216 240 96 40 1 : tunables 120 60 >>> 8 : slabdata 6 6 0 >>> >>> revoke_table 4 202 16 202 1 : tunables 120 60 >>> 8 : slabdata 1 1 0 >>> revoke_record 0 0 32 112 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> sgpool-128 2 2 4096 1 1 : tunables 24 12 >>> 8 : slabdata 2 2 0 >>> sgpool-64 2 2 2048 2 1 : tunables 24 12 >>> 8 : slabdata 1 1 0 >>> sgpool-32 2 4 1024 4 1 : tunables 54 27 >>> 8 : slabdata 1 1 0 >>> sgpool-16 2 8 512 8 1 : tunables 54 27 >>> 8 : slabdata 1 1 0 >>> sgpool-8 2 15 256 15 1 : tunables 120 60 >>> 8 : slabdata 1 1 0 >>> scsi_data_buffer 0 0 24 144 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> scsi_io_context 0 0 112 34 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> flow_cache 0 0 96 40 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> cfq_io_context 58 207 168 23 1 : tunables 120 60 >>> 8 : slabdata 9 9 0 >>> cfq_queue 56 308 136 28 1 : tunables 120 60 >>> 8 : slabdata 11 11 0 >>> >>> bsg_cmd 0 0 312 12 1 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> mqueue_inode_cache 1 4 896 4 1 : tunables 54 27 >>> 8 : slabdata 1 1 0 >>> isofs_inode_cache 0 0 608 6 1 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> minix_inode_cache 0 0 624 6 1 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> hugetlbfs_inode_cache 1 7 576 7 1 : tunables 54 >>> 27 8 : slabdata 1 1 0 >>> dnotify_cache 0 0 40 92 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> dquot 0 0 256 15 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> inotify_event_cache 0 0 40 92 1 : tunables 120 >>> 60 8 : slabdata 0 0 0 >>> inotify_watch_cache 94 159 72 53 1 : tunables 120 >>> 60 8 : slabdata 3 3 0 >>> >>> kioctx 0 0 384 10 1 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> kiocb 0 0 256 15 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> fasync_cache 0 0 24 144 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> shmem_inode_cache 878 1040 784 5 1 : tunables 54 27 >>> 8 : slabdata 208 208 0 >>> >>> pid_namespace 0 0 2112 3 2 : tunables 24 12 >>> 8 : slabdata 0 0 0 >>> nsproxy 0 0 56 67 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> posix_timers_cache 0 0 192 20 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> uid_cache 7 60 128 30 1 : tunables 120 60 >>> 8 : slabdata 2 2 0 >>> UNIX 128 220 704 11 2 : tunables 54 27 >>> 8 : slabdata 20 20 0 >>> >>> ip_mrt_cache 0 0 128 30 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> UDP-Lite 0 0 832 9 2 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> tcp_bind_bucket 15 118 64 59 1 : tunables 120 60 >>> 8 : slabdata 2 2 0 >>> >>> inet_peer_cache 1 59 64 59 1 : tunables 120 60 >>> 8 : slabdata 1 1 0 >>> secpath_cache 0 0 64 59 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> xfrm_dst_cache 0 0 384 10 1 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> ip_fib_alias 0 0 32 112 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> ip_fib_hash 15 106 72 53 1 : tunables 120 60 >>> 8 : slabdata 2 2 0 >>> ip_dst_cache 40 84 320 12 1 : tunables 54 27 >>> 8 : slabdata 7 7 0 >>> >>> arp_cache 8 15 256 15 1 : tunables 120 60 >>> 8 : slabdata 1 1 0 >>> RAW 33 35 768 5 1 : tunables 54 27 >>> 8 : slabdata 7 7 0 >>> UDP 11 36 832 9 2 : tunables 54 27 >>> 8 : slabdata 4 4 0 >>> tw_sock_TCP 4 20 192 20 1 : tunables 120 60 >>> 8 : slabdata 1 1 0 >>> >>> request_sock_TCP 0 0 128 30 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> TCP 16 24 1664 4 2 : tunables 24 12 >>> 8 : slabdata 6 6 0 >>> eventpoll_pwq 69 159 72 53 1 : tunables 120 60 >>> 8 : slabdata 3 3 0 >>> eventpoll_epi 69 150 128 30 1 : tunables 120 60 >>> 8 : slabdata 5 5 0 >>> >>> pfm_event_set 0 0 57344 1 16 : tunables 8 4 >>> 0 : slabdata 0 0 0 >>> pfm_context 0 0 8192 1 2 : tunables 8 4 >>> 0 : slabdata 0 0 0 >>> blkdev_integrity 0 0 112 34 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> blkdev_queue 10 12 2264 3 2 : tunables 24 12 >>> 8 : slabdata 4 4 0 >>> blkdev_requests 91 130 368 10 1 : tunables 54 27 >>> 8 : slabdata 13 13 27 >>> blkdev_ioc 56 371 72 53 1 : tunables 120 60 >>> 8 : slabdata 7 7 0 >>> >>> biovec-256 2 2 4096 1 1 : tunables 24 12 >>> 8 : slabdata 2 2 0 >>> biovec-128 2 4 2048 2 1 : tunables 24 12 >>> 8 : slabdata 2 2 0 >>> biovec-64 2 8 1024 4 1 : tunables 54 27 >>> 8 : slabdata 2 2 0 >>> biovec-16 2 30 256 15 1 : tunables 120 60 >>> 8 : slabdata 2 2 0 >>> biovec-4 2 118 64 59 1 : tunables 120 60 >>> 8 : slabdata 2 2 0 >>> biovec-1 223 606 16 202 1 : tunables 120 60 >>> 8 : slabdata 3 3 70 >>> >>> bio_integrity_payload 2 60 128 30 1 : tunables 120 >>> 60 8 : slabdata 2 2 0 >>> bio 205 330 128 30 1 : tunables 120 60 >>> 8 : slabdata 11 11 70 >>> sock_inode_cache 245 300 640 6 1 : tunables 54 27 >>> 8 : slabdata 50 50 0 >>> skbuff_fclone_cache 14 14 512 7 1 : tunables 54 >>> 27 8 : slabdata 2 2 0 >>> skbuff_head_cache 5121 5985 256 15 1 : tunables 120 60 >>> 8 : slabdata 399 399 68 >>> file_lock_cache 4 22 176 22 1 : tunables 120 60 >>> 8 : slabdata 1 1 0 >>> Acpi-Operand 889 1749 72 53 1 : tunables 120 60 >>> 8 : slabdata 33 33 0 >>> >>> Acpi-ParseExt 0 0 72 53 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> Acpi-Parse 0 0 48 77 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> Acpi-State 0 0 80 48 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> Acpi-Namespace 617 672 32 112 1 : tunables 120 60 >>> 8 : slabdata 6 6 0 >>> task_delay_info 389 884 112 34 1 : tunables 120 60 >>> 8 : slabdata 26 26 0 >>> >>> taskstats 0 0 328 12 1 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> page_cgroup 0 0 40 92 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> proc_inode_cache 1397 1446 608 6 1 : tunables 54 27 >>> 8 : slabdata 240 241 190 >>> sigqueue 29 96 160 24 1 : tunables 120 60 >>> 8 : slabdata 4 4 1 >>> radix_tree_node 193120 196672 552 7 1 : tunables 54 27 >>> 8 : slabdata 28096 28096 216 >>> bdev_cache 5 15 768 5 1 : tunables 54 27 >>> 8 : slabdata 3 3 0 >>> >>> sysfs_dir_cache 19120 19296 80 48 1 : tunables 120 60 >>> 8 : slabdata 402 402 0 >>> mnt_cache 30 105 256 15 1 : tunables 120 60 >>> 8 : slabdata 7 7 0 >>> inode_cache 1128 1176 560 7 1 : tunables 54 27 >>> 8 : slabdata 166 168 24 >>> dentry 4651 8189 208 19 1 : tunables 120 60 >>> 8 : slabdata 431 431 0 >>> filp 1563 2720 192 20 1 : tunables 120 60 >>> 8 : slabdata 136 136 242 >>> names_cache 142 142 4096 1 1 : tunables 24 12 >>> 8 : slabdata 142 142 96 >>> >>> key_jar 0 0 192 20 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> buffer_head 1129 3071 104 37 1 : tunables 120 60 >>> 8 : slabdata 83 83 0 >>> mm_struct 86 136 896 4 1 : tunables 54 27 >>> 8 : slabdata 34 34 1 >>> vm_area_struct 3406 4136 176 22 1 : tunables 120 60 >>> 8 : slabdata 188 188 26 >>> fs_cache 140 531 64 59 1 : tunables 120 60 >>> 8 : slabdata 9 9 1 >>> files_cache 83 150 768 5 1 : tunables 54 27 >>> 8 : slabdata 30 30 1 >>> signal_cache 325 388 960 4 1 : tunables 54 27 >>> 8 : slabdata 97 97 0 >>> sighand_cache 317 369 2112 3 2 : tunables 24 12 >>> 8 : slabdata 123 123 0 >>> task_xstate 155 256 512 8 1 : tunables 54 27 >>> 8 : slabdata 32 32 2 >>> task_struct 368 372 5872 1 2 : tunables 8 4 >>> 0 : slabdata 368 372 0 >>> anon_vma 966 1728 24 144 1 : tunables 120 60 >>> 8 : slabdata 12 12 0 >>> pid 377 960 128 30 1 : tunables 120 60 >>> 8 : slabdata 32 32 0 >>> >>> shared_policy_node 0 0 48 77 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> numa_policy 15 112 136 28 1 : tunables 120 60 >>> 8 : slabdata 4 4 0 >>> idr_layer_cache 284 322 544 7 1 : tunables 54 27 >>> 8 : slabdata 46 46 0 >>> >>> size-4194304(DMA) 0 0 4194304 1 1024 : tunables 1 1 >>> 0 : slabdata 0 0 0 >>> size-4194304 0 0 4194304 1 1024 : tunables 1 1 >>> 0 : slabdata 0 0 0 >>> size-2097152(DMA) 0 0 2097152 1 512 : tunables 1 1 >>> 0 : slabdata 0 0 0 >>> size-2097152 0 0 2097152 1 512 : tunables 1 1 >>> 0 : slabdata 0 0 0 >>> size-1048576(DMA) 0 0 1048576 1 256 : tunables 1 1 >>> 0 : slabdata 0 0 0 >>> size-1048576 0 0 1048576 1 256 : tunables 1 1 >>> 0 : slabdata 0 0 0 >>> size-524288(DMA) 0 0 524288 1 128 : tunables 1 1 >>> 0 : slabdata 0 0 0 >>> size-524288 0 0 524288 1 128 : tunables 1 1 >>> 0 : slabdata 0 0 0 >>> size-262144(DMA) 0 0 262144 1 64 : tunables 1 1 >>> 0 : slabdata 0 0 0 >>> size-262144 0 0 262144 1 64 : tunables 1 1 >>> 0 : slabdata 0 0 0 >>> size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 >>> 0 : slabdata 0 0 0 >>> size-131072 3 3 131072 1 32 : tunables 8 4 >>> 0 : slabdata 3 3 0 >>> size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 >>> 0 : slabdata 0 0 0 >>> size-65536 6 6 65536 1 16 : tunables 8 4 >>> 0 : slabdata 6 6 0 >>> size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 >>> 0 : slabdata 0 0 0 >>> size-32768 10 10 32768 1 8 : tunables 8 4 >>> 0 : slabdata 10 10 0 >>> >>> size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 >>> 0 : slabdata 0 0 0 >>> size-16384 44 44 16384 1 4 : tunables 8 4 >>> 0 : slabdata 44 44 0 >>> >>> size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 >>> 0 : slabdata 0 0 0 >>> size-8192 3611 3611 8192 1 2 : tunables 8 4 >>> 0 : slabdata 3611 3611 0 >>> >>> size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 >>> 8 : slabdata 0 0 0 >>> size-4096 1771 1771 4096 1 1 : tunables 24 12 >>> 8 : slabdata 1771 1771 0 >>> >>> size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 >>> 8 : slabdata 0 0 0 >>> size-2048 4609 4714 2048 2 1 : tunables 24 12 >>> 8 : slabdata 2357 2357 0 >>> >>> size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> size-1024 4829 4900 1024 4 1 : tunables 54 27 >>> 8 : slabdata 1225 1225 0 >>> >>> size-512(DMA) 0 0 512 8 1 : tunables 54 27 >>> 8 : slabdata 0 0 0 >>> size-512 1478 1520 512 8 1 : tunables 54 27 >>> 8 : slabdata 190 190 39 >>> >>> size-256(DMA) 0 0 256 15 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> size-256 4662 5550 256 15 1 : tunables 120 60 >>> 8 : slabdata 370 370 1 >>> >>> size-128(DMA) 0 0 128 30 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> size-64(DMA) 0 0 64 59 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> size-64 17232 29382 64 59 1 : tunables 120 60 >>> 8 : slabdata 498 498 0 >>> >>> size-32(DMA) 0 0 32 112 1 : tunables 120 60 >>> 8 : slabdata 0 0 0 >>> size-128 9907 16140 128 30 1 : tunables 120 60 >>> 8 : slabdata 538 538 0 >>> size-32 12487 13104 32 112 1 : tunables 120 60 >>> 8 : slabdata 117 117 0 >>> >>> kmem_cache 181 181 4224 1 2 : tunables 8 4 >>> 0 : slabdata 181 181 0 >>> >>> >>> Tasks: 278 total, 1 running, 276 sleeping, 0 stopped, 1 zombie >>> Cpu(s): 3.8%us, 0.1%sy, 0.0%ni, 96.0%id, 0.0%wa, 0.0%hi, 0.0%si, >>> 0.0%st >>> Mem: 198091444k total, 197636988k used, 454456k free, 4544k >>> buffers >>> Swap: 75505460k total, 8567448k used, 66938012k free, 29144008k cached >>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>> COMMAND >>> >>> 107 root 15 -5 0 0 0 D 10 0.0 5:06.43 >>> kswapd1 >>> >>> 19328 user1 20 0 66.5g 60g 2268 D 4 32.0 31:48.49 >>> R >>> >>> 1 root 20 0 1064 64 32 S 0 0.0 0:21.20 >>> init >>> >>> 2 root 15 -5 0 0 0 S 0 0.0 0:00.06 >>> kthreadd >>> >>> 3 root RT -5 0 0 0 S 0 0.0 0:00.24 >>> migration/0 >>> >>> 4 root 15 -5 0 0 0 S 0 0.0 1:01.12 >>> ksoftirqd/0 >>> >>> 5 root RT -5 0 0 0 S 0 0.0 0:00.30 >>> migration/1 >>> >>> 6 root 15 -5 0 0 0 S 0 0.0 0:00.50 >>> ksoftirqd/1 >>> >>> 7 root RT -5 0 0 0 S 0 0.0 0:00.22 >>> migration/2 >>> >>> 8 root 15 -5 0 0 0 S 0 0.0 0:00.36 >>> ksoftirqd/2 >>> >>> 9 root RT -5 0 0 0 S 0 0.0 0:00.28 >>> migration/3 >>> >>> 10 root 15 -5 0 0 0 S 0 0.0 0:00.60 >>> ksoftirqd/3 >>> >>> 11 root RT -5 0 0 0 S 0 0.0 0:00.18 >>> migration/4 >>> >>> 12 root 15 -5 0 0 0 S 0 0.0 0:00.40 >>> ksoftirqd/4 >>> >>> 13 root RT -5 0 0 0 S 0 0.0 0:00.26 >>> migration/5 >>> >>> 14 root 15 -5 0 0 0 S 0 0.0 0:00.76 >>> ksoftirqd/5 >>> >>> 15 root RT -5 0 0 0 S 0 0.0 0:00.20 >>> migration/6 >>> >>> 16 root 15 -5 0 0 0 S 0 0.0 0:00.36 >>> ksoftirqd/6 >>> >>> 17 root RT -5 0 0 0 S 0 0.0 0:00.26 >>> migration/7 >>> >>> 18 root 15 -5 0 0 0 S 0 0.0 0:00.68 >>> ksoftirqd/7 >>> >>> 19 root RT -5 0 0 0 S 0 0.0 0:00.88 >>> migration/8 >>> >>> 20 root 15 -5 0 0 0 S 0 0.0 0:07.70 >>> ksoftirqd/8 >>> >>> 21 root RT -5 0 0 0 S 0 0.0 0:01.12 >>> migration/9 >>> >>> 22 root 15 -5 0 0 0 S 0 0.0 0:01.20 >>> ksoftirqd/9 >>> >>> 23 root RT -5 0 0 0 S 0 0.0 0:03.50 >>> migration/10 >>> >>> 24 root 15 -5 0 0 0 S 0 0.0 0:01.22 >>> ksoftirqd/10 >>> >>> 25 root RT -5 0 0 0 S 0 0.0 0:04.84 >>> migration/11 >>> >>> 26 root 15 -5 0 0 0 S 0 0.0 0:01.90 >>> ksoftirqd/11 >>> >>> 27 root RT -5 0 0 0 S 0 0.0 0:01.46 >>> migration/12 >>> >>> 28 root 15 -5 0 0 0 S 0 0.0 0:01.42 >>> ksoftirqd/12 >>> >>> 29 root RT -5 0 0 0 S 0 0.0 0:01.62 >>> migration/13 >>> >>> 30 root 15 -5 0 0 0 S 0 0.0 0:01.84 >>> ksoftirqd/13 >>> >>> 31 root RT -5 0 0 0 S 0 0.0 0:01.90 >>> migration/14 >>> >>> 32 root 15 -5 0 0 0 S 0 0.0 0:01.18 >>> ksoftirqd/14 >>> -- >>> >>> Thanks, >>> -J >>> >>> On Mon, Apr 19, 2010 at 10:07 AM, Andreas Dilger < >>> andreas.dilger at oracle.com> wrote: >>> >>>> There is a known problem with the DLM LRU size that may be affecting >>>> you. It may be something else too. Please check /proc/{slabinfo,meminfo} to >>>> see what is using the memory on the client. >>>> >>>> Cheers, Andreas >>>> >>>> >>>> On 2010-04-19, at 10:43, Jagga Soorma <jagga13 at gmail.com> wrote: >>>> >>>> Hi Guys, >>>>> >>>>> My users are reporting some issues with memory on our lustre 1.8.1 >>>>> clients. It looks like when they submit a single job at a time the run time >>>>> was about 4.5 minutes. However, when they ran multiple jobs (10 or less) on >>>>> a client with 192GB of memory on a single node the run time for each job was >>>>> exceeding 3-4X the run time for the single process. They also noticed that >>>>> the swap space kept climbing even though there was plenty of free memory on >>>>> the system. Could this possibly be related to the lustre client? Does it >>>>> reserve any memory that is not accessible by any other process even though >>>>> it might not be in use? >>>>> >>>>> Thanks much, >>>>> -J >>>>> _______________________________________________ >>>>> Lustre-discuss mailing list >>>>> Lustre-discuss at lists.lustre.org >>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>>>> >>>> >>> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100419/4bbfb75c/attachment-0001.html
Also, here is the /proc/sys/lustre/memused for all my compute nodes: -- 580093506 138839275 26861811 1192253563 137585179 138928251 2214512411 1872972173 138846915 142122291 131465011 139968027 4626653800 188594515 23944887 2495517689 -- Can someone help me put some of this information together and make sense of this? I am pretty sure this is related to lustre but not sure what might be eating up the memory. Thanks, -J On Mon, Apr 19, 2010 at 1:28 PM, Jagga Soorma <jagga13 at gmail.com> wrote:> I have tried: > > echo 1 > /proc/sys/vm/drop_caches > & > echo 3 > /proc/sys/vm/drop_caches > > However the free memory does not change at all. Any ideas what might be > going on? > > -Simran > > > On Mon, Apr 19, 2010 at 11:37 AM, Jagga Soorma <jagga13 at gmail.com> wrote: > >> Could it be locking? I do have the flock option enabled. >> >> -- >> lustre_inode_cache 123 192 896 4 1 : tunables 54 27 >> 8 : slabdata 48 48 0 >> lov_oinfo 128 228 320 12 1 : tunables 54 27 8 >> : slabdata 19 19 0 >> ldlm_locks 1550 3992 512 8 1 : tunables 54 27 8 >> : slabdata 499 499 0 >> ldlm_resources 1449 3600 384 10 1 : tunables 54 27 8 >> : slabdata 360 360 0 >> -- >> >> Thanks, >> -J >> >> >> On Mon, Apr 19, 2010 at 11:26 AM, Jagga Soorma <jagga13 at gmail.com> wrote: >> >>> Here is something from April 12 that I see in the client logs. Not sure >>> if this is related: >>> >>> -- >>> Apr 12 14:51:16 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) >>> ino 424411146 page 0 (0) not covered by a lock (mmap?). check debug logs. >>> Apr 12 14:51:16 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) >>> ino 424411146 page 1480 (6062080) not covered by a lock (mmap?). check >>> debug logs. >>> Apr 12 14:51:16 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) >>> Skipped 1479 previous similar messages >>> Apr 12 14:51:17 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) >>> ino 424411146 page 273025 (1118310400) not covered by a lock (mmap?). check >>> debug logs. >>> Apr 12 14:51:17 manak kernel: Lustre: 7359:0:(rw.c:2092:ll_readpage()) >>> Skipped 271544 previous similar messages >>> -- >>> >>> -J >>> >>> >>> On Mon, Apr 19, 2010 at 11:02 AM, Jagga Soorma <jagga13 at gmail.com>wrote: >>> >>>> Andreas, >>>> >>>> I am seeing the problem again on one of my hosts and here is a live >>>> capture of the data. Can you assist with this? >>>> >>>> -- >>>> # free >>>> total used free shared buffers >>>> cached >>>> Mem: 198091444 197636852 454592 0 4260 >>>> 34251452 >>>> -/+ buffers/cache: 163381140 34710304 >>>> Swap: 75505460 10281796 65223664 >>>> >>>> # cat /proc/meminfo >>>> MemTotal: 198091444 kB >>>> MemFree: 458048 kB >>>> Buffers: 4268 kB >>>> Cached: 34099372 kB >>>> SwapCached: 7730744 kB >>>> Active: 62919152 kB >>>> Inactive: 34107188 kB >>>> SwapTotal: 75505460 kB >>>> SwapFree: 65220676 kB >>>> Dirty: 444 kB >>>> Writeback: 0 kB >>>> AnonPages: 58704728 kB >>>> Mapped: 12036 kB >>>> Slab: 99806476 kB >>>> SReclaimable: 118532 kB >>>> SUnreclaim: 99687944 kB >>>> PageTables: 131200 kB >>>> >>>> NFS_Unstable: 0 kB >>>> Bounce: 0 kB >>>> WritebackTmp: 0 kB >>>> CommitLimit: 174551180 kB >>>> Committed_AS: 65739660 kB >>>> >>>> VmallocTotal: 34359738367 kB >>>> VmallocUsed: 588416 kB >>>> VmallocChunk: 34359149923 kB >>>> HugePages_Total: 0 >>>> HugePages_Free: 0 >>>> HugePages_Rsvd: 0 >>>> HugePages_Surp: 0 >>>> Hugepagesize: 2048 kB >>>> DirectMap4k: 8432 kB >>>> DirectMap2M: 201308160 kB >>>> >>>> # cat /proc/slabinfo >>>> slabinfo - version: 2.1 >>>> # name <active_objs> <num_objs> <objsize> <objperslab> >>>> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata >>>> <active_slabs> <num_slabs> <sharedavail> >>>> nfs_direct_cache 0 0 128 30 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> nfs_write_data 36 44 704 11 2 : tunables 54 27 >>>> 8 : slabdata 4 4 0 >>>> nfs_read_data 32 33 704 11 2 : tunables 54 27 >>>> 8 : slabdata 3 3 0 >>>> nfs_inode_cache 0 0 984 4 1 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> nfs_page 0 0 128 30 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> rpc_buffers 8 8 2048 2 1 : tunables 24 12 >>>> 8 : slabdata 4 4 0 >>>> rpc_tasks 8 12 320 12 1 : tunables 54 27 >>>> 8 : slabdata 1 1 0 >>>> rpc_inode_cache 0 0 832 4 1 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> ll_async_page 8494811 8507076 320 12 1 : tunables 54 >>>> 27 8 : slabdata 708923 708923 216 >>>> ll_file_data 16 40 192 20 1 : tunables 120 60 >>>> 8 : slabdata 2 2 0 >>>> lustre_inode_cache 95 184 896 4 1 : tunables 54 >>>> 27 8 : slabdata 46 46 0 >>>> lov_oinfo 56 180 320 12 1 : tunables 54 27 >>>> 8 : slabdata 15 15 0 >>>> >>>> osc_quota_info 0 0 32 112 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> ll_qunit_cache 0 0 112 34 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> llcd_cache 0 0 3952 1 1 : tunables 24 12 >>>> 8 : slabdata 0 0 0 >>>> ptlrpc_cbdatas 0 0 32 112 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> interval_node 1680 5730 128 30 1 : tunables 120 60 >>>> 8 : slabdata 191 191 0 >>>> ldlm_locks 2255 6232 512 8 1 : tunables 54 27 >>>> 8 : slabdata 779 779 0 >>>> ldlm_resources 2227 5570 384 10 1 : tunables 54 27 >>>> 8 : slabdata 557 557 0 >>>> >>>> ll_import_cache 0 0 1248 3 1 : tunables 24 12 >>>> 8 : slabdata 0 0 0 >>>> ll_obdo_cache 0 459630919 208 19 1 : tunables 120 >>>> 60 8 : slabdata 0 24191101 0 >>>> >>>> ll_obd_dev_cache 13 13 5672 1 2 : tunables 8 4 >>>> 0 : slabdata 13 13 0 >>>> obd_lvfs_ctxt_cache 0 0 96 40 1 : tunables 120 >>>> 60 8 : slabdata 0 0 0 >>>> SDP 0 0 1728 4 2 : tunables 24 12 >>>> 8 : slabdata 0 0 0 >>>> fib6_nodes 7 59 64 59 1 : tunables 120 >>>> 60 8 : slabdata 1 1 0 >>>> ip6_dst_cache 10 24 320 12 1 : tunables 54 27 >>>> 8 : slabdata 2 2 0 >>>> >>>> ndisc_cache 3 30 256 15 1 : tunables 120 60 >>>> 8 : slabdata 2 2 0 >>>> RAWv6 35 36 960 4 1 : tunables 54 27 >>>> 8 : slabdata 9 9 0 >>>> UDPLITEv6 0 0 960 4 1 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> UDPv6 7 12 960 4 1 : tunables 54 27 >>>> 8 : slabdata 3 3 0 >>>> tw_sock_TCPv6 0 0 192 20 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> request_sock_TCPv6 0 0 192 20 1 : tunables 120 >>>> 60 8 : slabdata 0 0 0 >>>> TCPv6 3 4 1792 2 1 : tunables 24 12 >>>> 8 : slabdata 2 2 0 >>>> ib_mad 2051 2096 448 8 1 : tunables 54 27 >>>> 8 : slabdata 262 262 0 >>>> >>>> fuse_request 0 0 608 6 1 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> fuse_inode 0 0 704 11 2 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> kcopyd_job 0 0 360 11 1 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> dm_uevent 0 0 2608 3 2 : tunables 24 12 >>>> 8 : slabdata 0 0 0 >>>> dm_clone_bio_info 0 0 16 202 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> dm_rq_target_io 0 0 408 9 1 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> dm_target_io 0 0 24 144 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> dm_io 0 0 32 112 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> uhci_urb_priv 1 67 56 67 1 : tunables 120 60 >>>> 8 : slabdata 1 1 0 >>>> ext3_inode_cache 2472 2610 768 5 1 : tunables 54 27 >>>> 8 : slabdata 522 522 0 >>>> >>>> ext3_xattr 0 0 88 44 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> journal_handle 56 288 24 144 1 : tunables 120 60 >>>> 8 : slabdata 2 2 0 >>>> journal_head 216 240 96 40 1 : tunables 120 60 >>>> 8 : slabdata 6 6 0 >>>> >>>> revoke_table 4 202 16 202 1 : tunables 120 60 >>>> 8 : slabdata 1 1 0 >>>> revoke_record 0 0 32 112 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> sgpool-128 2 2 4096 1 1 : tunables 24 12 >>>> 8 : slabdata 2 2 0 >>>> sgpool-64 2 2 2048 2 1 : tunables 24 12 >>>> 8 : slabdata 1 1 0 >>>> sgpool-32 2 4 1024 4 1 : tunables 54 27 >>>> 8 : slabdata 1 1 0 >>>> sgpool-16 2 8 512 8 1 : tunables 54 27 >>>> 8 : slabdata 1 1 0 >>>> sgpool-8 2 15 256 15 1 : tunables 120 60 >>>> 8 : slabdata 1 1 0 >>>> scsi_data_buffer 0 0 24 144 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> scsi_io_context 0 0 112 34 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> flow_cache 0 0 96 40 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> cfq_io_context 58 207 168 23 1 : tunables 120 60 >>>> 8 : slabdata 9 9 0 >>>> cfq_queue 56 308 136 28 1 : tunables 120 60 >>>> 8 : slabdata 11 11 0 >>>> >>>> bsg_cmd 0 0 312 12 1 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> mqueue_inode_cache 1 4 896 4 1 : tunables 54 >>>> 27 8 : slabdata 1 1 0 >>>> isofs_inode_cache 0 0 608 6 1 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> minix_inode_cache 0 0 624 6 1 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> hugetlbfs_inode_cache 1 7 576 7 1 : tunables 54 >>>> 27 8 : slabdata 1 1 0 >>>> dnotify_cache 0 0 40 92 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> dquot 0 0 256 15 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> inotify_event_cache 0 0 40 92 1 : tunables 120 >>>> 60 8 : slabdata 0 0 0 >>>> inotify_watch_cache 94 159 72 53 1 : tunables 120 >>>> 60 8 : slabdata 3 3 0 >>>> >>>> kioctx 0 0 384 10 1 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> kiocb 0 0 256 15 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> fasync_cache 0 0 24 144 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> shmem_inode_cache 878 1040 784 5 1 : tunables 54 27 >>>> 8 : slabdata 208 208 0 >>>> >>>> pid_namespace 0 0 2112 3 2 : tunables 24 12 >>>> 8 : slabdata 0 0 0 >>>> nsproxy 0 0 56 67 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> posix_timers_cache 0 0 192 20 1 : tunables 120 >>>> 60 8 : slabdata 0 0 0 >>>> uid_cache 7 60 128 30 1 : tunables 120 60 >>>> 8 : slabdata 2 2 0 >>>> UNIX 128 220 704 11 2 : tunables 54 27 >>>> 8 : slabdata 20 20 0 >>>> >>>> ip_mrt_cache 0 0 128 30 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> UDP-Lite 0 0 832 9 2 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> tcp_bind_bucket 15 118 64 59 1 : tunables 120 60 >>>> 8 : slabdata 2 2 0 >>>> >>>> inet_peer_cache 1 59 64 59 1 : tunables 120 60 >>>> 8 : slabdata 1 1 0 >>>> secpath_cache 0 0 64 59 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> xfrm_dst_cache 0 0 384 10 1 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> ip_fib_alias 0 0 32 112 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> ip_fib_hash 15 106 72 53 1 : tunables 120 60 >>>> 8 : slabdata 2 2 0 >>>> ip_dst_cache 40 84 320 12 1 : tunables 54 27 >>>> 8 : slabdata 7 7 0 >>>> >>>> arp_cache 8 15 256 15 1 : tunables 120 60 >>>> 8 : slabdata 1 1 0 >>>> RAW 33 35 768 5 1 : tunables 54 27 >>>> 8 : slabdata 7 7 0 >>>> UDP 11 36 832 9 2 : tunables 54 27 >>>> 8 : slabdata 4 4 0 >>>> tw_sock_TCP 4 20 192 20 1 : tunables 120 60 >>>> 8 : slabdata 1 1 0 >>>> >>>> request_sock_TCP 0 0 128 30 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> TCP 16 24 1664 4 2 : tunables 24 12 >>>> 8 : slabdata 6 6 0 >>>> eventpoll_pwq 69 159 72 53 1 : tunables 120 60 >>>> 8 : slabdata 3 3 0 >>>> eventpoll_epi 69 150 128 30 1 : tunables 120 60 >>>> 8 : slabdata 5 5 0 >>>> >>>> pfm_event_set 0 0 57344 1 16 : tunables 8 4 >>>> 0 : slabdata 0 0 0 >>>> pfm_context 0 0 8192 1 2 : tunables 8 4 >>>> 0 : slabdata 0 0 0 >>>> blkdev_integrity 0 0 112 34 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> blkdev_queue 10 12 2264 3 2 : tunables 24 12 >>>> 8 : slabdata 4 4 0 >>>> blkdev_requests 91 130 368 10 1 : tunables 54 27 >>>> 8 : slabdata 13 13 27 >>>> blkdev_ioc 56 371 72 53 1 : tunables 120 60 >>>> 8 : slabdata 7 7 0 >>>> >>>> biovec-256 2 2 4096 1 1 : tunables 24 12 >>>> 8 : slabdata 2 2 0 >>>> biovec-128 2 4 2048 2 1 : tunables 24 12 >>>> 8 : slabdata 2 2 0 >>>> biovec-64 2 8 1024 4 1 : tunables 54 27 >>>> 8 : slabdata 2 2 0 >>>> biovec-16 2 30 256 15 1 : tunables 120 60 >>>> 8 : slabdata 2 2 0 >>>> biovec-4 2 118 64 59 1 : tunables 120 60 >>>> 8 : slabdata 2 2 0 >>>> biovec-1 223 606 16 202 1 : tunables 120 60 >>>> 8 : slabdata 3 3 70 >>>> >>>> bio_integrity_payload 2 60 128 30 1 : tunables 120 >>>> 60 8 : slabdata 2 2 0 >>>> bio 205 330 128 30 1 : tunables 120 >>>> 60 8 : slabdata 11 11 70 >>>> sock_inode_cache 245 300 640 6 1 : tunables 54 27 >>>> 8 : slabdata 50 50 0 >>>> skbuff_fclone_cache 14 14 512 7 1 : tunables 54 >>>> 27 8 : slabdata 2 2 0 >>>> skbuff_head_cache 5121 5985 256 15 1 : tunables 120 60 >>>> 8 : slabdata 399 399 68 >>>> file_lock_cache 4 22 176 22 1 : tunables 120 60 >>>> 8 : slabdata 1 1 0 >>>> Acpi-Operand 889 1749 72 53 1 : tunables 120 60 >>>> 8 : slabdata 33 33 0 >>>> >>>> Acpi-ParseExt 0 0 72 53 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> Acpi-Parse 0 0 48 77 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> Acpi-State 0 0 80 48 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> Acpi-Namespace 617 672 32 112 1 : tunables 120 60 >>>> 8 : slabdata 6 6 0 >>>> task_delay_info 389 884 112 34 1 : tunables 120 60 >>>> 8 : slabdata 26 26 0 >>>> >>>> taskstats 0 0 328 12 1 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> page_cgroup 0 0 40 92 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> proc_inode_cache 1397 1446 608 6 1 : tunables 54 27 >>>> 8 : slabdata 240 241 190 >>>> sigqueue 29 96 160 24 1 : tunables 120 60 >>>> 8 : slabdata 4 4 1 >>>> radix_tree_node 193120 196672 552 7 1 : tunables 54 27 >>>> 8 : slabdata 28096 28096 216 >>>> bdev_cache 5 15 768 5 1 : tunables 54 27 >>>> 8 : slabdata 3 3 0 >>>> >>>> sysfs_dir_cache 19120 19296 80 48 1 : tunables 120 60 >>>> 8 : slabdata 402 402 0 >>>> mnt_cache 30 105 256 15 1 : tunables 120 60 >>>> 8 : slabdata 7 7 0 >>>> inode_cache 1128 1176 560 7 1 : tunables 54 27 >>>> 8 : slabdata 166 168 24 >>>> dentry 4651 8189 208 19 1 : tunables 120 60 >>>> 8 : slabdata 431 431 0 >>>> filp 1563 2720 192 20 1 : tunables 120 60 >>>> 8 : slabdata 136 136 242 >>>> names_cache 142 142 4096 1 1 : tunables 24 12 >>>> 8 : slabdata 142 142 96 >>>> >>>> key_jar 0 0 192 20 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> buffer_head 1129 3071 104 37 1 : tunables 120 60 >>>> 8 : slabdata 83 83 0 >>>> mm_struct 86 136 896 4 1 : tunables 54 27 >>>> 8 : slabdata 34 34 1 >>>> vm_area_struct 3406 4136 176 22 1 : tunables 120 60 >>>> 8 : slabdata 188 188 26 >>>> fs_cache 140 531 64 59 1 : tunables 120 60 >>>> 8 : slabdata 9 9 1 >>>> files_cache 83 150 768 5 1 : tunables 54 27 >>>> 8 : slabdata 30 30 1 >>>> signal_cache 325 388 960 4 1 : tunables 54 27 >>>> 8 : slabdata 97 97 0 >>>> sighand_cache 317 369 2112 3 2 : tunables 24 12 >>>> 8 : slabdata 123 123 0 >>>> task_xstate 155 256 512 8 1 : tunables 54 27 >>>> 8 : slabdata 32 32 2 >>>> task_struct 368 372 5872 1 2 : tunables 8 4 >>>> 0 : slabdata 368 372 0 >>>> anon_vma 966 1728 24 144 1 : tunables 120 60 >>>> 8 : slabdata 12 12 0 >>>> pid 377 960 128 30 1 : tunables 120 60 >>>> 8 : slabdata 32 32 0 >>>> >>>> shared_policy_node 0 0 48 77 1 : tunables 120 >>>> 60 8 : slabdata 0 0 0 >>>> numa_policy 15 112 136 28 1 : tunables 120 60 >>>> 8 : slabdata 4 4 0 >>>> idr_layer_cache 284 322 544 7 1 : tunables 54 27 >>>> 8 : slabdata 46 46 0 >>>> >>>> size-4194304(DMA) 0 0 4194304 1 1024 : tunables 1 >>>> 1 0 : slabdata 0 0 0 >>>> size-4194304 0 0 4194304 1 1024 : tunables 1 >>>> 1 0 : slabdata 0 0 0 >>>> size-2097152(DMA) 0 0 2097152 1 512 : tunables 1 >>>> 1 0 : slabdata 0 0 0 >>>> size-2097152 0 0 2097152 1 512 : tunables 1 >>>> 1 0 : slabdata 0 0 0 >>>> size-1048576(DMA) 0 0 1048576 1 256 : tunables 1 >>>> 1 0 : slabdata 0 0 0 >>>> size-1048576 0 0 1048576 1 256 : tunables 1 >>>> 1 0 : slabdata 0 0 0 >>>> size-524288(DMA) 0 0 524288 1 128 : tunables 1 1 >>>> 0 : slabdata 0 0 0 >>>> size-524288 0 0 524288 1 128 : tunables 1 1 >>>> 0 : slabdata 0 0 0 >>>> size-262144(DMA) 0 0 262144 1 64 : tunables 1 1 >>>> 0 : slabdata 0 0 0 >>>> size-262144 0 0 262144 1 64 : tunables 1 1 >>>> 0 : slabdata 0 0 0 >>>> size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 >>>> 0 : slabdata 0 0 0 >>>> size-131072 3 3 131072 1 32 : tunables 8 4 >>>> 0 : slabdata 3 3 0 >>>> size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 >>>> 0 : slabdata 0 0 0 >>>> size-65536 6 6 65536 1 16 : tunables 8 4 >>>> 0 : slabdata 6 6 0 >>>> size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 >>>> 0 : slabdata 0 0 0 >>>> size-32768 10 10 32768 1 8 : tunables 8 4 >>>> 0 : slabdata 10 10 0 >>>> >>>> size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 >>>> 0 : slabdata 0 0 0 >>>> size-16384 44 44 16384 1 4 : tunables 8 4 >>>> 0 : slabdata 44 44 0 >>>> >>>> size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 >>>> 0 : slabdata 0 0 0 >>>> size-8192 3611 3611 8192 1 2 : tunables 8 4 >>>> 0 : slabdata 3611 3611 0 >>>> >>>> size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 >>>> 8 : slabdata 0 0 0 >>>> size-4096 1771 1771 4096 1 1 : tunables 24 12 >>>> 8 : slabdata 1771 1771 0 >>>> >>>> size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 >>>> 8 : slabdata 0 0 0 >>>> size-2048 4609 4714 2048 2 1 : tunables 24 12 >>>> 8 : slabdata 2357 2357 0 >>>> >>>> size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> size-1024 4829 4900 1024 4 1 : tunables 54 27 >>>> 8 : slabdata 1225 1225 0 >>>> >>>> size-512(DMA) 0 0 512 8 1 : tunables 54 27 >>>> 8 : slabdata 0 0 0 >>>> size-512 1478 1520 512 8 1 : tunables 54 27 >>>> 8 : slabdata 190 190 39 >>>> >>>> size-256(DMA) 0 0 256 15 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> size-256 4662 5550 256 15 1 : tunables 120 60 >>>> 8 : slabdata 370 370 1 >>>> >>>> size-128(DMA) 0 0 128 30 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> size-64(DMA) 0 0 64 59 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> size-64 17232 29382 64 59 1 : tunables 120 60 >>>> 8 : slabdata 498 498 0 >>>> >>>> size-32(DMA) 0 0 32 112 1 : tunables 120 60 >>>> 8 : slabdata 0 0 0 >>>> size-128 9907 16140 128 30 1 : tunables 120 60 >>>> 8 : slabdata 538 538 0 >>>> size-32 12487 13104 32 112 1 : tunables 120 60 >>>> 8 : slabdata 117 117 0 >>>> >>>> kmem_cache 181 181 4224 1 2 : tunables 8 4 >>>> 0 : slabdata 181 181 0 >>>> >>>> >>>> Tasks: 278 total, 1 running, 276 sleeping, 0 stopped, 1 zombie >>>> Cpu(s): 3.8%us, 0.1%sy, 0.0%ni, 96.0%id, 0.0%wa, 0.0%hi, 0.0%si, >>>> 0.0%st >>>> Mem: 198091444k total, 197636988k used, 454456k free, 4544k >>>> buffers >>>> Swap: 75505460k total, 8567448k used, 66938012k free, 29144008k cached >>>> >>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>> COMMAND >>>> >>>> 107 root 15 -5 0 0 0 D 10 0.0 5:06.43 >>>> kswapd1 >>>> >>>> 19328 user1 20 0 66.5g 60g 2268 D 4 32.0 31:48.49 >>>> R >>>> >>>> 1 root 20 0 1064 64 32 S 0 0.0 0:21.20 >>>> init >>>> >>>> 2 root 15 -5 0 0 0 S 0 0.0 0:00.06 >>>> kthreadd >>>> >>>> 3 root RT -5 0 0 0 S 0 0.0 0:00.24 >>>> migration/0 >>>> >>>> 4 root 15 -5 0 0 0 S 0 0.0 1:01.12 >>>> ksoftirqd/0 >>>> >>>> 5 root RT -5 0 0 0 S 0 0.0 0:00.30 >>>> migration/1 >>>> >>>> 6 root 15 -5 0 0 0 S 0 0.0 0:00.50 >>>> ksoftirqd/1 >>>> >>>> 7 root RT -5 0 0 0 S 0 0.0 0:00.22 >>>> migration/2 >>>> >>>> 8 root 15 -5 0 0 0 S 0 0.0 0:00.36 >>>> ksoftirqd/2 >>>> >>>> 9 root RT -5 0 0 0 S 0 0.0 0:00.28 >>>> migration/3 >>>> >>>> 10 root 15 -5 0 0 0 S 0 0.0 0:00.60 >>>> ksoftirqd/3 >>>> >>>> 11 root RT -5 0 0 0 S 0 0.0 0:00.18 >>>> migration/4 >>>> >>>> 12 root 15 -5 0 0 0 S 0 0.0 0:00.40 >>>> ksoftirqd/4 >>>> >>>> 13 root RT -5 0 0 0 S 0 0.0 0:00.26 >>>> migration/5 >>>> >>>> 14 root 15 -5 0 0 0 S 0 0.0 0:00.76 >>>> ksoftirqd/5 >>>> >>>> 15 root RT -5 0 0 0 S 0 0.0 0:00.20 >>>> migration/6 >>>> >>>> 16 root 15 -5 0 0 0 S 0 0.0 0:00.36 >>>> ksoftirqd/6 >>>> >>>> 17 root RT -5 0 0 0 S 0 0.0 0:00.26 >>>> migration/7 >>>> >>>> 18 root 15 -5 0 0 0 S 0 0.0 0:00.68 >>>> ksoftirqd/7 >>>> >>>> 19 root RT -5 0 0 0 S 0 0.0 0:00.88 >>>> migration/8 >>>> >>>> 20 root 15 -5 0 0 0 S 0 0.0 0:07.70 >>>> ksoftirqd/8 >>>> >>>> 21 root RT -5 0 0 0 S 0 0.0 0:01.12 >>>> migration/9 >>>> >>>> 22 root 15 -5 0 0 0 S 0 0.0 0:01.20 >>>> ksoftirqd/9 >>>> >>>> 23 root RT -5 0 0 0 S 0 0.0 0:03.50 >>>> migration/10 >>>> >>>> 24 root 15 -5 0 0 0 S 0 0.0 0:01.22 >>>> ksoftirqd/10 >>>> >>>> 25 root RT -5 0 0 0 S 0 0.0 0:04.84 >>>> migration/11 >>>> >>>> 26 root 15 -5 0 0 0 S 0 0.0 0:01.90 >>>> ksoftirqd/11 >>>> >>>> 27 root RT -5 0 0 0 S 0 0.0 0:01.46 >>>> migration/12 >>>> >>>> 28 root 15 -5 0 0 0 S 0 0.0 0:01.42 >>>> ksoftirqd/12 >>>> >>>> 29 root RT -5 0 0 0 S 0 0.0 0:01.62 >>>> migration/13 >>>> >>>> 30 root 15 -5 0 0 0 S 0 0.0 0:01.84 >>>> ksoftirqd/13 >>>> >>>> 31 root RT -5 0 0 0 S 0 0.0 0:01.90 >>>> migration/14 >>>> >>>> 32 root 15 -5 0 0 0 S 0 0.0 0:01.18 >>>> ksoftirqd/14 >>>> -- >>>> >>>> Thanks, >>>> -J >>>> >>>> On Mon, Apr 19, 2010 at 10:07 AM, Andreas Dilger < >>>> andreas.dilger at oracle.com> wrote: >>>> >>>>> There is a known problem with the DLM LRU size that may be affecting >>>>> you. It may be something else too. Please check /proc/{slabinfo,meminfo} to >>>>> see what is using the memory on the client. >>>>> >>>>> Cheers, Andreas >>>>> >>>>> >>>>> On 2010-04-19, at 10:43, Jagga Soorma <jagga13 at gmail.com> wrote: >>>>> >>>>> Hi Guys, >>>>>> >>>>>> My users are reporting some issues with memory on our lustre 1.8.1 >>>>>> clients. It looks like when they submit a single job at a time the run time >>>>>> was about 4.5 minutes. However, when they ran multiple jobs (10 or less) on >>>>>> a client with 192GB of memory on a single node the run time for each job was >>>>>> exceeding 3-4X the run time for the single process. They also noticed that >>>>>> the swap space kept climbing even though there was plenty of free memory on >>>>>> the system. Could this possibly be related to the lustre client? Does it >>>>>> reserve any memory that is not accessible by any other process even though >>>>>> it might not be in use? >>>>>> >>>>>> Thanks much, >>>>>> -J >>>>>> _______________________________________________ >>>>>> Lustre-discuss mailing list >>>>>> Lustre-discuss at lists.lustre.org >>>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>>>>> >>>>> >>>> >>> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100419/37123355/attachment-0001.html
On 2010-04-19, at 11:16, Jagga Soorma wrote:> What is the known problem with the DLM LRU size?It is mostly a problem on the server, actually.> Here is what my slabinfo/meminfo look like on one of the clients. > I don''t see anything out of the ordinary: > > (then again there are no jobs currently running on this system) > > slabinfo - version: 2.1 > # name <active_objs> <num_objs> <objsize> <objperslab> > <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : > slabdata <active_slabs> <num_slabs> <sharedavail>> ll_async_page 326589 328572 320 12 1 : tunables 54 > 27 8 : slabdata 27381 27381 0This shows you have 326589 pages in the lustre filesystem cache, or about 1275MB of data. That shouldn''t be too much for a system with 192GB of RAM...> lustre_inode_cache 769 772 896 4 1 : tunables 54 > 27 8 : slabdata 193 193 0 > ldlm_locks 2624 3688 512 8 1 : tunables 54 > 27 8 : slabdata 461 461 0 > ldlm_resources 2002 3340 384 10 1 : tunables 54 > 27 8 : slabdata 334 334 0Only about 2600 locks on 770 files is fine (this is what the DLM LRU size would affect, if it were out of control, which it isn''t).> ll_obdo_cache 0 452282156 208 19 1 : tunables > 120 60 8 : slabdata 0 23804324 0This is really out of whack. The "obdo" struct should normally only be allocated for a short time and then freed again, but here you have 452M of them using over 90GB of RAM. It looks like a leak of some kind, which is a bit surprising since we have fairly tight checking for memory leaks in the Lustre code. Are you running some unusual workload that is maybe walking an unusual code path? What you can do to track down memory leaks is enable Lustre memory tracing, increase the size of the debug buffer to catch enough tracing to be useful, and then run your job to see what is causing the leak, dump the kernel debug log, and then run leak- finder.pl (attached, and also in Lustre sources): client# lctl set_param debug=+malloc client# lctl set_param debug_mb=256 client$ {run job} client# sync client# lctl dk /tmp/debug client# perl leak-finder.pl < /tmp/debug 2>&1 | grep "Leak.*oa" client# lctl set_param debug=-malloc client# lctl set_param debug_mb=32 Since this is a running system, it will report spurious leaks for some kinds of allocations that remain in memory for some time (e.g. cached pages, inodes, etc), but with the exception of uncommitted RPCs (of which there should be none after the sync) there should not be any leaked obdo.> On 2010-04-19, at 10:43, Jagga Soorma <jagga13 at gmail.com> wrote: >> My users are reporting some issues with memory on our lustre 1.8.1 >> clients. It looks like when they submit a single job at a time the >> run time was about 4.5 minutes. However, when they ran multiple >> jobs (10 or less) on a client with 192GB of memory on a single node >> the run time for each job was exceeding 3-4X the run time for the >> single process. They also noticed that the swap space kept >> climbing even though there was plenty of free memory on the >> system. Could this possibly be related to the lustre client? Does >> it reserve any memory that is not accessible by any other process >> even though it might not be in use? >Cheers, Andreas -- Andreas Dilger Principal Engineer, Lustre Group Oracle Corporation Canada Inc. -------------- next part -------------- A non-text attachment was scrubbed... Name: leak_finder.pl Type: text/x-perl-script Size: 2809 bytes Desc: not available Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100419/0682f3f8/attachment.bin
Hi Andreas, Thanks for your response. I will try to run the leak-finder script and hopefully it will point us in the right direction. This only seems to be happening on some of my clients: -- client112: ll_obdo_cache 0 0 208 19 1 : tunables 120 60 8 : slabdata 0 0 0 client108: ll_obdo_cache 0 0 208 19 1 : tunables 120 60 8 : slabdata 0 0 0 client110: ll_obdo_cache 0 0 208 19 1 : tunables 120 60 8 : slabdata 0 0 0 client107: ll_obdo_cache 0 0 208 19 1 : tunables 120 60 8 : slabdata 0 0 0 client111: ll_obdo_cache 0 0 208 19 1 : tunables 120 60 8 : slabdata 0 0 0 client109: ll_obdo_cache 0 0 208 19 1 : tunables 120 60 8 : slabdata 0 0 0 client102: ll_obdo_cache 5 38 208 19 1 : tunables 120 60 8 : slabdata 2 2 1 client114: ll_obdo_cache 0 0 208 19 1 : tunables 120 60 8 : slabdata 0 0 0 client105: ll_obdo_cache 0 0 208 19 1 : tunables 120 60 8 : slabdata 0 0 0 client103: ll_obdo_cache 0 0 208 19 1 : tunables 120 60 8 : slabdata 0 0 0 client104: ll_obdo_cache 0 433506280 208 19 1 : tunables 120 60 8 : slabdata 0 22816120 0 client116: ll_obdo_cache 0 457366746 208 19 1 : tunables 120 60 8 : slabdata 0 24071934 0 client113: ll_obdo_cache 0 456778867 208 19 1 : tunables 120 60 8 : slabdata 0 24040993 0 client106: ll_obdo_cache 0 456372267 208 19 1 : tunables 120 60 8 : slabdata 0 24019593 0 client115: ll_obdo_cache 0 449929310 208 19 1 : tunables 120 60 8 : slabdata 0 23680490 0 client101: ll_obdo_cache 0 454318101 208 19 1 : tunables 120 60 8 : slabdata 0 23911479 0 -- Hopefully this should help. Not sure which application might be causing the leaks. Currently R is the only app that users seem to be using heavily on these clients. Will let you know what I find. Thanks again, -J On Mon, Apr 19, 2010 at 9:04 PM, Andreas Dilger <andreas.dilger at oracle.com>wrote:> On 2010-04-19, at 11:16, Jagga Soorma wrote: > >> What is the known problem with the DLM LRU size? >> > > It is mostly a problem on the server, actually. > > Here is what my slabinfo/meminfo look like on one of the clients. I >> don''t see anything out of the ordinary: >> >> (then again there are no jobs currently running on this system) >> >> slabinfo - version: 2.1 >> # name <active_objs> <num_objs> <objsize> <objperslab> >> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata >> <active_slabs> <num_slabs> <sharedavail> >> > > ll_async_page 326589 328572 320 12 1 : tunables 54 27 8 >> : slabdata 27381 27381 0 >> > > This shows you have 326589 pages in the lustre filesystem cache, or about > 1275MB of data. That shouldn''t be too much for a system with 192GB of > RAM... > > lustre_inode_cache 769 772 896 4 1 : tunables 54 27 >> 8 : slabdata 193 193 0 >> ldlm_locks 2624 3688 512 8 1 : tunables 54 27 8 >> : slabdata 461 461 0 >> ldlm_resources 2002 3340 384 10 1 : tunables 54 27 8 >> : slabdata 334 334 0 >> > > Only about 2600 locks on 770 files is fine (this is what the DLM LRU size > would affect, if it were out of control, which it isn''t). > > ll_obdo_cache 0 452282156 208 19 1 : tunables 120 60 >> 8 : slabdata 0 23804324 0 >> > > This is really out of whack. The "obdo" struct should normally only be > allocated for a short time and then freed again, but here you have 452M of > them using over 90GB of RAM. It looks like a leak of some kind, which is a > bit surprising since we have fairly tight checking for memory leaks in the > Lustre code. > > Are you running some unusual workload that is maybe walking an unusual code > path? What you can do to track down memory leaks is enable Lustre memory > tracing, increase the size of the debug buffer to catch enough tracing to be > useful, and then run your job to see what is causing the leak, dump the > kernel debug log, and then run leak-finder.pl (attached, and also in > Lustre sources): > > client# lctl set_param debug=+malloc > client# lctl set_param debug_mb=256 > client$ {run job} > client# sync > client# lctl dk /tmp/debug > client# perl leak-finder.pl < /tmp/debug 2>&1 | grep "Leak.*oa" > client# lctl set_param debug=-malloc > client# lctl set_param debug_mb=32 > > Since this is a running system, it will report spurious leaks for some > kinds of allocations that remain in memory for some time (e.g. cached pages, > inodes, etc), but with the exception of uncommitted RPCs (of which there > should be none after the sync) there should not be any leaked obdo. > > On 2010-04-19, at 10:43, Jagga Soorma <jagga13 at gmail.com> wrote: >> >>> My users are reporting some issues with memory on our lustre 1.8.1 >>> clients. It looks like when they submit a single job at a time the run time >>> was about 4.5 minutes. However, when they ran multiple jobs (10 or less) on >>> a client with 192GB of memory on a single node the run time for each job was >>> exceeding 3-4X the run time for the single process. They also noticed that >>> the swap space kept climbing even though there was plenty of free memory on >>> the system. Could this possibly be related to the lustre client? Does it >>> reserve any memory that is not accessible by any other process even though >>> it might not be in use? >>> >> >> > Cheers, Andreas > -- > Andreas Dilger > Principal Engineer, Lustre Group > Oracle Corporation Canada Inc. >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100420/4ca7afc6/attachment-0001.html
Hi
Here is leak from our system, not sure if this is the same issue. Application is
molpro which is causing this leak. (pristine lustre 1.8.2)
???????????? total?????? used?????? free???? shared??? buffers???? cached
Mem:????? 32962040?? 32869120????? 92920????????? 0??????? 632??? 4201332
-/+ buffers/cache:?? 28667156??? 4294884
Swap:????? 1020116??? 1020116????????? 0
# perl leak-finder.pl < /tmp/debug 2>&1 | grep "Leak.*oa"
*** Leak: ((oa))=208 bytes allocated at ffff8106c7e39be0
(osc_request.c:osc_build_req:2278, debug file line 2834)
*** Leak: ((oa))=208 bytes allocated at ffff8102bbd96220
(osc_request.c:osc_build_req:2278, debug file line 7844)
*** Leak: ((oa))=208 bytes allocated at ffff81039196e970
(osc_request.c:osc_build_req:2278, debug file line 12308)
*** Leak: ((oa))=208 bytes allocated at ffff8103fc5833c0
(osc_request.c:osc_build_req:2278, debug file line 14137)
*** Leak: ((oa))=208 bytes allocated at ffff8103fc583560
(osc_request.c:osc_build_req:2278, debug file line 18273)
*** Leak: ((oa))=208 bytes allocated at ffff810196049080
(osc_request.c:osc_build_req:2278, debug file line 22003)
*** Leak: ((oa))=208 bytes allocated at ffff8103fc583080
(osc_request.c:osc_build_req:2278, debug file line 50346)
*** Leak: ((oa))=208 bytes allocated at ffff8102bbd967d0
(osc_request.c:osc_build_req:2278, debug file line 52119)
*** Leak: ((oa))=208 bytes allocated at ffff8103fc583220
(osc_request.c:osc_build_req:2278, debug file line 53941)
*** Leak: ((oa))=208 bytes allocated at ffff8102fe6b8700
(osc_request.c:osc_build_req:2278, debug file line 56284)
*** Leak: ((oa))=208 bytes allocated at ffff8106c7e39080
(osc_request.c:osc_build_req:2278, debug file line 62691)
*** Leak: ((oa))=208 bytes allocated at ffff8106bad438a0
(osc_request.c:osc_build_req:2278, debug file line 64219)
*** Leak: ((oa))=208 bytes allocated at ffff8106bad432f0
(osc_request.c:osc_build_req:2278, debug file line 64555)
*** Leak: ((oa))=208 bytes allocated at ffff8106c7e39630
(osc_request.c:osc_build_req:2278, debug file line 66742)
*** Leak: ((oa))=208 bytes allocated at ffff8102bbd968a0
(osc_request.c:osc_build_req:2278, debug file line 67819)
*** Leak: ((oa))=208 bytes allocated at ffff8102fe6b8e50
(osc_request.c:osc_build_req:2278, debug file line 79362)
*** Leak: ((oa))=208 bytes allocated at ffff8106c7e39cb0
(osc_request.c:osc_build_req:2278, debug file line 80852)
*** Leak: ((oa))=208 bytes allocated at ffff8106bad437d0
(osc_request.c:osc_build_req:2278, debug file line 81220)
*** Leak: ((oa))=208 bytes allocated at ffff8106bad43490
(osc_request.c:osc_build_req:2278, debug file line 81228)
*** Leak: ((oa))=208 bytes allocated at ffff8106c7e39d80
(osc_request.c:osc_build_req:2278, debug file line 82818)
*** Leak: ((oa))=208 bytes allocated at ffff810196049be0
(osc_request.c:osc_build_req:2278, debug file line 83205)
*** Leak: ((oa))=208 bytes allocated at ffff8106c7e39220
(osc_request.c:osc_build_req:2278, debug file line 84102)
*** Leak: ((oa))=208 bytes allocated at ffff8103fc583700
(osc_request.c:osc_build_req:2278, debug file line 86491)
*** Leak: ((oa))=208 bytes allocated at ffff8106c7e39490
(osc_request.c:osc_build_req:2278, debug file line 87882)
*** Leak: ((oa))=208 bytes allocated at ffff81034fbd1e50
(osc_request.c:osc_build_req:2278, debug file line 88296)
*** Leak: ((oa))=208 bytes allocated at ffff8101960498a0
(osc_request.c:osc_build_req:2278, debug file line 97721)
*** Leak: ((oa))=208 bytes allocated at ffff8103fc583630
(osc_request.c:osc_build_req:2278, debug file line 112202)
*** Leak: ((oa))=208 bytes allocated at ffff8102bbd96be0
(osc_request.c:osc_build_req:2278, debug file line 131547)
*** Leak: ((oa))=208 bytes allocated at ffff8102fe6b8cb0
(osc_request.c:osc_build_req:2278, debug file line 132286)
*** Leak: ((oa))=208 bytes allocated at ffff81034fbd1a40
(osc_request.c:osc_build_req:2278, debug file line 136195)
*** Leak: ((oa))=208 bytes allocated at ffff81034fbd13c0
(osc_request.c:osc_build_req:2278, debug file line 181450)
*** Leak: ((oa))=208 bytes allocated at ffff810630625970
(osc_request.c:osc_build_req:2278, debug file line 199146)
*** Leak: ((oa))=208 bytes allocated at ffff810630625be0
(osc_request.c:osc_build_req:2278, debug file line 199428)
*** Leak: ((oa))=208 bytes allocated at ffff810630625080
(osc_request.c:osc_build_req:2278, debug file line 200627)
*** Leak: ((oa))=208 bytes allocated at ffff81034fbd17d0
(osc_request.c:osc_build_req:2278, debug file line 206924)
*** Leak: ((oa))=208 bytes allocated at ffff81034fbd1150
(osc_request.c:osc_build_req:2278, debug file line 207207)
*** Leak: ((oa))=208 bytes allocated at ffff810630625a40
(osc_request.c:osc_build_req:2278, debug file line 210587)
*** Leak: ((oa))=208 bytes allocated at ffff8107402c7630
(osc_request.c:osc_build_req:2278, debug file line 211246)
*** Leak: ((oa))=208 bytes allocated at ffff8107402c72f0
(osc_request.c:osc_build_req:2278, debug file line 211639)
*** Leak: ((oa))=208 bytes allocated at ffff8107402c78a0
(osc_request.c:osc_build_req:2278, debug file line 212032)
*** Leak: ((oa))=208 bytes allocated at ffff8107402c7970
(osc_request.c:osc_build_req:2278, debug file line 212426)
*** Leak: ((oa))=208 bytes allocated at ffff8107402c77d0
(osc_request.c:osc_build_req:2278, debug file line 212727)
*** Leak: ((oa))=208 bytes allocated at ffff8107402c7be0
(osc_request.c:osc_build_req:2278, debug file line 213087)
*** Leak: ((oa))=208 bytes allocated at ffff8107402c7080
(osc_request.c:osc_build_req:2278, debug file line 224138)
*** Leak: ((oa))=208 bytes allocated at ffff8107402c7e50
(osc_request.c:osc_build_req:2278, debug file line 257123)
*** Leak: ((oa))=208 bytes allocated at ffff8107402c7220
(osc_request.c:osc_build_req:2278, debug file line 257499)
*** Leak: ((oa))=208 bytes allocated at ffff8107402c7150
(osc_request.c:osc_build_req:2278, debug file line 257948)
*** Leak: ((oa))=208 bytes allocated at ffff8107461292f0
(osc_request.c:osc_build_req:2278, debug file line 290707)
*** Leak: ((oa))=208 bytes allocated at ffff8103fd64fa40
(osc_request.c:osc_build_req:2278, debug file line 293579)
*** Leak: ((oa))=208 bytes allocated at ffff810344bc4560
(osc_request.c:osc_build_req:2278, debug file line 299682)
*** Leak: ((oa))=208 bytes allocated at ffff810344bc48a0
(osc_request.c:osc_build_req:2278, debug file line 310552)
BR,
Tommi
PS. Thanks for the LUG 2010
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100426/fe3fd945/attachment.html
Hi, On Mon, Apr 26, 2010 at 01:16:37AM -0700, Tommi T wrote:> Here is leak from our system, not sure if this is the same issue. > Application is molpro which is causing this leak. (pristine lustre 1.8.2) > > total used free shared buffers cached > Mem: 32962040 32869120 92920 0 632 4201332 > -/+ buffers/cache: 28667156 4294884 > Swap: 1020116 1020116 0 > > # perl leak-finder.pl < /tmp/debug 2>&1 | grep "Leak.*oa" > *** Leak: ((oa))=208 bytes allocated at ffff8106c7e39be0 > (osc_request.c:osc_build_req:2278, debug file line 2834)[..] interesting. could you please collect a debug log with a debug mask set to -1 and attach it to a bugzilla ticket? Thanks in advance. Cheers, Johann
Hi, On Tue, Apr 20, 2010 at 09:08:25AM -0700, Jagga Soorma wrote:> Thanks for your response.* I will try to run the leak-finder script and > hopefully it will point us in the right direction.* This only seems to be > happening on some of my clients:Could you please tell us what kernel you use on the client side?> client104: ll_obdo_cache********* 0 433506280*** 208** 19*** 1 : tunables* > 120** 60*** 8 : slabdata***** 0 22816120***** 0 > client116: ll_obdo_cache********* 0 457366746*** 208** 19*** 1 : tunables* > 120** 60*** 8 : slabdata***** 0 24071934***** 0 > client113: ll_obdo_cache********* 0 456778867*** 208** 19*** 1 : tunables* > 120** 60*** 8 : slabdata***** 0 24040993***** 0 > client106: ll_obdo_cache********* 0 456372267*** 208** 19*** 1 : tunables* > 120** 60*** 8 : slabdata***** 0 24019593***** 0 > client115: ll_obdo_cache********* 0 449929310*** 208** 19*** 1 : tunables* > 120** 60*** 8 : slabdata***** 0 23680490***** 0 > client101: ll_obdo_cache********* 0 454318101*** 208** 19*** 1 : tunables* > 120** 60*** 8 : slabdata***** 0 23911479***** 0 > -- > > Hopefully this should help.* Not sure which application might be causing > the leaks.* Currently R is the only app that users seem to be using > heavily on these clients.* Will let you know what I find.Tommi Tervo has filed a bugzilla ticket for this issue, see https://bugzilla.lustre.org/show_bug.cgi?id=22701 Could you please add a comment to this ticket to describe the behavior of the application "R" (fork many threads, write to many files, use direct i/o, ...)? Cheers, Johann
Hi Johann, I am actually using 1.8.1 and not 1.8.2: # rpm -qa | grep -i lustre lustre-client-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default lustre-client-modules-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default My kernel version on the SLES 11 clients is: # uname -r 2.6.27.29-0.1-default My kernel version on the RHEL 5.3 mds/oss servers is: # uname -r 2.6.18-128.7.1.el5_lustre.1.8.1.1 Please let me know if you need any further information. I am still trying to get the user to help me run his app so that I can run the leak finder script to capture more information. Regards, -Simran On Tue, Apr 27, 2010 at 7:20 AM, Johann Lombardi <johann at sun.com> wrote:> Hi, > > On Tue, Apr 20, 2010 at 09:08:25AM -0700, Jagga Soorma wrote: > > Thanks for your response.* I will try to run the leak-finder script and > > hopefully it will point us in the right direction.* This only seems to be > > happening on some of my clients: > > Could you please tell us what kernel you use on the client side? > > > client104: ll_obdo_cache********* 0 433506280*** 208** 19*** 1 : > tunables* > > 120** 60*** 8 : slabdata***** 0 22816120***** 0 > > client116: ll_obdo_cache********* 0 457366746*** 208** 19*** 1 : > tunables* > > 120** 60*** 8 : slabdata***** 0 24071934***** 0 > > client113: ll_obdo_cache********* 0 456778867*** 208** 19*** 1 : > tunables* > > 120** 60*** 8 : slabdata***** 0 24040993***** 0 > > client106: ll_obdo_cache********* 0 456372267*** 208** 19*** 1 : > tunables* > > 120** 60*** 8 : slabdata***** 0 24019593***** 0 > > client115: ll_obdo_cache********* 0 449929310*** 208** 19*** 1 : > tunables* > > 120** 60*** 8 : slabdata***** 0 23680490***** 0 > > client101: ll_obdo_cache********* 0 454318101*** 208** 19*** 1 : > tunables* > > 120** 60*** 8 : slabdata***** 0 23911479***** 0 > > -- > > > > Hopefully this should help.* Not sure which application might be > causing > > the leaks.* Currently R is the only app that users seem to be using > > heavily on these clients.* Will let you know what I find. > > Tommi Tervo has filed a bugzilla ticket for this issue, see > https://bugzilla.lustre.org/show_bug.cgi?id=22701 > > Could you please add a comment to this ticket to describe the > behavior of the application "R" (fork many threads, write to > many files, use direct i/o, ...)? > > Cheers, > Johann >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100428/4773244d/attachment.html
Hello Jagga,
I checked the data, and indeed this does not look like a lustre memory
leak, rather than a slab fragmentation, which assumes there might be a
kernel issue here. From the slabinfo (I only keep three first columns here):
name <active_objs> <num_objs>
ll_obdo_cache 0 452282156 208
means that there are no active objects, but the memory pages are not
released back from slab allocator to the free pool (the num value is
huge). That looks like a slab fragmentation - you can get more
description at
http://kerneltrap.org/Linux/Slab_Defragmentation
Checking your mails, I wonder if this only happens on clients which
have SLES11 installed? As the RAM size is around 192Gb, I assume they
are NUMA systems?
If so, SLES11 has defrag_ratio tunables in /sys/kernel/slab/xxx
From the source of get_any_partial()
#ifdef CONFIG_NUMA
/*
* The defrag ratio allows a configuration of the tradeoffs between
* inter node defragmentation and node local allocations. A lower
* defrag_ratio increases the tendency to do local allocations
* instead of attempting to obtain partial slabs from other nodes.
*
* If the defrag_ratio is set to 0 then kmalloc() always
* returns node local objects. If the ratio is higher then kmalloc()
* may return off node objects because partial slabs are obtained
* from other nodes and filled up.
*
* If /sys/kernel/slab/xx/defrag_ratio is set to 100 (which makes
* defrag_ratio = 1000) then every (well almost) allocation will
* first attempt to defrag slab caches on other nodes. This means
* scanning over all nodes to look for partial slabs which may be
* expensive if we do it every time we are trying to find a slab
* with available objects.
*/
Could you please verify that your clients have defrag_ratio tunable and
try to use various values?
It looks like the value of 100 should be the best, unless there is a
bug, then may be even 0 gets the desired result?
Best regards,
Dmitry
Jagga Soorma wrote:> Hi Johann,
>
> I am actually using 1.8.1 and not 1.8.2:
>
> # rpm -qa | grep -i lustre
> lustre-client-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default
> lustre-client-modules-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default
>
> My kernel version on the SLES 11 clients is:
> # uname -r
> 2.6.27.29-0.1-default
>
> My kernel version on the RHEL 5.3 mds/oss servers is:
> # uname -r
> 2.6.18-128.7.1.el5_lustre.1.8.1.1
>
> Please let me know if you need any further information. I am still
> trying to get the user to help me run his app so that I can run the
> leak finder script to capture more information.
>
> Regards,
> -Simran
>
> On Tue, Apr 27, 2010 at 7:20 AM, Johann Lombardi <johann at sun.com
> <mailto:johann at sun.com>> wrote:
>
> Hi,
>
> On Tue, Apr 20, 2010 at 09:08:25AM -0700, Jagga Soorma wrote:
> > Thanks for your response.* I will try to run the leak-finder
> script and
> > hopefully it will point us in the right direction.* This only
> seems to be
> > happening on some of my clients:
>
> Could you please tell us what kernel you use on the client side?
>
> > client104: ll_obdo_cache********* 0 433506280*** 208** 19***
> 1 : tunables*
> > 120** 60*** 8 : slabdata***** 0 22816120***** 0
> > client116: ll_obdo_cache********* 0 457366746*** 208** 19***
> 1 : tunables*
> > 120** 60*** 8 : slabdata***** 0 24071934***** 0
> > client113: ll_obdo_cache********* 0 456778867*** 208** 19***
> 1 : tunables*
> > 120** 60*** 8 : slabdata***** 0 24040993***** 0
> > client106: ll_obdo_cache********* 0 456372267*** 208** 19***
> 1 : tunables*
> > 120** 60*** 8 : slabdata***** 0 24019593***** 0
> > client115: ll_obdo_cache********* 0 449929310*** 208** 19***
> 1 : tunables*
> > 120** 60*** 8 : slabdata***** 0 23680490***** 0
> > client101: ll_obdo_cache********* 0 454318101*** 208** 19***
> 1 : tunables*
> > 120** 60*** 8 : slabdata***** 0 23911479***** 0
> > --
> >
> > Hopefully this should help.* Not sure which application might
> be causing
> > the leaks.* Currently R is the only app that users seem to be
> using
> > heavily on these clients.* Will let you know what I find.
>
> Tommi Tervo has filed a bugzilla ticket for this issue, see
> https://bugzilla.lustre.org/show_bug.cgi?id=22701
>
> Could you please add a comment to this ticket to describe the
> behavior of the application "R" (fork many threads, write to
> many files, use direct i/o, ...)?
>
> Cheers,
> Johann
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100519/2649f91a/attachment.html
Hi Dmitry, I am still running into this issue on some nodes: client109: ll_obdo_cache 0 152914489 208 19 1 : tunables 120 60 8 : slabdata 0 8048131 0 client102: ll_obdo_cache 0 308526883 208 19 1 : tunables 120 60 8 : slabdata 0 16238257 0 How can I calculate how much memory this is holding on to. My system shows a lot of memory that is being used up but none of the jobs are using that much memory. Also, these clients are running a smp sles 11 kernel but I can''t find any /sys/kernel/slab directory. Linux client102 2.6.27.29-0.1-default #1 SMP 2009-08-15 17:53:59 +0200 x86_64 x86_64 x86_64 GNU/Linux What makes you say that this does not look like a lustre memory leak? I thought all the ll_* objects in slabinfo are lustre related? To me it looks like lustre is holding on to this memory but I don''t know much about lustre internals. Also, memused on these systems are: client102: 2353666940 client109: 2421645924 Any help would be greatly appreciated. Thanks, -J On Wed, May 19, 2010 at 8:08 AM, Dmitry Zogin <dmitry.zoguine at oracle.com>wrote:> Hello Jagga, > > I checked the data, and indeed this does not look like a lustre memory > leak, rather than a slab fragmentation, which assumes there might be a > kernel issue here. From the slabinfo (I only keep three first columns here): > > > name <active_objs> <num_objs> > ll_obdo_cache 0 452282156 208 > > means that there are no active objects, but the memory pages are not > released back from slab allocator to the free pool (the num value is huge). > That looks like a slab fragmentation - you can get more description at > http://kerneltrap.org/Linux/Slab_Defragmentation > > Checking your mails, I wonder if this only happens on clients which have > SLES11 installed? As the RAM size is around 192Gb, I assume they are NUMA > systems? > If so, SLES11 has defrag_ratio tunables in /sys/kernel/slab/xxx > From the source of get_any_partial() > > #ifdef CONFIG_NUMA > > /* > * The defrag ratio allows a configuration of the tradeoffs between > * inter node defragmentation and node local allocations. A lower > * defrag_ratio increases the tendency to do local allocations > * instead of attempting to obtain partial slabs from other nodes. > * > * If the defrag_ratio is set to 0 then kmalloc() always > * returns node local objects. If the ratio is higher then > kmalloc() > * may return off node objects because partial slabs are obtained > * from other nodes and filled up. > * > * If /sys/kernel/slab/xx/defrag_ratio is set to 100 (which makes > * defrag_ratio = 1000) then every (well almost) allocation will > * first attempt to defrag slab caches on other nodes. This means > * scanning over all nodes to look for partial slabs which may be > * expensive if we do it every time we are trying to find a slab > * with available objects. > */ > > Could you please verify that your clients have defrag_ratio tunable and try > to use various values? > It looks like the value of 100 should be the best, unless there is a bug, > then may be even 0 gets the desired result? > > Best regards, > Dmitry > > > Jagga Soorma wrote: > > Hi Johann, > > I am actually using 1.8.1 and not 1.8.2: > > # rpm -qa | grep -i lustre > lustre-client-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default > lustre-client-modules-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default > > My kernel version on the SLES 11 clients is: > # uname -r > 2.6.27.29-0.1-default > > My kernel version on the RHEL 5.3 mds/oss servers is: > # uname -r > 2.6.18-128.7.1.el5_lustre.1.8.1.1 > > Please let me know if you need any further information. I am still trying > to get the user to help me run his app so that I can run the leak finder > script to capture more information. > > Regards, > -Simran > > On Tue, Apr 27, 2010 at 7:20 AM, Johann Lombardi <johann at sun.com> wrote: > >> Hi, >> >> On Tue, Apr 20, 2010 at 09:08:25AM -0700, Jagga Soorma wrote: >> > Thanks for your response.* I will try to run the leak-finder script >> and >> > hopefully it will point us in the right direction.* This only seems to >> be >> > happening on some of my clients: >> >> Could you please tell us what kernel you use on the client side? >> >> > client104: ll_obdo_cache********* 0 433506280*** 208** 19*** 1 : >> tunables* >> > 120** 60*** 8 : slabdata***** 0 22816120***** 0 >> > client116: ll_obdo_cache********* 0 457366746*** 208** 19*** 1 : >> tunables* >> > 120** 60*** 8 : slabdata***** 0 24071934***** 0 >> > client113: ll_obdo_cache********* 0 456778867*** 208** 19*** 1 : >> tunables* >> > 120** 60*** 8 : slabdata***** 0 24040993***** 0 >> > client106: ll_obdo_cache********* 0 456372267*** 208** 19*** 1 : >> tunables* >> > 120** 60*** 8 : slabdata***** 0 24019593***** 0 >> > client115: ll_obdo_cache********* 0 449929310*** 208** 19*** 1 : >> tunables* >> > 120** 60*** 8 : slabdata***** 0 23680490***** 0 >> > client101: ll_obdo_cache********* 0 454318101*** 208** 19*** 1 : >> tunables* >> > 120** 60*** 8 : slabdata***** 0 23911479***** 0 >> > -- >> > >> > Hopefully this should help.* Not sure which application might be >> causing >> > the leaks.* Currently R is the only app that users seem to be using >> > heavily on these clients.* Will let you know what I find. >> >> Tommi Tervo has filed a bugzilla ticket for this issue, see >> https://bugzilla.lustre.org/show_bug.cgi?id=22701 >> >> Could you please add a comment to this ticket to describe the >> behavior of the application "R" (fork many threads, write to >> many files, use direct i/o, ...)? >> >> Cheers, >> Johann >> > > ------------------------------ > > _______________________________________________ > Lustre-discuss mailing listLustre-discuss at lists.lustre.orghttp://lists.lustre.org/mailman/listinfo/lustre-discuss > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100826/f8abaf3f/attachment-0001.html
On 2010-08-26, at 18:42, Jagga Soorma wrote:> I am still running into this issue on some nodes: > > client109: ll_obdo_cache 0 152914489 208 19 1 : tunables 120 60 8 : slabdata 0 8048131 0 > client102: ll_obdo_cache 0 308526883 208 19 1 : tunables 120 60 8 : slabdata 0 16238257 0 > > How can I calculate how much memory this is holding on to.If you do "head -1 /proc/slabinfo" it reports the column descriptions. The "slabdata" will section reports numslabs=16238257, and pagesperslab=1, so tis is 16238257 pages of memory, or about 64GB of RAM on client102. Ouch.> My system shows a lot of memory that is being used up but none of the jobs are using that much memory. Also, these clients are running a smp sles 11 kernel but I can''t find any /sys/kernel/slab directory. > > Linux client102 2.6.27.29-0.1-default #1 SMP 2009-08-15 17:53:59 +0200 x86_64 x86_64 x86_64 GNU/Linux > > What makes you say that this does not look like a lustre memory leak? I thought all the ll_* objects in slabinfo are lustre related?It''s true that the ll_obdo_cache objects are allocated by Lustre, but the above data shows 0 of those objects in use, so the kernel _should_ be freeing the unused slab objects. This particular data type (obdo) is only ever in use temporarily during system calls on the client, and should never be allocated for a long time. For some reason the kernel is not freeing the empty slab pages. That is the responsibility of the kernel, and not Lustre.> To me it looks like lustre is holding on to this memory but I don''t know much about lustre internals. > > Also, memused on these systems are: > > client102: 2353666940 > client109: 2421645924This shows that Lustre is actively using about 2.4GB of memory allocations. It is not tracking the 64GB of memory in the obdo_cache slab, because it has freed that memory (even though the kernel has not freed those pages).> Any help would be greatly appreciated.The only suggestion I have is that if you unmount Lustre and unload the modules (lustre_rmmod) it will free up this memory. Otherwise, searching for problems with the slab cache on this kernel may turn up something.> On Wed, May 19, 2010 at 8:08 AM, Dmitry Zogin <dmitry.zoguine at oracle.com> wrote: > Hello Jagga, > > I checked the data, and indeed this does not look like a lustre memory leak, rather than a slab fragmentation, which assumes there might be a kernel issue here. From the slabinfo (I only keep three first columns here): > > > name <active_objs> <num_objs> > ll_obdo_cache 0 452282156 208 > > means that there are no active objects, but the memory pages are not released back from slab allocator to the free pool (the num value is huge). That looks like a slab fragmentation - you can get more description at > http://kerneltrap.org/Linux/Slab_Defragmentation > > Checking your mails, I wonder if this only happens on clients which have SLES11 installed? As the RAM size is around 192Gb, I assume they are NUMA systems? > If so, SLES11 has defrag_ratio tunables in /sys/kernel/slab/xxx > From the source of get_any_partial() > > #ifdef CONFIG_NUMA > > /* > * The defrag ratio allows a configuration of the tradeoffs between > * inter node defragmentation and node local allocations. A lower > * defrag_ratio increases the tendency to do local allocations > * instead of attempting to obtain partial slabs from other nodes. > * > * If the defrag_ratio is set to 0 then kmalloc() always > * returns node local objects. If the ratio is higher then kmalloc() > * may return off node objects because partial slabs are obtained > * from other nodes and filled up. > * > * If /sys/kernel/slab/xx/defrag_ratio is set to 100 (which makes > * defrag_ratio = 1000) then every (well almost) allocation will > * first attempt to defrag slab caches on other nodes. This means > * scanning over all nodes to look for partial slabs which may be > * expensive if we do it every time we are trying to find a slab > * with available objects. > */ > > Could you please verify that your clients have defrag_ratio tunable and try to use various values? > It looks like the value of 100 should be the best, unless there is a bug, then may be even 0 gets the desired result? > > Best regards, > Dmitry > > > Jagga Soorma wrote: >> Hi Johann, >> >> I am actually using 1.8.1 and not 1.8.2: >> >> # rpm -qa | grep -i lustre >> lustre-client-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default >> lustre-client-modules-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default >> >> My kernel version on the SLES 11 clients is: >> # uname -r >> 2.6.27.29-0.1-default >> >> My kernel version on the RHEL 5.3 mds/oss servers is: >> # uname -r >> 2.6.18-128.7.1.el5_lustre.1.8.1.1 >> >> Please let me know if you need any further information. I am still trying to get the user to help me run his app so that I can run the leak finder script to capture more information. >> >> Regards, >> -Simran >> >> On Tue, Apr 27, 2010 at 7:20 AM, Johann Lombardi <johann at sun.com> wrote: >> Hi, >> >> On Tue, Apr 20, 2010 at 09:08:25AM -0700, Jagga Soorma wrote: >> > Thanks for your response.* I will try to run the leak-finder script and >> > hopefully it will point us in the right direction.* This only seems to be >> > happening on some of my clients: >> >> Could you please tell us what kernel you use on the client side? >> >> > client104: ll_obdo_cache********* 0 433506280*** 208** 19*** 1 : tunables* >> > 120** 60*** 8 : slabdata***** 0 22816120***** 0 >> > client116: ll_obdo_cache********* 0 457366746*** 208** 19*** 1 : tunables* >> > 120** 60*** 8 : slabdata***** 0 24071934***** 0 >> > client113: ll_obdo_cache********* 0 456778867*** 208** 19*** 1 : tunables* >> > 120** 60*** 8 : slabdata***** 0 24040993***** 0 >> > client106: ll_obdo_cache********* 0 456372267*** 208** 19*** 1 : tunables* >> > 120** 60*** 8 : slabdata***** 0 24019593***** 0 >> > client115: ll_obdo_cache********* 0 449929310*** 208** 19*** 1 : tunables* >> > 120** 60*** 8 : slabdata***** 0 23680490***** 0 >> > client101: ll_obdo_cache********* 0 454318101*** 208** 19*** 1 : tunables* >> > 120** 60*** 8 : slabdata***** 0 23911479***** 0 >> > -- >> > >> > Hopefully this should help.* Not sure which application might be causing >> > the leaks.* Currently R is the only app that users seem to be using >> > heavily on these clients.* Will let you know what I find. >> >> Tommi Tervo has filed a bugzilla ticket for this issue, see >> https://bugzilla.lustre.org/show_bug.cgi?id=22701 >> >> Could you please add a comment to this ticket to describe the >> behavior of the application "R" (fork many threads, write to >> many files, use direct i/o, ...)? >> >> Cheers, >> Johann >> >> >> _______________________________________________ >> Lustre-discuss mailing list >> >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> >> > >Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.
Actually there was a bug fixed in 1.8.4 when obdo structures can be allocated and freed outside of OBDO_ALLOC/OBDO_FREE macros. That could lead to the slab fragmentation and pseudo-leak. The patch is in the attachment 30664 for bz 21980 Dmitry Andreas Dilger wrote:> On 2010-08-26, at 18:42, Jagga Soorma wrote: > >> I am still running into this issue on some nodes: >> >> client109: ll_obdo_cache 0 152914489 208 19 1 : tunables 120 60 8 : slabdata 0 8048131 0 >> client102: ll_obdo_cache 0 308526883 208 19 1 : tunables 120 60 8 : slabdata 0 16238257 0 >> >> How can I calculate how much memory this is holding on to. >> > > If you do "head -1 /proc/slabinfo" it reports the column descriptions. > > The "slabdata" will section reports numslabs=16238257, and pagesperslab=1, so tis is 16238257 pages of memory, or about 64GB of RAM on client102. Ouch. > > >> My system shows a lot of memory that is being used up but none of the jobs are using that much memory. Also, these clients are running a smp sles 11 kernel but I can''t find any /sys/kernel/slab directory. >> >> Linux client102 2.6.27.29-0.1-default #1 SMP 2009-08-15 17:53:59 +0200 x86_64 x86_64 x86_64 GNU/Linux >> >> What makes you say that this does not look like a lustre memory leak? I thought all the ll_* objects in slabinfo are lustre related? >> > > It''s true that the ll_obdo_cache objects are allocated by Lustre, but the above data shows 0 of those objects in use, so the kernel _should_ be freeing the unused slab objects. This particular data type (obdo) is only ever in use temporarily during system calls on the client, and should never be allocated for a long time. > > For some reason the kernel is not freeing the empty slab pages. That is the responsibility of the kernel, and not Lustre. > > >> To me it looks like lustre is holding on to this memory but I don''t know much about lustre internals. >> >> Also, memused on these systems are: >> >> client102: 2353666940 >> client109: 2421645924 >> > > This shows that Lustre is actively using about 2.4GB of memory allocations. It is not tracking the 64GB of memory in the obdo_cache slab, because it has freed that memory (even though the kernel has not freed those pages). > > >> Any help would be greatly appreciated. >> > > The only suggestion I have is that if you unmount Lustre and unload the modules (lustre_rmmod) it will free up this memory. Otherwise, searching for problems with the slab cache on this kernel may turn up something. > > >> On Wed, May 19, 2010 at 8:08 AM, Dmitry Zogin <dmitry.zoguine at oracle.com> wrote: >> Hello Jagga, >> >> I checked the data, and indeed this does not look like a lustre memory leak, rather than a slab fragmentation, which assumes there might be a kernel issue here. From the slabinfo (I only keep three first columns here): >> >> >> name <active_objs> <num_objs> >> ll_obdo_cache 0 452282156 208 >> >> means that there are no active objects, but the memory pages are not released back from slab allocator to the free pool (the num value is huge). That looks like a slab fragmentation - you can get more description at >> http://kerneltrap.org/Linux/Slab_Defragmentation >> >> Checking your mails, I wonder if this only happens on clients which have SLES11 installed? As the RAM size is around 192Gb, I assume they are NUMA systems? >> If so, SLES11 has defrag_ratio tunables in /sys/kernel/slab/xxx >> From the source of get_any_partial() >> >> #ifdef CONFIG_NUMA >> >> /* >> * The defrag ratio allows a configuration of the tradeoffs between >> * inter node defragmentation and node local allocations. A lower >> * defrag_ratio increases the tendency to do local allocations >> * instead of attempting to obtain partial slabs from other nodes. >> * >> * If the defrag_ratio is set to 0 then kmalloc() always >> * returns node local objects. If the ratio is higher then kmalloc() >> * may return off node objects because partial slabs are obtained >> * from other nodes and filled up. >> * >> * If /sys/kernel/slab/xx/defrag_ratio is set to 100 (which makes >> * defrag_ratio = 1000) then every (well almost) allocation will >> * first attempt to defrag slab caches on other nodes. This means >> * scanning over all nodes to look for partial slabs which may be >> * expensive if we do it every time we are trying to find a slab >> * with available objects. >> */ >> >> Could you please verify that your clients have defrag_ratio tunable and try to use various values? >> It looks like the value of 100 should be the best, unless there is a bug, then may be even 0 gets the desired result? >> >> Best regards, >> Dmitry >> >> >> Jagga Soorma wrote: >> >>> Hi Johann, >>> >>> I am actually using 1.8.1 and not 1.8.2: >>> >>> # rpm -qa | grep -i lustre >>> lustre-client-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default >>> lustre-client-modules-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default >>> >>> My kernel version on the SLES 11 clients is: >>> # uname -r >>> 2.6.27.29-0.1-default >>> >>> My kernel version on the RHEL 5.3 mds/oss servers is: >>> # uname -r >>> 2.6.18-128.7.1.el5_lustre.1.8.1.1 >>> >>> Please let me know if you need any further information. I am still trying to get the user to help me run his app so that I can run the leak finder script to capture more information. >>> >>> Regards, >>> -Simran >>> >>> On Tue, Apr 27, 2010 at 7:20 AM, Johann Lombardi <johann at sun.com> wrote: >>> Hi, >>> >>> On Tue, Apr 20, 2010 at 09:08:25AM -0700, Jagga Soorma wrote: >>> >>>> Thanks for your response.* I will try to run the leak-finder script and >>>> hopefully it will point us in the right direction.* This only seems to be >>>> happening on some of my clients: >>>> >>> Could you please tell us what kernel you use on the client side? >>> >>> >>>> client104: ll_obdo_cache********* 0 433506280*** 208** 19*** 1 : tunables* >>>> 120** 60*** 8 : slabdata***** 0 22816120***** 0 >>>> client116: ll_obdo_cache********* 0 457366746*** 208** 19*** 1 : tunables* >>>> 120** 60*** 8 : slabdata***** 0 24071934***** 0 >>>> client113: ll_obdo_cache********* 0 456778867*** 208** 19*** 1 : tunables* >>>> 120** 60*** 8 : slabdata***** 0 24040993***** 0 >>>> client106: ll_obdo_cache********* 0 456372267*** 208** 19*** 1 : tunables* >>>> 120** 60*** 8 : slabdata***** 0 24019593***** 0 >>>> client115: ll_obdo_cache********* 0 449929310*** 208** 19*** 1 : tunables* >>>> 120** 60*** 8 : slabdata***** 0 23680490***** 0 >>>> client101: ll_obdo_cache********* 0 454318101*** 208** 19*** 1 : tunables* >>>> 120** 60*** 8 : slabdata***** 0 23911479***** 0 >>>> -- >>>> >>>> Hopefully this should help.* Not sure which application might be causing >>>> the leaks.* Currently R is the only app that users seem to be using >>>> heavily on these clients.* Will let you know what I find. >>>> >>> Tommi Tervo has filed a bugzilla ticket for this issue, see >>> https://bugzilla.lustre.org/show_bug.cgi?id=22701 >>> >>> Could you please add a comment to this ticket to describe the >>> behavior of the application "R" (fork many threads, write to >>> many files, use direct i/o, ...)? >>> >>> Cheers, >>> Johann >>> >>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>> >>> >>> >>> >> > > > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100830/e6b83e22/attachment-0001.html
It looks like 1.8.4 is the most recent stable release for 1.8.x, so I will plan on upgrading to this new release and see if this resolves my memory leak. Is there a reason why SLES 11 SP1 is not being tested for these new lustre clients? Why is the kernel for SLES11 staying at 2.6.27.39-0.3.1? Thanks, -Simran On Mon, Aug 30, 2010 at 6:50 PM, Dmitry Zogin <dmitry.zoguine at oracle.com>wrote:> Actually there was a bug fixed in 1.8.4 when obdo structures can be > allocated and freed outside of OBDO_ALLOC/OBDO_FREE macros. That could lead > to the slab fragmentation and pseudo-leak. > The patch is in the attachment 30664 for bz 21980 > > Dmitry > > > Andreas Dilger wrote: > > On 2010-08-26, at 18:42, Jagga Soorma wrote: > > > I am still running into this issue on some nodes: > > client109: ll_obdo_cache 0 152914489 208 19 1 : tunables 120 60 8 : slabdata 0 8048131 0 > client102: ll_obdo_cache 0 308526883 208 19 1 : tunables 120 60 8 : slabdata 0 16238257 0 > > How can I calculate how much memory this is holding on to. > > > If you do "head -1 /proc/slabinfo" it reports the column descriptions. > > The "slabdata" will section reports numslabs=16238257, and pagesperslab=1, so tis is 16238257 pages of memory, or about 64GB of RAM on client102. Ouch. > > > > My system shows a lot of memory that is being used up but none of the jobs are using that much memory. Also, these clients are running a smp sles 11 kernel but I can''t find any /sys/kernel/slab directory. > > Linux client102 2.6.27.29-0.1-default #1 SMP 2009-08-15 17:53:59 +0200 x86_64 x86_64 x86_64 GNU/Linux > > What makes you say that this does not look like a lustre memory leak? I thought all the ll_* objects in slabinfo are lustre related? > > > It''s true that the ll_obdo_cache objects are allocated by Lustre, but the above data shows 0 of those objects in use, so the kernel _should_ be freeing the unused slab objects. This particular data type (obdo) is only ever in use temporarily during system calls on the client, and should never be allocated for a long time. > > For some reason the kernel is not freeing the empty slab pages. That is the responsibility of the kernel, and not Lustre. > > > > To me it looks like lustre is holding on to this memory but I don''t know much about lustre internals. > > Also, memused on these systems are: > > client102: 2353666940 > client109: 2421645924 > > > This shows that Lustre is actively using about 2.4GB of memory allocations. It is not tracking the 64GB of memory in the obdo_cache slab, because it has freed that memory (even though the kernel has not freed those pages). > > > > Any help would be greatly appreciated. > > > The only suggestion I have is that if you unmount Lustre and unload the modules (lustre_rmmod) it will free up this memory. Otherwise, searching for problems with the slab cache on this kernel may turn up something. > > > > On Wed, May 19, 2010 at 8:08 AM, Dmitry Zogin <dmitry.zoguine at oracle.com> <dmitry.zoguine at oracle.com> wrote: > Hello Jagga, > > I checked the data, and indeed this does not look like a lustre memory leak, rather than a slab fragmentation, which assumes there might be a kernel issue here. From the slabinfo (I only keep three first columns here): > > > name <active_objs> <num_objs> > ll_obdo_cache 0 452282156 208 > > means that there are no active objects, but the memory pages are not released back from slab allocator to the free pool (the num value is huge). That looks like a slab fragmentation - you can get more description at http://kerneltrap.org/Linux/Slab_Defragmentation > > Checking your mails, I wonder if this only happens on clients which have SLES11 installed? As the RAM size is around 192Gb, I assume they are NUMA systems? > If so, SLES11 has defrag_ratio tunables in /sys/kernel/slab/xxx > From the source of get_any_partial() > > #ifdef CONFIG_NUMA > > /* > * The defrag ratio allows a configuration of the tradeoffs between > * inter node defragmentation and node local allocations. A lower > * defrag_ratio increases the tendency to do local allocations > * instead of attempting to obtain partial slabs from other nodes. > * > * If the defrag_ratio is set to 0 then kmalloc() always > * returns node local objects. If the ratio is higher then kmalloc() > * may return off node objects because partial slabs are obtained > * from other nodes and filled up. > * > * If /sys/kernel/slab/xx/defrag_ratio is set to 100 (which makes > * defrag_ratio = 1000) then every (well almost) allocation will > * first attempt to defrag slab caches on other nodes. This means > * scanning over all nodes to look for partial slabs which may be > * expensive if we do it every time we are trying to find a slab > * with available objects. > */ > > Could you please verify that your clients have defrag_ratio tunable and try to use various values? > It looks like the value of 100 should be the best, unless there is a bug, then may be even 0 gets the desired result? > > Best regards, > Dmitry > > > Jagga Soorma wrote: > > > Hi Johann, > > I am actually using 1.8.1 and not 1.8.2: > > # rpm -qa | grep -i lustre > lustre-client-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default > lustre-client-modules-1.8.1.1-2.6.27.29_0.1_lustre.1.8.1.1_default > > My kernel version on the SLES 11 clients is: > # uname -r > 2.6.27.29-0.1-default > > My kernel version on the RHEL 5.3 mds/oss servers is: > # uname -r > 2.6.18-128.7.1.el5_lustre.1.8.1.1 > > Please let me know if you need any further information. I am still trying to get the user to help me run his app so that I can run the leak finder script to capture more information. > > Regards, > -Simran > > On Tue, Apr 27, 2010 at 7:20 AM, Johann Lombardi <johann at sun.com> <johann at sun.com> wrote: > Hi, > > On Tue, Apr 20, 2010 at 09:08:25AM -0700, Jagga Soorma wrote: > > > Thanks for your response.* I will try to run the leak-finder script and > hopefully it will point us in the right direction.* This only seems to be > happening on some of my clients: > > > Could you please tell us what kernel you use on the client side? > > > > client104: ll_obdo_cache********* 0 433506280*** 208** 19*** 1 : tunables* > 120** 60*** 8 : slabdata***** 0 22816120***** 0 > client116: ll_obdo_cache********* 0 457366746*** 208** 19*** 1 : tunables* > 120** 60*** 8 : slabdata***** 0 24071934***** 0 > client113: ll_obdo_cache********* 0 456778867*** 208** 19*** 1 : tunables* > 120** 60*** 8 : slabdata***** 0 24040993***** 0 > client106: ll_obdo_cache********* 0 456372267*** 208** 19*** 1 : tunables* > 120** 60*** 8 : slabdata***** 0 24019593***** 0 > client115: ll_obdo_cache********* 0 449929310*** 208** 19*** 1 : tunables* > 120** 60*** 8 : slabdata***** 0 23680490***** 0 > client101: ll_obdo_cache********* 0 454318101*** 208** 19*** 1 : tunables* > 120** 60*** 8 : slabdata***** 0 23911479***** 0 > -- > > Hopefully this should help.* Not sure which application might be causing > the leaks.* Currently R is the only app that users seem to be using > heavily on these clients.* Will let you know what I find. > > > Tommi Tervo has filed a bugzilla ticket for this issue, seehttps://bugzilla.lustre.org/show_bug.cgi?id=22701 > > Could you please add a comment to this ticket to describe the > behavior of the application "R" (fork many threads, write to > many files, use direct i/o, ...)? > > Cheers, > Johann > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.orghttp://lists.lustre.org/mailman/listinfo/lustre-discuss > > > > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. > > _______________________________________________ > Lustre-discuss mailing listLustre-discuss at lists.lustre.orghttp://lists.lustre.org/mailman/listinfo/lustre-discuss > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100831/7bd710f1/attachment.html
On 2010-08-31, at 13:12, Jagga Soorma wrote:> It looks like 1.8.4 is the most recent stable release for 1.8.x, so I will plan on upgrading to this new release and see if this resolves my memory leak. Is there a reason why SLES 11 SP1 is not being tested for these new lustre clients? Why is the kernel for SLES11 staying at 2.6.27.39-0.3.1?Unfortunately, SLES decided to make a major change the kernel in their stable SLES11 release, from 2.6.27 to 2.6.32. This requires a considerable amount of change in Lustre to work with the new kernel, which wasn''t completed in time for the 1.8.4 feature cutoff. The support for SLES11 SP1 is largely complete and should be available in the 1.8.5 release. The only good thing to come out of this disruptive change is that SLES11 SP1''s 2.6.32 kernel is nearly the same as the RHEL6 2.6.32 kernel so it will hopefully mean we can share the patches for both of these kernels moving forward. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc.