Hello! We''ve got one MDS (Lustre 2.0) with 4 processors (Intel(R) Core(TM)2 Quad CPU Q9500 @ 2.83GHz). Our CPU usage is HIGH - 80-100% . With top command we see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2973 root 15 0 0 0 0 R 35.0 0.0 1509:09 socknal_sd02 2972 root 15 0 0 0 0 S 33.5 0.0 1132:13 socknal_sd01 2971 root 15 0 0 0 0 S 19.8 0.0 1129:12 socknal_sd00 2974 root 16 0 0 0 0 R 13.7 0.0 830:47.46 socknal_sd03 Is it normal situation? Can we improve performance at mds ? Maybe setting any parameters? (Only about 70 clients mount to Lustre and not all of them work with it.) With regards, Popova Ekaterina
Hi, More details are needed here. What type of interconnect are you using? What are your clients doing? etc. -cf On 07/04/2011 11:08 PM, Ekaterina Popova wrote:> Hello! > > We''ve got one MDS (Lustre 2.0) with 4 processors (Intel(R) Core(TM)2 > Quad CPU Q9500 @ 2.83GHz). > Our CPU usage is HIGH - 80-100% . > With top command we see: > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 2973 root 15 0 0 0 0 R 35.0 0.0 1509:09 socknal_sd02 > 2972 root 15 0 0 0 0 S 33.5 0.0 1132:13 socknal_sd01 > 2971 root 15 0 0 0 0 S 19.8 0.0 1129:12 socknal_sd00 > 2974 root 16 0 0 0 0 R 13.7 0.0 830:47.46 socknal_sd03 > > Is it normal situation? Can we improve performance at mds ? Maybe > setting any parameters? > (Only about 70 clients mount to Lustre and not all of them work with it.) > > With regards, > Popova Ekaterina > > > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss
I think it''s TCP/IP according to the process list. It''d help to find out where the CPU time was spent, e.g. by oprofile. - Isaac On Wed, Jul 06, 2011 at 12:14:54PM -0600, Colin Faber wrote:> Hi, > > More details are needed here. What type of interconnect are you using? > What are your clients doing? etc. > > -cf > > > On 07/04/2011 11:08 PM, Ekaterina Popova wrote: > > Hello! > > > > We''ve got one MDS (Lustre 2.0) with 4 processors (Intel(R) Core(TM)2 > > Quad CPU Q9500 @ 2.83GHz). > > Our CPU usage is HIGH - 80-100% . > > With top command we see: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 2973 root 15 0 0 0 0 R 35.0 0.0 1509:09 socknal_sd02 > > 2972 root 15 0 0 0 0 S 33.5 0.0 1132:13 socknal_sd01 > > 2971 root 15 0 0 0 0 S 19.8 0.0 1129:12 socknal_sd00 > > 2974 root 16 0 0 0 0 R 13.7 0.0 830:47.46 socknal_sd03 > > > > Is it normal situation? Can we improve performance at mds ? Maybe > > setting any parameters? > > (Only about 70 clients mount to Lustre and not all of them work with it.) > > > > With regards, > > Popova Ekaterina > > > > > > > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People''s Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________
Agreed however more details are still needed. -cf -----Original message----- From: Isaac Huang Sent: 07/07/2011, 12:31 am To: Colin Faber Cc: Ekaterina Popova; lustre-discuss at lists.lustre.org Subject: Re: [Lustre-discuss] high CPU usage on MDS I think it''s TCP/IP according to the process list. It''d help to find out where the CPU time was spent, e.g. by oprofile. - Isaac On Wed, Jul 06, 2011 at 12:14:54PM -0600, Colin Faber wrote:> Hi, > > More details are needed here. What type of interconnect are you using? > What are your clients doing? etc. > > -cf > > > On 07/04/2011 11:08 PM, Ekaterina Popova wrote: > > Hello! > > > > We''ve got one MDS (Lustre 2.0) with 4 processors (Intel(R) Core(TM)2 > > Quad CPU Q9500 @ 2.83GHz). > > Our CPU usage is HIGH - 80-100% . > > With top command we see: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 2973 root 15 0 0 0 0 R 35.0 0.0 1509:09 socknal_sd02 > > 2972 root 15 0 0 0 0 S 33.5 0.0 1132:13 socknal_sd01 > > 2971 root 15 0 0 0 0 S 19.8 0.0 1129:12 socknal_sd00 > > 2974 root 16 0 0 0 0 R 13.7 0.0 830:47.46 socknal_sd03 > > > > Is it normal situation? Can we improve performance at mds ? Maybe > > setting any parameters? > > (Only about 70 clients mount to Lustre and not all of them work with it.) > > > > With regards, > > Popova Ekaterina > > > > > > > > > > _______________________________________________ > > Lustre-discuss mailing list > > Lustre-discuss at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-discuss > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People''s Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________
On 07/07/2011 06:10 PM, Colin Faber wrote:> Agreed however more details are still needed. > > -cf > -----Original message----- > From: Isaac Huang > Sent: 07/07/2011, 12:31 am > To: Colin Faber > Cc: Ekaterina Popova; lustre-discuss at lists.lustre.org > Subject: Re: [Lustre-discuss] high CPU usage on MDS > > > > I think it''s TCP/IP according to the process list. It''d help to find > out where the CPU time was spent, e.g. by oprofile. > > - Isaac > > On Wed, Jul 06, 2011 at 12:14:54PM -0600, Colin Faber wrote: > >> Hi, >> >> More details are needed here. What type of interconnect are you using? >> What are your clients doing? etc. >> >> -cf >> >> >> On 07/04/2011 11:08 PM, Ekaterina Popova wrote: >> >>> Hello! >>> >>> We''ve got one MDS (Lustre 2.0) with 4 processors (Intel(R) Core(TM)2 >>> Quad CPU Q9500 @ 2.83GHz). >>> Our CPU usage is HIGH - 80-100% . >>> With top command we see: >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>> 2973 root 15 0 0 0 0 R 35.0 0.0 1509:09 socknal_sd02 >>> 2972 root 15 0 0 0 0 S 33.5 0.0 1132:13 socknal_sd01 >>> 2971 root 15 0 0 0 0 S 19.8 0.0 1129:12 socknal_sd00 >>> 2974 root 16 0 0 0 0 R 13.7 0.0 830:47.46 socknal_sd03 >>> >>> Is it normal situation? Can we improve performance at mds ? Maybe >>> setting any parameters? >>> (Only about 70 clients mount to Lustre and not all of them work with it.) >>> >>> With regards, >>> Popova Ekaterina >>> >>> >>> >>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> > ______________________________________________________________________ > This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. > > Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. > > Xyratex Technology Limited (03134912), Registered in England& Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. > > The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People''s Republic of China and Xyratex Japan Limited registered in Japan. > ______________________________________________________________________ > >Hi, We have two 1Gb Ethernet cards, one card is using for internal cluster, another one is for outer network (it is almost not used). We see large network traffic in MDS on internal network card (~200M/s in case of 150 jobs running in grid cluster). Memory using is low (total mem - 4G, used mem - 1G). At changelog on client we have such records: ... 5463915 06UNLNK 06:21:17.11299887 2011.07.05 0x0 t=[0x2000061c4:0x13af5:0x0] p[0x200005233:0x12fd3:0x0] app.upmf.properties285631923419962629.TMP 5463916 01CREAT 06:31:17.30895460 2011.07.05 0x0 t=[0x2000061c4:0x13b4d:0x0] p[0x200005233:0x12fd3:0x0] app.upmf.properties5508069877142384203.TMP 5463917 06UNLNK 06:31:47.60574777 2011.07.05 0x0 t=[0x2000061c4:0x13b4d:0x0] p[0x200005233:0x12fd3:0x0] app.upmf.properties5508069877142384203.TMP 5463918 01CREAT 06:31:47.77574595 2011.07.05 0x0 t=[0x2000061c4:0x13b55:0x0] p[0x200005233:0x12fd3:0x0] app.upmf.properties5753247530971764727.TMP 5463919 06UNLNK 06:31:47.219573079 2011.07.05 0x0 t=[0x2000061c4:0x13b55:0x0] p[0x200005233:0x12fd3:0x0] app.upmf.properties5753247530971764727.TMP On MDS: opreport --symbols -p/lib/modules/2.6.18-prep/ CPU: Core 2, speed 2833 MHz (estimated) Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000 samples % image name app name symbol name 3445413 3.1207 lnet.ko lnet.ko lnet_finalize 3367306 3.0499 lvfs.ko lvfs.ko lprocfs_counter_add 3224905 2.9210 ptlrpc.ko ptlrpc.ko ptlrpc_main_check_event 3207810 2.9055 vmlinux vmlinux task_rq_lock 3029057 2.7436 vmlinux vmlinux memset 2890069 2.6177 vmlinux vmlinux schedule 2292712 2.0766 ptlrpc.ko ptlrpc.ko ptlrpc_main 1997346 1.8091 lnet.ko lnet.ko lnet_parse 1777756 1.6102 vmlinux vmlinux kfree 1769944 1.6031 obdclass.ko obdclass.ko key_fini 1745169 1.5807 lnet.ko lnet.ko LNetMDBind 1646427 1.4913 vmlinux vmlinux __wake_up 1573871 1.4255 obdclass.ko obdclass.ko keys_fill 1556594 1.4099 atl1e.ko atl1e.ko atl1e_intr 1546613 1.4008 vmlinux vmlinux __list_add 1512194 1.3697 ptlrpc.ko ptlrpc.ko ptlrpc_server_handle_request 1398966 1.2671 vmlinux vmlinux list_del 1254363 1.1361 lvfs.ko lvfs.ko lprocfs_counter_sub 1206541 1.0928 vmlinux vmlinux try_to_wake_up 1188367 1.0764 obdclass.ko obdclass.ko keys_fini 1158540 1.0494 obdclass.ko obdclass.ko class_handle2object 1032197 0.9349 ptlrpc.ko ptlrpc.ko ptlrpc_server_drop_request 1027855 0.9310 vmlinux vmlinux thread_return 1013642 0.9181 vmlinux vmlinux tcp_sendmsg 978190 0.8860 vmlinux vmlinux copy_user_generic_unrolled 961920 0.8713 lnet.ko lnet.ko LNetPut 929284 0.8417 obdclass.ko obdclass.ko lu_context_init 911345 0.8255 vmlinux vmlinux kmem_cache_free 879200 0.7963 vmlinux vmlinux mwait_idle 843759 0.7642 atl1e.ko atl1e.ko atl1e_xmit_frame 808896 0.7327 vmlinux vmlinux __kmalloc 806832 0.7308 ksocklnd.ko ksocklnd.ko ksocknal_receive 762878 0.6910 ptlrpc.ko ptlrpc.ko ptlrpc_at_add_timed 753020 0.6821 ptlrpc.ko ptlrpc.ko lustre_msg_buf_v2 731792 0.6628 vmlinux vmlinux __wake_up_common 680931 0.6168 vmlinux vmlinux tcp_recvmsg 674703 0.6111 vmlinux vmlinux remove_wait_queue 649026 0.5879 vmlinux vmlinux skb_release_head_state 648489 0.5874 mdt.ko mdt.ko mdt_handle_common 620536 0.5621 ptlrpc.ko ptlrpc.ko request_in_callback 616810 0.5587 vmlinux vmlinux memcpy 605268 0.5482 vmlinux vmlinux find_busiest_group 588503 0.5330 vmlinux vmlinux __switch_to 584848 0.5297 obdclass.ko obdclass.ko lu_context_key_get 566552 0.5132 atl1e.ko atl1e.ko atl1e_clean_tx_irq 527686 0.4780 vmlinux vmlinux tcp_v4_rcv 524791 0.4753 ksocklnd.ko ksocklnd.ko ksocknal_find_conn_locked 495425 0.4487 ptlrpc.ko ptlrpc.ko reply_out_callback 482759 0.4373 vmlinux vmlinux kmem_cache_alloc 459759 0.4164 ptlrpc.ko ptlrpc.ko ptlrpc_send_reply 457681 0.4145 obdclass.ko obdclass.ko htable_lookup 438283 0.3970 ksocklnd.ko ksocklnd.ko ksocknal_transmit 432296 0.3916 ksocklnd.ko ksocklnd.ko ksocknal_scheduler 426023 0.3859 lnet.ko lnet.ko lnet_md_unlink 422453 0.3826 vmlinux vmlinux __down_read 416777 0.3775 vmlinux vmlinux dev_queue_xmit 390667 0.3538 ptlrpc.ko ptlrpc.ko __req_capsule_get 389829 0.3531 lnet.ko lnet.ko lnet_send 385353 0.3490 ptlrpc.ko ptlrpc.ko lustre_pack_reply_v2 379540 0.3438 vmlinux vmlinux prepare_to_wait 375104 0.3398 vmlinux vmlinux tcp_transmit_skb 365540 0.3311 ksocklnd.ko ksocklnd.ko ksocknal_alloc_tx 363096 0.3289 vmlinux vmlinux memmove 349697 0.3167 lnet.ko lnet.ko lnet_complete_msg_locked 344320 0.3119 ptlrpc.ko ptlrpc.ko sptlrpc_wireflavor2policy 334530 0.3030 obdclass.ko obdclass.ko lu_context_exit 333776 0.3023 vmlinux vmlinux effective_prio 332232 0.3009 ptlrpc.ko ptlrpc.ko at_measured 328450 0.2975 vmlinux vmlinux skb_release_data 326971 0.2962 obdclass.ko obdclass.ko lu_object_put 323959 0.2934 ptlrpc.ko ptlrpc.ko ptlrpc_server_free_request 322772 0.2924 vmlinux vmlinux tcp_ack 322667 0.2923 lnet.ko lnet.ko LNetMEAttach 313380 0.2838 mdt.ko mdt.ko mdt_getattr_internal 311371 0.2820 vmlinux vmlinux __alloc_skb 308274 0.2792 ptlrpc.ko ptlrpc.ko ldlm_resource_find 306783 0.2779 lnet.ko lnet.ko lnet_return_credits_locked 303357 0.2748 vmlinux vmlinux ip_route_input 302618 0.2741 obdclass.ko obdclass.ko lu_context_fini 297917 0.2698 vmlinux vmlinux enqueue_task 291245 0.2638 lnet.ko lnet.ko lnet_recv_put 289224 0.2620 vmlinux vmlinux __qdisc_run 288297 0.2611 osd_ldiskfs.ko osd_ldiskfs.ko osd_attr_get 283552 0.2568 ptlrpc.ko ptlrpc.ko lock_res_and_lock 282665 0.2560 ptlrpc.ko ptlrpc.ko target_pack_pool_reply 270775 0.2453 vmlinux vmlinux ip_queue_xmit 269148 0.2438 lnet.ko lnet.ko lnet_enq_event_locked 266185 0.2411 vmlinux vmlinux __find_get_block 262071 0.2374 vmlinux vmlinux resched_task 261675 0.2370 lnet.ko lnet.ko lnet_match_md 260547 0.2360 ip_tables.ko ip_tables.ko ipt_do_table 257705 0.2334 vmlinux vmlinux tcp_rcv_established 250976 0.2273 ksocklnd.ko ksocklnd.ko ksocknal_queue_tx_locked 249668 0.2261 vmlinux vmlinux local_bh_enable_ip 248794 0.2253 vmlinux vmlinux ip_output 245507 0.2224 ksocklnd.ko ksocklnd.ko ksocknal_read_callback 241913 0.2191 vmlinux vmlinux put_page 238653 0.2162 vmlinux vmlinux recalc_task_prio 232245 0.2104 lnet.ko lnet.ko lnet_lookup_cookie 230806 0.2091 ptlrpc.ko ptlrpc.ko null_accept 228674 0.2071 obdclass.ko obdclass.ko class_handle_hash 228161 0.2067 ptlrpc.ko ptlrpc.ko sptlrpc_svc_alloc_rs 228019 0.2065 ptlrpc.ko ptlrpc.ko sptlrpc_target_export_check 225989 0.2047 obdclass.ko obdclass.ko lu_object_find_at 221477 0.2006 oprofiled oprofiled odb_update_node_with_offset 213570 0.1934 ptlrpc.ko ptlrpc.ko null_alloc_rs 213379 0.1933 ptlrpc.ko ptlrpc.ko req_capsule_has_field 212791 0.1927 vmlinux vmlinux tcp_cleanup_rbuf 212646 0.1926 vmlinux vmlinux list_add 211718 0.1918 ksocklnd.ko ksocklnd.ko ksocknal_process_receive 211447 0.1915 vmlinux vmlinux sock_wfree 210107 0.1903 vmlinux vmlinux sched_clock 207013 0.1875 ptlrpc.ko ptlrpc.ko __req_capsule_offset 206226 0.1868 ksocklnd.ko ksocklnd.ko ksocknal_recv_iov 204788 0.1855 vmlinux vmlinux local_bh_disable 202609 0.1835 libcfs.ko libcfs.ko cfs_alloc_flags_to_gfp 201355 0.1824 vmlinux vmlinux do_gettimeofday 200403 0.1815 ptlrpc.ko ptlrpc.ko sptlrpc_svc_unwrap_request 200133 0.1813 ksocklnd.ko ksocklnd.ko ksocknal_data_ready 196847 0.1783 ksocklnd.ko ksocklnd.ko ksocknal_free_tx 194325 0.1760 ptlrpc.ko ptlrpc.ko ptlrpc_master_callback 191462 0.1734 ptlrpc.ko ptlrpc.ko ptlrpc_check_req 191407 0.1734 lnet.ko lnet.ko lnet_commit_md 191155 0.1731 libcfs.ko libcfs.ko lc_watchdog_touch 190496 0.1725 mdt.ko mdt.ko mdt_getattr 189965 0.1721 obdclass.ko obdclass.ko class_export_put 186893 0.1693 atl1e.ko atl1e.ko atl1e_clean_rx_irq 186683 0.1691 lnet.ko lnet.ko lnet_find_peer_locked 186143 0.1686 ldiskfs.ko ldiskfs.ko ldiskfs_find_entry 185352 0.1679 iptable_filter.ko iptable_filter.ko ipt_hook 184570 0.1672 ptlrpc.ko ptlrpc.ko ldlm_resource_get 184553 0.1672 ksocklnd.ko ksocklnd.ko ksocknal_find_peer_locked 182066 0.1649 vmlinux vmlinux add_wait_queue_exclusive 179692 0.1628 vmlinux vmlinux cond_resched 177681 0.1609 ksocklnd.ko ksocklnd.ko ksocknal_tx_prep 172136 0.1559 ptlrpc.ko ptlrpc.ko null_free_rs 168889 0.1530 libstdc++.so.6.0.8 opreport /usr/lib64/ libstdc++.so.6.0.8 167529 0.1517 ptlrpc.ko ptlrpc.ko lustre_msg_get_opc 166144 0.1505 ksocklnd.ko ksocklnd.ko ksocknal_lib_recv_iov 162940 0.1476 vmlinux vmlinux recalc_sigpending_tsk 161651 0.1464 ptlrpc.ko ptlrpc.ko sptlrpc_svc_ctx_decref 161047 0.1459 oprofiled oprofiled find_kernel_image 160557 0.1454 lnet.ko lnet.ko lnet_nid2ni_locked 159886 0.1448 ptlrpc.ko ptlrpc.ko lustre_shrink_msg_v2 158071 0.1432 obdclass.ko obdclass.ko class_export_get 157994 0.1431 vmlinux vmlinux __up_read 157418 0.1426 ldiskfs.ko ldiskfs.ko __ldiskfs_get_inode_loc 157222 0.1424 libcfs.ko libcfs.ko cfs_block_sigs 154417 0.1399 vmlinux vmlinux csum_partial 153571 0.1391 lnet.ko lnet.ko LNetMDAttach 149419 0.1353 vmlinux vmlinux __activate_task 149345 0.1353 vmlinux vmlinux __tcp_push_pending_frames 148491 0.1345 ksocklnd.ko ksocklnd.ko ksocknal_launch_packet 147683 0.1338 vmlinux vmlinux sock_recvmsg 144704 0.1311 ptlrpc.ko ptlrpc.ko ptlrpc_connection_addref 144254 0.1307 obdclass.ko obdclass.ko lu_object_find 139774 0.1266 vmlinux vmlinux pfifo_fast_enqueue 139234 0.1261 lnet.ko lnet.ko lnet_try_match_md 138095 0.1251 vmlinux vmlinux do_gettimeoffset_tsc 137704 0.1247 ptlrpc.ko ptlrpc.ko req_capsule_shrink 137273 0.1243 vmlinux vmlinux lock_timer_base 137021 0.1241 vmlinux vmlinux release_sock 135984 0.1232 mdt.ko mdt.ko mdt_unpack_req_pack_rep 135442 0.1227 vmlinux vmlinux generic_getxattr 132666 0.1202 obdclass.ko obdclass.ko class_connected_export 131849 0.1194 ptlrpc.ko ptlrpc.ko ptlrpc_buf_need_swab 131417 0.1190 libcfs.ko libcfs.ko cfs_alloc 131039 0.1187 ptlrpc.ko ptlrpc.ko target_send_reply 131009 0.1187 libcfs.ko libcfs.ko cfs_hash_add 130974 0.1186 vmlinux vmlinux __mod_timer 128345 0.1162 osd_ldiskfs.ko osd_ldiskfs.ko osd_object_read_lock 126437 0.1145 vmlinux vmlinux netif_rx 125451 0.1136 vmlinux vmlinux sock_common_recvmsg 124448 0.1127 ptlrpc.ko ptlrpc.ko ldlm_lock_put 123629 0.1120 lnet.ko lnet.ko lnet_post_send_locked 120179 0.1089 ptlrpc.ko ptlrpc.ko lustre_free_reply_state 119279 0.1080 ptlrpc.ko ptlrpc.ko ptl_send_buf 118334 0.1072 vmlinux vmlinux default_wake_function 117874 0.1068 ptlrpc.ko ptlrpc.ko sptlrpc_svc_wrap_reply 117338 0.1063 vmlinux vmlinux dequeue_task 116492 0.1055 opreport opreport extra_images::matcher::operator()(std::string const&) const 115787 0.1049 vmlinux vmlinux tcp_current_mss 114841 0.1040 mdt.ko mdt.ko mdt_regular_handle 114397 0.1036 vmlinux vmlinux __copy_skb_header 113796 0.1031 ptlrpc.ko ptlrpc.ko ptlrpc_server_post_idle_rqbds 112548 0.1019 vmlinux vmlinux _atomic_dec_and_lock 112275 0.1017 vmlinux vmlinux find_first_bit 112062 0.1015 vmlinux vmlinux find_next_bit 109904 0.0995 vmlinux vmlinux skb_clone 109041 0.0988 libcfs.ko libcfs.ko lc_watchdog_disable 107246 0.0971 ptlrpc.ko ptlrpc.ko __lustre_unpack_msg 106395 0.0964 ptlrpc.ko ptlrpc.ko __req_format_is_sane 105582 0.0956 mdt.ko mdt.ko mdt_shrink_reply 104623 0.0948 ptlrpc.ko ptlrpc.ko ptlrpc_update_export_timer 103974 0.0942 vmlinux vmlinux __rcu_process_callbacks 102272 0.0926 lnet.ko lnet.ko lib_md_build 100840 0.0913 lnet.ko lnet.ko lnet_md_deconstruct 100757 0.0913 ptlrpc.ko ptlrpc.ko lustre_msg_size_v2 99939 0.0905 vmlinux vmlinux ip_rcv 99033 0.0897 ptlrpc.ko ptlrpc.ko ldlm_lock_create 98275 0.0890 vmlinux vmlinux mod_timer 97811 0.0886 vmlinux vmlinux kfree_skbmem 96558 0.0875 mdd.ko mdd.ko mdd_get_md 96390 0.0873 ptlrpc.ko ptlrpc.ko req_capsule_init_area 95672 0.0867 mdt.ko mdt.ko mdt_check_ucred 95263 0.0863 lnet.ko lnet.ko lnet_ni_recv 94561 0.0856 mdt.ko mdt.ko mdt_reint_open 93791 0.0850 mdd.ko mdd.ko mdd_read_lock 93714 0.0849 osd_ldiskfs.ko osd_ldiskfs.ko osd_object_auth 90762 0.0822 vmlinux vmlinux IRQ0x4a_interrupt 90128 0.0816 ksocklnd.ko ksocklnd.ko ksocknal_recv 89651 0.0812 obdclass.ko obdclass.ko lu_time_key_exit 89647 0.0812 vmlinux vmlinux netif_receive_skb 88498 0.0802 mdt.ko mdt.ko mdt_pack_attr2body 87280 0.0791 ksocklnd.ko ksocklnd.ko ksocknal_new_packet 86200 0.0781 mdt.ko mdt.ko mdt_lock_handle_init 84561 0.0766 ptlrpc.ko ptlrpc.ko req_capsule_filled_sizes 84364 0.0764 ksocklnd.ko ksocklnd.ko ksocknal_send 83359 0.0755 vmlinux vmlinux nf_iterate 82758 0.0750 ptlrpc.ko ptlrpc.ko ldlm_inodebits_compat_queue 82041 0.0743 oprofiled oprofiled sfile_find 81178 0.0735 ksocklnd.ko ksocklnd.ko ksocknal_match_tx_v3 80916 0.0733 vmlinux vmlinux xattr_resolve_name 80849 0.0732 mdd.ko mdd.ko mdd_ucred_key_fini 80835 0.0732 osd_ldiskfs.ko osd_ldiskfs.ko osd_xattr_get 80829 0.0732 lustre.ko lustre.ko vvp_session_key_fini 80522 0.0729 mdt.ko mdt.ko mdt_object_lock 80471 0.0729 vmlinux vmlinux process_backlog 80218 0.0727 ptlrpc.ko ptlrpc.ko __req_msg 80150 0.0726 vmlinux vmlinux alternate_node_alloc 79896 0.0724 lvfs.ko lvfs.ko upcall_cache_get_entry 78040 0.0707 ptlrpc.ko ptlrpc.ko ldlm_handle_enqueue0 77957 0.0706 ptlrpc.ko ptlrpc.ko ldlm_lock_remove_from_lru 77256 0.0700 vmlinux vmlinux tcp_event_data_recv 77006 0.0697 vmlinux vmlinux lock_sock 75922 0.0688 mdd.ko mdd.ko mdd_attr_get_internal 74796 0.0677 ptlrpc.ko ptlrpc.ko target_committed_to_req 74788 0.0677 ksocklnd.ko ksocklnd.ko ksocknal_process_transmit 74656 0.0676 cmm.ko cmm.ko cml_attr_get 74576 0.0675 ksocklnd.ko ksocklnd.ko ksocknal_next_tx_carrier 73666 0.0667 lnet.ko lnet.ko lnet_ni_send 73296 0.0664 ksocklnd.ko ksocklnd.ko ksocknal_tx_done 73023 0.0661 lnet.ko lnet.ko lnet_setpayloadbuffer 72641 0.0658 osd_ldiskfs.ko osd_ldiskfs.ko lu_device_is_osd 72051 0.0653 vmlinux vmlinux skb_copy_datagram_iovec 71925 0.0651 ptlrpc.ko ptlrpc.ko ptlrpc_server_allow_normal 70839 0.0642 vmlinux vmlinux local_bh_enable 70244 0.0636 vmlinux vmlinux sock_sendmsg 70131 0.0635 vmlinux vmlinux get_task_mm 69625 0.0631 opreport opreport (anonymous namespace)::module_matcher::operator()(std::string const&) const 69573 0.0630 vmlinux vmlinux main_timer_handler 68370 0.0619 mdd.ko mdd.ko mdd_iattr_get 67955 0.0616 ptlrpc.ko ptlrpc.ko ptlrpc_connection_put 67875 0.0615 ptlrpc.ko ptlrpc.ko ldlm_pool_add 66775 0.0605 mdd.ko mdd.ko __mdd_lmm_get 66361 0.0601 vmlinux vmlinux __sk_dst_check 66330 0.0601 ksocklnd.ko ksocklnd.ko ksocknal_pack_msg_v2 66036 0.0598 oprofiled oprofiled pop_buffer_value 65776 0.0596 mdd.ko mdd.ko mdd_capainfo_key_fini 64314 0.0583 lnet.ko lnet.ko lnet_match_blocked_msg 64257 0.0582 vmlinux vmlinux recalc_sigpending 64184 0.0581 ptlrpc.ko ptlrpc.ko unlock_res_and_lock 64009 0.0580 vmlinux vmlinux tcp_check_space 63851 0.0578 libc-2.5.so top _IO_vfscanf 62692 0.0568 vmlinux vmlinux tcp_rcv_space_adjust 62667 0.0568 ldiskfs.ko ldiskfs.ko ldiskfs_xattr_get 62548 0.0567 ptlrpc.ko ptlrpc.ko req_capsule_extend 61313 0.0555 mdd.ko mdd.ko md_capainfo 61159 0.0554 libcfs.ko libcfs.ko cfs_waitq_active 59511 0.0539 vmlinux vmlinux __do_IRQ 59201 0.0536 ptlrpc.ko ptlrpc.ko req_capsule_init 58802 0.0533 vmlinux vmlinux interleave_nodes 58475 0.0530 ptlrpc.ko ptlrpc.ko lustre_shrink_msg 58425 0.0529 mdt.ko mdt.ko mdt_object_find 57699 0.0523 ksocklnd.ko ksocklnd.ko ksocknal_lib_send_iov 57284 0.0519 ptlrpc.ko ptlrpc.ko sptlrpc_svc_free_rs 57257 0.0519 oprofile.ko oprofile.ko sync_buffer 57155 0.0518 lnet.ko lnet.ko lnet_extract_iov 56945 0.0516 lnet.ko lnet.ko lnet_peer_alive_locked 56904 0.0515 osc.ko osc.ko osc_session_init 56737 0.0514 mdd.ko mdd.ko mdd_la_get 56521 0.0512 mdt.ko mdt.ko mdt_body_reverse_idmap 56498 0.0512 osd_ldiskfs.ko osd_ldiskfs.ko osd_dev 56151 0.0509 lov.ko lov.ko lov_session_key_init 55877 0.0506 vmlinux vmlinux __write_lock_failed 55655 0.0504 ldiskfs.ko ldiskfs.ko ldiskfs_get_blocks_handle 55472 0.0502 ptlrpc.ko ptlrpc.ko ldlm_cli_enqueue_local 54412 0.0493 vmlinux vmlinux ip_local_deliver 52606 0.0476 ptlrpc.ko ptlrpc.ko ptlrpc_server_request_pending 51767 0.0469 vmlinux vmlinux __do_softirq 51562 0.0467 ptlrpc.ko ptlrpc.ko req_capsule_field_present 51549 0.0467 vmlinux vmlinux inet_sendmsg 51414 0.0466 mdt.ko mdt.ko mdt_intent_policy 51296 0.0465 ksocklnd.ko ksocklnd.ko ksocknal_queue_tx_msg_v2 51009 0.0462 libcfs.ko libcfs.ko cfs_hash_rlock 50977 0.0462 vmlinux vmlinux __brelse 50811 0.0460 oprofile.ko oprofile.ko add_event_entry ... With regards, Popova Ekaterina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20110719/b648c771/attachment-0001.html
Hi there, I can''t see any unexpected information in your oprofile, although a few functions have high values in oprofile, but I think they are still in acceptable range. Since you are using GigE, which means data copy might contribute to cpu-usage. Although ksocklnd can support zero-copy, but it''s not used for small messages which is plenty on MDS. The way to change it is adding this to modprobe.conf of MDS: "ksocklnd zc_min_payload=1" It will enable zero-copy for payload larger than 4K (1 page), although I''m not sure how much this can help. Regards Liang On Jul 19, 2011, at 2:33 PM, Ekaterina Popova wrote:> On 07/07/2011 06:10 PM, Colin Faber wrote: >> >> Agreed however more details are still needed. >> >> -cf >> -----Original message----- >> From: Isaac Huang >> Sent: 07/07/2011, 12:31 am >> To: Colin Faber >> Cc: Ekaterina Popova; lustre-discuss at lists.lustre.org >> Subject: Re: [Lustre-discuss] high CPU usage on MDS >> >> >> >> I think it''s TCP/IP according to the process list. It''d help to find >> out where the CPU time was spent, e.g. by oprofile. >> >> - Isaac >> >> On Wed, Jul 06, 2011 at 12:14:54PM -0600, Colin Faber wrote: >> >>> Hi, >>> >>> More details are needed here. What type of interconnect are you using? >>> What are your clients doing? etc. >>> >>> -cf >>> >>> >>> On 07/04/2011 11:08 PM, Ekaterina Popova wrote: >>> >>>> Hello! >>>> >>>> We''ve got one MDS (Lustre 2.0) with 4 processors (Intel(R) Core(TM)2 >>>> Quad CPU Q9500 @ 2.83GHz). >>>> Our CPU usage is HIGH - 80-100% . >>>> With top command we see: >>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>> 2973 root 15 0 0 0 0 R 35.0 0.0 1509:09 socknal_sd02 >>>> 2972 root 15 0 0 0 0 S 33.5 0.0 1132:13 socknal_sd01 >>>> 2971 root 15 0 0 0 0 S 19.8 0.0 1129:12 socknal_sd00 >>>> 2974 root 16 0 0 0 0 R 13.7 0.0 830:47.46 socknal_sd03 >>>> >>>> Is it normal situation? Can we improve performance at mds ? Maybe >>>> setting any parameters? >>>> (Only about 70 clients mount to Lustre and not all of them work with it.) >>>> >>>> With regards, >>>> Popova Ekaterina >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>> >> ______________________________________________________________________ >> This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. >> >> Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. >> >> Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. >> >> The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People''s Republic of China and Xyratex Japan Limited registered in Japan. >> ______________________________________________________________________ >> >> > > Hi, > We have two 1Gb Ethernet cards, one card is using for internal cluster, another > one is for outer network (it is almost not used). We see large network traffic > in MDS on internal network card (~200M/s in case of 150 jobs running in grid > cluster). Memory using is low (total mem - 4G, used mem - 1G). At changelog on > client we have such records: > ... > 5463915 06UNLNK 06:21:17.11299887 2011.07.05 0x0 t=[0x2000061c4:0x13af5:0x0] p> [0x200005233:0x12fd3:0x0] app.upmf.properties285631923419962629.TMP > 5463916 01CREAT 06:31:17.30895460 2011.07.05 0x0 t=[0x2000061c4:0x13b4d:0x0] p> [0x200005233:0x12fd3:0x0] app.upmf.properties5508069877142384203.TMP > 5463917 06UNLNK 06:31:47.60574777 2011.07.05 0x0 t=[0x2000061c4:0x13b4d:0x0] p> [0x200005233:0x12fd3:0x0] app.upmf.properties5508069877142384203.TMP > 5463918 01CREAT 06:31:47.77574595 2011.07.05 0x0 t=[0x2000061c4:0x13b55:0x0] p> [0x200005233:0x12fd3:0x0] app.upmf.properties5753247530971764727.TMP > 5463919 06UNLNK 06:31:47.219573079 2011.07.05 0x0 t=[0x2000061c4:0x13b55:0x0] p> [0x200005233:0x12fd3:0x0] app.upmf.properties5753247530971764727.TMP > > On MDS: > > opreport --symbols -p /lib/modules/2.6.18-prep/ > > CPU: Core 2, speed 2833 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask > of 0x00 (Unhalted core cycles) count 100000 > samples % image name app name symbol name > 3445413 3.1207 lnet.ko lnet.ko > lnet_finalize > 3367306 3.0499 lvfs.ko lvfs.ko > lprocfs_counter_add > 3224905 2.9210 ptlrpc.ko ptlrpc.ko > ptlrpc_main_check_event > 3207810 2.9055 vmlinux vmlinux task_rq_lock > 3029057 2.7436 vmlinux vmlinux memset > 2890069 2.6177 vmlinux vmlinux schedule > 2292712 2.0766 ptlrpc.ko ptlrpc.ko ptlrpc_main > 1997346 1.8091 lnet.ko lnet.ko lnet_parse > 1777756 1.6102 vmlinux vmlinux kfree > 1769944 1.6031 obdclass.ko obdclass.ko key_fini > 1745169 1.5807 lnet.ko lnet.ko LNetMDBind > 1646427 1.4913 vmlinux vmlinux __wake_up > 1573871 1.4255 obdclass.ko obdclass.ko keys_fill > 1556594 1.4099 atl1e.ko atl1e.ko atl1e_intr > 1546613 1.4008 vmlinux vmlinux __list_add > 1512194 1.3697 ptlrpc.ko ptlrpc.ko > ptlrpc_server_handle_request > 1398966 1.2671 vmlinux vmlinux list_del > 1254363 1.1361 lvfs.ko lvfs.ko > lprocfs_counter_sub > 1206541 1.0928 vmlinux vmlinux > try_to_wake_up > 1188367 1.0764 obdclass.ko obdclass.ko keys_fini > 1158540 1.0494 obdclass.ko obdclass.ko > class_handle2object > 1032197 0.9349 ptlrpc.ko ptlrpc.ko > ptlrpc_server_drop_request > 1027855 0.9310 vmlinux vmlinux > thread_return > 1013642 0.9181 vmlinux vmlinux tcp_sendmsg > 978190 0.8860 vmlinux vmlinux > copy_user_generic_unrolled > 961920 0.8713 lnet.ko lnet.ko LNetPut > 929284 0.8417 obdclass.ko obdclass.ko > lu_context_init > 911345 0.8255 vmlinux vmlinux > kmem_cache_free > 879200 0.7963 vmlinux vmlinux mwait_idle > 843759 0.7642 atl1e.ko atl1e.ko > atl1e_xmit_frame > 808896 0.7327 vmlinux vmlinux __kmalloc > 806832 0.7308 ksocklnd.ko ksocklnd.ko > ksocknal_receive > 762878 0.6910 ptlrpc.ko ptlrpc.ko > ptlrpc_at_add_timed > 753020 0.6821 ptlrpc.ko ptlrpc.ko > lustre_msg_buf_v2 > 731792 0.6628 vmlinux vmlinux > __wake_up_common > 680931 0.6168 vmlinux vmlinux tcp_recvmsg > 674703 0.6111 vmlinux vmlinux > remove_wait_queue > 649026 0.5879 vmlinux vmlinux > skb_release_head_state > 648489 0.5874 mdt.ko mdt.ko > mdt_handle_common > 620536 0.5621 ptlrpc.ko ptlrpc.ko > request_in_callback > 616810 0.5587 vmlinux vmlinux memcpy > 605268 0.5482 vmlinux vmlinux > find_busiest_group > 588503 0.5330 vmlinux vmlinux __switch_to > 584848 0.5297 obdclass.ko obdclass.ko > lu_context_key_get > 566552 0.5132 atl1e.ko atl1e.ko > atl1e_clean_tx_irq > 527686 0.4780 vmlinux vmlinux tcp_v4_rcv > 524791 0.4753 ksocklnd.ko ksocklnd.ko > ksocknal_find_conn_locked > 495425 0.4487 ptlrpc.ko ptlrpc.ko > reply_out_callback > 482759 0.4373 vmlinux vmlinux > kmem_cache_alloc > 459759 0.4164 ptlrpc.ko ptlrpc.ko > ptlrpc_send_reply > 457681 0.4145 obdclass.ko obdclass.ko > htable_lookup > 438283 0.3970 ksocklnd.ko ksocklnd.ko > ksocknal_transmit > 432296 0.3916 ksocklnd.ko ksocklnd.ko > ksocknal_scheduler > 426023 0.3859 lnet.ko lnet.ko > lnet_md_unlink > 422453 0.3826 vmlinux vmlinux __down_read > 416777 0.3775 vmlinux vmlinux > dev_queue_xmit > 390667 0.3538 ptlrpc.ko ptlrpc.ko > __req_capsule_get > 389829 0.3531 lnet.ko lnet.ko lnet_send > 385353 0.3490 ptlrpc.ko ptlrpc.ko > lustre_pack_reply_v2 > 379540 0.3438 vmlinux vmlinux > prepare_to_wait > 375104 0.3398 vmlinux vmlinux > tcp_transmit_skb > 365540 0.3311 ksocklnd.ko ksocklnd.ko > ksocknal_alloc_tx > 363096 0.3289 vmlinux vmlinux memmove > 349697 0.3167 lnet.ko lnet.ko > lnet_complete_msg_locked > 344320 0.3119 ptlrpc.ko ptlrpc.ko > sptlrpc_wireflavor2policy > 334530 0.3030 obdclass.ko obdclass.ko > lu_context_exit > 333776 0.3023 vmlinux vmlinux > effective_prio > 332232 0.3009 ptlrpc.ko ptlrpc.ko at_measured > 328450 0.2975 vmlinux vmlinux > skb_release_data > 326971 0.2962 obdclass.ko obdclass.ko > lu_object_put > 323959 0.2934 ptlrpc.ko ptlrpc.ko > ptlrpc_server_free_request > 322772 0.2924 vmlinux vmlinux tcp_ack > 322667 0.2923 lnet.ko lnet.ko LNetMEAttach > 313380 0.2838 mdt.ko mdt.ko > mdt_getattr_internal > 311371 0.2820 vmlinux vmlinux __alloc_skb > 308274 0.2792 ptlrpc.ko ptlrpc.ko > ldlm_resource_find > 306783 0.2779 lnet.ko lnet.ko > lnet_return_credits_locked > 303357 0.2748 vmlinux vmlinux > ip_route_input > 302618 0.2741 obdclass.ko obdclass.ko > lu_context_fini > 297917 0.2698 vmlinux vmlinux enqueue_task > 291245 0.2638 lnet.ko lnet.ko > lnet_recv_put > 289224 0.2620 vmlinux vmlinux __qdisc_run > 288297 0.2611 osd_ldiskfs.ko osd_ldiskfs.ko osd_attr_get > 283552 0.2568 ptlrpc.ko ptlrpc.ko > lock_res_and_lock > 282665 0.2560 ptlrpc.ko ptlrpc.ko > target_pack_pool_reply > 270775 0.2453 vmlinux vmlinux > ip_queue_xmit > 269148 0.2438 lnet.ko lnet.ko > lnet_enq_event_locked > 266185 0.2411 vmlinux vmlinux > __find_get_block > 262071 0.2374 vmlinux vmlinux resched_task > 261675 0.2370 lnet.ko lnet.ko > lnet_match_md > 260547 0.2360 ip_tables.ko ip_tables.ko ipt_do_table > 257705 0.2334 vmlinux vmlinux > tcp_rcv_established > 250976 0.2273 ksocklnd.ko ksocklnd.ko > ksocknal_queue_tx_locked > 249668 0.2261 vmlinux vmlinux > local_bh_enable_ip > 248794 0.2253 vmlinux vmlinux ip_output > 245507 0.2224 ksocklnd.ko ksocklnd.ko > ksocknal_read_callback > 241913 0.2191 vmlinux vmlinux put_page > 238653 0.2162 vmlinux vmlinux > recalc_task_prio > 232245 0.2104 lnet.ko lnet.ko > lnet_lookup_cookie > 230806 0.2091 ptlrpc.ko ptlrpc.ko null_accept > 228674 0.2071 obdclass.ko obdclass.ko > class_handle_hash > 228161 0.2067 ptlrpc.ko ptlrpc.ko > sptlrpc_svc_alloc_rs > 228019 0.2065 ptlrpc.ko ptlrpc.ko > sptlrpc_target_export_check > 225989 0.2047 obdclass.ko obdclass.ko > lu_object_find_at > 221477 0.2006 oprofiled oprofiled > odb_update_node_with_offset > 213570 0.1934 ptlrpc.ko ptlrpc.ko > null_alloc_rs > 213379 0.1933 ptlrpc.ko ptlrpc.ko > req_capsule_has_field > 212791 0.1927 vmlinux vmlinux > tcp_cleanup_rbuf > 212646 0.1926 vmlinux vmlinux list_add > 211718 0.1918 ksocklnd.ko ksocklnd.ko > ksocknal_process_receive > 211447 0.1915 vmlinux vmlinux sock_wfree > 210107 0.1903 vmlinux vmlinux sched_clock > 207013 0.1875 ptlrpc.ko ptlrpc.ko > __req_capsule_offset > 206226 0.1868 ksocklnd.ko ksocklnd.ko > ksocknal_recv_iov > 204788 0.1855 vmlinux vmlinux > local_bh_disable > 202609 0.1835 libcfs.ko libcfs.ko > cfs_alloc_flags_to_gfp > 201355 0.1824 vmlinux vmlinux > do_gettimeofday > 200403 0.1815 ptlrpc.ko ptlrpc.ko > sptlrpc_svc_unwrap_request > 200133 0.1813 ksocklnd.ko ksocklnd.ko > ksocknal_data_ready > 196847 0.1783 ksocklnd.ko ksocklnd.ko > ksocknal_free_tx > 194325 0.1760 ptlrpc.ko ptlrpc.ko > ptlrpc_master_callback > 191462 0.1734 ptlrpc.ko ptlrpc.ko > ptlrpc_check_req > 191407 0.1734 lnet.ko lnet.ko > lnet_commit_md > 191155 0.1731 libcfs.ko libcfs.ko > lc_watchdog_touch > 190496 0.1725 mdt.ko mdt.ko mdt_getattr > 189965 0.1721 obdclass.ko obdclass.ko > class_export_put > 186893 0.1693 atl1e.ko atl1e.ko > atl1e_clean_rx_irq > 186683 0.1691 lnet.ko lnet.ko > lnet_find_peer_locked > 186143 0.1686 ldiskfs.ko ldiskfs.ko > ldiskfs_find_entry > 185352 0.1679 iptable_filter.ko iptable_filter.ko ipt_hook > 184570 0.1672 ptlrpc.ko ptlrpc.ko > ldlm_resource_get > 184553 0.1672 ksocklnd.ko ksocklnd.ko > ksocknal_find_peer_locked > 182066 0.1649 vmlinux vmlinux > add_wait_queue_exclusive > 179692 0.1628 vmlinux vmlinux cond_resched > 177681 0.1609 ksocklnd.ko ksocklnd.ko > ksocknal_tx_prep > 172136 0.1559 ptlrpc.ko ptlrpc.ko null_free_rs > 168889 0.1530 libstdc++.so.6.0.8 opreport /usr/lib64/ > libstdc++.so.6.0.8 > 167529 0.1517 ptlrpc.ko ptlrpc.ko > lustre_msg_get_opc > 166144 0.1505 ksocklnd.ko ksocklnd.ko > ksocknal_lib_recv_iov > 162940 0.1476 vmlinux vmlinux > recalc_sigpending_tsk > 161651 0.1464 ptlrpc.ko ptlrpc.ko > sptlrpc_svc_ctx_decref > 161047 0.1459 oprofiled oprofiled > find_kernel_image > 160557 0.1454 lnet.ko lnet.ko > lnet_nid2ni_locked > 159886 0.1448 ptlrpc.ko ptlrpc.ko > lustre_shrink_msg_v2 > 158071 0.1432 obdclass.ko obdclass.ko > class_export_get > 157994 0.1431 vmlinux vmlinux __up_read > 157418 0.1426 ldiskfs.ko ldiskfs.ko > __ldiskfs_get_inode_loc > 157222 0.1424 libcfs.ko libcfs.ko > cfs_block_sigs > 154417 0.1399 vmlinux vmlinux csum_partial > 153571 0.1391 lnet.ko lnet.ko LNetMDAttach > 149419 0.1353 vmlinux vmlinux > __activate_task > 149345 0.1353 vmlinux vmlinux > __tcp_push_pending_frames > 148491 0.1345 ksocklnd.ko ksocklnd.ko > ksocknal_launch_packet > 147683 0.1338 vmlinux vmlinux sock_recvmsg > 144704 0.1311 ptlrpc.ko ptlrpc.ko > ptlrpc_connection_addref > 144254 0.1307 obdclass.ko obdclass.ko > lu_object_find > 139774 0.1266 vmlinux vmlinux > pfifo_fast_enqueue > 139234 0.1261 lnet.ko lnet.ko > lnet_try_match_md > 138095 0.1251 vmlinux vmlinux > do_gettimeoffset_tsc > 137704 0.1247 ptlrpc.ko ptlrpc.ko > req_capsule_shrink > 137273 0.1243 vmlinux vmlinux > lock_timer_base > 137021 0.1241 vmlinux vmlinux release_sock > 135984 0.1232 mdt.ko mdt.ko > mdt_unpack_req_pack_rep > 135442 0.1227 vmlinux vmlinux > generic_getxattr > 132666 0.1202 obdclass.ko obdclass.ko > class_connected_export > 131849 0.1194 ptlrpc.ko ptlrpc.ko > ptlrpc_buf_need_swab > 131417 0.1190 libcfs.ko libcfs.ko cfs_alloc > 131039 0.1187 ptlrpc.ko ptlrpc.ko > target_send_reply > 131009 0.1187 libcfs.ko libcfs.ko cfs_hash_add > 130974 0.1186 vmlinux vmlinux __mod_timer > 128345 0.1162 osd_ldiskfs.ko osd_ldiskfs.ko > osd_object_read_lock > 126437 0.1145 vmlinux vmlinux netif_rx > 125451 0.1136 vmlinux vmlinux > sock_common_recvmsg > 124448 0.1127 ptlrpc.ko ptlrpc.ko > ldlm_lock_put > 123629 0.1120 lnet.ko lnet.ko > lnet_post_send_locked > 120179 0.1089 ptlrpc.ko ptlrpc.ko > lustre_free_reply_state > 119279 0.1080 ptlrpc.ko ptlrpc.ko ptl_send_buf > 118334 0.1072 vmlinux vmlinux > default_wake_function > 117874 0.1068 ptlrpc.ko ptlrpc.ko > sptlrpc_svc_wrap_reply > 117338 0.1063 vmlinux vmlinux dequeue_task > 116492 0.1055 opreport opreport > extra_images::matcher::operator()(std::string const&) const > 115787 0.1049 vmlinux vmlinux > tcp_current_mss > 114841 0.1040 mdt.ko mdt.ko > mdt_regular_handle > 114397 0.1036 vmlinux vmlinux > __copy_skb_header > 113796 0.1031 ptlrpc.ko ptlrpc.ko > ptlrpc_server_post_idle_rqbds > 112548 0.1019 vmlinux vmlinux > _atomic_dec_and_lock > 112275 0.1017 vmlinux vmlinux > find_first_bit > 112062 0.1015 vmlinux vmlinux > find_next_bit > 109904 0.0995 vmlinux vmlinux skb_clone > 109041 0.0988 libcfs.ko libcfs.ko > lc_watchdog_disable > 107246 0.0971 ptlrpc.ko ptlrpc.ko > __lustre_unpack_msg > 106395 0.0964 ptlrpc.ko ptlrpc.ko > __req_format_is_sane > 105582 0.0956 mdt.ko mdt.ko > mdt_shrink_reply > 104623 0.0948 ptlrpc.ko ptlrpc.ko > ptlrpc_update_export_timer > 103974 0.0942 vmlinux vmlinux > __rcu_process_callbacks > 102272 0.0926 lnet.ko lnet.ko lib_md_build > 100840 0.0913 lnet.ko lnet.ko > lnet_md_deconstruct > 100757 0.0913 ptlrpc.ko ptlrpc.ko > lustre_msg_size_v2 > 99939 0.0905 vmlinux vmlinux ip_rcv > 99033 0.0897 ptlrpc.ko ptlrpc.ko > ldlm_lock_create > 98275 0.0890 vmlinux vmlinux mod_timer > 97811 0.0886 vmlinux vmlinux kfree_skbmem > 96558 0.0875 mdd.ko mdd.ko mdd_get_md > 96390 0.0873 ptlrpc.ko ptlrpc.ko > req_capsule_init_area > 95672 0.0867 mdt.ko mdt.ko > mdt_check_ucred > 95263 0.0863 lnet.ko lnet.ko lnet_ni_recv > 94561 0.0856 mdt.ko mdt.ko > mdt_reint_open > 93791 0.0850 mdd.ko mdd.ko > mdd_read_lock > 93714 0.0849 osd_ldiskfs.ko osd_ldiskfs.ko > osd_object_auth > 90762 0.0822 vmlinux vmlinux > IRQ0x4a_interrupt > 90128 0.0816 ksocklnd.ko ksocklnd.ko > ksocknal_recv > 89651 0.0812 obdclass.ko obdclass.ko > lu_time_key_exit > 89647 0.0812 vmlinux vmlinux > netif_receive_skb > 88498 0.0802 mdt.ko mdt.ko > mdt_pack_attr2body > 87280 0.0791 ksocklnd.ko ksocklnd.ko > ksocknal_new_packet > 86200 0.0781 mdt.ko mdt.ko > mdt_lock_handle_init > 84561 0.0766 ptlrpc.ko ptlrpc.ko > req_capsule_filled_sizes > 84364 0.0764 ksocklnd.ko ksocklnd.ko > ksocknal_send > 83359 0.0755 vmlinux vmlinux nf_iterate > 82758 0.0750 ptlrpc.ko ptlrpc.ko > ldlm_inodebits_compat_queue > 82041 0.0743 oprofiled oprofiled sfile_find > 81178 0.0735 ksocklnd.ko ksocklnd.ko > ksocknal_match_tx_v3 > 80916 0.0733 vmlinux vmlinux > xattr_resolve_name > 80849 0.0732 mdd.ko mdd.ko > mdd_ucred_key_fini > 80835 0.0732 osd_ldiskfs.ko osd_ldiskfs.ko > osd_xattr_get > 80829 0.0732 lustre.ko lustre.ko > vvp_session_key_fini > 80522 0.0729 mdt.ko mdt.ko > mdt_object_lock > 80471 0.0729 vmlinux vmlinux > process_backlog > 80218 0.0727 ptlrpc.ko ptlrpc.ko __req_msg > 80150 0.0726 vmlinux vmlinux > alternate_node_alloc > 79896 0.0724 lvfs.ko lvfs.ko > upcall_cache_get_entry > 78040 0.0707 ptlrpc.ko ptlrpc.ko > ldlm_handle_enqueue0 > 77957 0.0706 ptlrpc.ko ptlrpc.ko > ldlm_lock_remove_from_lru > 77256 0.0700 vmlinux vmlinux > tcp_event_data_recv > 77006 0.0697 vmlinux vmlinux lock_sock > 75922 0.0688 mdd.ko mdd.ko > mdd_attr_get_internal > 74796 0.0677 ptlrpc.ko ptlrpc.ko > target_committed_to_req > 74788 0.0677 ksocklnd.ko ksocklnd.ko > ksocknal_process_transmit > 74656 0.0676 cmm.ko cmm.ko cml_attr_get > 74576 0.0675 ksocklnd.ko ksocklnd.ko > ksocknal_next_tx_carrier > 73666 0.0667 lnet.ko lnet.ko lnet_ni_send > 73296 0.0664 ksocklnd.ko ksocklnd.ko > ksocknal_tx_done > 73023 0.0661 lnet.ko lnet.ko > lnet_setpayloadbuffer > 72641 0.0658 osd_ldiskfs.ko osd_ldiskfs.ko > lu_device_is_osd > 72051 0.0653 vmlinux vmlinux > skb_copy_datagram_iovec > 71925 0.0651 ptlrpc.ko ptlrpc.ko > ptlrpc_server_allow_normal > 70839 0.0642 vmlinux vmlinux > local_bh_enable > 70244 0.0636 vmlinux vmlinux sock_sendmsg > 70131 0.0635 vmlinux vmlinux get_task_mm > 69625 0.0631 opreport opreport (anonymous > namespace)::module_matcher::operator()(std::string const&) const > 69573 0.0630 vmlinux vmlinux > main_timer_handler > 68370 0.0619 mdd.ko mdd.ko > mdd_iattr_get > 67955 0.0616 ptlrpc.ko ptlrpc.ko > ptlrpc_connection_put > 67875 0.0615 ptlrpc.ko ptlrpc.ko > ldlm_pool_add > 66775 0.0605 mdd.ko mdd.ko > __mdd_lmm_get > 66361 0.0601 vmlinux vmlinux > __sk_dst_check > 66330 0.0601 ksocklnd.ko ksocklnd.ko > ksocknal_pack_msg_v2 > 66036 0.0598 oprofiled oprofiled > pop_buffer_value > 65776 0.0596 mdd.ko mdd.ko > mdd_capainfo_key_fini > 64314 0.0583 lnet.ko lnet.ko > lnet_match_blocked_msg > 64257 0.0582 vmlinux vmlinux > recalc_sigpending > 64184 0.0581 ptlrpc.ko ptlrpc.ko > unlock_res_and_lock > 64009 0.0580 vmlinux vmlinux > tcp_check_space > 63851 0.0578 libc-2.5.so top _IO_vfscanf > 62692 0.0568 vmlinux vmlinux > tcp_rcv_space_adjust > 62667 0.0568 ldiskfs.ko ldiskfs.ko > ldiskfs_xattr_get > 62548 0.0567 ptlrpc.ko ptlrpc.ko > req_capsule_extend > 61313 0.0555 mdd.ko mdd.ko md_capainfo > 61159 0.0554 libcfs.ko libcfs.ko > cfs_waitq_active > 59511 0.0539 vmlinux vmlinux __do_IRQ > 59201 0.0536 ptlrpc.ko ptlrpc.ko > req_capsule_init > 58802 0.0533 vmlinux vmlinux > interleave_nodes > 58475 0.0530 ptlrpc.ko ptlrpc.ko > lustre_shrink_msg > 58425 0.0529 mdt.ko mdt.ko > mdt_object_find > 57699 0.0523 ksocklnd.ko ksocklnd.ko > ksocknal_lib_send_iov > 57284 0.0519 ptlrpc.ko ptlrpc.ko > sptlrpc_svc_free_rs > 57257 0.0519 oprofile.ko oprofile.ko sync_buffer > 57155 0.0518 lnet.ko lnet.ko > lnet_extract_iov > 56945 0.0516 lnet.ko lnet.ko > lnet_peer_alive_locked > 56904 0.0515 osc.ko osc.ko > osc_session_init > 56737 0.0514 mdd.ko mdd.ko mdd_la_get > 56521 0.0512 mdt.ko mdt.ko > mdt_body_reverse_idmap > 56498 0.0512 osd_ldiskfs.ko osd_ldiskfs.ko osd_dev > 56151 0.0509 lov.ko lov.ko > lov_session_key_init > 55877 0.0506 vmlinux vmlinux > __write_lock_failed > 55655 0.0504 ldiskfs.ko ldiskfs.ko > ldiskfs_get_blocks_handle > 55472 0.0502 ptlrpc.ko ptlrpc.ko > ldlm_cli_enqueue_local > 54412 0.0493 vmlinux vmlinux > ip_local_deliver > 52606 0.0476 ptlrpc.ko ptlrpc.ko > ptlrpc_server_request_pending > 51767 0.0469 vmlinux vmlinux __do_softirq > 51562 0.0467 ptlrpc.ko ptlrpc.ko > req_capsule_field_present > 51549 0.0467 vmlinux vmlinux inet_sendmsg > 51414 0.0466 mdt.ko mdt.ko > mdt_intent_policy > 51296 0.0465 ksocklnd.ko ksocklnd.ko > ksocknal_queue_tx_msg_v2 > 51009 0.0462 libcfs.ko libcfs.ko > cfs_hash_rlock > 50977 0.0462 vmlinux vmlinux __brelse > 50811 0.0460 oprofile.ko oprofile.ko > add_event_entry > ... > > With regards, > Popova Ekaterina > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20110720/081a2cfc/attachment-0001.html