search for: ptlrpc

...first. > > But we are still having problems.. I''m sorry to > say. we have an I/O server that is fine until > we start Lustre. It starts spewing lustre call traces : > > Call > Trace:<ffffffffa02fa089>{:libcfs:lcw_update_time+22} > <ffffffffa03e06e3>{:ptlrpc:ptlrpc_main+1408} > <ffffffff8013327d>{default_wake_function+0} > <ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0} > <ffffffffa03e0156>{:ptlrpc:ptlrpc_retry_rqbds+0} > <ffffffff80110ebb>{child_rip+8} > <ffffffffa03e0163>{:ptlrpc:ptlr...

on our cluster that has been running lustre for about 1 month. I have 1 MDT/MGS and 1 OSS with 2 OST''s. Our cluster uses all Gige and has about 608 nodes 1854 cores. We have allot of jobs that die, and/or go into high IO wait, strace shows processes stuck in fstat(). The big problem is (i think) I would like some feedback on it that of these 608 nodes 209 of them have in dmesg

...th lustre 1.6.4.2 and infiniband. Under load, the clients hand about every 10 minutes which is really bad for a production machine. The only way to fix the hang is to reboot the server. My users are getting extremely impatient :-/ I see this on the clients- LustreError: 2814:0:(client.c:975:ptlrpc_expire_one_request()) @@@ timeout (sent at 1202756629, 301s ago) req at ffff8100af233600 x1796079/ t0 o6->data-OST0000_UUID at 192.168.64.71@o2ib:28 lens 336/336 ref 1 fl Rpc:/0/0 rc 0/-22 Lustre: data-OST0000-osc-ffff810139ce4800: Connection to service data- OST0000 via nid 192.168.64.71...

...fs/hash.o /usr/src/redhat/BUILD/lustre-2.0.0.1/libcfs/libcfs/hash.c: In function `cfs_hash_getref'': /usr/src/redhat/BUILD/lustre-2.0.0.1/libcfs/libcfs/hash.c:212: warning: implicit declaration of function `atomic_inc_not_zero'' CC [M] /usr/src/redhat/BUILD/lustre-2.0.0.1/lustre/ptlrpc/service.o /usr/src/redhat/BUILD/lustre-2.0.0.1/lustre/ptlrpc/service.c: In function `ptlrpc_at_check_timed'': /usr/src/redhat/BUILD/lustre-2.0.0.1/lustre/ptlrpc/service.c:1168: warning: implicit declaration of function `atomic_inc_not_zero'' I manged to overcome the first two error...

...t; the use count of the module is one, but I don''t see where it''s used. # umount /mnt/lustre # ifconfig ib0 down # modprobe -r ko2iblnd FATAL: Module ko2iblnd is in use. # lsmod | grep ko2 ko2iblnd 143136 1 lnet 258088 5 lustre,ksocklnd,ko2iblnd,ptlrpc,obdclass libcfs 189784 12 osc,mgc,lustre,lov,lquota,mdc,ksocklnd,ko2iblnd,ptlrpc,obdclass,lnet,lvf s rdma_cm 65940 4 ko2iblnd,ib_iser,rdma_ucm,ib_sdp ib_core 88576 16 ko2iblnd,ib_iser,rdma_ucm,ib_ucm,ib_srp,ib_sdp,rdma_cm,ib_cm,iw_cm,ib_lo cal_s...

...t on the export server can take a real pounding (ive seen it push 300MB/sec) so I don''t know why nfs is crashing it. On the nfs export server i see these messages-- Lustre: 4224:0:(o2iblnd_cb.c:412:kiblnd_handle_rx()) PUT_NACK from 192.168.64.70 at o2ib LustreError: 4400:0:(client.c:969:ptlrpc_expire_one_request()) @@@ timeout (sent at 1197415542, 100s ago) req at ffff810827bfbc00 x38827/t0 o36->data-MDT0000_UUID at 192.168.64.70@o2ib:12 lens 14256/672 ref 1 fl Rpc:/0/0 rc 0/-22 Lustre: data-MDT0000-mdc-ffff81082d702000: Connection to service data-MDT0000 via nid 192.168.64.70 at o2i...

...b64/lustre/tests/llmount.sh" I have tried to kill it first with -2 upto -9 but the process will not budge. Here is the top lines from perf top 37.39% [osc] [k] osc_set_info_async 27.14% [lov] [k] lov_set_info_async 4.13% [kernel] [k] kfree 3.57% [ptlrpc] [k] ptlrpc_set_destroy 3.14% [kernel] [k] mutex_unlock 3.10% [lustre] [k] ll_wr_max_cached_mb 3.00% [kernel] [k] mutex_lock 2.82% [ptlrpc] [k] ptlrpc_prep_set 2.52% [kernel] [k] __kmalloc Thanks, Andrew > > Also, j...

...>] :libcfs:lbug_with_loc+0x7a/0xc0 Jan 31 10:45:24 opteron-ren-11 kernel: [<ffffffff887c2a65>] :lustre:ll_file_flock+0x295/0x550 Jan 31 10:45:24 opteron-ren-11 kernel: [<ffffffff8025fa48>] do_page_fault+0x476/0x7b5 Jan 31 10:45:24 opteron-ren-11 kernel: [<ffffffff8868fba0>] :ptlrpc:ldlm_flock_completion_ast+0x0/0x690 Jan 31 10:45:24 opteron-ren-11 kernel: [<ffffffff802ac2ff>] audit_syscall_entry+0x141/0x174 Jan 31 10:45:24 opteron-ren-11 kernel: [<ffffffff802d39d6>] sys_flock+0x117/0x150 Jan 31 10:45:24 opteron-ren-11 kernel: [<ffffffff8025729c>] tracesys...

...> formalization formal reintegration model with "proofs" of recovery > correctness and concurrency control description > > C-reintegration reintegration, including concurrency control, 1000 > integration with ptlrpc > > S-compound implementation of the compound operations on 1000 > the server > > S-reintegration reintegration of batches on the server, thread 1000 > scheduling > > S-undo keeping u...

Hi All, Attached is a proposed patch, which should solve some issues with mixed endians. Unfortunately I was not able to access the latest code on the CVS to see what other fixes might be there. -------------- next part -------------- A non-text attachment was scrubbed... Name: pack_generic.diff Type: text/x-patch Size: 2816 bytes Desc: not available Url :

Hi All, Attached is a proposed patch, which should solve some issues with mixed endians. Unfortunately I was not able to access the latest code on the CVS to see what other fixes might be there. -------------- next part -------------- A non-text attachment was scrubbed... Name: pack_generic.diff Type: text/x-patch Size: 2816 bytes Desc: not available Url :

...t; > formalization formal reintegration model with "proofs" of recovery > correctness and concurrency control description > > C-reintegration reintegration, including concurrency control, 1000 > integration with ptlrpc > > S-compound implementation of the compound operations on 1000 > the server > > S-reintegration reintegration of batches on the server, thread 1000 > scheduling > > S-undo keeping undo...

...age repeated 188807 times Jul 7 14:45:11 com01 kernel: BUG: soft lockup - CPU#15 stuck for 10s! [ll_ost_118:12180]Jul 7 14:45:11 com01 kernel: CPU 15: Jul 7 14:45:11 com01 kernel: Modules linked in: obdfilter(U) fsfilt_ldiskfs(U) ost(U) mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) ldiskfs(U) crc16(U) autofs4(U) hidp(U) rfcomm(U) l2cap(U) bluetooth(U) sunrpc(U) dm_multipath(U) scsi_dh(U) video(U) hwmon(U) backlight(U) sbs(U) i2c_ec(U) i2c_core(U) button(U) battery(U) asus_acpi(U) acpi_memhotplug(U) ac(U) ipv6(U) xfrm_nalgo(U) crypto_ap...