search for: nfsd_dispatch

...op ffff9c8bfcaa8080 opcnt 3 #3: 9: status 0 Aug 30 14:16:12 hvmnas01 kernel: nfsv4 compound returned 0 Aug 30 14:16:12 hvmnas01 kernel: --> nfsd4_store_cache_entry slot ffff9c8bd7bba000 Aug 30 14:16:12 hvmnas01 kernel: renewing client (clientid 5d694c2d/12bb1419) Aug 30 14:16:12 hvmnas01 kernel: nfsd_dispatch: vers 4 proc 1 Aug 30 14:16:12 hvmnas01 kernel: nfsv4 compound op #1/3: 53 (OP_SEQUENCE) Aug 30 14:16:12 hvmnas01 kernel: __find_in_sessionid_hashtbl: 1567181869:314250265:6:0 Aug 30 14:16:12 hvmnas01 kernel: nfsd4_sequence: slotid 0 Aug 30 14:16:12 hvmnas01 kernel: check_slot_seqid enter. seqid 12...

Good morning, in order to post proper documentation, what logs (or log level) do I need to troubleshoot a Centos 7.6.1810 3.10.0-957.27.2.el7.x86_64 tha tis running a NFS server on top LVM on top of XFS on top of VDO on top of MDAM on a 6 SSD disk RAID6 ? This physical NFS server is servign 2 NFS v4.2 shares to 2 physical KVM virtualization hosts. When remote NFS clients start doing intensive

...t_this_handle+205/368] [journal_start+149/196] [ext3_dirty_inode+116/268] [__mark_inode_dirty+50/168] [update_atime+75/80] [do_generic_file_read+1158/1172] [generic_file_read+147/400] [file_read_actor+0/224] [nfsd_read+477/636] [nfsd3_proc_read+295/388] [nfsd_dispatch+207/406] [svc_process+655/1264] [nfsd+566/944] [arch_kernel_thread+40/56] This sleep_on is the one at line 136 of transcation.c(start_this_handle), where it is waiting for t_state to stop being T_LOCKED. The related kjournald is at: kjournald Call Trace: [sleep_on+75/124]...

...9>] __dentry_open+0x154/0x238 [79915.815307] [<ffffffffa048441d>] :nfsd:nfsd_open+0x141/0x175 [79915.815318] [<ffffffffa0484655>] :nfsd:nfsd_write+0xc5/0xe2 [79915.815332] [<ffffffffa048b5fc>] :nfsd:nfsd3_proc_write+0xe6/0x105 [79915.815344] [<ffffffffa0480245>] :nfsd:nfsd_dispatch+0xde/0x1b6 [79915.815373] [<ffffffffa03f25c5>] :sunrpc:svc_process+0x408/0x6e7 [79915.815378] [<ffffffff8042a3c4>] __down_read+0x12/0xa1 [79915.815387] [<ffffffffa048067c>] :nfsd:nfsd+0x0/0x2a4 [79915.815396] [<ffffffffa0480810>] :nfsd:nfsd+0x194/0x2a4 [79915.815401] [&...

...;] ? vfs_rename+0x3ab/0x440 [<ffffffffa03e3aca>] ? nfsd_rename+0x47a/0x4d0 [nfsd] [<ffffffffa03eb4f1>] ? nfsd3_proc_rename+0xd1/0x1a0 [nfsd] [<ffffffffa03ecb85>] ? decode_fh+0x55/0x80 [nfsd] [<ffffffffa03ecddc>] ? decode_filename+0x1c/0x70 [nfsd] [<ffffffffa03dd43e>] ? nfsd_dispatch+0xfe/0x240 [nfsd] [<ffffffffa033b614>] ? svc_process_common+0x344/0x640 [sunrpc] [<ffffffff81063330>] ? default_wake_function+0x0/0x20 [<ffffffffa033bc50>] ? svc_process+0x110/0x160 [sunrpc] [<ffffffffa03ddb62>] ? nfsd+0xc2/0x160 [nfsd] [<ffffffffa03ddaa0>] ? nfsd+0x0/...

...e+0x11f/0x380 Aug 6 15:22:05 adams kernel: [<c011bc9a>] default_wake_function+0x2a/0x30 Aug 6 15:22:05 adams kernel: [<c011bcda>] __wake_up_common+0x3a/0x70 Aug 6 15:22:05 adams kernel: [<c01d8808>] nfsd_proc_write+0xa8/0x130 Aug 6 15:22:05 adams kernel: [<c01d7818>] nfsd_dispatch+0xe8/0x1f5 Aug 6 15:22:05 adams kernel: [<c01d7730>] nfsd_dispatch+0x0/0x1f5 Aug 6 15:22:05 adams kernel: [<c03b8120>] svc_process+0x480/0x64c Aug 6 15:22:05 adams kernel: [<c01d747b>] nfsd+0x26b/0x520 Aug 6 15:22:05 adams kernel: [<c010b356>] work_resched+0x5/0x16 A...

...;inet_sendmsg+0/90> Trace; 8017020d <net_bh+175/1d0> Trace; 880c32c8 <[nfsd]nfscache+18768/1e01f> Trace; 880aa540 <[nfsd]nfsd_procedures3+e0/2c0> Trace; 880a41b3 <[nfsd]nfsd3_proc_write+bb/d4> Trace; 880aa540 <[nfsd]nfsd_procedures3+e0/2c0> Trace; 8809c3f1 <[nfsd]nfsd_dispatch+d5/164> Trace; 880aa540 <[nfsd]nfsd_procedures3+e0/2c0> Trace; 88085958 <[sunrpc]svc_process+2bc/574> Trace; 880aa400 <[nfsd]nfsd_svcstats+0/40> Trace; 880a9edc <[nfsd]nfsd_version3+0/10> Trace; 8809c229 <[nfsd]nfsd+ed/1e0> Trace; 80109213 <kernel_thread+23/30&gt...

...lt;ffffffff88dddccf>] :nfsd:nfsd_open+0x20/0x184 Jun 7 19:45:21 sraid3 kernel: [<ffffffff88dddffb>] :nfsd:nfsd_write+0x89/0xd5 Jun 7 19:45:21 sraid3 kernel: [<ffffffff88de4b1a>] :nfsd:nfsd3_proc_write+0xea/0x109 Jun 7 19:45:21 sraid3 kernel: [<ffffffff88dda1db>] :nfsd:nfsd_dispatch+0xd8/0x1d6 Jun 7 19:45:21 sraid3 kernel: [<ffffffff88d5a651>] :sunrpc:svc_process+0x454/0x71b Jun 7 19:45:21 sraid3 kernel: [<ffffffff80064644>] __down_read+0x12/0x92 Jun 7 19:45:21 sraid3 kernel: [<ffffffff88dda5a1>] :nfsd:nfsd+0x0/0x2cb Jun 7 19:45:21 sraid3 kernel: [&l...

...kernel: [<ffffffff80047fc8>] vfs_unlink+0xc2/0x108 Dec 11 18:46:33 cpu3 kernel: [<ffffffff8857c57a>] :nfsd:nfsd_unlink+0x1de/0x24b Dec 11 18:46:33 cpu3 kernel: [<ffffffff88583e9a>] :nfsd:nfsd3_proc_remove+0xa8/0xb5 Dec 11 18:46:33 cpu3 kernel: [<ffffffff885791c4>] :nfsd:nfsd_dispatch+0xd7/0x198 Dec 11 18:46:33 cpu3 kernel: [<ffffffff88488514>] :sunrpc:svc_process+0x44d/0x70b Dec 11 18:46:33 cpu3 kernel: [<ffffffff800625bf>] __down_read+0x12/0x92 Dec 11 18:46:33 cpu3 kernel: [<ffffffff8857954d>] :nfsd:nfsd+0x0/0x2db Dec 11 18:46:33 cpu3 kernel: [<fffffff...

...lt;c01079f8>] __down_failed [kernel] 0x8 [<f8a8a00c>] .text.lock.vfs [nfsd] 0x5 [<c0118c39>] __wake_up [kernel] 0x49 [<f8a8d5e8>] nfsd3_proc_lookup [nfsd] 0xd8 [<f8a96b0c>] nfsd_procedures3 [nfsd] 0x6c [<f8a96b0c>] nfsd_procedures3 [nfsd] 0x6c [<f8a84667>] nfsd_dispatch [nfsd] 0xb7 [<f8a14047>] svc_process_Rsmp_6ad37799 [sunrpc] 0x347 [<f8a9645c>] nfsd_version3 [nfsd] 0x0 [<f8a9647c>] nfsd_program [nfsd] 0x0 [<f8a84442>] nfsd [nfsd] 0x252 [<f8a96444>] nfsd_list [nfsd] 0x0 [<f8a841f0>] nfsd [nfsd] 0x0 [<c0107286>] ke...

...9902e7>] svcauth_unix_set_client+0xa7/0xb5 [sunrpc] Feb 12 05:52:03 tier2-home kernel: [<f8c93351>] nfsd3_proc_lookup+0xa9/0xb3 [nfsd] Feb 12 05:52:03 tier2-home kernel: [<f8c952aa>] nfs3svc_decode_diropargs+0x0/0xfa [nfsd] Feb 12 05:52:03 tier2-home kernel: [<f8c896a2>] nfsd_dispatch+0xba/0x170 [nfsd] Feb 12 05:52:03 tier2-home kernel: [<f898d459>] svc_process+0x41b/0x6ce [sunrpc] Feb 12 05:52:03 tier2-home kernel: [<f8c89482>] nfsd+0x1cc/0x332 [nfsd] Feb 12 05:52:03 tier2-home kernel: [<f8c892b6>] nfsd+0x0/0x332 [nfsd] Feb 12 05:52:03 tier2-home kernel:...

...04:05:54 datastore01 kernel: [1210991.390878] [<ffffffffa0557609>] nfsd_write+0xf9/0x110 [nfsd] Apr 13 04:05:54 datastore01 kernel: [1210991.390886] [<ffffffffa055e93c>] nfsd3_proc_write+0xbc/0x150 [nfsd] Apr 13 04:05:54 datastore01 kernel: [1210991.390891] [<ffffffffa0550dab>] nfsd_dispatch+0xeb/0x230 [nfsd] Apr 13 04:05:54 datastore01 kernel: [1210991.390907] [<ffffffffa048a7c8>] svc_process_common+0x328/0x6d0 [sunrpc] Apr 13 04:05:54 datastore01 kernel: [1210991.390917] [<ffffffffa048aebf>] svc_process+0xff/0x150 [sunrpc] Apr 13 04:05:54 datastore01 kernel: [1210991.39...

...:08:24 server1 kernel: [<c0317627>] __cond_resched+0x14/0x3b Feb 4 00:08:24 server1 kernel: [<f8b40ee2>] nfsd3_proc_write+0xbf/0xd5 [nfsd] Feb 4 00:08:24 server1 kernel: [<f8b42f90>] nfs3svc_decode_writeargs+0x0/0x243 [nfsd] Feb 4 00:08:24 server1 kernel: [<f8b35947>] nfsd_dispatch+0xba/0x16f [nfsd] Feb 4 00:08:24 server1 kernel: [<f8ace74c>] svc_process+0x432/0x6e2 [sunrpc] Feb 4 00:08:24 server1 kernel: [<f8b355eb>] nfsd+0x2a7/0x549 [nfsd] Feb 4 00:08:24 server1 kernel: [<f8b35344>] nfsd+0x0/0x549 [nfsd] Feb 4 00:08:24 server1 kernel: [<c01041dd...