Dear Experts, We are running lustre 2.4.1 with a combined MDT/MGS disk-server mounted with 4 device-mappers as 4 OSTs. Recently the setup suffered from high system load and long hang when trying to lfs df -h from client. could someone shed light on the situation? Any help would be greatly appreciated $ sudo tail -n 100 /var/log/messages Jun 5 14:56:02 disk-server kernel: Lustre: lustre-OST0001: recovery is timed out, evict stale exports Jun 5 14:56:45 disk-server kernel: LustreError: 58949:0:(ldlm_resource.c:1165:ldlm_resource_get()) lustre-OST0001: lvbo_init failed for resource 0xc4:0x0: rc = -2 Jun 5 14:57:27 disk-server kernel: Lustre: lustre-OST0001: Client 7e1e6422-c17b-841e-1703-36dda2141083 (at 192.168.1.61@o2ib) reconnecting, waiting for 4 clients in recovery for 0:42 Jun 5 14:59:54 disk-server kernel: Lustre: lustre-OST0000: Not available for connect from 192.168.1.61@o2ib (stopping) Jun 5 15:02:00 disk-server kernel: LustreError: 11-0: lustre-OST0000-osc-MDT0000: Communicating with 0@lo, operation ost_connect failed with -19. Jun 5 15:02:17 disk-server kernel: LustreError: 4943:0:(ofd_obd.c:1338:ofd_create()) lustre-OST0002: unable to precreate: rc = -30 Jun 5 15:02:17 disk-server kernel: LustreError: 39979:0:(osp_precreate.c:484:osp_precreate_send()) lustre-OST0002-osc-MDT0000: can't precreate: rc = -30 Jun 5 15:02:17 disk-server kernel: LustreError: 39979:0:(osp_precreate.c:484:osp_precreate_send()) lustre-OST0002-osc-MDT0000: can't precreate: rc = -30 Jun 5 15:02:17 disk-server kernel: LustreError: 39979:0:(osp_precreate.c:989:osp_precreate_thread()) lustre-OST0002-osc-MDT0000: cannot precreate objects: rc = -30 Jun 5 15:04:59 disk-server kernel: LustreError: 4950:0:(tgt_lastrcvd.c:577:tgt_client_new()) lustre-OST0002: Failed to write client lcd at idx 2, rc -30 Jun 5 15:06:35 disk-server kernel: Lustre: lustre-OST0001: recovery is timed out, evict stale exports Jun 5 15:07:18 disk-server kernel: LustreError: 58949:0:(ldlm_resource.c:1165:ldlm_resource_get()) lustre-OST0001: lvbo_init failed for resource 0xc4:0x0: rc = -2 Jun 5 15:08:00 disk-server kernel: Lustre: lustre-OST0001: Client 7e1e6422-c17b-841e-1703-36dda2141083 (at 192.168.1.61@o2ib) reconnecting, waiting for 4 clients in recovery for 0:42 Jun 5 15:09:54 disk-server kernel: Lustre: lustre-OST0000: Not available for connect from 192.168.1.61@o2ib (stopping) some infromation [user@disk-server]$ uname -r 2.6.32-358.18.1.el6_lustre.x86_64 [user@disk-server]$ rpm -qa|grep lustre kernel-2.6.32-358.18.1.el6_lustre.x86_64 lustre-iokit-1.4.0-1.noarch lustre-ldiskfs-4.1.0-2.6.32_358.18.1.el6_lustre.x86_64.x86_64 lustre-debuginfo-2.4.1-2.6.32_358.18.1.el6_lustre.x86_64.x86_64 lustre-2.4.1-2.6.32_358.18.1.el6_lustre.x86_64.x86_64 lustre-tests-2.4.1-2.6.32_358.18.1.el6_lustre.x86_64.x86_64 lustre-modules-2.4.1-2.6.32_358.18.1.el6_lustre.x86_64.x86_64 lustre-ldiskfs-debuginfo-4.1.0-2.6.32_358.18.1.el6_lustre.x86_64.x86_64 lustre-osd-ldiskfs-2.4.1-2.6.32_358.18.1.el6_lustre.x86_64.x86_64 kernel-firmware-2.6.32-358.18.1.el6_lustre.x86_64 [user@disk-server]$ lctl dl 0 UP osd-ldiskfs lustre-MDT0000-osd lustre-MDT0000-osd_UUID 12 1 UP mgc MGC192.168.1.254@o2ib 48090e07-ab8e-54f2-0d40-1bf99a31c774 5 2 UP ost OSS OSS_uuid 3 3 UP mgs MGS MGS 13 4 UP mgc MGC117.103.97.241@tcp 9442eac9-407e-56fd-3b6b-91e3e39beb97 5 5 UP mds MDS MDS_uuid 3 6 UP lod lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4 7 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 17 8 UP mdd lustre-MDD0000 lustre-MDD0000_UUID 4 9 UP qmt lustre-QMT0000 lustre-QMT0000_UUID 4 10 UP osp lustre-OST0001-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 11 UP osd-ldiskfs lustre-OST0000-osd lustre-OST0000-osd_UUID 5 12 ST obdfilter lustre-OST0000 lustre-OST0000_UUID 5 13 UP lwp lustre-MDT0000-lwp-OST0000 lustre-MDT0000-lwp-OST0000_UUID 5 14 UP osp lustre-OST0002-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 15 UP osp lustre-OST0003-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 16 UP osp lustre-OST0000-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 17 UP lwp lustre-MDT0000-lwp-MDT0000 lustre-MDT0000-lwp-MDT0000_UUID 5 18 UP osd-ldiskfs lustre-OST0002-osd lustre-OST0002-osd_UUID 5 19 UP obdfilter lustre-OST0002 lustre-OST0002_UUID 9 20 UP lwp lustre-MDT0000-lwp-OST0002 lustre-MDT0000-lwp-OST0002_UUID 5 21 UP osd-ldiskfs lustre-OST0001-osd lustre-OST0001-osd_UUID 5 22 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 11 23 UP lwp lustre-MDT0000-lwp-OST0001 lustre-MDT0000-lwp-OST0001_UUID 5 24 UP osd-ldiskfs lustre-OST0003-osd lustre-OST0003-osd_UUID 5 25 UP obdfilter lustre-OST0003 lustre-OST0003_UUID 11 26 UP lwp lustre-MDT0000-lwp-OST0003 lustre-MDT0000-lwp-OST0003_UUID 5 _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss