Josephine Palencia
2008-Jul-27 21:02 UTC
[Lustre-discuss] Adding OST''s not increasing storage?
Test Setup: (combining them first while ENOMEM error fixed for mgs hardware) ------ mds00w.psc.teragrid.org: combined mdt/mgs and client oss00w.psc.teragrid.org: ost1 oss00w.psc.teragrid.org: ost2 operon22.psc.edu : ost3 and client Combined mgs/mdt on mds00w: ----------------------------- root at mds00w ~]# df -h /dev/sda8 9.9G 489M 8.9G 6% /mnt/test/mdt Added 1 ost from oss00w with 1.4TB ------------------ [root at oss00w ~]# df -h /dev/sda7 1.3T 1.1G 1.3T 1% /mnt/test/ost0 Shows 1.3T storage correctly on mds00w as client. root at mds00w ~]# df -h mds00w at tcp0:/testfs 1.3T 1.1G 1.3T 1% /mnt/testfs Added 2nd ost from oss01w with 1.4TB: ------------------ [root at oss01w ~]# df -h /dev/sda7 1.3T 1.1G 1.3T 1% /mnt/test/ost1 mds00w shows both osts active ------------------- root at mds00w ~]# cat /proc/fs/lustre/lov/testfs-clilov-ffff81007c659000/target_obd 0: testfs-OST0000_UUID ACTIVE 1: testfs-OST0001_UUID ACTIVE [root at mds00w ~]# cat /proc/fs/lustre/lov/testfs-MDT0000-mdtlov/target_obd 0: testfs-OST0000_UUID ACTIVE 1: testfs-OST0001_UUID ACTIVE but df from mds00w shows only 1.3T storage available (other 1.3T in used?) Should be around 2.6T Adding a third ost with only 150GB storage gives 142GB storage showing on both clients. Total storage should be around 2.7TB. ------------- [root at operon22 ~]# df -h /dev/hdb1 151G 1.1G 142G 1% /mnt/test/ost3 mds00w at tcp0:/testfs 2.8T 2.6T 142G 95% /mnt/testfs [root at mds00w ~]# df -h /dev/sda8 9.9G 489M 8.9G 6% /mnt/test/mdt mds00w at tcp0:/testfs 2.8T 2.6T 142G 95% /mnt/testfs [root at mds00w ~]# cat /proc/fs/lustre/devices 0 UP mgs MGS MGS 11 1 UP mgc MGC128.182.112.60 at tcp e6af805d-1e32-b002-d315-54fb78e7e558 5 2 UP lov testfs-MDT0000-mdtlov testfs-MDT0000-mdtlov_UUID 4 3 UP mdt testfs-MDT0000 testfs-MDT0000_UUID 7 4 UP mds mdd_obd-testfs-MDT0000-0 mdd_obd_uuid-testfs-MDT0000-0 3 5 UP osc testfs-OST0000-osc-MDT0000 testfs-MDT0000-mdtlov_UUID 5 6 UP lov testfs-clilov-ffff8101278aec00 f327440b-f5e9-c9cc-66fc-7a6001402368 4 7 UP lmv testfs-clilmv-ffff8101278aec00 f327440b-f5e9-c9cc-66fc-7a6001402368 4 8 UP mdc testfs-MDT0000-mdc-ffff8101278aec00 f327440b-f5e9-c9cc-66fc-7a6001402368 5 9 UP osc testfs-OST0000-osc-ffff8101278aec00 f327440b-f5e9-c9cc-66fc-7a6001402368 5 10 UP osc testfs-OST0001-osc-ffff8101278aec00 f327440b-f5e9-c9cc-66fc-7a6001402368 5 11 UP osc testfs-OST0001-osc-MDT0000 testfs-MDT0000-mdtlov_UUID 5 12 UP osc testfs-OST0002-osc-ffff8101278aec00 f327440b-f5e9-c9cc-66fc-7a6001402368 5 13 UP osc testfs-OST0002-osc-MDT0000 testfs-MDT0000-mdtlov_UUID 5 Show all OST''s active. [root at mds00w ~]# cat /proc/fs/lustre/lov/testfs-clilov-ffff8101278aec00/target_obd 0: testfs-OST0000_UUID ACTIVE 1: testfs-OST0001_UUID ACTIVE 2: testfs-OST0002_UUID ACTIVE [root at mds00w ~]# cat /proc/fs/lustre/lov/testfs-MDT0000-mdtlov/target_obd 0: testfs-OST0000_UUID ACTIVE 1: testfs-OST0001_UUID ACTIVE 2: testfs-OST0002_UUID ACTIVE Encountering issues deactivating/activating OST''s as well but that''s in another email. Thanks, josephin PS. Version: lustre: 1.9.50 Kernel-2.6.18-92.1.6-lustre-1.9.50 #2 SMP x86_64
Andreas Dilger
2008-Jul-29 22:23 UTC
[Lustre-discuss] Adding OST''s not increasing storage?
On Jul 27, 2008 17:02 -0400, Josephine Palencia wrote:> Added 2nd ost from oss01w with 1.4TB: > ------------------ > [root at oss01w ~]# df -h > /dev/sda7 1.3T 1.1G 1.3T 1% /mnt/test/ost1 > > but df from mds00w shows only 1.3T storage available (other 1.3T in used?) > Should be around 2.6TThis is probably bug 14951. The OSTs are added, but the clients are not handling it quite correctly. Unmounting and remounting them will correct this until it is fixed correctly. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.