Lustre devel - Apr 2007 - [Bug 12365] New: Unable to fill OST

Please don''t reply to lustre-devel. Instead, comment in Bugzilla by
using the following link:
https://bugzilla.lustre.org/show_bug.cgi?id=12365



This is really weird.  Unfortunately, I can''t reproduce it.  It
occurred on a
clean filesystem of 3 loopback OSTs - all I did was run sanity test 78 a few
times (and it failed but shouldn''t have for similar reasons.)

[root@vm1 tests]# lfs setstripe /mnt/lustre/foo 0 -1 -1
[root@vm1 tests]# ./directio rdwr /mnt/lustre/foo/f78 0 50 1048576
directio on /mnt/lustre/foo/f78 for 50x1048576 bytes
Write error Success (rc = 48234496, len = 52428800)
[root@vm1 tests]# lctl df
usage: df <input> [output]
[root@vm1 tests]# lfs df /mnt/lustre
UUID                 1K-blocks      Used Available  Use% Mounted on
lustre-MDT0000_UUID      34984      6380     28604   18% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      46856     22808     24048   48% /mnt/lustre[OST:0]
lustre-OST0001_UUID      46856     22280     24576   47% /mnt/lustre[OST:1]
lustre-OST0002_UUID      46856     23304     23552   49% /mnt/lustre[OST:2]

filesystem summary:     140568     68392     72176   48% /mnt/lustre

Note that the write failed but none of the OSTs are full.  The console shows:

LustreError: 1418:0:(obd_class.h:928:obd_brw_rqset()) error from callback: rc =
-28

I was also able to reproduce the problem using dd:

[root@vm1 tests]# lfs df /mnt/lustre
UUID                 1K-blocks      Used Available  Use% Mounted on
lustre-MDT0000_UUID      34984      6372     28612   18% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      46856      6920     39936   14% /mnt/lustre[OST:0]
lustre-OST0001_UUID      46856      6920     39936   14% /mnt/lustre[OST:1]
lustre-OST0002_UUID      46856     41800      5056   89% /mnt/lustre[OST:2]

filesystem summary:     140568     55640     84928   39% /mnt/lustre

[root@vm1 tests]# dd if=/dev/zero of=/mnt/lustre/test1 bs=1M count=35
dd: writing `/mnt/lustre/test1'': No space left on device
21+0 records in
20+0 records out
[root@vm1 tests]# !lfs
lfs df /mnt/lustre
UUID                 1K-blocks      Used Available  Use% Mounted on
lustre-MDT0000_UUID      34984      6372     28612   18% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      46856      6920     39936   14% /mnt/lustre[OST:0]
lustre-OST0001_UUID      46856     27528     19328   58% /mnt/lustre[OST:1]
lustre-OST0002_UUID      46856     42760      4096   91% /mnt/lustre[OST:2]

filesystem summary:     140568     77208     63360   54% /mnt/lustre

Console shows:

LustreError: 30415:0:(client.c:574:ptlrpc_check_status()) @@@ type
=PTL_RPC_MSG_ERR, err == -28  req@ca2f0a00 x106735/t0
o4->lustre-OST0000_UUID@192.168.1.128@tcp:28 lens 384/352 ref 2 fl Rpc:R/0/0
rc
0/-28

I wanted to see if this affected v1_6_0_RC4, so I shut down Lustre and started
that and it worked.  Unfortunately, when I restarted HEAD, it also worked.  I
have been unable to reproduce this problem.