Lustre discuss - Apr 2008 - Inode count differences between client and MDS

Hi,

The MDT and the client don''t agree on the inode count. They do agree on
inode-used however. (see below)

Any reason for this ?

Thanks
/Jakob

mds:~# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
[snip]
/dev/lustre_pool/mgt   16384      17   16367    1% /mnt/lustreconf/mgt
/dev/lustre_pool/mdt 201326592   18722 201307870  1% /mnt/lustreconf/mdt


client1:~# df -i
Filesystem          Inodes   IUsed   IFree IUse% Mounted on
[snip]
mds:/lustre       75380878   18722 75362156    1% /mnt/lustre

On Wed, Apr 09, 2008 at 11:20:04AM +0200, Jakob Goldbach
wrote:
> Hi,
> 
> The MDT and the client don''t agree on the inode count. They do
agree on
> inode-used however. (see below)
> 
> Any reason for this ?
The inode count you get on the client side also takes into account the
number of inodes available on the OSTs and the default stripe count.

Johann

On Thu, 2008-04-10 at 20:00 +0200, Johann Lombardi
wrote:
> The inode count you get on the client side also takes into account the
> number of inodes available on the OSTs and the default stripe count.
> 
The stripecount is 1. (from /proc/../lov/../stripecount) - The OSTs have
plenty ~400M inodes. The MDT has about ~200M (from tune2fs -l /dev/mdt).
Still, only 75M show up on the client. 

And even stranger - when creating new files - both used count AND
avail-count increases. (see below)

/Jakob  


client# lfs df  -i 
UUID                Inodes     IUsed     IFree IUse% Mounted on
foo-MDT0000_UUID  75382000     19935  75362065    0% /mnt/foo[MDT:0]
foo-OST0000_UUID 456794112      9603 456784509    0% /mnt/foo[OST:0]
foo-OST0001_UUID 456794112      9734 456784378    0% /mnt/foo[OST:1]


client# touch a b c
client# lfs df  -i
UUID                Inodes     IUsed     IFree IUse% Mounted on
foo-MDT0000_UUID  75382003     19938  75362065    0% /mnt/foo[MDT:0]
foo-OST0000_UUID 456794112      9603 456784509    0% /mnt/foo[OST:0]
foo-OST0001_UUID 456794112      9734 456784378    0% /mnt/foo[OST:1]

Hello!

On Apr 11, 2008, at 3:48 AM, Jakob Goldbach wrote:
>> The inode count you get on the client side also takes into account  
>> the
>> number of inodes available on the OSTs and the default stripe count.
> The stripecount is 1. (from /proc/../lov/../stripecount) - The OSTs  
> have
> plenty ~400M inodes. The MDT has about ~200M (from tune2fs -l /dev/ 
> mdt).
> Still, only 75M show up on the client.
There is another variable to this - we look at amount of free blocks  
(in assumption
of every file taking at least one block) and decrease avail number of  
inodes
if there are not enough free blocks to cover available inodes.
> And even stranger - when creating new files - both used count AND
> avail-count increases. (see below)
Because they are empty files, so they don;t consume any blocks yet.
What if you create them nonempty?

Bye,
     Oleg

Hi 

> There is another variable to this - we look at amount of free blocks  
> (in assumption
> of every file taking at least one block) and decrease avail number of  
> inodes
> if there are not enough free blocks to cover available inodes.
> 
I can see that the avail blocks on the MDS is close to avaiable onode
count - There is a low more on the OSS''es

But why is the MDS looked at here. The fileblocks are on the OSS - not
the MDS.

> Because they are empty files, so they don;t consume any blocks yet.
> What if you create them nonempty?
> 
Tried creating a small file. Same thing

client# df -i
Filesystem            Inodes   IUsed    IFree IUse% Mounted on
mds at tcp:/foo        75383282   21667 75361615    1% /mnt/foo

client# cp /etc/fstab myfile
client# df -i
Filesystem            Inodes   IUsed    IFree IUse% Mounted on
mds at tcp:/foo        75383283   21668 75361615    1% /mnt/foo

Typos:

On Mon, 2008-04-14 at 00:19 +0200, Jakob Goldbach wrote:
> I can see that the avail blocks on the MDS is close to avaiable onode
...available inode
> count - There is a low more on the OSS''es
...a lot more on the OSS.

Hello!

On Apr 13, 2008, at 6:19 PM, Jakob Goldbach wrote:
>> There is another variable to this - we look at amount of free blocks
>> (in assumption
>> of every file taking at least one block) and decrease avail number of
>> inodes
>> if there are not enough free blocks to cover available inodes.
> I can see that the avail blocks on the MDS is close to avaiable onode
> count - There is a low more on the OSS''es
Hm, looking at the code - indeed, we do this with MDS too, here is the
relevant comment:
  * We need to hack the return value for the free inode counts because
  * the current EA code requires one filesystem block per inode with  
EAs,
  * so it is possible to run out of blocks before we run out of inodes.
  *
  * This can be removed when the ext3 EA code is fixed.

This is no longer totally true, and only true if you have very wide
striping that does not fit into reserved EA space in inode.

Is this under-reporting breaking anything for you, I wonder?

Bye,
     Oleg

On Sun, 2008-04-13 at 19:46 -0400, Oleg Drokin wrote:
> Hm, looking at the code - indeed, we do this with MDS too, here is the
[snip]

Thanks for clarifying this, Oleg.
> Is this under-reporting breaking anything for you, I wonder?
> 
No - I was just worried what happened to all my inodes. I planned about
200M inodes on the MDS and less than half showed up on the clients.

Do you want me to file a bugreport on this ?

/Jakob

Hello!

On Apr 14, 2008, at 3:43 AM, Jakob Goldbach wrote:
>
>> Is this under-reporting breaking anything for you, I wonder?
>>
> No - I was just worried what happened to all my inodes. I planned  
> about
> 200M inodes on the MDS and less than half showed up on the clients.
> Do you want me to file a bugreport on this ?
I do not think this is serious enough to warrant a bug, personally.

Bye,
     Oleg