Gluster users - Aug 2012 - dht hashing based on basename only?

Dear experts,

we started to test gluster (3.3.0) on a small cluster grown over some
time (~200 cores and ~100 TB in heterogeneous hardware). Our data
typically are organised in a tree of directories containing a set of
files with always the same filenames, s.a.
  dir1
  |- file1 (100 MB)
  |- file2 (1 MB)
  dir2
  |- file1 (100 MB)
  |- file2 (1 MB)
  dir3
  |- file1 (100 MB)
  |- file2 (1 MB)
  ...

To gain some experience with rebalancing we set up a volume with one
brick only, copied some data as described above, added a second brick
and started the rebalancing. The result was that all files of a given
name ended up on the same brick. In our case, this leads to a very
inhomogeneous distribution of data volume since the different types of
files have very different sizes.

Looking at the implementation in the dht translator and checking
calculated hashes it seems that only the basename is used for the hash
calculation of a given file. With all directories having the same
mappings for the hash intervals to bricks, this would explain our
observation if only this file hash is used. However, I also see hashes
calculated for directories but it's not clear to me for what they are
used?

Do I miss something here? Is this behaviour intended? Is there a
(supported) way to still distribute the files homogeneously to all
bricks? E.g. by using the full path for the hashing (which is actually
what I understood from the manual), or by shuffling the hash intervals
per directory?

There must be other people having many directories containing the same
set of files. Any recommendations on how to handle this?

Many thanks,
    Jochen

On 08/14/2012 03:44 AM, Jochen Klein wrote:
> Looking at the implementation in the dht translator and checking
> calculated hashes it seems that only the basename is used for the hash
> calculation of a given file. With all directories having the same
> mappings for the hash intervals to bricks, this would explain our
> observation if only this file hash is used. However, I also see hashes
> calculated for directories but it's not clear to me for what they are
> used?
> 
> Do I miss something here? Is this behaviour intended? Is there a
> (supported) way to still distribute the files homogeneously to all
> bricks? E.g. by using the full path for the hashing (which is actually
> what I understood from the manual), or by shuffling the hash intervals
> per directory?
I tripped over the same issue a while ago.  Yes, the file hashes use only the
basename.  However, it's not (or at least shouldn't be) true that all
directories have the same mappings for the hash intervals.  The same ranges are
used, but rotated into different orders.  So, using letters for hash values,
different directories might have:

	A-I on brick1, J-R on brick2, S-Z on brick3
	A-I on brick2, J-R on brick3, S-Z on brick1
	A-I on brick3, J-R on brick1, S-Z on brick2

I just ran a quick test creating a bunch of directories on a simple two-brick
distributed volume.  Sure enough, about half of the directories got one order,
and the other half got another.  If this isn't working the same way for you
(check using "getxattr -x -n trusted.glusterfs.dht" on each per-brick
copy of
each directory) then it's probably a bug and we'll have to figure out
why.

Personally, I'd prefer if all directories *did* have the same hash layout,
so
that those layouts could be inherited instead of having to be set separately on
each and every directory of a potentially-petabyte volume.  That would require
that the hash include some directory-specific value (such as the parent GFID)
as well as the basename, but that seems a small price to pay.  In other words,
though right now it's a bit non-obvious how the layouts and hashing work,
some
day they might work as you (and I) had expected.

Hi Jochen,

The reason why you see such kind of distribution is:

1. You started off with a single brick volume, and hence the complete hash is
the single brick.
2. You added 2 brick, and issued rebalance. Now during rewriting of layout
(rebalancing), layout's are calculated to reduce the amount data migrated.
So, overlaps are considered.

In your scenario, 1st brick always got the start range, and others the remaining
in a round robin.

With regards,
Shishir