Gluster users - Mar 2012 - dht log entries in fuse client after successful expansion/rebalance

Hi

I'm using Gluster 3.2.5.  After expanding a 2x2 Distributed-Replicate
volume to 3x2 and performing a full rebalance fuse clients log the
following messages for every directory access:

[2012-03-08 10:53:56.953030] I [dht-common.c:524:dht_revalidate_cbk]
1-bfd-dht: mismatching layouts for /linux-3.2.9/tools/power/cpupower/bench
[2012-03-08 10:53:56.953065] I [dht-layout.c:682:dht_layout_dir_mismatch]
1-bfd-dht: subvol: bfd-replicate-2; inode layout - 0 - 0; disk layout -
2863311530 - 4294967295
[2012-03-08 10:53:56.953080] I [dht-common.c:524:dht_revalidate_cbk]
1-bfd-dht: mismatching layouts for /linux-3.2.9/tools/power/cpupower/bench
[2012-03-08 10:53:56.953218] I [dht-layout.c:682:dht_layout_dir_mismatch]
1-bfd-dht: subvol: bfd-replicate-0; inode layout - 0 - 2147483646; disk
layout - 0 - 1431655764
[2012-03-08 10:53:56.953239] I [dht-common.c:524:dht_revalidate_cbk]
1-bfd-dht: mismatching layouts for /linux-3.2.9/tools/power/cpupower/bench
[2012-03-08 10:53:56.966991] I [dht-layout.c:682:dht_layout_dir_mismatch]
1-bfd-dht: subvol: bfd-replicate-0; inode layout - 2147483647 -
4294967295; disk layout - 0 - 1431655764
[2012-03-08 10:53:56.967017] I [dht-common.c:524:dht_revalidate_cbk]
1-bfd-dht: mismatching layouts for /linux-3.2.9/tools/power/cpupower/debug
[2012-03-08 10:53:56.967118] I [dht-layout.c:682:dht_layout_dir_mismatch]
1-bfd-dht: subvol: bfd-replicate-1; inode layout - 0 - 2147483646; disk
layout - 1431655765 - 2863311529

I have tried doing a self-heal on the client and rebalancing the volume
again but the messages persist.  After remounting the volume the messages
stop.  The rebalance reports success in adjusting the layout and
redistributing files and I can see from looking at the bricks that
pre-expansion files have been moved to the new brick and post expansion
files are going to all three but the excessive client logging effects
performance and would take up lots of space when under heavy use.  Does
anybody know what might be happening here and how I can avoid these
messages after expansion without remounting or turning off/down client
logging?

Thanks.
Tim

On Fri, 9 Mar 2012 15:44:48 +0000
Tim Robinson <Tim.Robinson at humedica.com> wrote:
> mismatching layouts for .../bench
> subvol: bfd-replicate-2;
>	inode layout - 0 - 0;
>	disk layout - 2863311530 - 4294967295
> mismatching layouts for .../bench
> subvol: bfd-replicate-0;
>	inode layout - 0 - 2147483646;
>	disk layout - 0 - 1431655764
> mismatching layouts .../bench
> subvol: bfd-replicate-0;
>	inode layout - 2147483647 - 4294967295;
>	disk layout - 0 - 1431655764
> mismatching layouts for .../debug
> subvol: bfd-replicate-1;
>	inode layout - 0 - 2147483646;
>	disk layout - 1431655765 - 2863311529
It looks like either your mail program or mine scrambled these lines a
bit, so I've edited to show the most important bits.  Basically there
are three messages for .../bench, corresponding to the three
subvolumes, and then messages for .../debug on two out of three
subvolumes.  I'd be worried if we saw these repeated for the same
directory on the same subvolume, but (so far) that doesn't seem to be
the case.  AFAICT the problem is just that the layout-revalidation code
is being a lot more verbose than necessary.  This activity should tail
off pretty quickly.
> I have tried doing a self-heal on the client and rebalancing the
> volume again but the messages persist.  After remounting the volume
> the messages stop.  The rebalance reports success in adjusting the
> layout and redistributing files and I can see from looking at the
> bricks that pre-expansion files have been moved to the new brick and
> post expansion files are going to all three but the excessive client
> logging effects performance and would take up lots of space when
> under heavy use.  Does anybody know what might be happening here and
> how I can avoid these messages after expansion without remounting or
> turning off/down client logging?
We support dynamic reconfiguration on the servers, but not on the
clients, so without remounting there's no good way to reduce the client
log level.  I'd do it with gdb, but I don't think I can recommend that
for the sort of environment where a remount would be precluded.