Gluster users - Feb 2016 - Unable to get volume stat after lots of small files created

Hi,

I have three nodes cluster running GlusterFS v3.7.6 just built and are
in the progress running various tests.

For the small file test, after ~120,000 64K files written to one
volume, access to the volume slow down significantly. "ls -lR
/gluster_nj" took 3+ minutes to complete. 120,000 files under ~1000
directories under volume root.

Two questions :
1. Any tuning I can do to improve small file meta data query?
2. Right after creation of the 120,000 small files, the gluster CLI
function also get impacted:

sanlnjpnfsbu01 /etc/glusterfs # gluster volume profile rsvol1 info
Another transaction is in progress for rsvol1. Please try again after sometime.

I verified no profile command ran on anywhere else in the cluster.
Retried the command and getting same error for 30+ minutes.

At the time when "volume profile" command failed, I can see below
error in etc-glusterfs-glusterd.vol.log :

[2016-02-10 20:25:49.757758] W
[glusterd-locks.c:577:glusterd_mgmt_v3_lock]
(-->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(__glusterd_handle_cli_profil
e_volume+0x2eb) [0x7f401889796b]
-->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(glusterd_op_txn_begin+0x51a)
[0x7f40188973ba] -->/usr/lib64/glusterfs/3.
7.6/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_lock+0x1ed)
[0x7f4018938add] ) 0-management: Lock for rsvol1 held by
2e3e526a-1e9a-4f6d-abee-c42d84f3bb16
[2016-02-10 20:25:49.757776] E [MSGID: 106119]
[glusterd-handler.c:728:glusterd_op_txn_begin] 0-management: Unable to
acquire lock for rsvol1

Any idea what's preventing the profile command acquiring lock for the
volume?

Thanks,
Fred

On 02/11/2016 02:44 AM, Fred Yang wrote:
> Hi,
> 
> I have three nodes cluster running GlusterFS v3.7.6 just built and are
> in the progress running various tests.
> 
> For the small file test, after ~120,000 64K files written to one
> volume, access to the volume slow down significantly. "ls -lR
> /gluster_nj" took 3+ minutes to complete. 120,000 files under ~1000
> directories under volume root.
> 
> Two questions :
> 1. Any tuning I can do to improve small file meta data query?
> 2. Right after creation of the 120,000 small files, the gluster CLI
> function also get impacted:
Heavy I/O workload should not impact management plane.
> 
> sanlnjpnfsbu01 /etc/glusterfs # gluster volume profile rsvol1 info
> Another transaction is in progress for rsvol1. Please try again after
sometime.
> 
> I verified no profile command ran on anywhere else in the cluster.
> Retried the command and getting same error for 30+ minutes.
> 
> At the time when "volume profile" command failed, I can see below
> error in etc-glusterfs-glusterd.vol.log :
> 
> [2016-02-10 20:25:49.757758] W
> [glusterd-locks.c:577:glusterd_mgmt_v3_lock]
>
(-->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(__glusterd_handle_cli_profil
> e_volume+0x2eb) [0x7f401889796b]
>
-->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(glusterd_op_txn_begin+0x51a)
> [0x7f40188973ba] -->/usr/lib64/glusterfs/3.
> 7.6/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_lock+0x1ed)
> [0x7f4018938add] ) 0-management: Lock for rsvol1 held by
> 2e3e526a-1e9a-4f6d-abee-c42d84f3bb16
> [2016-02-10 20:25:49.757776] E [MSGID: 106119]
> [glusterd-handler.c:728:glusterd_op_txn_begin] 0-management: Unable to
> acquire lock for rsvol1
By any chance do you have any script which triggers gluster commands
like volume status in some time intervals? The above log indicates that
a lock on volume rsvol1 (by some gluster command) has been acquired by
node 2e3e526a-1e9a-4f6d-abee-c42d84f3bb16 which is the UUID of the
server and the same can be found from gluster peer status command. If
you don't get any clue from this information mind sharing the glusterd
log files?
> 
> Any idea what's preventing the profile command acquiring lock for the
volume?
> 
> Thanks,
> Fred
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>