Koby, Bradley
2014-Aug-12 13:53 UTC
[Gluster-users] Gluster log files being flooded with messages.
Hello, I am new to Gluster, having recently been made admin of this after a sudden departure of a fellow co-worker. There does not seem to be any visible degradation of Gluster, and replication seems to be happening as well. However, there are two log files that are filling up so fast, that the previous administrator had added some cron jobs to purge the files every 20min. The running version is 3.3.0. Any help or ideas would be greatly appreciated! Thanks. millions of these type of entries every 20 min in glustershd.log [2014-08-12 09:28:23.309811] E [afr-self-heald.c:287:_remove_stale_index] 0-NewRHS-replicate-0: fd1abea4-472d-4db2-8ffd-13e4cb4325fe: Failed to remove index on NewRHS-client-0 - No such file or directory [2014-08-12 09:28:23.310745] E [afr-self-heald.c:287:_remove_stale_index] 0-NewRHS-replicate-0: d8e4419f-1ba5-4a42-9c4d-1058e4bcbaf3: Failed to remove index on NewRHS-client-0 - No such file or directory [2014-08-12 09:28:23.311651] E [afr-self-heald.c:287:_remove_stale_index] 0-NewRHS-replicate-0: 7de3320a-eff7-43f6-ab88-67ce765df6a4: Failed to remove index on NewRHS-client-0 - No such file or directory and about 4 million of these type of entries every 20min in mnt-gluster-NewRHS-start.log [2014-08-12 09:34:54.492495] E [posix.c:223:posix_stat] 0-NewRHS-posix: lstat on /mnt/gluster/NewRHS/start/.glusterfs/a0/79/a079b4de-2998-48ea-9dca-835858ae2ff1 failed: No such file or directory [2014-08-12 09:34:54.492528] I [server3_1-fops.c:1085:server_unlink_cbk] 0-NewRHS-server: 715465019: UNLINK <gfid:7c0d26c5-07b2-4ff4-8b8c-e2dc2e6eba47>/34c985ca-2273-4c00-ae42-3c2df7ecd40a (a079b4de-2998-48ea-9dca-835858ae2ff1) ==> -1 (No such file or directory) [2014-08-12 09:34:54.493213] W [marker-quota.c:2047:mq_inspect_directory_xattr] 0-NewRHS-marker: cannot add a new contribution node [2014-08-12 09:34:54.493322] E [posix.c:223:posix_stat] 0-NewRHS-posix: lstat on /mnt/gluster/NewRHS/start/.glusterfs/11/74/117433f7-ea58-486f-9305-fdebc98e0f0a failed: No such file or directory [2014-08-12 09:34:54.493355] I [server3_1-fops.c:1085:server_unlink_cbk] 0-NewRHS-server: 715465021: UNLINK <gfid:7c0d26c5-07b2-4ff4-8b8c-e2dc2e6eba47>/79493dd2-a267-4dcb-b813-a9de10833de7 (117433f7-ea58-486f-9305-fdebc98e0f0a) ==> -1 (No such file or directory) Some basic info....... gluster> volume info NewRHS Volume Name: NewRHS Type: Replicate Volume ID: 6a63d285-d14c-4ac8-8405-3db771d25e34 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: mualglup01.mcs.muohio.edu:/mnt/gluster/NewRHS/start Brick2: mualglup02.mcs.muohio.edu:/mnt/gluster/NewRHS/start Options Reconfigured: auth.allow: 10.0.72.135,10.0.93.*,192.168.251.*,127.0.0.1,10.0.72.132,10.0.72.133 nfs.disable: off features.quota: on nfs.register-with-portmap: 1 diagnostics.client-log-level: ERROR gluster> peer status Number of Peers: 1 Hostname: mualglup02 Uuid: 33ab9b77-b49b-43af-8edd-f7c39b19841b State: Peer in Cluster (Connected) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140812/6b62a9a9/attachment.html>
Mr. John J. Hoffman
2014-Aug-12 17:23 UTC
[Gluster-users] Gluster log files being flooded with messages.
Bradley, You're not the only one that's happened to ;) It looks like you have some inconsistencies between the 2 peered nodes. The following will give you some more insight > gluster volume heal help Try to see if its attempting to heal files > gluster volume heal NewRHS info also if any heals have failed > gluster volume heal NewRHS info heal-failed or you're in a split-brain situation (files exist on both peers, but differ). > gluster volume heal NewRHS split-brain On 08/12/14 09:53, Koby, Bradley wrote:> Hello, > > I am new to Gluster, having recently been made admin of this after a > sudden departure of a fellow co-worker. There does not seem to be any > visible degradation of Gluster, and replication seems to be happening as > well. However, there are two log files that are filling up so fast, > that the previous administrator had added some cron jobs to purge the > files every 20min. > > The running version is 3.3.0. > > > Any help or ideas would be greatly appreciated! > > Thanks. > > > millions of these type of entries every 20 min in glustershd.log > > [2014-08-12 09:28:23.309811] E > [afr-self-heald.c:287:_remove_stale_index] 0-NewRHS-replicate-0: > fd1abea4-472d-4db2-8ffd-13e4cb4325fe: Failed to remove index on > NewRHS-client-0 - No such file or directory > [2014-08-12 09:28:23.310745] E > [afr-self-heald.c:287:_remove_stale_index] 0-NewRHS-replicate-0: > d8e4419f-1ba5-4a42-9c4d-1058e4bcbaf3: Failed to remove index on > NewRHS-client-0 - No such file or directory > [2014-08-12 09:28:23.311651] E > [afr-self-heald.c:287:_remove_stale_index] 0-NewRHS-replicate-0: > 7de3320a-eff7-43f6-ab88-67ce765df6a4: Failed to remove index on > NewRHS-client-0 - No such file or directory > > and > > about 4 million of these type of entries every 20min in > mnt-gluster-NewRHS-start.log > > [2014-08-12 09:34:54.492495] E [posix.c:223:posix_stat] 0-NewRHS-posix: > lstat on > /mnt/gluster/NewRHS/start/.glusterfs/a0/79/a079b4de-2998-48ea-9dca-835858ae2ff1 > failed: No such file or directory > [2014-08-12 09:34:54.492528] I [server3_1-fops.c:1085:server_unlink_cbk] > 0-NewRHS-server: 715465019: UNLINK > <gfid:7c0d26c5-07b2-4ff4-8b8c-e2dc2e6eba47>/34c985ca-2273-4c00-ae42-3c2df7ecd40a > (a079b4de-2998-48ea-9dca-835858ae2ff1) ==> -1 (No such file or directory) > [2014-08-12 09:34:54.493213] W > [marker-quota.c:2047:mq_inspect_directory_xattr] 0-NewRHS-marker: cannot > add a new contribution node > [2014-08-12 09:34:54.493322] E [posix.c:223:posix_stat] 0-NewRHS-posix: > lstat on > /mnt/gluster/NewRHS/start/.glusterfs/11/74/117433f7-ea58-486f-9305-fdebc98e0f0a > failed: No such file or directory > [2014-08-12 09:34:54.493355] I [server3_1-fops.c:1085:server_unlink_cbk] > 0-NewRHS-server: 715465021: UNLINK > <gfid:7c0d26c5-07b2-4ff4-8b8c-e2dc2e6eba47>/79493dd2-a267-4dcb-b813-a9de10833de7 > (117433f7-ea58-486f-9305-fdebc98e0f0a) ==> -1 (No such file or directory) > > > Some basic info....... > > gluster> volume info NewRHS > > Volume Name: NewRHS > Type: Replicate > Volume ID: 6a63d285-d14c-4ac8-8405-3db771d25e34 > Status: Started > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: mualglup01.mcs.muohio.edu:/mnt/gluster/NewRHS/start > Brick2: mualglup02.mcs.muohio.edu:/mnt/gluster/NewRHS/start > Options Reconfigured: > auth.allow: > 10.0.72.135,10.0.93.*,192.168.251.*,127.0.0.1,10.0.72.132,10.0.72.133 > nfs.disable: off > features.quota: on > nfs.register-with-portmap: 1 > diagnostics.client-log-level: ERROR > > > gluster> peer status > Number of Peers: 1 > > Hostname: mualglup02 > Uuid: 33ab9b77-b49b-43af-8edd-f7c39b19841b > State: Peer in Cluster (Connected) > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users >-- John J. Hoffman, CISSP Computer Engineer, AFRL/RCMI AFRL DoD Supercomputing Resource Center (DSRC) Air Force Research Laboratory 2435 Fifth Street Area B, Bldg 676, Rm 205 Wright-Patterson AFB, OH 45433-7802 john.hoffman at wpafb.af.mil jhoffman at afrl.hpc.mil (937) 904-5129 Caution: This message may contain competitive, sensitive or other non-public information not intended for disclosure outside official government channels. Do not disseminate this message without the approval of the Office of the Assistant Secretary of the Air Force for Acquisition. If you received this message in error, please notify the sender by reply e-mail and delete all copies of this message.