Gluster users - Dec 2013 - Where does the 'date' string in '/var/log/glusterfs/gl.log' come from?

Admittedly I should search the source, but I wonder if anyone knows this 
offhand.

Background:  of our  84 ROCKS (6.1) -provisioned compute nodes, 4 have picked 
up an 'advanced date'  in the /var/log/glusterfs/gl.log file - that date
string is running about 5-6 hours ahead of the system date and all the Gluster 
servers (which are identical and correct).  The time advancement does not 
appear to be identical tho it's hard to tell since it only shows on errors
and
those update irregularly.

All the clients are the same version and all the servers are the same (gluster 
v 3.4.0-8.el6.x86_64

This would not be of interest except that those 4 clients are losing files, 
unable to reliably do IO, etc on the gluster fs.  They don't appear to be 
having problems with NFS mounts, nor with a Fraunhofer FS that is also mounted 
on each node,

Rebooting 2 of them has no effect - they come right back with an advanced 
date.


---
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
---
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131209/9cde5ba3/attachment.html>

Hi Harry,

Did you setup ntp on each of the node, and sync the time to one single
source?

Thanks.


On Tue, Dec 10, 2013 at 12:44 PM, harry mangalam <harry.mangalam at
uci.edu>wrote:
>  Admittedly I should search the source, but I wonder if anyone knows this
> offhand.
>
>
>
> Background: of our 84 ROCKS (6.1) -provisioned compute nodes, 4 have
> picked up an 'advanced date' in the /var/log/glusterfs/gl.log file
- that
> date string is running about 5-6 hours ahead of the system date and all the
> Gluster servers (which are identical and correct). The time advancement
> does not appear to be identical tho it's hard to tell since it only
shows
> on errors and those update irregularly.
>
>
>
> All the clients are the same version and all the servers are the same
> (gluster v 3.4.0-8.el6.x86_64
>
>
>
> This would not be of interest except that those 4 clients are losing
> files, unable to reliably do IO, etc on the gluster fs. They don't
appear
> to be having problems with NFS mounts, nor with a Fraunhofer FS that is
> also mounted on each node,
>
>
>
> Rebooting 2 of them has no effect - they come right back with an advanced
> date.
>
>
>
>
>
> ---
>
> Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
>
> [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
>
> 415 South Circle View Dr, Irvine, CA, 92697 [shipping]
>
> MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
>
> ---
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>


-- 
Sharuzzaman Ahmat Raslan
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131210/d0de4ecd/attachment.html>

On 12/10/2013 10:14 AM, harry mangalam wrote:
> Admittedly I should search the source, but I wonder if anyone knows this
> offhand.
>
> Background: of our 84 ROCKS (6.1) -provisioned compute nodes, 4 have
> picked up an 'advanced date' in the /var/log/glusterfs/gl.log file
-
> that date string is running about 5-6 hours ahead of the system date and
> all the Gluster servers (which are identical and correct). The time
> advancement does not appear to be identical tho it's hard to tell since
> it only shows on errors and those update irregularly.
The timestamps in the log file are by default in UTC. That could 
possibly explain why the timestamps look advanced in the log file.
>
> All the clients are the same version and all the servers are the same
> (gluster v 3.4.0-8.el6.x86_64
>
> This would not be of interest except that those 4 clients are losing
> files, unable to reliably do IO, etc on the gluster fs. They don't
> appear to be having problems with NFS mounts, nor with a Fraunhofer FS
> that is also mounted on each node,
Do you observe anything in the client log files of these machines that 
indicate I/O problems?

Thanks,
Vijay