Dan Bretherton
2013-Mar-08 15:04 UTC
[Gluster-users] Re. Gluster machines slowing down over time
> *Tom Hall* thattommyhall at gmail.com > <mailto:gluster-users%40gluster.org?Subject=Re:%20Re%3A%20%5BGluster-users%5D%20Gluster%20machines%20slowing%20down%20over%20time&In-Reply-To=%3CCAMvQ07jForynyik2NYmYnKq-kAa5wHoZLwOU8HDtYXKChM2ZXQ%40mail.gmail.com%3E> > /Wed Dec 12 12:47:09 UTC 2012/ > > * Previous message: [Gluster-users] Gluster machines slowing down > over time > <http://gluster.org/pipermail/gluster-users/2012-December/034974.html> > * Next message: [Gluster-users] glusterfs-3.4.0qa5 released > <http://gluster.org/pipermail/gluster-users/2012-December/034973.html> > * *Messages sorted by:* [ date ] > <http://gluster.org/pipermail/gluster-users/2012-December/date.html#34976> > [ thread ] > <http://gluster.org/pipermail/gluster-users/2012-December/thread.html#34976> > [ subject ] > <http://gluster.org/pipermail/gluster-users/2012-December/subject.html#34976> > [ author ] > <http://gluster.org/pipermail/gluster-users/2012-December/author.html#34976> > > > ------------------------------------------------------------------------ > On Wed, Dec 12, 2012 at 10:41 AM, Brian Candler <B.Candler at pobox.com <http://supercolony.gluster.org/mailman/listinfo/gluster-users>> wrote: > > >/ On Tue, Dec 11, 2012 at 04:13:15PM +0000, Tom Hall wrote: > />/ > I have 2 gluster servers in replicated mode on EC2 with ~4G RAM > />/ > > />/ > CPU and RAM look fine but over time the system becomes sluggish, > />/ > particularly networking. > />/ > > />/ > I notice when sshing into the machine takes ages and running remote > />/ > commands with capistrano takes longer and longer. > />/ > />/ Do you have swap configured? Is the system going into swap? > / > > No swap (default on that size EC2 instance) > total used free shared buffers cached > Mem: 3750 3731 19 0 274 2929 > -/+ buffers/cache: 526 3224 > Swap: 0 0 0 > > Caching heavily but not using so much RAM > > > >/ > / > > >/ > /Have a look at output of "free", "vmstat 5" (for a minute or two), > >/ "iostat 5 -x" (ditto) > />/ > />/ > /I'll get it next time we experience the issue. > Seems to slow down over a week or so so it might be a while... > > Thanks guys! > TomDear All- I too have been experiencing a gradual slowing of GlusterFS over time, accompanied by increases in CPU load and memory usage. However, the storage server CPU load does not seem to be related to the I/O load imposed by users. The following three Ganglia plots for the past month illustrate the point. http://www.nerc-essc.ac.uk/~dab/GlusterFS_Ganglia_CPU_Mem_MonthTo8thMarch2013.jpeg http://www.nerc-essc.ac.uk/~dab/Rocks_Ganglia_CPU_Mem_MonthTo8thMarch2013.jpeg http://www.nerc-essc.ac.uk/~dab/GlusterFS-remus_Ganglia_CPU_Mem_MonthTo8thMarch2013.jpeg The first plot shows the average storage server CPU load and memory usage rising during the course of the past month, which became almost unmanageable and prompted me to restart all the servers a few days ago. The second plot shows the CPU load and memory usage on my small Rocks compute cluster, which is responsible for most of the I/O load on the GlusterFS cluster. The third chart shows CPU and memory usage on a typical storage server, which has been restarted a couple of times in the past month. The first restart was at the beginning of week 7 on the plot, and the second was a couple days ago at the same time as the other servers. There is no evidence that the I/O load on the storage cluster has been increasing over the past month, so why does the CPU load on the storage servers gradually increase (along with memory usage) until they have to be restarted? Restarting the all the servers can be very disruptive because it seems to take a long time for them all to start talking to each other properly again. Once everything has settled down everyone breathes a sigh of relief and enjoys relatively fast and responsive I/O for a few days, before things start to slow down again. It looks as if I'm going to have to start doing this routinely every weekend, but this isn't how I was planning to spend my weekends for the foreseeable future. Are there any patches in the pipeline that are likely to address this issue? -Dan -- Dan Bretherton ESSC Computer System Manager Department of Meteorology Harry Pitt Building, 3 Earley Gate University of Reading Reading, RG6 7BE (or RG6 6AL for postal service deliveries) UK Tel. +44 118 378 5205, Fax: +44 118 378 6413 -- ## Please sponsor me to run in VSO's 30km Race to the Eye ## ## http://www.justgiving.com/DanBretherton ## -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130308/598da05e/attachment.html>