similar to: log file spewing on one node, but not the others

What is the uptime of the affected node ? There is a similar error reported in?https://access.redhat.com/solutions/5518661? which could?indicate?a possible problem in a memory area named ?lru? .Have you noticed any ECC errors in dmesg/IPMI of the system ? At least I would reboot the node and run hardware diagnostics to check that everything is fine. Best Regards,Strahil Nikolov? Sent from Yahoo

On 1/30/2018 6:31 AM, Raghavendra Gowdappa wrote: > > > ----- Original Message ----- >> From: "Dan Ragle" <daniel at Biblestuph.com> >> To: "Raghavendra Gowdappa" <rgowdapp at redhat.com>, "Ravishankar N" <ravishankar at redhat.com> >> Cc: gluster-users at gluster.org, "Csaba Henk" <chenk at redhat.com>,

On 2/3/2018 8:58 AM, Dan Ragle wrote: > > > On 2/2/2018 2:13 AM, Nithya Balachandran wrote: >> Hi Dan, >> >> It sounds like you might be running into [1]. The patch has been >> posted upstream and the fix should be in the next release. >> In the meantime, I'm afraid there is no way to get around this without >> restarting the process. >>

On 2/2/2018 2:13 AM, Nithya Balachandran wrote: > Hi Dan, > > It sounds like you might be running into [1]. The patch has been posted > upstream and the fix should be in the next release. > In the meantime, I'm afraid there is no way to get around this without > restarting the process. > > Regards, > Nithya > >

----- Original Message ----- > From: "Dan Ragle" <daniel at Biblestuph.com> > To: "Raghavendra Gowdappa" <rgowdapp at redhat.com>, "Ravishankar N" <ravishankar at redhat.com> > Cc: gluster-users at gluster.org, "Csaba Henk" <chenk at redhat.com>, "Niels de Vos" <ndevos at redhat.com>, "Nithya >

On 1/29/2018 2:36 AM, Raghavendra Gowdappa wrote: > > > ----- Original Message ----- >> From: "Ravishankar N" <ravishankar at redhat.com> >> To: "Dan Ragle" <daniel at Biblestuph.com>, gluster-users at gluster.org >> Cc: "Csaba Henk" <chenk at redhat.com>, "Niels de Vos" <ndevos at redhat.com>,

Hi Dan, I had a suggestion and a question in my previous response. Let us know whether the suggestion helps and please let us know about your data-set (like how many directories/files and how these directories/files are organised) to understand the problem better. <snip> > In the > meantime can you remount glusterfs with options > --entry-timeout=0 and

Hi Dan, It sounds like you might be running into [1]. The patch has been posted upstream and the fix should be in the next release. In the meantime, I'm afraid there is no way to get around this without restarting the process. Regards, Nithya [1]https://bugzilla.redhat.com/show_bug.cgi?id=1541264 On 2 February 2018 at 02:57, Dan Ragle <daniel at biblestuph.com> wrote: > >

----- Original Message ----- > From: "Ravishankar N" <ravishankar at redhat.com> > To: "Dan Ragle" <daniel at Biblestuph.com>, gluster-users at gluster.org > Cc: "Csaba Henk" <chenk at redhat.com>, "Niels de Vos" <ndevos at redhat.com>, "Nithya Balachandran" <nbalacha at redhat.com>, > "Raghavendra

On 01/27/2018 02:29 AM, Dan Ragle wrote: > > On 1/25/2018 8:21 PM, Ravishankar N wrote: >> >> >> On 01/25/2018 11:04 PM, Dan Ragle wrote: >>> *sigh* trying again to correct formatting ... apologize for the >>> earlier mess. >>> >>> Having a memory issue with Gluster 3.12.4 and not sure how to >>> troubleshoot. I don't

Some corrections about the previous mails. Problem does not happen when no volumes created. Problem happens volumes created but in stopped state. Problem also happens when volumes started state. Below is the 5 stack traces taken by 10 min intervals and volumes stopped state. --1-- Thread 8 (Thread 0x7f413f3a7700 (LWP 104249)): #0 0x0000003d99c0f00d in nanosleep () from /lib64/libpthread.so.0 #1

On Tue, Sep 5, 2017 at 6:13 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Some corrections about the previous mails. Problem does not happen > when no volumes created. > Problem happens volumes created but in stopped state. Problem also > happens when volumes started state. > Below is the 5 stack traces taken by 10 min intervals and volumes stopped > state. > As

On Mon, 4 Sep 2017 at 20:04, Serkan ?oban <cobanserkan at gmail.com> wrote: > I have been using a 60 server 1560 brick 3.7.11 cluster without > problems for 1 years. I did not see this problem with it. > Note that this problem does not happen when I install packages & start > glusterd & peer probe and create the volumes. But after glusterd > restart. > > Also

Here you can find 10 stack trace samples from glusterd. I wait 10 seconds between each trace. https://www.dropbox.com/s/9f36goq5xn3p1yt/glusterd_pstack.zip?dl=0 Content of the first stack trace is here: Thread 8 (Thread 0x7f7a8cd4e700 (LWP 43069)): #0 0x0000003aa5c0f00d in nanosleep () from /lib64/libpthread.so.0 #1 0x000000303f837d57 in ?? () from /usr/lib64/libglusterfs.so.0 #2

Hi, You can find pstack sampes here: https://www.dropbox.com/s/6gw8b6tng8puiox/pstack_with_debuginfo.zip?dl=0 Here is the first one: Thread 8 (Thread 0x7f92879ae700 (LWP 78909)): #0 0x0000003d99c0f00d in nanosleep () from /lib64/libpthread.so.0 #1 0x000000310fe37d57 in gf_timer_proc () from /usr/lib64/libglusterfs.so.0 #2 0x0000003d99c07aa1 in start_thread () from /lib64/libpthread.so.0 #3

Serkan, I have gone through other mails in the mail thread as well but responding to this one specifically. Is this a source install or an RPM install ? If this is an RPM install, could you please install the glusterfs-debuginfo RPM and retry to capture the gdb backtrace. If this is a source install, then you'll need to configure the build with --enable-debug and reinstall and retry

Hi Milind, Anything new about the issue? Can you able to find the problem, anything else you need? I will continue with two clusters each 40 servers, so I will not be able to provide any further info for 80 servers. On Fri, Sep 1, 2017 at 10:30 AM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Hi, > You can find pstack sampes here: >

No worries Serkan, You can continue to use your 40 node clusters. The backtrace has resolved the function names and it *should* be sufficient to debug the issue. Thanks for letting us know. We'll post on this thread again to notify you about the findings. On Sat, Sep 2, 2017 at 2:42 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Hi Milind, > > Anything new about the

Not yet. Gaurav will be taking a look at it tomorrow. On Wed, 23 Aug 2017 at 20:14, Serkan ?oban <cobanserkan at gmail.com> wrote: > Hi Atin, > > Do you have time to check the logs? > > On Wed, Aug 23, 2017 at 10:02 AM, Serkan ?oban <cobanserkan at gmail.com> > wrote: > > Same thing happens with 3.12.rc0. This time perf top shows hanging in > >

I am working on it and will share my findings as soon as possible. Thanks Gaurav On Thu, Aug 24, 2017 at 3:58 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Restarting glusterd causes the same thing. I tried with 3.12.rc0, > 3.10.5. 3.8.15, 3.7.20 all same behavior. > My OS is centos 6.9, I tried with centos 6.8 problem remains... > Only way to a healthy state is