search for: oban

...ver cluster gets > rebooted, glusterd process hangs. > > Could you please provide the logs which led glusterd to hang for all the > cases along with gusterd process utilization. > > > Thanks > Gaurav > > > > > > > On Tue, Aug 29, 2017 at 2:44 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: >> >> Here is the requested logs: >> >> https://www.dropbox.com/s/vt187h0gtu5doip/gluster_logs_20_40_80_servers.zip?dl=0 >> >> >> On Tue, Aug 29, 2017 at 7:48 AM, Gaurav Yadav <gyadav at redhat.com> wrote: >&...

glusterd returned to normal, here is the logs: https://www.dropbox.com/s/41jx2zn3uizvr53/80servers_glusterd_normal_status.zip?dl=0 On Tue, Aug 29, 2017 at 1:47 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Here is the logs after stopping all three volumes and restarting > glusterd in all nodes. I waited 70 minutes after glusterd restart but > it is still consuming %100 CPU. > https://www.dropbox.com/s/pzl0f198v03twx3/80servers_after_glusterd_resta...

...ything significant. > > Can you send me gluster logs along with command-history-logs for these > scenarios: > Scenario1 : 20 servers > Scenario2 : 40 servers > Scenario3: 80 Servers > > > Thanks > Gaurav > > > > On Mon, Aug 28, 2017 at 11:22 AM, Serkan ?oban <cobanserkan at gmail.com> > wrote: >> >> Hi Gaurav, >> Any progress about the problem? >> >> On Thursday, August 24, 2017, Serkan ?oban <cobanserkan at gmail.com> wrote: >>> >>> Thank you Gaurav, >>> Here is more findings: &gt...

...e volume. However, you have mentioned that when a node from 80 server cluster gets rebooted, glusterd process hangs. Could you please provide the logs which led glusterd to hang for all the cases along with gusterd process utilization. Thanks Gaurav On Tue, Aug 29, 2017 at 2:44 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Here is the requested logs: > https://www.dropbox.com/s/vt187h0gtu5doip/gluster_logs_ > 20_40_80_servers.zip?dl=0 > > > On Tue, Aug 29, 2017 at 7:48 AM, Gaurav Yadav <gyadav at redhat.com> wrote: > > Till now I haven't fou...

Hi Gaurav, Any progress about the problem? On Thursday, August 24, 2017, Serkan ?oban <cobanserkan at gmail.com> wrote: > Thank you Gaurav, > Here is more findings: > Problem does not happen using only 20 servers each has 68 bricks. > (peer probe only 20 servers) > If we use 40 servers with single volume, glusterd cpu %100 state > continues for 5 minutes and...

Till now I haven't found anything significant. Can you send me gluster logs along with command-history-logs for these scenarios: Scenario1 : 20 servers Scenario2 : 40 servers Scenario3: 80 Servers Thanks Gaurav On Mon, Aug 28, 2017 at 11:22 AM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Hi Gaurav, > Any progress about the problem? > > On Thursday, August 24, 2017, Serkan ?oban <cobanserkan at gmail.com> wrote: > >> Thank you Gaurav, >> Here is more findings: >> Problem does not happen using only 20 s...

...ainst 16+3 as the rebuild time are bottlenecked by matrix operations which scale as the square of the number of data stripes. There are some savings because of larger data chunks but we ended up using 8+3 and heal times are about half compared to 16+3. -Alastair On 30 June 2017 at 02:22, Serkan ?oban <cobanserkan at gmail.com> wrote: > >Thanks for the reply. We will mainly use this for archival - near-cold > storage. > Archival usage is good for EC > > >Anything, from your experience, to keep in mind while planning large > installations? > I am using 3.7.11 and...

----- Original Message ----- > From: "Ben Turner" <bturner at redhat.com> > To: "Serkan ?oban" <cobanserkan at gmail.com> > Cc: "Gluster Users" <gluster-users at gluster.org> > Sent: Sunday, September 3, 2017 2:30:31 PM > Subject: Re: [Gluster-users] Glusterd proccess hangs on reboot > > ----- Original Message ----- > > From: "Milind Cha...

I am working on it and will share my findings as soon as possible. Thanks Gaurav On Thu, Aug 24, 2017 at 3:58 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Restarting glusterd causes the same thing. I tried with 3.12.rc0, > 3.10.5. 3.8.15, 3.7.20 all same behavior. > My OS is centos 6.9, I tried with centos 6.8 problem remains... > Only way to a healthy state is destroy gluster config/rpms, reinst...

i usually change event threads to 4. But those logs are from a default installation. On Sun, Sep 3, 2017 at 9:52 PM, Ben Turner <bturner at redhat.com> wrote: > ----- Original Message ----- >> From: "Ben Turner" <bturner at redhat.com> >> To: "Serkan ?oban" <cobanserkan at gmail.com> >> Cc: "Gluster Users" <gluster-users at gluster.org> >> Sent: Sunday, September 3, 2017 2:30:31 PM >> Subject: Re: [Gluster-users] Glusterd proccess hangs on reboot >> >> ----- Original Message ----- >> >...

...state. with 80 servers we have no working state yet... On Thu, Aug 24, 2017 at 1:33 PM, Gaurav Yadav <gyadav at redhat.com> wrote: > > I am working on it and will share my findings as soon as possible. > > > Thanks > Gaurav > > On Thu, Aug 24, 2017 at 3:58 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: >> >> Restarting glusterd causes the same thing. I tried with 3.12.rc0, >> 3.10.5. 3.8.15, 3.7.20 all same behavior. >> My OS is centos 6.9, I tried with centos 6.8 problem remains... >> Only way to a healthy state is destroy glu...

...rries Serkan, You can continue to use your 40 node clusters. The backtrace has resolved the function names and it *should* be sufficient to debug the issue. Thanks for letting us know. We'll post on this thread again to notify you about the findings. On Sat, Sep 2, 2017 at 2:42 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Hi Milind, > > Anything new about the issue? Can you able to find the problem, > anything else you need? > I will continue with two clusters each 40 servers, so I will not be > able to provide any further info for 80 servers. > > On...

...e bottlenecked by matrix operations which scale as the square > of the number of data stripes. There are some savings because of larger > data chunks but we ended up using 8+3 and heal times are about half compared > to 16+3. > > -Alastair > > On 30 June 2017 at 02:22, Serkan ?oban <cobanserkan at gmail.com> wrote: >> >> >Thanks for the reply. We will mainly use this for archival - near-cold >> > storage. >> Archival usage is good for EC >> >> >Anything, from your experience, to keep in mind while planning large >> >...

----- Original Message ----- > From: "Milind Changire" <mchangir at redhat.com> > To: "Serkan ?oban" <cobanserkan at gmail.com> > Cc: "Gluster Users" <gluster-users at gluster.org> > Sent: Saturday, September 2, 2017 11:44:40 PM > Subject: Re: [Gluster-users] Glusterd proccess hangs on reboot > > No worries Serkan, > You can continue to use your 40 no...

An upstream bug would be ideal as github issue is mainly used for enhancements. In the mean time, could you point to the exact failure shown at the command line and the log entry from cli.log? On Wed, Aug 23, 2017 at 12:10 AM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Hi, I think this is the line limiting brick count: > https://github.com/gluster/glusterfs/blob/c136024613c697fec87aaff3a07086 > 2b92c57977/cli/src/cli-cmd-parser.c#L84 > > Can gluster-devs increase this limit? Should I open a github issue? &...

...te: > Could you be able to provide the pstack dump of the glusterd process? > > On Wed, 23 Aug 2017 at 20:22, Atin Mukherjee <amukherj at redhat.com> wrote: >> >> Not yet. Gaurav will be taking a look at it tomorrow. >> >> On Wed, 23 Aug 2017 at 20:14, Serkan ?oban <cobanserkan at gmail.com> wrote: >>> >>> Hi Atin, >>> >>> Do you have time to check the logs? >>> >>> On Wed, Aug 23, 2017 at 10:02 AM, Serkan ?oban <cobanserkan at gmail.com> >>> wrote: >>> > Same thing happens...

Not yet. Gaurav will be taking a look at it tomorrow. On Wed, 23 Aug 2017 at 20:14, Serkan ?oban <cobanserkan at gmail.com> wrote: > Hi Atin, > > Do you have time to check the logs? > > On Wed, Aug 23, 2017 at 10:02 AM, Serkan ?oban <cobanserkan at gmail.com> > wrote: > > Same thing happens with 3.12.rc0. This time perf top shows hanging in > > libglu...

...ting glusterd causes the same thing. I tried with 3.12.rc0, 3.10.5. 3.8.15, 3.7.20 all same behavior. My OS is centos 6.9, I tried with centos 6.8 problem remains... Only way to a healthy state is destroy gluster config/rpms, reinstall and recreate volumes. On Thu, Aug 24, 2017 at 8:49 AM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Here you can find 10 stack trace samples from glusterd. I wait 10 > seconds between each trace. > https://www.dropbox.com/s/9f36goq5xn3p1yt/glusterd_pstack.zip?dl=0 > > Content of the first stack trace is here: > > Thread 8 (Thread 0x7f...

On Mon, Sep 4, 2017 at 5:28 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > >1. On 80 nodes cluster, did you reboot only one node or multiple ones? > Tried both, result is same, but the logs/stacks are from stopping and > starting glusterd only on one server while others are running. > > >2. Are you sure that...

...e and create the volumes. But after glusterd restart. Also note that this still happens without any volumes. So it is not related with brick count I think... On Mon, Sep 4, 2017 at 5:08 PM, Atin Mukherjee <amukherj at redhat.com> wrote: > > > On Mon, Sep 4, 2017 at 5:28 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: >> >> >1. On 80 nodes cluster, did you reboot only one node or multiple ones? >> Tried both, result is same, but the logs/stacks are from stopping and >> starting glusterd only on one server while others are running. >> >&g...