search for: cobanserkan

...er gets > rebooted, glusterd process hangs. > > Could you please provide the logs which led glusterd to hang for all the > cases along with gusterd process utilization. > > > Thanks > Gaurav > > > > > > > On Tue, Aug 29, 2017 at 2:44 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: >> >> Here is the requested logs: >> >> https://www.dropbox.com/s/vt187h0gtu5doip/gluster_logs_20_40_80_servers.zip?dl=0 >> >> >> On Tue, Aug 29, 2017 at 7:48 AM, Gaurav Yadav <gyadav at redhat.com> wrote: >> > Till no...

glusterd returned to normal, here is the logs: https://www.dropbox.com/s/41jx2zn3uizvr53/80servers_glusterd_normal_status.zip?dl=0 On Tue, Aug 29, 2017 at 1:47 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Here is the logs after stopping all three volumes and restarting > glusterd in all nodes. I waited 70 minutes after glusterd restart but > it is still consuming %100 CPU. > https://www.dropbox.com/s/pzl0f198v03twx3/80servers_after_glusterd_restart.zip?dl=0 >...

...gnificant. > > Can you send me gluster logs along with command-history-logs for these > scenarios: > Scenario1 : 20 servers > Scenario2 : 40 servers > Scenario3: 80 Servers > > > Thanks > Gaurav > > > > On Mon, Aug 28, 2017 at 11:22 AM, Serkan ?oban <cobanserkan at gmail.com> > wrote: >> >> Hi Gaurav, >> Any progress about the problem? >> >> On Thursday, August 24, 2017, Serkan ?oban <cobanserkan at gmail.com> wrote: >>> >>> Thank you Gaurav, >>> Here is more findings: >>> Proble...

...However, you have mentioned that when a node from 80 server cluster gets rebooted, glusterd process hangs. Could you please provide the logs which led glusterd to hang for all the cases along with gusterd process utilization. Thanks Gaurav On Tue, Aug 29, 2017 at 2:44 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Here is the requested logs: > https://www.dropbox.com/s/vt187h0gtu5doip/gluster_logs_ > 20_40_80_servers.zip?dl=0 > > > On Tue, Aug 29, 2017 at 7:48 AM, Gaurav Yadav <gyadav at redhat.com> wrote: > > Till now I haven't found anything sign...

Hi Gaurav, Any progress about the problem? On Thursday, August 24, 2017, Serkan ?oban <cobanserkan at gmail.com> wrote: > Thank you Gaurav, > Here is more findings: > Problem does not happen using only 20 servers each has 68 bricks. > (peer probe only 20 servers) > If we use 40 servers with single volume, glusterd cpu %100 state > continues for 5 minutes and it goes to norm...

Till now I haven't found anything significant. Can you send me gluster logs along with command-history-logs for these scenarios: Scenario1 : 20 servers Scenario2 : 40 servers Scenario3: 80 Servers Thanks Gaurav On Mon, Aug 28, 2017 at 11:22 AM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Hi Gaurav, > Any progress about the problem? > > On Thursday, August 24, 2017, Serkan ?oban <cobanserkan at gmail.com> wrote: > >> Thank you Gaurav, >> Here is more findings: >> Problem does not happen using only 20 servers each has...

...3 as the rebuild time are bottlenecked by matrix operations which scale as the square of the number of data stripes. There are some savings because of larger data chunks but we ended up using 8+3 and heal times are about half compared to 16+3. -Alastair On 30 June 2017 at 02:22, Serkan ?oban <cobanserkan at gmail.com> wrote: > >Thanks for the reply. We will mainly use this for archival - near-cold > storage. > Archival usage is good for EC > > >Anything, from your experience, to keep in mind while planning large > installations? > I am using 3.7.11 and only problem is...

----- Original Message ----- > From: "Ben Turner" <bturner at redhat.com> > To: "Serkan ?oban" <cobanserkan at gmail.com> > Cc: "Gluster Users" <gluster-users at gluster.org> > Sent: Sunday, September 3, 2017 2:30:31 PM > Subject: Re: [Gluster-users] Glusterd proccess hangs on reboot > > ----- Original Message ----- > > From: "Milind Changire" <mchang...

I am working on it and will share my findings as soon as possible. Thanks Gaurav On Thu, Aug 24, 2017 at 3:58 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Restarting glusterd causes the same thing. I tried with 3.12.rc0, > 3.10.5. 3.8.15, 3.7.20 all same behavior. > My OS is centos 6.9, I tried with centos 6.8 problem remains... > Only way to a healthy state is destroy gluster config/rpms, reinstall > and rec...

...change event threads to 4. But those logs are from a default installation. On Sun, Sep 3, 2017 at 9:52 PM, Ben Turner <bturner at redhat.com> wrote: > ----- Original Message ----- >> From: "Ben Turner" <bturner at redhat.com> >> To: "Serkan ?oban" <cobanserkan at gmail.com> >> Cc: "Gluster Users" <gluster-users at gluster.org> >> Sent: Sunday, September 3, 2017 2:30:31 PM >> Subject: Re: [Gluster-users] Glusterd proccess hangs on reboot >> >> ----- Original Message ----- >> > From: "Milind Ch...

...th 80 servers we have no working state yet... On Thu, Aug 24, 2017 at 1:33 PM, Gaurav Yadav <gyadav at redhat.com> wrote: > > I am working on it and will share my findings as soon as possible. > > > Thanks > Gaurav > > On Thu, Aug 24, 2017 at 3:58 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: >> >> Restarting glusterd causes the same thing. I tried with 3.12.rc0, >> 3.10.5. 3.8.15, 3.7.20 all same behavior. >> My OS is centos 6.9, I tried with centos 6.8 problem remains... >> Only way to a healthy state is destroy gluster config/rpms...

...kan, You can continue to use your 40 node clusters. The backtrace has resolved the function names and it *should* be sufficient to debug the issue. Thanks for letting us know. We'll post on this thread again to notify you about the findings. On Sat, Sep 2, 2017 at 2:42 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Hi Milind, > > Anything new about the issue? Can you able to find the problem, > anything else you need? > I will continue with two clusters each 40 servers, so I will not be > able to provide any further info for 80 servers. > > On Fri, Sep 1, 201...

...ecked by matrix operations which scale as the square > of the number of data stripes. There are some savings because of larger > data chunks but we ended up using 8+3 and heal times are about half compared > to 16+3. > > -Alastair > > On 30 June 2017 at 02:22, Serkan ?oban <cobanserkan at gmail.com> wrote: >> >> >Thanks for the reply. We will mainly use this for archival - near-cold >> > storage. >> Archival usage is good for EC >> >> >Anything, from your experience, to keep in mind while planning large >> > installations?...

----- Original Message ----- > From: "Milind Changire" <mchangir at redhat.com> > To: "Serkan ?oban" <cobanserkan at gmail.com> > Cc: "Gluster Users" <gluster-users at gluster.org> > Sent: Saturday, September 2, 2017 11:44:40 PM > Subject: Re: [Gluster-users] Glusterd proccess hangs on reboot > > No worries Serkan, > You can continue to use your 40 node clusters. > &gt...

An upstream bug would be ideal as github issue is mainly used for enhancements. In the mean time, could you point to the exact failure shown at the command line and the log entry from cli.log? On Wed, Aug 23, 2017 at 12:10 AM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Hi, I think this is the line limiting brick count: > https://github.com/gluster/glusterfs/blob/c136024613c697fec87aaff3a07086 > 2b92c57977/cli/src/cli-cmd-parser.c#L84 > > Can gluster-devs increase this limit? Should I open a github issue? > > On Mon,...

...ell. I use the disks in raid mode on all of them. I've tried a few non-Dell boxes and they all perform well even though some of them are very old. I've also tried disabling Idrac, the Idrac nic, virtual storage for Idrac with no sucess.. On Mon, Feb 26, 2018 at 9:28 AM, Serkan ?oban <cobanserkan at gmail.com> wrote: > I don't think it is related with iDRAC itself but some configuration > is wrong or there is some hw error. > Did you check battery of raid controller? Do you use disks in jbod > mode or raid mode? > > On Mon, Feb 26, 2018 at 6:12 PM, Ryan Wilkinson &...

...Could you be able to provide the pstack dump of the glusterd process? > > On Wed, 23 Aug 2017 at 20:22, Atin Mukherjee <amukherj at redhat.com> wrote: >> >> Not yet. Gaurav will be taking a look at it tomorrow. >> >> On Wed, 23 Aug 2017 at 20:14, Serkan ?oban <cobanserkan at gmail.com> wrote: >>> >>> Hi Atin, >>> >>> Do you have time to check the logs? >>> >>> On Wed, Aug 23, 2017 at 10:02 AM, Serkan ?oban <cobanserkan at gmail.com> >>> wrote: >>> > Same thing happens with 3.12.rc0....

Not yet. Gaurav will be taking a look at it tomorrow. On Wed, 23 Aug 2017 at 20:14, Serkan ?oban <cobanserkan at gmail.com> wrote: > Hi Atin, > > Do you have time to check the logs? > > On Wed, Aug 23, 2017 at 10:02 AM, Serkan ?oban <cobanserkan at gmail.com> > wrote: > > Same thing happens with 3.12.rc0. This time perf top shows hanging in > > libglusterfs.so and be...

...terd causes the same thing. I tried with 3.12.rc0, 3.10.5. 3.8.15, 3.7.20 all same behavior. My OS is centos 6.9, I tried with centos 6.8 problem remains... Only way to a healthy state is destroy gluster config/rpms, reinstall and recreate volumes. On Thu, Aug 24, 2017 at 8:49 AM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Here you can find 10 stack trace samples from glusterd. I wait 10 > seconds between each trace. > https://www.dropbox.com/s/9f36goq5xn3p1yt/glusterd_pstack.zip?dl=0 > > Content of the first stack trace is here: > > Thread 8 (Thread 0x7f7a8cd4e700 (LWP...

On Mon, Sep 4, 2017 at 5:28 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > >1. On 80 nodes cluster, did you reboot only one node or multiple ones? > Tried both, result is same, but the logs/stacks are from stopping and > starting glusterd only on one server while others are running. > > >2. Are you sure that pstack output wa...