similar to: Glusterfs brick logs after upgrading from 3.6 to 3.8

[From http://blog.nixpanic.net/2017/07/glusterfs-3814-is-here.html and also on https://planet.gluster.org/] GlusterFS 3.8.14 is here, 3.8 even closer to End-Of-Life The 10th of the month has passed again, that means a 3.8.x update can't be far out. So, here it is, we're announcing the [9]availability of glusterfs-3.8.14. Note that this is one of the last updates in the 3.8

I managed to workaround this issue by addding "[arch=amd64]" to my apt source list for gluster like this: deb [arch=amd64] http://download.gluster.org/pub/gluster/glusterfs/3.8/3.8.14/Debian/jessie/apt jessie main In case that can help any others with the same siutation (where they also have i386 arch enabled on the computer). > -------- Original Message -------- > Subject:

What would the fix be exactly? The apt repos are built the same way they've been built for the last 3+ years and you're the first person to trip over whatever it is you're tripping over. And there have never been packages for i386 for Debian. ----- Original Message ----- > From: "mabi" <mabi at protonmail.ch> > To: "Gluster Users" <gluster-users

Hello, I want to upgrade from 3.8.11 to 3.8.14 on my Debian 8 (jessie) servers but it looks like the official GlusterFS apt repository has a mistake as you can see here: Get:14 http://download.gluster.org jessie InRelease [2'083 B] Get:15 http://download.gluster.org jessie/main amd64 Packages [1'602 B] Fetched 23.7 kB in 2s (10.6 kB/s) W: Failed to fetch

I also just noticed quite a few of the following warning messages in the quotad.log log file: [2017-08-01 07:59:27.834202] W [MSGID: 108027] [afr-common.c:2496:afr_discover_done] 0-myvolume-replicate-0: no read subvols for (null) > -------- Original Message -------- > Subject: [Gluster-users] Quotas not working after adding arbiter brick to replica 2 > Local Time: August 1, 2017 8:49 AM

Hello, As you might have read in my previous post on the mailing list I have added an arbiter node to my GlusterFS 3.8.11 replica 2 volume. After some healing issues and help of Ravi that could get fixed but now I just noticed that my quotas are all gone. When I run the following command: glusterfs volume quota myvolume list There is no output... In the /var/log/glusterfs/quotad.log I can see the

This is exactly the problem, Systemctl stop glusterd does *not* kill the brick processes. On CentOS with gluster 3.10.x there is also a service, meant to only stop glusterfsd (brick processes). I think the reboot process may not be properly stopping glusterfsd or network or firewall may be stopped before glusterfsd and so the nodes go into the long timeout. Once again , in my case a simple

If your VMs use ext4 also check this: https://joejulian.name/blog/keeping-your-vms-from-going- read-only-when-encountering-a-ping-timeout-in-glusterfs/ I asked him what to do for VMs using XFS and he said he could not find a fix (setting to change) for those. HTH, Diego On Sep 8, 2017 6:19 AM, "Diego Remolina" <dijuremo at gmail.com> wrote: > The issue of I/O stopping may

This is the qemu log of instance: [2017-09-08 09:31:48.381077] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded in the last 1 seconds, disconnecting. [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (-->

I think this should be considered a bug If you have a server crash, glusterfsd process obviously doesn't exit properly and thus this could least to IO stop ? And server crashes are the main reason to use a redundant filesystem like gluster Il 8 set 2017 12:43 PM, "Diego Remolina" <dijuremo at gmail.com> ha scritto: This is exactly the problem, Systemctl stop glusterd does

Some corrections about the previous mails. Problem does not happen when no volumes created. Problem happens volumes created but in stopped state. Problem also happens when volumes started state. Below is the 5 stack traces taken by 10 min intervals and volumes stopped state. --1-- Thread 8 (Thread 0x7f413f3a7700 (LWP 104249)): #0 0x0000003d99c0f00d in nanosleep () from /lib64/libpthread.so.0 #1

On Tue, Sep 5, 2017 at 6:13 PM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Some corrections about the previous mails. Problem does not happen > when no volumes created. > Problem happens volumes created but in stopped state. Problem also > happens when volumes started state. > Below is the 5 stack traces taken by 10 min intervals and volumes stopped > state. > As

Hi Milind, Anything new about the issue? Can you able to find the problem, anything else you need? I will continue with two clusters each 40 servers, so I will not be able to provide any further info for 80 servers. On Fri, Sep 1, 2017 at 10:30 AM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Hi, > You can find pstack sampes here: >

Restarting glusterd causes the same thing. I tried with 3.12.rc0, 3.10.5. 3.8.15, 3.7.20 all same behavior. My OS is centos 6.9, I tried with centos 6.8 problem remains... Only way to a healthy state is destroy gluster config/rpms, reinstall and recreate volumes. On Thu, Aug 24, 2017 at 8:49 AM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Here you can find 10 stack trace samples

Hi Diego, indeed glusterfsd processes are runnin and it is the reason I do server reboot instead of systemctl glusterd stop. Is killall different from reboot in a way glusterfsd processes are terminated in CentOS (init 1?)? However I will try this and let you know. -ps On Fri, Sep 8, 2017 at 12:19 PM, Diego Remolina <dijuremo at gmail.com> wrote: > The issue of I/O stopping may also

----- Original Message ----- > From: "Milind Changire" <mchangir at redhat.com> > To: "Serkan ?oban" <cobanserkan at gmail.com> > Cc: "Gluster Users" <gluster-users at gluster.org> > Sent: Saturday, September 2, 2017 11:44:40 PM > Subject: Re: [Gluster-users] Glusterd proccess hangs on reboot > > No worries Serkan, > You can

Thank you Gaurav, Here is more findings: Problem does not happen using only 20 servers each has 68 bricks. (peer probe only 20 servers) If we use 40 servers with single volume, glusterd cpu %100 state continues for 5 minutes and it goes to normal state. with 80 servers we have no working state yet... On Thu, Aug 24, 2017 at 1:33 PM, Gaurav Yadav <gyadav at redhat.com> wrote: > > I am

Till now I haven't found anything significant. Can you send me gluster logs along with command-history-logs for these scenarios: Scenario1 : 20 servers Scenario2 : 40 servers Scenario3: 80 Servers Thanks Gaurav On Mon, Aug 28, 2017 at 11:22 AM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Hi Gaurav, > Any progress about the problem? > > On Thursday, August 24,

Hi Atin, Do you have time to check the logs? On Wed, Aug 23, 2017 at 10:02 AM, Serkan ?oban <cobanserkan at gmail.com> wrote: > Same thing happens with 3.12.rc0. This time perf top shows hanging in > libglusterfs.so and below is the glusterd logs, which are different > from 3.10. > With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see > brick processes come

i usually change event threads to 4. But those logs are from a default installation. On Sun, Sep 3, 2017 at 9:52 PM, Ben Turner <bturner at redhat.com> wrote: > ----- Original Message ----- >> From: "Ben Turner" <bturner at redhat.com> >> To: "Serkan ?oban" <cobanserkan at gmail.com> >> Cc: "Gluster Users" <gluster-users at