search for: palousetech

...you disable cluster.eager-lock and see if that makes any difference: # gluster volume set <VOL> cluster.eager-lock off Do also capture the volume profile again if you still see performance issues after disabling eager-lock. -Krutika On Wed, May 30, 2018 at 6:55 AM, Jim Kusznir <jim at palousetech.com> wrote: > I also finally found the following in my system log on one server: > > [10679.524491] INFO: task glusterclogro:14933 blocked for more than 120 > seconds. > [10679.525826] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > disables this message. &gt...

I would check disks status and accessibility of mount points where your gluster volumes reside. On Tue, May 29, 2018, 22:28 Jim Kusznir <jim at palousetech.com> wrote: > On one ovirt server, I'm now seeing these messages: > [56474.239725] blk_update_request: 63 callbacks suppressed > [56474.239732] blk_update_request: I/O error, dev dm-2, sector 0 > [56474.240602] blk_update_request: I/O error, dev dm-2, sector 3905945472 > [564...

...Nothing amiss with volume info and status. Can you check the agent.log and broker.log - will be under /var/log/ovirt-hosted-engine-ha/ Also the gluster client logs - under /var/log/glusterfs/rhev-data-center-mnt-glusterSD<volume>.log On Wed, May 30, 2018 at 12:08 PM, Jim Kusznir <jim at palousetech.com> wrote: > I believe the gluster data store for the engine is up and working > correctly. The rest are not mounted, as the engine hasn't managed to start > correctly yet. I did perform the copy-a-junk-file into the data store, > then check to ensure its there on another hos...

...l there. I have some more VMs I need to get back up and running. How do I fix this? ovirt1, for unknown reasons, will not work. Attempts to bring it online fail, and I haven't figured out what log file to look in yet for more details. On Wed, May 30, 2018 at 9:36 AM, Jim Kusznir <jim at palousetech.com> wrote: > Hi all again: > > I'm now subscribed to gluster-users as well, so I should get any replies > from that side too. > > At this point, I am seeing acceptable (although slower than I expect) > performance much of the time, with periodic massive spikes in laten...

On Thu, May 31, 2018 at 3:16 AM, Jim Kusznir <jim at palousetech.com> wrote: > I've been back at it, and still am unable to get more than one of my > physical nodes to come online in ovirt, nor am I able to get more than the > two gluster volumes (storage domains) to show online within ovirt. > > In Storage -> Volumes, they all show of...

...ber.com:/gluster/brick1/engine > ovirt2.* shows decent numbers. Is everything OK with the brick on ovirt1? > Are the bricks of engine volume on both these servers identical in terms > of their config? > > -Krutika > > > On Wed, May 30, 2018 at 3:07 PM, Jim Kusznir <jim at palousetech.com> wrote: > >> Hi: >> >> Thank you. I was finally able to get my cluster minimally functional >> again (its 2am local here; had to be up by 6am). I set the >> cluster.eager-lock off, but I'm still seeing performance issues. Here's >> the engine...

...latencies on the brick at ovirt1.nwfiber.com:/gluster/brick1/engine ovirt2.* shows decent numbers. Is everything OK with the brick on ovirt1? Are the bricks of engine volume on both these servers identical in terms of their config? -Krutika On Wed, May 30, 2018 at 3:07 PM, Jim Kusznir <jim at palousetech.com> wrote: > Hi: > > Thank you. I was finally able to get my cluster minimally functional > again (its 2am local here; had to be up by 6am). I set the > cluster.eager-lock off, but I'm still seeing performance issues. Here's > the engine volume: > > Brick: ov...

Do you see errors reported in the mount logs for the volume? If so, could you attach the logs? Any issues with your underlying disks. Can you also attach output of volume profiling? On Wed, May 30, 2018 at 12:13 AM, Jim Kusznir <jim at palousetech.com> wrote: > Ok, things have gotten MUCH worse this morning. I'm getting random errors > from VMs, right now, about a third of my VMs have been paused due to > storage issues, and most of the remaining VMs are not performing well. > > At this point, I am in full EMERGENCY m...

On Sun, Aug 6, 2017 at 4:42 AM, Jim Kusznir <jim at palousetech.com> wrote: > Well, after a very stressful weekend, I think I have things largely > working. Turns out that most of the above issues were caused by the linux > permissions of the exports for all three volumes (they had been reset to > 600; setting them to 774 or 770 fixed many of t...

[Adding gluster-users to look at the heal issue] On Tue, May 29, 2018 at 9:17 AM, Jim Kusznir <jim at palousetech.com> wrote: > Hello: > > I've been having some cluster and gluster performance issues lately. I > also found that my cluster was out of date, and was trying to apply updates > (hoping to fix some of these), and discovered the ovirt 4.1 repos were > taken completely offlin...