Roman
2015-Aug-04 13:06 UTC
[Gluster-users] back to problems: gluster 3.5.4, qemu and debian 8
Hi all, I'm back and tested those things. Michael was right. I've enabled read-ahead option and nothing changed. So the thing causes the problem with libgfapi and d8 virtio drivers is performance.write-behind. If it is off, everything works perfect. If I set it On, different problems are as result (which were confirmed by other user with same configuration). 2015-07-20 1:53 GMT+03:00 Roman <romeo.r at gmail.com>:> Thanks for your reply. > I will test these options as soon as I'm back from ma vacation (2 weeks > from now on). I'll be too far from servers to change something even on > testing volume 8) > > 2015-07-19 23:47 GMT+03:00 Niels de Vos <ndevos at redhat.com>: > >> On Sat, Jul 18, 2015 at 03:56:37PM +0000, Michael Mol wrote: >> > I think you'll find it's the write-behind that was killing you. >> > Write-behind opens you up to a number of data consistency issues, and I >> > strongly disrecommend it unless you have a rock-solid infrastructure >> from >> > the writer all the way to the disk the data ultimately sits on. >> >> The suggestion to disable these two options was to change the access >> pattern done by Qemu+libgfapi. Without looking into the source code, I >> do not know how write-behind and read-ahead play together. Anything that >> is "written behind" should get flushed when a "read ahead" operation >> overlaps a "written behind" area. There have been issues with >> write-behind before, where the flushing was not done in some corner >> case (https://github.com/gluster/glusterfs/commit/b0515e2a). Maybe this >> is something similar. >> >> > I bet that if you re-enabled read-ahead, you won't see the problem. Just >> > leave write-behind off. >> >> Indeed, write-behind is most likely. Both results of one of the options >> disabled at the time would be interesting to have. >> >> Once it is clear which option causes the problem, we can analyze the >> access pattern and hopefully fix the xlator. >> >> Thanks, >> Niels >> >> >> > On Sat, Jul 18, 2015, 10:44 AM Roman <romeo.r at gmail.com> wrote: >> > >> > solved after I've added (thanks to Niels de Vos) these options to the >> > volumes: >> > >> > performance.read-ahead: off >> > >> > performance.write-behind: off >> > >> > >> > 2015-07-15 17:23 GMT+03:00 Roman <romeo.r at gmail.com>: >> > >> > hey, >> > >> > I've updated the bug, if some1 has some ideas - share plz. >> > >> > https://bugzilla.redhat.com/show_bug.cgi?id=1242913 >> > >> > >> > 2015-07-14 19:14 GMT+03:00 Kaushal M <kshlmster at gmail.com>: >> > >> > Just a wild guess. What is the filesystem used for the debian 8 >> > installation? It could be the culprit. >> > >> > On Tue, Jul 14, 2015 at 7:27 PM, Roman <romeo.r at gmail.com> wrote: >> > > I've done this way: installed debian8 on local disks using netinstall >> iso, >> > > created a template of it and then cloned (full clone) it to glusterfs >> > > storage backend. VM boots and runs fine... untill I start to install >> > > something massive (DE ie). Last time it was mate failed to install >> due to >> > > python-gtk2 package problems (complaing that it could not compile it) >> > > >> > >> > > 2015-07-14 16:37 GMT+03:00 Scott Harvanek <scott.harvanek at login.com>: >> > >> >> > >> What happens if you install from a full CD and not a net-install? >> > >> >> > >> Limit the variables. Currently you are relying on remote mirrors and >> > >> Internet connectivity. >> > >> >> > >> It's either a Proxmox or Debian issue, I really don't think it's >> Gluster. >> > >> We have hundreds of Jessie installs running on GlusterFS backends. >> > >> >> > >> -- >> > >> Scott H. >> > >> Login, LLC. >> > >> >> > >> >> > >> >> > >> Roman >> > >> July 14, 2015 at 9:30 AM >> > >> Hey, >> > >> >> > >> thanks for reply. >> > >> If it would be networking related, it would affect everything. But >> it is >> > >> only debian 8 which won't install. >> > >> And yes, i did iperf test between gluster and proxmox nodes. Its ok. >> > >> Installation fails on every node, where i try to install d8. >> Sometimes it >> > >> goes well (today 1 of 6 tries was fine). Other distros install fine. >> > >> Sometimes installation process finishes, but VM won't start, just >> hangs >> > >> with errors like in this attached. >> > >> >> > >> >> > >> >> > >> >> > >> -- >> > >> Best regards, >> > >> Roman. >> > >> Scott Harvanek >> > >> July 14, 2015 at 9:17 AM >> > >> We don't have this issue, I'll take a stab tho- >> > >> >> > >> Have you confirmed everything is good on the network side of things? >> > >> MTU/Loss/Errors? >> > >> >> > >> Is your inconsistency linked to one specific brick? Have you tried >> > running >> > >> a replica instead of distributed? >> > >> >> > >> >> > >> >> > >> _______________________________________________ >> > >> Gluster-users mailing list >> > >> > >> Gluster-users at gluster.org >> > >> http://www.gluster.org/mailman/listinfo/gluster-users >> > >> Roman >> > >> July 14, 2015 at 6:38 AM >> > >> here is one of the errors example. its like files that debian >> installer >> > >> copies to the virtual disk that is located on glusterfs storage are >> > getting >> > >> corrupted. >> > >> in-target is /dev/vda1 >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> -- >> > >> Best regards, >> > >> Roman. >> > >> _______________________________________________ >> > >> Gluster-users mailing list >> > >> Gluster-users at gluster.org >> > >> http://www.gluster.org/mailman/listinfo/gluster-users >> > >> Roman >> > >> July 14, 2015 at 4:50 AM >> > >> Ubuntu 14.04 LTS base install and then mate install were fine! >> > >> >> > >> >> > >> >> > >> >> > >> -- >> > >> Best regards, >> > >> Roman. >> > >> _______________________________________________ >> > >> Gluster-users mailing list >> > >> Gluster-users at gluster.org >> > >> http://www.gluster.org/mailman/listinfo/gluster-users >> > >> Roman >> > >> July 13, 2015 at 7:35 PM >> > >> Bah... the randomness of this issue is killing me. >> > >> Not only HA volumes are affected. Got an error during installation >> of d8 >> > >> with mate (on python-gtk2 pkg) on Distributed volume also. >> > >> I've checked the MD5SUM of installation iso, its ok. >> > >> >> > >> Shortly after that on the same VE node I installed D7 with Gnome >> without >> > >> any problem on the HA glusterf volume. >> > >> >> > >> And on the same VE node I've installed D8 with both Mate and Gnome >> using >> > >> local storage disks without problems. There is a bug somewhere in >> > gluster or >> > >> qemu... Proxmox uses RH kernel btw: >> > >> >> > >> Linux services 2.6.32-37-pve >> > >> QEMU emulator version 2.2.1 >> > >> glusterfs 3.6.4 >> > >> >> > >> any ideas? >> > >> I'm ready to help to investigate this bug. >> > >> When sun will shine, I'll try to install latest Ubuntu also. But now >> I'm >> > >> going to sleep. >> > >> >> > >> >> > >> >> > >> >> > >> -- >> > >> Best regards, >> > >> Roman. >> > >> _______________________________________________ >> > >> Gluster-users mailing list >> > >> Gluster-users at gluster.org >> > >> http://www.gluster.org/mailman/listinfo/gluster-users >> > >> >> > >> >> > > >> > > >> > > >> > > -- >> > > Best regards, >> > > Roman. >> > > >> > > _______________________________________________ >> > > Gluster-users mailing list >> > > Gluster-users at gluster.org >> > > http://www.gluster.org/mailman/listinfo/gluster-users >> > >> > >> > >> > -- >> > >> > Best regards, >> > Roman. >> > >> > >> > >> > -- >> > >> > Best regards, >> > Roman. >> > >> > _______________________________________________ >> > Gluster-users mailing list >> > Gluster-users at gluster.org >> > http://www.gluster.org/mailman/listinfo/gluster-users >> >> > _______________________________________________ >> > Gluster-users mailing list >> > Gluster-users at gluster.org >> > http://www.gluster.org/mailman/listinfo/gluster-users >> >> > > > -- > Best regards, > Roman. >-- Best regards, Roman. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150804/9c882fe9/attachment.html>
Niels de Vos
2015-Aug-06 02:26 UTC
[Gluster-users] back to problems: gluster 3.5.4, qemu and debian 8
On Tue, Aug 04, 2015 at 04:06:50PM +0300, Roman wrote:> Hi all, > I'm back and tested those things. > Michael was right. I've enabled read-ahead option and nothing changed. > So the thing causes the problem with libgfapi and d8 virtio drivers > is performance.write-behind. If it is off, everything works perfect. If I > set it On, different problems are as result (which were confirmed by other > user with same configuration).Thanks for the follow up, Roman! We'll need to think about how to trace the I/O pattern that the installation does, and see if we can emulate that in a test-case. Once we can easily reproduce the problem, investigating the cause gets easier. I've got some more questions that should help us: - Do you know if any non-Proxmox environments had the same issue? - Is it possible to notice the failure in the guest and abort? When we track the I/O pattern, the failure would be at the end, and not somewhere in the trace (almost impossible to find). Thanks, Niels> > 2015-07-20 1:53 GMT+03:00 Roman <romeo.r at gmail.com>: > > > Thanks for your reply. > > I will test these options as soon as I'm back from ma vacation (2 weeks > > from now on). I'll be too far from servers to change something even on > > testing volume 8) > > > > 2015-07-19 23:47 GMT+03:00 Niels de Vos <ndevos at redhat.com>: > > > >> On Sat, Jul 18, 2015 at 03:56:37PM +0000, Michael Mol wrote: > >> > I think you'll find it's the write-behind that was killing you. > >> > Write-behind opens you up to a number of data consistency issues, and I > >> > strongly disrecommend it unless you have a rock-solid infrastructure > >> from > >> > the writer all the way to the disk the data ultimately sits on. > >> > >> The suggestion to disable these two options was to change the access > >> pattern done by Qemu+libgfapi. Without looking into the source code, I > >> do not know how write-behind and read-ahead play together. Anything that > >> is "written behind" should get flushed when a "read ahead" operation > >> overlaps a "written behind" area. There have been issues with > >> write-behind before, where the flushing was not done in some corner > >> case (https://github.com/gluster/glusterfs/commit/b0515e2a). Maybe this > >> is something similar. > >> > >> > I bet that if you re-enabled read-ahead, you won't see the problem. Just > >> > leave write-behind off. > >> > >> Indeed, write-behind is most likely. Both results of one of the options > >> disabled at the time would be interesting to have. > >> > >> Once it is clear which option causes the problem, we can analyze the > >> access pattern and hopefully fix the xlator. > >> > >> Thanks, > >> Niels > >> > >> > >> > On Sat, Jul 18, 2015, 10:44 AM Roman <romeo.r at gmail.com> wrote: > >> > > >> > solved after I've added (thanks to Niels de Vos) these options to the > >> > volumes: > >> > > >> > performance.read-ahead: off > >> > > >> > performance.write-behind: off > >> > > >> > > >> > 2015-07-15 17:23 GMT+03:00 Roman <romeo.r at gmail.com>: > >> > > >> > hey, > >> > > >> > I've updated the bug, if some1 has some ideas - share plz. > >> > > >> > https://bugzilla.redhat.com/show_bug.cgi?id=1242913 > >> > > >> > > >> > 2015-07-14 19:14 GMT+03:00 Kaushal M <kshlmster at gmail.com>: > >> > > >> > Just a wild guess. What is the filesystem used for the debian 8 > >> > installation? It could be the culprit. > >> > > >> > On Tue, Jul 14, 2015 at 7:27 PM, Roman <romeo.r at gmail.com> wrote: > >> > > I've done this way: installed debian8 on local disks using netinstall > >> iso, > >> > > created a template of it and then cloned (full clone) it to glusterfs > >> > > storage backend. VM boots and runs fine... untill I start to install > >> > > something massive (DE ie). Last time it was mate failed to install > >> due to > >> > > python-gtk2 package problems (complaing that it could not compile it) > >> > > > >> > > >> > > 2015-07-14 16:37 GMT+03:00 Scott Harvanek <scott.harvanek at login.com>: > >> > >> > >> > >> What happens if you install from a full CD and not a net-install? > >> > >> > >> > >> Limit the variables. Currently you are relying on remote mirrors and > >> > >> Internet connectivity. > >> > >> > >> > >> It's either a Proxmox or Debian issue, I really don't think it's > >> Gluster. > >> > >> We have hundreds of Jessie installs running on GlusterFS backends. > >> > >> > >> > >> -- > >> > >> Scott H. > >> > >> Login, LLC. > >> > >> > >> > >> > >> > >> > >> > >> Roman > >> > >> July 14, 2015 at 9:30 AM > >> > >> Hey, > >> > >> > >> > >> thanks for reply. > >> > >> If it would be networking related, it would affect everything. But > >> it is > >> > >> only debian 8 which won't install. > >> > >> And yes, i did iperf test between gluster and proxmox nodes. Its ok. > >> > >> Installation fails on every node, where i try to install d8. > >> Sometimes it > >> > >> goes well (today 1 of 6 tries was fine). Other distros install fine. > >> > >> Sometimes installation process finishes, but VM won't start, just > >> hangs > >> > >> with errors like in this attached. > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> -- > >> > >> Best regards, > >> > >> Roman. > >> > >> Scott Harvanek > >> > >> July 14, 2015 at 9:17 AM > >> > >> We don't have this issue, I'll take a stab tho- > >> > >> > >> > >> Have you confirmed everything is good on the network side of things? > >> > >> MTU/Loss/Errors? > >> > >> > >> > >> Is your inconsistency linked to one specific brick? Have you tried > >> > running > >> > >> a replica instead of distributed? > >> > >> > >> > >> > >> > >> > >> > >> _______________________________________________ > >> > >> Gluster-users mailing list > >> > > >> > >> Gluster-users at gluster.org > >> > >> http://www.gluster.org/mailman/listinfo/gluster-users > >> > >> Roman > >> > >> July 14, 2015 at 6:38 AM > >> > >> here is one of the errors example. its like files that debian > >> installer > >> > >> copies to the virtual disk that is located on glusterfs storage are > >> > getting > >> > >> corrupted. > >> > >> in-target is /dev/vda1 > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> -- > >> > >> Best regards, > >> > >> Roman. > >> > >> _______________________________________________ > >> > >> Gluster-users mailing list > >> > >> Gluster-users at gluster.org > >> > >> http://www.gluster.org/mailman/listinfo/gluster-users > >> > >> Roman > >> > >> July 14, 2015 at 4:50 AM > >> > >> Ubuntu 14.04 LTS base install and then mate install were fine! > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> -- > >> > >> Best regards, > >> > >> Roman. > >> > >> _______________________________________________ > >> > >> Gluster-users mailing list > >> > >> Gluster-users at gluster.org > >> > >> http://www.gluster.org/mailman/listinfo/gluster-users > >> > >> Roman > >> > >> July 13, 2015 at 7:35 PM > >> > >> Bah... the randomness of this issue is killing me. > >> > >> Not only HA volumes are affected. Got an error during installation > >> of d8 > >> > >> with mate (on python-gtk2 pkg) on Distributed volume also. > >> > >> I've checked the MD5SUM of installation iso, its ok. > >> > >> > >> > >> Shortly after that on the same VE node I installed D7 with Gnome > >> without > >> > >> any problem on the HA glusterf volume. > >> > >> > >> > >> And on the same VE node I've installed D8 with both Mate and Gnome > >> using > >> > >> local storage disks without problems. There is a bug somewhere in > >> > gluster or > >> > >> qemu... Proxmox uses RH kernel btw: > >> > >> > >> > >> Linux services 2.6.32-37-pve > >> > >> QEMU emulator version 2.2.1 > >> > >> glusterfs 3.6.4 > >> > >> > >> > >> any ideas? > >> > >> I'm ready to help to investigate this bug. > >> > >> When sun will shine, I'll try to install latest Ubuntu also. But now > >> I'm > >> > >> going to sleep. > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> -- > >> > >> Best regards, > >> > >> Roman. > >> > >> _______________________________________________ > >> > >> Gluster-users mailing list > >> > >> Gluster-users at gluster.org > >> > >> http://www.gluster.org/mailman/listinfo/gluster-users > >> > >> > >> > >> > >> > > > >> > > > >> > > > >> > > -- > >> > > Best regards, > >> > > Roman. > >> > > > >> > > _______________________________________________ > >> > > Gluster-users mailing list > >> > > Gluster-users at gluster.org > >> > > http://www.gluster.org/mailman/listinfo/gluster-users > >> > > >> > > >> > > >> > -- > >> > > >> > Best regards, > >> > Roman. > >> > > >> > > >> > > >> > -- > >> > > >> > Best regards, > >> > Roman. > >> > > >> > _______________________________________________ > >> > Gluster-users mailing list > >> > Gluster-users at gluster.org > >> > http://www.gluster.org/mailman/listinfo/gluster-users > >> > >> > _______________________________________________ > >> > Gluster-users mailing list > >> > Gluster-users at gluster.org > >> > http://www.gluster.org/mailman/listinfo/gluster-users > >> > >> > > > > > > -- > > Best regards, > > Roman. > > > > > > -- > Best regards, > Roman.