Krutika Dhananjay
2016-Jul-27 02:38 UTC
[Gluster-users] [Gluster-devel] 3.7.13 & proxmox/qemu
Yes please, could you file a bug against glusterfs for this issue? -Krutika On Wed, Jul 27, 2016 at 1:39 AM, David Gossage <dgossage at carouselchecks.com> wrote:> Has a bug report been filed for this issue or should l I create one with > the logs and results provided so far? > > *David Gossage* > *Carousel Checks Inc. | System Administrator* > *Office* 708.613.2284 > > On Fri, Jul 22, 2016 at 12:53 PM, David Gossage < > dgossage at carouselchecks.com> wrote: > >> >> >> >> On Fri, Jul 22, 2016 at 9:37 AM, Vijay Bellur <vbellur at redhat.com> wrote: >> >>> On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen <samppah at neutraali.net> >>> wrote: >>> > Here is a quick way how to test this: >>> > GlusterFS 3.7.13 volume with default settings with brick on ZFS >>> dataset. gluster-test1 is server and gluster-test2 is client mounting with >>> FUSE. >>> > >>> > Writing file with oflag=direct is not ok: >>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct >>> count=1 bs=1024000 >>> > dd: failed to open ?file?: Invalid argument >>> > >>> > Enable network.remote-dio on Gluster Volume: >>> > [root at gluster-test1 gluster]# gluster volume set gluster >>> network.remote-dio enable >>> > volume set: success >>> > >>> > Writing small file with oflag=direct is ok: >>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct >>> count=1 bs=1024000 >>> > 1+0 records in >>> > 1+0 records out >>> > 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s >>> > >>> > Writing bigger file with oflag=direct is ok: >>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct >>> count=100 bs=1M >>> > 100+0 records in >>> > 100+0 records out >>> > 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s >>> > >>> > Enable Sharding on Gluster Volume: >>> > [root at gluster-test1 gluster]# gluster volume set gluster >>> features.shard enable >>> > volume set: success >>> > >>> > Writing small file with oflag=direct is ok: >>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct >>> count=1 bs=1M >>> > 1+0 records in >>> > 1+0 records out >>> > 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s >>> > >>> > Writing bigger file with oflag=direct is not ok: >>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct >>> count=100 bs=1M >>> > dd: error writing ?file3?: Operation not permitted >>> > dd: closing output file ?file3?: Operation not permitted >>> > >>> >>> >>> Thank you for these tests! would it be possible to share the brick and >>> client logs? >>> >> >> Not sure if his tests are same as my setup but here is what I end up with >> >> Volume Name: glustershard >> Type: Replicate >> Volume ID: 0cc4efb6-3836-4caa-b24a-b3afb6e407c3 >> Status: Started >> Number of Bricks: 1 x 3 = 3 >> Transport-type: tcp >> Bricks: >> Brick1: 192.168.71.10:/gluster1/shard1/1 >> Brick2: 192.168.71.11:/gluster1/shard2/1 >> Brick3: 192.168.71.12:/gluster1/shard3/1 >> Options Reconfigured: >> features.shard-block-size: 64MB >> features.shard: on >> server.allow-insecure: on >> storage.owner-uid: 36 >> storage.owner-gid: 36 >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> network.remote-dio: enable >> cluster.eager-lock: enable >> performance.stat-prefetch: off >> performance.io-cache: off >> performance.quick-read: off >> cluster.self-heal-window-size: 1024 >> cluster.background-self-heal-count: 16 >> nfs.enable-ino32: off >> nfs.addr-namelookup: off >> nfs.disable: on >> performance.read-ahead: off >> performance.readdir-ahead: on >> >> >> >> dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/192.168.71.11\:_glustershard/ >> oflag=direct count=100 bs=1M >> 81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/ __DIRECT_IO_TEST__ >> .trashcan/ >> [root at ccengine2 ~]# dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/ >> 192.168.71.11\:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test >> oflag=direct count=100 bs=1M >> dd: error writing ?/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test?: >> Operation not permitted >> >> creates the 64M file in expected location then the shard is 0 >> >> # file: gluster1/shard1/1/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test >> >> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >> trusted.afr.dirty=0x000000000000000000000000 >> trusted.bit-rot.version=0x0200000000000000579231f3000e16e7 >> trusted.gfid=0xec6de302b35f427985639ca3e25d9df0 >> trusted.glusterfs.shard.block-size=0x0000000004000000 >> >> trusted.glusterfs.shard.file-size=0x0000000004000000000000000000000000000000000000010000000000000000 >> >> >> # file: gluster1/shard1/1/.shard/ec6de302-b35f-4279-8563-9ca3e25d9df0.1 >> >> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >> trusted.afr.dirty=0x000000000000000000000000 >> trusted.gfid=0x2bfd3cc8a727489b9a0474241548fe80 >> >> >>> Regards, >>> Vijay >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-users >> >> >> > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-devel >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160727/24271465/attachment.html>
On Tue, Jul 26, 2016 at 9:38 PM, Krutika Dhananjay <kdhananj at redhat.com> wrote:> Yes please, could you file a bug against glusterfs for this issue? >https://bugzilla.redhat.com/show_bug.cgi?id=1360785> > > -Krutika > > On Wed, Jul 27, 2016 at 1:39 AM, David Gossage < > dgossage at carouselchecks.com> wrote: > >> Has a bug report been filed for this issue or should l I create one with >> the logs and results provided so far? >> >> *David Gossage* >> *Carousel Checks Inc. | System Administrator* >> *Office* 708.613.2284 >> >> On Fri, Jul 22, 2016 at 12:53 PM, David Gossage < >> dgossage at carouselchecks.com> wrote: >> >>> >>> >>> >>> On Fri, Jul 22, 2016 at 9:37 AM, Vijay Bellur <vbellur at redhat.com> >>> wrote: >>> >>>> On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen < >>>> samppah at neutraali.net> wrote: >>>> > Here is a quick way how to test this: >>>> > GlusterFS 3.7.13 volume with default settings with brick on ZFS >>>> dataset. gluster-test1 is server and gluster-test2 is client mounting with >>>> FUSE. >>>> > >>>> > Writing file with oflag=direct is not ok: >>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct >>>> count=1 bs=1024000 >>>> > dd: failed to open ?file?: Invalid argument >>>> > >>>> > Enable network.remote-dio on Gluster Volume: >>>> > [root at gluster-test1 gluster]# gluster volume set gluster >>>> network.remote-dio enable >>>> > volume set: success >>>> > >>>> > Writing small file with oflag=direct is ok: >>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct >>>> count=1 bs=1024000 >>>> > 1+0 records in >>>> > 1+0 records out >>>> > 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s >>>> > >>>> > Writing bigger file with oflag=direct is ok: >>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct >>>> count=100 bs=1M >>>> > 100+0 records in >>>> > 100+0 records out >>>> > 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s >>>> > >>>> > Enable Sharding on Gluster Volume: >>>> > [root at gluster-test1 gluster]# gluster volume set gluster >>>> features.shard enable >>>> > volume set: success >>>> > >>>> > Writing small file with oflag=direct is ok: >>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct >>>> count=1 bs=1M >>>> > 1+0 records in >>>> > 1+0 records out >>>> > 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s >>>> > >>>> > Writing bigger file with oflag=direct is not ok: >>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct >>>> count=100 bs=1M >>>> > dd: error writing ?file3?: Operation not permitted >>>> > dd: closing output file ?file3?: Operation not permitted >>>> > >>>> >>>> >>>> Thank you for these tests! would it be possible to share the brick and >>>> client logs? >>>> >>> >>> Not sure if his tests are same as my setup but here is what I end up with >>> >>> Volume Name: glustershard >>> Type: Replicate >>> Volume ID: 0cc4efb6-3836-4caa-b24a-b3afb6e407c3 >>> Status: Started >>> Number of Bricks: 1 x 3 = 3 >>> Transport-type: tcp >>> Bricks: >>> Brick1: 192.168.71.10:/gluster1/shard1/1 >>> Brick2: 192.168.71.11:/gluster1/shard2/1 >>> Brick3: 192.168.71.12:/gluster1/shard3/1 >>> Options Reconfigured: >>> features.shard-block-size: 64MB >>> features.shard: on >>> server.allow-insecure: on >>> storage.owner-uid: 36 >>> storage.owner-gid: 36 >>> cluster.server-quorum-type: server >>> cluster.quorum-type: auto >>> network.remote-dio: enable >>> cluster.eager-lock: enable >>> performance.stat-prefetch: off >>> performance.io-cache: off >>> performance.quick-read: off >>> cluster.self-heal-window-size: 1024 >>> cluster.background-self-heal-count: 16 >>> nfs.enable-ino32: off >>> nfs.addr-namelookup: off >>> nfs.disable: on >>> performance.read-ahead: off >>> performance.readdir-ahead: on >>> >>> >>> >>> dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/192.168.71.11\:_glustershard/ >>> oflag=direct count=100 bs=1M >>> 81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/ __DIRECT_IO_TEST__ >>> .trashcan/ >>> [root at ccengine2 ~]# dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/ >>> 192.168.71.11\:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test >>> oflag=direct count=100 bs=1M >>> dd: error writing ?/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test?: >>> Operation not permitted >>> >>> creates the 64M file in expected location then the shard is 0 >>> >>> # file: >>> gluster1/shard1/1/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test >>> >>> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>> trusted.afr.dirty=0x000000000000000000000000 >>> trusted.bit-rot.version=0x0200000000000000579231f3000e16e7 >>> trusted.gfid=0xec6de302b35f427985639ca3e25d9df0 >>> trusted.glusterfs.shard.block-size=0x0000000004000000 >>> >>> trusted.glusterfs.shard.file-size=0x0000000004000000000000000000000000000000000000010000000000000000 >>> >>> >>> # file: gluster1/shard1/1/.shard/ec6de302-b35f-4279-8563-9ca3e25d9df0.1 >>> >>> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>> trusted.afr.dirty=0x000000000000000000000000 >>> trusted.gfid=0x2bfd3cc8a727489b9a0474241548fe80 >>> >>> >>>> Regards, >>>> Vijay >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >>> >>> >> >> _______________________________________________ >> Gluster-devel mailing list >> Gluster-devel at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-devel >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160727/bd58e53d/attachment.html>
So far both dd commands that failed previously worked fine on 3.7.14 Once I deleted old content from test volume it mounted to oVirt via storage add when previously it would error out. I am now creating a test VM with default disk caching settings (pretty sure oVirt is defaulting to none rather than writeback/through). So far all shards are being created properly. Load is sky rocketing but I have all 3 gluster bricks running off 1 hard drive on test box so I would expect horrible io/load issues with that. Very promising so far. Thank you developers for your help in working through this. Once I have the VM installed and running will test for a few days and make sure it doesn't have any freeze or locking issues then will roll this out to working cluster. *David Gossage* *Carousel Checks Inc. | System Administrator* *Office* 708.613.2284 On Wed, Jul 27, 2016 at 8:37 AM, David Gossage <dgossage at carouselchecks.com> wrote:> On Tue, Jul 26, 2016 at 9:38 PM, Krutika Dhananjay <kdhananj at redhat.com> > wrote: > >> Yes please, could you file a bug against glusterfs for this issue? >> > > https://bugzilla.redhat.com/show_bug.cgi?id=1360785 > > >> >> >> -Krutika >> >> On Wed, Jul 27, 2016 at 1:39 AM, David Gossage < >> dgossage at carouselchecks.com> wrote: >> >>> Has a bug report been filed for this issue or should l I create one with >>> the logs and results provided so far? >>> >>> *David Gossage* >>> *Carousel Checks Inc. | System Administrator* >>> *Office* 708.613.2284 >>> >>> On Fri, Jul 22, 2016 at 12:53 PM, David Gossage < >>> dgossage at carouselchecks.com> wrote: >>> >>>> >>>> >>>> >>>> On Fri, Jul 22, 2016 at 9:37 AM, Vijay Bellur <vbellur at redhat.com> >>>> wrote: >>>> >>>>> On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen < >>>>> samppah at neutraali.net> wrote: >>>>> > Here is a quick way how to test this: >>>>> > GlusterFS 3.7.13 volume with default settings with brick on ZFS >>>>> dataset. gluster-test1 is server and gluster-test2 is client mounting with >>>>> FUSE. >>>>> > >>>>> > Writing file with oflag=direct is not ok: >>>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct >>>>> count=1 bs=1024000 >>>>> > dd: failed to open ?file?: Invalid argument >>>>> > >>>>> > Enable network.remote-dio on Gluster Volume: >>>>> > [root at gluster-test1 gluster]# gluster volume set gluster >>>>> network.remote-dio enable >>>>> > volume set: success >>>>> > >>>>> > Writing small file with oflag=direct is ok: >>>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct >>>>> count=1 bs=1024000 >>>>> > 1+0 records in >>>>> > 1+0 records out >>>>> > 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s >>>>> > >>>>> > Writing bigger file with oflag=direct is ok: >>>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct >>>>> count=100 bs=1M >>>>> > 100+0 records in >>>>> > 100+0 records out >>>>> > 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s >>>>> > >>>>> > Enable Sharding on Gluster Volume: >>>>> > [root at gluster-test1 gluster]# gluster volume set gluster >>>>> features.shard enable >>>>> > volume set: success >>>>> > >>>>> > Writing small file with oflag=direct is ok: >>>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct >>>>> count=1 bs=1M >>>>> > 1+0 records in >>>>> > 1+0 records out >>>>> > 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s >>>>> > >>>>> > Writing bigger file with oflag=direct is not ok: >>>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct >>>>> count=100 bs=1M >>>>> > dd: error writing ?file3?: Operation not permitted >>>>> > dd: closing output file ?file3?: Operation not permitted >>>>> > >>>>> >>>>> >>>>> Thank you for these tests! would it be possible to share the brick and >>>>> client logs? >>>>> >>>> >>>> Not sure if his tests are same as my setup but here is what I end up >>>> with >>>> >>>> Volume Name: glustershard >>>> Type: Replicate >>>> Volume ID: 0cc4efb6-3836-4caa-b24a-b3afb6e407c3 >>>> Status: Started >>>> Number of Bricks: 1 x 3 = 3 >>>> Transport-type: tcp >>>> Bricks: >>>> Brick1: 192.168.71.10:/gluster1/shard1/1 >>>> Brick2: 192.168.71.11:/gluster1/shard2/1 >>>> Brick3: 192.168.71.12:/gluster1/shard3/1 >>>> Options Reconfigured: >>>> features.shard-block-size: 64MB >>>> features.shard: on >>>> server.allow-insecure: on >>>> storage.owner-uid: 36 >>>> storage.owner-gid: 36 >>>> cluster.server-quorum-type: server >>>> cluster.quorum-type: auto >>>> network.remote-dio: enable >>>> cluster.eager-lock: enable >>>> performance.stat-prefetch: off >>>> performance.io-cache: off >>>> performance.quick-read: off >>>> cluster.self-heal-window-size: 1024 >>>> cluster.background-self-heal-count: 16 >>>> nfs.enable-ino32: off >>>> nfs.addr-namelookup: off >>>> nfs.disable: on >>>> performance.read-ahead: off >>>> performance.readdir-ahead: on >>>> >>>> >>>> >>>> dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/192.168.71.11\:_glustershard/ >>>> oflag=direct count=100 bs=1M >>>> 81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/ __DIRECT_IO_TEST__ >>>> .trashcan/ >>>> [root at ccengine2 ~]# dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/ >>>> 192.168.71.11\:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test >>>> oflag=direct count=100 bs=1M >>>> dd: error writing ?/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test?: >>>> Operation not permitted >>>> >>>> creates the 64M file in expected location then the shard is 0 >>>> >>>> # file: >>>> gluster1/shard1/1/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test >>>> >>>> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>> trusted.afr.dirty=0x000000000000000000000000 >>>> trusted.bit-rot.version=0x0200000000000000579231f3000e16e7 >>>> trusted.gfid=0xec6de302b35f427985639ca3e25d9df0 >>>> trusted.glusterfs.shard.block-size=0x0000000004000000 >>>> >>>> trusted.glusterfs.shard.file-size=0x0000000004000000000000000000000000000000000000010000000000000000 >>>> >>>> >>>> # file: gluster1/shard1/1/.shard/ec6de302-b35f-4279-8563-9ca3e25d9df0.1 >>>> >>>> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>> trusted.afr.dirty=0x000000000000000000000000 >>>> trusted.gfid=0x2bfd3cc8a727489b9a0474241548fe80 >>>> >>>> >>>>> Regards, >>>>> Vijay >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>> >>>> >>>> >>> >>> _______________________________________________ >>> Gluster-devel mailing list >>> Gluster-devel at gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-devel >>> >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160802/99ab583e/attachment.html>