Krutika Dhananjay
2016-Aug-03 04:26 UTC
[Gluster-users] [Gluster-devel] 3.7.13 & proxmox/qemu
Glad the fixes worked for you. Thanks for that update! -Krutika On Tue, Aug 2, 2016 at 7:31 PM, David Gossage <dgossage at carouselchecks.com> wrote:> So far both dd commands that failed previously worked fine on 3.7.14 > > Once I deleted old content from test volume it mounted to oVirt via > storage add when previously it would error out. I am now creating a test > VM with default disk caching settings (pretty sure oVirt is defaulting to > none rather than writeback/through). So far all shards are being created > properly. > > Load is sky rocketing but I have all 3 gluster bricks running off 1 hard > drive on test box so I would expect horrible io/load issues with that. > > Very promising so far. Thank you developers for your help in working > through this. > > Once I have the VM installed and running will test for a few days and make > sure it doesn't have any freeze or locking issues then will roll this out > to working cluster. > > > > *David Gossage* > *Carousel Checks Inc. | System Administrator* > *Office* 708.613.2284 > > On Wed, Jul 27, 2016 at 8:37 AM, David Gossage < > dgossage at carouselchecks.com> wrote: > >> On Tue, Jul 26, 2016 at 9:38 PM, Krutika Dhananjay <kdhananj at redhat.com> >> wrote: >> >>> Yes please, could you file a bug against glusterfs for this issue? >>> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1360785 >> >> >>> >>> >>> -Krutika >>> >>> On Wed, Jul 27, 2016 at 1:39 AM, David Gossage < >>> dgossage at carouselchecks.com> wrote: >>> >>>> Has a bug report been filed for this issue or should l I create one >>>> with the logs and results provided so far? >>>> >>>> *David Gossage* >>>> *Carousel Checks Inc. | System Administrator* >>>> *Office* 708.613.2284 >>>> >>>> On Fri, Jul 22, 2016 at 12:53 PM, David Gossage < >>>> dgossage at carouselchecks.com> wrote: >>>> >>>>> >>>>> >>>>> >>>>> On Fri, Jul 22, 2016 at 9:37 AM, Vijay Bellur <vbellur at redhat.com> >>>>> wrote: >>>>> >>>>>> On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen < >>>>>> samppah at neutraali.net> wrote: >>>>>> > Here is a quick way how to test this: >>>>>> > GlusterFS 3.7.13 volume with default settings with brick on ZFS >>>>>> dataset. gluster-test1 is server and gluster-test2 is client mounting with >>>>>> FUSE. >>>>>> > >>>>>> > Writing file with oflag=direct is not ok: >>>>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct >>>>>> count=1 bs=1024000 >>>>>> > dd: failed to open ?file?: Invalid argument >>>>>> > >>>>>> > Enable network.remote-dio on Gluster Volume: >>>>>> > [root at gluster-test1 gluster]# gluster volume set gluster >>>>>> network.remote-dio enable >>>>>> > volume set: success >>>>>> > >>>>>> > Writing small file with oflag=direct is ok: >>>>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct >>>>>> count=1 bs=1024000 >>>>>> > 1+0 records in >>>>>> > 1+0 records out >>>>>> > 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s >>>>>> > >>>>>> > Writing bigger file with oflag=direct is ok: >>>>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 >>>>>> oflag=direct count=100 bs=1M >>>>>> > 100+0 records in >>>>>> > 100+0 records out >>>>>> > 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s >>>>>> > >>>>>> > Enable Sharding on Gluster Volume: >>>>>> > [root at gluster-test1 gluster]# gluster volume set gluster >>>>>> features.shard enable >>>>>> > volume set: success >>>>>> > >>>>>> > Writing small file with oflag=direct is ok: >>>>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 >>>>>> oflag=direct count=1 bs=1M >>>>>> > 1+0 records in >>>>>> > 1+0 records out >>>>>> > 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s >>>>>> > >>>>>> > Writing bigger file with oflag=direct is not ok: >>>>>> > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 >>>>>> oflag=direct count=100 bs=1M >>>>>> > dd: error writing ?file3?: Operation not permitted >>>>>> > dd: closing output file ?file3?: Operation not permitted >>>>>> > >>>>>> >>>>>> >>>>>> Thank you for these tests! would it be possible to share the brick and >>>>>> client logs? >>>>>> >>>>> >>>>> Not sure if his tests are same as my setup but here is what I end up >>>>> with >>>>> >>>>> Volume Name: glustershard >>>>> Type: Replicate >>>>> Volume ID: 0cc4efb6-3836-4caa-b24a-b3afb6e407c3 >>>>> Status: Started >>>>> Number of Bricks: 1 x 3 = 3 >>>>> Transport-type: tcp >>>>> Bricks: >>>>> Brick1: 192.168.71.10:/gluster1/shard1/1 >>>>> Brick2: 192.168.71.11:/gluster1/shard2/1 >>>>> Brick3: 192.168.71.12:/gluster1/shard3/1 >>>>> Options Reconfigured: >>>>> features.shard-block-size: 64MB >>>>> features.shard: on >>>>> server.allow-insecure: on >>>>> storage.owner-uid: 36 >>>>> storage.owner-gid: 36 >>>>> cluster.server-quorum-type: server >>>>> cluster.quorum-type: auto >>>>> network.remote-dio: enable >>>>> cluster.eager-lock: enable >>>>> performance.stat-prefetch: off >>>>> performance.io-cache: off >>>>> performance.quick-read: off >>>>> cluster.self-heal-window-size: 1024 >>>>> cluster.background-self-heal-count: 16 >>>>> nfs.enable-ino32: off >>>>> nfs.addr-namelookup: off >>>>> nfs.disable: on >>>>> performance.read-ahead: off >>>>> performance.readdir-ahead: on >>>>> >>>>> >>>>> >>>>> dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/192.168.71.11\:_glustershard/ >>>>> oflag=direct count=100 bs=1M >>>>> 81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/ __DIRECT_IO_TEST__ >>>>> .trashcan/ >>>>> [root at ccengine2 ~]# dd if=/dev/zero >>>>> of=/rhev/data-center/mnt/glusterSD/192.168.71.11\:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test >>>>> oflag=direct count=100 bs=1M >>>>> dd: error writing ?/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test?: >>>>> Operation not permitted >>>>> >>>>> creates the 64M file in expected location then the shard is 0 >>>>> >>>>> # file: >>>>> gluster1/shard1/1/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test >>>>> >>>>> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>>> trusted.afr.dirty=0x000000000000000000000000 >>>>> trusted.bit-rot.version=0x0200000000000000579231f3000e16e7 >>>>> trusted.gfid=0xec6de302b35f427985639ca3e25d9df0 >>>>> trusted.glusterfs.shard.block-size=0x0000000004000000 >>>>> >>>>> trusted.glusterfs.shard.file-size=0x0000000004000000000000000000000000000000000000010000000000000000 >>>>> >>>>> >>>>> # file: gluster1/shard1/1/.shard/ec6de302-b35f-4279-8563-9ca3e25d9df0.1 >>>>> >>>>> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>>> trusted.afr.dirty=0x000000000000000000000000 >>>>> trusted.gfid=0x2bfd3cc8a727489b9a0474241548fe80 >>>>> >>>>> >>>>>> Regards, >>>>>> Vijay >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>> >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Gluster-devel mailing list >>>> Gluster-devel at gluster.org >>>> http://www.gluster.org/mailman/listinfo/gluster-devel >>>> >>> >>> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160803/67ea8c95/attachment.html>
Lindsay Mathieson
2016-Aug-03 12:45 UTC
[Gluster-users] [Gluster-devel] 3.7.13 & proxmox/qemu
On 3/08/2016 2:26 PM, Krutika Dhananjay wrote:> Once I deleted old content from test volume it mounted to oVirt via > storage add when previously it would error out. I am now creating a > test VM with default disk caching settings (pretty sure oVirt is > defaulting to none rather than writeback/through). So far all shards > are being created properly.I can confirm that it works with ProxMox VM's in direct (no cache mode) as well.> > Load is sky rocketing but I have all 3 gluster bricks running off 1 > hard drive on test box so I would expect horrible io/load issues with > that.Ha! Same config for my test Host :) -- Lindsay Mathieson