Samuli Heinonen
2016-Jul-22 14:03 UTC
[Gluster-users] [Gluster-devel] 3.7.13 & proxmox/qemu
Here is a quick way how to test this: GlusterFS 3.7.13 volume with default settings with brick on ZFS dataset. gluster-test1 is server and gluster-test2 is client mounting with FUSE. Writing file with oflag=direct is not ok: [root at gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 bs=1024000 dd: failed to open ?file?: Invalid argument Enable network.remote-dio on Gluster Volume: [root at gluster-test1 gluster]# gluster volume set gluster network.remote-dio enable volume set: success Writing small file with oflag=direct is ok: [root at gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 bs=1024000 1+0 records in 1+0 records out 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s Writing bigger file with oflag=direct is ok: [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 bs=1M 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s Enable Sharding on Gluster Volume: [root at gluster-test1 gluster]# gluster volume set gluster features.shard enable volume set: success Writing small file with oflag=direct is ok: [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=1 bs=1M 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s Writing bigger file with oflag=direct is not ok: [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 bs=1M dd: error writing ?file3?: Operation not permitted dd: closing output file ?file3?: Operation not permitted -samuli> On 22 Jul 2016, at 16:12, Vijay Bellur <vbellur at redhat.com> wrote: > > 2016-07-22 1:54 GMT-04:00 Frank Rothenstein <f.rothenstein at bodden-kliniken.de>: >> The point is that even if all other backend storage filesystems do correctly >> untill 3.7.11 there was no error on ZFS. Something happened nobody ever >> could explain in the release of 3.7.12 that makes FUSE-mount _in ovirt_ (it >> partly uses dd with iflag=direct , using iflag=direct yourself gives also >> errors on the FUSE-mounts ) unusable. >> >> So 3.7.11 is the last usable version when using ZFS on bricks, afaik. >> > > Can you please share the exact dd command that causes this problem? > > Thanks, > Vijay > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users
On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen <samppah at neutraali.net> wrote:> Here is a quick way how to test this: > GlusterFS 3.7.13 volume with default settings with brick on ZFS dataset. gluster-test1 is server and gluster-test2 is client mounting with FUSE. > > Writing file with oflag=direct is not ok: > [root at gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 bs=1024000 > dd: failed to open ?file?: Invalid argument > > Enable network.remote-dio on Gluster Volume: > [root at gluster-test1 gluster]# gluster volume set gluster network.remote-dio enable > volume set: success > > Writing small file with oflag=direct is ok: > [root at gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 bs=1024000 > 1+0 records in > 1+0 records out > 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s > > Writing bigger file with oflag=direct is ok: > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 bs=1M > 100+0 records in > 100+0 records out > 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s > > Enable Sharding on Gluster Volume: > [root at gluster-test1 gluster]# gluster volume set gluster features.shard enable > volume set: success > > Writing small file with oflag=direct is ok: > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=1 bs=1M > 1+0 records in > 1+0 records out > 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s > > Writing bigger file with oflag=direct is not ok: > [root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 bs=1M > dd: error writing ?file3?: Operation not permitted > dd: closing output file ?file3?: Operation not permitted >Thank you for these tests! would it be possible to share the brick and client logs? Regards, Vijay