Tiemen Ruiten
2015-Nov-13 08:56 UTC
[Gluster-users] rsync to gluster mount: self-heal and bad performance
Hello Ernie, list,
No, that's not the case. The volume is mounted through glusterfs-fuse - on
the same server running one of the bricks. The fstab:
# /etc/fstab
# Created by anaconda on Tue Aug 18 18:10:49 2015
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=56778fed-bf3f-435e-8c32-edaa8c707f29 / xfs defaults
0 0
UUID=a44e32ed-cfbe-4ba0-896f-1efff9397ba1 /boot xfs defaults
0 0
UUID=a344d2bc-266d-4905-85b1-fbb7fe927659 swap
swap defaults
0 0
/dev/vdb1 /data/brick xfs defaults 1 2
iron2:/lpxassets /mnt/lpxassets glusterfs _netdev,acl 0 0
On 12 November 2015 at 22:50, Ernie Dunbar <maillist at lightspeed.ca>
wrote:
> Hi Tiemen
>
> It sounds like you're trying to rsync files onto your Gluster server,
> rather than to the Gluster filesystem. You want to copy these files into
> the mounted filesystem (typically on some other system than the Gluster
> servers), because Gluster is designed to handle it that way.
>
> I can't remember the nitty gritty details about why this is, but
I've made
> this mistake before as well. Hope that helps. :)
>
>
> On 2015-11-12 11:31, Tiemen Ruiten wrote:
>
>> Hello,
>>
>> While rsyncing to a directory mounted through glusterfs fuse,
>> performance is very bad and it appears every synced file generates a
>> (metadata) self-heal.
>>
>> The volume is mounted with option acl and acl's are set on a
>> subdirectory.
>>
>> Setup is as follows:
>>
>> Two Centos 7 VM's (KVM), with Gluster 3.7.6 and one physical CentOS
6
>> node, also Gluster 3.7.6. Physical node functions as arbiter. So
it's
>> a replica 3 arbiter 1 volume. The bricks are LVM's with XFS
>> filesystem.
>>
>> While I don't think I should expect top performance for rsync on
>> Gluster, I wouldn't expect every file synced to trigger a
self-heal.
>> Anything I can do to improve this? Should I file a bug?
>>
>> Another thing that looks related, I see a lot of these messages,
>> especially when doing IO:
>>
>> [2015-11-12 19:25:42.185904] I [dict.c:473:dict_get]
>>
>>
(-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x121)
>> [0x7fdcc2d31161]
>>
>>
-->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x242)
>> [0x7fdcc2b1b212] -->/lib64/libglusterfs.so.0(dict_get+0xac)
>> [0x7fdcd5e770cc] ) 0-dict: !this || key=system.posix_acl_default
>> [Invalid argument]
>>
>> --
>>
>> Tiemen Ruiten
>> Systems Engineer
>> R&D Media
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
--
Tiemen Ruiten
Systems Engineer
R&D Media
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20151113/77e487d1/attachment.html>
Tiemen Ruiten
2015-Nov-15 10:22 UTC
[Gluster-users] rsync to gluster mount: self-heal and bad performance
Any other suggestions? On 13 November 2015 at 09:56, Tiemen Ruiten <t.ruiten at rdmedia.com> wrote:> Hello Ernie, list, > > No, that's not the case. The volume is mounted through glusterfs-fuse - on > the same server running one of the bricks. The fstab: > > # /etc/fstab > # Created by anaconda on Tue Aug 18 18:10:49 2015 > # > # Accessible filesystems, by reference, are maintained under '/dev/disk' > # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info > # > UUID=56778fed-bf3f-435e-8c32-edaa8c707f29 / xfs defaults > 0 0 > UUID=a44e32ed-cfbe-4ba0-896f-1efff9397ba1 /boot xfs defaults > 0 0 > UUID=a344d2bc-266d-4905-85b1-fbb7fe927659 swap swap defaults > 0 0 > /dev/vdb1 /data/brick xfs defaults 1 2 > iron2:/lpxassets /mnt/lpxassets glusterfs _netdev,acl 0 0 > > > > > On 12 November 2015 at 22:50, Ernie Dunbar <maillist at lightspeed.ca> wrote: > >> Hi Tiemen >> >> It sounds like you're trying to rsync files onto your Gluster server, >> rather than to the Gluster filesystem. You want to copy these files into >> the mounted filesystem (typically on some other system than the Gluster >> servers), because Gluster is designed to handle it that way. >> >> I can't remember the nitty gritty details about why this is, but I've >> made this mistake before as well. Hope that helps. :) >> >> >> On 2015-11-12 11:31, Tiemen Ruiten wrote: >> >>> Hello, >>> >>> While rsyncing to a directory mounted through glusterfs fuse, >>> performance is very bad and it appears every synced file generates a >>> (metadata) self-heal. >>> >>> The volume is mounted with option acl and acl's are set on a >>> subdirectory. >>> >>> Setup is as follows: >>> >>> Two Centos 7 VM's (KVM), with Gluster 3.7.6 and one physical CentOS 6 >>> node, also Gluster 3.7.6. Physical node functions as arbiter. So it's >>> a replica 3 arbiter 1 volume. The bricks are LVM's with XFS >>> filesystem. >>> >>> While I don't think I should expect top performance for rsync on >>> Gluster, I wouldn't expect every file synced to trigger a self-heal. >>> Anything I can do to improve this? Should I file a bug? >>> >>> Another thing that looks related, I see a lot of these messages, >>> especially when doing IO: >>> >>> [2015-11-12 19:25:42.185904] I [dict.c:473:dict_get] >>> >>> (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x121) >>> [0x7fdcc2d31161] >>> >>> -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x242) >>> [0x7fdcc2b1b212] -->/lib64/libglusterfs.so.0(dict_get+0xac) >>> [0x7fdcd5e770cc] ) 0-dict: !this || key=system.posix_acl_default >>> [Invalid argument] >>> >>> -- >>> >>> Tiemen Ruiten >>> Systems Engineer >>> R&D Media >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > > > -- > Tiemen Ruiten > Systems Engineer > R&D Media >-- Tiemen Ruiten Systems Engineer R&D Media -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151115/cb18e045/attachment.html>