Raghavendra Gowdappa
2018-Oct-08 04:05 UTC
[Gluster-users] [Gluster-devel] Glusterfs and Structured data
+Gluster-users <gluster-users at gluster.org> On Mon, Oct 8, 2018 at 9:34 AM Raghavendra Gowdappa <rgowdapp at redhat.com> wrote:> > > On Fri, Feb 9, 2018 at 4:30 PM Raghavendra Gowdappa <rgowdapp at redhat.com> > wrote: > >> >> >> ----- Original Message ----- >> > From: "Pranith Kumar Karampuri" <pkarampu at redhat.com> >> > To: "Raghavendra G" <raghavendra at gluster.com> >> > Cc: "Gluster Devel" <gluster-devel at gluster.org> >> > Sent: Friday, February 9, 2018 2:30:59 PM >> > Subject: Re: [Gluster-devel] Glusterfs and Structured data >> > >> > >> > >> > On Thu, Feb 8, 2018 at 12:05 PM, Raghavendra G < >> raghavendra at gluster.com > >> > wrote: >> > >> > >> > >> > >> > >> > On Tue, Feb 6, 2018 at 8:15 PM, Vijay Bellur < vbellur at redhat.com > >> wrote: >> > >> > >> > >> > >> > >> > On Sun, Feb 4, 2018 at 3:39 AM, Raghavendra Gowdappa < >> rgowdapp at redhat.com > >> > wrote: >> > >> > >> > All, >> > >> > One of our users pointed out to the documentation that glusterfs is not >> good >> > for storing "Structured data" [1], while discussing an issue [2]. >> > >> > >> > As far as I remember, the content around structured data in the Install >> Guide >> > is from a FAQ that was being circulated in Gluster, Inc. indicating the >> > startup's market positioning. Most of that was based on not wanting to >> get >> > into performance based comparisons of storage systems that are >> frequently >> > seen in the structured data space. >> > >> > >> > Does any of you have more context on the feasibility of storing >> "structured >> > data" on Glusterfs? Is one of the reasons for such a suggestion >> "staleness >> > of metadata" as encountered in bugs like [3]? >> > >> > >> > There are challenges that distributed storage systems face when exposed >> to >> > applications that were written for a local filesystem interface. We have >> > encountered problems with applications like tar [4] that are not in the >> > realm of "Structured data". If we look at the common theme across all >> these >> > problems, it is related to metadata & read after write consistency >> issues >> > with the default translator stack that gets exposed on the client side. >> > While the default stack is optimal for other scenarios, it does seem >> that a >> > category of applications needing strict metadata consistency is not well >> > served by that. We have observed that disabling a few performance >> > translators and tuning cache timeouts for VFS/FUSE have helped to >> overcome >> > some of them. The WIP effort on timestamp consistency across the >> translator >> > stack, patches that have been merged as a result of the bugs that you >> > mention & other fixes for outstanding issues should certainly help in >> > catering to these workloads better with the file interface. >> > >> > There are deployments that I have come across where glusterfs is used >> for >> > storing structured data. gluster-block & qemu-libgfapi overcome the >> metadata >> > consistency problem by exposing a file as a block device & by disabling >> most >> > of the performance translators in the default stack. Workloads that have >> > been deemed problematic with the file interface for the reasons alluded >> > above, function well with the block interface. >> > >> > I agree that gluster-block due to its usage of a subset of glusterfs >> fops >> > (mostly reads/writes I guess), runs into less number of consistency >> issues. >> > However, as you've mentioned we seem to disable perf xlator stack in our >> > tests/use-cases till now. Note that perf xlator stack is one of worst >> > offenders as far as the metadata consistency is concerned (relatively >> less >> > scenarios of data inconsistency). So, I wonder, >> > * what would be the scenario if we enable perf xlator stack for >> > gluster-block? >> > * Is performance on gluster-block satisfactory so that we don't need >> these >> > xlators? >> > - Or is it that these xlators are not useful for the workload usually >> run on >> > gluster-block (For random read/write workload, read/write caching >> xlators >> > offer less or no advantage)? >> > >> > Yes. They are not useful. Block/VM files are opened with O_DIRECT, so we >> > don't enable caching at any layer in glusterfs. md-cache could be >> useful for >> > serving fstat from glusterfs. But apart from that I don't see any other >> > xlator contributing much. >> > >> > >> > >> > - Or theoretically the workload is ought to benefit from perf xlators, >> but we >> > don't see them in our results (there are open bugs to this effect)? >> > >> > I am asking these questions to ascertain priority on fixing perf >> xlators for >> > (meta)data inconsistencies. If we offer a different solution for these >> > workloads, the need for fixing these issues will be less. >> > >> > My personal opinion is that both block and fs should work correctly. >> i.e. >> > caching xlators shouldn't lead to inconsistency issues. >> >> +1. That's my personal opinion too. We'll try to fix these issues. >> However, we need to qualify the fixes. It would be helpful if community can >> help here. We'll let community know when the fixes are in. >> > > There has been some progress on this. Details can be found at: > https://www.mail-archive.com/gluster-devel at gluster.org/msg14877.html > > >> > It would be better >> > if we are in a position where we choose a workload on block vs fs based >> on >> > their performance for that workload and nothing else. Block/VM usecases >> > change the workload of the application for glusterfs, so for small file >> > operations the kind of performance you see on block can never be >> achieved by >> > glusterfs with the current architecture/design. >> > >> > >> > >> > >> > >> > >> > >> > I feel that we have come a long way from the time the install guide was >> > written and an update for removing the "staleness of content" might be >> in >> > order there :-). >> > >> > Regards, >> > Vijay >> > >> > [4] https://bugzilla.redhat.com/show_bug.cgi?id=1058526 >> > >> > >> > >> > [1] http://docs.gluster.org/en/latest/Install-Guide/Overview/ >> > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1512691 >> > [3] https://bugzilla.redhat.com/show_bug.cgi?id=1390050 >> > >> > regards, >> > Raghavendra >> > _______________________________________________ >> > Gluster-devel mailing list >> > Gluster-devel at gluster.org >> > http://lists.gluster.org/mailman/listinfo/gluster-devel >> > >> > >> > _______________________________________________ >> > Gluster-devel mailing list >> > Gluster-devel at gluster.org >> > http://lists.gluster.org/mailman/listinfo/gluster-devel >> > >> > >> > >> > -- >> > Raghavendra G >> > >> > _______________________________________________ >> > Gluster-devel mailing list >> > Gluster-devel at gluster.org >> > http://lists.gluster.org/mailman/listinfo/gluster-devel >> > >> > >> > >> > -- >> > Pranith >> > >> > _______________________________________________ >> > Gluster-devel mailing list >> > Gluster-devel at gluster.org >> > http://lists.gluster.org/mailman/listinfo/gluster-devel >> _______________________________________________ >> Gluster-devel mailing list >> Gluster-devel at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-devel >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181008/c448db60/attachment.html>