Hi, When we designed and wrote sharding feature in GlusterFS, our focus had been single-writer-to-large-files use cases, chief among these being the virtual machine image store use-case. Sharding, for the uninitiated, is a feature that was introduced in glusterfs-3.7.0 release with 'experimental' status. Here is some documentation that explains what it does at a high level: http://www.gluster.org/community/documentation/index.php/Features/sharding-xlator https://gluster.readthedocs.org/en/release-3.7.0/Features/shard/ We have now reached that stage where the feature is considered stable for the VM store use case after several rounds of testing (thanks to Lindsay Mathieson, Paul Cuzner and Satheesaran Sundaramoorthi), bug fixing and reviews (thanks to Pranith Karampuri). Also in this regard, patches have been sent to make sharding work with geo-replication, thanks to Kotresh's efforts (testing still in progress). We would love to hear from you on what you think of the feature and where it could be improved. Specifically, the following are the questions we are seeking feedback on: a) your experience testing sharding with VM store use-case - any bugs you ran into, any performance issues, etc b) what are the other large-file use-cases (apart from the VM store workload) you know of or use, where you think having sharding capability will be useful. Based on your feedback we will start work on making sharding work in other workloads and/or with other existing GlusterFS features. Thanks, Krutika -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151203/9248b75b/attachment.html>
Hi Guys, sorry for the late reply, my attention tends to be somewhat sporadic due to work and the large number of rescue dogs/cats I care for :) On 3/12/2015 8:34 PM, Krutika Dhananjay wrote:> We would love to hear from you on what you think of the feature and > where it could be improved. > Specifically, the following are the questions we are seeking feedback on: > a) your experience testing sharding with VM store use-case - any bugs > you ran into, any performance issues, etcTesting was initially somewhat stressful as I regularly encountered file corruption. However I don't think that was due to bugs, rather incorrect settings for the VM usecase. Once I got that sorted out it has been very stable - I have really stressed failure modes we run into at work - nodes going down while heavy writes were happening. Live migrations during heals. gluster software being killed while VM were running on the host. So far its held up without a hitch. To that end, one thing I think should be made more obvious is the settings required for VM Hosting: quick-read=off read-ahead=off io-cache=off stat-prefetch=off eager-lock=enable remote-dio=enable quorum-type=auto server-quorum-type=server They are quite crucial and very easy to miss in the online docs. And they are only recommended with noo mention that you will corrupt KVM VM's if you live migrate them between gluster nodes without them set. Also the virt group is missing from the debian packages. Setting them does seem to have slowed sequential writes by about 10% but I need to test that more. Something related - sharding is useful because it makes heals much more granular and hence faster. To that end it would be really useful if there was a heal info variant that gave a overview of the process - rather than list the shards that are being healed, just a aggregate total, e.g. $ gluster volume heal datastore1 status volume datastore1 - split brain: 0 - Wounded:65 - healing:4 It gives one a easy feeling of progress - heals aren't happening faster, but it would feel that way :) Also, it would be great if the heal info command could return faster, sometimes it takes over a minute. Thanks for the great work, Lindsay -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151209/6195332e/attachment.html>