Looks like a problem with caching. Can you please try by disabling all performance translators? The following configuration commands would disable performance translators in the gluster client stack: gluster volume set <volname> performance.quick-read off gluster volume set <volname> performance.io-cache off gluster volume set <volname> performance.write-behind off gluster volume set <volname> performance.stat-prefetch off gluster volume set <volname> performance.read-ahead off gluster volume set <volname> performance.readdir-ahead off gluster volume set <volname> performance.open-behind off gluster volume set <volname> performance.client-io-threads off Thanks, Vijay On Mon, May 22, 2017 at 9:46 AM, Christopher Schmidt <fakod666 at gmail.com> wrote:> Hi all, > > has anyone ever successfully deployed a Kafka (Cluster) on GlusterFS > volumes? > > I my case it's a Kafka Kubernetes-StatefulSet and a Heketi GlusterFS. > Needless to say that I am getting a lot of filesystem related exceptions > like this one: > > Failed to read `log header` from file channel `sun.nio.ch.FileChannelImpl at 67afa54a`. > Expected to read 12 bytes, but reached end of file after reading 0 bytes. > Started read from position 123065680. > > I limited the amount of exceptions with the log.flush.interval.messages=1 > option, but not all... > > best Christopher > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170522/96dae439/attachment.html>
This may be asking too much, but can you explain why or how it's even possible to bypass the cache like this, Vijay? On May 22, 2017 7:41:40 AM PDT, Vijay Bellur <vbellur at redhat.com> wrote:>Looks like a problem with caching. Can you please try by disabling all >performance translators? The following configuration commands would >disable >performance translators in the gluster client stack: > >gluster volume set <volname> performance.quick-read off >gluster volume set <volname> performance.io-cache off >gluster volume set <volname> performance.write-behind off >gluster volume set <volname> performance.stat-prefetch off >gluster volume set <volname> performance.read-ahead off >gluster volume set <volname> performance.readdir-ahead off >gluster volume set <volname> performance.open-behind off >gluster volume set <volname> performance.client-io-threads off > >Thanks, >Vijay > > > >On Mon, May 22, 2017 at 9:46 AM, Christopher Schmidt ><fakod666 at gmail.com> >wrote: > >> Hi all, >> >> has anyone ever successfully deployed a Kafka (Cluster) on GlusterFS >> volumes? >> >> I my case it's a Kafka Kubernetes-StatefulSet and a Heketi GlusterFS. >> Needless to say that I am getting a lot of filesystem related >exceptions >> like this one: >> >> Failed to read `log header` from file channel >`sun.nio.ch.FileChannelImpl at 67afa54a`. >> Expected to read 12 bytes, but reached end of file after reading 0 >bytes. >> Started read from position 123065680. >> >> I limited the amount of exceptions with the >log.flush.interval.messages=1 >> option, but not all... >> >> best Christopher >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >>-- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170522/711c5dd9/attachment.html>
OK, seems that this works now. A couple of questions: - What do you think, are all these options necessary for Kafka? - You wrote that there have to be kind of application profiles. So to find out, which set of options work is currently a matter of testing (and hope)? Or are there any experiences for MongoDB / ProstgreSQL / Zookeeper etc.? - I am using Heketi and Dynamik Storage Provisioning together with Kubernetes. Can I set this volume options somehow by default or by volume plugin? Thanks for you help... really appreciated.. Christopher Vijay Bellur <vbellur at redhat.com> schrieb am Mo., 22. Mai 2017 um 16:41 Uhr:> Looks like a problem with caching. Can you please try by disabling all > performance translators? The following configuration commands would disable > performance translators in the gluster client stack: > > gluster volume set <volname> performance.quick-read off > gluster volume set <volname> performance.io-cache off > gluster volume set <volname> performance.write-behind off > gluster volume set <volname> performance.stat-prefetch off > gluster volume set <volname> performance.read-ahead off > gluster volume set <volname> performance.readdir-ahead off > gluster volume set <volname> performance.open-behind off > gluster volume set <volname> performance.client-io-threads off > > Thanks, > Vijay > > > > On Mon, May 22, 2017 at 9:46 AM, Christopher Schmidt <fakod666 at gmail.com> > wrote: > >> Hi all, >> >> has anyone ever successfully deployed a Kafka (Cluster) on GlusterFS >> volumes? >> >> I my case it's a Kafka Kubernetes-StatefulSet and a Heketi GlusterFS. >> Needless to say that I am getting a lot of filesystem related exceptions >> like this one: >> >> Failed to read `log header` from file channel >> `sun.nio.ch.FileChannelImpl at 67afa54a`. Expected to read 12 bytes, but >> reached end of file after reading 0 bytes. Started read from position >> 123065680. >> >> I limited the amount of exceptions with the log.flush.interval.messages=1 >> option, but not all... >> >> best Christopher >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170523/939a8da8/attachment.html>