Raghavendra Gowdappa
2019-Jun-03 06:46 UTC
[Gluster-users] write request hung in write-behind
On Mon, Jun 3, 2019 at 11:57 AM Xie Changlong <zgrep at 139.com> wrote:> Hi all > > Test gluster 3.8.4-54.15 gnfs, i saw a write request hung in write-behind > followed by 1545 FLUSH requests. I found a similar > bugfix https://bugzilla.redhat.com/show_bug.cgi?id=1626787, but not sure > if it's the right one. > > [xlator.performance.write-behind.wb_inode] > path=/575/1e/5751e318f21f605f2aac241bf042e7a8.jpg > inode=0x7f51775b71a0 > window_conf=1073741824 > window_current=293822 > transit-size=293822 > dontsync=0 > > [.WRITE] > request-ptr=0x7f516eec2060 > refcount=1 > wound=yes > generation-number=1 > req->op_ret=293822 > req->op_errno=0 > sync-attempts=1 > sync-in-progress=yes >Note that the sync is still in progress. This means, write-behind has wound the write-request to its children and yet to receive the response (unless there is a bug in accounting of sync-in-progress). So, its likely that there are callstacks into children of write-behind, which are not complete yet. Are you sure the deepest hung call-stack is in write-behind? Can you check for frames with "complete=0"? size=293822> offset=1048576 > lied=-1 > append=0 > fulfilled=0 > go=-1 > > [.FLUSH] > request-ptr=0x7f517c2badf0 > refcount=1 > wound=no > generation-number=2 > req->op_ret=-1 > req->op_errno=116 > sync-attempts=0 > > [.FLUSH] > request-ptr=0x7f5173e9f7b0 > refcount=1 > wound=no > generation-number=2 > req->op_ret=0 > req->op_errno=0 > sync-attempts=0 > > [.FLUSH] > request-ptr=0x7f51640b8ca0 > refcount=1 > wound=no > generation-number=2 > req->op_ret=0 > req->op_errno=0 > sync-attempts=0 > > [.FLUSH] > request-ptr=0x7f516f3979d0 > refcount=1 > wound=no > generation-number=2 > req->op_ret=0 > req->op_errno=0 > sync-attempts=0 > > [.FLUSH] > request-ptr=0x7f516f6ac8d0 > refcount=1 > wound=no > generation-number=2 > req->op_ret=0 > req->op_errno=0 > sync-attempts=0 > > > Any comments would be appreciated! > > Thanks > -Xie > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190603/e162ffdc/attachment.html>