Shaik Salam
2019-Jan-21 12:06 UTC
[Gluster-users] Unable to create new volume due to pending operations
Hi, We have deployed glustrerfs as containers on openshift orgin. We are unable to create new volume for opeshift pods observed following error. Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Server busy. Retry operation later.. We could observe above issue due to pending operations on heketi db. We have exported db and removed pending operations (volumes,bricks) on db, lvs from physical hosts and imported. But heketi trying to delete volumes which are removed from heketi as part of pending operations in db and still we are unable to create volumes facing server busy. Can you please let me know still heketi where it is getting volumes id which are not available on heketi deb?? why we are unable to create volumes ?? (no info on glusterd,glusterfsd logs)?? [negroni] Started POST /volumes [heketi] WARNING 2019/01/21 11:48:31 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 260.577?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/21 11:48:39 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 221.477?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 151.896?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 125.387?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 168.23?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 123.231?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 160.416?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 124.439?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 126.748?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 138.377?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 138.477?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 267.79?s BR Salam =====-----=====-----====Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190121/726cfc5c/attachment.html>
Shaik Salam
2019-Jan-21 13:47 UTC
[Gluster-users] [Bugs] Unable to create new volume due to pending operations
Hi, Can you please reply on my issues. I think its already known issue I feel. Can you please let me know still heketi where it is getting volumes id which are not available on heketi db?? why we are unable to create volumes ?? (no info on glusterd,glusterfsd logs)?? BR Salam From: "Shaik Salam" <shaik.salam at tcs.com> To: Date: 01/21/2019 05:47 PM Subject: [Bugs] Unable to create new volume due to pending operations Sent by: bugs-bounces at gluster.org "External email. Open with Caution" Hi, We have deployed glustrerfs as containers on openshift orgin. We are unable to create new volume for opeshift pods observed following error. Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Server busy. Retry operation later.. We could observe above issue due to pending operations on heketi db. We have exported db and removed pending operations (volumes,bricks) on db, lvs from physical hosts and imported. But heketi trying to delete volumes which are removed from heketi as part of pending operations in db and still we are unable to create volumes facing server busy. Can you please let me know still heketi where it is getting volumes id which are not available on heketi deb?? why we are unable to create volumes ?? (no info on glusterd,glusterfsd logs)?? [negroni] Started POST /volumes [heketi] WARNING 2019/01/21 11:48:31 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 260.577?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/21 11:48:39 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 221.477?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 151.896?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 125.387?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 168.23?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 123.231?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 160.416?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 124.439?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 126.748?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 138.377?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 138.477?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 267.79?s BR Salam =====-----=====-----====Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you _______________________________________________ Bugs mailing list Bugs at gluster.org https://lists.gluster.org/mailman/listinfo/bugs -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190121/38b44434/attachment.html>