Shaik Salam
2019-Jan-24 07:06 UTC
[Gluster-users] Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
H Madhu, Could you please have look my issue If you have time (atleast workaround). I am unable to send mail to "John Mulligan" <John_Mulligan at redhat.com>" who is currently handling issue https://bugzilla.redhat.com/show_bug.cgi?id=1636912 BR Salam From: Shaik Salam/HYD/TCS To: "John Mulligan" <John_Mulligan at redhat.com>, "Michael Adam" <madam at redhat.com>, "Madhu Rajanna" <mrajanna at redhat.com> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org> Date: 01/24/2019 12:21 PM Subject: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy Hi All, We are facing also following issue on openshift origin while we are creating pvc for pods. (atlease provide workaround to move further) Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Server busy. Retry operation later.. Please find heketidb dump and log [negroni] Completed 429 Too Many Requests in 250.763?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 169.08?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 148.125?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 496.624?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 101.673?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 209.681?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 103.595?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 297.594?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 96.75?s [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 477.007?s [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 165.38?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 488.253?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 171.836?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 208.59?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 125.141?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 138.687?s [negroni] Started POST /volumes BR Salam =====-----=====-----====Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190124/6eac1d1e/attachment.html> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: heketi-gluster.db.txt URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190124/6eac1d1e/attachment.txt> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: heketi.log.txt URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190124/6eac1d1e/attachment-0001.txt>
Madhu Rajanna
2019-Jan-24 07:21 UTC
[Gluster-users] Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
HI Shaik, can you provide me the outpout of $heketi-cli server operations info from heketi pod as a workround you can try restarting the heketi pod. This will cause the current operations to go stale, but other pending pvcs may go to Bound state Regards, Madhu R On Thu, Jan 24, 2019 at 12:36 PM Shaik Salam <shaik.salam at tcs.com> wrote:> H Madhu, > > Could you please have look my issue If you have time (atleast workaround). > I am unable to send mail to "John Mulligan" <John_Mulligan at redhat.com>" > who is currently handling issue > https://bugzilla.redhat.com/show_bug.cgi?id=1636912 > > BR > Salam > > > From: Shaik Salam/HYD/TCS > To: "John Mulligan" <John_Mulligan at redhat.com>, "Michael Adam" < > madam at redhat.com>, "Madhu Rajanna" <mrajanna at redhat.com> > Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org> > Date: 01/24/2019 12:21 PM > Subject: Failed to provision volume with StorageClass > "glusterfs-storage": glusterfs: server busy > ------------------------------ > > > > > Hi All, > > We are facing also following issue on openshift origin while we are > creating pvc for pods. (atlease provide workaround to move further) > > Failed to provision volume with StorageClass "glusterfs-storage": > glusterfs: create volume err: error creating volume > Failed to provision volume with StorageClass "glusterfs-storage": > glusterfs: create volume err: error creating volume Server busy. Retry > operation later.. > > Please find heketidb dump and log > > [negroni] Completed 429 Too Many Requests in 250.763?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 169.08?s > [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c > [negroni] Completed 404 Not Found in 148.125?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 496.624?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 101.673?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 209.681?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 103.595?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 297.594?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 96.75?s > [negroni] Started POST /volumes > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 477.007?s > [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 165.38?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 488.253?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 171.836?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 208.59?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 125.141?s > [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 > [negroni] Completed 404 Not Found in 138.687?s > [negroni] Started POST /volumes > > > BR > Salam > > =====-----=====-----====> Notice: The information contained in this e-mail > message and/or attachments to it may contain > confidential or privileged information. If you are > not the intended recipient, any dissemination, use, > review, distribution, printing or copying of the > information contained in this e-mail message > and/or attachments to it are strictly prohibited. If > you have received this communication in error, > please notify us by reply e-mail or telephone and > immediately and permanently delete the message > and any attachments. Thank you > >-- Madhu Rajanna Software Engineer Red Hat Bangalore, India <https://www.redhat.com> mrajanna at redhat.com M: +91-9741133155 <https://red.ht/sig> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190124/16908e87/attachment.html>