Amar Tumballi Suryanarayan
2019-Jan-24 06:28 UTC
[Gluster-users] Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume
Looks like the requests to create volume has increased significantly. Heketi can handle 8 parallel volume create requests, and looks like there are 429 volume create requests pending. I am not an expert in this. Added few more people in CC to help when they get to see this. -Amar On Thu, Jan 24, 2019 at 11:28 AM Shaik Salam <shaik.salam at tcs.com> wrote:> Hi Surya, > > Could you please help us to resolve below issue (at lease workaround for > creating volume) > Attached db dump and log. Please let me know any other things need to > check. > Please guide us. > > BR > Salam > > > > From: Shaik Salam/HYD/TCS > To: "Amar Tumballi Suryanarayan" <atumball at redhat.com>, > bugs at gluster.org, "gluster-users at gluster.org List" < > gluster-users at gluster.org> > Cc: "Murali Kottakota" <murali.kottakota at tcs.com>, "Sanju Rakonde" > <srakonde at redhat.com> > Date: 01/23/2019 06:19 PM > Subject: Failed to provision volume with StorageClass > "glusterfs-storage": glusterfs: create volume err: error creating volume > ------------------------------ > > > > > > Hi, > > We are facing also following issue on openshift origin while we are > creating pvc for pods. > > Failed to provision volume with StorageClass "glusterfs-storage": > glusterfs: create volume err: error creating volume > Failed to provision volume with StorageClass "glusterfs-storage": > glusterfs: create volume err: error creating volume Server busy. Retry > operation later.. > > Please find heketidb dump and log > > [negroni] Completed 429 Too Many Requests in 250.763?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 169.08?s > [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c > [negroni] Completed 404 Not Found in 148.125?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 496.624?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 101.673?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 209.681?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 103.595?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 297.594?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 96.75?s > [negroni] Started POST /volumes > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 477.007?s > [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 165.38?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 488.253?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 171.836?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 208.59?s > [negroni] Started POST /volumes > [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds > limit (8) > [negroni] Completed 429 Too Many Requests in 125.141?s > [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 > [negroni] Completed 404 Not Found in 138.687?s > [negroni] Started POST /volumes > > > BR > Salam > > =====-----=====-----====> Notice: The information contained in this e-mail > message and/or attachments to it may contain > confidential or privileged information. If you are > not the intended recipient, any dissemination, use, > review, distribution, printing or copying of the > information contained in this e-mail message > and/or attachments to it are strictly prohibited. If > you have received this communication in error, > please notify us by reply e-mail or telephone and > immediately and permanently delete the message > and any attachments. Thank you > >-- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190124/6d9f646b/attachment.html>
Shaik Salam
2019-Jan-24 06:38 UTC
[Gluster-users] Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume
Hi All, Could you please help us to resolve issue (atleast workaround). 429 volumes are not requested at all in cluster. I am trying to create only one volume at a time. BR Salam From: "Amar Tumballi Suryanarayan" <atumball at redhat.com> To: "Shaik Salam" <shaik.salam at tcs.com> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org>, "Madhu Rajanna" <mrajanna at redhat.com>, "Raghavendra Talur" <rtalur at redhat.com>, "Michael Adam" <madam at redhat.com> Date: 01/24/2019 11:59 AM Subject: Re: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume "External email. Open with Caution" Looks like the requests to create volume has increased significantly. Heketi can handle 8 parallel volume create requests, and looks like there are 429 volume create requests pending. I am not an expert in this. Added few more people in CC to help when they get to see this. -Amar On Thu, Jan 24, 2019 at 11:28 AM Shaik Salam <shaik.salam at tcs.com> wrote: Hi Surya, Could you please help us to resolve below issue (at lease workaround for creating volume) Attached db dump and log. Please let me know any other things need to check. Please guide us. BR Salam From: Shaik Salam/HYD/TCS To: "Amar Tumballi Suryanarayan" <atumball at redhat.com>, bugs at gluster.org, "gluster-users at gluster.org List" < gluster-users at gluster.org> Cc: "Murali Kottakota" <murali.kottakota at tcs.com>, "Sanju Rakonde" <srakonde at redhat.com> Date: 01/23/2019 06:19 PM Subject: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Hi, We are facing also following issue on openshift origin while we are creating pvc for pods. Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Server busy. Retry operation later.. Please find heketidb dump and log [negroni] Completed 429 Too Many Requests in 250.763?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 169.08?s [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c [negroni] Completed 404 Not Found in 148.125?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 496.624?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 101.673?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 209.681?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 103.595?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 297.594?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 96.75?s [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 477.007?s [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 165.38?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 488.253?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 171.836?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 208.59?s [negroni] Started POST /volumes [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 125.141?s [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 [negroni] Completed 404 Not Found in 138.687?s [negroni] Started POST /volumes BR Salam =====-----=====-----====Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you -- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190124/8266d2e6/attachment.html>