Punit Dambiwal
2015-Mar-25 09:39 UTC
[Gluster-users] VM failed to start | Bad volume specification
Hi All, With the help of gluster community and ovirt-china community...my issue got resolved... The main root cause was the following :- 1. the glob operation takes quite a long time, longer than the ioprocess default 60s.. 2. python-ioprocess updated which makes a single change of configuration file doesn't work properly, only because this we should hack the code manually... Solution (Need to do on all the hosts) :- 1. Add the the ioprocess timeout value in the /etc/vdsm/vdsm.conf file as :- ------------ [irs] process_pool_timeout = 180 ------------- 2. Check /usr/share/vdsm/storage/outOfProcess.py, line 71 and see whether there is still "IOProcess(DEFAULT_TIMEOUT)" in it,if yes...then changing the configuration file takes no effect because now timeout is the third parameter not the second of IOProcess.__init__(). 3. Change IOProcess(DEFAULT_TIMEOUT) to IOProcess(timeout=DEFAULT_TIMEOUT) and remove the /usr/share/vdsm/storage/outOfProcess.pyc file and restart vdsm and supervdsm service on all hosts.... Thanks, Punit Dambiwal On Mon, Mar 23, 2015 at 9:18 AM, Punit Dambiwal <hypunit at gmail.com> wrote:> Hi All, > > Still i am facing the same issue...please help me to overcome this issue... > > Thanks, > punit > > On Fri, Mar 20, 2015 at 12:22 AM, Thomas Holkenbrink < > thomas.holkenbrink at fibercloud.com> wrote: > >> I?ve seen this before. The system thinks the storage system us up and >> running and then attempts to utilize it. >> >> The way I got around it was to put a delay in the startup of the gluster >> Node on the interface that the clients use to communicate. >> >> >> >> I use a bonded link, I then add a LINKDELAY to the interface to get the >> underlying system up and running before the network comes up. This then >> causes Network dependent features to wait for the network to finish. >> >> It adds about 10seconds to the startup time, in our environment it works >> well, you may not need as long of a delay. >> >> >> >> CentOS >> >> root at gls1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0 >> >> >> >> DEVICE=bond0 >> >> ONBOOT=yes >> >> BOOTPROTO=static >> >> USERCTL=no >> >> NETMASK=255.255.248.0 >> >> IPADDR=10.10.1.17 >> >> MTU=9000 >> >> IPV6INIT=no >> >> IPV6_AUTOCONF=no >> >> NETWORKING_IPV6=no >> >> NM_CONTROLLED=no >> >> LINKDELAY=10 >> >> NAME="System Storage Bond0" >> >> >> >> >> >> >> >> >> >> Hi Michal, >> >> >> >> The Storage domain is up and running and mounted on all the host >> nodes...as i updated before that it was working perfectly before but just >> after reboot can not make the VM poweron... >> >> >> >> [image: Inline image 1] >> >> >> >> [image: Inline image 2] >> >> >> >> [root at cpu01 log]# gluster volume info >> >> >> >> Volume Name: ds01 >> >> Type: Distributed-Replicate >> >> Volume ID: 369d3fdc-c8eb-46b7-a33e-0a49f2451ff6 >> >> Status: Started >> >> Number of Bricks: 48 x 2 = 96 >> >> Transport-type: tcp >> >> Bricks: >> >> Brick1: cpu01:/bricks/1/vol1 >> >> Brick2: cpu02:/bricks/1/vol1 >> >> Brick3: cpu03:/bricks/1/vol1 >> >> Brick4: cpu04:/bricks/1/vol1 >> >> Brick5: cpu01:/bricks/2/vol1 >> >> Brick6: cpu02:/bricks/2/vol1 >> >> Brick7: cpu03:/bricks/2/vol1 >> >> Brick8: cpu04:/bricks/2/vol1 >> >> Brick9: cpu01:/bricks/3/vol1 >> >> Brick10: cpu02:/bricks/3/vol1 >> >> Brick11: cpu03:/bricks/3/vol1 >> >> Brick12: cpu04:/bricks/3/vol1 >> >> Brick13: cpu01:/bricks/4/vol1 >> >> Brick14: cpu02:/bricks/4/vol1 >> >> Brick15: cpu03:/bricks/4/vol1 >> >> Brick16: cpu04:/bricks/4/vol1 >> >> Brick17: cpu01:/bricks/5/vol1 >> >> Brick18: cpu02:/bricks/5/vol1 >> >> Brick19: cpu03:/bricks/5/vol1 >> >> Brick20: cpu04:/bricks/5/vol1 >> >> Brick21: cpu01:/bricks/6/vol1 >> >> Brick22: cpu02:/bricks/6/vol1 >> >> Brick23: cpu03:/bricks/6/vol1 >> >> Brick24: cpu04:/bricks/6/vol1 >> >> Brick25: cpu01:/bricks/7/vol1 >> >> Brick26: cpu02:/bricks/7/vol1 >> >> Brick27: cpu03:/bricks/7/vol1 >> >> Brick28: cpu04:/bricks/7/vol1 >> >> Brick29: cpu01:/bricks/8/vol1 >> >> Brick30: cpu02:/bricks/8/vol1 >> >> Brick31: cpu03:/bricks/8/vol1 >> >> Brick32: cpu04:/bricks/8/vol1 >> >> Brick33: cpu01:/bricks/9/vol1 >> >> Brick34: cpu02:/bricks/9/vol1 >> >> Brick35: cpu03:/bricks/9/vol1 >> >> Brick36: cpu04:/bricks/9/vol1 >> >> Brick37: cpu01:/bricks/10/vol1 >> >> Brick38: cpu02:/bricks/10/vol1 >> >> Brick39: cpu03:/bricks/10/vol1 >> >> Brick40: cpu04:/bricks/10/vol1 >> >> Brick41: cpu01:/bricks/11/vol1 >> >> Brick42: cpu02:/bricks/11/vol1 >> >> Brick43: cpu03:/bricks/11/vol1 >> >> Brick44: cpu04:/bricks/11/vol1 >> >> Brick45: cpu01:/bricks/12/vol1 >> >> Brick46: cpu02:/bricks/12/vol1 >> >> Brick47: cpu03:/bricks/12/vol1 >> >> Brick48: cpu04:/bricks/12/vol1 >> >> Brick49: cpu01:/bricks/13/vol1 >> >> Brick50: cpu02:/bricks/13/vol1 >> >> Brick51: cpu03:/bricks/13/vol1 >> >> Brick52: cpu04:/bricks/13/vol1 >> >> Brick53: cpu01:/bricks/14/vol1 >> >> Brick54: cpu02:/bricks/14/vol1 >> >> Brick55: cpu03:/bricks/14/vol1 >> >> Brick56: cpu04:/bricks/14/vol1 >> >> Brick57: cpu01:/bricks/15/vol1 >> >> Brick58: cpu02:/bricks/15/vol1 >> >> Brick59: cpu03:/bricks/15/vol1 >> >> Brick60: cpu04:/bricks/15/vol1 >> >> Brick61: cpu01:/bricks/16/vol1 >> >> Brick62: cpu02:/bricks/16/vol1 >> >> Brick63: cpu03:/bricks/16/vol1 >> >> Brick64: cpu04:/bricks/16/vol1 >> >> Brick65: cpu01:/bricks/17/vol1 >> >> Brick66: cpu02:/bricks/17/vol1 >> >> Brick67: cpu03:/bricks/17/vol1 >> >> Brick68: cpu04:/bricks/17/vol1 >> >> Brick69: cpu01:/bricks/18/vol1 >> >> Brick70: cpu02:/bricks/18/vol1 >> >> Brick71: cpu03:/bricks/18/vol1 >> >> Brick72: cpu04:/bricks/18/vol1 >> >> Brick73: cpu01:/bricks/19/vol1 >> >> Brick74: cpu02:/bricks/19/vol1 >> >> Brick75: cpu03:/bricks/19/vol1 >> >> Brick76: cpu04:/bricks/19/vol1 >> >> Brick77: cpu01:/bricks/20/vol1 >> >> Brick78: cpu02:/bricks/20/vol1 >> >> Brick79: cpu03:/bricks/20/vol1 >> >> Brick80: cpu04:/bricks/20/vol1 >> >> Brick81: cpu01:/bricks/21/vol1 >> >> Brick82: cpu02:/bricks/21/vol1 >> >> Brick83: cpu03:/bricks/21/vol1 >> >> Brick84: cpu04:/bricks/21/vol1 >> >> Brick85: cpu01:/bricks/22/vol1 >> >> Brick86: cpu02:/bricks/22/vol1 >> >> Brick87: cpu03:/bricks/22/vol1 >> >> Brick88: cpu04:/bricks/22/vol1 >> >> Brick89: cpu01:/bricks/23/vol1 >> >> Brick90: cpu02:/bricks/23/vol1 >> >> Brick91: cpu03:/bricks/23/vol1 >> >> Brick92: cpu04:/bricks/23/vol1 >> >> Brick93: cpu01:/bricks/24/vol1 >> >> Brick94: cpu02:/bricks/24/vol1 >> >> Brick95: cpu03:/bricks/24/vol1 >> >> Brick96: cpu04:/bricks/24/vol1 >> >> Options Reconfigured: >> >> diagnostics.count-fop-hits: on >> >> diagnostics.latency-measurement: on >> >> nfs.disable: on >> >> user.cifs: enable >> >> auth.allow: 10.10.0.* >> >> performance.quick-read: off >> >> performance.read-ahead: off >> >> performance.io-cache: off >> >> performance.stat-prefetch: off >> >> cluster.eager-lock: enable >> >> network.remote-dio: enable >> >> cluster.quorum-type: auto >> >> cluster.server-quorum-type: server >> >> storage.owner-uid: 36 >> >> storage.owner-gid: 36 >> >> server.allow-insecure: on >> >> network.ping-timeout: 100 >> >> [root at cpu01 log]# >> >> >> >> ----------------------------------------- >> >> >> >> [root at cpu01 log]# gluster volume status >> >> Status of volume: ds01 >> >> Gluster process Port Online >> Pid >> >> >> ------------------------------------------------------------------------------ >> >> Brick cpu01:/bricks/1/vol1 49152 Y >> 33474 >> >> Brick cpu02:/bricks/1/vol1 49152 Y >> 40717 >> >> Brick cpu03:/bricks/1/vol1 49152 Y >> 18080 >> >> Brick cpu04:/bricks/1/vol1 49152 Y >> 40447 >> >> Brick cpu01:/bricks/2/vol1 49153 Y >> 33481 >> >> Brick cpu02:/bricks/2/vol1 49153 Y >> 40724 >> >> Brick cpu03:/bricks/2/vol1 49153 Y >> 18086 >> >> Brick cpu04:/bricks/2/vol1 49153 Y >> 40453 >> >> Brick cpu01:/bricks/3/vol1 49154 Y >> 33489 >> >> Brick cpu02:/bricks/3/vol1 49154 Y >> 40731 >> >> Brick cpu03:/bricks/3/vol1 49154 Y >> 18097 >> >> Brick cpu04:/bricks/3/vol1 49154 Y >> 40460 >> >> Brick cpu01:/bricks/4/vol1 49155 Y >> 33495 >> >> Brick cpu02:/bricks/4/vol1 49155 Y >> 40738 >> >> Brick cpu03:/bricks/4/vol1 49155 Y >> 18103 >> >> Brick cpu04:/bricks/4/vol1 49155 Y >> 40468 >> >> Brick cpu01:/bricks/5/vol1 49156 Y >> 33502 >> >> Brick cpu02:/bricks/5/vol1 49156 Y >> 40745 >> >> Brick cpu03:/bricks/5/vol1 49156 Y >> 18110 >> >> Brick cpu04:/bricks/5/vol1 49156 Y >> 40474 >> >> Brick cpu01:/bricks/6/vol1 49157 Y >> 33509 >> >> Brick cpu02:/bricks/6/vol1 49157 Y >> 40752 >> >> Brick cpu03:/bricks/6/vol1 49157 Y >> 18116 >> >> Brick cpu04:/bricks/6/vol1 49157 Y >> 40481 >> >> Brick cpu01:/bricks/7/vol1 49158 Y >> 33516 >> >> Brick cpu02:/bricks/7/vol1 49158 Y >> 40759 >> >> Brick cpu03:/bricks/7/vol1 49158 Y >> 18122 >> >> Brick cpu04:/bricks/7/vol1 49158 Y >> 40488 >> >> Brick cpu01:/bricks/8/vol1 49159 Y >> 33525 >> >> Brick cpu02:/bricks/8/vol1 49159 Y >> 40766 >> >> Brick cpu03:/bricks/8/vol1 49159 Y >> 18130 >> >> Brick cpu04:/bricks/8/vol1 49159 Y >> 40495 >> >> Brick cpu01:/bricks/9/vol1 49160 Y >> 33530 >> >> Brick cpu02:/bricks/9/vol1 49160 Y >> 40773 >> >> Brick cpu03:/bricks/9/vol1 49160 Y >> 18137 >> >> Brick cpu04:/bricks/9/vol1 49160 Y >> 40502 >> >> Brick cpu01:/bricks/10/vol1 49161 Y >> 33538 >> >> Brick cpu02:/bricks/10/vol1 49161 Y >> 40780 >> >> Brick cpu03:/bricks/10/vol1 49161 Y >> 18143 >> >> Brick cpu04:/bricks/10/vol1 49161 Y >> 40509 >> >> Brick cpu01:/bricks/11/vol1 49162 Y >> 33544 >> >> Brick cpu02:/bricks/11/vol1 49162 Y >> 40787 >> >> Brick cpu03:/bricks/11/vol1 49162 Y >> 18150 >> >> Brick cpu04:/bricks/11/vol1 49162 Y >> 40516 >> >> Brick cpu01:/bricks/12/vol1 49163 Y >> 33551 >> >> Brick cpu02:/bricks/12/vol1 49163 Y >> 40794 >> >> Brick cpu03:/bricks/12/vol1 49163 Y >> 18157 >> >> Brick cpu04:/bricks/12/vol1 49163 Y >> 40692 >> >> Brick cpu01:/bricks/13/vol1 49164 Y >> 33558 >> >> Brick cpu02:/bricks/13/vol1 49164 Y >> 40801 >> >> Brick cpu03:/bricks/13/vol1 49164 Y >> 18165 >> >> Brick cpu04:/bricks/13/vol1 49164 Y >> 40700 >> >> Brick cpu01:/bricks/14/vol1 49165 Y >> 33566 >> >> Brick cpu02:/bricks/14/vol1 49165 Y >> 40809 >> >> Brick cpu03:/bricks/14/vol1 49165 Y >> 18172 >> >> Brick cpu04:/bricks/14/vol1 49165 Y >> 40706 >> >> Brick cpu01:/bricks/15/vol1 49166 Y >> 33572 >> >> Brick cpu02:/bricks/15/vol1 49166 Y >> 40815 >> >> Brick cpu03:/bricks/15/vol1 49166 Y >> 18179 >> >> Brick cpu04:/bricks/15/vol1 49166 Y >> 40714 >> >> Brick cpu01:/bricks/16/vol1 49167 Y >> 33579 >> >> Brick cpu02:/bricks/16/vol1 49167 Y >> 40822 >> >> Brick cpu03:/bricks/16/vol1 49167 Y >> 18185 >> >> Brick cpu04:/bricks/16/vol1 49167 Y >> 40722 >> >> Brick cpu01:/bricks/17/vol1 49168 Y >> 33586 >> >> Brick cpu02:/bricks/17/vol1 49168 Y >> 40829 >> >> Brick cpu03:/bricks/17/vol1 49168 Y >> 18192 >> >> Brick cpu04:/bricks/17/vol1 49168 Y >> 40727 >> >> Brick cpu01:/bricks/18/vol1 49169 Y >> 33593 >> >> Brick cpu02:/bricks/18/vol1 49169 Y >> 40836 >> >> Brick cpu03:/bricks/18/vol1 49169 Y >> 18201 >> >> Brick cpu04:/bricks/18/vol1 49169 Y >> 40735 >> >> Brick cpu01:/bricks/19/vol1 49170 Y >> 33600 >> >> Brick cpu02:/bricks/19/vol1 49170 Y >> 40843 >> >> Brick cpu03:/bricks/19/vol1 49170 Y >> 18207 >> >> Brick cpu04:/bricks/19/vol1 49170 Y >> 40741 >> >> Brick cpu01:/bricks/20/vol1 49171 Y >> 33608 >> >> Brick cpu02:/bricks/20/vol1 49171 Y >> 40850 >> >> Brick cpu03:/bricks/20/vol1 49171 Y >> 18214 >> >> Brick cpu04:/bricks/20/vol1 49171 Y >> 40748 >> >> Brick cpu01:/bricks/21/vol1 49172 Y >> 33614 >> >> Brick cpu02:/bricks/21/vol1 49172 Y >> 40858 >> >> Brick cpu03:/bricks/21/vol1 49172 Y >> 18222 >> >> Brick cpu04:/bricks/21/vol1 49172 Y >> 40756 >> >> Brick cpu01:/bricks/22/vol1 49173 Y >> 33621 >> >> Brick cpu02:/bricks/22/vol1 49173 Y >> 40864 >> >> Brick cpu03:/bricks/22/vol1 49173 Y >> 18227 >> >> Brick cpu04:/bricks/22/vol1 49173 Y >> 40762 >> >> Brick cpu01:/bricks/23/vol1 49174 Y >> 33626 >> >> Brick cpu02:/bricks/23/vol1 49174 Y >> 40869 >> >> Brick cpu03:/bricks/23/vol1 49174 Y >> 18234 >> >> Brick cpu04:/bricks/23/vol1 49174 Y >> 40769 >> >> Brick cpu01:/bricks/24/vol1 49175 Y >> 33631 >> >> Brick cpu02:/bricks/24/vol1 49175 Y >> 40874 >> >> Brick cpu03:/bricks/24/vol1 49175 Y >> 18239 >> >> Brick cpu04:/bricks/24/vol1 49175 Y >> 40774 >> >> Self-heal Daemon on localhost N/A Y >> 33361 >> >> Self-heal Daemon on cpu05 N/A Y >> 2353 >> >> Self-heal Daemon on cpu04 N/A Y >> 40786 >> >> Self-heal Daemon on cpu02 N/A Y >> 32442 >> >> Self-heal Daemon on cpu03 N/A Y >> 18664 >> >> >> >> Task Status of Volume ds01 >> >> >> ------------------------------------------------------------------------------ >> >> Task : Rebalance >> >> ID : 5db24b30-4b9f-4b65-8910-a7a0a6d327a4 >> >> Status : completed >> >> >> >> [root at cpu01 log]# >> >> >> >> [root at cpu01 log]# gluster pool list >> >> UUID Hostname State >> >> 626c9360-8c09-480f-9707-116e67cc38e6 cpu02 Connected >> >> dc475d62-b035-4ee6-9006-6f03bf68bf24 cpu05 Connected >> >> 41b5b2ff-3671-47b4-b477-227a107e718d cpu03 Connected >> >> c0afe114-dfa7-407d-bad7-5a3f97a6f3fc cpu04 Connected >> >> 9b61b0a5-be78-4ac2-b6c0-2db588da5c35 localhost Connected >> >> [root at cpu01 log]# >> >> >> >> [image: Inline image 3] >> >> >> >> Thanks, >> >> Punit >> >> >> >> On Thu, Mar 19, 2015 at 2:53 PM, Michal Skrivanek < >> michal.skrivanek at redhat.com> wrote: >> >> >> On Mar 19, 2015, at 03:18 , Punit Dambiwal <hypunit at gmail.com> wrote: >> >> > Hi All, >> > >> > Is there any one have any idea about this problem...it seems it's bug >> either in Ovirt or Glusterfs...that's why no one has the idea about >> it....please correct me if i am wrong?. >> >> Hi, >> as I said, storage access times out; so it seems to me as a gluster setup >> problem, the storage domain you have your VMs on is not working? >> >> Thanks, >> michal >> >> >> > >> > Thanks, >> > Punit >> > >> > On Wed, Mar 18, 2015 at 5:05 PM, Punit Dambiwal <hypunit at gmail.com> >> wrote: >> > Hi Michal, >> > >> > Would you mind to let me know the possible messedup things...i will >> check and try to resolve it....still i am communicating gluster community >> to resolve this issue... >> > >> > But in the ovirt....gluster setup is quite straight....so how come it >> will be messedup with reboot ?? if it can be messedup with reboot then it >> seems not good and stable technology for the production storage.... >> > >> > Thanks, >> > Punit >> > >> > On Wed, Mar 18, 2015 at 3:51 PM, Michal Skrivanek < >> michal.skrivanek at redhat.com> wrote: >> > >> > On Mar 18, 2015, at 03:33 , Punit Dambiwal <hypunit at gmail.com> wrote: >> > >> > > Hi, >> > > >> > > Is there any one from community can help me to solve this issue...?? >> > > >> > > Thanks, >> > > Punit >> > > >> > > On Tue, Mar 17, 2015 at 12:52 PM, Punit Dambiwal <hypunit at gmail.com> >> wrote: >> > > Hi, >> > > >> > > I am facing one strange issue with ovirt/glusterfs....still didn't >> find this issue is related with glusterfs or Ovirt.... >> > > >> > > Ovirt :- 3.5.1 >> > > Glusterfs :- 3.6.1 >> > > Host :- 4 Hosts (Compute+ Storage)...each server has 24 bricks >> > > Guest VM :- more then 100 >> > > >> > > Issue :- When i deploy this cluster first time..it work well for >> me(all the guest VM created and running successfully)....but suddenly one >> day my one of the host node rebooted and none of the VM can boot up >> now...and failed with the following error "Bad Volume Specification" >> > > >> > > VMId :- d877313c18d9783ca09b62acf5588048 >> > > >> > > VDSM Logs :- http://ur1.ca/jxabi >> > >> > you've got timeouts while accessing storage?so I guess something got >> messed up on reboot, it may also be just a gluster misconfiguration? >> > >> > > Engine Logs :- http://ur1.ca/jxabv >> > > >> > > ------------------------ >> > > [root at cpu01 ~]# vdsClient -s 0 getVolumeInfo >> e732a82f-bae9-4368-8b98-dedc1c3814de 00000002-0002-0002-0002-000000000145 >> 6d123509-6867-45cf-83a2-6d679b77d3c5 9030bb43-6bc9-462f-a1b9-f6d5a02fb180 >> > > status = OK >> > > domain = e732a82f-bae9-4368-8b98-dedc1c3814de >> > > capacity = 21474836480 >> > > voltype = LEAF >> > > description >> > > parent = 00000000-0000-0000-0000-000000000000 >> > > format = RAW >> > > image = 6d123509-6867-45cf-83a2-6d679b77d3c5 >> > > uuid = 9030bb43-6bc9-462f-a1b9-f6d5a02fb180 >> > > disktype = 2 >> > > legality = LEGAL >> > > mtime = 0 >> > > apparentsize = 21474836480 >> > > truesize = 4562972672 >> > > type = SPARSE >> > > children = [] >> > > pool >> > > ctime = 1422676305 >> > > --------------------- >> > > >> > > I opened same thread earlier but didn't get any perfect answers to >> solve this issue..so i reopen it... >> > > >> > > https://www.mail-archive.com/users at ovirt.org/msg25011.html >> > > >> > > Thanks, >> > > Punit >> > > >> > > >> > > >> > >> > >> > >> >> >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150325/197e4a24/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.png Type: image/png Size: 11779 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150325/197e4a24/attachment.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 10223 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150325/197e4a24/attachment-0001.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.png Type: image/png Size: 22346 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150325/197e4a24/attachment-0002.png>
Kaushal M
2015-Mar-25 10:52 UTC
[Gluster-users] VM failed to start | Bad volume specification
Awesome Punit! I'm happy to have been a part of the debugging process. ~kaushal On Wed, Mar 25, 2015 at 3:09 PM, Punit Dambiwal <hypunit at gmail.com> wrote:> Hi All, > > With the help of gluster community and ovirt-china community...my issue > got resolved... > > The main root cause was the following :- > > 1. the glob operation takes quite a long time, longer than the ioprocess > default 60s.. > 2. python-ioprocess updated which makes a single change of configuration > file doesn't work properly, only because this we should hack the code > manually... > > Solution (Need to do on all the hosts) :- > > 1. Add the the ioprocess timeout value in the /etc/vdsm/vdsm.conf file as > :- > > ------------ > [irs] > process_pool_timeout = 180 > ------------- > > 2. Check /usr/share/vdsm/storage/outOfProcess.py, line 71 and see whether > there is still "IOProcess(DEFAULT_TIMEOUT)" in it,if yes...then changing > the configuration file takes no effect because now timeout is the third > parameter not the second of IOProcess.__init__(). > > 3. Change IOProcess(DEFAULT_TIMEOUT) to > IOProcess(timeout=DEFAULT_TIMEOUT) and remove the > /usr/share/vdsm/storage/outOfProcess.pyc file and restart vdsm and > supervdsm service on all hosts.... > > Thanks, > Punit Dambiwal > > > On Mon, Mar 23, 2015 at 9:18 AM, Punit Dambiwal <hypunit at gmail.com> wrote: > >> Hi All, >> >> Still i am facing the same issue...please help me to overcome this >> issue... >> >> Thanks, >> punit >> >> On Fri, Mar 20, 2015 at 12:22 AM, Thomas Holkenbrink < >> thomas.holkenbrink at fibercloud.com> wrote: >> >>> I?ve seen this before. The system thinks the storage system us up and >>> running and then attempts to utilize it. >>> >>> The way I got around it was to put a delay in the startup of the gluster >>> Node on the interface that the clients use to communicate. >>> >>> >>> >>> I use a bonded link, I then add a LINKDELAY to the interface to get the >>> underlying system up and running before the network comes up. This then >>> causes Network dependent features to wait for the network to finish. >>> >>> It adds about 10seconds to the startup time, in our environment it works >>> well, you may not need as long of a delay. >>> >>> >>> >>> CentOS >>> >>> root at gls1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0 >>> >>> >>> >>> DEVICE=bond0 >>> >>> ONBOOT=yes >>> >>> BOOTPROTO=static >>> >>> USERCTL=no >>> >>> NETMASK=255.255.248.0 >>> >>> IPADDR=10.10.1.17 >>> >>> MTU=9000 >>> >>> IPV6INIT=no >>> >>> IPV6_AUTOCONF=no >>> >>> NETWORKING_IPV6=no >>> >>> NM_CONTROLLED=no >>> >>> LINKDELAY=10 >>> >>> NAME="System Storage Bond0" >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> Hi Michal, >>> >>> >>> >>> The Storage domain is up and running and mounted on all the host >>> nodes...as i updated before that it was working perfectly before but just >>> after reboot can not make the VM poweron... >>> >>> >>> >>> [image: Inline image 1] >>> >>> >>> >>> [image: Inline image 2] >>> >>> >>> >>> [root at cpu01 log]# gluster volume info >>> >>> >>> >>> Volume Name: ds01 >>> >>> Type: Distributed-Replicate >>> >>> Volume ID: 369d3fdc-c8eb-46b7-a33e-0a49f2451ff6 >>> >>> Status: Started >>> >>> Number of Bricks: 48 x 2 = 96 >>> >>> Transport-type: tcp >>> >>> Bricks: >>> >>> Brick1: cpu01:/bricks/1/vol1 >>> >>> Brick2: cpu02:/bricks/1/vol1 >>> >>> Brick3: cpu03:/bricks/1/vol1 >>> >>> Brick4: cpu04:/bricks/1/vol1 >>> >>> Brick5: cpu01:/bricks/2/vol1 >>> >>> Brick6: cpu02:/bricks/2/vol1 >>> >>> Brick7: cpu03:/bricks/2/vol1 >>> >>> Brick8: cpu04:/bricks/2/vol1 >>> >>> Brick9: cpu01:/bricks/3/vol1 >>> >>> Brick10: cpu02:/bricks/3/vol1 >>> >>> Brick11: cpu03:/bricks/3/vol1 >>> >>> Brick12: cpu04:/bricks/3/vol1 >>> >>> Brick13: cpu01:/bricks/4/vol1 >>> >>> Brick14: cpu02:/bricks/4/vol1 >>> >>> Brick15: cpu03:/bricks/4/vol1 >>> >>> Brick16: cpu04:/bricks/4/vol1 >>> >>> Brick17: cpu01:/bricks/5/vol1 >>> >>> Brick18: cpu02:/bricks/5/vol1 >>> >>> Brick19: cpu03:/bricks/5/vol1 >>> >>> Brick20: cpu04:/bricks/5/vol1 >>> >>> Brick21: cpu01:/bricks/6/vol1 >>> >>> Brick22: cpu02:/bricks/6/vol1 >>> >>> Brick23: cpu03:/bricks/6/vol1 >>> >>> Brick24: cpu04:/bricks/6/vol1 >>> >>> Brick25: cpu01:/bricks/7/vol1 >>> >>> Brick26: cpu02:/bricks/7/vol1 >>> >>> Brick27: cpu03:/bricks/7/vol1 >>> >>> Brick28: cpu04:/bricks/7/vol1 >>> >>> Brick29: cpu01:/bricks/8/vol1 >>> >>> Brick30: cpu02:/bricks/8/vol1 >>> >>> Brick31: cpu03:/bricks/8/vol1 >>> >>> Brick32: cpu04:/bricks/8/vol1 >>> >>> Brick33: cpu01:/bricks/9/vol1 >>> >>> Brick34: cpu02:/bricks/9/vol1 >>> >>> Brick35: cpu03:/bricks/9/vol1 >>> >>> Brick36: cpu04:/bricks/9/vol1 >>> >>> Brick37: cpu01:/bricks/10/vol1 >>> >>> Brick38: cpu02:/bricks/10/vol1 >>> >>> Brick39: cpu03:/bricks/10/vol1 >>> >>> Brick40: cpu04:/bricks/10/vol1 >>> >>> Brick41: cpu01:/bricks/11/vol1 >>> >>> Brick42: cpu02:/bricks/11/vol1 >>> >>> Brick43: cpu03:/bricks/11/vol1 >>> >>> Brick44: cpu04:/bricks/11/vol1 >>> >>> Brick45: cpu01:/bricks/12/vol1 >>> >>> Brick46: cpu02:/bricks/12/vol1 >>> >>> Brick47: cpu03:/bricks/12/vol1 >>> >>> Brick48: cpu04:/bricks/12/vol1 >>> >>> Brick49: cpu01:/bricks/13/vol1 >>> >>> Brick50: cpu02:/bricks/13/vol1 >>> >>> Brick51: cpu03:/bricks/13/vol1 >>> >>> Brick52: cpu04:/bricks/13/vol1 >>> >>> Brick53: cpu01:/bricks/14/vol1 >>> >>> Brick54: cpu02:/bricks/14/vol1 >>> >>> Brick55: cpu03:/bricks/14/vol1 >>> >>> Brick56: cpu04:/bricks/14/vol1 >>> >>> Brick57: cpu01:/bricks/15/vol1 >>> >>> Brick58: cpu02:/bricks/15/vol1 >>> >>> Brick59: cpu03:/bricks/15/vol1 >>> >>> Brick60: cpu04:/bricks/15/vol1 >>> >>> Brick61: cpu01:/bricks/16/vol1 >>> >>> Brick62: cpu02:/bricks/16/vol1 >>> >>> Brick63: cpu03:/bricks/16/vol1 >>> >>> Brick64: cpu04:/bricks/16/vol1 >>> >>> Brick65: cpu01:/bricks/17/vol1 >>> >>> Brick66: cpu02:/bricks/17/vol1 >>> >>> Brick67: cpu03:/bricks/17/vol1 >>> >>> Brick68: cpu04:/bricks/17/vol1 >>> >>> Brick69: cpu01:/bricks/18/vol1 >>> >>> Brick70: cpu02:/bricks/18/vol1 >>> >>> Brick71: cpu03:/bricks/18/vol1 >>> >>> Brick72: cpu04:/bricks/18/vol1 >>> >>> Brick73: cpu01:/bricks/19/vol1 >>> >>> Brick74: cpu02:/bricks/19/vol1 >>> >>> Brick75: cpu03:/bricks/19/vol1 >>> >>> Brick76: cpu04:/bricks/19/vol1 >>> >>> Brick77: cpu01:/bricks/20/vol1 >>> >>> Brick78: cpu02:/bricks/20/vol1 >>> >>> Brick79: cpu03:/bricks/20/vol1 >>> >>> Brick80: cpu04:/bricks/20/vol1 >>> >>> Brick81: cpu01:/bricks/21/vol1 >>> >>> Brick82: cpu02:/bricks/21/vol1 >>> >>> Brick83: cpu03:/bricks/21/vol1 >>> >>> Brick84: cpu04:/bricks/21/vol1 >>> >>> Brick85: cpu01:/bricks/22/vol1 >>> >>> Brick86: cpu02:/bricks/22/vol1 >>> >>> Brick87: cpu03:/bricks/22/vol1 >>> >>> Brick88: cpu04:/bricks/22/vol1 >>> >>> Brick89: cpu01:/bricks/23/vol1 >>> >>> Brick90: cpu02:/bricks/23/vol1 >>> >>> Brick91: cpu03:/bricks/23/vol1 >>> >>> Brick92: cpu04:/bricks/23/vol1 >>> >>> Brick93: cpu01:/bricks/24/vol1 >>> >>> Brick94: cpu02:/bricks/24/vol1 >>> >>> Brick95: cpu03:/bricks/24/vol1 >>> >>> Brick96: cpu04:/bricks/24/vol1 >>> >>> Options Reconfigured: >>> >>> diagnostics.count-fop-hits: on >>> >>> diagnostics.latency-measurement: on >>> >>> nfs.disable: on >>> >>> user.cifs: enable >>> >>> auth.allow: 10.10.0.* >>> >>> performance.quick-read: off >>> >>> performance.read-ahead: off >>> >>> performance.io-cache: off >>> >>> performance.stat-prefetch: off >>> >>> cluster.eager-lock: enable >>> >>> network.remote-dio: enable >>> >>> cluster.quorum-type: auto >>> >>> cluster.server-quorum-type: server >>> >>> storage.owner-uid: 36 >>> >>> storage.owner-gid: 36 >>> >>> server.allow-insecure: on >>> >>> network.ping-timeout: 100 >>> >>> [root at cpu01 log]# >>> >>> >>> >>> ----------------------------------------- >>> >>> >>> >>> [root at cpu01 log]# gluster volume status >>> >>> Status of volume: ds01 >>> >>> Gluster process Port Online >>> Pid >>> >>> >>> ------------------------------------------------------------------------------ >>> >>> Brick cpu01:/bricks/1/vol1 49152 Y >>> 33474 >>> >>> Brick cpu02:/bricks/1/vol1 49152 Y >>> 40717 >>> >>> Brick cpu03:/bricks/1/vol1 49152 Y >>> 18080 >>> >>> Brick cpu04:/bricks/1/vol1 49152 Y >>> 40447 >>> >>> Brick cpu01:/bricks/2/vol1 49153 Y >>> 33481 >>> >>> Brick cpu02:/bricks/2/vol1 49153 Y >>> 40724 >>> >>> Brick cpu03:/bricks/2/vol1 49153 Y >>> 18086 >>> >>> Brick cpu04:/bricks/2/vol1 49153 Y >>> 40453 >>> >>> Brick cpu01:/bricks/3/vol1 49154 Y >>> 33489 >>> >>> Brick cpu02:/bricks/3/vol1 49154 Y >>> 40731 >>> >>> Brick cpu03:/bricks/3/vol1 49154 Y >>> 18097 >>> >>> Brick cpu04:/bricks/3/vol1 49154 Y >>> 40460 >>> >>> Brick cpu01:/bricks/4/vol1 49155 Y >>> 33495 >>> >>> Brick cpu02:/bricks/4/vol1 49155 Y >>> 40738 >>> >>> Brick cpu03:/bricks/4/vol1 49155 Y >>> 18103 >>> >>> Brick cpu04:/bricks/4/vol1 49155 Y >>> 40468 >>> >>> Brick cpu01:/bricks/5/vol1 49156 Y >>> 33502 >>> >>> Brick cpu02:/bricks/5/vol1 49156 Y >>> 40745 >>> >>> Brick cpu03:/bricks/5/vol1 49156 Y >>> 18110 >>> >>> Brick cpu04:/bricks/5/vol1 49156 Y >>> 40474 >>> >>> Brick cpu01:/bricks/6/vol1 49157 Y >>> 33509 >>> >>> Brick cpu02:/bricks/6/vol1 49157 Y >>> 40752 >>> >>> Brick cpu03:/bricks/6/vol1 49157 Y >>> 18116 >>> >>> Brick cpu04:/bricks/6/vol1 49157 Y >>> 40481 >>> >>> Brick cpu01:/bricks/7/vol1 49158 Y >>> 33516 >>> >>> Brick cpu02:/bricks/7/vol1 49158 Y >>> 40759 >>> >>> Brick cpu03:/bricks/7/vol1 49158 Y >>> 18122 >>> >>> Brick cpu04:/bricks/7/vol1 49158 Y >>> 40488 >>> >>> Brick cpu01:/bricks/8/vol1 49159 Y >>> 33525 >>> >>> Brick cpu02:/bricks/8/vol1 49159 Y >>> 40766 >>> >>> Brick cpu03:/bricks/8/vol1 49159 Y >>> 18130 >>> >>> Brick cpu04:/bricks/8/vol1 49159 Y >>> 40495 >>> >>> Brick cpu01:/bricks/9/vol1 49160 Y >>> 33530 >>> >>> Brick cpu02:/bricks/9/vol1 49160 Y >>> 40773 >>> >>> Brick cpu03:/bricks/9/vol1 49160 Y >>> 18137 >>> >>> Brick cpu04:/bricks/9/vol1 49160 Y >>> 40502 >>> >>> Brick cpu01:/bricks/10/vol1 49161 Y >>> 33538 >>> >>> Brick cpu02:/bricks/10/vol1 49161 Y >>> 40780 >>> >>> Brick cpu03:/bricks/10/vol1 49161 Y >>> 18143 >>> >>> Brick cpu04:/bricks/10/vol1 49161 Y >>> 40509 >>> >>> Brick cpu01:/bricks/11/vol1 49162 Y >>> 33544 >>> >>> Brick cpu02:/bricks/11/vol1 49162 Y >>> 40787 >>> >>> Brick cpu03:/bricks/11/vol1 49162 Y >>> 18150 >>> >>> Brick cpu04:/bricks/11/vol1 49162 Y >>> 40516 >>> >>> Brick cpu01:/bricks/12/vol1 49163 Y >>> 33551 >>> >>> Brick cpu02:/bricks/12/vol1 49163 Y >>> 40794 >>> >>> Brick cpu03:/bricks/12/vol1 49163 Y >>> 18157 >>> >>> Brick cpu04:/bricks/12/vol1 49163 Y >>> 40692 >>> >>> Brick cpu01:/bricks/13/vol1 49164 Y >>> 33558 >>> >>> Brick cpu02:/bricks/13/vol1 49164 Y >>> 40801 >>> >>> Brick cpu03:/bricks/13/vol1 49164 Y >>> 18165 >>> >>> Brick cpu04:/bricks/13/vol1 49164 Y >>> 40700 >>> >>> Brick cpu01:/bricks/14/vol1 49165 Y >>> 33566 >>> >>> Brick cpu02:/bricks/14/vol1 49165 Y >>> 40809 >>> >>> Brick cpu03:/bricks/14/vol1 49165 Y >>> 18172 >>> >>> Brick cpu04:/bricks/14/vol1 49165 Y >>> 40706 >>> >>> Brick cpu01:/bricks/15/vol1 49166 Y >>> 33572 >>> >>> Brick cpu02:/bricks/15/vol1 49166 Y >>> 40815 >>> >>> Brick cpu03:/bricks/15/vol1 49166 Y >>> 18179 >>> >>> Brick cpu04:/bricks/15/vol1 49166 Y >>> 40714 >>> >>> Brick cpu01:/bricks/16/vol1 49167 Y >>> 33579 >>> >>> Brick cpu02:/bricks/16/vol1 49167 Y >>> 40822 >>> >>> Brick cpu03:/bricks/16/vol1 49167 Y >>> 18185 >>> >>> Brick cpu04:/bricks/16/vol1 49167 Y >>> 40722 >>> >>> Brick cpu01:/bricks/17/vol1 49168 Y >>> 33586 >>> >>> Brick cpu02:/bricks/17/vol1 49168 Y >>> 40829 >>> >>> Brick cpu03:/bricks/17/vol1 49168 Y >>> 18192 >>> >>> Brick cpu04:/bricks/17/vol1 49168 Y >>> 40727 >>> >>> Brick cpu01:/bricks/18/vol1 49169 Y >>> 33593 >>> >>> Brick cpu02:/bricks/18/vol1 49169 Y >>> 40836 >>> >>> Brick cpu03:/bricks/18/vol1 49169 Y >>> 18201 >>> >>> Brick cpu04:/bricks/18/vol1 49169 Y >>> 40735 >>> >>> Brick cpu01:/bricks/19/vol1 49170 Y >>> 33600 >>> >>> Brick cpu02:/bricks/19/vol1 49170 Y >>> 40843 >>> >>> Brick cpu03:/bricks/19/vol1 49170 Y >>> 18207 >>> >>> Brick cpu04:/bricks/19/vol1 49170 Y >>> 40741 >>> >>> Brick cpu01:/bricks/20/vol1 49171 Y >>> 33608 >>> >>> Brick cpu02:/bricks/20/vol1 49171 Y >>> 40850 >>> >>> Brick cpu03:/bricks/20/vol1 49171 Y >>> 18214 >>> >>> Brick cpu04:/bricks/20/vol1 49171 Y >>> 40748 >>> >>> Brick cpu01:/bricks/21/vol1 49172 Y >>> 33614 >>> >>> Brick cpu02:/bricks/21/vol1 49172 Y >>> 40858 >>> >>> Brick cpu03:/bricks/21/vol1 49172 Y >>> 18222 >>> >>> Brick cpu04:/bricks/21/vol1 49172 Y >>> 40756 >>> >>> Brick cpu01:/bricks/22/vol1 49173 Y >>> 33621 >>> >>> Brick cpu02:/bricks/22/vol1 49173 Y >>> 40864 >>> >>> Brick cpu03:/bricks/22/vol1 49173 Y >>> 18227 >>> >>> Brick cpu04:/bricks/22/vol1 49173 Y >>> 40762 >>> >>> Brick cpu01:/bricks/23/vol1 49174 Y >>> 33626 >>> >>> Brick cpu02:/bricks/23/vol1 49174 Y >>> 40869 >>> >>> Brick cpu03:/bricks/23/vol1 49174 Y >>> 18234 >>> >>> Brick cpu04:/bricks/23/vol1 49174 Y >>> 40769 >>> >>> Brick cpu01:/bricks/24/vol1 49175 Y >>> 33631 >>> >>> Brick cpu02:/bricks/24/vol1 49175 Y >>> 40874 >>> >>> Brick cpu03:/bricks/24/vol1 49175 Y >>> 18239 >>> >>> Brick cpu04:/bricks/24/vol1 49175 Y >>> 40774 >>> >>> Self-heal Daemon on localhost N/A Y >>> 33361 >>> >>> Self-heal Daemon on cpu05 N/A Y >>> 2353 >>> >>> Self-heal Daemon on cpu04 N/A Y >>> 40786 >>> >>> Self-heal Daemon on cpu02 N/A Y >>> 32442 >>> >>> Self-heal Daemon on cpu03 N/A Y >>> 18664 >>> >>> >>> >>> Task Status of Volume ds01 >>> >>> >>> ------------------------------------------------------------------------------ >>> >>> Task : Rebalance >>> >>> ID : 5db24b30-4b9f-4b65-8910-a7a0a6d327a4 >>> >>> Status : completed >>> >>> >>> >>> [root at cpu01 log]# >>> >>> >>> >>> [root at cpu01 log]# gluster pool list >>> >>> UUID Hostname State >>> >>> 626c9360-8c09-480f-9707-116e67cc38e6 cpu02 Connected >>> >>> dc475d62-b035-4ee6-9006-6f03bf68bf24 cpu05 Connected >>> >>> 41b5b2ff-3671-47b4-b477-227a107e718d cpu03 Connected >>> >>> c0afe114-dfa7-407d-bad7-5a3f97a6f3fc cpu04 Connected >>> >>> 9b61b0a5-be78-4ac2-b6c0-2db588da5c35 localhost Connected >>> >>> [root at cpu01 log]# >>> >>> >>> >>> [image: Inline image 3] >>> >>> >>> >>> Thanks, >>> >>> Punit >>> >>> >>> >>> On Thu, Mar 19, 2015 at 2:53 PM, Michal Skrivanek < >>> michal.skrivanek at redhat.com> wrote: >>> >>> >>> On Mar 19, 2015, at 03:18 , Punit Dambiwal <hypunit at gmail.com> wrote: >>> >>> > Hi All, >>> > >>> > Is there any one have any idea about this problem...it seems it's bug >>> either in Ovirt or Glusterfs...that's why no one has the idea about >>> it....please correct me if i am wrong?. >>> >>> Hi, >>> as I said, storage access times out; so it seems to me as a gluster >>> setup problem, the storage domain you have your VMs on is not working? >>> >>> Thanks, >>> michal >>> >>> >>> > >>> > Thanks, >>> > Punit >>> > >>> > On Wed, Mar 18, 2015 at 5:05 PM, Punit Dambiwal <hypunit at gmail.com> >>> wrote: >>> > Hi Michal, >>> > >>> > Would you mind to let me know the possible messedup things...i will >>> check and try to resolve it....still i am communicating gluster community >>> to resolve this issue... >>> > >>> > But in the ovirt....gluster setup is quite straight....so how come it >>> will be messedup with reboot ?? if it can be messedup with reboot then it >>> seems not good and stable technology for the production storage.... >>> > >>> > Thanks, >>> > Punit >>> > >>> > On Wed, Mar 18, 2015 at 3:51 PM, Michal Skrivanek < >>> michal.skrivanek at redhat.com> wrote: >>> > >>> > On Mar 18, 2015, at 03:33 , Punit Dambiwal <hypunit at gmail.com> wrote: >>> > >>> > > Hi, >>> > > >>> > > Is there any one from community can help me to solve this issue...?? >>> > > >>> > > Thanks, >>> > > Punit >>> > > >>> > > On Tue, Mar 17, 2015 at 12:52 PM, Punit Dambiwal <hypunit at gmail.com> >>> wrote: >>> > > Hi, >>> > > >>> > > I am facing one strange issue with ovirt/glusterfs....still didn't >>> find this issue is related with glusterfs or Ovirt.... >>> > > >>> > > Ovirt :- 3.5.1 >>> > > Glusterfs :- 3.6.1 >>> > > Host :- 4 Hosts (Compute+ Storage)...each server has 24 bricks >>> > > Guest VM :- more then 100 >>> > > >>> > > Issue :- When i deploy this cluster first time..it work well for >>> me(all the guest VM created and running successfully)....but suddenly one >>> day my one of the host node rebooted and none of the VM can boot up >>> now...and failed with the following error "Bad Volume Specification" >>> > > >>> > > VMId :- d877313c18d9783ca09b62acf5588048 >>> > > >>> > > VDSM Logs :- http://ur1.ca/jxabi >>> > >>> > you've got timeouts while accessing storage?so I guess something got >>> messed up on reboot, it may also be just a gluster misconfiguration? >>> > >>> > > Engine Logs :- http://ur1.ca/jxabv >>> > > >>> > > ------------------------ >>> > > [root at cpu01 ~]# vdsClient -s 0 getVolumeInfo >>> e732a82f-bae9-4368-8b98-dedc1c3814de 00000002-0002-0002-0002-000000000145 >>> 6d123509-6867-45cf-83a2-6d679b77d3c5 9030bb43-6bc9-462f-a1b9-f6d5a02fb180 >>> > > status = OK >>> > > domain = e732a82f-bae9-4368-8b98-dedc1c3814de >>> > > capacity = 21474836480 >>> > > voltype = LEAF >>> > > description >>> > > parent = 00000000-0000-0000-0000-000000000000 >>> > > format = RAW >>> > > image = 6d123509-6867-45cf-83a2-6d679b77d3c5 >>> > > uuid = 9030bb43-6bc9-462f-a1b9-f6d5a02fb180 >>> > > disktype = 2 >>> > > legality = LEGAL >>> > > mtime = 0 >>> > > apparentsize = 21474836480 >>> > > truesize = 4562972672 >>> > > type = SPARSE >>> > > children = [] >>> > > pool >>> > > ctime = 1422676305 >>> > > --------------------- >>> > > >>> > > I opened same thread earlier but didn't get any perfect answers to >>> solve this issue..so i reopen it... >>> > > >>> > > https://www.mail-archive.com/users at ovirt.org/msg25011.html >>> > > >>> > > Thanks, >>> > > Punit >>> > > >>> > > >>> > > >>> > >>> > >>> > >>> >>> >>> >> >> > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150325/45e891cc/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 10223 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150325/45e891cc/attachment.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.png Type: image/png Size: 11779 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150325/45e891cc/attachment-0001.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.png Type: image/png Size: 22346 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150325/45e891cc/attachment-0002.png>