thr3ads.net - Gluster users - [Gluster-users] VM failed to start

If this information is useful, please help other people find it:
Share via:

Punit Dambiwal

2015-Mar-23 01:18 UTC

[Gluster-users] VM failed to start | Bad volume specification

Hi All,

Still i am facing the same issue...please help me to overcome this issue...

Thanks,
punit

On Fri, Mar 20, 2015 at 12:22 AM, Thomas Holkenbrink <
thomas.holkenbrink at fibercloud.com> wrote:
>  I?ve seen this before. The system thinks the storage system us up and
> running and then attempts to utilize it.
>
> The way I got around it was to put a delay in the startup of the gluster
> Node on the interface that the clients use to communicate.
>
>
>
> I use a bonded link, I then add a LINKDELAY to the interface to get the
> underlying system up and running before the network comes up. This then
> causes Network dependent features to wait for the network to finish.
>
> It adds about 10seconds to the startup time, in our environment it works
> well, you may not need as long of a delay.
>
>
>
> CentOS
>
> root at gls1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
>
>
>
> DEVICE=bond0
>
> ONBOOT=yes
>
> BOOTPROTO=static
>
> USERCTL=no
>
> NETMASK=255.255.248.0
>
> IPADDR=10.10.1.17
>
> MTU=9000
>
> IPV6INIT=no
>
> IPV6_AUTOCONF=no
>
> NETWORKING_IPV6=no
>
> NM_CONTROLLED=no
>
> LINKDELAY=10
>
> NAME="System Storage Bond0"
>
>
>
>
>
>
>
>
>
> Hi Michal,
>
>
>
> The Storage domain is up and running and mounted on all the host
> nodes...as i updated before that it was working perfectly before but just
> after reboot can not make the VM poweron...
>
>
>
> [image: Inline image 1]
>
>
>
> [image: Inline image 2]
>
>
>
> [root at cpu01 log]# gluster volume info
>
>
>
> Volume Name: ds01
>
> Type: Distributed-Replicate
>
> Volume ID: 369d3fdc-c8eb-46b7-a33e-0a49f2451ff6
>
> Status: Started
>
> Number of Bricks: 48 x 2 = 96
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: cpu01:/bricks/1/vol1
>
> Brick2: cpu02:/bricks/1/vol1
>
> Brick3: cpu03:/bricks/1/vol1
>
> Brick4: cpu04:/bricks/1/vol1
>
> Brick5: cpu01:/bricks/2/vol1
>
> Brick6: cpu02:/bricks/2/vol1
>
> Brick7: cpu03:/bricks/2/vol1
>
> Brick8: cpu04:/bricks/2/vol1
>
> Brick9: cpu01:/bricks/3/vol1
>
> Brick10: cpu02:/bricks/3/vol1
>
> Brick11: cpu03:/bricks/3/vol1
>
> Brick12: cpu04:/bricks/3/vol1
>
> Brick13: cpu01:/bricks/4/vol1
>
> Brick14: cpu02:/bricks/4/vol1
>
> Brick15: cpu03:/bricks/4/vol1
>
> Brick16: cpu04:/bricks/4/vol1
>
> Brick17: cpu01:/bricks/5/vol1
>
> Brick18: cpu02:/bricks/5/vol1
>
> Brick19: cpu03:/bricks/5/vol1
>
> Brick20: cpu04:/bricks/5/vol1
>
> Brick21: cpu01:/bricks/6/vol1
>
> Brick22: cpu02:/bricks/6/vol1
>
> Brick23: cpu03:/bricks/6/vol1
>
> Brick24: cpu04:/bricks/6/vol1
>
> Brick25: cpu01:/bricks/7/vol1
>
> Brick26: cpu02:/bricks/7/vol1
>
> Brick27: cpu03:/bricks/7/vol1
>
> Brick28: cpu04:/bricks/7/vol1
>
> Brick29: cpu01:/bricks/8/vol1
>
> Brick30: cpu02:/bricks/8/vol1
>
> Brick31: cpu03:/bricks/8/vol1
>
> Brick32: cpu04:/bricks/8/vol1
>
> Brick33: cpu01:/bricks/9/vol1
>
> Brick34: cpu02:/bricks/9/vol1
>
> Brick35: cpu03:/bricks/9/vol1
>
> Brick36: cpu04:/bricks/9/vol1
>
> Brick37: cpu01:/bricks/10/vol1
>
> Brick38: cpu02:/bricks/10/vol1
>
> Brick39: cpu03:/bricks/10/vol1
>
> Brick40: cpu04:/bricks/10/vol1
>
> Brick41: cpu01:/bricks/11/vol1
>
> Brick42: cpu02:/bricks/11/vol1
>
> Brick43: cpu03:/bricks/11/vol1
>
> Brick44: cpu04:/bricks/11/vol1
>
> Brick45: cpu01:/bricks/12/vol1
>
> Brick46: cpu02:/bricks/12/vol1
>
> Brick47: cpu03:/bricks/12/vol1
>
> Brick48: cpu04:/bricks/12/vol1
>
> Brick49: cpu01:/bricks/13/vol1
>
> Brick50: cpu02:/bricks/13/vol1
>
> Brick51: cpu03:/bricks/13/vol1
>
> Brick52: cpu04:/bricks/13/vol1
>
> Brick53: cpu01:/bricks/14/vol1
>
> Brick54: cpu02:/bricks/14/vol1
>
> Brick55: cpu03:/bricks/14/vol1
>
> Brick56: cpu04:/bricks/14/vol1
>
> Brick57: cpu01:/bricks/15/vol1
>
> Brick58: cpu02:/bricks/15/vol1
>
> Brick59: cpu03:/bricks/15/vol1
>
> Brick60: cpu04:/bricks/15/vol1
>
> Brick61: cpu01:/bricks/16/vol1
>
> Brick62: cpu02:/bricks/16/vol1
>
> Brick63: cpu03:/bricks/16/vol1
>
> Brick64: cpu04:/bricks/16/vol1
>
> Brick65: cpu01:/bricks/17/vol1
>
> Brick66: cpu02:/bricks/17/vol1
>
> Brick67: cpu03:/bricks/17/vol1
>
> Brick68: cpu04:/bricks/17/vol1
>
> Brick69: cpu01:/bricks/18/vol1
>
> Brick70: cpu02:/bricks/18/vol1
>
> Brick71: cpu03:/bricks/18/vol1
>
> Brick72: cpu04:/bricks/18/vol1
>
> Brick73: cpu01:/bricks/19/vol1
>
> Brick74: cpu02:/bricks/19/vol1
>
> Brick75: cpu03:/bricks/19/vol1
>
> Brick76: cpu04:/bricks/19/vol1
>
> Brick77: cpu01:/bricks/20/vol1
>
> Brick78: cpu02:/bricks/20/vol1
>
> Brick79: cpu03:/bricks/20/vol1
>
> Brick80: cpu04:/bricks/20/vol1
>
> Brick81: cpu01:/bricks/21/vol1
>
> Brick82: cpu02:/bricks/21/vol1
>
> Brick83: cpu03:/bricks/21/vol1
>
> Brick84: cpu04:/bricks/21/vol1
>
> Brick85: cpu01:/bricks/22/vol1
>
> Brick86: cpu02:/bricks/22/vol1
>
> Brick87: cpu03:/bricks/22/vol1
>
> Brick88: cpu04:/bricks/22/vol1
>
> Brick89: cpu01:/bricks/23/vol1
>
> Brick90: cpu02:/bricks/23/vol1
>
> Brick91: cpu03:/bricks/23/vol1
>
> Brick92: cpu04:/bricks/23/vol1
>
> Brick93: cpu01:/bricks/24/vol1
>
> Brick94: cpu02:/bricks/24/vol1
>
> Brick95: cpu03:/bricks/24/vol1
>
> Brick96: cpu04:/bricks/24/vol1
>
> Options Reconfigured:
>
> diagnostics.count-fop-hits: on
>
> diagnostics.latency-measurement: on
>
> nfs.disable: on
>
> user.cifs: enable
>
> auth.allow: 10.10.0.*
>
> performance.quick-read: off
>
> performance.read-ahead: off
>
> performance.io-cache: off
>
> performance.stat-prefetch: off
>
> cluster.eager-lock: enable
>
> network.remote-dio: enable
>
> cluster.quorum-type: auto
>
> cluster.server-quorum-type: server
>
> storage.owner-uid: 36
>
> storage.owner-gid: 36
>
> server.allow-insecure: on
>
> network.ping-timeout: 100
>
> [root at cpu01 log]#
>
>
>
> -----------------------------------------
>
>
>
> [root at cpu01 log]# gluster volume status
>
> Status of volume: ds01
>
> Gluster process                                         Port    Online  Pid
>
>
>
------------------------------------------------------------------------------
>
> Brick cpu01:/bricks/1/vol1                              49152   Y
> 33474
>
> Brick cpu02:/bricks/1/vol1                              49152   Y
> 40717
>
> Brick cpu03:/bricks/1/vol1                              49152   Y
> 18080
>
> Brick cpu04:/bricks/1/vol1                              49152   Y
> 40447
>
> Brick cpu01:/bricks/2/vol1                              49153   Y
> 33481
>
> Brick cpu02:/bricks/2/vol1                              49153   Y
> 40724
>
> Brick cpu03:/bricks/2/vol1                              49153   Y
> 18086
>
> Brick cpu04:/bricks/2/vol1                              49153   Y
> 40453
>
> Brick cpu01:/bricks/3/vol1                              49154   Y
> 33489
>
> Brick cpu02:/bricks/3/vol1                              49154   Y
> 40731
>
> Brick cpu03:/bricks/3/vol1                              49154   Y
> 18097
>
> Brick cpu04:/bricks/3/vol1                              49154   Y
> 40460
>
> Brick cpu01:/bricks/4/vol1                              49155   Y
> 33495
>
> Brick cpu02:/bricks/4/vol1                              49155   Y
> 40738
>
> Brick cpu03:/bricks/4/vol1                              49155   Y
> 18103
>
> Brick cpu04:/bricks/4/vol1                              49155   Y
> 40468
>
> Brick cpu01:/bricks/5/vol1                              49156   Y
> 33502
>
> Brick cpu02:/bricks/5/vol1                              49156   Y
> 40745
>
> Brick cpu03:/bricks/5/vol1                              49156   Y
> 18110
>
> Brick cpu04:/bricks/5/vol1                              49156   Y
> 40474
>
> Brick cpu01:/bricks/6/vol1                              49157   Y
> 33509
>
> Brick cpu02:/bricks/6/vol1                              49157   Y
> 40752
>
> Brick cpu03:/bricks/6/vol1                              49157   Y
> 18116
>
> Brick cpu04:/bricks/6/vol1                              49157   Y
> 40481
>
> Brick cpu01:/bricks/7/vol1                              49158   Y
> 33516
>
> Brick cpu02:/bricks/7/vol1                              49158   Y
> 40759
>
> Brick cpu03:/bricks/7/vol1                              49158   Y
> 18122
>
> Brick cpu04:/bricks/7/vol1                              49158   Y
> 40488
>
> Brick cpu01:/bricks/8/vol1                              49159   Y
> 33525
>
> Brick cpu02:/bricks/8/vol1                              49159   Y
> 40766
>
> Brick cpu03:/bricks/8/vol1                              49159   Y
> 18130
>
> Brick cpu04:/bricks/8/vol1                              49159   Y
> 40495
>
> Brick cpu01:/bricks/9/vol1                              49160   Y
> 33530
>
> Brick cpu02:/bricks/9/vol1                              49160   Y
> 40773
>
> Brick cpu03:/bricks/9/vol1                              49160   Y
> 18137
>
> Brick cpu04:/bricks/9/vol1                              49160   Y
> 40502
>
> Brick cpu01:/bricks/10/vol1                             49161   Y
> 33538
>
> Brick cpu02:/bricks/10/vol1                             49161   Y
> 40780
>
> Brick cpu03:/bricks/10/vol1                             49161   Y
> 18143
>
> Brick cpu04:/bricks/10/vol1                             49161   Y
> 40509
>
> Brick cpu01:/bricks/11/vol1                             49162   Y
> 33544
>
> Brick cpu02:/bricks/11/vol1                             49162   Y
> 40787
>
> Brick cpu03:/bricks/11/vol1                             49162   Y
> 18150
>
> Brick cpu04:/bricks/11/vol1                             49162   Y
> 40516
>
> Brick cpu01:/bricks/12/vol1                             49163   Y
> 33551
>
> Brick cpu02:/bricks/12/vol1                             49163   Y
> 40794
>
> Brick cpu03:/bricks/12/vol1                             49163   Y
> 18157
>
> Brick cpu04:/bricks/12/vol1                             49163   Y
> 40692
>
> Brick cpu01:/bricks/13/vol1                             49164   Y
> 33558
>
> Brick cpu02:/bricks/13/vol1                             49164   Y
> 40801
>
> Brick cpu03:/bricks/13/vol1                             49164   Y
> 18165
>
> Brick cpu04:/bricks/13/vol1                             49164   Y
> 40700
>
> Brick cpu01:/bricks/14/vol1                             49165   Y
> 33566
>
> Brick cpu02:/bricks/14/vol1                             49165   Y
> 40809
>
> Brick cpu03:/bricks/14/vol1                             49165   Y
> 18172
>
> Brick cpu04:/bricks/14/vol1                             49165   Y
> 40706
>
> Brick cpu01:/bricks/15/vol1                             49166   Y
> 33572
>
> Brick cpu02:/bricks/15/vol1                             49166   Y
> 40815
>
> Brick cpu03:/bricks/15/vol1                             49166   Y
> 18179
>
> Brick cpu04:/bricks/15/vol1                             49166   Y
> 40714
>
> Brick cpu01:/bricks/16/vol1                             49167   Y
> 33579
>
> Brick cpu02:/bricks/16/vol1                             49167   Y
> 40822
>
> Brick cpu03:/bricks/16/vol1                             49167   Y
> 18185
>
> Brick cpu04:/bricks/16/vol1                             49167   Y
> 40722
>
> Brick cpu01:/bricks/17/vol1                             49168   Y
> 33586
>
> Brick cpu02:/bricks/17/vol1                             49168   Y
> 40829
>
> Brick cpu03:/bricks/17/vol1                             49168   Y
> 18192
>
> Brick cpu04:/bricks/17/vol1                             49168   Y
> 40727
>
> Brick cpu01:/bricks/18/vol1                             49169   Y
> 33593
>
> Brick cpu02:/bricks/18/vol1                             49169   Y
> 40836
>
> Brick cpu03:/bricks/18/vol1                             49169   Y
> 18201
>
> Brick cpu04:/bricks/18/vol1                             49169   Y
> 40735
>
> Brick cpu01:/bricks/19/vol1                             49170   Y
> 33600
>
> Brick cpu02:/bricks/19/vol1                             49170   Y
> 40843
>
> Brick cpu03:/bricks/19/vol1                             49170   Y
> 18207
>
> Brick cpu04:/bricks/19/vol1                             49170   Y
> 40741
>
> Brick cpu01:/bricks/20/vol1                             49171   Y
> 33608
>
> Brick cpu02:/bricks/20/vol1                             49171   Y
> 40850
>
> Brick cpu03:/bricks/20/vol1                             49171   Y
> 18214
>
> Brick cpu04:/bricks/20/vol1                             49171   Y
> 40748
>
> Brick cpu01:/bricks/21/vol1                             49172   Y
> 33614
>
> Brick cpu02:/bricks/21/vol1                             49172   Y
> 40858
>
> Brick cpu03:/bricks/21/vol1                             49172   Y
> 18222
>
> Brick cpu04:/bricks/21/vol1                             49172   Y
> 40756
>
> Brick cpu01:/bricks/22/vol1                             49173   Y
> 33621
>
> Brick cpu02:/bricks/22/vol1                             49173   Y
> 40864
>
> Brick cpu03:/bricks/22/vol1                             49173   Y
> 18227
>
> Brick cpu04:/bricks/22/vol1                             49173   Y
> 40762
>
> Brick cpu01:/bricks/23/vol1                             49174   Y
> 33626
>
> Brick cpu02:/bricks/23/vol1                             49174   Y
> 40869
>
> Brick cpu03:/bricks/23/vol1                             49174   Y
> 18234
>
> Brick cpu04:/bricks/23/vol1                             49174   Y
> 40769
>
> Brick cpu01:/bricks/24/vol1                             49175   Y
> 33631
>
> Brick cpu02:/bricks/24/vol1                             49175   Y
> 40874
>
> Brick cpu03:/bricks/24/vol1                             49175   Y
> 18239
>
> Brick cpu04:/bricks/24/vol1                             49175   Y
> 40774
>
> Self-heal Daemon on localhost                           N/A     Y
> 33361
>
> Self-heal Daemon on cpu05                               N/A     Y
> 2353
>
> Self-heal Daemon on cpu04                               N/A     Y
> 40786
>
> Self-heal Daemon on cpu02                               N/A     Y
> 32442
>
> Self-heal Daemon on cpu03                               N/A     Y
> 18664
>
>
>
> Task Status of Volume ds01
>
>
>
------------------------------------------------------------------------------
>
> Task                 : Rebalance
>
> ID                   : 5db24b30-4b9f-4b65-8910-a7a0a6d327a4
>
> Status               : completed
>
>
>
> [root at cpu01 log]#
>
>
>
> [root at cpu01 log]# gluster pool list
>
> UUID                                    Hostname        State
>
> 626c9360-8c09-480f-9707-116e67cc38e6    cpu02           Connected
>
> dc475d62-b035-4ee6-9006-6f03bf68bf24    cpu05           Connected
>
> 41b5b2ff-3671-47b4-b477-227a107e718d    cpu03           Connected
>
> c0afe114-dfa7-407d-bad7-5a3f97a6f3fc    cpu04           Connected
>
> 9b61b0a5-be78-4ac2-b6c0-2db588da5c35    localhost       Connected
>
> [root at cpu01 log]#
>
>
>
> [image: Inline image 3]
>
>
>
> Thanks,
>
> Punit
>
>
>
> On Thu, Mar 19, 2015 at 2:53 PM, Michal Skrivanek <
> michal.skrivanek at redhat.com> wrote:
>
>
> On Mar 19, 2015, at 03:18 , Punit Dambiwal <hypunit at gmail.com>
wrote:
>
> > Hi All,
> >
> > Is there any one have any idea about this problem...it seems it's
bug
> either in Ovirt or Glusterfs...that's why no one has the idea about
> it....please correct me if i am wrong?.
>
> Hi,
> as I said, storage access times out; so it seems to me as a gluster setup
> problem, the storage domain you have your VMs on is not working?
>
> Thanks,
> michal
>
>
> >
> > Thanks,
> > Punit
> >
> > On Wed, Mar 18, 2015 at 5:05 PM, Punit Dambiwal <hypunit at
gmail.com>
> wrote:
> > Hi Michal,
> >
> > Would you mind to let me know the possible messedup things...i will
> check and try to resolve it....still i am communicating gluster community
> to resolve this issue...
> >
> > But in the ovirt....gluster setup is quite straight....so how come it
> will be messedup with reboot ?? if it can be messedup with reboot then it
> seems not good and stable technology for the production storage....
> >
> > Thanks,
> > Punit
> >
> > On Wed, Mar 18, 2015 at 3:51 PM, Michal Skrivanek <
> michal.skrivanek at redhat.com> wrote:
> >
> > On Mar 18, 2015, at 03:33 , Punit Dambiwal <hypunit at
gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Is there any one from community can help me to solve this
issue...??
> > >
> > > Thanks,
> > > Punit
> > >
> > > On Tue, Mar 17, 2015 at 12:52 PM, Punit Dambiwal <hypunit at
gmail.com>
> wrote:
> > > Hi,
> > >
> > > I am facing one strange issue with ovirt/glusterfs....still
didn't
> find this issue is related with glusterfs or Ovirt....
> > >
> > > Ovirt :- 3.5.1
> > > Glusterfs :- 3.6.1
> > > Host :- 4 Hosts (Compute+ Storage)...each server has 24 bricks
> > > Guest VM :- more then 100
> > >
> > > Issue :- When i deploy this cluster first time..it work well for
> me(all the guest VM created and running successfully)....but suddenly one
> day my one of the host node rebooted and none of the VM can boot up
> now...and failed with the following error "Bad Volume
Specification"
> > >
> > > VMId :- d877313c18d9783ca09b62acf5588048
> > >
> > > VDSM Logs :- http://ur1.ca/jxabi
> >
> > you've got timeouts while accessing storage?so I guess something
got
> messed up on reboot, it may also be just a gluster misconfiguration?
> >
> > > Engine Logs :- http://ur1.ca/jxabv
> > >
> > > ------------------------
> > > [root at cpu01 ~]# vdsClient -s 0 getVolumeInfo
> e732a82f-bae9-4368-8b98-dedc1c3814de 00000002-0002-0002-0002-000000000145
> 6d123509-6867-45cf-83a2-6d679b77d3c5 9030bb43-6bc9-462f-a1b9-f6d5a02fb180
> > >         status = OK
> > >         domain = e732a82f-bae9-4368-8b98-dedc1c3814de
> > >         capacity = 21474836480
> > >         voltype = LEAF
> > >         description > > >         parent =
00000000-0000-0000-0000-000000000000
> > >         format = RAW
> > >         image = 6d123509-6867-45cf-83a2-6d679b77d3c5
> > >         uuid = 9030bb43-6bc9-462f-a1b9-f6d5a02fb180
> > >         disktype = 2
> > >         legality = LEGAL
> > >         mtime = 0
> > >         apparentsize = 21474836480
> > >         truesize = 4562972672
> > >         type = SPARSE
> > >         children = []
> > >         pool > > >         ctime = 1422676305
> > > ---------------------
> > >
> > > I opened same thread earlier but didn't get any perfect
answers to
> solve this issue..so i reopen it...
> > >
> > > https://www.mail-archive.com/users at ovirt.org/msg25011.html
> > >
> > > Thanks,
> > > Punit
> > >
> > >
> > >
> >
> >
> >
>
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150323/854652af/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 10223 bytes
Desc: not available
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150323/854652af/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 11779 bytes
Desc: not available
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150323/854652af/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 22346 bytes
Desc: not available
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150323/854652af/attachment-0002.png>

Punit Dambiwal

2015-Mar-25 09:39 UTC

head link

[Gluster-users] VM failed to start | Bad volume specification

Hi All,

With the help of gluster community and ovirt-china community...my issue got
resolved...

The main root cause was the following :-

1. the glob operation takes quite a long time, longer than the ioprocess
default 60s..
2. python-ioprocess updated which makes a single change of configuration
file doesn't work properly, only because this we should hack the code
manually...

 Solution (Need to do on all the hosts) :-

 1. Add the the ioprocess timeout value in the /etc/vdsm/vdsm.conf file as
 :-

------------
[irs]
process_pool_timeout = 180
-------------

2. Check /usr/share/vdsm/storage/outOfProcess.py, line 71 and see whether
there is  still "IOProcess(DEFAULT_TIMEOUT)" in it,if yes...then
changing
the configuration file takes no effect because now timeout is the third
parameter not the second of IOProcess.__init__().

3. Change IOProcess(DEFAULT_TIMEOUT) to
IOProcess(timeout=DEFAULT_TIMEOUT) and remove the
 /usr/share/vdsm/storage/outOfProcess.pyc file and restart vdsm and
supervdsm service on all hosts....

Thanks,
Punit Dambiwal


On Mon, Mar 23, 2015 at 9:18 AM, Punit Dambiwal <hypunit at gmail.com>
wrote:
> Hi All,
>
> Still i am facing the same issue...please help me to overcome this issue...
>
> Thanks,
> punit
>
> On Fri, Mar 20, 2015 at 12:22 AM, Thomas Holkenbrink <
> thomas.holkenbrink at fibercloud.com> wrote:
>
>>  I?ve seen this before. The system thinks the storage system us up and
>> running and then attempts to utilize it.
>>
>> The way I got around it was to put a delay in the startup of the
gluster
>> Node on the interface that the clients use to communicate.
>>
>>
>>
>> I use a bonded link, I then add a LINKDELAY to the interface to get the
>> underlying system up and running before the network comes up. This then
>> causes Network dependent features to wait for the network to finish.
>>
>> It adds about 10seconds to the startup time, in our environment it
works
>> well, you may not need as long of a delay.
>>
>>
>>
>> CentOS
>>
>> root at gls1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
>>
>>
>>
>> DEVICE=bond0
>>
>> ONBOOT=yes
>>
>> BOOTPROTO=static
>>
>> USERCTL=no
>>
>> NETMASK=255.255.248.0
>>
>> IPADDR=10.10.1.17
>>
>> MTU=9000
>>
>> IPV6INIT=no
>>
>> IPV6_AUTOCONF=no
>>
>> NETWORKING_IPV6=no
>>
>> NM_CONTROLLED=no
>>
>> LINKDELAY=10
>>
>> NAME="System Storage Bond0"
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Hi Michal,
>>
>>
>>
>> The Storage domain is up and running and mounted on all the host
>> nodes...as i updated before that it was working perfectly before but
just
>> after reboot can not make the VM poweron...
>>
>>
>>
>> [image: Inline image 1]
>>
>>
>>
>> [image: Inline image 2]
>>
>>
>>
>> [root at cpu01 log]# gluster volume info
>>
>>
>>
>> Volume Name: ds01
>>
>> Type: Distributed-Replicate
>>
>> Volume ID: 369d3fdc-c8eb-46b7-a33e-0a49f2451ff6
>>
>> Status: Started
>>
>> Number of Bricks: 48 x 2 = 96
>>
>> Transport-type: tcp
>>
>> Bricks:
>>
>> Brick1: cpu01:/bricks/1/vol1
>>
>> Brick2: cpu02:/bricks/1/vol1
>>
>> Brick3: cpu03:/bricks/1/vol1
>>
>> Brick4: cpu04:/bricks/1/vol1
>>
>> Brick5: cpu01:/bricks/2/vol1
>>
>> Brick6: cpu02:/bricks/2/vol1
>>
>> Brick7: cpu03:/bricks/2/vol1
>>
>> Brick8: cpu04:/bricks/2/vol1
>>
>> Brick9: cpu01:/bricks/3/vol1
>>
>> Brick10: cpu02:/bricks/3/vol1
>>
>> Brick11: cpu03:/bricks/3/vol1
>>
>> Brick12: cpu04:/bricks/3/vol1
>>
>> Brick13: cpu01:/bricks/4/vol1
>>
>> Brick14: cpu02:/bricks/4/vol1
>>
>> Brick15: cpu03:/bricks/4/vol1
>>
>> Brick16: cpu04:/bricks/4/vol1
>>
>> Brick17: cpu01:/bricks/5/vol1
>>
>> Brick18: cpu02:/bricks/5/vol1
>>
>> Brick19: cpu03:/bricks/5/vol1
>>
>> Brick20: cpu04:/bricks/5/vol1
>>
>> Brick21: cpu01:/bricks/6/vol1
>>
>> Brick22: cpu02:/bricks/6/vol1
>>
>> Brick23: cpu03:/bricks/6/vol1
>>
>> Brick24: cpu04:/bricks/6/vol1
>>
>> Brick25: cpu01:/bricks/7/vol1
>>
>> Brick26: cpu02:/bricks/7/vol1
>>
>> Brick27: cpu03:/bricks/7/vol1
>>
>> Brick28: cpu04:/bricks/7/vol1
>>
>> Brick29: cpu01:/bricks/8/vol1
>>
>> Brick30: cpu02:/bricks/8/vol1
>>
>> Brick31: cpu03:/bricks/8/vol1
>>
>> Brick32: cpu04:/bricks/8/vol1
>>
>> Brick33: cpu01:/bricks/9/vol1
>>
>> Brick34: cpu02:/bricks/9/vol1
>>
>> Brick35: cpu03:/bricks/9/vol1
>>
>> Brick36: cpu04:/bricks/9/vol1
>>
>> Brick37: cpu01:/bricks/10/vol1
>>
>> Brick38: cpu02:/bricks/10/vol1
>>
>> Brick39: cpu03:/bricks/10/vol1
>>
>> Brick40: cpu04:/bricks/10/vol1
>>
>> Brick41: cpu01:/bricks/11/vol1
>>
>> Brick42: cpu02:/bricks/11/vol1
>>
>> Brick43: cpu03:/bricks/11/vol1
>>
>> Brick44: cpu04:/bricks/11/vol1
>>
>> Brick45: cpu01:/bricks/12/vol1
>>
>> Brick46: cpu02:/bricks/12/vol1
>>
>> Brick47: cpu03:/bricks/12/vol1
>>
>> Brick48: cpu04:/bricks/12/vol1
>>
>> Brick49: cpu01:/bricks/13/vol1
>>
>> Brick50: cpu02:/bricks/13/vol1
>>
>> Brick51: cpu03:/bricks/13/vol1
>>
>> Brick52: cpu04:/bricks/13/vol1
>>
>> Brick53: cpu01:/bricks/14/vol1
>>
>> Brick54: cpu02:/bricks/14/vol1
>>
>> Brick55: cpu03:/bricks/14/vol1
>>
>> Brick56: cpu04:/bricks/14/vol1
>>
>> Brick57: cpu01:/bricks/15/vol1
>>
>> Brick58: cpu02:/bricks/15/vol1
>>
>> Brick59: cpu03:/bricks/15/vol1
>>
>> Brick60: cpu04:/bricks/15/vol1
>>
>> Brick61: cpu01:/bricks/16/vol1
>>
>> Brick62: cpu02:/bricks/16/vol1
>>
>> Brick63: cpu03:/bricks/16/vol1
>>
>> Brick64: cpu04:/bricks/16/vol1
>>
>> Brick65: cpu01:/bricks/17/vol1
>>
>> Brick66: cpu02:/bricks/17/vol1
>>
>> Brick67: cpu03:/bricks/17/vol1
>>
>> Brick68: cpu04:/bricks/17/vol1
>>
>> Brick69: cpu01:/bricks/18/vol1
>>
>> Brick70: cpu02:/bricks/18/vol1
>>
>> Brick71: cpu03:/bricks/18/vol1
>>
>> Brick72: cpu04:/bricks/18/vol1
>>
>> Brick73: cpu01:/bricks/19/vol1
>>
>> Brick74: cpu02:/bricks/19/vol1
>>
>> Brick75: cpu03:/bricks/19/vol1
>>
>> Brick76: cpu04:/bricks/19/vol1
>>
>> Brick77: cpu01:/bricks/20/vol1
>>
>> Brick78: cpu02:/bricks/20/vol1
>>
>> Brick79: cpu03:/bricks/20/vol1
>>
>> Brick80: cpu04:/bricks/20/vol1
>>
>> Brick81: cpu01:/bricks/21/vol1
>>
>> Brick82: cpu02:/bricks/21/vol1
>>
>> Brick83: cpu03:/bricks/21/vol1
>>
>> Brick84: cpu04:/bricks/21/vol1
>>
>> Brick85: cpu01:/bricks/22/vol1
>>
>> Brick86: cpu02:/bricks/22/vol1
>>
>> Brick87: cpu03:/bricks/22/vol1
>>
>> Brick88: cpu04:/bricks/22/vol1
>>
>> Brick89: cpu01:/bricks/23/vol1
>>
>> Brick90: cpu02:/bricks/23/vol1
>>
>> Brick91: cpu03:/bricks/23/vol1
>>
>> Brick92: cpu04:/bricks/23/vol1
>>
>> Brick93: cpu01:/bricks/24/vol1
>>
>> Brick94: cpu02:/bricks/24/vol1
>>
>> Brick95: cpu03:/bricks/24/vol1
>>
>> Brick96: cpu04:/bricks/24/vol1
>>
>> Options Reconfigured:
>>
>> diagnostics.count-fop-hits: on
>>
>> diagnostics.latency-measurement: on
>>
>> nfs.disable: on
>>
>> user.cifs: enable
>>
>> auth.allow: 10.10.0.*
>>
>> performance.quick-read: off
>>
>> performance.read-ahead: off
>>
>> performance.io-cache: off
>>
>> performance.stat-prefetch: off
>>
>> cluster.eager-lock: enable
>>
>> network.remote-dio: enable
>>
>> cluster.quorum-type: auto
>>
>> cluster.server-quorum-type: server
>>
>> storage.owner-uid: 36
>>
>> storage.owner-gid: 36
>>
>> server.allow-insecure: on
>>
>> network.ping-timeout: 100
>>
>> [root at cpu01 log]#
>>
>>
>>
>> -----------------------------------------
>>
>>
>>
>> [root at cpu01 log]# gluster volume status
>>
>> Status of volume: ds01
>>
>> Gluster process                                         Port    Online
>>  Pid
>>
>>
>>
------------------------------------------------------------------------------
>>
>> Brick cpu01:/bricks/1/vol1                              49152   Y
>> 33474
>>
>> Brick cpu02:/bricks/1/vol1                              49152   Y
>> 40717
>>
>> Brick cpu03:/bricks/1/vol1                              49152   Y
>> 18080
>>
>> Brick cpu04:/bricks/1/vol1                              49152   Y
>> 40447
>>
>> Brick cpu01:/bricks/2/vol1                              49153   Y
>> 33481
>>
>> Brick cpu02:/bricks/2/vol1                              49153   Y
>> 40724
>>
>> Brick cpu03:/bricks/2/vol1                              49153   Y
>> 18086
>>
>> Brick cpu04:/bricks/2/vol1                              49153   Y
>> 40453
>>
>> Brick cpu01:/bricks/3/vol1                              49154   Y
>> 33489
>>
>> Brick cpu02:/bricks/3/vol1                              49154   Y
>> 40731
>>
>> Brick cpu03:/bricks/3/vol1                              49154   Y
>> 18097
>>
>> Brick cpu04:/bricks/3/vol1                              49154   Y
>> 40460
>>
>> Brick cpu01:/bricks/4/vol1                              49155   Y
>> 33495
>>
>> Brick cpu02:/bricks/4/vol1                              49155   Y
>> 40738
>>
>> Brick cpu03:/bricks/4/vol1                              49155   Y
>> 18103
>>
>> Brick cpu04:/bricks/4/vol1                              49155   Y
>> 40468
>>
>> Brick cpu01:/bricks/5/vol1                              49156   Y
>> 33502
>>
>> Brick cpu02:/bricks/5/vol1                              49156   Y
>> 40745
>>
>> Brick cpu03:/bricks/5/vol1                              49156   Y
>> 18110
>>
>> Brick cpu04:/bricks/5/vol1                              49156   Y
>> 40474
>>
>> Brick cpu01:/bricks/6/vol1                              49157   Y
>> 33509
>>
>> Brick cpu02:/bricks/6/vol1                              49157   Y
>> 40752
>>
>> Brick cpu03:/bricks/6/vol1                              49157   Y
>> 18116
>>
>> Brick cpu04:/bricks/6/vol1                              49157   Y
>> 40481
>>
>> Brick cpu01:/bricks/7/vol1                              49158   Y
>> 33516
>>
>> Brick cpu02:/bricks/7/vol1                              49158   Y
>> 40759
>>
>> Brick cpu03:/bricks/7/vol1                              49158   Y
>> 18122
>>
>> Brick cpu04:/bricks/7/vol1                              49158   Y
>> 40488
>>
>> Brick cpu01:/bricks/8/vol1                              49159   Y
>> 33525
>>
>> Brick cpu02:/bricks/8/vol1                              49159   Y
>> 40766
>>
>> Brick cpu03:/bricks/8/vol1                              49159   Y
>> 18130
>>
>> Brick cpu04:/bricks/8/vol1                              49159   Y
>> 40495
>>
>> Brick cpu01:/bricks/9/vol1                              49160   Y
>> 33530
>>
>> Brick cpu02:/bricks/9/vol1                              49160   Y
>> 40773
>>
>> Brick cpu03:/bricks/9/vol1                              49160   Y
>> 18137
>>
>> Brick cpu04:/bricks/9/vol1                              49160   Y
>> 40502
>>
>> Brick cpu01:/bricks/10/vol1                             49161   Y
>> 33538
>>
>> Brick cpu02:/bricks/10/vol1                             49161   Y
>> 40780
>>
>> Brick cpu03:/bricks/10/vol1                             49161   Y
>> 18143
>>
>> Brick cpu04:/bricks/10/vol1                             49161   Y
>> 40509
>>
>> Brick cpu01:/bricks/11/vol1                             49162   Y
>> 33544
>>
>> Brick cpu02:/bricks/11/vol1                             49162   Y
>> 40787
>>
>> Brick cpu03:/bricks/11/vol1                             49162   Y
>> 18150
>>
>> Brick cpu04:/bricks/11/vol1                             49162   Y
>> 40516
>>
>> Brick cpu01:/bricks/12/vol1                             49163   Y
>> 33551
>>
>> Brick cpu02:/bricks/12/vol1                             49163   Y
>> 40794
>>
>> Brick cpu03:/bricks/12/vol1                             49163   Y
>> 18157
>>
>> Brick cpu04:/bricks/12/vol1                             49163   Y
>> 40692
>>
>> Brick cpu01:/bricks/13/vol1                             49164   Y
>> 33558
>>
>> Brick cpu02:/bricks/13/vol1                             49164   Y
>> 40801
>>
>> Brick cpu03:/bricks/13/vol1                             49164   Y
>> 18165
>>
>> Brick cpu04:/bricks/13/vol1                             49164   Y
>> 40700
>>
>> Brick cpu01:/bricks/14/vol1                             49165   Y
>> 33566
>>
>> Brick cpu02:/bricks/14/vol1                             49165   Y
>> 40809
>>
>> Brick cpu03:/bricks/14/vol1                             49165   Y
>> 18172
>>
>> Brick cpu04:/bricks/14/vol1                             49165   Y
>> 40706
>>
>> Brick cpu01:/bricks/15/vol1                             49166   Y
>> 33572
>>
>> Brick cpu02:/bricks/15/vol1                             49166   Y
>> 40815
>>
>> Brick cpu03:/bricks/15/vol1                             49166   Y
>> 18179
>>
>> Brick cpu04:/bricks/15/vol1                             49166   Y
>> 40714
>>
>> Brick cpu01:/bricks/16/vol1                             49167   Y
>> 33579
>>
>> Brick cpu02:/bricks/16/vol1                             49167   Y
>> 40822
>>
>> Brick cpu03:/bricks/16/vol1                             49167   Y
>> 18185
>>
>> Brick cpu04:/bricks/16/vol1                             49167   Y
>> 40722
>>
>> Brick cpu01:/bricks/17/vol1                             49168   Y
>> 33586
>>
>> Brick cpu02:/bricks/17/vol1                             49168   Y
>> 40829
>>
>> Brick cpu03:/bricks/17/vol1                             49168   Y
>> 18192
>>
>> Brick cpu04:/bricks/17/vol1                             49168   Y
>> 40727
>>
>> Brick cpu01:/bricks/18/vol1                             49169   Y
>> 33593
>>
>> Brick cpu02:/bricks/18/vol1                             49169   Y
>> 40836
>>
>> Brick cpu03:/bricks/18/vol1                             49169   Y
>> 18201
>>
>> Brick cpu04:/bricks/18/vol1                             49169   Y
>> 40735
>>
>> Brick cpu01:/bricks/19/vol1                             49170   Y
>> 33600
>>
>> Brick cpu02:/bricks/19/vol1                             49170   Y
>> 40843
>>
>> Brick cpu03:/bricks/19/vol1                             49170   Y
>> 18207
>>
>> Brick cpu04:/bricks/19/vol1                             49170   Y
>> 40741
>>
>> Brick cpu01:/bricks/20/vol1                             49171   Y
>> 33608
>>
>> Brick cpu02:/bricks/20/vol1                             49171   Y
>> 40850
>>
>> Brick cpu03:/bricks/20/vol1                             49171   Y
>> 18214
>>
>> Brick cpu04:/bricks/20/vol1                             49171   Y
>> 40748
>>
>> Brick cpu01:/bricks/21/vol1                             49172   Y
>> 33614
>>
>> Brick cpu02:/bricks/21/vol1                             49172   Y
>> 40858
>>
>> Brick cpu03:/bricks/21/vol1                             49172   Y
>> 18222
>>
>> Brick cpu04:/bricks/21/vol1                             49172   Y
>> 40756
>>
>> Brick cpu01:/bricks/22/vol1                             49173   Y
>> 33621
>>
>> Brick cpu02:/bricks/22/vol1                             49173   Y
>> 40864
>>
>> Brick cpu03:/bricks/22/vol1                             49173   Y
>> 18227
>>
>> Brick cpu04:/bricks/22/vol1                             49173   Y
>> 40762
>>
>> Brick cpu01:/bricks/23/vol1                             49174   Y
>> 33626
>>
>> Brick cpu02:/bricks/23/vol1                             49174   Y
>> 40869
>>
>> Brick cpu03:/bricks/23/vol1                             49174   Y
>> 18234
>>
>> Brick cpu04:/bricks/23/vol1                             49174   Y
>> 40769
>>
>> Brick cpu01:/bricks/24/vol1                             49175   Y
>> 33631
>>
>> Brick cpu02:/bricks/24/vol1                             49175   Y
>> 40874
>>
>> Brick cpu03:/bricks/24/vol1                             49175   Y
>> 18239
>>
>> Brick cpu04:/bricks/24/vol1                             49175   Y
>> 40774
>>
>> Self-heal Daemon on localhost                           N/A     Y
>> 33361
>>
>> Self-heal Daemon on cpu05                               N/A     Y
>> 2353
>>
>> Self-heal Daemon on cpu04                               N/A     Y
>> 40786
>>
>> Self-heal Daemon on cpu02                               N/A     Y
>> 32442
>>
>> Self-heal Daemon on cpu03                               N/A     Y
>> 18664
>>
>>
>>
>> Task Status of Volume ds01
>>
>>
>>
------------------------------------------------------------------------------
>>
>> Task                 : Rebalance
>>
>> ID                   : 5db24b30-4b9f-4b65-8910-a7a0a6d327a4
>>
>> Status               : completed
>>
>>
>>
>> [root at cpu01 log]#
>>
>>
>>
>> [root at cpu01 log]# gluster pool list
>>
>> UUID                                    Hostname        State
>>
>> 626c9360-8c09-480f-9707-116e67cc38e6    cpu02           Connected
>>
>> dc475d62-b035-4ee6-9006-6f03bf68bf24    cpu05           Connected
>>
>> 41b5b2ff-3671-47b4-b477-227a107e718d    cpu03           Connected
>>
>> c0afe114-dfa7-407d-bad7-5a3f97a6f3fc    cpu04           Connected
>>
>> 9b61b0a5-be78-4ac2-b6c0-2db588da5c35    localhost       Connected
>>
>> [root at cpu01 log]#
>>
>>
>>
>> [image: Inline image 3]
>>
>>
>>
>> Thanks,
>>
>> Punit
>>
>>
>>
>> On Thu, Mar 19, 2015 at 2:53 PM, Michal Skrivanek <
>> michal.skrivanek at redhat.com> wrote:
>>
>>
>> On Mar 19, 2015, at 03:18 , Punit Dambiwal <hypunit at gmail.com>
wrote:
>>
>> > Hi All,
>> >
>> > Is there any one have any idea about this problem...it seems
it's bug
>> either in Ovirt or Glusterfs...that's why no one has the idea about
>> it....please correct me if i am wrong?.
>>
>> Hi,
>> as I said, storage access times out; so it seems to me as a gluster
setup
>> problem, the storage domain you have your VMs on is not working?
>>
>> Thanks,
>> michal
>>
>>
>> >
>> > Thanks,
>> > Punit
>> >
>> > On Wed, Mar 18, 2015 at 5:05 PM, Punit Dambiwal <hypunit at
gmail.com>
>> wrote:
>> > Hi Michal,
>> >
>> > Would you mind to let me know the possible messedup things...i
will
>> check and try to resolve it....still i am communicating gluster
community
>> to resolve this issue...
>> >
>> > But in the ovirt....gluster setup is quite straight....so how come
it
>> will be messedup with reboot ?? if it can be messedup with reboot then
it
>> seems not good and stable technology for the production storage....
>> >
>> > Thanks,
>> > Punit
>> >
>> > On Wed, Mar 18, 2015 at 3:51 PM, Michal Skrivanek <
>> michal.skrivanek at redhat.com> wrote:
>> >
>> > On Mar 18, 2015, at 03:33 , Punit Dambiwal <hypunit at
gmail.com> wrote:
>> >
>> > > Hi,
>> > >
>> > > Is there any one from community can help me to solve this
issue...??
>> > >
>> > > Thanks,
>> > > Punit
>> > >
>> > > On Tue, Mar 17, 2015 at 12:52 PM, Punit Dambiwal <hypunit
at gmail.com>
>> wrote:
>> > > Hi,
>> > >
>> > > I am facing one strange issue with ovirt/glusterfs....still
didn't
>> find this issue is related with glusterfs or Ovirt....
>> > >
>> > > Ovirt :- 3.5.1
>> > > Glusterfs :- 3.6.1
>> > > Host :- 4 Hosts (Compute+ Storage)...each server has 24
bricks
>> > > Guest VM :- more then 100
>> > >
>> > > Issue :- When i deploy this cluster first time..it work well
for
>> me(all the guest VM created and running successfully)....but suddenly
one
>> day my one of the host node rebooted and none of the VM can boot up
>> now...and failed with the following error "Bad Volume
Specification"
>> > >
>> > > VMId :- d877313c18d9783ca09b62acf5588048
>> > >
>> > > VDSM Logs :- http://ur1.ca/jxabi
>> >
>> > you've got timeouts while accessing storage?so I guess
something got
>> messed up on reboot, it may also be just a gluster misconfiguration?
>> >
>> > > Engine Logs :- http://ur1.ca/jxabv
>> > >
>> > > ------------------------
>> > > [root at cpu01 ~]# vdsClient -s 0 getVolumeInfo
>> e732a82f-bae9-4368-8b98-dedc1c3814de
00000002-0002-0002-0002-000000000145
>> 6d123509-6867-45cf-83a2-6d679b77d3c5
9030bb43-6bc9-462f-a1b9-f6d5a02fb180
>> > >         status = OK
>> > >         domain = e732a82f-bae9-4368-8b98-dedc1c3814de
>> > >         capacity = 21474836480
>> > >         voltype = LEAF
>> > >         description >> > >         parent =
00000000-0000-0000-0000-000000000000
>> > >         format = RAW
>> > >         image = 6d123509-6867-45cf-83a2-6d679b77d3c5
>> > >         uuid = 9030bb43-6bc9-462f-a1b9-f6d5a02fb180
>> > >         disktype = 2
>> > >         legality = LEGAL
>> > >         mtime = 0
>> > >         apparentsize = 21474836480
>> > >         truesize = 4562972672
>> > >         type = SPARSE
>> > >         children = []
>> > >         pool >> > >         ctime = 1422676305
>> > > ---------------------
>> > >
>> > > I opened same thread earlier but didn't get any perfect
answers to
>> solve this issue..so i reopen it...
>> > >
>> > > https://www.mail-archive.com/users at ovirt.org/msg25011.html
>> > >
>> > > Thanks,
>> > > Punit
>> > >
>> > >
>> > >
>> >
>> >
>> >
>>
>>
>>
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150325/197e4a24/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 11779 bytes
Desc: not available
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150325/197e4a24/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 10223 bytes
Desc: not available
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150325/197e4a24/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 22346 bytes
Desc: not available
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150325/197e4a24/attachment-0002.png>

Gluster users - Mar 2015 - VM failed to start | Bad volume specification

[Gluster-users] VM failed to start | Bad volume specification

[Gluster-users] VM failed to start | Bad volume specification