Hello Vijay, Atin and Avra,
Thanks a lot for your advises.
Because users are using, currently I cannot stop our server but I'm
planning to restart the suspicious host gluster13.
I use the glusterd version 3.6.1 for all servers and 3.6.0.29 for clients.
The OS is CentOS 6.6.
In /var/lib/glusterd/glusterd.info, ?operating-version=1? is found for all
server hosts.
I don?t know how should I check the .cmd_log_history,
but I tried to find ?volume status? in the log as following
$ grep ?volume status? .cmd_log_history
Basically I could find ?volume status : SUCESS? for almost all node
excepted with gluster13. In .cmd_log_history in gluster13, I found ?volume
status testvol : FAILED : Locking failed on gluster13. Please check log
file for details.?
Best regards,
Kondo
2015-04-21 18:27 GMT+09:00 Atin Mukherjee <amukherj at redhat.com>:
>
>
> On 04/21/2015 02:47 PM, Avra Sengupta wrote:
> > In the logs I see, glusterd_lock() being used. This api is called only
> > in older versions of gluster or if you have a cluster version is less
> > then 30600. So along with the version of glusterfs used, could you
also
> > let us know what is the cluster version. You can check it as
> > "operating-version" in /var/lib/glusterd/glusterd.info file.
> Additionally please check whether concurrent volume operations were
> triggered by checking .cmd_log_history across all the nodes, if so, this
> could result into stale locks.
>
> ~Atin
> >
> > Regards,
> > Avra
> >
> > On 04/21/2015 02:34 PM, Avra Sengupta wrote:
> >> Hi Kondo,
> >>
> >> Can u also mention the version of gluster you are using.
> >>
> >> +Adding gluster-users
> >>
> >> Regards,
> >> Avra
> >> On 04/21/2015 02:27 PM, Avra Sengupta wrote:
> >>> Hi Kondo,
> >>>
> >>> I went through the gluster13 logs you had sent. Seems like
something
> >>> on that machine is holding the lock and is not releasing it.
There
> >>> are ways in which the system might end up in this scenario. I
will
> >>> try and explain the same with an example.
> >>>
> >>> Let's say I have gluster 11, gluster12, and gluster 13 in
my cluster.
> >>> I initiate a command from gluster11. Now the first thing that
command
> >>> does is, it holds a lock on all the nodes in the cluster on
behalf of
> >>> gluster11. Once the command does what's intended, it's
last act
> >>> before ending is to unlock all the nodes in the cluster. Now,
only
> >>> the node that has issued the lock, can issue the unlock.
> >>>
> >>> In your case what has happened is some command after
successfully
> >>> acquired the lock on gluster13. Now the node which initiated
the
> >>> command, went down or glusterd on that node went down before
it could
> >>> complete the command and it never got to send the unlock to
gluster13.
> >>>
> >>> There's a workaround to it. You can restart glusterd on
gluster13 and
> >>> it should work fine.
> >>>
> >>> Regards,
> >>> Avra
> >>>
> >>> On 04/20/2015 06:55 PM, kenji kondo wrote:
> >>>> Hello Vijay,
> >>>> Maybe this is very rare case. But is there any idea?
> >>>>
> >>>> Thanks,
> >>>> Kondo
> >>>>
> >>>> 2015-04-15 9:47 GMT+09:00 Vijaikumar M <vmallika at
redhat.com
> >>>> <mailto:vmallika at redhat.com>>:
> >>>>
> >>>> Adding Avra...
> >>>>
> >>>> Thanks,
> >>>> Vijay
> >>>>
> >>>>
> >>>> -------- Forwarded Message --------
> >>>> Subject: Re: [Gluster-users] Quota trouble
> >>>> Date: Wed, 15 Apr 2015 00:27:26 +0900
> >>>> From: kenji kondo <kkay.jp at gmail.com>
> >>>> <mailto:kkay.jp at gmail.com>
> >>>> To: Vijaikumar M <vmallika at redhat.com>
> >>>> <mailto:vmallika at redhat.com>
> >>>>
> >>>>
> >>>>
> >>>> Hi Vijay,
> >>>>
> >>>> Thanks for your comments.
> >>>>
> >>>>
> >>>> The lock error occurs at one server it's called
"gluster13".
> >>>>
> >>>> In the gluster13, I tried to create new volume and
start quota.
> >>>> But it failed as below,
> >>>>
> >>>>
> >>>> In both host gluster10 and gluster13, ran below
> >>>>
> >>>> $ sudo mkdir /export11/testbrick1
> >>>>
> >>>> $ sudo mkdir /export11/testbrick2
> >>>>
> >>>> In gluster13, ran below
> >>>>
> >>>> $ sudo /usr/sbin/gluster volume create testvol2
> >>>> gluster13:/export11/testbrick1
gluster13:/export11/testbrick2
> >>>>
> >>>> volume create: testvol2: failed: Locking failed on
gluster13.
> >>>> Please check log file for details.
> >>>>
> >>>> $ sudo /usr/sbin/gluster volume create testvol2
> >>>> gluster10:/export11/testbrick1
gluster10:/export11/testbrick2
> >>>>
> >>>> volume create: testvol2: failed: Locking failed on
gluster13.
> >>>> Please check log file for details.
> >>>>
> >>>> But I recived error messages above.
> >>>>
> >>>> On the other hand, in gluster10, it was success.
> >>>>
> >>>> Again, in gluster13, I tried to run quota, but it
failed as below.
> >>>>
> >>>> $ sudo /usr/sbin/gluster volume quota testvol2 enable
> >>>>
> >>>> quota command failed : Locking failed on gluster13.
Please check
> >>>> log file for details.
> >>>>
> >>>>
> >>>> Could you find attached?
> >>>>
> >>>> We can find error messages in the log of gluster13.
> >>>>
> >>>>
> >>>> Best regards,
> >>>>
> >>>> Kondo
> >>>>
> >>>>
> >>>>
> >>>> 2015-04-13 19:38 GMT+09:00 Vijaikumar M <vmallika
at redhat.com
> >>>> <mailto:vmallika at redhat.com>>:
> >>>>
> >>>> Hi Kondo,
> >>>>
> >>>> The lock error you mentioned is because, another
operation
> >>>> is still running on the volume and hence not able
to acquire
> >>>> the lock.
> >>>> This is bug of not displaying proper error
message, we are
> >>>> working on fixing this issue.
> >>>>
> >>>> I was not able to find any clue on why quotad is
not running.
> >>>>
> >>>> I wanted to check, if we can manually start quotad
something
> >>>> like below:
> >>>>
> >>>> # /usr/local/sbin/glusterfs -s localhost
--volfile-id
> >>>> gluster/quotad -p
/var/lib/glusterd/quotad/run/quotad.pid -l
> >>>> /var/log/glusterfs/quotad.log -S
> >>>> /var/run/gluster/myquotad.socket --xlator-option
> >>>> *replicate*.data-self-heal=off --xlator-option
> >>>> *replicate*.metadata-self-heal=off --xlator-option
> >>>> *replicate*.entry-self-heal=off
> >>>>
> >>>> or
> >>>>
> >>>> create a new temporary volume, and enable quota on
this
> >>>> volume. (quotad will be same for all the volume
which has
> >>>> quota enabled)
> >>>>
> >>>>
> >>>> Thanks,
> >>>> Vijay
> >>>>
> >>>>
> >>>> On Sunday 12 April 2015 07:05 PM, kenji kondo
wrote:
> >>>>> Hi Vijay,
> >>>>>
> >>>>> Thank you for your suggestion. But I'm
sorry, it's
> >>>>> difficult to access from outside because my
glusterfs
> >>>>> system is closed.
> >>>>> I will give up if there is no clue information
in attached
> >>>>> log.
> >>>>>
> >>>>> Best regards,
> >>>>> Kondo
> >>>>>
> >>>>>
> >>>>> 2015-04-09 15:40 GMT+09:00 Vijaikumar M
> >>>>> <vmallika at redhat.com <mailto:vmallika
at redhat.com>>:
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thursday 09 April 2015 11:58 AM,
Vijaikumar M wrote:
> >>>>>>
> >>>>>>
> >>>>>> On Wednesday 08 April 2015 09:57 PM,
kenji kondo wrote:
> >>>>>>> Hi Vijay,
> >>>>>>>
> >>>>>>> I checked the all of the setting.
> >>>>>>> The all are
'features.quota=on' when I set quota
> >>>>>>> enable and the all are
'features.quota=off' when I
> >>>>>>> set quota disable.
> >>>>>>>
> >>>>>>> But I could find new issue.
> >>>>>>> When I checked a volume status for
all server, in one
> >>>>>>> of the servers I received the
error message as below.
> >>>>>>>
> >>>>>>> $ sudo /usr/sbin/gluster volume
status testvol
> >>>>>>> Locking failed on gluster13.
Please check log file
> >>>>>>> for details.
> >>>>>>>
> >>>>>>> In etc-glusterfs-glusterd.vol.log
of problem server,
> >>>>>>> I found error messages as below.
> >>>>>>> [2015-04-08 08:40:04.782644] I
> >>>>>>> [mem-pool.c:545:mem_pool_destroy]
0-management:
> >>>>>>> size=588 max=0 total=0
> >>>>>>> [2015-04-08 08:40:04.782685] I
> >>>>>>> [mem-pool.c:545:mem_pool_destroy]
0-management:
> >>>>>>> size=124 max=0 total=0
> >>>>>>> [2015-04-08 08:40:04.782848] W
> >>>>>>> [socket.c:611:__socket_rwv]
0-management: readv on
> >>>>>>>
/var/run/14b05cd492843e6e288e290c2d63093c.socket
> >>>>>>> failed (Invalid arguments)
> >>>>>>> [2015-04-08 08:40:04.805407] I
[MSGID: 106006]
> >>>>>>>
[glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify]
> >>>>>>> 0-management: nfs has disconnected
from glusterd.
> >>>>>>> [2015-04-08 08:43:02.439001] I
> >>>>>>>
> >>>>>>>
[glusterd-handler.c:3803:__glusterd_handle_status_volume]
> >>>>>>> 0-management: Received status
volume req for volume
> >>>>>>> testvol
> >>>>>>> [2015-04-08 08:43:02.460581] E
> >>>>>>>
[glusterd-utils.c:148:glusterd_lock] 0-management:
> >>>>>>> Unable to get lock for uuid:
> >>>>>>>
03a32bce-ec63-4dc3-a287-4901a55dd8c9, lock held by:
> >>>>>>>
03a32bce-ec63-4dc3-a287-4901a55dd8c9
> >>>>>>> [2015-04-08 08:43:02.460632] E
> >>>>>>>
[glusterd-op-sm.c:6584:glusterd_op_sm] 0-management:
> >>>>>>> handler returned: -1
> >>>>>>> [2015-04-08 08:43:02.460654] E
> >>>>>>>
[glusterd-syncop.c:105:gd_collate_errors] 0-: Locking
> >>>>>>> failed on gluster13. Please check
log file for details.
> >>>>>>> [2015-04-08 08:43:02.461409] E
> >>>>>>>
[glusterd-syncop.c:1602:gd_sync_task_begin]
> >>>>>>> 0-management: Locking Peers
Failed.
> >>>>>>> [2015-04-08 08:43:43.698168] I
> >>>>>>>
> >>>>>>>
[glusterd-handler.c:3803:__glusterd_handle_status_volume]
> >>>>>>> 0-management: Received status
volume req for volume
> >>>>>>> testvol
> >>>>>>> [2015-04-08 08:43:43.698813] E
> >>>>>>>
[glusterd-utils.c:148:glusterd_lock] 0-management:
> >>>>>>> Unable to get lock for uuid:
> >>>>>>>
03a32bce-ec63-4dc3-a287-4901a55dd8c9, lock held by:
> >>>>>>>
03a32bce-ec63-4dc3-a287-4901a55dd8c9
> >>>>>>> [2015-04-08 08:43:43.698898] E
> >>>>>>>
[glusterd-op-sm.c:6584:glusterd_op_sm] 0-management:
> >>>>>>> handler returned: -1
> >>>>>>> [2015-04-08 08:43:43.698994] E
> >>>>>>>
[glusterd-syncop.c:105:gd_collate_errors] 0-: Locking
> >>>>>>> failed on gluster13. Please check
log file for details.
> >>>>>>> [2015-04-08 08:43:43.702126] E
> >>>>>>>
[glusterd-syncop.c:1602:gd_sync_task_begin]
> >>>>>>> 0-management: Locking Peers
Failed.
> >>>>>>> [2015-04-08 08:44:01.277139] I
> >>>>>>>
> >>>>>>>
[glusterd-handler.c:3803:__glusterd_handle_status_volume]
> >>>>>>> 0-management: Received status
volume req for volume
> >>>>>>> testvol
> >>>>>>> [2015-04-08 08:44:01.277560] E
> >>>>>>>
[glusterd-utils.c:148:glusterd_lock] 0-management:
> >>>>>>> Unable to get lock for uuid:
> >>>>>>>
03a32bce-ec63-4dc3-a287-4901a55dd8c9, lock held by:
> >>>>>>>
03a32bce-ec63-4dc3-a287-4901a55dd8c9
> >>>>>>> [2015-04-08 08:44:01.277639] E
> >>>>>>>
[glusterd-op-sm.c:6584:glusterd_op_sm] 0-management:
> >>>>>>> handler returned: -1
> >>>>>>> [2015-04-08 08:44:01.277676] E
> >>>>>>>
[glusterd-syncop.c:105:gd_collate_errors] 0-: Locking
> >>>>>>> failed on gluster13. Please check
log file for details.
> >>>>>>> [2015-04-08 08:44:01.281514] E
> >>>>>>>
[glusterd-syncop.c:1602:gd_sync_task_begin]
> >>>>>>> 0-management: Locking Peers
Failed.
> >>>>>>> [2015-04-08 08:45:42.599796] I
> >>>>>>>
> >>>>>>>
[glusterd-handler.c:3803:__glusterd_handle_status_volume]
> >>>>>>> 0-management: Received status
volume req for volume
> >>>>>>> testvol
> >>>>>>> [2015-04-08 08:45:42.600343] E
> >>>>>>>
[glusterd-utils.c:148:glusterd_lock] 0-management:
> >>>>>>> Unable to get lock for uuid:
> >>>>>>>
03a32bce-ec63-4dc3-a287-4901a55dd8c9, lock held by:
> >>>>>>>
03a32bce-ec63-4dc3-a287-4901a55dd8c9
> >>>>>>> [2015-04-08 08:45:42.600417] E
> >>>>>>>
[glusterd-op-sm.c:6584:glusterd_op_sm] 0-management:
> >>>>>>> handler returned: -1
> >>>>>>> [2015-04-08 08:45:42.600482] E
> >>>>>>>
[glusterd-syncop.c:105:gd_collate_errors] 0-: Locking
> >>>>>>> failed on gluster13. Please check
log file for details.
> >>>>>>> [2015-04-08 08:45:42.601039] E
> >>>>>>>
[glusterd-syncop.c:1602:gd_sync_task_begin]
> >>>>>>> 0-management: Locking Peers
Failed.
> >>>>>>>
> >>>>>>> Does this situation relate to my
quota problems?
> >>>>>>>
> >>>>>>
> >>>>>> This is a glusterd different issue.
Can we get the
> >>>>>> glusterd logs from gluster13?
> >>>>>> Can get access to these machines, so
that we can debug
> >>>>>> live?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Vijay
> >>>>>>
> >>>>> Regarding quota issue, quota feature is
enabled
> >>>>> successfully. I am wondering why quotad is
not started.
> >>>>> If we get the access to the machine, it
will be easier
> >>>>> to debug the issue.
> >>>>>
> >>>>> Thanks,
> >>>>> Vijay
> >>>>>
> >>>>>
> >>>>>>>
> >>>>>>> Best regards,
> >>>>>>> Kondo
> >>>>>>>
> >>>>>>>
> >>>>>>> 2015-04-08 15:14 GMT+09:00
Vijaikumar M
> >>>>>>> <vmallika at redhat.com
<mailto:vmallika at redhat.com>>:
> >>>>>>>
> >>>>>>> Hi Kondo,
> >>>>>>>
> >>>>>>> I suspect, in one of the node
quota feature is
> >>>>>>> not set for some reason and
hence quotad is not
> >>>>>>> starting.
> >>>>>>>
> >>>>>>> On all the nodes can you check
if below option is
> >>>>>>> set to 'on'
> >>>>>>>
> >>>>>>> # grep quota
/var/lib/glusterd/vols/<volname>/info
> >>>>>>> features.quota=on
> >>>>>>>
> >>>>>>>
> >>>>>>> Also can I get brick logs from
all the nodes?
> >>>>>>>
> >>>>>>> Also can you create a
temporary volume and enable
> >>>>>>> the quota here and see if see
quota works fine
> >>>>>>> with this volume?
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Vijay
> >>>>>>>
> >>>>>>> On Tuesday 07 April 2015 08:34
PM, kenji kondo
> >>>>>>> wrote:
> >>>>>>>> Hi Vijay,
> >>>>>>>>
> >>>>>>>> Could you find attached?
> >>>>>>>> I got logs of server and
client.
> >>>>>>>> As same as before, I could
not create a file
> >>>>>>>> after quota usage-limit
setting.
> >>>>>>>>
> >>>>>>>> Best regards,
> >>>>>>>> Kondo
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> 2015-04-07 18:34 GMT+09:00
Vijaikumar M
> >>>>>>>> <vmallika at redhat.com
<mailto:vmallika at redhat.com
> >>:
> >>>>>>>>
> >>>>>>>> Hi Konda,
> >>>>>>>>
> >>>>>>>> Can we get all the log
files?
> >>>>>>>>
> >>>>>>>> # gluster volume quota
<volname> disable
> >>>>>>>> # gluster volume quota
<volname> enable
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Now copy all the logs
files.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Vijay
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tuesday 07 April
2015 12:39 PM, K.Kondo
> >>>>>>>> wrote:
> >>>>>>>>> Thank you very
much ! Vijay
> >>>>>>>>> I want to use a
quota because each volume
> >>>>>>>>> became too big.
> >>>>>>>>>
> >>>>>>>>> Best regard
> >>>>>>>>> Kondo
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 2015/04/07
15:18?Vijaikumar M
> >>>>>>>>> <vmallika at
redhat.com
> >>>>>>>>>
<mailto:vmallika at redhat.com>> ??????:
> >>>>>>>>>
> >>>>>>>>>> Hi Kondo,
> >>>>>>>>>>
> >>>>>>>>>> I couldn?t
find clue from the logs. I will
> >>>>>>>>>> discuss about
this issue with my
> >>>>>>>>>> colleagues
today.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Vijay
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Monday 06
April 2015 10:56 PM, kenji
> >>>>>>>>>> kondo wrote:
> >>>>>>>>>>> Hello
Vijay,
> >>>>>>>>>>> Is there
something idea for this?
> >>>>>>>>>>> Best
regards,
> >>>>>>>>>>> Kondo
> >>>>>>>>>>>
> >>>>>>>>>>> 2015-03-31
22:46 GMT+09:00 kenji kondo
> >>>>>>>>>>>
<kkay.jp at gmail.com
> >>>>>>>>>>>
<mailto:kkay.jp at gmail.com>>:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi
Vijay,
> >>>>>>>>>>>
> >>>>>>>>>>>
I'm sorry for late reply.
> >>>>>>>>>>> I
could get the debug mode log as
> >>>>>>>>>>>
attached.
> >>>>>>>>>>> In
this test, unfortunately the quota
> >>>>>>>>>>> did
not work as same as before.
> >>>>>>>>>>>
> >>>>>>>>>>> Could
you find the cause of my problem?
> >>>>>>>>>>>
> >>>>>>>>>>> Best
regards,
> >>>>>>>>>>> Kondo
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
2015-03-25 17:20 GMT+09:00 Vijaikumar
> >>>>>>>>>>> M
<vmallika at redhat.com
> >>>>>>>>>>>
<mailto:vmallika at redhat.com>>:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi
Kondo,
> >>>>>>>>>>>
> >>>>>>>>>>>
For some reason quota enable was
> >>>>>>>>>>>
not successful. We may have
> >>>>>>>>>>>
re-try enabling quota.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
Thanks,
> >>>>>>>>>>>
Vijay
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On
Tuesday 24 March 2015 07:08
> >>>>>>>>>>>
PM, kenji kondo wrote:
> >>>>>>>>>>>>
Hi Vijay,
> >>>>>>>>>>>>
Thanks for your checking.
> >>>>>>>>>>>>
Unfortunately, currently I can't
> >>>>>>>>>>>>
stop the service because many
> >>>>>>>>>>>>
users are using.
> >>>>>>>>>>>>
But, I want to know this cause
> >>>>>>>>>>>>
of this trouble, so I will plan
> >>>>>>>>>>>>
to stop. Please wait to get the
> >>>>>>>>>>>> log.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
Best regards,
> >>>>>>>>>>>>
Kondo
> >>>>>>>>>>>>
> >>>>>>>>>>>>
2015-03-24 17:01 GMT+09:00
> >>>>>>>>>>>>
Vijaikumar M
> >>>>>>>>>>>>
<vmallika at redhat.com
> >>>>>>>>>>>>
<mailto:vmallika at redhat.com>>:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
Hi Kondo,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
I couldn't find much clue in
> >>>>>>>>>>>>
the glusterd logs, other
> >>>>>>>>>>>>
than the error message you
> >>>>>>>>>>>>
mentioned below.
> >>>>>>>>>>>>
Can you try disabling and
> >>>>>>>>>>>>
enabling the quota again and
> >>>>>>>>>>>>
see if this start quotad?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
Try below command:
> >>>>>>>>>>>>
# gluster volume quota
> >>>>>>>>>>>>
<volname> disable
> >>>>>>>>>>>>
> >>>>>>>>>>>>
wait for all quota process
> >>>>>>>>>>>>
to terminate
> >>>>>>>>>>>>
#ps -ef | quota
> >>>>>>>>>>>>
> >>>>>>>>>>>>
# service glusterd stop
> >>>>>>>>>>>>
# glusterd -LDEBUG
> >>>>>>>>>>>>
# gluster volume quota
> >>>>>>>>>>>>
<volname> enable
> >>>>>>>>>>>>
> >>>>>>>>>>>>
Now verify if quotad is
> running
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
Thanks,
> >>>>>>>>>>>>
Vijay
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
On Monday 23 March 2015
> >>>>>>>>>>>>
06:24 PM, kenji kondo wrote:
> >>>>>>>>>>>>>
Hi Vijay,
> >>>>>>>>>>>>>
As you pointed out, the
> >>>>>>>>>>>>>
quotad is not running in
> >>>>>>>>>>>>>
the all of server.
> >>>>>>>>>>>>>
I checked the volume status
> >>>>>>>>>>>>>
and got following log.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
Quota Daemon on
> >>>>>>>>>>>>>
gluster25N/ANN/A
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
So, I attached requested
> >>>>>>>>>>>>>
log
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
'etc-glusterfs-glusterd.vol.log'.
> >>>>>>>>>>>>>
The error messages can be
> >>>>>>>>>>>>>
found in the log.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
[2015-03-19
> >>>>>>>>>>>>>
11:51:07.457697] E
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
[glusterd-quota.c:1467:glusterd_op_stage_quota]
> >>>>>>>>>>>>>
0-management: Quota is
> >>>>>>>>>>>>>
disabled, please enable quota
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
If you want more some
> >>>>>>>>>>>>>
information to solve this
> >>>>>>>>>>>>>
problems, please ask me.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
Best regards,
> >>>>>>>>>>>>>
Kondo
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
2015-03-23 16:04 GMT+09:00
> >>>>>>>>>>>>>
Vijaikumar M
> >>>>>>>>>>>>>
<vmallika at redhat.com
> >>>>>>>>>>>>>
<mailto:vmallika at redhat.com
> >>:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
Hi Kondo,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
Can you please verify
> >>>>>>>>>>>>>
if quotad is running?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
root at rh1:~ *# gluster
> >>>>>>>>>>>>>
volume status*
> >>>>>>>>>>>>>
Status of volume: vol1
> >>>>>>>>>>>>>
Gluster process TCP
> >>>>>>>>>>>>>
Port RDMA Port Online
> >>>>>>>>>>>>> Pid
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
>
------------------------------------------------------------------------------
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
Brick
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
rh1:/var/opt/gluster/bricks/b1/dir
> >>>>>>>>>>>>>
49152 0 Y 1858
> >>>>>>>>>>>>>
NFS Server on localhost
> >>>>>>>>>>>>>
2049 0 Y 1879
> >>>>>>>>>>>>>
*Quota Daemon on
> >>>>>>>>>>>>>
localhost N/A N/A
> >>>>>>>>>>>>>
Y 1914 **
> >>>>>>>>>>>>>
*
> >>>>>>>>>>>>>
Task Status of Volume
> vol1
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
>
------------------------------------------------------------------------------
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
There are no active
> >>>>>>>>>>>>>
volume tasks
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
root at rh1:~ # *ps -ef |
> >>>>>>>>>>>>>
grep quotad*
> >>>>>>>>>>>>>
root 1914 1 0
> >>>>>>>>>>>>>
12:29 ? 00:00:00
> >>>>>>>>>>>>>
/usr/local/sbin/glusterfs
> >>>>>>>>>>>>>
-s localhost
> >>>>>>>>>>>>>
--volfile-id
> >>>>>>>>>>>>>
gluster/quotad -p
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
/var/lib/glusterd/quotad/run/quotad.pid
> >>>>>>>>>>>>>
-l
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
*/var/log/glusterfs/quotad.log*-S
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
/var/run/gluster/bb6ab82f70f555fd5c0e188fa4e09584.socket
> >>>>>>>>>>>>>
--xlator-option
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
*replicate*.data-self-heal=off
> >>>>>>>>>>>>>
--xlator-option
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
*replicate*.metadata-self-heal=off
> >>>>>>>>>>>>>
--xlator-option
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
*replicate*.entry-self-heal=off
> >>>>>>>>>>>>>
root 1970 1511 0
> >>>>>>>>>>>>>
12:31 pts/1 00:00:00
> >>>>>>>>>>>>>
grep quotad
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
root at rh1:~ # *gluster
> >>>>>>>>>>>>>
volume info*
> >>>>>>>>>>>>>
Volume Name: vol1
> >>>>>>>>>>>>>
Type: Distribute
> >>>>>>>>>>>>>
Volume ID:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
a55519ec-65d1-4741-9ad3-f94020fc9b21
> >>>>>>>>>>>>>
Status: Started
> >>>>>>>>>>>>>
Number of Bricks: 1
> >>>>>>>>>>>>>
Transport-type: tcp
> >>>>>>>>>>>>>
Bricks:
> >>>>>>>>>>>>>
Brick1:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
rh1:/var/opt/gluster/bricks/b1/dir
> >>>>>>>>>>>>>
Options Reconfigured:
> >>>>>>>>>>>>>
*features.quota: on**
> >>>>>>>>>>>>>
*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
If quotad is not
> >>>>>>>>>>>>>
running, can you please
> >>>>>>>>>>>>>
provide glusterd logs
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
'usr-local-etc-glusterfs-glusterd.vol.log'.
> >>>>>>>>>>>>>
I will check is there
> >>>>>>>>>>>>>
are any issues starting
> >>>>>>>>>>>>>
quotad.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
Thanks,
> >>>>>>>>>>>>>
Vihay
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
On Monday 23 March 2015
> >>>>>>>>>>>>>
11:54 AM, K.Kondo wrote:
> >>>>>>>>>>>>>>
Hi Vijay,
> >>>>>>>>>>>>>>
I could not find
> >>>>>>>>>>>>>>
the"quotad.log" in
> >>>>>>>>>>>>>>
directory
> >>>>>>>>>>>>>>
/var/log/glusterfs in
> >>>>>>>>>>>>>>
both servers and
> >>>>>>>>>>>>>>
client. But other test
> >>>>>>>>>>>>>>
server has the log.
> >>>>>>>>>>>>>>
Do you know why there
> >>>>>>>>>>>>>>
is no the file?
> >>>>>>>>>>>>>>
Thanks,
> >>>>>>>>>>>>>>
Kondo
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
2015/03/23 13:41?
> >>>>>>>>>>>>>>
Vijaikumar M
> >>>>>>>>>>>>>>
<vmallika at redhat.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
<mailto:vmallika at redhat.com>>
> >>>>>>>>>>>>>>
??????:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
Hi Kondo,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
log file 'quotad.log'
> >>>>>>>>>>>>>>>
is missing in the
> >>>>>>>>>>>>>>>
attachment.Can you
> >>>>>>>>>>>>>>>
provide this log file
> >>>>>>>>>>>>>>>
as well?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
Thanks,
> >>>>>>>>>>>>>>>
Vijay
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
On Monday 23 March
> >>>>>>>>>>>>>>>
2015 09:50 AM, kenji
> >>>>>>>>>>>>>>>
kondo wrote:
> >>>>>>>>>>>>>>>>
Hi Vijay,
> >>>>>>>>>>>>>>>>
Could you find the
> >>>>>>>>>>>>>>>>
attached?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
Best regards,
> >>>>>>>>>>>>>>>>
Kondo
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
2015-03-23 12:53
> >>>>>>>>>>>>>>>>
GMT+09:00 Vijaikumar
> >>>>>>>>>>>>>>>>
M
> >>>>>>>>>>>>>>>>
<vmallika at redhat.com
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
<mailto:vmallika at redhat.com>>:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
Hi Kondo,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
Can you please
> >>>>>>>>>>>>>>>>
provide below
> >>>>>>>>>>>>>>>>
mentioned
> >>>>>>>>>>>>>>>>
gluterfs logs?
> >>>>>>>>>>>>>>>>
client logs
> >>>>>>>>>>>>>>>>
(name of this
> >>>>>>>>>>>>>>>>
log will be
> >>>>>>>>>>>>>>>>
prefixed with
> >>>>>>>>>>>>>>>>
mount-point
> >>>>>>>>>>>>>>>> dirname)
> >>>>>>>>>>>>>>>>
brick logs
> >>>>>>>>>>>>>>>>
quotad logs
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
Thanks,
> >>>>>>>>>>>>>>>>
Vijay
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
On Friday 20
> >>>>>>>>>>>>>>>>
March 2015 06:31
> >>>>>>>>>>>>>>>>
PM, kenji kondo
> >>>>>>>>>>>>>>>>
wrote:
> >>>>>>>>>>>>>>>>>
Hi, Vijay and
> >>>>>>>>>>>>>>>>>
Peter
> >>>>>>>>>>>>>>>>>
Thanks for your
> >>>>>>>>>>>>>>>>>
reply.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
I create new
> >>>>>>>>>>>>>>>>>
volume
> >>>>>>>>>>>>>>>>>
"testvol" with
> >>>>>>>>>>>>>>>>>
two bricks and
> >>>>>>>>>>>>>>>>>
set quota to
> >>>>>>>>>>>>>>>>>
simplify this
> >>>>>>>>>>>>>>>>>
problem.
> >>>>>>>>>>>>>>>>>
I got the
> >>>>>>>>>>>>>>>>>
glusterfs log
> >>>>>>>>>>>>>>>>>
as following
> >>>>>>>>>>>>>>>>>
after try to
> >>>>>>>>>>>>>>>>>
create a
> >>>>>>>>>>>>>>>>>
directory and
> >>>>>>>>>>>>>>>>> file.
> >>>>>>>>>>>>>>>>>
BTW, my
> >>>>>>>>>>>>>>>>>
glusterd was
> >>>>>>>>>>>>>>>>>
upgraded from
> >>>>>>>>>>>>>>>>>
older version,
> >>>>>>>>>>>>>>>>>
although I
> >>>>>>>>>>>>>>>>>
don't know
> >>>>>>>>>>>>>>>>>
related to it.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
Best regards,
> >>>>>>>>>>>>>>>>>
Kondo
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.931016]
> I
> >>>>>>>>>>>>>>>>>
[MSGID: 100030]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[glusterfsd.c:1998:main]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-/usr/sbin/glusterfs:
> >>>>>>>>>>>>>>>>>
Started running
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
/usr/sbin/glusterfs
> >>>>>>>>>>>>>>>>>
version
> >>>>>>>>>>>>>>>>>
3.6.0.29 (args:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
/usr/sbin/glusterfs
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
--volfile-server=gluster10
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
--volfile-id=testvol
> >>>>>>>>>>>>>>>>>
testvol)
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.944850]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[dht-shared.c:337:dht_init_regex]
> >>>>>>>>>>>>>>>>>
0-testvol-dht:
> >>>>>>>>>>>>>>>>>
using regex
> >>>>>>>>>>>>>>>>>
rsync-hash-regex
> > >>>>>>>>>>>>>>>>>
^\.(.+)\.[^.]+$
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.946256]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client.c:2280:notify]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-0:
> >>>>>>>>>>>>>>>>>
parent
> >>>>>>>>>>>>>>>>>
translators are
> >>>>>>>>>>>>>>>>>
ready,
> >>>>>>>>>>>>>>>>>
attempting
> >>>>>>>>>>>>>>>>>
connect on
> >>>>>>>>>>>>>>>>>
transport
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.950674]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client.c:2280:notify]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-1:
> >>>>>>>>>>>>>>>>>
parent
> >>>>>>>>>>>>>>>>>
translators are
> >>>>>>>>>>>>>>>>>
ready,
> >>>>>>>>>>>>>>>>>
attempting
> >>>>>>>>>>>>>>>>>
connect on
> >>>>>>>>>>>>>>>>>
transport
> >>>>>>>>>>>>>>>>>
Final graph:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
+------------------------------------------------------------------------------+
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
1: volume
> >>>>>>>>>>>>>>>>>
testvol-client-0
> >>>>>>>>>>>>>>>>>
2: type
> >>>>>>>>>>>>>>>>>
protocol/client
> >>>>>>>>>>>>>>>>>
3: option
> >>>>>>>>>>>>>>>>>
ping-timeout 42
> >>>>>>>>>>>>>>>>>
4: option
> >>>>>>>>>>>>>>>>>
remote-host
> >>>>>>>>>>>>>>>>>
gluster24
> >>>>>>>>>>>>>>>>>
5: option
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
remote-subvolume /export25/brick
> >>>>>>>>>>>>>>>>>
6: option
> >>>>>>>>>>>>>>>>>
transport-type
> >>>>>>>>>>>>>>>>>
socket
> >>>>>>>>>>>>>>>>>
7: option
> >>>>>>>>>>>>>>>>>
send-gids true
> >>>>>>>>>>>>>>>>>
8: end-volume
> >>>>>>>>>>>>>>>>>
9:
> >>>>>>>>>>>>>>>>>
10: volume
> >>>>>>>>>>>>>>>>>
testvol-client-1
> >>>>>>>>>>>>>>>>>
11: type
> >>>>>>>>>>>>>>>>>
protocol/client
> >>>>>>>>>>>>>>>>>
12: option
> >>>>>>>>>>>>>>>>>
ping-timeout 42
> >>>>>>>>>>>>>>>>>
13: option
> >>>>>>>>>>>>>>>>>
remote-host
> >>>>>>>>>>>>>>>>>
gluster25
> >>>>>>>>>>>>>>>>>
14: option
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
remote-subvolume /export25/brick
> >>>>>>>>>>>>>>>>>
15: option
> >>>>>>>>>>>>>>>>>
transport-type
> >>>>>>>>>>>>>>>>>
socket
> >>>>>>>>>>>>>>>>>
16: option
> >>>>>>>>>>>>>>>>>
send-gids true
> >>>>>>>>>>>>>>>>>
17: end-volume
> >>>>>>>>>>>>>>>>>
18:
> >>>>>>>>>>>>>>>>>
19: volume
> >>>>>>>>>>>>>>>>>
testvol-dht
> >>>>>>>>>>>>>>>>>
20: type
> >>>>>>>>>>>>>>>>>
> cluster/distribute
> >>>>>>>>>>>>>>>>>
21: subvolumes
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
testvol-client-0 testvol-client-1
> >>>>>>>>>>>>>>>>>
22: end-volume
> >>>>>>>>>>>>>>>>>
23:
> >>>>>>>>>>>>>>>>>
24: volume
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
testvol-write-behind
> >>>>>>>>>>>>>>>>>
25: type
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
performance/write-behind
> >>>>>>>>>>>>>>>>>
26: subvolumes
> >>>>>>>>>>>>>>>>>
testvol-dht
> >>>>>>>>>>>>>>>>>
27: end-volume
> >>>>>>>>>>>>>>>>>
28:
> >>>>>>>>>>>>>>>>>
29: volume
> >>>>>>>>>>>>>>>>>
> testvol-read-ahead
> >>>>>>>>>>>>>>>>>
30: type
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
performance/read-ahead
> >>>>>>>>>>>>>>>>>
31: subvolumes
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
testvol-write-behind
> >>>>>>>>>>>>>>>>>
32: end-volume
> >>>>>>>>>>>>>>>>>
33:
> >>>>>>>>>>>>>>>>>
34: volume
> >>>>>>>>>>>>>>>>>
testvol-io-cache
> >>>>>>>>>>>>>>>>>
35: type
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
performance/io-cache
> >>>>>>>>>>>>>>>>>
36: subvolumes
> >>>>>>>>>>>>>>>>>
> testvol-read-ahead
> >>>>>>>>>>>>>>>>>
37: end-volume
> >>>>>>>>>>>>>>>>>
38:
> >>>>>>>>>>>>>>>>>
39: volume
> >>>>>>>>>>>>>>>>>
> testvol-quick-read
> >>>>>>>>>>>>>>>>>
40: type
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
performance/quick-read
> >>>>>>>>>>>>>>>>>
41: subvolumes
> >>>>>>>>>>>>>>>>>
testvol-io-cache
> >>>>>>>>>>>>>>>>>
42: end-volume
> >>>>>>>>>>>>>>>>>
43:
> >>>>>>>>>>>>>>>>>
44: volume
> >>>>>>>>>>>>>>>>>
testvol-md-cache
> >>>>>>>>>>>>>>>>>
45: type
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
performance/md-cache
> >>>>>>>>>>>>>>>>>
46: subvolumes
> >>>>>>>>>>>>>>>>>
> testvol-quick-read
> >>>>>>>>>>>>>>>>>
47: end-volume
> >>>>>>>>>>>>>>>>>
48:
> >>>>>>>>>>>>>>>>>
49: volume
> >>>>>>>>>>>>>>>>>
testvol
> >>>>>>>>>>>>>>>>>
50: type
> >>>>>>>>>>>>>>>>>
debug/io-stats
> >>>>>>>>>>>>>>>>>
51: option
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
latency-measurement
> >>>>>>>>>>>>>>>>>
off
> >>>>>>>>>>>>>>>>>
52: option
> >>>>>>>>>>>>>>>>>
count-fop-hits
> off
> >>>>>>>>>>>>>>>>>
53: subvolumes
> >>>>>>>>>>>>>>>>>
testvol-md-cache
> >>>>>>>>>>>>>>>>>
54: end-volume
> >>>>>>>>>>>>>>>>>
55:
> >>>>>>>>>>>>>>>>>
56: volume
> >>>>>>>>>>>>>>>>>
meta-autoload
> >>>>>>>>>>>>>>>>>
57: type meta
> >>>>>>>>>>>>>>>>>
58: subvolumes
> >>>>>>>>>>>>>>>>>
testvol
> >>>>>>>>>>>>>>>>>
59: end-volume
> >>>>>>>>>>>>>>>>>
60:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
+------------------------------------------------------------------------------+
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.955337]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[rpc-clnt.c:1759:rpc_clnt_reconfig]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-0:
> >>>>>>>>>>>>>>>>>
changing port
> >>>>>>>>>>>>>>>>>
to 49155 (from 0)
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.957549]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[rpc-clnt.c:1759:rpc_clnt_reconfig]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-1:
> >>>>>>>>>>>>>>>>>
changing port
> >>>>>>>>>>>>>>>>>
to 49155 (from 0)
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.959889]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> [client-handshake.c:1415:select_server_supported_programs]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-0:
> >>>>>>>>>>>>>>>>>
Using Program
> >>>>>>>>>>>>>>>>>
GlusterFS 3.3,
> >>>>>>>>>>>>>>>>>
Num (1298437),
> >>>>>>>>>>>>>>>>>
Version (330)
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.960090]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> [client-handshake.c:1415:select_server_supported_programs]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-1:
> >>>>>>>>>>>>>>>>>
Using Program
> >>>>>>>>>>>>>>>>>
GlusterFS 3.3,
> >>>>>>>>>>>>>>>>>
Num (1298437),
> >>>>>>>>>>>>>>>>>
Version (330)
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.960376]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client-handshake.c:1200:client_setvolume_cbk]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-0:
> >>>>>>>>>>>>>>>>>
Connected to
> >>>>>>>>>>>>>>>>>
testvol-client-0,
> >>>>>>>>>>>>>>>>>
attached to
> >>>>>>>>>>>>>>>>>
remote volume
> >>>>>>>>>>>>>>>>>
> '/export25/brick'.
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.960405]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client-handshake.c:1212:client_setvolume_cbk]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-0:
> >>>>>>>>>>>>>>>>>
Server and
> >>>>>>>>>>>>>>>>>
Client
> >>>>>>>>>>>>>>>>>
lk-version
> >>>>>>>>>>>>>>>>>
numbers are not
> >>>>>>>>>>>>>>>>>
same, reopening
> >>>>>>>>>>>>>>>>>
the fds
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.960471]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client-handshake.c:1200:client_setvolume_cbk]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-1:
> >>>>>>>>>>>>>>>>>
Connected to
> >>>>>>>>>>>>>>>>>
testvol-client-1,
> >>>>>>>>>>>>>>>>>
attached to
> >>>>>>>>>>>>>>>>>
remote volume
> >>>>>>>>>>>>>>>>>
> '/export25/brick'.
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.960478]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client-handshake.c:1212:client_setvolume_cbk]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-1:
> >>>>>>>>>>>>>>>>>
Server and
> >>>>>>>>>>>>>>>>>
Client
> >>>>>>>>>>>>>>>>>
lk-version
> >>>>>>>>>>>>>>>>>
numbers are not
> >>>>>>>>>>>>>>>>>
same, reopening
> >>>>>>>>>>>>>>>>>
the fds
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.962288]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[fuse-bridge.c:5042:fuse_graph_setup]
> >>>>>>>>>>>>>>>>>
0-fuse:
> >>>>>>>>>>>>>>>>>
switched to
> >>>>>>>>>>>>>>>>> graph
0
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.962351]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client-handshake.c:188:client_set_lk_version_cbk]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-1:
> >>>>>>>>>>>>>>>>>
Server lk
> >>>>>>>>>>>>>>>>>
version = 1
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.962362]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client-handshake.c:188:client_set_lk_version_cbk]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-0:
> >>>>>>>>>>>>>>>>>
Server lk
> >>>>>>>>>>>>>>>>>
version = 1
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:42:52.962424]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[fuse-bridge.c:3971:fuse_init]
> >>>>>>>>>>>>>>>>>
0-glusterfs-fuse:
> >>>>>>>>>>>>>>>>>
FUSE inited
> >>>>>>>>>>>>>>>>>
with protocol
> >>>>>>>>>>>>>>>>>
versions:
> >>>>>>>>>>>>>>>>>
glusterfs 7.22
> >>>>>>>>>>>>>>>>>
kernel 7.14
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:13.352234]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[glusterfsd-mgmt.c:56:mgmt_cbk_spec]
> >>>>>>>>>>>>>>>>>
0-mgmt: Volume
> >>>>>>>>>>>>>>>>>
file changed
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.518667]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[dht-shared.c:337:dht_init_regex]
> >>>>>>>>>>>>>>>>>
2-testvol-dht:
> >>>>>>>>>>>>>>>>>
using regex
> >>>>>>>>>>>>>>>>>
rsync-hash-regex
> > >>>>>>>>>>>>>>>>>
^\.(.+)\.[^.]+$
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.520034]
> W
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[graph.c:344:_log_if_unknown_option]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-quota: option
> >>>>>>>>>>>>>>>>>
'timeout' is
> >>>>>>>>>>>>>>>>>
not recognized
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.520091]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client.c:2280:notify]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-client-0:
> >>>>>>>>>>>>>>>>>
parent
> >>>>>>>>>>>>>>>>>
translators are
> >>>>>>>>>>>>>>>>>
ready,
> >>>>>>>>>>>>>>>>>
attempting
> >>>>>>>>>>>>>>>>>
connect on
> >>>>>>>>>>>>>>>>>
transport
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.524546]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client.c:2280:notify]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-client-1:
> >>>>>>>>>>>>>>>>>
parent
> >>>>>>>>>>>>>>>>>
translators are
> >>>>>>>>>>>>>>>>>
ready,
> >>>>>>>>>>>>>>>>>
attempting
> >>>>>>>>>>>>>>>>>
connect on
> >>>>>>>>>>>>>>>>>
transport
> >>>>>>>>>>>>>>>>>
Final graph:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
+------------------------------------------------------------------------------+
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
1: volume
> >>>>>>>>>>>>>>>>>
testvol-client-0
> >>>>>>>>>>>>>>>>>
2: type
> >>>>>>>>>>>>>>>>>
protocol/client
> >>>>>>>>>>>>>>>>>
3: option
> >>>>>>>>>>>>>>>>>
ping-timeout 42
> >>>>>>>>>>>>>>>>>
4: option
> >>>>>>>>>>>>>>>>>
remote-host
> >>>>>>>>>>>>>>>>>
gluster24
> >>>>>>>>>>>>>>>>>
5: option
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
remote-subvolume /export25/brick
> >>>>>>>>>>>>>>>>>
6: option
> >>>>>>>>>>>>>>>>>
transport-type
> >>>>>>>>>>>>>>>>>
socket
> >>>>>>>>>>>>>>>>>
7: option
> >>>>>>>>>>>>>>>>>
send-gids true
> >>>>>>>>>>>>>>>>>
8: end-volume
> >>>>>>>>>>>>>>>>>
9:
> >>>>>>>>>>>>>>>>>
10: volume
> >>>>>>>>>>>>>>>>>
testvol-client-1
> >>>>>>>>>>>>>>>>>
11: type
> >>>>>>>>>>>>>>>>>
protocol/client
> >>>>>>>>>>>>>>>>>
12: option
> >>>>>>>>>>>>>>>>>
ping-timeout 42
> >>>>>>>>>>>>>>>>>
13: option
> >>>>>>>>>>>>>>>>>
remote-host
> >>>>>>>>>>>>>>>>>
gluster25
> >>>>>>>>>>>>>>>>>
14: option
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
remote-subvolume /export25/brick
> >>>>>>>>>>>>>>>>>
15: option
> >>>>>>>>>>>>>>>>>
transport-type
> >>>>>>>>>>>>>>>>>
socket
> >>>>>>>>>>>>>>>>>
16: option
> >>>>>>>>>>>>>>>>>
send-gids true
> >>>>>>>>>>>>>>>>>
17: end-volume
> >>>>>>>>>>>>>>>>>
18:
> >>>>>>>>>>>>>>>>>
19: volume
> >>>>>>>>>>>>>>>>>
testvol-dht
> >>>>>>>>>>>>>>>>>
20: type
> >>>>>>>>>>>>>>>>>
> cluster/distribute
> >>>>>>>>>>>>>>>>>
21: subvolumes
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
testvol-client-0 testvol-client-1
> >>>>>>>>>>>>>>>>>
22: end-volume
> >>>>>>>>>>>>>>>>>
23:
> >>>>>>>>>>>>>>>>>
24: volume
> >>>>>>>>>>>>>>>>>
testvol-quota
> >>>>>>>>>>>>>>>>>
25: type
> >>>>>>>>>>>>>>>>>
features/quota
> >>>>>>>>>>>>>>>>>
26: option
> >>>>>>>>>>>>>>>>>
timeout 0
> >>>>>>>>>>>>>>>>>
27: option
> >>>>>>>>>>>>>>>>>
deem-statfs off
> >>>>>>>>>>>>>>>>>
28: subvolumes
> >>>>>>>>>>>>>>>>>
testvol-dht
> >>>>>>>>>>>>>>>>>
29: end-volume
> >>>>>>>>>>>>>>>>>
30:
> >>>>>>>>>>>>>>>>>
31: volume
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
testvol-write-behind
> >>>>>>>>>>>>>>>>>
32: type
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
performance/write-behind
> >>>>>>>>>>>>>>>>>
33: subvolumes
> >>>>>>>>>>>>>>>>>
testvol-quota
> >>>>>>>>>>>>>>>>>
34: end-volume
> >>>>>>>>>>>>>>>>>
35:
> >>>>>>>>>>>>>>>>>
36: volume
> >>>>>>>>>>>>>>>>>
> testvol-read-ahead
> >>>>>>>>>>>>>>>>>
37: type
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
performance/read-ahead
> >>>>>>>>>>>>>>>>>
38: subvolumes
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
testvol-write-behind
> >>>>>>>>>>>>>>>>>
39: end-volume
> >>>>>>>>>>>>>>>>>
40:
> >>>>>>>>>>>>>>>>>
41: volume
> >>>>>>>>>>>>>>>>>
testvol-io-cache
> >>>>>>>>>>>>>>>>>
42: type
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
performance/io-cache
> >>>>>>>>>>>>>>>>>
43: subvolumes
> >>>>>>>>>>>>>>>>>
> testvol-read-ahead
> >>>>>>>>>>>>>>>>>
44: end-volume
> >>>>>>>>>>>>>>>>>
45:
> >>>>>>>>>>>>>>>>>
46: volume
> >>>>>>>>>>>>>>>>>
> testvol-quick-read
> >>>>>>>>>>>>>>>>>
47: type
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
performance/quick-read
> >>>>>>>>>>>>>>>>>
48: subvolumes
> >>>>>>>>>>>>>>>>>
testvol-io-cache
> >>>>>>>>>>>>>>>>>
49: end-volume
> >>>>>>>>>>>>>>>>>
50:
> >>>>>>>>>>>>>>>>>
51: volume
> >>>>>>>>>>>>>>>>>
testvol-md-cache
> >>>>>>>>>>>>>>>>>
52: type
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
performance/md-cache
> >>>>>>>>>>>>>>>>>
53: subvolumes
> >>>>>>>>>>>>>>>>>
> testvol-quick-read
> >>>>>>>>>>>>>>>>>
54: end-volume
> >>>>>>>>>>>>>>>>>
55:
> >>>>>>>>>>>>>>>>>
56: volume
> >>>>>>>>>>>>>>>>>
testvol
> >>>>>>>>>>>>>>>>>
57: type
> >>>>>>>>>>>>>>>>>
debug/io-stats
> >>>>>>>>>>>>>>>>>
58: option
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
latency-measurement
> >>>>>>>>>>>>>>>>>
off
> >>>>>>>>>>>>>>>>>
59: option
> >>>>>>>>>>>>>>>>>
count-fop-hits
> off
> >>>>>>>>>>>>>>>>>
60: subvolumes
> >>>>>>>>>>>>>>>>>
testvol-md-cache
> >>>>>>>>>>>>>>>>>
61: end-volume
> >>>>>>>>>>>>>>>>>
62:
> >>>>>>>>>>>>>>>>>
63: volume
> >>>>>>>>>>>>>>>>>
meta-autoload
> >>>>>>>>>>>>>>>>>
64: type meta
> >>>>>>>>>>>>>>>>>
65: subvolumes
> >>>>>>>>>>>>>>>>>
testvol
> >>>>>>>>>>>>>>>>>
66: end-volume
> >>>>>>>>>>>>>>>>>
67:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
>
+------------------------------------------------------------------------------+
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.530005]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[rpc-clnt.c:1759:rpc_clnt_reconfig]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-client-1:
> >>>>>>>>>>>>>>>>>
changing port
> >>>>>>>>>>>>>>>>>
to 49155 (from 0)
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.530047]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[rpc-clnt.c:1759:rpc_clnt_reconfig]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-client-0:
> >>>>>>>>>>>>>>>>>
changing port
> >>>>>>>>>>>>>>>>>
to 49155 (from 0)
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.539062]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> [client-handshake.c:1415:select_server_supported_programs]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-client-1:
> >>>>>>>>>>>>>>>>>
Using Program
> >>>>>>>>>>>>>>>>>
GlusterFS 3.3,
> >>>>>>>>>>>>>>>>>
Num (1298437),
> >>>>>>>>>>>>>>>>>
Version (330)
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.539299]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> [client-handshake.c:1415:select_server_supported_programs]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-client-0:
> >>>>>>>>>>>>>>>>>
Using Program
> >>>>>>>>>>>>>>>>>
GlusterFS 3.3,
> >>>>>>>>>>>>>>>>>
Num (1298437),
> >>>>>>>>>>>>>>>>>
Version (330)
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.539462]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client-handshake.c:1200:client_setvolume_cbk]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-client-1:
> >>>>>>>>>>>>>>>>>
Connected to
> >>>>>>>>>>>>>>>>>
testvol-client-1,
> >>>>>>>>>>>>>>>>>
attached to
> >>>>>>>>>>>>>>>>>
remote volume
> >>>>>>>>>>>>>>>>>
> '/export25/brick'.
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.539485]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client-handshake.c:1212:client_setvolume_cbk]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-client-1:
> >>>>>>>>>>>>>>>>>
Server and
> >>>>>>>>>>>>>>>>>
Client
> >>>>>>>>>>>>>>>>>
lk-version
> >>>>>>>>>>>>>>>>>
numbers are not
> >>>>>>>>>>>>>>>>>
same, reopening
> >>>>>>>>>>>>>>>>>
the fds
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.539729]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client-handshake.c:1200:client_setvolume_cbk]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-client-0:
> >>>>>>>>>>>>>>>>>
Connected to
> >>>>>>>>>>>>>>>>>
testvol-client-0,
> >>>>>>>>>>>>>>>>>
attached to
> >>>>>>>>>>>>>>>>>
remote volume
> >>>>>>>>>>>>>>>>>
> '/export25/brick'.
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.539751]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client-handshake.c:1212:client_setvolume_cbk]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-client-0:
> >>>>>>>>>>>>>>>>>
Server and
> >>>>>>>>>>>>>>>>>
Client
> >>>>>>>>>>>>>>>>>
lk-version
> >>>>>>>>>>>>>>>>>
numbers are not
> >>>>>>>>>>>>>>>>>
same, reopening
> >>>>>>>>>>>>>>>>>
the fds
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.542878]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[fuse-bridge.c:5042:fuse_graph_setup]
> >>>>>>>>>>>>>>>>>
0-fuse:
> >>>>>>>>>>>>>>>>>
switched to
> >>>>>>>>>>>>>>>>> graph
2
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.542959]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client-handshake.c:188:client_set_lk_version_cbk]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-client-1:
> >>>>>>>>>>>>>>>>>
Server lk
> >>>>>>>>>>>>>>>>>
version = 1
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:47:15.542987]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client-handshake.c:188:client_set_lk_version_cbk]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-client-0:
> >>>>>>>>>>>>>>>>>
Server lk
> >>>>>>>>>>>>>>>>>
version = 1
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:48:04.586291]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client.c:2289:notify]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-0:
> >>>>>>>>>>>>>>>>>
current graph
> >>>>>>>>>>>>>>>>>
is no longer
> >>>>>>>>>>>>>>>>>
active,
> >>>>>>>>>>>>>>>>>
destroying
> >>>>>>>>>>>>>>>>>
rpc_client
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:48:04.586360]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client.c:2289:notify]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-1:
> >>>>>>>>>>>>>>>>>
current graph
> >>>>>>>>>>>>>>>>>
is no longer
> >>>>>>>>>>>>>>>>>
active,
> >>>>>>>>>>>>>>>>>
destroying
> >>>>>>>>>>>>>>>>>
rpc_client
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:48:04.586378]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client.c:2215:client_rpc_notify]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-0:
> >>>>>>>>>>>>>>>>>
disconnected
> >>>>>>>>>>>>>>>>>
from
> >>>>>>>>>>>>>>>>>
testvol-client-0.
> >>>>>>>>>>>>>>>>>
Client process
> >>>>>>>>>>>>>>>>>
will keep
> >>>>>>>>>>>>>>>>>
trying to
> >>>>>>>>>>>>>>>>>
connect to
> >>>>>>>>>>>>>>>>>
glusterd until
> >>>>>>>>>>>>>>>>>
brick's port is
> >>>>>>>>>>>>>>>>>
available
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:48:04.586430]
> I
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client.c:2215:client_rpc_notify]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
0-testvol-client-1:
> >>>>>>>>>>>>>>>>>
disconnected
> >>>>>>>>>>>>>>>>>
from
> >>>>>>>>>>>>>>>>>
testvol-client-1.
> >>>>>>>>>>>>>>>>>
Client process
> >>>>>>>>>>>>>>>>>
will keep
> >>>>>>>>>>>>>>>>>
trying to
> >>>>>>>>>>>>>>>>>
connect to
> >>>>>>>>>>>>>>>>>
glusterd until
> >>>>>>>>>>>>>>>>>
brick's port is
> >>>>>>>>>>>>>>>>>
available
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:48:04.589552]
> W
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client-rpc-fops.c:306:client3_3_mkdir_cbk]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-client-0:
> >>>>>>>>>>>>>>>>>
remote
> >>>>>>>>>>>>>>>>>
operation
> >>>>>>>>>>>>>>>>>
failed:
> >>>>>>>>>>>>>>>>>
Transport
> >>>>>>>>>>>>>>>>>
endpoint is not
> >>>>>>>>>>>>>>>>>
connected.
> >>>>>>>>>>>>>>>>>
Path: /test/a
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:48:04.589608]
> W
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[fuse-bridge.c:481:fuse_entry_cbk]
> >>>>>>>>>>>>>>>>>
0-glusterfs-fuse:
> >>>>>>>>>>>>>>>>>
78: MKDIR()
> >>>>>>>>>>>>>>>>>
/test/a => -1
> >>>>>>>>>>>>>>>>>
(Transport
> >>>>>>>>>>>>>>>>>
endpoint is not
> >>>>>>>>>>>>>>>>>
connected)
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:48:11.073349]
> W
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[client-rpc-fops.c:2212:client3_3_create_cbk]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2-testvol-client-1:
> >>>>>>>>>>>>>>>>>
remote
> >>>>>>>>>>>>>>>>>
operation
> >>>>>>>>>>>>>>>>>
failed:
> >>>>>>>>>>>>>>>>>
Transport
> >>>>>>>>>>>>>>>>>
endpoint is not
> >>>>>>>>>>>>>>>>>
connected.
> >>>>>>>>>>>>>>>>>
Path: /test/f
> >>>>>>>>>>>>>>>>>
[2015-03-20
> >>>>>>>>>>>>>>>>>
03:48:11.073419]
> W
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
[fuse-bridge.c:1937:fuse_create_cbk]
> >>>>>>>>>>>>>>>>>
0-glusterfs-fuse:
> >>>>>>>>>>>>>>>>>
82: /test/f =>
> >>>>>>>>>>>>>>>>>
-1 (Transport
> >>>>>>>>>>>>>>>>>
endpoint is not
> >>>>>>>>>>>>>>>>>
connected)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
2015-03-20
> >>>>>>>>>>>>>>>>>
11:27 GMT+09:00
> >>>>>>>>>>>>>>>>>
Vijaikumar M
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
<vmallika at redhat.com
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
<mailto:vmallika at redhat.com>>:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
Hi Kondo,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
Can you
> >>>>>>>>>>>>>>>>>
please
> >>>>>>>>>>>>>>>>>
provide all
> >>>>>>>>>>>>>>>>>
the
> >>>>>>>>>>>>>>>>>
glusterfs
> >>>>>>>>>>>>>>>>>
log files?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
Thanks,
> >>>>>>>>>>>>>>>>>
Vijay
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
On Friday
> >>>>>>>>>>>>>>>>>
20 March
> >>>>>>>>>>>>>>>>>
2015 07:33
> >>>>>>>>>>>>>>>>>
AM, K.Kondo
> >>>>>>>>>>>>>>>>>
wrote:
> >>>>>>>>>>>>>>>>>>
Hello,
> >>>>>>>>>>>>>>>>>>
experts
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
I had a
> >>>>>>>>>>>>>>>>>>
trouble
> >>>>>>>>>>>>>>>>>>
about quota.
> >>>>>>>>>>>>>>>>>>
I set
> >>>>>>>>>>>>>>>>>>
quota to
> >>>>>>>>>>>>>>>>>>
one
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
distributed volume
> >>>>>>>>>>>>>>>>>>
"vol12" as
> >>>>>>>>>>>>>>>>>>
bellow.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
gluster>
> >>>>>>>>>>>>>>>>>>
volume
> >>>>>>>>>>>>>>>>>>
quota
> >>>>>>>>>>>>>>>>>>
vol12 enable
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
volume
> >>>>>>>>>>>>>>>>>>
quota :
> >>>>>>>>>>>>>>>>>>
success
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
gluster>
> >>>>>>>>>>>>>>>>>>
volume
> >>>>>>>>>>>>>>>>>>
quota
> >>>>>>>>>>>>>>>>>>
vol12
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
limit-usage /test
> >>>>>>>>>>>>>>>>>>
10GB
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
volume
> >>>>>>>>>>>>>>>>>>
quota :
> >>>>>>>>>>>>>>>>>>
success
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
But I
> >>>>>>>>>>>>>>>>>>
couldn't
> >>>>>>>>>>>>>>>>>>
create a
> >>>>>>>>>>>>>>>>>>
file and
> >>>>>>>>>>>>>>>>>>
directory
> >>>>>>>>>>>>>>>>>>
with below
> >>>>>>>>>>>>>>>>>>
error
> >>>>>>>>>>>>>>>>>>
message.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
In a
> >>>>>>>>>>>>>>>>>>
client host,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
$cd test
> >>>>>>>>>>>>>>>>>>
(mounted
> >>>>>>>>>>>>>>>>>>
using fuse)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
$mkdir a
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
mkdir:
> >>>>>>>>>>>>>>>>>>
cannot
> >>>>>>>>>>>>>>>>>>
create
> >>>>>>>>>>>>>>>>>>
directory
> >>>>>>>>>>>>>>>>>>
`a':
> >>>>>>>>>>>>>>>>>>
Transport
> >>>>>>>>>>>>>>>>>>
endpoint
> >>>>>>>>>>>>>>>>>>
is not
> >>>>>>>>>>>>>>>>>>
connected
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> Additionally,
> >>>>>>>>>>>>>>>>>>
I couldn't
> >>>>>>>>>>>>>>>>>>
check
> >>>>>>>>>>>>>>>>>>
quota
> >>>>>>>>>>>>>>>>>>
status
> >>>>>>>>>>>>>>>>>>
using
> >>>>>>>>>>>>>>>>>>
gluster
> >>>>>>>>>>>>>>>>>>
command.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
gluster>
> >>>>>>>>>>>>>>>>>>
volume
> >>>>>>>>>>>>>>>>>>
quota
> >>>>>>>>>>>>>>>>>>
vol12 list
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
Path
> >>>>>>>>>>>>>>>>>>
Hard-limit
> >>>>>>>>>>>>>>>>>>
Soft-limit
> >>>>>>>>>>>>>>>>>>
Used
> >>>>>>>>>>>>>>>>>>
Available
> >>>>>>>>>>>>>>>>>>
Soft-limit
> >>>>>>>>>>>>>>>>>>
exceeded?
> >>>>>>>>>>>>>>>>>>
Hard-limit
> >>>>>>>>>>>>>>>>>>
exceeded?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
>
---------------------------------------------------------------------------------------------------------------------------
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
Here,
> >>>>>>>>>>>>>>>>>>
this
> >>>>>>>>>>>>>>>>>>
command
> >>>>>>>>>>>>>>>>>>
stops, so
> >>>>>>>>>>>>>>>>>>
I have to
> >>>>>>>>>>>>>>>>>>
do Ctrl-C.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
Gluster
> >>>>>>>>>>>>>>>>>>
version is
> >>>>>>>>>>>>>>>>>>
3.6.1 and
> >>>>>>>>>>>>>>>>>>
3.6.0.29
> >>>>>>>>>>>>>>>>>>
for server
> >>>>>>>>>>>>>>>>>>
and client
> >>>>>>>>>>>>>>>>>>
> respectively.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
Any idea
> >>>>>>>>>>>>>>>>>>
for this?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
Best
> regards,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
K. Kondo
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
_______________________________________________
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
Gluster-users mailing list
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
Gluster-users at gluster.org
> >>>>>>>>>>>>>>>>>>
<mailto:Gluster-users at gluster.org>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
http://www.gluster.org/mailman/listinfo/gluster-users
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >
> >
> >
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> >
>
> --
> ~Atin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150421/2272da1f/attachment.html>