thr3ads.net - Gluster users - [Gluster-users] gluster 3.7.3 - volume heal info hangs

If this information is useful, please help other people find it:
Share via:

Andreas Mather

2015-Sep-24 13:29 UTC

[Gluster-users] gluster 3.7.3 - volume heal info hangs - unknown heal status

Hi Anuradha!

Thanks for your reply! Attached you can find the dump files. As I'm not
sure if they make their way through as attachments, here're links to them
as well:

brick1 - http://pastebin.com/3ivkhuRH
brick2 - http://pastebin.com/77sT1mut

- Andreas




On Thu, Sep 24, 2015 at 3:18 PM, Anuradha Talur <atalur at redhat.com>
wrote:
>
>
> ----- Original Message -----
> > From: "Andreas Mather" <andreas at allaboutapps.at>
> > To: "Gluster-users at gluster.org List" <gluster-users at
gluster.org>
> > Sent: Thursday, September 24, 2015 1:24:12 PM
> > Subject: [Gluster-users] gluster 3.7.3 - volume heal info hangs -
> unknown     heal status
> >
> > Hi!
> >
> > Our provider had network maintenance this night, so 2 of our 4 servers
> got
> > disconnected and reconnected. Since we knew this was coming, we
shifted
> all
> > work load off the affected servers. This morning, most of the cluster
> seems
> > fine, but for one volume, no heal info can be retrieved, so we
basically
> > don't know about the healing state of the volume. The volume is a
> replica 2
> > volume between vhost4-int/brick1 and vhost3-int/brick2.
> >
> > The volume is accessible, but since I don't get any heal info, I
don't
> know
> > if it is probably replicated. Any help to resolve this situation is
> highly
> > appreciated.
> >
> > hangs forever:
> > [root at vhost4 ~]# gluster volume heal vol4 info
> >
> > glfsheal-vol4.log:
> > [2015-09-24 07:47:59.284723] I [MSGID: 101190]
> > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started
thread
> with
> > index 1
> > [2015-09-24 07:47:59.293735] I [MSGID: 101190]
> > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started
thread
> with
> > index 2
> > [2015-09-24 07:47:59.294061] I [MSGID: 104045]
[glfs-master.c:95:notify]
> > 0-gfapi: New graph 76686f73-7434-2e61-6c6c-61626f757461 (0) coming up
> > [2015-09-24 07:47:59.294081] I [MSGID: 114020] [client.c:2118:notify]
> > 0-vol4-client-1: parent translators are ready, attempting connect on
> > transport
> > [2015-09-24 07:47:59.309470] I [MSGID: 114020] [client.c:2118:notify]
> > 0-vol4-client-2: parent translators are ready, attempting connect on
> > transport
> > [2015-09-24 07:47:59.310525] I [rpc-clnt.c:1819:rpc_clnt_reconfig]
> > 0-vol4-client-1: changing port to 49155 (from 0)
> > [2015-09-24 07:47:59.315958] I [MSGID: 114057]
> > [client-handshake.c:1437:select_server_supported_programs]
> 0-vol4-client-1:
> > Using Program GlusterFS 3.3, Num (1298437), Version (330)
> > [2015-09-24 07:47:59.316481] I [MSGID: 114046]
> > [client-handshake.c:1213:client_setvolume_cbk] 0-vol4-client-1:
> Connected to
> > vol4-client-1, attached to remote volume
'/storage/brick2/brick2'.
> > [2015-09-24 07:47:59.316495] I [MSGID: 114047]
> > [client-handshake.c:1224:client_setvolume_cbk] 0-vol4-client-1: Server
> and
> > Client lk-version numbers are not same, reopening the fds
> > [2015-09-24 07:47:59.316538] I [MSGID: 108005]
> [afr-common.c:3960:afr_notify]
> > 0-vol4-replicate-0: Subvolume 'vol4-client-1' came back up;
going online.
> > [2015-09-24 07:47:59.317150] I [MSGID: 114035]
> > [client-handshake.c:193:client_set_lk_version_cbk] 0-vol4-client-1:
> Server
> > lk version = 1
> > [2015-09-24 07:47:59.320898] I [rpc-clnt.c:1819:rpc_clnt_reconfig]
> > 0-vol4-client-2: changing port to 49154 (from 0)
> > [2015-09-24 07:47:59.325633] I [MSGID: 114057]
> > [client-handshake.c:1437:select_server_supported_programs]
> 0-vol4-client-2:
> > Using Program GlusterFS 3.3, Num (1298437), Version (330)
> > [2015-09-24 07:47:59.325780] I [MSGID: 114046]
> > [client-handshake.c:1213:client_setvolume_cbk] 0-vol4-client-2:
> Connected to
> > vol4-client-2, attached to remote volume
'/storage/brick1/brick1'.
> > [2015-09-24 07:47:59.325791] I [MSGID: 114047]
> > [client-handshake.c:1224:client_setvolume_cbk] 0-vol4-client-2: Server
> and
> > Client lk-version numbers are not same, reopening the fds
> > [2015-09-24 07:47:59.333346] I [MSGID: 114035]
> > [client-handshake.c:193:client_set_lk_version_cbk] 0-vol4-client-2:
> Server
> > lk version = 1
> > [2015-09-24 07:47:59.334545] I [MSGID: 108031]
> > [afr-common.c:1745:afr_local_discovery_cbk] 0-vol4-replicate-0:
selecting
> > local read_child vol4-client-2
> > [2015-09-24 07:47:59.335833] I [MSGID: 104041]
> > [glfs-resolve.c:862:__glfs_active_subvol] 0-vol4: switched to graph
> > 76686f73-7434-2e61-6c6c-61626f757461 (0)
> >
> > Questions to this output:
> > -) Why does it report " Using Program GlusterFS 3.3, Num
(1298437),
> Version
> > (330) ". We run 3.7.3 ?!
> > -) guster logs timestamps in UTC not taking server timezone into
> account. Is
> > there a way to fix this?
> >
> > etc-glusterfs-glusterd.vol.log:
> > no logs to after volume heal info command
> >
> > storage-brick1-brick1.log:
> > [2015-09-24 07:47:59.325720] I [login.c:81:gf_auth] 0-auth/login:
allowed
> > user names: 67ef1559-d3a1-403a-b8e7-fb145c3acf4e
> > [2015-09-24 07:47:59.325743] I [MSGID: 115029]
> > [server-handshake.c:610:server_setvolume] 0-vol4-server: accepted
client
> > from
> >
vhost4.allaboutapps.at-14900-2015/09/24-07:47:59:282313-vol4-client-2-0-0
> > (version: 3.7.3)
> >
> > storage-brick2-brick2.log:
> > no logs to after volume heal info command
> >
> >
> Hi Andreas,
>
> Could you please provide the following information so that we can
> understand why the command is hanging?
> When the command is hung, run the following command from one of the
> servers:
> `gluster volume statedump <volname>`
> This command will generate statedumps of glusterfsd processes in the
> servers. You can find them at /var/run/gluster . A typical statedump for a
> brick has
"<brick-path>.<pid-of-brick>.dump.<timestamp>" as
its name. Could
> you please attach them and respond?
>
> > Thanks,
> >
> > - Andreas
> >
> >
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
>
> --
> Thanks,
> Anuradha.
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150924/ce1a81c5/attachment.html>
-------------- next part --------------
DUMP-START-TIME: 2015-09-24 13:19:53.350649

[mallinfo]
mallinfo_arena=5488640
mallinfo_ordblks=399
mallinfo_smblks=237
mallinfo_hblks=21
mallinfo_hblkhd=30515200
mallinfo_usmblks=0
mallinfo_fsmblks=24400
mallinfo_uordblks=1774464
mallinfo_fordblks=3714176
mallinfo_keepcost=115808

[global.glusterfs - Memory usage]
num_types=146

[global.glusterfs - usage-type gf_common_mt_dnscache6 memusage]
size=16
num_allocs=1
max_size=16
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_event_pool memusage]
size=12712
num_allocs=2
max_size=12712
max_num_allocs=2
total_allocs=2

[global.glusterfs - usage-type gf_common_mt_xlator_t memusage]
size=44880
num_allocs=17
max_size=44880
max_num_allocs=17
total_allocs=17

[global.glusterfs - usage-type gf_common_mt_xlator_list_t memusage]
size=512
num_allocs=32
max_size=512
max_num_allocs=32
total_allocs=32

[global.glusterfs - usage-type gf_common_mt_volume_opt_list_t memusage]
size=456
num_allocs=19
max_size=456
max_num_allocs=19
total_allocs=19

[global.glusterfs - usage-type gf_common_mt_gf_timer_t memusage]
size=112
num_allocs=2
max_size=168
max_num_allocs=3
total_allocs=262064

[global.glusterfs - usage-type gf_common_mt_gf_timer_registry_t memusage]
size=168
num_allocs=1
max_size=168
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_iobuf memusage]
size=109536
num_allocs=8
max_size=109536
max_num_allocs=8
total_allocs=8

[global.glusterfs - usage-type gf_common_mt_iobuf_arena memusage]
size=1872
num_allocs=9
max_size=1872
max_num_allocs=9
total_allocs=9

[global.glusterfs - usage-type gf_common_mt_iobuf_pool memusage]
size=1776
num_allocs=1
max_size=1776
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_asprintf memusage]
size=257
num_allocs=12
max_size=362
max_num_allocs=14
total_allocs=60

[global.glusterfs - usage-type gf_common_mt_strdup memusage]
size=2382
num_allocs=105
max_size=3561
max_num_allocs=125
total_allocs=5734

[global.glusterfs - usage-type gf_common_mt_socket_private_t memusage]
size=1344
num_allocs=3
max_size=1344
max_num_allocs=3
total_allocs=3

[global.glusterfs - usage-type gf_common_mt_char memusage]
size=4135
num_allocs=57
max_size=24820
max_num_allocs=90
total_allocs=1737

[global.glusterfs - usage-type gf_common_mt_mem_pool memusage]
size=1200
num_allocs=10
max_size=1200
max_num_allocs=10
total_allocs=10

[global.glusterfs - usage-type gf_common_mt_long memusage]
size=8911520
num_allocs=10
max_size=8911520
max_num_allocs=10
total_allocs=10

[global.glusterfs - usage-type gf_common_mt_rpcsvc_auth_list memusage]
size=288
num_allocs=4
max_size=288
max_num_allocs=4
total_allocs=4

[global.glusterfs - usage-type gf_common_mt_rpcsvc_t memusage]
size=216
num_allocs=1
max_size=216
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_rpcsvc_program_t memusage]
size=240
num_allocs=2
max_size=240
max_num_allocs=2
total_allocs=2

[global.glusterfs - usage-type gf_common_mt_rpcsvc_listener_t memusage]
size=160
num_allocs=1
max_size=160
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_rpcsvc_wrapper_t memusage]
size=32
num_allocs=1
max_size=32
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_rpcclnt_t memusage]
size=312
num_allocs=1
max_size=312
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_rpcclnt_savedframe_t memusage]
size=200
num_allocs=1
max_size=200
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_rpc_trans_t memusage]
size=8352
num_allocs=3
max_size=8352
max_num_allocs=3
total_allocs=3

[global.glusterfs - usage-type gf_common_mt_glusterfs_graph_t memusage]
size=200
num_allocs=1
max_size=200
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_rpcclnt_cb_program_t memusage]
size=88
num_allocs=1
max_size=88
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_cliententry_t memusage]
size=2048
num_allocs=1
max_size=2048
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_clienttable_t memusage]
size=32
num_allocs=1
max_size=32
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_ereg memusage]
size=49152
num_allocs=1
max_size=49152
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_syncenv memusage]
size=15528
num_allocs=1
max_size=15528
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_sock_mt_lock_array memusage]
size=1640
num_allocs=1
max_size=1640
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gfd_mt_server_cmdline_t memusage]
size=40
num_allocs=1
max_size=40
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gfd_mt_xlator_cmdline_option_t memusage]
size=80
num_allocs=2
max_size=80
max_num_allocs=2
total_allocs=2

[global.glusterfs - usage-type gfd_mt_char memusage]
size=48
num_allocs=4
max_size=48
max_num_allocs=4
total_allocs=4

[global.glusterfs - usage-type gfd_mt_call_pool_t memusage]
size=48
num_allocs=1
max_size=48
max_num_allocs=1
total_allocs=1

[mempool]
-----=-----
pool-name=vol4-server:fd_t
hot-count=15
cold-count=1009
padded_sizeof=108
alloc-count=614205
max-alloc=46
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-server:dentry_t
hot-count=33
cold-count=16351
padded_sizeof=84
alloc-count=55
max-alloc=35
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-server:inode_t
hot-count=35
cold-count=16349
padded_sizeof=156
alloc-count=3701
max-alloc=37
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-trash:fd_t
hot-count=0
cold-count=1024
padded_sizeof=108
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-trash:dentry_t
hot-count=0
cold-count=32768
padded_sizeof=84
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-trash:inode_t
hot-count=4
cold-count=32764
padded_sizeof=156
alloc-count=4
max-alloc=4
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-trash:trash_local_t
hot-count=0
cold-count=64
padded_sizeof=8628
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-changetimerecorder:gf_ctr_local_t
hot-count=0
cold-count=64
padded_sizeof=16540
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-changelog:rpcsvc_request_t
hot-count=0
cold-count=8
padded_sizeof=2828
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-changelog:changelog_local_t
hot-count=0
cold-count=64
padded_sizeof=116
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-bitrot-stub:br_stub_local_t
hot-count=0
cold-count=512
padded_sizeof=84
alloc-count=43
max-alloc=2
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-locks:pl_local_t
hot-count=0
cold-count=32
padded_sizeof=148
alloc-count=481323
max-alloc=15
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-upcall:upcall_local_t
hot-count=0
cold-count=512
padded_sizeof=108
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-marker:marker_local_t
hot-count=0
cold-count=128
padded_sizeof=332
alloc-count=5232
max-alloc=1
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-quota:quota_local_t
hot-count=0
cold-count=64
padded_sizeof=476
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-server:rpcsvc_request_t
hot-count=0
cold-count=512
padded_sizeof=2828
alloc-count=94634730
max-alloc=118
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:struct saved_frame
hot-count=0
cold-count=8
padded_sizeof=124
alloc-count=7
max-alloc=2
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:struct rpc_req
hot-count=0
cold-count=8
padded_sizeof=588
alloc-count=7
max-alloc=2
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:rpcsvc_request_t
hot-count=0
cold-count=8
padded_sizeof=2828
alloc-count=52
max-alloc=1
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:log_buf_t
hot-count=1
cold-count=255
padded_sizeof=140
alloc-count=1367
max-alloc=7
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:data_t
hot-count=62
cold-count=16322
padded_sizeof=52
alloc-count=212015133
max-alloc=275
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:data_pair_t
hot-count=45
cold-count=16339
padded_sizeof=68
alloc-count=82432080
max-alloc=231
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:dict_t
hot-count=21
cold-count=4075
padded_sizeof=140
alloc-count=70306347
max-alloc=140
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:call_stub_t
hot-count=0
cold-count=1024
padded_sizeof=3764
alloc-count=115110858
max-alloc=119
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:call_stack_t
hot-count=0
cold-count=1024
padded_sizeof=1708
alloc-count=94441912
max-alloc=118
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:call_frame_t
hot-count=0
cold-count=4096
padded_sizeof=172
alloc-count=565746952
max-alloc=595
pool-misses=0
cur-stdalloc=0
max-stdalloc=0

[iobuf.global]
iobuf_pool=0x7fed67a30360
iobuf_pool.default_page_size=131072
iobuf_pool.arena_size=12976128
iobuf_pool.arena_cnt=8
iobuf_pool.request_misses=0

[purge.1]
purge.1.mem_base=0x7fed66a10000
purge.1.active_cnt=0
purge.1.passive_cnt=1024
purge.1.alloc_cnt=173428399
purge.1.max_active=32
purge.1.page_size=128

[purge.2]
purge.2.mem_base=0x7fed669d0000
purge.2.active_cnt=0
purge.2.passive_cnt=512
purge.2.alloc_cnt=93650111
purge.2.max_active=8
purge.2.page_size=512

[purge.3]
purge.3.mem_base=0x7fed668d0000
purge.3.active_cnt=0
purge.3.passive_cnt=512
purge.3.alloc_cnt=1039934
purge.3.max_active=6
purge.3.page_size=2048

[purge.4]
purge.4.mem_base=0x7fed64b32000
purge.4.active_cnt=0
purge.4.passive_cnt=128
purge.4.alloc_cnt=14148832
purge.4.max_active=11
purge.4.page_size=8192

[purge.5]
purge.5.mem_base=0x7fed64932000
purge.5.active_cnt=0
purge.5.passive_cnt=64
purge.5.alloc_cnt=12274302
purge.5.max_active=7
purge.5.page_size=32768

[purge.6]
purge.6.mem_base=0x7fed64532000
purge.6.active_cnt=0
purge.6.passive_cnt=32
purge.6.alloc_cnt=17562156
purge.6.max_active=20
purge.6.page_size=131072

[purge.7]
purge.7.mem_base=0x7fed64332000
purge.7.active_cnt=0
purge.7.passive_cnt=8
purge.7.alloc_cnt=1324877
purge.7.max_active=4
purge.7.page_size=262144

[purge.8]
purge.8.mem_base=0x7fed64132000
purge.8.active_cnt=0
purge.8.passive_cnt=2
purge.8.alloc_cnt=28
purge.8.max_active=1
purge.8.page_size=1048576

[global.callpool]
callpool_address=0x7fed67a4f3f0
callpool.cnt=0

[active graph - 1]

[protocol/server.vol4-server - Memory usage]
num_types=151

[protocol/server.vol4-server - usage-type gf_common_mt_event_pool memusage]
size=16
num_allocs=1
max_size=16
max_num_allocs=1
total_allocs=1

[protocol/server.vol4-server - usage-type gf_common_mt_fdentry_t memusage]
size=26624
num_allocs=13
max_size=32768
max_num_allocs=16
total_allocs=1345

[protocol/server.vol4-server - usage-type gf_common_mt_fdtable_t memusage]
size=832
num_allocs=13
max_size=960
max_num_allocs=15
total_allocs=457

[protocol/server.vol4-server - usage-type gf_common_mt_fd_ctx memusage]
size=4320
num_allocs=15
max_size=13248
max_num_allocs=46
total_allocs=613920

[protocol/server.vol4-server - usage-type gf_common_mt_inode_ctx memusage]
size=9072
num_allocs=21
max_size=9936
max_num_allocs=23
total_allocs=2322

[protocol/server.vol4-server - usage-type gf_common_mt_list_head memusage]
size=1273488
num_allocs=2
max_size=1273488
max_num_allocs=2
total_allocs=2

[protocol/server.vol4-server - usage-type gf_common_mt_inode_table_t memusage]
size=200
num_allocs=1
max_size=200
max_num_allocs=1
total_allocs=1

[protocol/server.vol4-server - usage-type gf_common_mt_volume_opt_list_t
memusage]
size=72
num_allocs=3
max_size=72
max_num_allocs=3
total_allocs=3

[protocol/server.vol4-server - usage-type gf_common_mt_auth_handle_t memusage]
size=48
num_allocs=2
max_size=48
max_num_allocs=2
total_allocs=2

[protocol/server.vol4-server - usage-type gf_common_mt_asprintf memusage]
size=116
num_allocs=5
max_size=562
max_num_allocs=9
total_allocs=10319

[protocol/server.vol4-server - usage-type gf_common_mt_strdup memusage]
size=35628
num_allocs=979
max_size=35665
max_num_allocs=981
total_allocs=14338333

[protocol/server.vol4-server - usage-type gf_common_mt_socket_private_t
memusage]
size=6272
num_allocs=14
max_size=7168
max_num_allocs=16
total_allocs=470

[protocol/server.vol4-server - usage-type gf_common_mt_char memusage]
size=102
num_allocs=6
max_size=12792
max_num_allocs=130
total_allocs=163893301

[protocol/server.vol4-server - usage-type gf_common_mt_mem_pool memusage]
size=480
num_allocs=4
max_size=480
max_num_allocs=4
total_allocs=4

[protocol/server.vol4-server - usage-type gf_common_mt_long memusage]
size=5490688
num_allocs=4
max_size=5490688
max_num_allocs=4
total_allocs=4

[protocol/server.vol4-server - usage-type gf_common_mt_rpcsvc_auth_list
memusage]
size=288
num_allocs=4
max_size=288
max_num_allocs=4
total_allocs=4

[protocol/server.vol4-server - usage-type gf_common_mt_rpcsvc_t memusage]
size=216
num_allocs=1
max_size=216
max_num_allocs=1
total_allocs=1

[protocol/server.vol4-server - usage-type gf_common_mt_rpcsvc_program_t
memusage]
size=360
num_allocs=3
max_size=360
max_num_allocs=3
total_allocs=3

[protocol/server.vol4-server - usage-type gf_common_mt_rpcsvc_listener_t
memusage]
size=160
num_allocs=1
max_size=160
max_num_allocs=1
total_allocs=1

[protocol/server.vol4-server - usage-type gf_common_mt_rpcsvc_wrapper_t
memusage]
size=32
num_allocs=1
max_size=96
max_num_allocs=3
total_allocs=457

[protocol/server.vol4-server - usage-type gf_common_mt_rpc_trans_t memusage]
size=38976
num_allocs=14
max_size=44544
max_num_allocs=16
total_allocs=470

[protocol/server.vol4-server - usage-type gf_common_mt_fd_lk_ctx_t memusage]
size=360
num_allocs=15
max_size=1104
max_num_allocs=46
total_allocs=613920

[protocol/server.vol4-server - usage-type gf_common_mt_client_t memusage]
size=1664
num_allocs=26
max_size=1920
max_num_allocs=30
total_allocs=914

[protocol/server.vol4-server - usage-type gf_common_mt_client_ctx memusage]
size=1664
num_allocs=13
max_size=1920
max_num_allocs=15
total_allocs=457

[protocol/server.vol4-server - usage-type gf_server_mt_server_conf_t memusage]
size=49744
num_allocs=14
max_size=49808
max_num_allocs=16
total_allocs=458

[xlator.protocol.server.priv]
server.total-bytes-read=436940921558
server.total-bytes-write=1602403362372
conn.1.bound_xl./storage/brick1/brick1.hashsize=14057
conn.1.bound_xl./storage/brick1/brick1.name=/storage/brick1/brick1/inode
conn.1.bound_xl./storage/brick1/brick1.lru_limit=16384
conn.1.bound_xl./storage/brick1/brick1.active_size=17
conn.1.bound_xl./storage/brick1/brick1.lru_size=18
conn.1.bound_xl./storage/brick1/brick1.purge_size=0

[conn.1.bound_xl./storage/brick1/brick1.active.1]
gfid=8afa4e92-6ba5-45f2-b7f6-c64851cee04b
nlookup=4322
fd-count=1
ref=1
ia_type=2

[conn.1.bound_xl./storage/brick1/brick1.active.2]
gfid=3141a578-6198-49d6-b462-ee66a5bdfde8
nlookup=90
fd-count=1
ref=1
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/shoperience-dev.qcow2
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.active.3]
gfid=f1028e08-1911-4757-b6ae-790893a49f6d
nlookup=275
fd-count=0
ref=3
ia_type=2

[xlator.features.locks.vol4-locks.inode]
path=/configs/old
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.active.4]
gfid=db3b8cd8-eb14-4ac8-9fd3-9914cc044a9c
nlookup=164
fd-count=2
ref=3
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/avnet-dev.qcow2
mandatory=0
inodelk-count=1
lock-dump.domain.domain=vol4-replicate-0:self-heal
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid =
18446744073709551610, owner=74580429ce7f0000, client=0x7fed540ba720,
connection-id=vhost4.allaboutapps.at-4704-2015/08/27-13:19:02:16249-vol4-client-2-0-0,
granted at 2015-09-23 22:18:37
lock-dump.domain.domain=vol4-replicate-0:metadata
lock-dump.domain.domain=vol4-replicate-0

[conn.1.bound_xl./storage/brick1/brick1.active.5]
gfid=0b1a02a4-7e97-4504-9f26-b6313691fec0
nlookup=170
fd-count=1
ref=1
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/reports-dev-1.qcow2
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.active.6]
gfid=5c34060b-43fe-409e-8c14-e9aa8b144910
nlookup=158
fd-count=1
ref=1
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/playground-dev-1.qcow2
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.active.7]
gfid=45f5abc5-69c0-4641-9b09-dcd7eeedc37f
nlookup=196
fd-count=2
ref=3
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/wuestenrot-dev.qcow2
mandatory=0
inodelk-count=1
lock-dump.domain.domain=vol4-replicate-0:self-heal
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid =
18446744073709551610, owner=204fba39be7f0000, client=0x7fed4c009e40,
connection-id=vhost3.allaboutapps.at-3643-2015/09/10-10:10:42:68099-vol4-client-2-0-1,
granted at 2015-09-23 22:26:50
lock-dump.domain.domain=vol4-replicate-0:metadata
lock-dump.domain.domain=vol4-replicate-0

[conn.1.bound_xl./storage/brick1/brick1.active.8]
gfid=97a5c60b-dd69-483f-a809-081ead34dc8b
nlookup=207
fd-count=1
ref=2
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/porsche-staging-1.qcow2
mandatory=0
inodelk-count=1
lock-dump.domain.domain=vol4-replicate-0:self-heal
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 19585,
owner=814c0000, client=0x7fed540f0da0,
connection-id=vhost4.allaboutapps.at-19585-2015/09/24-13:19:41:672689-vol4-client-2-0-0,
granted at 2015-09-24 13:19:43
lock-dump.domain.domain=vol4-replicate-0:metadata
lock-dump.domain.domain=vol4-replicate-0

[conn.1.bound_xl./storage/brick1/brick1.active.9]
gfid=5eadf1c9-b7d9-45e4-aff9-4a9ab8843be4
nlookup=201
fd-count=1
ref=1
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/haas-sohn-dev-1.qcow2
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.active.10]
gfid=5e27a281-1ace-4aa6-a6d2-1bdde1a38eb7
nlookup=215
fd-count=1
ref=1
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/novartis-staging.qcow2
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.active.11]
gfid=0dd0cff5-0070-45f2-8fd9-8aa647f1e7b9
nlookup=206
fd-count=1
ref=1
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/novartis-dev.qcow2
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.active.12]
gfid=3d41c61f-1f4c-462b-b8d8-0ffcc4f51932
nlookup=224
fd-count=1
ref=1
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/krebshilfe.qcow2
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.active.13]
gfid=66afe389-ecff-40de-8bb5-8ed9489b37b5
nlookup=732
fd-count=1
ref=1
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/guntamatic-dev-1.img.qcow2
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.active.14]
gfid=1c7a3140-881e-461d-9adc-475f49d68604
nlookup=1393
fd-count=1
ref=1
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/win7.qcow2
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.active.15]
gfid=73e12d91-615f-4383-9bb9-ed67d5bb467c
nlookup=3744
fd-count=0
ref=13
ia_type=2

[xlator.features.locks.vol4-locks.inode]
path=/configs
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.active.16]
gfid=00000000-0000-0000-0000-000000000005
nlookup=4231
fd-count=0
ref=1
ia_type=2

[xlator.features.locks.vol4-locks.inode]
path=/.trashcan
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.active.17]
gfid=00000000-0000-0000-0000-000000000001
nlookup=4914
fd-count=0
ref=1
ia_type=2

[xlator.features.locks.vol4-locks.inode]
path=/
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.1]
gfid=b6d5ec08-fcee-4c98-b8cd-f7519c3f551d
nlookup=620
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/old/abhof-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.2]
gfid=6abf53a9-6a02-430f-9991-b84935bf61b2
nlookup=620
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/old/abhof-staging.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.3]
gfid=7bc8905d-6fa4-4bbc-a639-222877f3e2e3
nlookup=1475
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/old/cashquizz-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.4]
gfid=00000000-0000-0000-0000-000000000006
nlookup=2366
fd-count=0
ref=0
ia_type=2

[xlator.features.locks.vol4-locks.inode]
path=/.trashcan/internal_op
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.5]
gfid=47fc7997-3792-406b-a6ac-dcf644ce662f
nlookup=1562
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/guntamatic-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.6]
gfid=6e63d4b2-89be-4d88-818f-6215d8dde0e5
nlookup=1562
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/haas-sohn-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.7]
gfid=409425ac-a11f-4252-bbc7-11a3ffa43a79
nlookup=1562
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/novartis-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.8]
gfid=b7382d70-f4b2-4449-975c-6f9c6f112ce2
nlookup=1562
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/novartis-staging.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.9]
gfid=bac5626f-ad39-4102-93d2-e9c4a1a20ad4
nlookup=1562
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/playground-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.10]
gfid=ebec8823-19f6-4e87-9e71-305cc178dd3f
nlookup=1562
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/porsche-staging.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.11]
gfid=1f6ce2ab-f9c0-4d9a-aad0-03726452ede0
nlookup=1562
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/reports-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.12]
gfid=bbc52dbe-136d-4559-b177-8ff800df4b5f
nlookup=1562
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/wuestenrot-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.13]
gfid=887e107c-27a4-4906-979a-c5ee8a899965
nlookup=1563
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/avnet-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.14]
gfid=6f5ce908-1a73-43af-909d-26073906f7fd
nlookup=634
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/win7.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.15]
gfid=7bb24cb9-1f8c-471a-a874-a31699abd6c2
nlookup=163
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/krebshilfe.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.16]
gfid=2b36054f-eda1-4ec9-8216-d9d7a260bf18
nlookup=11
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/shoperience-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick1/brick1.lru.17]
gfid=89f50322-270c-4a66-9275-813f8005b12d
nlookup=3334
fd-count=0
ref=0
ia_type=2

[conn.1.bound_xl./storage/brick1/brick1.lru.18]
gfid=c75d90b1-a4e8-477d-9b83-a6547f88c7a8
nlookup=2615
fd-count=0
ref=0
ia_type=2
conn.1.id=vhost4.allaboutapps.at-4702-2015/08/27-13:19:02:14144-vol4-client-2-0-0
conn.1.ref=1
conn.1.bound_xl=/storage/brick1/brick1
conn.2.id=vhost3.allaboutapps.at-4030-2015/09/10-10:11:10:614419-vol4-client-2-0-1
conn.2.ref=1
conn.2.bound_xl=/storage/brick1/brick1
conn.3.id=vhost3.allaboutapps.at-3643-2015/09/10-10:10:42:68099-vol4-client-2-0-1
conn.3.ref=1
conn.3.bound_xl=/storage/brick1/brick1
conn.4.id=vhost1.allaboutapps.at-4298-2015/09/10-07:12:25:821602-vol4-client-2-0-3
conn.4.ref=1
conn.4.bound_xl=/storage/brick1/brick1
conn.5.id=vhost1.allaboutapps.at-3924-2015/09/10-07:11:57:288087-vol4-client-2-0-3
conn.5.ref=1
conn.5.bound_xl=/storage/brick1/brick1
conn.6.id=vhost4.allaboutapps.at-19585-2015/09/24-13:19:41:672689-vol4-client-2-0-0
conn.6.ref=1
conn.6.bound_xl=/storage/brick1/brick1
conn.7.id=vhost1.allaboutapps.at-3916-2015/09/10-07:11:56:285421-vol4-client-2-0-4
conn.7.ref=1
conn.7.bound_xl=/storage/brick1/brick1
conn.8.id=vhost2.allaboutapps.at-3062-2015/09/10-07:59:06:556223-vol4-client-2-0-2
conn.8.ref=1
conn.8.bound_xl=/storage/brick1/brick1
conn.9.id=vhost2.allaboutapps.at-3064-2015/09/10-07:59:06:558627-vol4-client-2-0-2
conn.9.ref=1
conn.9.bound_xl=/storage/brick1/brick1
conn.10.id=vhost2.allaboutapps.at-3630-2015/09/10-07:59:36:520290-vol4-client-2-0-0
conn.10.ref=1
conn.10.bound_xl=/storage/brick1/brick1
conn.11.id=vhost3.allaboutapps.at-3641-2015/09/10-10:10:42:65055-vol4-client-2-0-1
conn.11.ref=1
conn.11.bound_xl=/storage/brick1/brick1
conn.12.id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-2-0-0
conn.12.ref=1
conn.12.bound_xl=/storage/brick1/brick1

[debug/io-stats./storage/brick1/brick1 - Memory usage]
num_types=145

[debug/io-stats./storage/brick1/brick1 - usage-type gf_common_mt_strdup
memusage]
size=1445
num_allocs=55
max_size=1529
max_num_allocs=58
total_allocs=1410

[debug/io-stats./storage/brick1/brick1 - usage-type gf_io_stats_mt_ios_conf
memusage]
size=5504
num_allocs=1
max_size=5504
max_num_allocs=1
total_allocs=1

[debug/io-stats./storage/brick1/brick1 - usage-type gf_io_stats_mt_ios_fd
memusage]
size=7728
num_allocs=14
max_size=9384
max_num_allocs=17
total_allocs=1366

[debug/io-stats./storage/brick1/brick1 - usage-type gf_io_stats_mt_ios_stat
memusage]
size=6552
num_allocs=51
max_size=6704
max_num_allocs=52
total_allocs=53

[debug/io-stats./storage/brick1/brick1 - usage-type gf_io_stats_mt_ios_stat_list
memusage]
size=5280
num_allocs=165
max_size=5280
max_num_allocs=165
total_allocs=165

[features/read-only.vol4-read-only - Memory usage]
num_types=142

[features/read-only.vol4-read-only - usage-type gf_read_only_mt_priv_t memusage]
size=4
num_allocs=1
max_size=4
max_num_allocs=1
total_allocs=1

[features/worm.vol4-worm - Memory usage]
num_types=142

[features/worm.vol4-worm - usage-type gf_read_only_mt_priv_t memusage]
size=4
num_allocs=1
max_size=4
max_num_allocs=1
total_allocs=1

[features/quota.vol4-quota - Memory usage]
num_types=152

[features/quota.vol4-quota - usage-type gf_common_mt_asprintf memusage]
size=25
num_allocs=1
max_size=108
max_num_allocs=2
total_allocs=2

[features/quota.vol4-quota - usage-type gf_common_mt_mem_pool memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1

[features/quota.vol4-quota - usage-type gf_common_mt_long memusage]
size=30464
num_allocs=1
max_size=30464
max_num_allocs=1
total_allocs=1

[features/quota.vol4-quota - usage-type gf_quota_mt_quota_priv_t memusage]
size=104
num_allocs=1
max_size=104
max_num_allocs=1
total_allocs=1

[xlators.features.quota.priv]
soft-timeout=60
hard-timeout=5
alert-time=86400
quota-on=0
statfs=0
volume-uuid=vol4
validation-count=0

[features/index.vol4-index - Memory usage]
num_types=144

[features/index.vol4-index - usage-type gf_index_mt_priv_t memusage]
size=160
num_allocs=1
max_size=160
max_num_allocs=1
total_allocs=1

[features/index.vol4-index - usage-type gf_index_inode_ctx_t memusage]
size=896
num_allocs=28
max_size=928
max_num_allocs=29
total_allocs=42

[features/index.vol4-index - usage-type gf_index_fd_ctx_t memusage]
size=16
num_allocs=1
max_size=16
max_num_allocs=1
total_allocs=4321

[features/barrier.vol4-barrier - Memory usage]
num_types=142

[features/barrier.vol4-barrier - usage-type gf_barrier_mt_priv_t memusage]
size=56
num_allocs=1
max_size=56
max_num_allocs=1
total_allocs=1

[xlator.features.barrier.priv]
barrier.enabled=0
barrier.timeout=120

[features/marker.vol4-marker - Memory usage]
num_types=150

[features/marker.vol4-marker - usage-type gf_common_mt_asprintf memusage]
size=27
num_allocs=1
max_size=27
max_num_allocs=1
total_allocs=1

[features/marker.vol4-marker - usage-type gf_common_mt_mem_pool memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1

[features/marker.vol4-marker - usage-type gf_common_mt_long memusage]
size=42496
num_allocs=1
max_size=42496
max_num_allocs=1
total_allocs=1

[features/marker.vol4-marker - usage-type gf_marker_mt_marker_conf_t memusage]
size=80
num_allocs=1
max_size=80
max_num_allocs=1
total_allocs=1

[performance/io-threads.vol4-io-threads - Memory usage]
num_types=142

[performance/io-threads.vol4-io-threads - usage-type gf_iot_mt_iot_conf_t
memusage]
size=376
num_allocs=1
max_size=376
max_num_allocs=1
total_allocs=1

[performance/io-threads.vol4-io-threads]
maximum_threads_count=16
current_threads_count=16
sleep_count=16
idle_time=120
stack_size=1048576
high_priority_threads=16
normal_priority_threads=16
low_priority_threads=16
least_priority_threads=1
cached least rate=4
least rate limit=0

[features/upcall.vol4-upcall - Memory usage]
num_types=145

[features/upcall.vol4-upcall - usage-type gf_common_mt_asprintf memusage]
size=27
num_allocs=1
max_size=27
max_num_allocs=1
total_allocs=1

[features/upcall.vol4-upcall - usage-type gf_common_mt_mem_pool memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1

[features/upcall.vol4-upcall - usage-type gf_common_mt_long memusage]
size=55296
num_allocs=1
max_size=55296
max_num_allocs=1
total_allocs=1

[features/upcall.vol4-upcall - usage-type gf_upcall_mt_private_t memusage]
size=48
num_allocs=1
max_size=48
max_num_allocs=1
total_allocs=1

[features/locks.vol4-locks - Memory usage]
num_types=150

[features/locks.vol4-locks - usage-type gf_common_mt_asprintf memusage]
size=22
num_allocs=1
max_size=74
max_num_allocs=7
total_allocs=966561

[features/locks.vol4-locks - usage-type gf_common_mt_strdup memusage]
size=1924
num_allocs=81
max_size=10100
max_num_allocs=193
total_allocs=14800447

[features/locks.vol4-locks - usage-type gf_common_mt_mem_pool memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1

[features/locks.vol4-locks - usage-type gf_common_mt_long memusage]
size=4736
num_allocs=1
max_size=4736
max_num_allocs=1
total_allocs=1

[features/locks.vol4-locks - usage-type gf_locks_mt_pl_dom_list_t memusage]
size=6864
num_allocs=78
max_size=6952
max_num_allocs=79
total_allocs=109

[features/locks.vol4-locks - usage-type gf_locks_mt_pl_inode_t memusage]
size=5632
num_allocs=32
max_size=5808
max_num_allocs=33
total_allocs=50

[features/locks.vol4-locks - usage-type gf_locks_mt_posix_lock_t memusage]
size=360
num_allocs=5
max_size=504
max_num_allocs=7
total_allocs=179

[features/locks.vol4-locks - usage-type gf_locks_mt_pl_inode_lock_t memusage]
size=6720
num_allocs=3
max_size=257600
max_num_allocs=115
total_allocs=14318286

[features/locks.vol4-locks - usage-type gf_locks_mt_posix_locks_private_t
memusage]
size=16
num_allocs=1
max_size=16
max_num_allocs=1
total_allocs=1

[features/locks.vol4-locks - usage-type gf_locks_mt_pl_fdctx_t memusage]
size=224
num_allocs=14
max_size=736
max_num_allocs=46
total_allocs=9496

[features/access-control.vol4-access-control - Memory usage]
num_types=145

[features/access-control.vol4-access-control - usage-type gf_posix_acl_mt_ctx_t
memusage]
size=1088
num_allocs=34
max_size=1152
max_num_allocs=36
total_allocs=1417

[features/access-control.vol4-access-control - usage-type
gf_posix_acl_mt_posix_ace_t memusage]
size=32
num_allocs=1
max_size=32
max_num_allocs=1
total_allocs=1

[features/access-control.vol4-access-control - usage-type gf_posix_acl_mt_conf_t
memusage]
size=16
num_allocs=1
max_size=16
max_num_allocs=1
total_allocs=1

[features/bitrot-stub.vol4-bitrot-stub - Memory usage]
num_types=159

[features/bitrot-stub.vol4-bitrot-stub - usage-type gf_common_mt_asprintf
memusage]
size=33
num_allocs=1
max_size=123
max_num_allocs=46
total_allocs=1476321

[features/bitrot-stub.vol4-bitrot-stub - usage-type gf_common_mt_mem_pool
memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1

[features/bitrot-stub.vol4-bitrot-stub - usage-type gf_common_mt_long memusage]
size=43008
num_allocs=1
max_size=43008
max_num_allocs=1
total_allocs=1

[features/bitrot-stub.vol4-bitrot-stub - usage-type gf_br_stub_mt_private_t
memusage]
size=4232
num_allocs=1
max_size=4232
max_num_allocs=1
total_allocs=1

[features/bitrot-stub.vol4-bitrot-stub - usage-type gf_br_stub_mt_inode_ctx_t
memusage]
size=1296
num_allocs=27
max_size=1392
max_num_allocs=29
total_allocs=45

[features/bitrot-stub.vol4-bitrot-stub - usage-type gf_br_stub_mt_br_stub_fd_t
memusage]
size=336
num_allocs=14
max_size=408
max_num_allocs=17
total_allocs=1375

[features/changelog.vol4-changelog - Memory usage]
num_types=156

[features/changelog.vol4-changelog - usage-type gf_common_mt_volume_opt_list_t
memusage]
size=24
num_allocs=1
max_size=24
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_common_mt_asprintf memusage]
size=65
num_allocs=2
max_size=141
max_num_allocs=4
total_allocs=108

[features/changelog.vol4-changelog - usage-type gf_common_mt_strdup memusage]
size=158
num_allocs=6
max_size=244
max_num_allocs=10
total_allocs=10

[features/changelog.vol4-changelog - usage-type gf_common_mt_socket_private_t
memusage]
size=448
num_allocs=1
max_size=448
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_common_mt_mem_pool memusage]
size=240
num_allocs=2
max_size=240
max_num_allocs=2
total_allocs=2

[features/changelog.vol4-changelog - usage-type gf_common_mt_long memusage]
size=30048
num_allocs=2
max_size=30048
max_num_allocs=2
total_allocs=2

[features/changelog.vol4-changelog - usage-type gf_common_mt_rpcsvc_auth_list
memusage]
size=288
num_allocs=4
max_size=288
max_num_allocs=4
total_allocs=4

[features/changelog.vol4-changelog - usage-type gf_common_mt_rpcsvc_t memusage]
size=216
num_allocs=1
max_size=216
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_common_mt_rpcsvc_program_t
memusage]
size=240
num_allocs=2
max_size=240
max_num_allocs=2
total_allocs=2

[features/changelog.vol4-changelog - usage-type gf_common_mt_rpcsvc_listener_t
memusage]
size=160
num_allocs=1
max_size=160
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_common_mt_rpcsvc_wrapper_t
memusage]
size=32
num_allocs=1
max_size=32
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_common_mt_rpc_trans_t
memusage]
size=2784
num_allocs=1
max_size=2784
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_common_mt_rbuf_t memusage]
size=32
num_allocs=1
max_size=32
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_common_mt_rlist_t memusage]
size=768
num_allocs=4
max_size=768
max_num_allocs=4
total_allocs=4

[features/changelog.vol4-changelog - usage-type gf_common_mt_rvec_t memusage]
size=4194432
num_allocs=4
max_size=4194432
max_num_allocs=4
total_allocs=4

[features/changelog.vol4-changelog - usage-type gf_changelog_mt_priv_t memusage]
size=856
num_allocs=1
max_size=856
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_changelog_mt_rt_t memusage]
size=4
num_allocs=1
max_size=4
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_changelog_mt_ev_dispatcher_t
memusage]
size=24
num_allocs=1
max_size=24
max_num_allocs=1
total_allocs=1

[features/changetimerecorder.vol4-changetimerecorder - Memory usage]
num_types=146

[features/changetimerecorder.vol4-changetimerecorder - usage-type
gf_common_mt_asprintf memusage]
size=84
num_allocs=2
max_size=161
max_num_allocs=2
total_allocs=9

[features/changetimerecorder.vol4-changetimerecorder - usage-type
gf_common_mt_mem_pool memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1

[features/changetimerecorder.vol4-changetimerecorder - usage-type
gf_common_mt_long memusage]
size=1058560
num_allocs=1
max_size=1058560
max_num_allocs=1
total_allocs=1

[features/changetimerecorder.vol4-changetimerecorder - usage-type
gf_mt_sql_connection_t memusage]
size=4144
num_allocs=1
max_size=4144
max_num_allocs=1
total_allocs=1

[features/changetimerecorder.vol4-changetimerecorder - usage-type
gf_mt_db_conn_node_t memusage]
size=104
num_allocs=1
max_size=104
max_num_allocs=1
total_allocs=1

[features/changetimerecorder.vol4-changetimerecorder - usage-type
gf_ctr_mt_private_t memusage]
size=72
num_allocs=1
max_size=72
max_num_allocs=1
total_allocs=1

[features/trash.vol4-trash - Memory usage]
num_types=145

[features/trash.vol4-trash - usage-type gf_common_mt_inode_ctx memusage]
size=1728
num_allocs=4
max_size=1728
max_num_allocs=4
total_allocs=4

[features/trash.vol4-trash - usage-type gf_common_mt_list_head memusage]
size=1273488
num_allocs=2
max_size=1273488
max_num_allocs=2
total_allocs=2

[features/trash.vol4-trash - usage-type gf_common_mt_inode_table_t memusage]
size=200
num_allocs=1
max_size=200
max_num_allocs=1
total_allocs=1

[features/trash.vol4-trash - usage-type gf_common_mt_asprintf memusage]
size=97
num_allocs=5
max_size=185
max_num_allocs=6
total_allocs=8

[features/trash.vol4-trash - usage-type gf_common_mt_strdup memusage]
size=92
num_allocs=6
max_size=104
max_num_allocs=7
total_allocs=8

[features/trash.vol4-trash - usage-type gf_common_mt_mem_pool memusage]
size=480
num_allocs=4
max_size=480
max_num_allocs=4
total_allocs=4

[features/trash.vol4-trash - usage-type gf_common_mt_long memusage]
size=8527104
num_allocs=4
max_size=8527104
max_num_allocs=4
total_allocs=4

[features/trash.vol4-trash - usage-type gf_trash_mt_trash_private_t memusage]
size=64
num_allocs=1
max_size=64
max_num_allocs=1
total_allocs=1

[storage/posix.vol4-posix - Memory usage]
num_types=149

[storage/posix.vol4-posix - usage-type gf_common_mt_inode_ctx memusage]
size=6048
num_allocs=14
max_size=8208
max_num_allocs=19
total_allocs=1379

[storage/posix.vol4-posix - usage-type gf_common_mt_strdup memusage]
size=23
num_allocs=1
max_size=102
max_num_allocs=2
total_allocs=4777

[storage/posix.vol4-posix - usage-type gf_common_mt_char memusage]
size=256
num_allocs=1
max_size=1138
max_num_allocs=40
total_allocs=36228451

[storage/posix.vol4-posix - usage-type gf_posix_mt_posix_fd memusage]
size=672
num_allocs=14
max_size=8256
max_num_allocs=172
total_allocs=609610

[storage/posix.vol4-posix - usage-type gf_posix_mt_posix_private memusage]
size=608
num_allocs=1
max_size=608
max_num_allocs=1
total_allocs=1

[storage/posix.vol4-posix - usage-type gf_posix_mt_trash_path memusage]
size=43
num_allocs=1
max_size=43
max_num_allocs=1
total_allocs=1

[storage/posix.vol4-posix]
base_path=/storage/brick1/brick1
base_path_length=22
max_read=387819008
max_write=-490912860
nr_files=-600100

DUMP-END-TIME: 2015-09-24 13:19:53.355057
-------------- next part --------------
DUMP-START-TIME: 2015-09-24 13:19:52.349361

[mallinfo]
mallinfo_arena=5394432
mallinfo_ordblks=311
mallinfo_smblks=430
mallinfo_hblks=21
mallinfo_hblkhd=30515200
mallinfo_usmblks=0
mallinfo_fsmblks=44608
mallinfo_uordblks=1787728
mallinfo_fordblks=3606704
mallinfo_keepcost=115808

[global.glusterfs - Memory usage]
num_types=146

[global.glusterfs - usage-type gf_common_mt_dnscache6 memusage]
size=16
num_allocs=1
max_size=16
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_event_pool memusage]
size=12712
num_allocs=2
max_size=12712
max_num_allocs=2
total_allocs=2

[global.glusterfs - usage-type gf_common_mt_xlator_t memusage]
size=44880
num_allocs=17
max_size=44880
max_num_allocs=17
total_allocs=17

[global.glusterfs - usage-type gf_common_mt_xlator_list_t memusage]
size=512
num_allocs=32
max_size=512
max_num_allocs=32
total_allocs=32

[global.glusterfs - usage-type gf_common_mt_volume_opt_list_t memusage]
size=456
num_allocs=19
max_size=456
max_num_allocs=19
total_allocs=19

[global.glusterfs - usage-type gf_common_mt_gf_timer_t memusage]
size=112
num_allocs=2
max_size=168
max_num_allocs=3
total_allocs=132261

[global.glusterfs - usage-type gf_common_mt_gf_timer_registry_t memusage]
size=168
num_allocs=1
max_size=168
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_iobuf memusage]
size=109536
num_allocs=8
max_size=109536
max_num_allocs=8
total_allocs=8

[global.glusterfs - usage-type gf_common_mt_iobuf_arena memusage]
size=1872
num_allocs=9
max_size=1872
max_num_allocs=9
total_allocs=9

[global.glusterfs - usage-type gf_common_mt_iobuf_pool memusage]
size=1776
num_allocs=1
max_size=1776
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_asprintf memusage]
size=257
num_allocs=12
max_size=362
max_num_allocs=14
total_allocs=50

[global.glusterfs - usage-type gf_common_mt_strdup memusage]
size=2382
num_allocs=105
max_size=3482
max_num_allocs=125
total_allocs=2139

[global.glusterfs - usage-type gf_common_mt_socket_private_t memusage]
size=1344
num_allocs=3
max_size=1344
max_num_allocs=3
total_allocs=3

[global.glusterfs - usage-type gf_common_mt_char memusage]
size=4135
num_allocs=57
max_size=24820
max_num_allocs=90
total_allocs=902

[global.glusterfs - usage-type gf_common_mt_mem_pool memusage]
size=1200
num_allocs=10
max_size=1200
max_num_allocs=10
total_allocs=10

[global.glusterfs - usage-type gf_common_mt_long memusage]
size=8911520
num_allocs=10
max_size=8911520
max_num_allocs=10
total_allocs=10

[global.glusterfs - usage-type gf_common_mt_rpcsvc_auth_list memusage]
size=288
num_allocs=4
max_size=288
max_num_allocs=4
total_allocs=4

[global.glusterfs - usage-type gf_common_mt_rpcsvc_t memusage]
size=216
num_allocs=1
max_size=216
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_rpcsvc_program_t memusage]
size=240
num_allocs=2
max_size=240
max_num_allocs=2
total_allocs=2

[global.glusterfs - usage-type gf_common_mt_rpcsvc_listener_t memusage]
size=160
num_allocs=1
max_size=160
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_rpcsvc_wrapper_t memusage]
size=32
num_allocs=1
max_size=32
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_rpcclnt_t memusage]
size=312
num_allocs=1
max_size=312
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_rpcclnt_savedframe_t memusage]
size=200
num_allocs=1
max_size=200
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_rpc_trans_t memusage]
size=8352
num_allocs=3
max_size=8352
max_num_allocs=3
total_allocs=3

[global.glusterfs - usage-type gf_common_mt_glusterfs_graph_t memusage]
size=200
num_allocs=1
max_size=200
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_rpcclnt_cb_program_t memusage]
size=88
num_allocs=1
max_size=88
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_cliententry_t memusage]
size=2048
num_allocs=1
max_size=2048
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_clienttable_t memusage]
size=32
num_allocs=1
max_size=32
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_ereg memusage]
size=49152
num_allocs=1
max_size=49152
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_syncenv memusage]
size=15528
num_allocs=1
max_size=15528
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_sock_mt_lock_array memusage]
size=1640
num_allocs=1
max_size=1640
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gfd_mt_server_cmdline_t memusage]
size=40
num_allocs=1
max_size=40
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gfd_mt_xlator_cmdline_option_t memusage]
size=80
num_allocs=2
max_size=80
max_num_allocs=2
total_allocs=2

[global.glusterfs - usage-type gfd_mt_char memusage]
size=48
num_allocs=4
max_size=48
max_num_allocs=4
total_allocs=4

[global.glusterfs - usage-type gfd_mt_call_pool_t memusage]
size=48
num_allocs=1
max_size=48
max_num_allocs=1
total_allocs=1

[mempool]
-----=-----
pool-name=vol4-server:fd_t
hot-count=16
cold-count=1008
padded_sizeof=108
alloc-count=3975
max-alloc=26
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-server:dentry_t
hot-count=33
cold-count=16351
padded_sizeof=84
alloc-count=48
max-alloc=35
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-server:inode_t
hot-count=35
cold-count=16349
padded_sizeof=156
alloc-count=861
max-alloc=37
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-trash:fd_t
hot-count=0
cold-count=1024
padded_sizeof=108
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-trash:dentry_t
hot-count=0
cold-count=32768
padded_sizeof=84
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-trash:inode_t
hot-count=4
cold-count=32764
padded_sizeof=156
alloc-count=4
max-alloc=4
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-trash:trash_local_t
hot-count=0
cold-count=64
padded_sizeof=8628
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-changetimerecorder:gf_ctr_local_t
hot-count=0
cold-count=64
padded_sizeof=16540
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-changelog:rpcsvc_request_t
hot-count=0
cold-count=8
padded_sizeof=2828
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-changelog:changelog_local_t
hot-count=0
cold-count=64
padded_sizeof=116
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-bitrot-stub:br_stub_local_t
hot-count=0
cold-count=512
padded_sizeof=84
alloc-count=36
max-alloc=2
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-locks:pl_local_t
hot-count=0
cold-count=32
padded_sizeof=148
alloc-count=244674
max-alloc=2
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-upcall:upcall_local_t
hot-count=0
cold-count=512
padded_sizeof=108
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-marker:marker_local_t
hot-count=0
cold-count=128
padded_sizeof=332
alloc-count=2432
max-alloc=1
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-quota:quota_local_t
hot-count=0
cold-count=64
padded_sizeof=476
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=vol4-server:rpcsvc_request_t
hot-count=3
cold-count=509
padded_sizeof=2828
alloc-count=47631155
max-alloc=44
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:struct saved_frame
hot-count=0
cold-count=8
padded_sizeof=124
alloc-count=4
max-alloc=2
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:struct rpc_req
hot-count=0
cold-count=8
padded_sizeof=588
alloc-count=4
max-alloc=2
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:rpcsvc_request_t
hot-count=0
cold-count=8
padded_sizeof=2828
alloc-count=26
max-alloc=1
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:log_buf_t
hot-count=1
cold-count=255
padded_sizeof=140
alloc-count=469
max-alloc=6
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:data_t
hot-count=65
cold-count=16319
padded_sizeof=52
alloc-count=118921737
max-alloc=186
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:data_pair_t
hot-count=45
cold-count=16339
padded_sizeof=68
alloc-count=45167948
max-alloc=127
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:dict_t
hot-count=24
cold-count=4072
padded_sizeof=140
alloc-count=41400994
max-alloc=62
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:call_stub_t
hot-count=0
cold-count=1024
padded_sizeof=3764
alloc-count=58236099
max-alloc=45
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:call_stack_t
hot-count=3
cold-count=1021
padded_sizeof=1708
alloc-count=47533715
max-alloc=44
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:call_frame_t
hot-count=12
cold-count=4084
padded_sizeof=172
alloc-count=272654693
max-alloc=144
pool-misses=0
cur-stdalloc=0
max-stdalloc=0

[iobuf.global]
iobuf_pool=0x7fdba06c7360
iobuf_pool.default_page_size=131072
iobuf_pool.arena_size=12976128
iobuf_pool.arena_cnt=8
iobuf_pool.request_misses=0

[purge.1]
purge.1.mem_base=0x7fdb9f601000
purge.1.active_cnt=0
purge.1.passive_cnt=1024
purge.1.alloc_cnt=76058713
purge.1.max_active=14
purge.1.page_size=128

[purge.2]
purge.2.mem_base=0x7fdb9f5c1000
purge.2.active_cnt=0
purge.2.passive_cnt=512
purge.2.alloc_cnt=57639641
purge.2.max_active=8
purge.2.page_size=512

[purge.3]
purge.3.mem_base=0x7fdb9f4c1000
purge.3.active_cnt=0
purge.3.passive_cnt=512
purge.3.alloc_cnt=606557
purge.3.max_active=6
purge.3.page_size=2048

[purge.4]
purge.4.mem_base=0x7fdb9d723000
purge.4.active_cnt=0
purge.4.passive_cnt=128
purge.4.alloc_cnt=7710876
purge.4.max_active=9
purge.4.page_size=8192

[purge.5]
purge.5.mem_base=0x7fdb9d523000
purge.5.active_cnt=0
purge.5.passive_cnt=64
purge.5.alloc_cnt=7037568
purge.5.max_active=7
purge.5.page_size=32768

[purge.6]
purge.6.mem_base=0x7fdb9d123000
purge.6.active_cnt=0
purge.6.passive_cnt=32
purge.6.alloc_cnt=2712024
purge.6.max_active=7
purge.6.page_size=131072

[purge.7]
purge.7.mem_base=0x7fdb9cf23000
purge.7.active_cnt=0
purge.7.passive_cnt=8
purge.7.alloc_cnt=468445
purge.7.max_active=3
purge.7.page_size=262144

[purge.8]
purge.8.mem_base=0x7fdb9cd23000
purge.8.active_cnt=0
purge.8.passive_cnt=2
purge.8.alloc_cnt=11
purge.8.max_active=1
purge.8.page_size=1048576

[global.callpool]
callpool_address=0x7fdba06e63f0
callpool.cnt=3

[global.callpool.stack.1]
stack=0x7fdb9cacadc4
uid=0
gid=0
pid=19585
unique=27
lk-owner=814c0000
op=stack
type=0
cnt=4

[global.callpool.stack.1.frame.1]
frame=0x7fdb9cc78204
ref_count=0
translator=vol4-locks
complete=0
parent=vol4-io-threads
wind_from=default_inodelk_resume
wind_to=FIRST_CHILD(this)->fops->inodelk
unwind_to=default_inodelk_cbk

[global.callpool.stack.1.frame.2]
frame=0x7fdb9cc798dc
ref_count=1
translator=vol4-io-threads
complete=0
parent=/storage/brick2/brick2
wind_from=io_stats_inodelk
wind_to=FIRST_CHILD(this)->fops->inodelk
unwind_to=io_stats_inodelk_cbk

[global.callpool.stack.1.frame.3]
frame=0x7fdb9cc7b7c4
ref_count=1
translator=/storage/brick2/brick2
complete=0
parent=vol4-server
wind_from=server_inodelk_resume
wind_to=bound_xl->fops->inodelk
unwind_to=server_inodelk_cbk

[global.callpool.stack.1.frame.4]
frame=0x7fdb9cc79d90
ref_count=1
translator=vol4-server
complete=0

[global.callpool.stack.2]
stack=0x7fdb9cacdc78
uid=0
gid=0
pid=-6
unique=17234
lk-owner=204fba39be7f0000
op=stack
type=0
cnt=4

[global.callpool.stack.2.frame.1]
frame=0x7fdb9cc7bdd0
ref_count=0
translator=vol4-locks
complete=0
parent=vol4-io-threads
wind_from=default_inodelk_resume
wind_to=FIRST_CHILD(this)->fops->inodelk
unwind_to=default_inodelk_cbk

[global.callpool.stack.2.frame.2]
frame=0x7fdb9cc7a850
ref_count=1
translator=vol4-io-threads
complete=0
parent=/storage/brick2/brick2
wind_from=io_stats_inodelk
wind_to=FIRST_CHILD(this)->fops->inodelk
unwind_to=io_stats_inodelk_cbk

[global.callpool.stack.2.frame.3]
frame=0x7fdb9cc79ae0
ref_count=1
translator=/storage/brick2/brick2
complete=0
parent=vol4-server
wind_from=server_inodelk_resume
wind_to=bound_xl->fops->inodelk
unwind_to=server_inodelk_cbk

[global.callpool.stack.2.frame.4]
frame=0x7fdb9cc77290
ref_count=1
translator=vol4-server
complete=0

[global.callpool.stack.3]
stack=0x7fdb9cad1f30
uid=0
gid=0
pid=-6
unique=2017760
lk-owner=74580429ce7f0000
op=stack
type=0
cnt=4

[global.callpool.stack.3.frame.1]
frame=0x7fdb9cc7b66c
ref_count=0
translator=vol4-locks
complete=0
parent=vol4-io-threads
wind_from=default_inodelk_resume
wind_to=FIRST_CHILD(this)->fops->inodelk
unwind_to=default_inodelk_cbk

[global.callpool.stack.3.frame.2]
frame=0x7fdb9cc78cc4
ref_count=1
translator=vol4-io-threads
complete=0
parent=/storage/brick2/brick2
wind_from=io_stats_inodelk
wind_to=FIRST_CHILD(this)->fops->inodelk
unwind_to=io_stats_inodelk_cbk

[global.callpool.stack.3.frame.3]
frame=0x7fdb9cc7a0ec
ref_count=1
translator=/storage/brick2/brick2
complete=0
parent=vol4-server
wind_from=server_inodelk_resume
wind_to=bound_xl->fops->inodelk
unwind_to=server_inodelk_cbk

[global.callpool.stack.3.frame.4]
frame=0x7fdb9cc7be7c
ref_count=1
translator=vol4-server
complete=0

[active graph - 1]

[protocol/server.vol4-server - Memory usage]
num_types=151

[protocol/server.vol4-server - usage-type gf_common_mt_event_pool memusage]
size=16
num_allocs=1
max_size=16
max_num_allocs=1
total_allocs=1

[protocol/server.vol4-server - usage-type gf_common_mt_fdentry_t memusage]
size=26624
num_allocs=13
max_size=32768
max_num_allocs=16
total_allocs=430

[protocol/server.vol4-server - usage-type gf_common_mt_fdtable_t memusage]
size=832
num_allocs=13
max_size=896
max_num_allocs=14
total_allocs=148

[protocol/server.vol4-server - usage-type gf_common_mt_fd_ctx memusage]
size=4608
num_allocs=16
max_size=7488
max_num_allocs=26
total_allocs=3824

[protocol/server.vol4-server - usage-type gf_common_mt_inode_ctx memusage]
size=13392
num_allocs=31
max_size=14256
max_num_allocs=33
total_allocs=855

[protocol/server.vol4-server - usage-type gf_common_mt_list_head memusage]
size=1273488
num_allocs=2
max_size=1273488
max_num_allocs=2
total_allocs=2

[protocol/server.vol4-server - usage-type gf_common_mt_inode_table_t memusage]
size=200
num_allocs=1
max_size=200
max_num_allocs=1
total_allocs=1

[protocol/server.vol4-server - usage-type gf_common_mt_volume_opt_list_t
memusage]
size=72
num_allocs=3
max_size=72
max_num_allocs=3
total_allocs=3

[protocol/server.vol4-server - usage-type gf_common_mt_auth_handle_t memusage]
size=48
num_allocs=2
max_size=48
max_num_allocs=2
total_allocs=2

[protocol/server.vol4-server - usage-type gf_common_mt_iobref memusage]
size=72
num_allocs=3
max_size=1128
max_num_allocs=47
total_allocs=95262256

[protocol/server.vol4-server - usage-type gf_common_mt_asprintf memusage]
size=116
num_allocs=5
max_size=782
max_num_allocs=14
total_allocs=2199

[protocol/server.vol4-server - usage-type gf_common_mt_strdup memusage]
size=12813
num_allocs=364
max_size=12833
max_num_allocs=364
total_allocs=11339211

[protocol/server.vol4-server - usage-type gf_common_mt_socket_private_t
memusage]
size=6272
num_allocs=14
max_size=6720
max_num_allocs=15
total_allocs=149

[protocol/server.vol4-server - usage-type gf_common_mt_char memusage]
size=208
num_allocs=12
max_size=12792
max_num_allocs=130
total_allocs=88233773

[protocol/server.vol4-server - usage-type gf_common_mt_mem_pool memusage]
size=480
num_allocs=4
max_size=480
max_num_allocs=4
total_allocs=4

[protocol/server.vol4-server - usage-type gf_common_mt_long memusage]
size=5490688
num_allocs=4
max_size=5490688
max_num_allocs=4
total_allocs=4

[protocol/server.vol4-server - usage-type gf_common_mt_rpcsvc_auth_list
memusage]
size=288
num_allocs=4
max_size=288
max_num_allocs=4
total_allocs=4

[protocol/server.vol4-server - usage-type gf_common_mt_rpcsvc_t memusage]
size=216
num_allocs=1
max_size=216
max_num_allocs=1
total_allocs=1

[protocol/server.vol4-server - usage-type gf_common_mt_rpcsvc_program_t
memusage]
size=360
num_allocs=3
max_size=360
max_num_allocs=3
total_allocs=3

[protocol/server.vol4-server - usage-type gf_common_mt_rpcsvc_listener_t
memusage]
size=160
num_allocs=1
max_size=160
max_num_allocs=1
total_allocs=1

[protocol/server.vol4-server - usage-type gf_common_mt_rpcsvc_wrapper_t
memusage]
size=32
num_allocs=1
max_size=96
max_num_allocs=3
total_allocs=136

[protocol/server.vol4-server - usage-type gf_common_mt_rpc_trans_t memusage]
size=38976
num_allocs=14
max_size=41760
max_num_allocs=15
total_allocs=149

[protocol/server.vol4-server - usage-type gf_common_mt_fd_lk_ctx_t memusage]
size=384
num_allocs=16
max_size=624
max_num_allocs=26
total_allocs=3824

[protocol/server.vol4-server - usage-type gf_common_mt_client_t memusage]
size=1664
num_allocs=26
max_size=1792
max_num_allocs=28
total_allocs=296

[protocol/server.vol4-server - usage-type gf_common_mt_client_ctx memusage]
size=1664
num_allocs=13
max_size=1792
max_num_allocs=14
total_allocs=148

[protocol/server.vol4-server - usage-type gf_common_mt_iobrefs memusage]
size=384
num_allocs=3
max_size=6016
max_num_allocs=47
total_allocs=95262256

[protocol/server.vol4-server - usage-type gf_server_mt_server_conf_t memusage]
size=49744
num_allocs=14
max_size=49776
max_num_allocs=15
total_allocs=149

[protocol/server.vol4-server - usage-type gf_server_mt_state_t memusage]
size=6096
num_allocs=3
max_size=89408
max_num_allocs=44
total_allocs=47533726

[xlator.protocol.server.priv]
server.total-bytes-read=251572376
server.total-bytes-write=113779360
conn.1.bound_xl./storage/brick2/brick2.hashsize=14057
conn.1.bound_xl./storage/brick2/brick2.name=/storage/brick2/brick2/inode
conn.1.bound_xl./storage/brick2/brick2.lru_limit=16384
conn.1.bound_xl./storage/brick2/brick2.active_size=17
conn.1.bound_xl./storage/brick2/brick2.lru_size=18
conn.1.bound_xl./storage/brick2/brick2.purge_size=0

[conn.1.bound_xl./storage/brick2/brick2.active.1]
gfid=b3c53027-57c5-4ad9-8f49-d705667933e1
nlookup=2070
fd-count=2
ref=2
ia_type=2

[xlator.features.locks.vol4-locks.inode]
path=<gfid:b3c53027-57c5-4ad9-8f49-d705667933e1>
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.active.2]
gfid=3141a578-6198-49d6-b462-ee66a5bdfde8
nlookup=78
fd-count=1
ref=2
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/shoperience-dev.qcow2
mandatory=0
inodelk-count=1
lock-dump.domain.domain=vol4-replicate-0:self-heal
lock-dump.domain.domain=vol4-replicate-0:metadata
lock-dump.domain.domain=vol4-replicate-0
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=2396364800, len=4096, pid
= 0, owner=4c150a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23

[conn.1.bound_xl./storage/brick2/brick2.active.3]
gfid=f1028e08-1911-4757-b6ae-790893a49f6d
nlookup=63
fd-count=0
ref=3
ia_type=2

[xlator.features.locks.vol4-locks.inode]
path=/configs/old
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.active.4]
gfid=db3b8cd8-eb14-4ac8-9fd3-9914cc044a9c
nlookup=53
fd-count=2
ref=6
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/avnet-dev.qcow2
mandatory=0
inodelk-count=3
lock-dump.domain.domain=vol4-replicate-0:self-heal
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid =
18446744073709551610, owner=74580429ce7f0000, client=0x7fdb8c0ef4f0,
connection-id=vhost4.allaboutapps.at-4704-2015/08/27-13:19:02:16249-vol4-client-1-0-3,
granted at 2015-09-23 22:18:37
lock-dump.domain.domain=vol4-replicate-0:metadata
lock-dump.domain.domain=vol4-replicate-0
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=1996185600, len=28672,
pid = 0, owner=e0140a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[1](BLOCKED)=type=WRITE, whence=0, start=0, len=0, pid =
18446744073709551610, owner=74580429ce7f0000, client=0x7fdb8c0ef4f0,
connection-id=vhost4.allaboutapps.at-4704-2015/08/27-13:19:02:16249-vol4-client-1-0-3,
blocked at 2015-09-23 22:18:37

[conn.1.bound_xl./storage/brick2/brick2.active.5]
gfid=0b1a02a4-7e97-4504-9f26-b6313691fec0
nlookup=49
fd-count=1
ref=2
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/reports-dev-1.qcow2
mandatory=0
inodelk-count=1
lock-dump.domain.domain=vol4-replicate-0:self-heal
lock-dump.domain.domain=vol4-replicate-0:metadata
lock-dump.domain.domain=vol4-replicate-0
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=1591920128, len=17408,
pid = 0, owner=08140a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23

[conn.1.bound_xl./storage/brick2/brick2.active.6]
gfid=5c34060b-43fe-409e-8c14-e9aa8b144910
nlookup=31
fd-count=1
ref=2
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/playground-dev-1.qcow2
mandatory=0
inodelk-count=1
lock-dump.domain.domain=vol4-replicate-0:self-heal
lock-dump.domain.domain=vol4-replicate-0:metadata
lock-dump.domain.domain=vol4-replicate-0
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=591859712, len=1024, pid
= 0, owner=90160a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23

[conn.1.bound_xl./storage/brick2/brick2.active.7]
gfid=45f5abc5-69c0-4641-9b09-dcd7eeedc37f
nlookup=61
fd-count=2
ref=7
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/wuestenrot-dev.qcow2
mandatory=0
inodelk-count=4
lock-dump.domain.domain=vol4-replicate-0:self-heal
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid =
18446744073709551610, owner=204fba39be7f0000, client=0x7fdb8c0ba490,
connection-id=vhost3.allaboutapps.at-3643-2015/09/10-10:10:42:68099-vol4-client-1-0-0,
granted at 2015-09-23 22:26:50
lock-dump.domain.domain=vol4-replicate-0:metadata
lock-dump.domain.domain=vol4-replicate-0
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=1722875904, len=45056,
pid = 0, owner=74140a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[1](ACTIVE)=type=WRITE, whence=0, start=3647254528, len=20480,
pid = 0, owner=74140a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[2](BLOCKED)=type=WRITE, whence=0, start=0, len=0, pid =
18446744073709551610, owner=204fba39be7f0000, client=0x7fdb8c0ba490,
connection-id=vhost3.allaboutapps.at-3643-2015/09/10-10:10:42:68099-vol4-client-1-0-0,
blocked at 2015-09-23 22:26:50

[conn.1.bound_xl./storage/brick2/brick2.active.8]
gfid=97a5c60b-dd69-483f-a809-081ead34dc8b
nlookup=66
fd-count=1
ref=6
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/porsche-staging-1.qcow2
mandatory=0
inodelk-count=4
lock-dump.domain.domain=vol4-replicate-0:self-heal
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid = 19585,
owner=814c0000, client=0x7fdb84005890,
connection-id=vhost4.allaboutapps.at-19585-2015/09/24-13:19:41:672689-vol4-client-1-0-0,
granted at 2015-09-24 13:19:43
lock-dump.domain.domain=vol4-replicate-0:metadata
lock-dump.domain.domain=vol4-replicate-0
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=2044358656, len=4096, pid
= 0, owner=b8150a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[1](ACTIVE)=type=WRITE, whence=0, start=266702848, len=4096, pid
= 0, owner=b8150a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[2](BLOCKED)=type=WRITE, whence=0, start=0, len=0, pid = 19585,
owner=814c0000, client=0x7fdb84005890,
connection-id=vhost4.allaboutapps.at-19585-2015/09/24-13:19:41:672689-vol4-client-1-0-0,
blocked at 2015-09-24 13:19:43

[conn.1.bound_xl./storage/brick2/brick2.active.9]
gfid=5eadf1c9-b7d9-45e4-aff9-4a9ab8843be4
nlookup=53
fd-count=1
ref=12
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/haas-sohn-dev-1.qcow2
mandatory=0
inodelk-count=11
lock-dump.domain.domain=vol4-replicate-0:self-heal
lock-dump.domain.domain=vol4-replicate-0:metadata
lock-dump.domain.domain=vol4-replicate-0
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=955105280, len=4096, pid
= 0, owner=d4170a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[1](ACTIVE)=type=WRITE, whence=0, start=2115424256, len=4096, pid
= 0, owner=d4170a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[2](ACTIVE)=type=WRITE, whence=0, start=2114916352, len=4096, pid
= 0, owner=d4170a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[3](ACTIVE)=type=WRITE, whence=0, start=2114576384, len=4096, pid
= 0, owner=d4170a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[4](ACTIVE)=type=WRITE, whence=0, start=2114564096, len=8192, pid
= 0, owner=d4170a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[5](ACTIVE)=type=WRITE, whence=0, start=2114490368, len=4096, pid
= 0, owner=d4170a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[6](ACTIVE)=type=WRITE, whence=0, start=2114482176, len=4096, pid
= 0, owner=d4170a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[7](ACTIVE)=type=WRITE, whence=0, start=2114387968, len=4096, pid
= 0, owner=d4170a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[8](ACTIVE)=type=WRITE, whence=0, start=2114326528, len=4096, pid
= 0, owner=d4170a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[9](ACTIVE)=type=WRITE, whence=0, start=2111340544, len=4096, pid
= 0, owner=d4170a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23
inodelk.inodelk[10](ACTIVE)=type=WRITE, whence=0, start=2097545216, len=8192,
pid = 0, owner=d4170a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23

[conn.1.bound_xl./storage/brick2/brick2.active.10]
gfid=5e27a281-1ace-4aa6-a6d2-1bdde1a38eb7
nlookup=59
fd-count=1
ref=1
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/novartis-staging.qcow2
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.active.11]
gfid=0dd0cff5-0070-45f2-8fd9-8aa647f1e7b9
nlookup=45
fd-count=1
ref=2
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/novartis-dev.qcow2
mandatory=0
inodelk-count=1
lock-dump.domain.domain=vol4-replicate-0:self-heal
lock-dump.domain.domain=vol4-replicate-0:metadata
lock-dump.domain.domain=vol4-replicate-0
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=1933933056, len=14848,
pid = 0, owner=9c130a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23

[conn.1.bound_xl./storage/brick2/brick2.active.12]
gfid=3d41c61f-1f4c-462b-b8d8-0ffcc4f51932
nlookup=49
fd-count=1
ref=2
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/krebshilfe.qcow2
mandatory=0
inodelk-count=1
lock-dump.domain.domain=vol4-replicate-0:self-heal
lock-dump.domain.domain=vol4-replicate-0:metadata
lock-dump.domain.domain=vol4-replicate-0
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=1328116224, len=1024, pid
= 0, owner=68170a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23

[conn.1.bound_xl./storage/brick2/brick2.active.13]
gfid=00000000-0000-0000-0000-000000000005
nlookup=101
fd-count=0
ref=1
ia_type=2

[xlator.features.locks.vol4-locks.inode]
path=/.trashcan
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.active.14]
gfid=73e12d91-615f-4383-9bb9-ed67d5bb467c
nlookup=135
fd-count=0
ref=13
ia_type=2

[xlator.features.locks.vol4-locks.inode]
path=/configs
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.active.15]
gfid=66afe389-ecff-40de-8bb5-8ed9489b37b5
nlookup=72
fd-count=1
ref=2
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/guntamatic-dev-1.img.qcow2
mandatory=0
inodelk-count=1
lock-dump.domain.domain=vol4-replicate-0:self-heal
lock-dump.domain.domain=vol4-replicate-0
inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=2642804224, len=10240,
pid = 0, owner=58120a647d7f0000, client=0x7fdb8400f4c0,
connection-id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3,
granted at 2015-09-23 22:17:23

[conn.1.bound_xl./storage/brick2/brick2.active.16]
gfid=1c7a3140-881e-461d-9adc-475f49d68604
nlookup=562
fd-count=1
ref=1
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/win7.qcow2
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.active.17]
gfid=00000000-0000-0000-0000-000000000001
nlookup=12
fd-count=0
ref=1
ia_type=2

[xlator.features.locks.vol4-locks.inode]
path=/
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.1]
gfid=b6d5ec08-fcee-4c98-b8cd-f7519c3f551d
nlookup=13
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/old/abhof-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.2]
gfid=6abf53a9-6a02-430f-9991-b84935bf61b2
nlookup=13
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/old/abhof-staging.xml
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.3]
gfid=7bc8905d-6fa4-4bbc-a639-222877f3e2e3
nlookup=13
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/old/cashquizz-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.4]
gfid=89f50322-270c-4a66-9275-813f8005b12d
nlookup=98
fd-count=0
ref=0
ia_type=2

[conn.1.bound_xl./storage/brick2/brick2.lru.5]
gfid=c75d90b1-a4e8-477d-9b83-a6547f88c7a8
nlookup=102
fd-count=0
ref=0
ia_type=2

[conn.1.bound_xl./storage/brick2/brick2.lru.6]
gfid=00000000-0000-0000-0000-000000000006
nlookup=61
fd-count=0
ref=0
ia_type=2

[conn.1.bound_xl./storage/brick2/brick2.lru.7]
gfid=887e107c-27a4-4906-979a-c5ee8a899965
nlookup=53
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/avnet-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.8]
gfid=6e63d4b2-89be-4d88-818f-6215d8dde0e5
nlookup=53
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/haas-sohn-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.9]
gfid=409425ac-a11f-4252-bbc7-11a3ffa43a79
nlookup=53
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/novartis-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.10]
gfid=ebec8823-19f6-4e87-9e71-305cc178dd3f
nlookup=53
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/porsche-staging.xml
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.11]
gfid=2b36054f-eda1-4ec9-8216-d9d7a260bf18
nlookup=6
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/shoperience-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.12]
gfid=bbc52dbe-136d-4559-b177-8ff800df4b5f
nlookup=53
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/wuestenrot-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.13]
gfid=47fc7997-3792-406b-a6ac-dcf644ce662f
nlookup=53
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/guntamatic-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.14]
gfid=7bb24cb9-1f8c-471a-a874-a31699abd6c2
nlookup=38
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/krebshilfe.xml
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.15]
gfid=b7382d70-f4b2-4449-975c-6f9c6f112ce2
nlookup=53
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/novartis-staging.xml
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.16]
gfid=bac5626f-ad39-4102-93d2-e9c4a1a20ad4
nlookup=53
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/playground-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.17]
gfid=1f6ce2ab-f9c0-4d9a-aad0-03726452ede0
nlookup=53
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/reports-dev.xml
mandatory=0

[conn.1.bound_xl./storage/brick2/brick2.lru.18]
gfid=6f5ce908-1a73-43af-909d-26073906f7fd
nlookup=53
fd-count=0
ref=0
ia_type=1

[xlator.features.locks.vol4-locks.inode]
path=/configs/win7.xml
mandatory=0
conn.1.id=vhost3.allaboutapps.at-3641-2015/09/10-10:10:42:65055-vol4-client-1-0-0
conn.1.ref=1
conn.1.bound_xl=/storage/brick2/brick2
conn.2.id=vhost2.allaboutapps.at-3064-2015/09/10-07:59:06:558627-vol4-client-1-0-2
conn.2.ref=1
conn.2.bound_xl=/storage/brick2/brick2
conn.3.id=vhost2.allaboutapps.at-3062-2015/09/10-07:59:06:556223-vol4-client-1-0-2
conn.3.ref=1
conn.3.bound_xl=/storage/brick2/brick2
conn.4.id=vhost4.allaboutapps.at-19585-2015/09/24-13:19:41:672689-vol4-client-1-0-0
conn.4.ref=2
conn.4.bound_xl=/storage/brick2/brick2
conn.5.id=vhost4.allaboutapps.at-4702-2015/08/27-13:19:02:14144-vol4-client-1-0-3
conn.5.ref=1
conn.5.bound_xl=/storage/brick2/brick2
conn.6.id=vhost1.allaboutapps.at-3924-2015/09/10-07:11:57:288087-vol4-client-1-0-5
conn.6.ref=1
conn.6.bound_xl=/storage/brick2/brick2
conn.7.id=vhost1.allaboutapps.at-4298-2015/09/10-07:12:25:821602-vol4-client-1-0-4
conn.7.ref=1
conn.7.bound_xl=/storage/brick2/brick2
conn.8.id=vhost2.allaboutapps.at-3630-2015/09/10-07:59:36:520290-vol4-client-1-0-2
conn.8.ref=1
conn.8.bound_xl=/storage/brick2/brick2
conn.9.id=vhost4.allaboutapps.at-4704-2015/08/27-13:19:02:16249-vol4-client-1-0-3
conn.9.ref=2
conn.9.bound_xl=/storage/brick2/brick2
conn.10.id=vhost1.allaboutapps.at-3916-2015/09/10-07:11:56:285421-vol4-client-1-0-4
conn.10.ref=1
conn.10.bound_xl=/storage/brick2/brick2
conn.11.id=vhost3.allaboutapps.at-4030-2015/09/10-10:11:10:614419-vol4-client-1-0-0
conn.11.ref=1
conn.11.bound_xl=/storage/brick2/brick2
conn.12.id=vhost4.allaboutapps.at-5057-2015/08/27-13:20:19:469245-vol4-client-1-0-3
conn.12.ref=1
conn.12.bound_xl=/storage/brick2/brick2

[debug/io-stats./storage/brick2/brick2 - Memory usage]
num_types=145

[debug/io-stats./storage/brick2/brick2 - usage-type gf_common_mt_strdup
memusage]
size=1337
num_allocs=49
max_size=1603
max_num_allocs=59
total_allocs=756

[debug/io-stats./storage/brick2/brick2 - usage-type gf_io_stats_mt_ios_conf
memusage]
size=5504
num_allocs=1
max_size=5504
max_num_allocs=1
total_allocs=1

[debug/io-stats./storage/brick2/brick2 - usage-type gf_io_stats_mt_ios_fd
memusage]
size=7728
num_allocs=14
max_size=13248
max_num_allocs=24
total_allocs=718

[debug/io-stats./storage/brick2/brick2 - usage-type gf_io_stats_mt_ios_stat
memusage]
size=5640
num_allocs=45
max_size=5792
max_num_allocs=46
total_allocs=47

[debug/io-stats./storage/brick2/brick2 - usage-type gf_io_stats_mt_ios_stat_list
memusage]
size=4224
num_allocs=132
max_size=4224
max_num_allocs=132
total_allocs=132

[features/read-only.vol4-read-only - Memory usage]
num_types=142

[features/read-only.vol4-read-only - usage-type gf_read_only_mt_priv_t memusage]
size=4
num_allocs=1
max_size=4
max_num_allocs=1
total_allocs=1

[features/worm.vol4-worm - Memory usage]
num_types=142

[features/worm.vol4-worm - usage-type gf_read_only_mt_priv_t memusage]
size=4
num_allocs=1
max_size=4
max_num_allocs=1
total_allocs=1

[features/quota.vol4-quota - Memory usage]
num_types=152

[features/quota.vol4-quota - usage-type gf_common_mt_asprintf memusage]
size=25
num_allocs=1
max_size=108
max_num_allocs=2
total_allocs=2

[features/quota.vol4-quota - usage-type gf_common_mt_mem_pool memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1

[features/quota.vol4-quota - usage-type gf_common_mt_long memusage]
size=30464
num_allocs=1
max_size=30464
max_num_allocs=1
total_allocs=1

[features/quota.vol4-quota - usage-type gf_quota_mt_quota_priv_t memusage]
size=104
num_allocs=1
max_size=104
max_num_allocs=1
total_allocs=1

[xlators.features.quota.priv]
soft-timeout=60
hard-timeout=5
alert-time=86400
quota-on=0
statfs=0
volume-uuid=vol4
validation-count=0

[features/index.vol4-index - Memory usage]
num_types=144

[features/index.vol4-index - usage-type gf_index_mt_priv_t memusage]
size=160
num_allocs=1
max_size=160
max_num_allocs=1
total_allocs=1

[features/index.vol4-index - usage-type gf_index_inode_ctx_t memusage]
size=768
num_allocs=24
max_size=800
max_num_allocs=25
total_allocs=34

[features/index.vol4-index - usage-type gf_index_fd_ctx_t memusage]
size=32
num_allocs=2
max_size=48
max_num_allocs=3
total_allocs=2069

[features/barrier.vol4-barrier - Memory usage]
num_types=142

[features/barrier.vol4-barrier - usage-type gf_barrier_mt_priv_t memusage]
size=56
num_allocs=1
max_size=56
max_num_allocs=1
total_allocs=1

[xlator.features.barrier.priv]
barrier.enabled=0
barrier.timeout=120

[features/marker.vol4-marker - Memory usage]
num_types=150

[features/marker.vol4-marker - usage-type gf_common_mt_asprintf memusage]
size=27
num_allocs=1
max_size=27
max_num_allocs=1
total_allocs=1

[features/marker.vol4-marker - usage-type gf_common_mt_mem_pool memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1

[features/marker.vol4-marker - usage-type gf_common_mt_long memusage]
size=42496
num_allocs=1
max_size=42496
max_num_allocs=1
total_allocs=1

[features/marker.vol4-marker - usage-type gf_marker_mt_marker_conf_t memusage]
size=80
num_allocs=1
max_size=80
max_num_allocs=1
total_allocs=1

[performance/io-threads.vol4-io-threads - Memory usage]
num_types=142

[performance/io-threads.vol4-io-threads - usage-type gf_iot_mt_iot_conf_t
memusage]
size=376
num_allocs=1
max_size=376
max_num_allocs=1
total_allocs=1

[performance/io-threads.vol4-io-threads]
maximum_threads_count=16
current_threads_count=16
sleep_count=16
idle_time=120
stack_size=1048576
high_priority_threads=16
normal_priority_threads=16
low_priority_threads=16
least_priority_threads=1
cached least rate=432
least rate limit=0

[features/upcall.vol4-upcall - Memory usage]
num_types=145

[features/upcall.vol4-upcall - usage-type gf_common_mt_asprintf memusage]
size=27
num_allocs=1
max_size=27
max_num_allocs=1
total_allocs=1

[features/upcall.vol4-upcall - usage-type gf_common_mt_mem_pool memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1

[features/upcall.vol4-upcall - usage-type gf_common_mt_long memusage]
size=55296
num_allocs=1
max_size=55296
max_num_allocs=1
total_allocs=1

[features/upcall.vol4-upcall - usage-type gf_upcall_mt_private_t memusage]
size=48
num_allocs=1
max_size=48
max_num_allocs=1
total_allocs=1

[features/locks.vol4-locks - Memory usage]
num_types=150

[features/locks.vol4-locks - usage-type gf_common_mt_asprintf memusage]
size=22
num_allocs=1
max_size=74
max_num_allocs=5
total_allocs=489233

[features/locks.vol4-locks - usage-type gf_common_mt_strdup memusage]
size=3490
num_allocs=94
max_size=8243
max_num_allocs=155
total_allocs=11576279

[features/locks.vol4-locks - usage-type gf_common_mt_mem_pool memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1

[features/locks.vol4-locks - usage-type gf_common_mt_long memusage]
size=4736
num_allocs=1
max_size=4736
max_num_allocs=1
total_allocs=1

[features/locks.vol4-locks - usage-type gf_locks_mt_pl_dom_list_t memusage]
size=5808
num_allocs=66
max_size=5896
max_num_allocs=67
total_allocs=87

[features/locks.vol4-locks - usage-type gf_locks_mt_pl_inode_t memusage]
size=5632
num_allocs=32
max_size=5632
max_num_allocs=32
total_allocs=44

[features/locks.vol4-locks - usage-type gf_locks_mt_posix_lock_t memusage]
size=360
num_allocs=5
max_size=504
max_num_allocs=7
total_allocs=40

[features/locks.vol4-locks - usage-type gf_locks_mt_pl_inode_lock_t memusage]
size=62720
num_allocs=28
max_size=212800
max_num_allocs=95
total_allocs=11331409

[features/locks.vol4-locks - usage-type gf_locks_mt_posix_locks_private_t
memusage]
size=16
num_allocs=1
max_size=16
max_num_allocs=1
total_allocs=1

[features/locks.vol4-locks - usage-type gf_locks_mt_pl_fdctx_t memusage]
size=224
num_allocs=14
max_size=384
max_num_allocs=24
total_allocs=1609

[features/access-control.vol4-access-control - Memory usage]
num_types=145

[features/access-control.vol4-access-control - usage-type gf_posix_acl_mt_ctx_t
memusage]
size=1088
num_allocs=34
max_size=1152
max_num_allocs=36
total_allocs=51

[features/access-control.vol4-access-control - usage-type
gf_posix_acl_mt_posix_ace_t memusage]
size=32
num_allocs=1
max_size=32
max_num_allocs=1
total_allocs=1

[features/access-control.vol4-access-control - usage-type gf_posix_acl_mt_conf_t
memusage]
size=16
num_allocs=1
max_size=16
max_num_allocs=1
total_allocs=1

[features/bitrot-stub.vol4-bitrot-stub - Memory usage]
num_types=159

[features/bitrot-stub.vol4-bitrot-stub - usage-type gf_common_mt_asprintf
memusage]
size=33
num_allocs=1
max_size=63
max_num_allocs=16
total_allocs=731303

[features/bitrot-stub.vol4-bitrot-stub - usage-type gf_common_mt_mem_pool
memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1

[features/bitrot-stub.vol4-bitrot-stub - usage-type gf_common_mt_long memusage]
size=43008
num_allocs=1
max_size=43008
max_num_allocs=1
total_allocs=1

[features/bitrot-stub.vol4-bitrot-stub - usage-type gf_br_stub_mt_private_t
memusage]
size=4232
num_allocs=1
max_size=4232
max_num_allocs=1
total_allocs=1

[features/bitrot-stub.vol4-bitrot-stub - usage-type gf_br_stub_mt_inode_ctx_t
memusage]
size=1296
num_allocs=27
max_size=1392
max_num_allocs=29
total_allocs=39

[features/bitrot-stub.vol4-bitrot-stub - usage-type gf_br_stub_mt_br_stub_fd_t
memusage]
size=336
num_allocs=14
max_size=576
max_num_allocs=24
total_allocs=727

[features/changelog.vol4-changelog - Memory usage]
num_types=156

[features/changelog.vol4-changelog - usage-type gf_common_mt_volume_opt_list_t
memusage]
size=24
num_allocs=1
max_size=24
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_common_mt_asprintf memusage]
size=65
num_allocs=2
max_size=141
max_num_allocs=4
total_allocs=56

[features/changelog.vol4-changelog - usage-type gf_common_mt_strdup memusage]
size=158
num_allocs=6
max_size=244
max_num_allocs=10
total_allocs=10

[features/changelog.vol4-changelog - usage-type gf_common_mt_socket_private_t
memusage]
size=448
num_allocs=1
max_size=448
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_common_mt_mem_pool memusage]
size=240
num_allocs=2
max_size=240
max_num_allocs=2
total_allocs=2

[features/changelog.vol4-changelog - usage-type gf_common_mt_long memusage]
size=30048
num_allocs=2
max_size=30048
max_num_allocs=2
total_allocs=2

[features/changelog.vol4-changelog - usage-type gf_common_mt_rpcsvc_auth_list
memusage]
size=288
num_allocs=4
max_size=288
max_num_allocs=4
total_allocs=4

[features/changelog.vol4-changelog - usage-type gf_common_mt_rpcsvc_t memusage]
size=216
num_allocs=1
max_size=216
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_common_mt_rpcsvc_program_t
memusage]
size=240
num_allocs=2
max_size=240
max_num_allocs=2
total_allocs=2

[features/changelog.vol4-changelog - usage-type gf_common_mt_rpcsvc_listener_t
memusage]
size=160
num_allocs=1
max_size=160
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_common_mt_rpcsvc_wrapper_t
memusage]
size=32
num_allocs=1
max_size=32
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_common_mt_rpc_trans_t
memusage]
size=2784
num_allocs=1
max_size=2784
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_common_mt_rbuf_t memusage]
size=32
num_allocs=1
max_size=32
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_common_mt_rlist_t memusage]
size=768
num_allocs=4
max_size=768
max_num_allocs=4
total_allocs=4

[features/changelog.vol4-changelog - usage-type gf_common_mt_rvec_t memusage]
size=4194432
num_allocs=4
max_size=4194432
max_num_allocs=4
total_allocs=4

[features/changelog.vol4-changelog - usage-type gf_changelog_mt_priv_t memusage]
size=856
num_allocs=1
max_size=856
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_changelog_mt_rt_t memusage]
size=4
num_allocs=1
max_size=4
max_num_allocs=1
total_allocs=1

[features/changelog.vol4-changelog - usage-type gf_changelog_mt_ev_dispatcher_t
memusage]
size=24
num_allocs=1
max_size=24
max_num_allocs=1
total_allocs=1

[features/changetimerecorder.vol4-changetimerecorder - Memory usage]
num_types=146

[features/changetimerecorder.vol4-changetimerecorder - usage-type
gf_common_mt_asprintf memusage]
size=84
num_allocs=2
max_size=161
max_num_allocs=2
total_allocs=9

[features/changetimerecorder.vol4-changetimerecorder - usage-type
gf_common_mt_mem_pool memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1

[features/changetimerecorder.vol4-changetimerecorder - usage-type
gf_common_mt_long memusage]
size=1058560
num_allocs=1
max_size=1058560
max_num_allocs=1
total_allocs=1

[features/changetimerecorder.vol4-changetimerecorder - usage-type
gf_mt_sql_connection_t memusage]
size=4144
num_allocs=1
max_size=4144
max_num_allocs=1
total_allocs=1

[features/changetimerecorder.vol4-changetimerecorder - usage-type
gf_mt_db_conn_node_t memusage]
size=104
num_allocs=1
max_size=104
max_num_allocs=1
total_allocs=1

[features/changetimerecorder.vol4-changetimerecorder - usage-type
gf_ctr_mt_private_t memusage]
size=72
num_allocs=1
max_size=72
max_num_allocs=1
total_allocs=1

[features/trash.vol4-trash - Memory usage]
num_types=145

[features/trash.vol4-trash - usage-type gf_common_mt_inode_ctx memusage]
size=1728
num_allocs=4
max_size=1728
max_num_allocs=4
total_allocs=4

[features/trash.vol4-trash - usage-type gf_common_mt_list_head memusage]
size=1273488
num_allocs=2
max_size=1273488
max_num_allocs=2
total_allocs=2

[features/trash.vol4-trash - usage-type gf_common_mt_inode_table_t memusage]
size=200
num_allocs=1
max_size=200
max_num_allocs=1
total_allocs=1

[features/trash.vol4-trash - usage-type gf_common_mt_asprintf memusage]
size=97
num_allocs=5
max_size=185
max_num_allocs=6
total_allocs=8

[features/trash.vol4-trash - usage-type gf_common_mt_strdup memusage]
size=92
num_allocs=6
max_size=104
max_num_allocs=7
total_allocs=8

[features/trash.vol4-trash - usage-type gf_common_mt_mem_pool memusage]
size=480
num_allocs=4
max_size=480
max_num_allocs=4
total_allocs=4

[features/trash.vol4-trash - usage-type gf_common_mt_long memusage]
size=8527104
num_allocs=4
max_size=8527104
max_num_allocs=4
total_allocs=4

[features/trash.vol4-trash - usage-type gf_trash_mt_trash_private_t memusage]
size=64
num_allocs=1
max_size=64
max_num_allocs=1
total_allocs=1

[storage/posix.vol4-posix - Memory usage]
num_types=149

[storage/posix.vol4-posix - usage-type gf_common_mt_inode_ctx memusage]
size=1728
num_allocs=4
max_size=2160
max_num_allocs=5
total_allocs=6

[storage/posix.vol4-posix - usage-type gf_common_mt_strdup memusage]
size=23
num_allocs=1
max_size=102
max_num_allocs=2
total_allocs=2210

[storage/posix.vol4-posix - usage-type gf_common_mt_char memusage]
size=256
num_allocs=1
max_size=629
max_num_allocs=13
total_allocs=20090440

[storage/posix.vol4-posix - usage-type gf_posix_mt_posix_fd memusage]
size=672
num_allocs=14
max_size=1248
max_num_allocs=26
total_allocs=1756

[storage/posix.vol4-posix - usage-type gf_posix_mt_posix_private memusage]
size=608
num_allocs=1
max_size=608
max_num_allocs=1
total_allocs=1

[storage/posix.vol4-posix - usage-type gf_posix_mt_trash_path memusage]
size=43
num_allocs=1
max_size=43
max_num_allocs=1
total_allocs=1

[storage/posix.vol4-posix]
base_path=/storage/brick2/brick2
base_path_length=22
max_read=6684672
max_write=382638521
nr_files=-133

DUMP-END-TIME: 2015-09-24 13:19:52.351716

Anuradha Talur

2015-Oct-05 11:01 UTC

head link

[Gluster-users] gluster 3.7.3 - volume heal info hangs - unknown heal status

----- Original Message -----> From: "Andreas Mather" <andreas at allaboutapps.at>
> To: "Anuradha Talur" <atalur at redhat.com>
> Cc: "Gluster-users at gluster.org List" <gluster-users at
gluster.org>
> Sent: Thursday, September 24, 2015 6:59:38 PM
> Subject: Re: [Gluster-users] gluster 3.7.3 - volume heal info hangs -
unknown heal status
> 
> Hi Anuradha!
> 
> Thanks for your reply! Attached you can find the dump files. As I'm not
> sure if they make their way through as attachments, here're links to
them
> as well:
> 
> brick1 - http://pastebin.com/3ivkhuRH
> brick2 - http://pastebin.com/77sT1mutHi,

I see some blocked locks from the statedump.
Could you let me know what kind of workload you had when you observed the
hang?> 
> - Andreas
> 
> 
> 
> 
> On Thu, Sep 24, 2015 at 3:18 PM, Anuradha Talur <atalur at
redhat.com> wrote:
> 
> >
> >
> > ----- Original Message -----
> > > From: "Andreas Mather" <andreas at
allaboutapps.at>
> > > To: "Gluster-users at gluster.org List"
<gluster-users at gluster.org>
> > > Sent: Thursday, September 24, 2015 1:24:12 PM
> > > Subject: [Gluster-users] gluster 3.7.3 - volume heal info hangs -
> > unknown     heal status
> > >
> > > Hi!
> > >
> > > Our provider had network maintenance this night, so 2 of our 4
servers
> > got
> > > disconnected and reconnected. Since we knew this was coming, we
shifted
> > all
> > > work load off the affected servers. This morning, most of the
cluster
> > seems
> > > fine, but for one volume, no heal info can be retrieved, so we
basically
> > > don't know about the healing state of the volume. The volume
is a
> > replica 2
> > > volume between vhost4-int/brick1 and vhost3-int/brick2.
> > >
> > > The volume is accessible, but since I don't get any heal
info, I don't
> > know
> > > if it is probably replicated. Any help to resolve this situation
is
> > highly
> > > appreciated.
> > >
> > > hangs forever:
> > > [root at vhost4 ~]# gluster volume heal vol4 info
> > >
> > > glfsheal-vol4.log:
> > > [2015-09-24 07:47:59.284723] I [MSGID: 101190]
> > > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started
thread
> > with
> > > index 1
> > > [2015-09-24 07:47:59.293735] I [MSGID: 101190]
> > > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started
thread
> > with
> > > index 2
> > > [2015-09-24 07:47:59.294061] I [MSGID: 104045]
[glfs-master.c:95:notify]
> > > 0-gfapi: New graph 76686f73-7434-2e61-6c6c-61626f757461 (0)
coming up
> > > [2015-09-24 07:47:59.294081] I [MSGID: 114020]
[client.c:2118:notify]
> > > 0-vol4-client-1: parent translators are ready, attempting connect
on
> > > transport
> > > [2015-09-24 07:47:59.309470] I [MSGID: 114020]
[client.c:2118:notify]
> > > 0-vol4-client-2: parent translators are ready, attempting connect
on
> > > transport
> > > [2015-09-24 07:47:59.310525] I
[rpc-clnt.c:1819:rpc_clnt_reconfig]
> > > 0-vol4-client-1: changing port to 49155 (from 0)
> > > [2015-09-24 07:47:59.315958] I [MSGID: 114057]
> > > [client-handshake.c:1437:select_server_supported_programs]
> > 0-vol4-client-1:
> > > Using Program GlusterFS 3.3, Num (1298437), Version (330)
> > > [2015-09-24 07:47:59.316481] I [MSGID: 114046]
> > > [client-handshake.c:1213:client_setvolume_cbk] 0-vol4-client-1:
> > Connected to
> > > vol4-client-1, attached to remote volume
'/storage/brick2/brick2'.
> > > [2015-09-24 07:47:59.316495] I [MSGID: 114047]
> > > [client-handshake.c:1224:client_setvolume_cbk] 0-vol4-client-1:
Server
> > and
> > > Client lk-version numbers are not same, reopening the fds
> > > [2015-09-24 07:47:59.316538] I [MSGID: 108005]
> > [afr-common.c:3960:afr_notify]
> > > 0-vol4-replicate-0: Subvolume 'vol4-client-1' came back
up; going online.
> > > [2015-09-24 07:47:59.317150] I [MSGID: 114035]
> > > [client-handshake.c:193:client_set_lk_version_cbk]
0-vol4-client-1:
> > Server
> > > lk version = 1
> > > [2015-09-24 07:47:59.320898] I
[rpc-clnt.c:1819:rpc_clnt_reconfig]
> > > 0-vol4-client-2: changing port to 49154 (from 0)
> > > [2015-09-24 07:47:59.325633] I [MSGID: 114057]
> > > [client-handshake.c:1437:select_server_supported_programs]
> > 0-vol4-client-2:
> > > Using Program GlusterFS 3.3, Num (1298437), Version (330)
> > > [2015-09-24 07:47:59.325780] I [MSGID: 114046]
> > > [client-handshake.c:1213:client_setvolume_cbk] 0-vol4-client-2:
> > Connected to
> > > vol4-client-2, attached to remote volume
'/storage/brick1/brick1'.
> > > [2015-09-24 07:47:59.325791] I [MSGID: 114047]
> > > [client-handshake.c:1224:client_setvolume_cbk] 0-vol4-client-2:
Server
> > and
> > > Client lk-version numbers are not same, reopening the fds
> > > [2015-09-24 07:47:59.333346] I [MSGID: 114035]
> > > [client-handshake.c:193:client_set_lk_version_cbk]
0-vol4-client-2:
> > Server
> > > lk version = 1
> > > [2015-09-24 07:47:59.334545] I [MSGID: 108031]
> > > [afr-common.c:1745:afr_local_discovery_cbk] 0-vol4-replicate-0:
selecting
> > > local read_child vol4-client-2
> > > [2015-09-24 07:47:59.335833] I [MSGID: 104041]
> > > [glfs-resolve.c:862:__glfs_active_subvol] 0-vol4: switched to
graph
> > > 76686f73-7434-2e61-6c6c-61626f757461 (0)
> > >
> > > Questions to this output:
> > > -) Why does it report " Using Program GlusterFS 3.3, Num
(1298437),
> > Version
> > > (330) ". We run 3.7.3 ?!
> > > -) guster logs timestamps in UTC not taking server timezone into
> > account. Is
> > > there a way to fix this?
> > >
> > > etc-glusterfs-glusterd.vol.log:
> > > no logs to after volume heal info command
> > >
> > > storage-brick1-brick1.log:
> > > [2015-09-24 07:47:59.325720] I [login.c:81:gf_auth] 0-auth/login:
allowed
> > > user names: 67ef1559-d3a1-403a-b8e7-fb145c3acf4e
> > > [2015-09-24 07:47:59.325743] I [MSGID: 115029]
> > > [server-handshake.c:610:server_setvolume] 0-vol4-server: accepted
client
> > > from
> > >
vhost4.allaboutapps.at-14900-2015/09/24-07:47:59:282313-vol4-client-2-0-0
> > > (version: 3.7.3)
> > >
> > > storage-brick2-brick2.log:
> > > no logs to after volume heal info command
> > >
> > >
> > Hi Andreas,
> >
> > Could you please provide the following information so that we can
> > understand why the command is hanging?
> > When the command is hung, run the following command from one of the
> > servers:
> > `gluster volume statedump <volname>`
> > This command will generate statedumps of glusterfsd processes in the
> > servers. You can find them at /var/run/gluster . A typical statedump
for a
> > brick has
"<brick-path>.<pid-of-brick>.dump.<timestamp>" as
its name. Could
> > you please attach them and respond?
> >
> > > Thanks,
> > >
> > > - Andreas
> > >
> > >
> > >
> > > _______________________________________________
> > > Gluster-users mailing list
> > > Gluster-users at gluster.org
> > > http://www.gluster.org/mailman/listinfo/gluster-users
> >
> > --
> > Thanks,
> > Anuradha.
> >
> 
-- 
Thanks,
Anuradha.

Gluster users - Oct 2015 - gluster 3.7.3 - volume heal info hangs - unknown heal status

[Gluster-users] gluster 3.7.3 - volume heal info hangs - unknown heal status

[Gluster-users] gluster 3.7.3 - volume heal info hangs - unknown heal status