Jim Prewett
2018-Feb-26 21:19 UTC
[Gluster-users] Problems with write-behind with large files on Gluster 3.8.4
Hello, I'm having problems when write-behind is enabled on Gluster 3.8.4. I have 2 Gluster servers each with a single brick that is mirrored between them. The code causing these issues reads two data files each approx. 128G in size. It opens a third file, mmap()'s that file, and subsequently reads and writes to it. The third file, on sucessful runs (without write-behind enabled) is ultimately approx. 224G in size. The servers have the IP addresses 172.17.2.254 and 172.17.2.255 and the client has the IP address 172.17.1.61. These are all IP over InfiniBand. I'm attaching logfiles for the brick and for the volume from each of the servers and for the client. I'm also attaching the output of "gluster volume info" and "gluster volume get <volume> all". I have only noticed problems with write-behind being enabled with this one particular workload. When I ran it under strace, I see it seeking all over the place and reading and writing little bits of data to/from the third file. For now, I'm leaving write-behind disabled. What are the performance implications of this for jobs that don't have this strange access pattern? My co-worker who usually maintains the Gluster filesystems here is busy having a baby right now and I've gotten it while he's out, so I'm /really/ new to Gluster and am not confident that anything is correct in my configuration (nor do I have a specific reason to doubt its correctness! :) I have checked the InfiniBand fabric for errors and do not see any beyond the normal PortXmitWait counter. There is no firewall on any of these machines. Their system clocks seem to all be synchronized. Is there anything additional I can provide to help diagnose this problem? Thanks for any help you can provide! :) Jim James E. Prewett Jim at Prewett.org download at hpc.unm.edu Systems Team Leader LoGS: http://www.hpc.unm.edu/~download/LoGS/ Designated Security Officer OpenPGP key: pub 1024D/31816D93 HPC Systems Engineer III UNM HPC 505.277.8210 -------------- next part -------------- Option Value ------ ----- cluster.lookup-unhashed on cluster.lookup-optimize on cluster.min-free-disk 10% cluster.min-free-inodes 5% cluster.rebalance-stats off cluster.subvols-per-directory (null) cluster.readdir-optimize off cluster.rsync-hash-regex (null) cluster.extra-hash-regex (null) cluster.dht-xattr-name trusted.glusterfs.dht cluster.randomize-hash-range-by-gfid off cluster.rebal-throttle normal cluster.lock-migration off cluster.local-volume-name (null) cluster.weighted-rebalance on cluster.switch-pattern (null) cluster.entry-change-log on cluster.read-subvolume (null) cluster.read-subvolume-index -1 cluster.read-hash-mode 1 cluster.background-self-heal-count 8 cluster.metadata-self-heal on cluster.data-self-heal on cluster.entry-self-heal on cluster.self-heal-daemon on cluster.heal-timeout 600 cluster.self-heal-window-size 1 cluster.data-change-log on cluster.metadata-change-log on cluster.data-self-heal-algorithm (null) cluster.eager-lock on disperse.eager-lock on cluster.quorum-type none cluster.quorum-count (null) cluster.choose-local true cluster.self-heal-readdir-size 1KB cluster.post-op-delay-secs 1 cluster.ensure-durability on cluster.consistent-metadata no cluster.heal-wait-queue-length 128 cluster.favorite-child-policy none cluster.stripe-block-size 128KB cluster.stripe-coalesce true diagnostics.latency-measurement on diagnostics.dump-fd-stats off diagnostics.count-fop-hits on diagnostics.brick-log-level INFO diagnostics.client-log-level INFO diagnostics.brick-sys-log-level CRITICAL diagnostics.client-sys-log-level CRITICAL diagnostics.brick-logger (null) diagnostics.client-logger (null) diagnostics.brick-log-format (null) diagnostics.client-log-format (null) diagnostics.brick-log-buf-size 5 diagnostics.client-log-buf-size 5 diagnostics.brick-log-flush-timeout 120 diagnostics.client-log-flush-timeout 120 diagnostics.stats-dump-interval 0 diagnostics.fop-sample-interval 0 diagnostics.fop-sample-buf-size 65535 diagnostics.stats-dnscache-ttl-sec 86400 performance.cache-max-file-size 0 performance.cache-min-file-size 0 performance.cache-refresh-timeout 1 performance.cache-priority performance.cache-size 8GB performance.io-thread-count 24 performance.high-prio-threads 16 performance.normal-prio-threads 16 performance.low-prio-threads 16 performance.least-prio-threads 1 performance.enable-least-priority on performance.least-rate-limit 0 performance.cache-size 8GB performance.flush-behind on performance.nfs.flush-behind on performance.write-behind-window-size 8MB performance.resync-failed-syncs-after-fsyncon performance.nfs.write-behind-window-size1MB performance.strict-o-direct off performance.nfs.strict-o-direct off performance.strict-write-ordering off performance.nfs.strict-write-ordering off performance.lazy-open yes performance.read-after-open no performance.read-ahead-page-count 4 performance.md-cache-timeout 1 performance.cache-swift-metadata true features.encryption off encryption.master-key (null) encryption.data-key-size 256 encryption.block-size 4096 network.frame-timeout 1800 network.ping-timeout 42 network.tcp-window-size (null) features.lock-heal off features.grace-timeout 10 network.remote-dio disable client.event-threads 8 network.ping-timeout 42 network.tcp-window-size (null) network.inode-lru-limit 16384 auth.allow * auth.reject (null) transport.keepalive (null) server.allow-insecure (null) server.root-squash off server.anonuid 65534 server.anongid 65534 server.statedump-path /var/run/gluster server.outstanding-rpc-limit 512 features.lock-heal off features.grace-timeout 10 server.ssl (null) auth.ssl-allow * server.manage-gids off server.dynamic-auth on client.send-gids on server.gid-timeout 300 server.own-thread (null) server.event-threads 32 ssl.own-cert (null) ssl.private-key (null) ssl.ca-list (null) ssl.crl-path (null) ssl.certificate-depth (null) ssl.cipher-list (null) ssl.dh-param (null) ssl.ec-curve (null) performance.write-behind off performance.read-ahead on performance.readdir-ahead on performance.io-cache on performance.quick-read on performance.open-behind on performance.stat-prefetch on performance.client-io-threads on performance.nfs.write-behind on performance.nfs.read-ahead off performance.nfs.io-cache off performance.nfs.quick-read off performance.nfs.stat-prefetch off performance.nfs.io-threads off performance.force-readdirp true features.uss off features.snapshot-directory .snaps features.show-snapshot-directory off network.compression off network.compression.window-size -15 network.compression.mem-level 8 network.compression.min-size 0 network.compression.compression-level -1 network.compression.debug false features.limit-usage (null) features.quota-timeout 0 features.default-soft-limit 80% features.soft-timeout 60 features.hard-timeout 5 features.alert-time 86400 features.quota-deem-statfs off geo-replication.indexing off geo-replication.indexing off geo-replication.ignore-pid-check off geo-replication.ignore-pid-check off features.quota off features.inode-quota off features.bitrot disable debug.trace off debug.log-history no debug.log-file no debug.exclude-ops (null) debug.include-ops (null) debug.error-gen off debug.error-failure (null) debug.error-number (null) debug.random-failure off debug.error-fops (null) nfs.enable-ino32 no nfs.mem-factor 15 nfs.export-dirs on nfs.export-volumes on nfs.addr-namelookup off nfs.dynamic-volumes off nfs.register-with-portmap on nfs.outstanding-rpc-limit 16 nfs.port 2049 nfs.rpc-auth-unix on nfs.rpc-auth-null on nfs.rpc-auth-allow all nfs.rpc-auth-reject none nfs.ports-insecure off nfs.trusted-sync off nfs.trusted-write off nfs.volume-access read-write nfs.export-dir nfs.disable on nfs.nlm on nfs.acl on nfs.mount-udp off nfs.mount-rmtab /var/lib/glusterd/nfs/rmtab nfs.rpc-statd /sbin/rpc.statd nfs.server-aux-gids off nfs.drc off nfs.drc-size 0x20000 nfs.read-size (1 * 1048576ULL) nfs.write-size (1 * 1048576ULL) nfs.readdir-size (1 * 1048576ULL) nfs.rdirplus on nfs.exports-auth-enable (null) nfs.auth-refresh-interval-sec (null) nfs.auth-cache-ttl-sec (null) features.read-only off features.worm off features.worm-file-level off features.default-retention-period 120 features.retention-mode relax features.auto-commit-period 180 storage.linux-aio off storage.batch-fsync-mode reverse-fsync storage.batch-fsync-delay-usec 0 storage.owner-uid -1 storage.owner-gid -1 storage.node-uuid-pathinfo off storage.health-check-interval 30 storage.build-pgfid off storage.bd-aio off config.transport tcp,rdma cluster.server-quorum-type off cluster.server-quorum-ratio 0 changelog.changelog off changelog.changelog-dir (null) changelog.encoding ascii changelog.rollover-time 15 changelog.fsync-interval 5 changelog.changelog-barrier-timeout 120 changelog.capture-del-path off features.barrier disable features.barrier-timeout 120 features.trash off features.trash-dir .trashcan features.trash-eliminate-path (null) features.trash-max-filesize 5MB features.trash-internal-op off cluster.enable-shared-storage disable cluster.write-freq-threshold 0 cluster.read-freq-threshold 0 cluster.tier-pause off cluster.tier-promote-frequency 120 cluster.tier-demote-frequency 3600 cluster.watermark-hi 90 cluster.watermark-low 75 cluster.tier-mode cache cluster.tier-max-promote-file-size 0 cluster.tier-max-mb 4000 cluster.tier-max-files 10000 features.ctr-enabled off features.record-counters off features.ctr-record-metadata-heat off features.ctr_link_consistency off features.ctr_lookupheal_link_timeout 300 features.ctr_lookupheal_inode_timeout 300 features.ctr-sql-db-cachesize 1000 features.ctr-sql-db-wal-autocheckpoint 1000 locks.trace off locks.mandatory-locking off cluster.disperse-self-heal-daemon enable cluster.quorum-reads no client.bind-insecure (null) ganesha.enable off features.shard off features.shard-block-size 4MB features.scrub-throttle lazy features.scrub-freq biweekly features.scrub false features.expiry-time 120 features.cache-invalidation off features.cache-invalidation-timeout 60 features.leases off features.lease-lock-recall-timeout 60 disperse.background-heals 8 disperse.heal-wait-qlength 128 cluster.heal-timeout 600 dht.force-readdirp on disperse.read-policy round-robin cluster.shd-max-threads 1 cluster.shd-wait-qlength 1024 cluster.locking-scheme full cluster.granular-entry-heal no -------------- next part -------------- Volume Name: scratch Type: Stripe Volume ID: 38f6c032-b6bf-4923-8751-5bd93650e2d0 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp,rdma Bricks: Brick1: server01ib:/data/brick1/scratch Brick2: server02ib:/data/brick1/scratch Options Reconfigured: config.transport: tcp,rdma performance.cache-size: 8GB diagnostics.count-fop-hits: on diagnostics.latency-measurement: on performance.readdir-ahead: on cluster.lookup-optimize: on client.event-threads: 8 performance.client-io-threads: on performance.io-thread-count: 24 performance.write-behind-window-size: 8MB server.event-threads: 32 performance.write-behind: off performance.stat-prefetch: on performance.resync-failed-syncs-after-fsync: on server.outstanding-rpc-limit: 512 -------------- next part -------------- [2018-02-22 18:07:45.241970] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022 [2018-02-22 18:07:45.242018] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.242681] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022 [2018-02-22 18:07:45.242704] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.242728] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022 [2018-02-22 18:07:45.242737] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.242753] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022 [2018-02-22 18:07:45.242762] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.242786] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022 [2018-02-22 18:07:45.242795] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.243058] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022 [2018-02-22 18:07:45.243071] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.243090] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022 [2018-02-22 18:07:45.243098] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.243116] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022 [2018-02-22 18:07:45.243124] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.243141] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022 [2018-02-22 18:07:45.243149] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.243384] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022 [2018-02-22 18:07:45.243396] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.243416] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f6462b85000, wc.byte_len = 45056, post->reused = 1 [2018-02-22 18:07:45.243692] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f6462b85000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:45.244143] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f6461e73000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:47.159515] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f6462b7d000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:47.171016] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f6462b75000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:47.171548] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f6462b77000, wc.byte_len = 131072, post->reused = 1 -------------- next part -------------- [2018-02-22 18:07:45.241029] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023 [2018-02-22 18:07:45.241095] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.241269] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023 [2018-02-22 18:07:45.241333] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.241375] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023 [2018-02-22 18:07:45.241394] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.241461] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023 [2018-02-22 18:07:45.241482] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.241512] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023 [2018-02-22 18:07:45.241529] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2018-02-22 18:07:45.241560] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f10e7351000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:45.242274] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f10e7349000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:47.158767] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f10e733d000, wc.byte_len = 131072, post->reused = 1 -------------- next part -------------- [2018-02-22 18:07:45.221277] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.221309] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.221320] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54987 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.221334] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.221365] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.221368] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2018-02-22 18:07:45.221576] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-02-22 18:07:45.210946 (xid=0x54974) [2018-02-22 18:07:45.221596] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.224781] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54988 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.224902] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54989 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.225011] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.225074] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.225630] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.225697] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.225753] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.225801] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.225830] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54990 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.226338] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54991 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.226383] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54992 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.226452] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54993 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.226947] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54994 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.227452] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54995 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.227491] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54996 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.227524] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54997 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.227568] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54998 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.227615] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54999 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.227654] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.227738] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.227801] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.227863] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.227920] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.227972] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.228767] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.228803] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790331: READ => -1 gfid=3a86afe5-7392-49a5-b60a-e0c93b050c01 fd=0x2b2f23f023b4 (Transport endpoint is not connected) [2018-02-22 18:07:45.228826] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.228864] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790330: READ => -1 gfid=3a86afe5-7392-49a5-b60a-e0c93b050c01 fd=0x2b2f23f023b4 (Transport endpoint is not connected) [2018-02-22 18:07:45.228872] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.228919] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.228966] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.228989] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument] [2018-02-22 18:07:45.229009] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.229050] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.229102] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.229122] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.220065 (xid=0x54986) [2018-02-22 18:07:45.229151] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.229193] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.229408] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549aa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 35 times between [2018-02-22 18:07:45.221365] and [2018-02-22 18:07:45.229445] [2018-02-22 18:07:45.229449] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154) [2018-02-22 18:07:45.229488] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ab Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 37 times between [2018-02-22 18:07:45.221334] and [2018-02-22 18:07:45.229507] [2018-02-22 18:07:45.229524] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.229511] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed [2018-02-22 18:07:45.229549] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54385 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.229561] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.229572] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ac Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed" repeated 37 times between [2018-02-22 18:07:45.221309] and [2018-02-22 18:07:45.229571] [2018-02-22 18:07:45.229587] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.229583] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.229620] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2018-02-22 18:07:45.229623] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop The message "W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)" repeated 38 times between [2018-02-22 18:07:45.221277] and [2018-02-22 18:07:45.229677] [2018-02-22 18:07:45.229678] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.229702] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ad Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.229713] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.229736] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.220053 (xid=0x5437b) [2018-02-22 18:07:45.229753] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.229769] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.229800] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.229812] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ae Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.229866] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.220767 (xid=0x5437c) The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.229623] and [2018-02-22 18:07:45.229838] [2018-02-22 18:07:45.229883] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.229867] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.229903] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549af Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.229713] and [2018-02-22 18:07:45.229913] [2018-02-22 18:07:45.229928] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.229978] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.229991] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed" repeated 2 times between [2018-02-22 18:07:45.229800] and [2018-02-22 18:07:45.229990] [2018-02-22 18:07:45.230004] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.222057 (xid=0x5437d) [2018-02-22 18:07:45.230037] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.230004] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.230040] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.230083] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.230088] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.230126] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154) [2018-02-22 18:07:45.230130] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.222694 (xid=0x5437e) [2018-02-22 18:07:45.230129] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed [2018-02-22 18:07:45.230149] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.230163] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54386 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.230165] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.230176] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.230004] and [2018-02-22 18:07:45.230183] [2018-02-22 18:07:45.230193] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.230187] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154) The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.230088] and [2018-02-22 18:07:45.230195] [2018-02-22 18:07:45.230220] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.230214] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed [2018-02-22 18:07:45.230236] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.230242] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54387 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.230234] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.230245] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.230255] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.230257] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.230267] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.224869 (xid=0x5437f) [2018-02-22 18:07:45.230273] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.230264] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.230274] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.230308] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.230314] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.230344] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed" repeated 2 times between [2018-02-22 18:07:45.230257] and [2018-02-22 18:07:45.230367] [2018-02-22 18:07:45.230368] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.230377] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.230389] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.225018 (xid=0x54380) The message "W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)" repeated 2 times between [2018-02-22 18:07:45.230314] and [2018-02-22 18:07:45.230420] [2018-02-22 18:07:45.230423] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.230422] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.230424] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154) [2018-02-22 18:07:45.230454] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.230466] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed [2018-02-22 18:07:45.230476] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54388 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.230497] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154) [2018-02-22 18:07:45.230502] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.226409 (xid=0x54381) [2018-02-22 18:07:45.230514] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54389 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 7 times between [2018-02-22 18:07:45.229883] and [2018-02-22 18:07:45.230485] [2018-02-22 18:07:45.230492] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.230513] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed [2018-02-22 18:07:45.230536] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.230535] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.230542] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.230557] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.230565] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.230570] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154) [2018-02-22 18:07:45.230588] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.230572] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed [2018-02-22 18:07:45.230600] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 5 times between [2018-02-22 18:07:45.230245] and [2018-02-22 18:07:45.230600] [2018-02-22 18:07:45.230624] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed" repeated 2 times between [2018-02-22 18:07:45.230422] and [2018-02-22 18:07:45.230623] [2018-02-22 18:07:45.230612] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.230625] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154) [2018-02-22 18:07:45.230618] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.230635] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.230652] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.227685 (xid=0x54382) [2018-02-22 18:07:45.230665] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.230659] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed [2018-02-22 18:07:45.230660] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.230612] and [2018-02-22 18:07:45.230674] The message "W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)" repeated 2 times between [2018-02-22 18:07:45.230565] and [2018-02-22 18:07:45.230688] [2018-02-22 18:07:45.230677] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.230690] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.230717] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ba Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.230728] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.230740] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.230763] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549bb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.230764] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.228029 (xid=0x54383) [2018-02-22 18:07:45.230762] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.230773] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.230784] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.230782] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.230786] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.230788] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154) [2018-02-22 18:07:45.230804] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549bc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.230677] and [2018-02-22 18:07:45.230802] [2018-02-22 18:07:45.230829] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.230819] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed [2018-02-22 18:07:45.230845] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.230864] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.230867] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.230852] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.230874] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549bd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.230898] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-02-22 18:07:45.229052 (xid=0x54384) [2018-02-22 18:07:45.230900] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.230914] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.230914] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.230901] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.230660] and [2018-02-22 18:07:45.230923] [2018-02-22 18:07:45.230928] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.230946] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549be Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.230956] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.230961] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790385: READ => -1 gfid=3a86afe5-7392-49a5-b60a-e0c93b050c01 fd=0x2b2f23f023b4 (Transport endpoint is not connected) [2018-02-22 18:07:45.230960] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.230972] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.230986] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.230990] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.231005] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549bf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.230990] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154) [2018-02-22 18:07:45.231008] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed [2018-02-22 18:07:45.231015] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231032] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:45.231042] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument] [2018-02-22 18:07:45.231029] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.231031] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.231073] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231080] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.231078] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:45.231085] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154) [2018-02-22 18:07:45.231104] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.231099] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231112] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.231123] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.231130] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.231130] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231144] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.231136] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.231164] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.231148] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.231149] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.231175] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231184] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.231207] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.231209] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.231210] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.231211] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.231219] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231238] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.231231] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.231255] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231264] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54390 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.231261] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.231261] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.231275] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231283] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.231279] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.231297] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.231303] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.231306] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.231310] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.231324] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.231333] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54391 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.231326] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231340] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.231341] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231335] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.231377] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.231388] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.231383] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.231381] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b2t1_0001.fits (3a86afe5-7392-49a5-b60a-e0c93b050c01) [Transport endpoint is not connected] [2018-02-22 18:07:45.231417] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.231419] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231432] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.231434] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b2t1_0001.fits (3a86afe5-7392-49a5-b60a-e0c93b050c01) [Transport endpoint is not connected] [2018-02-22 18:07:45.231446] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.231452] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.231463] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 2-scratch-dht: Found anomalies in /download/work/kstovall/57910/drx_57910_B0656+14_b2t1_0001.fits (gfid = 3a86afe5-7392-49a5-b60a-e0c93b050c01). Holes=1 overlaps=0 [2018-02-22 18:07:45.231456] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231471] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.231488] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: 3a86afe5-7392-49a5-b60a-e0c93b050c01: failed to resolve (Transport endpoint is not connected) [2018-02-22 18:07:45.231490] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.231486] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.231501] E [fuse-bridge.c:796:fuse_getattr_resume] 0-glusterfs-fuse: 790393: GETATTR 47480857159808 (3a86afe5-7392-49a5-b60a-e0c93b050c01) resolution failed [2018-02-22 18:07:45.231494] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.231509] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.231516] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54392 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.231515] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.231520] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231540] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.231538] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231542] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.231564] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.231570] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54393 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.231580] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.231587] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231594] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.231604] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.231617] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.231610] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.231626] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.231638] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231638] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.231664] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54394 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.231653] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.231667] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.231688] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231694] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.231696] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.231714] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231715] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.231721] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.231747] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.231751] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.231765] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.231789] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231800] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.231801] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.231816] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.231823] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54395 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.231824] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ca Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.231822] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.231833] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231828] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.231850] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.231843] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231865] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549cb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.231863] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.231873] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.231879] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231885] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.231899] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.231917] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.231919] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.231922] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.231928] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54396 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.231925] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.231954] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549cc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.231959] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.231949] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231967] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.231985] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.232000] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54398 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.231999] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.232002] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.232025] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.232025] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.232021] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232039] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.232048] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549cd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.232050] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.232060] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54397 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.232057] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232064] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.232075] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232073] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.232092] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.232098] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.232100] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.232104] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.232114] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ce Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.232121] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.232136] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54399 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.232138] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232159] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.232156] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.232154] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232183] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.232194] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549cf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.232203] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232214] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.232216] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.232226] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.232226] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.232241] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.232249] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232262] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.232267] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.232268] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.232279] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.232276] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.232302] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.232303] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232314] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.232320] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.232329] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.232328] W [MSGID: 114031] [client-rpc-fops.c:1495:client3_3_fstat_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232335] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.232344] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.232334] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232356] W [MSGID: 114031] [client-rpc-fops.c:1495:client3_3_fstat_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232364] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232400] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232404] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.232448] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.232456] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.232471] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.232479] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.232476] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.232493] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.232496] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.232268] and [2018-02-22 18:07:45.232525] [2018-02-22 18:07:45.232532] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.232552] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.232564] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.232571] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.232364] and [2018-02-22 18:07:45.232579] [2018-02-22 18:07:45.232592] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.232592] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.232632] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.232654] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.232668] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232680] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.232708] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.232703] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.232706] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.232726] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.232757] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.233456] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.233454] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.233478] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.233485] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.233494] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.233483] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.233498] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.233513] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.233540] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.233532] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.233580] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.233592] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.233602] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.233614] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.233630] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.233670] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.233692] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.233690] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.233706] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.233739] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.233756] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.233755] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.233766] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.233756] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.233795] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.233833] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.233835] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.233860] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.233871] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.233892] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.233900] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.233908] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.233926] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.233936] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.233937] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.233939] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.233960] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.233967] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.233978] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.233996] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.234011] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.234023] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.234016] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.234035] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.234065] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.234077] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.234079] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.234096] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.234100] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.234106] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.234119] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.234115] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.234136] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.234146] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.234154] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.234157] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.234174] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.234184] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.234197] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.234220] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.234218] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.234231] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.234227] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.234247] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.234256] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549da Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.234262] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop" repeated 3 times between [2018-02-22 18:07:45.234065] and [2018-02-22 18:07:45.234264] [2018-02-22 18:07:45.234283] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.234511] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.234513] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.234530] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.234540] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.234556] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.234567] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.234572] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.234583] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.234593] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.234613] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.234622] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.234629] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.234638] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.234646] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.234670] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.234677] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.234692] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.234707] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.234716] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.234725] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.234738] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.234746] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543aa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.234751] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.234754] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.234784] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.234828] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.234834] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.234842] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.234859] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ab Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.234867] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.234879] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.234880] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.234881] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.234907] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ac Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.234917] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.234929] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.234923] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.234954] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ad Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.234953] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.234965] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.234988] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.234996] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.234998] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.235015] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ae Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.235024] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.235037] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.235059] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.235075] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.235083] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543af Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.235094] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.235119] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.235182] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.235197] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.235196] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.235205] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.235231] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.235258] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.235260] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.235273] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.235280] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.235297] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.235321] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.235336] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.235344] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.235352] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.235362] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.235399] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.235413] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.235412] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.235423] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.235436] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.235444] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.235460] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.235465] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.235486] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.235495] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.235501] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.235514] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.235529] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.235537] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.235558] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.236105] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2018-02-22 18:07:45.236245] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234482 (xid=0x549dc) [2018-02-22 18:07:45.236264] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.236558] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234491 (xid=0x549de) [2018-02-22 18:07:45.236672] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234655 (xid=0x549df) [2018-02-22 18:07:45.236899] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234699 (xid=0x549e0) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2018-02-22 18:07:45.236264] and [2018-02-22 18:07:45.236914] [2018-02-22 18:07:45.236931] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237010] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234804 (xid=0x549e1) [2018-02-22 18:07:45.237025] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237045] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.237061] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.237069] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ee Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.237076] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237135] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234822 (xid=0x549e2) [2018-02-22 18:07:45.237357] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(PORTMAP) op(PORTBYBRICK(1)) called at 2018-02-22 18:07:45.234954 (xid=0x549e3) [2018-02-22 18:07:45.237150] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237431] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.237430] W [MSGID: 114032] [client-handshake.c:1508:client_query_portmap_cbk] 2-scratch-client-0: received RPC status error, try again later [Transport endpoint is not connected] [2018-02-22 18:07:45.237450] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.237458] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ef Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.237466] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.234283] and [2018-02-22 18:07:45.237488] [2018-02-22 18:07:45.237510] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.237529] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.237541] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.237541] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234979 (xid=0x549e4) [2018-02-22 18:07:45.237549] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237555] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237566] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.237554] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.237585] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.237584] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.237594] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237623] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.237657] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.237658] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.237675] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.237673] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235016 (xid=0x549e5) [2018-02-22 18:07:45.237690] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237696] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237698] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.237710] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.237721] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.237736] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.237748] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237763] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.237771] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.237765] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.237784] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.237799] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235054 (xid=0x549e6) [2018-02-22 18:07:45.237797] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237805] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.237813] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.237827] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.237814] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237835] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.237842] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237862] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.237886] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.237888] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.237905] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.237913] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237926] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.237934] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235159 (xid=0x549e7) [2018-02-22 18:07:45.237942] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.237941] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237953] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.237950] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.237965] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument] The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.237862] and [2018-02-22 18:07:45.237976] [2018-02-22 18:07:45.237986] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238005] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238012] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.237950] and [2018-02-22 18:07:45.238020] [2018-02-22 18:07:45.238034] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238038] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument] [2018-02-22 18:07:45.238033] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238070] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238068] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238079] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235589 (xid=0x549e8) [2018-02-22 18:07:45.238080] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238097] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238100] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238110] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238122] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238111] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238129] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238147] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238174] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238183] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238182] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238195] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238207] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238208] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238220] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238214] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235705 (xid=0x549e9) [2018-02-22 18:07:45.238229] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.238195] and [2018-02-22 18:07:45.238243] [2018-02-22 18:07:45.238247] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238253] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238266] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238278] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238287] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238297] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238302] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238303] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238327] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fe Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238337] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238357] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235740 (xid=0x549ea) [2018-02-22 18:07:45.238381] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238392] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238411] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238418] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ff Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238424] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.238297] and [2018-02-22 18:07:45.238446] [2018-02-22 18:07:45.238469] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235777 (xid=0x549eb) [2018-02-22 18:07:45.238465] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238485] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238491] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238507] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a00 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238516] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238530] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238536] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238531] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238556] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a01 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238579] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235812 (xid=0x549ec) [2018-02-22 18:07:45.238566] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238583] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238579] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238595] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238610] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238621] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a02 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238629] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238643] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238651] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238644] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238672] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a03 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238682] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238692] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238696] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238709] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235840 (xid=0x549ed) [2018-02-22 18:07:45.238711] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238729] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a05 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238732] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238739] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238752] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238752] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238761] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238788] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a04 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238798] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238812] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238823] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238824] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238837] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a06 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238847] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238862] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238869] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238865] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238885] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a07 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238893] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238903] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238907] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238922] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238935] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238943] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.238956] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.238964] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.238958] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.238977] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a09 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.238990] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239004] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239007] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239024] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239038] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a08 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239047] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239060] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239069] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239062] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239081] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239090] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239100] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239105] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239119] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239132] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239140] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239153] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239160] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239155] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239174] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239183] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239190] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239207] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239208] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239221] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239233] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239243] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239244] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239254] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239259] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239268] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.239279] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239270] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.239286] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239296] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ce Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.239293] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239307] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a10 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239305] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239309] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2018-02-22 18:07:45.239328] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239333] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.239345] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239330] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239361] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a11 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239383] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2018-02-22 18:07:45.239233] and [2018-02-22 18:07:45.239384] [2018-02-22 18:07:45.239410] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239401] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239428] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a12 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239432] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235590 (xid=0x543b8) [2018-02-22 18:07:45.239449] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239453] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239457] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239459] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239483] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a13 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239484] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239500] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239506] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239523] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239528] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239541] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a14 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239551] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239557] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235614 (xid=0x543b9) [2018-02-22 18:07:45.239573] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239573] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239574] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239589] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239607] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a17 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239616] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239625] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.239633] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239638] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.239644] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239650] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239657] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543cf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.239666] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a16 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239676] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239682] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235622 (xid=0x543ba) [2018-02-22 18:07:45.239685] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239696] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239704] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.239712] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239733] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239748] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239758] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a15 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239769] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239789] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239799] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235647 (xid=0x543bb) [2018-02-22 18:07:45.239798] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.239803] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239826] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.239824] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.239833] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a18 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239797] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239827] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239835] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239856] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.239851] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239881] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239884] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239899] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a19 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239898] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239911] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239921] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239925] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.239932] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.239936] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.239947] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.239947] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.239950] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239954] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235673 (xid=0x543bc) [2018-02-22 18:07:45.239956] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239969] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239969] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.239988] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.239994] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.239998] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.240006] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240016] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240025] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.240026] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240036] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.240035] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.240043] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240039] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240058] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240062] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240073] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240077] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.240082] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235687 (xid=0x543bd) [2018-02-22 18:07:45.240100] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240099] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240084] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240115] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240098] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240127] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240136] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240149] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240164] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240166] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240173] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240184] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240193] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240203] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240210] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240211] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240220] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240222] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235708 (xid=0x543be) [2018-02-22 18:07:45.240237] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240239] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240243] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240257] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240267] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a20 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240275] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240286] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240291] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.240297] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240311] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a21 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240316] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.240321] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240335] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.240339] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240342] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235728 (xid=0x543bf) [2018-02-22 18:07:45.240353] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240368] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.240368] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240357] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240406] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.240360] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240424] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.240426] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240428] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.240438] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a22 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240435] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240449] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240452] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.240460] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240471] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.240473] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.240484] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a23 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240490] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.240487] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235746 (xid=0x543c0) [2018-02-22 18:07:45.240482] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240507] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240513] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.240490] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240517] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240493] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240533] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.240541] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240548] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.240558] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240561] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240567] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.240580] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240589] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.240592] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a24 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240602] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed" repeated 2 times between [2018-02-22 18:07:45.240473] and [2018-02-22 18:07:45.240601] [2018-02-22 18:07:45.240616] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235760 (xid=0x543c1) [2018-02-22 18:07:45.240611] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240611] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240622] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240633] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240656] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240661] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240668] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a25 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240676] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.240677] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240689] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240702] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240710] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a26 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240719] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240719] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240732] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240746] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240757] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a27 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240761] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240764] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235783 (xid=0x543c2) [2018-02-22 18:07:45.240768] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240777] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240777] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240797] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240805] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.240815] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.240798] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240828] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.240831] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a28 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240835] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240841] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240854] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240857] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.240870] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240880] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a29 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240874] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240888] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240899] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240913] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240910] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.236943 (xid=0x543c3) [2018-02-22 18:07:45.240911] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240920] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240919] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240924] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.240950] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.240956] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.240965] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.240986] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.240978] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.240999] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.241006] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.241001] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241015] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241033] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.241038] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(PORTMAP) op(PORTBYBRICK(1)) called at 2018-02-22 18:07:45.237021 (xid=0x543c4) [2018-02-22 18:07:45.241034] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.241048] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.241055] W [MSGID: 114032] [client-handshake.c:1508:client_query_portmap_cbk] 2-scratch-client-1: received RPC status error, try again later [Transport endpoint is not connected] [2018-02-22 18:07:45.241043] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.241068] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543da Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.241079] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241080] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.241102] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.241106] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.241113] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.241108] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.241111] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.241130] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543db Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.241142] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241149] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.237611 (xid=0x543c5) [2018-02-22 18:07:45.241149] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241150] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.241166] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.241173] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.241172] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.241174] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241187] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.241189] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241221] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.241245] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.241259] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.241263] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.241273] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.241274] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.241282] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543dc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.241282] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241291] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241288] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.241295] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.241293] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.237634 (xid=0x543c6) [2018-02-22 18:07:45.241313] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.241315] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.241321] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.241330] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.241335] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543dd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.241334] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.241335] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241351] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.241357] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.241344] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241348] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241378] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.241389] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a30 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.241389] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.241408] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.241405] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241421] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.241430] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a31 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.241434] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.241439] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241442] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.241442] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.241474] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543de Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.241483] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.237740 (xid=0x543c7) [2018-02-22 18:07:45.241477] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.241499] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.241485] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241500] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241516] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.241528] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543df Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.241537] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.241539] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241552] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.241555] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.241568] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.241579] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.241587] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241595] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.241598] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.241611] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.241612] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.241618] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a32 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.241628] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.241621] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-02-22 18:07:45.237917 (xid=0x543c8) [2018-02-22 18:07:45.241616] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.241637] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241648] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241650] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.241649] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.241636] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241672] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.241674] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.241680] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790398: READ => -1 gfid=cb243c85-02fe-4e37-b03f-80f9187e72df fd=0x2b2f23f02210 (Transport endpoint is not connected) [2018-02-22 18:07:45.241690] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a33 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.241685] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.241704] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241708] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.241706] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241722] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.241726] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.241741] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed" repeated 2 times between [2018-02-22 18:07:45.241612] and [2018-02-22 18:07:45.241740] [2018-02-22 18:07:45.241741] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.241753] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument] [2018-02-22 18:07:45.241754] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241767] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.241786] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a34 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.241784] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.241791] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.241784] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.241795] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241810] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.241820] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241824] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) The message "W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)" repeated 2 times between [2018-02-22 18:07:45.241726] and [2018-02-22 18:07:45.241840] [2018-02-22 18:07:45.241846] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.241855] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.241848] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-02-22 18:07:45.238059 (xid=0x543c9) [2018-02-22 18:07:45.241842] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.241826] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.241881] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.241885] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a35 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.241886] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241890] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241894] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241910] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.241908] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.241927] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790399: READ => -1 gfid=cb243c85-02fe-4e37-b03f-80f9187e72df fd=0x2b2f23f02210 (Transport endpoint is not connected) [2018-02-22 18:07:45.241907] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.241940] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.241947] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a36 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.241945] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.241962] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.241962] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.241980] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.241974] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.241991] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument] [2018-02-22 18:07:45.241998] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.242004] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.242008] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a37 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.242008] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.242000] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242023] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242018] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.242040] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.242049] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242054] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.242063] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.242067] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.242065] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.242084] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.238080 (xid=0x543ca) [2018-02-22 18:07:45.242095] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.242086] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.242101] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242107] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242109] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.242125] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.242130] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.242132] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.242137] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.242141] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a38 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.242156] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.242162] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242164] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242176] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.242191] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.242199] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a39 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.242201] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.242204] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.242223] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.238146 (xid=0x543cb) [2018-02-22 18:07:45.242220] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242249] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242246] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242252] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.242274] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.242277] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.242288] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ea Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.242299] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument] [2018-02-22 18:07:45.242312] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.242313] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.242332] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.242330] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790424: READ => -1 gfid=cb243c85-02fe-4e37-b03f-80f9187e72df fd=0x2b2f23f02210 (Transport endpoint is not connected) [2018-02-22 18:07:45.242341] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.242354] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543eb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.242356] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.242366] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.242246] and [2018-02-22 18:07:45.242363] [2018-02-22 18:07:45.242384] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242369] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.242403] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.242408] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.238325 (xid=0x543cc) [2018-02-22 18:07:45.242408] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.242436] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.242435] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.242431] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242436] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.242460] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.242461] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242476] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.242468] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.242489] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.242489] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.242493] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ec Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.242497] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242526] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.242528] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.242511] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242530] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.242530] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.242524] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.242552] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.242555] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.238346 (xid=0x543cd) [2018-02-22 18:07:45.242567] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.242578] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ed Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.242571] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242579] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242594] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.242607] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.242605] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.242638] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ee Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.242579] and [2018-02-22 18:07:45.242647] [2018-02-22 18:07:45.242658] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.242680] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ef Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed" repeated 2 times between [2018-02-22 18:07:45.242567] and [2018-02-22 18:07:45.242679] [2018-02-22 18:07:45.242689] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242701] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.242702] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.242718] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop" repeated 3 times between [2018-02-22 18:07:45.242607] and [2018-02-22 18:07:45.242733] [2018-02-22 18:07:45.242748] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.242740] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.242762] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.242761] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.242762] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.242771] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b2t2_0001.fits (cb243c85-02fe-4e37-b03f-80f9187e72df) [Transport endpoint is not connected] [2018-02-22 18:07:45.242782] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.242689] and [2018-02-22 18:07:45.242777] [2018-02-22 18:07:45.242797] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.242795] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.242801] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b2t2_0001.fits (cb243c85-02fe-4e37-b03f-80f9187e72df) [Transport endpoint is not connected] [2018-02-22 18:07:45.242795] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242817] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.242831] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 2-scratch-dht: Found anomalies in /download/work/kstovall/57910/drx_57910_B0656+14_b2t2_0001.fits (gfid = cb243c85-02fe-4e37-b03f-80f9187e72df). Holes=1 overlaps=0 [2018-02-22 18:07:45.242836] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.242845] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242845] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.242853] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.242857] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: cb243c85-02fe-4e37-b03f-80f9187e72df: failed to resolve (Transport endpoint is not connected) [2018-02-22 18:07:45.242870] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.242845] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.242875] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.242879] E [fuse-bridge.c:796:fuse_getattr_resume] 0-glusterfs-fuse: 790426: GETATTR 47480857159432 (cb243c85-02fe-4e37-b03f-80f9187e72df) resolution failed [2018-02-22 18:07:45.242879] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242906] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.242913] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.242917] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.242917] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.242927] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.242934] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a40 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.242937] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.242952] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.242969] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.242979] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.242979] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.242989] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a41 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.242998] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243012] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.243018] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243035] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.243047] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a42 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.243050] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243050] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.243060] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243068] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.243070] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.243084] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a43 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.243087] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.243077] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.243082] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.243104] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243089] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243106] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243128] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.243132] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.243140] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.243141] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.243153] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a44 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.243136] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243159] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.243168] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.243171] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243177] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243188] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.243203] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.243203] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.243215] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.243207] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243226] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243236] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.243247] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.243256] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.243255] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.243258] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.243265] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243257] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.243287] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a45 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.243289] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.243294] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.243296] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243321] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.243318] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.243327] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.243346] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.243355] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a46 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.243356] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243346] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243364] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243388] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.243383] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.243388] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243405] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.243419] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a47 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.243428] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243442] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.243447] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243444] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.243466] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a48 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.243475] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243487] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.243503] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243507] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.243520] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a49 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.243528] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243535] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.243540] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243537] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.243559] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.243569] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243582] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.243574] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.243588] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243604] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.243598] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.243616] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.243621] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.243624] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243631] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243646] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.243649] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243648] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.243656] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.243679] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.243688] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.243688] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243698] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.243700] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.243707] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.243717] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.243729] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.243733] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243735] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243739] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243759] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.243764] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.243766] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.243773] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.243785] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243786] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.243791] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.243797] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.243799] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243806] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243816] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.243832] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.243838] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.243833] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243852] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.243862] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243872] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.243881] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.243889] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fe Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.243889] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.243897] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243916] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.243918] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.243928] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.243934] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.243940] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.243952] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.243958] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.243962] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243967] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.243973] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.243985] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.243992] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244002] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ff Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244003] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244010] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244023] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244027] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244043] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244057] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54400 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244065] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244075] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244076] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244092] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54401 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244092] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244100] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244112] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244118] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244136] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244149] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54402 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244158] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244167] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244168] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244183] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54403 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244185] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244196] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244204] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244223] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244231] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.244228] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244245] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.244250] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54404 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244254] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a50 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.244264] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244265] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244271] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244279] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.244273] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244274] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.244293] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54405 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244297] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.244302] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244310] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a51 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.244314] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244320] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244322] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244325] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244341] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54406 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244344] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244351] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244363] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.244364] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244383] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244391] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54407 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244388] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244403] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244413] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244425] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244432] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54408 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244430] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.244445] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244443] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244460] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.244473] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a52 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.244476] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244483] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244496] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.244517] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.244522] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.244533] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a53 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.244542] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244552] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.244561] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.244568] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a54 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.244579] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.244579] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244590] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.244603] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.244610] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a55 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.244620] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244621] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.244633] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.244644] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.244651] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a56 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.244660] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244663] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.244674] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.244676] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244692] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244699] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54409 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244708] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244724] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244734] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244741] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244745] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244750] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244762] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244775] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244774] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244788] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244784] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244797] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244806] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244824] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244832] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244837] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244843] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244858] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244865] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244866] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244883] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244886] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.244899] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.244903] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.244915] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.244918] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.244933] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a57 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.244832] and [2018-02-22 18:07:45.244923] [2018-02-22 18:07:45.244949] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.244951] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.244941] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244967] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a58 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.244965] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.244977] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.244976] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.244991] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.244993] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.245009] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a59 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.245020] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.245017] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.245017] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.245042] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.245046] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.245051] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.245059] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.245068] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.245078] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.245077] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.245087] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.245097] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.245099] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.245101] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.245126] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.245129] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.245134] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.245156] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.245169] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.245177] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.245180] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.245186] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.245196] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.245201] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.245208] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.245212] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.245216] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.245225] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.245232] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54410 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.245230] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.245248] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.245254] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.245248] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.245272] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.245281] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54411 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.245288] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.245289] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.245298] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:45.245314] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54412 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:45.245313] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:45.245322] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.245339] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.245367] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:45.245389] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.245404] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.245422] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.245432] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.245443] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.245454] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.245461] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.245468] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.245471] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.245499] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.245507] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.245507] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.245522] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a60 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.245529] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.245567] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.245592] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:45.245601] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:45.245608] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a61 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:45.245614] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:45.245654] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:45.246260] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-0: changing port to 49154 (from 0) [2018-02-22 18:07:45.247605] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-1: changing port to 49154 (from 0) [2018-02-22 18:07:45.254977] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-02-22 18:07:45.255615] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-0: Connected to scratch-client-0, attached to remote volume '/data/brick1/scratch'. [2018-02-22 18:07:45.255630] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-0: Server and Client lk-version numbers are not same, reopening the fds [2018-02-22 18:07:45.255641] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-0: 4 fds open - Delaying child_up until they are re-opened [2018-02-22 18:07:45.255830] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-02-22 18:07:45.256099] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP [2018-02-22 18:07:45.256197] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-0: Server lk version = 1 [2018-02-22 18:07:45.256226] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-1: Connected to scratch-client-1, attached to remote volume '/data/brick1/scratch'. [2018-02-22 18:07:45.256238] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-1: Server and Client lk-version numbers are not same, reopening the fds [2018-02-22 18:07:45.256247] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-1: 4 fds open - Delaying child_up until they are re-opened [2018-02-22 18:07:45.256661] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP [2018-02-22 18:07:45.256739] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-1: Server lk version = 1 [2018-02-22 18:07:46.186045] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2018-02-22 18:07:46.186183] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184758 (xid=0x54a6e) [2018-02-22 18:07:46.186197] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:46.186286] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184793 (xid=0x54a6f) [2018-02-22 18:07:46.186379] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184809 (xid=0x54a70) [2018-02-22 18:07:46.186472] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184824 (xid=0x54a71) [2018-02-22 18:07:46.186557] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184837 (xid=0x54a72) [2018-02-22 18:07:46.186641] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184851 (xid=0x54a73) [2018-02-22 18:07:46.186718] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184881 (xid=0x54a74) [2018-02-22 18:07:46.186794] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184919 (xid=0x54a75) [2018-02-22 18:07:46.186879] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184933 (xid=0x54a76) [2018-02-22 18:07:46.186963] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184948 (xid=0x54a77) [2018-02-22 18:07:46.187044] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184971 (xid=0x54a78) [2018-02-22 18:07:46.187128] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184992 (xid=0x54a79) [2018-02-22 18:07:46.187206] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185049 (xid=0x54a7a) [2018-02-22 18:07:46.187282] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185090 (xid=0x54a7b) [2018-02-22 18:07:46.187367] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185110 (xid=0x54a7c) [2018-02-22 18:07:46.187454] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185124 (xid=0x54a7d) [2018-02-22 18:07:46.187547] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185137 (xid=0x54a7e) [2018-02-22 18:07:46.187635] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185164 (xid=0x54a7f) [2018-02-22 18:07:46.187712] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185186 (xid=0x54a80) [2018-02-22 18:07:46.187799] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185308 (xid=0x54a81) [2018-02-22 18:07:46.187886] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185351 (xid=0x54a82) [2018-02-22 18:07:46.187973] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185391 (xid=0x54a83) [2018-02-22 18:07:46.188050] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185410 (xid=0x54a84) [2018-02-22 18:07:46.188126] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185447 (xid=0x54a85) [2018-02-22 18:07:46.188203] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185462 (xid=0x54a86) [2018-02-22 18:07:46.188281] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185485 (xid=0x54a87) [2018-02-22 18:07:46.188369] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185498 (xid=0x54a88) [2018-02-22 18:07:46.188470] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185514 (xid=0x54a89) [2018-02-22 18:07:46.188757] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184866 (xid=0x5441f) [2018-02-22 18:07:46.188678] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2018-02-22 18:07:46.188773] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:46.188866] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184890 (xid=0x54420) [2018-02-22 18:07:46.188960] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184903 (xid=0x54421) [2018-02-22 18:07:46.189053] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184958 (xid=0x54422) [2018-02-22 18:07:46.189149] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184985 (xid=0x54423) [2018-02-22 18:07:46.189243] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185005 (xid=0x54424) [2018-02-22 18:07:46.189338] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185019 (xid=0x54425) [2018-02-22 18:07:46.189446] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185032 (xid=0x54426) [2018-02-22 18:07:46.189543] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185058 (xid=0x54427) [2018-02-22 18:07:46.189638] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185072 (xid=0x54428) [2018-02-22 18:07:46.189735] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185103 (xid=0x54429) [2018-02-22 18:07:46.189834] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185150 (xid=0x5442a) [2018-02-22 18:07:46.189930] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185173 (xid=0x5442b) [2018-02-22 18:07:46.190028] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185198 (xid=0x5442c) [2018-02-22 18:07:46.190126] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185211 (xid=0x5442d) [2018-02-22 18:07:46.190224] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185224 (xid=0x5442e) [2018-02-22 18:07:46.190323] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185236 (xid=0x5442f) [2018-02-22 18:07:46.190427] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185249 (xid=0x54430) [2018-02-22 18:07:46.190529] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185265 (xid=0x54431) [2018-02-22 18:07:46.190629] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185281 (xid=0x54432) [2018-02-22 18:07:46.190729] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185295 (xid=0x54433) [2018-02-22 18:07:46.190830] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185322 (xid=0x54434) [2018-02-22 18:07:46.190931] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185334 (xid=0x54435) [2018-02-22 18:07:46.191033] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185366 (xid=0x54436) [2018-02-22 18:07:46.191135] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185422 (xid=0x54437) [2018-02-22 18:07:46.191237] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185437 (xid=0x54438) [2018-02-22 18:07:46.191342] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185472 (xid=0x54439) [2018-02-22 18:07:46.191457] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185524 (xid=0x5443a) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 27 times between [2018-02-22 18:07:46.188773] and [2018-02-22 18:07:46.191472] [2018-02-22 18:07:47.148084] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.148118] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.148126] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 28 times between [2018-02-22 18:07:46.186197] and [2018-02-22 18:07:47.148136] [2018-02-22 18:07:47.148142] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.148200] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.148218] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.148217] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.148229] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.148259] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154) [2018-02-22 18:07:47.148272] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.148271] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed [2018-02-22 18:07:47.148281] W [fuse-bridge.c:1291:fuse_err_cbk] 0-glusterfs-fuse: 790447: FLUSH() ERR => -1 (Transport endpoint is not connected) [2018-02-22 18:07:47.148279] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.148310] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.148314] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.148346] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.148355] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.148362] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.148369] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.148410] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.148435] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.148436] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.148450] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.148457] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.148476] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.148505] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.148513] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.148520] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.148527] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.148538] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.148560] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.148568] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.148575] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.148596] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.148617] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.148626] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.148666] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.148678] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.148702] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.148712] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.148761] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.148765] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.148807] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.148803] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.148816] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.148846] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.148849] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.148858] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.148872] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.148888] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.148920] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.148924] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.148922] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.148954] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.148961] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.148964] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.148973] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a90 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.148994] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.149000] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.149008] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.149029] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.149045] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.149058] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54440 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.149077] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.149091] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.149126] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.149132] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.149141] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.149158] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a91 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.149162] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.149177] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54441 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.149168] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.149185] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.149185] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.149228] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.149255] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.149275] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54442 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.149274] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.149287] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.149325] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.149343] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.149345] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.149379] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54443 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.149390] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.149420] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.149422] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.149450] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.149479] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54444 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.149504] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.149572] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.149606] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.149607] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.149627] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54445 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.149635] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.149668] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.149694] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.149713] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.149723] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54446 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.149732] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.149753] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.149797] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.149811] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54447 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.149810] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.149820] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.149857] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.149891] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.149891] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.149904] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54448 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.149911] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.149935] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.149952] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.149966] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.149973] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54449 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.149985] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150013] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.150057] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.150067] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.150066] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.150078] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150104] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.150136] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.150138] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.150162] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.150174] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150193] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.150217] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.150225] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.150219] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.150232] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150246] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.150259] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.150258] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.150266] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150262] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.150293] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.150319] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.150321] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.150335] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.150348] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150369] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.150403] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.150410] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.150423] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.150433] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a92 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.150430] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.150447] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150454] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.150467] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150470] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.150484] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.150510] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.150520] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.150527] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54451 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.150535] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150558] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.150585] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.150598] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.150604] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54453 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.150613] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150640] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.150683] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.150697] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.150707] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54455 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.150726] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150738] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.150752] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.150759] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54454 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.150759] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.150768] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150794] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.150805] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.150820] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.150823] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.150832] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a93 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.150836] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.150854] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54456 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.150846] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150868] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.150866] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150890] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.150916] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.150925] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.150932] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a94 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.150940] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.150940] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.150979] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.150980] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.150998] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.151012] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a95 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.151010] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54450 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.151007] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.151025] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.151065] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.151078] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.151109] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.151120] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.151131] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.151146] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54452 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.151159] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.151176] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.151196] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.151200] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.151207] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54457 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.151222] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.151236] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.151251] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.151261] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54458 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.151262] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.151273] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.151306] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.151311] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.151328] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a96 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.151327] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.151343] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.151350] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.151384] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.151384] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.151396] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54459 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.151406] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.151414] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.151423] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.151429] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.151434] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.151436] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.151465] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.151471] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.151482] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.151491] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.151504] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.151523] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.151539] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.151539] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.151566] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.151576] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.151598] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.151609] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.151617] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.151635] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.151653] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.151668] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.151686] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.151688] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.151697] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.151710] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.151731] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.151745] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.151753] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54461 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.151753] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.151779] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.151804] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.151818] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.151819] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.151844] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54462 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.151857] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.151899] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.151930] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.151948] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.151958] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.151970] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.152001] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.152033] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.152042] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.152053] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54465 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.152062] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.152089] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.152092] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.152107] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.152116] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54463 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.152127] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.152140] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.152153] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.152164] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54464 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.152168] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.152173] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.152207] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.152233] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.152243] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.152250] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54467 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.152251] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-0: changing port to 49154 (from 0) [2018-02-22 18:07:47.152258] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.152270] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.152296] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.152304] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.152310] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54469 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.152318] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.152330] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.152332] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.152358] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.152375] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54466 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.152388] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.152422] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.152443] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.152460] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.152469] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54460 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.152481] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.152508] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.152550] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.152572] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.152582] W [MSGID: 103070] [rdma.c:4283:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x2b2f1dfc3000, wc.byte_len = 0, post->reused = 2 [2018-02-22 18:07:47.152588] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54468 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.152636] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.152667] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.152681] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.152709] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.152729] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5446a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.152734] W [MSGID: 103070] [rdma.c:4283:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x2b2f1dfc5000, wc.byte_len = 0, post->reused = 3 [2018-02-22 18:07:47.152751] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.152791] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.152796] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.152817] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.152829] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5446b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.152847] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.152868] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.152909] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.152922] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.152930] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5446e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.152942] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.152948] W [MSGID: 103070] [rdma.c:4283:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x2b2f1dfc3000, wc.byte_len = 0, post->reused = 3 [2018-02-22 18:07:47.152970] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.153025] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.153060] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.153083] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5446d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.153104] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.153142] W [MSGID: 103070] [rdma.c:4283:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x2b2f1dfc3000, wc.byte_len = 0, post->reused = 4 [2018-02-22 18:07:47.153142] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.153646] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.153648] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.153682] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a9f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.153698] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.153722] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.153886] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.153901] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.153908] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.153918] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.153936] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.153978] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.153993] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.154004] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.154015] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154031] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.154087] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.154104] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.154116] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.154128] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154144] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.154170] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.154188] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.154195] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.154196] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152215 (xid=0x54a99) [2018-02-22 18:07:47.154206] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154229] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154230] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.154239] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.154270] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.154282] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.154292] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154317] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.154320] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.154344] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.154369] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.154382] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154410] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.154507] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.154506] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152542 (xid=0x54a9a) [2018-02-22 18:07:47.154518] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.154532] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.154535] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154542] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154561] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.154580] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.154596] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.154606] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.154617] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154632] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.154646] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.154650] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.154653] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.154662] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154696] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.154701] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.154701] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152568 (xid=0x54a9b) [2018-02-22 18:07:47.154714] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.154730] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.154732] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154758] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.154743] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154756] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.154792] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.154801] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aaa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.154813] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154837] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.154857] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.154858] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.154879] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aab Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.154893] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154905] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.154917] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.154926] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aac Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.154929] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.154918] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152591 (xid=0x54a9c) [2018-02-22 18:07:47.154940] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154951] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.154980] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.155029] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.155041] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.155049] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aad Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.155059] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155091] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.155087] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152742 (xid=0x54a9d) [2018-02-22 18:07:47.155110] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155131] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.155147] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.155157] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aae Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.155168] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155206] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.155240] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.155252] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.155262] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aaf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.155271] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155284] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153129 (xid=0x54a9e) [2018-02-22 18:07:47.155312] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155318] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.155368] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.155382] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.155389] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.155397] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155425] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.155452] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.155461] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.155468] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.155474] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155495] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.155526] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.155535] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.155542] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.155553] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155576] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.155585] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.155593] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2018-02-22 18:07:47.155600] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.155610] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.155621] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155641] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.155661] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.155666] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.155677] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.155686] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155707] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.155719] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.155721] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.155727] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.155736] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155738] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153148 (xid=0x5446c) [2018-02-22 18:07:47.155760] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155763] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.155766] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.155798] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.155813] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.155827] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155848] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.155865] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.155866] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.155893] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.155903] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155913] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153170 (xid=0x5446f) [2018-02-22 18:07:47.155918] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.155938] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.155950] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.155961] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.155967] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.155992] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156036] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.156073] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.156086] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.156094] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.156104] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156100] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2018-02-22 18:07:47.153187 (xid=0x54470) [2018-02-22 18:07:47.156128] W [MSGID: 114032] [client-handshake.c:1625:client_dump_version_cbk] 2-scratch-client-1: received RPC status error [Transport endpoint is not connected] [2018-02-22 18:07:47.156129] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.156140] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.156158] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.156169] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.156178] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156195] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.156197] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.156225] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.156238] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aba Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.156251] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156272] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.156275] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153204 (xid=0x54471) [2018-02-22 18:07:47.156287] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.156309] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.156298] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156319] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156303] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.156358] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.156403] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.156414] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.156413] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.156421] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156454] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.156485] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.156485] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.156501] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.156509] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156504] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153220 (xid=0x54472) [2018-02-22 18:07:47.156529] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156543] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.156564] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.156583] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.156592] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abe Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.156601] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156636] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.156665] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.156676] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.156686] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.156693] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156701] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153237 (xid=0x54473) [2018-02-22 18:07:47.156723] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156728] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.156766] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.156774] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.156780] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.156787] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156790] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.156816] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.156824] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.156832] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.156830] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153269 (xid=0x54474) [2018-02-22 18:07:47.156854] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156855] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156876] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.156865] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.156904] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.156916] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.156923] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.156927] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156937] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.156936] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.156947] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.156957] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.156982] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.156994] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.156991] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153302 (xid=0x54475) [2018-02-22 18:07:47.157009] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157010] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157025] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.157034] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157042] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.157043] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157060] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.157064] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.157085] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.157087] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.157101] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157101] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153334 (xid=0x54476) [2018-02-22 18:07:47.157106] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.157107] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.157118] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157124] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.157132] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:47.157034] and [2018-02-22 18:07:47.157127] [2018-02-22 18:07:47.157142] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.157138] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157158] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.157171] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.157179] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.157180] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.157179] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.157195] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157192] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157201] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.157208] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157221] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.157225] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153366 (xid=0x54477) [2018-02-22 18:07:47.157233] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.157248] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.157245] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157259] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157266] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.157273] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157285] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.157295] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157299] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.157301] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.157312] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157325] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.157333] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157338] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.157340] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.157354] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157357] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153393 (xid=0x54478) [2018-02-22 18:07:47.157367] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.157382] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157384] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157389] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.157402] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157459] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.157497] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.157498] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.157503] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.157514] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54490 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.157519] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157521] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157526] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aca Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.157532] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157532] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.157547] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153416 (xid=0x54479) [2018-02-22 18:07:47.157562] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157565] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:47.157459] and [2018-02-22 18:07:47.157565] [2018-02-22 18:07:47.157575] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157589] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54acb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.157600] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157612] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.157613] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157628] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54acc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.157630] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.157638] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.157656] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.157670] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.157668] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153441 (xid=0x5447a) [2018-02-22 18:07:47.157676] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54491 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.157685] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157690] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157700] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.157712] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.157718] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.157725] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54492 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.157732] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157742] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.157751] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157757] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54acd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:47.157600] and [2018-02-22 18:07:47.157766] [2018-02-22 18:07:47.157767] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.157788] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.157803] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153758 (xid=0x5447b) [2018-02-22 18:07:47.157809] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.157823] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ace Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.157822] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157823] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157835] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157881] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.157906] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.157907] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157919] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54acf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.157925] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157940] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153782 (xid=0x5447c) [2018-02-22 18:07:47.157955] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.157957] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.157973] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.157987] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.157993] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.157999] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158031] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.158067] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.158067] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153910 (xid=0x5447d) [2018-02-22 18:07:47.158077] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.158084] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.158082] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158090] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158108] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.158129] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.158137] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.158144] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54493 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.158150] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158186] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.158183] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153928 (xid=0x5447e) [2018-02-22 18:07:47.158198] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158209] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.158218] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.158218] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.158225] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158264] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.158288] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.158289] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.158305] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.158312] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158309] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153945 (xid=0x5447f) [2018-02-22 18:07:47.158325] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158456] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153964 (xid=0x54480) [2018-02-22 18:07:47.158474] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158475] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.158492] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.158498] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.158505] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:47.158264] and [2018-02-22 18:07:47.158515] [2018-02-22 18:07:47.158529] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.158544] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.158550] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54494 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.158556] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158581] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153990 (xid=0x54481) [2018-02-22 18:07:47.158600] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158600] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.158649] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.158658] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.158665] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.158672] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158704] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.158708] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154202 (xid=0x54482) [2018-02-22 18:07:47.158722] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158735] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.158744] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.158743] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.158750] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158794] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.158817] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.158817] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.158829] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.158837] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158839] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154251 (xid=0x54483) [2018-02-22 18:07:47.158854] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158876] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.158919] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.158933] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.158939] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.158945] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.158979] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154278 (xid=0x54484) [2018-02-22 18:07:47.158986] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.158997] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.159017] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.159033] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.159039] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.159045] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.159087] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.159103] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.159110] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154297 (xid=0x54485) [2018-02-22 18:07:47.159118] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.159130] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54495 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.159127] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.159141] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.159178] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.159284] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154326 (xid=0x54486) [2018-02-22 18:07:47.159298] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.159432] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154349 (xid=0x54487) [2018-02-22 18:07:47.159569] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154507 (xid=0x54488) [2018-02-22 18:07:47.159703] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154546 (xid=0x54489) [2018-02-22 18:07:47.159837] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154685 (xid=0x5448a) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 4 times between [2018-02-22 18:07:47.159298] and [2018-02-22 18:07:47.159852] [2018-02-22 18:07:47.159919] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.159942] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.159948] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ada Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.160013] W [fuse-bridge.c:2312:fuse_writev_cbk] 0-glusterfs-fuse: 790458: WRITE => -1 gfid=91f97edd-97a3-4948-9d9d-469c2d174853 fd=0x2b2f23f02440 (Transport endpoint is not connected) [2018-02-22 18:07:47.159959] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.160040] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.161163] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.161183] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.161195] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54496 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.161205] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.161225] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.161258] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.161260] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.161280] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54adb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.161290] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.161318] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.161360] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.161395] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.161407] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54497 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.161418] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.161448] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.161473] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.161492] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.161501] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54498 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.161510] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.161523] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.161566] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.161580] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54499 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.161579] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.161589] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.161626] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.161659] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.161660] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.161678] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.161687] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.161712] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.161734] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.161752] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.161761] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.161770] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.161785] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.161828] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.161841] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.161840] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.161850] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.161882] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.161914] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.161925] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.161934] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54adc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.161943] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.161970] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.162001] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.162012] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.162021] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54add Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.162030] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.162052] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.162082] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.162083] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.162101] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.162109] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.162139] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.162171] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.162184] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.162194] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ade Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.162202] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.162231] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.162255] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.162274] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.162283] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54adf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.162292] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.162296] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.162334] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.162346] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.162355] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.162363] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.162401] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.162434] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.162447] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.162446] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.162456] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.162497] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.162530] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.162531] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.162549] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.162558] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.162589] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.162614] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.162633] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.162642] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.162650] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.162682] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.162714] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.162725] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.162734] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.162742] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.162764] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.162805] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.162816] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.162825] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.162833] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.162866] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.162901] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.162913] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.162912] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.162922] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.162948] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.162976] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.162987] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.162996] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.163006] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.163040] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.163073] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.163074] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.163093] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.163104] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.163119] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.163147] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.163159] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.163167] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.163176] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.163212] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.163234] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.163253] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.163262] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.163270] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.163296] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.163337] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.163348] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.163356] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.163365] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.163415] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.163451] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.163462] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.163471] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.163480] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.163518] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.163550] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.163562] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.163562] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.163571] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.163616] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.163649] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.163660] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.163669] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.163677] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.163692] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.163720] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.163721] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.163738] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.163747] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.163787] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.163819] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.163831] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.163841] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.163855] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.163897] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.163931] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.163942] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.163953] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.163962] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.164005] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.164029] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.164048] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.164057] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.164066] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.164080] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.164189] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-02-22 18:07:47.164228] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.164245] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.164254] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.164263] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.164280] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.164343] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.164355] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.164365] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.164382] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.164427] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.164462] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008) [2018-02-22 18:07:47.164474] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed [2018-02-22 18:07:47.164487] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544aa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1) [2018-02-22 18:07:47.164496] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.164540] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop [2018-02-22 18:07:47.165116] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.165133] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.165143] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.165155] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.165162] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2018-02-22 18:07:47.165193] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.165235] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.165248] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.165247] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.165258] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.165299] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164108 (xid=0x54aea) [2018-02-22 18:07:47.165311] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.165316] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.165346] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.165348] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.165367] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.165384] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.165469] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.165522] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164130 (xid=0x54aeb) [2018-02-22 18:07:47.165539] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.165572] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.165589] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.165588] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.165599] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.165621] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.165668] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164175 (xid=0x54aec) [2018-02-22 18:07:47.165682] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.165746] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.165748] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.165770] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.165802] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164210 (xid=0x54aed) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:47.165682] and [2018-02-22 18:07:47.165815] [2018-02-22 18:07:47.165822] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.165876] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.165889] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.165888] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.165899] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.165926] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS Handshake) op(SETVOLUME(1)) called at 2018-02-22 18:07:47.164238 (xid=0x54aee) [2018-02-22 18:07:47.165941] W [MSGID: 114032] [client-handshake.c:1088:client_setvolume_cbk] 2-scratch-client-0: received RPC status error [Transport endpoint is not connected] [2018-02-22 18:07:47.165953] I [MSGID: 114051] [client-handshake.c:1259:client_setvolume_cbk] 2-scratch-client-0: sending CHILD_CONNECTING event [2018-02-22 18:07:47.165963] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.166001] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.166002] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.166020] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.166029] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.166037] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164320 (xid=0x54aef) [2018-02-22 18:07:47.166082] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.166163] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.166185] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164582 (xid=0x54af0) [2018-02-22 18:07:47.166187] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.166212] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2018-02-22 18:07:47.166029] and [2018-02-22 18:07:47.166224] [2018-02-22 18:07:47.166282] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164646 (xid=0x54af1) [2018-02-22 18:07:47.166269] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.166300] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.166404] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.166422] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.166421] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.166433] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.166456] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.166508] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164697 (xid=0x54af2) [2018-02-22 18:07:47.166518] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:47.166531] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:47.166525] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.166552] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.166565] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.166653] W [fuse-bridge.c:2312:fuse_writev_cbk] 0-glusterfs-fuse: 790460: WRITE => -1 gfid=91f97edd-97a3-4948-9d9d-469c2d174853 fd=0x2b2f23f02440 (Transport endpoint is not connected) [2018-02-22 18:07:47.166688] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164725 (xid=0x54af3) [2018-02-22 18:07:47.166704] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.166947] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2018-02-22 18:07:47.167042] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b3t1_0001.fits (dace8bc0-f4d7-48b2-98ab-a0f768b3be8c) [Transport endpoint is not connected] [2018-02-22 18:07:47.167069] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164600 (xid=0x544ac) [2018-02-22 18:07:47.167085] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.167085] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.167099] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b3t1_0001.fits (dace8bc0-f4d7-48b2-98ab-a0f768b3be8c) [Transport endpoint is not connected] [2018-02-22 18:07:47.167128] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afe Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.167126] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.167145] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.167152] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 2-scratch-dht: Found anomalies in /download/work/kstovall/57910/drx_57910_B0656+14_b3t1_0001.fits (gfid = dace8bc0-f4d7-48b2-98ab-a0f768b3be8c). Holes=1 overlaps=0 [2018-02-22 18:07:47.167191] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: dace8bc0-f4d7-48b2-98ab-a0f768b3be8c: failed to resolve (Transport endpoint is not connected) [2018-02-22 18:07:47.167205] E [fuse-bridge.c:796:fuse_getattr_resume] 0-glusterfs-fuse: 790462: GETATTR 47480857159620 (dace8bc0-f4d7-48b2-98ab-a0f768b3be8c) resolution failed [2018-02-22 18:07:47.167242] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164622 (xid=0x544ad) The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:47.166456] and [2018-02-22 18:07:47.167174] [2018-02-22 18:07:47.167257] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.167304] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.167319] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.167326] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aff Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.167334] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.167356] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.167463] W [MSGID: 114031] [client-rpc-fops.c:1495:client3_3_fstat_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.167478] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164669 (xid=0x544ae) [2018-02-22 18:07:47.167501] W [MSGID: 114031] [client-rpc-fops.c:1495:client3_3_fstat_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.167502] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.167554] W [fuse-bridge.c:767:fuse_attr_cbk] 0-glusterfs-fuse: 790467: FSTAT() /download/work/kstovall/57910/drx_57910_B0656+14_b3t2_0001.fits => -1 (Transport endpoint is not connected) [2018-02-22 18:07:47.167560] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.167575] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.167582] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b00 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.167590] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.167609] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.167656] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(PORTMAP) op(PORTBYBRICK(1)) called at 2018-02-22 18:07:47.164738 (xid=0x544af) [2018-02-22 18:07:47.167669] W [MSGID: 114032] [client-handshake.c:1508:client_query_portmap_cbk] 2-scratch-client-1: received RPC status error, try again later [Transport endpoint is not connected] [2018-02-22 18:07:47.167761] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164755 (xid=0x544b0) [2018-02-22 18:07:47.167774] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.167861] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008) [2018-02-22 18:07:47.167873] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed [2018-02-22 18:07:47.167880] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b01 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:47.167890] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.167909] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:47.167910] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.165522 (xid=0x544b1) [2018-02-22 18:07:47.167924] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.168078] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.165551 (xid=0x544b2) [2018-02-22 18:07:47.168222] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.165676 (xid=0x544b3) [2018-02-22 18:07:47.168423] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.165710 (xid=0x544b4) [2018-02-22 18:07:47.168568] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.166138 (xid=0x544b5) [2018-02-22 18:07:47.168712] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.166343 (xid=0x544b6) [2018-02-22 18:07:47.168856] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.166378 (xid=0x544b7) [2018-02-22 18:07:47.169003] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.166492 (xid=0x544b8) [2018-02-22 18:07:47.172418] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-0: changing port to 49154 (from 0) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 7 times between [2018-02-22 18:07:47.167924] and [2018-02-22 18:07:47.169018] [2018-02-22 18:07:47.176708] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-02-22 18:07:47.177249] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-0: Connected to scratch-client-0, attached to remote volume '/data/brick1/scratch'. [2018-02-22 18:07:47.177261] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-0: Server and Client lk-version numbers are not same, reopening the fds [2018-02-22 18:07:47.177270] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-0: 4 fds open - Delaying child_up until they are re-opened [2018-02-22 18:07:47.177839] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP [2018-02-22 18:07:47.177894] W [MSGID: 114031] [client-rpc-fops.c:799:client3_3_statfs_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:47.177935] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-0: Server lk version = 1 [2018-02-22 18:07:48.123815] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected] [2018-02-22 18:07:48.548708] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-1: changing port to 49154 (from 0) [2018-02-22 18:07:48.124421] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected] [2018-02-22 18:07:48.553517] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-02-22 18:07:48.553937] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-1: Connected to scratch-client-1, attached to remote volume '/data/brick1/scratch'. [2018-02-22 18:07:48.553946] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-1: Server and Client lk-version numbers are not same, reopening the fds [2018-02-22 18:07:48.553953] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-1: 4 fds open - Delaying child_up until they are re-opened [2018-02-22 18:07:48.554300] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP [2018-02-22 18:07:48.554458] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-1: Server lk version = 1 [2018-02-22 18:07:49.139511] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:49.139545] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:49.139561] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b2e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:49.139578] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.139584] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2018-02-22 18:07:49.139616] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:49.139744] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2018-02-22 18:07:49.137922 (xid=0x54b13) [2018-02-22 18:07:49.139762] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected] [2018-02-22 18:07:49.139816] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:49.139836] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:49.139847] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b2f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:49.139859] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138017 (xid=0x54b14) [2018-02-22 18:07:49.139861] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.139883] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.139896] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:49.139960] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:49.139976] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:49.139981] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138034 (xid=0x54b15) [2018-02-22 18:07:49.139986] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b30 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:49.140001] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.140009] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.140016] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:49.140093] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:49.140094] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138210 (xid=0x54b16) [2018-02-22 18:07:49.140112] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:49.140119] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.140124] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b31 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:49.140146] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.140164] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:49.140219] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:49.140219] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138307 (xid=0x54b17) [2018-02-22 18:07:49.140236] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:49.140243] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.140248] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b32 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:49.140270] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.140277] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:49.140343] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138332 (xid=0x54b18) [2018-02-22 18:07:49.140361] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:49.140361] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.140405] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:49.140418] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b33 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:49.140429] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.140458] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:49.140491] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138404 (xid=0x54b19) [2018-02-22 18:07:49.140505] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.140609] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138432 (xid=0x54b1a) [2018-02-22 18:07:49.140622] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.140643] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:49.140664] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:49.140671] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b34 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:49.140678] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.140690] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:49.140711] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138454 (xid=0x54b1b) [2018-02-22 18:07:49.140726] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.140825] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138475 (xid=0x54b1c) [2018-02-22 18:07:49.140924] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138506 (xid=0x54b1d) [2018-02-22 18:07:49.141023] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138533 (xid=0x54b1e) [2018-02-22 18:07:49.141122] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138560 (xid=0x54b1f) [2018-02-22 18:07:49.141211] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138607 (xid=0x54b20) [2018-02-22 18:07:49.141311] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138631 (xid=0x54b21) [2018-02-22 18:07:49.141443] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138654 (xid=0x54b22) [2018-02-22 18:07:49.141546] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138679 (xid=0x54b23) [2018-02-22 18:07:49.141646] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138725 (xid=0x54b24) [2018-02-22 18:07:49.141735] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138747 (xid=0x54b25) [2018-02-22 18:07:49.141835] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138773 (xid=0x54b26) [2018-02-22 18:07:49.141934] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138845 (xid=0x54b27) [2018-02-22 18:07:49.142023] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138891 (xid=0x54b28) [2018-02-22 18:07:49.142112] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138961 (xid=0x54b29) [2018-02-22 18:07:49.142211] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138986 (xid=0x54b2a) [2018-02-22 18:07:49.142312] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139037 (xid=0x54b2b) [2018-02-22 18:07:49.142433] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139086 (xid=0x54b2c) [2018-02-22 18:07:49.142523] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139164 (xid=0x54b2d) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 18 times between [2018-02-22 18:07:49.140726] and [2018-02-22 18:07:49.142536] [2018-02-22 18:07:49.142797] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2018-02-22 18:07:49.142922] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2018-02-22 18:07:49.137938 (xid=0x544c6) [2018-02-22 18:07:49.142939] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected] [2018-02-22 18:07:49.142965] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected] [2018-02-22 18:07:49.142984] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected] [2018-02-22 18:07:49.143128] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138060 (xid=0x544c7) [2018-02-22 18:07:49.143144] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.143145] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2018-02-22 18:07:49.143175] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2018-02-22 18:07:49.143201] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2018-02-22 18:07:49.143219] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2018-02-22 18:07:49.143244] W [fuse-bridge.c:471:fuse_entry_cbk] 0-glusterfs-fuse: 790481: LOOKUP() /download => -1 (Transport endpoint is not connected) [2018-02-22 18:07:49.143247] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138083 (xid=0x544c8) [2018-02-22 18:07:49.143261] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.143359] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138105 (xid=0x544c9) [2018-02-22 18:07:49.143482] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138127 (xid=0x544ca) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:49.143261] and [2018-02-22 18:07:49.143503] [2018-02-22 18:07:49.143515] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected] [2018-02-22 18:07:49.143574] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected] [2018-02-22 18:07:49.143605] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected] [2018-02-22 18:07:49.143639] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected] [2018-02-22 18:07:49.143638] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138147 (xid=0x544cb) [2018-02-22 18:07:49.143670] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.143768] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2018-02-22 18:07:49.143801] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2018-02-22 18:07:49.143820] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138169 (xid=0x544cc) [2018-02-22 18:07:49.143840] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2018-02-22 18:07:49.143841] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.143875] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2018-02-22 18:07:49.143912] W [fuse-bridge.c:471:fuse_entry_cbk] 0-glusterfs-fuse: 790485: LOOKUP() /download => -1 (Transport endpoint is not connected) [2018-02-22 18:07:49.143999] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138194 (xid=0x544cd) [2018-02-22 18:07:49.144019] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.144169] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138231 (xid=0x544ce) [2018-02-22 18:07:49.144295] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138253 (xid=0x544cf) [2018-02-22 18:07:49.144404] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138273 (xid=0x544d0) [2018-02-22 18:07:49.144541] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138294 (xid=0x544d1) [2018-02-22 18:07:49.144695] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138348 (xid=0x544d2) [2018-02-22 18:07:49.144847] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138582 (xid=0x544d3) [2018-02-22 18:07:49.145003] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138707 (xid=0x544d4) [2018-02-22 18:07:49.145155] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138796 (xid=0x544d5) [2018-02-22 18:07:49.145311] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138819 (xid=0x544d6) [2018-02-22 18:07:49.145422] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138863 (xid=0x544d7) [2018-02-22 18:07:49.145567] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138910 (xid=0x544d8) [2018-02-22 18:07:49.145721] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138936 (xid=0x544d9) [2018-02-22 18:07:49.145871] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139010 (xid=0x544da) [2018-02-22 18:07:49.146021] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139057 (xid=0x544db) [2018-02-22 18:07:49.146173] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139105 (xid=0x544dc) [2018-02-22 18:07:49.146326] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139130 (xid=0x544dd) [2018-02-22 18:07:49.146488] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139190 (xid=0x544de) [2018-02-22 18:07:49.146629] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140064 (xid=0x544df) [2018-02-22 18:07:49.146783] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140327 (xid=0x544e0) [2018-02-22 18:07:49.146935] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140501 (xid=0x544e1) [2018-02-22 18:07:49.147086] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140522 (xid=0x544e2) [2018-02-22 18:07:49.147236] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140545 (xid=0x544e3) [2018-02-22 18:07:49.147393] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140564 (xid=0x544e4) [2018-02-22 18:07:49.147536] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140582 (xid=0x544e5) [2018-02-22 18:07:49.147641] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140601 (xid=0x544e6) [2018-02-22 18:07:49.147790] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140623 (xid=0x544e7) [2018-02-22 18:07:49.147846] W [fuse-bridge.c:1291:fuse_err_cbk] 0-glusterfs-fuse: 790480: FLUSH() ERR => -1 (Transport endpoint is not connected) [2018-02-22 18:07:49.152660] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-0: changing port to 49154 (from 0) The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 26 times between [2018-02-22 18:07:49.144019] and [2018-02-22 18:07:49.147810] [2018-02-22 18:07:49.156308] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154) [2018-02-22 18:07:49.156348] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed [2018-02-22 18:07:49.156358] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b37 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0) [2018-02-22 18:07:49.156377] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.156418] W [fuse-bridge.c:1291:fuse_err_cbk] 0-glusterfs-fuse: 790496: FLUSH() ERR => -1 (Transport endpoint is not connected) [2018-02-22 18:07:49.156449] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop [2018-02-22 18:07:49.157895] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-02-22 18:07:49.158352] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-0: Connected to scratch-client-0, attached to remote volume '/data/brick1/scratch'. [2018-02-22 18:07:49.158366] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-0: Server and Client lk-version numbers are not same, reopening the fds [2018-02-22 18:07:49.158708] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-0: 2 fds open - Delaying child_up until they are re-opened [2018-02-22 18:07:49.159196] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP [2018-02-22 18:07:49.159242] W [MSGID: 114031] [client-rpc-fops.c:799:client3_3_statfs_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected] [2018-02-22 18:07:49.159319] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-0: Server lk version = 1 [2018-02-22 18:07:49.181472] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download/work/kstovall/57910/jp.pbs (e9f57eb0-e8a2-40dc-b56f-59eea34941a6) [Transport endpoint is not connected] -------------- next part -------------- [2018-02-22 18:07:45.226382] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7fd5ba679000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:45.236646] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-0 [2018-02-22 18:07:45.236796] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits [2018-02-22 18:07:45.236854] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b2_0001.fits [2018-02-22 18:07:45.236910] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log [2018-02-22 18:07:45.236966] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine2.log [2018-02-22 18:07:45.237007] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-0 [2018-02-22 18:07:45.262237] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-1 (version: 3.8.4) [2018-02-22 18:07:45.416142] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler276-32676-2018/02/22-16:07:03:805703-scratch-client-1-22-0 [2018-02-22 18:07:45.416218] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /slzhou/slzhou/CeO2-331/NEB1/neb-new/PBE-d2/09/SCF/OSZICAR [2018-02-22 18:07:45.416254] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /slzhou/slzhou/CeO2-331/NEB1/neb-new/PBE-d2/09/SCF/OUTCAR [2018-02-22 18:07:45.416306] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler276-32676-2018/02/22-16:07:03:805703-scratch-client-1-22-0 [2018-02-22 18:07:46.191329] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7fd5ba799000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:46.192099] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7fd5ba8e3000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:46.192498] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7fd5ba401000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:46.192876] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7fd5ba7c5000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:46.195455] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-1 [2018-02-22 18:07:46.195537] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits [2018-02-22 18:07:46.195589] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log [2018-02-22 18:07:46.195634] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b2_0001.fits [2018-02-22 18:07:46.195675] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine2.log [2018-02-22 18:07:46.195742] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-1 [2018-02-22 18:07:48.559889] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-2 (version: 3.8.4) [2018-02-22 18:07:49.144466] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7fd5bbf7b000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:49.145265] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7fd5ba53b000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:49.149583] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-2 [2018-02-22 18:07:49.149670] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits [2018-02-22 18:07:49.149726] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log [2018-02-22 18:07:49.149825] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-2 -------------- next part -------------- [2018-02-22 18:07:45.226574] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f32cabd1000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:45.228378] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-0 [2018-02-22 18:07:45.228601] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b2_0001.fits [2018-02-22 18:07:45.228681] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits [2018-02-22 18:07:45.228737] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log [2018-02-22 18:07:45.228793] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine2.log [2018-02-22 18:07:45.230202] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1021) [2018-02-22 18:07:45.230273] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1021) failed [2018-02-22 18:07:45.230292] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54974, Program: GlusterFS 3.3, ProgVers: 330, Proc: 12) to rpc-transport (rdma.scratch-server) [2018-02-22 18:07:45.230386] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/debug/io-stats.so(+0x10e82) [0x7f32dc180e82] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x1cac6) [0x7f32d7dd4ac6] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed [2018-02-22 18:07:45.230463] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-0 [2018-02-22 18:07:45.261717] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-1 (version: 3.8.4) [2018-02-22 18:07:45.416150] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler276-32676-2018/02/22-16:07:03:805703-scratch-client-0-22-0 [2018-02-22 18:07:45.416244] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /slzhou/slzhou/CeO2-331/NEB1/neb-new/PBE-d2/09/SCF/OSZICAR [2018-02-22 18:07:45.416315] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /slzhou/slzhou/CeO2-331/NEB1/neb-new/PBE-d2/09/SCF/OUTCAR [2018-02-22 18:07:45.416373] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler276-32676-2018/02/22-16:07:03:805703-scratch-client-0-22-0 [2018-02-22 18:07:46.191486] W [MSGID: 103004] [rdma.c:2327:__gf_rdma_register_local_mr_for_rdma] 0-rpc-transport/rdma: registering memory for IBV_ACCESS_LOCAL_WRITE failed [Invalid argument] [2018-02-22 18:07:46.191549] W [MSGID: 103004] [rdma.c:3701:gf_rdma_do_reads] 0-rpc-transport/rdma: registering local memory for rdma read failed [2018-02-22 18:07:46.191570] W [MSGID: 103061] [rdma.c:3969:gf_rdma_recv_request] 0-rpc-transport/rdma: rdma read from peer (172.17.1.61:1023) failed [2018-02-22 18:07:46.191600] W [MSGID: 103063] [rdma.c:4084:gf_rdma_process_recv] 0-rpc-transport/rdma: receiving a request from peer (172.17.1.61:1023) failed [2018-02-22 18:07:46.191611] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f32caced000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:46.192513] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f32ca41d000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:46.192880] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f32ca9c7000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:46.193399] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f32ca7d3000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:46.193948] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-1 [2018-02-22 18:07:46.194011] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1023) [2018-02-22 18:07:46.194085] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits [2018-02-22 18:07:46.194086] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1023) failed [2018-02-22 18:07:46.194118] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log [2018-02-22 18:07:46.194131] W [MSGID: 103026] [rdma.c:3670:gf_rdma_do_reads] 0-rpc-transport/rdma: transport not connected to peer (172.17.1.61:1023), not doing rdma reads [2018-02-22 18:07:46.194191] W [MSGID: 103061] [rdma.c:3969:gf_rdma_recv_request] 0-rpc-transport/rdma: rdma read from peer (172.17.1.61:1023) failed [2018-02-22 18:07:46.194191] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine2.log [2018-02-22 18:07:46.194214] W [MSGID: 103063] [rdma.c:4084:gf_rdma_process_recv] 0-rpc-transport/rdma: receiving a request from peer (172.17.1.61:1023) failed [2018-02-22 18:07:46.194252] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1023) [2018-02-22 18:07:46.194264] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1023) failed [2018-02-22 18:07:46.194342] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54a72, Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (rdma.scratch-server) [2018-02-22 18:07:46.194288] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54a6f, Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (rdma.scratch-server) [2018-02-22 18:07:46.194404] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/debug/io-stats.so(+0x112bb) [0x7f32dc1812bb] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x1c651) [0x7f32d7dd4651] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed [2018-02-22 18:07:46.194452] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/debug/io-stats.so(+0x112bb) [0x7f32dc1812bb] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x1c651) [0x7f32d7dd4651] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed [2018-02-22 18:07:46.194562] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-1 [2018-02-22 18:07:47.170678] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f32cabaf000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:47.170781] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-2 (version: 3.8.4) [2018-02-22 18:07:47.171160] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f32ca9eb000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:47.171941] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-2 [2018-02-22 18:07:47.171968] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1023) [2018-02-22 18:07:47.172145] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1023) failed [2018-02-22 18:07:47.172168] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54aee, Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (rdma.scratch-server) [2018-02-22 18:07:47.172164] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glusterfs: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-2 [2018-02-22 18:07:47.172214] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2cf25) [0x7f32d7de4f25] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2cec3) [0x7f32d7de4ec3] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed [2018-02-22 18:07:47.172248] E [server.c:207:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2cf25) [0x7f32d7de4f25] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2cec3) [0x7f32d7de4ec3] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x882e) [0x7f32d7dc082e] ) 0-: Reply submission failed [2018-02-22 18:07:47.183288] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-3 (version: 3.8.4) [2018-02-22 18:07:49.144790] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f32ca3b7000, wc.byte_len = 131072, post->reused = 1 [2018-02-22 18:07:49.147498] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-3 [2018-02-22 18:07:49.147530] W [MSGID: 103026] [rdma.c:3670:gf_rdma_do_reads] 0-rpc-transport/rdma: transport not connected to peer (172.17.1.61:1023), not doing rdma reads [2018-02-22 18:07:49.147592] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits [2018-02-22 18:07:49.147596] W [MSGID: 103061] [rdma.c:3969:gf_rdma_recv_request] 0-rpc-transport/rdma: rdma read from peer (172.17.1.61:1023) failed [2018-02-22 18:07:49.147646] W [MSGID: 103063] [rdma.c:4084:gf_rdma_process_recv] 0-rpc-transport/rdma: receiving a request from peer (172.17.1.61:1023) failed [2018-02-22 18:07:49.147659] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log [2018-02-22 18:07:49.147655] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1023) [2018-02-22 18:07:49.147797] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1023) failed [2018-02-22 18:07:49.147824] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54b13, Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (rdma.scratch-server) [2018-02-22 18:07:49.148988] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/debug/io-stats.so(+0x12857) [0x7f32dc182857] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2b427) [0x7f32d7de3427] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed [2018-02-22 18:07:49.149072] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-3 [2018-02-22 18:07:49.164480] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-4 (version: 3.8.4) [2018-02-22 18:07:46.194027] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b2_0001.fits
Raghavendra Gowdappa
2018-Feb-27 02:44 UTC
[Gluster-users] Problems with write-behind with large files on Gluster 3.8.4
+csaba On Tue, Feb 27, 2018 at 2:49 AM, Jim Prewett <download at carc.unm.edu> wrote:> > Hello, > > I'm having problems when write-behind is enabled on Gluster 3.8.4. > > I have 2 Gluster servers each with a single brick that is mirrored between > them. The code causing these issues reads two data files each approx. 128G > in size. It opens a third file, mmap()'s that file, and subsequently reads > and writes to it. The third file, on sucessful runs (without write-behind > enabled) is ultimately approx. 224G in size. >What exactly is the problem you are facing with write-behind enabled? Is it that the file size is smaller?> The servers have the IP addresses 172.17.2.254 and 172.17.2.255 and the > client has the IP address 172.17.1.61. These are all IP over InfiniBand. > > I'm attaching logfiles for the brick and for the volume from each of the > servers and for the client. I'm also attaching the output of "gluster > volume info" and "gluster volume get <volume> all". > > I have only noticed problems with write-behind being enabled with this one > particular workload. When I ran it under strace, I see it seeking all over > the place and reading and writing little bits of data to/from the third > file. >What is the pattern you see when write-behind is disabled? Can you attach strace of the application for both scenarios - write-behind enabled and disabled? Can you also explain the workload and its data access pattern?> For now, I'm leaving write-behind disabled. What are the performance > implications of this for jobs that don't have this strange access pattern? >Disabling write-behind can bring down performance for sequential workloads.> My co-worker who usually maintains the Gluster filesystems here is busy > having a baby right now and I've gotten it while he's out, so I'm /really/ > new to Gluster and am not confident that anything is correct in my > configuration (nor do I have a specific reason to doubt its correctness! :) > > I have checked the InfiniBand fabric for errors and do not see any beyond > the normal PortXmitWait counter. There is no firewall on any of these > machines. Their system clocks seem to all be synchronized. > > Is there anything additional I can provide to help diagnose this problem? > > Thanks for any help you can provide! :) > > Jim > > James E. Prewett Jim at Prewett.org download at hpc.unm.edu > Systems Team Leader LoGS: http://www.hpc.unm.edu/~download/LoGS/ > Designated Security Officer OpenPGP key: pub 1024D/31816D93 > HPC Systems Engineer III UNM HPC 505.277.8210 > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180227/11a73396/attachment.html>