So this may seem like a very newb question, but I don't quite understand why this isn't working. The configuration I'm using is very simple, and I've included both vol files below. The problem seems to be that the client is sending something over the IPoIB interface AFTER it's already started working over the ib-verbs interface... atleast, that's what I gather from the following errors when run in debug mode... [2009-06-08 16:58:20] D [ib-verbs.c:507:ib_verbs_receive] transport/ib-verbs: server: corrupt header received [2009-06-08 16:58:20] D [ib-verbs.c:1156:ib_verbs_recv_completion_proc] transport/ib-verbs: pollin notification to server failed, disconnecting transport [2009-06-08 16:58:20] E [ib-verbs.c:1996:ib_verbs_event_handler] transport/ib-verbs: server: pollin received on tcp socket (peer: 172.21.1.154:1023) after handshake is complete [2009-06-08 16:58:20] D [ib-verbs.c:1865:ib_verbs_handshake_pollerr] transport/ib-verbs: server: peer disconnected, cleaning up [2009-06-08 16:58:20] N [server-protocol.c:7796:notify] server: 172.21.1.154:1023 disconnected If I'm missing something in my configuration, please feel free to point it out to me. Any help is greatly appreciated as I plan on going much larger than just NFS type filesystem setups, if the performance can be tweaked good enough. -- Jason Williams Systems Administrator Johns Hopkins University Physics and Astronomy Department ---- Configuration Follows ---- # file: /etc/glusterfs/glusterfs-server.vol volume posix type storage/posix option directory /export/scratch end-volume volume locks type features/locks option mandatory-locks on subvolumes posix end-volume volume brick type performance/io-threads option thread-count 8 subvolumes locks end-volume volume server type protocol/server option client-volume-filename /etc/glusterfs/glusterfs-client.vol option transport-type ib-verbs/server # option transport-type tcp/server option transport.ib-verbs.listen-path 172.21.0.2 option auth.addr.brick.allow * # Edit and add list of allowed clients comma separated IP addrs(names) here subvolumes brick end-volume # file: /etc/glusterfs/glusterfs-client.vol volume remote type protocol/client option transport-type ib-verbs/client # option transport-type tcp/client option remote-host 172.21.0.2 # can be IP or hostname option remote-subvolume brick end-volume volume writebehind type performance/write-behind option block-size 1MB option window-size 4MB subvolumes remote end-volume volume cache type performance/io-cache option cache-size 512MB subvolumes writebehind end-volume