search for: rdma

I've run into a problem with Gluster stability with the RDMA transport. Below is a description of the environment, a simple script that can replicate the problem, and log files from my test system. I can work around the problem by using the TCP transport over IPoIB but would like some input onto what may be making the RDMA transport fail in this case. ===...

...storage.health-check-interval 30 storage.build-pgfid off storage.bd-aio off config.transport tcp,rdma cluster.server-quorum-type off cluster.server-quorum-ratio 0 changelog.changelog off changelog.changelog-dir...

Hey guys, So, we're testing Gluster RDMA storage, and are having some issues. Things are working...just not as we expected them. THere isn't a whole lot in the way, that I've foudn on docs for gluster rdma, aside from basically "install gluster-rdma", create a volume with transport=rdma, and mount w/ transport=rdma.......

Signed-off-by: Yuval Shaia <yuval.shaia at oracle.com> --- hw/Kconfig | 1 + hw/rdma/Kconfig | 4 + hw/rdma/Makefile.objs | 2 + hw/rdma/virtio/virtio-rdma-ib.c | 287 ++++++++++++++++++++ hw/rdma/virtio/virtio-rdma-ib.h | 93 +++++++ hw/rdma/virtio/virtio-rdma-main.c | 185 +++++++++++++ hw/vi...

...e:10034: checking whether to enable OpenIB gen2 support configure:10138: cp conftest.c build && make -d modules CC=gcc -f /home/mhebenst/lustre-2.3.0/build/Makefile LUSTRE_LINUX_CONFIG=/adm in/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/.config LINUXINCLUDE= -I/usr/local/ofed/3.5/src/compat-rdma-3.5/include -I/admin/extra/l inux-2.6.32-358.2.1.el6.x86_64.crt1/arch/x86/include -I/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/arch/x86/include/generated -I /admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/include -I/admin/extra/linux-2.6.32-358.2.1.el6.x86_64.crt1/include -I/admin/extra/l...

Did any RDMA/Ethernet users see this Gluster error? If so do you know what caused it and how to fix? If you haven't seen it, what RPMs and configuration do you use specific to RDMA/Ethernet? [2011-11-10 10:30:20.595801] C [rdma.c:2417:rdma_connect_qp]0-rpc-transport/rdma: Failed to modify QP to RTR [201...

On 4/11/19 4:01 AM, Yuval Shaia wrote: > +++ b/drivers/infiniband/hw/virtio/Kconfig > @@ -0,0 +1,6 @@ > +config INFINIBAND_VIRTIO_RDMA > + tristate "VirtIO Paravirtualized RDMA Driver" > + depends on NETDEVICES && ETHERNET && PCI && INET > + ---help--- > + This driver provides low-level support for VirtIO Paravirtual > + RDMA adapter. Does this driver really depend on Ethernet,...

Dear all, I faced a problem with a glusterfs volume (pure distributed, _not_ dispersed) over RDMA transport. One user had a directory with a large number of files (50,000 files) and just doing an "ls" in this directory yields a "Transport endpoint not connected" error. The effect is, that "ls" only shows some files, but not all. The respective log file shows thi...

Is there a document with some guidelines for setting up bricks with tcp or rdma transport? I'm looking at a new deployment where the storage cluster hosts connect via 10GigE, but clients are on 1GigE. Over time, there will be 10GigE clients, but the majority will remain on 1GigE. In this setup, should the storage bricks use tcp or rdma? If tcp is the better choice, and a...

OK - finished some tests over tcp and ironed out a lot of problems. rdma is next; should be snap now.... [I must admit that this is my 1st foray into the land of IB, so some of the following may be obvious to a non-naive admin..] except that while I can create and start the volume with rdma as transport: ================================== root at pbs3:~ 622 $ gluste...

Dear Gluster-Users, I am experiencing RDMA problems. I have installed Ubuntu 16.04.4 running with 4.15.0-13-generic kernel, MLNX_OFED_LINUX-4.3-1.0.1.0-ubuntu16.04-x86_64 to 4 different servers. All of them has Mellanox ConnectX-4 LX dual port NICs. These four servers are connected via Mellanox SN2100 Switch. I have installed GlusterFS...

Hi, I've recently install glusterfs on a cluster of several machines, and although it works very nicely when I use TCP connections I completely fail to use RDMA connections. I have 10 machines with IB NICs, and RDMA does work between them (verified using ib_write_bw -r and several other apps). When I configure a volume with RDMA I have a problem with the port at the client side. It keeps trying to get to port 24008, no matter what port the server actuall...

Stefan, Sounds like a brick process is not running. I have notice some strangeness in my lab when using RDMA, I often have to forcibly restart the brick process, often as in every single time I do a major operation, add a new volume, remove a volume, stop a volume, etc. gluster volume status <vol> Does any of the self heal daemons show N/A? If that's the case, try forcing a restart on the volu...

...5/19 12:35 PM, Yuval Shaia wrote: > > On Thu, Apr 11, 2019 at 07:02:15PM +0200, Cornelia Huck wrote: > > > On Thu, 11 Apr 2019 14:01:54 +0300 > > > Yuval Shaia <yuval.shaia at oracle.com> wrote: > > > > > > > Data center backends use more and more RDMA or RoCE devices and more and > > > > more software runs in virtualized environment. > > > > There is a need for a standard to enable RDMA/RoCE on Virtual Machines. > > > > > > > > Virtio is the optimal solution since is the de-facto para-virtualizato...

...5/19 12:35 PM, Yuval Shaia wrote: > > On Thu, Apr 11, 2019 at 07:02:15PM +0200, Cornelia Huck wrote: > > > On Thu, 11 Apr 2019 14:01:54 +0300 > > > Yuval Shaia <yuval.shaia at oracle.com> wrote: > > > > > > > Data center backends use more and more RDMA or RoCE devices and more and > > > > more software runs in virtualized environment. > > > > There is a need for a standard to enable RDMA/RoCE on Virtual Machines. > > > > > > > > Virtio is the optimal solution since is the de-facto para-virtualizato...

Forgot to mention, sometimes I have to do force start other volumes as well, its hard to determine which brick process is locked up from the logs. Status of volume: rhev_vms_primary Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick spidey.ib.runlevelone.lan:/gluster/brick/rhev_vms_primary 0 49157 Y 15666 Brick deadpool.ib.runlevelone.lan:/gluster/brick/rhev_vms_primary 0 49156 Y...

Hi all, I have downloaded and installed the following: [root at compute-0-0 ~]# rpm -qa | grep gluster glusterfs-rdma-3.3beta2-1 glusterfs-fuse-3.3beta2-1 glusterfs-core-3.3beta2-1 I have also compiled all those from source but they didn't work either. The problem I'm having is that when I start glusterd I get the following and I'm not sure how to fix it: [2012-01-26 23:51:01.800533] I [glusterd.c:57...

...eaddirp on Can I turn off these safely? (or what precisely do they do?) I also assured that all glusterd processes have unlimited locked memory. Just to state it clearly: I do _not_ see any data corruption. Just the directory listings do not work (in very rare cases) with rdma transport: "ls" shows only a part of the files. but then I do: stat /path/to/known/filename it succeeds, and even md5sum /path/to/known/filename/that/does/not/get/listed/with/ls yields the correct result. best wishes, Stefan > Am 30.05.2018 um 03:00 schrieb Dan Lavu <dan...

Is infiniband itself working fine? You can run tools like ibv_rc_pingpong to find out. On Wed, Apr 25, 2018 at 12:23 PM, Necati E. SISECI <siseci at gmail.com> wrote: > Dear Gluster-Users, > > I am experiencing RDMA problems. > > I have installed Ubuntu 16.04.4 running with 4.15.0-13-generic kernel, > MLNX_OFED_LINUX-4.3-1.0.1.0-ubuntu16.04-x86_64 to 4 different servers. > All of them has Mellanox ConnectX-4 LX dual port NICs. These four servers > are connected via Mellanox SN2100 Switch. > &...

On Thu, Apr 11, 2019 at 07:02:15PM +0200, Cornelia Huck wrote: > On Thu, 11 Apr 2019 14:01:54 +0300 > Yuval Shaia <yuval.shaia at oracle.com> wrote: > > > Data center backends use more and more RDMA or RoCE devices and more and > > more software runs in virtualized environment. > > There is a need for a standard to enable RDMA/RoCE on Virtual Machines. > > > > Virtio is the optimal solution since is the de-facto para-virtualizaton > > technology and also because...