Amar Tumballi Suryanarayan
2019-Jan-24 05:54 UTC
[Gluster-users] Can't write to volume using vim/nano
I suspect this is a bug with 'Transport: rdma' part. We have called out for de-scoping that feature as we are lacking experts in that domain right now. Recommend you to use IPoIB option, and use tcp/socket transport type (which is default). That should mostly fix all the issues. -Amar On Thu, Jan 24, 2019 at 5:31 AM Jim Kinney <jim.kinney at gmail.com> wrote:> That really sounds like a bug with the sharding. I'm not using sharding on > my setup and files are writeable (vim) with 2 bytes and no errors occur. > Perhaps the small size is cached until it's large enough to trigger a write > > On Wed, 2019-01-23 at 21:46 -0200, Lindolfo Meira wrote: > > Also I noticed that any subsequent write (after the first write with 340 > > bytes or more), regardless the size, will work as expected. > > > > Lindolfo Meira, MSc > > Diretor Geral, Centro Nacional de Supercomputa??o > > Universidade Federal do Rio Grande do Sul > > +55 (51) 3308-3139 > > > On Wed, 23 Jan 2019, Lindolfo Meira wrote: > > > Just checked: when the write is >= 340 bytes, everything works as > > supposed. If the write is smaller, the error takes place. And when it > > does, nothing is logged on the server. The client, however, logs the > > following: > > > [2019-01-23 23:28:54.554664] W [MSGID: 103046] > > [rdma.c:3502:gf_rdma_decode_header] 0-rpc-transport/rdma: received a msg > > of type RDMA_ERROR > > > [2019-01-23 23:28:54.554728] W [MSGID: 103046] > > [rdma.c:3939:gf_rdma_process_recv] 0-rpc-transport/rdma: peer > > (172.24.1.6:49152), couldn't encode or decode the msg properly or write > > chunks were not provided for replies that were bigger than > > RDMA_INLINE_THRESHOLD (2048) > > > [2019-01-23 23:28:54.554765] W [MSGID: 114031] > > [client-rpc-fops_v2.c:680:client4_0_writev_cbk] 0-gfs-client-5: remote > > operation failed [Transport endpoint is not connected] > > > [2019-01-23 23:28:54.554850] W [fuse-bridge.c:1436:fuse_err_cbk] > > 0-glusterfs-fuse: 1723199: FLUSH() ERR => -1 (Transport endpoint is not > > connected) > > > > > Lindolfo Meira, MSc > > Diretor Geral, Centro Nacional de Supercomputa??o > > Universidade Federal do Rio Grande do Sul > > +55 (51) 3308-3139 > > > On Wed, 23 Jan 2019, Lindolfo Meira wrote: > > > Hi Jim. Thanks for taking the time. > > > Sorry I didn't express myself properly. It's not a simple matter of > > permissions. Users can write to the volume alright. It's when vim and nano > > are used, or when small file writes are performed (by cat or echo), that > > it doesn't work. The file is updated with the write in the server, but it > > shows up as empty in the client. > > > I guess it has something to do with the size of the write, because I ran a > > test writing to a file one byte at a time, and it never showed up as > > having any content in the client (although in the server it kept growing > > accordingly). > > > I should point out that I'm using a sharded volume. But when I was testing > > a striped volume, it also happened. Output of "gluster volume info" > > follows bellow: > > > Volume Name: gfs > > Type: Distribute > > Volume ID: b5ef065f-1ba2-481f-8108-e8f6d2d3f036 > > Status: Started > > Snapshot Count: 0 > > Number of Bricks: 6 > > Transport-type: rdma > > Bricks: > > Brick1: pfs01-ib:/mnt/data > > Brick2: pfs02-ib:/mnt/data > > Brick3: pfs03-ib:/mnt/data > > Brick4: pfs04-ib:/mnt/data > > Brick5: pfs05-ib:/mnt/data > > Brick6: pfs06-ib:/mnt/data > > Options Reconfigured: > > nfs.disable: on > > features.shard: on > > > > > Lindolfo Meira, MSc > > Diretor Geral, Centro Nacional de Supercomputa??o > > Universidade Federal do Rio Grande do Sul > > +55 (51) 3308-3139 > > > On Wed, 23 Jan 2019, Jim Kinney wrote: > > > Check permissions on the mount. I have multiple dozens of systems > > mounting 18 "exports" using fuse and it works for multiple user > > read/write based on user access permissions to the mount point space. > > /home is mounted for 150+ users plus another dozen+ lab storage spaces. > > I do manage user access with freeIPA across all systems to keep things > > consistent. > > On Wed, 2019-01-23 at 19:31 -0200, Lindolfo Meira wrote: > > Am I missing something here? A mere write operation, using vim or > > nano, cannot be performed on a gluster volume mounted over fuse! What > > gives? > > Lindolfo Meira, MScDiretor Geral, Centro Nacional de > > Supercomputa??oUniversidade Federal do Rio Grande do Sul+55 (51) > > 3308-3139_______________________________________________Gluster-users > > mailing > > listGluster-users at gluster.org > > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > -- > > James P. Kinney III > > > Every time you stop a school, you will have to build a jail. What you > > gain at one end you lose at the other. It's like feeding a dog on his > > own tail. It won't fatten the dog. > > - Speech 11/23/1900 Mark Twain > > > http://heretothereideas.blogspot.com/ > > > > -- > > James P. Kinney III Every time you stop a school, you will have to build a > jail. What you gain at one end you lose at the other. It's like feeding a > dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark > Twain http://heretothereideas.blogspot.com/ > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190124/d8af31cd/attachment.html>
I have rdma capability. Will test and report back. I'm still on v 3.12. On January 24, 2019 12:54:26 AM EST, Amar Tumballi Suryanarayan <atumball at redhat.com> wrote:>I suspect this is a bug with 'Transport: rdma' part. We have called out >for >de-scoping that feature as we are lacking experts in that domain right >now. >Recommend you to use IPoIB option, and use tcp/socket transport type >(which >is default). That should mostly fix all the issues. > >-Amar > >On Thu, Jan 24, 2019 at 5:31 AM Jim Kinney <jim.kinney at gmail.com> >wrote: > >> That really sounds like a bug with the sharding. I'm not using >sharding on >> my setup and files are writeable (vim) with 2 bytes and no errors >occur. >> Perhaps the small size is cached until it's large enough to trigger a >write >> >> On Wed, 2019-01-23 at 21:46 -0200, Lindolfo Meira wrote: >> >> Also I noticed that any subsequent write (after the first write with >340 >> >> bytes or more), regardless the size, will work as expected. >> >> >> >> Lindolfo Meira, MSc >> >> Diretor Geral, Centro Nacional de Supercomputa??o >> >> Universidade Federal do Rio Grande do Sul >> >> +55 (51) 3308-3139 >> >> >> On Wed, 23 Jan 2019, Lindolfo Meira wrote: >> >> >> Just checked: when the write is >= 340 bytes, everything works as >> >> supposed. If the write is smaller, the error takes place. And when it >> >> does, nothing is logged on the server. The client, however, logs the >> >> following: >> >> >> [2019-01-23 23:28:54.554664] W [MSGID: 103046] >> >> [rdma.c:3502:gf_rdma_decode_header] 0-rpc-transport/rdma: received a >msg >> >> of type RDMA_ERROR >> >> >> [2019-01-23 23:28:54.554728] W [MSGID: 103046] >> >> [rdma.c:3939:gf_rdma_process_recv] 0-rpc-transport/rdma: peer >> >> (172.24.1.6:49152), couldn't encode or decode the msg properly or >write >> >> chunks were not provided for replies that were bigger than >> >> RDMA_INLINE_THRESHOLD (2048) >> >> >> [2019-01-23 23:28:54.554765] W [MSGID: 114031] >> >> [client-rpc-fops_v2.c:680:client4_0_writev_cbk] 0-gfs-client-5: >remote >> >> operation failed [Transport endpoint is not connected] >> >> >> [2019-01-23 23:28:54.554850] W [fuse-bridge.c:1436:fuse_err_cbk] >> >> 0-glusterfs-fuse: 1723199: FLUSH() ERR => -1 (Transport endpoint is >not >> >> connected) >> >> >> >> >> Lindolfo Meira, MSc >> >> Diretor Geral, Centro Nacional de Supercomputa??o >> >> Universidade Federal do Rio Grande do Sul >> >> +55 (51) 3308-3139 >> >> >> On Wed, 23 Jan 2019, Lindolfo Meira wrote: >> >> >> Hi Jim. Thanks for taking the time. >> >> >> Sorry I didn't express myself properly. It's not a simple matter of >> >> permissions. Users can write to the volume alright. It's when vim and >nano >> >> are used, or when small file writes are performed (by cat or echo), >that >> >> it doesn't work. The file is updated with the write in the server, >but it >> >> shows up as empty in the client. >> >> >> I guess it has something to do with the size of the write, because I >ran a >> >> test writing to a file one byte at a time, and it never showed up as >> >> having any content in the client (although in the server it kept >growing >> >> accordingly). >> >> >> I should point out that I'm using a sharded volume. But when I was >testing >> >> a striped volume, it also happened. Output of "gluster volume info" >> >> follows bellow: >> >> >> Volume Name: gfs >> >> Type: Distribute >> >> Volume ID: b5ef065f-1ba2-481f-8108-e8f6d2d3f036 >> >> Status: Started >> >> Snapshot Count: 0 >> >> Number of Bricks: 6 >> >> Transport-type: rdma >> >> Bricks: >> >> Brick1: pfs01-ib:/mnt/data >> >> Brick2: pfs02-ib:/mnt/data >> >> Brick3: pfs03-ib:/mnt/data >> >> Brick4: pfs04-ib:/mnt/data >> >> Brick5: pfs05-ib:/mnt/data >> >> Brick6: pfs06-ib:/mnt/data >> >> Options Reconfigured: >> >> nfs.disable: on >> >> features.shard: on >> >> >> >> >> Lindolfo Meira, MSc >> >> Diretor Geral, Centro Nacional de Supercomputa??o >> >> Universidade Federal do Rio Grande do Sul >> >> +55 (51) 3308-3139 >> >> >> On Wed, 23 Jan 2019, Jim Kinney wrote: >> >> >> Check permissions on the mount. I have multiple dozens of systems >> >> mounting 18 "exports" using fuse and it works for multiple user >> >> read/write based on user access permissions to the mount point space. >> >> /home is mounted for 150+ users plus another dozen+ lab storage >spaces. >> >> I do manage user access with freeIPA across all systems to keep >things >> >> consistent. >> >> On Wed, 2019-01-23 at 19:31 -0200, Lindolfo Meira wrote: >> >> Am I missing something here? A mere write operation, using vim or >> >> nano, cannot be performed on a gluster volume mounted over fuse! What >> >> gives? >> >> Lindolfo Meira, MScDiretor Geral, Centro Nacional de >> >> Supercomputa??oUniversidade Federal do Rio Grande do Sul+55 (51) >> >> 3308-3139_______________________________________________Gluster-users >> >> mailing >> >> listGluster-users at gluster.org >> >> >> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> -- >> >> James P. Kinney III >> >> >> Every time you stop a school, you will have to build a jail. What you >> >> gain at one end you lose at the other. It's like feeding a dog on his >> >> own tail. It won't fatten the dog. >> >> - Speech 11/23/1900 Mark Twain >> >> >> http://heretothereideas.blogspot.com/ >> >> >> >> -- >> >> James P. Kinney III Every time you stop a school, you will have to >build a >> jail. What you gain at one end you lose at the other. It's like >feeding a >> dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 >Mark >> Twain http://heretothereideas.blogspot.com/ >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > > >-- >Amar Tumballi (amarts)-- Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190124/802211af/attachment.html>