hi, nbalacha, Sorry for my later reply. I have tried to turn off cluster.readdir-optimize, it's of no help. I create a similar volume with default option, and mv command seems ok, but when I enable quota, mv will lost file, the volume option is list as below: Brick168: node33:/data13/bricks/test4 Options Reconfigured: features.quota-deem-statfs: on features.inode-quota: on features.quota: on transport.address-family: inet nfs.disable: on performance.client-io-threads: off so I think it seems a bug of quota machanism, but I cant understand why. Howerer, when i create a similar volume with only 2x2 bricks and enable quota, mv will lost file occasionally?if i dont make a mistake. without quota, everything seems ok. Yours Yu Date: Wed, 5 Sep 2018 14:45:36 +0530> From: Nithya Balachandran <nbalacha at redhat.com> > To: yu sun <sunyu1949 at gmail.com> > Cc: gluster-users <gluster-users at gluster.org> > Subject: Re: [Gluster-users] mv lost some files ? > Message-ID: > <CAOUCJ=g9wd2zkcBd5> VBzt9-nJmPww6WJgjsm8p7xvPTTmqZLQ at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On 5 September 2018 at 14:10, Nithya Balachandran <nbalacha at redhat.com> > wrote: > > > > > > > On 5 September 2018 at 14:02, Nithya Balachandran <nbalacha at redhat.com> > > wrote: > > > >> Hi, > >> > >> Please try turning off cluster.readdir-optimize and see it if helps. > >> > > > > You can also try turning off parallel-readdir. > > > > > Please ignore this - it is not enabled on your volume. > > > > > > >> If not, please send us the client mount logs and a tcpdump of when the > >> *ls* is performed from the client. Use the following to capture the > >> dump: > >> > >> tcpdump -i any -s 0 -w /var/tmp/dirls.pcap tcp and not port 22 > >> > >> > >> > >> Thanks, > >> Nithya > >> > >> > >>> > >>> Raghavendra Gowdappa <rgowdapp at redhat.com> ?2018?9?5??? ??12:40??? > >>> > >>>> > >>>> > >>>> > >>>>> > >>>>>> > >>>>>> On Tue, Sep 4, 2018 at 5:28 PM, yu sun <sunyu1949 at gmail.com> wrote: > >>>>>> > >>>>>>> Hi all: > >>>>>>> > >>>>>>> I have a replicated volume project2 with info: > >>>>>>> Volume Name: project2 Type: Distributed-Replicate Volume ID: > >>>>>>> 60175b8e-de0e-4409-81ae-7bb5eb5cacbf Status: Started Snapshot > >>>>>>> Count: 0 Number of Bricks: 84 x 2 = 168 Transport-type: tcp > Bricks: Brick1: > >>>>>>> node20:/data2/bricks/project2 Brick2: > node21:/data2/bricks/project2 Brick3: > >>>>>>> node22:/data2/bricks/project2 Brick4: > node23:/data2/bricks/project2 Brick5: > >>>>>>> node24:/data2/bricks/project2 Brick6: > node25:/data2/bricks/project2 Brick7: > >>>>>>> node26:/data2/bricks/project2 Brick8: > node27:/data2/bricks/project2 Brick9: > >>>>>>> node28:/data2/bricks/project2 Brick10: > node29:/data2/bricks/project2 > >>>>>>> Brick11: node30:/data2/bricks/project2 Brick12: > >>>>>>> node31:/data2/bricks/project2 Brick13: > node32:/data2/bricks/project2 > >>>>>>> Brick14: node33:/data2/bricks/project2 Brick15: > >>>>>>> node20:/data3/bricks/project2 Brick16: > node21:/data3/bricks/project2 > >>>>>>> Brick17: node22:/data3/bricks/project2 Brick18: > >>>>>>> node23:/data3/bricks/project2 Brick19: > node24:/data3/bricks/project2 > >>>>>>> Brick20: node25:/data3/bricks/project2 Brick21: > >>>>>>> node26:/data3/bricks/project2 Brick22: > node27:/data3/bricks/project2 > >>>>>>> Brick23: node28:/data3/bricks/project2 Brick24: > >>>>>>> node29:/data3/bricks/project2 Brick25: > node30:/data3/bricks/project2 > >>>>>>> Brick26: node31:/data3/bricks/project2 Brick27: > >>>>>>> node32:/data3/bricks/project2 Brick28: > node33:/data3/bricks/project2 > >>>>>>> Brick29: node20:/data4/bricks/project2 Brick30: > >>>>>>> node21:/data4/bricks/project2 Brick31: > node22:/data4/bricks/project2 > >>>>>>> Brick32: node23:/data4/bricks/project2 Brick33: > >>>>>>> node24:/data4/bricks/project2 Brick34: > node25:/data4/bricks/project2 > >>>>>>> Brick35: node26:/data4/bricks/project2 Brick36: > >>>>>>> node27:/data4/bricks/project2 Brick37: > node28:/data4/bricks/project2 > >>>>>>> Brick38: node29:/data4/bricks/project2 Brick39: > >>>>>>> node30:/data4/bricks/project2 Brick40: > node31:/data4/bricks/project2 > >>>>>>> Brick41: node32:/data4/bricks/project2 Brick42: > >>>>>>> node33:/data4/bricks/project2 Brick43: > node20:/data5/bricks/project2 > >>>>>>> Brick44: node21:/data5/bricks/project2 Brick45: > >>>>>>> node22:/data5/bricks/project2 Brick46: > node23:/data5/bricks/project2 > >>>>>>> Brick47: node24:/data5/bricks/project2 Brick48: > >>>>>>> node25:/data5/bricks/project2 Brick49: > node26:/data5/bricks/project2 > >>>>>>> Brick50: node27:/data5/bricks/project2 Brick51: > >>>>>>> node28:/data5/bricks/project2 Brick52: > node29:/data5/bricks/project2 > >>>>>>> Brick53: node30:/data5/bricks/project2 Brick54: > >>>>>>> node31:/data5/bricks/project2 Brick55: > node32:/data5/bricks/project2 > >>>>>>> Brick56: node33:/data5/bricks/project2 Brick57: > >>>>>>> node20:/data6/bricks/project2 Brick58: > node21:/data6/bricks/project2 > >>>>>>> Brick59: node22:/data6/bricks/project2 Brick60: > >>>>>>> node23:/data6/bricks/project2 Brick61: > node24:/data6/bricks/project2 > >>>>>>> Brick62: node25:/data6/bricks/project2 Brick63: > >>>>>>> node26:/data6/bricks/project2 Brick64: > node27:/data6/bricks/project2 > >>>>>>> Brick65: node28:/data6/bricks/project2 Brick66: > >>>>>>> node29:/data6/bricks/project2 Brick67: > node30:/data6/bricks/project2 > >>>>>>> Brick68: node31:/data6/bricks/project2 Brick69: > >>>>>>> node32:/data6/bricks/project2 Brick70: > node33:/data6/bricks/project2 > >>>>>>> Brick71: node20:/data7/bricks/project2 Brick72: > >>>>>>> node21:/data7/bricks/project2 Brick73: > node22:/data7/bricks/project2 > >>>>>>> Brick74: node23:/data7/bricks/project2 Brick75: > >>>>>>> node24:/data7/bricks/project2 Brick76: > node25:/data7/bricks/project2 > >>>>>>> Brick77: node26:/data7/bricks/project2 Brick78: > >>>>>>> node27:/data7/bricks/project2 Brick79: > node28:/data7/bricks/project2 > >>>>>>> Brick80: node29:/data7/bricks/project2 Brick81: > >>>>>>> node30:/data7/bricks/project2 Brick82: > node31:/data7/bricks/project2 > >>>>>>> Brick83: node32:/data7/bricks/project2 Brick84: > >>>>>>> node33:/data7/bricks/project2 Brick85: > node20:/data8/bricks/project2 > >>>>>>> Brick86: node21:/data8/bricks/project2 Brick87: > >>>>>>> node22:/data8/bricks/project2 Brick88: > node23:/data8/bricks/project2 > >>>>>>> Brick89: node24:/data8/bricks/project2 Brick90: > >>>>>>> node25:/data8/bricks/project2 Brick91: > node26:/data8/bricks/project2 > >>>>>>> Brick92: node27:/data8/bricks/project2 Brick93: > >>>>>>> node28:/data8/bricks/project2 Brick94: > node29:/data8/bricks/project2 > >>>>>>> Brick95: node30:/data8/bricks/project2 Brick96: > >>>>>>> node31:/data8/bricks/project2 Brick97: > node32:/data8/bricks/project2 > >>>>>>> Brick98: node33:/data8/bricks/project2 Brick99: > >>>>>>> node20:/data9/bricks/project2 Brick100: > node21:/data9/bricks/project2 > >>>>>>> Brick101: node22:/data9/bricks/project2 Brick102: > >>>>>>> node23:/data9/bricks/project2 Brick103: > node24:/data9/bricks/project2 > >>>>>>> Brick104: node25:/data9/bricks/project2 Brick105: > >>>>>>> node26:/data9/bricks/project2 Brick106: > node27:/data9/bricks/project2 > >>>>>>> Brick107: node28:/data9/bricks/project2 Brick108: > >>>>>>> node29:/data9/bricks/project2 Brick109: > node30:/data9/bricks/project2 > >>>>>>> Brick110: node31:/data9/bricks/project2 Brick111: > >>>>>>> node32:/data9/bricks/project2 Brick112: > node33:/data9/bricks/project2 > >>>>>>> Brick113: node20:/data10/bricks/project2 Brick114: > >>>>>>> node21:/data10/bricks/project2 Brick115: > node22:/data10/bricks/project2 > >>>>>>> Brick116: node23:/data10/bricks/project2 Brick117: > >>>>>>> node24:/data10/bricks/project2 Brick118: > node25:/data10/bricks/project2 > >>>>>>> Brick119: node26:/data10/bricks/project2 Brick120: > >>>>>>> node27:/data10/bricks/project2 Brick121: > node28:/data10/bricks/project2 > >>>>>>> Brick122: node29:/data10/bricks/project2 Brick123: > >>>>>>> node30:/data10/bricks/project2 Brick124: > node31:/data10/bricks/project2 > >>>>>>> Brick125: node32:/data10/bricks/project2 Brick126: > >>>>>>> node33:/data10/bricks/project2 Brick127: > node20:/data11/bricks/project2 > >>>>>>> Brick128: node21:/data11/bricks/project2 Brick129: > >>>>>>> node22:/data11/bricks/project2 Brick130: > node23:/data11/bricks/project2 > >>>>>>> Brick131: node24:/data11/bricks/project2 Brick132: > >>>>>>> node25:/data11/bricks/project2 Brick133: > node26:/data11/bricks/project2 > >>>>>>> Brick134: node27:/data11/bricks/project2 Brick135: > >>>>>>> node28:/data11/bricks/project2 Brick136: > node29:/data11/bricks/project2 > >>>>>>> Brick137: node30:/data11/bricks/project2 Brick138: > >>>>>>> node31:/data11/bricks/project2 Brick139: > node32:/data11/bricks/project2 > >>>>>>> Brick140: node33:/data11/bricks/project2 Brick141: > >>>>>>> node20:/data12/bricks/project2 Brick142: > node21:/data12/bricks/project2 > >>>>>>> Brick143: node22:/data12/bricks/project2 Brick144: > >>>>>>> node23:/data12/bricks/project2 Brick145: > node24:/data12/bricks/project2 > >>>>>>> Brick146: node25:/data12/bricks/project2 Brick147: > >>>>>>> node26:/data12/bricks/project2 Brick148: > node27:/data12/bricks/project2 > >>>>>>> Brick149: node28:/data12/bricks/project2 Brick150: > >>>>>>> node29:/data12/bricks/project2 Brick151: > node30:/data12/bricks/project2 > >>>>>>> Brick152: node31:/data12/bricks/project2 Brick153: > >>>>>>> node32:/data12/bricks/project2 Brick154: > node33:/data12/bricks/project2 > >>>>>>> Brick155: node20:/data13/bricks/project2 Brick156: > >>>>>>> node21:/data13/bricks/project2 Brick157: > node22:/data13/bricks/project2 > >>>>>>> Brick158: node23:/data13/bricks/project2 Brick159: > >>>>>>> node24:/data13/bricks/project2 Brick160: > node25:/data13/bricks/project2 > >>>>>>> Brick161: node26:/data13/bricks/project2 Brick162: > >>>>>>> node27:/data13/bricks/project2 Brick163: > node28:/data13/bricks/project2 > >>>>>>> Brick164: node29:/data13/bricks/project2 Brick165: > >>>>>>> node30:/data13/bricks/project2 Brick166: > node31:/data13/bricks/project2 > >>>>>>> Brick167: node32:/data13/bricks/project2 Brick168: > >>>>>>> node33:/data13/bricks/project2 Options Reconfigured: > >>>>>>> performance.force-readdirp: on performance.write-behind: off > >>>>>>> performance.stat-prefetch: on performance.client-io-threads: on > >>>>>>> nfs.disable: on transport.address-family: inet features.quota: on > >>>>>>> features.inode-quota: on features.quota-deem-statfs: on > >>>>>>> cluster.readdir-optimize: on cluster.lookup-optimize: on > >>>>>>> dht.force-readdirp: off client.event-threads: 10 > server.event-threads: 10 > >>>>>>> performance.readdir-ahead: on performance.io-cache: on > >>>>>>> performance.flush-behind: on performance.cache-size: 5GB > >>>>>>> performance.cache-max-file-size: 1MB > performance.write-behind-window-size: > >>>>>>> 10MB performance.read-ahead: off network.remote-dio: enable > >>>>>>> performance.strict-o-direct: disable performance.io-thread-count: > 25 > >>>>>>> > >>>>>>> > >>>>>>> the volume looks ok, and I mount this volume on my client machine: > >>>>>>> mount -t glusterfs -o oom-score-adj=-999 -o direct-io-mode=disable > >>>>>>> -o use-readdirp=no node20:/project2 /mnt/project2 > >>>>>>> > >>>>>>> I have a directory in /mnt/project2/, but when I mv the directory > to > >>>>>>> other dirs, files in the dir lost while tree or ls the dir, some > files > >>>>>>> missing, my operations is list as below: > >>>>>>> > >>>>>> > >>>>>> Looks very similar to: > >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1118762 > >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1337394 > >>>>>> > >>>>>> > >>>>>> > >>>>>>> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ mkdir test-dir > >>>>>>> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ tree > >>>>>>> . > >>>>>>> ??? face_landmarks > >>>>>>> ? ??? alive > >>>>>>> ? ??? logs_20180823_28 > >>>>>>> ? ??? info_000000.out > >>>>>>> ? ??? info_000001.out > >>>>>>> ? ??? info_000002.out > >>>>>>> ? ??? info_000003.out > >>>>>>> ? ??? info_000004.out > >>>>>>> ? ??? info_000005.out > >>>>>>> ? ??? info_000006.out > >>>>>>> ? ??? info_000007.out > >>>>>>> ? ??? info_000008.out > >>>>>>> ? ??? info_000009.out > >>>>>>> ? ??? info_000010.out > >>>>>>> ? ??? info_000011.out > >>>>>>> ? ??? info_000012.out > >>>>>>> ? ??? info_000013.out > >>>>>>> ? ??? info_000014.out > >>>>>>> ? ??? info_000015.out > >>>>>>> ? ??? info_000016.out > >>>>>>> ? ??? info_000017.out > >>>>>>> ? ??? info_000018.out > >>>>>>> ? ??? info_000019.out > >>>>>>> ??? test-dir > >>>>>>> > >>>>>>> 4 directories, 20 files > >>>>>>> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ mv > >>>>>>> face_landmarks/ test-dir/ > >>>>>>> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ tree > >>>>>>> . > >>>>>>> ??? test-dir > >>>>>>> ??? face_landmarks > >>>>>>> > >>>>>>> 2 directories, 0 files > >>>>>>> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ cd > >>>>>>> test-dir/face_landmarks/ > >>>>>>> root at ml-gpu-ser129.nmg01 > :/mnt/project2/371_37829/test-dir/face_landmarks$ > >>>>>>> ls > >>>>>>> root at ml-gpu-ser129.nmg01 > :/mnt/project2/371_37829/test-dir/face_landmarks$ > >>>>>>> cd .. > >>>>>>> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829/test-dir$ mv > >>>>>>> face_landmarks/ .. > >>>>>>> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829/test-dir$ cd .. > >>>>>>> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ tree > >>>>>>> . > >>>>>>> ??? face_landmarks > >>>>>>> ? ??? alive > >>>>>>> ??? test-dir > >>>>>>> > >>>>>>> 3 directories, 0 files > >>>>>>> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ > >>>>>>> > >>>>>>> I think i make some mistakes with volume option, buti i am not > sure, > >>>>>>> so how can i find the lost files? the files seems still int the > directory, > >>>>>>> because i cant remove the directory and rm tell me "Not empty > directory" > >>>>>>> > >>>>>> > >>>>>> Its likely that src and dst of mv having same gfid and that's > causing > >>>>>> the issues. Can you look into both src and dst paths on all bricks? > Union > >>>>>> of contents of both directories should give all the files in the src > >>>>>> directory before mv. Once found you can, > >>>>>> * keep a backup of contents of src and dst on all bricks > >>>>>> * remove trusted.gfid xattr on src and dst from all bricks > >>>>>> * remove gfid handle (.glusterfs/<first two characters of > >>>>>> gfid>/<second set of two characters of gfid>/<gfid> on each brick) > >>>>>> * disable readdirplus in entire stack (maybe you can use a tmp mount > >>>>>> for this) [1] > >>>>>> * stat src and dst on a mount point with readdirplus disabled. > >>>>>> * Now you'll see two directories src and dst on mountpoint. You can > >>>>>> copy the contents of both into a new directory > >>>>>> > >>>>>> [1] https://lists.gluster.org/pipermail/gluster-users/2017-March > >>>>>> /030148.html > >>>>>> > >>>>>> > >>>>>>> > >>>>>>> Any suggestions is appreciated. > >>>>>>> Many Thanks > >>>>>>> > >>>>>>> Best regards > >>>>>>> Yu > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> Gluster-users mailing list > >>>>>>> Gluster-users at gluster.org > >>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users > >>>>>>> > >>>>>> > >>>>>> > >>>> > >>> _______________________________________________ > >>> Gluster-users mailing list > >>> Gluster-users at gluster.org > >>> https://lists.gluster.org/mailman/listinfo/gluster-users > >>> > >> > >> > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.gluster.org/pipermail/gluster-users/attachments/20180905/b869cf39/attachment-0001.html > > > > ------------------------------ > > Message: 2 > Date: Wed, 5 Sep 2018 10:21:51 +0000 > From: Krishna Verma <kverma at cadence.com> > To: Gluster Users <gluster-users at gluster.org> > Subject: [Gluster-users] Master-Slave-Replicated Volume size is > different > Message-ID: > < > BY2PR07MB9683D3DB74BD7A1F3AAA04CD8020 at BY2PR07MB968.namprd07.prod.outlook.com > > > > Content-Type: text/plain; charset="us-ascii" > > Hi All, > > I have a geo replication setup of 2*1 with replicated volume. > > The replicated volume is showing difference in "used size" between master > and slave. > > At master "/repvol" showing only 2% use. > > [root at noi-foreman02 repvol]# df -hT /repvol > Filesystem Type Size Used Avail Use% Mounted > on > gluster-poc-noida:/glusterep fuse.glusterfs 30G 402M 28G 2% /repvol > [root at noi-foreman02 repvol]# > > Whereas at Slave it shows 71% use. > > [root at sj-kverma repvol]# df -hT /repvol > Filesystem Type Size Used Avail Use% Mounted on > gluster-poc-sj:glusterep fuse.glusterfs 30G 20G 8.2G 71% /repvol > You have new mail in /var/spool/mail/root > [root at sj-kverma repvol]# > > Could anyone please help to replicated it correctly with actual size. > There is no data at master and slave. > > /Krishna > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.gluster.org/pipermail/gluster-users/attachments/20180905/2ab493bd/attachment-0001.html > > > > ------------------------------ > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > End of Gluster-users Digest, Vol 125, Issue 8 > ********************************************* >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180905/53506c5a/attachment.html>