Dan Bretherton
2010-Nov-28 23:33 UTC
[Gluster-users] Directory rename prevented by mismatching layouts in 3.1
Hello All, I am having a lot of problems with a distributed, replicated volume in GlusterFS 3.1. A user tried to rename a directory using "mv ANALYSIS ANALYSIS-6HR", but received the error "operation not permitted". He then saw that both the original and the newly named directories were there, but neither contained the data that had been in ANALYSIS. I was called in to have a look, and saw another permission error when the user tried to remame ANALYSIS-6HR back to ANALYSIS, and another one on trying to rename ANALYSIS-6HR to ANALYSIS-6HR-TT. Now all three versions are there but none contains the files that were originaly in ANALYSIS. The directories with the first two names are now owned by root, as shown below in the parent directory listing. [root at nemo1 logs]# ls -l /glusterfs/atmos/users/kih/INTERIM total 244 drwxr-x--- 2 root root 16384 Nov 26 16:57 ANALYSIS drwxr-xr-x 2 root root 16384 Nov 26 17:04 ANALYSIS-6HR drwxr-xr-x 2 kih essc 16384 Nov 26 16:58 ANALYSIS-6HR-TT The glusterfs-atmos.log file errors resulting from the above command are shown at the bottom of this message. Running a self heal (i.e. ls -alR) on the volume doesn't solve the problem, only gives more of the same errors. The files are actually located in ANALYSIS-6HR-TT, where they should be after the two rename operations, but only on one of the replicated bricks. They do not show up in the mounted GlusterFS volume (NFS or native GlusterFS client). Does anybody know how to make the files visible again and stop this behaviour from occurring? -Dan; See log file extract below: [2010-11-28 22:57:53.57596] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS [2010-11-28 22:57:53.57761] I [dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol: atmos-replicate-2; inode layout - 0 - 0; disk layout - 0 - 2147483646 [2010-11-28 22:57:53.57781] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS [2010-11-28 22:57:53.57872] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS [2010-11-28 22:57:53.57927] I [dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol: atmos-replicate-3; inode layout - 0 - 0; disk layout - 2147483647 - 4294967295 [2010-11-28 22:57:53.57945] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS [2010-11-28 22:57:53.69267] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS [2010-11-28 22:57:53.69392] I [dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol: atmos-replicate-2; inode layout - 0 - 0; disk layout - 0 - 2147483646 [2010-11-28 22:57:53.69410] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS [2010-11-28 22:57:53.69523] I [dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol: atmos-replicate-3; inode layout - 0 - 0; disk layout - 2147483647 - 4294967295 [2010-11-28 22:57:53.69542] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS [2010-11-28 22:57:53.69581] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS [2010-11-28 22:57:53.163971] I [dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol: atmos-replicate-1; inode layout - 1073741823 - 2147483645; disk layout - 2147483646 - 3221225468 [2010-11-28 22:57:53.164051] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR [2010-11-28 22:57:53.164098] I [dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol: atmos-replicate-2; inode layout - 2147483646 - 3221225468; disk layout - 3221225469 - 4294967295 [2010-11-28 22:57:53.164116] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR [2010-11-28 22:57:53.164159] I [dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol: atmos-replicate-3; inode layout - 3221225469 - 4294967295; disk layout - 0 - 1073741822 [2010-11-28 22:57:53.164176] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR [2010-11-28 22:57:53.164695] I [dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol: atmos-replicate-0; inode layout - 0 - 1073741822; disk layout - 1073741823 - 2147483645 [2010-11-28 22:57:53.164712] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR [2010-11-28 22:57:53.178847] I [dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol: atmos-replicate-1; inode layout - 1073741823 - 2147483645; disk layout - 2147483646 - 3221225468 [2010-11-28 22:57:53.178884] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR [2010-11-28 22:57:53.178942] I [dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol: atmos-replicate-2; inode layout - 2147483646 - 3221225468; disk layout - 3221225469 - 4294967295 [2010-11-28 22:57:53.178960] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR [2010-11-28 22:57:53.179267] I [dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol: atmos-replicate-3; inode layout - 3221225469 - 4294967295; disk layout - 0 - 1073741822 [2010-11-28 22:57:53.179286] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR [2010-11-28 22:57:53.179582] I [dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol: atmos-replicate-0; inode layout - 0 - 1073741822; disk layout - 1073741823 - 2147483645 [2010-11-28 22:57:53.179600] I [dht-common.c:415:dht_revalidate_cbk] atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR