Susant Palai
2015-Oct-30 07:22 UTC
[Gluster-users] Missing files after add new bricks and remove old ones - how to restore files
Hi Marco,
Can you send the stat of the files from the removed-brick?
Susant
----- Original Message -----
From: "Marco Lorenzo Crociani" <marcoc at
prismatelecomtesting.com>
To: gluster-users at gluster.org
Sent: Tuesday, 27 October, 2015 6:58:51 PM
Subject: [Gluster-users] Missing files after add new bricks and remove old ones
- how to restore files
Hi,
we had a 2 node glusterfs cluster.
we had a 2x2 Distributed-Replicate volume on it.
It was:
Brick1: s20gfs.ovirt:/gluster/VOL/brick1
Brick2: s21gfs.ovirt:/gluster/VOL/brick2
Brick3: s20gfs.ovirt:/gluster/VOL/brick3
Brick4: s21gfs.ovirt:/gluster/VOL/brick4
We added more nodes to the cluster. So I wanted to redistribute the
bricks on the nodes.
I added 2 new bricks to the volume.
gluster volume add-brick VOL s22gfs.ovirt:/gluster/VOL/brick2
x`s23gfs.ovirt:/gluster/VOL/brick3
then I removed 2 old bricks
gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1
s21gfs.ovirt:/gluster/VOL/brick2 start
checked the status
gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1
s21gfs.ovirt:/gluster/VOL/brick2 status
when it was completed and I saw data on the new bricks I run:
gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1
s21gfs.ovirt:/gluster/VOL/brick2 commit
Result:
some files missing from the volume. Those file are still on the removed
bricks.
First question. Have I followed a wrong procedure?
Second question. How to restore those files? I can't re-add back those
brick to the volume because it tolds me:
volume add-brick: failed: /gluster/VOL/brick1 is already part of a volume
If I use the force option I get the old files or they get erased?
Should I rsync from the "unmounted" brick to the "mounted"
volume or to
an "umounted" brick that is part of the volume?
Regards,
--
Marco Crociani
Prisma Telecom Testing S.r.l.
via Petrocchi, 4 20127 MILANO ITALY
Phone: +39 02 26113507
Fax: +39 02 26113597
e-mail: marcoc at prismatelecomtesting.com
web: http://www.prismatelecomtesting.com
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
Marco Lorenzo Crociani
2015-Oct-30 16:51 UTC
[Gluster-users] Missing files after add new bricks and remove old ones - how to restore files
Hi Susant, here the stats: [root at s20 brick1]# stat .* * File: `.' Size: 78 Blocks: 0 IO Block: 4096 directory Device: 811h/2065d Inode: 2481712637 Links: 7 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-30 15:26:57.565475699 +0100 Modify: 2015-08-04 12:30:56.604846056 +0200 Change: 2015-10-27 14:21:12.981420157 +0100 File: `..' Size: 50 Blocks: 0 IO Block: 4096 directory Device: 811h/2065d Inode: 495824 Links: 6 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-27 11:27:36.956230769 +0100 Modify: 2015-08-04 11:25:27.893410342 +0200 Change: 2015-08-04 11:25:27.893410342 +0200 File: `.glusterfs' Size: 8192 Blocks: 24 IO Block: 4096 directory Device: 811h/2065d Inode: 2481712643 Links: 261 Access: (0600/drw-------) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-30 15:31:14.988775452 +0100 Modify: 2015-08-04 12:31:40.803715075 +0200 Change: 2015-08-04 12:31:40.803715075 +0200 File: `.trashcan' Size: 24 Blocks: 0 IO Block: 4096 directory Device: 811h/2065d Inode: 495865 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 18:32:17.369070847 +0100 Modify: 2015-08-04 11:36:11.357529000 +0200 Change: 2015-10-26 18:32:17.368070850 +0100 File: `lost+found' Size: 6 Blocks: 0 IO Block: 4096 directory Device: 811h/2065d Inode: 2481712624 Links: 2 Access: (0700/drwx------) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 18:55:08.274323554 +0100 Modify: 2014-01-18 21:48:37.000000000 +0100 Change: 2015-10-26 18:55:08.259323594 +0100 File: `rh' Size: 6 Blocks: 0 IO Block: 4096 directory Device: 811h/2065d Inode: 495961 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 14:02:02.294771698 +0100 Modify: 2015-03-26 13:22:19.000000000 +0100 Change: 2015-10-26 18:32:17.384070805 +0100 File: `zimbra' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 811h/2065d Inode: 495969 Links: 50 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-30 15:27:44.899346957 +0100 Modify: 2015-10-26 18:32:17.733069841 +0100 Change: 2015-10-26 18:32:17.733069841 +0100 [root at s21 brick2]# stat .* * File: `.' Size: 78 Blocks: 0 IO Block: 4096 directory Device: 811h/2065d Inode: 501309 Links: 7 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-30 17:45:23.983929018 +0100 Modify: 2015-08-04 12:30:56.602392330 +0200 Change: 2015-10-26 18:32:17.327779305 +0100 File: `..' Size: 50 Blocks: 0 IO Block: 4096 directory Device: 811h/2065d Inode: 2484780736 Links: 6 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-30 17:45:20.800922878 +0100 Modify: 2015-08-04 11:25:27.942732803 +0200 Change: 2015-08-04 11:25:27.942732803 +0200 File: `.glusterfs' Size: 8192 Blocks: 24 IO Block: 4096 directory Device: 811h/2065d Inode: 501323 Links: 261 Access: (0600/drw-------) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-08-04 11:36:13.776967886 +0200 Modify: 2015-08-04 12:31:40.801477366 +0200 Change: 2015-08-04 12:31:40.801477366 +0200 File: `.trashcan' Size: 24 Blocks: 0 IO Block: 4096 directory Device: 811h/2065d Inode: 2484780773 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 18:32:17.324779299 +0100 Modify: 2015-08-04 11:36:11.357529000 +0200 Change: 2015-10-26 18:32:17.368779386 +0100 File: `lost+found' Size: 6 Blocks: 0 IO Block: 4096 directory Device: 811h/2065d Inode: 501268 Links: 2 Access: (0700/drwx------) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 18:32:17.371779392 +0100 Modify: 2014-01-18 21:48:37.000000000 +0100 Change: 2015-10-26 18:55:08.260516194 +0100 File: `rh' Size: 6 Blocks: 0 IO Block: 4096 directory Device: 811h/2065d Inode: 2484780842 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 18:32:17.386779422 +0100 Modify: 2015-03-26 13:22:19.000000000 +0100 Change: 2015-10-26 18:32:17.384779418 +0100 File: `zimbra' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 811h/2065d Inode: 2484780856 Links: 50 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 18:34:10.300004939 +0100 Modify: 2015-10-26 18:32:17.733780116 +0100 Change: 2015-10-26 18:32:17.733780116 +0100 [root at s20 brick1]# stat zimbra/jdk-1.7.0_45/db/bin/* File: `zimbra/jdk-1.7.0_45/db/bin/dblook' Size: 5740 Blocks: 16 IO Block: 4096 regular file Device: 811h/2065d Inode: 509260 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.509803807 +0100 Modify: 2012-11-02 12:23:32.000000000 +0100 Change: 2015-10-26 15:31:30.510803804 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/derby_common.bat' Size: 2426 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 509263 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.510803804 +0100 Modify: 2012-11-01 10:49:22.000000000 +0100 Change: 2015-10-26 15:31:30.511803801 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/ij' Size: 5876 Blocks: 16 IO Block: 4096 regular file Device: 811h/2065d Inode: 509264 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.512803798 +0100 Modify: 2012-11-02 12:23:32.000000000 +0100 Change: 2015-10-26 15:31:30.512803798 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/NetworkServerControl' Size: 5801 Blocks: 16 IO Block: 4096 regular file Device: 811h/2065d Inode: 509257 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.507803811 +0100 Modify: 2012-11-02 12:23:32.000000000 +0100 Change: 2015-10-26 15:31:30.508803809 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/setEmbeddedCP.bat' Size: 1278 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 509269 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.513803795 +0100 Modify: 2012-11-01 10:49:22.000000000 +0100 Change: 2015-10-26 15:31:30.514803792 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/setNetworkClientCP.bat' Size: 1284 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 509272 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.514803792 +0100 Modify: 2012-11-01 10:49:22.000000000 +0100 Change: 2015-10-26 15:31:30.515803790 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/setNetworkServerCP' Size: 1075 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 509273 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.516803788 +0100 Modify: 2012-11-01 10:49:22.000000000 +0100 Change: 2015-10-26 15:31:30.517803786 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/startNetworkServer' Size: 5807 Blocks: 16 IO Block: 4096 regular file Device: 811h/2065d Inode: 509276 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.517803786 +0100 Modify: 2012-11-02 12:23:32.000000000 +0100 Change: 2015-10-26 15:31:30.518803783 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/startNetworkServer.bat' Size: 1397 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 509278 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.519803780 +0100 Modify: 2012-11-01 10:49:22.000000000 +0100 Change: 2015-10-26 15:31:30.520803777 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/sysinfo.bat' Size: 1389 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 509284 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-30 17:47:26.048360639 +0100 Modify: 2012-11-01 10:49:22.000000000 +0100 Change: 2015-10-26 15:31:30.521803774 +0100 [root at s21 brick2]# stat zimbra/jdk-1.7.0_45/db/bin/* File: `zimbra/jdk-1.7.0_45/db/bin/dblook' Size: 5740 Blocks: 16 IO Block: 4096 regular file Device: 811h/2065d Inode: 2484798182 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.508105065 +0100 Modify: 2012-11-02 12:23:32.000000000 +0100 Change: 2015-10-26 15:31:30.509105067 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/derby_common.bat' Size: 2426 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 2484798185 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.510105069 +0100 Modify: 2012-11-01 10:49:22.000000000 +0100 Change: 2015-10-26 15:31:30.511105071 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/ij' Size: 5876 Blocks: 16 IO Block: 4096 regular file Device: 811h/2065d Inode: 2484798186 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.511105071 +0100 Modify: 2012-11-02 12:23:32.000000000 +0100 Change: 2015-10-26 15:31:30.512105073 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/NetworkServerControl' Size: 5801 Blocks: 16 IO Block: 4096 regular file Device: 811h/2065d Inode: 2484798179 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.507105063 +0100 Modify: 2012-11-02 12:23:32.000000000 +0100 Change: 2015-10-26 15:31:30.508105065 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/setEmbeddedCP.bat' Size: 1278 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 2484798191 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.513105075 +0100 Modify: 2012-11-01 10:49:22.000000000 +0100 Change: 2015-10-26 15:31:30.513105075 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/setNetworkClientCP.bat' Size: 1284 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 2484798194 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.514105077 +0100 Modify: 2012-11-01 10:49:22.000000000 +0100 Change: 2015-10-26 15:31:30.515105079 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/setNetworkServerCP' Size: 1075 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 2484798195 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.515105079 +0100 Modify: 2012-11-01 10:49:22.000000000 +0100 Change: 2015-10-26 15:31:30.516105081 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/startNetworkServer' Size: 5807 Blocks: 16 IO Block: 4096 regular file Device: 811h/2065d Inode: 2484798198 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.517105083 +0100 Modify: 2012-11-02 12:23:32.000000000 +0100 Change: 2015-10-26 15:31:30.518105085 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/startNetworkServer.bat' Size: 1397 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 2484798200 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.519105087 +0100 Modify: 2012-11-01 10:49:22.000000000 +0100 Change: 2015-10-26 15:31:30.519105087 +0100 File: `zimbra/jdk-1.7.0_45/db/bin/sysinfo.bat' Size: 1389 Blocks: 8 IO Block: 4096 regular file Device: 811h/2065d Inode: 2484798206 Links: 2 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-10-26 15:31:30.520105089 +0100 Modify: 2012-11-01 10:49:22.000000000 +0100 Change: 2015-10-26 15:31:30.521105091 +0100 Thanks, Marco On 30/10/2015 08:22, Susant Palai wrote:> Hi Marco, > Can you send the stat of the files from the removed-brick? > > Susant > > ----- Original Message ----- > From: "Marco Lorenzo Crociani" <marcoc at prismatelecomtesting.com> > To: gluster-users at gluster.org > Sent: Tuesday, 27 October, 2015 6:58:51 PM > Subject: [Gluster-users] Missing files after add new bricks and remove old ones - how to restore files > > Hi, > we had a 2 node glusterfs cluster. > we had a 2x2 Distributed-Replicate volume on it. > > It was: > Brick1: s20gfs.ovirt:/gluster/VOL/brick1 > Brick2: s21gfs.ovirt:/gluster/VOL/brick2 > Brick3: s20gfs.ovirt:/gluster/VOL/brick3 > Brick4: s21gfs.ovirt:/gluster/VOL/brick4 > > We added more nodes to the cluster. So I wanted to redistribute the > bricks on the nodes. > I added 2 new bricks to the volume. > > gluster volume add-brick VOL s22gfs.ovirt:/gluster/VOL/brick2 > x`s23gfs.ovirt:/gluster/VOL/brick3 > > then I removed 2 old bricks > gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1 > s21gfs.ovirt:/gluster/VOL/brick2 start > > checked the status > gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1 > s21gfs.ovirt:/gluster/VOL/brick2 status > > when it was completed and I saw data on the new bricks I run: > gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1 > s21gfs.ovirt:/gluster/VOL/brick2 commit > > Result: > some files missing from the volume. Those file are still on the removed > bricks. > > First question. Have I followed a wrong procedure? > > Second question. How to restore those files? I can't re-add back those > brick to the volume because it tolds me: > volume add-brick: failed: /gluster/VOL/brick1 is already part of a volume > If I use the force option I get the old files or they get erased? > Should I rsync from the "unmounted" brick to the "mounted" volume or to > an "umounted" brick that is part of the volume? > > Regards, >-- Marco Crociani Prisma Telecom Testing S.r.l. via Petrocchi, 4 20127 MILANO ITALY Phone: +39 02 26113507 Fax: +39 02 26113597 e-mail: marcoc at prismatelecomtesting.com web: http://www.prismatelecomtesting.com Questa email (e I suoi allegati) costituisce informazione riservata e confidenziale e pu? essere soggetto a legal privilege. Pu? essere utilizzata esclusivamente dai suoi destinatari legittimi. Se avete ricevuto questa email per errore, siete pregati di informarne immediatamente il mittente e quindi cancellarla. A meno che non siate stati a ci? espressamente autorizzati, la diffusione o la riproduzione di questa email o del suo contenuto non sono consentiti. Salvo che questa email sia espressamente qualificata come offerta o accettazione contrattuale, il mittente non intende con questa email dare vita ad un vincolo giuridico e questa email non pu? essere interpretata quale offerta o accettazione che possa dare vita ad un contratto. Qualsiasi opinione manifestata in questa email ? un'opinione personale del mittente, salvo che il mittente dichiari espressamente che si tratti di un'opinione di Prisma Engineering. ******************************************************************************* This e-mail (including any attachments) is private and confidential, and may be privileged. It is for the exclusive use of the intended recipient(s). If you have received this email in error, please inform the sender immediately and then delete this email. Unless you have been given specific permission to do so, please do not distribute or copy this email or its contents. Unless the text of this email specifically states that it is a contractual offer or acceptance, the sender does not intend to create a legal relationship and this email shall not constitute an offer or acceptance which could give rise to a contract. Any views expressed in this communication are those of the individual sender, except where the sender specifically states them to be the views of Prisma Engineering.
Marco Lorenzo Crociani
2015-Nov-05 16:27 UTC
[Gluster-users] Missing files after add new bricks and remove old ones - how to restore files
Any news? glusterfs version is 3.7.5 On 30/10/2015 17:51, Marco Lorenzo Crociani wrote:> Hi Susant, > here the stats: > > [root at s20 brick1]# stat .* * > File: `.' > Size: 78 Blocks: 0 IO Block: 4096 directory > Device: 811h/2065d Inode: 2481712637 Links: 7 > Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-30 15:26:57.565475699 +0100 > Modify: 2015-08-04 12:30:56.604846056 +0200 > Change: 2015-10-27 14:21:12.981420157 +0100 > File: `..' > Size: 50 Blocks: 0 IO Block: 4096 directory > Device: 811h/2065d Inode: 495824 Links: 6 > Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-27 11:27:36.956230769 +0100 > Modify: 2015-08-04 11:25:27.893410342 +0200 > Change: 2015-08-04 11:25:27.893410342 +0200 > File: `.glusterfs' > Size: 8192 Blocks: 24 IO Block: 4096 directory > Device: 811h/2065d Inode: 2481712643 Links: 261 > Access: (0600/drw-------) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-30 15:31:14.988775452 +0100 > Modify: 2015-08-04 12:31:40.803715075 +0200 > Change: 2015-08-04 12:31:40.803715075 +0200 > File: `.trashcan' > Size: 24 Blocks: 0 IO Block: 4096 directory > Device: 811h/2065d Inode: 495865 Links: 3 > Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 18:32:17.369070847 +0100 > Modify: 2015-08-04 11:36:11.357529000 +0200 > Change: 2015-10-26 18:32:17.368070850 +0100 > File: `lost+found' > Size: 6 Blocks: 0 IO Block: 4096 directory > Device: 811h/2065d Inode: 2481712624 Links: 2 > Access: (0700/drwx------) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 18:55:08.274323554 +0100 > Modify: 2014-01-18 21:48:37.000000000 +0100 > Change: 2015-10-26 18:55:08.259323594 +0100 > File: `rh' > Size: 6 Blocks: 0 IO Block: 4096 directory > Device: 811h/2065d Inode: 495961 Links: 2 > Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 14:02:02.294771698 +0100 > Modify: 2015-03-26 13:22:19.000000000 +0100 > Change: 2015-10-26 18:32:17.384070805 +0100 > File: `zimbra' > Size: 4096 Blocks: 8 IO Block: 4096 directory > Device: 811h/2065d Inode: 495969 Links: 50 > Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-30 15:27:44.899346957 +0100 > Modify: 2015-10-26 18:32:17.733069841 +0100 > Change: 2015-10-26 18:32:17.733069841 +0100 > > > > [root at s21 brick2]# stat .* * > File: `.' > Size: 78 Blocks: 0 IO Block: 4096 directory > Device: 811h/2065d Inode: 501309 Links: 7 > Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-30 17:45:23.983929018 +0100 > Modify: 2015-08-04 12:30:56.602392330 +0200 > Change: 2015-10-26 18:32:17.327779305 +0100 > File: `..' > Size: 50 Blocks: 0 IO Block: 4096 directory > Device: 811h/2065d Inode: 2484780736 Links: 6 > Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-30 17:45:20.800922878 +0100 > Modify: 2015-08-04 11:25:27.942732803 +0200 > Change: 2015-08-04 11:25:27.942732803 +0200 > File: `.glusterfs' > Size: 8192 Blocks: 24 IO Block: 4096 directory > Device: 811h/2065d Inode: 501323 Links: 261 > Access: (0600/drw-------) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-08-04 11:36:13.776967886 +0200 > Modify: 2015-08-04 12:31:40.801477366 +0200 > Change: 2015-08-04 12:31:40.801477366 +0200 > File: `.trashcan' > Size: 24 Blocks: 0 IO Block: 4096 directory > Device: 811h/2065d Inode: 2484780773 Links: 3 > Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 18:32:17.324779299 +0100 > Modify: 2015-08-04 11:36:11.357529000 +0200 > Change: 2015-10-26 18:32:17.368779386 +0100 > File: `lost+found' > Size: 6 Blocks: 0 IO Block: 4096 directory > Device: 811h/2065d Inode: 501268 Links: 2 > Access: (0700/drwx------) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 18:32:17.371779392 +0100 > Modify: 2014-01-18 21:48:37.000000000 +0100 > Change: 2015-10-26 18:55:08.260516194 +0100 > File: `rh' > Size: 6 Blocks: 0 IO Block: 4096 directory > Device: 811h/2065d Inode: 2484780842 Links: 2 > Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 18:32:17.386779422 +0100 > Modify: 2015-03-26 13:22:19.000000000 +0100 > Change: 2015-10-26 18:32:17.384779418 +0100 > File: `zimbra' > Size: 4096 Blocks: 8 IO Block: 4096 directory > Device: 811h/2065d Inode: 2484780856 Links: 50 > Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 18:34:10.300004939 +0100 > Modify: 2015-10-26 18:32:17.733780116 +0100 > Change: 2015-10-26 18:32:17.733780116 +0100 > > > > > > > [root at s20 brick1]# stat zimbra/jdk-1.7.0_45/db/bin/* > File: `zimbra/jdk-1.7.0_45/db/bin/dblook' > Size: 5740 Blocks: 16 IO Block: 4096 regular file > Device: 811h/2065d Inode: 509260 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.509803807 +0100 > Modify: 2012-11-02 12:23:32.000000000 +0100 > Change: 2015-10-26 15:31:30.510803804 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/derby_common.bat' > Size: 2426 Blocks: 8 IO Block: 4096 regular file > Device: 811h/2065d Inode: 509263 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.510803804 +0100 > Modify: 2012-11-01 10:49:22.000000000 +0100 > Change: 2015-10-26 15:31:30.511803801 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/ij' > Size: 5876 Blocks: 16 IO Block: 4096 regular file > Device: 811h/2065d Inode: 509264 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.512803798 +0100 > Modify: 2012-11-02 12:23:32.000000000 +0100 > Change: 2015-10-26 15:31:30.512803798 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/NetworkServerControl' > Size: 5801 Blocks: 16 IO Block: 4096 regular file > Device: 811h/2065d Inode: 509257 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.507803811 +0100 > Modify: 2012-11-02 12:23:32.000000000 +0100 > Change: 2015-10-26 15:31:30.508803809 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/setEmbeddedCP.bat' > Size: 1278 Blocks: 8 IO Block: 4096 regular file > Device: 811h/2065d Inode: 509269 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.513803795 +0100 > Modify: 2012-11-01 10:49:22.000000000 +0100 > Change: 2015-10-26 15:31:30.514803792 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/setNetworkClientCP.bat' > Size: 1284 Blocks: 8 IO Block: 4096 regular file > Device: 811h/2065d Inode: 509272 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.514803792 +0100 > Modify: 2012-11-01 10:49:22.000000000 +0100 > Change: 2015-10-26 15:31:30.515803790 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/setNetworkServerCP' > Size: 1075 Blocks: 8 IO Block: 4096 regular file > Device: 811h/2065d Inode: 509273 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.516803788 +0100 > Modify: 2012-11-01 10:49:22.000000000 +0100 > Change: 2015-10-26 15:31:30.517803786 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/startNetworkServer' > Size: 5807 Blocks: 16 IO Block: 4096 regular file > Device: 811h/2065d Inode: 509276 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.517803786 +0100 > Modify: 2012-11-02 12:23:32.000000000 +0100 > Change: 2015-10-26 15:31:30.518803783 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/startNetworkServer.bat' > Size: 1397 Blocks: 8 IO Block: 4096 regular file > Device: 811h/2065d Inode: 509278 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.519803780 +0100 > Modify: 2012-11-01 10:49:22.000000000 +0100 > Change: 2015-10-26 15:31:30.520803777 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/sysinfo.bat' > Size: 1389 Blocks: 8 IO Block: 4096 regular file > Device: 811h/2065d Inode: 509284 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-30 17:47:26.048360639 +0100 > Modify: 2012-11-01 10:49:22.000000000 +0100 > Change: 2015-10-26 15:31:30.521803774 +0100 > > > [root at s21 brick2]# stat zimbra/jdk-1.7.0_45/db/bin/* > File: `zimbra/jdk-1.7.0_45/db/bin/dblook' > Size: 5740 Blocks: 16 IO Block: 4096 regular file > Device: 811h/2065d Inode: 2484798182 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.508105065 +0100 > Modify: 2012-11-02 12:23:32.000000000 +0100 > Change: 2015-10-26 15:31:30.509105067 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/derby_common.bat' > Size: 2426 Blocks: 8 IO Block: 4096 regular file > Device: 811h/2065d Inode: 2484798185 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.510105069 +0100 > Modify: 2012-11-01 10:49:22.000000000 +0100 > Change: 2015-10-26 15:31:30.511105071 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/ij' > Size: 5876 Blocks: 16 IO Block: 4096 regular file > Device: 811h/2065d Inode: 2484798186 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.511105071 +0100 > Modify: 2012-11-02 12:23:32.000000000 +0100 > Change: 2015-10-26 15:31:30.512105073 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/NetworkServerControl' > Size: 5801 Blocks: 16 IO Block: 4096 regular file > Device: 811h/2065d Inode: 2484798179 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.507105063 +0100 > Modify: 2012-11-02 12:23:32.000000000 +0100 > Change: 2015-10-26 15:31:30.508105065 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/setEmbeddedCP.bat' > Size: 1278 Blocks: 8 IO Block: 4096 regular file > Device: 811h/2065d Inode: 2484798191 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.513105075 +0100 > Modify: 2012-11-01 10:49:22.000000000 +0100 > Change: 2015-10-26 15:31:30.513105075 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/setNetworkClientCP.bat' > Size: 1284 Blocks: 8 IO Block: 4096 regular file > Device: 811h/2065d Inode: 2484798194 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.514105077 +0100 > Modify: 2012-11-01 10:49:22.000000000 +0100 > Change: 2015-10-26 15:31:30.515105079 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/setNetworkServerCP' > Size: 1075 Blocks: 8 IO Block: 4096 regular file > Device: 811h/2065d Inode: 2484798195 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.515105079 +0100 > Modify: 2012-11-01 10:49:22.000000000 +0100 > Change: 2015-10-26 15:31:30.516105081 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/startNetworkServer' > Size: 5807 Blocks: 16 IO Block: 4096 regular file > Device: 811h/2065d Inode: 2484798198 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.517105083 +0100 > Modify: 2012-11-02 12:23:32.000000000 +0100 > Change: 2015-10-26 15:31:30.518105085 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/startNetworkServer.bat' > Size: 1397 Blocks: 8 IO Block: 4096 regular file > Device: 811h/2065d Inode: 2484798200 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.519105087 +0100 > Modify: 2012-11-01 10:49:22.000000000 +0100 > Change: 2015-10-26 15:31:30.519105087 +0100 > File: `zimbra/jdk-1.7.0_45/db/bin/sysinfo.bat' > Size: 1389 Blocks: 8 IO Block: 4096 regular file > Device: 811h/2065d Inode: 2484798206 Links: 2 > Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) > Access: 2015-10-26 15:31:30.520105089 +0100 > Modify: 2012-11-01 10:49:22.000000000 +0100 > Change: 2015-10-26 15:31:30.521105091 +0100 > > > Thanks, > > Marco > > > > On 30/10/2015 08:22, Susant Palai wrote: >> Hi Marco, >> Can you send the stat of the files from the removed-brick? >> >> Susant >> >> ----- Original Message ----- >> From: "Marco Lorenzo Crociani" <marcoc at prismatelecomtesting.com> >> To: gluster-users at gluster.org >> Sent: Tuesday, 27 October, 2015 6:58:51 PM >> Subject: [Gluster-users] Missing files after add new bricks and >> remove old ones - how to restore files >> >> Hi, >> we had a 2 node glusterfs cluster. >> we had a 2x2 Distributed-Replicate volume on it. >> >> It was: >> Brick1: s20gfs.ovirt:/gluster/VOL/brick1 >> Brick2: s21gfs.ovirt:/gluster/VOL/brick2 >> Brick3: s20gfs.ovirt:/gluster/VOL/brick3 >> Brick4: s21gfs.ovirt:/gluster/VOL/brick4 >> >> We added more nodes to the cluster. So I wanted to redistribute the >> bricks on the nodes. >> I added 2 new bricks to the volume. >> >> gluster volume add-brick VOL s22gfs.ovirt:/gluster/VOL/brick2 >> x`s23gfs.ovirt:/gluster/VOL/brick3 >> >> then I removed 2 old bricks >> gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1 >> s21gfs.ovirt:/gluster/VOL/brick2 start >> >> checked the status >> gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1 >> s21gfs.ovirt:/gluster/VOL/brick2 status >> >> when it was completed and I saw data on the new bricks I run: >> gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1 >> s21gfs.ovirt:/gluster/VOL/brick2 commit >> >> Result: >> some files missing from the volume. Those file are still on the removed >> bricks. >> >> First question. Have I followed a wrong procedure? >> >> Second question. How to restore those files? I can't re-add back those >> brick to the volume because it tolds me: >> volume add-brick: failed: /gluster/VOL/brick1 is already part of a >> volume >> If I use the force option I get the old files or they get erased? >> Should I rsync from the "unmounted" brick to the "mounted" volume or to >> an "umounted" brick that is part of the volume? >> >> Regards, >> > >-- Marco Crociani Prisma Telecom Testing S.r.l. via Petrocchi, 4 20127 MILANO ITALY Phone: +39 02 26113507 Fax: +39 02 26113597 e-mail: marcoc at prismatelecomtesting.com web: http://www.prismatelecomtesting.com Questa email (e I suoi allegati) costituisce informazione riservata e confidenziale e pu? essere soggetto a legal privilege. Pu? essere utilizzata esclusivamente dai suoi destinatari legittimi. Se avete ricevuto questa email per errore, siete pregati di informarne immediatamente il mittente e quindi cancellarla. A meno che non siate stati a ci? espressamente autorizzati, la diffusione o la riproduzione di questa email o del suo contenuto non sono consentiti. Salvo che questa email sia espressamente qualificata come offerta o accettazione contrattuale, il mittente non intende con questa email dare vita ad un vincolo giuridico e questa email non pu? essere interpretata quale offerta o accettazione che possa dare vita ad un contratto. Qualsiasi opinione manifestata in questa email ? un'opinione personale del mittente, salvo che il mittente dichiari espressamente che si tratti di un'opinione di Prisma Engineering. ******************************************************************************* This e-mail (including any attachments) is private and confidential, and may be privileged. It is for the exclusive use of the intended recipient(s). If you have received this email in error, please inform the sender immediately and then delete this email. Unless you have been given specific permission to do so, please do not distribute or copy this email or its contents. Unless the text of this email specifically states that it is a contractual offer or acceptance, the sender does not intend to create a legal relationship and this email shall not constitute an offer or acceptance which could give rise to a contract. Any views expressed in this communication are those of the individual sender, except where the sender specifically states them to be the views of Prisma Engineering.