[Gluster-users] 3.8.3 Bitrot signature process

Amudhan P
Thu, 22 Sep 2016 05:55:28 -0000

Hi Kotresh,

2280 is a brick process, i have not tried with dist-rep volume?

I have not seen any fd in bitd process in any of the node's and bitd
process usage always 0% CPU and randomly it goes 0.3% CPU.



Thanks,
Amudhan

On Thursday, September 22, 2016, Kotresh Hiremath Ravishankar <
khiremat at redhat.com> wrote:
> Hi Amudhan,
>
> No, bitrot signer is a different process by itself and is not part of
brick process.
> I believe the process 2280 is a brick process ? Did you check with
dist-rep volume?
> Is the same behavior being observed there as well? We need to figure out
why brick
> process is holding that fd for such a long time.
>
> Thanks and Regards,
> Kotresh H R
>
> ----- Original Message -----
>> From: "Amudhan P" <amudhan83 at gmail.com>
>> To: "Kotresh Hiremath Ravishankar" <khiremat at
redhat.com>
>> Sent: Wednesday, September 21, 2016 8:15:33 PM
>> Subject: Re: [Gluster-users] 3.8.3 Bitrot signature process
>>
>> Hi Kotresh,
>>
>> As soon as fd closes from brick1 pid, i can see bitrot signature for
the
>> file in brick.
>>
>> So, it looks like fd opened by brick process to calculate signature.
>>
>> output of the file:
>>
>> -rw-r--r-- 2 root root 250M Sep 21 18:32
>> /media/disk1/brick1/data/G/test59-bs10M-c100.nul
>>
>> getfattr: Removing leading '/' from absolute path names
>> # file: media/disk1/brick1/data/G/test59-bs10M-c100.nul
>> trusted.bit-rot.signature=0x010200000000000000e9474e4cc6
73c0c227a6e807e04aa4ab1f88d3744243950a290869c53daa65df
>> trusted.bit-rot.version=0x020000000000000057d6af3200012a13
>> trusted.ec.config=0x0000080501000200
>> trusted.ec.size=0x000000003e800000
>> trusted.ec.version=0x0000000000001f400000000000001f40
>> trusted.gfid=0x4c091145429448468fffe358482c63e1
>>
>> stat /media/disk1/brick1/data/G/test59-bs10M-c100.nul
>>   File: ?/media/disk1/brick1/data/G/test59-bs10M-c100.nul?
>>   Size: 262144000       Blocks: 512000     IO Block: 4096   regular
file
>> Device: 811h/2065d      Inode: 402653311   Links: 2
>> Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/
root)
>> Access: 2016-09-21 18:34:43.722712751 +0530
>> Modify: 2016-09-21 18:32:41.650712946 +0530
>> Change: 2016-09-21 19:14:41.698708914 +0530
>>  Birth: -
>>
>>
>> In other 2 bricks in same set, still signature is not updated for the
same
>> file.
>>
>>
>> On Wed, Sep 21, 2016 at 6:48 PM, Amudhan P <amudhan83 at
gmail.com> wrote:
>>
>> > Hi Kotresh,
>> >
>> > I am very sure, No read was going on from mount point.
>> >
>> > Again i did same test but after writing data to mount point. I
have
>> > unmounted mount point.
>> >
>> > after 120 seconds i am seeing this file fd entry in brick 1 pid
>> >
>> > getfattr -m. -e hex -d test59-bs10
>> > # file: test59-bs10M-c100.nul
>> > trusted.bit-rot.version=0x020000000000000057bed574000ed534
>> > trusted.ec.config=0x0000080501000200
>> > trusted.ec.size=0x000000003e800000
>> > trusted.ec.version=0x0000000000001f400000000000001f40
>> > trusted.gfid=0x4c091145429448468fffe358482c63e1
>> >
>> >
>> > ls -l /proc/2280/fd
>> > lr-x------ 1 root root 64 Sep 21 13:08 19 ->
/media/disk1/brick1/.
>> > glusterfs/4c/09/4c091145-4294-4846-8fff-e358482c63e1
>> >
>> > Volume is a EC - 4+1
>> >
>> > On Wed, Sep 21, 2016 at 6:17 PM, Kotresh Hiremath Ravishankar <
>> > khiremat at redhat.com> wrote:
>> >
>> >> Hi Amudhan,
>> >>
>> >> If you see the ls output, some process has a fd opened in the
backend.
>> >> That is the reason bitrot is not considering for the signing.
>> >> Could you please observe, after 120 secs of closure of
>> >> "/media/disk2/brick2/.glusterfs/6e/7c/6e7c49e6-094e-4435-
>> >> 85bf-f21f99fd8764"
>> >> the signing happens. If so we need to figure out who holds
this fd for
>> >> such a long time.
>> >> And also we need to figure is this issue specific to EC
volume.
>> >>
>> >> Thanks and Regards,
>> >> Kotresh H R
>> >>
>> >> ----- Original Message -----
>> >> > From: "Amudhan P" <amudhan83 at
gmail.com>
>> >> > To: "Kotresh Hiremath Ravishankar" <khiremat
at redhat.com>
>> >> > Cc: "Gluster Users" <gluster-users at
gluster.org>
>> >> > Sent: Wednesday, September 21, 2016 4:56:40 PM
>> >> > Subject: Re: [Gluster-users] 3.8.3 Bitrot signature
process
>> >> >
>> >> > Hi Kotresh,
>> >> >
>> >> >
>> >> > Writing new file.
>> >> >
>> >> > getfattr -m. -e hex -d /media/disk2/brick2/data/G/
test58-bs10M-c100.nul
>> >> > getfattr: Removing leading '/' from absolute path
names
>> >> > # file: media/disk2/brick2/data/G/test58-bs10M-c100.nul
>> >> >
trusted.bit-rot.version=0x020000000000000057da8b23000b120e
>> >> > trusted.ec.config=0x0000080501000200
>> >> > trusted.ec.size=0x000000003e800000
>> >> > trusted.ec.version=0x0000000000001f400000000000001f40
>> >> > trusted.gfid=0x6e7c49e6094e443585bff21f99fd8764
>> >> >
>> >> >
>> >> > Running ls -l in brick 2 pid
>> >> >
>> >> > ls -l /proc/30162/fd
>> >> >
>> >> > lr-x------ 1 root root 64 Sep 21 16:22 59 ->
>> >> > /media/disk2/brick2/.glusterfs/quanrantine
>> >> > lrwx------ 1 root root 64 Sep 21 16:22 6 ->
>> >> >
/var/lib/glusterd/vols/glsvol1/run/10.1.2.2-media-disk2-brick2.pid
>> >> > lr-x------ 1 root root 64 Sep 21 16:25 60 ->
>> >> > /media/disk2/brick2/.glusterfs/6e/7c/6e7c49e6-094e-4435-
>> >> 85bf-f21f99fd8764
>> >> > lr-x------ 1 root root 64 Sep 21 16:22 61 ->
>> >> > /media/disk2/brick2/.glusterfs/quanrantine
>> >> >
>> >> >
>> >> > find /media/disk2/ -samefile
>> >> > /media/disk2/brick2/.glusterfs/6e/7c/6e7c49e6-094e-4435-
>> >> 85bf-f21f99fd8764
>> >> > /media/disk2/brick2/.glusterfs/6e/7c/6e7c49e6-094e-4435-
>> >> 85bf-f21f99fd8764
>> >> > /media/disk2/brick2/data/G/test58-bs10M-c100.nul
>> >> >
>> >> >
>> >> >
>> >> > On Wed, Sep 21, 2016 at 3:28 PM, Kotresh Hiremath
Ravishankar <
>> >> > khiremat at redhat.com> wrote:
>> >> >
>> >> > > Hi Amudhan,
>> >> > >
>> >> > > Don't grep for the filename, glusterfs maintains
hardlink in
>> >> .glusterfs
>> >> > > directory
>> >> > > for each file. Just check 'ls -l
/proc/<respective brick pid>/fd'
for
>> >> any
>> >> > > fds opened
>> >> > > for a file in .glusterfs and check if it's the
same file.
>> >> > >
>> >> > > Thanks and Regards,
>> >> > > Kotresh H R
>> >> > >
>> >> > > ----- Original Message -----
>> >> > > > From: "Amudhan P" <amudhan83 at
gmail.com>
>> >> > > > To: "Kotresh Hiremath Ravishankar"
<khiremat at redhat.com>
>> >> > > > Cc: "Gluster Users" <gluster-users
at gluster.org>
>> >> > > > Sent: Wednesday, September 21, 2016 1:33:10 PM
>> >> > > > Subject: Re: [Gluster-users] 3.8.3 Bitrot
signature process
>> >> > > >
>> >> > > > Hi Kotresh,
>> >> > > >
>> >> > > > i have used below command to verify any open fd
for file.
>> >> > > >
>> >> > > > "ls -l /proc/*/fd | grep filename".
>> >> > > >
>> >> > > > as soon as write completes there no open
fd's, if there is any
>> >> alternate
>> >> > > > option. please let me know will also try that.
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > Also, below is my scrub status in my test
setup. number of
skipped
>> >> files
>> >> > > > slow reducing day by day. I think files are
skipped due to
bitrot
>> >> > > signature
>> >> > > > process is not completed yet.
>> >> > > >
>> >> > > > where can i see scrub skipped files?
>> >> > > >
>> >> > > >
>> >> > > > Volume name : glsvol1
>> >> > > >
>> >> > > > State of scrub: Active (Idle)
>> >> > > >
>> >> > > > Scrub impact: normal
>> >> > > >
>> >> > > > Scrub frequency: daily
>> >> > > >
>> >> > > > Bitrot error log location:
/var/log/glusterfs/bitd.log
>> >> > > >
>> >> > > > Scrubber error log location:
/var/log/glusterfs/scrub.log
>> >> > > >
>> >> > > >
>> >> > > >
========================================================>> >> >
> >
>> >> > > > Node: localhost
>> >> > > >
>> >> > > > Number of Scrubbed files: 1644
>> >> > > >
>> >> > > > Number of Skipped files: 1001
>> >> > > >
>> >> > > > Last completed scrub time: 2016-09-20 11:59:58
>> >> > > >
>> >> > > > Duration of last scrub (D:M:H:M:S): 0:0:39:26
>> >> > > >
>> >> > > > Error count: 0
>> >> > > >
>> >> > > >
>> >> > > >
========================================================>> >> >
> >
>> >> > > > Node: 10.1.2.3
>> >> > > >
>> >> > > > Number of Scrubbed files: 1644
>> >> > > >
>> >> > > > Number of Skipped files: 1001
>> >> > > >
>> >> > > > Last completed scrub time: 2016-09-20 10:50:00
>> >> > > >
>> >> > > > Duration of last scrub (D:M:H:M:S): 0:0:38:17
>> >> > > >
>> >> > > > Error count: 0
>> >> > > >
>> >> > > >
>> >> > > >
========================================================>> >> >
> >
>> >> > > > Node: 10.1.2.4
>> >> > > >
>> >> > > > Number of Scrubbed files: 981
>> >> > > >
>> >> > > > Number of Skipped files: 1664
>> >> > > >
>> >> > > > Last completed scrub time: 2016-09-20 12:38:01
>> >> > > >
>> >> > > > Duration of last scrub (D:M:H:M:S): 0:0:35:19
>> >> > > >
>> >> > > > Error count: 0
>> >> > > >
>> >> > > >
>> >> > > >
========================================================>> >> >
> >
>> >> > > > Node: 10.1.2.1
>> >> > > >
>> >> > > > Number of Scrubbed files: 1263
>> >> > > >
>> >> > > > Number of Skipped files: 1382
>> >> > > >
>> >> > > > Last completed scrub time: 2016-09-20 11:57:21
>> >> > > >
>> >> > > > Duration of last scrub (D:M:H:M:S): 0:0:37:17
>> >> > > >
>> >> > > > Error count: 0
>> >> > > >
>> >> > > >
>> >> > > >
========================================================>> >> >
> >
>> >> > > > Node: 10.1.2.2
>> >> > > >
>> >> > > > Number of Scrubbed files: 1644
>> >> > > >
>> >> > > > Number of Skipped files: 1001
>> >> > > >
>> >> > > > Last completed scrub time: 2016-09-20 11:59:25
>> >> > > >
>> >> > > > Duration of last scrub (D:M:H:M:S): 0:0:39:18
>> >> > > >
>> >> > > > Error count: 0
>> >> > > >
>> >> > > >
========================================================>> >> >
> >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > Thanks
>> >> > > > Amudhan
>> >> > > >
>> >> > > >
>> >> > > > On Wed, Sep 21, 2016 at 11:45 AM, Kotresh
Hiremath Ravishankar <
>> >> > > > khiremat at redhat.com> wrote:
>> >> > > >
>> >> > > > > Hi Amudhan,
>> >> > > > >
>> >> > > > > I don't think it's the limitation
with read data from the
brick.
>> >> > > > > To limit the usage of CPU, throttling is
done using token
bucket
>> >> > > > > algorithm. The log message showed is
related to it. But even
then
>> >> > > > > I think it should not take 12 minutes for
check-sum
calculation
>> >> unless
>> >> > > > > there is an fd open (might be internal).
Could you please
cross
>> >> verify
>> >> > > > > if there are any fd opened on that file by
looking into
/proc? I
>> >> will
>> >> > > > > also test it out in the mean time and get
back to you.
>> >> > > > >
>> >> > > > > Thanks and Regards,
>> >> > > > > Kotresh H R
>> >> > > > >
>> >> > > > > ----- Original Message -----
>> >> > > > > > From: "Amudhan P"
<amudhan83 at gmail.com>
>> >> > > > > > To: "Kotresh Hiremath
Ravishankar" <khiremat at redhat.com>
>> >> > > > > > Cc: "Gluster Users"
<gluster-users at gluster.org>
>> >> > > > > > Sent: Tuesday, September 20, 2016
3:19:28 PM
>> >> > > > > > Subject: Re: [Gluster-users] 3.8.3
Bitrot signature process
>> >> > > > > >
>> >> > > > > > Hi Kotresh,
>> >> > > > > >
>> >> > > > > > Please correct me if i am wrong, Once
a file write completes
>> >> and as
>> >> > > soon
>> >> > > > > as
>> >> > > > > > closes fds, bitrot waits for 120
seconds and starts hashing
and
>> >> > > update
>> >> > > > > > signature for the file in brick.
>> >> > > > > >
>> >> > > > > > But, what i am feeling that bitrot
takes too much of time to
>> >> complete
>> >> > > > > > hashing.
>> >> > > > > >
>> >> > > > > > below is test result i would like to
share.
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > writing data in below path using dd :
>> >> > > > > >
>> >> > > > > > /mnt/gluster/data/G (mount point)
>> >> > > > > > -rw-r--r-- 1 root root  10M Sep 20
12:19 test53-bs10M-c1.nul
>> >> > > > > > -rw-r--r-- 1 root root 100M Sep 20
12:19
test54-bs10M-c10.nul
>> >> > > > > >
>> >> > > > > > No any other write or read process is
going on.
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > Checking file data in one of the
brick.
>> >> > > > > >
>> >> > > > > > -rw-r--r-- 2 root root 2.5M Sep 20
12:23 test53-bs10M-c1.nul
>> >> > > > > > -rw-r--r-- 2 root root  25M Sep 20
12:23
test54-bs10M-c10.nul
>> >> > > > > >
>> >> > > > > > file's stat and getfattr info
from brick, after write
process
>> >> > > completed.
>> >> > > > > >
>> >> > > > > >
gfstst-node5:/media/disk2/brick2/data/G$ stat
>> >> test53-bs10M-c1.nul
>> >> > > > > >   File: ?test53-bs10M-c1.nul?
>> >> > > > > >   Size: 2621440         Blocks: 5120
IO Block: 4096
>> >>  regular
>> >> > > file
>> >> > > > > > Device: 821h/2081d      Inode:
536874168   Links: 2
>> >> > > > > > Access: (0644/-rw-r--r--)  Uid: (
0/    root)   Gid: (
  0/
>> >> > > root)
>> >> > > > > > Access: 2016-09-20 12:23:28.798886647
+0530
>> >> > > > > > Modify: 2016-09-20 12:23:28.994886646
+0530
>> >> > > > > > Change: 2016-09-20 12:23:28.998886646
+0530
>> >> > > > > >  Birth: -
>> >> > > > > >
>> >> > > > > >
gfstst-node5:/media/disk2/brick2/data/G$ stat
>> >> test54-bs10M-c10.nul
>> >> > > > > >   File: ?test54-bs10M-c10.nul?
>> >> > > > > >   Size: 26214400        Blocks: 51200
IO Block: 4096
>> >>  regular
>> >> > > file
>> >> > > > > > Device: 821h/2081d      Inode:
536874169   Links: 2
>> >> > > > > > Access: (0644/-rw-r--r--)  Uid: (
0/    root)   Gid: (
  0/
>> >> > > root)
>> >> > > > > > Access: 2016-09-20 12:23:42.902886624
+0530
>> >> > > > > > Modify: 2016-09-20 12:23:44.378886622
+0530
>> >> > > > > > Change: 2016-09-20 12:23:44.378886622
+0530
>> >> > > > > >  Birth: -
>> >> > > > > >
>> >> > > > > >
gfstst-node5:/media/disk2/brick2/data/G$ sudo getfattr -m.
-e
>> >> hex -d
>> >> > > > > > test53-bs10M-c1.nul
>> >> > > > > > # file: test53-bs10M-c1.nul
>> >> > > > > >
trusted.bit-rot.version=0x020000000000000057daa7b50002e5b4
>> >> > > > > > trusted.ec.config=0x0000080501000200
>> >> > > > > > trusted.ec.size=0x0000000000a00000
>> >> > > > > >
trusted.ec.version=0x00000000000000500000000000000050
>> >> > > > > >
trusted.gfid=0xe2416bd1aae4403c88f44286273bbe99
>> >> > > > > >
>> >> > > > > >
gfstst-node5:/media/disk2/brick2/data/G$ sudo getfattr -m.
-e
>> >> hex -d
>> >> > > > > > test54-bs10M-c10.nul
>> >> > > > > > # file: test54-bs10M-c10.nul
>> >> > > > > >
trusted.bit-rot.version=0x020000000000000057daa7b50002e5b4
>> >> > > > > > trusted.ec.config=0x0000080501000200
>> >> > > > > > trusted.ec.size=0x0000000006400000
>> >> > > > > >
trusted.ec.version=0x00000000000003200000000000000320
>> >> > > > > >
trusted.gfid=0x54e018dd8c5a4bd79e0317729d8a57c5
>> >> > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > file's stat and getfattr info
from brick, after bitrot
signature
>> >> > > updated.
>> >> > > > > >
>> >> > > > > >
gfstst-node5:/media/disk2/brick2/data/G$ stat
>> >> test53-bs10M-c1.nul
>> >> > > > > >   File: ?test53-bs10M-c1.nul?
>> >> > > > > >   Size: 2621440         Blocks: 5120
IO Block: 4096
>> >>  regular
>> >> > > file
>> >> > > > > > Device: 821h/2081d      Inode:
536874168   Links: 2
>> >> > > > > > Access: (0644/-rw-r--r--)  Uid: (
0/    root)   Gid: (
  0/
>> >> > > root)
>> >> > > > > > Access: 2016-09-20 12:25:31.494886450
+0530
>> >> > > > > > Modify: 2016-09-20 12:23:28.994886646
+0530
>> >> > > > > > Change: 2016-09-20 12:27:00.994886307
+0530
>> >> > > > > >  Birth: -
>> >> > > > > >
>> >> > > > > >
>> >> > > > > >
gfstst-node5:/media/disk2/brick2/data/G$ sudo getfattr -m.
-e
>> >> hex -d
>> >> > > > > > test53-bs10M-c1.nul
>> >> > > > > > # file: test53-bs10M-c1.nul
>> >> > > > > >
trusted.bit-rot.signature=0x0102000000000000006de7493c5c
>> >> > > > >
90f643357c268fbaaf461c1567e0334e4948023ce17268403aa37a
>> >> > > > > >
trusted.bit-rot.version=0x020000000000000057daa7b50002e5b4
>> >> > > > > > trusted.ec.config=0x0000080501000200
>> >> > > > > > trusted.ec.size=0x0000000000a00000
>> >> > > > > >
trusted.ec.version=0x00000000000000500000000000000050
>> >> > > > > >
trusted.gfid=0xe2416bd1aae4403c88f44286273bbe99
>> >> > > > > >
>> >> > > > > >
>> >> > > > > >
gfstst-node5:/media/disk2/brick2/data/G$ stat
>> >> test54-bs10M-c10.nul
>> >> > > > > >   File: ?test54-bs10M-c10.nul?
>> >> > > > > >   Size: 26214400        Blocks: 51200
IO Block: 4096
>> >>  regular
>> >> > > file
>> >> > > > > > Device: 821h/2081d      Inode:
536874169   Links: 2
>> >> > > > > > Access: (0644/-rw-r--r--)  Uid: (
0/    root)   Gid: (
  0/
>> >> > > root)
>> >> > > > > > Access: 2016-09-20 12:25:47.510886425
+0530
>> >> > > > > > Modify: 2016-09-20 12:23:44.378886622
+0530
>> >> > > > > > Change: 2016-09-20 12:38:05.954885243
+0530
>> >> > > > > >  Birth: -
>> >> > > > > >
>> >> > > > > >
>> >> > > > > >
gfstst-node5:/media/disk2/brick2/data/G$ sudo getfattr -m.
-e
>> >> hex -d
>> >> > > > > > test54-bs10M-c10.nul
>> >> > > > > > # file: test54-bs10M-c10.nul
>> >> > > > > >
trusted.bit-rot.signature=0x010200000000000000394c345f0b
>> >> > > > >
0c63ee652627a62eed069244d35c4d5134e4f07d4eabb51afda47e
>> >> > > > > >
trusted.bit-rot.version=0x020000000000000057daa7b50002e5b4
>> >> > > > > > trusted.ec.config=0x0000080501000200
>> >> > > > > > trusted.ec.size=0x0000000006400000
>> >> > > > > >
trusted.ec.version=0x00000000000003200000000000000320
>> >> > > > > >
trusted.gfid=0x54e018dd8c5a4bd79e0317729d8a57c5
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > (Actual time taken for reading file
from brick for md5sum)
>> >> > > > > >
>> >> > > > > >
gfstst-node5:/media/disk2/brick2/data/G$ time md5sum
>> >> > > test53-bs10M-c1.nul
>> >> > > > > > 8354dcaa18a1ecb52d0895bf00888c44
test53-bs10M-c1.nul
>> >> > > > > >
>> >> > > > > > real    0m0.045s
>> >> > > > > > user    0m0.007s
>> >> > > > > > sys     0m0.003s
>> >> > > > > >
>> >> > > > > >
gfstst-node5:/media/disk2/brick2/data/G$ time md5sum
>> >> > > > > test54-bs10M-c10.nul
>> >> > > > > > bed3c0a4a1407f584989b4009e9ce33f
test54-bs10M-c10.nul
>> >> > > > > >
>> >> > > > > > real    0m0.166s
>> >> > > > > > user    0m0.062s
>> >> > > > > > sys     0m0.011s
>> >> > > > > >
>> >> > > > > > As you can see that
'test54-bs10M-c10.nul' file took around
12
>> >> > > minutes to
>> >> > > > > > update bitort signature (pls refer
stat output for the
file).
>> >> > > > > >
>> >> > > > > > what would be the cause for such a
slow read?. Any
limitation
>> >> in read
>> >> > > > > data
>> >> > > > > > from brick?
>> >> > > > > >
>> >> > > > > > Also, i am seeing this line bitd.log,
what does this mean?
>> >> > > > > > [bit-rot.c:1784:br_rate_limit_signer]
0-glsvol1-bit-rot-0:
>> >> [Rate
>> >> > > Limit
>> >> > > > > > Info] "tokens/sec (rate):
131072, maxlimit: 524288
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > Thanks
>> >> > > > > > Amudhan P
>> >> > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > On Mon, Sep 19, 2016 at 1:00 PM,
Kotresh Hiremath
Ravishankar <
>> >> > > > > > khiremat at redhat.com> wrote:
>> >> > > > > >
>> >> > > > > > > Hi Amudhan,
>> >> > > > > > >
>> >> > > > > > > Thanks for testing out the
bitrot feature and sorry for
the
>> >> delayed
>> >> > > > > > > response.
>> >> > > > > > > Please find the answers inline.
>> >> > > > > > >
>> >> > > > > > > Thanks and Regards,
>> >> > > > > > > Kotresh H R
>> >> > > > > > >
>> >> > > > > > > ----- Original Message -----
>> >> > > > > > > > From: "Amudhan P"
<amudhan83 at gmail.com>
>> >> > > > > > > > To: "Gluster
Users" <gluster-users at gluster.org>
>> >> > > > > > > > Sent: Friday, September 16,
2016 4:14:10 PM
>> >> > > > > > > > Subject: Re:
[Gluster-users] 3.8.3 Bitrot signature
process
>> >> > > > > > > >
>> >> > > > > > > > Hi,
>> >> > > > > > > >
>> >> > > > > > > > Can anyone reply to this
mail.
>> >> > > > > > > >
>> >> > > > > > > > On Tue, Sep 13, 2016 at
12:49 PM, Amudhan P <
>> >> > > amudhan83 at gmail.com >
>> >> > > > > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > Hi,
>> >> > > > > > > >
>> >> > > > > > > > I am testing bitrot feature
in Gluster 3.8.3 with
disperse
>> >> EC
>> >> > > volume
>> >> > > > > 4+1.
>> >> > > > > > > >
>> >> > > > > > > > When i write single small
file (< 10MB) after 2 seconds
i
>> >> can see
>> >> > > > > bitrot
>> >> > > > > > > > signature in bricks for the
file, but when i write
multiple
>> >> files
>> >> > > > > with
>> >> > > > > > > > different size ( > 10MB)
it takes long time (> 24hrs)
to see
>> >> > > bitrot
>> >> > > > > > > > signature in all the files.
>> >> > > > > > >
>> >> > > > > > >    The default timeout for
signing to happen is 120
seconds.
>> >> So the
>> >> > > > > > > signing will happen
>> >> > > > > > >   120 secs after the last fd
gets closed on that file. So
if
>> >> the
>> >> > > file
>> >> > > > > is
>> >> > > > > > > being written
>> >> > > > > > >   continuously, it will not be
signed until 120 secs after
>> >> it's
>> >> > > last
>> >> > > > > fd is
>> >> > > > > > > closed.
>> >> > > > > > > >
>> >> > > > > > > > My questions are.
>> >> > > > > > > > 1. I have enabled scrub
schedule as hourly and throttle
as
>> >> > > normal,
>> >> > > > > does
>> >> > > > > > > this
>> >> > > > > > > > make any impact in delaying
bitrot signature?
>> >> > > > > > >       No.
>> >> > > > > > > > 2. other than
"bitd.log" where else i can watch current
>> >> status of
>> >> > > > > bitrot,
>> >> > > > > > > > like number of files added
for signature and file
status?
>> >> > > > > > >      Signature will happen after
120 sec of last fd
closure,
>> >> as
>> >> > > said
>> >> > > > > above.
>> >> > > > > > >      There is not status command
which tracks the
signature
>> >> of the
>> >> > > > > files.
>> >> > > > > > >      But there is bitrot status
command which tracks the
>> >> number of
>> >> > > > > files
>> >> > > > > > >      scrubbed.
>> >> > > > > > >
>> >> > > > > > >      #gluster vol bitrot
<volname> scrub status
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > > > 3. where i can confirm that
all the files in the brick
are
>> >> bitrot
>> >> > > > > signed?
>> >> > > > > > >
>> >> > > > > > >      As said, signing
information of all the files is not
>> >> tracked.
>> >> > > > > > >
>> >> > > > > > > > 4. is there any file read
size limit in bitrot?
>> >> > > > > > >
>> >> > > > > > >      I didn't get. Could you
please elaborate this ?
>> >> > > > > > >
>> >> > > > > > > > 5. options for tuning
bitrot for faster signing of
files?
>> >> > > > > > >
>> >> > > > > > >      Bitrot feature is mainly to
detect silent corruption
>> >> > > (bitflips) of
>> >> > > > > > > files due to long
>> >> > > > > > >      term storage. Hence the
default is 120 sec of last fd
>> >> > > closure, the
>> >> > > > > > > signing happens.
>> >> > > > > > >      But there is a tune able
which can change the default
>> >> 120 sec
>> >> > > but
>> >> > > > > > > that's only for
>> >> > > > > > >      testing purposes and we
don't recommend it.
>> >> > > > > > >
>> >> > > > > > >       gluster vol get master
features.expiry-time
>> >> > > > > > >
>> >> > > > > > >      For testing purposes, you
can change this default and
>> >> test.
>> >> > > > > > > >
>> >> > > > > > > > Thanks
>> >> > > > > > > > Amudhan
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > >
_______________________________________________
>> >> > > > > > > > Gluster-users mailing list
>> >> > > > > > > > Gluster-users at
gluster.org
>> >> > > > > > > >
http://www.gluster.org/mailman/listinfo/gluster-users
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160922/a48c92ea/attachment.html>