Gluster users - Oct 2017 - how to verify bitrot signed file manually?

resending mail.

On Fri, Sep 22, 2017 at 5:30 PM, Amudhan P <amudhan83 at gmail.com> wrote:
> ok, from bitrot code I figured out gluster using sha256 hashing algo.
>
>
> Now coming to the problem, during scrub run in my cluster some of my files
> were marked as bad in few set of nodes.
> I just wanted to confirm bad file. so, I have used "sha256sum"
tool in
> Linux to manually get file hash.
>
> here is the result.
>
> file-1, file-2 marked as bad by scrub and file-3 is healthy.
>
> file-1 sha256 and bitrot signature value matches but still it's been
> marked as bad.
>
> file-2 sha256 and bitrot signature value don't match, could be a victim
of
> bitrot or bitflip.file is still readable without any issue and no errors
> found in the drive.
>
> file-3 sha256 and bitrot signature matches and healthy.
>
>
> file-1 output from
>
> "sha256sum" = "71eada9352b1352aaef0f806d3d561
> 768ce2df905ded1668f665e06eca2d0bd4"
>
>
> "getfattr -m. -e hex -d "
> # file: file-1
> trusted.bit-rot.bad-file=0x3100
> trusted.bit-rot.signature=0x01020000000000000071eada9352
> b1352aaef0f806d3d561768ce2df905ded1668f665e06eca2d0bd4
> trusted.bit-rot.version=0x020000000000000058e4f3b40006793d
> trusted.ec.config=0x0000080a02000200
> trusted.ec.dirty=0x00000000000000000000000000000000
> trusted.ec.size=0x0000000718996701
> trusted.ec.version=0x0000000000038c4c0000000000038c4d
> trusted.gfid=0xf078a24134fe4f9bb953eca8c28dea9a
>
> output scrub log:
> [2017-09-02 13:02:20.311160] A [MSGID: 118023]
[bit-rot-scrub.c:244:bitd_compare_ckum]
> 0-qubevaultdr-bit-rot-0: CORRUPTION DETECTED: Object /file-1 {Brick:
> /media/disk16/brick16 | GFID: f078a241-34fe-4f9b-b953-eca8c28dea9a}
> [2017-09-02 13:02:20.311579] A [MSGID: 118024]
[bit-rot-scrub.c:264:bitd_compare_ckum]
> 0-qubevaultdr-bit-rot-0: Marking /file-1 [GFID:
f078a241-34fe-4f9b-b953-eca8c28dea9a
> | Brick: /media/disk16/brick16] as corrupted..
>
> file-2 output from
>
> "sha256sum" = "c41ef9c81faed4f3e6010ea67984c3
> cfefd842f98ee342939151f9250972dcda"
>
>
> "getfattr -m. -e hex -d "
> # file: file-2
> trusted.bit-rot.bad-file=0x3100
> trusted.bit-rot.signature=0x0102000000000000009162cb17d4
> f0bee676fcb7830c5286d05b8e8940d14f3d117cb90b7b1defc129
> trusted.bit-rot.version=0x020000000000000058e4f3b400019bb2
> trusted.ec.config=0x0000080a02000200
> trusted.ec.dirty=0x00000000000000000000000000000000
> trusted.ec.size=0x00000000403433f6
> trusted.ec.version=0x000000000000201a000000000000201b
> trusted.gfid=0xa50012b0a632477c99232313928d239a
>
> output scrub log:
> [2017-09-02 05:18:14.003156] A [MSGID: 118023]
[bit-rot-scrub.c:244:bitd_compare_ckum]
> 0-qubevaultdr-bit-rot-0: CORRUPTION DETECTED: Object /file-2 {Brick:
> /media/disk13/brick13 | GFID: a50012b0-a632-477c-9923-2313928d239a}
> [2017-09-02 05:18:14.006629] A [MSGID: 118024]
[bit-rot-scrub.c:264:bitd_compare_ckum]
> 0-qubevaultdr-bit-rot-0: Marking /file-2 [GFID:
a50012b0-a632-477c-9923-2313928d239a
> | Brick: /media/disk13/brick13] as corrupted..
>
>
> file-3 output from
>
> "sha256sum" = "a590735b3c8936cc7ca9835128a19c
> 38a3f79c8fd53fddc031a9349b7e273f27"
>
>
> "getfattr -m. -e hex -d "
> # file: file-3
> trusted.bit-rot.signature=0x010200000000000000a590735b3c
> 8936cc7ca9835128a19c38a3f79c8fd53fddc031a9349b7e273f27
> trusted.bit-rot.version=0x020000000000000058e4f3b400019bb2
> trusted.ec.config=0x0000080a02000200
> trusted.ec.dirty=0x00000000000000000000000000000000
> trusted.ec.size=0x000000003530fc96
> trusted.ec.version=0x0000000000001a980000000000001a99
> trusted.gfid=0x10d8920e42cd42cf9448b8bf3941c192
>
>
>
> most of the bitrot bad files are in the set of new nodes and data were
> uploaded using gluster 3.10.1. no drive issues are any kind of error msgs
> in logs.
>
> what could be gone wrong?
>
> regards
> Amudhan
>
> On Thu, Sep 21, 2017 at 1:23 PM, Amudhan P <amudhan83 at gmail.com>
wrote:
>
>> Hi,
>>
>> I have a file in my brick which was signed by bitrot and latter when
>> running scrub it was marked as bad.
>>
>> Now, I want to verify file again manually. just to clarify my doubt
>>
>> how can I do this?
>>
>>
>> regards
>> Amudhan
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170925/9cf659d7/attachment.html>

Hi Amudhan,

Sorry for the late response as I was busy with other things. You are right
bitrot uses sha256 for checksum.
If file-1, file-2 are marked bad, the I/O should be errored out with EIO.
If that is not happening, we need
to look further into it. But what's the file contents of file-1 and file-2
on the replica bricks ? Are they
matching ?

Thanks and Regards,
Kotresh HR

On Mon, Sep 25, 2017 at 4:19 PM, Amudhan P <amudhan83 at gmail.com> wrote:
> resending mail.
>
>
> On Fri, Sep 22, 2017 at 5:30 PM, Amudhan P <amudhan83 at gmail.com>
wrote:
>
>> ok, from bitrot code I figured out gluster using sha256 hashing algo.
>>
>>
>> Now coming to the problem, during scrub run in my cluster some of my
>> files were marked as bad in few set of nodes.
>> I just wanted to confirm bad file. so, I have used
"sha256sum" tool in
>> Linux to manually get file hash.
>>
>> here is the result.
>>
>> file-1, file-2 marked as bad by scrub and file-3 is healthy.
>>
>> file-1 sha256 and bitrot signature value matches but still it's
been
>> marked as bad.
>>
>> file-2 sha256 and bitrot signature value don't match, could be a
victim
>> of bitrot or bitflip.file is still readable without any issue and no
errors
>> found in the drive.
>>
>> file-3 sha256 and bitrot signature matches and healthy.
>>
>>
>> file-1 output from
>>
>> "sha256sum" = "71eada9352b1352aaef0f806d3d56
>> 1768ce2df905ded1668f665e06eca2d0bd4"
>>
>>
>> "getfattr -m. -e hex -d "
>> # file: file-1
>> trusted.bit-rot.bad-file=0x3100
>> trusted.bit-rot.signature=0x01020000000000000071eada9352b135
>> 2aaef0f806d3d561768ce2df905ded1668f665e06eca2d0bd4
>> trusted.bit-rot.version=0x020000000000000058e4f3b40006793d
>> trusted.ec.config=0x0000080a02000200
>> trusted.ec.dirty=0x00000000000000000000000000000000
>> trusted.ec.size=0x0000000718996701
>> trusted.ec.version=0x0000000000038c4c0000000000038c4d
>> trusted.gfid=0xf078a24134fe4f9bb953eca8c28dea9a
>>
>> output scrub log:
>> [2017-09-02 13:02:20.311160] A [MSGID: 118023]
>> [bit-rot-scrub.c:244:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0:
>> CORRUPTION DETECTED: Object /file-1 {Brick: /media/disk16/brick16 |
GFID:
>> f078a241-34fe-4f9b-b953-eca8c28dea9a}
>> [2017-09-02 13:02:20.311579] A [MSGID: 118024]
>> [bit-rot-scrub.c:264:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0:
Marking
>> /file-1 [GFID: f078a241-34fe-4f9b-b953-eca8c28dea9a | Brick:
>> /media/disk16/brick16] as corrupted..
>>
>> file-2 output from
>>
>> "sha256sum" = "c41ef9c81faed4f3e6010ea67984c
>> 3cfefd842f98ee342939151f9250972dcda"
>>
>>
>> "getfattr -m. -e hex -d "
>> # file: file-2
>> trusted.bit-rot.bad-file=0x3100
>> trusted.bit-rot.signature=0x0102000000000000009162cb17d4f0be
>> e676fcb7830c5286d05b8e8940d14f3d117cb90b7b1defc129
>> trusted.bit-rot.version=0x020000000000000058e4f3b400019bb2
>> trusted.ec.config=0x0000080a02000200
>> trusted.ec.dirty=0x00000000000000000000000000000000
>> trusted.ec.size=0x00000000403433f6
>> trusted.ec.version=0x000000000000201a000000000000201b
>> trusted.gfid=0xa50012b0a632477c99232313928d239a
>>
>> output scrub log:
>> [2017-09-02 05:18:14.003156] A [MSGID: 118023]
>> [bit-rot-scrub.c:244:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0:
>> CORRUPTION DETECTED: Object /file-2 {Brick: /media/disk13/brick13 |
GFID:
>> a50012b0-a632-477c-9923-2313928d239a}
>> [2017-09-02 05:18:14.006629] A [MSGID: 118024]
>> [bit-rot-scrub.c:264:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0:
Marking
>> /file-2 [GFID: a50012b0-a632-477c-9923-2313928d239a | Brick:
>> /media/disk13/brick13] as corrupted..
>>
>>
>> file-3 output from
>>
>> "sha256sum" = "a590735b3c8936cc7ca9835128a19
>> c38a3f79c8fd53fddc031a9349b7e273f27"
>>
>>
>> "getfattr -m. -e hex -d "
>> # file: file-3
>> trusted.bit-rot.signature=0x010200000000000000a590735b3c8936
>> cc7ca9835128a19c38a3f79c8fd53fddc031a9349b7e273f27
>> trusted.bit-rot.version=0x020000000000000058e4f3b400019bb2
>> trusted.ec.config=0x0000080a02000200
>> trusted.ec.dirty=0x00000000000000000000000000000000
>> trusted.ec.size=0x000000003530fc96
>> trusted.ec.version=0x0000000000001a980000000000001a99
>> trusted.gfid=0x10d8920e42cd42cf9448b8bf3941c192
>>
>>
>>
>> most of the bitrot bad files are in the set of new nodes and data were
>> uploaded using gluster 3.10.1. no drive issues are any kind of error
msgs
>> in logs.
>>
>> what could be gone wrong?
>>
>> regards
>> Amudhan
>>
>> On Thu, Sep 21, 2017 at 1:23 PM, Amudhan P <amudhan83 at
gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a file in my brick which was signed by bitrot and latter
when
>>> running scrub it was marked as bad.
>>>
>>> Now, I want to verify file again manually. just to clarify my doubt
>>>
>>> how can I do this?
>>>
>>>
>>> regards
>>> Amudhan
>>>
>>
>>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>


-- 
Thanks and Regards,
Kotresh H R
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170929/514e7a0c/attachment.html>

my volume is distributed disperse volume 8+2 EC.
file1 and file2 are different files lying in same brick. I am able to read
the file from mount point without any issue because of EC it reads rest of
the available blocks in other nodes.

my question is "file1" sha256 value matches bitrot signature value but
still, it is also marked as bad by scrubber daemon. why is that?



On Fri, Sep 29, 2017 at 12:52 PM, Kotresh Hiremath Ravishankar <
khiremat at redhat.com> wrote:
> Hi Amudhan,
>
> Sorry for the late response as I was busy with other things. You are right
> bitrot uses sha256 for checksum.
> If file-1, file-2 are marked bad, the I/O should be errored out with EIO.
> If that is not happening, we need
> to look further into it. But what's the file contents of file-1 and
file-2
> on the replica bricks ? Are they
> matching ?
>
> Thanks and Regards,
> Kotresh HR
>
> On Mon, Sep 25, 2017 at 4:19 PM, Amudhan P <amudhan83 at gmail.com>
wrote:
>
>> resending mail.
>>
>>
>> On Fri, Sep 22, 2017 at 5:30 PM, Amudhan P <amudhan83 at
gmail.com> wrote:
>>
>>> ok, from bitrot code I figured out gluster using sha256 hashing
algo.
>>>
>>>
>>> Now coming to the problem, during scrub run in my cluster some of
my
>>> files were marked as bad in few set of nodes.
>>> I just wanted to confirm bad file. so, I have used
"sha256sum" tool in
>>> Linux to manually get file hash.
>>>
>>> here is the result.
>>>
>>> file-1, file-2 marked as bad by scrub and file-3 is healthy.
>>>
>>> file-1 sha256 and bitrot signature value matches but still it's
been
>>> marked as bad.
>>>
>>> file-2 sha256 and bitrot signature value don't match, could be
a victim
>>> of bitrot or bitflip.file is still readable without any issue and
no errors
>>> found in the drive.
>>>
>>> file-3 sha256 and bitrot signature matches and healthy.
>>>
>>>
>>> file-1 output from
>>>
>>> "sha256sum" = "71eada9352b1352aaef0f806d3d56
>>> 1768ce2df905ded1668f665e06eca2d0bd4"
>>>
>>>
>>> "getfattr -m. -e hex -d "
>>> # file: file-1
>>> trusted.bit-rot.bad-file=0x3100
>>> trusted.bit-rot.signature=0x01020000000000000071eada9352b135
>>> 2aaef0f806d3d561768ce2df905ded1668f665e06eca2d0bd4
>>> trusted.bit-rot.version=0x020000000000000058e4f3b40006793d
>>> trusted.ec.config=0x0000080a02000200
>>> trusted.ec.dirty=0x00000000000000000000000000000000
>>> trusted.ec.size=0x0000000718996701
>>> trusted.ec.version=0x0000000000038c4c0000000000038c4d
>>> trusted.gfid=0xf078a24134fe4f9bb953eca8c28dea9a
>>>
>>> output scrub log:
>>> [2017-09-02 13:02:20.311160] A [MSGID: 118023]
>>> [bit-rot-scrub.c:244:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0:
>>> CORRUPTION DETECTED: Object /file-1 {Brick: /media/disk16/brick16 |
GFID:
>>> f078a241-34fe-4f9b-b953-eca8c28dea9a}
>>> [2017-09-02 13:02:20.311579] A [MSGID: 118024]
>>> [bit-rot-scrub.c:264:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0:
>>> Marking /file-1 [GFID: f078a241-34fe-4f9b-b953-eca8c28dea9a |
Brick:
>>> /media/disk16/brick16] as corrupted..
>>>
>>> file-2 output from
>>>
>>> "sha256sum" = "c41ef9c81faed4f3e6010ea67984c
>>> 3cfefd842f98ee342939151f9250972dcda"
>>>
>>>
>>> "getfattr -m. -e hex -d "
>>> # file: file-2
>>> trusted.bit-rot.bad-file=0x3100
>>> trusted.bit-rot.signature=0x0102000000000000009162cb17d4f0be
>>> e676fcb7830c5286d05b8e8940d14f3d117cb90b7b1defc129
>>> trusted.bit-rot.version=0x020000000000000058e4f3b400019bb2
>>> trusted.ec.config=0x0000080a02000200
>>> trusted.ec.dirty=0x00000000000000000000000000000000
>>> trusted.ec.size=0x00000000403433f6
>>> trusted.ec.version=0x000000000000201a000000000000201b
>>> trusted.gfid=0xa50012b0a632477c99232313928d239a
>>>
>>> output scrub log:
>>> [2017-09-02 05:18:14.003156] A [MSGID: 118023]
>>> [bit-rot-scrub.c:244:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0:
>>> CORRUPTION DETECTED: Object /file-2 {Brick: /media/disk13/brick13 |
GFID:
>>> a50012b0-a632-477c-9923-2313928d239a}
>>> [2017-09-02 05:18:14.006629] A [MSGID: 118024]
>>> [bit-rot-scrub.c:264:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0:
>>> Marking /file-2 [GFID: a50012b0-a632-477c-9923-2313928d239a |
Brick:
>>> /media/disk13/brick13] as corrupted..
>>>
>>>
>>> file-3 output from
>>>
>>> "sha256sum" = "a590735b3c8936cc7ca9835128a19
>>> c38a3f79c8fd53fddc031a9349b7e273f27"
>>>
>>>
>>> "getfattr -m. -e hex -d "
>>> # file: file-3
>>> trusted.bit-rot.signature=0x010200000000000000a590735b3c8936
>>> cc7ca9835128a19c38a3f79c8fd53fddc031a9349b7e273f27
>>> trusted.bit-rot.version=0x020000000000000058e4f3b400019bb2
>>> trusted.ec.config=0x0000080a02000200
>>> trusted.ec.dirty=0x00000000000000000000000000000000
>>> trusted.ec.size=0x000000003530fc96
>>> trusted.ec.version=0x0000000000001a980000000000001a99
>>> trusted.gfid=0x10d8920e42cd42cf9448b8bf3941c192
>>>
>>>
>>>
>>> most of the bitrot bad files are in the set of new nodes and data
were
>>> uploaded using gluster 3.10.1. no drive issues are any kind of
error msgs
>>> in logs.
>>>
>>> what could be gone wrong?
>>>
>>> regards
>>> Amudhan
>>>
>>> On Thu, Sep 21, 2017 at 1:23 PM, Amudhan P <amudhan83 at
gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a file in my brick which was signed by bitrot and latter
when
>>>> running scrub it was marked as bad.
>>>>
>>>> Now, I want to verify file again manually. just to clarify my
doubt
>>>>
>>>> how can I do this?
>>>>
>>>>
>>>> regards
>>>> Amudhan
>>>>
>>>
>>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
> --
> Thanks and Regards,
> Kotresh H R
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171003/d144df0c/attachment.html>