thr3ads.net - freebsd stable - Replicable file-system corruption due to fsck/ufs [Apr 2019]

If this information is useful, please help other people find it:
Share via:

Jamie Landeg-Jones

2019-Apr-11 03:47 UTC

Replicable file-system corruption due to fsck/ufs

I've noticed a replicable disk corruption by fsck_ufs/ffs on sparse files.

This is on amd/64 12-stable-20190409, but I first noticed it on
12-stable-20190326.

I didn't notice it on my previous build of 12-stable-20190107, but I
may not have had any relevant sparse files at the time, so I don't know
if that version was affected. 12-release worked OK.

Here is a simplified replicable example. Thinking about it just now, I
suspect it's triggered by files which end in sparseness.

Can anyone else replicate this, or has my machine gone nuts?

Cheers, Jamie

 | root at thompson# l
 | total 12
 | 4 drwxr-x---   2 root  wheel  -   512 11 Apr 04:08 ./
 | 4 drwxr-xr-x  16 root  wheel  - 1,024 11 Apr 04:08 ../
 | 4 -rw-r-----   1 root  wheel  -    43 11 Apr 04:08 typescript
 |
 | root at thompson# dd if=/dev/zero bs=1m count=2048 of=test.img
 | 2048+0 records in
 | 2048+0 records out
 | 2147483648 bytes transferred in 4.127411 secs (520298036 bytes/sec)
 |
 | root at thompson# l
 | total 2097708
 |       4 drwxr-x---   2 root  wheel  -           512 11 Apr 04:08 ./
 |       4 drwxr-xr-x  16 root  wheel  -         1,024 11 Apr 04:08 ../
 | 2097696 -rw-r-----   1 root  wheel  - 2,147,483,648 11 Apr 04:08 test.img
 |       4 -rw-r-----   1 root  wheel  -            43 11 Apr 04:08 typescript
 |
 | root at thompson# mdconfig test.img
 | md1
 |
 | root at thompson# newfs /dev/md1
 | /dev/md1: 2048.0MB (4194304 sectors) block size 32768, fragment size 4096
 |         using 4 cylinder groups of 512.03MB, 16385 blks, 65664 inodes.
 | super-block backups (for fsck_ffs -b #) at:
 |  192, 1048832, 2097472, 3146112
 |
 | root at thompson# md mnt
 | mnt
 |
 | root at thompson# mount /dev/md1 mnt
 |
 | root at thompson# cd mnt/
 | ~/x/mnt ~/x
 |
 | root at thompson# df .
 | Filesystem 1K-blocks Used     Avail Capacity  Mounted on
 | /dev/md1   2,031,132    8 1,868,636     0%    /root/x/mnt
 |
 | root at thompson# l
 | total 12
 | 4 drwxr-xr-x  3 root  wheel     - 512 11 Apr 04:09 ./
 | 4 drwxr-x---  3 root  wheel     - 512 11 Apr 04:09 ../
 | 4 drwxrwxr-x  2 root  operator  - 512 11 Apr 04:09 .snap/
 |
 | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
 | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
 | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
 | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
 | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
 | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
 | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
 | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
 | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
 |
 | root at thompson# l
 | total 652
 |   4 drwxr-xr-x  3 root  wheel     -           512 11 Apr 04:14 ./
 |   4 drwxr-x---  3 root  wheel     -           512 11 Apr 04:09 ../
 |   4 drwxrwxr-x  2 root  operator  -           512 11 Apr 04:09 .snap/
 | 640 -rw-r-----  1 root  wheel     - 9,663,676,605 11 Apr 04:14 test
 |
 | root at thompson# sha256 -r test > sha256.out
 |
 | root at thompson# cd ..
 | ~/x ~/x/mnt
 |
 | root at thompson# umount mnt
 |
 | root at thompson# fsck /dev/md1
 | ** /dev/md1
 | ** Last Mounted on /root/x/mnt
 | ** Phase 1 - Check Blocks and Sizes
 | INODE 4: FILE SIZE 9663676605 BEYOND END OF ALLOCATED FILE, SIZE SHOULD BE
1342210048
 | ADJUST? [yn] y
 |
 | ** Phase 2 - Check Pathnames
 | ** Phase 3 - Check Connectivity
 | ** Phase 4 - Check Reference Counts
 | ** Phase 5 - Check Cyl groups
 | 4 files, 163 used, 507620 free (20 frags, 63450 blocks, 0.0% fragmentation)
 |
 | ***** FILE SYSTEM IS CLEAN *****
 |
 | ***** FILE SYSTEM WAS MODIFIED *****
 |
 | root at thompson# fsck /dev/md1
 | ** /dev/md1
 | ** Last Mounted on /root/x/mnt
 | ** Phase 1 - Check Blocks and Sizes
 | PARTIALLY TRUNCATED INODE I=4
 | SALVAGE? [yn] y
 |
 | INCORRECT BLOCK COUNT I=4 (1280 should be 256)
 | CORRECT? [yn] y
 |
 | INODE 4: FILE SIZE 1342210048 BEYOND END OF ALLOCATED FILE, SIZE SHOULD BE
268468224
 | ADJUST? [yn] y
 |
 | ** Phase 2 - Check Pathnames
 | ** Phase 3 - Check Connectivity
 | ** Phase 4 - Check Reference Counts
 | ** Phase 5 - Check Cyl groups
 | FREE BLK COUNT(S) WRONG IN SUPERBLK
 | SALVAGE? [yn] y
 |
 | SUMMARY INFORMATION BAD
 | SALVAGE? [yn] y
 |
 | BLK(S) MISSING IN BIT MAPS
 | SALVAGE? [yn] y
 |
 | 4 files, 35 used, 507748 free (20 frags, 63466 blocks, 0.0% fragmentation)
 |
 | ***** FILE SYSTEM IS CLEAN *****
 |
 | ***** FILE SYSTEM WAS MODIFIED *****
 |
 | root at thompson# fsck /dev/md1
 | ** /dev/md1
 | ** Last Mounted on /root/x/mnt
 | ** Phase 1 - Check Blocks and Sizes
 | PARTIALLY TRUNCATED INODE I=4
 | SALVAGE? [yn] y
 |
 | INCORRECT BLOCK COUNT I=4 (256 should be 128)
 | CORRECT? [yn] y
 |
 | INODE 4: FILE SIZE 268468224 BEYOND END OF ALLOCATED FILE, SIZE SHOULD BE
134610944
 | ADJUST? [yn] y
 |
 | ** Phase 2 - Check Pathnames
 | ** Phase 3 - Check Connectivity
 | ** Phase 4 - Check Reference Counts
 | ** Phase 5 - Check Cyl groups
 | FREE BLK COUNT(S) WRONG IN SUPERBLK
 | SALVAGE? [yn] y
 |
 | SUMMARY INFORMATION BAD
 | SALVAGE? [yn] y
 |
 | BLK(S) MISSING IN BIT MAPS
 | SALVAGE? [yn] y
 |
 | 4 files, 19 used, 507764 free (20 frags, 63468 blocks, 0.0% fragmentation)
 |
 | ***** FILE SYSTEM IS CLEAN *****
 |
 | ***** FILE SYSTEM WAS MODIFIED *****
 |
 | root at thompson# fsck /dev/md1
 | ** /dev/md1
 | ** Last Mounted on /root/x/mnt
 | ** Phase 1 - Check Blocks and Sizes
 | ** Phase 2 - Check Pathnames
 | ** Phase 3 - Check Connectivity
 | ** Phase 4 - Check Reference Counts
 | ** Phase 5 - Check Cyl groups
 | 4 files, 19 used, 507764 free (20 frags, 63468 blocks, 0.0% fragmentation)
 |
 | ***** FILE SYSTEM IS CLEAN *****
 |
 | root at thompson# fsck /dev/md1
 | ** /dev/md1
 | ** Last Mounted on /root/x/mnt
 | ** Phase 1 - Check Blocks and Sizes
 | ** Phase 2 - Check Pathnames
 | ** Phase 3 - Check Connectivity
 | ** Phase 4 - Check Reference Counts
 | ** Phase 5 - Check Cyl groups
 | 4 files, 19 used, 507764 free (20 frags, 63468 blocks, 0.0% fragmentation)
 |
 | ***** FILE SYSTEM IS CLEAN *****
 |
 | root at thompson# mount /dev/md1 mnt
 |
 | root at thompson# cd mnt/
 | ~/x/mnt ~/x
 |
 | root at thompson# l
 | total 80
 |  4 drwxr-xr-x  3 root  wheel     -         512 11 Apr 04:14 ./
 |  4 drwxr-x---  3 root  wheel     -         512 11 Apr 04:09 ../
 |  4 drwxrwxr-x  2 root  operator  -         512 11 Apr 04:09 .snap/
 |  4 -rw-r-----  1 root  wheel     -          70 11 Apr 04:14 sha256.out
 | 64 -rw-r-----  1 root  wheel     - 134,610,944 11 Apr 04:14 test
 |
 | root at thompson# cat sha256.out
 | 76b042e7fbb3ed1914cf600a0b5ed8e10b8d917a006dbbff774a996c9bbce941 test
 |
 | root at thompson# sha256 -r test
 | 6b1a548d057244632b5d2897f8c17177236c262c6af54cc0a9db5ddc8285fbd4 test

Peter Holm

2019-Apr-11 04:36 UTC

head link

Replicable file-system corruption due to fsck/ufs

On Thu, Apr 11, 2019 at 04:47:43AM +0100, Jamie Landeg-Jones
wrote:> I've noticed a replicable disk corruption by fsck_ufs/ffs on sparse
files.
> 
> This is on amd/64 12-stable-20190409, but I first noticed it on
> 12-stable-20190326.
> 
> I didn't notice it on my previous build of 12-stable-20190107, but I
> may not have had any relevant sparse files at the time, so I don't know
> if that version was affected. 12-release worked OK.
> 
> Here is a simplified replicable example. Thinking about it just now, I
> suspect it's triggered by files which end in sparseness.
> 
> Can anyone else replicate this, or has my machine gone nuts?
> 
> Cheers, Jamie
> 
>  | root at thompson# l
>  | total 12
>  | 4 drwxr-x---   2 root  wheel  -   512 11 Apr 04:08 ./
>  | 4 drwxr-xr-x  16 root  wheel  - 1,024 11 Apr 04:08 ../
>  | 4 -rw-r-----   1 root  wheel  -    43 11 Apr 04:08 typescript
>  |
>  | root at thompson# dd if=/dev/zero bs=1m count=2048 of=test.img
>  | 2048+0 records in
>  | 2048+0 records out
>  | 2147483648 bytes transferred in 4.127411 secs (520298036 bytes/sec)
>  |
>  | root at thompson# l
>  | total 2097708
>  |       4 drwxr-x---   2 root  wheel  -           512 11 Apr 04:08 ./
>  |       4 drwxr-xr-x  16 root  wheel  -         1,024 11 Apr 04:08 ../
>  | 2097696 -rw-r-----   1 root  wheel  - 2,147,483,648 11 Apr 04:08
test.img
>  |       4 -rw-r-----   1 root  wheel  -            43 11 Apr 04:08
typescript
>  |
>  | root at thompson# mdconfig test.img
>  | md1
>  |
>  | root at thompson# newfs /dev/md1
>  | /dev/md1: 2048.0MB (4194304 sectors) block size 32768, fragment size
4096
>  |         using 4 cylinder groups of 512.03MB, 16385 blks, 65664 inodes.
>  | super-block backups (for fsck_ffs -b #) at:
>  |  192, 1048832, 2097472, 3146112
>  |
>  | root at thompson# md mnt
>  | mnt
>  |
>  | root at thompson# mount /dev/md1 mnt
>  |
>  | root at thompson# cd mnt/
>  | ~/x/mnt ~/x
>  |
>  | root at thompson# df .
>  | Filesystem 1K-blocks Used     Avail Capacity  Mounted on
>  | /dev/md1   2,031,132    8 1,868,636     0%    /root/x/mnt
>  |
>  | root at thompson# l
>  | total 12
>  | 4 drwxr-xr-x  3 root  wheel     - 512 11 Apr 04:09 ./
>  | 4 drwxr-x---  3 root  wheel     - 512 11 Apr 04:09 ../
>  | 4 drwxrwxr-x  2 root  operator  - 512 11 Apr 04:09 .snap/
>  |
>  | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
>  | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
>  | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
>  | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
>  | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
>  | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
>  | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
>  | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
>  | root at thompson# echo "testing 1...2...3..." >> test ;
truncate -s +1g test
>  |
>  | root at thompson# l
>  | total 652
>  |   4 drwxr-xr-x  3 root  wheel     -           512 11 Apr 04:14 ./
>  |   4 drwxr-x---  3 root  wheel     -           512 11 Apr 04:09 ../
>  |   4 drwxrwxr-x  2 root  operator  -           512 11 Apr 04:09 .snap/
>  | 640 -rw-r-----  1 root  wheel     - 9,663,676,605 11 Apr 04:14 test
>  |
>  | root at thompson# sha256 -r test > sha256.out
>  |
>  | root at thompson# cd ..
>  | ~/x ~/x/mnt
>  |
>  | root at thompson# umount mnt
>  |
>  | root at thompson# fsck /dev/md1
>  | ** /dev/md1
>  | ** Last Mounted on /root/x/mnt
>  | ** Phase 1 - Check Blocks and Sizes
>  | INODE 4: FILE SIZE 9663676605 BEYOND END OF ALLOCATED FILE, SIZE SHOULD
BE 1342210048
>  | ADJUST? [yn] y
>  |
>  | ** Phase 2 - Check Pathnames
>  | ** Phase 3 - Check Connectivity
>  | ** Phase 4 - Check Reference Counts
>  | ** Phase 5 - Check Cyl groups
>  | 4 files, 163 used, 507620 free (20 frags, 63450 blocks, 0.0%
fragmentation)
>  |
>  | ***** FILE SYSTEM IS CLEAN *****
>  |
>  | ***** FILE SYSTEM WAS MODIFIED *****
>  |
>  | root at thompson# fsck /dev/md1
>  | ** /dev/md1
>  | ** Last Mounted on /root/x/mnt
>  | ** Phase 1 - Check Blocks and Sizes
>  | PARTIALLY TRUNCATED INODE I=4
>  | SALVAGE? [yn] y
>  |
>  | INCORRECT BLOCK COUNT I=4 (1280 should be 256)
>  | CORRECT? [yn] y
>  |
>  | INODE 4: FILE SIZE 1342210048 BEYOND END OF ALLOCATED FILE, SIZE SHOULD
BE 268468224
>  | ADJUST? [yn] y
>  |
>  | ** Phase 2 - Check Pathnames
>  | ** Phase 3 - Check Connectivity
>  | ** Phase 4 - Check Reference Counts
>  | ** Phase 5 - Check Cyl groups
>  | FREE BLK COUNT(S) WRONG IN SUPERBLK
>  | SALVAGE? [yn] y
>  |
>  | SUMMARY INFORMATION BAD
>  | SALVAGE? [yn] y
>  |
>  | BLK(S) MISSING IN BIT MAPS
>  | SALVAGE? [yn] y
>  |
>  | 4 files, 35 used, 507748 free (20 frags, 63466 blocks, 0.0%
fragmentation)
>  |
>  | ***** FILE SYSTEM IS CLEAN *****
>  |
>  | ***** FILE SYSTEM WAS MODIFIED *****
>  |
>  | root at thompson# fsck /dev/md1
>  | ** /dev/md1
>  | ** Last Mounted on /root/x/mnt
>  | ** Phase 1 - Check Blocks and Sizes
>  | PARTIALLY TRUNCATED INODE I=4
>  | SALVAGE? [yn] y
>  |
>  | INCORRECT BLOCK COUNT I=4 (256 should be 128)
>  | CORRECT? [yn] y
>  |
>  | INODE 4: FILE SIZE 268468224 BEYOND END OF ALLOCATED FILE, SIZE SHOULD
BE 134610944
>  | ADJUST? [yn] y
>  |
>  | ** Phase 2 - Check Pathnames
>  | ** Phase 3 - Check Connectivity
>  | ** Phase 4 - Check Reference Counts
>  | ** Phase 5 - Check Cyl groups
>  | FREE BLK COUNT(S) WRONG IN SUPERBLK
>  | SALVAGE? [yn] y
>  |
>  | SUMMARY INFORMATION BAD
>  | SALVAGE? [yn] y
>  |
>  | BLK(S) MISSING IN BIT MAPS
>  | SALVAGE? [yn] y
>  |
>  | 4 files, 19 used, 507764 free (20 frags, 63468 blocks, 0.0%
fragmentation)
>  |
>  | ***** FILE SYSTEM IS CLEAN *****
>  |
>  | ***** FILE SYSTEM WAS MODIFIED *****
>  |
>  | root at thompson# fsck /dev/md1
>  | ** /dev/md1
>  | ** Last Mounted on /root/x/mnt
>  | ** Phase 1 - Check Blocks and Sizes
>  | ** Phase 2 - Check Pathnames
>  | ** Phase 3 - Check Connectivity
>  | ** Phase 4 - Check Reference Counts
>  | ** Phase 5 - Check Cyl groups
>  | 4 files, 19 used, 507764 free (20 frags, 63468 blocks, 0.0%
fragmentation)
>  |
>  | ***** FILE SYSTEM IS CLEAN *****
>  |
>  | root at thompson# fsck /dev/md1
>  | ** /dev/md1
>  | ** Last Mounted on /root/x/mnt
>  | ** Phase 1 - Check Blocks and Sizes
>  | ** Phase 2 - Check Pathnames
>  | ** Phase 3 - Check Connectivity
>  | ** Phase 4 - Check Reference Counts
>  | ** Phase 5 - Check Cyl groups
>  | 4 files, 19 used, 507764 free (20 frags, 63468 blocks, 0.0%
fragmentation)
>  |
>  | ***** FILE SYSTEM IS CLEAN *****
>  |
>  | root at thompson# mount /dev/md1 mnt
>  |
>  | root at thompson# cd mnt/
>  | ~/x/mnt ~/x
>  |
>  | root at thompson# l
>  | total 80
>  |  4 drwxr-xr-x  3 root  wheel     -         512 11 Apr 04:14 ./
>  |  4 drwxr-x---  3 root  wheel     -         512 11 Apr 04:09 ../
>  |  4 drwxrwxr-x  2 root  operator  -         512 11 Apr 04:09 .snap/
>  |  4 -rw-r-----  1 root  wheel     -          70 11 Apr 04:14 sha256.out
>  | 64 -rw-r-----  1 root  wheel     - 134,610,944 11 Apr 04:14 test
>  |
>  | root at thompson# cat sha256.out
>  | 76b042e7fbb3ed1914cf600a0b5ed8e10b8d917a006dbbff774a996c9bbce941 test
>  |
>  | root at thompson# sha256 -r test
>  | 6b1a548d057244632b5d2897f8c17177236c262c6af54cc0a9db5ddc8285fbd4 test
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"
I see this even with a single truncate on HEAD.

$ ./truncate10.sh
96 -rw-r--r--  1 root  wheel  1073741824 11 apr. 06:33 test
** /dev/md10a
** Last Mounted on /mnt
** Phase 1 - Check Blocks and Sizes
INODE 3: FILE SIZE 1073741824 BEYOND END OF ALLOCATED FILE, SIZE SHOULD BE
268435456
ADJUST? yes


-- 
Peter

freebsd stable - Apr 2019 - Replicable file-system corruption due to fsck/ufs

Replicable file-system corruption due to fsck/ufs

Replicable file-system corruption due to fsck/ufs