likun
2017-Jan-03 03:33 UTC
[Gluster-users] read error: Input/output error but not in brain split stat
Hello, everyone. Today, one of our glusterfs consumer reported an error: java.io.FileNotFoundException: /mnt/glusterfs/gps/data/2017-01/02/4.txt (Input/output error). Then I tried to access this file: # cd /mnt/glusterfs/gps/data/2017-01/02 # more 4.txt # cat 4.txt cat: read error: Input/output error I supposed there was something wrong with glusterfs, so I logined into glusterfs system, found nothing wrong, the output of 'gluster volume status|info' and 'gluster volume heal gvol0 info' command appended at the end. So, question is what's wrong? Then I logined into the bricks that hold this file, the size of the file 4.txt was different: core at ab09 /export/gluster/sdb3/vol/gps/data/2017-01/02 $ ls -l 4.txt -rw-r--r--. 2 root root 7490 Jan 2 00:00 4.txt core at ac08 /export/gluster/sdb3/vol/gps/data/2017-01/02 $ ls -l 4.txt -rw-r--r--. 2 root root 9046 Jan 2 00:00 4.txt core at ad08 /export/gluster/sdb3/vol/gps/data/2017-01/02 $ ls -l 4.txt -rw-r--r--. 2 root root 9046 Jan 2 00:00 4.txt And the getfattr output: core at ab09 /export/gluster/sdb3/vol/gps/data/2017-01/02 # getfattr -m . -d -e hex 4.txt # file: 4.txt security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f7 43a733000 trusted.afr.gvol0-client-51=0x000000000000000000000000 trusted.afr.gvol0-client-52=0x000000000000000000000000 trusted.afr.gvol0-client-53=0x000000000000000000000000 trusted.bit-rot.version=0x02000000000000005850b03d00072c12 trusted.gfid=0xa61d7b2fd53f4eb9be7a656b0915dbea trusted.glusterfs.quota.363d3c2d-5ccb-497e-9ecc-6eaf5f435026.contri.1=0x0000 000000001e000000000000000001 trusted.pgfid.363d3c2d-5ccb-497e-9ecc-6eaf5f435026=0x00000001 core at ac08 /export/gluster/sdb3/vol/gps/data/2017-01/02 # getfattr -m . -d -e hex 4.txt # file: 4.txt security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f7 43a733000 trusted.afr.gvol0-client-51=0x000000000000000000000000 trusted.afr.gvol0-client-52=0x000000000000000000000000 trusted.afr.gvol0-client-53=0x000000000000000000000000 trusted.bit-rot.version=0x02000000000000005850b03d0007711e trusted.gfid=0xa61d7b2fd53f4eb9be7a656b0915dbea trusted.glusterfs.quota.363d3c2d-5ccb-497e-9ecc-6eaf5f435026.contri.1=0x0000 0000000024000000000000000001 trusted.pgfid.363d3c2d-5ccb-497e-9ecc-6eaf5f435026=0x00000001 core at ad08 /export/gluster/sdb3/vol/gps/data/2017-01/02 # getfattr -m . -d -e hex 4.txt # file: 4.txt security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f7 43a733000 trusted.afr.gvol0-client-51=0x000000000000000000000000 trusted.afr.gvol0-client-52=0x000000000000000000000000 trusted.afr.gvol0-client-53=0x000000000000000000000000 trusted.bit-rot.version=0x02000000000000005861dcf900054117 trusted.gfid=0xa61d7b2fd53f4eb9be7a656b0915dbea trusted.glusterfs.quota.363d3c2d-5ccb-497e-9ecc-6eaf5f435026.contri.1=0x0000 0000000024000000000000000001 trusted.pgfid.363d3c2d-5ccb-497e-9ecc-6eaf5f435026=0x00000001 Anyone know what was wrong with here ? Brain split ? but gluster volume heal info didn't report it . Is this a bug or something? What caused this? Although a "gluster volume heal gvol0 full" command handled this, synced this file, the impact to glusterfs consumer was real, and would happen again in the near future, I supposed. How can this be avoided, or recommend some preactive action to handle this ? I'm running official gluster-centos:gluster3u8_centos7 image (version 3.8.5) in container with kubernetes. Likun Appendix: # gluster volume status Status of volume: gvol0 Gluster process TCP Port RDMA Port Online Pid ---------------------------------------------------------------------------- -- Brick 10.3.3.9:/mnt/brick1/vol 49152 0 Y 318 Brick 10.3.3.19:/mnt/brick1/vol 49152 0 Y 316 Brick 10.3.3.29:/mnt/brick1/vol 49152 0 Y 316 Brick 10.3.3.10:/mnt/brick1/vol 49152 0 Y 316 Brick 10.3.3.20:/mnt/brick1/vol 49152 0 Y 318 Brick 10.3.3.30:/mnt/brick1/vol 49152 0 Y 339 Brick 10.3.3.9:/mnt/brick2/vol 49153 0 Y 337 Brick 10.3.3.19:/mnt/brick2/vol 49153 0 Y 335 Brick 10.3.3.29:/mnt/brick2/vol 49153 0 Y 335 Brick 10.3.3.11:/mnt/brick1/vol 49152 0 Y 318 Brick 10.3.3.21:/mnt/brick1/vol 49152 0 Y 316 Brick 10.3.3.39:/mnt/brick1/vol 49152 0 Y 744 Brick 10.3.3.12:/mnt/brick1/vol 49152 0 Y 316 Brick 10.3.3.22:/mnt/brick1/vol 49152 0 Y 287 Brick 10.3.3.40:/mnt/brick1/vol 49152 0 Y 333 Brick 10.3.3.13:/mnt/brick1/vol 49152 0 Y 318 Brick 10.3.3.31:/mnt/brick1/vol 49152 0 Y 308 Brick 10.3.3.41:/mnt/brick1/vol 49152 0 Y 331 Brick 10.3.3.23:/mnt/brick1/vol 49152 0 Y 322 Brick 10.3.3.32:/mnt/brick1/vol 49152 0 Y 308 Brick 10.3.3.42:/mnt/brick1/vol 49152 0 Y 324 Brick 10.3.3.33:/mnt/brick1/vol 49152 0 Y 308 Brick 10.3.3.43:/mnt/brick1/vol 49152 0 Y 341 Brick 10.3.3.10:/mnt/brick2/vol 49153 0 Y 335 Brick 10.3.3.20:/mnt/brick2/vol 49153 0 Y 337 Brick 10.3.3.30:/mnt/brick2/vol 49153 0 Y 355 Brick 10.3.3.40:/mnt/brick2/vol 49153 0 Y 339 Brick 10.3.3.11:/mnt/brick2/vol 49153 0 Y 337 Brick 10.3.3.21:/mnt/brick2/vol 49153 0 Y 335 Brick 10.3.3.39:/mnt/brick2/vol 49153 0 Y 763 Brick 10.3.3.12:/mnt/brick2/vol 49153 0 Y 335 Brick 10.3.3.22:/mnt/brick2/vol 49153 0 Y 306 Brick 10.3.3.31:/mnt/brick2/vol 49153 0 Y 327 Brick 10.3.3.13:/mnt/brick2/vol 49153 0 Y 337 Brick 10.3.3.32:/mnt/brick2/vol 49153 0 Y 327 Brick 10.3.3.41:/mnt/brick2/vol 49153 0 Y 336 Brick 10.3.3.23:/mnt/brick2/vol 49153 0 Y 330 Brick 10.3.3.33:/mnt/brick2/vol 49153 0 Y 327 Brick 10.3.3.42:/mnt/brick2/vol 49153 0 Y 343 Brick 10.3.3.9:/mnt/brick3/vol 49154 0 Y 356 Brick 10.3.3.19:/mnt/brick3/vol 49154 0 Y 354 Brick 10.3.3.29:/mnt/brick3/vol 49154 0 Y 354 Brick 10.3.3.10:/mnt/brick3/vol 49154 0 Y 354 Brick 10.3.3.20:/mnt/brick3/vol 49154 0 Y 356 Brick 10.3.3.43:/mnt/brick2/vol 49153 0 Y 336 Brick 10.3.3.11:/mnt/brick3/vol 49154 0 Y 356 Brick 10.3.3.21:/mnt/brick3/vol 49154 0 Y 354 Brick 10.3.3.39:/mnt/brick3/vol 49154 0 Y 782 Brick 10.3.3.12:/mnt/brick3/vol 49154 0 Y 354 Brick 10.3.3.30:/mnt/brick3/vol 49154 0 Y 363 Brick 10.3.3.40:/mnt/brick3/vol 49154 0 Y 345 Brick 10.3.3.22:/mnt/brick3/vol 49154 0 Y 325 Brick 10.3.3.31:/mnt/brick3/vol 49154 0 Y 346 Brick 10.3.3.41:/mnt/brick3/vol 49154 0 Y 343 Brick 10.3.3.13:/mnt/brick3/vol 49154 0 Y 356 Brick 10.3.3.32:/mnt/brick3/vol 49154 0 Y 346 Brick 10.3.3.42:/mnt/brick3/vol 49154 0 Y 349 Brick 10.3.3.23:/mnt/brick3/vol 49154 0 Y 338 Brick 10.3.3.33:/mnt/brick3/vol 49154 0 Y 346 Brick 10.3.3.43:/mnt/brick3/vol 49154 0 Y 349 Brick 10.3.3.9:/mnt/brick4/vol 49155 0 Y 375 Brick 10.3.3.19:/mnt/brick4/vol 49155 0 Y 373 Brick 10.3.3.29:/mnt/brick4/vol 49155 0 Y 373 Brick 10.3.3.10:/mnt/brick4/vol 49155 0 Y 373 Brick 10.3.3.20:/mnt/brick4/vol 49155 0 Y 375 Brick 10.3.3.30:/mnt/brick4/vol 49155 0 Y 345 Brick 10.3.3.11:/mnt/brick4/vol 49155 0 Y 375 Brick 10.3.3.21:/mnt/brick4/vol 49155 0 Y 373 Brick 10.3.3.39:/mnt/brick4/vol 49155 0 Y 801 Brick 10.3.3.12:/mnt/brick4/vol 49155 0 Y 373 Brick 10.3.3.22:/mnt/brick4/vol 49155 0 Y 344 Brick 10.3.3.40:/mnt/brick4/vol 49155 0 Y 351 Brick 10.3.3.13:/mnt/brick4/vol 49155 0 Y 375 Brick 10.3.3.31:/mnt/brick4/vol 49155 0 Y 365 Brick 10.3.3.41:/mnt/brick4/vol 49155 0 Y 355 Brick 10.3.3.10:/mnt/brick5/vol 49156 0 Y 392 Brick 10.3.3.33:/mnt/brick4/vol 49155 0 Y 365 Brick 10.3.3.43:/mnt/brick4/vol 49155 0 Y 331 Brick 10.3.3.23:/mnt/brick4/vol 49155 0 Y 347 Brick 10.3.3.32:/mnt/brick4/vol 49155 0 Y 365 Brick 10.3.3.42:/mnt/brick4/vol 49155 0 Y 355 Brick 10.3.3.9:/mnt/brick5/vol 49156 0 Y 394 Brick 10.3.3.19:/mnt/brick5/vol 49156 0 Y 392 Brick 10.3.3.29:/mnt/brick5/vol 49156 0 Y 392 Brick 10.3.3.20:/mnt/brick5/vol 49156 0 Y 394 Brick 10.3.3.30:/mnt/brick5/vol 49156 0 Y 332 Brick 10.3.3.40:/mnt/brick5/vol 49156 0 Y 357 Brick 10.3.3.11:/mnt/brick5/vol 49156 0 Y 394 Brick 10.3.3.21:/mnt/brick5/vol 49156 0 Y 392 Brick 10.3.3.39:/mnt/brick5/vol 49156 0 Y 820 Brick 10.3.3.12:/mnt/brick5/vol 49156 0 Y 392 Brick 10.3.3.22:/mnt/brick5/vol 49156 0 Y 363 Brick 10.3.3.31:/mnt/brick5/vol 49156 0 Y 384 Brick 10.3.3.13:/mnt/brick5/vol 49156 0 Y 394 Brick 10.3.3.23:/mnt/brick5/vol 49156 0 Y 352 Brick 10.3.3.41:/mnt/brick5/vol 49156 0 Y 349 Brick 10.3.3.10:/mnt/brick6/vol 49157 0 Y 411 Brick 10.3.3.32:/mnt/brick5/vol 49156 0 Y 384 Brick 10.3.3.42:/mnt/brick5/vol 49156 0 Y 335 Brick 10.3.3.11:/mnt/brick6/vol 49157 0 Y 413 Brick 10.3.3.33:/mnt/brick5/vol 49156 0 Y 384 Brick 10.3.3.43:/mnt/brick5/vol 49156 0 Y 355 Brick 10.3.3.9:/mnt/brick6/vol 49157 0 Y 413 Brick 10.3.3.19:/mnt/brick6/vol 49157 0 Y 411 Brick 10.3.3.29:/mnt/brick6/vol 49157 0 Y 411 Brick 10.3.3.20:/mnt/brick6/vol 49157 0 Y 413 Brick 10.3.3.30:/mnt/brick6/vol 49157 0 Y 350 Brick 10.3.3.40:/mnt/brick6/vol 49157 0 Y 363 Brick 10.3.3.21:/mnt/brick6/vol 49157 0 Y 411 Brick 10.3.3.31:/mnt/brick6/vol 49157 0 Y 403 Brick 10.3.3.41:/mnt/brick6/vol 49157 0 Y 361 Brick 10.3.3.12:/mnt/brick6/vol 49157 0 Y 411 Brick 10.3.3.22:/mnt/brick6/vol 49157 0 Y 382 Brick 10.3.3.39:/mnt/brick6/vol 49157 0 Y 839 Brick 10.3.3.13:/mnt/brick6/vol 49157 0 Y 413 Brick 10.3.3.32:/mnt/brick6/vol 49157 0 Y 403 Brick 10.3.3.42:/mnt/brick6/vol 49157 0 Y 361 Brick 10.3.3.23:/mnt/brick6/vol 49157 0 Y 359 Brick 10.3.3.33:/mnt/brick6/vol 49157 0 Y 403 Brick 10.3.3.43:/mnt/brick6/vol 49157 0 Y 360 Brick 10.3.3.9:/mnt/brick7/vol 49158 0 Y 432 Brick 10.3.3.19:/mnt/brick7/vol 49158 0 Y 430 Brick 10.3.3.29:/mnt/brick7/vol 49158 0 Y 430 Brick 10.3.3.10:/mnt/brick7/vol 49158 0 Y 430 Brick 10.3.3.20:/mnt/brick7/vol 49158 0 Y 432 Brick 10.3.3.30:/mnt/brick7/vol 49158 0 Y 369 Brick 10.3.3.11:/mnt/brick7/vol 49158 0 Y 432 Brick 10.3.3.21:/mnt/brick7/vol 49158 0 Y 430 Brick 10.3.3.39:/mnt/brick7/vol 49158 0 Y 858 Brick 10.3.3.12:/mnt/brick7/vol 49158 0 Y 430 Brick 10.3.3.22:/mnt/brick7/vol 49158 0 Y 401 Brick 10.3.3.40:/mnt/brick7/vol 49158 0 Y 369 Self-heal Daemon on localhost N/A N/A Y 15582 Quota Daemon on localhost N/A N/A Y 15590 Bitrot Daemon on localhost N/A N/A Y 15598 Scrubber Daemon on localhost N/A N/A Y 15607 Self-heal Daemon on 10.3.3.42 N/A N/A Y 1383 Quota Daemon on 10.3.3.42 N/A N/A Y 1391 Bitrot Daemon on 10.3.3.42 N/A N/A Y 1399 Scrubber Daemon on 10.3.3.42 N/A N/A Y 1408 Self-heal Daemon on 10.3.3.23 N/A N/A Y 4002 Quota Daemon on 10.3.3.23 N/A N/A Y 4010 Bitrot Daemon on 10.3.3.23 N/A N/A Y 4018 Scrubber Daemon on 10.3.3.23 N/A N/A Y 4027 Self-heal Daemon on 10.3.3.31 N/A N/A Y 15002 Quota Daemon on 10.3.3.31 N/A N/A Y 15010 Bitrot Daemon on 10.3.3.31 N/A N/A Y 15018 Scrubber Daemon on 10.3.3.31 N/A N/A Y 15027 Self-heal Daemon on 10.3.3.43 N/A N/A Y 9882 Quota Daemon on 10.3.3.43 N/A N/A Y 9890 Bitrot Daemon on 10.3.3.43 N/A N/A Y 9898 Scrubber Daemon on 10.3.3.43 N/A N/A Y 9907 Self-heal Daemon on 10.3.3.22 N/A N/A Y 15096 Quota Daemon on 10.3.3.22 N/A N/A Y 15104 Bitrot Daemon on 10.3.3.22 N/A N/A Y 15112 Scrubber Daemon on 10.3.3.22 N/A N/A Y 15121 Self-heal Daemon on 10.3.3.41 N/A N/A Y 9732 Quota Daemon on 10.3.3.41 N/A N/A Y 9740 Bitrot Daemon on 10.3.3.41 N/A N/A Y 9748 Scrubber Daemon on 10.3.3.41 N/A N/A Y 9757 Self-heal Daemon on 10.3.3.40 N/A N/A Y 5869 Quota Daemon on 10.3.3.40 N/A N/A Y 5877 Bitrot Daemon on 10.3.3.40 N/A N/A Y 5885 Scrubber Daemon on 10.3.3.40 N/A N/A Y 5894 Self-heal Daemon on 10.3.3.21 N/A N/A Y 15588 Quota Daemon on 10.3.3.21 N/A N/A Y 15596 Bitrot Daemon on 10.3.3.21 N/A N/A Y 15604 Scrubber Daemon on 10.3.3.21 N/A N/A Y 15613 Self-heal Daemon on ad06.pek.prod.com N/A N/A Y 16011 Quota Daemon on ad06.pek.prod.com N/A N/A Y 16019 Bitrot Daemon on ad06.pek.prod.com N/A N/A Y 16027 Scrubber Daemon on ad06.pek.prod.com N/A N/A Y 16036 Self-heal Daemon on 10.3.3.13 N/A N/A Y 15312 Quota Daemon on 10.3.3.13 N/A N/A Y 15320 Bitrot Daemon on 10.3.3.13 N/A N/A Y 15328 Scrubber Daemon on 10.3.3.13 N/A N/A Y 15337 Self-heal Daemon on 10.3.3.32 N/A N/A Y 15524 Quota Daemon on 10.3.3.32 N/A N/A Y 15532 Bitrot Daemon on 10.3.3.32 N/A N/A Y 15540 Scrubber Daemon on 10.3.3.32 N/A N/A Y 15549 Self-heal Daemon on 10.3.3.30 N/A N/A Y 5690 Quota Daemon on 10.3.3.30 N/A N/A Y 5698 Bitrot Daemon on 10.3.3.30 N/A N/A Y 5706 Scrubber Daemon on 10.3.3.30 N/A N/A Y 5715 Self-heal Daemon on 10.3.3.9 N/A N/A Y 15370 Quota Daemon on 10.3.3.9 N/A N/A Y 15378 Bitrot Daemon on 10.3.3.9 N/A N/A Y 15386 Scrubber Daemon on 10.3.3.9 N/A N/A Y 15395 Self-heal Daemon on 10.3.3.19 N/A N/A Y 15381 Quota Daemon on 10.3.3.19 N/A N/A Y 15389 Bitrot Daemon on 10.3.3.19 N/A N/A Y 15397 Scrubber Daemon on 10.3.3.19 N/A N/A Y 15406 Self-heal Daemon on 10.3.3.10 N/A N/A Y 15264 Quota Daemon on 10.3.3.10 N/A N/A Y 15272 Bitrot Daemon on 10.3.3.10 N/A N/A Y 15280 Scrubber Daemon on 10.3.3.10 N/A N/A Y 15289 Self-heal Daemon on 10.3.3.33 N/A N/A Y 15359 Quota Daemon on 10.3.3.33 N/A N/A Y 15367 Bitrot Daemon on 10.3.3.33 N/A N/A Y 15375 Scrubber Daemon on 10.3.3.33 N/A N/A Y 15384 Self-heal Daemon on 10.3.3.29 N/A N/A Y 15437 Quota Daemon on 10.3.3.29 N/A N/A Y 15445 Bitrot Daemon on 10.3.3.29 N/A N/A Y 15453 Scrubber Daemon on 10.3.3.29 N/A N/A Y 15462 Self-heal Daemon on 10.3.3.12 N/A N/A Y 15322 Quota Daemon on 10.3.3.12 N/A N/A Y 15330 Bitrot Daemon on 10.3.3.12 N/A N/A Y 15338 Scrubber Daemon on 10.3.3.12 N/A N/A Y 15347 Self-heal Daemon on 10.3.3.20 N/A N/A Y 15435 Quota Daemon on 10.3.3.20 N/A N/A Y 15443 Bitrot Daemon on 10.3.3.20 N/A N/A Y 15451 Scrubber Daemon on 10.3.3.20 N/A N/A Y 15460 Task Status of Volume gvol0 ---------------------------------------------------------------------------- -- There are no active volume tasks [root at aa08 /]# gluster volume heal gvol0 info Brick 10.3.3.9:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.19:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.29:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.10:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.20:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.30:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.9:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.19:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.29:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.11:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.21:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.39:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.12:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.22:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.40:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.13:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.31:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.41:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.23:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.32:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.42:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.33:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.43:/mnt/brick1/vol Status: Connected Number of entries: 0 Brick 10.3.3.10:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.20:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.30:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.40:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.11:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.21:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.39:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.12:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.22:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.31:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.13:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.32:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.41:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.23:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.33:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.42:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.9:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.19:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.29:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.10:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.20:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.43:/mnt/brick2/vol Status: Connected Number of entries: 0 Brick 10.3.3.11:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.21:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.39:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.12:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.30:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.40:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.22:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.31:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.41:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.13:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.32:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.42:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.23:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.33:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.43:/mnt/brick3/vol Status: Connected Number of entries: 0 Brick 10.3.3.9:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.19:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.29:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.10:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.20:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.30:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.11:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.21:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.39:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.12:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.22:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.40:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.13:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.31:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.41:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.10:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.33:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.43:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.23:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.32:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.42:/mnt/brick4/vol Status: Connected Number of entries: 0 Brick 10.3.3.9:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.19:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.29:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.20:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.30:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.40:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.11:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.21:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.39:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.12:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.22:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.31:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.13:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.23:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.41:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.10:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.32:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.42:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.11:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.33:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.43:/mnt/brick5/vol Status: Connected Number of entries: 0 Brick 10.3.3.9:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.19:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.29:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.20:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.30:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.40:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.21:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.31:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.41:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.12:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.22:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.39:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.13:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.32:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.42:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.23:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.33:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.43:/mnt/brick6/vol Status: Connected Number of entries: 0 Brick 10.3.3.9:/mnt/brick7/vol Status: Connected Number of entries: 0 Brick 10.3.3.19:/mnt/brick7/vol Status: Connected Number of entries: 0 Brick 10.3.3.29:/mnt/brick7/vol Status: Connected Number of entries: 0 Brick 10.3.3.10:/mnt/brick7/vol Status: Connected Number of entries: 0 Brick 10.3.3.20:/mnt/brick7/vol Status: Connected Number of entries: 0 Brick 10.3.3.30:/mnt/brick7/vol Status: Connected Number of entries: 0 Brick 10.3.3.11:/mnt/brick7/vol Status: Connected Number of entries: 0 Brick 10.3.3.21:/mnt/brick7/vol Status: Connected Number of entries: 0 Brick 10.3.3.39:/mnt/brick7/vol Status: Connected Number of entries: 0 Brick 10.3.3.12:/mnt/brick7/vol Status: Connected Number of entries: 0 Brick 10.3.3.22:/mnt/brick7/vol Status: Connected Number of entries: 0 Brick 10.3.3.40:/mnt/brick7/vol Status: Connected Number of entries: 0 # gluster volume info Volume Name: gvol0 Type: Distributed-Replicate Volume ID: cc9f0101-0bc7-4a40-a813-a7e540593a2b Status: Started Snapshot Count: 0 Number of Bricks: 44 x 3 = 132 Transport-type: tcp Bricks: Brick1: 10.3.3.9:/mnt/brick1/vol Brick2: 10.3.3.19:/mnt/brick1/vol Brick3: 10.3.3.29:/mnt/brick1/vol Brick4: 10.3.3.10:/mnt/brick1/vol Brick5: 10.3.3.20:/mnt/brick1/vol Brick6: 10.3.3.30:/mnt/brick1/vol Brick7: 10.3.3.9:/mnt/brick2/vol Brick8: 10.3.3.19:/mnt/brick2/vol Brick9: 10.3.3.29:/mnt/brick2/vol Brick10: 10.3.3.11:/mnt/brick1/vol Brick11: 10.3.3.21:/mnt/brick1/vol Brick12: 10.3.3.39:/mnt/brick1/vol Brick13: 10.3.3.12:/mnt/brick1/vol Brick14: 10.3.3.22:/mnt/brick1/vol Brick15: 10.3.3.40:/mnt/brick1/vol Brick16: 10.3.3.13:/mnt/brick1/vol Brick17: 10.3.3.31:/mnt/brick1/vol Brick18: 10.3.3.41:/mnt/brick1/vol Brick19: 10.3.3.23:/mnt/brick1/vol Brick20: 10.3.3.32:/mnt/brick1/vol Brick21: 10.3.3.42:/mnt/brick1/vol Brick22: 10.3.3.33:/mnt/brick1/vol Brick23: 10.3.3.43:/mnt/brick1/vol Brick24: 10.3.3.10:/mnt/brick2/vol Brick25: 10.3.3.20:/mnt/brick2/vol Brick26: 10.3.3.30:/mnt/brick2/vol Brick27: 10.3.3.40:/mnt/brick2/vol Brick28: 10.3.3.11:/mnt/brick2/vol Brick29: 10.3.3.21:/mnt/brick2/vol Brick30: 10.3.3.39:/mnt/brick2/vol Brick31: 10.3.3.12:/mnt/brick2/vol Brick32: 10.3.3.22:/mnt/brick2/vol Brick33: 10.3.3.31:/mnt/brick2/vol Brick34: 10.3.3.13:/mnt/brick2/vol Brick35: 10.3.3.32:/mnt/brick2/vol Brick36: 10.3.3.41:/mnt/brick2/vol Brick37: 10.3.3.23:/mnt/brick2/vol Brick38: 10.3.3.33:/mnt/brick2/vol Brick39: 10.3.3.42:/mnt/brick2/vol Brick40: 10.3.3.9:/mnt/brick3/vol Brick41: 10.3.3.19:/mnt/brick3/vol Brick42: 10.3.3.29:/mnt/brick3/vol Brick43: 10.3.3.10:/mnt/brick3/vol Brick44: 10.3.3.20:/mnt/brick3/vol Brick45: 10.3.3.43:/mnt/brick2/vol Brick46: 10.3.3.11:/mnt/brick3/vol Brick47: 10.3.3.21:/mnt/brick3/vol Brick48: 10.3.3.39:/mnt/brick3/vol Brick49: 10.3.3.12:/mnt/brick3/vol Brick50: 10.3.3.30:/mnt/brick3/vol Brick51: 10.3.3.40:/mnt/brick3/vol Brick52: 10.3.3.22:/mnt/brick3/vol Brick53: 10.3.3.31:/mnt/brick3/vol Brick54: 10.3.3.41:/mnt/brick3/vol Brick55: 10.3.3.13:/mnt/brick3/vol Brick56: 10.3.3.32:/mnt/brick3/vol Brick57: 10.3.3.42:/mnt/brick3/vol Brick58: 10.3.3.23:/mnt/brick3/vol Brick59: 10.3.3.33:/mnt/brick3/vol Brick60: 10.3.3.43:/mnt/brick3/vol Brick61: 10.3.3.9:/mnt/brick4/vol Brick62: 10.3.3.19:/mnt/brick4/vol Brick63: 10.3.3.29:/mnt/brick4/vol Brick64: 10.3.3.10:/mnt/brick4/vol Brick65: 10.3.3.20:/mnt/brick4/vol Brick66: 10.3.3.30:/mnt/brick4/vol Brick67: 10.3.3.11:/mnt/brick4/vol Brick68: 10.3.3.21:/mnt/brick4/vol Brick69: 10.3.3.39:/mnt/brick4/vol Brick70: 10.3.3.12:/mnt/brick4/vol Brick71: 10.3.3.22:/mnt/brick4/vol Brick72: 10.3.3.40:/mnt/brick4/vol Brick73: 10.3.3.13:/mnt/brick4/vol Brick74: 10.3.3.31:/mnt/brick4/vol Brick75: 10.3.3.41:/mnt/brick4/vol Brick76: 10.3.3.10:/mnt/brick5/vol Brick77: 10.3.3.33:/mnt/brick4/vol Brick78: 10.3.3.43:/mnt/brick4/vol Brick79: 10.3.3.23:/mnt/brick4/vol Brick80: 10.3.3.32:/mnt/brick4/vol Brick81: 10.3.3.42:/mnt/brick4/vol Brick82: 10.3.3.9:/mnt/brick5/vol Brick83: 10.3.3.19:/mnt/brick5/vol Brick84: 10.3.3.29:/mnt/brick5/vol Brick85: 10.3.3.20:/mnt/brick5/vol Brick86: 10.3.3.30:/mnt/brick5/vol Brick87: 10.3.3.40:/mnt/brick5/vol Brick88: 10.3.3.11:/mnt/brick5/vol Brick89: 10.3.3.21:/mnt/brick5/vol Brick90: 10.3.3.39:/mnt/brick5/vol Brick91: 10.3.3.12:/mnt/brick5/vol Brick92: 10.3.3.22:/mnt/brick5/vol Brick93: 10.3.3.31:/mnt/brick5/vol Brick94: 10.3.3.13:/mnt/brick5/vol Brick95: 10.3.3.23:/mnt/brick5/vol Brick96: 10.3.3.41:/mnt/brick5/vol Brick97: 10.3.3.10:/mnt/brick6/vol Brick98: 10.3.3.32:/mnt/brick5/vol Brick99: 10.3.3.42:/mnt/brick5/vol Brick100: 10.3.3.11:/mnt/brick6/vol Brick101: 10.3.3.33:/mnt/brick5/vol Brick102: 10.3.3.43:/mnt/brick5/vol Brick103: 10.3.3.9:/mnt/brick6/vol Brick104: 10.3.3.19:/mnt/brick6/vol Brick105: 10.3.3.29:/mnt/brick6/vol Brick106: 10.3.3.20:/mnt/brick6/vol Brick107: 10.3.3.30:/mnt/brick6/vol Brick108: 10.3.3.40:/mnt/brick6/vol Brick109: 10.3.3.21:/mnt/brick6/vol Brick110: 10.3.3.31:/mnt/brick6/vol Brick111: 10.3.3.41:/mnt/brick6/vol Brick112: 10.3.3.12:/mnt/brick6/vol Brick113: 10.3.3.22:/mnt/brick6/vol Brick114: 10.3.3.39:/mnt/brick6/vol Brick115: 10.3.3.13:/mnt/brick6/vol Brick116: 10.3.3.32:/mnt/brick6/vol Brick117: 10.3.3.42:/mnt/brick6/vol Brick118: 10.3.3.23:/mnt/brick6/vol Brick119: 10.3.3.33:/mnt/brick6/vol Brick120: 10.3.3.43:/mnt/brick6/vol Brick121: 10.3.3.9:/mnt/brick7/vol Brick122: 10.3.3.19:/mnt/brick7/vol Brick123: 10.3.3.29:/mnt/brick7/vol Brick124: 10.3.3.10:/mnt/brick7/vol Brick125: 10.3.3.20:/mnt/brick7/vol Brick126: 10.3.3.30:/mnt/brick7/vol Brick127: 10.3.3.11:/mnt/brick7/vol Brick128: 10.3.3.21:/mnt/brick7/vol Brick129: 10.3.3.39:/mnt/brick7/vol Brick130: 10.3.3.12:/mnt/brick7/vol Brick131: 10.3.3.22:/mnt/brick7/vol Brick132: 10.3.3.40:/mnt/brick7/vol Options Reconfigured: nfs.disable: on performance.readdir-ahead: on transport.address-family: inet features.quota: on features.inode-quota: on features.quota-deem-statfs: on cluster.quorum-type: auto features.quota-timeout: 10 features.bitrot: on features.scrub: Active performance.cache-size: 4GB diagnostics.latency-measurement: on diagnostics.count-fop-hits: on -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20170103/220de743/attachment.html>
Ravishankar N
2017-Jan-04 02:30 UTC
[Gluster-users] read error: Input/output error but not in brain split stat
On 01/03/2017 09:03 AM, likun wrote:> > Then I tried to access this file: > > # cd /mnt/glusterfs/gps/data/2017-01/02 > > # more 4.txt > > # cat 4.txt > > cat: read error: Input/output error > > I supposed there was something wrong with glusterfs, so I logined into > glusterfs system, found nothing wrong, the output of ?gluster volume > status|info? and ?gluster volume heal gvol0 info? command appended at > the end. So, question is what?s wrong? >What did the mount log say at this time? Also were there errors/warnings on the 3 brick logs where the file resides at the same time? Regards, Ravi -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20170104/bbff2674/attachment.html>