Eugene M. Zheganin
2015-Nov-05 19:29 UTC
unable to boot a healthy zfs pool: all block copies unavailable
Hi. Today one of my zfs pool disks dies, I was unable to change it on the fly (video board was blocking it) so I powered off, changed disk (not in root pool) and all of a sudden I realized that i cannot boot: ZFS: i/o error - all block copies unavailable ZFS: can't read MOS of pool zroot gptzfsboot: failed to mount default pool zroot It was first reboot since October, 16th, when I installed recent -STABLE and upgraded zpool. I was pretty confident that I've installed loaders, but I tried to reinstall them - no luck. Then I built today's STABLE and installed loaders from it - same issue. I've even tried to install less recent loaders from a server nearby - same issue. Two years ago I have encountered similar (if not identical) issue: https://lists.freebsd.org/pipermail/freebsd-stable/2013-December/076317.html The main difference was it was i386. Now I have an amd64 machine. I even updated it's BIOS, and I still cannot boot. Zpool is fine: I'm writing this message from this exact machine, however, I had to boot it from today -STABLE from an USB stick. I've read about the zfsboottest utility and tried it on my unbootable pool - after the bried info about it (traated as healthy) it said "OK". I guess no errors were encountered. So... what can I do to restore the ability to boot from my root pool ? Thanks. Eugene. P.S. Some info about the pool below. let me know if it's not enough - I'll post more. [root at bsdrookie:/]# zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 1,79T 690G 1,12T - 14% 37% 1.00x ONLINE - [root at bsdrookie:/]# zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 704G 1,05T 2,07G legacy zroot/crypted 516G 1,06T 502G - zroot/jails 2,16G 1,05T 2,03G /usr/local/public/jails zroot/tmp 223M 1,05T 223M /tmp zroot/usr 182G 1,05T 17,0G /usr zroot/usr/home 145G 1,05T 145G /usr/home zroot/usr/ports 17,0G 1,05T 2,79G /usr/ports zroot/usr/ports/distfiles 14,3G 1,05T 14,3G /usr/ports/distfiles zroot/usr/ports/packages 384K 1,05T 384K /usr/ports/packages zroot/usr/public 2,11G 1,05T 2,11G /usr/local/public zroot/usr/src 1,56G 1,05T 1,56G /usr/src zroot/var 1,19G 1,05T 83,7M /var zroot/var/crash 992M 1,05T 992M /var/crash zroot/var/db 113M 1,05T 39,7M /var/db zroot/var/db/pkg 73,1M 1,05T 73,1M /var/db/pkg zroot/var/empty 144K 1,05T 144K /var/empty zroot/var/log 2,53M 1,05T 2,53M /var/log zroot/var/mail 272K 1,05T 272K /var/mail zroot/var/run 520K 1,05T 520K /var/run zroot/var/tmp 22,7M 1,05T 22,7M /var/tmp
Andriy Gapon
2015-Nov-05 21:58 UTC
unable to boot a healthy zfs pool: all block copies unavailable
On 05/11/2015 21:29, Eugene M. Zheganin wrote:> Hi. > > Today one of my zfs pool disks dies, I was unable to change it on the > fly (video board was blocking it) so I powered off, changed disk (not in > root pool) and all of a sudden I realized that i cannot boot: > > ZFS: i/o error - all block copies unavailable > ZFS: can't read MOS of pool zroot > gptzfsboot: failed to mount default pool zroot > > It was first reboot since October, 16th, when I installed recent -STABLE > and upgraded zpool. I was pretty confident that I've installed loaders, > but I tried to reinstall them - no luck. Then I built today's STABLE and > installed loaders from it - same issue. I've even tried to install less > recent loaders from a server nearby - same issue. > > Two years ago I have encountered similar (if not identical) issue: > https://lists.freebsd.org/pipermail/freebsd-stable/2013-December/076317.html > The main difference was it was i386. Now I have an amd64 machine. I even > updated it's BIOS, and I still cannot boot. Zpool is fine: I'm writing > this message from this exact machine, however, I had to boot it from > today -STABLE from an USB stick. > > I've read about the zfsboottest utility and tried it on my unbootable > pool - after the bried info about it (traated as healthy) it said "OK". > I guess no errors were encountered. > > So... what can I do to restore the ability to boot from my root pool ?It could be that your BIOS is not able to read past 1TB (512 * INT_MAX). That seems to be a rather common problem for consumer motherboards. Here is an example of how it looked for me: https://people.freebsd.org/~avg/IMAG1099.jpg Fortunately, it wasn't a root pool that got the error.> P.S. Some info about the pool below. let me know if it's not enough - > I'll post more. > > [root at bsdrookie:/]# zpool list > NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > zroot 1,79T 690G 1,12T - 14% 37% 1.00x ONLINE - > > [root at bsdrookie:/]# zfs list > NAME USED AVAIL REFER MOUNTPOINT > zroot 704G 1,05T 2,07G legacy > zroot/crypted 516G 1,06T 502G - > zroot/jails 2,16G 1,05T 2,03G /usr/local/public/jails > zroot/tmp 223M 1,05T 223M /tmp > zroot/usr 182G 1,05T 17,0G /usr > zroot/usr/home 145G 1,05T 145G /usr/home > zroot/usr/ports 17,0G 1,05T 2,79G /usr/ports > zroot/usr/ports/distfiles 14,3G 1,05T 14,3G /usr/ports/distfiles > zroot/usr/ports/packages 384K 1,05T 384K /usr/ports/packages > zroot/usr/public 2,11G 1,05T 2,11G /usr/local/public > zroot/usr/src 1,56G 1,05T 1,56G /usr/src > zroot/var 1,19G 1,05T 83,7M /var > zroot/var/crash 992M 1,05T 992M /var/crash > zroot/var/db 113M 1,05T 39,7M /var/db > zroot/var/db/pkg 73,1M 1,05T 73,1M /var/db/pkg > zroot/var/empty 144K 1,05T 144K /var/empty > zroot/var/log 2,53M 1,05T 2,53M /var/log > zroot/var/mail 272K 1,05T 272K /var/mail > zroot/var/run 520K 1,05T 520K /var/run > zroot/var/tmp 22,7M 1,05T 22,7M /var/tmp > > > _______________________________________________ > freebsd-stable at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org" >-- Andriy Gapon
Michael B. Eichorn
2015-Nov-06 00:11 UTC
unable to boot a healthy zfs pool: all block copies unavailable
On Fri, 2015-11-06 at 00:29 +0500, Eugene M. Zheganin wrote:> Hi. > > Today one of my zfs pool disks dies, I was unable to change it on the > fly (video board was blocking it) so I powered off, changed disk (not > in > root pool) and all of a sudden I realized that i cannot boot: > > ZFS: i/o error - all block copies unavailable > ZFS: can't read MOS of pool zroot > gptzfsboot: failed to mount default pool zroot > > It was first reboot since October, 16th, when I installed recent > -STABLE > and upgraded zpool. I was pretty confident that I've installed loaders, > but I tried to reinstall them - no luck. Then I built today's STABLE > and > installed loaders from it - same issue. I've even tried to install less > recent loaders from a server nearby - same issue. > > Two years ago I have encountered similar (if not identical) issue: > https://lists.freebsd.org/pipermail/freebsd-stable/2013-December/076317 > .html > The main difference was it was i386. Now I have an amd64 machine. I > even > updated it's BIOS, and I still cannot boot. Zpool is fine: I'm writing > this message from this exact machine, however, I had to boot it from > today -STABLE from an USB stick. > > I've read about the zfsboottest utility and tried it on my unbootable > pool - after the bried info about it (traated as healthy) it said "OK". > I guess no errors were encountered. > > So... what can I do to restore the ability to boot from my root pool ? > > Thanks. > EugeneI had something similar happen to me in the last month with a the installer zfs setup. Unfortunately it was a critical system so I didn't have time for diagnostics. My pants-on-fire solution was that since the pool was a mirror, swap a disk out with my spare, reinstall, and send/receive the data back. Not pretty, but it was my personal mail/file/web/everything-not-on-the-firewall server. The pool was accessible from the installer shell and on import to the fresh system, it just wouldn't boot. Also in my case it was RELEASE not STABLE, and yes it was consumer grade hardware. TLDR: Not just you. Reinstall? -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 5761 bytes Desc: not available URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20151105/7557e410/attachment.bin>