zfs discuss - May 2010 - Mirroring USB Drive with Laptop for Backup purposes

Hi,

Just wondering whether mirroring a USB drive with main laptop disk for 
backup purposes is recommended or not.

Current setup, single root pool set up on 200GB internal laptop drive :

$ zpool status
   pool: rpool
  state: ONLINE
  scrub: non requested
config :
	NAME        STATE     READ WRITE CKSUM
	rpool       ONLINE       0     0     0
	  c5t0d0s0  ONLINE       0     0     0


I have a 320GB external USB drive which I''d like to configure as a 
mirror of this root pool (I know it will only use 200GB of the eternal 
one, not worried about that).

Plan would be to connect the USB drive, once or twice a week, let it 
resilver, and then disconnect again. Connecting USB drive 24/7 would 
AFAIK have performance issues for the Laptop.

This would have the added benefit of the USB drive being bootable.

- Recommended or not ?
- Are there known issues with this type of setup ?


cheers

Matt

Hi Matt,

Don''t know if it''s recommended or not, but I''ve been
doing it for close
to 3 years on my OpenSolaris laptop, it saved me a few times like last 
week when my internal drive died :)

/peter

On 2010-05-04 20.33, Matt Keenan wrote:
> Hi,
>
> Just wondering whether mirroring a USB drive with main laptop disk for
> backup purposes is recommended or not.
>
> Current setup, single root pool set up on 200GB internal laptop drive :
>
> $ zpool status
> pool: rpool
> state: ONLINE
> scrub: non requested
> config :
> NAME STATE READ WRITE CKSUM
> rpool ONLINE 0 0 0
> c5t0d0s0 ONLINE 0 0 0
>
>
> I have a 320GB external USB drive which I''d like to configure as a
> mirror of this root pool (I know it will only use 200GB of the eternal
> one, not worried about that).
>
> Plan would be to connect the USB drive, once or twice a week, let it
> resilver, and then disconnect again. Connecting USB drive 24/7 would
> AFAIK have performance issues for the Laptop.
>
> This would have the added benefit of the USB drive being bootable.
>
> - Recommended or not ?
> - Are there known issues with this type of setup ?
>
>
> cheers
>
> Matt
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Matt Keenan
> 
> Just wondering whether mirroring a USB drive with main laptop disk for
> backup purposes is recommended or not.
> 
> Plan would be to connect the USB drive, once or twice a week, let it
> resilver, and then disconnect again. Connecting USB drive 24/7 would
> AFAIK have performance issues for the Laptop.
MMmmm...  If it works, sounds good.  But I don''t think it''ll
work as
expected, for a number of reasons, outlined below.

The suggestion I would have instead, would be to make the external drive its
own separate zpool, and then you can incrementally "zfs send | zfs
receive"
onto the external.

Here are the obstacles I think you''ll have with your proposed solution:

#1 I think all the entire used portion of the filesystem needs to resilver
every time.  I don''t think there''s any such thing as an
incremental
resilver.  

#2 How would you plan to disconnect the drive?  If you zpool detach it, I
think it''s no longer a mirror, and not mountable.  If you simply yank
out
the plug ... although that might work, it would certainly be nonideal.  If
you power off, disconnect, and power on ... Again, it should probably be
fine, but it''s not designed to be used that way intentionally, so your
results ... are probably as-yet untested.

I don''t want to go on.  This list could go on forever.  I will strongly
encourage you to simply use "zfs send | zfs receive" because
that''s a
standard practice thing to do.  It is known that the external drive is not
bootable this way, but that''s why you have this article on how to make
it
bootable:

http://docs.sun.com/app/docs/doc/819-5461/ghzur?l=en&a=view

> This would have the added benefit of the USB drive being bootable.
By default, AFAIK, that''s not correct.  When you mirror rpool to
another
device, by default the 2nd device is not bootable, because it''s just
got an
rpool in there.  No boot loader.

Even if you do this mirror idea, which I believe will be slower and less
reliable than "zfs send | zfs receive" you still haven''t
gained anything as
compared to the "zfs send | zfs receive" procedure, which is known to
work
reliable with optimal performance.

* Edward Ned Harvey (solaris2 at nedharvey.com) wrote:
> > From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> > bounces at opensolaris.org] On Behalf Of Matt Keenan
> > 
> > Just wondering whether mirroring a USB drive with main laptop disk for
> > backup purposes is recommended or not.
> > 
> > Plan would be to connect the USB drive, once or twice a week, let it
> > resilver, and then disconnect again. Connecting USB drive 24/7 would
> > AFAIK have performance issues for the Laptop.
> 
> MMmmm...  If it works, sounds good.  But I don''t think
it''ll work as
> expected, for a number of reasons, outlined below.
It used to work for James Gosling.

http://blogs.sun.com/jag/entry/solaris_and_os_x

[snip]
 
> > This would have the added benefit of the USB drive being bootable.
> 
> By default, AFAIK, that''s not correct.  When you mirror rpool to
> another device, by default the 2nd device is not bootable, because
> it''s just got an rpool in there.  No boot loader.
That''s true, but easily fixed (just like for any other mirrored pool
configuration).

installgrub -m /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/{disk}
> Even if you do this mirror idea, which I believe will be slower and
> less reliable than "zfs send | zfs receive" you still
haven''t gained
> anything as compared to the "zfs send | zfs receive" procedure,
which
> is known to work reliable with optimal performance.
How about ease-of-use, all you have to do is plug in the usb disk and
zfs will ''do the right thing''.  You don''t have to
remember to run zfs
send | zfs receive, or bother with figuring out what to send/recv etc
etc etc.

Cheers,

-- 
Glenn

Glenn Lagasse wrote:
> 
> How about ease-of-use, all you have to do is plug in the usb disk and
> zfs will ''do the right thing''.  You don''t have
to remember to run zfs
> send | zfs receive, or bother with figuring out what to send/recv etc
> etc etc.
It should be possible to automate that via syseventd/syseventconfd. 
Sadly the documentation is a bit... um... sparse...

-- 
Carson

On Wed, May 05, 2010 at 04:34:13PM -0400, Edward Ned Harvey
wrote:
> The suggestion I would have instead, would be to make the external drive
its
> own separate zpool, and then you can incrementally "zfs send | zfs
receive"
> onto the external.
I''d suggest doing both, to different destinations :)  Each kind of
"backup" serves different, complementary purposes.
> #1 I think all the entire used portion of the filesystem needs to resilver
> every time.  I don''t think there''s any such thing as an
incremental
> resilver.  
Incorrect. It will play forward all the (still-live) blocks from txg''s
between the time it was last online and now. 

That said, I''d also recommend a scrub on a regular basis, once the
resilver has completed, and that will trawl through all the data and
take all that time you were worried about anyway.  For a 200G disk,
full, over usb, I''d expect around 4-5 hours.  That''s fine for
a "leave
running overnight" workflow.

This is the benefit of this kind of "backup" - as well as being almost
brainless to initiate, it''s able to automatically repair marginal
sectors on the laptop disk if they become unreadable, saving you from
the hassle of trying to restore damaged files.

The send|recv kind of backup is much better for restoring data from
old snapshots (if the target is larger than the source and keeps them
longer), and recovering from accidentally destroying both mirrored
copies of data due to operator error. 
> #2 How would you plan to disconnect the drive?  If you zpool detach it, I
> think it''s no longer a mirror, and not mountable.
That''s correct - which is why you would use "zpool offline".

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100506/41fdf726/attachment.bin>

On Wed, 5 May 2010, Edward Ned Harvey wrote:
>
> Here are the obstacles I think you''ll have with your proposed
solution:
>
> #1 I think all the entire used portion of the filesystem needs to resilver
> every time.  I don''t think there''s any such thing as an
incremental
> resilver.
It sounds like you are not sure.  Maybe you should be sure.  Yes, I do 
think that it is a wise idea if you are really sure.

See "Transactional pruning" at

   http://blogs.sun.com/bonwick/entry/smokin_mirrors

and then "Top-down resilvering".
>> This would have the added benefit of the USB drive being bootable.
>
> By default, AFAIK, that''s not correct.  When you mirror rpool to
another
> device, by default the 2nd device is not bootable, because it''s
just got an
> rpool in there.  No boot loader.
Unless it was added at install time, or the user added a boot loader. 
It is quite doable since it is the normal case as when a system is 
installed onto a mirror pair of disks.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Thu, 6 May 2010, Daniel Carosone wrote:
>
> That said, I''d also recommend a scrub on a regular basis, once the
> resilver has completed, and that will trawl through all the data and
> take all that time you were worried about anyway.  For a 200G disk,
> full, over usb, I''d expect around 4-5 hours.  That''s fine
for a "leave
> running overnight" workflow.
When I have simply powered down a mirror disk in a USB-based mirrored 
pair, I have noticed that it seems that zfs must be doing its own 
little secret scrub of the restored disk without me requesting it even 
though ''zpool status'' does not mention it and it says the disk
is
resilvered.  The flashing lights annoyed me so I exported and imported 
the pool and then the flashing lights were gone.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

Based on comments, some people say nay, some say yah..... so I decided 
to give it a spin, and see
how I get on.

To make my mirror bootable I followed instructions posted here :
   http://www.taiter.com/blog/2009/04/opensolaris-200811-adding-disk.html

I plan to do a quick write up myself of my own experience, but so far 
everything is working fine.

Mirror size is 200GB (Smallest disk, happens to be laptop disk), once I 
attached the USB drive, it
started resilvering straight away, and only took 1hr 45mins to complete 
and it resilvered 120G !!
This I was very impressed with.

So far I''ve not noticed any system performance degradation with the 
drive attached. I did a quick test, yanked out the drive, degrades rpool 
as expected, but system continues to function fine.

I also did a quick test to see of the USB drive was indeed bootable, by 
connecting to another laptop, it booted perfectly.

Connecting the USB drive back to original laptop, the pool comes back 
online and resilvers seamlessly.

This is automatic 24/7 backup at it''s best...

One thing I did notice, I powered down yesterday whilst USB was 
attached, this morning when booting up, I did so without the USB 
attached, laptop failed to boot, I had to connect the USB drive and it 
booted up fine.

Key would be to degrade the pool before shutdown, e.g. disconnect USB 
drive, might try using zpool offline and see how that works.

If I encounter issues, I''ll post again.

cheers

Matt

On 05/ 5/10 09:34 PM, Edward Ned Harvey wrote:
>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
>> bounces at opensolaris.org] On Behalf Of Matt Keenan
>>
>> Just wondering whether mirroring a USB drive with main laptop disk for
>> backup purposes is recommended or not.
>>
>> Plan would be to connect the USB drive, once or twice a week, let it
>> resilver, and then disconnect again. Connecting USB drive 24/7 would
>> AFAIK have performance issues for the Laptop.
>>      
> MMmmm...  If it works, sounds good.  But I don''t think
it''ll work as
> expected, for a number of reasons, outlined below.
>
> The suggestion I would have instead, would be to make the external drive
its
> own separate zpool, and then you can incrementally "zfs send | zfs
receive"
> onto the external.
>
> Here are the obstacles I think you''ll have with your proposed
solution:
>
> #1 I think all the entire used portion of the filesystem needs to resilver
> every time.  I don''t think there''s any such thing as an
incremental
> resilver.
>
> #2 How would you plan to disconnect the drive?  If you zpool detach it, I
> think it''s no longer a mirror, and not mountable.  If you simply
yank out
> the plug ... although that might work, it would certainly be nonideal.  If
> you power off, disconnect, and power on ... Again, it should probably be
> fine, but it''s not designed to be used that way intentionally, so
your
> results ... are probably as-yet untested.
>
> I don''t want to go on.  This list could go on forever.  I will
strongly
> encourage you to simply use "zfs send | zfs receive" because
that''s a
> standard practice thing to do.  It is known that the external drive is not
> bootable this way, but that''s why you have this article on how to
make it
> bootable:
>
> http://docs.sun.com/app/docs/doc/819-5461/ghzur?l=en&a=view
>
>
>    
>> This would have the added benefit of the USB drive being bootable.
>>      
> By default, AFAIK, that''s not correct.  When you mirror rpool to
another
> device, by default the 2nd device is not bootable, because it''s
just got an
> rpool in there.  No boot loader.
>
> Even if you do this mirror idea, which I believe will be slower and less
> reliable than "zfs send | zfs receive" you still haven''t
gained anything as
> compared to the "zfs send | zfs receive" procedure, which is
known to work
> reliable with optimal performance.
>
>

After some playing around I''ve noticed some kinks particularly around 
booting.

Some scenarios :

- Poweroff with USB drive connected or removed, Solaris will not boot 
unless USB drive is
   connected, and in some cases need to be attached to the exact same 
USB port when last
   attached. Is this a bug ?

- Take USB drive offline via zfs offline, and poweroff, this is much 
nastier, as machine would not
   boot at all regardless of whether USB drive was connected or not. I 
had to boot into LiveCD,
   zpool import (whilst USB was attached), and bring USB drive back 
online via zfs online pool
   disk.

Exact steps on what I did :
   http://blogs.sun.com/mattman/entry/bootable_usb_mirror

As I find other caveats I''ll add them... But it looks like having the 
drive connected at all times is preferable.

cheers

Matt

On 05/ 6/10 12:11 PM, Matt Keenan wrote:
>
> Based on comments, some people say nay, some say yah..... so I decided 
> to give it a spin, and see
> how I get on.
>
> To make my mirror bootable I followed instructions posted here :
>   http://www.taiter.com/blog/2009/04/opensolaris-200811-adding-disk.html
>
> I plan to do a quick write up myself of my own experience, but so far 
> everything is working fine.
>
> Mirror size is 200GB (Smallest disk, happens to be laptop disk), once 
> I attached the USB drive, it
> started resilvering straight away, and only took 1hr 45mins to 
> complete and it resilvered 120G !!
> This I was very impressed with.
>
> So far I''ve not noticed any system performance degradation with
the
> drive attached. I did a quick test, yanked out the drive, degrades 
> rpool as expected, but system continues to function fine.
>
> I also did a quick test to see of the USB drive was indeed bootable, 
> by connecting to another laptop, it booted perfectly.
>
> Connecting the USB drive back to original laptop, the pool comes back 
> online and resilvers seamlessly.
>
> This is automatic 24/7 backup at it''s best...
>
> One thing I did notice, I powered down yesterday whilst USB was 
> attached, this morning when booting up, I did so without the USB 
> attached, laptop failed to boot, I had to connect the USB drive and it 
> booted up fine.
>
> Key would be to degrade the pool before shutdown, e.g. disconnect USB 
> drive, might try using zpool offline and see how that works.
>
> If I encounter issues, I''ll post again.
>
> cheers
>
> Matt
>
> On 05/ 5/10 09:34 PM, Edward Ned Harvey wrote:
>>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
>>> bounces at opensolaris.org] On Behalf Of Matt Keenan
>>>
>>> Just wondering whether mirroring a USB drive with main laptop disk
for
>>> backup purposes is recommended or not.
>>>
>>> Plan would be to connect the USB drive, once or twice a week, let
it
>>> resilver, and then disconnect again. Connecting USB drive 24/7
would
>>> AFAIK have performance issues for the Laptop.
>> MMmmm...  If it works, sounds good.  But I don''t think
it''ll work as
>> expected, for a number of reasons, outlined below.
>>
>> The suggestion I would have instead, would be to make the external 
>> drive its
>> own separate zpool, and then you can incrementally "zfs send | zfs
>> receive"
>> onto the external.
>>
>> Here are the obstacles I think you''ll have with your proposed
solution:
>>
>> #1 I think all the entire used portion of the filesystem needs to 
>> resilver
>> every time.  I don''t think there''s any such thing as
an incremental
>> resilver.
>>
>> #2 How would you plan to disconnect the drive?  If you zpool detach 
>> it, I
>> think it''s no longer a mirror, and not mountable.  If you
simply yank
>> out
>> the plug ... although that might work, it would certainly be 
>> nonideal.  If
>> you power off, disconnect, and power on ... Again, it should probably
be
>> fine, but it''s not designed to be used that way intentionally,
so your
>> results ... are probably as-yet untested.
>>
>> I don''t want to go on.  This list could go on forever.  I will
strongly
>> encourage you to simply use "zfs send | zfs receive" because
that''s a
>> standard practice thing to do.  It is known that the external drive 
>> is not
>> bootable this way, but that''s why you have this article on how
to
>> make it
>> bootable:
>>
>> http://docs.sun.com/app/docs/doc/819-5461/ghzur?l=en&a=view
>>
>>
>>> This would have the added benefit of the USB drive being bootable.
>> By default, AFAIK, that''s not correct.  When you mirror rpool
to another
>> device, by default the 2nd device is not bootable, because
it''s just
>> got an
>> rpool in there.  No boot loader.
>>
>> Even if you do this mirror idea, which I believe will be slower and
less
>> reliable than "zfs send | zfs receive" you still
haven''t gained
>> anything as
>> compared to the "zfs send | zfs receive" procedure, which is
known to
>> work
>> reliable with optimal performance.
>>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

> From: Matt Keenan [mailto:mattman at opensolaris.org]
> 
> After some playing around I''ve noticed some kinks particularly
around
> booting.
I''m going to continue encouraging you to staying
"mainstream," because what
people do the most is usually what''s supported the best.  I think
you''ll
have a more reliable system overall if you incrementally "zfs send | zfs
receive" to your external storage.

Unless you just want to do what you''re doing for academic reasons.  I
sure
as heck do all kinds of crazy unadvisable computer stuff for academic
purposes.  And you should see the kind of ridiculous stuff my brother
builds!  He still has a P4 with 9 PCI cards in it, for some sort of ham
radio data repeater including modem and stuff, asterisk and radio project.
It''s crazy for sure.

On Fri, May 7, 2010 at 2:51 AM, Matt Keenan <mattman at opensolaris.org>
wrote:
> - Poweroff with USB drive connected or removed, Solaris will not boot
unless
> USB drive is
> ?connected, and in some cases need to be attached to the exact same USB
port
> when last
> ?attached. Is this a bug ?
There''s a known issue in recent builds about booting with part of a
mirrored boot drive unavailable. That''s most likely what
you''re
hitting.

The drive should be on the same USB port because the device path is
saved in the zpool.cache. If you removed the zpool.cache, it wouldn''t
matter where the drive was plugged in.
> - Take USB drive offline via zfs offline, and poweroff, this is much
I think this is another manifestation of the same bug.

-B

-- 
Brandon High : bhigh at freaks.com

On 05/07/2010 11:08 AM, Edward Ned Harvey wrote:
> I''m going to continue encouraging you to staying
"mainstream," because what
> people do the most is usually what''s supported the best.
If I may be the contrarian, I hope Matt keeps experimenting with this, 
files bugs, and they get fixed.  His use case is very compelling - I 
know lots of SOHO folks who could really use a NAS where this ''just
worked''

The ZFS team has done well by thinking liberally about conventional 
assumptions.

-Bill

-- 
Bill McGonigle, Owner
BFC Computing, LLC
http://bfccomputing.com/
Telephone: +1.603.448.4440
Email, IM, VOIP: bill at bfccomputing.com
VCard: http://bfccomputing.com/vcard/bill.vcf
Social networks: bill_mcgonigle/bill.mcgonigle

> - Poweroff with USB drive connected or removed, Solaris will not boot 
> unless USB drive is
>   connected, and in some cases need to be attached to the exact same 
> USB port when last
>   attached. Is this a bug ?

Possibly hitting this?
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6923585

Yours
Markus Kovero

On 05/ 7/10 10:07 PM, Bill McGonigle wrote:
> On 05/07/2010 11:08 AM, Edward Ned Harvey wrote:
>> I''m going to continue encouraging you to staying
"mainstream,"
>> because what
>> people do the most is usually what''s supported the best.
>
> If I may be the contrarian, I hope Matt keeps experimenting with this, 
> files bugs, and they get fixed.  His use case is very compelling - I 
> know lots of SOHO folks who could really use a NAS where this
''just
> worked''
>
> The ZFS team has done well by thinking liberally about conventional 
> assumptions.
>
> -Bill
>
My plan indeed is to continue with this setup (going to upgrade to 138 
to resolve my reboot issue). This particular use case for me is 
definitely compelling, the simply fact that I can plug my USB drive into 
another laptop and boot into the exact same environment is reason enough 
for me to continue with this setup and see how things go.

Mind you doing occasional zfs send''s to another backup drive might be 
something I''ll do aswell :-)

cheers

Matt

>>>>> "bh" == Brandon High <bhigh at freaks.com>
writes:
    bh> The drive should be on the same USB port because the device
    bh> path is saved in the zpool.cache. If you removed the
    bh> zpool.cache, it wouldn''t matter where the drive was plugged
    bh> in.

I thought it was supposed to go by devid.

There was a bug a while ago that Solaris won''t calculate devid for
devices that say over SCSI they are ``removeable'''' because, in
the
sense that a DynaMO or DVD-R is ``removeable'''', the serial
number
returned by various identity commands or mode pages isn''t bound to any
set of stored bits, and the way devid''s are used throughout Solaris
means they are like a namespace or an array-of for a set of bit-stores
so it''s not appropriate for a DVD-R drive to have a devid.  A DVD disc
could have one, though---in fact a release of a pressed disc could
appropriately have a non-serialized devid.  However USB stick
designers used to working with Microsoft don''t bother to think through
how the SCSI architecture should work in a sane world because they are
used to reading chatty-idiot Microsoft manuals, so they fill out the
page like a beaurocratic form with whatever feels appropriate and mark
USB sticks ``removeable'''', which according to the standard and
to a
sane implementer is a warning that the virtual SCSI disk attached to
the virtual SCSI host adapter inside the USB pod might be soldered to
removeable FLASH chips.  It''s quite stupid because before the OS has
even determined what kind of USB device is plugged in, it already
knows the device is removeable in that sense, just like it knows
hot-swap SATA is removeable.  USB is no more removeable, even in
practical use, than SATA.  (eSATA!  *slap*) Even in the case of CF
readers, it''s probably wrong most of the time to set the removeable
SCSI flag because the connection that''s severable is between the
virtual SCSI adapter in the ``reader'''' and the virtual SCSI
disk in
the CF/SD/... card, while the removeable flag indicates severability
between SCSI disk and storage medium.  In the CF/SD/... reader case
the serial number in the IDENTIFY command or mode pages will come from
CF/SD/... and remain bound to the bits.  The only case that might call
for setting the bit is where the adapter is synthesizing a fake mode
page where the removeable bit appears, but even then the bit should be
clear so long as any serialized fields in other commands and mode
pages are still serialized somehow (whether synthesized or not).
Actual removeable in-the-scsi-standard''s-sense HARD DISK drives mostly
don''t exist, and real removeable things in the real world attach as
optical where an understanding of their removeability is embedded in
the driver: ANYTHING the cd driver attaches will be treated
removeable.

consequently the bit is useless to the way solaris is using it, and
does little more than break USB support in ways like this, but the
developers refuse to let go of their dreams about what the bit was
supposed to mean even though a flood of reality has guaranteed at this
point their dream will never come true.  I think there was some
magical simon-sez flag they added to /kernel/drv/whatever.conf so the
bug could be closed, so you might go hunting for that flag in which
they will surely want you to encode in a baroque case-sensitive
undocumented notation that ``The Microtraveler model 477217045 serial
80502813 attached to driver/hub/hub/port/function has a LYING
REMOVEABLE FLAG'''', but maybe you can somehow set it to
''*'' and rejoin
reality.  Still this won''t help you on livecd''s. 
It''s probably wiser
to walk away from USB unless/until there''s a serious will to adopt the
practical mindset needed to support it reasonably.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100510/33b7a23d/attachment.bin>

On Mon, May 10, 2010 at 11:04 AM, Miles Nordin <carton at ivy.net>
wrote:
>>>>>> "bh" == Brandon High <bhigh at
freaks.com> writes:
> ? ?bh> The drive should be on the same USB port because the device
> ? ?bh> path is saved in the zpool.cache. If you removed the
>
> I thought it was supposed to go by devid.
zpool.cache saves the device path to make importing pools faster. It
would be nice if there was a boot flag you could give it to ignore the
file...

zdb -C shows the contents of zpool.cache. It''s got guid, path, devid,
phys_path and other info in it.

-B

-- 
Brandon High : bhigh at freaks.com

On May 11, 2010, at 3:26 PM, Brandon High wrote:
> On Mon, May 10, 2010 at 11:04 AM, Miles Nordin <carton at ivy.net>
wrote:
>>>>>>> "bh" == Brandon High <bhigh at
freaks.com> writes:
>>    bh> The drive should be on the same USB port because the device
>>    bh> path is saved in the zpool.cache. If you removed the
>> 
>> I thought it was supposed to go by devid.
> 
> zpool.cache saves the device path to make importing pools faster. It
> would be nice if there was a boot flag you could give it to ignore the
> file...
boot single user and mv it (just like we''ve done for fstab/vfstab for
the past 30+ years :-)
> zdb -C shows the contents of zpool.cache. It''s got guid, path,
devid,
> phys_path and other info in it.

yep.
 -- richard

-- 
ZFS storage and performance consulting at http://www.RichardElling.com

On Tue, May 11, 2010 at 8:17 PM, Richard Elling
<richard.elling at gmail.com> wrote:
> boot single user and mv it (just like we''ve done for fstab/vfstab
for
> the past 30+ years :-)
It would be nice to have a grub menu item that ignores the cache, so
if you know you''ve removed a USB drive, you don''t need to muck
about
in single user then reboot again.

But who needs usability? This is unix, man.

-B

-- 
Brandon High : bhigh at freaks.com

On May 11, 2010, at 8:31 PM, Brandon High wrote:
> On Tue, May 11, 2010 at 8:17 PM, Richard Elling
> <richard.elling at gmail.com> wrote:
>> boot single user and mv it (just like we''ve done for
fstab/vfstab for
>> the past 30+ years :-)
> 
> It would be nice to have a grub menu item that ignores the cache, so
> if you know you''ve removed a USB drive, you don''t need to
muck about
> in single user then reboot again.
> 
> But who needs usability? This is unix, man.
I must have missed something.  For the past few years I have routinely
booted with unimportable pools because I often use ramdisks.  Sure,
I get FMA messages, but that doesn''t affect the boot. OTOH, I
don''t try
to "backup" using mirrors.
 -- richard

-- 
ZFS storage and performance consulting at http://www.RichardElling.com

On 10 maj 2010, at 20.04, Miles Nordin wrote:
>>>>>> "bh" == Brandon High <bhigh at
freaks.com> writes:
> 
>    bh> The drive should be on the same USB port because the device
>    bh> path is saved in the zpool.cache. If you removed the
>    bh> zpool.cache, it wouldn''t matter where the drive was
plugged
>    bh> in.
> 
> I thought it was supposed to go by devid.
> 
> There was a bug a while ago that Solaris won''t calculate devid for
> devices that say over SCSI they are ``removeable''''
because, in the
> sense that a DynaMO or DVD-R is ``removeable'''', the
serial number
> returned by various identity commands or mode pages isn''t bound to
any
> set of stored bits, and the way devid''s are used throughout
Solaris
> means they are like a namespace or an array-of for a set of bit-stores
> so it''s not appropriate for a DVD-R drive to have a devid.  A DVD
disc
> could have one, though---in fact a release of a pressed disc could
> appropriately have a non-serialized devid.  However USB stick
> designers used to working with Microsoft don''t bother to think
through
> how the SCSI architecture should work in a sane world because they are
> used to reading chatty-idiot Microsoft manuals, so they fill out the
> page like a beaurocratic form with whatever feels appropriate and mark
> USB sticks ``removeable'''', which according to the
standard and to a
> sane implementer is a warning that the virtual SCSI disk attached to
> the virtual SCSI host adapter inside the USB pod might be soldered to
> removeable FLASH chips.  It''s quite stupid because before the OS
has
> even determined what kind of USB device is plugged in, it already
> knows the device is removeable in that sense, just like it knows
> hot-swap SATA is removeable.  USB is no more removeable, even in
> practical use, than SATA.  (eSATA!  *slap*) Even in the case of CF
> readers, it''s probably wrong most of the time to set the
removeable
> SCSI flag because the connection that''s severable is between the
> virtual SCSI adapter in the ``reader'''' and the virtual
SCSI disk in
> the CF/SD/... card, while the removeable flag indicates severability
> between SCSI disk and storage medium.  In the CF/SD/... reader case
> the serial number in the IDENTIFY command or mode pages will come from
> CF/SD/... and remain bound to the bits.  The only case that might call
> for setting the bit is where the adapter is synthesizing a fake mode
> page where the removeable bit appears, but even then the bit should be
> clear so long as any serialized fields in other commands and mode
> pages are still serialized somehow (whether synthesized or not).
> Actual removeable in-the-scsi-standard''s-sense HARD DISK drives
mostly
> don''t exist, and real removeable things in the real world attach
as
> optical where an understanding of their removeability is embedded in
> the driver: ANYTHING the cd driver attaches will be treated
> removeable.
> 
> consequently the bit is useless to the way solaris is using it, and
> does little more than break USB support in ways like this, but the
> developers refuse to let go of their dreams about what the bit was
> supposed to mean even though a flood of reality has guaranteed at this
> point their dream will never come true.  I think there was some
> magical simon-sez flag they added to /kernel/drv/whatever.conf so the
> bug could be closed, so you might go hunting for that flag in which
> they will surely want you to encode in a baroque case-sensitive
> undocumented notation that ``The Microtraveler model 477217045 serial
> 80502813 attached to driver/hub/hub/port/function has a LYING
> REMOVEABLE FLAG'''', but maybe you can somehow set it to
''*'' and rejoin
> reality.  Still this won''t help you on livecd''s. 
It''s probably wiser
> to walk away from USB unless/until there''s a serious will to adopt
the
> practical mindset needed to support it reasonably.
I''m sorry if I am slow, but I don''t get it;
Whys isn''t devid calculated for removable devices?
What does the serial number has to do with devid?

/ragge

On 12 maj 2010, at 05.31, Brandon High wrote:
> On Tue, May 11, 2010 at 8:17 PM, Richard Elling
> <richard.elling at gmail.com> wrote:
>> boot single user and mv it (just like we''ve done for
fstab/vfstab for
>> the past 30+ years :-)
> 
> It would be nice to have a grub menu item that ignores the cache, so
> if you know you''ve removed a USB drive, you don''t need to
muck about
> in single user then reboot again.
> 
> But who needs usability? This is unix, man.
But that shouldn''t ever be needed, except in this case where there
is a bug, should it?

Implementing options to work around every thinkable bug would be an
interesting problem, but I believe the problem is to hard to be
solved in reasonable time.

/ragge

On Tue, May 11, 2010 at 10:13 PM, Richard Elling
<richard.elling at gmail.com> wrote:
>> But who needs usability? This is unix, man.
>
> I must have missed something. ?For the past few years I have routinely
> booted with unimportable pools because I often use ramdisks. ?Sure,
> I get FMA messages, but that doesn''t affect the boot. OTOH, I
don''t try
> to "backup" using mirrors.
If you boot from usb and move your rpool from one port to another, you
can''t boot. If you plug your boot sata drive into a different port on
the motherboard, you can''t boot. Apparently if you are missing a
device from your rpool mirror, you can''t boot. I haven''t tried
booting
in single user mode in any of these cases because it''s been easier to
undo the change.

If it was possible to pass in a flag from grub to ignore the cache, it
would make life a little easier in such cases.

-B

-- 
Brandon High : bhigh at freaks.com

On May 12, 2010, at 3:23 AM, Brandon High wrote:
> On Tue, May 11, 2010 at 10:13 PM, Richard Elling <richard.elling at
gmail.com
> > wrote:
>>> But who needs usability? This is unix, man.
>>
>> I must have missed something.  For the past few years I have  
>> routinely
>> booted with unimportable pools because I often use ramdisks.  Sure,
>> I get FMA messages, but that doesn''t affect the boot. OTOH, I
don''t
>> try
>> to "backup" using mirrors.
>
> (..)
> If it was possible to pass in a flag from grub to ignore the cache, it
> would make life a little easier in such cases.
Recently I have been working on a zpool that refuses to import. During  
my work I had to boot the server many times in failsafe mode to be  
able to remove the zpool.cache file, so Brandon''s suggestions sounds  
very reasonable at first.

However, I realized that if you import using "zpool import -R /altroot  
your_pool" -- it does NOT create a new zpool.cache.

So, as long as you use -R, you can safely import pools without  
creating a new zpool.cache file and your next reboot will not screw up  
the system.

Basically there''s no real need to a grub option (actually for a kernel
parameter) -- if you have a problem, you go failsafe mode and remove  
the file, then in your tests you attempt to import using -R so the  
cache is not re-created and you don''t need to go into failsafe mode  
ever again.

best regards,
Eduardo Bragatto.

>>>>> "bh" == Brandon High <bhigh at freaks.com>
writes:
    bh> If you boot from usb and move your rpool from one port to
    bh> another, you can''t boot. If you plug your boot sata drive
into
    bh> a different port on the motherboard, you can''t
    bh> boot. Apparently if you are missing a device from your rpool
    bh> mirror, you can''t boot.

yeah, this is retarded and should get fixed.

    bh> zpool.cache saves the device path to make importing pools
    bh> faster. It would be nice if there was a boot flag you could
    bh> give it to ignore the file...

I''ve no doubt this is true but ISTR it''s not related to the
booting
problem above becuase I do not think zpool.cache is used to find the
root pool.  It''s only used for finding other pools.  ISTR the root
pool is found through devid''s that grub reads from the label on the
BIOS device it picks, and then passes to the kernel.  note that
zpool.cache is ON THE POOL, so it can''t be used to find the pool (ok,
it can---on x86 it can be sync''ed into the boot archive, and on SPARC
it can be read through the PROM---but although I could be wrong ISTR
this is not what''s actually done).

I think you''ll find you CAN move drives among sata ports, just not
among controller types, because the devid is a blob generated by the
disk driver, and pci-ide and AHCI will yeild up different devid''s for
the same disk.  Grub never calculates a devid, just reads one from the
label (reads a devid that some earlier kernel got from pci-ide or ahci
and wrote into the label).  so when ports and device names change,
rewriting labels is helpful but not urgent.  When disk drivers change,
rewriting labels is urgent.

yeah, the fact that ramdisk booting isn''t possible with opensolaris
makes tihs whole situation a lot more serious than it was back when
SXCE was still available for download.  Is there any way to make a
devid-proof rescue boot option?  Is there a way to make grub boot an
iso image off the hard disk for example?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100512/9b6970bf/attachment.bin>

On Wed, May 12, 2010 at 1:39 PM, Miles Nordin <carton at ivy.net>
wrote:
> root pool. ?It''s only used for finding other pools. ?ISTR the root
> pool is found through devid''s that grub reads from the label on
the
> BIOS device it picks, and then passes to the kernel. ?note that
Ok, that makes more sense with what I''ve experienced. The devid for a
USB device must change as it moves from port to port. When moving a
USB stick I built on one system to another, I had to run bootadm
update-archive from the LiveCD.
> I think you''ll find you CAN move drives among sata ports, just not
> among controller types, because the devid is a blob generated by the
The testing that I did with moving the sata port may have been across
controllers.
> devid-proof rescue boot option? ?Is there a way to make grub boot an
> iso image off the hard disk for example?
It''s probably possible to use the LiveCD / LiveUSB with a custom grub
menu item.

-B

-- 
Brandon High : bhigh at freaks.com

>>>>> "bh" == Brandon High <bhigh at freaks.com>
writes:
    bh> The devid for a USB device must change as it moves from port
    bh> to port.

I guess it was tl;dr the first time I said this, but:

  the old theory was that a USB device does not get a devid because it
  is marked ``removeable'''' in some arcane SCSI page, for the
same
  reason it doesn''t make sense to give a CD-ROM a devid because its
  medium can be removed.  

I don''t know if this has changed, or if it''s even
what''s really going
on.  but like I said without the ramdisk boot option it''s more
important to fix this type of problem, so if someone has a workaround
please share!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100513/97d2d079/attachment.bin>

On 12 maj 2010, at 22.39, Miles Nordin wrote:
>>>>>> "bh" == Brandon High <bhigh at
freaks.com> writes:
> 
>    bh> If you boot from usb and move your rpool from one port to
>    bh> another, you can''t boot. If you plug your boot sata
drive into
>    bh> a different port on the motherboard, you can''t
>    bh> boot. Apparently if you are missing a device from your rpool
>    bh> mirror, you can''t boot.
> 
> yeah, this is retarded and should get fixed.
> 
>    bh> zpool.cache saves the device path to make importing pools
>    bh> faster. It would be nice if there was a boot flag you could
>    bh> give it to ignore the file...
> 
> I''ve no doubt this is true but ISTR it''s not related to
the booting
> problem above becuase I do not think zpool.cache is used to find the
> root pool.  It''s only used for finding other pools.  ISTR the root
> pool is found through devid''s that grub reads from the label on
the
> BIOS device it picks, and then passes to the kernel.  note that
> zpool.cache is ON THE POOL, so it can''t be used to find the pool
(ok,
> it can---on x86 it can be sync''ed into the boot archive, and on
SPARC
> it can be read through the PROM---but although I could be wrong ISTR
> this is not what''s actually done).
> 
> I think you''ll find you CAN move drives among sata ports, just not
> among controller types, because the devid is a blob generated by the
> disk driver, and pci-ide and AHCI will yeild up different devid''s
for
> the same disk.  Grub never calculates a devid, just reads one from the
> label (reads a devid that some earlier kernel got from pci-ide or ahci
> and wrote into the label).  so when ports and device names change,
> rewriting labels is helpful but not urgent.  When disk drivers change,
> rewriting labels is urgent.
Are you talking about booting here?

Because with the OS booted, the devid should only be a hint, and if it
is found to not be correct the the disks should be found with their
guids by searching all /dev/dsk/* devices, shouldn''t they?

So, if everything worked as expected, I don''t see any reason at all to
ever have to remove/ignore the zpool.cache. (Well - except for the case
when you don''t want the OS to import pools that where imported before
shutdown/crash - but that hasn''t been discussed here yet.)

Finding disks at boot is a very different animal and have it''s
limitations, though GRUB tries to get work around some of them.

Am I missing something?

/ragge