zfs discuss - Sep 2010 - Best practice for Sol10U9 ZIL -- mirrored or not?

Best practice in Solaris 10 U8 and older was to use a mirrored ZIL.

With the ability to remove slog devices in Solaris 10 U9, we''re
thinking we may get more bang for our buck to use two slog devices for
improved IOPS performance instead of needing the redundancy so much.

Any thoughts on this?

If we lost our slog devices and had to reboot, would the system come up
(eg could we "remove" failed slog devices from the zpool so the zpool
would come online..)

Thanks,
Ray

+------------------------------------------------------------------------------
| On 2010-09-16 18:08:46, Ray Van Dolson wrote:
| 
| Best practice in Solaris 10 U8 and older was to use a mirrored ZIL.
| 
| With the ability to remove slog devices in Solaris 10 U9, we''re
| thinking we may get more bang for our buck to use two slog devices for
| improved IOPS performance instead of needing the redundancy so much.
| 
| Any thoughts on this?
| 
| If we lost our slog devices and had to reboot, would the system come up
| (eg could we "remove" failed slog devices from the zpool so the
zpool
| would come online..)

The ability to remove the slogs isn''t really the win here,
it''s import -F. The
problem is: If the ZIL dies, you will lose whatever writes were in-flight.

I''ve just deployed some SSD ZIL (on U9), and decided to mirror them.
Cut the
two SSDs into 1GB and 31GB partitions, mirrored the two 1GBs as slog and have
the two 31GB as L2ARC.

So far extremely happy with it. Running a scrub during production hours,
before, was unheard of. (And, well, "production" for mail storage is
basically
all hours, so.)

As for running non-mirrored slogs... dunno. Our customers would be pretty
pissed if we lost any mail, so I doubt I will do so. My SSDs were only $90
each, though, so cost is hardly a factor for us.

Cheers.
-- 
bdha
cyberpunk is dead. long live cyberpunk.

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Bryan Horstmann-Allen
>
> The ability to remove the slogs isn''t really the win here,
it''s import
> -F. The
Disagree.

Although I agree the -F is important and good, I think the log device
removal is the main win.  Prior to log device removal, if you lose your
slog, then you lose your whole pool, and probably your system halts (or does
something equally bad, which isn''t strictly halting).  Therefore you
want
your slog to be as redundant as the rest of your pool.

With log device removal, if you lose a slog while the system is up, worst
case is performance degradation.

With log device removal, there''s only one thing you have to worry
about:
Your slog goes bad, and undetected.  So the system keeps writing to it,
unaware that it will never be able to read, and therefore when you get a
system crash, and for the first time your system tries to read that device,
you lose information.  Not your whole pool.  You lose up to 30 sec of writes
that the system thought it wrote, but never did.  You require the -F to
import.

Historically, people always recommend mirroring your log device, even with
log device removal, to protect against the above situation.  But in a recent
conversation including Neil, it seems there might be a bug which causes the
log device mirror to be ignored during import, thus rendering the mirror
useless in the above situation.

Neil, or anyone, is there any confirmation or development on that bug?

Given all of this, I would say it''s recommended to forget about
mirroring
log devices for now.  In the past, the recommendation was "Yes
mirror."
Right now, it''s "No don''t mirror," and after the bug
is fixed, the
recommendation will again become "Yes, mirror."

On 09/17/10 06:24, Edward Ned Harvey wrote:
>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
>> bounces at opensolaris.org] On Behalf Of Bryan Horstmann-Allen
>>
>> The ability to remove the slogs isn''t really the win here,
it''s import
>> -F. The
>>     
>
> Disagree.
>
> Although I agree the -F is important and good, I think the log device
> removal is the main win.  Prior to log device removal, if you lose your
> slog, then you lose your whole pool, and probably your system halts (or
does
> something equally bad, which isn''t strictly halting).  Therefore
you want
> your slog to be as redundant as the rest of your pool.
>
> With log device removal, if you lose a slog while the system is up, worst
> case is performance degradation.
>
> With log device removal, there''s only one thing you have to worry
about:
> Your slog goes bad, and undetected.  So the system keeps writing to it,
> unaware that it will never be able to read, and therefore when you get a
> system crash, and for the first time your system tries to read that device,
> you lose information.  Not your whole pool.  You lose up to 30 sec of
writes
> that the system thought it wrote, but never did.  You require the -F to
> import.
>   
The default is  now 5 seconds (zfs_txg_timeout).
> Historically, people always recommend mirroring your log device, even with
> log device removal, to protect against the above situation.  But in a
recent
> conversation including Neil, it seems there might be a bug which causes the
> log device mirror to be ignored during import, thus rendering the mirror
> useless in the above situation.
>
> Neil, or anyone, is there any confirmation or development on that bug?
>   
I believe George responded on that thread that we do handle log mirrors 
correctly.
That is, if one side fails to checksum a block we do indeed check the 
other side.
I should have been more cautious with my concern. I think I said I
don''t
know if we handle
it correctly, and George confirmed we do. Sorry for the false alarm.
> Given all of this, I would say it''s recommended to forget about
mirroring
> log devices for now.  In the past, the recommendation was "Yes
mirror."
> Right now, it''s "No don''t mirror," and after
the bug is fixed, the
> recommendation will again become "Yes, mirror."
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100917/ce955cbb/attachment.html>

> From: Neil Perrin [mailto:neil.perrin at oracle.com]
> 
> > you lose information.  Not your whole pool.  You lose up to 
> > 30 sec of writes
> 
> The default is? now 5 seconds (zfs_txg_timeout).
When did that become default?  Should I *ever* say 30 sec anymore?

In my world, the oldest machine is 10u6.  (Except one machine named
"dinosaur" that is sol8)

> I believe George responded on that thread that we do handle log mirrors
> correctly.
> That is, if one side fails to checksum a block we do indeed check the
> other side.
> I should have been more cautious with my concern. I think I said I
> don''t know if we handle
> it correctly, and George confirmed we do. Sorry for the false alarm.
Great.  ;-)  Thank you.

So the recommendation is still to mirror log devices, because the
recommendation will naturally be ultra-conservative.  ;-)  The risk is far
smaller now than it was before.  So make up your own mind.  If you are
willing to risk 5sec or 30sec of data in the situation of (a) undetected
failed log device *and* (b) ungraceful system crash, then you are willing to
run with unmirrored log devices.  In no situation does the filesystem become
inconsistent or corrupt.  In the worst case, you have a filesystem which is
consistent with a valid filesystem state, a few seconds before the system
crash.  (Assuming you have a zpool recent enough to support log device
removal.)

On Sep 17, 2010, at 20:32, Edward Ned Harvey wrote:
> When did that become default?  Should I *ever* say 30 sec anymore?
June 8, 2010, revision  12586:b118bbd65be9:

http://src.opensolaris.org/source/history/onnv/onnv-gate/usr/src/uts/common/fs/zfs/txg.c

On 09/17/10 18:32, Edward Ned Harvey wrote:
>> From: Neil Perrin [mailto:neil.perrin at oracle.com]
>>
>>     
>>> you lose information.  Not your whole pool.  You lose up to 
>>> 30 sec of writes
>>>       
>> The default is  now 5 seconds (zfs_txg_timeout).
>>     
>
> When did that become default?
It was changed more recently than I remember in snv_143 as part of a of 
set of
bug fixes: 6494473, 6743992, 6936821, 6956464. They were integrated on 
6/8/10.
>   Should I *ever* say 30 sec anymore?
>   
Well for versions before snv_143 then 30 seconds is correct.  I was just
giving a heads up that it has changed.
> In my world, the oldest machine is 10u6.  (Except one machine named
> "dinosaur" that is sol8)
>
>
>   
>> I believe George responded on that thread that we do handle log mirrors
>> correctly.
>> That is, if one side fails to checksum a block we do indeed check the
>> other side.
>> I should have been more cautious with my concern. I think I said I
>> don''t know if we handle
>> it correctly, and George confirmed we do. Sorry for the false alarm.
>>     
>
> Great.  ;-)  Thank you.
>
> So the recommendation is still to mirror log devices, because the
> recommendation will naturally be ultra-conservative.  ;-)  The risk is far
> smaller now than it was before.  So make up your own mind.  If you are
> willing to risk 5sec or 30sec of data in the situation of (a) undetected
> failed log device *and* (b) ungraceful system crash, then you are willing
to
> run with unmirrored log devices.  In no situation does the filesystem
become
> inconsistent or corrupt.  In the worst case, you have a filesystem which is
> consistent with a valid filesystem state, a few seconds before the system
> crash.  (Assuming you have a zpool recent enough to support log device
> removal.)
>
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100917/b6fcddb2/attachment.html>

On 09/18/10 04:46 PM, Neil Perrin wrote:
> On 09/17/10 18:32, Edward Ned Harvey wrote:
>>> From: Neil Perrin [mailto:neil.perrin at oracle.com]
>>>
>>>      
>>>> you lose information.  Not your whole pool.  You lose up to
>>>> 30 sec of writes
>>>>        
>>> The default is  now 5 seconds (zfs_txg_timeout).
>>>      
>>
>> When did that become default?
>
> It was changed more recently than I remember in snv_143 as part of a 
> of set of
> bug fixes: 6494473, 6743992, 6936821, 6956464. They were integrated on 
> 6/8/10.
>
>>    Should I *ever* say 30 sec anymore?
>>    
>
> Well for versions before snv_143 then 30 seconds is correct.  I was just
> giving a heads up that it has changed.
>
In the context of this thread, was the change integrated in update 9?

-- 
Ian.

On 09/17/10 23:31, Ian Collins wrote:
> On 09/18/10 04:46 PM, Neil Perrin wrote:
>> On 09/17/10 18:32, Edward Ned Harvey wrote:
>>>> From: Neil Perrin [mailto:neil.perrin at oracle.com]
>>>>
>>>>     
>>>>> you lose information.  Not your whole pool.  You lose up to
>>>>> 30 sec of writes
>>>>>        
>>>> The default is  now 5 seconds (zfs_txg_timeout).
>>>>      
>>>
>>> When did that become default?
>>
>> It was changed more recently than I remember in snv_143 as part of a 
>> of set of
>> bug fixes: 6494473, 6743992, 6936821, 6956464. They were integrated 
>> on 6/8/10.
>>
>>>    Should I *ever* say 30 sec anymore?
>>>    
>>
>> Well for versions before snv_143 then 30 seconds is correct.  I was
just
>> giving a heads up that it has changed.
>>
>
> In the context of this thread, was the change integrated in update 9?
>
- No. It looks like it''s destined for Update 10.