zfs discuss - Jan 2010 - $100 SSD = >5x faster dedupe

I''ve written about my slow-to-dedupe RAIDZ.

After a week of.....waiting....I finally bought a little $100 30G OCZ
Vertex and plugged it in as a cache.

After <2 hours of warmup, my zfs send/receive rate on the pool
is
>16MB/sec (reading and writing each at 16MB as measured by zpool
iostat).
That''s up from <3MB/sec, with a RAM-only cache on a 6GB machine.

The SSD has about 8GB utilized right now, and the L2ARC benefit is amazing.
Quite an amazing improvement for $100...recommend you don''t dedupe
without one.

mike

Make that 25MB/sec, and rising...
So it''s 8x faster now.

mike
-- 
This message posted from opensolaris.org

Are you using the SSD for l2arc or zil or both?
-- 
This message posted from opensolaris.org

Just l2arc. Guess I can always repartition later.

mike


On Sun, Jan 3, 2010 at 11:39 AM, Jack Kielsmeier <jackal at netins.net>
wrote:
> Are you using the SSD for l2arc or zil or both?
> --
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

> Just l2arc. Guess I can always repartition later.
> 
> mike
> 
> 
> On Sun, Jan 3, 2010 at 11:39 AM, Jack Kielsmeier
> <jackal at netins.net> wrote:
> > Are you using the SSD for l2arc or zil or both?
> > --
> > This message posted from opensolaris.org
> > _______________________________________________
> > zfs-discuss mailing list
> > zfs-discuss at opensolaris.org
> >
> http://mail.opensolaris.org/mailman/listinfo/zfs-discu
> ss
> >
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discu
> ss
This is good to know. I have considered purchasing a SSD, but have not wanted to
make the investment not knowing how much it would help. It is suggested not to
put zil on a device external to the disks in the pool unless you mirror the zil
device. This is suggested to prevent data loss if the zil device dies.

With L2arc, no such redundancy is needed. So, with a $100 SSD, if you can get 8x
the performance out of your dedup''d dataset, and you don''t
have to worry about "what if the device fails", I''d call that
an awesome investment.
-- 
This message posted from opensolaris.org

On Sun, 3 Jan 2010, Jack Kielsmeier wrote:
>
> help. It is suggested not to put zil on a device external to the 
> disks in the pool unless you mirror the zil device. This is 
> suggested to prevent data loss if the zil device dies.
The reason why it is suggested that the intent log reside in the same 
chassis where the pool disks live is so that it is easier to move the 
pool to a different computer if need be.  This is an independent 
problem from mirroring.  If the intent log device is easily moved to 
the new computer, then there is not really a problem.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

> On Sun, 3 Jan 2010, Jack Kielsmeier wrote:
> >
> > help. It is suggested not to put zil on a device
> external to the 
> > disks in the pool unless you mirror the zil device.
> This is 
> > suggested to prevent data loss if the zil device
> dies.
> 
> The reason why it is suggested that the intent log
> reside in the same 
> chassis where the pool disks live is so that it is
> easier to move the 
> pool to a different computer if need be.  This is an
> independent 
> problem from mirroring.  If the intent log device is
> easily moved to 
> the new computer, then there is not really a problem.
> 
> Bob
> --
> Bob Friesenhahn
> bfriesen at simple.dallas.tx.us,
> http://www.simplesystems.org/users/bfriesen/
> GraphicsMagick Maintainer,
>    http://www.GraphicsMagick.org/
> ____________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discu
> ss
Ah, it looks like the information I was recalling is now outdated with the zil:

http://bugs.opensolaris.org/view_bug.do?bug_id=6707530
(this now appears to be fixed, and have been fixed for some time (since snv_96).
-- 
This message posted from opensolaris.org

On Jan 3, 2010, at 4:05 PM, Jack Kielsmeier wrote:
>
> With L2arc, no such redundancy is needed. So, with a $100 SSD, if  
> you can get 8x the performance out of your dedup''d dataset, and
you
> don''t have to worry about "what if the device fails",
I''d call that
> an awesome investment.
AFAIK, the L2ARC is not persistent. This is not much of a problem for  
servers,
which tend to stay up for long periods of time.

Upon reboot, there will be a lot of random reads as the DTT is reloaded.
  -- richard

On Sun, Jan 03, 2010 at 08:26:47PM -0800, Richard Elling
wrote:
> On Jan 3, 2010, at 4:05 PM, Jack Kielsmeier wrote:
> >
> >With L2arc, no such redundancy is needed. So, with a $100 SSD, if  
> >you can get 8x the performance out of your dedup''d dataset,
and you
> >don''t have to worry about "what if the device
fails", I''d call that
> >an awesome investment.
> 
> AFAIK, the L2ARC is not persistent.
Correct, until this CR is fixed:

	6662467 L2ARC content to survive reboots
	http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6662467

Please don''t ask for a specific ETA - I can''t give one yet.

Brendan
> This is not much of a problem for  
> servers,
> which tend to stay up for long periods of time.
> 
> Upon reboot, there will be a lot of random reads as the DTT is reloaded.
>  -- richard
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
-- 
Brendan Gregg, Sun Microsystems Fishworks.    http://blogs.sun.com/brendan

On Thu, Dec 31, 2009 at 9:37 PM, Michael Herf <mbherf at gmail.com> wrote:
> I''ve written about my slow-to-dedupe RAIDZ.
>
> After a week of.....waiting....I finally bought a little $100 30G OCZ
> Vertex and plugged it in as a cache.
>
> After <2 hours of warmup, my zfs send/receive rate on the pool is
> >16MB/sec (reading and writing each at 16MB as measured by zpool
> iostat).
> That''s up from <3MB/sec, with a RAM-only cache on a 6GB
machine.
>
> The SSD has about 8GB utilized right now, and the L2ARC benefit is amazing.
> Quite an amazing improvement for $100...recommend you don''t dedupe
without
> one.
>
> mike
> _______________________________________________
>
this is nice to know.  I just  ordered 3 64 gb kingston ssd''s (model
snv125-S2)

I was going to use 2 for the root pool and 1 for L2ARC (i was also
considering slicing the 2 for rpool into 20 gb/42 gb and using the other 2
slices for l2arc as well...)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100104/dcdeacc9/attachment.html>

Michael Herf wrote:
> I''ve written about my slow-to-dedupe RAIDZ.
> 
> After a week of.....waiting....I finally bought a
> little $100 30G OCZ
> Vertex and plugged it in as a cache.
> 
> After <2 hours of warmup, my zfs send/receive rate on
> the pool is
> >16MB/sec (reading and writing each at 16MB as
> measured by zpool
> iostat).
> That''s up from <3MB/sec, with a RAM-only cache on a
> 6GB machine.
> 
> The SSD has about 8GB utilized right now, and the
> L2ARC benefit is amazing.
> Quite an amazing improvement for $100...recommend you
> don''t dedupe without one.
I did something similar, but with a SCSI drive.  I keep a large external USB
drive as a "last ditch" recovery pool which is synchronized hourly
from the main pool.  Kind of like a poor man''s tape backup.

When I enabled dedup=verify on the USB pool, the sync performance went south,
because the USB drive had to read stripes to verify that they were actual dups. 
Since I had an unused 146GB SCSI drive plugged in, I made the SCSI drive L2ARC
for the USB pool.  Write performance skyrocketed by a factor of 6 and is now
faster than when there was no dedupe enabled.

Marty
-- 
This message posted from opensolaris.org

Marty Scholes wrote:
>
> I did something similar, but with a SCSI drive.  I keep a large external
USB drive as a "last ditch" recovery pool which is synchronized hourly
from the main pool.  Kind of like a poor man''s tape backup.
>
> When I enabled dedup=verify on the USB pool, the sync performance went
south, because the USB drive had to read stripes to verify that they were actual
dups.
Why did you set dedup=verify on the USB pool?

-- 
Ian.

Ian wrote:
> Why did you set dedup=verify on the USB pool?
Because that is my last-ditch copy of the data and MUST be correct.  At the same
time, I want to cram as much data as possible into the pool.

If I ever go to the USB pool, something has already gone horribly wrong and I am
desperate.  I can''t comprehend the anxiety I would have if one or more
stripes had a birthday collision giving me silent data corruption that I found
out about months or years later.

It''s probably paranoid, but a level of paranoia I can live with.

Good question, by the way.
-- 
This message posted from opensolaris.org

Sorry to hijack the thread, but can you explain your setup? Sounds
interesting, but need more info...

Thanks!

--Tiernan

On Jan 7, 2010 11:56 PM, "Marty Scholes" <martyscholes at
yahoo.com> wrote:

Ian wrote: > Why did you set dedup=verify on the USB pool?
Because that is my last-ditch copy of the data and MUST be correct.  At the
same time, I want to cram as much data as possible into the pool.

If I ever go to the USB pool, something has already gone horribly wrong and
I am desperate.  I can''t comprehend the anxiety I would have if one or
more
stripes had a birthday collision giving me silent data corruption that I
found out about months or years later.

It''s probably paranoid, but a level of paranoia I can live with.

Good question, by the way.

-- This message posted from
opensolaris.org_______________________________________________

zfs-discuss mailing list zfs-discuss at opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zf...
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100108/2bd2dc99/attachment.html>

--- On Thu, 1/7/10, Tiernan OToole <lsmartman at gmail.com>
wrote:
> Sorry to hijack the thread, but can you
> explain your setup? Sounds interesting, but need more
> info...
This is just a home setup to amuse me and placate my three boys, each of whom
has several Windows instances running under Virtualbox.

Server is a Sun v40z: quad 2.4 GHz Opteron with 16GB.  Internal bays hold a pair
of 73GB drives as a mirrored rpool and a pair of 36GB drives for spares to the
array plus a 146GB drive I use as cache to the usb pool (a single 320GB sata
drive).

The array is an HP MSA30 with 14x36GB drives configured as RAIDZ3 using the
spares listed above with auto snapshots as the tank pool. Tank is synchronized
hourly to the usb pool.

It''s all connected via four HP 4000M switches (one at the server and
one at each workstation) which are meshed via gigabit fiber.

Two workstations are triple-head sunrays.

One station is a single sunray 150 integrated unit.

This is a work in progress with plenty of headroom to grow.  I started the build
in November and have less than $1200 into it so far.

Thanks for letting me hijack the thread by sharing!

Cheers,
Marty