I have an approx 700GB (of data) FS that I had dedup turned on for. (See previous posts.) I turned on dedup after the FS was populated, and was not sure dedup was working. I had another copy of the data, so I removed the data, and then tried to destroy the snapshots I had taken. The first two didn''t take too long, but the last one (the oldest) has taken literally hours now. I''ve rebooted and tried starting over, but it hasn''t made a difference. I realize that I did things in the wrong order. I should have removed the oldest snapshot first, on to the newest, and then removed the data in the FS itself. But still, it shouldn''t take hours, should it? I made sure the machine was otherwise idle, and did an ''iostat'', which shows about 5KB/sec reads and virtually no writes to the pool. Any ideas where to look? I''d just remove the FS entirely at this point, but I''d have to destroy the snapshot first, so I''m in the same boat, yes? TIA, Paul
This sounds like the known issue about the dedupe map not fitting in ram. When blocks are freed, dedupe scans the whole map to ensure each block is not is use before releasing it. This takes a veeery long time if the map doesn''t fit in ram. If you can try adding more ram to the system. -- This message posted from opensolaris.org
7:51pm, Richard Jahnel wrote:> This sounds like the known issue about the dedupe map not fitting in ram. > > When blocks are freed, dedupe scans the whole map to ensure each block is not is use before releasing it. This takes a veeery long time if the map doesn''t fit in ram. > > If you can try adding more ram to the system. > --Thanks for the info. Unfortunately, I''m not sure I''ll be able to add more RAM any time soon. But I''m certainly going to try, as this is the primary backup server for our Oracle databases. Thanks again, Paul PS It''s got 8GB right now. You think doubling that to 16GB would cut it? Is there a way to see how big the map is, anyway?
On 04/14/10 19:51, Richard Jahnel wrote:> This sounds like the known issue about the dedupe map not fitting in ram.Indeed, but this is not correct:> When blocks are freed, dedupe scans the whole map to ensure each block is not is use before releasing it.That''s not correct. dedup uses a data structure which is indexed by the hash of the contents of each block. That hash function is effectively random, so it needs to access a *random* part of the map for each free which means that it (as you correctly stated):> ... takes a veeery long time if the map doesn''t fit in ram. > > If you can try adding more ram to the system.Adding a flash-based ssd as an cache/L2ARC device is also very effective; random i/o to ssd is much faster than random i/o to spinning rust. - Bill
On Wed, Apr 14, 2010 at 09:04:50PM -0500, Paul Archer wrote:> I realize that I did things in the wrong order. I should have removed the > oldest snapshot first, on to the newest, and then removed the data in the > FS itself.For the problem in question, this is irrelevant. As discussed in the rest of the thread, you''ll hit this when doing anyting that requires updating the ref counts on a large number of DDT entries. The only way snapshot order can really make a big difference is if you arrange for it to do so in advance. If you know you have a large amount of data to delete from a filesystem: - snapshot at the start - start deleting - snapshot fast and frequently during the deletion - let the snapshots go, later, at a controlled pace, to limit the rate of actual block frees. -- Dan. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100415/3f831ee3/attachment.bin>
Thank you for the corrections. Also I forgot about using an SSD to assist. My bad. =) -- This message posted from opensolaris.org
3:26pm, Daniel Carosone wrote:> On Wed, Apr 14, 2010 at 09:04:50PM -0500, Paul Archer wrote: >> I realize that I did things in the wrong order. I should have removed the >> oldest snapshot first, on to the newest, and then removed the data in the >> FS itself. > > For the problem in question, this is irrelevant. As discussed in the > rest of the thread, you''ll hit this when doing anyting that requires > updating the ref counts on a large number of DDT entries. > > The only way snapshot order can really make a big difference is if you > arrange for it to do so in advance. If you know you have a large > amount of data to delete from a filesystem: > - snapshot at the start > - start deleting > - snapshot fast and frequently during the deletion > - let the snapshots go, later, at a controlled pace, to limit the > rate of actual block frees. >That''s a great idea. I wish I had thought of/heard of it before I deleted the data in my dedup''ed FS. Paul