-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all. Sorry if I am asking a FAQ, but I haven''t found a really authorizative answer to this. Most references are old, incomplete or of "I have heard of" kind. I am running Solaris 10 Update 9, and my pool is v22. I recently got two 40GB SSD I plan to add to my pool. My idea is this: 1. Format each SSD as 39GB+1GB. 2. Use the TWO 39GB''s as L2ARC, with no redundancy. 3. Use the TWO 1GB''s as mirrored ZIL. 1GB of ZIL seems more than enough for my needs. I have synchronous writes, but they are, 99.9% of the time, <1MB/s, with occasional bursts. My main concern here is about pool stability if there have any kind of problem with the SSD''s. Especifically: 1. Is the L2ARC data stored in the SSD checksummed?. If so, can I expect that ZFS goes directly to the disk if the checksum is wrong?. 2. Can I import a POOL if one/both L2ARC''s are not available?. 3. What happend if a L2ARC device, suddenly, "dissappears"?. 4. Any idea if L2ARC content will be persistent across system rebooting "eventually"? 5. Can I import a POOL if one/both ZIL devices are not available?. My pool is v22. I know that I can remove ZIL devices since v19, but I don''t know if I can remove them AFTER they are physically unavailable, of before importing the pool (after a reboot). 6. Can I remove a ZIL device after ZFS consider it "faulty"?. 7. What if a ZIL device "dissapears", suddenly?. I know that I could lose "committed" transactions in-fight, but will the machine crash?. Will it fallback to ZIL on harddisk?. 8. Since my ZIL will be mirrored, I assume that the OS will actually will look for transactions to be replayed in both devices (AFAIK, the ZIL chain is considered done when the checksum of the last block is not valid, and I wonder how this interacts with ZIL device mirroring). 9. If a ZIL device mirrored goes offline/online, will it resilver from the other side, or it will simply get new transactions, since old transactions are irrelevant after ?30? seconds?. 10. What happens if my 1GB of ZIL is too optimistic?. Will ZFS use the disks or it will stop writers until flushing ZIL to the HDs?. Anything else I should consider?. As you can see, my concerns concentrate in what happens if the SSDs go bad or "somebody" unplugs them "live". I have backup of (most) of my data, but rebuilding a 12TB pool from backups, in a production machine, in a remote hosting, would be something I rather avoid :-p. I know that hybrid HD+SSD pools were a bit flacky in the past (you lost the ZIL device, you kiss goodbye to your ZPOOL, in the pre-v19 days), and I want to know what terrain I am getting into. PS: I plan to upgrade to S10 U10 when available, and I will upgrade the ZPOOL version after a while. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTlxjxplgi5GaxT1NAQLi9AP/VW2LQqij6y25KQ3c5EDBWvnnL1Z7R65j BJ0N1EbWW6ZdkQ9uFoLNJBVb8xPgwpTOKuy5g8FTwrjs1Sc5a3E3DbRDUg75faE5 4IOgCi0gtIVyrxGEQ2AAhnKHGcto/2gB9Y5KRiibBeysbqNvr0HXQsko7WRauP96 N1L1TqFsN8E=sDRY -----END PGP SIGNATURE-----
Are you truly new to ZFS? Or do you work for NetApp or EMC or somebody else that is curious? - Mike On Aug 29, 2011, at 9:15 PM, Jesus Cea wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi all. Sorry if I am asking a FAQ, but I haven''t found a really > authorizative answer to this. Most references are old, incomplete or > of "I have heard of" kind. > > I am running Solaris 10 Update 9, and my pool is v22. > > I recently got two 40GB SSD I plan to add to my pool. My idea is this: > > 1. Format each SSD as 39GB+1GB. > 2. Use the TWO 39GB''s as L2ARC, with no redundancy. > 3. Use the TWO 1GB''s as mirrored ZIL. > > 1GB of ZIL seems more than enough for my needs. I have synchronous > writes, but they are, 99.9% of the time, <1MB/s, with occasional bursts. > > My main concern here is about pool stability if there have any kind of > problem with the SSD''s. Especifically: > > 1. Is the L2ARC data stored in the SSD checksummed?. If so, can I > expect that ZFS goes directly to the disk if the checksum is wrong?. > > 2. Can I import a POOL if one/both L2ARC''s are not available?. > > 3. What happend if a L2ARC device, suddenly, "dissappears"?. > > 4. Any idea if L2ARC content will be persistent across system > rebooting "eventually"? > > 5. Can I import a POOL if one/both ZIL devices are not available?. My > pool is v22. I know that I can remove ZIL devices since v19, but I > don''t know if I can remove them AFTER they are physically unavailable, > of before importing the pool (after a reboot). > > 6. Can I remove a ZIL device after ZFS consider it "faulty"?. > > 7. What if a ZIL device "dissapears", suddenly?. I know that I could > lose "committed" transactions in-fight, but will the machine crash?. > Will it fallback to ZIL on harddisk?. > > 8. Since my ZIL will be mirrored, I assume that the OS will actually > will look for transactions to be replayed in both devices (AFAIK, the > ZIL chain is considered done when the checksum of the last block is > not valid, and I wonder how this interacts with ZIL device mirroring). > > 9. If a ZIL device mirrored goes offline/online, will it resilver from > the other side, or it will simply get new transactions, since old > transactions are irrelevant after ?30? seconds?. > > 10. What happens if my 1GB of ZIL is too optimistic?. Will ZFS use the > disks or it will stop writers until flushing ZIL to the HDs?. > > Anything else I should consider?. > > As you can see, my concerns concentrate in what happens if the SSDs go > bad or "somebody" unplugs them "live". > > I have backup of (most) of my data, but rebuilding a 12TB pool from > backups, in a production machine, in a remote hosting, would be > something I rather avoid :-p. > > I know that hybrid HD+SSD pools were a bit flacky in the past (you > lost the ZIL device, you kiss goodbye to your ZPOOL, in the pre-v19 > days), and I want to know what terrain I am getting into. > > PS: I plan to upgrade to S10 U10 when available, and I will upgrade > the ZPOOL version after a while. > > - -- > Jesus Cea Avion _/_/ _/_/_/ _/_/_/ > jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ > jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ > . _/_/ _/_/ _/_/ _/_/ _/_/ > "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ > "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ > "El amor es poner tu felicidad en la felicidad de otro" - Leibniz > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQCVAwUBTlxjxplgi5GaxT1NAQLi9AP/VW2LQqij6y25KQ3c5EDBWvnnL1Z7R65j > BJ0N1EbWW6ZdkQ9uFoLNJBVb8xPgwpTOKuy5g8FTwrjs1Sc5a3E3DbRDUg75faE5 > 4IOgCi0gtIVyrxGEQ2AAhnKHGcto/2gB9Y5KRiibBeysbqNvr0HXQsko7WRauP96 > N1L1TqFsN8E> =sDRY > -----END PGP SIGNATURE----- > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Jesus Cea > > 1. Is the L2ARC data stored in the SSD checksummed?. If so, can I > expect that ZFS goes directly to the disk if the checksum is wrong?.Yup.> 2. Can I import a POOL if one/both L2ARC''s are not available?.Yup.> 3. What happend if a L2ARC device, suddenly, "dissappears"?.No problem. You lose the acceleration that it was giving you, and reads revert to the primary storage instead.> 4. Any idea if L2ARC content will be persistent across system > rebooting "eventually"?I hope so...> 5. Can I import a POOL if one/both ZIL devices are not available?. My > pool is v22. I know that I can remove ZIL devices since v19, but I > don''t know if I can remove them AFTER they are physically unavailable, > of before importing the pool (after a reboot).No problem. However - You can only import without the log devices by force. Because if a pool is offline, and you''re trying to import, the system has no way to know if there were any uncommitted transactions in the log devices unless it can read them. It will allow you to discard the ZIL if it''s unavailable, but it warns you harshly and you do so at your own risk. There are not a lot of situations where that matters though.> 6. Can I remove a ZIL device after ZFS consider it "faulty"?.Yup.> 7. What if a ZIL device "dissapears", suddenly?. I know that I could > lose "committed" transactions in-fight, but will the machine crash?. > Will it fallback to ZIL on harddisk?.No problem. It reverts to pool. The only risk you have is ... Suppose some write was already committed to log device, and then the log device disappears before the TXG is actually committed to pool. And then the system hard-crashes before the TXG is actually flushed to pool. That could result in one TXG loss, but having all of your SSD''s and system hard crash within seconds of each other is ... unlikely. I calculate that risk to be on the same order as meteor strike. ;-)> 8. Since my ZIL will be mirrored, I assume that the OS will actually > will look for transactions to be replayed in both devicesCorrect.> 9. If a ZIL device mirrored goes offline/online, will it resilver from > the other side, or it will simply get new transactions, since old > transactions are irrelevant after ?30? seconds?.I don''t know, but it doesn''t matter, does it? Worst case, there''s a few seconds of degraded performance while resilvering. So stop your pre-schooler from yanking disks out of your server and reinserting them if you want to prevent this. ;-)> 10. What happens if my 1GB of ZIL is too optimistic?. Will ZFS use the > disks or it will stop writers until flushing ZIL to the HDs?.Good question. I don''t know.> Anything else I should consider?. > > As you can see, my concerns concentrate in what happens if the SSDs go > bad or "somebody" unplugs them "live".Why are you more concerned about your SSD''s going offline as opposed to your HDD''s? In all but the most extreme cases, IMHO the best solution nowadays is either to use an unmirrored log device (since losing your log device does not mean pool destruction, and write performance might be better if you don''t have to wait for writing 2 devices)... Or disable the ZIL. If you disable the ZIL, you get maximum performance at minimum cost, and depending on how you use your system, it may be acceptable.> I have backup of (most) of my data, but rebuilding a 12TB pool from > backups, in a production machine, in a remote hosting, would be > something I rather avoid :-p. > > I know that hybrid HD+SSD pools were a bit flacky in the past (you > lost the ZIL device, you kiss goodbye to your ZPOOL, in the pre-v19 > days), and I want to know what terrain I am getting into.Those days are over. It''s solid and stable now... Since I guess a year ago, maybe two years.
On 08/30/11 08:31, Edward Ned Harvey wrote:>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >> bounces at opensolaris.org] On Behalf Of Jesus Cea >> >> >> 10. What happens if my 1GB of ZIL is too optimistic?. Will ZFS use the >> disks or it will stop writers until flushing ZIL to the HDs?. >> > > Good question. I don''t know. >- It will use the pool disks. Thanks Edward for answering the rest. Neil. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110830/8a7a9cf1/attachment.html>
On 08/30/11 15:31, Edward Ned Harvey wrote:>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >> bounces at opensolaris.org] On Behalf Of Jesus Cea >> >> 1. Is the L2ARC data stored in the SSD checksummed?. If so, can I >> expect that ZFS goes directly to the disk if the checksum is wrong?. > > Yup.Note the following is an implementation detail subject to change: It is NOT checksumed on disk only in memory, but the L2ARC data on disk is not used after reboot anyway just now. -- Darren J Moffat
On Tue, Aug 30, 2011 at 03:53:48PM +0100, Darren J Moffat wrote:> On 08/30/11 15:31, Edward Ned Harvey wrote: >>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >>> bounces at opensolaris.org] On Behalf Of Jesus Cea >>> >>> 1. Is the L2ARC data stored in the SSD checksummed?. If so, can I >>> expect that ZFS goes directly to the disk if the checksum is wrong?. >> >> Yup. > > Note the following is an implementation detail subject to change: > > It is NOT checksumed on disk only in memory, but the L2ARC data on disk > is not used after reboot anyway just now.It''s not checksummed on disk as a direct function of L2ARC storage with an L2ARC checksum because that''s unnecessary - the cached data is verified against the original zfs checksums. And, yes, if it fails it counts as a "bad read" and zfs tries again from the data pool. It''s checksummed on the way into memory, from either pool disk or l2arc disk. If it''s already in ARC memory, it''s just a hit and the checksum is not done each time - that would be ludicrously expensive, and is one of the ways non-ECC systems can corrupt data. L2ARC persistence may require adding checksumming to the L2ARC on-disk format, but presumably for the L2ARC metadata that will need to be stored persistently (and now only exists in ram), not so much for the cached pool data. -- Dan. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110831/0901912f/attachment.bin>
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30/08/11 06:48, Michael DeMan wrote:> Are you truly new to ZFS? Or do you work for NetApp or EMC or > somebody else that is curious?I am a Solaris Admin for the last 15 years, and a ZFS user from the very first public release. I think that hybrid storages (SSD+HD) are a huge opportunity for ZFS, but I am still seeing problem reports. Just a few days ago somebody posted in this list about being unable to delete a faulty SSD ZIL. I am trying to be cautious and apply due diligence. It is part of my job, after all... :) - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTmvH5Zlgi5GaxT1NAQLWCwP8Dv/YS1VSmIeCsOjF1IkFJO8TvyIodcPI AS/LaKWW2Px7QI/2ML6R4hX4Fylz1J3zBlAhIK7EAgCCrEJlmyPHkSdj9QskdWY+ gHfKMQCk0zkqtksC9WZSgrKHNYSOyI7PE0hDsZpJ35ai71IIZklJ/P+lSpEmPJZc opBiybNGlFQ=pa2r -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I have a new answer: interaction between dataset encryption and L2ARC and ZIL. 1. I am pretty sure (but not completely sure) that data stored in the ZIL is encrypted, if the destination dataset uses encryption. Can anybody confirm?. 2. What happens with L2ARC?. Since ARC is not encrypted (in RAM), is it encrypted when evicted to L2ARC?. Thanks for your time and attention!. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTnd/zZlgi5GaxT1NAQLYOgQAiEbcrklzj/79u9DysRSb1YuMx2J/FsO2 0qgH7KKAVUY6g7QJ2oWB0jDVwLRqJJVzunx4MMmxc+U0eiLoETRVOnYpy6wRNdme fwC5vl0EW2xTQirQde3OSCyyBaN+mKAq+FSd+IA2jQn6y3MLVWq2ucv3d8B6VQDd 32rYKHe6+70=tv4G -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 19/09/11 19:45, Jesus Cea wrote:> I have a new answer: interaction between dataset encryption and > L2ARC and ZIL.Question, a new question... :) - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTneAtplgi5GaxT1NAQJPXwP9EpD4tYoqFvsGGnAv51aNwNif1pcvuf6x swEqopVdZxIVHi5Sw7LXNCI6S4gL1WC0Fc2rSOhPj/2a98ihYQSCqn3g+RWlPiJy 0yHdcKfVkHVB3Wy/ReBBwI4RVZLIkjKXP8YN7yBBbPP9ZmrA9lMBFm7zkex6PLXe H2zgUQD8sa8=tbUK -----END PGP SIGNATURE-----
On 9/19/11 11:45 AM, Jesus Cea wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I have a new answer: interaction between dataset encryption and L2ARC > and ZIL. > > 1. I am pretty sure (but not completely sure) that data stored in the > ZIL is encrypted, if the destination dataset uses encryption. Can > anybody confirm?. >If the data set (file system/zvol) is encrypted then the user data is also encrypted. The ZIL meta data used to parse blocks and records is kept in the clear (in order to claim the blocks) but the user data is encrypted. Neil.
On 09/19/11 18:45, Jesus Cea wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I have a new answer: interaction between dataset encryption and L2ARC > and ZIL. > > 1. I am pretty sure (but not completely sure) that data stored in the > ZIL is encrypted, if the destination dataset uses encryption. Can > anybody confirm?.Of course if we didn''t do that we would be leaking user data.> 2. What happens with L2ARC?. Since ARC is not encrypted (in RAM), is > it encrypted when evicted to L2ARC?.Use of the L2ARC is disabled for data from encrypted datasets at this time. -- Darren J Moffat