Hi, I was discussing the common practice of disk eradication used by many firms for security. I was thinking this may be a useful feature of ZFS to have an option to eradicate data as its removed, meaning after the last reference/snapshot is done and a block is freed, then write the eradication patterns back to the removed blocks. By any chance, has this been discussed or considered before? Thanks, Brian
Typically this is called "Sanitization" and could be done as part of an evacuation of data from the disk in preparation for removal. You would want to specify the patterns to write and the number of passes. -- mark Brian Kolaci wrote:> Hi, > > I was discussing the common practice of disk eradication used by many > firms for security. I was thinking this may be a useful feature of > ZFS to have an option to eradicate data as its removed, meaning after > the last reference/snapshot is done and a block is freed, then write > the eradication patterns back to the removed blocks. > > By any chance, has this been discussed or considered before? > > Thanks, > > Brian > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- <http://www.sun.com> * Mark A. Carlson * Sr. Architect *Systems Group* Phone x69559 / 303-223-6139 Email Mark.Carlson at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091110/0165fb5c/attachment.html>
On Nov 10, 2009, at 20:55, Mark A. Carlson wrote:> Typically this is called "Sanitization" and could be done as part of > an evacuation of data from the disk in preparation for removal. > > You would want to specify the patterns to write and the number of > passes.See also "remanence": http://en.wikipedia.org/wiki/Data_remanence (S)ATA actually has a protocol command ("secure erase") that will cause the disk to over write all of its sectors, and not be usable until its done. This doesn''t exist in SCSI / SAS / FC as far as I know. Generally speaking one over write is sufficient to prevent data from being accessible, but various government standards specify anywhere between one and four passes: http://en.wikipedia.org/wiki/Data_erasure Degaussing or complete destruction is usually necessary for the "top secret" stuff. DBAN is a useful (open-source) utility that I tend to recommend for regular folk: http://www.dban.org/ While it could be useful, there are penalties in various jurisdictions for leaking data (especially with government-related stuff), so I''m not sure if Sun would want to potentially expose itself to inappropriate use that doesn''t clean everything properly. With ZFS encryption coming up, it could be sufficient to have your data sets encrypted and then simply "forget" the key. The data is still technically there, but (theoretically) completely inaccessible.
Excuse me for mentioning it but why not just use the format command?
format(1M)
analyze
Run read, write, compare tests, and data
purge. The data purge
function implements the National Computer Security Center Guide to
Understanding Data Remnance (NCSC-TG-025 version 2) Overwriting
Algorithm. See NOTES.
The NCSC-TG-025 algorithm for overwriting
meets the DoD 5200.28-M (ADP
Security Manual) Eraser Procedures specification. The NIST Guidelines
for Media Sanitization (NIST SP 800-88)
also reference this algorithm..
And if the disk is buggered (a very technical term). A great big hammer!
Mark A. Carlson wrote:
Typically this is called "Sanitization" and could be
done as part of
an evacuation of data from the disk in preparation for removal.
You would want to specify the patterns to write and the number of
passes.
-- mark
Brian Kolaci wrote:
Hi,
I was discussing the common practice of disk eradication used by many
firms for security. I was thinking this may be a useful feature of ZFS
to have an option to eradicate data as its removed, meaning after the
last reference/snapshot is done and a block is freed, then write the
eradication patterns back to the removed blocks.
By any chance, has this been discussed or considered before?
Thanks,
Brian
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
--
Mark A. Carlson
Sr. Architect
Systems Group
Phone x69559 / 303-223-6139
Email Mark.Carlson@Sun.COM
www.eagle.co.nz
This email is confidential and may be legally
privileged. If received in error please destroy and immediately notify
us.
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Thanks all, It was a government customer that I was talking too and it sounded like a good idea, however with the certification paper trails required today, I don''t think it would be of such a benefit after all. It may be useful on the disk evacuation, but they''re still going to need their paper trail with a certification that it was done and confirmed. David Magda wrote:> On Nov 10, 2009, at 20:55, Mark A. Carlson wrote: > >> Typically this is called "Sanitization" and could be done as part of >> an evacuation of data from the disk in preparation for removal. >> >> You would want to specify the patterns to write and the number of >> passes. > > See also "remanence": > > http://en.wikipedia.org/wiki/Data_remanence > > (S)ATA actually has a protocol command ("secure erase") that will cause > the disk to over write all of its sectors, and not be usable until its > done. This doesn''t exist in SCSI / SAS / FC as far as I know. > > Generally speaking one over write is sufficient to prevent data from > being accessible, but various government standards specify anywhere > between one and four passes: > > http://en.wikipedia.org/wiki/Data_erasure > > Degaussing or complete destruction is usually necessary for the "top > secret" stuff. DBAN is a useful (open-source) utility that I tend to > recommend for regular folk: > > http://www.dban.org/ > > While it could be useful, there are penalties in various jurisdictions > for leaking data (especially with government-related stuff), so I''m not > sure if Sun would want to potentially expose itself to inappropriate use > that doesn''t clean everything properly. > > With ZFS encryption coming up, it could be sufficient to have your data > sets encrypted and then simply "forget" the key. The data is still > technically there, but (theoretically) completely inaccessible. >
Brian Kolaci wrote:> Hi, > > I was discussing the common practice of disk eradication used by many > firms for security. I was thinking this may be a useful feature of ZFS > to have an option to eradicate data as its removed, meaning after the > last reference/snapshot is done and a block is freed, then write the > eradication patterns back to the removed blocks. > > By any chance, has this been discussed or considered before?Yes it has been discussed here before. It is one of the things I want to look at after ZFS Crypto and block pointer rewritter have integrated. Also in some juristictions if the data was always encrypted on disk then you don''t need to write any patterns to erase the blocks. So ZFS Crypto can help there. -- Darren J Moffat
This feature is described in this RFE: http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=4930014 Secure delete option: erase blocks after they''re freed cs On 11/11/09 09:17, Darren J Moffat wrote:> Brian Kolaci wrote: >> Hi, >> >> I was discussing the common practice of disk eradication used by many >> firms for security. I was thinking this may be a useful feature of >> ZFS to have an option to eradicate data as its removed, meaning after >> the last reference/snapshot is done and a block is freed, then write >> the eradication patterns back to the removed blocks. >> >> By any chance, has this been discussed or considered before? > > Yes it has been discussed here before. > > It is one of the things I want to look at after ZFS Crypto and block > pointer rewritter have integrated. > > Also in some juristictions if the data was always encrypted on disk then > you don''t need to write any patterns to erase the blocks. So ZFS Crypto > can help there. >
Hi, Well ... i think Darren should implement this as a part of zfs-crypto. Secure Delete on SSD looks like quite challenge, when wear leveling and bad block relocation kicks in ;) Regards Joerg Am 11.11.2009 um 17:53 schrieb Cindy Swearingen:> This feature is described in this RFE: > > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=4930014 > Secure delete option: erase blocks after they''re freed > > cs > > On 11/11/09 09:17, Darren J Moffat wrote: >> Brian Kolaci wrote: >>> Hi, >>> >>> I was discussing the common practice of disk eradication used by many firms for security. I was thinking this may be a useful feature of ZFS to have an option to eradicate data as its removed, meaning after the last reference/snapshot is done and a block is freed, then write the eradication patterns back to the removed blocks. >>> >>> By any chance, has this been discussed or considered before? >> Yes it has been discussed here before. >> It is one of the things I want to look at after ZFS Crypto and block pointer rewritter have integrated. >> Also in some juristictions if the data was always encrypted on disk then you don''t need to write any patterns to erase the blocks. So ZFS Crypto can help there. > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Joerg Moellenkamp wrote:> Hi, > > Well ... i think Darren should implement this as a part of zfs-crypto. Secure Delete on SSD looks like quite challenge, when wear leveling and bad block relocation kicks in ;)No I won''t be doing that as part of the zfs-crypto project. As I said some jurisdictions are happy that if the data is encrypted then overwrite of the blocks isn''t required. For those that aren''t use dd(1M) or format(1M) may be sufficient - if that isn''t then nothing short of physical destruction is likely good enough. If we choose to add block erasure on delete to ZFS then it is a completely separate and complementary feature to encryption. -- Darren J Moffat
On Wed, Nov 11, 2009 at 12:29 PM, Darren J Moffat <darrenm at opensolaris.org>wrote:> Joerg Moellenkamp wrote: > >> Hi, >> >> Well ... i think Darren should implement this as a part of zfs-crypto. >> Secure Delete on SSD looks like quite challenge, when wear leveling and bad >> block relocation kicks in ;) >> > > No I won''t be doing that as part of the zfs-crypto project. As I said some > jurisdictions are happy that if the data is encrypted then overwrite of the > blocks isn''t required. For those that aren''t use dd(1M) or format(1M) may > be sufficient - if that isn''t then nothing short of physical destruction is > likely good enough. > > If we choose to add block erasure on delete to ZFS then it is a completely > separate and complementary feature to encryption. > > -- > Darren J Moffat >http://www.dban.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091111/a17c316b/attachment.html>
On Wed, November 11, 2009 13:29, Darren J Moffat wrote:> No I won''t be doing that as part of the zfs-crypto project. As I said > some jurisdictions are happy that if the data is encrypted then > overwrite of the blocks isn''t required. For those that aren''t use > dd(1M) or format(1M) may be sufficient - if that isn''t then nothing > short of physical destruction is likely good enough.Does anyone know if SATA''s ''secure erase'' command available on SSDs?
On Wed, 2009-11-11 at 10:29 -0800, Darren J Moffat wrote:> Joerg Moellenkamp wrote: > > Hi, > > > > Well ... i think Darren should implement this as a part of > zfs-crypto. Secure Delete on SSD looks like quite challenge, when wear > leveling and bad block relocation kicks in ;) > > No I won''t be doing that as part of the zfs-crypto project. As I said > some jurisdictions are happy that if the data is encrypted then > overwrite of the blocks isn''t required. For those that aren''t use > dd(1M) or format(1M) may be sufficient - if that isn''t then nothing > short of physical destruction is likely good enough.note that "eradication" via overwrite makes no sense if the underlying storage uses copy-on-write, because there''s no guarantee that the newly written block actually will overlay the freed block. IMHO the sweet spot here may be to overwrite once with zeros (allowing the block to be compressed out of existance if the underlying storage is a compressed zvol or equivalent) or to use the TRIM command. (It may also be worthwhile for zvols exported via various protocols to themselves implement the TRIM command -- freeing the underlying storage). - Bill
Bill Sommerfeld wrote:> On Wed, 2009-11-11 at 10:29 -0800, Darren J Moffat wrote: >> Joerg Moellenkamp wrote: >>> Hi, >>> >>> Well ... i think Darren should implement this as a part of >> zfs-crypto. Secure Delete on SSD looks like quite challenge, when wear >> leveling and bad block relocation kicks in ;) >> >> No I won''t be doing that as part of the zfs-crypto project. As I said >> some jurisdictions are happy that if the data is encrypted then >> overwrite of the blocks isn''t required. For those that aren''t use >> dd(1M) or format(1M) may be sufficient - if that isn''t then nothing >> short of physical destruction is likely good enough. > > note that "eradication" via overwrite makes no sense if the underlying > storage uses copy-on-write, because there''s no guarantee that the newly > written block actually will overlay the freed block.Which is why this has to be a ZFS feature rather than something link GNU shred(1) which runs in userland.> IMHO the sweet spot here may be to overwrite once with zeros (allowing > the block to be compressed out of existance if the underlying storage is > a compressed zvol or equivalent) or to use the TRIM command.Exactly. -- Darren J Moffat
On Wed, 11 Nov 2009, Darren J Moffat wrote:>> >> note that "eradication" via overwrite makes no sense if the underlying >> storage uses copy-on-write, because there''s no guarantee that the newly >> written block actually will overlay the freed block. > > Which is why this has to be a ZFS feature rather than something link GNU > shred(1) which runs in userland.Zfs is absolutely useless for this if the underlying storage uses copy-on-write. Therefore, it is absolutely useless to put it in zfs. No one should even consider it. The use of encrypted blocks is much better, even though encrypted blocks may be subject to freeze-spray attack if the whole computer is compromised while it is still running. Otherwise use a sledge-hammer followed by incineration. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
On Nov 11, 2009, at 17:40, Bob Friesenhahn wrote:> Zfs is absolutely useless for this if the underlying storage uses > copy-on-write. Therefore, it is absolutely useless to put it in > zfs. No one should even consider it. > > The use of encrypted blocks is much better, even though encrypted > blocks may be subject to freeze-spray attack if the whole computer > is compromised while it is still running. Otherwise use a sledge- > hammer followed by incineration.There seem to be ''secure erase'' methods available for some SSDs:> Zeus Solid State Drives are available with secure erase methods to > support a wide variety of requirements. MilPurge provide secure > erase procedure that comply with several agency guidelines, > including: DoD 5220.22-M, NSA 130-2, AFSSI 5020, AR 380-19, and > Navso 5239. Additional capabilities include Intelligent Destructive > Purge where the flash media is physically damaged and rendered > totally and irrevocably unusable.http://www.stec-inc.com/products/zeus/ The Intel X25-M is reported to mark all cells as free / empty via ATA''s secure erase: http://ata.wiki.kernel.org/index.php/ATA_Secure_Erase Marking them and actually resetting them are two different things though. Hopefully more products will actually do a proper reset / wipe in the future.
Bob Friesenhahn wrote:> On Wed, 11 Nov 2009, Darren J Moffat wrote: >>> >>> note that "eradication" via overwrite makes no sense if the underlying >>> storage uses copy-on-write, because there''s no guarantee that the newly >>> written block actually will overlay the freed block. >> >> Which is why this has to be a ZFS feature rather than something link >> GNU shred(1) which runs in userland. > > Zfs is absolutely useless for this if the underlying storage uses > copy-on-write. Therefore, it is absolutely useless to put it in zfs. No > one should even consider it.I disagree. Sure there are cases where ZFS which is copy-on-write is sitting ontop of something that is copy-on-write (iSCSI luns backed by ZFS on another system for example) that doesn''t mean there are no cases where this is useful. For example in an appliance or other controlled environment it isn''t useless and it is being considered for example those use cases of ZFS. If we introduce this for ZFS it would likely be a per dataset or pool property and it will be documented that the whole storage stack needs to be taken into consideration for it to determined if this is actually effective.> The use of encrypted blocks is much better, even though encrypted blocks > may be subject to freeze-spray attack if the whole computer is > compromised while it is still running. Otherwise use a sledge-hammer > followed by incineration.Much better for jurisdictions that allow for that, but not all do. I know of at least one that wants even ciphertext blocks to overwritten. -- Darren J Moffat
On Wed, 11 Nov 2009, Darren J Moffat wrote:>> >> Zfs is absolutely useless for this if the underlying storage uses >> copy-on-write. Therefore, it is absolutely useless to put it in >> zfs. No one should even consider it. > > I disagree. Sure there are cases where ZFS which is copy-on-write > is sitting ontop of something that is copy-on-write (iSCSI luns > backed by ZFS on another system for example) that doesn''t mean there > are no cases where this is useful. For example in an appliance or > other controlled environment it isn''t useless and it is being > considered for example those use cases of ZFS.Perhaps in a product where the engineering team has examined every part of the product from top to bottom, it might have some use. Advertising that the feature exists might cause someone to believe that it actually works. There are plenty of devices in common use which are always/sometimes COW. These include SSDs and certain types of CD/DVD/WORM drives. For SSDs, COW is commonly called "wear leveling". In hard-drives, we have what is known as "bad block management". Given that many storage products are made in places in China, it seems unlikely that any government is going to trust assurances from a product vendor that their product never leaves behind copies of data. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
>>>>> "djm" == Darren J Moffat <darrenm at opensolaris.org> writes:>> encrypted blocks is much better, even though >> encrypted blocks may be subject to freeze-spray attack if the >> whole computer is compromised the idea of crypto deletion is to use many keys to encrypt the drive, and encrypt keys with other keys. When you want to delete something, forget the key that encrypted it. Its point is to defend against exactly the freeze-spray case by making undelete into a difficult cryptanalysis problem even for the drive''s authorized key-holding user. djm> Much better for jurisdictions that allow for that, but not all djm> do. I know of at least one that wants even ciphertext blocks djm> to overwritten. The appropriate answer depends when do they want it done, though. Do they want it done continuously while the machine is running whenever someone rm''s something? Or is it about ``losing'''' the data, about media containing encrypted blocks passing outside the campus, or just not knowing where something physically is at all times? If there is no requirement for the former case, crypto deletion is pointless. Any requirement for the second case can be met with ''dd'' or similar---it''s only the former that needs filesystem integration. It''s imagineable that the two cases might have different levels of cargo cult strictness, though. or...that ``jurisdictions'''' smart enough to express two separate requirements will be negotiable, and for those that don''t distinguish you can say ``oh, but I thought you only meant the latter case because you''ve already accepted a mishmash of products some of which do nothing at all about the former case.'''' -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091112/2c49eec4/attachment.bin>
Miles Nordin wrote:>>>>>> "djm" == Darren J Moffat <darrenm at opensolaris.org> writes: > > >> encrypted blocks is much better, even though > >> encrypted blocks may be subject to freeze-spray attack if the > >> whole computer is compromised > > the idea of crypto deletion is to use many keys to encrypt the drive, > and encrypt keys with other keys. When you want to delete something, > forget the key that encrypted it.Yes I know, remember I designed ZFS crypto and this is exactly one of the use case.> djm> Much better for jurisdictions that allow for that, but not all > djm> do. I know of at least one that wants even ciphertext blocks > djm> to overwritten. > > The appropriate answer depends when do they want it done, though. Do > they want it done continuously while the machine is running whenever > someone rm''s something? Or is it about ``losing'''' the data, about > media containing encrypted blocks passing outside the campus, or just > not knowing where something physically is at all times?I''m not in a position to discuss this jurisdictions requirements and rationale on a public mailing list. All I''m saying is that data destruction base only on key destruction/unavailability is not considered enough in some cases. -- Darren J Moffat
On Wed, 11 Nov 2009, David Magda wrote:> > There seem to be ''secure erase'' methods available for some SSDs:Unless the hardware and firmware of these devices has been inspected and validated by a certified third party which is well-versed in such analaysis, I would not trust such devices with significant phrases like "tritium core". Some secrets are very great and should not be trusted to a marketing department. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
> >> djm> Much better for jurisdictions that allow for that, but not all >> not knowing where something physically is at all times? > > I''m not in a position to discuss this jurisdictions requirements and rationale on a public mailing list. All I''m saying is that data destruction base only on key destruction/unavailability is not considered enough in some cases.Nevertheless i think secure delete cannot exist without cryptography, as more and more devices are available where you can''t control the placement of a block. As far as i know those secure delete in SSD is just capable to delete all data, not a single block. Thus the only save way to really delete securely would be the combination of both. When you can''t delete a block on device securely, it''s protected by the encryption as a last line of defense. However secure deletion by cryptography needs the secure deletion by overwriting the stuff, as you close the attack vector of simply waiting until the cryptographic algorithm is broken. One part can''t exist without the other ... Regards Joerg -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091115/22b46727/attachment.html>