zfs discuss - Feb 2010 - Intrusion Detection - powered by ZFS Checksumming ?

Hello, 

an idea popped into my mind while talking about security and intrusion
detection.

Host based ID may use Checksumming for file change tracking. It works like this:

Once installed and knowning the software is "OK", a baseline is
created.
Then in every check - verify the current status of the data  with the baseline
and report changes.

An example for this is AIDE.  

The difficult part is the checksumming - this takes time. 

My idea would be to use ZFS snapshots for this. 

baseline creation = create snapshot
baseline verification = verify the checksums of the objects and report objects
diffent

This could work for non-zvol environments. 

Is it possible to extract the checksums of ZFS objects with a command line tool
?

Regards, 
Robert
-- 
This message posted from opensolaris.org

On 08/02/2010 12:55, Lutz Schumann wrote:
> Hello,
>
> an idea popped into my mind while talking about security and intrusion
detection.
>
> Host based ID may use Checksumming for file change tracking. It works like
this:
>
> Once installed and knowning the software is "OK", a baseline is
created.
> Then in every check - verify the current status of the data  with the
baseline and report changes.
>
> An example for this is AIDE.
>
> The difficult part is the checksumming - this takes time.
>
> My idea would be to use ZFS snapshots for this.
>
> baseline creation = create snapshot
> baseline verification = verify the checksums of the objects and report
objects diffent
>
> This could work for non-zvol environments.
>
> Is it possible to extract the checksums of ZFS objects with a command line
tool ?
Only with the zdb(1M) tool but note that the checksums are NOT of files 
but of the ZFS blocks.

-- 
Darren J Moffat

> Only with the zdb(1M) tool but note that the
> checksums are NOT of files 
> but of the ZFS blocks.
Thanks - bocks, right (doh) - thats what I was missing. Damn it would be so nice
:(
-- 
This message posted from opensolaris.org

On Mon, Feb 08, 2010 at 11:24:56AM -0800, Lutz Schumann
wrote:
> > Only with the zdb(1M) tool but note that the
> > checksums are NOT of files 
> > but of the ZFS blocks.
> 
> Thanks - bocks, right (doh) - thats what I was missing. Damn it would be so
nice :(
If you''re comparing the current data to a snapshot baseline on the
same pool, it just means you need to compare more checksums (several
per file), it doesn''t invalidate the idea.  

There may also be other ways of checking quickly that the file data is
unmodified since snapshot X, but again it will require looking at zfs
internals. This is far from the first use case for an official
interface to get at this kind of data.  It''s quite similar to the
question of how to verify send|recv integrity from yesterday, for
example. As yet I don''t know of a concrete proposal of what such an
interface should look like (since there''s nothing to borrow from
POSIX), let alone an implementation.

It more complicated if you''re comparing checksums against an external
baseline reference (such as from a build) because block sizes and
checksum algorithms may vary between pools.  However, as you note,
that''s already catered for by existing tools, so they could work
together.

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100209/4b389ab5/attachment.bin>

May be look at rsync and  rsync lib (http://librsync.sourceforge.net/) code to
see if a ZFS API could be design to help rsync/librsync in the future as well as
diff.

It might be a good idea for POSIX to have a single checksum and a multi-checksum
interface.

One problem could be block sizes, if a file is re-written and is the same size
it may have different ZFS record sizes within, if it was written over a long
period of time (txg''s)(ignoring compression), and therefore you could
not use ZFS checksum to compare two files.

Side Note:
It would be nice if ZFS on every txg only wrote full record sizes unless it was
short on memory, or a file was closed. Maybe the txg could happen more often if
it just scanned for full recordsize''s writes and closed files. Or block
which had not be altered for three scan''s.????
-- 
This message posted from opensolaris.org

Damon Atkins <Damon_Atkins at yahoo.com.au> writes:
> One problem could be block sizes, if a file is re-written and is the
> same size it may have different ZFS record sizes within, if it was
> written over a long period of time (txg''s)(ignoring compression),
and
> therefore you could not use ZFS checksum to compare two files.
the record size used for a file is chosen when that file is created.  it
can''t change.  when the default record size for the dataset changes,
only new files will be affected.  ZFS *must* write a complete record
even if you change just one byte (unless it''s the tail record of
course), since there isn''t any better granularity for the block
pointers.

-- 
Kjetil T. Homme
Redpill Linpro AS - Changing the game

I would have thought that if I write 1k then ZFS txg times out in 30secs, then
the 1k will be written to disk in a 1k record block, and then if I write 4k then
30secs latter txg happen another 4k record size block will be written, and then
if I write 130k a 128k and 2k record block will be written.

Making the file have record sizes of
1k+4k+128k+2k
-- 
This message posted from opensolaris.org

On Feb 8, 2010, at 9:10 PM, Damon Atkins wrote:
> I would have thought that if I write 1k then ZFS txg times out in 30secs,
then the 1k will be written to disk in a 1k record block, and then if I write 4k
then 30secs latter txg happen another 4k record size block will be written, and
then if I write 130k a 128k and 2k record block will be written.
> 
> Making the file have record sizes of
> 1k+4k+128k+2k
Close. Once the max record size is achieved, it is not reduced.  So the
allocation is:
	1KB + 4KB + 128KB + 128KB

Physical writes tend to be coalesced, which is one reason why a transactional
system performs well.
 -- richard

Richard Elling <richard.elling at gmail.com> writes:
> On Feb 8, 2010, at 9:10 PM, Damon Atkins wrote:
>
>> I would have thought that if I write 1k then ZFS txg times out in
>> 30secs, then the 1k will be written to disk in a 1k record block, and
>> then if I write 4k then 30secs latter txg happen another 4k record
>> size block will be written, and then if I write 130k a 128k and 2k
>> record block will be written.
>> 
>> Making the file have record sizes of
>> 1k+4k+128k+2k
>
> Close. Once the max record size is achieved, it is not reduced.  So
> the allocation is: 1KB + 4KB + 128KB + 128KB
I think the above is easily misunderstood.  I assume the OP means
append, not rewrites, and in that case (with recordsize=128k):

* after the first write, the file will consist of a single 1 KiB record.
* after the first append, the file will consist of a single 5 KiB
  record.
* after the second append, one 128 KiB record and one 7 KiB record.

in each of these operations, the *whole* file will be rewritten to a new
location, but after a third append, only the tail record will be
rewritten.

-- 
Kjetil T. Homme
Redpill Linpro AS - Changing the game

On 02/09/10 08:18, Kjetil Torgrim Homme wrote:
> Richard Elling <richard.elling at gmail.com> writes:
> 
>> On Feb 8, 2010, at 9:10 PM, Damon Atkins wrote:
>>
>>> I would have thought that if I write 1k then ZFS txg times out in
>>> 30secs, then the 1k will be written to disk in a 1k record block,
and
>>> then if I write 4k then 30secs latter txg happen another 4k record
>>> size block will be written, and then if I write 130k a 128k and 2k
>>> record block will be written.
>>>
>>> Making the file have record sizes of
>>> 1k+4k+128k+2k
>> Close. Once the max record size is achieved, it is not reduced.  So
>> the allocation is: 1KB + 4KB + 128KB + 128KB
> 
> I think the above is easily misunderstood.  I assume the OP means
> append, not rewrites, and in that case (with recordsize=128k):
> 
> * after the first write, the file will consist of a single 1 KiB record.
> * after the first append, the file will consist of a single 5 KiB
>   record.
Good so far.
> * after the second append, one 128 KiB record and one 7 KiB record.
A long time ago we used to write short tail blocks, but not any more.
So after the 2nd append we actually have 2 128KB blocks.
> 
> in each of these operations, the *whole* file will be rewritten to a new
> location, but after a third append, only the tail record will be
> rewritten.
So after the third append we''d actually have 3 128KB blocks. The first
doesn''t
need to be re-written.

Neil.

Neil Perrin <Neil.Perrin at Sun.COM> writes:
> On 02/09/10 08:18, Kjetil Torgrim Homme wrote:
>> I think the above is easily misunderstood.  I assume the OP means
>> append, not rewrites, and in that case (with recordsize=128k):
>>
>> * after the first write, the file will consist of a single 1 KiB
record.
>> * after the first append, the file will consist of a single 5 KiB
>>   record.
>
> Good so far.
>
>> * after the second append, one 128 KiB record and one 7 KiB record.
>
> A long time ago we used to write short tail blocks, but not any more.
> So after the 2nd append we actually have 2 128KB blocks.
thanks a lot for the correction!

-- 
Kjetil T. Homme
Redpill Linpro AS - Changing the game