zfs discuss - Feb 2008 - modification to zdb to decompress blocks

Hi All,
I have modified zdb to do decompression in zdb_read_block.  Syntax is:

# zdb -R poolname:devid:blkno:psize:d,compression_type,lsize

Where compression_type can be lzjb or any other type compression that 
zdb uses, and
lsize is the size after compression.  I have used this with a modified 
mdb to allow one to
do the following:

    given a pathname for a file on a zfs file system, display the blocks 
(i.e., data) of the file.  The file
    system need not be mounted.

If anyone is interested, send me email.  I can send a webrev of the zdb 
changes for those interested.
As for the mdb changes, I sent a webrev of those a while ago, and have 
since added a rawzfs dmod.

I plan to present a paper at osdevcon in Prague in June that uses the 
modified zdb and mdb to
show the physical layout of a zfs file system.  (I should mention that, 
over time, I have found that
the ZFS on-disk format paper actually does tell you almost everything).

max

If you can''t file a RFE yourself (with the attached diffs), then  
yeah, i''d like to see them so i can do it.

cool stuff,
eric

On Feb 26, 2008, at 4:35 AM, max at bruningsystems.com wrote:
> Hi All,
> I have modified zdb to do decompression in zdb_read_block.  Syntax is:
>
> # zdb -R poolname:devid:blkno:psize:d,compression_type,lsize
>
> Where compression_type can be lzjb or any other type compression that
> zdb uses, and
> lsize is the size after compression.  I have used this with a modified
> mdb to allow one to
> do the following:
>
>     given a pathname for a file on a zfs file system, display the  
> blocks
> (i.e., data) of the file.  The file
>     system need not be mounted.
>
> If anyone is interested, send me email.  I can send a webrev of the  
> zdb
> changes for those interested.
> As for the mdb changes, I sent a webrev of those a while ago, and have
> since added a rawzfs dmod.
>
> I plan to present a paper at osdevcon in Prague in June that uses the
> modified zdb and mdb to
> show the physical layout of a zfs file system.  (I should mention  
> that,
> over time, I have found that
> the ZFS on-disk format paper actually does tell you almost  
> everything).
>
> max
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Hi Eric,

I am (once again) having trouble getting webrev to work.  I got it 
working back in September,
but since have re-installed my system once or twice (search for "using 
webrev" in the tools-discuss
mailing list).  So, rather than spin wheels for half a day
to try to get this to work, I am attaching the modified zdb.c file.
The changes are pretty simple, and currently this sends raw output to 
standard error.

Here is an example of how I use it.  In this example, I start with the 
blkptr from the uberblock
and use the decompress from zdb to dump the objset_phys_t.  From there, 
I use my modified
mdb to dump the objset_phys_t.  I am currently writing a paper that 
starts here and displays the
data for a file in an unmounted zfs file system.  The following should 
give you an idea
of where I am headed with this.  It is a bit convoluted since I have to 
go back and forth between
zdb and mdb to get what I want, but it works!


# zdb -uuuu laciedisk
Uberblock

        magic = 0000000000bab10c
        version = 10
        txg = 19148
        guid_sum = 17219723339164464949
        timestamp = 1203801884 UTC = Sat Feb 23 22:24:44 2008
        rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:4e00:200> 
DVA[1]=<0:1c0004e
00:200> DVA[2]=<0:380001200:200> fletcher4 lzjb LE contiguous 
birth=19148 fill=1
8 cksum=89aca5d29:38d271ef883:be5570b26779:1af282de579a51
#

The above output shows that the objset_phys_t contains 3 copies of the root
blkptr_t, one at disk block 0x4e00, the second at 0x1c0004e00, and the third
at 0x380001200.  Each of these is 0x200 bytes on the disk, and is on vdev 0
(the only disk in the pool).
We''ll need to find which device(s) make up the pool.  We''ll
use zpool(1) for
this.

# zpool status laciedisk
  pool: laciedisk
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        laciedisk   ONLINE       0     0     0
          c4t0d0p1  ONLINE       0     0     0

errors: No known data errors
#

Now we''ll use zdb to dump the raw data referred to in the uberblock. 
This
is an objset_phys_t.  The command below reads a 0x200 bytes from the
laciedisk zpool and the /dev/dsk/c4t0d0p1 device in the pool.  It reads 
block
number 0x4e00 and performs lzjb decompression.  The size after 
decompression is
0x400
bytes.  The output is placed in /tmp/objset.

# zdb -R laciedisk:c4t0d0p1:4e00:200:d,lzjb,400 2> /tmp/objset  <-- here 
I am using the modified zdb
Found vdev: /dev/dsk/c4t0d0p1
#

/tmp/objset contains the objset_phys_t data.  We use (a modified) mdb to 
print
the structure contents.

# mdb /tmp/objset
 > ::loadctf    <-- this loads current kernel CTF for use with raw data
 > 0::print -a -t zfs`objset_phys_t
{
    0 dnode_phys_t os_meta_dnode = {
        0 uint8_t dn_type = 0xa        <-- DMU_OT_DNODE (see 
uts/common/fs/zfs/sys/dmu.h)
        1 uint8_t dn_indblkshift = 0xe
        2 uint8_t dn_nlevels = 0x1     <-- no indirect blocks
        3 uint8_t dn_nblkptr = 0x3     <-- 3 copies in the blkptr_t
        4 uint8_t dn_bonustype = 0     <-- no data in bonus buffer
        5 uint8_t dn_checksum = 0
...

 From here, I dump the blkptr using 40::blkptr.  (0x40 is the offset 
within the objset_phys_t of
where the blkptr_t resides in /tmp/objset.  Then I go back to zdb to 
decompress the block, then
back to mdb to dump the dnode_phys_t array, then back to zdb, and so on.

Let me know if you have any ideas of how to make this simpler...

thanks,
max


eric kustarz wrote:
> If you can''t file a RFE yourself (with the attached diffs), then
yeah,
> i''d like to see them so i can do it.
>
> cool stuff,
> eric
>
> On Feb 26, 2008, at 4:35 AM, max at bruningsystems.com wrote:
>
>> Hi All,
>> I have modified zdb to do decompression in zdb_read_block.  Syntax is:
>>
>> # zdb -R poolname:devid:blkno:psize:d,compression_type,lsize
>>
>> Where compression_type can be lzjb or any other type compression that
>> zdb uses, and
>> lsize is the size after compression.  I have used this with a modified
>> mdb to allow one to
>> do the following:
>>
>>     given a pathname for a file on a zfs file system, display the
blocks
>> (i.e., data) of the file.  The file
>>     system need not be mounted.
>>
>> If anyone is interested, send me email.  I can send a webrev of the zdb
>> changes for those interested.
>> As for the mdb changes, I sent a webrev of those a while ago, and have
>> since added a rawzfs dmod.
>>
>> I plan to present a paper at osdevcon in Prague in June that uses the
>> modified zdb and mdb to
>> show the physical layout of a zfs file system.  (I should mention that,
>> over time, I have found that
>> the ZFS on-disk format paper actually does tell you almost everything).
>>
>> max
>>
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zdb.c
Type: text/x-csrc
Size: 64338 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080301/83b11cac/attachment.bin>

Hi,

Great stuff.

Does this change will make it into opensolaris? Looking at actual code I
couldn''t find the modification.

I try to replace zdb.c in the opensolaris main tree before compiling with
nightly but the compiler wasn''t happy with it. Can you write down the
right options?

bbr
 
 
This message posted from opensolaris.org

Hi Benjamin,

Benjamin Brumaire wrote:
> Hi,
>
> Great stuff.
>
> Does this change will make it into opensolaris? Looking at actual code I
couldn''t find the modification.
>
> I try to replace zdb.c in the opensolaris main tree before compiling with
nightly but the compiler wasn''t happy with it. Can you write down the
right options?
>
> bbr
>   
Attached is the modified version of zdb.c that I am using.  I suggest 
diffing it with the version on your machine and
pasting in the changes.  I suspect that for this to make it into 
opensolaris, I will have to make a "project" out of it,
and I simply do not have the time to go through the process.  At any 
rate, once you have the changed version in
usr/src/cmd/zdb/zdb.c, you should be able to do a "dmake all" in the
zdb
directory, and, assuming all the libraries
it uses have also been built, it should compile and work.

max

-------------- next part --------------
A non-text attachment was scrubbed...
Name: zdb.c
Type: text/x-csrc
Size: 64338 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080503/7884f57c/attachment.bin>

thanks for the quick reaction. I ve now a working binary for my system.

 I don''t understand why these changes should go through a project. The
hooks are already there so once the code is written no much work have to be
done. But it''s an other story. Lets decode lzjb blocks now :-)

bbr
 
 
This message posted from opensolaris.org