zfs discuss - Jan 2012 - (gang?)block layout question, and how to decipher ZDB output?

Hello, all

   I''m "playing" with ZDB again on another test system,
the rpool being uncompressed with 512-byte sectors.
Here''s some output that puzzles me (questions follow):

# zdb -dddddddd -bbbbbb rpool/ROOT/nightly-2012-01-31 260050
...
          1e80000   L0 DVA[0]=<0:200972e00:20200> 
DVA[1]=<0:391820a00:200> [L0 ZFS plain file] fletcher4 uncompressed LE 
gang unique single size=20000L/20000P birth=743480L/743480P fill=1 
cksum=40ed72fd1a7e:10813724cba2f082:99adc8fc26918419:77a6a1f23fa0600

          1ea0000   L0 DVA[0]=<0:206373000:20000> [L0 ZFS plain file] 
fletcher4 uncompressed LE contiguous unique single size=20000L/20000P 
birth=743481L/743481P fill=1 
cksum=3e813c892009:fe2666367dd7397:eab4fcda09e1eda:920af56056f956cb

          1ec0000   L0 DVA[0]=<0:206599400:20000> [L0 ZFS plain file] 
fletcher4 uncompressed LE contiguous unique single size=20000L/20000P 
birth=743481L/743481P fill=1 
cksum=4213d7748915:10958bc56b9df6f3:514903c1c2a34d4f:92b907df21b16050
...
          3220000   L0 DVA[0]=<0:191e41400:20200> 
DVA[1]=<0:39c762600:200> [L0 ZFS plain file] fletcher4 uncompressed LE 
gang unique single size=20000L/20000P birth=743482L/743482P fill=1 
cksum=3e7277e04a2c:fc89c0aa8f72a75:46132aa352fa4e80:9126a177ac19bfe4

          3240000   L0 DVA[0]=<0:191e41600:20200> 
DVA[1]=<0:39c762800:200> [L0 ZFS plain file] fletcher4 uncompressed LE 
gang unique single size=20000L/20000P birth=743482L/743482P fill=1 
cksum=7135eb7cbcd:34cb6fb5d8397c0:5d9d497e1e2f3942:72b840ca284a2130

          3260000   L0 DVA[0]=<0:191e41800:20200> 
DVA[1]=<0:39c7b9600:200> [L0 ZFS plain file] fletcher4 uncompressed LE 
gang unique single size=20000L/20000P birth=743482L/743482P fill=1 
cksum=0:0:0:0

          3280000   L0 DVA[0]=<0:191e70400:20200> 
DVA[1]=<0:39c7b9800:200> [L0 ZFS plain file] fletcher4 uncompressed LE 
gang unique single size=20000L/20000P birth=743482L/743482P fill=1 
cksum=0:0:0:0

                 segment [0000000000000000, 00000000032a0000) size 50.6M

1) Why does each almost entry have several DVA addresses?
   I believe this has to do with gang-blocking, but why is
   it not consistent (some blocks are single-DVA, most are
   double-DVA).

   I thought it might be due to fragmentation - ZFS failed
   to find enough contiguous addresses... is it realistic?

   In this case, what does the single "segment" entry mean
   then?

2) The last two blocks are reported to have zero values for
   checksums. Why is that, and why doesn''t it trigger a CKSUM
   error (I can read the whole file, no I/O errors)?

3) I tried to extract the blocks, they did not seem empty
   (that was my guess at why they would have zero checksums).
   However, I am not certain how to correctly extract such
   ganged blocks:

3.1) Trying the first DVA as-is fails (how does it know the
   "correct" power-of-two size for a direct request of on-VDEV
   data?):
# zdb -R rpool 0:191e70400:20200:r > /tmp/rp1
Found vdev: /dev/dsk/c4t0d0s0
Assertion failed: size <= (1ULL << 17) (0x20200 <= 0x20000), file 
../../../uts/common/fs/zfs/zio.c, line 493
Abort

3.2) Disabling assertions seems to help:
# zdb -AAA -R rpool 0:191e70400:20200:r > /tmp/rp3
Found vdev: /dev/dsk/c4t0d0s0

Comparing RP1 and RP3 files showed that the latter indeed has
0x200 bytes added at the end; first 0x20000 are identical...

3.3) Extracting a power-of-two sized block works, but is it
   everything there was in original on-disk data (sized 20200?)
# zdb -R rpool 0:191e70400:20000:r > /tmp/rp1
Found vdev: /dev/dsk/c4t0d0s0

# zdb -R rpool 0:39c7b9800:200:r > /tmp/rp2
Found vdev: /dev/dsk/c4t0d0s0

Thanks in advance for helping me decipher this all, :)
//Jim Klimov