Re: [Libguestfs] [PATCH v4 0/3] New API - find_block

NoxDaFox
Tue, 11 Oct 2016 11:13:09 -0000

2016-10-11 11:56 GMT+03:00 Pino Toscano <ptoscano@redhat.com>:

> On Saturday, 8 October 2016 18:27:21 CEST Matteo Cafasso wrote:
> > Patch ready for merging.
> >
> > v4:
> >
> >  - check return code of tsk_fs_attr_walk
> >  - pass TSK_FS_FILE_WALK_FLAG_NOSPARSE as additional flag to
> >  tsk_fs_attr_walk
> >
> > After discussing with TSK authors the behaviour is clear. [1]
>
> Thanks, this improves the situation a bit.
>
> > In case of COMPRESSED blocks, the callback will be called for all the
> > attributes no matter whether they are on disk or not (sparse). In
> > such cases, the block address will be 0. [2]
>
> Note that the API docs say:
>   For compressed and sparse attributes, the address *may* be zero.
> (emphasis is mine)
>

> My concern is that, if the address in such cases is
"unspecified", then
> the comparisons in "attrwalk_callback" are done against a
> random/unitialized value (which would be bad).
>

I understand your concerns. The data will not be wrong. Is the API
documentation being misleading.
The data *will* be 0 for SPARSE blocks and *might* be 0 or not for
compressed blocks based on certain criteria. See below.


> Also, if the block address would be zero, what's the point of having it
> among the blocks tsk_fs_attr_walk() iterates over?
>

This is due to the way NTFS organizes information and deals with its
compression and the way the API loops over them.

For each file or directory, there is a MFT (Master File Table) record which
consists in a linear repository of attributes (1Kb of size each).
Attributes can be resident within the MFT or non-resident according to
their size. The $DATA attribute storing the actual file content is an
example of typically non-resident ones.

Non-resident attributes are stored on disk in what is referred as data-runs
(contiguous blocks) which are then mapped within the attribute itself. A
typical file greater than 800 Bytes has the $DATA attribute containing a
map of data runs with their location on the disk. If the map itself is too
big for the $DATA attribute (this can happen if the actual content is too
fragmented), then extra records are created and their mapping is placed in
a special attribute called $ATTRIBUTE_LIST. [1]

When the given file is compressed (native NTFS compression, not application
level one), the algorithm goes on each data run within the attribute and:
[2]
 1 if the data run is zero filled, will set the corresponding blocks as
sparse and set their address to 0.
 2 if compressing the data run does not save any disk block, it will leave
it as is.
 3 if compressing the data run does save one or more blocks, the spared one
will be again marked as sparse and their address will be 0.

Note that the entire attribute will be marked as compressed no matter what
happened to the clusters on disk.

The logic loops through all non-resident attributes (which is what we want:
we want all the disk blocks allocated for that file). For each attribute,
it loops over all the blocks which that attributes maps and calls the
callback.

Our issue is the information at the origin of the sparse flag: the
information might come from the block (BAD/ALLOC/UNALLOC), or from the file
metadata (RAW,SPARSE,COMPRESSED,CONT, META). [3]

The tsk_fs_attr_walk() walks over the given attribute's blocks. In case we
are inspecting attributes of compressed files, the flag will report the
*file* status (COMPRESSED) yet will not able to tell us what the
compression algorithm did (1,2,3) to that block. It will still correctly
give us the address: 0 if sparse (case 1 or 3) or the correct number
otherwise (case 2 or 3).

[1]
https://en.wikipedia.org/wiki/NTFS#Attribute_lists.2C_attributes.2C_and_streams
[2] http://www.digital-evidence.org/fsfa/  -  Chapter 11
[3]
http://www.sleuthkit.org/sleuthkit/docs/api-docs/4.2/tsk__fs_8h.html#a1e6bf157f5d258191bf5d8ae31ee7148
-  "Note that some of these are set only by file_walk because they are
file-level details, such as compression and sparse."


> Thanks,
> --
> Pino Toscano
> _______________________________________________
> Libguestfs mailing list
> Libguestfs@redhat.com
> https://www.redhat.com/mailman/listinfo/libguestfs
>