similar to: [PATCH 0/2] blkls API to extract unallocated blocks

blkcat allows to extract files given their data units (clusters for NTFS, fragments for ExtX). This API allows to recover files which icat cannot. For example icat often fails retrieving deleted files on Ext3/Ext4 filesystems. As this API works at data unit level, some garbage at the beginning or at the end of the files is expected. An example of blkcat can be found here:

The filesystem_walk0 API parses the FS internals of a partition and returns a list of all the files and directories contained within. It list deleted files and directories as well. For each node, it reports its relative path, its inode and its allocation status. The output is serialised in XDR format and written to the given file. The command is similar to The Sleuth Kit "fls -rp

v2: - Rebase on top of master Matteo Cafasso (2): New API: download_blocks Added download_blocks API test daemon/sleuthkit.c | 41 ++++++++++++++++++++++++++- generator/actions.ml | 24 ++++++++++++++++ gobject/Makefile.inc | 2 ++ src/MAX_PROC_NR | 2 +- tests/tsk/Makefile.am | 1 +

With this API we complete the set of functions required to extract deleted files/data from most of the available filesystems. The function allows to extract data units (blocks) within a given range from a partition. The tests show an example on how the function can be used to retrieve deleted data. Matteo Cafasso (2): New API: download_blocks Added download_blocks API test

The ffind API allows to retrieve a file name from a device given its inode. The function returns a struct "tsknode" which contains the file name, its inode and it's allocation status. The struct will be employed by other APIs as well (fls, ifind etc..). $ ./run guestfish --ro -a /home/noxdafox/disks/ubuntu.qcow2 ><fs> run ><fs> ffind /dev/sda1 2 tsk_name: /

The Sleuth Kit is a filesystem forensic tool for accessing disk volumes and extracting digital evidence from. http://www.sleuthkit.org/ The icat API allows to download a file from a device given its metadata number (inode). It supports multiple filesystem types. The icat command allows to access to otherwise unreachable files such as filesystem data structures and deleted files which content is

As discussed in the topic: https://www.redhat.com/archives/libguestfs/2016-March/msg00018.html I'd like to add to libguestfs the disk forensics capabilities offered by The Sleuth Kit. http://www.sleuthkit.org/ The two APIs I'm adding with the patch are a simple example of which type of features TSK can enable. The icat API it's very similar to the previously added ntfscat_i but it

Renamed ffind API to find_inode. Renamed tsknode struct to tsk_node. Changed struct field from int64 to uint64. As pointed out on IRC it would be better to agree on some naming convention. One option would be to prefix all the forensics APIs with tsk_ as TSK (The Sleuth Kit) is the main tool used for implementing them. Other option could be giving generic names allowing us to change underlying

The command name is a bit confusing because it's similar to "cat" but act as "donwload". Therefore I am renaming it with a more clear name. At the same go I cleaned up a bit the code following the standards and improved the API documentation. This patch is ready for review. Code available at: https://github.com/noxdafox/libguestfs/tree/download_inode Matteo Cafasso (3):

"icat" name comes from the employed command line tool which might be replaced later on with a different implementation. The command name is a bit confusing because it's similar to "cat" but act as "donwload". Therefore I am renaming it with a more clear name. At the same time I cleaned up a bit the code and improved it's readability and code comments. This

Thanks, I have pushed this patch series. Could you consider changing: > + optional = Some "icat"; I think it would be nice to have a single feature, and to call the feature "sleuthkit" or "forensics" or something like that. We don't need to have one feature per API since installation of a single package (sleuthkit) is sufficient to make all the APIs

Patch ready for merging. v4: - check return code of tsk_fs_attr_walk - pass TSK_FS_FILE_WALK_FLAG_NOSPARSE as additional flag to tsk_fs_attr_walk After discussing with TSK authors the behaviour is clear. [1] In case of COMPRESSED blocks, the callback will be called for all the attributes no matter whether they are on disk or not (sparse). In such cases, the block address will be 0. [2] So

v6: - added metadata reallocation flag in tsk_flags Certain filesystems separate file name structures and metadata ones. Therefore, deleted entries with file name structures in an unallocated state might point to metadata structures which have been reallocated to new files. A flag set to 1 is generally an indication that the information gathered from the metadata structure (file

v3: - fixed attribute walk callback: checking against TSK_FS_BLOCK_FLAG_RAW flag would exclude compressed data blocks which are still important. Yet we want to exclude sparse blocks (TSK_FS_BLOCK_FLAG_SPARSE) as they are not stored on the disk. Matteo Cafasso (3): New API: internal_find_block New API: find_block find_block: added API tests daemon/tsk.c | 91

v7: - iterate over output file instead of reading it into memory Instead of reading the whole output file in memory and iterating over the resulting buffer, use XDR primitives to directly iterate over the file itself. This should reduce the API memory consumption. Patch ready for review. Code available at: https://github.com/noxdafox/libguestfs/tree/filesystem_walk Matteo Cafasso

v2: - use boolean field in struct - move refactoring to previous series Matteo Cafasso (3): New API: internal_find_block New API: find_block find_block: added API tests daemon/tsk.c | 90 ++++++++++++++++++++++++++++++++++++++++++++ generator/actions.ml | 25 ++++++++++++ src/MAX_PROC_NR | 2 +- src/tsk.c | 17 +++++++++

v8: - rebase on master - bump version to 1.33.37 - squash commits 1, 2, 3 Kept original commits messages when squashing them. Matteo Cafasso (3): New API: internal_filesystem_walk New API: filesystem_walk lib: Added filesystem_walk command tests daemon/Makefile.am | 4 +- daemon/tsk.c | 249 ++++++++++++++++++++++++++++++++++++++

The filesystem_walk API parses the FS internals of a partition and returns a list of all the files and directories contained within. It list deleted files and directories as well. For each node, it reports its relative path, its inode and its allocation status. This is the end user API for inspecting a disk partition content. The command can handle filenames with special characters. Example

v9: - add missing files: java/Makefile.inc, java/com/redhat/et/libguestfs/.gitignore, gobject/Makefile.inc - reserve space in tsk_dirent struct for future usage - use int instead of bool_t type - improve API documentation Matteo Cafasso (3): New API: internal_filesystem_walk New API: filesystem_walk lib: Added filesystem_walk command tests daemon/Makefile.am

The find_inode API allows the User to search all the entries referring to a given inode and returns a tsk_dirent structure for each of them. As I didn't want to change unrelated code, there is a little bit of code duplication at the moment. Plan is to refactor the logic in a dedicated set of patches. Matteo Cafasso (3): New API: internal_find_inode New API: find_inode find_inode: added