Richard W.M. Jones
2020-Sep-17 12:39 UTC
[Libguestfs] [PATCH v3 0/8] Windows BitLocker support.
As discussed in the emails today, this is the third version addressing most points from the v1/v2 review. You will need to pair this with the changes in libguestfs-common from this series: https://www.redhat.com/archives/libguestfs/2020-September/msg00050.html Rich.
Richard W.M. Jones
2020-Sep-17 12:39 UTC
[Libguestfs] [PATCH v3 1/8] New APIs: cryptsetup-open and cryptsetup-close.
This commit deprecates luks-open/luks-open-ro/luks-close for the more generic sounding names cryptsetup-open/cryptsetup-close, which also correspond directly to the cryptsetup commands. The optional cryptsetup-open readonly flag is used to replace the functionality of luks-open-ro. The optional cryptsetup-open crypttype parameter can be used to select the type (corresponding to cryptsetup open --type), which allows us to open BitLocker-encrypted disks with no extra effort. As a convenience the crypttype parameter may be omitted, and libguestfs will use a heuristic (based on vfs-type output) to try to determine the correct type to use. The deprecated functions and the new functions are all (re-)written in OCaml. There is no new test here, unfortunately. It would be nice to test Windows BitLocker support in this new API, however the Linux tools do not support creating BitLocker disks, and while it is possible to create one under Windows, the smallest compressed disk I could create is 37M because of a mixture of the minimum support size for BitLocker disks and the fact that encrypted parts of NTFS cannot be compressed. --- .gitignore | 1 + daemon/Makefile.am | 3 + daemon/cryptsetup.ml | 78 +++++++++++++++++++ daemon/luks.c | 83 -------------------- generator/actions_core.ml | 109 ++++++++++++++++----------- generator/actions_core_deprecated.ml | 52 +++++++++++++ generator/daemon.ml | 5 +- generator/proc_nr.ml | 2 + gobject/Makefile.inc | 2 + lib/MAX_PROC_NR | 2 +- lib/guestfs.pod | 17 +++-- 11 files changed, 214 insertions(+), 140 deletions(-) diff --git a/.gitignore b/.gitignore index ab4cd0667..39354de51 100644 --- a/.gitignore +++ b/.gitignore @@ -147,6 +147,7 @@ Makefile.in /daemon/btrfs.mli /daemon/callbacks.ml /daemon/caml-stubs.c +/daemon/cryptsetup.mli /daemon/daemon_config.ml /daemon/daemon_utils_tests /daemon/devsparts.mli diff --git a/daemon/Makefile.am b/daemon/Makefile.am index e57e5a506..480192e1c 100644 --- a/daemon/Makefile.am +++ b/daemon/Makefile.am @@ -39,6 +39,7 @@ generator_built = \ blkid.mli \ btrfs.mli \ callbacks.ml \ + cryptsetup.mli \ devsparts.mli \ file.mli \ filearch.mli \ @@ -269,6 +270,7 @@ SOURCES_MLI = \ btrfs.mli \ callbacks.mli \ chroot.mli \ + cryptsetup.mli \ daemon.mli \ daemon_config.mli \ devsparts.mli \ @@ -310,6 +312,7 @@ SOURCES_ML = \ chroot.ml \ blkid.ml \ btrfs.ml \ + cryptsetup.ml \ devsparts.ml \ file.ml \ filearch.ml \ diff --git a/daemon/cryptsetup.ml b/daemon/cryptsetup.ml new file mode 100644 index 000000000..05538057e --- /dev/null +++ b/daemon/cryptsetup.ml @@ -0,0 +1,78 @@ +(* guestfs-inspection + * Copyright (C) 2009-2020 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + *) + +open Printf +open Unix + +open Std_utils + +open Utils + +let cryptsetup_open ?(readonly = false) ?crypttype device key mapname + (* Sanity check: /dev/mapper/mapname must not exist already. Note + * that the device-mapper control device (/dev/mapper/control) is + * always there, so you can't ever have mapname == "control". + *) + let devmapper = sprintf "/dev/mapper/%s" mapname in + if is_block_device devmapper then + failwithf "%s: device already exists" devmapper; + + (* Heuristically determine the encryption type. *) + let crypttype + match crypttype with + | Some s -> s + | None -> + let t = Blkid.vfs_type (Mountable.of_device device) in + match t with + | "crypto_LUKS" -> "luks" + | "BitLocker" -> "bitlk" + | _ -> + failwithf "%s: unknown encrypted device type" t in + + (* Write the key to a temporary file. *) + let keyfile, chan = Filename.open_temp_file "crypt" ".key" in + output_string chan key; + close_out chan; + + let args = ref [] in + List.push_back_list args ["-d"; keyfile]; + if readonly then List.push_back args "--readonly"; + List.push_back_list args ["open"; device; mapname; "--type"; crypttype]; + + (* Make sure we always remove the temporary file. *) + protect ~f:(fun () -> ignore (command "cryptsetup" !args)) + ~finally:(fun () -> unlink keyfile); + + udev_settle () + +let cryptsetup_close device + (* Must be /dev/mapper/... *) + if not (String.is_prefix device "/dev/mapper/") then + failwithf "%s: you must call this on the /dev/mapper device created by cryptsetup-open" device; + + let mapname = String.sub device 12 (String.length device - 12) in + ignore (command "cryptsetup" ["close"; mapname]); + + udev_settle () + +(* Deprecated APIs for backwards compatibility. *) +let luks_open device key mapname + cryptsetup_open ~crypttype:"luks" device key mapname +let luks_open_ro device key mapname + cryptsetup_open ~crypttype:"luks" ~readonly:true device key mapname +let luks_close = cryptsetup_close diff --git a/daemon/luks.c b/daemon/luks.c index d631cb100..51ced585d 100644 --- a/daemon/luks.c +++ b/daemon/luks.c @@ -81,89 +81,6 @@ remove_temp (char *tempfile) free (tempfile); } -static int -luks_open (const char *device, const char *key, const char *mapname, - int readonly) -{ - /* Sanity check: /dev/mapper/mapname must not exist already. Note - * that the device-mapper control device (/dev/mapper/control) is - * always there, so you can't ever have mapname == "control". - */ - CLEANUP_FREE char *devmapper = NULL; - if (asprintf (&devmapper, "/dev/mapper/%s", mapname) == -1) { - reply_with_perror ("asprintf"); - return -1; - } - if (access (devmapper, F_OK) == 0) { - reply_with_error ("%s: device already exists", devmapper); - return -1; - } - - char *tempfile = write_key_to_temp (key); - if (!tempfile) - return -1; - - const char *argv[MAX_ARGS]; - size_t i = 0; - - ADD_ARG (argv, i, "cryptsetup"); - ADD_ARG (argv, i, "-d"); - ADD_ARG (argv, i, tempfile); - if (readonly) ADD_ARG (argv, i, "--readonly"); - ADD_ARG (argv, i, "luksOpen"); - ADD_ARG (argv, i, device); - ADD_ARG (argv, i, mapname); - ADD_ARG (argv, i, NULL); - - CLEANUP_FREE char *err = NULL; - int r = commandv (NULL, &err, (const char * const *) argv); - remove_temp (tempfile); - - if (r == -1) { - reply_with_error ("%s", err); - return -1; - } - - udev_settle (); - - return 0; -} - -int -do_luks_open (const char *device, const char *key, const char *mapname) -{ - return luks_open (device, key, mapname, 0); -} - -int -do_luks_open_ro (const char *device, const char *key, const char *mapname) -{ - return luks_open (device, key, mapname, 1); -} - -int -do_luks_close (const char *device) -{ - /* Must be /dev/mapper/... */ - if (! STRPREFIX (device, "/dev/mapper/")) { - reply_with_error ("luks_close: you must call this on the /dev/mapper device created by luks_open"); - return -1; - } - - const char *mapname = &device[12]; - - CLEANUP_FREE char *err = NULL; - int r = command (NULL, &err, "cryptsetup", "luksClose", mapname, NULL); - if (r == -1) { - reply_with_error ("%s", err); - return -1; - } - - udev_settle (); - - return 0; -} - static int luks_format (const char *device, const char *key, int keyslot, const char *cipher) diff --git a/generator/actions_core.ml b/generator/actions_core.ml index 9a24a8d78..2092c990e 100644 --- a/generator/actions_core.ml +++ b/generator/actions_core.ml @@ -5664,52 +5664,6 @@ will be able to see every block device. This command also clears the LVM cache and performs a volume group scan." }; - { defaults with - name = "luks_open"; added = (1, 5, 1); - style = RErr, [String (Device, "device"); String (Key, "key"); String (PlainString, "mapname")], []; - optional = Some "luks"; - shortdesc = "open a LUKS-encrypted block device"; - longdesc = "\ -This command opens a block device which has been encrypted -according to the Linux Unified Key Setup (LUKS) standard. - -C<device> is the encrypted block device or partition. - -The caller must supply one of the keys associated with the -LUKS block device, in the C<key> parameter. - -This creates a new block device called F</dev/mapper/mapname>. -Reads and writes to this block device are decrypted from and -encrypted to the underlying C<device> respectively. - -If this block device contains LVM volume groups, then -calling C<guestfs_lvm_scan> with the C<activate> -parameter C<true> will make them visible. - -Use C<guestfs_list_dm_devices> to list all device mapper -devices." }; - - { defaults with - name = "luks_open_ro"; added = (1, 5, 1); - style = RErr, [String (Device, "device"); String (Key, "key"); String (PlainString, "mapname")], []; - optional = Some "luks"; - shortdesc = "open a LUKS-encrypted block device read-only"; - longdesc = "\ -This is the same as C<guestfs_luks_open> except that a read-only -mapping is created." }; - - { defaults with - name = "luks_close"; added = (1, 5, 1); - style = RErr, [String (Device, "device")], []; - optional = Some "luks"; - shortdesc = "close a LUKS device"; - longdesc = "\ -This closes a LUKS device that was created earlier by -C<guestfs_luks_open> or C<guestfs_luks_open_ro>. The -C<device> parameter must be the name of the LUKS mapping -device (ie. F</dev/mapper/mapname>) and I<not> the name -of the underlying block device." }; - { defaults with name = "luks_format"; added = (1, 5, 2); style = RErr, [String (Device, "device"); String (Key, "key"); Int "keyslot"], []; @@ -9753,4 +9707,67 @@ is used)." }; longdesc = "\ This returns the UUID of the LUKS device C<device>." }; + { defaults with + name = "cryptsetup_open"; added = (1, 43, 2); + style = RErr, [String (Device, "device"); String (Key, "key"); String (PlainString, "mapname")], [OBool "readonly"; OString "crypttype"]; + impl = OCaml "Cryptsetup.cryptsetup_open"; + optional = Some "luks"; + test_excuse = "no way to format BitLocker, and smallest device is huge"; + shortdesc = "open an encrypted block device"; + longdesc = "\ +This command opens a block device which has been encrypted +according to the Linux Unified Key Setup (LUKS) standard, +Windows BitLocker, or some other types. + +C<device> is the encrypted block device or partition. + +The caller must supply one of the keys associated with the +encrypted block device, in the C<key> parameter. + +This creates a new block device called F</dev/mapper/mapname>. +Reads and writes to this block device are decrypted from and +encrypted to the underlying C<device> respectively. + +C<mapname> cannot be C<\"control\"> because that name is reserved +by device-mapper. + +If the optional C<crypttype> parameter is not present then +libguestfs tries to guess the correct type (for example +LUKS or BitLocker). However you can override this by +specifying one of the following types: + +=over 4 + +=item C<luks> + +A Linux LUKS device. + +=item C<bitlk> + +A Windows BitLocker device. + +=back + +The optional C<readonly> flag, if set to true, creates a +read-only mapping. + +If this block device contains LVM volume groups, then +calling C<guestfs_lvm_scan> with the C<activate> +parameter C<true> will make them visible. + +Use C<guestfs_list_dm_devices> to list all device mapper +devices." }; + + { defaults with + name = "cryptsetup_close"; added = (1, 43, 2); + style = RErr, [String (Device, "device")], []; + impl = OCaml "Cryptsetup.cryptsetup_close"; + optional = Some "luks"; + shortdesc = "close an encrypted device"; + longdesc = "\ +This closes an encrypted device that was created earlier by +C<guestfs_cryptsetup_open>. The C<device> parameter must be +the name of the mapping device (ie. F</dev/mapper/mapname>) +and I<not> the name of the underlying block device." }; + ] diff --git a/generator/actions_core_deprecated.ml b/generator/actions_core_deprecated.ml index 8556763b7..299f9854b 100644 --- a/generator/actions_core_deprecated.ml +++ b/generator/actions_core_deprecated.ml @@ -847,4 +847,56 @@ allows you to specify the new size (in bytes) explicitly." }; This rescans all block devices and rebuilds the list of LVM physical volumes, volume groups and logical volumes." }; + { defaults with + name = "luks_open"; added = (1, 5, 1); + style = RErr, [String (Device, "device"); String (Key, "key"); String (PlainString, "mapname")], []; + impl = OCaml "Cryptsetup.luks_open"; + optional = Some "luks"; + deprecated_by = Replaced_by "cryptsetup_open"; + shortdesc = "open a LUKS-encrypted block device"; + longdesc = "\ +This command opens a block device which has been encrypted +according to the Linux Unified Key Setup (LUKS) standard. + +C<device> is the encrypted block device or partition. + +The caller must supply one of the keys associated with the +LUKS block device, in the C<key> parameter. + +This creates a new block device called F</dev/mapper/mapname>. +Reads and writes to this block device are decrypted from and +encrypted to the underlying C<device> respectively. + +If this block device contains LVM volume groups, then +calling C<guestfs_lvm_scan> with the C<activate> +parameter C<true> will make them visible. + +Use C<guestfs_list_dm_devices> to list all device mapper +devices." }; + + { defaults with + name = "luks_open_ro"; added = (1, 5, 1); + style = RErr, [String (Device, "device"); String (Key, "key"); String (PlainString, "mapname")], []; + impl = OCaml "Cryptsetup.luks_open_ro"; + optional = Some "luks"; + deprecated_by = Replaced_by "cryptsetup_open"; + shortdesc = "open a LUKS-encrypted block device read-only"; + longdesc = "\ +This is the same as C<guestfs_luks_open> except that a read-only +mapping is created." }; + + { defaults with + name = "luks_close"; added = (1, 5, 1); + style = RErr, [String (Device, "device")], []; + impl = OCaml "Cryptsetup.luks_close"; + optional = Some "luks"; + deprecated_by = Replaced_by "cryptsetup_close"; + shortdesc = "close a LUKS device"; + longdesc = "\ +This closes a LUKS device that was created earlier by +C<guestfs_luks_open> or C<guestfs_luks_open_ro>. The +C<device> parameter must be the name of the LUKS mapping +device (ie. F</dev/mapper/mapname>) and I<not> the name +of the underlying block device." }; + ] diff --git a/generator/daemon.ml b/generator/daemon.ml index 9edef462c..b1047427b 100644 --- a/generator/daemon.ml +++ b/generator/daemon.ml @@ -776,7 +776,8 @@ let generate_daemon_caml_stubs () pr "Val_bool (%s)" n; | OInt _ -> assert false | OInt64 _ -> assert false - | OString _ -> assert false + | OString _ -> + pr "caml_copy_string (%s)" n | OStringList _ -> assert false ); pr ";\n"; @@ -792,7 +793,7 @@ let generate_daemon_caml_stubs () | Bool n -> pr "Val_bool (%s)" n | Int n -> pr "Val_int (%s)" n | Int64 n -> pr "caml_copy_int64 (%s)" n - | String ((PlainString|Device|Pathname|Dev_or_Path), n) -> + | String ((PlainString|Device|Pathname|Dev_or_Path|Key), n) -> pr "caml_copy_string (%s)" n | String ((Mountable|Mountable_or_Path), n) -> pr "guestfs_int_daemon_copy_mountable (%s)" n diff --git a/generator/proc_nr.ml b/generator/proc_nr.ml index d8e7e1282..30e42864f 100644 --- a/generator/proc_nr.ml +++ b/generator/proc_nr.ml @@ -514,6 +514,8 @@ let proc_nr = [ 505, "f2fs_expand"; 506, "lvm_scan"; 507, "luks_uuid"; +508, "cryptsetup_open"; +509, "cryptsetup_close"; ] (* End of list. If adding a new entry, add it at the end of the list diff --git a/gobject/Makefile.inc b/gobject/Makefile.inc index 84ad960ef..650f8ddac 100644 --- a/gobject/Makefile.inc +++ b/gobject/Makefile.inc @@ -69,6 +69,7 @@ guestfs_gobject_headers= \ include/guestfs-gobject/optargs-copy_file_to_device.h \ include/guestfs-gobject/optargs-copy_file_to_file.h \ include/guestfs-gobject/optargs-cpio_out.h \ + include/guestfs-gobject/optargs-cryptsetup_open.h \ include/guestfs-gobject/optargs-disk_create.h \ include/guestfs-gobject/optargs-download_blocks.h \ include/guestfs-gobject/optargs-e2fsck.h \ @@ -162,6 +163,7 @@ guestfs_gobject_sources= \ src/optargs-copy_file_to_device.c \ src/optargs-copy_file_to_file.c \ src/optargs-cpio_out.c \ + src/optargs-cryptsetup_open.c \ src/optargs-disk_create.c \ src/optargs-download_blocks.c \ src/optargs-e2fsck.c \ diff --git a/lib/MAX_PROC_NR b/lib/MAX_PROC_NR index 055b6671a..77afe238f 100644 --- a/lib/MAX_PROC_NR +++ b/lib/MAX_PROC_NR @@ -1 +1 @@ -507 +509 diff --git a/lib/guestfs.pod b/lib/guestfs.pod index d746a41b1..bce9eb79f 100644 --- a/lib/guestfs.pod +++ b/lib/guestfs.pod @@ -585,17 +585,18 @@ Libguestfs allows you to access Linux guests which have been encrypted using whole disk encryption that conforms to the Linux Unified Key Setup (LUKS) standard. This includes nearly all whole disk encryption systems used by modern -Linux guests. +Linux guests. Windows BitLocker is also supported. -Use L</guestfs_vfs_type> to identify LUKS-encrypted block -devices (it returns the string C<crypto_LUKS>). +Use L</guestfs_vfs_type> to identify encrypted block +devices. For LUKS it returns the string C<crypto_LUKS>. +For Windows BitLocker it returns C<BitLocker>. -Then open these devices by calling L</guestfs_luks_open>. +Then open these devices by calling L</guestfs_cryptsetup_open>. Obviously you will require the passphrase! -Opening a LUKS device creates a new device mapper device +Opening an encrypted device creates a new device mapper device called F</dev/mapper/mapname> (where C<mapname> is the -string you supply to L</guestfs_luks_open>). +string you supply to L</guestfs_cryptsetup_open>). Reads and writes to this mapper device are decrypted from and encrypted to the underlying block device respectively. @@ -603,11 +604,11 @@ LVM volume groups on the device can be made visible by calling L</guestfs_vgscan> followed by L</guestfs_vg_activate_all>. The logical volume(s) can now be mounted in the usual way. -Use the reverse process to close a LUKS device. Unmount +Use the reverse process to close an encrypted device. Unmount any logical volumes on it, deactivate the volume groups by calling C<guestfs_vg_activate (g, 0, ["/dev/VG"])>. Then close the mapper device by calling -L</guestfs_luks_close> on the F</dev/mapper/mapname> +L</guestfs_cryptsetup_close> on the F</dev/mapper/mapname> device (I<not> the underlying encrypted block device). =head2 MOUNT LOCAL -- 2.27.0
Richard W.M. Jones
2020-Sep-17 12:39 UTC
[Libguestfs] [PATCH v3 2/8] fish: Update documentation to refer to cryptsetup-open/close and BitLocker.
--- fish/guestfish.pod | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/fish/guestfish.pod b/fish/guestfish.pod index c4fdd17b7..9f086f110 100644 --- a/fish/guestfish.pod +++ b/fish/guestfish.pod @@ -857,34 +857,40 @@ it, eg: Libguestfs has some support for Linux guests encrypted according to the Linux Unified Key Setup (LUKS) standard, which includes nearly all -whole disk encryption systems used by modern Linux guests. Currently -only LVM-on-LUKS is supported. +whole disk encryption systems used by modern Linux guests, and Windows +BitLocker. Identify encrypted block devices and partitions using L</vfs-type>: ><fs> vfs-type /dev/sda2 crypto_LUKS -Then open those devices using L</luks-open>. This creates a -device-mapper device called F</dev/mapper/luksdev>. +or: - ><fs> luks-open /dev/sda2 luksdev + ><fs> vfs-type /dev/sda2 + BitLocker + +Then open those devices using L</cryptsetup-open>. +This creates a device-mapper device called F</dev/mapper/name>. + + ><fs> cryptsetup-open /dev/sda2 name Enter key or passphrase ("key"): <enter the passphrase> -Finally you have to tell LVM to scan for volume groups on -the newly created mapper device: +For Linux guests you have to tell LVM to scan for volume groups on the +newly created mapper device: vgscan vg-activate-all true -The logical volume(s) can now be mounted in the usual way. +The filesystems or logical volumes can now be mounted in the usual way. -Before closing a LUKS device you must unmount any logical volumes on -it and deactivate the volume groups by calling C<vg-activate false VG> -on each one. Then you can close the mapper device: +Before closing an encrypted device you must unmount any logical +volumes on it and deactivate the volume groups by calling +C<vg-activate false VG> on each one. Then you can close the mapper +device: vg-activate false /dev/VG - luks-close /dev/mapper/luksdev + cryptsetup-close /dev/mapper/name =head1 WINDOWS PATHS -- 2.27.0
Richard W.M. Jones
2020-Sep-17 12:39 UTC
[Libguestfs] [PATCH v3 3/8] daemon: Rewrite list-filesystems implementation imperatively.
Simple refactoring to make the code clearer, should have no other effect. --- daemon/listfs.ml | 64 ++++++++++++++++++++++++------------------------ 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/daemon/listfs.ml b/daemon/listfs.ml index 903f363d0..3b71471ae 100644 --- a/daemon/listfs.ml +++ b/daemon/listfs.ml @@ -28,38 +28,36 @@ let rec list_filesystems () let has_lvm2 = Optgroups.lvm2_available () in let has_ldm = Optgroups.ldm_available () in + let ret = ref [] in + (* Devices. *) let devices = Devsparts.list_devices () in let devices = List.filter is_not_partitioned_device devices in - let ret = List.filter_map check_with_vfs_type devices in + List.iter (check_with_vfs_type ret) devices; (* Partitions. *) let partitions = Devsparts.list_partitions () in let partitions = List.filter is_partition_can_hold_filesystem partitions in - let ret = ret @ List.filter_map check_with_vfs_type partitions in + List.iter (check_with_vfs_type ret) partitions; (* MD. *) let mds = Md.list_md_devices () in let mds = List.filter is_not_partitioned_device mds in - let ret = ret @ List.filter_map check_with_vfs_type mds in + List.iter (check_with_vfs_type ret) mds; (* LVM. *) - let ret - if has_lvm2 then ( - let lvs = Lvm.lvs () in - ret @ List.filter_map check_with_vfs_type lvs - ) - else ret in + if has_lvm2 then ( + let lvs = Lvm.lvs () in + List.iter (check_with_vfs_type ret) lvs + ); (* LDM. *) - let ret - if has_ldm then ( - let ldmvols = Ldm.list_ldm_volumes () in - ret @ List.filter_map check_with_vfs_type ldmvols - ) - else ret in + if has_ldm then ( + let ldmvols = Ldm.list_ldm_volumes () in + List.iter (check_with_vfs_type ret) ldmvols + ); - List.flatten ret + !ret (* Look to see if device can directly contain filesystem (RHBZ#590167). * Partitioned devices cannot contain filesystem, so we will exclude @@ -120,11 +118,10 @@ and is_mbr_extended parttype device partnum Parted.part_get_mbr_part_type device partnum = "extended" (* Use vfs-type to check for a filesystem of some sort of [device]. - * Returns [Some [device, vfs_type; ...]] if found (there may be - * multiple devices found in the case of btrfs), else [None] if nothing - * is found. + * Appends (device, vfs_type) to the ret parameter (there may be + * multiple devices found in the case of btrfs). *) -and check_with_vfs_type device +and check_with_vfs_type ret device let mountable = Mountable.of_device device in let vfs_type try Blkid.vfs_type mountable @@ -135,18 +132,18 @@ and check_with_vfs_type device "" in if vfs_type = "" then - Some [mountable, "unknown"] + List.push_back ret (mountable, "unknown") (* Ignore all "*_member" strings. In libblkid these are returned * for things which are members of some RAID or LVM set, most * importantly "LVM2_member" which is a PV. *) else if String.is_suffix vfs_type "_member" then - None + () (* Ignore LUKS-encrypted partitions. These are also containers, as above. *) else if vfs_type = "crypto_LUKS" then - None + () (* A single btrfs device can turn into many volumes. *) else if vfs_type = "btrfs" then ( @@ -162,15 +159,18 @@ and check_with_vfs_type device fun { Structs.btrfssubvolume_id = id } -> id <> default_volume ) vols in - Some ( - (mountable, vfs_type) (* whole device = default volume *) - :: List.map ( - fun { Structs.btrfssubvolume_path = path } -> - let mountable = Mountable.of_btrfsvol device path in - (mountable, "btrfs") - ) vols - ) + (* whole device = default volume *) + List.push_back ret (mountable, vfs_type); + + (* subvolumes *) + List.push_back_list ret ( + List.map ( + fun { Structs.btrfssubvolume_path = path } -> + let mountable = Mountable.of_btrfsvol device path in + (mountable, "btrfs") + ) vols + ) ) else - Some [mountable, vfs_type] + List.push_back ret (mountable, vfs_type) -- 2.27.0
Richard W.M. Jones
2020-Sep-17 12:40 UTC
[Libguestfs] [PATCH v3 4/8] daemon: Ignore BitLocker disks in list-filesystems API.
--- daemon/listfs.ml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/daemon/listfs.ml b/daemon/listfs.ml index 3b71471ae..a4c917d12 100644 --- a/daemon/listfs.ml +++ b/daemon/listfs.ml @@ -141,8 +141,8 @@ and check_with_vfs_type ret device else if String.is_suffix vfs_type "_member" then () - (* Ignore LUKS-encrypted partitions. These are also containers, as above. *) - else if vfs_type = "crypto_LUKS" then + (* Ignore encrypted partitions. These are also containers, as above. *) + else if vfs_type = "crypto_LUKS" || vfs_type = "BitLocker" then () (* A single btrfs device can turn into many volumes. *) -- 2.27.0
Richard W.M. Jones
2020-Sep-17 12:40 UTC
[Libguestfs] [PATCH v3 5/8] daemon: Reimplement list_dm_devices API in OCaml.
Simple refactoring. The only annoying point is requiring an extra module because of OCaml module dependency restrictions. --- .gitignore | 1 + daemon/Makefile.am | 3 ++ daemon/lvm.c | 76 --------------------------------------- daemon/lvm_dm.ml | 39 ++++++++++++++++++++ generator/actions_core.ml | 1 + 5 files changed, 44 insertions(+), 76 deletions(-) diff --git a/.gitignore b/.gitignore index 39354de51..71a84fd9d 100644 --- a/.gitignore +++ b/.gitignore @@ -164,6 +164,7 @@ Makefile.in /daemon/link.mli /daemon/listfs.mli /daemon/lvm.mli +/daemon/lvm_dm.mli /daemon/lvm-tokenization.c /daemon/md.mli /daemon/mount.mli diff --git a/daemon/Makefile.am b/daemon/Makefile.am index 480192e1c..038be592c 100644 --- a/daemon/Makefile.am +++ b/daemon/Makefile.am @@ -50,6 +50,7 @@ generator_built = \ link.mli \ listfs.mli \ lvm.mli \ + lvm_dm.mli \ md.mli \ mount.mli \ optgroups.ml \ @@ -289,6 +290,7 @@ SOURCES_MLI = \ link.mli \ listfs.mli \ lvm.mli \ + lvm_dm.mli \ lvm_utils.mli \ md.mli \ mount.mli \ @@ -321,6 +323,7 @@ SOURCES_ML = \ link.ml \ lvm.ml \ lvm_utils.ml \ + lvm_dm.ml \ findfs.ml \ md.ml \ mount.ml \ diff --git a/daemon/lvm.c b/daemon/lvm.c index a78b344db..039240866 100644 --- a/daemon/lvm.c +++ b/daemon/lvm.c @@ -764,82 +764,6 @@ do_lvm_canonical_lv_name (const char *device) return canonical; /* caller frees */ } -/* List everything in /dev/mapper which *isn't* an LV (RHBZ#688062). */ -char ** -do_list_dm_devices (void) -{ - CLEANUP_FREE_STRINGSBUF DECLARE_STRINGSBUF (ret); - struct dirent *d; - DIR *dir; - int r; - - dir = opendir ("/dev/mapper"); - if (!dir) { - reply_with_perror ("opendir: /dev/mapper"); - return NULL; - } - - while (1) { - CLEANUP_FREE char *devname = NULL; - - errno = 0; - d = readdir (dir); - if (d == NULL) break; - - /* Ignore . and .. */ - if (STREQ (d->d_name, ".") || STREQ (d->d_name, "..")) - continue; - - /* Ignore /dev/mapper/control which is used internally by dm. */ - if (STREQ (d->d_name, "control")) - continue; - - if (asprintf (&devname, "/dev/mapper/%s", d->d_name) == -1) { - reply_with_perror ("asprintf"); - closedir (dir); - return NULL; - } - - /* Ignore dm devices which are LVs. */ - r = lv_canonical (devname, NULL); - if (r == -1) { - closedir (dir); - return NULL; - } - if (r) - continue; - - /* Not an LV, so add it. */ - if (add_string (&ret, devname) == -1) { - closedir (dir); - return NULL; - } - } - - /* Did readdir fail? */ - if (errno != 0) { - reply_with_perror ("readdir: /dev/mapper"); - closedir (dir); - return NULL; - } - - /* Close the directory handle. */ - if (closedir (dir) == -1) { - reply_with_perror ("closedir: /dev/mapper"); - return NULL; - } - - /* Sort the output (may be empty). */ - if (ret.size > 0) - sort_strings (ret.argv, ret.size); - - /* NULL-terminate the list. */ - if (end_stringsbuf (&ret) == -1) - return NULL; - - return take_stringsbuf (&ret); -} - char * do_vgmeta (const char *vg, size_t *size_r) { diff --git a/daemon/lvm_dm.ml b/daemon/lvm_dm.ml new file mode 100644 index 000000000..474ba8377 --- /dev/null +++ b/daemon/lvm_dm.ml @@ -0,0 +1,39 @@ +(* guestfs-inspection + * Copyright (C) 2009-2020 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + *) + +open Unix +open Printf + +open Std_utils + +open Utils + +(* List everything in /dev/mapper which *isn't* an LV (RHBZ#688062). *) +let list_dm_devices () + let ds = Sys.readdir "/dev/mapper" in + let ds = Array.to_list ds in + let ds = List.sort compare ds in + + (* Ignore /dev/mapper/control which is used internally by d-m. *) + let ds = List.filter ((<>) "control") ds in + + let ds = List.map ((^) "/dev/mapper/") ds in + + (* Only keep devices which are _not_ LVs. *) + let ds = List.filter (fun d -> Lvm_utils.lv_canonical d = None) ds in + ds diff --git a/generator/actions_core.ml b/generator/actions_core.ml index 2092c990e..9d0c0fe76 100644 --- a/generator/actions_core.ml +++ b/generator/actions_core.ml @@ -6180,6 +6180,7 @@ parameter." }; { defaults with name = "list_dm_devices"; added = (1, 11, 15); style = RStringList (RDevice, "devices"), [], []; + impl = OCaml "Lvm_dm.list_dm_devices"; shortdesc = "list device mapper devices"; longdesc = "\ List all device mapper devices. -- 2.27.0
Richard W.M. Jones
2020-Sep-17 12:40 UTC
[Libguestfs] [PATCH v3 6/8] daemon: Search device-mapper devices for list-filesystems API.
In case any bare filesystems were decrypted using cryptsetup-open, they would appear as /dev/mapper/name devices. Since list-filesystems did not consider those when searching for filesystems, the unencrypted filesystems would not be returned. Note that previously this worked for LUKS because the common case (eg. for Fedora) was that whole devices were encrypted and thoes devices contained LVs, so luks-open + vgactivate would activate the LVs which would then be found by list-filesystems. For Windows BitLocker, the common case seems to be that each separate NTFS filesystem is contained in a separate BitLocker wrapper. --- daemon/listfs.ml | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/daemon/listfs.ml b/daemon/listfs.ml index a4c917d12..d8add5005 100644 --- a/daemon/listfs.ml +++ b/daemon/listfs.ml @@ -35,6 +35,14 @@ let rec list_filesystems () let devices = List.filter is_not_partitioned_device devices in List.iter (check_with_vfs_type ret) devices; + (* Device-mapper devices. + * We include these in case any encrypted devices contain + * direct filesystems. + *) + let devices = Lvm_dm.list_dm_devices () in + let devices = List.filter is_not_partitioned_device devices in + List.iter (check_with_vfs_type ret) devices; + (* Partitions. *) let partitions = Devsparts.list_partitions () in let partitions = List.filter is_partition_can_hold_filesystem partitions in @@ -64,6 +72,11 @@ let rec list_filesystems () * such devices. *) and is_not_partitioned_device device + let device + if String.is_prefix device "/dev/mapper/" then + Unix_utils.Realpath.realpath device + else + device in assert (String.is_prefix device "/dev/"); let dev_name = String.sub device 5 (String.length device - 5) in let dev_dir = "/sys/block/" ^ dev_name in -- 2.27.0
Richard W.M. Jones
2020-Sep-17 12:40 UTC
[Libguestfs] [PATCH v3 7/8] daemon: lvm_canonical_lv_name: Return EINVAL if called with non-LV.
Previously callers were unable to distinguish a regular error (like an I/O error) from the case where you call this API on something which is valid but not a logical volume. Set errno to a known value in this case. --- daemon/lvm.c | 2 +- generator/actions_core.ml | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/daemon/lvm.c b/daemon/lvm.c index 039240866..841dc4b6b 100644 --- a/daemon/lvm.c +++ b/daemon/lvm.c @@ -757,7 +757,7 @@ do_lvm_canonical_lv_name (const char *device) return NULL; if (r == 0) { - reply_with_error ("%s: not a logical volume", device); + reply_with_error_errno (EINVAL, "%s: not a logical volume", device); return NULL; } diff --git a/generator/actions_core.ml b/generator/actions_core.ml index 9d0c0fe76..806565b19 100644 --- a/generator/actions_core.ml +++ b/generator/actions_core.ml @@ -5994,7 +5994,8 @@ might find to the canonical name. For example, F</dev/mapper/VG-LV> is converted to F</dev/VG/LV>. This command returns an error if the C<lvname> parameter does -not refer to a logical volume. +not refer to a logical volume. In this case errno will be +set to C<EINVAL>. See also C<guestfs_is_lv>, C<guestfs_canonical_device_name>." }; -- 2.27.0
Richard W.M. Jones
2020-Sep-17 12:40 UTC
[Libguestfs] [PATCH v3 8/8] lib/canonical-name.c: Hide EINVAL error from underlying API call.
When guestfs_lvm_canonical_lv_name was called with a /dev/dm* or /dev/mapper* name which was not an LV then a noisy error would be printed. This would typically have happened with encrypted disks, and now happens very noticably when inspecting Windows BitLocker- encrypted guests. Using the modified error behaviour of this API from the previous commit we can now hide that error in that specific case. (Even in this case the underlying error message still gets logged in debug output). --- lib/canonical-name.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/lib/canonical-name.c b/lib/canonical-name.c index 052bbf12c..e0c7918b4 100644 --- a/lib/canonical-name.c +++ b/lib/canonical-name.c @@ -46,9 +46,14 @@ guestfs_impl_canonical_device_name (guestfs_h *g, const char *device) } else if (STRPREFIX (device, "/dev/mapper/") || STRPREFIX (device, "/dev/dm-")) { - /* XXX hide errors */ + guestfs_push_error_handler (g, NULL, NULL); ret = guestfs_lvm_canonical_lv_name (g, device); - if (ret == NULL) + guestfs_pop_error_handler (g); + /* EINVAL is expected if the device is somelike a LUKS- or + * BitLocker-encrypted volume, so simply return the original + * name in that case. + */ + if (ret == NULL && guestfs_last_errno (g) == EINVAL) ret = safe_strdup (g, device); } else -- 2.27.0
Martin Kletzander
2020-Oct-06 13:29 UTC
Re: [Libguestfs] [PATCH v3 0/8] Windows BitLocker support.
On Thu, Sep 17, 2020 at 01:39:56PM +0100, Richard W.M. Jones wrote:>As discussed in the emails today, this is the third version addressing >most points from the v1/v2 review. > >You will need to pair this with the changes in libguestfs-common from >this series: > >https://www.redhat.com/archives/libguestfs/2020-September/msg00050.html >I'm going to look at this, but I do not have a machine to try this on, so I will not be able to do any functional testing (i.e. the review will be very much incomplete). But from my limited testing I can tell you it breaks the build for me with the following errors: Making all in golang make[2]: Entering directory '/home/nert/dev/libguestfs/golang' ../run go install libguestfs.org/guestfs # libguestfs.org/guestfs src/libguestfs.org/guestfs/guestfs.go:3964:34: could not determine kind of name for C.GUESTFS_CRYPTSETUP_OPEN_CRYPTTYPE_BITMASK src/libguestfs.org/guestfs/guestfs.go:3960:34: could not determine kind of name for C.GUESTFS_CRYPTSETUP_OPEN_READONLY_BITMASK src/libguestfs.org/guestfs/guestfs.go:3925:10: could not determine kind of name for C.guestfs_cryptsetup_close src/libguestfs.org/guestfs/guestfs.go:3970:10: could not determine kind of name for C.guestfs_cryptsetup_open_argv make[2]: *** [Makefile:2509: pkg/linux_amd64/libguestfs.org/guestfs.a] Error 2 make[2]: Leaving directory '/home/nert/dev/libguestfs/golang' so just so you know before I get to looking at the code. Have a nice day, Martin>Rich. > > >_______________________________________________ >Libguestfs mailing list >Libguestfs@redhat.com >https://www.redhat.com/mailman/listinfo/libguestfs
Richard W.M. Jones
2020-Oct-06 14:22 UTC
Re: [Libguestfs] [PATCH v3 0/8] Windows BitLocker support.
On Tue, Oct 06, 2020 at 03:29:33PM +0200, Martin Kletzander wrote:> On Thu, Sep 17, 2020 at 01:39:56PM +0100, Richard W.M. Jones wrote: > >As discussed in the emails today, this is the third version addressing > >most points from the v1/v2 review. > > > >You will need to pair this with the changes in libguestfs-common from > >this series: > > > >https://www.redhat.com/archives/libguestfs/2020-September/msg00050.html > > > > I'm going to look at this, but I do not have a machine to try this on, so I will > not be able to do any functional testing (i.e. the review will be very much > incomplete). But from my limited testing I can tell you it breaks the build for > me with the following errors:I have --disable-golang ... Enabling it shows the exact same error. It's actually (another) bug caused by moving <guestfs.h> from lib/ to include/ so that the golang bindings are using the installed copy of guestfs.h (which lacks this newly added binding) rather than the local copy. I'll push a fix in a bit. Rich.> Making all in golang > make[2]: Entering directory '/home/nert/dev/libguestfs/golang' > ../run go install libguestfs.org/guestfs > # libguestfs.org/guestfs > src/libguestfs.org/guestfs/guestfs.go:3964:34: could not determine kind of name for C.GUESTFS_CRYPTSETUP_OPEN_CRYPTTYPE_BITMASK > src/libguestfs.org/guestfs/guestfs.go:3960:34: could not determine kind of name for C.GUESTFS_CRYPTSETUP_OPEN_READONLY_BITMASK > src/libguestfs.org/guestfs/guestfs.go:3925:10: could not determine kind of name for C.guestfs_cryptsetup_close > src/libguestfs.org/guestfs/guestfs.go:3970:10: could not determine kind of name for C.guestfs_cryptsetup_open_argv > make[2]: *** [Makefile:2509: pkg/linux_amd64/libguestfs.org/guestfs.a] Error 2 > make[2]: Leaving directory '/home/nert/dev/libguestfs/golang' > > so just so you know before I get to looking at the code. > > Have a nice day, > Martin > > >Rich. > > > > > >_______________________________________________ > >Libguestfs mailing list > >Libguestfs@redhat.com > >https://www.redhat.com/mailman/listinfo/libguestfs-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
Martin Kletzander
2020-Oct-09 09:47 UTC
Re: [Libguestfs] [PATCH v3 8/8] lib/canonical-name.c: Hide EINVAL error from underlying API call.
On Thu, Sep 17, 2020 at 01:40:04PM +0100, Richard W.M. Jones wrote:>When guestfs_lvm_canonical_lv_name was called with a /dev/dm* or >/dev/mapper* name which was not an LV then a noisy error would be >printed. This would typically have happened with encrypted disks, and >now happens very noticably when inspecting Windows BitLocker- >encrypted guests. > >Using the modified error behaviour of this API from the previous >commit we can now hide that error in that specific case. (Even in >this case the underlying error message still gets logged in debug >output). >--- > lib/canonical-name.c | 9 +++++++-- > 1 file changed, 7 insertions(+), 2 deletions(-) > >diff --git a/lib/canonical-name.c b/lib/canonical-name.c >index 052bbf12c..e0c7918b4 100644 >--- a/lib/canonical-name.c >+++ b/lib/canonical-name.c >@@ -46,9 +46,14 @@ guestfs_impl_canonical_device_name (guestfs_h *g, const char *device) > } > else if (STRPREFIX (device, "/dev/mapper/") || > STRPREFIX (device, "/dev/dm-")) { >- /* XXX hide errors */ >+ guestfs_push_error_handler (g, NULL, NULL);Doesn't this hide the other error which might be important as well? The only one I can find it the file not existing, but it means this function might fail without an error message if I understand it correctly.> ret = guestfs_lvm_canonical_lv_name (g, device); >- if (ret == NULL) >+ guestfs_pop_error_handler (g); >+ /* EINVAL is expected if the device is somelike a LUKS- or >+ * BitLocker-encrypted volume, so simply return the original >+ * name in that case. >+ */ >+ if (ret == NULL && guestfs_last_errno (g) == EINVAL) > ret = safe_strdup (g, device); > } > else >-- >2.27.0 > >_______________________________________________ >Libguestfs mailing list >Libguestfs@redhat.com >https://www.redhat.com/mailman/listinfo/libguestfs
Martin Kletzander
2020-Oct-09 15:02 UTC
Re: [Libguestfs] [PATCH v3 0/8] Windows BitLocker support.
On Thu, Sep 17, 2020 at 01:39:56PM +0100, Richard W.M. Jones wrote:>As discussed in the emails today, this is the third version addressing >most points from the v1/v2 review. > >You will need to pair this with the changes in libguestfs-common from >this series: > >https://www.redhat.com/archives/libguestfs/2020-September/msg00050.html >The series looks good, I went through as much as I possibly could and that's why it took me so much time, sorry for that. What I like about it is that it is essentially just a different --type for the cryptsetup command and the rest is just refactors and clean-ups. So I'd say ACK from me (if that means anything), it's just one thing that I wanted to try out (and maybe you could create a test for it) and it did not work as expected. Basically what I did was create a small disk, create one partition over the whole disk, then cryptsetup luksFormat the partition, open it and format it with a filesystem (without any LVM). That is one of the things you were adding support for, but it is not limited to Windows Bitlocker setup, it can just as well be a custom setup when installing any Linux distro. Even after quite a bit of fighting, rebuilding the appliance and so on I did not manage for it to show up in the list-filesystems or even do a cryptsetup-open on the partition even though it uses an appliance built from git master with the patches applied. But I'm quite sure I could've done something wrong, so if that works for you, that's enough. Still, since you cannot do the test for Bitlocker, my idea was that you could make the test for non-lvm parition encrypted by LUKS as that would check some of the other code. Have a nice weekend, Martin>Rich. > > >_______________________________________________ >Libguestfs mailing list >Libguestfs@redhat.com >https://www.redhat.com/mailman/listinfo/libguestfs
Richard W.M. Jones
2020-Oct-09 15:33 UTC
Re: [Libguestfs] [PATCH v3 0/8] Windows BitLocker support.
On Fri, Oct 09, 2020 at 05:02:57PM +0200, Martin Kletzander wrote:> Basically what I did was create a small disk, create one partition > over the whole disk, then cryptsetup luksFormat the partition, open > it and format it with a filesystem (without any LVM). That is one > of the things you were adding support for, but it is not limited to > Windows Bitlocker setup, it can just as well be a custom setup when > installing any Linux distro. > > Even after quite a bit of fighting, rebuilding the appliance and so > on I did not manage for it to show up in the list-filesystems or > even do a cryptsetup-open on the partition even though it uses an > appliance built from git master with the patches applied. But I'm > quite sure I could've done something wrong, so if that works for > you, that's enough.There's something in the test suite that already does this, so $ make && make -C test-data check should produce test-data/phony-guests/fedora-luks.img (see test-data/phony-guests/make-fedora-img.pl for how). This image can be opened: $ guestfish --ro -a test-data/phony-guests/fedora-luks.img -i Enter key or passphrase ("/dev/sda2"): FEDORA Welcome to guestfish, the guest filesystem shell for editing virtual machine filesystems and disk images. Type: ‘help’ for help on commands ‘man’ to read the manual ‘quit’ to quit the shell Operating system: Fedora release 14 (Phony) /dev/VG/Root mounted on / /dev/sda1 mounted on /boot ><fs> list-devices /dev/sda ><fs> list-partitions /dev/sda1 /dev/sda2 ><fs> vfs-type /dev/sda2 crypto_LUKS However ...> Still, since you cannot do the test for Bitlocker, my idea was that > you could make the test for non-lvm parition encrypted by LUKS as > that would check some of the other code.... cryptsetup cannot create a new BitLocker disk, which is rather unfortunate. I created a BitLocker disk using Windows, and I'll privately send you a link, but because of the cryptsetup problem there's no way to automate this kind of test. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
Possibly Parallel Threads
- Re: [PATCH v3 0/8] Windows BitLocker support.
- Re: [PATCH v3 0/8] Windows BitLocker support.
- [PATCH 1/7] New APIs: cryptsetup-open and cryptsetup-close.
- [PATCH v2 1/7] New APIs: cryptsetup-open and cryptsetup-close.
- Re: [PATCH v2 1/7] New APIs: cryptsetup-open and cryptsetup-close.