Richard W.M. Jones
2010-Aug-20 10:38 UTC
[Libguestfs] [PATCH febootstrap 0/8] Add support for building an ext2-based appliance
This patch series adds support for outputing an ext2-based appliance from febootstrap-supermin-helper. The usage is very simple, you just add '-f ext2' flag and the name of the appliance file that you want to write to. The implementation uses libext2fs, which is a very low level way to create ext2 filesystems from scratch. We'd like to use libguestfs, but that's an obvious reason why we can't do that :-) These patches are *not* finalized yet. However I doubt that patches 0001-0007 will change very much (and especially not 0001-0004), so those can be reviewed. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
Richard W.M. Jones
2010-Aug-20 10:38 UTC
[Libguestfs] [PATCH febootstrap 1/8] Move febootstrap-supermin-helper into helper/ subdirectory.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora -------------- next part -------------->From d515b7602c755049960308386782b568c48d2960 Mon Sep 17 00:00:00 2001From: Richard Jones <rjones at redhat.com> Date: Thu, 19 Aug 2010 13:21:37 +0100 Subject: [PATCH 1/8] Move febootstrap-supermin-helper into helper/ subdirectory. This will allow us to split up the monolithic C program into separate files for ease of modification. This patch is just file rearrangement. --- Makefile.am | 24 +- configure.ac | 2 +- febootstrap-supermin-helper.c | 997 -------------------------------- febootstrap-supermin-helper.pod | 109 ---- helper/Makefile.am | 47 ++ helper/febootstrap-supermin-helper.pod | 109 ++++ helper/main.c | 997 ++++++++++++++++++++++++++++++++ 7 files changed, 1156 insertions(+), 1129 deletions(-) delete mode 100644 febootstrap-supermin-helper.c delete mode 100644 febootstrap-supermin-helper.pod create mode 100644 helper/Makefile.am create mode 100644 helper/febootstrap-supermin-helper.pod create mode 100644 helper/main.c diff --git a/Makefile.am b/Makefile.am index bb3b505..9db86e9 100644 --- a/Makefile.am +++ b/Makefile.am @@ -19,7 +19,7 @@ ACLOCAL_AMFLAGS = -I m4 -SUBDIRS = lib examples +SUBDIRS = lib helper examples bin_SCRIPTS = \ febootstrap \ @@ -28,9 +28,6 @@ bin_SCRIPTS = \ febootstrap-minimize \ febootstrap-to-initramfs \ febootstrap-to-supermin -bin_PROGRAMS = \ - febootstrap-supermin-helper -DISTCLEANFILES = $(bin_SCRIPTS) febootstrap: febootstrap.sh rm -f $@ @@ -68,18 +65,13 @@ febootstrap-to-supermin: febootstrap-to-supermin.sh chmod 0555 $@-t mv $@-t $@ -febootstrap_supermin_helper_SOURCES = febootstrap-supermin-helper.c -febootstrap_supermin_helper_CFLAGS = -Wall -Ilib -febootstrap_supermin_helper_LDADD = $(LTLIBINTL) -Llib -lgnu - man_MANS = \ febootstrap.8 \ febootstrap-run.8 \ febootstrap-install.8 \ febootstrap-minimize.8 \ febootstrap-to-initramfs.8 \ - febootstrap-to-supermin.8 \ - febootstrap-supermin-helper.8 + febootstrap-to-supermin.8 if HAVE_PERLDOC @@ -143,16 +135,6 @@ febootstrap-to-supermin.8: febootstrap-to-supermin.pod febootstrap-to-supermin.txt: febootstrap-to-supermin.pod pod2text $< > $@ -febootstrap-supermin-helper.8: febootstrap-supermin-helper.pod - pod2man \ - --section 8 \ - -c "Virtualization Support" \ - --release "$(PACKAGE_NAME)-$(PACKAGE_VERSION)" \ - $< > $@ - -febootstrap-supermin-helper.txt: febootstrap-supermin-helper.pod - pod2text $< > $@ - endif EXTRA_DIST = \ @@ -173,6 +155,4 @@ EXTRA_DIST = \ febootstrap-to-supermin.8 febootstrap-to-supermin.txt \ febootstrap-to-supermin.pod \ febootstrap-to-supermin.sh \ - febootstrap-supermin-helper.8 febootstrap-supermin-helper.txt \ - febootstrap-supermin-helper.pod \ m4/gnulib-cache.m4 diff --git a/configure.ac b/configure.ac index 05597b1..1a94700 100644 --- a/configure.ac +++ b/configure.ac @@ -59,5 +59,5 @@ if test "x$YUM" = "xno" ; then fi AC_CONFIG_HEADERS([config.h]) -AC_CONFIG_FILES([lib/Makefile Makefile examples/Makefile]) +AC_CONFIG_FILES([Makefile lib/Makefile helper/Makefile examples/Makefile]) AC_OUTPUT diff --git a/febootstrap-supermin-helper.c b/febootstrap-supermin-helper.c deleted file mode 100644 index 1a43cf2..0000000 --- a/febootstrap-supermin-helper.c +++ /dev/null @@ -1,997 +0,0 @@ -/* febootstrap-supermin-helper reimplementation in C. - * Copyright (C) 2009-2010 Red Hat Inc. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. - */ - -/* This script builds the supermin appliance on the fly each - * time the appliance runs. - * - * *NOTE*: This program is designed to be very short-lived, and so we - * don't normally bother to free up any memory that we allocate. - * That's not completely true - we free up stuff if it's obvious and - * easy to free up, and ignore the rest. - */ - -#include <config.h> - -#include <stdio.h> -#include <stdlib.h> -#include <stdarg.h> -#include <stdint.h> -#include <string.h> -#include <errno.h> -#include <fcntl.h> -#include <inttypes.h> -#include <unistd.h> -#include <getopt.h> -#include <limits.h> -#include <fnmatch.h> -#include <dirent.h> -#include <sys/types.h> -#include <sys/time.h> -#include <sys/stat.h> -#include <assert.h> - -#include "error.h" -#include "filevercmp.h" -#include "fts_.h" -#include "full-write.h" -#include "hash.h" -#include "hash-pjw.h" -#include "xalloc.h" -#include "xvasprintf.h" - -/* Directory containing candidate kernels. We could make this - * configurable at some point. - */ -#define KERNELDIR "/boot" -#define MODULESDIR "/lib/modules" - -/* Buffer size used in copy operations throughout. Large for - * greatest efficiency. - */ -#define BUFFER_SIZE 65536 - -static struct timeval start_t; -static int verbose = 0; - -static void print_timestamped_message (const char *fs, ...); -static const char *create_kernel (const char *hostcpu, const char *kernel); -static void create_appliance (char **inputs, int nr_inputs, const char *whitelist, const char *modpath, const char *initrd); - -enum { HELP_OPTION = CHAR_MAX + 1 }; - -static const char *options = "k:vV"; -static const struct option long_options[] = { - { "help", 0, 0, HELP_OPTION }, - { "kmods", required_argument, 0, 'k' }, - { "verbose", 0, 0, 'v' }, - { "version", 0, 0, 'V' }, - { 0, 0, 0, 0 } -}; - -static void -usage (const char *progname) -{ - printf ("%s: build the supermin appliance on the fly\n" - "\n" - "Usage:\n" - " %s [-options] inputs [...] whitelist host_cpu kernel initrd\n" - " %s --help\n" - " %s --version\n" - "\n" - "This script is used by febootstrap to build the supermin appliance\n" - "(kernel and initrd output files). You should NOT need to run this\n" - "program directly except if you are debugging tricky supermin\n" - "appliance problems.\n" - "\n" - "NB: The kernel and initrd parameters are OUTPUT parameters. If\n" - "those files exist, they are overwritten by the output.\n" - "\n" - "Options:\n" - " --help\n" - " Display this help text and exit.\n" - " -k file | --kmods file\n" - " Specify kernel module whitelist.\n" - " --verbose | -v\n" - " Enable verbose messages (give multiple times for more verbosity).\n" - " --version | -V\n" - " Display version number and exit.\n", - progname, progname, progname, progname); -} - -int -main (int argc, char *argv[]) -{ - /* First thing: start the clock. */ - gettimeofday (&start_t, NULL); - - const char *whitelist = NULL; - - /* Command line arguments. */ - for (;;) { - int c = getopt_long (argc, argv, options, long_options, NULL); - if (c == -1) break; - - switch (c) { - case HELP_OPTION: - usage (argv[0]); - exit (EXIT_SUCCESS); - - case 'k': - whitelist = optarg; - break; - - case 'v': - verbose++; - break; - - case 'V': - printf (PACKAGE_NAME " " PACKAGE_VERSION "\n"); - exit (EXIT_SUCCESS); - - default: - usage (argv[0]); - exit (EXIT_FAILURE); - } - } - - char **inputs = &argv[optind]; - int nr_inputs = argc - optind - 3; - - if (nr_inputs < 1) { - usage (argv[0]); - exit (EXIT_FAILURE); - } - - /* See: https://bugzilla.redhat.com/show_bug.cgi?id=558593 */ - const char *hostcpu = argv[argc-3]; - - /* Output files. */ - const char *kernel = argv[argc-2]; - const char *initrd = argv[argc-1]; - - if (verbose) { - print_timestamped_message ("whitelist = %s, " - "host_cpu = %s, " - "kernel = %s, " - "initrd = %s", - whitelist ? : "(not specified)", - hostcpu, kernel, initrd); - int i; - for (i = 0; i < nr_inputs; ++i) - print_timestamped_message ("inputs[%d] = %s", i, inputs[i]); - } - - /* Remove the output files if they exist. */ - unlink (kernel); - unlink (initrd); - - /* Create kernel output file. */ - const char *modpath; - modpath = create_kernel (hostcpu, kernel); - - if (verbose) - print_timestamped_message ("finished creating kernel"); - - /* Create the appliance. */ - create_appliance (inputs, nr_inputs, whitelist, modpath, initrd); - - if (verbose) - print_timestamped_message ("finished creating appliance"); - - exit (EXIT_SUCCESS); -} - -/* Compute Y - X and return the result in milliseconds. - * Approximately the same as this code: - * http://www.mpp.mpg.de/~huber/util/timevaldiff.c - */ -static int64_t -timeval_diff (const struct timeval *x, const struct timeval *y) -{ - int64_t msec; - - msec = (y->tv_sec - x->tv_sec) * 1000; - msec += (y->tv_usec - x->tv_usec) / 1000; - return msec; -} - -static void -print_timestamped_message (const char *fs, ...) -{ - struct timeval tv; - gettimeofday (&tv, NULL); - - va_list args; - char *msg; - int err; - - va_start (args, fs); - err = vasprintf (&msg, fs, args); - va_end (args); - - if (err < 0) return; - - fprintf (stderr, "supermin helper [%05" PRIi64 "ms] %s\n", - timeval_diff (&start_t, &tv), msg); - - free (msg); -} - -static char **read_dir (const char *dir); -static char **filter_fnmatch (char **strings, const char *patt, int flags); -static char **filter_notmatching_substring (char **strings, const char *sub); -static void sort (char **strings, int (*compare) (const void *, const void *)); -static int isdir (const char *path); - -static int -reverse_filevercmp (const void *p1, const void *p2) -{ - const char *s1 = * (char * const *) p1; - const char *s2 = * (char * const *) p2; - - /* Note, arguments are reversed to achieve a reverse sort. */ - return filevercmp (s2, s1); -} - -/* Create the kernel. This chooses an appropriate kernel and makes a - * symlink to it. - * - * Look for the most recent kernel named vmlinuz-*.<arch>* which has a - * corresponding directory in /lib/modules/. If the architecture is - * x86, look for any x86 kernel. - * - * RHEL 5 didn't append the arch to the kernel name, so look for - * kernels without arch second. - * - * If no suitable kernel can be found, exit with an error. - * - * This function returns the module path (ie. /lib/modules/<version>). - */ -static const char * -create_kernel (const char *hostcpu, const char *kernel) -{ - char **all_files = read_dir (KERNELDIR); - - /* In original: ls -1dvr /boot/vmlinuz-*.$arch* 2>/dev/null | grep -v xen */ - const char *patt; - if (hostcpu[0] == 'i' && hostcpu[2] == '8' && hostcpu[3] == '6' && - hostcpu[4] == '\0') - patt = "vmlinuz-*.i?86*"; - else - patt = xasprintf ("vmlinuz-*.%s*", hostcpu); - - char **candidates; - candidates = filter_fnmatch (all_files, patt, FNM_NOESCAPE); - candidates = filter_notmatching_substring (candidates, "xen"); - - if (candidates[0] == NULL) { - /* In original: ls -1dvr /boot/vmlinuz-* 2>/dev/null | grep -v xen */ - patt = "vmlinuz-*"; - candidates = filter_fnmatch (all_files, patt, FNM_NOESCAPE); - candidates = filter_notmatching_substring (candidates, "xen"); - - if (candidates[0] == NULL) - goto no_kernels; - } - - sort (candidates, reverse_filevercmp); - - /* Choose the first candidate which has a corresponding /lib/modules - * directory. - */ - int i; - for (i = 0; candidates[i] != NULL; ++i) { - if (verbose >= 2) - fprintf (stderr, "candidate kernel: " KERNELDIR "/%s\n", candidates[i]); - - /* Ignore "vmlinuz-" at the beginning of the kernel name. */ - const char *version = &candidates[i][8]; - - /* /lib/modules/<version> */ - char *modpath = xasprintf (MODULESDIR "/%s", version); - - if (verbose >= 2) - fprintf (stderr, "checking modpath %s is a directory\n", modpath); - - if (isdir (modpath)) { - if (verbose >= 2) - fprintf (stderr, "picked %s because modpath %s exists\n", - candidates[i], modpath); - - char *tmp = xasprintf (KERNELDIR "/%s", candidates[i]); - - if (verbose >= 2) - fprintf (stderr, "creating symlink %s -> %s\n", kernel, tmp); - - if (symlink (tmp, kernel) == -1) - error (EXIT_FAILURE, errno, "symlink kernel"); - - free (tmp); - - return modpath; - } - } - - /* Print more diagnostics here than the old script did. */ - no_kernels: - fprintf (stderr, - "febootstrap-supermin-helper: failed to find a suitable kernel.\n" - "I looked for kernels in " KERNELDIR " and modules in " MODULESDIR - ".\n" - "If this is a Xen guest, and you only have Xen domU kernels\n" - "installed, try installing a fullvirt kernel (only for\n" - "febootstrap use, you shouldn't boot the Xen guest with it).\n"); - exit (EXIT_FAILURE); -} - -static void iterate_inputs (char **inputs, int nr_inputs); -static void iterate_input_directory (const char *dirname, int dirfd); -static void write_kernel_modules (const char *whitelist, const char *modpath); -static void write_hostfiles (const char *hostfiles_file); -static void write_to_fd (const void *buffer, size_t len); -static void write_file_to_fd (const char *filename); -static void write_file_len_to_fd (const char *filename, size_t len); -static void write_padding (size_t len); -static char **load_file (const char *filename); -static void cpio_append_fts_entry (FTSENT *entry); -static void cpio_append_stat (const char *filename, struct stat *); -static void cpio_append (const char *filename); -static void cpio_append_trailer (void); - -static int out_fd = -1; -static off_t out_offset = 0; - -/* Create the appliance. - * - * The initrd consists of these components concatenated together: - * - * (1) The base skeleton appliance that we constructed at build time. - * format = plain cpio - * (2) The host files which match wildcards in *.supermin.hostfiles. - * input format = plain text, output format = plain cpio - * (3) The modules from modpath which are on the module whitelist. - * output format = plain cpio - * - * The original shell script used the external cpio program to create - * parts (2) and (3), but we have decided it's going to be faster if - * we just write out the data outselves. The reasons are that - * external cpio is slow (particularly when used with SELinux because - * it does 512 byte reads), and the format that we're writing is - * narrow and well understood, because we only care that the Linux - * kernel can read it. - * - * This version contains some improvements over the C version written - * for libguestfs, in that we can have multiple base images (or - * hostfiles) or use a directory to store these files. - */ -static void -create_appliance (char **inputs, int nr_inputs, - const char *whitelist, - const char *modpath, - const char *initrd) -{ - out_fd = open (initrd, O_WRONLY | O_CREAT | O_TRUNC | O_NOCTTY, 0644); - if (out_fd == -1) - error (EXIT_FAILURE, errno, "open: %s", initrd); - out_offset = 0; - - iterate_inputs (inputs, nr_inputs); - - /* Kernel modules (3). */ - write_kernel_modules (whitelist, modpath); - - cpio_append_trailer (); - - /* Finish off and close output file. */ - if (close (out_fd) == -1) - error (EXIT_FAILURE, errno, "close: %s", initrd); -} - -/* Iterate over the inputs to find out what they are, visiting - * directories if specified. - */ -static void -iterate_inputs (char **inputs, int nr_inputs) -{ - int i; - for (i = 0; i < nr_inputs; ++i) { - if (verbose) - print_timestamped_message ("visiting %s", inputs[i]); - - int fd = open (inputs[i], O_RDONLY); - if (fd == -1) - error (EXIT_FAILURE, errno, "open: %s", inputs[i]); - - struct stat statbuf; - if (fstat (fd, &statbuf) == -1) - error (EXIT_FAILURE, errno, "fstat: %s", inputs[i]); - - /* Directory? */ - if (S_ISDIR (statbuf.st_mode)) - iterate_input_directory (inputs[i], fd); - else if (S_ISREG (statbuf.st_mode)) { - /* Is it a cpio file? */ - char buf[6]; - if (read (fd, buf, 6) == 6 && memcmp (buf, "070701", 6) == 0) - /* Yes, a cpio file. This is a skeleton appliance, case (1). */ - write_file_to_fd (inputs[i]); - else - /* No, must be hostfiles, case (2). */ - write_hostfiles (inputs[i]); - } - else - error (EXIT_FAILURE, 0, "%s: input is not a regular file or directory", - inputs[i]); - - close (fd); - } -} - -static void -iterate_input_directory (const char *dirname, int dirfd) -{ - char path[PATH_MAX]; - strcpy (path, dirname); - size_t len = strlen (dirname); - path[len++] = '/'; - - char *inputs[] = { path }; - - DIR *dir = fdopendir (dirfd); - if (dir == NULL) - error (EXIT_FAILURE, errno, "fdopendir: %s", dirname); - - struct dirent *d; - while ((errno = 0, d = readdir (dir)) != NULL) { - if (d->d_name[0] == '.') /* ignore ., .. and any hidden files. */ - continue; - - strcpy (&path[len], d->d_name); - iterate_inputs (inputs, 1); - } - - if (errno != 0) - error (EXIT_FAILURE, errno, "readdir: %s", dirname); - - if (closedir (dir) == -1) - error (EXIT_FAILURE, errno, "closedir: %s", dirname); -} - -/* Copy kernel modules. - * - * Find every file under modpath. - * - * Exclude all *.ko files, *except* ones which match names in - * the whitelist (which may contain wildcards). Include all - * other files. - * - * Add chosen files to the output. - * - * whitelist_file may be NULL, to include ALL kernel modules. - */ -static void -write_kernel_modules (const char *whitelist_file, const char *modpath) -{ - char **whitelist = NULL; - if (whitelist_file != NULL) - whitelist = load_file (whitelist_file); - - char *paths[2] = { (char *) modpath, NULL }; - FTS *fts = fts_open (paths, FTS_COMFOLLOW|FTS_PHYSICAL, NULL); - if (fts == NULL) - error (EXIT_FAILURE, errno, "write_kernel_modules: fts_open: %s", modpath); - - for (;;) { - errno = 0; - FTSENT *entry = fts_read (fts); - if (entry == NULL && errno != 0) - error (EXIT_FAILURE, errno, "write_kernel_modules: fts_read: %s", modpath); - if (entry == NULL) - break; - - /* Ignore directories being visited in post-order. */ - if (entry->fts_info & FTS_DP) - continue; - - /* Is it a *.ko file? */ - if (entry->fts_namelen >= 3 && - entry->fts_name[entry->fts_namelen-3] == '.' && - entry->fts_name[entry->fts_namelen-2] == 'k' && - entry->fts_name[entry->fts_namelen-1] == 'o') { - if (whitelist) { - /* Is it a *.ko file which is on the whitelist? */ - size_t j; - for (j = 0; whitelist[j] != NULL; ++j) { - int r; - r = fnmatch (whitelist[j], entry->fts_name, 0); - if (r == 0) { - /* It's on the whitelist, so include it. */ - if (verbose >= 2) - fprintf (stderr, "including kernel module %s (matches whitelist entry %s)\n", - entry->fts_name, whitelist[j]); - cpio_append_fts_entry (entry); - break; - } else if (r != FNM_NOMATCH) - error (EXIT_FAILURE, 0, "internal error: fnmatch ('%s', '%s', %d) returned unexpected non-zero value %d\n", - whitelist[j], entry->fts_name, 0, r); - } /* for (j) */ - } else { /* whitelist == NULL, always include */ - if (verbose >= 2) - fprintf (stderr, "including kernel module %s\n", entry->fts_name); - cpio_append_fts_entry (entry); - } - } else - /* It's some other sort of file, or a directory, always include. */ - cpio_append_fts_entry (entry); - } - - if (fts_close (fts) == -1) - error (EXIT_FAILURE, errno, "write_kernel_modules: fts_close: %s", modpath); -} - -/* Copy the host files. - * - * Read the list of entries in hostfiles (which may contain - * wildcards). Look them up in the filesystem, and add those files - * that exist. Ignore any files that don't exist or are not readable. - */ -static void -write_hostfiles (const char *hostfiles_file) -{ - char **hostfiles = load_file (hostfiles_file); - - /* Hostfiles list can contain "." before each path - ignore it. - * It also contains each directory name before we enter it. But - * we don't read that until we see a wildcard for that directory. - */ - size_t i, j; - for (i = 0; hostfiles[i] != NULL; ++i) { - char *hostfile = hostfiles[i]; - if (hostfile[0] == '.') - hostfile++; - - struct stat statbuf; - - /* Is it a wildcard? */ - if (strchr (hostfile, '*') || strchr (hostfile, '?')) { - char *dirname = xstrdup (hostfile); - char *patt = strrchr (dirname, '/'); - assert (patt); - *patt++ = '\0'; - - char **files = read_dir (dirname); - files = filter_fnmatch (files, patt, FNM_NOESCAPE); - - /* Add matching files. */ - for (j = 0; files[j] != NULL; ++j) { - char *tmp = xasprintf ("%s/%s", dirname, files[j]); - - if (verbose >= 2) - fprintf (stderr, "including host file %s (matches %s)\n", tmp, patt); - - cpio_append (tmp); - - free (tmp); - } - } - /* Else does this file/directory/whatever exist? */ - else if (lstat (hostfile, &statbuf) == 0) { - if (verbose >= 2) - fprintf (stderr, "including host file %s (directly referenced)\n", - hostfile); - - cpio_append_stat (hostfile, &statbuf); - } /* Ignore files that don't exist. */ - } -} - -/*----------*/ -/* Helper functions. */ - -static void -add_string (char ***argv, size_t *n_used, size_t *n_alloc, const char *str) -{ - char *new_str; - - if (*n_used >= *n_alloc) - *argv = x2nrealloc (*argv, n_alloc, sizeof (char *)); - - if (str) - new_str = xstrdup (str); - else - new_str = NULL; - - (*argv)[*n_used] = new_str; - - (*n_used)++; -} - -static size_t -count_strings (char *const *argv) -{ - size_t argc; - - for (argc = 0; argv[argc] != NULL; ++argc) - ; - return argc; -} - -struct dir_cache { - char *path; - char **files; -}; - -static size_t -dir_cache_hash (void const *x, size_t table_size) -{ - struct dir_cache const *p = x; - return hash_pjw (p->path, table_size); -} - -static bool -dir_cache_compare (void const *x, void const *y) -{ - struct dir_cache const *p = x; - struct dir_cache const *q = y; - return strcmp (p->path, q->path) == 0; -} - -/* Read a directory into a list of strings. - * - * Previously looked up directories are cached and returned quickly, - * saving some considerable amount of time compared to reading the - * directory over again. However this means you really must not - * alter the array of strings that are returned. - * - * Returns an empty list if the directory cannot be opened. - */ -static char ** -read_dir (const char *name) -{ - static Hash_table *ht = NULL; - - if (!ht) - ht = hash_initialize (1024, NULL, dir_cache_hash, dir_cache_compare, NULL); - - struct dir_cache key = { .path = (char *) name }; - struct dir_cache *p = hash_lookup (ht, &key); - if (p) - return p->files; - - char **files = NULL; - size_t n_used = 0, n_alloc = 0; - - DIR *dir = opendir (name); - if (!dir) { - /* If it fails to open, that's OK, skip to the end. */ - /*perror (name);*/ - goto done; - } - - for (;;) { - errno = 0; - struct dirent *d = readdir (dir); - if (d == NULL) { - if (errno != 0) - /* But if it fails here, after opening and potentially reading - * part of the directory, that's a proper failure - inform the - * user and exit. - */ - error (EXIT_FAILURE, errno, "%s", name); - break; - } - - add_string (&files, &n_used, &n_alloc, d->d_name); - } - - if (closedir (dir) == -1) - error (EXIT_FAILURE, errno, "closedir: %s", name); - - done: - /* NULL-terminate the array. */ - add_string (&files, &n_used, &n_alloc, NULL); - - /* Add it to the hash for next time. */ - p = xmalloc (sizeof *p); - p->path = (char *) name; - p->files = files; - p = hash_insert (ht, p); - assert (p != NULL); - - return files; -} - -/* Filter a list of strings and return only those matching the wildcard. */ -static char ** -filter_fnmatch (char **strings, const char *patt, int flags) -{ - char **out = NULL; - size_t n_used = 0, n_alloc = 0; - - int i, r; - for (i = 0; strings[i] != NULL; ++i) { - r = fnmatch (patt, strings[i], flags); - if (r == 0) - add_string (&out, &n_used, &n_alloc, strings[i]); - else if (r != FNM_NOMATCH) - error (EXIT_FAILURE, 0, "internal error: fnmatch ('%s', '%s', %d) returned unexpected non-zero value %d\n", - patt, strings[i], flags, r); - } - - add_string (&out, &n_used, &n_alloc, NULL); - return out; -} - -/* Filter a list of strings and return only those which DON'T contain sub. */ -static char ** -filter_notmatching_substring (char **strings, const char *sub) -{ - char **out = NULL; - size_t n_used = 0, n_alloc = 0; - - int i; - for (i = 0; strings[i] != NULL; ++i) { - if (strstr (strings[i], sub) == NULL) - add_string (&out, &n_used, &n_alloc, strings[i]); - } - - add_string (&out, &n_used, &n_alloc, NULL); - return out; -} - -/* Sort a list of strings, in place, with the comparison function supplied. */ -static void -sort (char **strings, int (*compare) (const void *, const void *)) -{ - qsort (strings, count_strings (strings), sizeof (char *), compare); -} - -/* Return true iff path exists and is a directory. This version - * follows symlinks. - */ -static int -isdir (const char *path) -{ - struct stat statbuf; - - if (stat (path, &statbuf) == -1) - return 0; - - return S_ISDIR (statbuf.st_mode); -} - -/* Copy contents of buffer to out_fd and keep out_offset correct. */ -static void -write_to_fd (const void *buffer, size_t len) -{ - if (full_write (out_fd, buffer, len) != len) - error (EXIT_FAILURE, errno, "write"); - out_offset += len; -} - -/* Copy contents of file to out_fd. */ -static void -write_file_to_fd (const char *filename) -{ - char buffer[BUFFER_SIZE]; - int fd2; - ssize_t r; - - if (verbose >= 2) - fprintf (stderr, "write_file_to_fd %s -> %d\n", filename, out_fd); - - fd2 = open (filename, O_RDONLY); - if (fd2 == -1) - error (EXIT_FAILURE, errno, "open: %s", filename); - for (;;) { - r = read (fd2, buffer, sizeof buffer); - if (r == 0) - break; - if (r == -1) { - if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) - continue; - error (EXIT_FAILURE, errno, "read: %s", filename); - } - write_to_fd (buffer, r); - } - - if (close (fd2) == -1) - error (EXIT_FAILURE, errno, "close: %s", filename); -} - -/* Copy file of given length to output, and fail if the file has - * changed size. - */ -static void -write_file_len_to_fd (const char *filename, size_t len) -{ - char buffer[BUFFER_SIZE]; - size_t count = 0; - - if (verbose >= 2) - fprintf (stderr, "write_file_to_fd %s -> %d\n", filename, out_fd); - - int fd2 = open (filename, O_RDONLY); - if (fd2 == -1) - error (EXIT_FAILURE, errno, "open: %s", filename); - for (;;) { - ssize_t r = read (fd2, buffer, sizeof buffer); - if (r == 0) - break; - if (r == -1) { - if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) - continue; - error (EXIT_FAILURE, errno, "read: %s", filename); - } - write_to_fd (buffer, r); - count += r; - if (count > len) - error (EXIT_FAILURE, 0, "write_file_len_to_fd: %s: file has increased in size\n", filename); - } - - if (close (fd2) == -1) - error (EXIT_FAILURE, errno, "close: %s", filename); - - if (count != len) - error (EXIT_FAILURE, 0, "febootstrap-supermin-helper: write_file_len_to_fd: %s: file has changed size\n", filename); -} - -/* Load in a file, returning a list of lines. */ -static char ** -load_file (const char *filename) -{ - char **lines = 0; - size_t n_used = 0, n_alloc = 0; - - FILE *fp; - fp = fopen (filename, "r"); - if (fp == NULL) - error (EXIT_FAILURE, errno, "fopen: %s", filename); - - char line[4096]; - while (fgets (line, sizeof line, fp)) { - size_t len = strlen (line); - if (len > 0 && line[len-1] == '\n') - line[len-1] = '\0'; - add_string (&lines, &n_used, &n_alloc, line); - } - - add_string (&lines, &n_used, &n_alloc, NULL); - return lines; -} - -/* Append the file pointed to by FTSENT to the cpio output. */ -static void -cpio_append_fts_entry (FTSENT *entry) -{ - if (entry->fts_info & FTS_NS || entry->fts_info & FTS_NSOK) - cpio_append (entry->fts_path); - else - cpio_append_stat (entry->fts_path, entry->fts_statp); -} - -/* Append the file named 'filename' to the cpio output. */ -static void -cpio_append (const char *filename) -{ - struct stat statbuf; - - if (lstat (filename, &statbuf) == -1) - error (EXIT_FAILURE, errno, "lstat: %s", filename); - cpio_append_stat (filename, &statbuf); -} - -/* Append the file to the cpio output. */ -#define PADDING(len) ((((len) + 3) & ~3) - (len)) - -#define CPIO_HEADER_LEN (6 + 13*8) - -static void -cpio_append_stat (const char *filename, struct stat *statbuf) -{ - const char *orig_filename = filename; - - if (*filename == '/') - filename++; - if (*filename == '\0') - filename = "."; - - if (verbose >= 2) - fprintf (stderr, "cpio_append_stat %s 0%o -> %d\n", - orig_filename, statbuf->st_mode, out_fd); - - /* Regular files and symlinks are the only ones that have a "body" - * in this cpio entry. - */ - int has_body = S_ISREG (statbuf->st_mode) || S_ISLNK (statbuf->st_mode); - - size_t len = strlen (filename) + 1; - - char header[CPIO_HEADER_LEN + 1]; - snprintf (header, sizeof header, - "070701" /* magic */ - "%08X" /* inode */ - "%08X" /* mode */ - "%08X" "%08X" /* uid, gid */ - "%08X" /* nlink */ - "%08X" /* mtime */ - "%08X" /* file length */ - "%08X" "%08X" /* device holding file major, minor */ - "%08X" "%08X" /* for specials, device major, minor */ - "%08X" /* name length (including \0 byte) */ - "%08X", /* checksum (not used by the kernel) */ - (unsigned) statbuf->st_ino, statbuf->st_mode, - statbuf->st_uid, statbuf->st_gid, - (unsigned) statbuf->st_nlink, (unsigned) statbuf->st_mtime, - has_body ? (unsigned) statbuf->st_size : 0, - major (statbuf->st_dev), minor (statbuf->st_dev), - major (statbuf->st_rdev), minor (statbuf->st_rdev), - (unsigned) len, 0); - - /* Write the header. */ - write_to_fd (header, CPIO_HEADER_LEN); - - /* Follow with the filename, and pad it. */ - write_to_fd (filename, len); - size_t padding_len = PADDING (CPIO_HEADER_LEN + len); - write_padding (padding_len); - - /* Follow with the file or symlink content, and pad it. */ - if (has_body) { - if (S_ISREG (statbuf->st_mode)) - write_file_len_to_fd (orig_filename, statbuf->st_size); - else if (S_ISLNK (statbuf->st_mode)) { - char tmp[PATH_MAX]; - if (readlink (orig_filename, tmp, sizeof tmp) == -1) - error (EXIT_FAILURE, errno, "readlink: %s", orig_filename); - write_to_fd (tmp, statbuf->st_size); - } - - padding_len = PADDING (statbuf->st_size); - write_padding (padding_len); - } -} - -/* CPIO voodoo. */ -static void -cpio_append_trailer (void) -{ - struct stat statbuf; - memset (&statbuf, 0, sizeof statbuf); - statbuf.st_nlink = 1; - cpio_append_stat ("TRAILER!!!", &statbuf); - - /* CPIO seems to pad up to the next block boundary, ie. up to - * the next 512 bytes. - */ - write_padding (((out_offset + 511) & ~511) - out_offset); - assert ((out_offset & 511) == 0); -} - -/* Write 'len' bytes of zeroes out. */ -static void -write_padding (size_t len) -{ - static const char buffer[512] = { 0 }; - - while (len > 0) { - size_t n = len < sizeof buffer ? len : sizeof buffer; - write_to_fd (buffer, n); - len -= n; - } -} diff --git a/febootstrap-supermin-helper.pod b/febootstrap-supermin-helper.pod deleted file mode 100644 index 29f4b95..0000000 --- a/febootstrap-supermin-helper.pod +++ /dev/null @@ -1,109 +0,0 @@ -=head1 NAME - -febootstrap-supermin-helper - Reconstruct initramfs from supermin appliance. - -=head1 SYNOPSIS - - febootstrap-supermin-helper supermin.img hostfiles.txt host_cpu kernel initrd - febootstrap-supermin-helper input [...] host_cpu kernel initrd - -=head1 DESCRIPTION - -I<febootstrap-supermin-helper> reconstructs a bootable kernel and -initramfs from a supermin appliance. First you should be familiar -with L<febootstrap(8)>, L<febootstrap-to-initramfs(8)> and -L<febootstrap-to-supermin(8)>. - -=head1 PARAMETERS - -Of the four or five required parameters, the first few are I<input> -files, and the last two are I<output> files. - -C<supermin.img> and C<hostfiles.txt> are the input files which -describe the supermin appliance. (You can also use a directory name -here which is searched for files). - -C<host_cpu> should be the host CPU, eg. C<x86_64> or C<i686>. - -C<kernel> and C<initrd> are the temporary output files that this -script produces. These output files are meant to be used just for -booting the appliance, and should be deleted straight afterwards. - -=head1 OPTIONS - -=over 4 - -=item B<-k file> | B<--kmods file> - -If this option is specified, then C<file> should be a list of -wildcards matching kernel module names, eg: - - virtio*.ko - scsi*.ko - piix.ko - -In this case, only kernel modules matching those wildcards will be -included in the output appliance. Note: You must resolve any -dependencies yourself as this does not pull in dependent modules -automatically. - -If this option is not specified, then every kernel module from the -host will be included. This is safer, but can produce rather large -appliances which need a lot more memory to boot. - -=back - -=head1 SPEED - -In libguestfs, on a mid-range Intel-based PC, we reconstruct the -initramfs using this script in around 1/5th of a second (assuming a -"hot cache" - it's rather slower when run the first time on a cold -cache). - -Some tips to improve performance: - -=over 4 - -=item * - -Use a kernel module whitelist (the C<--kmods> option), and only -list the kernel modules you really need. - -=item * - -Minimize the appliance, removing as much extraneous junk as possible. - -As well as using L<febootstrap-minimize(8)> it is worth checking for -anything that is not necessary for your particular application and -removing it by hand. - -=back - -=head1 SEE ALSO - -L<febootstrap(8)>, -L<febootstrap-to-initramfs(8)>, -L<febootstrap-to-supermin(8)>. - -=head1 AUTHORS - -Richard W.M. Jones <rjones @ redhat . com> - -=head1 COPYRIGHT - -(C) Copyright 2009-2010 Red Hat Inc., -L<http://people.redhat.com/~rjones/febootstrap>. - -This program is free software; you can redistribute it and/or modify -it under the terms of the GNU General Public License as published by -the Free Software Foundation; either version 2 of the License, or -(at your option) any later version. - -This program is distributed in the hope that it will be useful, -but WITHOUT ANY WARRANTY; without even the implied warranty of -MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -GNU General Public License for more details. - -You should have received a copy of the GNU General Public License -along with this program; if not, write to the Free Software -Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. diff --git a/helper/Makefile.am b/helper/Makefile.am new file mode 100644 index 0000000..518cfa6 --- /dev/null +++ b/helper/Makefile.am @@ -0,0 +1,47 @@ +# febootstrap Makefile.am +# (C) Copyright 2010 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +# +# Written by Richard W.M. Jones <rjones at redhat.com> + +bin_PROGRAMS = \ + febootstrap-supermin-helper +DISTCLEANFILES = $(bin_SCRIPTS) + +febootstrap_supermin_helper_SOURCES = main.c +febootstrap_supermin_helper_CFLAGS = -Wall -I../lib +febootstrap_supermin_helper_LDADD = $(LTLIBINTL) -L../lib -lgnu + +man_MANS = \ + febootstrap-supermin-helper.8 + +if HAVE_PERLDOC + +febootstrap-supermin-helper.8: febootstrap-supermin-helper.pod + pod2man \ + --section 8 \ + -c "Virtualization Support" \ + --release "$(PACKAGE_NAME)-$(PACKAGE_VERSION)" \ + $< > $@ + +febootstrap-supermin-helper.txt: febootstrap-supermin-helper.pod + pod2text $< > $@ + +endif + +EXTRA_DIST = \ + febootstrap-supermin-helper.8 febootstrap-supermin-helper.txt \ + febootstrap-supermin-helper.pod diff --git a/helper/febootstrap-supermin-helper.pod b/helper/febootstrap-supermin-helper.pod new file mode 100644 index 0000000..29f4b95 --- /dev/null +++ b/helper/febootstrap-supermin-helper.pod @@ -0,0 +1,109 @@ +=head1 NAME + +febootstrap-supermin-helper - Reconstruct initramfs from supermin appliance. + +=head1 SYNOPSIS + + febootstrap-supermin-helper supermin.img hostfiles.txt host_cpu kernel initrd + febootstrap-supermin-helper input [...] host_cpu kernel initrd + +=head1 DESCRIPTION + +I<febootstrap-supermin-helper> reconstructs a bootable kernel and +initramfs from a supermin appliance. First you should be familiar +with L<febootstrap(8)>, L<febootstrap-to-initramfs(8)> and +L<febootstrap-to-supermin(8)>. + +=head1 PARAMETERS + +Of the four or five required parameters, the first few are I<input> +files, and the last two are I<output> files. + +C<supermin.img> and C<hostfiles.txt> are the input files which +describe the supermin appliance. (You can also use a directory name +here which is searched for files). + +C<host_cpu> should be the host CPU, eg. C<x86_64> or C<i686>. + +C<kernel> and C<initrd> are the temporary output files that this +script produces. These output files are meant to be used just for +booting the appliance, and should be deleted straight afterwards. + +=head1 OPTIONS + +=over 4 + +=item B<-k file> | B<--kmods file> + +If this option is specified, then C<file> should be a list of +wildcards matching kernel module names, eg: + + virtio*.ko + scsi*.ko + piix.ko + +In this case, only kernel modules matching those wildcards will be +included in the output appliance. Note: You must resolve any +dependencies yourself as this does not pull in dependent modules +automatically. + +If this option is not specified, then every kernel module from the +host will be included. This is safer, but can produce rather large +appliances which need a lot more memory to boot. + +=back + +=head1 SPEED + +In libguestfs, on a mid-range Intel-based PC, we reconstruct the +initramfs using this script in around 1/5th of a second (assuming a +"hot cache" - it's rather slower when run the first time on a cold +cache). + +Some tips to improve performance: + +=over 4 + +=item * + +Use a kernel module whitelist (the C<--kmods> option), and only +list the kernel modules you really need. + +=item * + +Minimize the appliance, removing as much extraneous junk as possible. + +As well as using L<febootstrap-minimize(8)> it is worth checking for +anything that is not necessary for your particular application and +removing it by hand. + +=back + +=head1 SEE ALSO + +L<febootstrap(8)>, +L<febootstrap-to-initramfs(8)>, +L<febootstrap-to-supermin(8)>. + +=head1 AUTHORS + +Richard W.M. Jones <rjones @ redhat . com> + +=head1 COPYRIGHT + +(C) Copyright 2009-2010 Red Hat Inc., +L<http://people.redhat.com/~rjones/febootstrap>. + +This program is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation; either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program; if not, write to the Free Software +Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. diff --git a/helper/main.c b/helper/main.c new file mode 100644 index 0000000..1a43cf2 --- /dev/null +++ b/helper/main.c @@ -0,0 +1,997 @@ +/* febootstrap-supermin-helper reimplementation in C. + * Copyright (C) 2009-2010 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + */ + +/* This script builds the supermin appliance on the fly each + * time the appliance runs. + * + * *NOTE*: This program is designed to be very short-lived, and so we + * don't normally bother to free up any memory that we allocate. + * That's not completely true - we free up stuff if it's obvious and + * easy to free up, and ignore the rest. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdarg.h> +#include <stdint.h> +#include <string.h> +#include <errno.h> +#include <fcntl.h> +#include <inttypes.h> +#include <unistd.h> +#include <getopt.h> +#include <limits.h> +#include <fnmatch.h> +#include <dirent.h> +#include <sys/types.h> +#include <sys/time.h> +#include <sys/stat.h> +#include <assert.h> + +#include "error.h" +#include "filevercmp.h" +#include "fts_.h" +#include "full-write.h" +#include "hash.h" +#include "hash-pjw.h" +#include "xalloc.h" +#include "xvasprintf.h" + +/* Directory containing candidate kernels. We could make this + * configurable at some point. + */ +#define KERNELDIR "/boot" +#define MODULESDIR "/lib/modules" + +/* Buffer size used in copy operations throughout. Large for + * greatest efficiency. + */ +#define BUFFER_SIZE 65536 + +static struct timeval start_t; +static int verbose = 0; + +static void print_timestamped_message (const char *fs, ...); +static const char *create_kernel (const char *hostcpu, const char *kernel); +static void create_appliance (char **inputs, int nr_inputs, const char *whitelist, const char *modpath, const char *initrd); + +enum { HELP_OPTION = CHAR_MAX + 1 }; + +static const char *options = "k:vV"; +static const struct option long_options[] = { + { "help", 0, 0, HELP_OPTION }, + { "kmods", required_argument, 0, 'k' }, + { "verbose", 0, 0, 'v' }, + { "version", 0, 0, 'V' }, + { 0, 0, 0, 0 } +}; + +static void +usage (const char *progname) +{ + printf ("%s: build the supermin appliance on the fly\n" + "\n" + "Usage:\n" + " %s [-options] inputs [...] whitelist host_cpu kernel initrd\n" + " %s --help\n" + " %s --version\n" + "\n" + "This script is used by febootstrap to build the supermin appliance\n" + "(kernel and initrd output files). You should NOT need to run this\n" + "program directly except if you are debugging tricky supermin\n" + "appliance problems.\n" + "\n" + "NB: The kernel and initrd parameters are OUTPUT parameters. If\n" + "those files exist, they are overwritten by the output.\n" + "\n" + "Options:\n" + " --help\n" + " Display this help text and exit.\n" + " -k file | --kmods file\n" + " Specify kernel module whitelist.\n" + " --verbose | -v\n" + " Enable verbose messages (give multiple times for more verbosity).\n" + " --version | -V\n" + " Display version number and exit.\n", + progname, progname, progname, progname); +} + +int +main (int argc, char *argv[]) +{ + /* First thing: start the clock. */ + gettimeofday (&start_t, NULL); + + const char *whitelist = NULL; + + /* Command line arguments. */ + for (;;) { + int c = getopt_long (argc, argv, options, long_options, NULL); + if (c == -1) break; + + switch (c) { + case HELP_OPTION: + usage (argv[0]); + exit (EXIT_SUCCESS); + + case 'k': + whitelist = optarg; + break; + + case 'v': + verbose++; + break; + + case 'V': + printf (PACKAGE_NAME " " PACKAGE_VERSION "\n"); + exit (EXIT_SUCCESS); + + default: + usage (argv[0]); + exit (EXIT_FAILURE); + } + } + + char **inputs = &argv[optind]; + int nr_inputs = argc - optind - 3; + + if (nr_inputs < 1) { + usage (argv[0]); + exit (EXIT_FAILURE); + } + + /* See: https://bugzilla.redhat.com/show_bug.cgi?id=558593 */ + const char *hostcpu = argv[argc-3]; + + /* Output files. */ + const char *kernel = argv[argc-2]; + const char *initrd = argv[argc-1]; + + if (verbose) { + print_timestamped_message ("whitelist = %s, " + "host_cpu = %s, " + "kernel = %s, " + "initrd = %s", + whitelist ? : "(not specified)", + hostcpu, kernel, initrd); + int i; + for (i = 0; i < nr_inputs; ++i) + print_timestamped_message ("inputs[%d] = %s", i, inputs[i]); + } + + /* Remove the output files if they exist. */ + unlink (kernel); + unlink (initrd); + + /* Create kernel output file. */ + const char *modpath; + modpath = create_kernel (hostcpu, kernel); + + if (verbose) + print_timestamped_message ("finished creating kernel"); + + /* Create the appliance. */ + create_appliance (inputs, nr_inputs, whitelist, modpath, initrd); + + if (verbose) + print_timestamped_message ("finished creating appliance"); + + exit (EXIT_SUCCESS); +} + +/* Compute Y - X and return the result in milliseconds. + * Approximately the same as this code: + * http://www.mpp.mpg.de/~huber/util/timevaldiff.c + */ +static int64_t +timeval_diff (const struct timeval *x, const struct timeval *y) +{ + int64_t msec; + + msec = (y->tv_sec - x->tv_sec) * 1000; + msec += (y->tv_usec - x->tv_usec) / 1000; + return msec; +} + +static void +print_timestamped_message (const char *fs, ...) +{ + struct timeval tv; + gettimeofday (&tv, NULL); + + va_list args; + char *msg; + int err; + + va_start (args, fs); + err = vasprintf (&msg, fs, args); + va_end (args); + + if (err < 0) return; + + fprintf (stderr, "supermin helper [%05" PRIi64 "ms] %s\n", + timeval_diff (&start_t, &tv), msg); + + free (msg); +} + +static char **read_dir (const char *dir); +static char **filter_fnmatch (char **strings, const char *patt, int flags); +static char **filter_notmatching_substring (char **strings, const char *sub); +static void sort (char **strings, int (*compare) (const void *, const void *)); +static int isdir (const char *path); + +static int +reverse_filevercmp (const void *p1, const void *p2) +{ + const char *s1 = * (char * const *) p1; + const char *s2 = * (char * const *) p2; + + /* Note, arguments are reversed to achieve a reverse sort. */ + return filevercmp (s2, s1); +} + +/* Create the kernel. This chooses an appropriate kernel and makes a + * symlink to it. + * + * Look for the most recent kernel named vmlinuz-*.<arch>* which has a + * corresponding directory in /lib/modules/. If the architecture is + * x86, look for any x86 kernel. + * + * RHEL 5 didn't append the arch to the kernel name, so look for + * kernels without arch second. + * + * If no suitable kernel can be found, exit with an error. + * + * This function returns the module path (ie. /lib/modules/<version>). + */ +static const char * +create_kernel (const char *hostcpu, const char *kernel) +{ + char **all_files = read_dir (KERNELDIR); + + /* In original: ls -1dvr /boot/vmlinuz-*.$arch* 2>/dev/null | grep -v xen */ + const char *patt; + if (hostcpu[0] == 'i' && hostcpu[2] == '8' && hostcpu[3] == '6' && + hostcpu[4] == '\0') + patt = "vmlinuz-*.i?86*"; + else + patt = xasprintf ("vmlinuz-*.%s*", hostcpu); + + char **candidates; + candidates = filter_fnmatch (all_files, patt, FNM_NOESCAPE); + candidates = filter_notmatching_substring (candidates, "xen"); + + if (candidates[0] == NULL) { + /* In original: ls -1dvr /boot/vmlinuz-* 2>/dev/null | grep -v xen */ + patt = "vmlinuz-*"; + candidates = filter_fnmatch (all_files, patt, FNM_NOESCAPE); + candidates = filter_notmatching_substring (candidates, "xen"); + + if (candidates[0] == NULL) + goto no_kernels; + } + + sort (candidates, reverse_filevercmp); + + /* Choose the first candidate which has a corresponding /lib/modules + * directory. + */ + int i; + for (i = 0; candidates[i] != NULL; ++i) { + if (verbose >= 2) + fprintf (stderr, "candidate kernel: " KERNELDIR "/%s\n", candidates[i]); + + /* Ignore "vmlinuz-" at the beginning of the kernel name. */ + const char *version = &candidates[i][8]; + + /* /lib/modules/<version> */ + char *modpath = xasprintf (MODULESDIR "/%s", version); + + if (verbose >= 2) + fprintf (stderr, "checking modpath %s is a directory\n", modpath); + + if (isdir (modpath)) { + if (verbose >= 2) + fprintf (stderr, "picked %s because modpath %s exists\n", + candidates[i], modpath); + + char *tmp = xasprintf (KERNELDIR "/%s", candidates[i]); + + if (verbose >= 2) + fprintf (stderr, "creating symlink %s -> %s\n", kernel, tmp); + + if (symlink (tmp, kernel) == -1) + error (EXIT_FAILURE, errno, "symlink kernel"); + + free (tmp); + + return modpath; + } + } + + /* Print more diagnostics here than the old script did. */ + no_kernels: + fprintf (stderr, + "febootstrap-supermin-helper: failed to find a suitable kernel.\n" + "I looked for kernels in " KERNELDIR " and modules in " MODULESDIR + ".\n" + "If this is a Xen guest, and you only have Xen domU kernels\n" + "installed, try installing a fullvirt kernel (only for\n" + "febootstrap use, you shouldn't boot the Xen guest with it).\n"); + exit (EXIT_FAILURE); +} + +static void iterate_inputs (char **inputs, int nr_inputs); +static void iterate_input_directory (const char *dirname, int dirfd); +static void write_kernel_modules (const char *whitelist, const char *modpath); +static void write_hostfiles (const char *hostfiles_file); +static void write_to_fd (const void *buffer, size_t len); +static void write_file_to_fd (const char *filename); +static void write_file_len_to_fd (const char *filename, size_t len); +static void write_padding (size_t len); +static char **load_file (const char *filename); +static void cpio_append_fts_entry (FTSENT *entry); +static void cpio_append_stat (const char *filename, struct stat *); +static void cpio_append (const char *filename); +static void cpio_append_trailer (void); + +static int out_fd = -1; +static off_t out_offset = 0; + +/* Create the appliance. + * + * The initrd consists of these components concatenated together: + * + * (1) The base skeleton appliance that we constructed at build time. + * format = plain cpio + * (2) The host files which match wildcards in *.supermin.hostfiles. + * input format = plain text, output format = plain cpio + * (3) The modules from modpath which are on the module whitelist. + * output format = plain cpio + * + * The original shell script used the external cpio program to create + * parts (2) and (3), but we have decided it's going to be faster if + * we just write out the data outselves. The reasons are that + * external cpio is slow (particularly when used with SELinux because + * it does 512 byte reads), and the format that we're writing is + * narrow and well understood, because we only care that the Linux + * kernel can read it. + * + * This version contains some improvements over the C version written + * for libguestfs, in that we can have multiple base images (or + * hostfiles) or use a directory to store these files. + */ +static void +create_appliance (char **inputs, int nr_inputs, + const char *whitelist, + const char *modpath, + const char *initrd) +{ + out_fd = open (initrd, O_WRONLY | O_CREAT | O_TRUNC | O_NOCTTY, 0644); + if (out_fd == -1) + error (EXIT_FAILURE, errno, "open: %s", initrd); + out_offset = 0; + + iterate_inputs (inputs, nr_inputs); + + /* Kernel modules (3). */ + write_kernel_modules (whitelist, modpath); + + cpio_append_trailer (); + + /* Finish off and close output file. */ + if (close (out_fd) == -1) + error (EXIT_FAILURE, errno, "close: %s", initrd); +} + +/* Iterate over the inputs to find out what they are, visiting + * directories if specified. + */ +static void +iterate_inputs (char **inputs, int nr_inputs) +{ + int i; + for (i = 0; i < nr_inputs; ++i) { + if (verbose) + print_timestamped_message ("visiting %s", inputs[i]); + + int fd = open (inputs[i], O_RDONLY); + if (fd == -1) + error (EXIT_FAILURE, errno, "open: %s", inputs[i]); + + struct stat statbuf; + if (fstat (fd, &statbuf) == -1) + error (EXIT_FAILURE, errno, "fstat: %s", inputs[i]); + + /* Directory? */ + if (S_ISDIR (statbuf.st_mode)) + iterate_input_directory (inputs[i], fd); + else if (S_ISREG (statbuf.st_mode)) { + /* Is it a cpio file? */ + char buf[6]; + if (read (fd, buf, 6) == 6 && memcmp (buf, "070701", 6) == 0) + /* Yes, a cpio file. This is a skeleton appliance, case (1). */ + write_file_to_fd (inputs[i]); + else + /* No, must be hostfiles, case (2). */ + write_hostfiles (inputs[i]); + } + else + error (EXIT_FAILURE, 0, "%s: input is not a regular file or directory", + inputs[i]); + + close (fd); + } +} + +static void +iterate_input_directory (const char *dirname, int dirfd) +{ + char path[PATH_MAX]; + strcpy (path, dirname); + size_t len = strlen (dirname); + path[len++] = '/'; + + char *inputs[] = { path }; + + DIR *dir = fdopendir (dirfd); + if (dir == NULL) + error (EXIT_FAILURE, errno, "fdopendir: %s", dirname); + + struct dirent *d; + while ((errno = 0, d = readdir (dir)) != NULL) { + if (d->d_name[0] == '.') /* ignore ., .. and any hidden files. */ + continue; + + strcpy (&path[len], d->d_name); + iterate_inputs (inputs, 1); + } + + if (errno != 0) + error (EXIT_FAILURE, errno, "readdir: %s", dirname); + + if (closedir (dir) == -1) + error (EXIT_FAILURE, errno, "closedir: %s", dirname); +} + +/* Copy kernel modules. + * + * Find every file under modpath. + * + * Exclude all *.ko files, *except* ones which match names in + * the whitelist (which may contain wildcards). Include all + * other files. + * + * Add chosen files to the output. + * + * whitelist_file may be NULL, to include ALL kernel modules. + */ +static void +write_kernel_modules (const char *whitelist_file, const char *modpath) +{ + char **whitelist = NULL; + if (whitelist_file != NULL) + whitelist = load_file (whitelist_file); + + char *paths[2] = { (char *) modpath, NULL }; + FTS *fts = fts_open (paths, FTS_COMFOLLOW|FTS_PHYSICAL, NULL); + if (fts == NULL) + error (EXIT_FAILURE, errno, "write_kernel_modules: fts_open: %s", modpath); + + for (;;) { + errno = 0; + FTSENT *entry = fts_read (fts); + if (entry == NULL && errno != 0) + error (EXIT_FAILURE, errno, "write_kernel_modules: fts_read: %s", modpath); + if (entry == NULL) + break; + + /* Ignore directories being visited in post-order. */ + if (entry->fts_info & FTS_DP) + continue; + + /* Is it a *.ko file? */ + if (entry->fts_namelen >= 3 && + entry->fts_name[entry->fts_namelen-3] == '.' && + entry->fts_name[entry->fts_namelen-2] == 'k' && + entry->fts_name[entry->fts_namelen-1] == 'o') { + if (whitelist) { + /* Is it a *.ko file which is on the whitelist? */ + size_t j; + for (j = 0; whitelist[j] != NULL; ++j) { + int r; + r = fnmatch (whitelist[j], entry->fts_name, 0); + if (r == 0) { + /* It's on the whitelist, so include it. */ + if (verbose >= 2) + fprintf (stderr, "including kernel module %s (matches whitelist entry %s)\n", + entry->fts_name, whitelist[j]); + cpio_append_fts_entry (entry); + break; + } else if (r != FNM_NOMATCH) + error (EXIT_FAILURE, 0, "internal error: fnmatch ('%s', '%s', %d) returned unexpected non-zero value %d\n", + whitelist[j], entry->fts_name, 0, r); + } /* for (j) */ + } else { /* whitelist == NULL, always include */ + if (verbose >= 2) + fprintf (stderr, "including kernel module %s\n", entry->fts_name); + cpio_append_fts_entry (entry); + } + } else + /* It's some other sort of file, or a directory, always include. */ + cpio_append_fts_entry (entry); + } + + if (fts_close (fts) == -1) + error (EXIT_FAILURE, errno, "write_kernel_modules: fts_close: %s", modpath); +} + +/* Copy the host files. + * + * Read the list of entries in hostfiles (which may contain + * wildcards). Look them up in the filesystem, and add those files + * that exist. Ignore any files that don't exist or are not readable. + */ +static void +write_hostfiles (const char *hostfiles_file) +{ + char **hostfiles = load_file (hostfiles_file); + + /* Hostfiles list can contain "." before each path - ignore it. + * It also contains each directory name before we enter it. But + * we don't read that until we see a wildcard for that directory. + */ + size_t i, j; + for (i = 0; hostfiles[i] != NULL; ++i) { + char *hostfile = hostfiles[i]; + if (hostfile[0] == '.') + hostfile++; + + struct stat statbuf; + + /* Is it a wildcard? */ + if (strchr (hostfile, '*') || strchr (hostfile, '?')) { + char *dirname = xstrdup (hostfile); + char *patt = strrchr (dirname, '/'); + assert (patt); + *patt++ = '\0'; + + char **files = read_dir (dirname); + files = filter_fnmatch (files, patt, FNM_NOESCAPE); + + /* Add matching files. */ + for (j = 0; files[j] != NULL; ++j) { + char *tmp = xasprintf ("%s/%s", dirname, files[j]); + + if (verbose >= 2) + fprintf (stderr, "including host file %s (matches %s)\n", tmp, patt); + + cpio_append (tmp); + + free (tmp); + } + } + /* Else does this file/directory/whatever exist? */ + else if (lstat (hostfile, &statbuf) == 0) { + if (verbose >= 2) + fprintf (stderr, "including host file %s (directly referenced)\n", + hostfile); + + cpio_append_stat (hostfile, &statbuf); + } /* Ignore files that don't exist. */ + } +} + +/*----------*/ +/* Helper functions. */ + +static void +add_string (char ***argv, size_t *n_used, size_t *n_alloc, const char *str) +{ + char *new_str; + + if (*n_used >= *n_alloc) + *argv = x2nrealloc (*argv, n_alloc, sizeof (char *)); + + if (str) + new_str = xstrdup (str); + else + new_str = NULL; + + (*argv)[*n_used] = new_str; + + (*n_used)++; +} + +static size_t +count_strings (char *const *argv) +{ + size_t argc; + + for (argc = 0; argv[argc] != NULL; ++argc) + ; + return argc; +} + +struct dir_cache { + char *path; + char **files; +}; + +static size_t +dir_cache_hash (void const *x, size_t table_size) +{ + struct dir_cache const *p = x; + return hash_pjw (p->path, table_size); +} + +static bool +dir_cache_compare (void const *x, void const *y) +{ + struct dir_cache const *p = x; + struct dir_cache const *q = y; + return strcmp (p->path, q->path) == 0; +} + +/* Read a directory into a list of strings. + * + * Previously looked up directories are cached and returned quickly, + * saving some considerable amount of time compared to reading the + * directory over again. However this means you really must not + * alter the array of strings that are returned. + * + * Returns an empty list if the directory cannot be opened. + */ +static char ** +read_dir (const char *name) +{ + static Hash_table *ht = NULL; + + if (!ht) + ht = hash_initialize (1024, NULL, dir_cache_hash, dir_cache_compare, NULL); + + struct dir_cache key = { .path = (char *) name }; + struct dir_cache *p = hash_lookup (ht, &key); + if (p) + return p->files; + + char **files = NULL; + size_t n_used = 0, n_alloc = 0; + + DIR *dir = opendir (name); + if (!dir) { + /* If it fails to open, that's OK, skip to the end. */ + /*perror (name);*/ + goto done; + } + + for (;;) { + errno = 0; + struct dirent *d = readdir (dir); + if (d == NULL) { + if (errno != 0) + /* But if it fails here, after opening and potentially reading + * part of the directory, that's a proper failure - inform the + * user and exit. + */ + error (EXIT_FAILURE, errno, "%s", name); + break; + } + + add_string (&files, &n_used, &n_alloc, d->d_name); + } + + if (closedir (dir) == -1) + error (EXIT_FAILURE, errno, "closedir: %s", name); + + done: + /* NULL-terminate the array. */ + add_string (&files, &n_used, &n_alloc, NULL); + + /* Add it to the hash for next time. */ + p = xmalloc (sizeof *p); + p->path = (char *) name; + p->files = files; + p = hash_insert (ht, p); + assert (p != NULL); + + return files; +} + +/* Filter a list of strings and return only those matching the wildcard. */ +static char ** +filter_fnmatch (char **strings, const char *patt, int flags) +{ + char **out = NULL; + size_t n_used = 0, n_alloc = 0; + + int i, r; + for (i = 0; strings[i] != NULL; ++i) { + r = fnmatch (patt, strings[i], flags); + if (r == 0) + add_string (&out, &n_used, &n_alloc, strings[i]); + else if (r != FNM_NOMATCH) + error (EXIT_FAILURE, 0, "internal error: fnmatch ('%s', '%s', %d) returned unexpected non-zero value %d\n", + patt, strings[i], flags, r); + } + + add_string (&out, &n_used, &n_alloc, NULL); + return out; +} + +/* Filter a list of strings and return only those which DON'T contain sub. */ +static char ** +filter_notmatching_substring (char **strings, const char *sub) +{ + char **out = NULL; + size_t n_used = 0, n_alloc = 0; + + int i; + for (i = 0; strings[i] != NULL; ++i) { + if (strstr (strings[i], sub) == NULL) + add_string (&out, &n_used, &n_alloc, strings[i]); + } + + add_string (&out, &n_used, &n_alloc, NULL); + return out; +} + +/* Sort a list of strings, in place, with the comparison function supplied. */ +static void +sort (char **strings, int (*compare) (const void *, const void *)) +{ + qsort (strings, count_strings (strings), sizeof (char *), compare); +} + +/* Return true iff path exists and is a directory. This version + * follows symlinks. + */ +static int +isdir (const char *path) +{ + struct stat statbuf; + + if (stat (path, &statbuf) == -1) + return 0; + + return S_ISDIR (statbuf.st_mode); +} + +/* Copy contents of buffer to out_fd and keep out_offset correct. */ +static void +write_to_fd (const void *buffer, size_t len) +{ + if (full_write (out_fd, buffer, len) != len) + error (EXIT_FAILURE, errno, "write"); + out_offset += len; +} + +/* Copy contents of file to out_fd. */ +static void +write_file_to_fd (const char *filename) +{ + char buffer[BUFFER_SIZE]; + int fd2; + ssize_t r; + + if (verbose >= 2) + fprintf (stderr, "write_file_to_fd %s -> %d\n", filename, out_fd); + + fd2 = open (filename, O_RDONLY); + if (fd2 == -1) + error (EXIT_FAILURE, errno, "open: %s", filename); + for (;;) { + r = read (fd2, buffer, sizeof buffer); + if (r == 0) + break; + if (r == -1) { + if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) + continue; + error (EXIT_FAILURE, errno, "read: %s", filename); + } + write_to_fd (buffer, r); + } + + if (close (fd2) == -1) + error (EXIT_FAILURE, errno, "close: %s", filename); +} + +/* Copy file of given length to output, and fail if the file has + * changed size. + */ +static void +write_file_len_to_fd (const char *filename, size_t len) +{ + char buffer[BUFFER_SIZE]; + size_t count = 0; + + if (verbose >= 2) + fprintf (stderr, "write_file_to_fd %s -> %d\n", filename, out_fd); + + int fd2 = open (filename, O_RDONLY); + if (fd2 == -1) + error (EXIT_FAILURE, errno, "open: %s", filename); + for (;;) { + ssize_t r = read (fd2, buffer, sizeof buffer); + if (r == 0) + break; + if (r == -1) { + if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) + continue; + error (EXIT_FAILURE, errno, "read: %s", filename); + } + write_to_fd (buffer, r); + count += r; + if (count > len) + error (EXIT_FAILURE, 0, "write_file_len_to_fd: %s: file has increased in size\n", filename); + } + + if (close (fd2) == -1) + error (EXIT_FAILURE, errno, "close: %s", filename); + + if (count != len) + error (EXIT_FAILURE, 0, "febootstrap-supermin-helper: write_file_len_to_fd: %s: file has changed size\n", filename); +} + +/* Load in a file, returning a list of lines. */ +static char ** +load_file (const char *filename) +{ + char **lines = 0; + size_t n_used = 0, n_alloc = 0; + + FILE *fp; + fp = fopen (filename, "r"); + if (fp == NULL) + error (EXIT_FAILURE, errno, "fopen: %s", filename); + + char line[4096]; + while (fgets (line, sizeof line, fp)) { + size_t len = strlen (line); + if (len > 0 && line[len-1] == '\n') + line[len-1] = '\0'; + add_string (&lines, &n_used, &n_alloc, line); + } + + add_string (&lines, &n_used, &n_alloc, NULL); + return lines; +} + +/* Append the file pointed to by FTSENT to the cpio output. */ +static void +cpio_append_fts_entry (FTSENT *entry) +{ + if (entry->fts_info & FTS_NS || entry->fts_info & FTS_NSOK) + cpio_append (entry->fts_path); + else + cpio_append_stat (entry->fts_path, entry->fts_statp); +} + +/* Append the file named 'filename' to the cpio output. */ +static void +cpio_append (const char *filename) +{ + struct stat statbuf; + + if (lstat (filename, &statbuf) == -1) + error (EXIT_FAILURE, errno, "lstat: %s", filename); + cpio_append_stat (filename, &statbuf); +} + +/* Append the file to the cpio output. */ +#define PADDING(len) ((((len) + 3) & ~3) - (len)) + +#define CPIO_HEADER_LEN (6 + 13*8) + +static void +cpio_append_stat (const char *filename, struct stat *statbuf) +{ + const char *orig_filename = filename; + + if (*filename == '/') + filename++; + if (*filename == '\0') + filename = "."; + + if (verbose >= 2) + fprintf (stderr, "cpio_append_stat %s 0%o -> %d\n", + orig_filename, statbuf->st_mode, out_fd); + + /* Regular files and symlinks are the only ones that have a "body" + * in this cpio entry. + */ + int has_body = S_ISREG (statbuf->st_mode) || S_ISLNK (statbuf->st_mode); + + size_t len = strlen (filename) + 1; + + char header[CPIO_HEADER_LEN + 1]; + snprintf (header, sizeof header, + "070701" /* magic */ + "%08X" /* inode */ + "%08X" /* mode */ + "%08X" "%08X" /* uid, gid */ + "%08X" /* nlink */ + "%08X" /* mtime */ + "%08X" /* file length */ + "%08X" "%08X" /* device holding file major, minor */ + "%08X" "%08X" /* for specials, device major, minor */ + "%08X" /* name length (including \0 byte) */ + "%08X", /* checksum (not used by the kernel) */ + (unsigned) statbuf->st_ino, statbuf->st_mode, + statbuf->st_uid, statbuf->st_gid, + (unsigned) statbuf->st_nlink, (unsigned) statbuf->st_mtime, + has_body ? (unsigned) statbuf->st_size : 0, + major (statbuf->st_dev), minor (statbuf->st_dev), + major (statbuf->st_rdev), minor (statbuf->st_rdev), + (unsigned) len, 0); + + /* Write the header. */ + write_to_fd (header, CPIO_HEADER_LEN); + + /* Follow with the filename, and pad it. */ + write_to_fd (filename, len); + size_t padding_len = PADDING (CPIO_HEADER_LEN + len); + write_padding (padding_len); + + /* Follow with the file or symlink content, and pad it. */ + if (has_body) { + if (S_ISREG (statbuf->st_mode)) + write_file_len_to_fd (orig_filename, statbuf->st_size); + else if (S_ISLNK (statbuf->st_mode)) { + char tmp[PATH_MAX]; + if (readlink (orig_filename, tmp, sizeof tmp) == -1) + error (EXIT_FAILURE, errno, "readlink: %s", orig_filename); + write_to_fd (tmp, statbuf->st_size); + } + + padding_len = PADDING (statbuf->st_size); + write_padding (padding_len); + } +} + +/* CPIO voodoo. */ +static void +cpio_append_trailer (void) +{ + struct stat statbuf; + memset (&statbuf, 0, sizeof statbuf); + statbuf.st_nlink = 1; + cpio_append_stat ("TRAILER!!!", &statbuf); + + /* CPIO seems to pad up to the next block boundary, ie. up to + * the next 512 bytes. + */ + write_padding (((out_offset + 511) & ~511) - out_offset); + assert ((out_offset & 511) == 0); +} + +/* Write 'len' bytes of zeroes out. */ +static void +write_padding (size_t len) +{ + static const char buffer[512] = { 0 }; + + while (len > 0) { + size_t n = len < sizeof buffer ? len : sizeof buffer; + write_to_fd (buffer, n); + len -= n; + } +} -- 1.7.1
Richard W.M. Jones
2010-Aug-20 10:39 UTC
[Libguestfs] [PATCH febootstrap 2/8] Refactor febootstrap-supermin-helper code into separate files.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw -------------- next part -------------->From df48822a767b5f2d4f58ddd5d56420e3b79ed011 Mon Sep 17 00:00:00 2001From: Richard Jones <rjones at redhat.com> Date: Thu, 19 Aug 2010 13:41:12 +0100 Subject: [PATCH 2/8] Refactor febootstrap-supermin-helper code into separate files. This is just code movement. There is no change to the code. --- helper/Makefile.am | 7 +- helper/README | 11 + helper/appliance.c | 501 +++++++++++++++++++++++++++++++ helper/helper.h | 45 +++ helper/kernel.c | 127 ++++++++ helper/main.c | 842 +--------------------------------------------------- helper/utils.c | 281 +++++++++++++++++ 7 files changed, 974 insertions(+), 840 deletions(-) create mode 100644 helper/README create mode 100644 helper/appliance.c create mode 100644 helper/helper.h create mode 100644 helper/kernel.c create mode 100644 helper/utils.c diff --git a/helper/Makefile.am b/helper/Makefile.am index 518cfa6..fb21286 100644 --- a/helper/Makefile.am +++ b/helper/Makefile.am @@ -21,7 +21,12 @@ bin_PROGRAMS = \ febootstrap-supermin-helper DISTCLEANFILES = $(bin_SCRIPTS) -febootstrap_supermin_helper_SOURCES = main.c +febootstrap_supermin_helper_SOURCES = \ + helper.h \ + appliance.c \ + kernel.c \ + main.c \ + utils.c febootstrap_supermin_helper_CFLAGS = -Wall -I../lib febootstrap_supermin_helper_LDADD = $(LTLIBINTL) -L../lib -lgnu diff --git a/helper/README b/helper/README new file mode 100644 index 0000000..2577b31 --- /dev/null +++ b/helper/README @@ -0,0 +1,11 @@ +This directory contains the febootstrap-supermin-helper +reimplementation in C. + +This program builds the supermin appliance on the fly each time the +appliance runs (or in recent versions of libguestfs, the appliance is +cached as long as the host files don't change). + +*NOTE*: This program is designed to be very short-lived, and so we +don't normally bother to free up any memory that we allocate. That's +not completely true - we free up stuff if it's obvious and easy to +free up, and ignore the rest. diff --git a/helper/appliance.c b/helper/appliance.c new file mode 100644 index 0000000..c7200bd --- /dev/null +++ b/helper/appliance.c @@ -0,0 +1,501 @@ +/* febootstrap-supermin-helper reimplementation in C. + * Copyright (C) 2009-2010 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <unistd.h> +#include <limits.h> +#include <fcntl.h> +#include <errno.h> +#include <dirent.h> +#include <fnmatch.h> +#include <sys/stat.h> +#include <assert.h> + +#include "error.h" +#include "fts_.h" +#include "full-write.h" +#include "xalloc.h" +#include "xvasprintf.h" + +#include "helper.h" + +/* Buffer size used in copy operations throughout. Large for + * greatest efficiency. + */ +#define BUFFER_SIZE 65536 + +static void iterate_inputs (char **inputs, int nr_inputs); +static void iterate_input_directory (const char *dirname, int dirfd); +static void write_kernel_modules (const char *whitelist, const char *modpath); +static void write_hostfiles (const char *hostfiles_file); +static void write_to_fd (const void *buffer, size_t len); +static void write_file_to_fd (const char *filename); +static void write_file_len_to_fd (const char *filename, size_t len); +static void write_padding (size_t len); +static void cpio_append_fts_entry (FTSENT *entry); +static void cpio_append_stat (const char *filename, struct stat *); +static void cpio_append (const char *filename); +static void cpio_append_trailer (void); + +static int out_fd = -1; +static off_t out_offset = 0; + +/* Create the appliance. + * + * The initrd consists of these components concatenated together: + * + * (1) The base skeleton appliance that we constructed at build time. + * format = plain cpio + * (2) The host files which match wildcards in *.supermin.hostfiles. + * input format = plain text, output format = plain cpio + * (3) The modules from modpath which are on the module whitelist. + * output format = plain cpio + * + * The original shell script used the external cpio program to create + * parts (2) and (3), but we have decided it's going to be faster if + * we just write out the data outselves. The reasons are that + * external cpio is slow (particularly when used with SELinux because + * it does 512 byte reads), and the format that we're writing is + * narrow and well understood, because we only care that the Linux + * kernel can read it. + * + * This version contains some improvements over the C version written + * for libguestfs, in that we can have multiple base images (or + * hostfiles) or use a directory to store these files. + */ +void +create_appliance (char **inputs, int nr_inputs, + const char *whitelist, + const char *modpath, + const char *initrd) +{ + out_fd = open (initrd, O_WRONLY | O_CREAT | O_TRUNC | O_NOCTTY, 0644); + if (out_fd == -1) + error (EXIT_FAILURE, errno, "open: %s", initrd); + out_offset = 0; + + iterate_inputs (inputs, nr_inputs); + + /* Kernel modules (3). */ + write_kernel_modules (whitelist, modpath); + + cpio_append_trailer (); + + /* Finish off and close output file. */ + if (close (out_fd) == -1) + error (EXIT_FAILURE, errno, "close: %s", initrd); +} + +/* Iterate over the inputs to find out what they are, visiting + * directories if specified. + */ +static void +iterate_inputs (char **inputs, int nr_inputs) +{ + int i; + for (i = 0; i < nr_inputs; ++i) { + if (verbose) + print_timestamped_message ("visiting %s", inputs[i]); + + int fd = open (inputs[i], O_RDONLY); + if (fd == -1) + error (EXIT_FAILURE, errno, "open: %s", inputs[i]); + + struct stat statbuf; + if (fstat (fd, &statbuf) == -1) + error (EXIT_FAILURE, errno, "fstat: %s", inputs[i]); + + /* Directory? */ + if (S_ISDIR (statbuf.st_mode)) + iterate_input_directory (inputs[i], fd); + else if (S_ISREG (statbuf.st_mode)) { + /* Is it a cpio file? */ + char buf[6]; + if (read (fd, buf, 6) == 6 && memcmp (buf, "070701", 6) == 0) + /* Yes, a cpio file. This is a skeleton appliance, case (1). */ + write_file_to_fd (inputs[i]); + else + /* No, must be hostfiles, case (2). */ + write_hostfiles (inputs[i]); + } + else + error (EXIT_FAILURE, 0, "%s: input is not a regular file or directory", + inputs[i]); + + close (fd); + } +} + +static void +iterate_input_directory (const char *dirname, int dirfd) +{ + char path[PATH_MAX]; + strcpy (path, dirname); + size_t len = strlen (dirname); + path[len++] = '/'; + + char *inputs[] = { path }; + + DIR *dir = fdopendir (dirfd); + if (dir == NULL) + error (EXIT_FAILURE, errno, "fdopendir: %s", dirname); + + struct dirent *d; + while ((errno = 0, d = readdir (dir)) != NULL) { + if (d->d_name[0] == '.') /* ignore ., .. and any hidden files. */ + continue; + + strcpy (&path[len], d->d_name); + iterate_inputs (inputs, 1); + } + + if (errno != 0) + error (EXIT_FAILURE, errno, "readdir: %s", dirname); + + if (closedir (dir) == -1) + error (EXIT_FAILURE, errno, "closedir: %s", dirname); +} + +/* Copy kernel modules. + * + * Find every file under modpath. + * + * Exclude all *.ko files, *except* ones which match names in + * the whitelist (which may contain wildcards). Include all + * other files. + * + * Add chosen files to the output. + * + * whitelist_file may be NULL, to include ALL kernel modules. + */ +static void +write_kernel_modules (const char *whitelist_file, const char *modpath) +{ + char **whitelist = NULL; + if (whitelist_file != NULL) + whitelist = load_file (whitelist_file); + + char *paths[2] = { (char *) modpath, NULL }; + FTS *fts = fts_open (paths, FTS_COMFOLLOW|FTS_PHYSICAL, NULL); + if (fts == NULL) + error (EXIT_FAILURE, errno, "write_kernel_modules: fts_open: %s", modpath); + + for (;;) { + errno = 0; + FTSENT *entry = fts_read (fts); + if (entry == NULL && errno != 0) + error (EXIT_FAILURE, errno, "write_kernel_modules: fts_read: %s", modpath); + if (entry == NULL) + break; + + /* Ignore directories being visited in post-order. */ + if (entry->fts_info & FTS_DP) + continue; + + /* Is it a *.ko file? */ + if (entry->fts_namelen >= 3 && + entry->fts_name[entry->fts_namelen-3] == '.' && + entry->fts_name[entry->fts_namelen-2] == 'k' && + entry->fts_name[entry->fts_namelen-1] == 'o') { + if (whitelist) { + /* Is it a *.ko file which is on the whitelist? */ + size_t j; + for (j = 0; whitelist[j] != NULL; ++j) { + int r; + r = fnmatch (whitelist[j], entry->fts_name, 0); + if (r == 0) { + /* It's on the whitelist, so include it. */ + if (verbose >= 2) + fprintf (stderr, "including kernel module %s (matches whitelist entry %s)\n", + entry->fts_name, whitelist[j]); + cpio_append_fts_entry (entry); + break; + } else if (r != FNM_NOMATCH) + error (EXIT_FAILURE, 0, "internal error: fnmatch ('%s', '%s', %d) returned unexpected non-zero value %d\n", + whitelist[j], entry->fts_name, 0, r); + } /* for (j) */ + } else { /* whitelist == NULL, always include */ + if (verbose >= 2) + fprintf (stderr, "including kernel module %s\n", entry->fts_name); + cpio_append_fts_entry (entry); + } + } else + /* It's some other sort of file, or a directory, always include. */ + cpio_append_fts_entry (entry); + } + + if (fts_close (fts) == -1) + error (EXIT_FAILURE, errno, "write_kernel_modules: fts_close: %s", modpath); +} + +/* Copy the host files. + * + * Read the list of entries in hostfiles (which may contain + * wildcards). Look them up in the filesystem, and add those files + * that exist. Ignore any files that don't exist or are not readable. + */ +static void +write_hostfiles (const char *hostfiles_file) +{ + char **hostfiles = load_file (hostfiles_file); + + /* Hostfiles list can contain "." before each path - ignore it. + * It also contains each directory name before we enter it. But + * we don't read that until we see a wildcard for that directory. + */ + size_t i, j; + for (i = 0; hostfiles[i] != NULL; ++i) { + char *hostfile = hostfiles[i]; + if (hostfile[0] == '.') + hostfile++; + + struct stat statbuf; + + /* Is it a wildcard? */ + if (strchr (hostfile, '*') || strchr (hostfile, '?')) { + char *dirname = xstrdup (hostfile); + char *patt = strrchr (dirname, '/'); + assert (patt); + *patt++ = '\0'; + + char **files = read_dir (dirname); + files = filter_fnmatch (files, patt, FNM_NOESCAPE); + + /* Add matching files. */ + for (j = 0; files[j] != NULL; ++j) { + char *tmp = xasprintf ("%s/%s", dirname, files[j]); + + if (verbose >= 2) + fprintf (stderr, "including host file %s (matches %s)\n", tmp, patt); + + cpio_append (tmp); + + free (tmp); + } + } + /* Else does this file/directory/whatever exist? */ + else if (lstat (hostfile, &statbuf) == 0) { + if (verbose >= 2) + fprintf (stderr, "including host file %s (directly referenced)\n", + hostfile); + + cpio_append_stat (hostfile, &statbuf); + } /* Ignore files that don't exist. */ + } +} + +/* Copy contents of buffer to out_fd and keep out_offset correct. */ +static void +write_to_fd (const void *buffer, size_t len) +{ + if (full_write (out_fd, buffer, len) != len) + error (EXIT_FAILURE, errno, "write"); + out_offset += len; +} + +/* Copy contents of file to out_fd. */ +static void +write_file_to_fd (const char *filename) +{ + char buffer[BUFFER_SIZE]; + int fd2; + ssize_t r; + + if (verbose >= 2) + fprintf (stderr, "write_file_to_fd %s -> %d\n", filename, out_fd); + + fd2 = open (filename, O_RDONLY); + if (fd2 == -1) + error (EXIT_FAILURE, errno, "open: %s", filename); + for (;;) { + r = read (fd2, buffer, sizeof buffer); + if (r == 0) + break; + if (r == -1) { + if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) + continue; + error (EXIT_FAILURE, errno, "read: %s", filename); + } + write_to_fd (buffer, r); + } + + if (close (fd2) == -1) + error (EXIT_FAILURE, errno, "close: %s", filename); +} + +/* Copy file of given length to output, and fail if the file has + * changed size. + */ +static void +write_file_len_to_fd (const char *filename, size_t len) +{ + char buffer[BUFFER_SIZE]; + size_t count = 0; + + if (verbose >= 2) + fprintf (stderr, "write_file_to_fd %s -> %d\n", filename, out_fd); + + int fd2 = open (filename, O_RDONLY); + if (fd2 == -1) + error (EXIT_FAILURE, errno, "open: %s", filename); + for (;;) { + ssize_t r = read (fd2, buffer, sizeof buffer); + if (r == 0) + break; + if (r == -1) { + if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) + continue; + error (EXIT_FAILURE, errno, "read: %s", filename); + } + write_to_fd (buffer, r); + count += r; + if (count > len) + error (EXIT_FAILURE, 0, "write_file_len_to_fd: %s: file has increased in size\n", filename); + } + + if (close (fd2) == -1) + error (EXIT_FAILURE, errno, "close: %s", filename); + + if (count != len) + error (EXIT_FAILURE, 0, "febootstrap-supermin-helper: write_file_len_to_fd: %s: file has changed size\n", filename); +} + +/* Append the file pointed to by FTSENT to the cpio output. */ +static void +cpio_append_fts_entry (FTSENT *entry) +{ + if (entry->fts_info & FTS_NS || entry->fts_info & FTS_NSOK) + cpio_append (entry->fts_path); + else + cpio_append_stat (entry->fts_path, entry->fts_statp); +} + +/* Append the file named 'filename' to the cpio output. */ +static void +cpio_append (const char *filename) +{ + struct stat statbuf; + + if (lstat (filename, &statbuf) == -1) + error (EXIT_FAILURE, errno, "lstat: %s", filename); + cpio_append_stat (filename, &statbuf); +} + +/* Append the file to the cpio output. */ +#define PADDING(len) ((((len) + 3) & ~3) - (len)) + +#define CPIO_HEADER_LEN (6 + 13*8) + +static void +cpio_append_stat (const char *filename, struct stat *statbuf) +{ + const char *orig_filename = filename; + + if (*filename == '/') + filename++; + if (*filename == '\0') + filename = "."; + + if (verbose >= 2) + fprintf (stderr, "cpio_append_stat %s 0%o -> %d\n", + orig_filename, statbuf->st_mode, out_fd); + + /* Regular files and symlinks are the only ones that have a "body" + * in this cpio entry. + */ + int has_body = S_ISREG (statbuf->st_mode) || S_ISLNK (statbuf->st_mode); + + size_t len = strlen (filename) + 1; + + char header[CPIO_HEADER_LEN + 1]; + snprintf (header, sizeof header, + "070701" /* magic */ + "%08X" /* inode */ + "%08X" /* mode */ + "%08X" "%08X" /* uid, gid */ + "%08X" /* nlink */ + "%08X" /* mtime */ + "%08X" /* file length */ + "%08X" "%08X" /* device holding file major, minor */ + "%08X" "%08X" /* for specials, device major, minor */ + "%08X" /* name length (including \0 byte) */ + "%08X", /* checksum (not used by the kernel) */ + (unsigned) statbuf->st_ino, statbuf->st_mode, + statbuf->st_uid, statbuf->st_gid, + (unsigned) statbuf->st_nlink, (unsigned) statbuf->st_mtime, + has_body ? (unsigned) statbuf->st_size : 0, + major (statbuf->st_dev), minor (statbuf->st_dev), + major (statbuf->st_rdev), minor (statbuf->st_rdev), + (unsigned) len, 0); + + /* Write the header. */ + write_to_fd (header, CPIO_HEADER_LEN); + + /* Follow with the filename, and pad it. */ + write_to_fd (filename, len); + size_t padding_len = PADDING (CPIO_HEADER_LEN + len); + write_padding (padding_len); + + /* Follow with the file or symlink content, and pad it. */ + if (has_body) { + if (S_ISREG (statbuf->st_mode)) + write_file_len_to_fd (orig_filename, statbuf->st_size); + else if (S_ISLNK (statbuf->st_mode)) { + char tmp[PATH_MAX]; + if (readlink (orig_filename, tmp, sizeof tmp) == -1) + error (EXIT_FAILURE, errno, "readlink: %s", orig_filename); + write_to_fd (tmp, statbuf->st_size); + } + + padding_len = PADDING (statbuf->st_size); + write_padding (padding_len); + } +} + +/* CPIO voodoo. */ +static void +cpio_append_trailer (void) +{ + struct stat statbuf; + memset (&statbuf, 0, sizeof statbuf); + statbuf.st_nlink = 1; + cpio_append_stat ("TRAILER!!!", &statbuf); + + /* CPIO seems to pad up to the next block boundary, ie. up to + * the next 512 bytes. + */ + write_padding (((out_offset + 511) & ~511) - out_offset); + assert ((out_offset & 511) == 0); +} + +/* Write 'len' bytes of zeroes out. */ +static void +write_padding (size_t len) +{ + static const char buffer[512] = { 0 }; + + while (len > 0) { + size_t n = len < sizeof buffer ? len : sizeof buffer; + write_to_fd (buffer, n); + len -= n; + } +} diff --git a/helper/helper.h b/helper/helper.h new file mode 100644 index 0000000..fdf446e --- /dev/null +++ b/helper/helper.h @@ -0,0 +1,45 @@ +/* febootstrap-supermin-helper reimplementation in C. + * Copyright (C) 2009-2010 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + */ + +#ifndef FEBOOTSTRAP_SUPERMIN_HELPER_H +#define FEBOOTSTRAP_SUPERMIN_HELPER_H + +/* main.c */ +extern struct timeval start_t; +extern int verbose; + +/* appliance.c */ +extern void create_appliance (char **inputs, int nr_inputs, const char *whitelist, const char *modpath, const char *initrd); + +/* kernel.c */ +extern const char *create_kernel (const char *hostcpu, const char *kernel); + +/* utils.c */ +extern void print_timestamped_message (const char *fs, ...); +extern int64_t timeval_diff (const struct timeval *x, const struct timeval *y); +extern int reverse_filevercmp (const void *p1, const void *p2); +extern void add_string (char ***argv, size_t *n_used, size_t *n_alloc, const char *str); +extern size_t count_strings (char *const *argv); +extern char **read_dir (const char *name); +extern char **filter_fnmatch (char **strings, const char *patt, int flags); +extern char **filter_notmatching_substring (char **strings, const char *sub); +extern void sort (char **strings, int (*compare) (const void *, const void *)); +extern int isdir (const char *path); +extern char **load_file (const char *filename); + +#endif /* FEBOOTSTRAP_SUPERMIN_HELPER_H */ diff --git a/helper/kernel.c b/helper/kernel.c new file mode 100644 index 0000000..6b0607f --- /dev/null +++ b/helper/kernel.c @@ -0,0 +1,127 @@ +/* febootstrap-supermin-helper reimplementation in C. + * Copyright (C) 2009-2010 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <fnmatch.h> +#include <unistd.h> +#include <errno.h> + +#include "error.h" +#include "xvasprintf.h" + +#include "helper.h" + +/* Directory containing candidate kernels. We could make this + * configurable at some point. + */ +#define KERNELDIR "/boot" +#define MODULESDIR "/lib/modules" + +/* Create the kernel. This chooses an appropriate kernel and makes a + * symlink to it. + * + * Look for the most recent kernel named vmlinuz-*.<arch>* which has a + * corresponding directory in /lib/modules/. If the architecture is + * x86, look for any x86 kernel. + * + * RHEL 5 didn't append the arch to the kernel name, so look for + * kernels without arch second. + * + * If no suitable kernel can be found, exit with an error. + * + * This function returns the module path (ie. /lib/modules/<version>). + */ +const char * +create_kernel (const char *hostcpu, const char *kernel) +{ + char **all_files = read_dir (KERNELDIR); + + /* In original: ls -1dvr /boot/vmlinuz-*.$arch* 2>/dev/null | grep -v xen */ + const char *patt; + if (hostcpu[0] == 'i' && hostcpu[2] == '8' && hostcpu[3] == '6' && + hostcpu[4] == '\0') + patt = "vmlinuz-*.i?86*"; + else + patt = xasprintf ("vmlinuz-*.%s*", hostcpu); + + char **candidates; + candidates = filter_fnmatch (all_files, patt, FNM_NOESCAPE); + candidates = filter_notmatching_substring (candidates, "xen"); + + if (candidates[0] == NULL) { + /* In original: ls -1dvr /boot/vmlinuz-* 2>/dev/null | grep -v xen */ + patt = "vmlinuz-*"; + candidates = filter_fnmatch (all_files, patt, FNM_NOESCAPE); + candidates = filter_notmatching_substring (candidates, "xen"); + + if (candidates[0] == NULL) + goto no_kernels; + } + + sort (candidates, reverse_filevercmp); + + /* Choose the first candidate which has a corresponding /lib/modules + * directory. + */ + int i; + for (i = 0; candidates[i] != NULL; ++i) { + if (verbose >= 2) + fprintf (stderr, "candidate kernel: " KERNELDIR "/%s\n", candidates[i]); + + /* Ignore "vmlinuz-" at the beginning of the kernel name. */ + const char *version = &candidates[i][8]; + + /* /lib/modules/<version> */ + char *modpath = xasprintf (MODULESDIR "/%s", version); + + if (verbose >= 2) + fprintf (stderr, "checking modpath %s is a directory\n", modpath); + + if (isdir (modpath)) { + if (verbose >= 2) + fprintf (stderr, "picked %s because modpath %s exists\n", + candidates[i], modpath); + + char *tmp = xasprintf (KERNELDIR "/%s", candidates[i]); + + if (verbose >= 2) + fprintf (stderr, "creating symlink %s -> %s\n", kernel, tmp); + + if (symlink (tmp, kernel) == -1) + error (EXIT_FAILURE, errno, "symlink kernel"); + + free (tmp); + + return modpath; + } + } + + /* Print more diagnostics here than the old script did. */ + no_kernels: + fprintf (stderr, + "febootstrap-supermin-helper: failed to find a suitable kernel.\n" + "I looked for kernels in " KERNELDIR " and modules in " MODULESDIR + ".\n" + "If this is a Xen guest, and you only have Xen domU kernels\n" + "installed, try installing a fullvirt kernel (only for\n" + "febootstrap use, you shouldn't boot the Xen guest with it).\n"); + exit (EXIT_FAILURE); +} diff --git a/helper/main.c b/helper/main.c index 1a43cf2..a30e02c 100644 --- a/helper/main.c +++ b/helper/main.c @@ -16,61 +16,25 @@ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. */ -/* This script builds the supermin appliance on the fly each - * time the appliance runs. - * - * *NOTE*: This program is designed to be very short-lived, and so we - * don't normally bother to free up any memory that we allocate. - * That's not completely true - we free up stuff if it's obvious and - * easy to free up, and ignore the rest. - */ - #include <config.h> #include <stdio.h> #include <stdlib.h> -#include <stdarg.h> -#include <stdint.h> #include <string.h> #include <errno.h> -#include <fcntl.h> -#include <inttypes.h> #include <unistd.h> #include <getopt.h> #include <limits.h> -#include <fnmatch.h> -#include <dirent.h> #include <sys/types.h> #include <sys/time.h> -#include <sys/stat.h> #include <assert.h> #include "error.h" -#include "filevercmp.h" -#include "fts_.h" -#include "full-write.h" -#include "hash.h" -#include "hash-pjw.h" -#include "xalloc.h" -#include "xvasprintf.h" -/* Directory containing candidate kernels. We could make this - * configurable at some point. - */ -#define KERNELDIR "/boot" -#define MODULESDIR "/lib/modules" +#include "helper.h" -/* Buffer size used in copy operations throughout. Large for - * greatest efficiency. - */ -#define BUFFER_SIZE 65536 - -static struct timeval start_t; -static int verbose = 0; - -static void print_timestamped_message (const char *fs, ...); -static const char *create_kernel (const char *hostcpu, const char *kernel); -static void create_appliance (char **inputs, int nr_inputs, const char *whitelist, const char *modpath, const char *initrd); +struct timeval start_t; +int verbose = 0; enum { HELP_OPTION = CHAR_MAX + 1 }; @@ -195,803 +159,3 @@ main (int argc, char *argv[]) exit (EXIT_SUCCESS); } - -/* Compute Y - X and return the result in milliseconds. - * Approximately the same as this code: - * http://www.mpp.mpg.de/~huber/util/timevaldiff.c - */ -static int64_t -timeval_diff (const struct timeval *x, const struct timeval *y) -{ - int64_t msec; - - msec = (y->tv_sec - x->tv_sec) * 1000; - msec += (y->tv_usec - x->tv_usec) / 1000; - return msec; -} - -static void -print_timestamped_message (const char *fs, ...) -{ - struct timeval tv; - gettimeofday (&tv, NULL); - - va_list args; - char *msg; - int err; - - va_start (args, fs); - err = vasprintf (&msg, fs, args); - va_end (args); - - if (err < 0) return; - - fprintf (stderr, "supermin helper [%05" PRIi64 "ms] %s\n", - timeval_diff (&start_t, &tv), msg); - - free (msg); -} - -static char **read_dir (const char *dir); -static char **filter_fnmatch (char **strings, const char *patt, int flags); -static char **filter_notmatching_substring (char **strings, const char *sub); -static void sort (char **strings, int (*compare) (const void *, const void *)); -static int isdir (const char *path); - -static int -reverse_filevercmp (const void *p1, const void *p2) -{ - const char *s1 = * (char * const *) p1; - const char *s2 = * (char * const *) p2; - - /* Note, arguments are reversed to achieve a reverse sort. */ - return filevercmp (s2, s1); -} - -/* Create the kernel. This chooses an appropriate kernel and makes a - * symlink to it. - * - * Look for the most recent kernel named vmlinuz-*.<arch>* which has a - * corresponding directory in /lib/modules/. If the architecture is - * x86, look for any x86 kernel. - * - * RHEL 5 didn't append the arch to the kernel name, so look for - * kernels without arch second. - * - * If no suitable kernel can be found, exit with an error. - * - * This function returns the module path (ie. /lib/modules/<version>). - */ -static const char * -create_kernel (const char *hostcpu, const char *kernel) -{ - char **all_files = read_dir (KERNELDIR); - - /* In original: ls -1dvr /boot/vmlinuz-*.$arch* 2>/dev/null | grep -v xen */ - const char *patt; - if (hostcpu[0] == 'i' && hostcpu[2] == '8' && hostcpu[3] == '6' && - hostcpu[4] == '\0') - patt = "vmlinuz-*.i?86*"; - else - patt = xasprintf ("vmlinuz-*.%s*", hostcpu); - - char **candidates; - candidates = filter_fnmatch (all_files, patt, FNM_NOESCAPE); - candidates = filter_notmatching_substring (candidates, "xen"); - - if (candidates[0] == NULL) { - /* In original: ls -1dvr /boot/vmlinuz-* 2>/dev/null | grep -v xen */ - patt = "vmlinuz-*"; - candidates = filter_fnmatch (all_files, patt, FNM_NOESCAPE); - candidates = filter_notmatching_substring (candidates, "xen"); - - if (candidates[0] == NULL) - goto no_kernels; - } - - sort (candidates, reverse_filevercmp); - - /* Choose the first candidate which has a corresponding /lib/modules - * directory. - */ - int i; - for (i = 0; candidates[i] != NULL; ++i) { - if (verbose >= 2) - fprintf (stderr, "candidate kernel: " KERNELDIR "/%s\n", candidates[i]); - - /* Ignore "vmlinuz-" at the beginning of the kernel name. */ - const char *version = &candidates[i][8]; - - /* /lib/modules/<version> */ - char *modpath = xasprintf (MODULESDIR "/%s", version); - - if (verbose >= 2) - fprintf (stderr, "checking modpath %s is a directory\n", modpath); - - if (isdir (modpath)) { - if (verbose >= 2) - fprintf (stderr, "picked %s because modpath %s exists\n", - candidates[i], modpath); - - char *tmp = xasprintf (KERNELDIR "/%s", candidates[i]); - - if (verbose >= 2) - fprintf (stderr, "creating symlink %s -> %s\n", kernel, tmp); - - if (symlink (tmp, kernel) == -1) - error (EXIT_FAILURE, errno, "symlink kernel"); - - free (tmp); - - return modpath; - } - } - - /* Print more diagnostics here than the old script did. */ - no_kernels: - fprintf (stderr, - "febootstrap-supermin-helper: failed to find a suitable kernel.\n" - "I looked for kernels in " KERNELDIR " and modules in " MODULESDIR - ".\n" - "If this is a Xen guest, and you only have Xen domU kernels\n" - "installed, try installing a fullvirt kernel (only for\n" - "febootstrap use, you shouldn't boot the Xen guest with it).\n"); - exit (EXIT_FAILURE); -} - -static void iterate_inputs (char **inputs, int nr_inputs); -static void iterate_input_directory (const char *dirname, int dirfd); -static void write_kernel_modules (const char *whitelist, const char *modpath); -static void write_hostfiles (const char *hostfiles_file); -static void write_to_fd (const void *buffer, size_t len); -static void write_file_to_fd (const char *filename); -static void write_file_len_to_fd (const char *filename, size_t len); -static void write_padding (size_t len); -static char **load_file (const char *filename); -static void cpio_append_fts_entry (FTSENT *entry); -static void cpio_append_stat (const char *filename, struct stat *); -static void cpio_append (const char *filename); -static void cpio_append_trailer (void); - -static int out_fd = -1; -static off_t out_offset = 0; - -/* Create the appliance. - * - * The initrd consists of these components concatenated together: - * - * (1) The base skeleton appliance that we constructed at build time. - * format = plain cpio - * (2) The host files which match wildcards in *.supermin.hostfiles. - * input format = plain text, output format = plain cpio - * (3) The modules from modpath which are on the module whitelist. - * output format = plain cpio - * - * The original shell script used the external cpio program to create - * parts (2) and (3), but we have decided it's going to be faster if - * we just write out the data outselves. The reasons are that - * external cpio is slow (particularly when used with SELinux because - * it does 512 byte reads), and the format that we're writing is - * narrow and well understood, because we only care that the Linux - * kernel can read it. - * - * This version contains some improvements over the C version written - * for libguestfs, in that we can have multiple base images (or - * hostfiles) or use a directory to store these files. - */ -static void -create_appliance (char **inputs, int nr_inputs, - const char *whitelist, - const char *modpath, - const char *initrd) -{ - out_fd = open (initrd, O_WRONLY | O_CREAT | O_TRUNC | O_NOCTTY, 0644); - if (out_fd == -1) - error (EXIT_FAILURE, errno, "open: %s", initrd); - out_offset = 0; - - iterate_inputs (inputs, nr_inputs); - - /* Kernel modules (3). */ - write_kernel_modules (whitelist, modpath); - - cpio_append_trailer (); - - /* Finish off and close output file. */ - if (close (out_fd) == -1) - error (EXIT_FAILURE, errno, "close: %s", initrd); -} - -/* Iterate over the inputs to find out what they are, visiting - * directories if specified. - */ -static void -iterate_inputs (char **inputs, int nr_inputs) -{ - int i; - for (i = 0; i < nr_inputs; ++i) { - if (verbose) - print_timestamped_message ("visiting %s", inputs[i]); - - int fd = open (inputs[i], O_RDONLY); - if (fd == -1) - error (EXIT_FAILURE, errno, "open: %s", inputs[i]); - - struct stat statbuf; - if (fstat (fd, &statbuf) == -1) - error (EXIT_FAILURE, errno, "fstat: %s", inputs[i]); - - /* Directory? */ - if (S_ISDIR (statbuf.st_mode)) - iterate_input_directory (inputs[i], fd); - else if (S_ISREG (statbuf.st_mode)) { - /* Is it a cpio file? */ - char buf[6]; - if (read (fd, buf, 6) == 6 && memcmp (buf, "070701", 6) == 0) - /* Yes, a cpio file. This is a skeleton appliance, case (1). */ - write_file_to_fd (inputs[i]); - else - /* No, must be hostfiles, case (2). */ - write_hostfiles (inputs[i]); - } - else - error (EXIT_FAILURE, 0, "%s: input is not a regular file or directory", - inputs[i]); - - close (fd); - } -} - -static void -iterate_input_directory (const char *dirname, int dirfd) -{ - char path[PATH_MAX]; - strcpy (path, dirname); - size_t len = strlen (dirname); - path[len++] = '/'; - - char *inputs[] = { path }; - - DIR *dir = fdopendir (dirfd); - if (dir == NULL) - error (EXIT_FAILURE, errno, "fdopendir: %s", dirname); - - struct dirent *d; - while ((errno = 0, d = readdir (dir)) != NULL) { - if (d->d_name[0] == '.') /* ignore ., .. and any hidden files. */ - continue; - - strcpy (&path[len], d->d_name); - iterate_inputs (inputs, 1); - } - - if (errno != 0) - error (EXIT_FAILURE, errno, "readdir: %s", dirname); - - if (closedir (dir) == -1) - error (EXIT_FAILURE, errno, "closedir: %s", dirname); -} - -/* Copy kernel modules. - * - * Find every file under modpath. - * - * Exclude all *.ko files, *except* ones which match names in - * the whitelist (which may contain wildcards). Include all - * other files. - * - * Add chosen files to the output. - * - * whitelist_file may be NULL, to include ALL kernel modules. - */ -static void -write_kernel_modules (const char *whitelist_file, const char *modpath) -{ - char **whitelist = NULL; - if (whitelist_file != NULL) - whitelist = load_file (whitelist_file); - - char *paths[2] = { (char *) modpath, NULL }; - FTS *fts = fts_open (paths, FTS_COMFOLLOW|FTS_PHYSICAL, NULL); - if (fts == NULL) - error (EXIT_FAILURE, errno, "write_kernel_modules: fts_open: %s", modpath); - - for (;;) { - errno = 0; - FTSENT *entry = fts_read (fts); - if (entry == NULL && errno != 0) - error (EXIT_FAILURE, errno, "write_kernel_modules: fts_read: %s", modpath); - if (entry == NULL) - break; - - /* Ignore directories being visited in post-order. */ - if (entry->fts_info & FTS_DP) - continue; - - /* Is it a *.ko file? */ - if (entry->fts_namelen >= 3 && - entry->fts_name[entry->fts_namelen-3] == '.' && - entry->fts_name[entry->fts_namelen-2] == 'k' && - entry->fts_name[entry->fts_namelen-1] == 'o') { - if (whitelist) { - /* Is it a *.ko file which is on the whitelist? */ - size_t j; - for (j = 0; whitelist[j] != NULL; ++j) { - int r; - r = fnmatch (whitelist[j], entry->fts_name, 0); - if (r == 0) { - /* It's on the whitelist, so include it. */ - if (verbose >= 2) - fprintf (stderr, "including kernel module %s (matches whitelist entry %s)\n", - entry->fts_name, whitelist[j]); - cpio_append_fts_entry (entry); - break; - } else if (r != FNM_NOMATCH) - error (EXIT_FAILURE, 0, "internal error: fnmatch ('%s', '%s', %d) returned unexpected non-zero value %d\n", - whitelist[j], entry->fts_name, 0, r); - } /* for (j) */ - } else { /* whitelist == NULL, always include */ - if (verbose >= 2) - fprintf (stderr, "including kernel module %s\n", entry->fts_name); - cpio_append_fts_entry (entry); - } - } else - /* It's some other sort of file, or a directory, always include. */ - cpio_append_fts_entry (entry); - } - - if (fts_close (fts) == -1) - error (EXIT_FAILURE, errno, "write_kernel_modules: fts_close: %s", modpath); -} - -/* Copy the host files. - * - * Read the list of entries in hostfiles (which may contain - * wildcards). Look them up in the filesystem, and add those files - * that exist. Ignore any files that don't exist or are not readable. - */ -static void -write_hostfiles (const char *hostfiles_file) -{ - char **hostfiles = load_file (hostfiles_file); - - /* Hostfiles list can contain "." before each path - ignore it. - * It also contains each directory name before we enter it. But - * we don't read that until we see a wildcard for that directory. - */ - size_t i, j; - for (i = 0; hostfiles[i] != NULL; ++i) { - char *hostfile = hostfiles[i]; - if (hostfile[0] == '.') - hostfile++; - - struct stat statbuf; - - /* Is it a wildcard? */ - if (strchr (hostfile, '*') || strchr (hostfile, '?')) { - char *dirname = xstrdup (hostfile); - char *patt = strrchr (dirname, '/'); - assert (patt); - *patt++ = '\0'; - - char **files = read_dir (dirname); - files = filter_fnmatch (files, patt, FNM_NOESCAPE); - - /* Add matching files. */ - for (j = 0; files[j] != NULL; ++j) { - char *tmp = xasprintf ("%s/%s", dirname, files[j]); - - if (verbose >= 2) - fprintf (stderr, "including host file %s (matches %s)\n", tmp, patt); - - cpio_append (tmp); - - free (tmp); - } - } - /* Else does this file/directory/whatever exist? */ - else if (lstat (hostfile, &statbuf) == 0) { - if (verbose >= 2) - fprintf (stderr, "including host file %s (directly referenced)\n", - hostfile); - - cpio_append_stat (hostfile, &statbuf); - } /* Ignore files that don't exist. */ - } -} - -/*----------*/ -/* Helper functions. */ - -static void -add_string (char ***argv, size_t *n_used, size_t *n_alloc, const char *str) -{ - char *new_str; - - if (*n_used >= *n_alloc) - *argv = x2nrealloc (*argv, n_alloc, sizeof (char *)); - - if (str) - new_str = xstrdup (str); - else - new_str = NULL; - - (*argv)[*n_used] = new_str; - - (*n_used)++; -} - -static size_t -count_strings (char *const *argv) -{ - size_t argc; - - for (argc = 0; argv[argc] != NULL; ++argc) - ; - return argc; -} - -struct dir_cache { - char *path; - char **files; -}; - -static size_t -dir_cache_hash (void const *x, size_t table_size) -{ - struct dir_cache const *p = x; - return hash_pjw (p->path, table_size); -} - -static bool -dir_cache_compare (void const *x, void const *y) -{ - struct dir_cache const *p = x; - struct dir_cache const *q = y; - return strcmp (p->path, q->path) == 0; -} - -/* Read a directory into a list of strings. - * - * Previously looked up directories are cached and returned quickly, - * saving some considerable amount of time compared to reading the - * directory over again. However this means you really must not - * alter the array of strings that are returned. - * - * Returns an empty list if the directory cannot be opened. - */ -static char ** -read_dir (const char *name) -{ - static Hash_table *ht = NULL; - - if (!ht) - ht = hash_initialize (1024, NULL, dir_cache_hash, dir_cache_compare, NULL); - - struct dir_cache key = { .path = (char *) name }; - struct dir_cache *p = hash_lookup (ht, &key); - if (p) - return p->files; - - char **files = NULL; - size_t n_used = 0, n_alloc = 0; - - DIR *dir = opendir (name); - if (!dir) { - /* If it fails to open, that's OK, skip to the end. */ - /*perror (name);*/ - goto done; - } - - for (;;) { - errno = 0; - struct dirent *d = readdir (dir); - if (d == NULL) { - if (errno != 0) - /* But if it fails here, after opening and potentially reading - * part of the directory, that's a proper failure - inform the - * user and exit. - */ - error (EXIT_FAILURE, errno, "%s", name); - break; - } - - add_string (&files, &n_used, &n_alloc, d->d_name); - } - - if (closedir (dir) == -1) - error (EXIT_FAILURE, errno, "closedir: %s", name); - - done: - /* NULL-terminate the array. */ - add_string (&files, &n_used, &n_alloc, NULL); - - /* Add it to the hash for next time. */ - p = xmalloc (sizeof *p); - p->path = (char *) name; - p->files = files; - p = hash_insert (ht, p); - assert (p != NULL); - - return files; -} - -/* Filter a list of strings and return only those matching the wildcard. */ -static char ** -filter_fnmatch (char **strings, const char *patt, int flags) -{ - char **out = NULL; - size_t n_used = 0, n_alloc = 0; - - int i, r; - for (i = 0; strings[i] != NULL; ++i) { - r = fnmatch (patt, strings[i], flags); - if (r == 0) - add_string (&out, &n_used, &n_alloc, strings[i]); - else if (r != FNM_NOMATCH) - error (EXIT_FAILURE, 0, "internal error: fnmatch ('%s', '%s', %d) returned unexpected non-zero value %d\n", - patt, strings[i], flags, r); - } - - add_string (&out, &n_used, &n_alloc, NULL); - return out; -} - -/* Filter a list of strings and return only those which DON'T contain sub. */ -static char ** -filter_notmatching_substring (char **strings, const char *sub) -{ - char **out = NULL; - size_t n_used = 0, n_alloc = 0; - - int i; - for (i = 0; strings[i] != NULL; ++i) { - if (strstr (strings[i], sub) == NULL) - add_string (&out, &n_used, &n_alloc, strings[i]); - } - - add_string (&out, &n_used, &n_alloc, NULL); - return out; -} - -/* Sort a list of strings, in place, with the comparison function supplied. */ -static void -sort (char **strings, int (*compare) (const void *, const void *)) -{ - qsort (strings, count_strings (strings), sizeof (char *), compare); -} - -/* Return true iff path exists and is a directory. This version - * follows symlinks. - */ -static int -isdir (const char *path) -{ - struct stat statbuf; - - if (stat (path, &statbuf) == -1) - return 0; - - return S_ISDIR (statbuf.st_mode); -} - -/* Copy contents of buffer to out_fd and keep out_offset correct. */ -static void -write_to_fd (const void *buffer, size_t len) -{ - if (full_write (out_fd, buffer, len) != len) - error (EXIT_FAILURE, errno, "write"); - out_offset += len; -} - -/* Copy contents of file to out_fd. */ -static void -write_file_to_fd (const char *filename) -{ - char buffer[BUFFER_SIZE]; - int fd2; - ssize_t r; - - if (verbose >= 2) - fprintf (stderr, "write_file_to_fd %s -> %d\n", filename, out_fd); - - fd2 = open (filename, O_RDONLY); - if (fd2 == -1) - error (EXIT_FAILURE, errno, "open: %s", filename); - for (;;) { - r = read (fd2, buffer, sizeof buffer); - if (r == 0) - break; - if (r == -1) { - if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) - continue; - error (EXIT_FAILURE, errno, "read: %s", filename); - } - write_to_fd (buffer, r); - } - - if (close (fd2) == -1) - error (EXIT_FAILURE, errno, "close: %s", filename); -} - -/* Copy file of given length to output, and fail if the file has - * changed size. - */ -static void -write_file_len_to_fd (const char *filename, size_t len) -{ - char buffer[BUFFER_SIZE]; - size_t count = 0; - - if (verbose >= 2) - fprintf (stderr, "write_file_to_fd %s -> %d\n", filename, out_fd); - - int fd2 = open (filename, O_RDONLY); - if (fd2 == -1) - error (EXIT_FAILURE, errno, "open: %s", filename); - for (;;) { - ssize_t r = read (fd2, buffer, sizeof buffer); - if (r == 0) - break; - if (r == -1) { - if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) - continue; - error (EXIT_FAILURE, errno, "read: %s", filename); - } - write_to_fd (buffer, r); - count += r; - if (count > len) - error (EXIT_FAILURE, 0, "write_file_len_to_fd: %s: file has increased in size\n", filename); - } - - if (close (fd2) == -1) - error (EXIT_FAILURE, errno, "close: %s", filename); - - if (count != len) - error (EXIT_FAILURE, 0, "febootstrap-supermin-helper: write_file_len_to_fd: %s: file has changed size\n", filename); -} - -/* Load in a file, returning a list of lines. */ -static char ** -load_file (const char *filename) -{ - char **lines = 0; - size_t n_used = 0, n_alloc = 0; - - FILE *fp; - fp = fopen (filename, "r"); - if (fp == NULL) - error (EXIT_FAILURE, errno, "fopen: %s", filename); - - char line[4096]; - while (fgets (line, sizeof line, fp)) { - size_t len = strlen (line); - if (len > 0 && line[len-1] == '\n') - line[len-1] = '\0'; - add_string (&lines, &n_used, &n_alloc, line); - } - - add_string (&lines, &n_used, &n_alloc, NULL); - return lines; -} - -/* Append the file pointed to by FTSENT to the cpio output. */ -static void -cpio_append_fts_entry (FTSENT *entry) -{ - if (entry->fts_info & FTS_NS || entry->fts_info & FTS_NSOK) - cpio_append (entry->fts_path); - else - cpio_append_stat (entry->fts_path, entry->fts_statp); -} - -/* Append the file named 'filename' to the cpio output. */ -static void -cpio_append (const char *filename) -{ - struct stat statbuf; - - if (lstat (filename, &statbuf) == -1) - error (EXIT_FAILURE, errno, "lstat: %s", filename); - cpio_append_stat (filename, &statbuf); -} - -/* Append the file to the cpio output. */ -#define PADDING(len) ((((len) + 3) & ~3) - (len)) - -#define CPIO_HEADER_LEN (6 + 13*8) - -static void -cpio_append_stat (const char *filename, struct stat *statbuf) -{ - const char *orig_filename = filename; - - if (*filename == '/') - filename++; - if (*filename == '\0') - filename = "."; - - if (verbose >= 2) - fprintf (stderr, "cpio_append_stat %s 0%o -> %d\n", - orig_filename, statbuf->st_mode, out_fd); - - /* Regular files and symlinks are the only ones that have a "body" - * in this cpio entry. - */ - int has_body = S_ISREG (statbuf->st_mode) || S_ISLNK (statbuf->st_mode); - - size_t len = strlen (filename) + 1; - - char header[CPIO_HEADER_LEN + 1]; - snprintf (header, sizeof header, - "070701" /* magic */ - "%08X" /* inode */ - "%08X" /* mode */ - "%08X" "%08X" /* uid, gid */ - "%08X" /* nlink */ - "%08X" /* mtime */ - "%08X" /* file length */ - "%08X" "%08X" /* device holding file major, minor */ - "%08X" "%08X" /* for specials, device major, minor */ - "%08X" /* name length (including \0 byte) */ - "%08X", /* checksum (not used by the kernel) */ - (unsigned) statbuf->st_ino, statbuf->st_mode, - statbuf->st_uid, statbuf->st_gid, - (unsigned) statbuf->st_nlink, (unsigned) statbuf->st_mtime, - has_body ? (unsigned) statbuf->st_size : 0, - major (statbuf->st_dev), minor (statbuf->st_dev), - major (statbuf->st_rdev), minor (statbuf->st_rdev), - (unsigned) len, 0); - - /* Write the header. */ - write_to_fd (header, CPIO_HEADER_LEN); - - /* Follow with the filename, and pad it. */ - write_to_fd (filename, len); - size_t padding_len = PADDING (CPIO_HEADER_LEN + len); - write_padding (padding_len); - - /* Follow with the file or symlink content, and pad it. */ - if (has_body) { - if (S_ISREG (statbuf->st_mode)) - write_file_len_to_fd (orig_filename, statbuf->st_size); - else if (S_ISLNK (statbuf->st_mode)) { - char tmp[PATH_MAX]; - if (readlink (orig_filename, tmp, sizeof tmp) == -1) - error (EXIT_FAILURE, errno, "readlink: %s", orig_filename); - write_to_fd (tmp, statbuf->st_size); - } - - padding_len = PADDING (statbuf->st_size); - write_padding (padding_len); - } -} - -/* CPIO voodoo. */ -static void -cpio_append_trailer (void) -{ - struct stat statbuf; - memset (&statbuf, 0, sizeof statbuf); - statbuf.st_nlink = 1; - cpio_append_stat ("TRAILER!!!", &statbuf); - - /* CPIO seems to pad up to the next block boundary, ie. up to - * the next 512 bytes. - */ - write_padding (((out_offset + 511) & ~511) - out_offset); - assert ((out_offset & 511) == 0); -} - -/* Write 'len' bytes of zeroes out. */ -static void -write_padding (size_t len) -{ - static const char buffer[512] = { 0 }; - - while (len > 0) { - size_t n = len < sizeof buffer ? len : sizeof buffer; - write_to_fd (buffer, n); - len -= n; - } -} diff --git a/helper/utils.c b/helper/utils.c new file mode 100644 index 0000000..d956a80 --- /dev/null +++ b/helper/utils.c @@ -0,0 +1,281 @@ +/* febootstrap-supermin-helper reimplementation in C. + * Copyright (C) 2009-2010 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <stdint.h> +#include <string.h> +#include <dirent.h> +#include <errno.h> +#include <fnmatch.h> +#include <inttypes.h> +#include <sys/time.h> +#include <sys/stat.h> +#include <assert.h> + +#include "error.h" +#include "filevercmp.h" +#include "hash.h" +#include "hash-pjw.h" +#include "xalloc.h" + +#include "helper.h" + +/* Compute Y - X and return the result in milliseconds. + * Approximately the same as this code: + * http://www.mpp.mpg.de/~huber/util/timevaldiff.c + */ +extern int64_t +timeval_diff (const struct timeval *x, const struct timeval *y) +{ + int64_t msec; + + msec = (y->tv_sec - x->tv_sec) * 1000; + msec += (y->tv_usec - x->tv_usec) / 1000; + return msec; +} + +extern void +print_timestamped_message (const char *fs, ...) +{ + struct timeval tv; + gettimeofday (&tv, NULL); + + va_list args; + char *msg; + int err; + + va_start (args, fs); + err = vasprintf (&msg, fs, args); + va_end (args); + + if (err < 0) return; + + fprintf (stderr, "supermin helper [%05" PRIi64 "ms] %s\n", + timeval_diff (&start_t, &tv), msg); + + free (msg); +} + +extern int +reverse_filevercmp (const void *p1, const void *p2) +{ + const char *s1 = * (char * const *) p1; + const char *s2 = * (char * const *) p2; + + /* Note, arguments are reversed to achieve a reverse sort. */ + return filevercmp (s2, s1); +} + +extern void +add_string (char ***argv, size_t *n_used, size_t *n_alloc, const char *str) +{ + char *new_str; + + if (*n_used >= *n_alloc) + *argv = x2nrealloc (*argv, n_alloc, sizeof (char *)); + + if (str) + new_str = xstrdup (str); + else + new_str = NULL; + + (*argv)[*n_used] = new_str; + + (*n_used)++; +} + +extern size_t +count_strings (char *const *argv) +{ + size_t argc; + + for (argc = 0; argv[argc] != NULL; ++argc) + ; + return argc; +} + +struct dir_cache { + char *path; + char **files; +}; + +static size_t +dir_cache_hash (void const *x, size_t table_size) +{ + struct dir_cache const *p = x; + return hash_pjw (p->path, table_size); +} + +static bool +dir_cache_compare (void const *x, void const *y) +{ + struct dir_cache const *p = x; + struct dir_cache const *q = y; + return strcmp (p->path, q->path) == 0; +} + +/* Read a directory into a list of strings. + * + * Previously looked up directories are cached and returned quickly, + * saving some considerable amount of time compared to reading the + * directory over again. However this means you really must not + * alter the array of strings that are returned. + * + * Returns an empty list if the directory cannot be opened. + */ +extern char ** +read_dir (const char *name) +{ + static Hash_table *ht = NULL; + + if (!ht) + ht = hash_initialize (1024, NULL, dir_cache_hash, dir_cache_compare, NULL); + + struct dir_cache key = { .path = (char *) name }; + struct dir_cache *p = hash_lookup (ht, &key); + if (p) + return p->files; + + char **files = NULL; + size_t n_used = 0, n_alloc = 0; + + DIR *dir = opendir (name); + if (!dir) { + /* If it fails to open, that's OK, skip to the end. */ + /*perror (name);*/ + goto done; + } + + for (;;) { + errno = 0; + struct dirent *d = readdir (dir); + if (d == NULL) { + if (errno != 0) + /* But if it fails here, after opening and potentially reading + * part of the directory, that's a proper failure - inform the + * user and exit. + */ + error (EXIT_FAILURE, errno, "%s", name); + break; + } + + add_string (&files, &n_used, &n_alloc, d->d_name); + } + + if (closedir (dir) == -1) + error (EXIT_FAILURE, errno, "closedir: %s", name); + + done: + /* NULL-terminate the array. */ + add_string (&files, &n_used, &n_alloc, NULL); + + /* Add it to the hash for next time. */ + p = xmalloc (sizeof *p); + p->path = (char *) name; + p->files = files; + p = hash_insert (ht, p); + assert (p != NULL); + + return files; +} + +/* Filter a list of strings and return only those matching the wildcard. */ +extern char ** +filter_fnmatch (char **strings, const char *patt, int flags) +{ + char **out = NULL; + size_t n_used = 0, n_alloc = 0; + + int i, r; + for (i = 0; strings[i] != NULL; ++i) { + r = fnmatch (patt, strings[i], flags); + if (r == 0) + add_string (&out, &n_used, &n_alloc, strings[i]); + else if (r != FNM_NOMATCH) + error (EXIT_FAILURE, 0, "internal error: fnmatch ('%s', '%s', %d) returned unexpected non-zero value %d\n", + patt, strings[i], flags, r); + } + + add_string (&out, &n_used, &n_alloc, NULL); + return out; +} + +/* Filter a list of strings and return only those which DON'T contain sub. */ +extern char ** +filter_notmatching_substring (char **strings, const char *sub) +{ + char **out = NULL; + size_t n_used = 0, n_alloc = 0; + + int i; + for (i = 0; strings[i] != NULL; ++i) { + if (strstr (strings[i], sub) == NULL) + add_string (&out, &n_used, &n_alloc, strings[i]); + } + + add_string (&out, &n_used, &n_alloc, NULL); + return out; +} + +/* Sort a list of strings, in place, with the comparison function supplied. */ +extern void +sort (char **strings, int (*compare) (const void *, const void *)) +{ + qsort (strings, count_strings (strings), sizeof (char *), compare); +} + +/* Return true iff path exists and is a directory. This version + * follows symlinks. + */ +extern int +isdir (const char *path) +{ + struct stat statbuf; + + if (stat (path, &statbuf) == -1) + return 0; + + return S_ISDIR (statbuf.st_mode); +} + +/* Load in a file, returning a list of lines. */ +extern char ** +load_file (const char *filename) +{ + char **lines = 0; + size_t n_used = 0, n_alloc = 0; + + FILE *fp; + fp = fopen (filename, "r"); + if (fp == NULL) + error (EXIT_FAILURE, errno, "fopen: %s", filename); + + char line[4096]; + while (fgets (line, sizeof line, fp)) { + size_t len = strlen (line); + if (len > 0 && line[len-1] == '\n') + line[len-1] = '\0'; + add_string (&lines, &n_used, &n_alloc, line); + } + + add_string (&lines, &n_used, &n_alloc, NULL); + return lines; +} -- 1.7.1
Richard W.M. Jones
2010-Aug-20 10:39 UTC
[Libguestfs] [PATCH febootstrap 3/8] Change initrd -> appliance.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw -------------- next part -------------->From 861c38be3e8c135d93a072911b65ce88ac58fd9f Mon Sep 17 00:00:00 2001From: Richard Jones <rjones at redhat.com> Date: Thu, 19 Aug 2010 13:55:36 +0100 Subject: [PATCH 3/8] Change initrd -> appliance. We are going to be writing a general appliance, not necessarily an initrd. In the ext2 appliance case we will _also_ write an initrd, so we shouldn't confuse the two things. --- helper/appliance.c | 8 ++++---- helper/helper.h | 2 +- helper/main.c | 10 +++++----- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/helper/appliance.c b/helper/appliance.c index c7200bd..c6defc0 100644 --- a/helper/appliance.c +++ b/helper/appliance.c @@ -86,11 +86,11 @@ void create_appliance (char **inputs, int nr_inputs, const char *whitelist, const char *modpath, - const char *initrd) + const char *appliance) { - out_fd = open (initrd, O_WRONLY | O_CREAT | O_TRUNC | O_NOCTTY, 0644); + out_fd = open (appliance, O_WRONLY | O_CREAT | O_TRUNC | O_NOCTTY, 0644); if (out_fd == -1) - error (EXIT_FAILURE, errno, "open: %s", initrd); + error (EXIT_FAILURE, errno, "open: %s", appliance); out_offset = 0; iterate_inputs (inputs, nr_inputs); @@ -102,7 +102,7 @@ create_appliance (char **inputs, int nr_inputs, /* Finish off and close output file. */ if (close (out_fd) == -1) - error (EXIT_FAILURE, errno, "close: %s", initrd); + error (EXIT_FAILURE, errno, "close: %s", appliance); } /* Iterate over the inputs to find out what they are, visiting diff --git a/helper/helper.h b/helper/helper.h index fdf446e..e9618ba 100644 --- a/helper/helper.h +++ b/helper/helper.h @@ -24,7 +24,7 @@ extern struct timeval start_t; extern int verbose; /* appliance.c */ -extern void create_appliance (char **inputs, int nr_inputs, const char *whitelist, const char *modpath, const char *initrd); +extern void create_appliance (char **inputs, int nr_inputs, const char *whitelist, const char *modpath, const char *appliance); /* kernel.c */ extern const char *create_kernel (const char *hostcpu, const char *kernel); diff --git a/helper/main.c b/helper/main.c index a30e02c..d350b4e 100644 --- a/helper/main.c +++ b/helper/main.c @@ -126,15 +126,15 @@ main (int argc, char *argv[]) /* Output files. */ const char *kernel = argv[argc-2]; - const char *initrd = argv[argc-1]; + const char *appliance = argv[argc-1]; if (verbose) { print_timestamped_message ("whitelist = %s, " "host_cpu = %s, " "kernel = %s, " - "initrd = %s", + "appliance = %s", whitelist ? : "(not specified)", - hostcpu, kernel, initrd); + hostcpu, kernel, appliance); int i; for (i = 0; i < nr_inputs; ++i) print_timestamped_message ("inputs[%d] = %s", i, inputs[i]); @@ -142,7 +142,7 @@ main (int argc, char *argv[]) /* Remove the output files if they exist. */ unlink (kernel); - unlink (initrd); + unlink (appliance); /* Create kernel output file. */ const char *modpath; @@ -152,7 +152,7 @@ main (int argc, char *argv[]) print_timestamped_message ("finished creating kernel"); /* Create the appliance. */ - create_appliance (inputs, nr_inputs, whitelist, modpath, initrd); + create_appliance (inputs, nr_inputs, whitelist, modpath, appliance); if (verbose) print_timestamped_message ("finished creating appliance"); -- 1.7.1
Richard W.M. Jones
2010-Aug-20 10:40 UTC
[Libguestfs] [PATCH febootstrap 4/8] Refactor cpio code into separate file.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://et.redhat.com/~rjones/libguestfs/ See what it can do: http://et.redhat.com/~rjones/libguestfs/recipes.html -------------- next part -------------->From a241e323e209b0da30396408574c8e3d89b323e0 Mon Sep 17 00:00:00 2001From: Richard Jones <rjones at redhat.com> Date: Thu, 19 Aug 2010 14:23:11 +0100 Subject: [PATCH 4/8] Refactor cpio code into separate file. We create a 'writer' abstraction and (currently) a single implementation of this which can write cpio files. All cpio-related code is moved out of 'appliance.c' into 'cpio.c'. 'appliance.c' becomes a generic appliance builder. This commit should not change the semantics of the program. --- helper/Makefile.am | 1 + helper/appliance.c | 274 +++++---------------------------------------------- helper/cpio.c | 278 ++++++++++++++++++++++++++++++++++++++++++++++++++++ helper/helper.h | 26 +++++- helper/main.c | 3 +- 5 files changed, 333 insertions(+), 249 deletions(-) create mode 100644 helper/cpio.c diff --git a/helper/Makefile.am b/helper/Makefile.am index fb21286..985d4b2 100644 --- a/helper/Makefile.am +++ b/helper/Makefile.am @@ -24,6 +24,7 @@ DISTCLEANFILES = $(bin_SCRIPTS) febootstrap_supermin_helper_SOURCES = \ helper.h \ appliance.c \ + cpio.c \ kernel.c \ main.c \ utils.c diff --git a/helper/appliance.c b/helper/appliance.c index c6defc0..b7a46c0 100644 --- a/helper/appliance.c +++ b/helper/appliance.c @@ -22,7 +22,6 @@ #include <stdlib.h> #include <string.h> #include <unistd.h> -#include <limits.h> #include <fcntl.h> #include <errno.h> #include <dirent.h> @@ -32,32 +31,15 @@ #include "error.h" #include "fts_.h" -#include "full-write.h" #include "xalloc.h" #include "xvasprintf.h" #include "helper.h" -/* Buffer size used in copy operations throughout. Large for - * greatest efficiency. - */ -#define BUFFER_SIZE 65536 - -static void iterate_inputs (char **inputs, int nr_inputs); -static void iterate_input_directory (const char *dirname, int dirfd); -static void write_kernel_modules (const char *whitelist, const char *modpath); -static void write_hostfiles (const char *hostfiles_file); -static void write_to_fd (const void *buffer, size_t len); -static void write_file_to_fd (const char *filename); -static void write_file_len_to_fd (const char *filename, size_t len); -static void write_padding (size_t len); -static void cpio_append_fts_entry (FTSENT *entry); -static void cpio_append_stat (const char *filename, struct stat *); -static void cpio_append (const char *filename); -static void cpio_append_trailer (void); - -static int out_fd = -1; -static off_t out_offset = 0; +static void iterate_inputs (char **inputs, int nr_inputs, struct writer *); +static void iterate_input_directory (const char *dirname, int dirfd, struct writer *); +static void add_kernel_modules (const char *whitelist, const char *modpath, struct writer *); +static void add_hostfiles (const char *hostfiles_file, struct writer *); /* Create the appliance. * @@ -86,30 +68,24 @@ void create_appliance (char **inputs, int nr_inputs, const char *whitelist, const char *modpath, - const char *appliance) + const char *appliance, + struct writer *writer) { - out_fd = open (appliance, O_WRONLY | O_CREAT | O_TRUNC | O_NOCTTY, 0644); - if (out_fd == -1) - error (EXIT_FAILURE, errno, "open: %s", appliance); - out_offset = 0; + writer->wr_start (appliance); - iterate_inputs (inputs, nr_inputs); + iterate_inputs (inputs, nr_inputs, writer); /* Kernel modules (3). */ - write_kernel_modules (whitelist, modpath); - - cpio_append_trailer (); + add_kernel_modules (whitelist, modpath, writer); - /* Finish off and close output file. */ - if (close (out_fd) == -1) - error (EXIT_FAILURE, errno, "close: %s", appliance); + writer->wr_end (); } /* Iterate over the inputs to find out what they are, visiting * directories if specified. */ static void -iterate_inputs (char **inputs, int nr_inputs) +iterate_inputs (char **inputs, int nr_inputs, struct writer *writer) { int i; for (i = 0; i < nr_inputs; ++i) { @@ -126,16 +102,16 @@ iterate_inputs (char **inputs, int nr_inputs) /* Directory? */ if (S_ISDIR (statbuf.st_mode)) - iterate_input_directory (inputs[i], fd); + iterate_input_directory (inputs[i], fd, writer); else if (S_ISREG (statbuf.st_mode)) { /* Is it a cpio file? */ char buf[6]; if (read (fd, buf, 6) == 6 && memcmp (buf, "070701", 6) == 0) /* Yes, a cpio file. This is a skeleton appliance, case (1). */ - write_file_to_fd (inputs[i]); + writer->wr_cpio_file (inputs[i]); else /* No, must be hostfiles, case (2). */ - write_hostfiles (inputs[i]); + add_hostfiles (inputs[i], writer); } else error (EXIT_FAILURE, 0, "%s: input is not a regular file or directory", @@ -146,7 +122,7 @@ iterate_inputs (char **inputs, int nr_inputs) } static void -iterate_input_directory (const char *dirname, int dirfd) +iterate_input_directory (const char *dirname, int dirfd, struct writer *writer) { char path[PATH_MAX]; strcpy (path, dirname); @@ -165,7 +141,7 @@ iterate_input_directory (const char *dirname, int dirfd) continue; strcpy (&path[len], d->d_name); - iterate_inputs (inputs, 1); + iterate_inputs (inputs, 1, writer); } if (errno != 0) @@ -188,7 +164,8 @@ iterate_input_directory (const char *dirname, int dirfd) * whitelist_file may be NULL, to include ALL kernel modules. */ static void -write_kernel_modules (const char *whitelist_file, const char *modpath) +add_kernel_modules (const char *whitelist_file, const char *modpath, + struct writer *writer) { char **whitelist = NULL; if (whitelist_file != NULL) @@ -197,13 +174,13 @@ write_kernel_modules (const char *whitelist_file, const char *modpath) char *paths[2] = { (char *) modpath, NULL }; FTS *fts = fts_open (paths, FTS_COMFOLLOW|FTS_PHYSICAL, NULL); if (fts == NULL) - error (EXIT_FAILURE, errno, "write_kernel_modules: fts_open: %s", modpath); + error (EXIT_FAILURE, errno, "add_kernel_modules: fts_open: %s", modpath); for (;;) { errno = 0; FTSENT *entry = fts_read (fts); if (entry == NULL && errno != 0) - error (EXIT_FAILURE, errno, "write_kernel_modules: fts_read: %s", modpath); + error (EXIT_FAILURE, errno, "add_kernel_modules: fts_read: %s", modpath); if (entry == NULL) break; @@ -227,7 +204,7 @@ write_kernel_modules (const char *whitelist_file, const char *modpath) if (verbose >= 2) fprintf (stderr, "including kernel module %s (matches whitelist entry %s)\n", entry->fts_name, whitelist[j]); - cpio_append_fts_entry (entry); + writer->wr_fts_entry (entry); break; } else if (r != FNM_NOMATCH) error (EXIT_FAILURE, 0, "internal error: fnmatch ('%s', '%s', %d) returned unexpected non-zero value %d\n", @@ -236,15 +213,15 @@ write_kernel_modules (const char *whitelist_file, const char *modpath) } else { /* whitelist == NULL, always include */ if (verbose >= 2) fprintf (stderr, "including kernel module %s\n", entry->fts_name); - cpio_append_fts_entry (entry); + writer->wr_fts_entry (entry); } } else /* It's some other sort of file, or a directory, always include. */ - cpio_append_fts_entry (entry); + writer->wr_fts_entry (entry); } if (fts_close (fts) == -1) - error (EXIT_FAILURE, errno, "write_kernel_modules: fts_close: %s", modpath); + error (EXIT_FAILURE, errno, "add_kernel_modules: fts_close: %s", modpath); } /* Copy the host files. @@ -254,7 +231,7 @@ write_kernel_modules (const char *whitelist_file, const char *modpath) * that exist. Ignore any files that don't exist or are not readable. */ static void -write_hostfiles (const char *hostfiles_file) +add_hostfiles (const char *hostfiles_file, struct writer *writer) { char **hostfiles = load_file (hostfiles_file); @@ -287,7 +264,7 @@ write_hostfiles (const char *hostfiles_file) if (verbose >= 2) fprintf (stderr, "including host file %s (matches %s)\n", tmp, patt); - cpio_append (tmp); + writer->wr_file (tmp); free (tmp); } @@ -298,204 +275,7 @@ write_hostfiles (const char *hostfiles_file) fprintf (stderr, "including host file %s (directly referenced)\n", hostfile); - cpio_append_stat (hostfile, &statbuf); + writer->wr_file_stat (hostfile, &statbuf); } /* Ignore files that don't exist. */ } } - -/* Copy contents of buffer to out_fd and keep out_offset correct. */ -static void -write_to_fd (const void *buffer, size_t len) -{ - if (full_write (out_fd, buffer, len) != len) - error (EXIT_FAILURE, errno, "write"); - out_offset += len; -} - -/* Copy contents of file to out_fd. */ -static void -write_file_to_fd (const char *filename) -{ - char buffer[BUFFER_SIZE]; - int fd2; - ssize_t r; - - if (verbose >= 2) - fprintf (stderr, "write_file_to_fd %s -> %d\n", filename, out_fd); - - fd2 = open (filename, O_RDONLY); - if (fd2 == -1) - error (EXIT_FAILURE, errno, "open: %s", filename); - for (;;) { - r = read (fd2, buffer, sizeof buffer); - if (r == 0) - break; - if (r == -1) { - if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) - continue; - error (EXIT_FAILURE, errno, "read: %s", filename); - } - write_to_fd (buffer, r); - } - - if (close (fd2) == -1) - error (EXIT_FAILURE, errno, "close: %s", filename); -} - -/* Copy file of given length to output, and fail if the file has - * changed size. - */ -static void -write_file_len_to_fd (const char *filename, size_t len) -{ - char buffer[BUFFER_SIZE]; - size_t count = 0; - - if (verbose >= 2) - fprintf (stderr, "write_file_to_fd %s -> %d\n", filename, out_fd); - - int fd2 = open (filename, O_RDONLY); - if (fd2 == -1) - error (EXIT_FAILURE, errno, "open: %s", filename); - for (;;) { - ssize_t r = read (fd2, buffer, sizeof buffer); - if (r == 0) - break; - if (r == -1) { - if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) - continue; - error (EXIT_FAILURE, errno, "read: %s", filename); - } - write_to_fd (buffer, r); - count += r; - if (count > len) - error (EXIT_FAILURE, 0, "write_file_len_to_fd: %s: file has increased in size\n", filename); - } - - if (close (fd2) == -1) - error (EXIT_FAILURE, errno, "close: %s", filename); - - if (count != len) - error (EXIT_FAILURE, 0, "febootstrap-supermin-helper: write_file_len_to_fd: %s: file has changed size\n", filename); -} - -/* Append the file pointed to by FTSENT to the cpio output. */ -static void -cpio_append_fts_entry (FTSENT *entry) -{ - if (entry->fts_info & FTS_NS || entry->fts_info & FTS_NSOK) - cpio_append (entry->fts_path); - else - cpio_append_stat (entry->fts_path, entry->fts_statp); -} - -/* Append the file named 'filename' to the cpio output. */ -static void -cpio_append (const char *filename) -{ - struct stat statbuf; - - if (lstat (filename, &statbuf) == -1) - error (EXIT_FAILURE, errno, "lstat: %s", filename); - cpio_append_stat (filename, &statbuf); -} - -/* Append the file to the cpio output. */ -#define PADDING(len) ((((len) + 3) & ~3) - (len)) - -#define CPIO_HEADER_LEN (6 + 13*8) - -static void -cpio_append_stat (const char *filename, struct stat *statbuf) -{ - const char *orig_filename = filename; - - if (*filename == '/') - filename++; - if (*filename == '\0') - filename = "."; - - if (verbose >= 2) - fprintf (stderr, "cpio_append_stat %s 0%o -> %d\n", - orig_filename, statbuf->st_mode, out_fd); - - /* Regular files and symlinks are the only ones that have a "body" - * in this cpio entry. - */ - int has_body = S_ISREG (statbuf->st_mode) || S_ISLNK (statbuf->st_mode); - - size_t len = strlen (filename) + 1; - - char header[CPIO_HEADER_LEN + 1]; - snprintf (header, sizeof header, - "070701" /* magic */ - "%08X" /* inode */ - "%08X" /* mode */ - "%08X" "%08X" /* uid, gid */ - "%08X" /* nlink */ - "%08X" /* mtime */ - "%08X" /* file length */ - "%08X" "%08X" /* device holding file major, minor */ - "%08X" "%08X" /* for specials, device major, minor */ - "%08X" /* name length (including \0 byte) */ - "%08X", /* checksum (not used by the kernel) */ - (unsigned) statbuf->st_ino, statbuf->st_mode, - statbuf->st_uid, statbuf->st_gid, - (unsigned) statbuf->st_nlink, (unsigned) statbuf->st_mtime, - has_body ? (unsigned) statbuf->st_size : 0, - major (statbuf->st_dev), minor (statbuf->st_dev), - major (statbuf->st_rdev), minor (statbuf->st_rdev), - (unsigned) len, 0); - - /* Write the header. */ - write_to_fd (header, CPIO_HEADER_LEN); - - /* Follow with the filename, and pad it. */ - write_to_fd (filename, len); - size_t padding_len = PADDING (CPIO_HEADER_LEN + len); - write_padding (padding_len); - - /* Follow with the file or symlink content, and pad it. */ - if (has_body) { - if (S_ISREG (statbuf->st_mode)) - write_file_len_to_fd (orig_filename, statbuf->st_size); - else if (S_ISLNK (statbuf->st_mode)) { - char tmp[PATH_MAX]; - if (readlink (orig_filename, tmp, sizeof tmp) == -1) - error (EXIT_FAILURE, errno, "readlink: %s", orig_filename); - write_to_fd (tmp, statbuf->st_size); - } - - padding_len = PADDING (statbuf->st_size); - write_padding (padding_len); - } -} - -/* CPIO voodoo. */ -static void -cpio_append_trailer (void) -{ - struct stat statbuf; - memset (&statbuf, 0, sizeof statbuf); - statbuf.st_nlink = 1; - cpio_append_stat ("TRAILER!!!", &statbuf); - - /* CPIO seems to pad up to the next block boundary, ie. up to - * the next 512 bytes. - */ - write_padding (((out_offset + 511) & ~511) - out_offset); - assert ((out_offset & 511) == 0); -} - -/* Write 'len' bytes of zeroes out. */ -static void -write_padding (size_t len) -{ - static const char buffer[512] = { 0 }; - - while (len > 0) { - size_t n = len < sizeof buffer ? len : sizeof buffer; - write_to_fd (buffer, n); - len -= n; - } -} diff --git a/helper/cpio.c b/helper/cpio.c new file mode 100644 index 0000000..f9ffc3f --- /dev/null +++ b/helper/cpio.c @@ -0,0 +1,278 @@ +/* febootstrap-supermin-helper reimplementation in C. + * Copyright (C) 2009-2010 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <unistd.h> +#include <limits.h> +#include <fcntl.h> +#include <errno.h> +#include <sys/stat.h> +#include <assert.h> + +#include "error.h" +#include "fts_.h" +#include "full-write.h" +#include "xalloc.h" +#include "xvasprintf.h" + +#include "helper.h" + +/* Buffer size used in copy operations throughout. Large for + * greatest efficiency. + */ +#define BUFFER_SIZE 65536 + +static int out_fd = -1; +static off_t out_offset = 0; + +static void write_file_to_fd (const char *filename); +static void write_file_len_to_fd (const char *filename, size_t len); +static void write_padding (size_t len); +static void cpio_append_fts_entry (FTSENT *entry); +static void cpio_append_stat (const char *filename, const struct stat *); +static void cpio_append (const char *filename); +static void cpio_append_trailer (void); + +/* Copy contents of buffer to out_fd and keep out_offset correct. */ +static void +write_to_fd (const void *buffer, size_t len) +{ + if (full_write (out_fd, buffer, len) != len) + error (EXIT_FAILURE, errno, "write"); + out_offset += len; +} + +/* Copy contents of file to out_fd. */ +static void +write_file_to_fd (const char *filename) +{ + char buffer[BUFFER_SIZE]; + int fd2; + ssize_t r; + + if (verbose >= 2) + fprintf (stderr, "write_file_to_fd %s -> %d\n", filename, out_fd); + + fd2 = open (filename, O_RDONLY); + if (fd2 == -1) + error (EXIT_FAILURE, errno, "open: %s", filename); + for (;;) { + r = read (fd2, buffer, sizeof buffer); + if (r == 0) + break; + if (r == -1) { + if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) + continue; + error (EXIT_FAILURE, errno, "read: %s", filename); + } + write_to_fd (buffer, r); + } + + if (close (fd2) == -1) + error (EXIT_FAILURE, errno, "close: %s", filename); +} + +/* Copy file of given length to output, and fail if the file has + * changed size. + */ +static void +write_file_len_to_fd (const char *filename, size_t len) +{ + char buffer[BUFFER_SIZE]; + size_t count = 0; + + if (verbose >= 2) + fprintf (stderr, "write_file_to_fd %s -> %d\n", filename, out_fd); + + int fd2 = open (filename, O_RDONLY); + if (fd2 == -1) + error (EXIT_FAILURE, errno, "open: %s", filename); + for (;;) { + ssize_t r = read (fd2, buffer, sizeof buffer); + if (r == 0) + break; + if (r == -1) { + if (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR) + continue; + error (EXIT_FAILURE, errno, "read: %s", filename); + } + write_to_fd (buffer, r); + count += r; + if (count > len) + error (EXIT_FAILURE, 0, "write_file_len_to_fd: %s: file has increased in size\n", filename); + } + + if (close (fd2) == -1) + error (EXIT_FAILURE, errno, "close: %s", filename); + + if (count != len) + error (EXIT_FAILURE, 0, "febootstrap-supermin-helper: write_file_len_to_fd: %s: file has changed size\n", filename); +} + +/* Append the file pointed to by FTSENT to the cpio output. */ +static void +cpio_append_fts_entry (FTSENT *entry) +{ + if (entry->fts_info & FTS_NS || entry->fts_info & FTS_NSOK) + cpio_append (entry->fts_path); + else + cpio_append_stat (entry->fts_path, entry->fts_statp); +} + +/* Append the file named 'filename' to the cpio output. */ +static void +cpio_append (const char *filename) +{ + struct stat statbuf; + + if (lstat (filename, &statbuf) == -1) + error (EXIT_FAILURE, errno, "lstat: %s", filename); + cpio_append_stat (filename, &statbuf); +} + +/* Append the file to the cpio output. */ +#define PADDING(len) ((((len) + 3) & ~3) - (len)) + +#define CPIO_HEADER_LEN (6 + 13*8) + +static void +cpio_append_stat (const char *filename, const struct stat *statbuf) +{ + const char *orig_filename = filename; + + if (*filename == '/') + filename++; + if (*filename == '\0') + filename = "."; + + if (verbose >= 2) + fprintf (stderr, "cpio_append_stat %s 0%o -> %d\n", + orig_filename, statbuf->st_mode, out_fd); + + /* Regular files and symlinks are the only ones that have a "body" + * in this cpio entry. + */ + int has_body = S_ISREG (statbuf->st_mode) || S_ISLNK (statbuf->st_mode); + + size_t len = strlen (filename) + 1; + + char header[CPIO_HEADER_LEN + 1]; + snprintf (header, sizeof header, + "070701" /* magic */ + "%08X" /* inode */ + "%08X" /* mode */ + "%08X" "%08X" /* uid, gid */ + "%08X" /* nlink */ + "%08X" /* mtime */ + "%08X" /* file length */ + "%08X" "%08X" /* device holding file major, minor */ + "%08X" "%08X" /* for specials, device major, minor */ + "%08X" /* name length (including \0 byte) */ + "%08X", /* checksum (not used by the kernel) */ + (unsigned) statbuf->st_ino, statbuf->st_mode, + statbuf->st_uid, statbuf->st_gid, + (unsigned) statbuf->st_nlink, (unsigned) statbuf->st_mtime, + has_body ? (unsigned) statbuf->st_size : 0, + major (statbuf->st_dev), minor (statbuf->st_dev), + major (statbuf->st_rdev), minor (statbuf->st_rdev), + (unsigned) len, 0); + + /* Write the header. */ + write_to_fd (header, CPIO_HEADER_LEN); + + /* Follow with the filename, and pad it. */ + write_to_fd (filename, len); + size_t padding_len = PADDING (CPIO_HEADER_LEN + len); + write_padding (padding_len); + + /* Follow with the file or symlink content, and pad it. */ + if (has_body) { + if (S_ISREG (statbuf->st_mode)) + write_file_len_to_fd (orig_filename, statbuf->st_size); + else if (S_ISLNK (statbuf->st_mode)) { + char tmp[PATH_MAX]; + if (readlink (orig_filename, tmp, sizeof tmp) == -1) + error (EXIT_FAILURE, errno, "readlink: %s", orig_filename); + write_to_fd (tmp, statbuf->st_size); + } + + padding_len = PADDING (statbuf->st_size); + write_padding (padding_len); + } +} + +/* CPIO voodoo. */ +static void +cpio_append_trailer (void) +{ + struct stat statbuf; + memset (&statbuf, 0, sizeof statbuf); + statbuf.st_nlink = 1; + cpio_append_stat ("TRAILER!!!", &statbuf); + + /* CPIO seems to pad up to the next block boundary, ie. up to + * the next 512 bytes. + */ + write_padding (((out_offset + 511) & ~511) - out_offset); + assert ((out_offset & 511) == 0); +} + +/* Write 'len' bytes of zeroes out. */ +static void +write_padding (size_t len) +{ + static const char buffer[512] = { 0 }; + + while (len > 0) { + size_t n = len < sizeof buffer ? len : sizeof buffer; + write_to_fd (buffer, n); + len -= n; + } +} + +static void +cpio_start (const char *appliance) +{ + out_fd = open (appliance, O_WRONLY | O_CREAT | O_TRUNC | O_NOCTTY, 0644); + if (out_fd == -1) + error (EXIT_FAILURE, errno, "open: %s", appliance); + out_offset = 0; +} + +static void +cpio_end (void) +{ + cpio_append_trailer (); + + /* Finish off and close output file. */ + if (close (out_fd) == -1) + error (EXIT_FAILURE, errno, "close"); +} + +struct writer cpio_writer = { + .wr_start = cpio_start, + .wr_end = cpio_end, + .wr_file = cpio_append, + .wr_file_stat = cpio_append_stat, + .wr_fts_entry = cpio_append_fts_entry, + .wr_cpio_file = write_file_to_fd, +}; diff --git a/helper/helper.h b/helper/helper.h index e9618ba..75b6eb3 100644 --- a/helper/helper.h +++ b/helper/helper.h @@ -19,12 +19,36 @@ #ifndef FEBOOTSTRAP_SUPERMIN_HELPER_H #define FEBOOTSTRAP_SUPERMIN_HELPER_H +#include <sys/stat.h> +#include "fts_.h" + +struct writer { + /* Start a new appliance, finish one off. */ + void (*wr_start) (const char *appliance); + void (*wr_end) (void); + + /* Append the named host file to the appliance being built. The + * wr_file_stat form is used where we have already stat'd this file, + * to avoid having to stat it a second time. The wr_fts_entry form + * is used where the caller has an FTSENT. + */ + void (*wr_file) (const char *filename); + void (*wr_file_stat) (const char *filename, const struct stat *); + void (*wr_fts_entry) (FTSENT *entry); + + /* Append the contents of cpio file to the appliance being built. */ + void (*wr_cpio_file) (const char *cpio_file); +}; + /* main.c */ extern struct timeval start_t; extern int verbose; /* appliance.c */ -extern void create_appliance (char **inputs, int nr_inputs, const char *whitelist, const char *modpath, const char *appliance); +extern void create_appliance (char **inputs, int nr_inputs, const char *whitelist, const char *modpath, const char *appliance, struct writer *writer); + +/* cpio.c */ +extern struct writer cpio_writer; /* kernel.c */ extern const char *create_kernel (const char *hostcpu, const char *kernel); diff --git a/helper/main.c b/helper/main.c index d350b4e..a448b9e 100644 --- a/helper/main.c +++ b/helper/main.c @@ -152,7 +152,8 @@ main (int argc, char *argv[]) print_timestamped_message ("finished creating kernel"); /* Create the appliance. */ - create_appliance (inputs, nr_inputs, whitelist, modpath, appliance); + create_appliance (inputs, nr_inputs, whitelist, modpath, appliance, + &cpio_writer); if (verbose) print_timestamped_message ("finished creating appliance"); -- 1.7.1
Richard W.M. Jones
2010-Aug-20 10:40 UTC
[Libguestfs] [PATCH febootstrap 5/8] Add -f option for selecting the output format.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/ -------------- next part -------------->From 4215a34b9c3c8de59efee328fae5a7133e5c84b3 Mon Sep 17 00:00:00 2001From: Richard Jones <rjones at redhat.com> Date: Thu, 19 Aug 2010 14:58:03 +0100 Subject: [PATCH 5/8] Add -f option for selecting the output format. Only -f cpio is permitted by this commit. --- helper/febootstrap-supermin-helper.pod | 5 +++++ helper/main.c | 25 ++++++++++++++++++++++--- 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/helper/febootstrap-supermin-helper.pod b/helper/febootstrap-supermin-helper.pod index 29f4b95..6a2551b 100644 --- a/helper/febootstrap-supermin-helper.pod +++ b/helper/febootstrap-supermin-helper.pod @@ -33,6 +33,11 @@ booting the appliance, and should be deleted straight afterwards. =over 4 +=item B<-f cpio> | B<--format cpio> + +Select the output format. The default, and only possible output +format, is C<cpio> (ie. a Linux initramfs). + =item B<-k file> | B<--kmods file> If this option is specified, then C<file> should be a list of diff --git a/helper/main.c b/helper/main.c index a448b9e..f5a06cc 100644 --- a/helper/main.c +++ b/helper/main.c @@ -36,11 +36,14 @@ struct timeval start_t; int verbose = 0; +static const char *format = "cpio"; + enum { HELP_OPTION = CHAR_MAX + 1 }; -static const char *options = "k:vV"; +static const char *options = "f:k:vV"; static const struct option long_options[] = { { "help", 0, 0, HELP_OPTION }, + { "format", required_argument, 0, 'f' }, { "kmods", required_argument, 0, 'k' }, { "verbose", 0, 0, 'v' }, { "version", 0, 0, 'V' }, @@ -68,6 +71,8 @@ usage (const char *progname) "Options:\n" " --help\n" " Display this help text and exit.\n" + " -f cpio | --format cpio\n" + " Specify output format (default: cpio).\n" " -k file | --kmods file\n" " Specify kernel module whitelist.\n" " --verbose | -v\n" @@ -95,6 +100,10 @@ main (int argc, char *argv[]) usage (argv[0]); exit (EXIT_SUCCESS); + case 'f': + format = optarg; + break; + case 'k': whitelist = optarg; break; @@ -113,6 +122,17 @@ main (int argc, char *argv[]) } } + /* Select the correct writer module. */ + struct writer *writer; + + if (strcmp (format, "cpio") == 0) + writer = &cpio_writer; + else { + fprintf (stderr, "%s: incorrect output format (-f): must be cpio\n", + argv[0]); + exit (EXIT_FAILURE); + } + char **inputs = &argv[optind]; int nr_inputs = argc - optind - 3; @@ -152,8 +172,7 @@ main (int argc, char *argv[]) print_timestamped_message ("finished creating kernel"); /* Create the appliance. */ - create_appliance (inputs, nr_inputs, whitelist, modpath, appliance, - &cpio_writer); + create_appliance (inputs, nr_inputs, whitelist, modpath, appliance, writer); if (verbose) print_timestamped_message ("finished creating appliance"); -- 1.7.1
Richard W.M. Jones
2010-Aug-20 10:40 UTC
[Libguestfs] [PATCH febootstrap 6/8] Add -f ext2 option.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into Xen guests. http://et.redhat.com/~rjones/virt-p2v -------------- next part -------------->From 823e986203c084be03f241a20bc617c1fbbd4370 Mon Sep 17 00:00:00 2001From: Richard Jones <rjones at redhat.com> Date: Thu, 19 Aug 2010 15:21:17 +0100 Subject: [PATCH 6/8] Add -f ext2 option. This option doesn't work yet. The commit is just for adding the capability to pass the extra kernel/initrd/appliance parameters through 'main.c' and 'appliance.c'. --- helper/Makefile.am | 1 + helper/appliance.c | 3 +- helper/cpio.c | 3 +- helper/ext2.c | 46 ++++++++++++++++++++++++++++++++ helper/febootstrap-supermin-helper.pod | 38 ++++++++++++++++++++----- helper/helper.h | 16 +++++++++-- helper/main.c | 43 +++++++++++++++++++++++------- 7 files changed, 127 insertions(+), 23 deletions(-) create mode 100644 helper/ext2.c diff --git a/helper/Makefile.am b/helper/Makefile.am index 985d4b2..5c5966c 100644 --- a/helper/Makefile.am +++ b/helper/Makefile.am @@ -25,6 +25,7 @@ febootstrap_supermin_helper_SOURCES = \ helper.h \ appliance.c \ cpio.c \ + ext2.c \ kernel.c \ main.c \ utils.c diff --git a/helper/appliance.c b/helper/appliance.c index b7a46c0..2474b14 100644 --- a/helper/appliance.c +++ b/helper/appliance.c @@ -68,10 +68,11 @@ void create_appliance (char **inputs, int nr_inputs, const char *whitelist, const char *modpath, + const char *initrd, const char *appliance, struct writer *writer) { - writer->wr_start (appliance); + writer->wr_start (appliance, modpath, initrd); iterate_inputs (inputs, nr_inputs, writer); diff --git a/helper/cpio.c b/helper/cpio.c index f9ffc3f..1dd22b0 100644 --- a/helper/cpio.c +++ b/helper/cpio.c @@ -250,7 +250,8 @@ write_padding (size_t len) } static void -cpio_start (const char *appliance) +cpio_start (const char *appliance, + const char *modpath, const char *initrd) { out_fd = open (appliance, O_WRONLY | O_CREAT | O_TRUNC | O_NOCTTY, 0644); if (out_fd == -1) diff --git a/helper/ext2.c b/helper/ext2.c new file mode 100644 index 0000000..2900221 --- /dev/null +++ b/helper/ext2.c @@ -0,0 +1,46 @@ +/* febootstrap-supermin-helper reimplementation in C. + * Copyright (C) 2009-2010 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/stat.h> +#include <assert.h> + +#include "error.h" +#include "fts_.h" + +#include "helper.h" + +static void +ext2_start (const char *appliance, + const char *modpath, const char *initrd) +{ + abort (); +} + +struct writer ext2_writer = { + .wr_start = ext2_start, + /* .wr_end = , + .wr_file = , + .wr_file_stat = , + .wr_fts_entry = , + .wr_cpio_file = , */ +}; diff --git a/helper/febootstrap-supermin-helper.pod b/helper/febootstrap-supermin-helper.pod index 6a2551b..7a3dbd1 100644 --- a/helper/febootstrap-supermin-helper.pod +++ b/helper/febootstrap-supermin-helper.pod @@ -7,6 +7,8 @@ febootstrap-supermin-helper - Reconstruct initramfs from supermin appliance. febootstrap-supermin-helper supermin.img hostfiles.txt host_cpu kernel initrd febootstrap-supermin-helper input [...] host_cpu kernel initrd + febootstrap-supermin-helper -f ext2 input [...] host_cpu kernel initrd appliance + =head1 DESCRIPTION I<febootstrap-supermin-helper> reconstructs a bootable kernel and @@ -16,8 +18,8 @@ L<febootstrap-to-supermin(8)>. =head1 PARAMETERS -Of the four or five required parameters, the first few are I<input> -files, and the last two are I<output> files. +Of the required parameters, the first few are I<input> files, and the +last two or three are I<output> files. C<supermin.img> and C<hostfiles.txt> are the input files which describe the supermin appliance. (You can also use a directory name @@ -25,18 +27,38 @@ here which is searched for files). C<host_cpu> should be the host CPU, eg. C<x86_64> or C<i686>. -C<kernel> and C<initrd> are the temporary output files that this -script produces. These output files are meant to be used just for -booting the appliance, and should be deleted straight afterwards. +C<kernel>, C<initrd> and C<appliance> are the temporary output files +that this script produces. These output files are meant to be used +just for booting the appliance, and should be deleted straight +afterwards. The extra C<appliance> parameter is only required when +the format is C<ext2>. =head1 OPTIONS =over 4 -=item B<-f cpio> | B<--format cpio> +=item B<-f fmt> | B<--format fmt> + +Select the output format for the appliance. Possible formats are: + +=over 4 + +=item cpio + +A Linux initramfs. This is the default. -Select the output format. The default, and only possible output -format, is C<cpio> (ie. a Linux initramfs). +In this case you have to supply names for the C<kernel> +and C<initrd>, where the C<initrd> is the appliance. + +=item ext2 + +An ext2 filesystem. + +In this case you have to supply names for the C<kernel>, +a small C<initrd> which is used just to locate the appliance, +and the C<appliance> (the ext2 filesystem). + +=back =item B<-k file> | B<--kmods file> diff --git a/helper/helper.h b/helper/helper.h index 75b6eb3..5e62c17 100644 --- a/helper/helper.h +++ b/helper/helper.h @@ -23,8 +23,15 @@ #include "fts_.h" struct writer { - /* Start a new appliance, finish one off. */ - void (*wr_start) (const char *appliance); + /* Start building a new appliance. + * 'appliance' is the output appliance. + * 'initrd' is the mini-initrd to create (only used for ext2 output). + * 'modpath' is the kernel module path. + */ + void (*wr_start) (const char *appliance, + const char *modpath, const char *initrd); + + /* Finish off the appliance. */ void (*wr_end) (void); /* Append the named host file to the appliance being built. The @@ -45,11 +52,14 @@ extern struct timeval start_t; extern int verbose; /* appliance.c */ -extern void create_appliance (char **inputs, int nr_inputs, const char *whitelist, const char *modpath, const char *appliance, struct writer *writer); +extern void create_appliance (char **inputs, int nr_inputs, const char *whitelist, const char *modpath, const char *initrd, const char *appliance, struct writer *writer); /* cpio.c */ extern struct writer cpio_writer; +/* ext2.c */ +extern struct writer ext2_writer; + /* kernel.c */ extern const char *create_kernel (const char *hostcpu, const char *kernel); diff --git a/helper/main.c b/helper/main.c index f5a06cc..f8bf41d 100644 --- a/helper/main.c +++ b/helper/main.c @@ -57,6 +57,7 @@ usage (const char *progname) "\n" "Usage:\n" " %s [-options] inputs [...] whitelist host_cpu kernel initrd\n" + " %s -f ext2 inputs [...] whitelist host_cpu kernel initrd appliance\n" " %s --help\n" " %s --version\n" "\n" @@ -79,7 +80,7 @@ usage (const char *progname) " Enable verbose messages (give multiple times for more verbosity).\n" " --version | -V\n" " Display version number and exit.\n", - progname, progname, progname, progname); + progname, progname, progname, progname, progname); } int @@ -124,37 +125,56 @@ main (int argc, char *argv[]) /* Select the correct writer module. */ struct writer *writer; + int nr_outputs; - if (strcmp (format, "cpio") == 0) + if (strcmp (format, "cpio") == 0) { writer = &cpio_writer; + nr_outputs = 2; /* kernel and appliance (== initrd) */ + } + else if (strcmp (format, "ext2") == 0) { + writer = &ext2_writer; + nr_outputs = 3; /* kernel, initrd, appliance */ + } else { fprintf (stderr, "%s: incorrect output format (-f): must be cpio\n", argv[0]); exit (EXIT_FAILURE); } + /* [optind .. optind+nr_inputs-1] hostcpu [argc-nr_outputs-1 .. argc-1] + * <---- nr_inputs ----> 1 <---- nr_outputs ----> + */ char **inputs = &argv[optind]; - int nr_inputs = argc - optind - 3; + int nr_inputs = argc - nr_outputs - 1 - optind; + char **outputs = &argv[optind+nr_inputs+1]; + /*assert (outputs [nr_outputs] == NULL); + assert (inputs [nr_inputs + 1 + nr_outputs] == NULL);*/ if (nr_inputs < 1) { - usage (argv[0]); + fprintf (stderr, "%s: not enough files specified on the command line\n", + argv[0]); exit (EXIT_FAILURE); } /* See: https://bugzilla.redhat.com/show_bug.cgi?id=558593 */ - const char *hostcpu = argv[argc-3]; + const char *hostcpu = outputs[-1]; /* Output files. */ - const char *kernel = argv[argc-2]; - const char *appliance = argv[argc-1]; + const char *kernel = outputs[0]; + const char *initrd; + const char *appliance; + initrd = appliance = outputs[1]; + if (nr_outputs > 2) + appliance = outputs[2]; if (verbose) { print_timestamped_message ("whitelist = %s, " "host_cpu = %s, " "kernel = %s, " + "initrd = %s, " "appliance = %s", whitelist ? : "(not specified)", - hostcpu, kernel, appliance); + hostcpu, kernel, initrd, appliance); int i; for (i = 0; i < nr_inputs; ++i) print_timestamped_message ("inputs[%d] = %s", i, inputs[i]); @@ -162,7 +182,9 @@ main (int argc, char *argv[]) /* Remove the output files if they exist. */ unlink (kernel); - unlink (appliance); + unlink (initrd); + if (initrd != appliance) + unlink (appliance); /* Create kernel output file. */ const char *modpath; @@ -172,7 +194,8 @@ main (int argc, char *argv[]) print_timestamped_message ("finished creating kernel"); /* Create the appliance. */ - create_appliance (inputs, nr_inputs, whitelist, modpath, appliance, writer); + create_appliance (inputs, nr_inputs, whitelist, modpath, + initrd, appliance, writer); if (verbose) print_timestamped_message ("finished creating appliance"); -- 1.7.1
Richard W.M. Jones
2010-Aug-20 10:41 UTC
[Libguestfs] [PATCH febootstrap 7/8] Remove bogus 'whitelist' parameter from usage.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones New in Fedora 11: Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 70 libraries supprt'd http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw -------------- next part -------------->From 15dc21be8613b6f33878cd5dd82165f434c7f368 Mon Sep 17 00:00:00 2001From: Richard Jones <rjones at redhat.com> Date: Thu, 19 Aug 2010 16:11:44 +0100 Subject: [PATCH 7/8] Remove bogus 'whitelist' parameter from usage. Use the -k whitelist option to get a kernel whitelist. This is a bug in current febootstrap-supermin-helper. --- helper/main.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/helper/main.c b/helper/main.c index f8bf41d..d7dd25a 100644 --- a/helper/main.c +++ b/helper/main.c @@ -56,8 +56,8 @@ usage (const char *progname) printf ("%s: build the supermin appliance on the fly\n" "\n" "Usage:\n" - " %s [-options] inputs [...] whitelist host_cpu kernel initrd\n" - " %s -f ext2 inputs [...] whitelist host_cpu kernel initrd appliance\n" + " %s [-options] inputs [...] host_cpu kernel initrd\n" + " %s -f ext2 inputs [...] host_cpu kernel initrd appliance\n" " %s --help\n" " %s --version\n" "\n" -- 1.7.1
Richard W.M. Jones
2010-Aug-20 10:41 UTC
[Libguestfs] [PATCH febootstrap 8/8 INCOMPLETE] Add ext2 output module.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora -------------- next part -------------->From 5ed9c376904ca312d8c38a9dbd0bd8716f58ca7d Mon Sep 17 00:00:00 2001From: Richard Jones <rjones at redhat.com> Date: Thu, 19 Aug 2010 15:55:01 +0100 Subject: [PATCH 8/8] Add ext2 output module. --- README | 4 + configure.ac | 19 +++ helper/Makefile.am | 2 + helper/ext2.c | 429 +++++++++++++++++++++++++++++++++++++++++++++++++++- helper/ext2cpio.c | 358 +++++++++++++++++++++++++++++++++++++++++++ helper/ext2cpio.h | 42 +++++ 6 files changed, 848 insertions(+), 6 deletions(-) create mode 100644 helper/ext2cpio.c create mode 100644 helper/ext2cpio.h diff --git a/README b/README index b8df88e..5383dcb 100644 --- a/README +++ b/README @@ -38,6 +38,10 @@ Requirements qemu - If you want to test-run your systems. + libext2fs + /sbin/mke2fs + - These are part of e2fsprogs. + Building and installing ----------------------- diff --git a/configure.ac b/configure.ac index 1a94700..c91190a 100644 --- a/configure.ac +++ b/configure.ac @@ -58,6 +58,25 @@ if test "x$YUM" = "xno" ; then AC_MSG_FAILURE([yum program not found]) fi +AC_PATH_PROG([MKE2FS],[mke2fs],[no]) +if test "x$MKE2FS" = "xno" ; then + AC_MSG_FAILURE([mke2fs program not found (is /sbin in your current path?)]) +fi +AC_DEFINE_UNQUOTED([MKE2FS],["$MKE2FS"], + [Full path to the mke2fs program.]) + +AC_CHECK_LIB([com_err],[error_message],[],[ + AC_MSG_FAILURE([com_err library not found (part of e2fsprogs)]) +]) + +AC_CHECK_LIB([ext2fs],[ext2fs_file_open2],[],[ + AC_MSG_FAILURE([libext2fs library not found (part of e2fsprogs)]) +]) + +AC_CHECK_HEADER([ext2fs/ext2fs.h],[],[ + AC_MSG_FAILURE([Header <ext2fs/ext2fs.h> not found (part of e2fsprogs)]) +]) + AC_CONFIG_HEADERS([config.h]) AC_CONFIG_FILES([Makefile lib/Makefile helper/Makefile examples/Makefile]) AC_OUTPUT diff --git a/helper/Makefile.am b/helper/Makefile.am index 5c5966c..c43f906 100644 --- a/helper/Makefile.am +++ b/helper/Makefile.am @@ -26,6 +26,8 @@ febootstrap_supermin_helper_SOURCES = \ appliance.c \ cpio.c \ ext2.c \ + ext2cpio.c \ + ext2cpio.h \ kernel.c \ main.c \ utils.c diff --git a/helper/ext2.c b/helper/ext2.c index 2900221..f2ad92c 100644 --- a/helper/ext2.c +++ b/helper/ext2.c @@ -21,26 +21,443 @@ #include <stdio.h> #include <stdlib.h> #include <string.h> +#include <fcntl.h> +#include <unistd.h> +#include <errno.h> +#include <limits.h> #include <sys/stat.h> #include <assert.h> #include "error.h" #include "fts_.h" +#include "xvasprintf.h" #include "helper.h" +#include "ext2cpio.h" + +ext2_filsys fs; + +/* The ext2 image that we build always has a fixed size, and we 'hope' + * that the files fit in (otherwise we'll get an error). Note that + * the file is sparsely allocated. + * + * The downside of allocating a very large initial disk is that the + * fixed overhead of ext2 is larger (since ext2 calculates it based on + * the size of the disk). For a 1GB disk the overhead is + * approximately 16MB. + * + * In future, make this configurable, or determine it from the input + * files (XXX). + */ +#define APPLIANCE_SIZE (1024*1024*1024) static void ext2_start (const char *appliance, const char *modpath, const char *initrd) { - abort (); + initialize_ext2_error_table (); + + int fd = open (appliance, O_WRONLY | O_CREAT | O_TRUNC | O_NOCTTY, 0644); + if (fd == -1) + error (EXIT_FAILURE, errno, "open: %s", appliance); + + if (lseek (fd, APPLIANCE_SIZE - 1, SEEK_SET) == -1) + error (EXIT_FAILURE, errno, "lseek"); + + char c = 0; + if (write (fd, &c, 1) != 1) + error (EXIT_FAILURE, errno, "write"); + + if (close (fd) == -1) + error (EXIT_FAILURE, errno, "close"); + + /* Run mke2fs on the file. + * XXX Quoting, but this string doesn't come from an untrusted source. + */ + char *cmd = xasprintf ("%s -t ext2 -F%s '%s'", + MKE2FS, + verbose ? "" : "q", + appliance); + int r = system (cmd); + if (r == -1 || WEXITSTATUS (r) != 0) + error (EXIT_FAILURE, 0, "%s: failed", cmd); + free (cmd); + + if (verbose) + print_timestamped_message ("finished mke2fs"); + + /* Open the filesystem. */ + errcode_t err + ext2fs_open (appliance, EXT2_FLAG_RW, 0, 0, unix_io_manager, &fs); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_open: %s", error_message (err)); + + /* Bitmaps are not loaded by default, so load them. ext2fs_close will + * write out any changes. + */ + err = ext2fs_read_bitmaps (fs); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_read_bitmaps: %s", error_message (err)); +} + +static void +ext2_end (void) +{ + /* Write out changes and close. */ + errcode_t err = ext2fs_close (fs); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_close: %s", error_message (err)); +} + +void +ext2_mkdir (ext2_ino_t dir_ino, const char *dirname, const char *basename, + mode_t mode, uid_t uid, gid_t gid, + time_t ctime, time_t atime, time_t mtime) +{ + errcode_t err; + + mode = LINUX_S_IFDIR | (mode & 0777); + + /* Does the directory exist? This is legitimate: we just skip + * this case. + */ + ext2_ino_t ino; + err = ext2fs_namei (fs, EXT2_ROOT_INO, dir_ino, basename, &ino); + if (err == 0) + return; /* skip */ + + /* Otherwise, create it. */ + err = ext2fs_new_inode (fs, dir_ino, mode, 0, &ino); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_new_inode: %s", error_message (err)); + + err = ext2fs_mkdir (fs, dir_ino, ino, basename); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_mkdir: %s/%s: %s", + dirname, basename, error_message (err)); + + /* Copy the final permissions, UID etc. to the inode. */ + struct ext2_inode inode; + err = ext2fs_read_inode (fs, ino, &inode); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_read_inode: %s", error_message (err)); + inode.i_mode = mode; + inode.i_uid = uid; + inode.i_gid = gid; + inode.i_ctime = ctime; + inode.i_atime = atime; + inode.i_mtime = mtime; + err = ext2fs_write_inode (fs, ino, &inode); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_write_inode: %s", error_message (err)); +} + +void +ext2_empty_inode (ext2_ino_t dir_ino, const char *dirname, const char *basename, + mode_t mode, uid_t uid, gid_t gid, + time_t ctime, time_t atime, time_t mtime, + int major, int minor, int dir_ft, ext2_ino_t *ino_ret) +{ + errcode_t err; + struct ext2_inode inode; + ext2_ino_t ino; + + err = ext2fs_new_inode (fs, dir_ino, mode, 0, &ino); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_new_inode: %s", error_message (err)); + + memset (&inode, 0, sizeof inode); + inode.i_mode = mode; + inode.i_uid = uid; + inode.i_gid = gid; + inode.i_blocks = 0; + inode.i_links_count = 1; + inode.i_ctime = ctime; + inode.i_atime = atime; + inode.i_mtime = mtime; + inode.i_size = 0; + inode.i_block[0] = (minor & 0xff) | (major << 8) | ((minor & ~0xff) << 12); + + err = ext2fs_write_new_inode (fs, ino, &inode); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_write_inode: %s", error_message (err)); + + ext2_link (dir_ino, basename, ino, dir_ft); + + ext2fs_inode_alloc_stats2 (fs, ino, 1, 0); + + if (ino_ret) + *ino_ret = ino; +} + +/* You must create the file first with ext2_empty_inode. */ +void +ext2_write_file (ext2_ino_t ino, const char *buf, size_t size) +{ + errcode_t err; + ext2_file_t file; + err = ext2fs_file_open2 (fs, ino, NULL, EXT2_FILE_WRITE, &file); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_file_open2: %s", error_message (err)); + + /* ext2fs_file_write cannot deal with partial writes. You have + * to write the entire file in a single call. + */ + unsigned int written; + err = ext2fs_file_write (file, buf, size, &written); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_file_write: %s", error_message (err)); + if ((size_t) written != size) + error (EXIT_FAILURE, 0, + "ext2fs_file_write: size = %zu != written = %u\n", + size, written); + + err = ext2fs_file_flush (file); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_file_flush: %s", error_message (err)); + err = ext2fs_file_close (file); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_file_close: %s", error_message (err)); + + /* Update the true size in the inode. */ + struct ext2_inode inode; + err = ext2fs_read_inode (fs, ino, &inode); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_read_inode: %s", error_message (err)); + inode.i_size = size; + err = ext2fs_write_inode (fs, ino, &inode); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_write_inode: %s", error_message (err)); +} + +/* This is just a wrapper around ext2fs_link which calls + * ext2fs_expand_dir as necessary if the directory fills up. See + * definition of expand_dir in the sources of debugfs. + */ +void +ext2_link (ext2_ino_t dir_ino, const char *basename, ext2_ino_t ino, int dir_ft) +{ + errcode_t err; + + again: + err = ext2fs_link (fs, dir_ino, basename, ino, dir_ft); + + if (err == EXT2_ET_DIR_NO_SPACE) { + err = ext2fs_expand_dir (fs, dir_ino); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2_link: ext2fs_expand_dir: %s: %s", + basename, error_message (err)); + goto again; + } + + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_link: %s: %s", + basename, error_message (err)); +} + +static int +release_block (ext2_filsys fs, blk_t *blocknr, + int blockcnt, void *private) +{ + blk_t block; + + block = *blocknr; + ext2fs_block_alloc_stats (fs, block, -1); + return 0; +} + +/* unlink or rmdir path, if it exists. */ +void +ext2_clean_path (ext2_ino_t dir_ino, + const char *dirname, const char *basename, + int isdir) +{ + errcode_t err; + + ext2_ino_t ino; + err = ext2fs_lookup (fs, dir_ino, basename, strlen (basename), + NULL, &ino); + if (err == EXT2_ET_FILE_NOT_FOUND) + return; + + if (!isdir) { + struct ext2_inode inode; + err = ext2fs_read_inode (fs, ino, &inode); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_read_inode: %s", error_message (err)); + inode.i_links_count--; + err = ext2fs_write_inode (fs, ino, &inode); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_write_inode: %s", error_message (err)); + + err = ext2fs_unlink (fs, dir_ino, basename, 0, 0); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_unlink_inode: %s", error_message (err)); + + if (inode.i_links_count == 0) { + inode.i_dtime = time (NULL); + err = ext2fs_write_inode (fs, ino, &inode); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2fs_write_inode: %s", error_message (err)); + + if (ext2fs_inode_has_valid_blocks (&inode)) + ext2fs_block_iterate (fs, ino, BLOCK_FLAG_READ_ONLY, NULL, + release_block, NULL); + + ext2fs_inode_alloc_stats2 (fs, ino, -1, isdir); + } + } + /* else it's a directory, what to do? XXX */ +} + +/* Read in the whole file into memory. Check the size is still 'size'. */ +static char * +read_whole_file (const char *filename, size_t size) +{ + char *buf = malloc (size); + if (buf == NULL) + error (EXIT_FAILURE, errno, "malloc"); + + int fd = open (filename, O_RDONLY); + if (fd == -1) + error (EXIT_FAILURE, errno, "open: %s", filename); + + size_t n = 0; + char *p = buf; + + while (n < size) { + ssize_t r = read (fd, p, size - n); + if (r == -1) + error (EXIT_FAILURE, errno, "read: %s", filename); + if (r == 0) + error (EXIT_FAILURE, 0, + "error: file has changed size unexpectedly: %s", filename); + n += r; + p += r; + } + + if (close (fd) == -1) + error (EXIT_FAILURE, errno, "close: %s", filename); + + return buf; +} + +/* Add a file (or directory etc) from the host. */ +static void +ext2_file_stat (const char *orig_filename, const struct stat *statbuf) +{ + errcode_t err; + + if (verbose >= 2) + fprintf (stderr, "ext2_file_stat %s 0%o\n", + orig_filename, statbuf->st_mode); + + /* Sanity check the path. These rules are always true for the paths + * passed to us here from the appliance layer. The assertions just + * verify that the rules haven't changed. + */ + size_t n = strlen (orig_filename); + assert (n <= PATH_MAX); + assert (n > 0); + assert (orig_filename[0] == '/'); /* always absolute path */ + assert (n == 1 || orig_filename[n-1] != '/'); /* no trailing slash */ + + /* Don't make the root directory, it always exists. This simplifies + * the code that follows. + */ + if (n == 1) return; + + const char *dirname, *basename; + const char *p = strrchr (orig_filename, '/'); + ext2_ino_t dir_ino; + if (orig_filename == p) { /* "/foo" */ + dirname = "/"; + basename = orig_filename+1; + dir_ino = EXT2_ROOT_INO; + } else { /* "/foo/bar" */ + dirname = strndup (orig_filename, p-orig_filename); + basename = p+1; + + /* Look up the parent directory. */ + err = ext2fs_namei (fs, EXT2_ROOT_INO, EXT2_ROOT_INO, dirname, &dir_ino); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2: parent directory not found: %s: %s", + dirname, error_message (err)); + } + + ext2_clean_path (dir_ino, dirname, basename, S_ISDIR (statbuf->st_mode)); + + int dir_ft; + + /* Create regular file. */ + if (S_ISREG (statbuf->st_mode)) { + /* XXX Hard links get duplicated here. */ + ext2_ino_t ino; + ext2_empty_inode (dir_ino, dirname, basename, + statbuf->st_mode, statbuf->st_uid, statbuf->st_gid, + statbuf->st_ctime, statbuf->st_atime, statbuf->st_mtime, + 0, 0, EXT2_FT_REG_FILE, &ino); + + if (statbuf->st_size > 0) { + char *buf = read_whole_file (orig_filename, statbuf->st_size); + ext2_write_file (ino, buf, statbuf->st_size); + free (buf); + } + } + /* Create directory. */ + else if (S_ISDIR (statbuf->st_mode)) + ext2_mkdir (dir_ino, dirname, basename, + statbuf->st_mode, statbuf->st_uid, statbuf->st_gid, + statbuf->st_ctime, statbuf->st_atime, statbuf->st_mtime); + /* Create a special file. */ + else if (S_ISBLK (statbuf->st_mode)) { + dir_ft = EXT2_FT_BLKDEV; + goto make_special; + } + else if (S_ISCHR (statbuf->st_mode)) { + dir_ft = EXT2_FT_CHRDEV; + goto make_special; + } else if (S_ISFIFO (statbuf->st_mode)) { + dir_ft = EXT2_FT_FIFO; + goto make_special; + } else if (S_ISSOCK (statbuf->st_mode)) { + dir_ft = EXT2_FT_SOCK; + make_special: + ext2_empty_inode (dir_ino, dirname, basename, + statbuf->st_mode, statbuf->st_uid, statbuf->st_gid, + statbuf->st_ctime, statbuf->st_atime, statbuf->st_mtime, + major (statbuf->st_rdev), minor (statbuf->st_rdev), + dir_ft, NULL); + } +} + +static void +ext2_file (const char *filename) +{ + struct stat statbuf; + + if (lstat (filename, &statbuf) == -1) + error (EXIT_FAILURE, errno, "lstat: %s", filename); + ext2_file_stat (filename, &statbuf); +} + +/* In theory this could be optimized to avoid a namei lookup, but + * it probably wouldn't make much difference. + */ +static void +ext2_fts_entry (FTSENT *entry) +{ + if (entry->fts_info & FTS_NS || entry->fts_info & FTS_NSOK) + ext2_file (entry->fts_path); + else + ext2_file_stat (entry->fts_path, entry->fts_statp); } struct writer ext2_writer = { .wr_start = ext2_start, - /* .wr_end = , - .wr_file = , - .wr_file_stat = , - .wr_fts_entry = , - .wr_cpio_file = , */ + .wr_end = ext2_end, + .wr_file = ext2_file, + .wr_file_stat = ext2_file_stat, + .wr_fts_entry = ext2_fts_entry, + .wr_cpio_file = ext2_cpio_file, }; diff --git a/helper/ext2cpio.c b/helper/ext2cpio.c new file mode 100644 index 0000000..65b8bcb --- /dev/null +++ b/helper/ext2cpio.c @@ -0,0 +1,358 @@ +/* febootstrap-supermin-helper reimplementation in C. + * Copyright (C) 2009-2010 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + */ + +#include <config.h> + +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <fcntl.h> +#include <unistd.h> +#include <errno.h> +#include <limits.h> +#include <sys/stat.h> +#include <assert.h> + +#include "error.h" + +#include "helper.h" +#include "ext2cpio.h" + +/* This function must unpack the cpio file and add the files it + * contains to the ext2 filesystem. Essentially this is doing the + * same thing as the kernel init/initramfs.c code. Note that we + * assume that the cpio is uncompressed newc format and can't/won't + * deal with anything else. All this cpio parsing code is copied to + * some extent from init/initramfs.c in the kernel. + */ +#define N_ALIGN(len) ((((len) + 1) & ~3) + 2) + +static unsigned long cpio_ino, nlink; +static mode_t mode; +static unsigned long body_len, name_len; +static uid_t uid; +static gid_t gid; +static time_t mtime; +static int dev_major, dev_minor, rdev_major, rdev_minor; +static loff_t curr, next_header; +static FILE *fp; + +static void parse_header (char *s); +static int parse_next_entry (void); +static void skip_to_next_header (void); +static void read_file (void); +static char *read_whole_body (void); +static ext2_ino_t maybe_link (void); +static void add_link (ext2_ino_t real_ino); +static void clear_links (void); + +void +ext2_cpio_file (const char *cpio_file) +{ + fp = fopen (cpio_file, "r"); + if (fp == NULL) + error (EXIT_FAILURE, errno, "open: %s", cpio_file); + + curr = 0; + while (parse_next_entry ()) + ; + + fclose (fp); +} + +static int +parse_next_entry (void) +{ + clearerr (fp); + + char header[110]; + + /* Skip padding and synchronize with the next header. */ + again: + if (fread (&header[0], 4, 1, fp) != 1) { + if (feof (fp)) + return 0; + error (EXIT_FAILURE, errno, "read failure reading cpio file"); + } + curr += 4; + if (memcmp (header, "\0\0\0\0", 4) == 0) + goto again; + + /* Read the rest of the header field. */ + if (fread (&header[4], sizeof header - 4, 1, fp) != 1) + error (EXIT_FAILURE, errno, "read failure reading cpio file"); + curr += sizeof header - 4; + + if (verbose >= 2) + fprintf (stderr, "cpio header %s\n", header); + + if (memcmp (header, "070707", 6) == 0) + error (EXIT_FAILURE, 0, "incorrect cpio method: use -H newc option"); + if (memcmp (header, "070701", 6) != 0) + error (EXIT_FAILURE, 0, "input is not a cpio file"); + + parse_header (header); + + next_header = curr + N_ALIGN(name_len) + body_len; + next_header = (next_header + 3) & ~3; + if (name_len <= 0 || name_len > PATH_MAX) + skip_to_next_header (); + else if (S_ISLNK (mode)) { + if (body_len <= 0 || body_len > PATH_MAX) + skip_to_next_header (); + else + read_file (); + } + else if (!S_ISREG (mode) && body_len > 0) + skip_to_next_header (); /* only regular files have bodies */ + else + read_file (); /* could be file, directory, block special, ... */ + + return 1; +} + +static void +parse_header (char *s) +{ + unsigned long parsed[12]; + char buf[9]; + int i; + + buf[8] = '\0'; + for (i = 0, s += 6; i < 12; i++, s += 8) { + memcpy (buf, s, 8); + parsed[i] = strtoul (buf, NULL, 16); + } + cpio_ino = parsed[0]; /* fake inode number from cpio file */ + mode = parsed[1]; + uid = parsed[2]; + gid = parsed[3]; + nlink = parsed[4]; + mtime = parsed[5]; + body_len = parsed[6]; + dev_major = parsed[7]; + dev_minor = parsed[8]; + rdev_major = parsed[9]; + rdev_minor = parsed[10]; + name_len = parsed[11]; +} + +static void +skip_to_next_header (void) +{ + char buf[65536]; + + while (curr < next_header) { + size_t bytes = (size_t) (next_header - curr); + if (bytes > sizeof buf) + bytes = sizeof buf; + size_t r = fread (buf, 1, bytes, fp); + if (r == 0) + error (EXIT_FAILURE, errno, "error or unexpected end of cpio file"); + curr += r; + } +} + +/* Read any sort of file. The body will only be present for + * regular files and symlinks. + */ +static void +read_file (void) +{ + errcode_t err; + int dir_ft; + char name[N_ALIGN(name_len)+1]; /* asserted above this is <= PATH_MAX */ + + if (fread (name, N_ALIGN(name_len), 1, fp) != 1) + error (EXIT_FAILURE, errno, "read failure reading name field in cpio file"); + curr += N_ALIGN(name_len); + + name[name_len] = '\0'; + + if (verbose >= 2) + fprintf (stderr, "ext2 read_file %s %o\n", name, mode); + + if (strcmp (name, "TRAILER!!!") == 0) { + clear_links (); + goto skip; + } + + /* The name will be something like "bin/ls" or "./bin/ls". It won't + * (ever?) be an absolute path. Skip leading parts, and if it refers + * to the root directory just skip it entirely. + */ + char *dirname = name, *basename; + if (*dirname == '.') + dirname++; + if (*dirname == '/') + dirname++; + if (*dirname == '\0') + goto skip; + + ext2_ino_t dir_ino; + basename = strrchr (dirname, '/'); + if (basename == NULL) { + basename = dirname; + dir_ino = EXT2_ROOT_INO; + } else { + *basename++ = '\0'; + + /* Look up the parent directory. */ + err = ext2fs_namei (fs, EXT2_ROOT_INO, EXT2_ROOT_INO, dirname, &dir_ino); + if (err != 0) + error (EXIT_FAILURE, 0, "ext2: parent directory not found: %s: %s", + dirname, error_message (err)); + } + + if (verbose >= 2) + fprintf (stderr, "ext2 read_file dirname %s basename %s\n", + dirname, basename); + + ext2_clean_path (dir_ino, dirname, basename, S_ISDIR (mode)); + + /* Create a regular file. */ + if (S_ISREG (mode)) { + ext2_ino_t ml = maybe_link (); + ext2_ino_t ino; + if (ml <= 1) { + ext2_empty_inode (dir_ino, dirname, basename, + mode, uid, gid, mtime, mtime, mtime, + 0, 0, EXT2_FT_REG_FILE, &ino); + if (ml == 1) + add_link (ino); + } + else /* ml >= 2 */ { + /* It's a hard link back to a previous file. */ + ino = ml; + ext2_link (dir_ino, basename, ino, EXT2_FT_REG_FILE); + } + + if (body_len) { + char *buf = read_whole_body (); + ext2_write_file (ino, buf, body_len); + free (buf); + } + } + /* Create a symlink. */ + else if (S_ISLNK (mode)) { + ext2_ino_t ino; + ext2_empty_inode (dir_ino, dirname, basename, + mode, uid, gid, mtime, mtime, mtime, + 0, 0, EXT2_FT_SYMLINK, &ino); + + char *buf = read_whole_body (); + ext2_write_file (ino, buf, body_len); + free (buf); + } + /* Create a directory. */ + else if (S_ISDIR (mode)) { + ext2_mkdir (dir_ino, dirname, basename, + mode, uid, gid, mtime, mtime, mtime); + } + /* Create a special file. */ + else if (S_ISBLK (mode)) { + dir_ft = EXT2_FT_BLKDEV; + goto make_special; + } + else if (S_ISCHR (mode)) { + dir_ft = EXT2_FT_CHRDEV; + goto make_special; + } else if (S_ISFIFO (mode)) { + dir_ft = EXT2_FT_FIFO; + goto make_special; + } else if (S_ISSOCK (mode)) { + dir_ft = EXT2_FT_SOCK; + make_special: + /* Just like the kernel, we ignore special files with nlink > 1. */ + if (maybe_link () == 0) + ext2_empty_inode (dir_ino, dirname, basename, + mode, uid, gid, mtime, mtime, mtime, + rdev_major, rdev_minor, dir_ft, NULL); + } + + skip: + skip_to_next_header (); +} + +static char * +read_whole_body (void) +{ + char *buf = malloc (body_len); + if (buf == NULL) + error (EXIT_FAILURE, errno, "malloc"); + + size_t r = fread (buf, body_len, 1, fp); + if (r != 1) + error (EXIT_FAILURE, errno, "read failure reading body in cpio file"); + curr += body_len; + + return buf; +} + +struct links { + struct links *next; + unsigned long cpio_ino; /* fake ino from cpio file */ + int minor; + int major; + ext2_ino_t real_ino; /* real inode number on ext2 filesystem */ +}; +static struct links *links_head = NULL; + +/* If it's a hard link, return the linked inode number in the real + * ext2 filesystem. + * + * Returns: 0 = not a hard link + * 1 = possible unresolved hard link + * inode number = resolved hard link to this inode + */ +static ext2_ino_t +maybe_link (void) +{ + if (nlink >= 2) { + struct links *p; + for (p = links_head; p; p = p->next) { + if (p->cpio_ino != cpio_ino) + continue; + if (p->minor != dev_minor) + continue; + if (p->major != dev_major) + continue; + return p->real_ino; + } + return 1; + } + + return 0; +} + +static void +add_link (ext2_ino_t real_ino) +{ + struct links *p = malloc (sizeof (*p)); + p->cpio_ino = cpio_ino; + p->minor = dev_minor; + p->major = dev_major; + p->real_ino = real_ino; +} + +static void +clear_links (void) +{ + /* Don't bother to free the linked list in this short-lived program. */ + links_head = NULL; +} diff --git a/helper/ext2cpio.h b/helper/ext2cpio.h new file mode 100644 index 0000000..570e414 --- /dev/null +++ b/helper/ext2cpio.h @@ -0,0 +1,42 @@ +/* febootstrap-supermin-helper reimplementation in C. + * Copyright (C) 2009-2010 Red Hat Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + */ + +/* This is a private interface used between the two parts of the + * ext2 plugin. + */ + +#ifndef FEBOOTSTRAP_SUPERMIN_EXT2CPIO_H +#define FEBOOTSTRAP_SUPERMIN_EXT2CPIO_H + +/* Inlining is broken in the ext2fs header file. Disable it by + * defining the following: + */ +#define NO_INLINE_FUNCS + +#include <ext2fs/ext2fs.h> + +extern ext2_filsys fs; + +extern void ext2_mkdir (ext2_ino_t dir_ino, const char *dirname, const char *basename, mode_t mode, uid_t uid, gid_t gid, time_t ctime, time_t atime, time_t mtime); +extern void ext2_empty_inode (ext2_ino_t dir_ino, const char *dirname, const char *basename, mode_t mode, uid_t uid, gid_t gid, time_t ctime, time_t atime, time_t mtime, int major, int minor, int dir_ft, ext2_ino_t *ino_ret); +extern void ext2_write_file (ext2_ino_t ino, const char *buf, size_t size); +extern void ext2_link (ext2_ino_t dir_ino, const char *basename, ext2_ino_t ino, int dir_ft); +extern void ext2_cpio_file (const char *cpio_file); +extern void ext2_clean_path (ext2_ino_t dir_ino, const char *dirname, const char *basename, int isdir); + +#endif /* FEBOOTSTRAP_SUPERMIN_EXT2CPIO_H */ -- 1.7.1
Apparently Analagous Threads
- [PATCH 0/5] 5 conservative changes to errno handling
- [PATCH 0/8 v2 DISCUSSION ONLY] Connecting to live virtual machines
- [PATCH 0/9] Enhance virt-resize so it can really expand Linux and Windows guests
- [PATCH 0/7] Add libvirt domain to core API
- [PATCH 0/13 v2] Prepare for adding write support to hivex (Windows registry) library