Matteo Cafasso
2017-Apr-06 20:41 UTC
[Libguestfs] [PATCH v6 0/7] Feature: Yara file scanning
v6: - use new test functions - fix yara_detection struct field names - revert yara_load function to initial version With Pino we were exploring the idea of allowing Users to load multiple rule files with subsequent calls to yara_load API. https://www.redhat.com/archives/libguestfs/2016-November/msg00119.html It turns out impractical due to YARA API limitations. It is possible to load multiple rule source files into the compiler. Yet once compiled no further rule file can be added. This would make the yara_load API difficult to understand for the end User. The yara tool iself cannot scan files with more than a rule file. To combine multiple rule files the User is recommended to use the yarac tool. This makes the yara_load API more similar to the yara workflow. - further small fixes according to the v5 comments After further Matteo Cafasso (7): daemon: expose file upload logic appliance: add yara dependency New API: yara_load New API: yara_destroy New API: internal_yara_scan New API: yara_scan yara_scan: added API tests appliance/packagelist.in | 4 + configure.ac | 1 + daemon/Makefile.am | 4 +- daemon/cleanups.c | 9 + daemon/cleanups.h | 2 + daemon/daemon.h | 3 + daemon/upload.c | 70 +++---- daemon/yara.c | 303 +++++++++++++++++++++++++++++++ generator/Makefile.am | 3 + generator/actions.ml | 6 +- generator/actions_yara.ml | 92 ++++++++++ generator/actions_yara.mli | 22 +++ generator/proc_nr.ml | 3 + generator/structs.ml | 9 + gobject/Makefile.inc | 2 + java/Makefile.inc | 1 + java/com/redhat/et/libguestfs/.gitignore | 1 + lib/MAX_PROC_NR | 2 +- lib/Makefile.am | 1 + lib/yara.c | 127 +++++++++++++ m4/guestfs_daemon.m4 | 14 ++ tests/yara/Makefile.am | 26 +++ tests/yara/test-yara-scan.sh | 61 +++++++ 23 files changed, 731 insertions(+), 35 deletions(-) create mode 100644 daemon/yara.c create mode 100644 generator/actions_yara.ml create mode 100644 generator/actions_yara.mli create mode 100644 lib/yara.c create mode 100644 tests/yara/Makefile.am create mode 100755 tests/yara/test-yara-scan.sh -- 2.11.0
Matteo Cafasso
2017-Apr-06 20:41 UTC
[Libguestfs] [PATCH v6 1/7] daemon: expose file upload logic
Allows other modules to use the same logic for uploading files.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
---
daemon/daemon.h | 3 +++
daemon/upload.c | 70 ++++++++++++++++++++++++++++++++-------------------------
2 files changed, 42 insertions(+), 31 deletions(-)
diff --git a/daemon/daemon.h b/daemon/daemon.h
index abec087cd..797ec2dd9 100644
--- a/daemon/daemon.h
+++ b/daemon/daemon.h
@@ -259,6 +259,9 @@ extern int64_t ntfs_minimum_size (const char *device);
extern int swap_set_uuid (const char *device, const char *uuid);
extern int swap_set_label (const char *device, const char *label);
+/*-- in upload.c --*/
+extern int upload_to_fd (int fd);
+
/* ordinary daemon functions use these to indicate errors
* NB: you don't need to prefix the string with the current command,
* it is added automatically by the client-side RPC stubs.
diff --git a/daemon/upload.c b/daemon/upload.c
index 655baf29d..144bb246c 100644
--- a/daemon/upload.c
+++ b/daemon/upload.c
@@ -54,60 +54,68 @@ write_cb (void *data_vp, const void *buf, size_t len)
return 0;
}
+int
+upload_to_fd (int fd)
+{
+ int ret = 0, err = 0;
+ struct write_cb_data data = { .fd = fd, .written = 0 };
+
+ ret = receive_file (write_cb, &data);
+ if (ret == -1) { /* write error */
+ err = errno;
+ ret = cancel_receive ();
+ errno = err;
+ reply_with_error ("write error");
+ close (fd);
+ return -1;
+ }
+ if (ret == -2) { /* cancellation from library */
+ /* This error is ignored by the library since it initiated the
+ * cancel. Nevertheless we must send an error reply here.
+ */
+ reply_with_error ("file upload cancelled");
+ close (fd);
+ return -1;
+ }
+
+ if (close (fd) == -1) {
+ reply_with_perror ("close");
+ return -1;
+ }
+
+ return 0;
+}
+
/* Has one FileIn parameter. */
static int
upload (const char *filename, int flags, int64_t offset)
{
- struct write_cb_data data = { .written = 0 };
- int err, r, is_dev;
+ int err, is_dev, fd;
is_dev = STRPREFIX (filename, "/dev/");
if (!is_dev) CHROOT_IN;
- data.fd = open (filename, flags, 0666);
+ fd = open (filename, flags, 0666);
if (!is_dev) CHROOT_OUT;
- if (data.fd == -1) {
+ if (fd == -1) {
err = errno;
- r = cancel_receive ();
+ cancel_receive ();
errno = err;
reply_with_perror ("%s", filename);
return -1;
}
if (offset) {
- if (lseek (data.fd, offset, SEEK_SET) == -1) {
+ if (lseek (fd, offset, SEEK_SET) == -1) {
err = errno;
- r = cancel_receive ();
+ cancel_receive ();
errno = err;
reply_with_perror ("lseek: %s", filename);
return -1;
}
}
- r = receive_file (write_cb, &data);
- if (r == -1) { /* write error */
- err = errno;
- r = cancel_receive ();
- errno = err;
- reply_with_error ("write error: %s", filename);
- close (data.fd);
- return -1;
- }
- if (r == -2) { /* cancellation from library */
- /* This error is ignored by the library since it initiated the
- * cancel. Nevertheless we must send an error reply here.
- */
- reply_with_error ("file upload cancelled");
- close (data.fd);
- return -1;
- }
-
- if (close (data.fd) == -1) {
- reply_with_perror ("close: %s", filename);
- return -1;
- }
-
- return 0;
+ return upload_to_fd (fd);
}
/* Has one FileIn parameter. */
--
2.11.0
Matteo Cafasso
2017-Apr-06 20:41 UTC
[Libguestfs] [PATCH v6 2/7] appliance: add yara dependency
libyara3 on Debian/Ubuntu
yara on SUSE/RedHat
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
---
appliance/packagelist.in | 4 ++++
daemon/Makefile.am | 3 ++-
m4/guestfs_daemon.m4 | 14 ++++++++++++++
3 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/appliance/packagelist.in b/appliance/packagelist.in
index 5cf22768a..8846ce846 100644
--- a/appliance/packagelist.in
+++ b/appliance/packagelist.in
@@ -52,6 +52,7 @@ ifelse(REDHAT,1,
vim-minimal
xz
yajl
+ yara
zfs-fuse
)
@@ -86,6 +87,7 @@ dnl iproute has been renamed to iproute2
libsystemd-journal0
libtirpc1
libyajl2
+ libyara3
linux-image
dnl syslinux 'suggests' mtools, but in reality it's a hard
dependency:
mtools
@@ -129,6 +131,7 @@ ifelse(ARCHLINUX,1,
vim
xz
yajl
+ yara
)
ifelse(SUSE,1,
@@ -159,6 +162,7 @@ ifelse(SUSE,1,
systemd
vim
xz
+ yara
)
ifelse(FRUGALWARE,1,
diff --git a/daemon/Makefile.am b/daemon/Makefile.am
index 8632c3768..af4430f20 100644
--- a/daemon/Makefile.am
+++ b/daemon/Makefile.am
@@ -191,7 +191,8 @@ guestfsd_LDADD = \
$(SERVENT_LIB) \
$(PCRE_LIBS) \
$(TSK_LIBS) \
- $(RPC_LIBS)
+ $(RPC_LIBS) \
+ $(YARA_LIBS)
guestfsd_CPPFLAGS = \
-I$(top_srcdir)/gnulib/lib \
diff --git a/m4/guestfs_daemon.m4 b/m4/guestfs_daemon.m4
index eb5a6d5cf..522cd5f0e 100644
--- a/m4/guestfs_daemon.m4
+++ b/m4/guestfs_daemon.m4
@@ -126,3 +126,17 @@ AC_CHECK_LIB([tsk],[tsk_version_print],[
AC_DEFINE([HAVE_LIBTSK], [1], [Define to 1 if The Sleuth Kit library
(libtsk) is available.])
], [])
],[AC_MSG_WARN([The Sleuth Kit library (libtsk) not found])])
+
+dnl yara library (optional)
+PKG_CHECK_MODULES([YARA], [libyara],[
+ AC_SUBST([YARA_CFLAGS])
+ AC_SUBST([YARA_LIBS])
+ AC_DEFINE([HAVE_YARA],[1],[yara library found at compile time.])
+],[
+ AC_CHECK_LIB([yara],[yr_initialize],[
+ AC_CHECK_HEADER([yara.h],[
+ AC_SUBST([YARA_LIBS], [-lyara])
+ AC_DEFINE([HAVE_YARA], [1], [Define to 1 if Yara library is
available.])
+ ], [])
+ ],[AC_MSG_WARN([Yara library not found])])
+])
--
2.11.0
The yara_load API allows to load a set of Yara rules contained within a
file on the host.
Rules can be in binary format, as when compiled with yarac command, or
in source code format. In the latter case, the rules will be first
compiled and then loaded.
Subsequent calls of the yara_load API will result in the discard of the
previously loaded rules.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
---
daemon/Makefile.am | 1 +
daemon/cleanups.c | 9 ++
daemon/cleanups.h | 2 +
daemon/yara.c | 203 +++++++++++++++++++++++++++++++++++++++++++++
generator/Makefile.am | 3 +
generator/actions.ml | 3 +-
generator/actions_yara.ml | 48 +++++++++++
generator/actions_yara.mli | 21 +++++
generator/proc_nr.ml | 1 +
lib/MAX_PROC_NR | 2 +-
10 files changed, 291 insertions(+), 2 deletions(-)
create mode 100644 daemon/yara.c
create mode 100644 generator/actions_yara.ml
create mode 100644 generator/actions_yara.mli
diff --git a/daemon/Makefile.am b/daemon/Makefile.am
index af4430f20..e4679a8c5 100644
--- a/daemon/Makefile.am
+++ b/daemon/Makefile.am
@@ -168,6 +168,7 @@ guestfsd_SOURCES = \
wc.c \
xattr.c \
xfs.c \
+ yara.c \
zero.c \
zerofree.c
diff --git a/daemon/cleanups.c b/daemon/cleanups.c
index 092e493d7..3102cf94b 100644
--- a/daemon/cleanups.c
+++ b/daemon/cleanups.c
@@ -62,6 +62,15 @@ cleanup_close (void *ptr)
}
void
+cleanup_fclose (void *ptr)
+{
+ FILE *f = * (FILE **) ptr;
+
+ if (f)
+ fclose (f);
+}
+
+void
cleanup_aug_close (void *ptr)
{
augeas *aug = * (augeas **) ptr;
diff --git a/daemon/cleanups.h b/daemon/cleanups.h
index 6746e2744..a791244cb 100644
--- a/daemon/cleanups.h
+++ b/daemon/cleanups.h
@@ -26,6 +26,7 @@ extern void cleanup_free (void *ptr);
extern void cleanup_free_string_list (void *ptr);
extern void cleanup_unlink_free (void *ptr);
extern void cleanup_close (void *ptr);
+extern void cleanup_fclose (void *ptr);
extern void cleanup_aug_close (void *ptr);
extern void cleanup_free_stringsbuf (void *ptr);
@@ -35,6 +36,7 @@ extern void cleanup_free_stringsbuf (void *ptr);
__attribute__((cleanup(cleanup_free_string_list)))
#define CLEANUP_UNLINK_FREE __attribute__((cleanup(cleanup_unlink_free)))
#define CLEANUP_CLOSE __attribute__((cleanup(cleanup_close)))
+#define CLEANUP_FCLOSE __attribute__((cleanup(cleanup_fclose)))
#define CLEANUP_AUG_CLOSE __attribute__((cleanup(cleanup_aug_close)))
#define CLEANUP_FREE_STRINGSBUF
__attribute__((cleanup(cleanup_free_stringsbuf)))
#else
diff --git a/daemon/yara.c b/daemon/yara.c
new file mode 100644
index 000000000..0d33d83cd
--- /dev/null
+++ b/daemon/yara.c
@@ -0,0 +1,203 @@
+/* libguestfs - the guestfsd daemon
+ * Copyright (C) 2016 Red Hat Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301
USA.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <fcntl.h>
+#include <stdlib.h>
+#include <inttypes.h>
+#include <string.h>
+#include <unistd.h>
+#include <stdbool.h>
+#include <rpc/xdr.h>
+#include <rpc/types.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+
+#include "daemon.h"
+#include "actions.h"
+#include "optgroups.h"
+#include "guestfs_protocol.h"
+
+#ifdef HAVE_YARA
+
+#include <yara.h>
+
+#define CLEANUP_DESTROY_YARA_COMPILER \
+ __attribute__((cleanup(cleanup_destroy_yara_compiler)))
+
+struct write_callback_data {
+ int fd;
+ uint64_t written;
+};
+
+/* Yara compiled rules. */
+static YR_RULES *rules = NULL;
+static bool initialized = false;
+
+static int compile_rules_file (const char *);
+static void compile_error_callback (int, const char *, int, const char *, void
*);
+static void cleanup_destroy_yara_compiler (void *ptr);
+
+/* Has one FileIn parameter.
+ * Takes optional arguments, consult optargs_bitmask.
+ */
+int
+do_yara_load (void)
+{
+ int ret = 0;
+ CLEANUP_CLOSE int fd = -1;
+ char tmpfile[] = "/tmp/yaraXXXXXX";
+
+ fd = mkstemp (tmpfile);
+ if (fd == -1) {
+ reply_with_perror ("mkstemp");
+ return -1;
+ }
+
+ ret = upload_to_fd (fd);
+ if (ret < 0) {
+ unlink (tmpfile);
+ return -1;
+ }
+
+ /* Initialize yara only once. */
+ if (!initialized) {
+ ret = yr_initialize ();
+ if (ret != ERROR_SUCCESS) {
+ reply_with_error ("failed initializing yara");
+ unlink (tmpfile);
+ return -1;
+ }
+
+ initialized = true;
+ }
+
+ /* Destroy previously loaded rules. */
+ if (rules != NULL) {
+ yr_rules_destroy (rules);
+ rules = NULL;
+ }
+
+ /* Try to load the rules as compiled.
+ * If their are in source code format, compile them first.
+ */
+ ret = yr_rules_load (tmpfile, &rules);
+ if (ret == ERROR_INVALID_FILE)
+ ret = compile_rules_file (tmpfile);
+
+ unlink (tmpfile);
+
+ return (ret == ERROR_SUCCESS) ? 0 : -1;
+}
+
+/* Compile source code rules and load them.
+ * Return ERROR_SUCCESS on success, Yara error code type on error.
+ */
+static int
+compile_rules_file (const char *rules_path)
+{
+ int ret = 0;
+ CLEANUP_FCLOSE FILE *rule_file = NULL;
+ CLEANUP_DESTROY_YARA_COMPILER YR_COMPILER *compiler = NULL;
+
+ ret = yr_compiler_create (&compiler);
+ if (ret != ERROR_SUCCESS) {
+ reply_with_error ("yr_compiler_create");
+ return ret;
+ }
+
+ yr_compiler_set_callback (compiler, compile_error_callback, NULL);
+
+ rule_file = fopen (rules_path, "r");
+ if (rule_file == NULL) {
+ reply_with_error ("unable to open rules file");
+ return ret;
+ }
+
+ ret = yr_compiler_add_file (compiler, rule_file, NULL, NULL);
+ if (ret > 0)
+ return ret;
+
+ ret = yr_compiler_get_rules (compiler, &rules);
+ if (ret == ERROR_INSUFICIENT_MEMORY) {
+ errno = ENOMEM;
+ reply_with_perror ("yr_compiler_get_rules");
+ }
+
+ return ret;
+}
+
+/* Yara compilation error callback.
+ * Reports back the compilation error message.
+ * Prints compilation warnings if verbose.
+ */
+static void
+compile_error_callback(int level, const char *name, int line,
+ const char *message, void *data)
+{
+ if (level == YARA_ERROR_LEVEL_ERROR)
+ reply_with_error ("Yara error (line %d): %s", line, message);
+ else if (verbose)
+ fprintf (stderr, "Yara warning (line %d): %s\n", line, message);
+}
+
+/* Clean up yara handle on daemon exit. */
+void yara_finalize (void) __attribute__((destructor));
+
+void
+yara_finalize (void)
+{
+ int ret = 0;
+
+ if (!initialized)
+ return;
+
+ if (rules != NULL) {
+ yr_rules_destroy (rules);
+ rules = NULL;
+ }
+
+ ret = yr_finalize ();
+ if (ret != ERROR_SUCCESS)
+ perror ("yr_finalize");
+
+ initialized = false;
+}
+
+static void
+cleanup_destroy_yara_compiler (void *ptr)
+{
+ YR_COMPILER *compiler = * (YR_COMPILER **) ptr;
+
+ if (compiler != NULL)
+ yr_compiler_destroy (compiler);
+}
+
+int
+optgroup_libyara_available (void)
+{
+ return 1;
+}
+
+#else /* !HAVE_YARA */
+
+OPTGROUP_LIBYARA_NOT_AVAILABLE
+
+#endif /* !HAVE_YARA */
diff --git a/generator/Makefile.am b/generator/Makefile.am
index 31989cd4a..81b49cab1 100644
--- a/generator/Makefile.am
+++ b/generator/Makefile.am
@@ -45,6 +45,8 @@ sources = \
actions_tsk.mli \
authors.ml \
authors.mli \
+ actions_yara.ml \
+ actions_yara.mli \
bindtests.ml \
bindtests.mli \
c.ml \
@@ -132,6 +134,7 @@ objects = \
actions_properties.cmo \
actions_properties_deprecated.cmo \
actions_tsk.cmo \
+ actions_yara.cmo \
actions.cmo \
structs.cmo \
fish_commands.cmo \
diff --git a/generator/actions.ml b/generator/actions.ml
index fa0e6568a..4df3b2a32 100644
--- a/generator/actions.ml
+++ b/generator/actions.ml
@@ -50,7 +50,8 @@ let daemon_functions
Actions_core_deprecated.daemon_functions @
Actions_debug.daemon_functions @
Actions_hivex.daemon_functions @
- Actions_tsk.daemon_functions
+ Actions_tsk.daemon_functions @
+ Actions_yara.daemon_functions
(* Some post-processing of the basic lists of actions. *)
diff --git a/generator/actions_yara.ml b/generator/actions_yara.ml
new file mode 100644
index 000000000..3e55206ec
--- /dev/null
+++ b/generator/actions_yara.ml
@@ -0,0 +1,48 @@
+(* libguestfs
+ * Copyright (C) 2009-2017 Red Hat Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ *)
+
+(* Please read generator/README first. *)
+
+open Types
+
+(* Yara APIs. *)
+
+let daemon_functions = [
+ { defaults with
+ name = "yara_load"; added = (1, 37, 9);
+ style = RErr, [FileIn "filename"], [];
+ progress = true; cancellable = true;
+ optional = Some "libyara";
+ shortdesc = "load yara rules within libguestfs";
+ longdesc = "\
+Load a set of Yara rules from F<filename> within libguestfs appliance.
+
+Yara rules allow to categorize files based on textual or binary patterns
+within their content.
+See C<yara_scan> to see how to scan files with the loaded rules.
+
+Rules can be in binary format, as when compiled with yarac command, or
+in source code format. In the latter case, the rules will be first
+compiled and then loaded.
+
+Rules in source code format cannot include external files. In such cases,
+it is recommended to compile them first.
+
+Previously loaded rules will be destroyed." };
+
+]
diff --git a/generator/actions_yara.mli b/generator/actions_yara.mli
new file mode 100644
index 000000000..23eefe068
--- /dev/null
+++ b/generator/actions_yara.mli
@@ -0,0 +1,21 @@
+(* libguestfs
+ * Copyright (C) 2009-2017 Red Hat Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ *)
+
+(* Please read generator/README first. *)
+
+val daemon_functions : Types.action list
diff --git a/generator/proc_nr.ml b/generator/proc_nr.ml
index b3be31996..d50cc9efa 100644
--- a/generator/proc_nr.ml
+++ b/generator/proc_nr.ml
@@ -479,6 +479,7 @@ let proc_nr = [
469, "aug_transform";
470, "internal_find_inode";
471, "mksquashfs";
+472, "yara_load";
]
(* End of list. If adding a new entry, add it at the end of the list
diff --git a/lib/MAX_PROC_NR b/lib/MAX_PROC_NR
index c305aa5ae..68cfb10d1 100644
--- a/lib/MAX_PROC_NR
+++ b/lib/MAX_PROC_NR
@@ -1 +1 @@
-471
+472
--
2.11.0
The yara_destroy API allows to claim resources back via the removal of
the previously loaded Yara rules.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
---
daemon/yara.c | 14 ++++++++++++++
generator/actions_yara.ml | 8 ++++++++
generator/proc_nr.ml | 1 +
lib/MAX_PROC_NR | 2 +-
4 files changed, 24 insertions(+), 1 deletion(-)
diff --git a/daemon/yara.c b/daemon/yara.c
index 0d33d83cd..186a330c1 100644
--- a/daemon/yara.c
+++ b/daemon/yara.c
@@ -107,6 +107,20 @@ do_yara_load (void)
return (ret == ERROR_SUCCESS) ? 0 : -1;
}
+int
+do_yara_destroy (void)
+{
+ if (rules == NULL) {
+ reply_with_error ("no yara rules loaded");
+ return -1;
+ }
+
+ yr_rules_destroy (rules);
+ rules = NULL;
+
+ return 0;
+}
+
/* Compile source code rules and load them.
* Return ERROR_SUCCESS on success, Yara error code type on error.
*/
diff --git a/generator/actions_yara.ml b/generator/actions_yara.ml
index 3e55206ec..9d93d9f11 100644
--- a/generator/actions_yara.ml
+++ b/generator/actions_yara.ml
@@ -45,4 +45,12 @@ it is recommended to compile them first.
Previously loaded rules will be destroyed." };
+ { defaults with
+ name = "yara_destroy"; added = (1, 37, 9);
+ style = RErr, [], [];
+ optional = Some "libyara";
+ shortdesc = "destroy previously loaded yara rules";
+ longdesc = "\
+Destroy previously loaded Yara rules in order to free libguestfs
resources." };
+
]
diff --git a/generator/proc_nr.ml b/generator/proc_nr.ml
index d50cc9efa..d471b1a83 100644
--- a/generator/proc_nr.ml
+++ b/generator/proc_nr.ml
@@ -480,6 +480,7 @@ let proc_nr = [
470, "internal_find_inode";
471, "mksquashfs";
472, "yara_load";
+473, "yara_destroy";
]
(* End of list. If adding a new entry, add it at the end of the list
diff --git a/lib/MAX_PROC_NR b/lib/MAX_PROC_NR
index 68cfb10d1..8410b8b89 100644
--- a/lib/MAX_PROC_NR
+++ b/lib/MAX_PROC_NR
@@ -1 +1 @@
-472
+473
--
2.11.0
Matteo Cafasso
2017-Apr-06 20:41 UTC
[Libguestfs] [PATCH v6 5/7] New API: internal_yara_scan
The internal_yara_scan runs the Yara engine with the previously loaded
rules against the given file.
For each rule matching against the scanned file, a struct containing
the file name and the rule identifier is returned.
The gathered list of yara_detection structs is serialised into XDR format
and written to a file.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
---
daemon/yara.c | 86 ++++++++++++++++++++++++++++++++
generator/actions_yara.ml | 8 +++
generator/proc_nr.ml | 1 +
generator/structs.ml | 9 ++++
gobject/Makefile.inc | 2 +
java/Makefile.inc | 1 +
java/com/redhat/et/libguestfs/.gitignore | 1 +
lib/MAX_PROC_NR | 2 +-
8 files changed, 109 insertions(+), 1 deletion(-)
diff --git a/daemon/yara.c b/daemon/yara.c
index 186a330c1..80e3f692b 100644
--- a/daemon/yara.c
+++ b/daemon/yara.c
@@ -54,6 +54,8 @@ static bool initialized = false;
static int compile_rules_file (const char *);
static void compile_error_callback (int, const char *, int, const char *, void
*);
static void cleanup_destroy_yara_compiler (void *ptr);
+static int yara_rules_callback (int , void *, void *);
+static int send_detection_info (const char *, YR_RULE *);
/* Has one FileIn parameter.
* Takes optional arguments, consult optargs_bitmask.
@@ -121,6 +123,38 @@ do_yara_destroy (void)
return 0;
}
+/* Has one FileOut parameter. */
+int
+do_internal_yara_scan (const char *path)
+{
+ int ret = 0;
+ CLEANUP_CLOSE int fd = -1;
+
+ if (rules == NULL) {
+ reply_with_error ("no yara rules loaded");
+ return -1;
+ }
+
+ CHROOT_IN;
+ fd = open (path, O_RDONLY|O_CLOEXEC);
+ CHROOT_OUT;
+
+ if (fd < 0) {
+ reply_with_perror ("%s", path);
+ return -1;
+ }
+
+ reply (NULL, NULL); /* Reply message. */
+
+ ret = yr_rules_scan_fd (rules, fd, 0, yara_rules_callback, (void *) path, 0);
+ if (ret == ERROR_SUCCESS)
+ ret = send_file_end (0); /* File transfer end. */
+ else
+ send_file_end (1); /* Cancel file transfer. */
+
+ return 0;
+}
+
/* Compile source code rules and load them.
* Return ERROR_SUCCESS on success, Yara error code type on error.
*/
@@ -172,6 +206,58 @@ compile_error_callback(int level, const char *name, int
line,
fprintf (stderr, "Yara warning (line %d): %s\n", line, message);
}
+/* Yara scan callback, called by yr_rules_scan_file.
+ * Return 0 on success, -1 on error.
+ */
+static int
+yara_rules_callback (int code, void *message, void *data)
+{
+ int ret = 0;
+
+ if (code == CALLBACK_MSG_RULE_MATCHING)
+ ret = send_detection_info ((const char *)data, (YR_RULE *) message);
+
+ return (ret == 0) ? CALLBACK_CONTINUE : CALLBACK_ERROR;
+}
+
+/* Serialize file path and rule name and send it out.
+ * Return 0 on success, -1 on error.
+ */
+static int
+send_detection_info (const char *name, YR_RULE *rule)
+{
+ XDR xdr;
+ int ret = 0;
+ size_t len = 0;
+ CLEANUP_FREE char *buf = NULL;
+ struct guestfs_int_yara_detection detection;
+
+ detection.yara_name = (char *) name;
+ detection.yara_rule = (char *) rule->identifier;
+
+ /* Serialize detection struct. */
+ buf = malloc (GUESTFS_MAX_CHUNK_SIZE);
+ if (buf == NULL) {
+ perror ("malloc");
+ return -1;
+ }
+
+ xdrmem_create (&xdr, buf, GUESTFS_MAX_CHUNK_SIZE, XDR_ENCODE);
+
+ ret = xdr_guestfs_int_yara_detection (&xdr, &detection);
+ if (ret == 0) {
+ perror ("xdr_guestfs_int_yara_detection");
+ return -1;
+ }
+
+ len = xdr_getpos (&xdr);
+
+ xdr_destroy (&xdr);
+
+ /* Send serialised yara_detection out. */
+ return send_file_write (buf, len);
+}
+
/* Clean up yara handle on daemon exit. */
void yara_finalize (void) __attribute__((destructor));
diff --git a/generator/actions_yara.ml b/generator/actions_yara.ml
index 9d93d9f11..2166d6f0a 100644
--- a/generator/actions_yara.ml
+++ b/generator/actions_yara.ml
@@ -53,4 +53,12 @@ Previously loaded rules will be destroyed." };
longdesc = "\
Destroy previously loaded Yara rules in order to free libguestfs
resources." };
+ { defaults with
+ name = "internal_yara_scan"; added = (1, 37, 9);
+ style = RErr, [Pathname "path"; FileOut "filename"],
[];
+ visibility = VInternal;
+ optional = Some "libyara";
+ shortdesc = "scan a file with the loaded yara rules";
+ longdesc = "Internal function for yara_scan." };
+
]
diff --git a/generator/proc_nr.ml b/generator/proc_nr.ml
index d471b1a83..c7619638a 100644
--- a/generator/proc_nr.ml
+++ b/generator/proc_nr.ml
@@ -481,6 +481,7 @@ let proc_nr = [
471, "mksquashfs";
472, "yara_load";
473, "yara_destroy";
+474, "internal_yara_scan";
]
(* End of list. If adding a new entry, add it at the end of the list
diff --git a/generator/structs.ml b/generator/structs.ml
index c1c9b668e..834fa9c54 100644
--- a/generator/structs.ml
+++ b/generator/structs.ml
@@ -469,6 +469,15 @@ let structs = [
];
s_camel_name = "TSKDirent" };
+ (* Yara detection information. *)
+ { defaults with
+ s_name = "yara_detection";
+ s_cols = [
+ "yara_name", FString;
+ "yara_rule", FString;
+ ];
+ s_camel_name = "YaraDetection" };
+
] (* end of structs *)
let lookup_struct name diff --git a/gobject/Makefile.inc b/gobject/Makefile.inc
index 8fa8599d3..a6dcba022 100644
--- a/gobject/Makefile.inc
+++ b/gobject/Makefile.inc
@@ -49,6 +49,7 @@ guestfs_gobject_headers= \
include/guestfs-gobject/struct-version.h \
include/guestfs-gobject/struct-xattr.h \
include/guestfs-gobject/struct-xfsinfo.h \
+ include/guestfs-gobject/struct-yara_detection.h \
include/guestfs-gobject/optargs-add_domain.h \
include/guestfs-gobject/optargs-add_drive.h \
include/guestfs-gobject/optargs-add_drive_scratch.h \
@@ -141,6 +142,7 @@ guestfs_gobject_sources= \
src/struct-version.c \
src/struct-xattr.c \
src/struct-xfsinfo.c \
+ src/struct-yara_detection.c \
src/optargs-add_domain.c \
src/optargs-add_drive.c \
src/optargs-add_drive_scratch.c \
diff --git a/java/Makefile.inc b/java/Makefile.inc
index b0ddb5c3d..3a202cee0 100644
--- a/java/Makefile.inc
+++ b/java/Makefile.inc
@@ -46,4 +46,5 @@ java_built_sources = \
com/redhat/et/libguestfs/Version.java \
com/redhat/et/libguestfs/XAttr.java \
com/redhat/et/libguestfs/XFSInfo.java \
+ com/redhat/et/libguestfs/YaraDetection.java \
com/redhat/et/libguestfs/GuestFS.java
diff --git a/java/com/redhat/et/libguestfs/.gitignore
b/java/com/redhat/et/libguestfs/.gitignore
index 89d923949..bc03cb965 100644
--- a/java/com/redhat/et/libguestfs/.gitignore
+++ b/java/com/redhat/et/libguestfs/.gitignore
@@ -23,3 +23,4 @@ VG.java
Version.java
XAttr.java
XFSInfo.java
+YaraDetection.java
diff --git a/lib/MAX_PROC_NR b/lib/MAX_PROC_NR
index 8410b8b89..5f3bb9813 100644
--- a/lib/MAX_PROC_NR
+++ b/lib/MAX_PROC_NR
@@ -1 +1 @@
-473
+474
--
2.11.0
The yara_scan API parses the file generated by the daemon counterpart
function and returns the list of yara_detection structs to the user.
It writes the daemon's command output on a temporary file and parses it,
deserialising the XDR formatted yara_detection structs.
It returns to the caller the list of yara_detection structs generated by
the internal_yara_scan command.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
---
generator/actions.ml | 3 +-
generator/actions_yara.ml | 28 ++++++++++
generator/actions_yara.mli | 1 +
lib/Makefile.am | 1 +
lib/yara.c | 127 +++++++++++++++++++++++++++++++++++++++++++++
5 files changed, 159 insertions(+), 1 deletion(-)
create mode 100644 lib/yara.c
diff --git a/generator/actions.ml b/generator/actions.ml
index 4df3b2a32..d36a4f8a9 100644
--- a/generator/actions.ml
+++ b/generator/actions.ml
@@ -38,7 +38,8 @@ let non_daemon_functions
Actions_inspection_deprecated.non_daemon_functions @
Actions_properties.non_daemon_functions @
Actions_properties_deprecated.non_daemon_functions @
- Actions_tsk.non_daemon_functions
+ Actions_tsk.non_daemon_functions @
+ Actions_yara.non_daemon_functions
(* daemon_functions are any functions which cause some action
* to take place in the daemon.
diff --git a/generator/actions_yara.ml b/generator/actions_yara.ml
index 2166d6f0a..deb4190a9 100644
--- a/generator/actions_yara.ml
+++ b/generator/actions_yara.ml
@@ -22,6 +22,34 @@ open Types
(* Yara APIs. *)
+let non_daemon_functions = [
+ { defaults with
+ name = "yara_scan"; added = (1, 37, 9);
+ style = RStructList ("detections", "yara_detection"),
[Pathname "path"], [];
+ optional = Some "libyara";
+ progress = true; cancellable = true;
+ shortdesc = "scan a file with the loaded yara rules";
+ longdesc = "\
+Scan a file with the previously loaded Yara rules.
+
+For each matching rule, a C<yara_detection> structure is returned.
+
+The C<yara_detection> structure contains the following fields.
+
+=over 4
+
+=item 'yara_name'
+
+Path of the file matching a Yara rule.
+
+=item 'yara_rule'
+
+Identifier of the Yara rule which matched against the given file.
+
+=back" };
+
+]
+
let daemon_functions = [
{ defaults with
name = "yara_load"; added = (1, 37, 9);
diff --git a/generator/actions_yara.mli b/generator/actions_yara.mli
index 23eefe068..06b8116c4 100644
--- a/generator/actions_yara.mli
+++ b/generator/actions_yara.mli
@@ -18,4 +18,5 @@
(* Please read generator/README first. *)
+val non_daemon_functions : Types.action list
val daemon_functions : Types.action list
diff --git a/lib/Makefile.am b/lib/Makefile.am
index 063706f8f..fc55c2dcf 100644
--- a/lib/Makefile.am
+++ b/lib/Makefile.am
@@ -127,6 +127,7 @@ libguestfs_la_SOURCES = \
wait.c \
whole-file.c \
version.c \
+ yara.c \
libguestfs.syms
libguestfs_la_CPPFLAGS = \
diff --git a/lib/yara.c b/lib/yara.c
new file mode 100644
index 000000000..864766e7a
--- /dev/null
+++ b/lib/yara.c
@@ -0,0 +1,127 @@
+/* libguestfs
+ * Copyright (C) 2016 Red Hat Inc.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <string.h>
+#include <rpc/xdr.h>
+#include <rpc/types.h>
+
+#include "guestfs.h"
+#include "guestfs_protocol.h"
+#include "guestfs-internal.h"
+#include "guestfs-internal-all.h"
+#include "guestfs-internal-actions.h"
+
+static struct guestfs_yara_detection_list *parse_yara_detection_file (guestfs_h
*, const char *);
+static int deserialise_yara_detection_list (guestfs_h *, FILE *, struct
guestfs_yara_detection_list *);
+
+struct guestfs_yara_detection_list *
+guestfs_impl_yara_scan (guestfs_h *g, const char *path)
+{
+ int ret = 0;
+ CLEANUP_UNLINK_FREE char *tmpfile = NULL;
+
+ tmpfile = guestfs_int_make_temp_path (g, "yara_scan");
+ if (tmpfile == NULL)
+ return NULL;
+
+ ret = guestfs_internal_yara_scan (g, path, tmpfile);
+ if (ret < 0)
+ return NULL;
+
+ return parse_yara_detection_file (g, tmpfile); /* caller frees */
+}
+
+/* Parse the file content and return detections list.
+ * Return a list of yara_detection on success, NULL on error.
+ */
+static struct guestfs_yara_detection_list *
+parse_yara_detection_file (guestfs_h *g, const char *tmpfile)
+{
+ int ret = 0;
+ CLEANUP_FCLOSE FILE *fp = NULL;
+ struct guestfs_yara_detection_list *detections = NULL;
+
+ fp = fopen (tmpfile, "r");
+ if (fp == NULL) {
+ perrorf (g, "fopen: %s", tmpfile);
+ return NULL;
+ }
+
+ /* Initialise results array. */
+ detections = safe_malloc (g, sizeof (*detections));
+ detections->len = 8;
+ detections->val = safe_malloc (g, detections->len *
+ sizeof (*detections->val));
+
+ /* Deserialise buffer into detection list. */
+ ret = deserialise_yara_detection_list (g, fp, detections);
+ if (ret < 0) {
+ guestfs_free_yara_detection_list (detections);
+ return NULL;
+ }
+
+ return detections;
+}
+
+/* Deserialise the file content and populate the detection list.
+ * Return the number of deserialised detections, -1 on error.
+ */
+static int
+deserialise_yara_detection_list (guestfs_h *g, FILE *fp,
+ struct guestfs_yara_detection_list
*detections)
+{
+ XDR xdr;
+ int ret = 0;
+ uint32_t index = 0;
+ struct stat statbuf;
+
+ ret = fstat (fileno(fp), &statbuf);
+ if (ret == -1)
+ return -1;
+
+ xdrstdio_create (&xdr, fp, XDR_DECODE);
+
+ for (index = 0; xdr_getpos (&xdr) < statbuf.st_size; index++) {
+ if (index == detections->len) {
+ detections->len = 2 * detections->len;
+ detections->val = safe_realloc (g, detections->val,
+ detections->len *
+ sizeof (*detections->val));
+ }
+
+ /* Clear the entry so xdr logic will allocate necessary memory. */
+ memset (&detections->val[index], 0, sizeof (*detections->val));
+ ret = xdr_guestfs_int_yara_detection (&xdr, (guestfs_int_yara_detection
*)
+ &detections->val[index]);
+ if (ret == 0)
+ break;
+ }
+
+ xdr_destroy (&xdr);
+ detections->len = index;
+
+ return ret ? 0 : -1;
+}
--
2.11.0
Matteo Cafasso
2017-Apr-06 20:41 UTC
[Libguestfs] [PATCH v6 PATCH 7/7] yara_scan: added API tests
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
---
configure.ac | 1 +
tests/yara/Makefile.am | 26 +++++++++++++++++++
tests/yara/test-yara-scan.sh | 61 ++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 88 insertions(+)
create mode 100644 tests/yara/Makefile.am
create mode 100755 tests/yara/test-yara-scan.sh
diff --git a/configure.ac b/configure.ac
index a23416bc0..b2bdada00 100644
--- a/configure.ac
+++ b/configure.ac
@@ -296,6 +296,7 @@ AC_CONFIG_FILES([Makefile
tests/tsk/Makefile
tests/xfs/Makefile
tests/xml/Makefile
+ tests/yara/Makefile
tools/Makefile
utils/boot-analysis/Makefile
utils/boot-benchmark/Makefile
diff --git a/tests/yara/Makefile.am b/tests/yara/Makefile.am
new file mode 100644
index 000000000..e23d94e4c
--- /dev/null
+++ b/tests/yara/Makefile.am
@@ -0,0 +1,26 @@
+# libguestfs
+# Copyright (C) 2016 Red Hat Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+include $(top_srcdir)/subdir-rules.mk
+
+TESTS = \
+ test-yara-scan.sh
+
+TESTS_ENVIRONMENT = $(top_builddir)/run --test
+
+EXTRA_DIST = \
+ $(TESTS)
diff --git a/tests/yara/test-yara-scan.sh b/tests/yara/test-yara-scan.sh
new file mode 100755
index 000000000..501c459e4
--- /dev/null
+++ b/tests/yara/test-yara-scan.sh
@@ -0,0 +1,61 @@
+#!/bin/bash -
+# libguestfs
+# Copyright (C) 2016 Red Hat Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+# Test the yara_scan command.
+
+set -e
+
+$TEST_FUNCTIONS
+skip_if_skipped
+skip_unless_feature_available sleuthkit
+skip_unless_phony_guest blank-fs.img
+
+rm -f test-yara-rules.yar
+
+/bin/cat << EOF > test-yara-rules.yar
+rule TestRule
+{
+ strings:
+ \$my_text_string = "some text"
+
+ condition:
+ \$my_text_string
+}
+EOF
+
+output=$(
+guestfish --ro -a ../../test-data/phony-guests/blank-fs.img <<EOF
+run
+mount /dev/sda1 /
+write /text.txt "some text"
+yara-load test-yara-rules.yar
+yara-scan /text.txt
+umount /
+yara-destroy
+EOF
+)
+
+echo $output | grep -zq '{ yara_name: /text.txt yara_rule: TestRule }'
+if [ $? != 0 ]; then
+ echo "$0: TestRule not found in detections list."
+ echo "Detections list:"
+ echo $output
+ exit 1
+fi
+
+rm -f test-yara-rules.yar
--
2.11.0
Richard W.M. Jones
2017-Apr-18 07:53 UTC
Re: [Libguestfs] [PATCH v6 1/7] daemon: expose file upload logic
On Thu, Apr 06, 2017 at 11:41:01PM +0300, Matteo Cafasso wrote:> Allows other modules to use the same logic for uploading files. > > Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> > --- > daemon/daemon.h | 3 +++ > daemon/upload.c | 70 ++++++++++++++++++++++++++++++++------------------------- > 2 files changed, 42 insertions(+), 31 deletions(-) > > diff --git a/daemon/daemon.h b/daemon/daemon.h > index abec087cd..797ec2dd9 100644 > --- a/daemon/daemon.h > +++ b/daemon/daemon.h > @@ -259,6 +259,9 @@ extern int64_t ntfs_minimum_size (const char *device); > extern int swap_set_uuid (const char *device, const char *uuid); > extern int swap_set_label (const char *device, const char *label); > > +/*-- in upload.c --*/ > +extern int upload_to_fd (int fd); > + > /* ordinary daemon functions use these to indicate errors > * NB: you don't need to prefix the string with the current command, > * it is added automatically by the client-side RPC stubs. > diff --git a/daemon/upload.c b/daemon/upload.c > index 655baf29d..144bb246c 100644 > --- a/daemon/upload.c > +++ b/daemon/upload.c > @@ -54,60 +54,68 @@ write_cb (void *data_vp, const void *buf, size_t len) > return 0; > } > > +int > +upload_to_fd (int fd) > +{ > + int ret = 0, err = 0;Let's use 'r' instead of 'ret', since it's the same as what the previous code used and you're not actually returning 'ret'.> + struct write_cb_data data = { .fd = fd, .written = 0 }; > + > + ret = receive_file (write_cb, &data); > + if (ret == -1) { /* write error */ > + err = errno; > + ret = cancel_receive (); > + errno = err; > + reply_with_error ("write error");Compared to the old code:> - reply_with_error ("write error: %s", filename);this loses the filename from the error message. Pass the file name to upload_to_fd to avoid losing information. ...> - r = cancel_receive (); > + cancel_receive (); > errno = err; > reply_with_perror ("%s", filename); > return -1;and> err = errno; > - r = cancel_receive (); > + cancel_receive (); > errno = err; > reply_with_perror ("lseek: %s", filename); > return -1;These changes are a separate bug, so they should be in a separate commit. More importantly than that, you should use ignore_value (...) around cancel_receive, else Coverity will complain that the result is not used. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
Richard W.M. Jones
2017-Apr-18 07:54 UTC
Re: [Libguestfs] [PATCH v6 2/7] appliance: add yara dependency
On Thu, Apr 06, 2017 at 11:41:02PM +0300, Matteo Cafasso wrote:> libyara3 on Debian/Ubuntu > yara on SUSE/RedHatThis commit needs to also document the new dependency in docs/guestfs-building.pod Rich.> Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> > --- > appliance/packagelist.in | 4 ++++ > daemon/Makefile.am | 3 ++- > m4/guestfs_daemon.m4 | 14 ++++++++++++++ > 3 files changed, 20 insertions(+), 1 deletion(-) > > diff --git a/appliance/packagelist.in b/appliance/packagelist.in > index 5cf22768a..8846ce846 100644 > --- a/appliance/packagelist.in > +++ b/appliance/packagelist.in > @@ -52,6 +52,7 @@ ifelse(REDHAT,1, > vim-minimal > xz > yajl > + yara > zfs-fuse > ) > > @@ -86,6 +87,7 @@ dnl iproute has been renamed to iproute2 > libsystemd-journal0 > libtirpc1 > libyajl2 > + libyara3 > linux-image > dnl syslinux 'suggests' mtools, but in reality it's a hard dependency: > mtools > @@ -129,6 +131,7 @@ ifelse(ARCHLINUX,1, > vim > xz > yajl > + yara > ) > > ifelse(SUSE,1, > @@ -159,6 +162,7 @@ ifelse(SUSE,1, > systemd > vim > xz > + yara > ) > > ifelse(FRUGALWARE,1, > diff --git a/daemon/Makefile.am b/daemon/Makefile.am > index 8632c3768..af4430f20 100644 > --- a/daemon/Makefile.am > +++ b/daemon/Makefile.am > @@ -191,7 +191,8 @@ guestfsd_LDADD = \ > $(SERVENT_LIB) \ > $(PCRE_LIBS) \ > $(TSK_LIBS) \ > - $(RPC_LIBS) > + $(RPC_LIBS) \ > + $(YARA_LIBS) > > guestfsd_CPPFLAGS = \ > -I$(top_srcdir)/gnulib/lib \ > diff --git a/m4/guestfs_daemon.m4 b/m4/guestfs_daemon.m4 > index eb5a6d5cf..522cd5f0e 100644 > --- a/m4/guestfs_daemon.m4 > +++ b/m4/guestfs_daemon.m4 > @@ -126,3 +126,17 @@ AC_CHECK_LIB([tsk],[tsk_version_print],[ > AC_DEFINE([HAVE_LIBTSK], [1], [Define to 1 if The Sleuth Kit library (libtsk) is available.]) > ], []) > ],[AC_MSG_WARN([The Sleuth Kit library (libtsk) not found])]) > + > +dnl yara library (optional) > +PKG_CHECK_MODULES([YARA], [libyara],[ > + AC_SUBST([YARA_CFLAGS]) > + AC_SUBST([YARA_LIBS]) > + AC_DEFINE([HAVE_YARA],[1],[yara library found at compile time.]) > +],[ > + AC_CHECK_LIB([yara],[yr_initialize],[ > + AC_CHECK_HEADER([yara.h],[ > + AC_SUBST([YARA_LIBS], [-lyara]) > + AC_DEFINE([HAVE_YARA], [1], [Define to 1 if Yara library is available.]) > + ], []) > + ],[AC_MSG_WARN([Yara library not found])]) > +]) > -- > 2.11.0-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/
Richard W.M. Jones
2017-Apr-18 08:10 UTC
Re: [Libguestfs] [PATCH v6 3/7] New API: yara_load
On Thu, Apr 06, 2017 at 11:41:03PM +0300, Matteo Cafasso wrote:> +#include <yara.h> > + > +#define CLEANUP_DESTROY_YARA_COMPILER \ > + __attribute__((cleanup(cleanup_destroy_yara_compiler)))While we should probably get rid of HAVE_ATTRIBUTE_CLEANUP, while we still have it you need to use it here and provide the alternative for people who don't HAVE_ATTRIBUTE_CLEANUP.> +/* Has one FileIn parameter. > + * Takes optional arguments, consult optargs_bitmask. > + */ > +int > +do_yara_load (void) > +{ > + int ret = 0;You're not returning 'ret', so call it something else, eg. 'r'.> + CLEANUP_CLOSE int fd = -1; > + char tmpfile[] = "/tmp/yaraXXXXXX"; > + > + fd = mkstemp (tmpfile); > + if (fd == -1) { > + reply_with_perror ("mkstemp"); > + return -1; > + } > + > + ret = upload_to_fd (fd); > + if (ret < 0) {upload_to_fd returns 0 or -1, so only check for r == -1.> +static void > +compile_error_callback(int level, const char *name, int line, > + const char *message, void *data)Space before the opening parenthesis.> +let daemon_functions = [ > + { defaults with > + name = "yara_load"; added = (1, 37, 9); > + style = RErr, [FileIn "filename"], []; > + progress = true; cancellable = true; > + optional = Some "libyara"; > + shortdesc = "load yara rules within libguestfs"; > + longdesc = "\ > +Load a set of Yara rules from F<filename> within libguestfs appliance.This is still confusingly worded, but after examining the code I think I understand what you're trying to say. Just replace this first sentence with: "Upload a set of Yara rules from local file F<filename>.> +Yara rules allow to categorize files based on textual or binary patterns > +within their content. > +See C<yara_scan> to see how to scan files with the loaded rules.This should be: C<guestfs_yara_scan>. The generator will replace C<guestfs_ with the correct link. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
Richard W.M. Jones
2017-Apr-18 08:10 UTC
Re: [Libguestfs] [PATCH v6 4/7] New API: yara_destroy
On Thu, Apr 06, 2017 at 11:41:04PM +0300, Matteo Cafasso wrote:> The yara_destroy API allows to claim resources back via the removal of > the previously loaded Yara rules. > > Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>This one looks fine, ACK. Rich.> daemon/yara.c | 14 ++++++++++++++ > generator/actions_yara.ml | 8 ++++++++ > generator/proc_nr.ml | 1 + > lib/MAX_PROC_NR | 2 +- > 4 files changed, 24 insertions(+), 1 deletion(-) > > diff --git a/daemon/yara.c b/daemon/yara.c > index 0d33d83cd..186a330c1 100644 > --- a/daemon/yara.c > +++ b/daemon/yara.c > @@ -107,6 +107,20 @@ do_yara_load (void) > return (ret == ERROR_SUCCESS) ? 0 : -1; > } > > +int > +do_yara_destroy (void) > +{ > + if (rules == NULL) { > + reply_with_error ("no yara rules loaded"); > + return -1; > + } > + > + yr_rules_destroy (rules); > + rules = NULL; > + > + return 0; > +} > + > /* Compile source code rules and load them. > * Return ERROR_SUCCESS on success, Yara error code type on error. > */ > diff --git a/generator/actions_yara.ml b/generator/actions_yara.ml > index 3e55206ec..9d93d9f11 100644 > --- a/generator/actions_yara.ml > +++ b/generator/actions_yara.ml > @@ -45,4 +45,12 @@ it is recommended to compile them first. > > Previously loaded rules will be destroyed." }; > > + { defaults with > + name = "yara_destroy"; added = (1, 37, 9); > + style = RErr, [], []; > + optional = Some "libyara"; > + shortdesc = "destroy previously loaded yara rules"; > + longdesc = "\ > +Destroy previously loaded Yara rules in order to free libguestfs resources." }; > + > ] > diff --git a/generator/proc_nr.ml b/generator/proc_nr.ml > index d50cc9efa..d471b1a83 100644 > --- a/generator/proc_nr.ml > +++ b/generator/proc_nr.ml > @@ -480,6 +480,7 @@ let proc_nr = [ > 470, "internal_find_inode"; > 471, "mksquashfs"; > 472, "yara_load"; > +473, "yara_destroy"; > ] > > (* End of list. If adding a new entry, add it at the end of the list > diff --git a/lib/MAX_PROC_NR b/lib/MAX_PROC_NR > index 68cfb10d1..8410b8b89 100644 > --- a/lib/MAX_PROC_NR > +++ b/lib/MAX_PROC_NR > @@ -1 +1 @@ > -472 > +473 > -- > 2.11.0-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
Richard W.M. Jones
2017-Apr-18 08:13 UTC
Re: [Libguestfs] [PATCH v6 5/7] New API: internal_yara_scan
On Thu, Apr 06, 2017 at 11:41:05PM +0300, Matteo Cafasso wrote:> +/* Has one FileOut parameter. */ > +int > +do_internal_yara_scan (const char *path) > +{ > + int ret = 0;Not returning 'ret', so call it something else, eg. 'r'. There are further instances below. The rest of the commit looks fine, so ACK with the changes. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Richard W.M. Jones
2017-Apr-18 09:14 UTC
Re: [Libguestfs] [PATCH v6 6/7] New API: yara_scan
On Thu, Apr 06, 2017 at 11:41:06PM +0300, Matteo Cafasso wrote:> +let non_daemon_functions = [ > + { defaults with > + name = "yara_scan"; added = (1, 37, 9); > + style = RStructList ("detections", "yara_detection"), [Pathname "path"], []; > + optional = Some "libyara"; > + progress = true; cancellable = true; > + shortdesc = "scan a file with the loaded yara rules"; > + longdesc = "\ > +Scan a file with the previously loaded Yara rules. > + > +For each matching rule, a C<yara_detection> structure is returned. > + > +The C<yara_detection> structure contains the following fields. > + > +=over 4 > + > +=item 'yara_name'...> +=item 'yara_rule'You should use: =item C<yara_name>> +static struct guestfs_yara_detection_list *parse_yara_detection_file (guestfs_h *, const char *); > +static int deserialise_yara_detection_list (guestfs_h *, FILE *, struct guestfs_yara_detection_list *); > + > +struct guestfs_yara_detection_list * > +guestfs_impl_yara_scan (guestfs_h *g, const char *path) > +{ > + int ret = 0;s/ret/r/> + CLEANUP_UNLINK_FREE char *tmpfile = NULL; > + > + tmpfile = guestfs_int_make_temp_path (g, "yara_scan"); > + if (tmpfile == NULL) > + return NULL; > + > + ret = guestfs_internal_yara_scan (g, path, tmpfile); > + if (ret < 0) > + return NULL;This function returns 0 or -1, so only need to check for r == -1.> +/* Parse the file content and return detections list. > + * Return a list of yara_detection on success, NULL on error. > + */ > +static struct guestfs_yara_detection_list * > +parse_yara_detection_file (guestfs_h *g, const char *tmpfile) > +{ > + int ret = 0;s/ret/r/> + CLEANUP_FCLOSE FILE *fp = NULL; > + struct guestfs_yara_detection_list *detections = NULL; > + > + fp = fopen (tmpfile, "r"); > + if (fp == NULL) { > + perrorf (g, "fopen: %s", tmpfile); > + return NULL; > + } > + > + /* Initialise results array. */ > + detections = safe_malloc (g, sizeof (*detections)); > + detections->len = 8; > + detections->val = safe_malloc (g, detections->len * > + sizeof (*detections->val)); > + > + /* Deserialise buffer into detection list. */ > + ret = deserialise_yara_detection_list (g, fp, detections); > + if (ret < 0) {This returns 0 or -1, so check r == -1.> + guestfs_free_yara_detection_list (detections); > + return NULL;There is no call to error/perrorf/etc along this path.> + } > + > + return detections; > +} > + > +/* Deserialise the file content and populate the detection list. > + * Return the number of deserialised detections, -1 on error. > + */ > +static int > +deserialise_yara_detection_list (guestfs_h *g, FILE *fp, > + struct guestfs_yara_detection_list *detections) > +{ > + XDR xdr; > + int ret = 0;s/ret/r/> + uint32_t index = 0; > + struct stat statbuf; > + > + ret = fstat (fileno(fp), &statbuf); > + if (ret == -1)No call to perrorf.> + return -1; > + > + xdrstdio_create (&xdr, fp, XDR_DECODE); > + > + for (index = 0; xdr_getpos (&xdr) < statbuf.st_size; index++) { > + if (index == detections->len) { > + detections->len = 2 * detections->len; > + detections->val = safe_realloc (g, detections->val, > + detections->len * > + sizeof (*detections->val)); > + } > + > + /* Clear the entry so xdr logic will allocate necessary memory. */ > + memset (&detections->val[index], 0, sizeof (*detections->val)); > + ret = xdr_guestfs_int_yara_detection (&xdr, (guestfs_int_yara_detection *) > + &detections->val[index]); > + if (ret == 0) > + break; > + } > + > + xdr_destroy (&xdr); > + detections->len = index; > + > + return ret ? 0 : -1;No call to error/perrorf along the error path. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
Richard W.M. Jones
2017-Apr-18 09:14 UTC
Re: [Libguestfs] [PATCH v6 PATCH 7/7] yara_scan: added API tests
This one looks OK. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/