Yara is a rule based scanning engine aimed to help malware analysts in finding and classifying interesting samples. https://github.com/VirusTotal/yara This series adds Yara support to Libguestfs allowing to upload sets of rules and scanning files against them. Currently provided APIs: - yara_load: loads a set of rules - yara_destroy: free resources allocated by loaded rules - yara_scan: scans a file with the loaded rules Future APIs: - yara_scan_recursive: scan the entire FS starting from the given point (could it be a flag in yara_scan?) - yara_scan_inode: use TSK to scan files by inodes allowing to scan deleted or hidden files Code ready for review, available here: https://github.com/noxdafox/libguestfs/tree/yara Matteo Cafasso (6): appliance: add libyara dependency New API: yara_load New API: yara_destroy New API: internal_yara_scan New API: yara_scan yara_scan: added API tests appliance/packagelist.in | 1 + configure.ac | 1 + daemon/Makefile.am | 4 +- daemon/yara.c | 318 +++++++++++++++++++++++++++++++ generator/actions.ml | 59 ++++++ generator/structs.ml | 9 + gobject/Makefile.inc | 2 + java/Makefile.inc | 1 + java/com/redhat/et/libguestfs/.gitignore | 1 + m4/guestfs_daemon.m4 | 8 + src/MAX_PROC_NR | 2 +- src/Makefile.am | 1 + src/yara.c | 140 ++++++++++++++ tests/yara/Makefile.am | 26 +++ tests/yara/test-yara-scan.sh | 72 +++++++ 15 files changed, 643 insertions(+), 2 deletions(-) create mode 100644 daemon/yara.c create mode 100644 src/yara.c create mode 100644 tests/yara/Makefile.am create mode 100755 tests/yara/test-yara-scan.sh -- 2.10.1
Matteo Cafasso
2016-Nov-02 19:26 UTC
[Libguestfs] [PATCH 1/6] appliance: add libyara dependency
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
---
appliance/packagelist.in | 1 +
daemon/Makefile.am | 3 ++-
m4/guestfs_daemon.m4 | 8 ++++++++
3 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/appliance/packagelist.in b/appliance/packagelist.in
index f278f66..5982df8 100644
--- a/appliance/packagelist.in
+++ b/appliance/packagelist.in
@@ -232,6 +232,7 @@ jfsutils
kmod
less
libxml2
+libyara3
lsof
lsscsi
lvm2
diff --git a/daemon/Makefile.am b/daemon/Makefile.am
index 23f60eb..3a25f43 100644
--- a/daemon/Makefile.am
+++ b/daemon/Makefile.am
@@ -222,7 +222,8 @@ guestfsd_LDADD = \
$(LIBINTL) \
$(SERVENT_LIB) \
$(PCRE_LIBS) \
- $(TSK_LIBS)
+ $(TSK_LIBS) \
+ $(YARA_LIBS)
guestfsd_CPPFLAGS = \
-I$(top_srcdir)/gnulib/lib \
diff --git a/m4/guestfs_daemon.m4 b/m4/guestfs_daemon.m4
index 12123df..9dc7c6c 100644
--- a/m4/guestfs_daemon.m4
+++ b/m4/guestfs_daemon.m4
@@ -126,3 +126,11 @@ AC_CHECK_LIB([tsk],[tsk_version_print],[
AC_DEFINE([HAVE_LIBTSK], [1], [Define to 1 if The Sleuth Kit library
(libtsk) is available.])
], [])
],[AC_MSG_WARN([The Sleuth Kit library (libtsk) not found])])
+
+dnl yara library (optional)
+AC_CHECK_LIB([yara],[yr_initialize],[
+ AC_CHECK_HEADER([yara.h],[
+ AC_SUBST([YARA_LIBS], [-lyara])
+ AC_DEFINE([HAVE_LIBYARA], [1], [Define to 1 if Yara library is
available.])
+ ], [])
+],[AC_MSG_WARN([Yara library not found])])
--
2.10.1
The yara_load API allows to load a set of Yara rules contained within a
file on the host.
Rules can be in binary format, as when compiled with yarac command, or
in source code format. In the latter case, the rules will be first
compiled and then loaded.
Subsequent calls of the yara_load API will result in the discard of the
previously loaded rules.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
---
daemon/Makefile.am | 1 +
daemon/yara.c | 205 +++++++++++++++++++++++++++++++++++++++++++++++++++
generator/actions.ml | 15 ++++
src/MAX_PROC_NR | 2 +-
4 files changed, 222 insertions(+), 1 deletion(-)
create mode 100644 daemon/yara.c
diff --git a/daemon/Makefile.am b/daemon/Makefile.am
index 3a25f43..c385edc 100644
--- a/daemon/Makefile.am
+++ b/daemon/Makefile.am
@@ -200,6 +200,7 @@ guestfsd_SOURCES = \
wc.c \
xattr.c \
xfs.c \
+ yara.c \
zero.c \
zerofree.c
diff --git a/daemon/yara.c b/daemon/yara.c
new file mode 100644
index 0000000..0d085a8
--- /dev/null
+++ b/daemon/yara.c
@@ -0,0 +1,205 @@
+/* libguestfs - the guestfsd daemon
+ * Copyright (C) 2016 Red Hat Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301
USA.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <fcntl.h>
+#include <stdlib.h>
+#include <inttypes.h>
+#include <string.h>
+#include <unistd.h>
+#include <rpc/xdr.h>
+#include <rpc/types.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+
+#include "daemon.h"
+#include "actions.h"
+#include "optgroups.h"
+#include "guestfs_protocol.h"
+
+#ifdef HAVE_LIBYARA
+
+#include <yara.h>
+
+struct write_callback_data {
+ int fd;
+ uint64_t written;
+};
+
+/* Yara compiled rules. */
+static YR_RULES *rules = NULL;
+
+static int upload_rules_file (char *);
+static int compile_rules_file (const char *);
+static int write_callback (void *, const void *, size_t);
+
+/* Has one FileIn parameter. */
+int
+do_yara_load (void)
+{
+ int ret = 0;
+ char tmpfile[] = "/tmp/yaraXXXXXX";
+
+ ret = upload_rules_file (tmpfile);
+ if (ret < 0)
+ return -1;
+
+ ret = yr_initialize ();
+ if (ret != ERROR_SUCCESS) {
+ reply_with_error ("failed initializing yara");
+ unlink (tmpfile);
+ return -1;
+ }
+
+ /* Destroy previously loaded rules. */
+ if (rules != NULL) {
+ yr_rules_destroy (rules);
+ rules = NULL;
+ }
+
+ /* Try to load the rules as compiled.
+ * If their are in source code format, compile them first.
+ */
+ ret = yr_rules_load (tmpfile, &rules);
+ if (ret == ERROR_INVALID_FILE)
+ ret = compile_rules_file (tmpfile);
+
+ unlink (tmpfile);
+
+ if (ret != ERROR_SUCCESS) {
+ reply_with_error ("failed loading the rules");
+ yr_finalize ();
+ return -1;
+ }
+
+ ret = yr_finalize ();
+ if (ret != ERROR_SUCCESS) {
+ reply_with_error ("failed finalizing yara");
+ return -1;
+ }
+
+ return 0;
+}
+
+/* Upload rules file on a temporary file.
+ * Return 0 on success, -1 on error.
+ */
+static int
+upload_rules_file (char *rules_path)
+{
+ int ret = 0;
+ CLEANUP_CLOSE int fd = 0;
+ struct write_callback_data data = { .written = 0 };
+
+ data.fd = mkstemp (rules_path);
+ if (data.fd == -1) {
+ reply_with_perror ("mkstemp");
+ return -1;
+ }
+
+ ret = receive_file (write_callback, &data.fd);
+ if (ret == -1) {
+ /* Write error. */
+ cancel_receive ();
+ reply_with_error ("write error: %s", rules_path);
+ return -1;
+ }
+ if (ret == -2) {
+ /* Cancellation from library */
+ reply_with_error ("file upload cancelled");
+ return -1;
+ }
+
+ return 0;
+}
+
+/* Compile source code rules and load them.
+ * Return ERROR_SUCCESS on success, Yara error code type on error.
+ */
+static int
+compile_rules_file (const char *rules_path)
+{
+ int ret = 0;
+ FILE *rule_file = NULL;
+ YR_COMPILER *compiler = NULL;
+
+ ret = yr_compiler_create (&compiler);
+ if (ret != ERROR_SUCCESS) {
+ perror ("yr_compiler_create");
+ return ret;
+ }
+
+ rule_file = fopen (rules_path, "r");
+ if (rule_file == NULL) {
+ yr_compiler_destroy (compiler);
+ perror ("unable to open rules file");
+ return ret;
+ }
+
+ ret = yr_compiler_add_file (compiler, rule_file, NULL, rules_path);
+ if (ret > 0) {
+ yr_compiler_destroy (compiler);
+ fclose (rule_file);
+ perror ("yr_compiler_add_file");
+ return ret;
+ }
+
+ ret = yr_compiler_get_rules (compiler, &rules);
+ yr_compiler_destroy (compiler);
+ fclose (rule_file);
+
+ if (ret != ERROR_SUCCESS)
+ perror ("yr_compiler_get_rules");
+
+ return ret;
+}
+
+/* File upload callback, called by receive_file.
+ * Return 0 on success, -1 on error.
+ */
+static int
+write_callback (void *data_vp, const void *buf, size_t len)
+{
+ int ret;
+ struct write_callback_data *data = data_vp;
+
+ ret = xwrite (data->fd, buf, len);
+ if (ret == -1)
+ return -1;
+
+ data->written += len;
+
+ if (progress_hint > 0)
+ notify_progress (data->written, progress_hint);
+
+ return 0;
+}
+
+int
+optgroup_libyara_available (void)
+{
+ return 1;
+}
+
+#else /* !HAVE_LIBYARA */
+
+OPTGROUP_LIBYARA_NOT_AVAILABLE
+
+#endif /* !HAVE_LIBYARA */
diff --git a/generator/actions.ml b/generator/actions.ml
index 91a1819..14659bd 100644
--- a/generator/actions.ml
+++ b/generator/actions.ml
@@ -13253,6 +13253,21 @@ is removed." };
shortdesc = "search the entries associated to the given inode";
longdesc = "Internal function for find_inode." };
+ { defaults with
+ name = "yara_load"; added = (1, 35, 15);
+ style = RErr, [FileIn "filename";], [];
+ proc_nr = Some 471;
+ progress = true; cancellable = true;
+ optional = Some "libyara";
+ shortdesc = "load yara rules within libguestfs";
+ longdesc = "\
+Load a set of Yara rules from F<filename> within libguestfs appliance.
+
+Rules can be both in source code and compiled form.
+Rules in source code format cannot include external files. In such cases,
+it is recommended to compile them first.
+
+Previously loaded rules will be destroyed." };
]
(* Non-API meta-commands available only in guestfish.
diff --git a/src/MAX_PROC_NR b/src/MAX_PROC_NR
index 5f476b6..c305aa5 100644
--- a/src/MAX_PROC_NR
+++ b/src/MAX_PROC_NR
@@ -1 +1 @@
-470
+471
--
2.10.1
The yara_destroy API allows to claim resources back via the removal of
the previously loaded Yara rules.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
---
daemon/yara.c | 14 ++++++++++++++
generator/actions.ml | 9 +++++++++
src/MAX_PROC_NR | 2 +-
3 files changed, 24 insertions(+), 1 deletion(-)
diff --git a/daemon/yara.c b/daemon/yara.c
index 0d085a8..a6b9fe2 100644
--- a/daemon/yara.c
+++ b/daemon/yara.c
@@ -98,6 +98,20 @@ do_yara_load (void)
return 0;
}
+int
+do_yara_destroy (void)
+{
+ if (rules == NULL) {
+ reply_with_error ("no yara rules loaded");
+ return -1;
+ }
+
+ yr_rules_destroy (rules);
+ rules = NULL;
+
+ return 0;
+}
+
/* Upload rules file on a temporary file.
* Return 0 on success, -1 on error.
*/
diff --git a/generator/actions.ml b/generator/actions.ml
index 14659bd..966e012 100644
--- a/generator/actions.ml
+++ b/generator/actions.ml
@@ -13268,6 +13268,15 @@ Rules in source code format cannot include external
files. In such cases,
it is recommended to compile them first.
Previously loaded rules will be destroyed." };
+
+ { defaults with
+ name = "yara_destroy"; added = (1, 35, 15);
+ style = RErr, [], [];
+ proc_nr = Some 472;
+ optional = Some "libyara";
+ shortdesc = "destroy previously loaded yara rules";
+ longdesc = "\
+Destroy previously loaded Yara rules in order to free libguestfs
resources." };
]
(* Non-API meta-commands available only in guestfish.
diff --git a/src/MAX_PROC_NR b/src/MAX_PROC_NR
index c305aa5..68cfb10 100644
--- a/src/MAX_PROC_NR
+++ b/src/MAX_PROC_NR
@@ -1 +1 @@
-471
+472
--
2.10.1
The internal_yara_scan runs the Yara engine with the previously loaded
rules against the given file.
For each rule matching against the scanned file, a struct containing
the file name and the rule identifier is returned.
The gathered list of yara_detection structs is serialised into XDR format
and written to a file.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
---
daemon/yara.c | 99 ++++++++++++++++++++++++++++++++
generator/actions.ml | 10 ++++
generator/structs.ml | 9 +++
gobject/Makefile.inc | 2 +
java/Makefile.inc | 1 +
java/com/redhat/et/libguestfs/.gitignore | 1 +
src/MAX_PROC_NR | 2 +-
7 files changed, 123 insertions(+), 1 deletion(-)
diff --git a/daemon/yara.c b/daemon/yara.c
index a6b9fe2..099401c 100644
--- a/daemon/yara.c
+++ b/daemon/yara.c
@@ -49,6 +49,8 @@ static YR_RULES *rules = NULL;
static int upload_rules_file (char *);
static int compile_rules_file (const char *);
static int write_callback (void *, const void *, size_t);
+static int yara_rules_callback (int , void *, void *);
+static int send_detection_info (const char *, YR_RULE *);
/* Has one FileIn parameter. */
int
@@ -112,6 +114,51 @@ do_yara_destroy (void)
return 0;
}
+/* Has one FileOut parameter. */
+int
+do_internal_yara_scan (const char *path)
+{
+ int ret = 0;
+ CLEANUP_CLOSE int fd = 0;
+
+ if (rules == NULL) {
+ reply_with_error ("no yara rules loaded");
+ return -1;
+ }
+
+ ret = yr_initialize ();
+ if (ret != ERROR_SUCCESS) {
+ reply_with_error ("failed initializing yara");
+ return -1;
+ }
+
+ CHROOT_IN;
+ fd = open (path, O_RDONLY|O_CLOEXEC);
+ CHROOT_OUT;
+
+ if (fd < 0) {
+ reply_with_perror ("%s", path);
+ yr_finalize ();
+ return -1;
+ }
+
+ reply (NULL, NULL); /* Reply message. */
+
+ ret = yr_rules_scan_fd (rules, fd, 0, yara_rules_callback, (void *) path, 0);
+ if (ret == ERROR_SUCCESS)
+ ret = send_file_end (0); /* File transfer end. */
+ else
+ send_file_end (1); /* Cancel file transfer. */
+
+ ret = yr_finalize ();
+ if (ret != ERROR_SUCCESS) {
+ reply_with_error ("failed finalizing yara");
+ return -1;
+ }
+
+ return 0;
+}
+
/* Upload rules file on a temporary file.
* Return 0 on success, -1 on error.
*/
@@ -206,6 +253,58 @@ write_callback (void *data_vp, const void *buf, size_t len)
return 0;
}
+/* Yara scan callback, called by yr_rules_scan_file.
+ * Return 0 on success, -1 on error.
+ */
+static int
+yara_rules_callback (int code, void *message, void *data)
+{
+ int ret = 0;
+
+ if (code == CALLBACK_MSG_RULE_MATCHING)
+ ret = send_detection_info ((const char *)data, (YR_RULE *) message);
+
+ return (ret == 0) ? CALLBACK_CONTINUE : CALLBACK_ERROR;
+}
+
+/* Serialize file path and rule name and send it out.
+ * Return 0 on success, -1 on error.
+ */
+static int
+send_detection_info (const char *name, YR_RULE *rule)
+{
+ XDR xdr;
+ int ret = 0;
+ size_t len = 0;
+ struct guestfs_int_yara_detection detection;
+ CLEANUP_FREE char *buf = NULL, *fname = NULL;
+
+ detection.name = (char *) name;
+ detection.rule = (char *) rule->identifier;
+
+ /* Serialize detection struct. */
+ buf = malloc (GUESTFS_MAX_CHUNK_SIZE);
+ if (buf == NULL) {
+ perror ("malloc");
+ return -1;
+ }
+
+ xdrmem_create (&xdr, buf, GUESTFS_MAX_CHUNK_SIZE, XDR_ENCODE);
+
+ ret = xdr_guestfs_int_yara_detection (&xdr, &detection);
+ if (ret == 0) {
+ perror ("xdr_guestfs_int_yara_detection");
+ return -1;
+ }
+
+ len = xdr_getpos (&xdr);
+
+ xdr_destroy (&xdr);
+
+ /* Send serialised tsk_detection out. */
+ return send_file_write (buf, len);
+}
+
int
optgroup_libyara_available (void)
{
diff --git a/generator/actions.ml b/generator/actions.ml
index 966e012..61211f4 100644
--- a/generator/actions.ml
+++ b/generator/actions.ml
@@ -13277,6 +13277,16 @@ Previously loaded rules will be destroyed." };
shortdesc = "destroy previously loaded yara rules";
longdesc = "\
Destroy previously loaded Yara rules in order to free libguestfs
resources." };
+
+ { defaults with
+ name = "internal_yara_scan"; added = (1, 35, 15);
+ style = RErr, [Pathname "path"; FileOut "filename";],
[];
+ proc_nr = Some 473;
+ visibility = VInternal;
+ optional = Some "libyara";
+ shortdesc = "scan a file with the loaded yara rules";
+ longdesc = "Internal function for yara_scan." };
+
]
(* Non-API meta-commands available only in guestfish.
diff --git a/generator/structs.ml b/generator/structs.ml
index 029bc3a..3fa2ebc 100644
--- a/generator/structs.ml
+++ b/generator/structs.ml
@@ -468,6 +468,15 @@ let structs = [
];
s_camel_name = "TSKDirent" };
+ (* Yara detection information. *)
+ { defaults with
+ s_name = "yara_detection";
+ s_cols = [
+ "name", FString;
+ "rule", FString;
+ ];
+ s_camel_name = "YaraDetection" };
+
] (* end of structs *)
let lookup_struct name diff --git a/gobject/Makefile.inc b/gobject/Makefile.inc
index 149e4c6..a784b62 100644
--- a/gobject/Makefile.inc
+++ b/gobject/Makefile.inc
@@ -49,6 +49,7 @@ guestfs_gobject_headers= \
include/guestfs-gobject/struct-version.h \
include/guestfs-gobject/struct-xattr.h \
include/guestfs-gobject/struct-xfsinfo.h \
+ include/guestfs-gobject/struct-yara_detection.h \
include/guestfs-gobject/optargs-add_domain.h \
include/guestfs-gobject/optargs-add_drive.h \
include/guestfs-gobject/optargs-add_drive_scratch.h \
@@ -140,6 +141,7 @@ guestfs_gobject_sources= \
src/struct-version.c \
src/struct-xattr.c \
src/struct-xfsinfo.c \
+ src/struct-yara_detection.c \
src/optargs-add_domain.c \
src/optargs-add_drive.c \
src/optargs-add_drive_scratch.c \
diff --git a/java/Makefile.inc b/java/Makefile.inc
index 59b55eb..acf2a2f 100644
--- a/java/Makefile.inc
+++ b/java/Makefile.inc
@@ -46,4 +46,5 @@ java_built_sources = \
com/redhat/et/libguestfs/Version.java \
com/redhat/et/libguestfs/XAttr.java \
com/redhat/et/libguestfs/XFSInfo.java \
+ com/redhat/et/libguestfs/YaraDetection.java \
com/redhat/et/libguestfs/GuestFS.java
diff --git a/java/com/redhat/et/libguestfs/.gitignore
b/java/com/redhat/et/libguestfs/.gitignore
index 89d9239..bc03cb9 100644
--- a/java/com/redhat/et/libguestfs/.gitignore
+++ b/java/com/redhat/et/libguestfs/.gitignore
@@ -23,3 +23,4 @@ VG.java
Version.java
XAttr.java
XFSInfo.java
+YaraDetection.java
diff --git a/src/MAX_PROC_NR b/src/MAX_PROC_NR
index 68cfb10..8410b8b 100644
--- a/src/MAX_PROC_NR
+++ b/src/MAX_PROC_NR
@@ -1 +1 @@
-472
+473
--
2.10.1
The yara_scan API parses the file generated by the daemon counterpart
function and returns the list of yara_detection structs to the user.
It writes the daemon's command output on a temporary file and parses it,
deserialising the XDR formatted yara_detection structs.
It returns to the caller the list of yara_detection structs generated by
the internal_yara_scan command.
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
---
generator/actions.ml | 25 +++++++++
src/Makefile.am | 1 +
src/yara.c | 140 +++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 166 insertions(+)
create mode 100644 src/yara.c
diff --git a/generator/actions.ml b/generator/actions.ml
index 61211f4..59a75fc 100644
--- a/generator/actions.ml
+++ b/generator/actions.ml
@@ -3729,6 +3729,31 @@ Searches all the entries associated with the given inode.
For each entry, a C<tsk_dirent> structure is returned.
See C<filesystem_walk> for more information about C<tsk_dirent>
structures." };
+ { defaults with
+ name = "yara_scan"; added = (1, 35, 15);
+ style = RStructList ("detections", "yara_detection"),
[Pathname "path";], [];
+ optional = Some "libyara";
+ progress = true; cancellable = true;
+ shortdesc = "scan a file with the loaded yara rules";
+ longdesc = "\
+Scan a file with the previously loaded Yara rules.
+
+For each matching rule, a C<yara_detection> structure is returned.
+
+The C<tsk_dirent> structure contains the following fields.
+
+=over 4
+
+=item 'name'
+
+Path of the file matching a Yara rule.
+
+=item 'rule'
+
+Identifier of the Yara rule which matched against the given file.
+
+=back" };
+
]
(* daemon_functions are any functions which cause some action
diff --git a/src/Makefile.am b/src/Makefile.am
index 8150d99..812ffbb 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -133,6 +133,7 @@ libguestfs_la_SOURCES = \
wait.c \
whole-file.c \
version.c \
+ yara.c \
libguestfs.syms
libguestfs_la_CPPFLAGS = \
diff --git a/src/yara.c b/src/yara.c
new file mode 100644
index 0000000..0b924a2
--- /dev/null
+++ b/src/yara.c
@@ -0,0 +1,140 @@
+/* libguestfs
+ * Copyright (C) 2016 Red Hat Inc.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <string.h>
+#include <rpc/xdr.h>
+#include <rpc/types.h>
+
+#include "guestfs.h"
+#include "guestfs_protocol.h"
+#include "guestfs-internal.h"
+#include "guestfs-internal-all.h"
+#include "guestfs-internal-actions.h"
+
+static struct guestfs_yara_detection_list *parse_yara_detection_file (guestfs_h
*, const char *);
+static int deserialise_yara_detection_list (guestfs_h *, FILE *, struct
guestfs_yara_detection_list *);
+static char *make_temp_file (guestfs_h *, const char *);
+
+struct guestfs_yara_detection_list *
+guestfs_impl_yara_scan (guestfs_h *g, const char *path)
+{
+ int ret = 0;
+ CLEANUP_UNLINK_FREE char *tmpfile = NULL;
+
+ tmpfile = make_temp_file (g, "yara_scan");
+ if (tmpfile == NULL)
+ return NULL;
+
+ ret = guestfs_internal_yara_scan (g, path, tmpfile);
+ if (ret < 0)
+ return NULL;
+
+ return parse_yara_detection_file (g, tmpfile); /* caller frees */
+}
+
+/* Parse the file content and return detections list.
+ * Return a list of yara_detection on success, NULL on error.
+ */
+static struct guestfs_yara_detection_list *
+parse_yara_detection_file (guestfs_h *g, const char *tmpfile)
+{
+ int ret = 0;
+ CLEANUP_FCLOSE FILE *fp = NULL;
+ struct guestfs_yara_detection_list *detections = NULL;
+
+ fp = fopen (tmpfile, "r");
+ if (fp == NULL) {
+ perrorf (g, "fopen: %s", tmpfile);
+ return NULL;
+ }
+
+ /* Initialise results array. */
+ detections = safe_malloc (g, sizeof (*detections));
+ detections->len = 8;
+ detections->val = safe_malloc (g, detections->len *
+ sizeof (*detections->val));
+
+ /* Deserialise buffer into detection list. */
+ ret = deserialise_yara_detection_list (g, fp, detections);
+ if (ret < 0) {
+ guestfs_free_yara_detection_list (detections);
+ return NULL;
+ }
+
+ return detections;
+}
+
+/* Deserialise the file content and populate the detection list.
+ * Return the number of deserialised detections, -1 on error.
+ */
+static int
+deserialise_yara_detection_list (guestfs_h *g, FILE *fp,
+ struct guestfs_yara_detection_list
*detections)
+{
+ XDR xdr;
+ int ret = 0;
+ uint32_t index = 0;
+ struct stat statbuf;
+
+ ret = fstat (fileno(fp), &statbuf);
+ if (ret == -1)
+ return -1;
+
+ xdrstdio_create (&xdr, fp, XDR_DECODE);
+
+ for (index = 0; xdr_getpos (&xdr) < statbuf.st_size; index++) {
+ if (index == detections->len) {
+ detections->len = 2 * detections->len;
+ detections->val = safe_realloc (g, detections->val,
+ detections->len *
+ sizeof (*detections->val));
+ }
+
+ /* Clear the entry so xdr logic will allocate necessary memory. */
+ memset (&detections->val[index], 0, sizeof (*detections->val));
+ ret = xdr_guestfs_int_yara_detection (&xdr, (guestfs_int_yara_detection
*)
+ &detections->val[index]);
+ if (ret == 0)
+ break;
+ }
+
+ xdr_destroy (&xdr);
+ detections->len = index;
+
+ return ret ? 0 : -1;
+}
+
+static char *
+make_temp_file (guestfs_h *g, const char *name)
+{
+ int ret = 0;
+
+ ret = guestfs_int_lazy_make_tmpdir (g);
+ if (ret < 0)
+ return NULL;
+
+ return safe_asprintf (g, "%s/%s%d", g->tmpdir, name,
++g->unique);
+}
--
2.10.1
Signed-off-by: Matteo Cafasso <noxdafox@gmail.com>
---
configure.ac | 1 +
tests/yara/Makefile.am | 26 ++++++++++++++++
tests/yara/test-yara-scan.sh | 72 ++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 99 insertions(+)
create mode 100644 tests/yara/Makefile.am
create mode 100755 tests/yara/test-yara-scan.sh
diff --git a/configure.ac b/configure.ac
index 45bb935..0681010 100644
--- a/configure.ac
+++ b/configure.ac
@@ -285,6 +285,7 @@ AC_CONFIG_FILES([Makefile
tests/tsk/Makefile
tests/xfs/Makefile
tests/xml/Makefile
+ tests/yara/Makefile
tools/Makefile
utils/boot-analysis/Makefile
utils/boot-benchmark/Makefile
diff --git a/tests/yara/Makefile.am b/tests/yara/Makefile.am
new file mode 100644
index 0000000..e23d94e
--- /dev/null
+++ b/tests/yara/Makefile.am
@@ -0,0 +1,26 @@
+# libguestfs
+# Copyright (C) 2016 Red Hat Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+include $(top_srcdir)/subdir-rules.mk
+
+TESTS = \
+ test-yara-scan.sh
+
+TESTS_ENVIRONMENT = $(top_builddir)/run --test
+
+EXTRA_DIST = \
+ $(TESTS)
diff --git a/tests/yara/test-yara-scan.sh b/tests/yara/test-yara-scan.sh
new file mode 100755
index 0000000..a899e33
--- /dev/null
+++ b/tests/yara/test-yara-scan.sh
@@ -0,0 +1,72 @@
+#!/bin/bash -
+# libguestfs
+# Copyright (C) 2016 Red Hat Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+# Test the yara_scan command.
+
+set -e
+
+if [ -n "$SKIP_TEST_YARA_SCAN_SH" ]; then
+ echo "$0: test skipped because environment variable is set."
+ exit 77
+fi
+
+rm -f test-yara-rules.yar
+
+# Skip if Yara is not supported by the appliance.
+if ! guestfish add /dev/null : run : available "libyara"; then
+ echo "$0: skipped because Yara is not available in the appliance"
+ exit 77
+fi
+
+if [ ! -s ../../test-data/phony-guests/blank-fs.img ]; then
+ echo "$0: skipped because blank-fs.img is zero-sized"
+ exit 77
+fi
+
+/bin/cat << EOF > test-yara-rules.yar
+rule TestRule
+{
+ strings:
+ \$my_text_string = "some text"
+
+ condition:
+ \$my_text_string
+}
+EOF
+
+output=$(
+guestfish --ro -a ../../test-data/phony-guests/blank-fs.img <<EOF
+run
+mount /dev/sda1 /
+write /text.txt "some text"
+yara-load test-yara-rules.yar
+yara-scan /text.txt
+umount /
+yara-destroy
+EOF
+)
+
+echo $output | grep -zq '{ name: /text.txt rule: TestRule }'
+if [ $? != 0 ]; then
+ echo "$0: TestRule not found in detections list."
+ echo "Detections list:"
+ echo $output
+ exit 1
+fi
+
+rm -f test-yara-rules.yar
--
2.10.1
Pino Toscano
2016-Nov-07 14:07 UTC
Re: [Libguestfs] [PATCH 1/6] appliance: add libyara dependency
On Wednesday, 2 November 2016 21:26:19 CET Matteo Cafasso wrote:> Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> > --- > appliance/packagelist.in | 1 + > daemon/Makefile.am | 3 ++- > m4/guestfs_daemon.m4 | 8 ++++++++ > 3 files changed, 11 insertions(+), 1 deletion(-) > > diff --git a/appliance/packagelist.in b/appliance/packagelist.in > index f278f66..5982df8 100644 > --- a/appliance/packagelist.in > +++ b/appliance/packagelist.in > @@ -232,6 +232,7 @@ jfsutils > kmod > less > libxml2 > +libyara3This is specific for Debian/Ubuntu, so needs to be placed there.> lsof > lsscsi > lvm2 > diff --git a/daemon/Makefile.am b/daemon/Makefile.am > index 23f60eb..3a25f43 100644 > --- a/daemon/Makefile.am > +++ b/daemon/Makefile.am > @@ -222,7 +222,8 @@ guestfsd_LDADD = \ > $(LIBINTL) \ > $(SERVENT_LIB) \ > $(PCRE_LIBS) \ > - $(TSK_LIBS) > + $(TSK_LIBS) \ > + $(YARA_LIBS) > > guestfsd_CPPFLAGS = \ > -I$(top_srcdir)/gnulib/lib \ > diff --git a/m4/guestfs_daemon.m4 b/m4/guestfs_daemon.m4 > index 12123df..9dc7c6c 100644 > --- a/m4/guestfs_daemon.m4 > +++ b/m4/guestfs_daemon.m4 > @@ -126,3 +126,11 @@ AC_CHECK_LIB([tsk],[tsk_version_print],[ > AC_DEFINE([HAVE_LIBTSK], [1], [Define to 1 if The Sleuth Kit library (libtsk) is available.]) > ], []) > ],[AC_MSG_WARN([The Sleuth Kit library (libtsk) not found])]) > + > +dnl yara library (optional) > +AC_CHECK_LIB([yara],[yr_initialize],[ > + AC_CHECK_HEADER([yara.h],[ > + AC_SUBST([YARA_LIBS], [-lyara]) > + AC_DEFINE([HAVE_LIBYARA], [1], [Define to 1 if Yara library is available.]) > + ], []) > +],[AC_MSG_WARN([Yara library not found])])Newer version of YARA have pkg-config support, so that should be checked first, falling back to manual library detection. Thanks, -- Pino Toscano
On Wednesday, 2 November 2016 21:26:20 CET Matteo Cafasso wrote:> The yara_load API allows to load a set of Yara rules contained within a > file on the host. > > Rules can be in binary format, as when compiled with yarac command, or > in source code format. In the latter case, the rules will be first > compiled and then loaded.This paragraph may be worth added in the API longdesc as well, since it gives more hint to users.> +/* Yara compiled rules. */ > +static YR_RULES *rules = NULL;This is not properly freed at shutdown -- take a look at what is done for Augeas, aug_finalize.> +static int upload_rules_file (char *); > +static int compile_rules_file (const char *); > +static int write_callback (void *, const void *, size_t); > + > +/* Has one FileIn parameter. */ > +int > +do_yara_load (void) > +{ > + int ret = 0; > + char tmpfile[] = "/tmp/yaraXXXXXX"; > + > + ret = upload_rules_file (tmpfile); > + if (ret < 0) > + return -1; > + > + ret = yr_initialize ();The libyara API docs do not say whether this is doable multiple times, so I'd do that only once at all.> + if (ret != ERROR_SUCCESS) { > + reply_with_error ("failed initializing yara"); > + unlink (tmpfile); > + return -1; > + } > + > + /* Destroy previously loaded rules. */ > + if (rules != NULL) { > + yr_rules_destroy (rules); > + rules = NULL; > + } > + > + /* Try to load the rules as compiled. > + * If their are in source code format, compile them first. > + */ > + ret = yr_rules_load (tmpfile, &rules); > + if (ret == ERROR_INVALID_FILE) > + ret = compile_rules_file (tmpfile); > + > + unlink (tmpfile); > + > + if (ret != ERROR_SUCCESS) { > + reply_with_error ("failed loading the rules"); > + yr_finalize (); > + return -1; > + } > + > + ret = yr_finalize ();Accoding to the API documentation, yr_finalize "must be called when you are finished using the library". OTOH, libyara is used also after this point (by other APIs, for example), so we cannot use yr_finalize here.> + if (ret != ERROR_SUCCESS) { > + reply_with_error ("failed finalizing yara"); > + return -1; > + } > + > + return 0; > +} > + > +/* Upload rules file on a temporary file. > + * Return 0 on success, -1 on error. > + */ > +static int > +upload_rules_file (char *rules_path) > +{ > + int ret = 0; > + CLEANUP_CLOSE int fd = 0; > + struct write_callback_data data = { .written = 0 }; > + > + data.fd = mkstemp (rules_path); > + if (data.fd == -1) { > + reply_with_perror ("mkstemp"); > + return -1; > + } > + > + ret = receive_file (write_callback, &data.fd);'fd' happens to be the first member in the write_callback_data struct, so this passes the right address for the callback. OTOH, passing the address of the struct itself is a better option, so order changes in the members of the struct won't cause the callback to explode.> + if (ret == -1) { > + /* Write error. */ > + cancel_receive (); > + reply_with_error ("write error: %s", rules_path); > + return -1; > + } > + if (ret == -2) { > + /* Cancellation from library */ > + reply_with_error ("file upload cancelled"); > + return -1; > + } > + > + return 0; > +} > + > +/* Compile source code rules and load them. > + * Return ERROR_SUCCESS on success, Yara error code type on error. > + */ > +static int > +compile_rules_file (const char *rules_path) > +{ > + int ret = 0; > + FILE *rule_file = NULL;Sounds like you could copy the CLEANUP_FCLOSE stuff from the library to the daemon (see daemon/cleanups.{c,h}).> + YR_COMPILER *compiler = NULL;Likewise, adding a custom CLEANUP_YR_COMPILER would help making this function less leak prone.> + ret = yr_compiler_create (&compiler); > + if (ret != ERROR_SUCCESS) { > + perror ("yr_compiler_create");I'd issue reply_with_(p)error instead of perror directly in this function, avoiding generic error messages in do_yara_load.> + return ret; > + } > + > + rule_file = fopen (rules_path, "r"); > + if (rule_file == NULL) { > + yr_compiler_destroy (compiler); > + perror ("unable to open rules file"); > + return ret; > + } > + > + ret = yr_compiler_add_file (compiler, rule_file, NULL, rules_path); > + if (ret > 0) { > + yr_compiler_destroy (compiler); > + fclose (rule_file); > + perror ("yr_compiler_add_file");It looks like there is yr_compiler_* API for getting more information on the actual error -- please make use of it in error message, so it's easier to know why something failed. Thanks, -- Pino Toscano