similar to: [PATCH] lib: command: switch from select() to poll()

On Wednesday, 26 February 2020 15:08:24 CET Daniel P. Berrangé wrote: > On Wed, Feb 26, 2020 at 02:39:04PM +0100, Pino Toscano wrote: > > select() has a maximum value for the FDs it can monitor, and since > > the libguestfs library can be used in other applications, this limit > > may be hit by users in case lots of FDs are opened. > > > > As solution, switch to

On Wed, Feb 26, 2020 at 02:39:04PM +0100, Pino Toscano wrote: > select() has a maximum value for the FDs it can monitor, and since > the libguestfs library can be used in other applications, this limit > may be hit by users in case lots of FDs are opened. > > As solution, switch to poll(): it has a slightly better interface to > check what changed and for which FD, and it does

Inspired by libvirt's virCommand* internal mini-library, this adds some internal APIs for running commands. The first patch contains the new APIs. The subsequent patches change various parts of the library over to use it. Rich.

--- src/command.c | 64 +++++++++++++++++++++++++++++++++++++++++++------- src/guestfs-internal.h | 3 +++ 2 files changed, 58 insertions(+), 9 deletions(-) diff --git a/src/command.c b/src/command.c index 4bb469b..e26573d 100644 --- a/src/command.c +++ b/src/command.c @@ -360,7 +360,7 @@ debug_command (struct command *cmd) } static int -run_command (struct command *cmd)

Improve readability of commandrvf() by replacing bare int values for file descriptors with their symbolic names STD{IN,OUT,ERR}_FILENO. Also add PIPE_READ and PIPE_WRITE for referencing relevant ends of a pipe. --- daemon/guestfsd.c | 79 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 42 insertions(+), 37 deletions(-) diff --git a/daemon/guestfsd.c b/daemon/guestfsd.c

Hi, I attached the changes I made to a vanilla libguestfs-1.22.6 in order to make it work in mingw/win32. Added is also the patch required to make QEMU compatible (add a command to QMP that lists the supported devices (the regilat way you do it print it to stderr, which is difficult to redirect in win32)). This is done on behalf of Intel Corp. Thanks, Or (oberon in irc)

Commits 3c27f3d91e1566854747bbe844186783fc84f3a8 and 1b6f0daa9ae7fcc94e389232d0c397816cda973d added an internal API for running commands asynchronously. It is only used by the copy-in and copy-out APIs. Unfortunately this made the command code very complex: it was almost impossible to redirect stderr to a file, and there were a lot of long-range dependencies through the file. It was also buggy:

As the start of work to add remote support, I'm taking a close look at the protocol layer in the library. These are some small cleanups. Rich.

The last (only?) caller of guestfs_int_cmd_clear_close_files() disappeared in commit e4c396888056 ("lib/info: Remove /dev/fd hacking and pass a true filename to qemu-img info.", 2018-01-23), part of v1.37.36. Simplify the code by removing guestfs_int_cmd_clear_close_files(). Signed-off-by: Laszlo Ersek <lersek at redhat.com> --- lib/guestfs-internal.h | 1 - lib/command.c

Hi, attached there is the second version of the patch series adding copy_in and copy_out in the library, mostly moving them from guestfish. It also adds the copy_in usage in virt-customize, as aid in a new image building. Thanks, Pino Toscano (7): cmd: add a way to run (and wait) asynchronously commands cmd: add a child-setup callback cmd: add the possibility to get a fd to the process

When the COMMAND_FLAG_FOLD_STDOUT_ON_STDERR flag is passed to command*(), indicate that as stdout=e in debugging message. --- daemon/command.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/daemon/command.c b/daemon/command.c index 73fce56..2423a4e 100644 --- a/daemon/command.c +++ b/daemon/command.c @@ -185,6 +185,7 @@ commandrvf (char **stdoutput, char **stderror,

Various changes/fixes to use smaller stack frames. Rich.

I've been taking a long hard look at the protocol layer. It has evolved over a long time without any particular direction, and the result is, to say the least, not very organized. These patches take a first step at cleaning up the mess by abstracting out socket operations from the rest of the code. The purpose of this is to allow us to slot in a different connection layer under the

The original implementation utilies select(). In Linux select() typically supports up to 1024 file descriptors. This can be a problem when user tries to boot up many guests. Switching to poll() has minimum impact on existing code and has better scalibility. Up to 8192 file descriptors are supported in the current implementation. Signed-off-by: Wei Liu <wei.liu2@citrix.com> ---

Allows more sharing between the daemon and the inspection program. --- daemon/Makefile.am | 2 + daemon/cleanups.c | 80 ++++++++++ daemon/cleanups.h | 47 ++++++ daemon/command.c | 436 +++++++++++++++++++++++++++++++++++++++++++++++++++++ daemon/command.h | 41 +++++ daemon/daemon.h | 47 +----- daemon/guestfsd.c | 392 ----------------------------------------------- po/POTFILES

I noticed that R was burning up 100% of a CPU when a call to Sys.sleep() was made. Upon investigation, I discovered that R_checkActivityEx() in src/unix/sys-std.c was putting the entire timeout (in usec) into the struct timeval tv_usec member, leaving tv_sec set to 0. I don't know about other unix variants, but Solaris requires that the timeout value be normalized (i.e. 0 <= tv_usec

Mostly modelled after a snippet implemented in dib, it is an helper function to run multiple commands in parallel, waiting for all of them at once, and returning all their exit codes. It is possible to pass custom descriptors for collecting stdout and stderr of each command. Common_utils.run_command is adapted to use few helper methods used by run_commands, so all the existing code using it

GCC has two warnings related to large stack frames. We were already using the -Wframe-larger-than warning, but this reduces the threshold from 10000 to 5000 bytes. However that warning only covers the static part of frames (not alloca). So this change also enables -Wstack-usage=10000 which covers both the static and dynamic usage (alloca and variable length arrays). Multiple changes are made

Mostly modelled after a snippet implemented in dib, it is an helper function to run multiple commands in parallel, waiting for all of them at once, and returning all their exit codes. It is possible to pass custom descriptors for collecting stdout and stderr of each command. Common_utils.run_command is adapted to use it, so all the existing code using it keeps working. Add a simple unit tests

This is a more working version. Inspection (partially) succeeds on a real guest this time :-) You can test it out on a real guest (in this case, a CentOS disk image located at /tmp/centos-6.img) by doing: $ ./run guestfish -v -x -a /tmp/centos-6.img ><fs> run ><fs> debug sh "guestfs-inspection --verbose" which will print lots of debugging, and at the end the