Richard W.M. Jones
2018-Sep-19 16:23 UTC
[Libguestfs] [PATCH 0/3] v2v: -o rhv-upload: Add a test.
This adds a test of -o rhv-upload. Obviously for an upstream test we cannot require a working oVirt server. This test works by faking the ovirtsdk4 Python module, setting PYTHONPATH so that the fake module is picked up instead of the real module (if installed). However it's more complex than that because the nbdkit plugin also expects to talk to a working imageio HTTPS server. Therefore the fake ovirtsdk4 module also starts a web server running on a random port which looks sufficiently similar to a real imageio server to trick virt-v2v (but any data written is discarded). This test is designed to pick up only gross errors, such as the one where we recently shipped a non-working -o rhv-upload mode in RHEL 7.6 LP. Rich.
Richard W.M. Jones
2018-Sep-19 16:23 UTC
[Libguestfs] [PATCH 1/3] v2v: -o rhv-upload: Print commands before running them in verbose mode.
--- v2v/output_rhv_upload.ml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/v2v/output_rhv_upload.ml b/v2v/output_rhv_upload.ml index f03e1ede3..3bb2ca807 100644 --- a/v2v/output_rhv_upload.ml +++ b/v2v/output_rhv_upload.ml @@ -117,6 +117,8 @@ class output_rhv_upload output_alloc output_conn (* Check that nbdkit is available and new enough. *) let error_unless_nbdkit_working () + let cmd = "nbdkit --version >/dev/null" in + debug "%s" cmd; if 0 <> Sys.command "nbdkit --version >/dev/null" then error (f_"nbdkit is not installed or not working. It is required to use ‘-o rhv-upload’. See \"OUTPUT TO RHV\" in the virt-v2v(1) manual."); @@ -139,6 +141,7 @@ class output_rhv_upload output_alloc output_conn let cmd = sprintf "nbdkit %s %s --dump-plugin >/dev/null" Python_script.python (quote (Python_script.path plugin_script)) in + debug "%s" cmd; if Sys.command cmd <> 0 then error (f_"nbdkit %s plugin is not installed or not working. It is required if you want to use ‘-o rhv-upload’. -- 2.19.0.rc0
Richard W.M. Jones
2018-Sep-19 16:23 UTC
[Libguestfs] [PATCH 2/3] v2v: -o rhv-upload: Only set SSL context for https connections.
For real imageio servers the destination will always be https. This change has no effect there. However when testing we want to use an http server for simplicity. As there is no cafile in this case the call to ssl.create_default_context().load_verify_locations(cafile=...) will fail. --- v2v/rhv-upload-plugin.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py index 5cd6d5cab..6e35b5057 100644 --- a/v2v/rhv-upload-plugin.py +++ b/v2v/rhv-upload-plugin.py @@ -207,8 +207,11 @@ def open(readonly): else: destination_url = urlparse(transfer.proxy_url) - context = ssl.create_default_context() - context.load_verify_locations(cafile = params['rhv_cafile']) + if destination_url.scheme == "https": + context = ssl.create_default_context() + context.load_verify_locations(cafile = params['rhv_cafile']) + else: + context = None http = HTTPSConnection( destination_url.hostname, -- 2.19.0.rc0
Richard W.M. Jones
2018-Sep-19 16:23 UTC
[Libguestfs] [PATCH 3/3] v2v: -o rhv-upload: Add a test.
Previously this output method was almost completely untested. This commit adds a fake ovirtsdk4 module so we can test the -o rhv-upload method fairly completely without needing an actual oVirt instance around. --- v2v/Makefile.am | 2 + v2v/rhv-upload-plugin.py | 16 +- .../ovirtsdk4/__init__.py | 146 ++++++++++++++++++ .../ovirtsdk4/types.py | 125 +++++++++++++++ v2v/test-v2v-o-rhv-upload.sh | 51 ++++++ 5 files changed, 333 insertions(+), 7 deletions(-) diff --git a/v2v/Makefile.am b/v2v/Makefile.am index 14183572b..9a2b4391d 100644 --- a/v2v/Makefile.am +++ b/v2v/Makefile.am @@ -382,6 +382,7 @@ TESTS += \ test-v2v-o-openstack.sh \ test-v2v-o-qemu.sh \ test-v2v-o-rhv.sh \ + test-v2v-o-rhv-upload.sh \ test-v2v-o-vdsm-options.sh \ test-v2v-oa-option.sh \ test-v2v-of-option.sh \ @@ -539,6 +540,7 @@ EXTRA_DIST += \ test-v2v-o-qemu.sh \ test-v2v-o-rhv.ovf.expected \ test-v2v-o-rhv.sh \ + test-v2v-o-rhv-upload.sh \ test-v2v-o-rhv-upload-oo-query.sh \ test-v2v-o-vdsm-oo-query.sh \ test-v2v-o-vdsm-options.ovf.expected \ diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py index 6e35b5057..b27349103 100644 --- a/v2v/rhv-upload-plugin.py +++ b/v2v/rhv-upload-plugin.py @@ -210,14 +210,16 @@ def open(readonly): if destination_url.scheme == "https": context = ssl.create_default_context() context.load_verify_locations(cafile = params['rhv_cafile']) + http = HTTPSConnection( + destination_url.hostname, + destination_url.port, + context = context + ) else: - context = None - - http = HTTPSConnection( - destination_url.hostname, - destination_url.port, - context = context - ) + http = HTTPConnection( + destination_url.hostname, + destination_url.port, + ) # The first request is to fetch the features of the server. diff --git a/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py new file mode 100644 index 000000000..2ddb950ea --- /dev/null +++ b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/__init__.py @@ -0,0 +1,146 @@ +# -*- python -*- +# Copyright (C) 2018 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Fake ovirtsdk4 module used as a test harness. +# See v2v/test-v2v-o-rhv-upload.sh + +class Error(Exception): + pass +class NotFoundError(Error): + pass + +class Connection(object): + def __init__( + self, + url = None, + username = None, + password = None, + ca_file = None, + log = None, + insecure = False, + ): + pass + + def system_service(self): + return SystemService() + +class SystemService(object): + def data_centers_service(self): + return DataCentersService() + + def disks_service(self): + return DisksService() + + def image_transfers_service(self): + return ImageTransfersService() + + def storage_domains_service(self): + return StorageDomainsService() + + def vms_service(self): + return VmsService() + +class DataCentersService(object): + def list(self, search=None, case_sensitive=False): + return [] + +class DiskService(object): + def __init__(self, disk_id): + self._disk_id = disk_id + + def get(self): + return types.Disk() + + def remove(self): + pass + +class DisksService(object): + def add(self, disk=None): + return disk + + def disk_service(self, disk_id): + return DiskService(disk_id) + +class ImageTransferService(object): + def __init__(self): + self._finalized = False + + def get(self): + if self._finalized: + raise NotFoundError + else: + return types.ImageTransfer() + + def finalize(self): + self._finalized = True + +class ImageTransfersService(object): + def add(self, transfer): + return transfer + + def image_transfer_service(self, id): + return ImageTransferService() + +class StorageDomain(object): + id = "ba87af68-b630-4211-a73a-694c1a689405" + +class StorageDomainsService(object): + def list(self, search=None): + return [ StorageDomain() ] + +class VmsService(object): + def add(self, vm): + return vm + + def list(self, search=None): + return [] + +# Create a background thread running a web server which is +# simulating the imageio server. + +from http.server import HTTPServer, BaseHTTPRequestHandler +import random +import threading + +# Choose a random port number in range [50000,59999] +imageio_port = random.randint(50000,60000) + +class RequestHandler(BaseHTTPRequestHandler): + def do_OPTIONS(self): + self.send_response(200) + self.send_header("Content-type", "application/json; charset=UTF-8") + self.end_headers() + # Advertize only zero support. + self.wfile.write(b'''{ "features": [ "zero" ] }''') + + # eg. zero request. Just ignore it. + def do_PATCH(self): + self.send_response(200) + self.end_headers() + + # Flush request. Ignore it. + def do_PUT(self): + self.send_response(200) + self.end_headers() + +def server(): + server_address = ("", imageio_port) + httpd = HTTPServer(server_address, RequestHandler) + httpd.serve_forever() + +thread = threading.Thread(target = server, args = [], daemon = True) +thread.start() diff --git a/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/types.py b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/types.py new file mode 100644 index 000000000..9b3f557ee --- /dev/null +++ b/v2v/test-v2v-o-rhv-upload-module/ovirtsdk4/types.py @@ -0,0 +1,125 @@ +# -*- python -*- +# Copyright (C) 2018 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Fake ovirtsdk4 module used as a test harness. +# See v2v/test-v2v-o-rhv-upload.sh + +from enum import Enum +from ovirtsdk4 import imageio_port + +class Cluster(object): + def __init__(self, name): + pass + +class Configuration(object): + def __init__(self, type=None, data=None): + pass +class ConfigurationType(Enum): + OVA = 'ova' + OVF = 'ovf' + + def __init__(self, image): + self._image = image + + def __str__(self): + return self._image + +class DiskFormat(Enum): + COW = "cow" + RAW = "raw" + + def __init__(self, image): + self._image = image + + def __str__(self): + return self._image + +class DiskStatus(Enum): + ILLEGAL = "illegal" + LOCKED = "locked" + OK = "ok" + + def __init__(self, image): + self._image = image + + def __str__(self): + return self._image + +class Disk(object): + def __init__( + self, + id = None, + name = None, + description = None, + format = None, + initial_size = None, + provisioned_size = None, + sparse = False, + storage_domains = None + ): + pass + + id = 123 + status = DiskStatus.OK + +class ImageTransferPhase(Enum): + CANCELLED = 'cancelled' + FINALIZING_FAILURE = 'finalizing_failure' + FINALIZING_SUCCESS = 'finalizing_success' + FINISHED_FAILURE = 'finished_failure' + FINISHED_SUCCESS = 'finished_success' + INITIALIZING = 'initializing' + PAUSED_SYSTEM = 'paused_system' + PAUSED_USER = 'paused_user' + RESUMING = 'resuming' + TRANSFERRING = 'transferring' + UNKNOWN = 'unknown' + + def __init__(self, image): + self._image = image + + def __str__(self): + return self._image + +class ImageTransfer(object): + def __init__( + self, + disk = None, + host = None, + inactivity_timeout = None, + ): + pass + + id = 456 + phase = ImageTransferPhase.TRANSFERRING + transfer_url = "http://localhost:" + str(imageio_port) + "/" + +class Initialization(object): + def __init__(self, configuration): + pass + +class StorageDomain(object): + def __init__(self, name = None): + pass + +class Vm(object): + def __init__( + self, + cluster = None, + initialization = None + ): + pass diff --git a/v2v/test-v2v-o-rhv-upload.sh b/v2v/test-v2v-o-rhv-upload.sh new file mode 100755 index 000000000..8bda7cc0b --- /dev/null +++ b/v2v/test-v2v-o-rhv-upload.sh @@ -0,0 +1,51 @@ +#!/bin/bash - +# libguestfs virt-v2v test script +# Copyright (C) 2018 Red Hat Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +# Test -o rhv-upload. +# +# These uses a test harness (see +# v2v/test-v2v-o-rhv-upload-module/ovirtsdk4) to fake responses from +# oVirt. + +set -e +set -x + +$TEST_FUNCTIONS +skip_if_skipped +skip_if_backend uml +skip_unless_phony_guest windows.img + +libvirt_uri="test://$abs_top_builddir/test-data/phony-guests/guests.xml" +f=$top_builddir/test-data/phony-guests/windows.img + +export VIRT_TOOLS_DATA_DIR="$top_srcdir/test-data/fake-virt-tools" +export VIRTIO_WIN="$top_srcdir/test-data/fake-virtio-win" +export PYTHONPATH=$srcdir/test-v2v-o-rhv-upload-module:$PYTHONPATH + +# Run virt-v2v -o rhv-upload. +# +# The fake ovirtsdk4 module doesn't care about most of the options +# like -oc, -oo rhv-cafile, -op etc. Any values may be used. +$VG virt-v2v --debug-gc -v -x \ + -i libvirt -ic "$libvirt_uri" windows \ + -o rhv-upload \ + -oc https://example.com/ovirt-engine/api \ + -oo rhv-cafile=/dev/null \ + -oo rhv-direct \ + -op /dev/null \ + -os . -- 2.19.0.rc0
Nir Soffer
2018-Sep-19 17:26 UTC
Re: [Libguestfs] [PATCH 2/3] v2v: -o rhv-upload: Only set SSL context for https connections.
On Wed, Sep 19, 2018 at 7:24 PM Richard W.M. Jones <rjones@redhat.com> wrote:> For real imageio servers the destination will always be https. This > change has no effect there. > > However when testing we want to use an http server for simplicity. As > there is no cafile in this case the call to > ssl.create_default_context().load_verify_locations(cafile=...) will fail. > --- > v2v/rhv-upload-plugin.py | 7 +++++-- > 1 file changed, 5 insertions(+), 2 deletions(-) > > diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py > index 5cd6d5cab..6e35b5057 100644 > --- a/v2v/rhv-upload-plugin.py > +++ b/v2v/rhv-upload-plugin.py > @@ -207,8 +207,11 @@ def open(readonly): > else: > destination_url = urlparse(transfer.proxy_url) > > - context = ssl.create_default_context() > - context.load_verify_locations(cafile = params['rhv_cafile']) >This line was never needed. In imageio client we use: context = ssl.create_default_context( purpose=ssl.Purpose.SERVER_AUTH, cafile=cafile) if not secure: context.check_hostname = False context.verify_mode = ssl.CERT_NONE See https://github.com/oVirt/ovirt-imageio/blob/356d224f1124deb3d63125b1f3b3e583839bcbd9/common/ovirt_imageio_common/client.py#L52 So we can replace this with context = ssl.create_default_context(cafile = params.get('rhv_cafile'))> + if destination_url.scheme == "https": > + context = ssl.create_default_context() > + context.load_verify_locations(cafile = params['rhv_cafile']) > + else: > + context = None >This will create a default context inside HTTPSConnection.__init__, which will try to verify the server certificate and hostname and may fail if the certificates are not set up properly in the tests. Nir> > http = HTTPSConnection( > destination_url.hostname, > -- > 2.19.0.rc0 > > _______________________________________________ > Libguestfs mailing list > Libguestfs@redhat.com > https://www.redhat.com/mailman/listinfo/libguestfs >
Possibly Parallel Threads
- Re: [PATCH 2/3] v2v: -o rhv-upload: Only set SSL context for https connections.
- [PATCH v2 2/3] v2v: -o rhv-upload: Only set SSL context for https connections.
- [PATCH] v2v: -o rhv-upload: Fix upload when using https
- Re: [PATCH] v2v: -o rhv-upload: Fix upload when using https
- Re: [PATCH] v2v: -o rhv-upload: Fix upload when using https