Libguestfs - Apr 2010 - Fix 2 issues in ESX transfer

We were seeing 100% failure rates transferring 10G disk images from ESX on a
particular setup. We also weren't spotting the transfer failure, and dying
with
a strange error from libguestfs. These 2 patches fix the error check which
should have made it obvious what was failing, and the underlying error.

The check of transferred size against expected size was done after returning the
volume, so was never usefully happening.
---
 lib/Sys/VirtV2V/Transfer/ESX.pm |   12 +++++++-----
 1 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/lib/Sys/VirtV2V/Transfer/ESX.pm b/lib/Sys/VirtV2V/Transfer/ESX.pm
index 4d65d5e..fb0c6ea 100644
--- a/lib/Sys/VirtV2V/Transfer/ESX.pm
+++ b/lib/Sys/VirtV2V/Transfer/ESX.pm
@@ -131,16 +131,18 @@ sub get_volume
         my $died = $r->header('X-Died');
         die($died) if (defined($died));
 
+        # It reports success even if we didn't receive the whole file
+        die(user_message(__x("Didn't receive full volume. Received
{received} ".
+                             "of {total} bytes.",
+                             received => $self->{_v2v_received},
+                             total => $self->{_v2v_volsize})))
+            unless ($self->{_v2v_received} == $self->{_v2v_volsize});
+
         my $vol = $self->{_v2v_vol};
         $vol->close();
         return $vol;
     }
 
-    die(user_message(__x("Didn't receive full volume. Received
{received} of ".
-                         "{total} bytes.",
-                         received => $self->{_v2v_received},
-                         total => $self->{_v2v_volsize})))
-        unless ($self->{_v2v_received} == $self->{_v2v_volsize});
 
     if ($r->code == 401) {
         die(user_message(__x("Authentication error connecting to ".
-- 
1.6.6.1

We used to fetch the volume in a HEAD request. After receiving the header, which
contains the content length, we created the target volume and opened it for
writing before continuing to receive data. This created a problem if volume
creation took long enough for the transfer to time out.

To fix this, we get the size first with a HEAD request. Create the volume, then
do a second GET request to receive the data.

This fixes RHBZ#586816
---
 lib/Sys/VirtV2V/Transfer/ESX.pm |   30 ++++++++++++++++++++++++------
 1 files changed, 24 insertions(+), 6 deletions(-)

diff --git a/lib/Sys/VirtV2V/Transfer/ESX.pm b/lib/Sys/VirtV2V/Transfer/ESX.pm
index fb0c6ea..f638149 100644
--- a/lib/Sys/VirtV2V/Transfer/ESX.pm
+++ b/lib/Sys/VirtV2V/Transfer/ESX.pm
@@ -121,10 +121,22 @@ sub get_volume
         return $target->get_volume($volname);
     }
 
+    # Head request to get the size and create the volume
+    # We could do this with a single GET request. The problem with this is that
+    # you have to create the volume before writing to it. If the volume
creation
+    # takes a very long time, the transfer may fail in the mean time.
+    my $r = $self->head($url);
+    if ($r->is_success) {
+        $self->verify_certificate($r) unless ($self->{_v2v_noverify});
+        $self->create_volume($r);
+    } else {
+        $self->report_error($r);
+    }
+
     $self->{_v2v_received} = 0;
-    my $r = $self->SUPER::get($url,
-                              ':content_cb' => sub {
$self->handle_data(@_); },
-                              ':read_size_hint' => 64 * 1024);
+    $r = $self->get($url,
+                    ':content_cb' => sub {
$self->handle_data(@_); },
+                    ':read_size_hint' => 64 * 1024);
 
     if ($r->is_success) {
         # It reports success even if one of the callbacks died
@@ -143,6 +155,13 @@ sub get_volume
         return $vol;
     }
 
+    $self->report_error($r);
+}
+
+sub report_error
+{
+    my $self = shift;
+    my ($r) = @_;
 
     if ($r->code == 401) {
         die(user_message(__x("Authentication error connecting to ".
@@ -173,10 +192,9 @@ sub handle_data
 
     my ($data, $response) = @_;
 
-    # Create the volume if it hasn't been created already
-    if (!defined($self->{_v2v_vol}) && $response->is_success) {
+    # Verify the certificate of the get request the first time we're called
+    if ($self->{_v2v_received} == 0) {
         $self->verify_certificate($response) unless
($self->{_v2v_noverify});
-        $self->create_volume($response);
     }
 
     $self->{_v2v_received} += length($data);
-- 
1.6.6.1

On Wed, Apr 28, 2010 at 05:18:57PM +0100, Matthew Booth
wrote:
> We were seeing 100% failure rates transferring 10G disk images from ESX on
a
> particular setup. We also weren't spotting the transfer failure, and
dying with
> a strange error from libguestfs. These 2 patches fix the error check which
> should have made it obvious what was failing, and the underlying error.
Heh, I thought for a minute you were saying libguestfs wasn't
reporting an error!

ACK to both patches, they both make sense.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora