Libguestfs - Jun 2018 - [PATCH] v2v: rhv-upload: Disable Nagle algorithm

When sending a PUT request, the http header may be sent in the first
packet when calling con.endheaders(). When we send the first chunk, the
kernel may delay the send because the header packet was not acked yet.

We have seen PUT requests delayed by 40 milliseconds on the server side
during virt-v2v upload to ovirt. Here is example log from current RHEL
virt-v2v version, uploading to RHV 4.2.3:

2018-06-12 17:04:01,750 INFO    (Thread-2) [images] Writing 52736 bytes
at offset 0 flush False to /path/to/image for ticket
374bec27-930d-4097-8e41-e4bc23324eb0

2018-06-12 17:04:01,790 INFO    (Thread-2) [directio] Operation stats:
<Clock(total=0.04, read=0.04, write=0.00)>

The server spent 40 milliseconds reading 52736 bytes form
rhv_upload_plugin running on the same host.

This issue was fixed in python 3.5 by using the TCP_NO_DELAY option
after connecting[1]. I backported this change from python 3.5.

I tested the same change using imageio example upload script. With this
change and with optimized PATCH requests, upload time of 4G sparse image
was reduced from 7 minutes to 1 minute.

See this ovirt patch for more details:
https://gerrit.ovirt.org/#/c/92276/

This change is needed only for python 2.

[1] https://bugs.python.org/issue23302
---
 v2v/rhv-upload-plugin.py | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py
index ed99cc7a9..f8cd37e9f 100644
--- a/v2v/rhv-upload-plugin.py
+++ b/v2v/rhv-upload-plugin.py
@@ -23,7 +23,7 @@ import ssl
 import sys
 import time
 
-from http.client import HTTPSConnection
+from http.client import HTTPSConnection as _HTTPSConnection
 from urllib.parse import urlparse
 
 import ovirtsdk4 as sdk
@@ -38,6 +38,18 @@ timeout = 5*60
 # is no formal API here.
 params = None
 
+class HTTPSConnection(_HTTPSConnection):
+    def connect(self):
+        """
+        Using TCP_NO_DELAY avoids delays when sending small payload, such as
+        ovirt PATCH requests.
+
+        This issue was fixed in python 3.5, see:
+        https://bugs.python.org/issue23302
+        """
+        _HTTPSConnection.connect(self)
+        self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+
 def config(key, value):
     global params
 
-- 
2.17.1

On Fri, Jun 15, 2018 at 10:57:32PM +0300, Nir Soffer
wrote:
> When sending a PUT request, the http header may be sent in the first
> packet when calling con.endheaders(). When we send the first chunk, the
> kernel may delay the send because the header packet was not acked yet.
> 
> We have seen PUT requests delayed by 40 milliseconds on the server side
> during virt-v2v upload to ovirt. Here is example log from current RHEL
> virt-v2v version, uploading to RHV 4.2.3:
> 
> 2018-06-12 17:04:01,750 INFO    (Thread-2) [images] Writing 52736 bytes
> at offset 0 flush False to /path/to/image for ticket
> 374bec27-930d-4097-8e41-e4bc23324eb0
> 
> 2018-06-12 17:04:01,790 INFO    (Thread-2) [directio] Operation stats:
> <Clock(total=0.04, read=0.04, write=0.00)>
> 
> The server spent 40 milliseconds reading 52736 bytes form
> rhv_upload_plugin running on the same host.
> 
> This issue was fixed in python 3.5 by using the TCP_NO_DELAY option
> after connecting[1]. I backported this change from python 3.5.
> 
> I tested the same change using imageio example upload script. With this
> change and with optimized PATCH requests, upload time of 4G sparse image
> was reduced from 7 minutes to 1 minute.
> 
> See this ovirt patch for more details:
> https://gerrit.ovirt.org/#/c/92276/
> 
> This change is needed only for python 2.
> 
> [1] https://bugs.python.org/issue23302
> ---
>  v2v/rhv-upload-plugin.py | 14 +++++++++++++-
>  1 file changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py
> index ed99cc7a9..f8cd37e9f 100644
> --- a/v2v/rhv-upload-plugin.py
> +++ b/v2v/rhv-upload-plugin.py
> @@ -23,7 +23,7 @@ import ssl
>  import sys
>  import time
>  
> -from http.client import HTTPSConnection
> +from http.client import HTTPSConnection as _HTTPSConnection
>  from urllib.parse import urlparse
>  
>  import ovirtsdk4 as sdk
> @@ -38,6 +38,18 @@ timeout = 5*60
>  # is no formal API here.
>  params = None
>  
> +class HTTPSConnection(_HTTPSConnection):
> +    def connect(self):
> +        """
> +        Using TCP_NO_DELAY avoids delays when sending small payload, such
as
> +        ovirt PATCH requests.
> +
> +        This issue was fixed in python 3.5, see:
> +        https://bugs.python.org/issue23302
> +        """
> +        _HTTPSConnection.connect(self)
> +        self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
> +
>  def config(key, value):
>      global params
>  
ACK.

Note that we only need this downstream (in RHEL >= 7.6).  I have filed
a bug to include this patch:

  https://bugzilla.redhat.com/show_bug.cgi?id=1591960

In Fedora and RHEL >= 8 we're Python 3.6+ only, so the bug shouldn't
affect us there.

Thanks,

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v