Virtualization - Aug 2022 - [PATCH 0/1] Stalled communication on Hyper V Sockets

Hi,

After upgrading our VMs kernel, our golang software stack started facing
some communication stalling on HyperV platforms.

After receiving a few hundered MBs data from a vsocket in the guest,
the data flows suddenly stops, and the receiver is not notified. The
sender on the host times out.

Since the use case is very simple, we were surprised that it was not
reported, so we took time to investigate.

The issue was bisected to patch below
[b3f7fd54881bcba5dc529935f38] af_vsock: separate wait data
loop
>From there ftrace allowed to spot the problem, more details are in the
patch description.

In the end, note that the patch is not HyperV specific. So maybe the
reproduction rate is lower on other platforms.

Let me know what you think,
Fr?d?ric Dalleau

Fr?d?ric Dalleau (1):
  vsock: Fix stalled communication on vhost sockets

 net/vmw_vsock/af_vsock.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

-- 
2.34.1

After kernel upgrade, we faced some stalled communication on HyperV VMs
with our golang clients. We bisected the issue to patch
[b3f7fd54881bcba5dc529935f38] af_vsock: separate wait data loop

When this occurred the following sequence appeared in ftrace:
hvs_stream_has_data <- vsock_connectible_wait_data
hvs_channel_cb <- vmbus_on_event
hvs_notify_recv_pre_block <- vsock_connectible_wait_data

It turned out that it was possible to enter hvs_notify_recv_pre_block
with data pending in the stream. This is because the data condition must
be checked between prepare_to_wait and schedule, if not the receiver may
not be woken up if an event occurs after checking the condition and
before prepare_to_wait. This patch moves prepare_to_wait call before
the condition and at the end of loop, similar to what is done elsewhere
in the same file.

Signed-off-by: Fr?d?ric Dalleau <frederic.dalleau at docker.com>
---
 net/vmw_vsock/af_vsock.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/net/vmw_vsock/af_vsock.c b/net/vmw_vsock/af_vsock.c
index ee418701cdee..cd5e0dbcabe0 100644
--- a/net/vmw_vsock/af_vsock.c
+++ b/net/vmw_vsock/af_vsock.c
@@ -1904,9 +1904,9 @@ static int vsock_connectible_wait_data(struct sock *sk,
 	vsk = vsock_sk(sk);
 	err = 0;
 	transport = vsk->transport;
+	prepare_to_wait(sk_sleep(sk), wait, TASK_INTERRUPTIBLE);
 
 	while ((data = vsock_connectible_has_data(vsk)) == 0) {
-		prepare_to_wait(sk_sleep(sk), wait, TASK_INTERRUPTIBLE);
 
 		if (sk->sk_err != 0 ||
 		    (sk->sk_shutdown & RCV_SHUTDOWN) ||
@@ -1937,6 +1937,7 @@ static int vsock_connectible_wait_data(struct sock *sk,
 			err = -EAGAIN;
 			break;
 		}
+		prepare_to_wait(sk_sleep(sk), wait, TASK_INTERRUPTIBLE);
 	}
 
 	finish_wait(sk_sleep(sk), wait);
-- 
2.34.1