Michael S. Tsirkin
2020-Dec-28 21:38 UTC
[PATCH rfc 0/3] virtio-net: add tx-hash, rx-tstamp and tx-tstamp
On Mon, Dec 28, 2020 at 02:51:09PM -0500, Willem de Bruijn wrote:> On Mon, Dec 28, 2020 at 12:29 PM Michael S. Tsirkin <mst at redhat.com> wrote: > > > > On Mon, Dec 28, 2020 at 11:22:30AM -0500, Willem de Bruijn wrote: > > > From: Willem de Bruijn <willemb at google.com> > > > > > > RFC for three new features to the virtio network device: > > > > > > 1. pass tx flow hash and state to host, for routing + telemetry > > > 2. pass rx tstamp to guest, for better RTT estimation > > > 3. pass tx tstamp to host, for accurate pacing > > > > > > All three would introduce an extension to the virtio spec. > > > I assume this would require opening three ballots against v1.2 at > > > https://www.oasis-open.org/committees/ballots.php?wg_abbrev=virtio > > > > > > This RFC is to informally discuss the proposals first. > > > > > > The patchset is against v5.10. Evaluation additionally requires > > > changes to qemu and at least one back-end. I implemented preliminary > > > support in Linux vhost-net. Both patches available through github at > > > > > > https://github.com/wdebruij/linux/tree/virtio-net-txhash-1 > > > https://github.com/wdebruij/qemu/tree/virtio-net-txhash-1 > > > > Any data on what the benefits are? > > For the general method, yes. For this specific implementation, not yet. > > Swift congestion control is delay based. It won the best paper award > at SIGCOMM this year. That paper has a lot of data: > https://dl.acm.org/doi/pdf/10.1145/3387514.3406591 . Section 3.1 talks > about the different components that contribute to delay and how to > isolate them.And for the hashing part?> BBR and BBRv2 also have an explicit ProbeRTT phase as part of the design. > > The specific additional benefits for VM-based TCP depends on many > conditions, e.g., whether a vCPU is exclusively owned and pinned. But > the same reasoning should be even more applicable to this even longer > stack, especially in the worst case conditions.
Willem de Bruijn
2020-Dec-29 01:14 UTC
[PATCH rfc 0/3] virtio-net: add tx-hash, rx-tstamp and tx-tstamp
On Mon, Dec 28, 2020 at 7:47 PM Michael S. Tsirkin <mst at redhat.com> wrote:> > On Mon, Dec 28, 2020 at 02:51:09PM -0500, Willem de Bruijn wrote: > > On Mon, Dec 28, 2020 at 12:29 PM Michael S. Tsirkin <mst at redhat.com> wrote: > > > > > > On Mon, Dec 28, 2020 at 11:22:30AM -0500, Willem de Bruijn wrote: > > > > From: Willem de Bruijn <willemb at google.com> > > > > > > > > RFC for three new features to the virtio network device: > > > > > > > > 1. pass tx flow hash and state to host, for routing + telemetry > > > > 2. pass rx tstamp to guest, for better RTT estimation > > > > 3. pass tx tstamp to host, for accurate pacing > > > > > > > > All three would introduce an extension to the virtio spec. > > > > I assume this would require opening three ballots against v1.2 at > > > > https://www.oasis-open.org/committees/ballots.php?wg_abbrev=virtio > > > > > > > > This RFC is to informally discuss the proposals first. > > > > > > > > The patchset is against v5.10. Evaluation additionally requires > > > > changes to qemu and at least one back-end. I implemented preliminary > > > > support in Linux vhost-net. Both patches available through github at > > > > > > > > https://github.com/wdebruij/linux/tree/virtio-net-txhash-1 > > > > https://github.com/wdebruij/qemu/tree/virtio-net-txhash-1 > > > > > > Any data on what the benefits are? > > > > For the general method, yes. For this specific implementation, not yet. > > > > Swift congestion control is delay based. It won the best paper award > > at SIGCOMM this year. That paper has a lot of data: > > https://dl.acm.org/doi/pdf/10.1145/3387514.3406591 . Section 3.1 talks > > about the different components that contribute to delay and how to > > isolate them. > > And for the hashing part?A few concrete examples of error conditions that can be resolved are mentioned in the commits that add sk_rethink_txhash calls. Such as commit 7788174e8726 ("tcp: change IPv6 flow-label upon receiving spurious retransmission"): " Currently a Linux IPv6 TCP sender will change the flow label upon timeouts to potentially steer away from a data path that has gone bad. However this does not help if the problem is on the ACK path and the data path is healthy. In this case the receiver is likely to receive repeated spurious retransmission because the sender couldn't get the ACKs in time and has recurring timeouts. This patch adds another feature to mitigate this problem. It leverages the DSACK states in the receiver to change the flow label of the ACKs to speculatively re-route the ACK packets. In order to allow triggering on the second consecutive spurious RTO, the receiver changes the flow label upon sending a second consecutive DSACK for a sequence number below RCV.NXT. " I don't have quantitative data on the efficacy at scale at hand. Let me see what I can find. This will probably take some time, at least until people are back after the holidays. I didn't want to delay the patch, as the merge window was a nice time for RFC. But agreed that it deserves stronger justification.
Willem de Bruijn
2021-Jan-06 20:32 UTC
[PATCH rfc 0/3] virtio-net: add tx-hash, rx-tstamp and tx-tstamp
On Mon, Dec 28, 2020 at 8:15 PM Willem de Bruijn <willemdebruijn.kernel at gmail.com> wrote:> > On Mon, Dec 28, 2020 at 7:47 PM Michael S. Tsirkin <mst at redhat.com> wrote: > > > > On Mon, Dec 28, 2020 at 02:51:09PM -0500, Willem de Bruijn wrote: > > > On Mon, Dec 28, 2020 at 12:29 PM Michael S. Tsirkin <mst at redhat.com> wrote: > > > > > > > > On Mon, Dec 28, 2020 at 11:22:30AM -0500, Willem de Bruijn wrote: > > > > > From: Willem de Bruijn <willemb at google.com> > > > > > > > > > > RFC for three new features to the virtio network device: > > > > > > > > > > 1. pass tx flow hash and state to host, for routing + telemetry > > > > > 2. pass rx tstamp to guest, for better RTT estimation > > > > > 3. pass tx tstamp to host, for accurate pacing > > > > > > > > > > All three would introduce an extension to the virtio spec. > > > > > I assume this would require opening three ballots against v1.2 at > > > > > https://www.oasis-open.org/committees/ballots.php?wg_abbrev=virtio > > > > > > > > > > This RFC is to informally discuss the proposals first. > > > > > > > > > > The patchset is against v5.10. Evaluation additionally requires > > > > > changes to qemu and at least one back-end. I implemented preliminary > > > > > support in Linux vhost-net. Both patches available through github at > > > > > > > > > > https://github.com/wdebruij/linux/tree/virtio-net-txhash-1 > > > > > https://github.com/wdebruij/qemu/tree/virtio-net-txhash-1 > > > > > > > > Any data on what the benefits are? > > > > > > For the general method, yes. For this specific implementation, not yet. > > > > > > Swift congestion control is delay based. It won the best paper award > > > at SIGCOMM this year. That paper has a lot of data: > > > https://dl.acm.org/doi/pdf/10.1145/3387514.3406591 . Section 3.1 talks > > > about the different components that contribute to delay and how to > > > isolate them. > > > > And for the hashing part? > > A few concrete examples of error conditions that can be resolved are > mentioned in the commits that add sk_rethink_txhash calls. Such as > commit 7788174e8726 ("tcp: change IPv6 flow-label upon receiving > spurious retransmission"): > > " > Currently a Linux IPv6 TCP sender will change the flow label upon > timeouts to potentially steer away from a data path that has gone > bad. However this does not help if the problem is on the ACK path > and the data path is healthy. In this case the receiver is likely > to receive repeated spurious retransmission because the sender > couldn't get the ACKs in time and has recurring timeouts. > > This patch adds another feature to mitigate this problem. It > leverages the DSACK states in the receiver to change the flow > label of the ACKs to speculatively re-route the ACK packets. > In order to allow triggering on the second consecutive spurious > RTO, the receiver changes the flow label upon sending a second > consecutive DSACK for a sequence number below RCV.NXT. > " > > I don't have quantitative data on the efficacy at scale at hand. Let > me see what I can find. This will probably take some time, at least > until people are back after the holidays. I didn't want to delay the > patch, as the merge window was a nice time for RFC. But agreed that it > deserves stronger justification.The practical results mirror what the theory suggests: that in the presence of multiple paths, of which one goes bad, this method maintains connectivity where otherwise it would disconnect. When IPv6 FlowLabel was included in path selection (e.g., LAG/ECMP), flowlabel rotation on TCP timeout avoided the vast majority of TCP disconnections that would otherwise have occurred during to link failures in long-haul backbones, when an alternative path was available. So it's not a matter of percentages, just the existence of an alternative healthy path on which the packets will eventually land quite deterministically as it rotates the txhash on each timeout. This method can be deployed based on a variety of "bad connection" signals. Besides timeouts, the aforementioned spurious retransmits, for one. This TCP connection-level information can independent of flowlabel rotation be valuable information to the cloud provider to detect and pinpoint network issues. As mentioned before, ideally we can pass along such details of the type of signal along with the hash. But that also requires passing that state in the guest from the TCP layer to the virtio-net device. So left for separate later work. For now we just have the reserved space in the header. Michael, what is the best way to proceed with this? Send the patches for review to net-next, or should I start by opening ballots to https://www.oasis-open.org/committees/ballots.php?wg_abbrev=virtio first? Thanks.