Jason Wang
2023-Oct-25 01:18 UTC
[PATCH net-next 0/5] virtio-net: support dynamic coalescing moderation
On Tue, Oct 24, 2023 at 8:03?PM Heng Qi <hengqi at linux.alibaba.com> wrote:> > > > ? 2023/10/12 ??4:29, Jason Wang ??: > > On Thu, Oct 12, 2023 at 3:44?PM Heng Qi <hengqi at linux.alibaba.com> wrote: > >> Now, virtio-net already supports per-queue moderation parameter > >> setting. Based on this, we use the netdim library of linux to support > >> dynamic coalescing moderation for virtio-net. > >> > >> Due to hardware scheduling issues, we only tested rx dim. > > Do you have PPS numbers? And TX numbers are also important as the > > throughput could be misleading due to various reasons. > > Hi Jason! > > The comparison of rx netdim performance is as follows: > (the backend supporting tx dim is not yet ready)Thanks a lot for the numbers. I'd still expect the TX result as I did play tx interrupt coalescing about 10 years ago. I will start to review the series but let's try to have some TX numbers as well. Btw, it would be more convenient to have a raw PPS benchmark. E.g you can try to use a software or hardware packet generator. Thanks> > > I. Sockperf UDP > ================================================> 1. Env > rxq_0 is affinity to cpu_0 > > 2. Cmd > client: taskset -c 0 sockperf tp -p 8989 -i $IP -t 10 -m 16B > server: taskset -c 0 sockperf sr -p 8989 > > 3. Result > dim off: 1143277.00 rxpps, throughput 17.844 MBps, cpu is 100%. > dim on: 1124161.00 rxpps, throughput 17.610 MBps, cpu is 83.5%. > ================================================> > > II. Redis > ================================================> 1. Env > There are 8 rxqs and rxq_i is affinity to cpu_i. > > 2. Result > When all cpus are 100%, ops/sec of memtier_benchmark client is > dim off: 978437.23 > dim on: 1143638.28 > ================================================> > > III. Nginx > ================================================> 1. Env > There are 8 rxqs and rxq_i is affinity to cpu_i. > > 2. Result > When all cpus are 100%, requests/sec of wrk client is > dim off: 877931.67 > dim on: 1019160.31 > ================================================> > Thanks! > > > > > Thanks > > > >> @Test env > >> rxq0 has affinity to cpu0. > >> > >> @Test cmd > >> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} > >> server: taskset -c 0 sockperf sr --tcp > >> > >> @Test res > >> The second column is the ratio of the result returned by client > >> when rx dim is enabled to the result returned by client when > >> rx dim is disabled. > >> -------------------------------------- > >> | msg_size | rx_dim=on / rx_dim=off | > >> -------------------------------------- > >> | 14B | + 3% | > >> -------------------------------------- > >> | 100B | + 16% | > >> -------------------------------------- > >> | 500B | + 25% | > >> -------------------------------------- > >> | 1400B | + 28% | > >> -------------------------------------- > >> | 2048B | + 22% | > >> -------------------------------------- > >> | 4096B | + 5% | > >> -------------------------------------- > >> > >> --- > >> This patch set was part of the previous netdim patch set[1]. > >> [1] was split into a merged bugfix set[2] and the current set. > >> The previous relevant commentators have been Cced. > >> > >> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi at linux.alibaba.com/ > >> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi at linux.alibaba.com/ > >> > >> Heng Qi (5): > >> virtio-net: returns whether napi is complete > >> virtio-net: separate rx/tx coalescing moderation cmds > >> virtio-net: extract virtqueue coalescig cmd for reuse > >> virtio-net: support rx netdim > >> virtio-net: support tx netdim > >> > >> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- > >> 1 file changed, 322 insertions(+), 72 deletions(-) > >> > >> -- > >> 2.19.1.6.gb485710b > >> > >> > >
Michael S. Tsirkin
2023-Oct-25 05:53 UTC
[PATCH net-next 0/5] virtio-net: support dynamic coalescing moderation
On Wed, Oct 25, 2023 at 09:18:27AM +0800, Jason Wang wrote:> On Tue, Oct 24, 2023 at 8:03?PM Heng Qi <hengqi at linux.alibaba.com> wrote: > > > > > > > > ? 2023/10/12 ??4:29, Jason Wang ??: > > > On Thu, Oct 12, 2023 at 3:44?PM Heng Qi <hengqi at linux.alibaba.com> wrote: > > >> Now, virtio-net already supports per-queue moderation parameter > > >> setting. Based on this, we use the netdim library of linux to support > > >> dynamic coalescing moderation for virtio-net. > > >> > > >> Due to hardware scheduling issues, we only tested rx dim. > > > Do you have PPS numbers? And TX numbers are also important as the > > > throughput could be misleading due to various reasons. > > > > Hi Jason! > > > > The comparison of rx netdim performance is as follows: > > (the backend supporting tx dim is not yet ready) > > Thanks a lot for the numbers. > > I'd still expect the TX result as I did play tx interrupt coalescing > about 10 years ago. > > I will start to review the series but let's try to have some TX numbers as well. > > Btw, it would be more convenient to have a raw PPS benchmark. E.g you > can try to use a software or hardware packet generator. > > ThanksLatency results are also kind of interesting.> > > > > > I. Sockperf UDP > > ================================================> > 1. Env > > rxq_0 is affinity to cpu_0 > > > > 2. Cmd > > client: taskset -c 0 sockperf tp -p 8989 -i $IP -t 10 -m 16B > > server: taskset -c 0 sockperf sr -p 8989 > > > > 3. Result > > dim off: 1143277.00 rxpps, throughput 17.844 MBps, cpu is 100%. > > dim on: 1124161.00 rxpps, throughput 17.610 MBps, cpu is 83.5%. > > ================================================> > > > > > II. Redis > > ================================================> > 1. Env > > There are 8 rxqs and rxq_i is affinity to cpu_i. > > > > 2. Result > > When all cpus are 100%, ops/sec of memtier_benchmark client is > > dim off: 978437.23 > > dim on: 1143638.28 > > ================================================> > > > > > III. Nginx > > ================================================> > 1. Env > > There are 8 rxqs and rxq_i is affinity to cpu_i. > > > > 2. Result > > When all cpus are 100%, requests/sec of wrk client is > > dim off: 877931.67 > > dim on: 1019160.31 > > ================================================> > > > Thanks! > > > > > > > > Thanks > > > > > >> @Test env > > >> rxq0 has affinity to cpu0. > > >> > > >> @Test cmd > > >> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} > > >> server: taskset -c 0 sockperf sr --tcp > > >> > > >> @Test res > > >> The second column is the ratio of the result returned by client > > >> when rx dim is enabled to the result returned by client when > > >> rx dim is disabled. > > >> -------------------------------------- > > >> | msg_size | rx_dim=on / rx_dim=off | > > >> -------------------------------------- > > >> | 14B | + 3% | > > >> -------------------------------------- > > >> | 100B | + 16% | > > >> -------------------------------------- > > >> | 500B | + 25% | > > >> -------------------------------------- > > >> | 1400B | + 28% | > > >> -------------------------------------- > > >> | 2048B | + 22% | > > >> -------------------------------------- > > >> | 4096B | + 5% | > > >> -------------------------------------- > > >> > > >> --- > > >> This patch set was part of the previous netdim patch set[1]. > > >> [1] was split into a merged bugfix set[2] and the current set. > > >> The previous relevant commentators have been Cced. > > >> > > >> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi at linux.alibaba.com/ > > >> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi at linux.alibaba.com/ > > >> > > >> Heng Qi (5): > > >> virtio-net: returns whether napi is complete > > >> virtio-net: separate rx/tx coalescing moderation cmds > > >> virtio-net: extract virtqueue coalescig cmd for reuse > > >> virtio-net: support rx netdim > > >> virtio-net: support tx netdim > > >> > > >> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- > > >> 1 file changed, 322 insertions(+), 72 deletions(-) > > >> > > >> -- > > >> 2.19.1.6.gb485710b > > >> > > >> > > > >