Jason Wang
2023-Nov-02 04:33 UTC
[PATCH net-next 0/5] virtio-net: support dynamic coalescing moderation
On Wed, Nov 1, 2023 at 7:03?PM Heng Qi <hengqi at linux.alibaba.com> wrote:> > > > ? 2023/10/25 ??1:53, Michael S. Tsirkin ??: > > On Wed, Oct 25, 2023 at 09:18:27AM +0800, Jason Wang wrote: > >> On Tue, Oct 24, 2023 at 8:03?PM Heng Qi <hengqi at linux.alibaba.com> wrote: > >>> > >>> > >>> ? 2023/10/12 ??4:29, Jason Wang ??: > >>>> On Thu, Oct 12, 2023 at 3:44?PM Heng Qi <hengqi at linux.alibaba.com> wrote: > >>>>> Now, virtio-net already supports per-queue moderation parameter > >>>>> setting. Based on this, we use the netdim library of linux to support > >>>>> dynamic coalescing moderation for virtio-net. > >>>>> > >>>>> Due to hardware scheduling issues, we only tested rx dim. > >>>> Do you have PPS numbers? And TX numbers are also important as the > >>>> throughput could be misleading due to various reasons. > >>> Hi Jason! > >>> > >>> The comparison of rx netdim performance is as follows: > >>> (the backend supporting tx dim is not yet ready) > >> Thanks a lot for the numbers. > >> > >> I'd still expect the TX result as I did play tx interrupt coalescing > >> about 10 years ago. > >> > >> I will start to review the series but let's try to have some TX numbers as well. > >> > >> Btw, it would be more convenient to have a raw PPS benchmark. E.g you > >> can try to use a software or hardware packet generator. > >> > >> Thanks > > Latency results are also kind of interesting. > > I test the latency using sockperf pp: > > @Rx cmd > taskset -c 0 sockperf sr -p 8989 > > @Tx cmd > taskset -c 0 sockperf pp -i ${ip} -p 8989 -t 10 > > After running this cmd 5 times and averaging the results, > we get the following data: > > dim off: 17.7735 usec > dim on: 18.0110 usecLet's add those numbers to the changelog of the next version. Thanks> > Thanks! > > > > > > >>> > >>> I. Sockperf UDP > >>> ================================================> >>> 1. Env > >>> rxq_0 is affinity to cpu_0 > >>> > >>> 2. Cmd > >>> client: taskset -c 0 sockperf tp -p 8989 -i $IP -t 10 -m 16B > >>> server: taskset -c 0 sockperf sr -p 8989 > >>> > >>> 3. Result > >>> dim off: 1143277.00 rxpps, throughput 17.844 MBps, cpu is 100%. > >>> dim on: 1124161.00 rxpps, throughput 17.610 MBps, cpu is 83.5%. > >>> ================================================> >>> > >>> > >>> II. Redis > >>> ================================================> >>> 1. Env > >>> There are 8 rxqs and rxq_i is affinity to cpu_i. > >>> > >>> 2. Result > >>> When all cpus are 100%, ops/sec of memtier_benchmark client is > >>> dim off: 978437.23 > >>> dim on: 1143638.28 > >>> ================================================> >>> > >>> > >>> III. Nginx > >>> ================================================> >>> 1. Env > >>> There are 8 rxqs and rxq_i is affinity to cpu_i. > >>> > >>> 2. Result > >>> When all cpus are 100%, requests/sec of wrk client is > >>> dim off: 877931.67 > >>> dim on: 1019160.31 > >>> ================================================> >>> > >>> Thanks! > >>> > >>>> Thanks > >>>> > >>>>> @Test env > >>>>> rxq0 has affinity to cpu0. > >>>>> > >>>>> @Test cmd > >>>>> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} > >>>>> server: taskset -c 0 sockperf sr --tcp > >>>>> > >>>>> @Test res > >>>>> The second column is the ratio of the result returned by client > >>>>> when rx dim is enabled to the result returned by client when > >>>>> rx dim is disabled. > >>>>> -------------------------------------- > >>>>> | msg_size | rx_dim=on / rx_dim=off | > >>>>> -------------------------------------- > >>>>> | 14B | + 3% | > >>>>> -------------------------------------- > >>>>> | 100B | + 16% | > >>>>> -------------------------------------- > >>>>> | 500B | + 25% | > >>>>> -------------------------------------- > >>>>> | 1400B | + 28% | > >>>>> -------------------------------------- > >>>>> | 2048B | + 22% | > >>>>> -------------------------------------- > >>>>> | 4096B | + 5% | > >>>>> -------------------------------------- > >>>>> > >>>>> --- > >>>>> This patch set was part of the previous netdim patch set[1]. > >>>>> [1] was split into a merged bugfix set[2] and the current set. > >>>>> The previous relevant commentators have been Cced. > >>>>> > >>>>> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi at linux.alibaba.com/ > >>>>> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi at linux.alibaba.com/ > >>>>> > >>>>> Heng Qi (5): > >>>>> virtio-net: returns whether napi is complete > >>>>> virtio-net: separate rx/tx coalescing moderation cmds > >>>>> virtio-net: extract virtqueue coalescig cmd for reuse > >>>>> virtio-net: support rx netdim > >>>>> virtio-net: support tx netdim > >>>>> > >>>>> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- > >>>>> 1 file changed, 322 insertions(+), 72 deletions(-) > >>>>> > >>>>> -- > >>>>> 2.19.1.6.gb485710b > >>>>> > >>>>> > >>> >