search for: skb_array_consum

...struct skb_array tx_array; }; struct tun_flow_entry { @@ -515,7 +517,11 @@ static struct tun_struct *tun_enable_queue(struct tun_file *tfile) static void tun_queue_purge(struct tun_file *tfile) { - skb_queue_purge(&tfile->sk.sk_receive_queue); + struct sk_buff *skb; + + while ((skb = skb_array_consume(&tfile->tx_array)) != NULL) + kfree_skb(skb); + skb_queue_purge(&tfile->sk.sk_error_queue); } @@ -560,6 +566,8 @@ static void __tun_detach(struct tun_file *tfile, bool clean) tun->dev->reg_state == NETREG_REGISTERED) unregister_netdevice(tun->dev); } +...

..._file *tfile) > return tun; > } > > -static void tun_queue_purge(struct tun_file *tfile) > +static void tun_queue_purge(struct tun_struct *tun, struct tun_file *tfile) > { > + struct sk_buff *skb; > + > + if (tun->flags & IFF_TX_ARRAY) { > + while ((skb = skb_array_consume(&tfile->tx_array)) != NULL) > + kfree_skb(skb); > + } > + > skb_queue_purge(&tfile->sk.sk_receive_queue); > skb_queue_purge(&tfile->sk.sk_error_queue); > } > @@ -545,7 +555,7 @@ static void __tun_detach(struct tun_file *tfile, bool clean) > sy...

...tic struct tun_struct *tun_enable_queue(struct tun_file *tfile) return tun; } -static void tun_queue_purge(struct tun_file *tfile) +static void tun_queue_purge(struct tun_struct *tun, struct tun_file *tfile) { + struct sk_buff *skb; + + if (tun->flags & IFF_TX_ARRAY) { + while ((skb = skb_array_consume(&tfile->tx_array)) != NULL) + kfree_skb(skb); + } + skb_queue_purge(&tfile->sk.sk_receive_queue); skb_queue_purge(&tfile->sk.sk_error_queue); } @@ -545,7 +555,7 @@ static void __tun_detach(struct tun_file *tfile, bool clean) synchronize_net(); tun_flow_delete_by_q...

...tic struct tun_struct *tun_enable_queue(struct tun_file *tfile) return tun; } -static void tun_queue_purge(struct tun_file *tfile) +static void tun_queue_purge(struct tun_struct *tun, struct tun_file *tfile) { + struct sk_buff *skb; + + if (tun->flags & IFF_TX_ARRAY) { + while ((skb = skb_array_consume(&tfile->tx_array)) != NULL) + kfree_skb(skb); + } + skb_queue_purge(&tfile->sk.sk_receive_queue); skb_queue_purge(&tfile->sk.sk_error_queue); } @@ -545,7 +555,7 @@ static void __tun_detach(struct tun_file *tfile, bool clean) synchronize_net(); tun_flow_delete_by_q...

While running a multiple VM testscase with each VM running iperf traffic between others the following kernel NULL pointer exception was seen. Race appears when the tun driver instance of one VM calls skb_array_produce (from tun_net_xmit) and the the destined VM's skb_array_consume (from tun_ring_recv), which could run concurrently on another core. Due to which the sock_wfree gets called again from the tun_ring_recv context. The fix is to add write/read barrier calls to be sure that we get proper values in the tun_ring_recv context. Crash log [35321.580227] Unable to handl...

While running a multiple VM testscase with each VM running iperf traffic between others the following kernel NULL pointer exception was seen. Race appears when the tun driver instance of one VM calls skb_array_produce (from tun_net_xmit) and the the destined VM's skb_array_consume (from tun_ring_recv), which could run concurrently on another core. Due to which the sock_wfree gets called again from the tun_ring_recv context. The fix is to add write/read barrier calls to be sure that we get proper values in the tun_ring_recv context. Crash log [35321.580227] Unable to handl...

Hi all: This series tries to switch to use skb array in tun. This is used to eliminate the spinlock contention between producer and consumer. The conversion was straightforward: just introdce a tx skb array and use it instead of sk_receive_queue. A minor issue is to keep the tx_queue_len behaviour, since tun used to use it for the length of sk_receive_queue. This is done through: - add the

Hi all: This series tries to switch to use skb array in tun. This is used to eliminate the spinlock contention between producer and consumer. The conversion was straightforward: just introdce a tx skb array and use it instead of sk_receive_queue. A minor issue is to keep the tx_queue_len behaviour, since tun used to use it for the length of sk_receive_queue. This is done through: - add the

Hi all: This series tries to switch to use skb array in tun. This is used to eliminate the spinlock contention between producer and consumer. The conversion was straightforward: just introdce a tx skb array and use it instead of sk_receive_queue. A minor issue is to keep the tx_queue_len behaviour, since tun used to use it for the length of sk_receive_queue. This is done through: - add the

Hi all: This series tries to switch to use skb array in tun. This is used to eliminate the spinlock contention between producer and consumer. The conversion was straightforward: just introdce a tx skb array and use it instead of sk_receive_queue. A minor issue is to keep the tx_queue_len behaviour, since tun used to use it for the length of sk_receive_queue. This is done through: - add the

...ducer_lock. 2)Prior to this call the tun_net_xmit calls skb_orphan which calls the skb->destructor and sets skb->destructor and skb->sk as NULL. 2.a) These 2 writes are getting reordered 3) At the same time in the receive side (tun_ring_recv), which gets executed in another core calls skb_array_consume which pulls the skb from ptr ring, this pull is protected by a consumer lock. 4) eventually calling the skb->destructor (sock_wfree) with stale values. Also note that this issue is reproducible in a long run and doesn't happen immediately after the launch of multiple VM's (infact t...

...gt; +static int rx_batched; +module_param(rx_batched, int, 0444); +MODULE_PARM_DESC(rx_batched, "Number of packets batched in rx"); + /* Uncomment to enable debugging */ /* #define TUN_DEBUG 1 */ @@ -522,6 +526,7 @@ static void tun_queue_purge(struct tun_file *tfile) while ((skb = skb_array_consume(&tfile->tx_array)) != NULL) kfree_skb(skb); + skb_queue_purge(&tfile->sk.sk_write_queue); skb_queue_purge(&tfile->sk.sk_error_queue); } @@ -1140,10 +1145,36 @@ static struct sk_buff *tun_alloc_skb(struct tun_file *tfile, return skb; } +static void tun_rx_batched...

...call the tun_net_xmit calls skb_orphan which calls the > skb->destructor and sets skb->destructor and skb->sk as NULL. > > 2.a) These 2 writes are getting reordered > > 3) At the same time in the receive side (tun_ring_recv), which gets executed > in another core calls skb_array_consume which pulls the skb from ptr ring, > this pull is protected by a consumer lock. > > 4) eventually calling the skb->destructor (sock_wfree) with stale values. > > Also note that this issue is reproducible in a long run and doesn't happen > immediately after the launch of...

...gt; +static int rx_batched; +module_param(rx_batched, int, 0444); +MODULE_PARM_DESC(rx_batched, "Number of packets batched in rx"); + /* Uncomment to enable debugging */ /* #define TUN_DEBUG 1 */ @@ -522,6 +526,7 @@ static void tun_queue_purge(struct tun_file *tfile) while ((skb = skb_array_consume(&tfile->tx_array)) != NULL) kfree_skb(skb); + skb_queue_purge(&tfile->sk.sk_write_queue); skb_queue_purge(&tfile->sk.sk_error_queue); } @@ -1140,10 +1145,44 @@ static struct sk_buff *tun_alloc_skb(struct tun_file *tfile, return skb; } +static int tun_rx_batched(...

...rs/net/tun.c +++ b/drivers/net/tun.c @@ -218,6 +218,7 @@ struct tun_struct { struct list_head disabled; void *security; u32 flow_count; + u32 rx_batched; struct tun_pcpu_stats __percpu *pcpu_stats; }; @@ -522,6 +523,7 @@ static void tun_queue_purge(struct tun_file *tfile) while ((skb = skb_array_consume(&tfile->tx_array)) != NULL) kfree_skb(skb); + skb_queue_purge(&tfile->sk.sk_write_queue); skb_queue_purge(&tfile->sk.sk_error_queue); } @@ -1139,10 +1141,46 @@ static struct sk_buff *tun_alloc_skb(struct tun_file *tfile, return skb; } +static void tun_rx_batched...

...rs/net/tun.c +++ b/drivers/net/tun.c @@ -218,6 +218,7 @@ struct tun_struct { struct list_head disabled; void *security; u32 flow_count; + u32 rx_batched; struct tun_pcpu_stats __percpu *pcpu_stats; }; @@ -522,6 +523,7 @@ static void tun_queue_purge(struct tun_file *tfile) while ((skb = skb_array_consume(&tfile->tx_array)) != NULL) kfree_skb(skb); + skb_queue_purge(&tfile->sk.sk_write_queue); skb_queue_purge(&tfile->sk.sk_error_queue); } @@ -1140,10 +1142,45 @@ static struct sk_buff *tun_alloc_skb(struct tun_file *tfile, return skb; } +static void tun_rx_batched...

Hi: This series tries to implement tx batching support for vhost. This was done by using MSG_MORE as a hint for under layer socket. The backend (e.g tap) can then batch the packets temporarily in a list and submit it all once the number of bacthed exceeds a limitation. Tests shows obvious improvement on guest pktgen over over mlx4(noqueue) on host: Mpps -+%

Hi: This series tries to implement tx batching support for vhost. This was done by using MSG_MORE as a hint for under layer socket. The backend (e.g tap) can then batch the packets temporarily in a list and submit it all once the number of bacthed exceeds a limitation. Tests shows obvious improvement on guest pktgen over over mlx4(noqueue) on host: Mpps -+%

...> While running a multiple VM testscase with each VM running iperf > traffic between others the following kernel NULL pointer exception > was seen. > > Race appears when the tun driver instance of one VM calls skb_array_produce > (from tun_net_xmit) and the the destined VM's skb_array_consume > (from tun_ring_recv), which could run concurrently on another core. Due to > which the sock_wfree gets called again from the tun_ring_recv context. > > The fix is to add write/read barrier calls to be sure that we get proper > values in the tun_ring_recv context. > > Crash...

Users of ptr_ring expect that it's safe to give the data structure a pointer and have it be available to consumers, but that actually requires an smb_wmb or a stronger barrier. In absence of such barriers and on architectures that reorder writes, consumer might read an un=initialized value from an skb pointer stored in the skb array. This was observed causing crashes. To fix, add memory