Hi, Does anybody know whether the ring buffer between front end and back end will suffer from overflow? I just wonder if the ring buffer will be full and drop some packets when the Net I/O load is very heavy. BTW, If I want to change the size of i/o ring buffer, how should I do? I tried to reset the NET_TX_RING_SIZE and NET_RX_RING_SIZE in both front end and back end, but it seems doesn''t work. Thanks. Regards, Cong _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2011-09-30 at 05:18 +0100, David Xu wrote:> Hi, > > Does anybody know whether the ring buffer between front end and back > end will suffer from overflow? I just wonder if the ring buffer will > be full and drop some packets when the Net I/O load is very heavy.In the case of networking whichever end is putting stuff on the ring checks that there is enough room and will stop the queue when it cannot transmit any more and restart when room becomes available.> BTW, If I want to change the size of i/o ring buffer, how should I do? > I tried to reset the NET_TX_RING_SIZE and NET_RX_RING_SIZE in both > front end and back end, but it seems doesn''t work. Thanks.Currently the rings are limited to 1 page so if you want to increase the size you would need to add multipage ring support to the network protocol. There have been patches to do this for the blk protocol but I do not recall any for the net protocol. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, 2011/9/29 Ian Campbell <Ian.Campbell@citrix.com>:> On Fri, 2011-09-30 at 05:18 +0100, David Xu wrote: >> Hi, >> >> Does anybody know whether the ring buffer between front end and back >> end will suffer from overflow? I just wonder if the ring buffer will >> be full and drop some packets when the Net I/O load is very heavy. > > In the case of networking whichever end is putting stuff on the ring > checks that there is enough room and will stop the queue when it cannot > transmit any more and restart when room becomes available.You mean even there is not enough room in ring buffer, xen will * not drop the packets * and just delay the transmission. I used httperf to measure the performance of web server running in a VM (The workload in this VM is mixed, so it can not benefit from boost mechanism. The net i/o suffers from relatively high latency which depends on the number of VMs in the system). I found that with the increase of request rate in client side, the connection rate will drop and the connection time will increase dramatically. And the retransmission appears when the request rate is over than a quantum. So I doubted that the http/tcp connection suffer from the packets drop when the ring buffer is full because of high request rate.> >> BTW, If I want to change the size of i/o ring buffer, how should I do? >> I tried to reset the NET_TX_RING_SIZE and NET_RX_RING_SIZE in both >> front end and back end, but it seems doesn''t work. Thanks. > > Currently the rings are limited to 1 page so if you want to increase the > size you would need to add multipage ring support to the network > protocol. There have been patches to do this for the blk protocol but I > do not recall any for the net protocol.Yes, increasing the size is relatively hard. So I just want reduce the size of ring buffer to make sure my doubt described above. I directly set NET_TX_RING_SIZE and NET_RX_RING_SIZE to 128, but it doesn''t seem to work. Regards, Cong _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2011-09-30 at 15:44 +0100, David Xu wrote:> Hi, > > 2011/9/29 Ian Campbell <Ian.Campbell@citrix.com>: > > On Fri, 2011-09-30 at 05:18 +0100, David Xu wrote: > >> Hi, > >> > >> Does anybody know whether the ring buffer between front end and back > >> end will suffer from overflow? I just wonder if the ring buffer will > >> be full and drop some packets when the Net I/O load is very heavy. > > > > In the case of networking whichever end is putting stuff on the ring > > checks that there is enough room and will stop the queue when it cannot > > transmit any more and restart when room becomes available. > > You mean even there is not enough room in ring buffer, xen will * not > drop the packets * and just delay the transmission.It''s not Xen but rather the kernel back and front ends which is involved here. You can examine the hard_start_xmit functions in both netback and netfront to determine for yourself whether or not packets can be dropped and when.> I used httperf to > measure the performance of web server running in a VM (The workload in > this VM is mixed, so it can not benefit from boost mechanism. The net > i/o suffers from relatively high latency which depends on the number > of VMs in the system). I found that with the increase of request rate > in client side, the connection rate will drop and the connection time > will increase dramatically. And the retransmission appears when the > request rate is over than a quantum. So I doubted that the http/tcp > connection suffer from the packets drop when the ring buffer is full > because of high request rate. > > > > >> BTW, If I want to change the size of i/o ring buffer, how should I do? > >> I tried to reset the NET_TX_RING_SIZE and NET_RX_RING_SIZE in both > >> front end and back end, but it seems doesn''t work. Thanks. > > > > Currently the rings are limited to 1 page so if you want to increase the > > size you would need to add multipage ring support to the network > > protocol. There have been patches to do this for the blk protocol but I > > do not recall any for the net protocol. > > Yes, increasing the size is relatively hard. So I just want reduce the > size of ring buffer to make sure my doubt described above. I directly > set NET_TX_RING_SIZE and NET_RX_RING_SIZE to 128, but it doesn''t seem to work.You need to make sure both ends of the connection agree on the ring size. I''m afraid this is not a very common thing to want to do so if you want to persist with this approach I''m afraid you''ll have to do some debugging. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, I made the experiment with httperf again, and used tcpdump to capture the packets on both server side and client side. I found that if there was some retransmission (e.t. client send several syn to server because it didn''t receive ack in time), the vif only receive the last packet (syn) and missed the former ones. So I think there are some issues happened between the eth0(veth0) and vif in the dom0. But if I use a low latency scheduler written by myself (only modify the client scheduler and didn''t touch other parts ), there will not be retransmission or there is very few retransmission. I am not familiar with netback of xen. Can you give me some suggestion? Or which part source code I need to check to find the reason of packets loss between eth0(veth0) and vif in dom0? Thanks. Regards, Cong 2011/9/30 Ian Campbell <Ian.Campbell@eu.citrix.com>:> On Fri, 2011-09-30 at 15:44 +0100, David Xu wrote: >> Hi, >> >> 2011/9/29 Ian Campbell <Ian.Campbell@citrix.com>: >> > On Fri, 2011-09-30 at 05:18 +0100, David Xu wrote: >> >> Hi, >> >> >> >> Does anybody know whether the ring buffer between front end and back >> >> end will suffer from overflow? I just wonder if the ring buffer will >> >> be full and drop some packets when the Net I/O load is very heavy. >> > >> > In the case of networking whichever end is putting stuff on the ring >> > checks that there is enough room and will stop the queue when it cannot >> > transmit any more and restart when room becomes available. >> >> You mean even there is not enough room in ring buffer, xen will * not >> drop the packets * and just delay the transmission. > > It''s not Xen but rather the kernel back and front ends which is involved > here. You can examine the hard_start_xmit functions in both netback and > netfront to determine for yourself whether or not packets can be dropped > and when. > >> I used httperf to >> measure the performance of web server running in a VM (The workload in >> this VM is mixed, so it can not benefit from boost mechanism. The net >> i/o suffers from relatively high latency which depends on the number >> of VMs in the system). I found that with the increase of request rate >> in client side, the connection rate will drop and the connection time >> will increase dramatically. And the retransmission appears when the >> request rate is over than a quantum. So I doubted that the http/tcp >> connection suffer from the packets drop when the ring buffer is full >> because of high request rate. >> >> > >> >> BTW, If I want to change the size of i/o ring buffer, how should I do? >> >> I tried to reset the NET_TX_RING_SIZE and NET_RX_RING_SIZE in both >> >> front end and back end, but it seems doesn''t work. Thanks. >> > >> > Currently the rings are limited to 1 page so if you want to increase the >> > size you would need to add multipage ring support to the network >> > protocol. There have been patches to do this for the blk protocol but I >> > do not recall any for the net protocol. >> >> Yes, increasing the size is relatively hard. So I just want reduce the >> size of ring buffer to make sure my doubt described above. I directly >> set NET_TX_RING_SIZE and NET_RX_RING_SIZE to 128, but it doesn''t seem to work. > > You need to make sure both ends of the connection agree on the ring > size. > > I''m afraid this is not a very common thing to want to do so if you want > to persist with this approach I''m afraid you''ll have to do some > debugging. > > Ian. > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Please don''t top post. On Tue, 2011-10-04 at 16:17 +0100, David Xu wrote:> Hi, > > I made the experiment with httperf again, and used tcpdump to capture > the packets on both server side and client side. I found that if there > was some retransmission (e.t. client send several syn to server > because it didn''t receive ack in time), the vif only receive the last > packet (syn) and missed the former ones. So I think there are some > issues happened between the eth0(veth0) and vif in the dom0. But if I > use a low latency scheduler written by myself (only modify the client > scheduler and didn''t touch other parts ), there will not be > retransmission or there is very few retransmission. I am not familiar > with netback of xen. Can you give me some suggestion? Or which part > source code I need to check to find the reason of packets loss between > eth0(veth0) and vif in dom0? Thanks.The netback code lives in either drivers/xen/netback or drivers/net/xen-netback depending on the kernel you are using. The interesting code is generally split between netback.c and interface.c.>From the point of view of the kernel it is just a normal network driver.For packets going from dom0->domU you can look at the .ndo_start_xmit callback registered with the kernel (xenvif_start_xmit in mainline, something else in some other kernels) and trace from there to see how and where they can be dropped. For packets coming the other way (domU->dom0) you can either start from the place which calls netif_rx_ni() and work backwards or start from the interrupt handler, i.e. the function passed to bind_interdomain_evtchn_to_irqhandler() and work forwards. You might also find starting from the *_tx_action and *_rx_action functions useful starting points for tracing what is going on.. Ian.> > Regards, > Cong > > 2011/9/30 Ian Campbell <Ian.Campbell@eu.citrix.com>: > > On Fri, 2011-09-30 at 15:44 +0100, David Xu wrote: > >> Hi, > >> > >> 2011/9/29 Ian Campbell <Ian.Campbell@citrix.com>: > >> > On Fri, 2011-09-30 at 05:18 +0100, David Xu wrote: > >> >> Hi, > >> >> > >> >> Does anybody know whether the ring buffer between front end and back > >> >> end will suffer from overflow? I just wonder if the ring buffer will > >> >> be full and drop some packets when the Net I/O load is very heavy. > >> > > >> > In the case of networking whichever end is putting stuff on the ring > >> > checks that there is enough room and will stop the queue when it cannot > >> > transmit any more and restart when room becomes available. > >> > >> You mean even there is not enough room in ring buffer, xen will * not > >> drop the packets * and just delay the transmission. > > > > It''s not Xen but rather the kernel back and front ends which is involved > > here. You can examine the hard_start_xmit functions in both netback and > > netfront to determine for yourself whether or not packets can be dropped > > and when. > > > >> I used httperf to > >> measure the performance of web server running in a VM (The workload in > >> this VM is mixed, so it can not benefit from boost mechanism. The net > >> i/o suffers from relatively high latency which depends on the number > >> of VMs in the system). I found that with the increase of request rate > >> in client side, the connection rate will drop and the connection time > >> will increase dramatically. And the retransmission appears when the > >> request rate is over than a quantum. So I doubted that the http/tcp > >> connection suffer from the packets drop when the ring buffer is full > >> because of high request rate. > >> > >> > > >> >> BTW, If I want to change the size of i/o ring buffer, how should I do? > >> >> I tried to reset the NET_TX_RING_SIZE and NET_RX_RING_SIZE in both > >> >> front end and back end, but it seems doesn''t work. Thanks. > >> > > >> > Currently the rings are limited to 1 page so if you want to increase the > >> > size you would need to add multipage ring support to the network > >> > protocol. There have been patches to do this for the blk protocol but I > >> > do not recall any for the net protocol. > >> > >> Yes, increasing the size is relatively hard. So I just want reduce the > >> size of ring buffer to make sure my doubt described above. I directly > >> set NET_TX_RING_SIZE and NET_RX_RING_SIZE to 128, but it doesn''t seem to work. > > > > You need to make sure both ends of the connection agree on the ring > > size. > > > > I''m afraid this is not a very common thing to want to do so if you want > > to persist with this approach I''m afraid you''ll have to do some > > debugging. > > > > Ian. > > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, I want to use the libxc function xc_sched_credit_domain_set( int xc_handle, uint32_t domid, struct xen_domctl_sched_credit *sdom) in a user-level application. But I don''t know how to set the value of xc_handle. Is there somebody familiar with it? Thanks. Regards, Cong 2011/10/4 David Xu <davidxu06@gmail.com>:> Hi, > > I made the experiment with httperf again, and used tcpdump to capture > the packets on both server side and client side. I found that if there > was some retransmission (e.t. client send several syn to server > because it didn''t receive ack in time), the vif only receive the last > packet (syn) and missed the former ones. So I think there are some > issues happened between the eth0(veth0) and vif in the dom0. But if I > use a low latency scheduler written by myself (only modify the client > scheduler and didn''t touch other parts ), there will not be > retransmission or there is very few retransmission. I am not familiar > with netback of xen. Can you give me some suggestion? Or which part > source code I need to check to find the reason of packets loss between > eth0(veth0) and vif in dom0? Thanks. > > Regards, > Cong > > 2011/9/30 Ian Campbell <Ian.Campbell@eu.citrix.com>: >> On Fri, 2011-09-30 at 15:44 +0100, David Xu wrote: >>> Hi, >>> >>> 2011/9/29 Ian Campbell <Ian.Campbell@citrix.com>: >>> > On Fri, 2011-09-30 at 05:18 +0100, David Xu wrote: >>> >> Hi, >>> >> >>> >> Does anybody know whether the ring buffer between front end and back >>> >> end will suffer from overflow? I just wonder if the ring buffer will >>> >> be full and drop some packets when the Net I/O load is very heavy. >>> > >>> > In the case of networking whichever end is putting stuff on the ring >>> > checks that there is enough room and will stop the queue when it cannot >>> > transmit any more and restart when room becomes available. >>> >>> You mean even there is not enough room in ring buffer, xen will * not >>> drop the packets * and just delay the transmission. >> >> It''s not Xen but rather the kernel back and front ends which is involved >> here. You can examine the hard_start_xmit functions in both netback and >> netfront to determine for yourself whether or not packets can be dropped >> and when. >> >>> I used httperf to >>> measure the performance of web server running in a VM (The workload in >>> this VM is mixed, so it can not benefit from boost mechanism. The net >>> i/o suffers from relatively high latency which depends on the number >>> of VMs in the system). I found that with the increase of request rate >>> in client side, the connection rate will drop and the connection time >>> will increase dramatically. And the retransmission appears when the >>> request rate is over than a quantum. So I doubted that the http/tcp >>> connection suffer from the packets drop when the ring buffer is full >>> because of high request rate. >>> >>> > >>> >> BTW, If I want to change the size of i/o ring buffer, how should I do? >>> >> I tried to reset the NET_TX_RING_SIZE and NET_RX_RING_SIZE in both >>> >> front end and back end, but it seems doesn''t work. Thanks. >>> > >>> > Currently the rings are limited to 1 page so if you want to increase the >>> > size you would need to add multipage ring support to the network >>> > protocol. There have been patches to do this for the blk protocol but I >>> > do not recall any for the net protocol. >>> >>> Yes, increasing the size is relatively hard. So I just want reduce the >>> size of ring buffer to make sure my doubt described above. I directly >>> set NET_TX_RING_SIZE and NET_RX_RING_SIZE to 128, but it doesn''t seem to work. >> >> You need to make sure both ends of the connection agree on the ring >> size. >> >> I''m afraid this is not a very common thing to want to do so if you want >> to persist with this approach I''m afraid you''ll have to do some >> debugging. >> >> Ian. >> >> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stop top posting, I''ve asked you this before. On Fri, 2011-10-07 at 16:23 +0100, David Xu wrote:> Hi, > > I want to use the libxc function xc_sched_credit_domain_set( int > xc_handle, uint32_t domid, struct xen_domctl_sched_credit *sdom) in a > user-level application. But I don''t know how to set the value of > xc_handle. Is there somebody familiar with it? Thanks.You can''t use xc_* functions in the kernel, they are userspace. You could in theory look at the implementation of xc_sched_credit_domain_set to find out what that hypercall is and make the underlying hypercall instead, But: The hypercall in question here is a domctl which are not available / allowed from kernel space (for policy and semantic reasons rather than technological ones). As I explained elsewhere I don''t think the architecture you have in mind is correct. I think you should look at exporting the necessary statistics from the kernel to userspace and have a daemon implement the logic and mechanisms for changing the scheduling parameters as required. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
At 16:49 +0100 on 07 Oct (1318006180), Ian Campbell wrote:> Stop top posting, I''ve asked you this before.Also, please start a new thread for a new question; the subject line of your email didn''t match the contents.> On Fri, 2011-10-07 at 16:23 +0100, David Xu wrote: > > Hi, > > > > I want to use the libxc function xc_sched_credit_domain_set( int > > xc_handle, uint32_t domid, struct xen_domctl_sched_credit *sdom) in a > > user-level application. But I don''t know how to set the value of > > xc_handle. Is there somebody familiar with it? Thanks. > > You can''t use xc_* functions in the kernel, they are userspace.He''s programming in userspace. :) You need to call xc_interface_open() to get an xc_handle. Please read through the rest of xenctrl.h before asking any more coding questions here. You could also have found the answer by looking at any of the other programs in the tools/ directore that use the xc interface. Cheers, Tim. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, This time I want to call it in the user-space. I wrote a user-level application and call the xc_sched_credit_domain_set( int xc_handle, uint32_t domid, struct xen_domctl_sched_credit *sdom) in it. But I don''t know how to pass the xc_handle parameter in this function. Thanks. Regards, Cong 2011/10/7 Ian Campbell <Ian.Campbell@eu.citrix.com>:> Stop top posting, I''ve asked you this before. > > On Fri, 2011-10-07 at 16:23 +0100, David Xu wrote: >> Hi, >> >> I want to use the libxc function xc_sched_credit_domain_set( int >> xc_handle, uint32_t domid, struct xen_domctl_sched_credit *sdom) in a >> user-level application. But I don''t know how to set the value of >> xc_handle. Is there somebody familiar with it? Thanks. > > You can''t use xc_* functions in the kernel, they are userspace. You > could in theory look at the implementation of xc_sched_credit_domain_set > to find out what that hypercall is and make the underlying hypercall > instead, But: > > The hypercall in question here is a domctl which are not available / > allowed from kernel space (for policy and semantic reasons rather than > technological ones). > > As I explained elsewhere I don''t think the architecture you have in mind > is correct. I think you should look at exporting the necessary > statistics from the kernel to userspace and have a daemon implement the > logic and mechanisms for changing the scheduling parameters as required. > > Ian. > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Oct-07 16:46 UTC
Re: libxc handles (was Re: [Xen-devel] ring buffer overflow)
On Fri, 2011-10-07 at 16:56 +0100, Tim Deegan wrote:> At 16:49 +0100 on 07 Oct (1318006180), Ian Campbell wrote: > > Stop top posting, I''ve asked you this before. > > Also, please start a new thread for a new question; the subject line of > your email didn''t match the contents. > > > On Fri, 2011-10-07 at 16:23 +0100, David Xu wrote: > > > Hi, > > > > > > I want to use the libxc function xc_sched_credit_domain_set( int > > > xc_handle, uint32_t domid, struct xen_domctl_sched_credit *sdom) in a > > > user-level application. But I don''t know how to set the value of > > > xc_handle. Is there somebody familiar with it? Thanks. > > > > You can''t use xc_* functions in the kernel, they are userspace. > > He''s programming in userspace. :)So he is, I was confused by the existing context... Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2011-10-07 at 17:23 +0100, David Xu wrote:> Hi, > > This time I want to call it in the user-space. I wrote a user-level > application and call the xc_sched_credit_domain_set( int > xc_handle, uint32_t domid, struct xen_domctl_sched_credit *sdom) in > it. But I don''t know how to pass the xc_handle parameter in this > function. Thanks.Tim already answered this. But why do you persist in top posting? You''ve been asked not to _repeatedly_. Please re-read http://wiki.xen.org/xenwiki/AskingXenDevelQuestions. In particular the etiquette section. Perhaps I should start ignoring top posted emails. :-/ Ian.> > Regards, > Cong > > 2011/10/7 Ian Campbell <Ian.Campbell@eu.citrix.com>: > > Stop top posting, I''ve asked you this before. > > > > On Fri, 2011-10-07 at 16:23 +0100, David Xu wrote: > >> Hi, > >> > >> I want to use the libxc function xc_sched_credit_domain_set( int > >> xc_handle, uint32_t domid, struct xen_domctl_sched_credit *sdom) in a > >> user-level application. But I don''t know how to set the value of > >> xc_handle. Is there somebody familiar with it? Thanks. > > > > You can''t use xc_* functions in the kernel, they are userspace. You > > could in theory look at the implementation of xc_sched_credit_domain_set > > to find out what that hypercall is and make the underlying hypercall > > instead, But: > > > > The hypercall in question here is a domctl which are not available / > > allowed from kernel space (for policy and semantic reasons rather than > > technological ones). > > > > As I explained elsewhere I don''t think the architecture you have in mind > > is correct. I think you should look at exporting the necessary > > statistics from the kernel to userspace and have a daemon implement the > > logic and mechanisms for changing the scheduling parameters as required. > > > > Ian. > > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Sorry, I did not notice his email and I did not mean to. I will be more careful next time. Regards, Cong 2011/10/7 Ian Campbell <Ian.Campbell@citrix.com>:> On Fri, 2011-10-07 at 17:23 +0100, David Xu wrote: >> Hi, >> >> This time I want to call it in the user-space. I wrote a user-level >> application and call the xc_sched_credit_domain_set( int >> xc_handle, uint32_t domid, struct xen_domctl_sched_credit *sdom) in >> it. But I don''t know how to pass the xc_handle parameter in this >> function. Thanks. > > Tim already answered this. But why do you persist in top posting? You''ve > been asked not to _repeatedly_. > > Please re-read http://wiki.xen.org/xenwiki/AskingXenDevelQuestions. In > particular the etiquette section. > > Perhaps I should start ignoring top posted emails. :-/ > > Ian. > >> >> Regards, >> Cong >> >> 2011/10/7 Ian Campbell <Ian.Campbell@eu.citrix.com>: >> > Stop top posting, I''ve asked you this before. >> > >> > On Fri, 2011-10-07 at 16:23 +0100, David Xu wrote: >> >> Hi, >> >> >> >> I want to use the libxc function xc_sched_credit_domain_set( int >> >> xc_handle, uint32_t domid, struct xen_domctl_sched_credit *sdom) in a >> >> user-level application. But I don''t know how to set the value of >> >> xc_handle. Is there somebody familiar with it? Thanks. >> > >> > You can''t use xc_* functions in the kernel, they are userspace. You >> > could in theory look at the implementation of xc_sched_credit_domain_set >> > to find out what that hypercall is and make the underlying hypercall >> > instead, But: >> > >> > The hypercall in question here is a domctl which are not available / >> > allowed from kernel space (for policy and semantic reasons rather than >> > technological ones). >> > >> > As I explained elsewhere I don''t think the architecture you have in mind >> > is correct. I think you should look at exporting the necessary >> > statistics from the kernel to userspace and have a daemon implement the >> > logic and mechanisms for changing the scheduling parameters as required. >> > >> > Ian. >> > >> > >> > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel