Xen users - Apr 2013 - bad network perfornance too many frags messages

Hi,
With a 3.7.10 kernel from openssue 12.3 my paravirtualized vm got really 
poor network performances.
and I''have this messages every half of a second in the logs:
kernel netfront: Too many frags

Any idea about what could be the cause and what test I can do?

I''ve read something about MAX_SKB_FAGS and offloading but it is not
that
clear for me.

Thank you.

> Hi,
> With a 3.7.10 kernel from openssue 12.3 my paravirtualized vm got really
> poor network performances.
> and I''have this messages every half of a second in the logs:
> kernel netfront: Too many frags
> 
> Any idea about what could be the cause and what test I can do?
> 
> I''ve read something about MAX_SKB_FAGS and offloading but it is
not that
> clear for me.
> 
Does turning off scatter gather on the bridge help at all?

I did ethtool -K br0 sg off
and still have lots of  (3 per seconds) "netfront: Too many frags"
messages
in the VM''slogs.


The problem occurs with the host connect at 1GB. If I connected to a 100Mb
ports the VM''s are ok.


2013/4/8 James Harper <james.harper@bendigoit.com.au>
> > Hi,
> > With a 3.7.10 kernel from openssue 12.3 my paravirtualized vm got
really
> > poor network performances.
> > and I''have this messages every half of a second in the logs:
> > kernel netfront: Too many frags
> >
> > Any idea about what could be the cause and what test I can do?
> >
> > I''ve read something about MAX_SKB_FAGS and offloading but it
is not that
> > clear for me.
> >
>
> Does turning off scatter gather on the bridge help at all?
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

Someone will growl at you if you top post.
> 
> I did ethtool -K br0 sg off
> 
> and still have lots of  (3 per seconds) "netfront: Too many
frags" messages in
> the VM''slogs.
> 
> The problem occurs with the host connect at 1GB. If I connected to a 100Mb
> ports the VM''s are ok.
> 
Check xenstore to see if sg is enabled in the device setup.

xenstore-ls /local/domain/<domid>/device/vif/0

or something like that. Also check the backend path (shown in the output of the
above).

James
> 
> 
> 2013/4/8 James Harper <james.harper@bendigoit.com.au>
> 
> 
> 	> Hi,
> 	> With a 3.7.10 kernel from openssue 12.3 my paravirtualized vm got
> really
> 	> poor network performances.
> 	> and I''have this messages every half of a second in the logs:
> 	> kernel netfront: Too many frags
> 	>
> 	> Any idea about what could be the cause and what test I can do?
> 	>
> 	> I''ve read something about MAX_SKB_FAGS and offloading but it
is
> not that
> 	> clear for me.
> 	>
> 
> 	Does turning off scatter gather on the bridge help at all?
> 
>

On Mon, Apr 08, 2013 at 05:11:30AM +0100, Alexis wrote:
> Hi,
> With a 3.7.10 kernel from openssue 12.3 my paravirtualized vm got really 
> poor network performances.
> and I''have this messages every half of a second in the logs:
> kernel netfront: Too many frags
> 
> Any idea about what could be the cause and what test I can do?
> 
> I''ve read something about MAX_SKB_FAGS and offloading but it is
not that
> clear for me.
> 
What''s your Dom0 kernel version?


Wei.
> Thank you.
> 
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users

xenstore-ls /local/domain/14/device/vif/0

show me
backend = "/local/domain/0/backend/vif/14/0"
feature-sg = "1"


xenstore-ls /local/domain/0/backend/vif/14/0
feature-sg = "1"

So I guess it means sg is enable on both backend and frontend.


Sorry for the top post I was just "reply to all" in my mailer.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

uname -a
Linux yuva2 3.7.10-1.1-xen #1 SMP Thu Feb 28 15:06:29 UTC 2013 (82d3f21)
x86_64 x86_64 x86_64 GNU/Linux



2013/4/8 Wei Liu <wei.liu2@citrix.com>
> On Mon, Apr 08, 2013 at 05:11:30AM +0100, Alexis wrote:
> > Hi,
> > With a 3.7.10 kernel from openssue 12.3 my paravirtualized vm got
really
> > poor network performances.
> > and I''have this messages every half of a second in the logs:
> > kernel netfront: Too many frags
> >
> > Any idea about what could be the cause and what test I can do?
> >
> > I''ve read something about MAX_SKB_FAGS and offloading but it
is not that
> > clear for me.
> >
>
> What''s your Dom0 kernel version?
>
>
> Wei.
>
> > Thank you.
> >
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xen.org
> > http://lists.xen.org/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

On Mon, Apr 08, 2013 at 01:35:32PM +0100, Alexis Pellicier
wrote:
> xenstore-ls /local/domain/14/device/vif/0
> 
> show me
> backend = "/local/domain/0/backend/vif/14/0"
> feature-sg = "1"
> 
> 
> xenstore-ls /local/domain/0/backend/vif/14/0
> feature-sg = "1"
> 
> So I guess it means sg is enable on both backend and frontend.
> 
SG is always used whenever possible. This has nothing to do with the
bridge.

Unfortunately there is no option to disable it. If you fancy recompiling
backend driver, look at drivers/net/xen-netback/xenbus.c:netback_probe
and change ''sg = 1'' to ''sg = 0''.


Wei.
> 
> Sorry for the top post I was just "reply to all" in my mailer.
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users

Ok thank for your help
Here my report on what I did:

-edit sg=0 in xenbus.c
-recompile xenbus_be.ko netbk.ko xennet.ko
-replace these 3 modules in my Dom0 reboot
-replace these 3 modules in my DomU reboot
-xenstore-ls gave feature-sg=0 for backend
-xenstore-ls gave feature-sg=1 for frontend. (I couldn’t disable it)

My test still show me really bad transfer rate and those Too may frags
kernel error in DomU logs.

Any further clue?



2013/4/8 Wei Liu <wei.liu2@citrix.com>
> On Mon, Apr 08, 2013 at 01:35:32PM +0100, Alexis Pellicier wrote:
> > xenstore-ls /local/domain/14/device/vif/0
> >
> > show me
> > backend = "/local/domain/0/backend/vif/14/0"
> > feature-sg = "1"
> >
> >
> > xenstore-ls /local/domain/0/backend/vif/14/0
> > feature-sg = "1"
> >
> > So I guess it means sg is enable on both backend and frontend.
> >
>
> SG is always used whenever possible. This has nothing to do with the
> bridge.
>
> Unfortunately there is no option to disable it. If you fancy recompiling
> backend driver, look at drivers/net/xen-netback/xenbus.c:netback_probe
> and change ''sg = 1'' to ''sg = 0''.
>
>
> Wei.
>
> >
> > Sorry for the top post I was just "reply to all" in my
mailer.
> >
>
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xen.org
> > http://lists.xen.org/xen-users
>
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

I''ve just checked on other server running opensuse12.2 ( 3.4.33 kernel)
both on the Dom0 and DomU.

sg-feature is on either on the backend and the frontend and transfer rate
is ok on the DomU with no error in the logs.

I was wondering if it could be related to opensuse distro?





2013/4/8 Alexis Pellicier <alexis.pellicier@nds.k12.tr>
> Ok thank for your help
> Here my report on what I did:
>
> -edit sg=0 in xenbus.c
> -recompile xenbus_be.ko netbk.ko xennet.ko
> -replace these 3 modules in my Dom0 reboot
> -replace these 3 modules in my DomU reboot
> -xenstore-ls gave feature-sg=0 for backend
> -xenstore-ls gave feature-sg=1 for frontend. (I couldn’t disable it)
>
> My test still show me really bad transfer rate and those Too may frags
> kernel error in DomU logs.
>
> Any further clue?
>
>
>
> 2013/4/8 Wei Liu <wei.liu2@citrix.com>
>
>> On Mon, Apr 08, 2013 at 01:35:32PM +0100, Alexis Pellicier wrote:
>> > xenstore-ls /local/domain/14/device/vif/0
>> >
>> > show me
>> > backend = "/local/domain/0/backend/vif/14/0"
>> > feature-sg = "1"
>> >
>> >
>> > xenstore-ls /local/domain/0/backend/vif/14/0
>> > feature-sg = "1"
>> >
>> > So I guess it means sg is enable on both backend and frontend.
>> >
>>
>> SG is always used whenever possible. This has nothing to do with the
>> bridge.
>>
>> Unfortunately there is no option to disable it. If you fancy
recompiling
>> backend driver, look at drivers/net/xen-netback/xenbus.c:netback_probe
>> and change ''sg = 1'' to ''sg = 0''.
>>
>>
>> Wei.
>>
>> >
>> > Sorry for the top post I was just "reply to all" in my
mailer.
>> >
>>
>> > _______________________________________________
>> > Xen-users mailing list
>> > Xen-users@lists.xen.org
>> > http://lists.xen.org/xen-users
>>
>>
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

On 4/8/2013 12:35 PM, Alexis Pellicier wrote:
> Any further clue?
I started seeing this "too many frags" error popping up in
dom0''s over
the last few months, when using the latest 4.2 builds. I''ve seen it
with
both Linux (various distros) and Windows domUs. For me, usually it 
results in Xen disabling the network adapter.

The workaround that I use is turning off TSO/GSO inside the domU:

ethtool -K eth0 tx off tso off gso off

You may also need to turn off GSO from the dom0 side:

ethtool -k vifX.0 tx off gso off

-John

ethtool -K vifx.0 tso off from Dom0 solves the issue
or
ethtool -K vifx.0 sg off from Dom0 solves the issue

Thank you!!