Anyone know how I can automatically set the MTU on a vif when I create
the domain? I am using AoE to provide a SAN for my virtual machines and
I create a bridge called stbr0 which I connect to eth1 in my domU. I
want it to have an mtu of 9000 for performance reasons.
I am using CentOS 5 and I see that the default vif creation script is
vif-bridge. I have tried changing vif-bridge like so:
case "$command" in
online)
setup_bridge_port "$vif"
add_to_bridge "$bridge" "$vif"
if [ ${bridge} = "stbr0" ]; then
ip link set ${dev} mtu 9000
fi
so if the bridge we are adding the vif to is stbr0 then we set the mtu
to 9000. But this causes the xm create command to hang and I''m not sure
why.
Any suggestions on how to get a vif set up with an mtu of 9000? It
doesn''t seem that I can just say mtu= in the vif= line in my domain
config file.
--
Tracy R Reed Read my blog at http://ultraviolet.org
Key fingerprint = D4A8 4860 535C ABF8 BA97 25A6 F4F2 1829 9615 02AD
Non-GPG signed mail gets read only if I can find it among the spam.
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
I would put a line somewhere in your startup (after the network is up) to set the MTU. It can be done on the command line using the ifconfig command. I haven''t done this myself, but it seems like it should be fairly straightforward. BTW, which AoE unit are you using (presuming it''s Coraid)? Is it the 15xx series, or the 16xx series? I''m looking at buying one this week and would be very interested in how it performs in this type of setup. <offtopic slightly> I''m looking at buying a new server and passing the AoE functions through to one or more DomU''s via exported ethernet interfaces (I will have 6 or so) If it''s not feasable, I would love to know BEFORE I buy it. :) </offtopic slightly> Hth, Phil. ----- Original Message ----- From: "Tracy R Reed" <treed@ultraviolet.org> To: <xen-users@lists.xensource.com> Sent: Tuesday, March 25, 2008 7:05 AM Subject: [Xen-users] Changing MTU of vif> Anyone know how I can automatically set the MTU on a vif when I create the > domain? I am using AoE to provide a SAN for my virtual machines and I > create a bridge called stbr0 which I connect to eth1 in my domU. I want it > to have an mtu of 9000 for performance reasons. > > I am using CentOS 5 and I see that the default vif creation script is > vif-bridge. I have tried changing vif-bridge like so: > > case "$command" in > online) > setup_bridge_port "$vif" > add_to_bridge "$bridge" "$vif" > if [ ${bridge} = "stbr0" ]; then > ip link set ${dev} mtu 9000 > fi > > so if the bridge we are adding the vif to is stbr0 then we set the mtu to > 9000. But this causes the xm create command to hang and I''m not sure why. > > Any suggestions on how to get a vif set up with an mtu of 9000? It doesn''t > seem that I can just say mtu= in the vif= line in my domain config file. > > -- > Tracy R Reed Read my blog at http://ultraviolet.org > Key fingerprint = D4A8 4860 535C ABF8 BA97 25A6 F4F2 1829 9615 02AD > Non-GPG signed mail gets read only if I can find it among the spam. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Phillip Bennett wrote:> I would put a line somewhere in your startup (after the network is up) > to set the MTU. It can be done on the command line using the ifconfig > command.You mean in the network startup in the DomU? That won''t change the MTU on the vif in the Dom0. I do have it set inside the DomU to be set to 9000. I can do it on the command line in the dom0 after startup but that is a real pain and has the nasty consequence that your disk behaves very unreliably until you look through that long list of vifs and figure out which one belongs to your domU and connected to the storage bridge and manually change it. Not acceptable.> BTW, which AoE unit are you using (presuming it''s Coraid)? Is it the > 15xx series, or the 16xx series? I''m looking at buying one this week > and would be very interested in how it performs in this type of setup.Neither. I have built my own. A supermicro 3u 15 drive case with triple redundant power, an Intel mobo with Intel cpu and 2G of RAM, and a pair of Gig-e network cards bonded with 802.3ad, and an Adaptec 31605 RAID controller. I run qaoed on it by just slicing off a piece of disk with LVM and point qaoed at the lv. I have two of these set up identically and the dom0 does software mirroring between them. So I could have a network link fail or even a whole disk server fail and not have any problems. I purchased a couple of the Coraid units about a year ago and found that their implementation of LVM is not flexible enough. It can only export whole disk devices and cannot export just a 50G slice of disk which is what I need. So I had to send them back. They don''t run Linux and Linux''s LVM in their Coraid units. They run Plan 9.> <offtopic slightly> > I''m looking at buying a new server and passing the AoE functions through > to one or more DomU''s via exported ethernet interfaces (I will have 6 or > so) If it''s not feasable, I would love to know BEFORE I buy it. :) > </offtopic slightly>AoE with Xen works perfectly for me. The only problem is that it is rather challenging to set up. You should be a pretty experienced sysadmin to get it going. I have set up a website at http://xenaoe.org where I am documenting my whole setup and promoting the idea. I still need to get my initrd up there (pretty custom, diskless cpu nodes, etc.) and a number of other fixes and optimizations which I have made. It has been a LOT of work to get it all going. So I am working to share everything I have learned with the rest of you folks so you can set this up also. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tracy R Reed wrote:> case "$command" in > online) > setup_bridge_port "$vif" > add_to_bridge "$bridge" "$vif" > if [ ${bridge} = "stbr0" ]; then > ip link set ${dev} mtu 9000 > fiOops...that ${dev} should have been ${vif}. But either way I still have the same problem. The xm create command just hangs. Ok, another take on this: Anyone know how I can debug what is going on inside the vif-bridge script? xm create is swallowing up its stdin and I am going through the python code trying to figure out where I can tap into it myself to see what is going on. I do see people pasting what seems to be output from this script running set -x but I don''t know how to get that. I have included some echo statements to a file in vif-bridge and that seems to work so I know that the $vif and $bridge are set to what I expect. I can run the above ip link command from the command line after having created the domain and everything seems to work. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Subject: Re: [Xen-users] Changing MTU of vif > > Tracy R Reed wrote: > > case "$command" in > > online) > > setup_bridge_port "$vif" > > add_to_bridge "$bridge" "$vif" > > if [ ${bridge} = "stbr0" ]; then > > ip link set ${dev} mtu 9000 > > fi > > Oops...that ${dev} should have been ${vif}. But either way I still have > the same problem. The xm create command just hangs. > > Ok, another take on this: Anyone know how I can debug what is going on > inside the vif-bridge script?You can, for example, put "set -x" at the top of the /etc/xen/scripts/vif-common.sh script and then you get the output of that in /var/log/xen-hotplug.log. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I''ve been vaguely following this discussion, and AFAIK, xen does not provide a mechanism to keep the backend and frontend MTU''s in sync. You''d need a xenstore settings in the frontend and the backend, eg the backend would specify a minimum possible MTU and a maximum possible MTU, and the frontend would specify the MTU it uses. Minimum and maximum MTU would be specified at domain configuration time (defaulting to 1500/1500 probably). The backend would set it''s interface to the value the frontend specifies, eg: Backend: backend/vif/<id>/<vif>/min-MTU = 576 backend/vif/<id>/<vif>/max-MTU = 1500 Frontend Device/vif/<vif>/MTU = 1500 Possibly it would only be legal to change the MTU value while the interface was not in the "connected" state... Does anyone know if, on a linux bridge, there is any reason why all attached devices wouldn''t have the same MTU? James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Phillip Bennett wrote:> I see your problem. I had the impression you were exporting the > hardware to the DomU. (I guess that''s what I am going to do, I just > assumed) I guess actually finding the interface you need would be a lot > harder.. I haven''t had to do this, so can''t offer anything useful.Actually, it shouldn''t be hard at all if I could just get my trivial change to vif-bridge to work: case "$command" in online) setup_bridge_port "$vif" add_to_bridge "$bridge" "$vif" if [ ${bridge} = "stbr0" ]; then ip link set ${dev} mtu 9000 fi That should set the mtu on the vif being connected to my SAN bridge when the domain is created. But for some reason this seems to hang.> That sounds like a sweet setup! Do you think it''s faster than the > Coraid units then? How is it pricewise? (I''m guessing it''s a LOT > cheaper...)I don''t know if it is any faster since I use nearly identical hardware: Same mobo, chassis, cpu, etc. Main different is the RAID controller. It is quite a bit cheaper. But the main reason I like it is that it runs Linux so I can use Linux''s LVM to slice the disk however I want. Coraids volume management is a bit crude.> Fair enough. I''d say I''m faily experienced... Hopefully enough to get > it up and running satisfactorily. :)Cool. Keep an eye on xenaoe.org as I am writing up a detailed howto and will be opening up all of my code in svn soon. Yesterday I added a HOWTO build your own zero single point of failure cluster which so far includes a parts/price list. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Harper wrote:> I''ve been vaguely following this discussion, and AFAIK, xen does not > provide a mechanism to keep the backend and frontend MTU''s in sync.This isn''t really what I''m asking for although it would be nice. I just want to automatically set the MTU on the vif when the domain is created. I can setup my network scripts inside the domain to hard code eth1 (always my SAN interface in a domU) to MTU 9000.> Possibly it would only be legal to change the MTU value while the > interface was not in the "connected" state...I have noticed that I cannot change the MTU on the bridge when other devices are attached with a different MTU. I can set the MTU of the vif attached to the bridge to 9000. Only after setting the vif to 9000 can I set the bridge to 9000. Then I also have to set the MTU on eth1 in the domU to 9000. Then everything works perfectly. I can set MTU=9000 in the /etc/sysconfig/network-scripts/ifcfg-eth1 file. But I need to find a way to automate setting MTU to 9000 on the bridge and vif. I notice that if I create a domU and do all of the above and get the MTU on everything set to 9000 and then I create another domU the MTU on the bridge gets set back to 1500 and the new vif is also 1500. The first vif created on the SAN bridge still has an MTU of 9000. So now I have to set the new vif to 9000, then I can set the bridge back to 9000 and everything works fine again until I add another domU at which point I have to do this dance again.> Does anyone know if, on a linux bridge, there is any reason why all > attached devices wouldn''t have the same MTU?I can''t think of any reason. It would seem to make sense. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fischer, Anna wrote:> You can, for example, put "set -x" at the top of the /etc/xen/scripts/vif-common.sh script and then you get the output of that in /var/log/xen-hotplug.log.This was actually the first thing I tried but I put set +x instead of -x which apparently doesn''t work. :) Thanks for the tip! I have since corrected that and now get the following in the xen-hotplug.log: + add_to_bridge stbr0 vif28.1 + local bridge=stbr0 + local dev=vif28.1 + ''['' -e /sys/class/net/stbr0/brif/vif28.1 '']'' + brctl addif stbr0 vif28.1 + ip link set vif28.1 up + ''['' stbr0 = stbr0 '']'' + ip link set vif28.1 mtu 9000 SIOCSIFMTU: Invalid argument ++ sigerr ++ fatal ''/etc/xen/scripts/vif-bridge failed; error detected.'' So we are in the add_to_bridge function and my little test for checking that this is the vif being added to my SAN bridge works and then we try to set the mtu to 9000 and get "Invalid argument". Strange thing is that this command works from the command line right after I normally set up a domain so I''m not sure what the problem is. I should be able to set the mtu on the vif and then the mtu on the bridge. While looking through the scripts I noticed that the bridge is taken down momentarily while setting everything up so that would explain why the mtu on the bridge keeps getting reset to 1500 every time I create a domain. Entire output is here in case anyone is interested: http://pastebin.ca/958683 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
unsbscribe _________________________________________________________________ Send a smile, make someone laugh, have some fun! Start now! http://g.msn.ca/ca55/208 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tracy R Reed wrote:> Anyone know how I can automatically set the MTU on a vif when I create > the domain?It was pointed out on IRC that perhaps the reason I cannot set the mtu at this point in the script is that the frontend inside the domU may not be up yet. And as I need to get this system into production and don''t want to spend any more time on this issue I have just written a script and cron''d it to run every minute that will go through /sys/class/net/stbr0/brif/ and set the mtu on all of the interfaces it finds there to 9000. I''ll be keeping an eye out for a more proper way to solve this problem but I''m done for now. Thanks all! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Am Wednesday, den 26 March hub Tracy R Reed folgendes in die Tasten:> Tracy R Reed wrote: > >Anyone know how I can automatically set the MTU on a vif when I create > >the domain? > > It was pointed out on IRC that perhaps the reason I cannot set the mtu > at this point in the script is that the frontend inside the domU may not > be up yet. And as I need to get this system into production and don''t > want to spend any more time on this issue I have just written a script > and cron''d it to run every minute that will go through > /sys/class/net/stbr0/brif/ and set the mtu on all of the interfaces it > finds there to 9000.> I''ll be keeping an eye out for a more proper way to solve this problem > but I''m done for now. Thanks all!#define HACK_VALUE As you wrote it is some kind of timeing related problem. Maybe the following hack might help: Instead of just executing the MTU fiddling stuff right when setting up the vif, delay the setup for $time like this: ( sleep $time set mtu ... )& HTH Ciao Max -- Follow the white penguin. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wednesday 26 March 2008 01:42:34 pm Tracy R Reed wrote:> case "$command" in > online) > setup_bridge_port "$vif" > add_to_bridge "$bridge" "$vif" > if [ ${bridge} = "stbr0" ]; then > ip link set ${dev} mtu 9000 > fiYour cron idea is nice. Have you tried just changing the order of setting the MTU, ie - before it is added to the bridge?: case "$command" in online) setup_bridge_port "$vif" if [ ${bridge} = "stbr0" ]; then ip link set stbr0 mtu 9000 ip link set ${dev} mtu 9000 add_to_bridge "$bridge" "$vif" fi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> AoE with Xen works perfectly for me. The only problem is that it is rather > challenging to set up. You should be a pretty experienced sysadmin to get it > going. > > I have set up a website at http://xenaoe.org where I am documenting my whole > setup and promoting the idea. I still need to get my initrd up there (prettyI''d like to have heard your thoughts on the various AOE target implimentations (vblade, kernel-vblade, qaoed, etc...) ---- I did come up with a crude fix for your vif-bridge problem... log debug "Successful vif-bridge $command for $vif, bridge $bridge." if [ "$command" == "online" ] then success fi after the "success" command, I added the following (sleep 10; ifconfig "$vif" mtu 5200) & e.g. log debug "Successful vif-bridge $command for $vif, bridge $bridge." if [ "$command" == "online" ] then success do_without_error ifconfig "$vif" mtu 5200 (sleep 10; ifconfig "$vif" mtu 5200) & fi If you''re not familiar with the syntax, the parenthesis cause a another shell to be forked with the enclosed commands, and the ampersand puts it in the background, so the main vif-bridge script basically forks a child to sleep and tweak the mtu after a delay. It is not pretty, it is not clean, but then having a script like the following run every 60 seconds isn''t clean or pretty either: #!/bin/bash # cd /sys/class/net/xenbr0/brif/ for f in * ; do echo ifconfig $f mtu 5200; ifconfig $f mtu 5200; done (why did I pick 5200? Because I wanted the system to use 4096 byte aoe transfers. ) (Oh, and from what I have seen, the MTU on the bridge device doesn''t need to be set... it is automatically the smallest MTU of any of the interfaces attached to the bridge.) -Tom _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
"Tracy R Reed" <treed@ultraviolet.org> Wrote:> Phillip Bennett wrote: >> I would put a line somewhere in your startup (after the network is up) to >> set the MTU. It can be done on the command line using the ifconfig >> command. > > You mean in the network startup in the DomU? That won''t change the MTU on > the vif in the Dom0. I do have it set inside the DomU to be set to 9000. I > can do it on the command line in the dom0 after startup but that is a real > pain and has the nasty consequence that your disk behaves very unreliably > until you look through that long list of vifs and figure out which one > belongs to your domU and connected to the storage bridge and manually > change it. Not acceptable. >I see your problem. I had the impression you were exporting the hardware to the DomU. (I guess that''s what I am going to do, I just assumed) I guess actually finding the interface you need would be a lot harder.. I haven''t had to do this, so can''t offer anything useful.>> BTW, which AoE unit are you using (presuming it''s Coraid)? Is it the >> 15xx series, or the 16xx series? I''m looking at buying one this week and >> would be very interested in how it performs in this type of setup. > > Neither. I have built my own. A supermicro 3u 15 drive case with triple > redundant power, an Intel mobo with Intel cpu and 2G of RAM, and a pair of > Gig-e network cards bonded with 802.3ad, and an Adaptec 31605 RAID > controller. I run qaoed on it by just slicing off a piece of disk with LVM > and point qaoed at the lv. I have two of these set up identically and the > dom0 does software mirroring between them. So I could have a network link > fail or even a whole disk server fail and not have any problems. > > I purchased a couple of the Coraid units about a year ago and found that > their implementation of LVM is not flexible enough. It can only export > whole disk devices and cannot export just a 50G slice of disk which is > what I need. So I had to send them back. They don''t run Linux and Linux''s > LVM in their Coraid units. They run Plan 9.That sounds like a sweet setup! Do you think it''s faster than the Coraid units then? How is it pricewise? (I''m guessing it''s a LOT cheaper...)>> <offtopic slightly> >> I''m looking at buying a new server and passing the AoE functions through >> to one or more DomU''s via exported ethernet interfaces (I will have 6 or >> so) If it''s not feasable, I would love to know BEFORE I buy it. :) >> </offtopic slightly> > > AoE with Xen works perfectly for me. The only problem is that it is rather > challenging to set up. You should be a pretty experienced sysadmin to get > it going.Fair enough. I''d say I''m faily experienced... Hopefully enough to get it up and running satisfactorily. :)> I have set up a website at http://xenaoe.org where I am documenting my > whole setup and promoting the idea. I still need to get my initrd up there > (pretty custom, diskless cpu nodes, etc.) and a number of other fixes and > optimizations which I have made. It has been a LOT of work to get it all > going. So I am working to share everything I have learned with the rest of > you folks so you can set this up also.Looks great. I''ll be sure to read it regularly. Good Luck! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users