Greetings: I''m trying to set up an entirely diskless Xen system and have found a problem. Before I spend a lot more time hunting it down, I wanted to see if anyone else had already figured it out. I have a machine which has a disk with the following configuration: /dev/hda1 - Linux image /dev/hda2 - Swap /dev/hda5 - Linux image /dev/hda6 - swap If I boot with DOM0 using /dev/hda1 and DOM1 using /dev/hda5, everything works fine. DOM0 gets a "real" IP address, plus 169.254.1.0 (after running xen_nat_enable). Booting DOM1 from /dev/hda5 works, and gets it 169.254.1.1. If I move the contents /dev/hda5 to an NFS server, I can boot DOM1 diskless and everything is fine. If I move the contents of /dev/hda1 to the NFS server, I can boot DOM0 fine via NFS, but if I boot DOM1 from either /dev/hda6 or NFS, the 169.254.1.1 IP address does not work (when I boot diskless, this prevents it from booting, of course, but when I boot from the disk it boots fine but has broken networking). An ifconfig shows the address of the lan card as being correct, but no packets actually get delivered. If I instead assign it a "real" IP address on it''s subnet, everything works as well. This has lead me to conclude that booting DOM0 diskless breaks the NAT functionality, and I''m wondering if any networking guru''s out there might have any ideas about where to go from here (or if someone has DOM0+NAT working diskless, I''d love to hear about that as well). I suspect that it is somehow related to the fact that booting diskless causes the LAN card to be initialed very early in the boot process by the kernel, and it must not be getting initialized the same as if it was set up later. Thanks, Bret ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> If I move the contents of /dev/hda1 to the NFS server, I can boot DOM0 > fine via NFS, but if I boot DOM1 from either /dev/hda6 or NFS, the > 169.254.1.1 IP address does not work (when I boot diskless, this > prevents it from booting, of course, but when I boot from the disk it > boots fine but has broken networking). An ifconfig shows the address of > the lan card as being correct, but no packets actually get delivered. If > I instead assign it a "real" IP address on it''s subnet, everything works > as well.169.254/16 addresses are VMM-local: they''re never allowed out on the LAN. For your purposes of accessing an external NFS server use some other subnet e.g. 192.168/16. (If you''re using Xen 1.2, still give dom0 the 169.254.1.0 alias so that you can get the console. Console works totally differently in 1.3 so this is no longer required.) Ian ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Thu, 2004-04-08 at 10:27, Ian Pratt wrote:> 169.254/16 addresses are VMM-local: they''re never allowed out on > the LAN. > > For your purposes of accessing an external NFS server use some > other subnet e.g. 192.168/16.I don''t think it is a routing problem, and (despite my long post :-), I must have not been clear enough. If I boot DOM0 diskfull , and then boot DOM1 diskful or diskless, I can ping DOM1 from DOM0 (ping 169.254.1.1 works), but if I boot DOM0 diskless, and DOM1 either diskless or diskfull, the ping command fails. Bret ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> On Thu, 2004-04-08 at 10:27, Ian Pratt wrote: > > > 169.254/16 addresses are VMM-local: they''re never allowed out on > > the LAN. > > > > For your purposes of accessing an external NFS server use some > > other subnet e.g. 192.168/16. > > I don''t think it is a routing problem, and (despite my long post :-), I > must have not been clear enough. > > If I boot DOM0 diskfull , and then boot DOM1 diskful or diskless, I can > ping DOM1 from DOM0 (ping 169.254.1.1 works), but if I boot DOM0 > diskless, and DOM1 either diskless or diskfull, the ping command fails.I don''t think NAT''ing NFS traffic is going to work -- to my knowledge Linux iptables hasn''t got state-full masquerading for NFS. It''s a bit tricky with all the portmapper and sunrpc interaction. Any reason why you can''t just give both dom0 and dom1 LAN-visible IP addresses? However, it is odd that you can''t ping dom1 even if it is booted diskfull. It might be worth trying to back this change out, but I can''t see why it would cause problems: http://xen.bkbits.net:8080/xeno-1.2.bk/diffs/xen/net/dev.c@1.73?nav=index.html|src/.|src/xen|src/xen/net|hist/xen/net/dev.c Ian ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Greetings: Ian and I exchanged more email off list, and he eventually asked the right question. For reasons that I don''t understand (but presumably which are related to how things get added to the routing tables differently if the kernel initialized the networking instead of the startup scripts) the routing tables are slightly different if DOM0 is diskless. On a diskfull system, the relevant portions look like: Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 and on a diskless system the look like: Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 lo 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 Notice that the diskless system''s first entry is to lo, not to eth0. This was the root of the problem. I fixed it with: route del -net 169.254.0.0 gw 0.0.0.0 netmask 255.255.0.0 lo With that change everything works as expected, and I can boot totally disless. I had looked at the routing tables earlier, but missed that difference. Thanks Ian. Bret ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel