Hello all! Anyone here use AoE? I cannot seem to find an AoE specific mailing list. Not sure I want to bother the Coraid folks since I am not using Coraid equipment. Is there a better place for this query? I am so close to getting my cluster mounted with root (and then Xen domains) on AoE but my initrd is having problems getting AoE set up. I am using kernel vmlinuz-2.6.16.13-xen. AoE is compiled in statically. /sys/module/aoe/parameters/ contains a file called aoe_iflist AoE is mentioned in the dmesg when it boots. /sys/module/aoe/parameters/aoe_iflist contains the active ethernet interface. If I echo interface names into /dev/aoe/interfaces they appear in /sys/module/aoe/parameters/aoe_iflist so I know aoe is loaded and doing something. But when I do echo 1 > /dev/aoe/discover nothing happens. Nothing in the dmesg etc. Not sure where to go from here. I have vblade running on another machine which is directly connected to the diskless cpu node. No hub or switch involved, just cross connected ethernet. If I run tethereal on the vblade machine looking for some sort of AoE related traffic I see nothing coming across the wire. I do see dhcp requests coming from that interface when the box boots due to PXE trying to find a server on that interface etc. So I know the physical ethernet link is good. There really isn''t much written about AoE. It''s so simple that it seems like there shouldn''t be much to write about. Any ideas? Thanks! -- Tracy R Reed http://ultraviolet.org A: Because we read from top to bottom, left to right Q: Why should I start my reply below the quoted text _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wednesday 12 July 2006 1:26 pm, Tracy R Reed wrote:> Hello all! > > Anyone here use AoE? I cannot seem to find an AoE specific mailing list.i use it, but not (yet) with Xen> my initrd is having problems getting AoE set up. I am using kernel > vmlinuz-2.6.16.13-xen. > > AoE is compiled in statically.the AoE driver merged in the mainline Linux kernel is very old; better try to use the Coraid''s one (if compiled as a module, no need to change the kernel)> /sys/module/aoe/parameters/aoe_iflist contains the active > ethernet interface. If I echo interface names into /dev/aoe/interfaces > they appear in /sys/module/aoe/parameters/aoe_iflist so I know aoe is > loaded and doing something. > > But when I do echo 1 > /dev/aoe/discover nothing happens.are you using AoE in the Dom0, or DomU? if DomU, you can''t use routed network, since AoE isn''t IP, even on bridge, i wouldn''t be surprised if the bridge get''s in the way of discovery. also, if you use it in DomU, each packet would pass at least twice thought Dom0 before getting to the DomU. in short, the AoE initiator should be in Dom0> ethernet. If I run tethereal on the vblade machine looking for some sort > of AoE related traffic I see nothing coming across the wire. I do seeAgain, since AoE isn''t IP, you should look for ethernet frames type 0x88A2 -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Javier Guerra wrote:> the AoE driver merged in the mainline Linux kernel is very old; better try to > use the Coraid''s one (if compiled as a module, no need to change the kernel)Ah...ok, I''ll check this out.> are you using AoE in the Dom0, or DomU? if DomU, you can''t use routed network, > since AoE isn''t IP, even on bridge, i wouldn''t be surprised if the bridge > get''s in the way of discovery. also, if you use it in DomU, each packet > would pass at least twice thought Dom0 before getting to the DomU.I am using AoE in the Dom0. This machine is completely diskless. The machine boots, pxe gets an ip, it tftp''s xen, a kernel, and an initrd and mounts the initrd and runs the init script which loads some modules, loads the xen module, ip''s up an ethernet interface, and then we do echo 1 > /dev/aoe/discover to find aoe devices. As I understand it the bridges are ethernet bridges so they should be able to pass raw ethernet frames. But that does not matter because Dom0 will be getting the AoE devices and passing them to the DomU''s as block devices to be used.> in short, the AoE initiator should be in Dom0It definitely is.>> ethernet. If I run tethereal on the vblade machine looking for some sort >> of AoE related traffic I see nothing coming across the wire. I do see > > Again, since AoE isn''t IP, you should look for ethernet frames type 0x88A2Hmm...I thought ethereal would show us everything by default, even non-ip packets. I am compiling a non-xen kernel now just as a test to see if AoE works any better without xen. -- Tracy R Reed http://ultraviolet.org A: Because we read from top to bottom, left to right Q: Why should I start my reply below the quoted text _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thursday 13 July 2006 7:09 pm, Tracy R Reed wrote:> > in short, the AoE initiator should be in Dom0 > > It definitely is.good. nice setup, btw, i too like diskless boxes. hopefully it won''t be long until it''s practical to have some more advanced networks for this kind of things (Infiniband, or at least RDMA over 10GE)> > Again, since AoE isn''t IP, you should look for ethernet frames type > > 0x88A2 > > Hmm...I thought ethereal would show us everything by default, even > non-ip packets.maybe, but if you filter by src IP number, you only get IP packets. better filter by type 88A2 or by src MAC address> I am compiling a non-xen kernel now just as a test to see if AoE works > any better without xen.somebody did get Xen running over AoE, so it shouldn''t be a kernel patch incompatibility. maybe the ethx you''re using for AoE gets in the xen bridge? do you do your aoe-discover before or after setting up the Xen network? -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users