Alexander Petry
2007-Jul-31 11:14 UTC
[Xen-users] [Q] How to access a Mellanox InfiniBand HCA from domU directly
Hi list!
I would like to access my InfiniBand HCA from a domU, so I first tried
the smart-io.hg Xen tree but it did not build right away. As a temporary
workaround I tried to let the domU directly access the HCA. The steps I
have taken have been:
1. In dom0:
# lspci -v
03:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx
HCA] (rev a0)
Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]
Flags: fast devsel, IRQ 16
Memory at f0800000 (64-bit, non-prefetchable) [size=1M]
Memory at f0000000 (64-bit, prefetchable) [size=8M]
Capabilities: [40] Power Management version 2
Capabilities: [48] Vital Product Data
Capabilities: [90] Message Signalled Interrupts: Mask- 64bit+
Queue=0/5 Enable-
Capabilities: [84] MSI-X: Enable- Mask- TabSize=32
Capabilities: [60] Express Endpoint IRQ 0
With that information I edited my grub config to exclude this particular
PCI device:
module /boot/vmlinuz-2.6.19-4-server root=/dev/sda1 ro console=tty0
console=ttyS0,9600n8 max_loop=64 pciback.hide=(0000:03:00.0)
2. After a reboot, I edited the domU config so that it includes the line:
pci = [''0000:03:00.0'']
The output of lspci -v executed in the user domain is:
0000:00:00.0 InfiniBand: Mellanox Technology: Unknown device 6274 (rev a0)
Subsystem: Mellanox Technology: Unknown device 6274
Flags: fast devsel, IRQ 16
Memory at f0800000 (64-bit, non-prefetchable) [size=1M]
Memory at f0000000 (64-bit, prefetchable) [size=8M]
Capabilities: [40] Power Management version 2
Capabilities: [48] Vital Product Data
Capabilities: [90] Message Signalled Interrupts: 64bit+
Queue=0/5 Enable-
Capabilities: [84] #11 [001f]
Capabilities: [60] #10 [0001]
The module cannot be loaded and prints the following error:
xen-domU0:~# modprobe ib_mthca
[90558.241638] ib_mthca: Mellanox InfiniBand HCA driver v0.08 (February
14, 2006)
[90558.241651] ib_mthca: Initializing 0000:00:00.0
[90559.245730] ib_mthca 0000:00:00.0: QUERY_FW command failed, aborting.
[90559.245792] ib_mthca: probe of 0000:00:00.0 failed with error -11
The output of ''xm info'' is:
host : xen-o-matic
release : 2.6.19-4-server
version : #2 SMP Thu Apr 5 06:07:34 UTC 2007
machine : i686
nr_cpus : 2
nr_nodes : 1
sockets_per_node : 1
cores_per_socket : 2
threads_per_core : 1
cpu_mhz : 2009
hw_caps :
178bfbff:ebd3fbff:00000000:00000010:00002001:00000000:0000001f
total_memory : 2030
free_memory : 1367
xen_major : 3
xen_minor : 0
xen_extra : .3-0
xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
xen_pagesize : 4096
platform_params : virt_start=0xf5800000
xen_changeset : unavailable
cc_compiler : gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)
cc_compile_by : buildd
cc_compile_domain : buildd
cc_compile_date : Sat Mar 24 16:01:08 UTC 2007
xend_config_format : 2
Does anybody have an idea?
Best regards,
Alexander Petry
--
==========================================| Dipl. Inf. Alexander Petry
|
| Head Competence Center for HPC |
| Institut fuer Techno- und |
| Wirtschaftsmathematik (ITWM) |
| Fraunhofer-Platz 1 |
| |
| D-67663 Kaiserslautern |
==========================================| E-Mail: petry@itwm.fhg.de
|
| Internet: http://www.itwm.fhg.de |
==========================================
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Alexander Petry
2007-Jul-31 14:55 UTC
[Xen-users] [Q] How to access a Mellanox InfiniBand HCA from domU directly
Hi list!
I would like to access my InfiniBand HCA from a domU, so I first tried
the smart-io.hg Xen tree but it did not build right away. As a temporary
workaround I tried to let the domU directly access the HCA. The steps I
have taken have been:
1. In dom0:
# lspci -v
03:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx
HCA] (rev a0)
Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]
Flags: fast devsel, IRQ 16
Memory at f0800000 (64-bit, non-prefetchable) [size=1M]
Memory at f0000000 (64-bit, prefetchable) [size=8M]
Capabilities: [40] Power Management version 2
Capabilities: [48] Vital Product Data
Capabilities: [90] Message Signalled Interrupts: Mask- 64bit+
Queue=0/5 Enable-
Capabilities: [84] MSI-X: Enable- Mask- TabSize=32
Capabilities: [60] Express Endpoint IRQ 0
With that information I edited my grub config to exclude this particular
PCI device:
module /boot/vmlinuz-2.6.19-4-server root=/dev/sda1 ro console=tty0
console=ttyS0,9600n8 max_loop=64 pciback.hide=(0000:03:00.0)
2. After a reboot, I edited the domU config so that it includes the line:
pci = [''0000:03:00.0'']
The output of lspci -v executed in the user domain is:
0000:00:00.0 InfiniBand: Mellanox Technology: Unknown device 6274 (rev a0)
Subsystem: Mellanox Technology: Unknown device 6274
Flags: fast devsel, IRQ 16
Memory at f0800000 (64-bit, non-prefetchable) [size=1M]
Memory at f0000000 (64-bit, prefetchable) [size=8M]
Capabilities: [40] Power Management version 2
Capabilities: [48] Vital Product Data
Capabilities: [90] Message Signalled Interrupts: 64bit+
Queue=0/5 Enable-
Capabilities: [84] #11 [001f]
Capabilities: [60] #10 [0001]
The module cannot be loaded and prints the following error:
xen-domU0:~# modprobe ib_mthca
[90558.241638] ib_mthca: Mellanox InfiniBand HCA driver v0.08 (February
14, 2006)
[90558.241651] ib_mthca: Initializing 0000:00:00.0
[90559.245730] ib_mthca 0000:00:00.0: QUERY_FW command failed, aborting.
[90559.245792] ib_mthca: probe of 0000:00:00.0 failed with error -11
The output of ''xm info'' is:
host : xen-o-matic
release : 2.6.19-4-server
version : #2 SMP Thu Apr 5 06:07:34 UTC 2007
machine : i686
nr_cpus : 2
nr_nodes : 1
sockets_per_node : 1
cores_per_socket : 2
threads_per_core : 1
cpu_mhz : 2009
hw_caps :
178bfbff:ebd3fbff:00000000:00000010:00002001:00000000:0000001f
total_memory : 2030
free_memory : 1367
xen_major : 3
xen_minor : 0
xen_extra : .3-0
xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
xen_pagesize : 4096
platform_params : virt_start=0xf5800000
xen_changeset : unavailable
cc_compiler : gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)
cc_compile_by : buildd
cc_compile_domain : buildd
cc_compile_date : Sat Mar 24 16:01:08 UTC 2007
xend_config_format : 2
Does anybody have an idea?
Best regards,
Alexander Petry
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Matthew Haas
2007-Aug-01 22:03 UTC
Re: [Xen-users] [Q] How to access a Mellanox InfiniBand HCA from domU directly
Howdy,
Wondering if anyone can shed any light on this:
I had a hard drive crash in a machine (P4 2.6GHz, Debian Etch, Xen
3.0.3), and being lazy, I took a new drive, partitioned and formatted,
mounted as a slave into an existing machine, and did a:
"cp -a /source/* /dest/"
Off a rescue disk, to get all the data copied over. I then went in and
changed hostname and IP configuration in the appropriate places,
restored both machines to operating order, and started them up.
Both machines come up and are accessible on the network. However,
before too much time passes, the following messages start to appear on
the console of both machines, and continue over time:
peth0: received packet with own address as source address
Googling for some clue, apparently this message appeared in certain
Xen configurations, and was (presumably?) due to IPv6 configuration
issues.. I don''t even have IPv6 compiled into the kernels on these
machines, so that doesn''t appear to be my issue.
I''ve scrounged through the Xen scripts and believe I see where peth0
is being created and brought online, but I see nothing there that
references an existing file to get ID from, or anything that would seem
to create an interface with duplicate settings as the other machine.
Seems like some sort of static data was duplicated during the copy and
is not auto-generated, so both machines see each other''s traffic on
peth0? Question is.. what/where?
Any clues/suggestions would be greatly appreciated.
Thanks!
-Matthew
--
Matthew Haas
Visiting Instructor
Corning Community College
Computer & Information Science
http://lab46.corning-cc.edu/haas/home/
"Writing should be like breathing;
It is one of those important things we do." -- me
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Matthew Haas
2007-Aug-01 22:07 UTC
[Xen-users] peth0: received packet with own address as source address
*** RESEND: I neglected to change subject line on previous post.
*** Sorry for the confusion/misleading subject. Been a long day.
Howdy,
Wondering if anyone can shed any light on this:
I had a hard drive crash in a machine (P4 2.6GHz, Debian Etch, Xen
3.0.3), and being lazy, I took a new drive, partitioned and formatted,
mounted as a slave into an existing machine, and did a:
"cp -a /source/* /dest/"
Off a rescue disk, to get all the data copied over. I then went in and
changed hostname and IP configuration in the appropriate places,
restored both machines to operating order, and started them up.
Both machines come up and are accessible on the network. However,
before too much time passes, the following messages start to appear on
the console of both machines, and continue over time:
peth0: received packet with own address as source address
Googling for some clue, apparently this message appeared in certain
Xen configurations, and was (presumably?) due to IPv6 configuration
issues.. I don''t even have IPv6 compiled into the kernels on these
machines, so that doesn''t appear to be my issue.
I''ve scrounged through the Xen scripts and believe I see where peth0
is being created and brought online, but I see nothing there that
references an existing file to get ID from, or anything that would seem
to create an interface with duplicate settings as the other machine.
Seems like some sort of static data was duplicated during the copy and
is not auto-generated, so both machines see each other''s traffic on
peth0? Question is.. what/where?
Any clues/suggestions would be greatly appreciated.
Thanks!
-Matthew
--
Matthew Haas
Visiting Instructor
Corning Community College
Computer & Information Science
http://lab46.corning-cc.edu/haas/home/
"Writing should be like breathing;
It is one of those important things we do." -- me
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Dylan Martin
2007-Aug-01 22:24 UTC
Re: [Xen-users] peth0: received packet with own address as source address
Just a dumb idea: are you sure the two boxes have different mac addresses? Slightly less dumb idea: grep through /etc looking for your IP address and MAC address. Maybe the system is consulting some dumb file and getting the wrong address that way. Much less dumb idea: start up tcpdump on both machines and see what actual traffic is causing the error.> *** RESEND: I neglected to change subject line on previous post. > *** Sorry for the confusion/misleading subject. Been a long day. > > Howdy, > > Wondering if anyone can shed any light on this: > > I had a hard drive crash in a machine (P4 2.6GHz, Debian Etch, Xen > 3.0.3), and being lazy, I took a new drive, partitioned and formatted, > mounted as a slave into an existing machine, and did a: > > "cp -a /source/* /dest/" > > Off a rescue disk, to get all the data copied over. I then went in and > changed hostname and IP configuration in the appropriate places, > restored both machines to operating order, and started them up. > > Both machines come up and are accessible on the network. However, > before too much time passes, the following messages start to appear on > the console of both machines, and continue over time: > > peth0: received packet with own address as source address > > Googling for some clue, apparently this message appeared in certain > Xen configurations, and was (presumably?) due to IPv6 configuration > issues.. I don''t even have IPv6 compiled into the kernels on these > machines, so that doesn''t appear to be my issue. > > I''ve scrounged through the Xen scripts and believe I see where peth0 > is being created and brought online, but I see nothing there that > references an existing file to get ID from, or anything that would seem > to create an interface with duplicate settings as the other machine. > > Seems like some sort of static data was duplicated during the copy and > is not auto-generated, so both machines see each other''s traffic on > peth0? Question is.. what/where? > > Any clues/suggestions would be greatly appreciated. > > Thanks! > > -Matthew > -- > Matthew Haas > Visiting Instructor > Corning Community College > Computer & Information Science > http://lab46.corning-cc.edu/haas/home/ > > "Writing should be like breathing; > It is one of those important things we do." -- me > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Matthew Haas
2007-Aug-01 22:42 UTC
Re: [Xen-users] peth0: received packet with own address as source address
Dylan Martin wrote:> Just a dumb idea: are you sure the two boxes have different mac > addresses? >Yes, eth0 on both machines have unique MAC addresses.> Slightly less dumb idea: grep through /etc looking for your IP > address and MAC address. Maybe the system is consulting some dumb > file and getting the wrong address that way. >One of the first things I did (just for IP), and I just did it again looking for MAC address (upper and lowercase)... still came up empty.> Much less dumb idea: start up tcpdump on both machines and see > what actual traffic is causing the error. >I see the following when running tcpdump on peth0 from machine B (the duplicated one-- machine A is the original): 18:33:58.091906 fe:ff:ff:ff:ff:ff (oui Unknown) > Broadcast, ethertype Unknown (0x88a2), length 60: 0x0000: 1000 ffff ff01 0000 0000 0000 0000 0000 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 0x0020: 0000 0000 0000 0000 0000 0000 0000 18:33:58.091907 fe:ff:ff:ff:ff:ff (oui Unknown) > Broadcast, ethertype Unknown (0x88a2), length 60: 0x0000: 1000 ffff ff01 0000 0000 0000 0000 0000 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 0x0020: 0000 0000 0000 0000 0000 0000 0000 18:33:58.091908 MM:AA:CC:HI:NE:BB (oui Unknown) > Broadcast, ethertype Unknown (0x88a2), length 60: 0x0000: 1000 ffff ff01 0000 0000 0000 0000 0000 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 0x0020: 0000 0000 0000 0000 0000 0000 0000 18:33:58.122515 fe:ff:ff:ff:ff:ff (oui Unknown) > Broadcast, ethertype Unknown (0x88a2), length 60: 0x0000: 1000 ffff ff01 0000 0000 0000 0000 0000 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 0x0020: 0000 0000 0000 0000 0000 0000 0000 I occasionally see the other machine''s MAC pop up as well. Running tcpdump on Machine A yields the same looking output. FE:FF:FF:FF:FF:FF is the MAC assigned to the peth0, vifs, xenbr0s on what appears to be all my Xen machines. Not sure what to make of it, aside from it is trying to broadcast an ethernet frame of unknown type. Thanks for the suggestions. -Matthew -- Matthew Haas Visiting Instructor Corning Community College Computer & Information Science http://lab46.corning-cc.edu/haas/home/ "Writing should be like breathing; It is one of those important things we do." -- me _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users