Lustre discuss - Dec 2011 - Problems with lustre router setup IB <-> TCP

Hello, 


I have a big problem finding an example how to setup a lustre-router. 

Maybe someone can direct me to the right direction.



Inside the server room we have a lustre storage system with 1 combined MDS/OSS
node and 3 OSS nodes.

Around 100 compute nodes connect to this lustre storage over IB. So right now
all lnet communication is done

over IB. We have to connect now a small number of workstations outside the
server room to the lustre storage system.

I would prefer to do this with a lustre router, instead of enabling
communication for the whole system over IB and TCP.




##################################################
#                server room                     #   
# +---------+                                    #
# | MDS/OSS |                                    #
# +---------+                                    #
# |   OSS   |\                                   #                  
+-------------+
# +---------+ \                                  #                   |
workstation |
# |   OSS   |  \                                 #                 
/+-------------+
# +---------+   \+-----------+      +--------+   #   +------------+/
# |   OSS   |    | IB-switch |--IB--| router |--TCP--| TCP-switch | 
# +---------+   /+-----------+      +--------+   #   +------------+\
#              /                                 #                 
\+-------------+
# +---------+ /                                  #                   |
workstation |
# | clients |/                                   #                  
+-------------+
# +---------+                                    #
#                                                #
##################################################

Markus Kirsch
Consulting

ict ag - Aachen
information ? communication ? technology
a Fujitsu Company
Kackertstra?e 20
52072 Aachen
www.ictag.net
www.hpcline.de
www.ictag-shop.de

Tel: +49 241 88949 156
Fax: +49 241 88949 110
Mobil: +49 177 88949 22
Mail: mak at ictag.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20111219/90c97746/attachment.html

Hi Markus,

I have a working test setup of this, seems it just depends on having the
correct modprobe-confs for the involved machines.

On the MDS/OSS we have /etc/modprobe.d/lustre.conf with
> options lnet networks=o2ib0(ib0) routes="tcp1
192.168.0.[3-4]@o2ib0"
dead_router_check_interval=300
where 192.168.0.[3-4] are the (IBo)IPs of the two LNET-routers we installed.

On the LNET-router box, we have
> options lnet forwarding="enabled"
> options lnet accept=all
> options lnet networks="o2ib0(ib0),tcp1(eth0)" accept_timeout=15
And on the clients in the ethernet part, we have
> options lnet networks=tcp1(eth0) routes="o2ib
10.1.1.[15,18]@tcp1"
dead_router_check_interval=300
where 10.1.1.[15,18] are the IPs of the LNET routers on the ethernet side.

Cheers,
Thomas

On 12/19/2011 02:46 PM, Markus Kirsch wrote:
> Hello,
> 
>  
> 
> I have a big problem finding an example how to setup a lustre-router.
> 
> Maybe someone can direct me to the right direction.
> 
>  
> 
> Inside the server room we have a lustre storage system with 1 combined
> MDS/OSS node and 3 OSS nodes.
> 
> Around 100 compute nodes connect to this lustre storage over IB. So
> right now all lnet communication is done
> 
> over IB. We have to connect now a small number of workstations outside
> the server room to the lustre storage system.
> 
> I would prefer to do this with a lustre router, instead of enabling
> communication for the whole system over IB and TCP.
> 
>  
> 
>  
> ##################################################
> #                server room                     #  
> # +---------+                                    #
> # | MDS/OSS |                                    #
> # +---------+                                    #
> # |   OSS   |\                                   #                  
> +-------------+
> # +---------+ \                                  #                   |
> workstation |
> # |   OSS   |  \                                 #                 
> /+-------------+
> # +---------+   \+-----------+      +--------+   #   +------------+/
> # |   OSS   |    | IB-switch |--IB--| router |--TCP--| TCP-switch |
> # +---------+   /+-----------+      +--------+   #   +------------+\
> #              /                                 #                 
> \+-------------+
> # +---------+ /                                  #                   |
> workstation |
> # | clients |/                                   #                  
> +-------------+
> # +---------+                                    #
> #                                                #
> ##################################################
>  
> Markus Kirsch
> Consulting
>  
> ict ag - Aachen
> information ? communication ? technology
> a Fujitsu Company
> Kackertstra?e 20
> 52072 Aachen
> www.ictag.net
> www.hpcline.de
> www.ictag-shop.de
>  
> Tel: +49 241 88949 156
> Fax: +49 241 88949 110
> Mobil: +49 177 88949 22
> Mail: mak at ictag.net
>  
> 
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
-- 
--------------------------------------------------------------------
Thomas Roth
Department: Informationstechnologie
Location: SB3 1.262
Phone: +49-6159-71 1453  Fax: +49-6159-71 2986

GSI Helmholtzzentrum f?r Schwerionenforschung GmbH
Planckstra?e 1
64291 Darmstadt
www.gsi.de

Gesellschaft mit beschr?nkter Haftung
Sitz der Gesellschaft: Darmstadt
Handelsregister: Amtsgericht Darmstadt, HRB 1528

Gesch?ftsf?hrung: Professor Dr. Dr. h.c. Horst St?cker,
Dr. Hartmut Eickhoff

Vorsitzende des Aufsichtsrates: Dr. Beatrix Vierkorn-Rudolph
Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt