Different interfaces need to be declared with different LNET networks - something like "networks=tcp0(eth0),tcp1(eth1)". Of course, that assumes your clients are configured to use a mix of tcp0 and tcp1 for connections (with each client only using one of the two). This is really only useful in corner cases, when you''re doing something strange; if eth0 and eth1 are in the same subnet (as in your example), this is almost certainly not productive. A better bet might be to use a single LNET, and bond the two interfaces together - either as an active/passive pair, or active/active (e.g., LACP). Then you''d declare networks=tcp0(bond0), give the bond a single IP address, and client traffic would be split across the two members in the bond more like you probably expect (given the limits of the bond protocol you''re using). -- Mike Shuey On Tue, Jun 25, 2013 at 8:06 AM, Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>wrote:> hello friends, > > I need to comunicate my OSS by two ethernet TCP interfaces: eth0 and eth1. > > I have configured this feature in my modprobe.d with: > > “options lnet networks=tcp0(eth0,eth1)” > > And I can see two interfaces with: > > lctl --net tcp interface_list > sa-d4-01.ceta-ciemat.es: (192.168.11.15/255.255.255.0) npeer 0 nroute 2 > sa-d4-01.ceta-ciemat.es: (192.168.11.35/255.255.255.0) npeer 0 nroute 0 > > But, the clients only can communicate with the first interface: > > lctl ping 192.168.11.15 > 12345-0@lo > 12345-192.168.11.15@tcp > lctl ping 192.168.11.35 > failed to ping 192.168.11.35@tcp: Input/output error > > > Any suggestions how to “enable” the second interface? > > > thank in advance > > *Alfonso Pardo Diaz * > *System Administrator / Researcher * > *c/ Sola nº 1; 10200 TRUJILLO, SPAIN * > *Tel: +34 927 65 93 17 Fax: +34 927 32 32 37 * > > [image: CETA-Ciemat logo] <http://www.ceta-ciemat.es/> > > ---------------------------- Confidencialidad: Este mensaje y sus ficheros > adjuntos se dirige exclusivamente a su destinatario y puede contener > información privilegiada o confidencial. Si no es vd. el destinatario > indicado, queda notificado de que la utilización, divulgación y/o copia sin > autorización está prohibida en virtud de la legislación vigente. Si ha > recibido este mensaje por error, le rogamos que nos lo comunique > inmediatamente respondiendo al mensaje y proceda a su destrucción. > Disclaimer: This message and its attached files is intended exclusively for > its recipients and may contain confidential information. If you received > this e-mail in error you are hereby notified that any dissemination, copy > or disclosure of this communication is strictly prohibited and may be > unlawful. In this case, please notify us by a reply and delete this email > and its contents immediately. ---------------------------- > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >_______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
From: Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org<mailto:alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>> Date: Tuesday, June 25, 2013 5:22 AM To: Michael Shuey <shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org<mailto:shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org>> Cc: WC-Discuss <WC-Discuss.Migration-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org<mailto:WC-Discuss.Migration@intel.com>>, "lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:lustre-discuss@lists.lustre.org>" <lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:lustre-discuss-aLEFhgZF4x4@public.gmane.orgstre.org>> Subject: Re: [Lustre-discuss] Lustre over two TCP interfaces thank Michael, This is my second step, I will change the lnet with “options lnet networks=tcp0(bond0,bond1)” because my machines has 4 nics. I have a bond0 and bond1 with LACP. I need to comunicate the clients with two network for HA network. If the bond0 network is down, the clients can reach the OSS by the second network bond1. If I change the modprobe with “options lnet networks=tcp0(bond0),tcp1(bond1)”, how the clients mount the filesystem to reach the OSS by two network? The issue here is not Lustre, but TCP/IP routing. If you have two IP address on a single node, and another machine can see both addresses, typically one interface is picked as ''closer'' and traffic flows through one interface only. This is outside of Lustre, and will happen with any application. The client mount address is still the address exported by the MGS. That does not change with bonding, except you use the bond IP instead of the individual IPs. When you configure multiple address and.or networks for a server node (MGS,MDS, OSS) the clients are given the list at mount time, and will independently try each address in the list until they connect. Again, the start with the ''closest'' address by hop rules. Cliffw From: Michael Shuey<mailto:shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> Sent: Tuesday, June 25, 2013 2:14 PM To: Alfonso Pardo<mailto:alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org> Cc: lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> ; WC-Discuss<mailto:WC-Discuss.Migration-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> Subject: Re: [Lustre-discuss] Lustre over two TCP interfaces Different interfaces need to be declared with different LNET networks - something like "networks=tcp0(eth0),tcp1(eth1)". Of course, that assumes your clients are configured to use a mix of tcp0 and tcp1 for connections (with each client only using one of the two). This is really only useful in corner cases, when you''re doing something strange; if eth0 and eth1 are in the same subnet (as in your example), this is almost certainly not productive. A better bet might be to use a single LNET, and bond the two interfaces together - either as an active/passive pair, or active/active (e.g., LACP). Then you''d declare networks=tcp0(bond0), give the bond a single IP address, and client traffic would be split across the two members in the bond more like you probably expect (given the limits of the bond protocol you''re using). -- Mike Shuey On Tue, Jun 25, 2013 at 8:06 AM, Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org<mailto:alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>> wrote: hello friends, I need to comunicate my OSS by two ethernet TCP interfaces: eth0 and eth1. I have configured this feature in my modprobe.d with: “options lnet networks=tcp0(eth0,eth1)” And I can see two interfaces with: lctl --net tcp interface_list sa-d4-01.ceta-ciemat.es<http://sa-d4-01.ceta-ciemat.es>: (192.168.11.15/255.255.255.0<http://192.168.11.15/255.255.255.0>) npeer 0 nroute 2 sa-d4-01.ceta-ciemat.es<http://sa-d4-01.ceta-ciemat.es>: (192.168.11.35/255.255.255.0<http://192.168.11.35/255.255.255.0>) npeer 0 nroute 0 But, the clients only can communicate with the first interface: lctl ping 192.168.11.15 12345-0@lo 12345-192.168.11.15@tcp lctl ping 192.168.11.35 failed to ping 192.168.11.35@tcp: Input/output error Any suggestions how to “enable” the second interface? thank in advance Alfonso Pardo Diaz System Administrator / Researcher c/ Sola nº 1; 10200 TRUJILLO, SPAIN Tel: +34 927 65 93 17<tel:%2B34%20927%2065%2093%2017> Fax: +34 927 32 32 37 [CETA-Ciemat logo]<http://www.ceta-ciemat.es/> ---------------------------- Confidencialidad: Este mensaje y sus ficheros adjuntos se dirige exclusivamente a su destinatario y puede contener información privilegiada o confidencial. Si no es vd. el destinatario indicado, queda notificado de que la utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente respondiendo al mensaje y proceda a su destrucción. Disclaimer: This message and its attached files is intended exclusively for its recipients and may contain confidential information. If you received this e-mail in error you are hereby notified that any dissemination, copy or disclosure of this communication is strictly prohibited and may be unlawful. In this case, please notify us by a reply and delete this email and its contents immediately. ---------------------------- _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org<mailto:Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> http://lists.lustre.org/mailman/listinfo/lustre-discuss
Unless something has changed in the new versions of lustre, I don't think lustre can do failover between nids on the same machine. It can choose the available nid at mount time, but if an active nid goes away after you are mounted then the client chooses the failover nid, and this must be on a different server. Check the archives for more discussion in this topic :) -----Original Message----- From: Alfonso Pardo [alfonso.pardo@ciemat.es<mailto:alfonso.pardo@ciemat.es>] Sent: Tuesday, June 25, 2013 07:23 AM Central Standard Time To: Michael Shuey Cc: WC-Discuss; lustre-discuss@lists.lustre.org Subject: Re: [Lustre-discuss] Lustre over two TCP interfaces thank Michael, This is my second step, I will change the lnet with “options lnet networks=tcp0(bond0,bond1)” because my machines has 4 nics. I have a bond0 and bond1 with LACP. I need to comunicate the clients with two network for HA network. If the bond0 network is down, the clients can reach the OSS by the second network bond1. If I change the modprobe with “options lnet networks=tcp0(bond0),tcp1(bond1)”, how the clients mount the filesystem to reach the OSS by two network? From: Michael Shuey<mailto:shuey@purdue.edu> Sent: Tuesday, June 25, 2013 2:14 PM To: Alfonso Pardo<mailto:alfonso.pardo@ciemat.es> Cc: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org> ; WC-Discuss<mailto:WC-Discuss.Migration@intel.com> Subject: Re: [Lustre-discuss] Lustre over two TCP interfaces Different interfaces need to be declared with different LNET networks - something like "networks=tcp0(eth0),tcp1(eth1)". Of course, that assumes your clients are configured to use a mix of tcp0 and tcp1 for connections (with each client only using one of the two). This is really only useful in corner cases, when you're doing something strange; if eth0 and eth1 are in the same subnet (as in your example), this is almost certainly not productive. A better bet might be to use a single LNET, and bond the two interfaces together - either as an active/passive pair, or active/active (e.g., LACP). Then you'd declare networks=tcp0(bond0), give the bond a single IP address, and client traffic would be split across the two members in the bond more like you probably expect (given the limits of the bond protocol you're using). -- Mike Shuey On Tue, Jun 25, 2013 at 8:06 AM, Alfonso Pardo <alfonso.pardo@ciemat.es<mailto:alfonso.pardo@ciemat.es>> wrote: hello friends, I need to comunicate my OSS by two ethernet TCP interfaces: eth0 and eth1. I have configured this feature in my modprobe.d with: “options lnet networks=tcp0(eth0,eth1)” And I can see two interfaces with: lctl --net tcp interface_list sa-d4-01.ceta-ciemat.es<http://sa-d4-01.ceta-ciemat.es>: (192.168.11.15/255.255.255.0<http://192.168.11.15/255.255.255.0>) npeer 0 nroute 2 sa-d4-01.ceta-ciemat.es<http://sa-d4-01.ceta-ciemat.es>: (192.168.11.35/255.255.255.0<http://192.168.11.35/255.255.255.0>) npeer 0 nroute 0 But, the clients only can communicate with the first interface: lctl ping 192.168.11.15 12345-0@lo 12345-192.168.11.15@tcp lctl ping 192.168.11.35 failed to ping 192.168.11.35@tcp: Input/output error Any suggestions how to “enable” the second interface? thank in advance Alfonso Pardo Diaz System Administrator / Researcher c/ Sola nº 1; 10200 TRUJILLO, SPAIN Tel: +34 927 65 93 17<tel:%2B34%20927%2065%2093%2017> Fax: +34 927 32 32 37 [CETA-Ciemat logo]<http://www.ceta-ciemat.es/> ---------------------------- Confidencialidad: Este mensaje y sus ficheros adjuntos se dirige exclusivamente a su destinatario y puede contener información privilegiada o confidencial. Si no es vd. el destinatario indicado, queda notificado de que la utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente respondiendo al mensaje y proceda a su destrucción. Disclaimer: This message and its attached files is intended exclusively for its recipients and may contain confidential information. If you received this e-mail in error you are hereby notified that any dissemination, copy or disclosure of this communication is strictly prohibited and may be unlawful. In this case, please notify us by a reply and delete this email and its contents immediately. ---------------------------- _______________________________________________ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org<mailto:Lustre-discuss@lists.lustre.org> http://lists.lustre.org/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
On 06/26/2013 04:16 PM, Alfonso Pardo wrote:> But.... if I configure the OST assigning to the first interface of the > OSS (bond0) and as failover OSS the second inteface of the OSS. If the > bond0 network down, the client will try to connect to the failover, that > is the second interface of the OSS. > is it possible?I stand to be corrected, but no, I don't think so. As I understand it the failover code looks for a different server instance, rather than a different nid. See http://lists.opensfs.org/pipermail/lustre-devel-opensfs.org/2012-August/000028.html> *From:* Brian O'Connor <mailto:briano@sgi.com> > *Sent:* Wednesday, June 26, 2013 1:09 AM > *To:* 'Alfonso Pardo' <mailto:alfonso.pardo@ciemat.es> ; 'Michael Shuey' > <mailto:shuey@purdue.edu> > *Cc:* 'WC-Discuss' <mailto:WC-Discuss.Migration@intel.com> ; > mailto:lustre-discuss@lists.lustre.org > *Subject:* RE: [Lustre-discuss] Lustre over two TCP interfaces > Unless something has changed in the new versions of lustre, I don't > think lustre can do failover between nids on the same machine. > > It can choose the available nid at mount time, but if an active nid goes > away after you are mounted then the client chooses the failover nid, and > this must be on a different server. > > Check the archives for more discussion in this topic :) > > > > -----Original Message----- > *From: *Alfonso Pardo [alfonso.pardo@ciemat.es > <mailto:alfonso.pardo@ciemat.es>] > *Sent: *Tuesday, June 25, 2013 07:23 AM Central Standard Time > *To: *Michael Shuey > *Cc: *WC-Discuss; lustre-discuss@lists.lustre.org > *Subject: *Re: [Lustre-discuss] Lustre over two TCP interfaces > > thank Michael, > This is my second step, I will change the lnet with “options lnet > networks=tcp0(bond0,bond1)” because my machines has 4 nics. I have a > bond0 and bond1 with LACP. I need to comunicate the clients with two > network for HA network. > If the bond0 network is down, the clients can reach the OSS by the > second network bond1. > If I change the modprobe with “options lnet > networks=tcp0(bond0),tcp1(bond1)”, how the clients mount the filesystem > to reach the OSS by two network? > *From:* Michael Shuey <mailto:shuey@purdue.edu> > *Sent:* Tuesday, June 25, 2013 2:14 PM > *To:* Alfonso Pardo <mailto:alfonso.pardo@ciemat.es> > *Cc:* lustre-discuss@lists.lustre.org > <mailto:lustre-discuss@lists.lustre.org> ; WC-Discuss > <mailto:WC-Discuss.Migration@intel.com> > *Subject:* Re: [Lustre-discuss] Lustre over two TCP interfaces > Different interfaces need to be declared with different LNET networks - > something like "networks=tcp0(eth0),tcp1(eth1)". Of course, that > assumes your clients are configured to use a mix of tcp0 and tcp1 for > connections (with each client only using one of the two). This is > really only useful in corner cases, when you're doing something strange; > if eth0 and eth1 are in the same subnet (as in your example), this is > almost certainly not productive. > A better bet might be to use a single LNET, and bond the two interfaces > together - either as an active/passive pair, or active/active (e.g., > LACP). Then you'd declare networks=tcp0(bond0), give the bond a single > IP address, and client traffic would be split across the two members in > the bond more like you probably expect (given the limits of the bond > protocol you're using). > -- > Mike Shuey > > > On Tue, Jun 25, 2013 at 8:06 AM, Alfonso Pardo <alfonso.pardo@ciemat.es > <mailto:alfonso.pardo@ciemat.es>> wrote: > > hello friends, > I need to comunicate my OSS by two ethernet TCP interfaces: eth0 and > eth1. > I have configured this feature in my modprobe.d with: > “options lnet networks=tcp0(eth0,eth1)” > And I can see two interfaces with: > lctl --net tcp interface_list > sa-d4-01.ceta-ciemat.es <http://sa-d4-01.ceta-ciemat.es>: > (192.168.11.15/255.255.255.0 <http://192.168.11.15/255.255.255.0>) > npeer 0 nroute 2 > sa-d4-01.ceta-ciemat.es <http://sa-d4-01.ceta-ciemat.es>: > (192.168.11.35/255.255.255.0 <http://192.168.11.35/255.255.255.0>) > npeer 0 nroute 0 > But, the clients only can communicate with the first interface: > lctl ping 192.168.11.15 > 12345-0@lo > 12345-192.168.11.15@tcp > lctl ping 192.168.11.35 > failed to ping 192.168.11.35@tcp: Input/output error > Any suggestions how to “enable” the second interface? > thank in advance > > /Alfonso Pardo Diaz/ > /*System Administrator / Researcher*/ > /c/ Sola nº 1; 10200 TRUJILLO, SPAIN/ > /Tel: +34 927 65 93 17 <tel:%2B34%20927%2065%2093%2017> Fax: +34 927 > 32 32 37/ > > CETA-Ciemat logo <http://www.ceta-ciemat.es/> > > ---------------------------- Confidencialidad: Este mensaje y sus > ficheros adjuntos se dirige exclusivamente a su destinatario y puede > contener información privilegiada o confidencial. Si no es vd. el > destinatario indicado, queda notificado de que la utilización, > divulgación y/o copia sin autorización está prohibida en virtud de > la legislación vigente. Si ha recibido este mensaje por error, le > rogamos que nos lo comunique inmediatamente respondiendo al mensaje > y proceda a su destrucción. Disclaimer: This message and its > attached files is intended exclusively for its recipients and may > contain confidential information. If you received this e-mail in > error you are hereby notified that any dissemination, copy or > disclosure of this communication is strictly prohibited and may be > unlawful. In this case, please notify us by a reply and delete this > email and its contents immediately. ---------------------------- > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org <mailto:Lustre-discuss@lists.lustre.org> > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- Brian O'Connor ------------------------------------------------------------- SGI Consulting Email: briano@sgi.com, Mobile +61 417 746 452 Phone: +61 3 9963 1900, Fax: +61 3 9963 1902 691 Burke Road, Camberwell, Victoria, 3124 AUSTRALIA http://www.sgi.com/support/services ------------------------------------------------------------- _______________________________________________ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
oooh! Thanks for you reply! May be another way is a floating IP between two interfaces with IPVS (corosync). -----Mensaje original----- From: Brian O'Connor Sent: Wednesday, June 26, 2013 10:15 AM To: Alfonso Pardo Cc: 'Michael Shuey' ; 'WC-Discuss' ; lustre-discuss@lists.lustre.org Subject: Re: [Lustre-discuss] Lustre over two TCP interfaces On 06/26/2013 04:16 PM, Alfonso Pardo wrote:> But.... if I configure the OST assigning to the first interface of the > OSS (bond0) and as failover OSS the second inteface of the OSS. If the > bond0 network down, the client will try to connect to the failover, that > is the second interface of the OSS. > is it possible?I stand to be corrected, but no, I don't think so. As I understand it the failover code looks for a different server instance, rather than a different nid. See http://lists.opensfs.org/pipermail/lustre-devel-opensfs.org/2012-August/000028.html> *From:* Brian O'Connor <mailto:briano@sgi.com> > *Sent:* Wednesday, June 26, 2013 1:09 AM > *To:* 'Alfonso Pardo' <mailto:alfonso.pardo@ciemat.es> ; 'Michael Shuey' > <mailto:shuey@purdue.edu> > *Cc:* 'WC-Discuss' <mailto:WC-Discuss.Migration@intel.com> ; > mailto:lustre-discuss@lists.lustre.org > *Subject:* RE: [Lustre-discuss] Lustre over two TCP interfaces > Unless something has changed in the new versions of lustre, I don't > think lustre can do failover between nids on the same machine. > > It can choose the available nid at mount time, but if an active nid goes > away after you are mounted then the client chooses the failover nid, and > this must be on a different server. > > Check the archives for more discussion in this topic :) > > > > -----Original Message----- > *From: *Alfonso Pardo [alfonso.pardo@ciemat.es > <mailto:alfonso.pardo@ciemat.es>] > *Sent: *Tuesday, June 25, 2013 07:23 AM Central Standard Time > *To: *Michael Shuey > *Cc: *WC-Discuss; lustre-discuss@lists.lustre.org > *Subject: *Re: [Lustre-discuss] Lustre over two TCP interfaces > > thank Michael, > This is my second step, I will change the lnet with “options lnet > networks=tcp0(bond0,bond1)” because my machines has 4 nics. I have a > bond0 and bond1 with LACP. I need to comunicate the clients with two > network for HA network. > If the bond0 network is down, the clients can reach the OSS by the > second network bond1. > If I change the modprobe with “options lnet > networks=tcp0(bond0),tcp1(bond1)”, how the clients mount the filesystem > to reach the OSS by two network? > *From:* Michael Shuey <mailto:shuey@purdue.edu> > *Sent:* Tuesday, June 25, 2013 2:14 PM > *To:* Alfonso Pardo <mailto:alfonso.pardo@ciemat.es> > *Cc:* lustre-discuss@lists.lustre.org > <mailto:lustre-discuss@lists.lustre.org> ; WC-Discuss > <mailto:WC-Discuss.Migration@intel.com> > *Subject:* Re: [Lustre-discuss] Lustre over two TCP interfaces > Different interfaces need to be declared with different LNET networks - > something like "networks=tcp0(eth0),tcp1(eth1)". Of course, that > assumes your clients are configured to use a mix of tcp0 and tcp1 for > connections (with each client only using one of the two). This is > really only useful in corner cases, when you're doing something strange; > if eth0 and eth1 are in the same subnet (as in your example), this is > almost certainly not productive. > A better bet might be to use a single LNET, and bond the two interfaces > together - either as an active/passive pair, or active/active (e.g., > LACP). Then you'd declare networks=tcp0(bond0), give the bond a single > IP address, and client traffic would be split across the two members in > the bond more like you probably expect (given the limits of the bond > protocol you're using). > -- > Mike Shuey > > > On Tue, Jun 25, 2013 at 8:06 AM, Alfonso Pardo <alfonso.pardo@ciemat.es > <mailto:alfonso.pardo@ciemat.es>> wrote: > > hello friends, > I need to comunicate my OSS by two ethernet TCP interfaces: eth0 and > eth1. > I have configured this feature in my modprobe.d with: > “options lnet networks=tcp0(eth0,eth1)” > And I can see two interfaces with: > lctl --net tcp interface_list > sa-d4-01.ceta-ciemat.es <http://sa-d4-01.ceta-ciemat.es>: > (192.168.11.15/255.255.255.0 <http://192.168.11.15/255.255.255.0>) > npeer 0 nroute 2 > sa-d4-01.ceta-ciemat.es <http://sa-d4-01.ceta-ciemat.es>: > (192.168.11.35/255.255.255.0 <http://192.168.11.35/255.255.255.0>) > npeer 0 nroute 0 > But, the clients only can communicate with the first interface: > lctl ping 192.168.11.15 > 12345-0@lo > 12345-192.168.11.15@tcp > lctl ping 192.168.11.35 > failed to ping 192.168.11.35@tcp: Input/output error > Any suggestions how to “enable” the second interface? > thank in advance > > /Alfonso Pardo Diaz/ > /*System Administrator / Researcher*/ > /c/ Sola nº 1; 10200 TRUJILLO, SPAIN/ > /Tel: +34 927 65 93 17 <tel:%2B34%20927%2065%2093%2017> Fax: +34 927 > 32 32 37/ > > CETA-Ciemat logo <http://www.ceta-ciemat.es/> > > ---------------------------- Confidencialidad: Este mensaje y sus > ficheros adjuntos se dirige exclusivamente a su destinatario y puede > contener información privilegiada o confidencial. Si no es vd. el > destinatario indicado, queda notificado de que la utilización, > divulgación y/o copia sin autorización está prohibida en virtud de > la legislación vigente. Si ha recibido este mensaje por error, le > rogamos que nos lo comunique inmediatamente respondiendo al mensaje > y proceda a su destrucción. Disclaimer: This message and its > attached files is intended exclusively for its recipients and may > contain confidential information. If you received this e-mail in > error you are hereby notified that any dissemination, copy or > disclosure of this communication is strictly prohibited and may be > unlawful. In this case, please notify us by a reply and delete this > email and its contents immediately. ---------------------------- > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@lists.lustre.org > <mailto:Lustre-discuss@lists.lustre.org> > http://lists.lustre.org/mailman/listinfo/lustre-discuss >-- Brian O'Connor ------------------------------------------------------------- SGI Consulting Email: briano@sgi.com, Mobile +61 417 746 452 Phone: +61 3 9963 1900, Fax: +61 3 9963 1902 691 Burke Road, Camberwell, Victoria, 3124 AUSTRALIA http://www.sgi.com/support/services ------------------------------------------------------------- _______________________________________________ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
That will probably be slow - the machine you use to proxy the IPVS address would be a bottleneck. Out of curiosity, what problem are you trying to solve here? Do you anticipate whole-subnet outages to be an issue (and if so, why)? -- Mike Shuey On Wed, Jun 26, 2013 at 4:53 AM, Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>wrote:> oooh! > > > Thanks for you reply! May be another way is a floating IP between two > interfaces with IPVS (corosync). > > -----Mensaje original----- From: Brian O''Connor > Sent: Wednesday, June 26, 2013 10:15 AM > To: Alfonso Pardo > Cc: ''Michael Shuey'' ; ''WC-Discuss'' ; lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org<lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> > Subject: Re: [Lustre-discuss] Lustre over two TCP interfaces > > > > > > On 06/26/2013 04:16 PM, Alfonso Pardo wrote: > >> But.... if I configure the OST assigning to the first interface of the >> OSS (bond0) and as failover OSS the second inteface of the OSS. If the >> bond0 network down, the client will try to connect to the failover, that >> is the second interface of the OSS. >> is it possible? >> > > > I stand to be corrected, but no, I don''t think so. As I understand it > the failover code looks for a different server instance, rather than a > different nid. > > See > > http://lists.opensfs.org/**pipermail/lustre-devel-** > opensfs.org/2012-August/**000028.html<http://lists.opensfs.org/pipermail/lustre-devel-opensfs.org/2012-August/000028.html> > > > *From:* Brian O''Connor <mailto:briano-sJ/iWh9BUns@public.gmane.org> >> *Sent:* Wednesday, June 26, 2013 1:09 AM >> *To:* ''Alfonso Pardo'' <mailto:alfonso.pardo@ciemat.**es<alfonso.pardo@ciemat.es>> >> ; ''Michael Shuey'' >> <mailto:shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> >> *Cc:* ''WC-Discuss'' <mailto:WC-Discuss.Migration@**intel.com<WC-Discuss.Migration-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>> >> ; >> mailto:lustre-discuss@lists.**lustre.org<lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> >> *Subject:* RE: [Lustre-discuss] Lustre over two TCP interfaces >> Unless something has changed in the new versions of lustre, I don''t >> think lustre can do failover between nids on the same machine. >> >> It can choose the available nid at mount time, but if an active nid goes >> away after you are mounted then the client chooses the failover nid, and >> this must be on a different server. >> >> Check the archives for more discussion in this topic :) >> >> >> >> -----Original Message----- >> *From: *Alfonso Pardo [alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org >> <mailto:alfonso.pardo@ciemat.**es <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>>] >> *Sent: *Tuesday, June 25, 2013 07:23 AM Central Standard Time >> *To: *Michael Shuey >> *Cc: *WC-Discuss; lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org<lustre-discuss@lists.lustre.org> >> *Subject: *Re: [Lustre-discuss] Lustre over two TCP interfaces >> >> thank Michael, >> This is my second step, I will change the lnet with “options lnet >> networks=tcp0(bond0,bond1)” because my machines has 4 nics. I have a >> bond0 and bond1 with LACP. I need to comunicate the clients with two >> network for HA network. >> If the bond0 network is down, the clients can reach the OSS by the >> second network bond1. >> If I change the modprobe with “options lnet >> networks=tcp0(bond0),tcp1(**bond1)”, how the clients mount the filesystem >> to reach the OSS by two network? >> *From:* Michael Shuey <mailto:shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> >> *Sent:* Tuesday, June 25, 2013 2:14 PM >> *To:* Alfonso Pardo <mailto:alfonso.pardo@ciemat.**es<alfonso.pardo@ciemat.es> >> > >> *Cc:* lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org <lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> >> <mailto:lustre-discuss@lists.**lustre.org<lustre-discuss-aLEFhgZF4x4kHE2CFeF4+A@public.gmane.orgg>> >> ; WC-Discuss >> <mailto:WC-Discuss.Migration@**intel.com <WC-Discuss.Migration-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> >> > >> *Subject:* Re: [Lustre-discuss] Lustre over two TCP interfaces >> Different interfaces need to be declared with different LNET networks - >> something like "networks=tcp0(eth0),tcp1(**eth1)". Of course, that >> assumes your clients are configured to use a mix of tcp0 and tcp1 for >> connections (with each client only using one of the two). This is >> really only useful in corner cases, when you''re doing something strange; >> if eth0 and eth1 are in the same subnet (as in your example), this is >> almost certainly not productive. >> A better bet might be to use a single LNET, and bond the two interfaces >> together - either as an active/passive pair, or active/active (e.g., >> LACP). Then you''d declare networks=tcp0(bond0), give the bond a single >> IP address, and client traffic would be split across the two members in >> the bond more like you probably expect (given the limits of the bond >> protocol you''re using). >> -- >> Mike Shuey >> >> >> On Tue, Jun 25, 2013 at 8:06 AM, Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org >> <mailto:alfonso.pardo@ciemat.**es <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>>> wrote: >> >> hello friends, >> I need to comunicate my OSS by two ethernet TCP interfaces: eth0 and >> eth1. >> I have configured this feature in my modprobe.d with: >> “options lnet networks=tcp0(eth0,eth1)” >> And I can see two interfaces with: >> lctl --net tcp interface_list >> sa-d4-01.ceta-ciemat.es <http://sa-d4-01.ceta-ciemat.**es<http://sa-d4-01.ceta-ciemat.es> >> >: >> (192.168.11.15/255.255.255.0 <http://192.168.11.15/255.255.**255.0<http://192.168.11.15/255.255.255.0> >> >) >> npeer 0 nroute 2 >> sa-d4-01.ceta-ciemat.es <http://sa-d4-01.ceta-ciemat.**es<http://sa-d4-01.ceta-ciemat.es> >> >: >> (192.168.11.35/255.255.255.0 <http://192.168.11.35/255.255.**255.0<http://192.168.11.35/255.255.255.0> >> >) >> npeer 0 nroute 0 >> But, the clients only can communicate with the first interface: >> lctl ping 192.168.11.15 >> 12345-0@lo >> 12345-192.168.11.15@tcp >> lctl ping 192.168.11.35 >> failed to ping 192.168.11.35@tcp: Input/output error >> Any suggestions how to “enable” the second interface? >> thank in advance >> >> /Alfonso Pardo Diaz/ >> /*System Administrator / Researcher*/ >> /c/ Sola nº 1; 10200 TRUJILLO, SPAIN/ >> /Tel: +34 927 65 93 17 <tel:%2B34%20927%2065%2093%**2017> Fax: +34 >> 927 >> 32 32 37/ >> >> CETA-Ciemat logo <http://www.ceta-ciemat.es/> >> >> ---------------------------- Confidencialidad: Este mensaje y sus >> ficheros adjuntos se dirige exclusivamente a su destinatario y puede >> contener información privilegiada o confidencial. Si no es vd. el >> destinatario indicado, queda notificado de que la utilización, >> divulgación y/o copia sin autorización está prohibida en virtud de >> la legislación vigente. Si ha recibido este mensaje por error, le >> rogamos que nos lo comunique inmediatamente respondiendo al mensaje >> y proceda a su destrucción. Disclaimer: This message and its >> attached files is intended exclusively for its recipients and may >> contain confidential information. If you received this e-mail in >> error you are hereby notified that any dissemination, copy or >> disclosure of this communication is strictly prohibited and may be >> unlawful. In this case, please notify us by a reply and delete this >> email and its contents immediately. ---------------------------- >> >> ______________________________**_________________ >> Lustre-discuss mailing list >> Lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org <Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org><mailto: >> Lustre-discuss@lists.**lustre.org <Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org>> >> http://lists.lustre.org/**mailman/listinfo/lustre-**discuss<http://lists.lustre.org/mailman/listinfo/lustre-discuss> >> >> > > -- > Brian O''Connor > ------------------------------**------------------------------**- > SGI Consulting > Email: briano-sJ/iWh9BUns@public.gmane.org, Mobile +61 417 746 452 > Phone: +61 3 9963 1900, Fax: +61 3 9963 1902 > 691 Burke Road, Camberwell, Victoria, 3124 > AUSTRALIA > http://www.sgi.com/support/**services<http://www.sgi.com/support/services> > ------------------------------**------------------------------**- >_______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Hi Alfonso, I guess, you have two switches, with 2 interfaces (bond0) connected to one switch and the other 2 interfaces (bond1) to the second switch. --- What you need to do is merge the switches using a ''stacking'' cable (if the switches are stackable) and create a single trunk using 2 ports from each switch. Then create a single bond on the Linux side using all the 4 Interfaces (and have just 1 IP). Use bonding mode balance-rr or 0 without LACP to get load balancing across all the 4 NICs. If the switches aren''t stackable and a single trunk cannot be created on the switch side, then use bonding mode balance-alb or 6 on the Linux side. No changes need to be done to the cabling in either case. --- This way you get Load Balancing and H/A across NICs. Indivar Nair On Wed, Jun 26, 2013 at 5:47 PM, Michael Shuey <shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> wrote:> That will probably be slow - the machine you use to proxy the IPVS address > would be a bottleneck. Out of curiosity, what problem are you trying to > solve here? Do you anticipate whole-subnet outages to be an issue (and if > so, why)? > > -- > Mike Shuey > > > On Wed, Jun 26, 2013 at 4:53 AM, Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>wrote: > >> oooh! >> >> >> Thanks for you reply! May be another way is a floating IP between two >> interfaces with IPVS (corosync). >> >> -----Mensaje original----- From: Brian O''Connor >> Sent: Wednesday, June 26, 2013 10:15 AM >> To: Alfonso Pardo >> Cc: ''Michael Shuey'' ; ''WC-Discuss'' ; lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org<lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> >> Subject: Re: [Lustre-discuss] Lustre over two TCP interfaces >> >> >> >> >> >> On 06/26/2013 04:16 PM, Alfonso Pardo wrote: >> >>> But.... if I configure the OST assigning to the first interface of the >>> OSS (bond0) and as failover OSS the second inteface of the OSS. If the >>> bond0 network down, the client will try to connect to the failover, that >>> is the second interface of the OSS. >>> is it possible? >>> >> >> >> I stand to be corrected, but no, I don''t think so. As I understand it >> the failover code looks for a different server instance, rather than a >> different nid. >> >> See >> >> http://lists.opensfs.org/**pipermail/lustre-devel-** >> opensfs.org/2012-August/**000028.html<http://lists.opensfs.org/pipermail/lustre-devel-opensfs.org/2012-August/000028.html> >> >> >> *From:* Brian O''Connor <mailto:briano-sJ/iWh9BUns@public.gmane.org> >>> *Sent:* Wednesday, June 26, 2013 1:09 AM >>> *To:* ''Alfonso Pardo'' <mailto:alfonso.pardo@ciemat.**es<alfonso.pardo@ciemat.es>> >>> ; ''Michael Shuey'' >>> <mailto:shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> >>> *Cc:* ''WC-Discuss'' <mailto:WC-Discuss.Migration@**intel.com<WC-Discuss.Migration-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>> >>> ; >>> mailto:lustre-discuss@lists.**lustre.org<lustre-discuss-aLEFhgZF4x4kHE2CFeF4+A@public.gmane.orgg> >>> *Subject:* RE: [Lustre-discuss] Lustre over two TCP interfaces >>> Unless something has changed in the new versions of lustre, I don''t >>> think lustre can do failover between nids on the same machine. >>> >>> It can choose the available nid at mount time, but if an active nid goes >>> away after you are mounted then the client chooses the failover nid, and >>> this must be on a different server. >>> >>> Check the archives for more discussion in this topic :) >>> >>> >>> >>> -----Original Message----- >>> *From: *Alfonso Pardo [alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org >>> <mailto:alfonso.pardo@ciemat.**es <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>>] >>> *Sent: *Tuesday, June 25, 2013 07:23 AM Central Standard Time >>> *To: *Michael Shuey >>> *Cc: *WC-Discuss; lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org<lustre-discuss@lists.lustre.org> >>> *Subject: *Re: [Lustre-discuss] Lustre over two TCP interfaces >>> >>> thank Michael, >>> This is my second step, I will change the lnet with “options lnet >>> networks=tcp0(bond0,bond1)” because my machines has 4 nics. I have a >>> bond0 and bond1 with LACP. I need to comunicate the clients with two >>> network for HA network. >>> If the bond0 network is down, the clients can reach the OSS by the >>> second network bond1. >>> If I change the modprobe with “options lnet >>> networks=tcp0(bond0),tcp1(**bond1)”, how the clients mount the >>> filesystem >>> to reach the OSS by two network? >>> *From:* Michael Shuey <mailto:shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> >>> *Sent:* Tuesday, June 25, 2013 2:14 PM >>> *To:* Alfonso Pardo <mailto:alfonso.pardo@ciemat.**es<alfonso.pardo@ciemat.es> >>> > >>> *Cc:* lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org<lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> >>> <mailto:lustre-discuss@lists.**lustre.org<lustre-discuss-aLEFhgZF4x42wxj9z9LmtA@public.gmane.orgrg>> >>> ; WC-Discuss >>> <mailto:WC-Discuss.Migration@**intel.com<WC-Discuss.Migration-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> >>> > >>> *Subject:* Re: [Lustre-discuss] Lustre over two TCP interfaces >>> Different interfaces need to be declared with different LNET networks - >>> something like "networks=tcp0(eth0),tcp1(**eth1)". Of course, that >>> assumes your clients are configured to use a mix of tcp0 and tcp1 for >>> connections (with each client only using one of the two). This is >>> really only useful in corner cases, when you''re doing something strange; >>> if eth0 and eth1 are in the same subnet (as in your example), this is >>> almost certainly not productive. >>> A better bet might be to use a single LNET, and bond the two interfaces >>> together - either as an active/passive pair, or active/active (e.g., >>> LACP). Then you''d declare networks=tcp0(bond0), give the bond a single >>> IP address, and client traffic would be split across the two members in >>> the bond more like you probably expect (given the limits of the bond >>> protocol you''re using). >>> -- >>> Mike Shuey >>> >>> >>> On Tue, Jun 25, 2013 at 8:06 AM, Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org >>> <mailto:alfonso.pardo@ciemat.**es <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>>> wrote: >>> >>> hello friends, >>> I need to comunicate my OSS by two ethernet TCP interfaces: eth0 and >>> eth1. >>> I have configured this feature in my modprobe.d with: >>> “options lnet networks=tcp0(eth0,eth1)” >>> And I can see two interfaces with: >>> lctl --net tcp interface_list >>> sa-d4-01.ceta-ciemat.es <http://sa-d4-01.ceta-ciemat.**es<http://sa-d4-01.ceta-ciemat.es> >>> >: >>> (192.168.11.15/255.255.255.0 <http://192.168.11.15/255.255.**255.0<http://192.168.11.15/255.255.255.0> >>> >) >>> npeer 0 nroute 2 >>> sa-d4-01.ceta-ciemat.es <http://sa-d4-01.ceta-ciemat.**es<http://sa-d4-01.ceta-ciemat.es> >>> >: >>> (192.168.11.35/255.255.255.0 <http://192.168.11.35/255.255.**255.0<http://192.168.11.35/255.255.255.0> >>> >) >>> npeer 0 nroute 0 >>> But, the clients only can communicate with the first interface: >>> lctl ping 192.168.11.15 >>> 12345-0@lo >>> 12345-192.168.11.15@tcp >>> lctl ping 192.168.11.35 >>> failed to ping 192.168.11.35@tcp: Input/output error >>> Any suggestions how to “enable” the second interface? >>> thank in advance >>> >>> /Alfonso Pardo Diaz/ >>> /*System Administrator / Researcher*/ >>> /c/ Sola nº 1; 10200 TRUJILLO, SPAIN/ >>> /Tel: +34 927 65 93 17 <tel:%2B34%20927%2065%2093%**2017> Fax: +34 >>> 927 >>> 32 32 37/ >>> >>> CETA-Ciemat logo <http://www.ceta-ciemat.es/> >>> >>> ---------------------------- Confidencialidad: Este mensaje y sus >>> ficheros adjuntos se dirige exclusivamente a su destinatario y puede >>> contener información privilegiada o confidencial. Si no es vd. el >>> destinatario indicado, queda notificado de que la utilización, >>> divulgación y/o copia sin autorización está prohibida en virtud de >>> la legislación vigente. Si ha recibido este mensaje por error, le >>> rogamos que nos lo comunique inmediatamente respondiendo al mensaje >>> y proceda a su destrucción. Disclaimer: This message and its >>> attached files is intended exclusively for its recipients and may >>> contain confidential information. If you received this e-mail in >>> error you are hereby notified that any dissemination, copy or >>> disclosure of this communication is strictly prohibited and may be >>> unlawful. In this case, please notify us by a reply and delete this >>> email and its contents immediately. ---------------------------- >>> >>> ______________________________**_________________ >>> Lustre-discuss mailing list >>> Lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org <Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org><mailto: >>> Lustre-discuss@lists.**lustre.org <Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org>> >>> http://lists.lustre.org/**mailman/listinfo/lustre-**discuss<http://lists.lustre.org/mailman/listinfo/lustre-discuss> >>> >>> >> >> -- >> Brian O''Connor >> ------------------------------**------------------------------**- >> SGI Consulting >> Email: briano-sJ/iWh9BUns@public.gmane.org, Mobile +61 417 746 452 >> Phone: +61 3 9963 1900, Fax: +61 3 9963 1902 >> 691 Burke Road, Camberwell, Victoria, 3124 >> AUSTRALIA >> http://www.sgi.com/support/**services<http://www.sgi.com/support/services> >> ------------------------------**------------------------------**- >> > > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss > >_______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Then follow the instructions in my earlier mail. No need to have bond0 and bond1. You will achieve high-availability even with one bonded interface. Cheers, Indivar Nair On Thu, Jun 27, 2013 at 11:55 AM, Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>wrote:> Yes I have two swtiches, one to the bond0 interface and other switch to > the second bond1 interface. > > > > *From:* Indivar Nair <indivar.nair-OZBC9qKnwSOacSAJ64KLvA@public.gmane.org> > *Sent:* Wednesday, June 26, 2013 8:05 PM > *To:* Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org> > *Cc:* Michael Shuey <shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> ; WC-Discuss<WC-Discuss.Migration@intel.com>; > lustre-discuss <lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> > *Subject:* Re: [Lustre-discuss] Lustre over two TCP interfaces > > Hi Alfonso, > > I guess, you have two switches, with 2 interfaces (bond0) connected to one > switch and the other 2 interfaces (bond1) to the second switch. > > --- > > What you need to do is merge the switches using a ''stacking'' cable (if the > switches are stackable) and create a single trunk using 2 ports from each > switch. > Then create a single bond on the Linux side using all the 4 Interfaces > (and have just 1 IP). > > Use bonding mode balance-rr or 0 without LACP to get load balancing across > all the 4 NICs. > > If the switches aren''t stackable and a single trunk cannot be created on > the switch side, then use bonding mode balance-alb or 6 on the Linux side. > > No changes need to be done to the cabling in either case. > > --- > > This way you get Load Balancing and H/A across NICs. > > > > Indivar Nair > > > > > On Wed, Jun 26, 2013 at 5:47 PM, Michael Shuey <shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> wrote: > >> That will probably be slow - the machine you use to proxy the IPVS >> address would be a bottleneck. Out of curiosity, what problem are you >> trying to solve here? Do you anticipate whole-subnet outages to be an >> issue (and if so, why)? >> >> -- >> Mike Shuey >> >> >> On Wed, Jun 26, 2013 at 4:53 AM, Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>wrote: >> >>> oooh! >>> >>> >>> Thanks for you reply! May be another way is a floating IP between two >>> interfaces with IPVS (corosync). >>> >>> -----Mensaje original----- From: Brian O''Connor >>> Sent: Wednesday, June 26, 2013 10:15 AM >>> To: Alfonso Pardo >>> Cc: ''Michael Shuey'' ; ''WC-Discuss'' ; lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org<lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> >>> Subject: Re: [Lustre-discuss] Lustre over two TCP interfaces >>> >>> >>> >>> >>> >>> On 06/26/2013 04:16 PM, Alfonso Pardo wrote: >>> >>>> But.... if I configure the OST assigning to the first interface of the >>>> OSS (bond0) and as failover OSS the second inteface of the OSS. If the >>>> bond0 network down, the client will try to connect to the failover, that >>>> is the second interface of the OSS. >>>> is it possible? >>>> >>> >>> >>> I stand to be corrected, but no, I don''t think so. As I understand it >>> the failover code looks for a different server instance, rather than a >>> different nid. >>> >>> See >>> >>> http://lists.opensfs.org/**pipermail/lustre-devel-** >>> opensfs.org/2012-August/**000028.html<http://lists.opensfs.org/pipermail/lustre-devel-opensfs.org/2012-August/000028.html> >>> >>> >>> *From:* Brian O''Connor <mailto:briano-sJ/iWh9BUns@public.gmane.org> >>>> *Sent:* Wednesday, June 26, 2013 1:09 AM >>>> *To:* ''Alfonso Pardo'' <mailto:alfonso.pardo@ciemat.**es<alfonso.pardo@ciemat.es>> >>>> ; ''Michael Shuey'' >>>> <mailto:shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> >>>> *Cc:* ''WC-Discuss'' <mailto:WC-Discuss.Migration@**intel.com<WC-Discuss.Migration-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>> >>>> ; >>>> mailto:lustre-discuss@lists.**lustre.org<lustre-discuss-aLEFhgZF4x42wxj9z9LmtA@public.gmane.orgrg> >>>> *Subject:* RE: [Lustre-discuss] Lustre over two TCP interfaces >>>> Unless something has changed in the new versions of lustre, I don''t >>>> think lustre can do failover between nids on the same machine. >>>> >>>> It can choose the available nid at mount time, but if an active nid goes >>>> away after you are mounted then the client chooses the failover nid, and >>>> this must be on a different server. >>>> >>>> Check the archives for more discussion in this topic :) >>>> >>>> >>>> >>>> -----Original Message----- >>>> *From: *Alfonso Pardo [alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org >>>> <mailto:alfonso.pardo@ciemat.**es <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>>] >>>> *Sent: *Tuesday, June 25, 2013 07:23 AM Central Standard Time >>>> *To: *Michael Shuey >>>> *Cc: *WC-Discuss; lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org<lustre-discuss@lists.lustre.org> >>>> *Subject: *Re: [Lustre-discuss] Lustre over two TCP interfaces >>>> >>>> thank Michael, >>>> This is my second step, I will change the lnet with “options lnet >>>> networks=tcp0(bond0,bond1)” because my machines has 4 nics. I have a >>>> bond0 and bond1 with LACP. I need to comunicate the clients with two >>>> network for HA network. >>>> If the bond0 network is down, the clients can reach the OSS by the >>>> second network bond1. >>>> If I change the modprobe with “options lnet >>>> networks=tcp0(bond0),tcp1(**bond1)”, how the clients mount the >>>> filesystem >>>> to reach the OSS by two network? >>>> *From:* Michael Shuey <mailto:shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> >>>> *Sent:* Tuesday, June 25, 2013 2:14 PM >>>> *To:* Alfonso Pardo <mailto:alfonso.pardo@ciemat.**es<alfonso.pardo@ciemat.es> >>>> > >>>> *Cc:* lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org<lustre-discuss-aLEFhgZF4x4kHE2CFeF4+A@public.gmane.orgg> >>>> <mailto:lustre-discuss@lists.**lustre.org<lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.orgorg>> >>>> ; WC-Discuss >>>> <mailto:WC-Discuss.Migration@**intel.com<WC-Discuss.Migration-ral2JQCrhuE@public.gmane.orgm> >>>> > >>>> *Subject:* Re: [Lustre-discuss] Lustre over two TCP interfaces >>>> Different interfaces need to be declared with different LNET networks - >>>> something like "networks=tcp0(eth0),tcp1(**eth1)". Of course, that >>>> assumes your clients are configured to use a mix of tcp0 and tcp1 for >>>> connections (with each client only using one of the two). This is >>>> really only useful in corner cases, when you''re doing something strange; >>>> if eth0 and eth1 are in the same subnet (as in your example), this is >>>> almost certainly not productive. >>>> A better bet might be to use a single LNET, and bond the two interfaces >>>> together - either as an active/passive pair, or active/active (e.g., >>>> LACP). Then you''d declare networks=tcp0(bond0), give the bond a single >>>> IP address, and client traffic would be split across the two members in >>>> the bond more like you probably expect (given the limits of the bond >>>> protocol you''re using). >>>> -- >>>> Mike Shuey >>>> >>>> >>>> On Tue, Jun 25, 2013 at 8:06 AM, Alfonso Pardo <alfonso.pardo@ciemat.es >>>> <mailto:alfonso.pardo@ciemat.**es <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>>> wrote: >>>> >>>> hello friends, >>>> I need to comunicate my OSS by two ethernet TCP interfaces: eth0 and >>>> eth1. >>>> I have configured this feature in my modprobe.d with: >>>> “options lnet networks=tcp0(eth0,eth1)” >>>> And I can see two interfaces with: >>>> lctl --net tcp interface_list >>>> sa-d4-01.ceta-ciemat.es <http://sa-d4-01.ceta-ciemat.**es<http://sa-d4-01.ceta-ciemat.es> >>>> >: >>>> (192.168.11.15/255.255.255.0 <http://192.168.11.15/255.255.**255.0<http://192.168.11.15/255.255.255.0> >>>> >) >>>> npeer 0 nroute 2 >>>> sa-d4-01.ceta-ciemat.es <http://sa-d4-01.ceta-ciemat.**es<http://sa-d4-01.ceta-ciemat.es> >>>> >: >>>> (192.168.11.35/255.255.255.0 <http://192.168.11.35/255.255.**255.0<http://192.168.11.35/255.255.255.0> >>>> >) >>>> npeer 0 nroute 0 >>>> But, the clients only can communicate with the first interface: >>>> lctl ping 192.168.11.15 >>>> 12345-0@lo >>>> 12345-192.168.11.15@tcp >>>> lctl ping 192.168.11.35 >>>> failed to ping 192.168.11.35@tcp: Input/output error >>>> Any suggestions how to “enable” the second interface? >>>> thank in advance >>>> >>>> /Alfonso Pardo Diaz/ >>>> /*System Administrator / Researcher*/ >>>> /c/ Sola nº 1; 10200 TRUJILLO, SPAIN/ >>>> /Tel: +34 927 65 93 17 <tel:%2B34%20927%2065%2093%**2017> Fax: +34 >>>> 927 >>>> 32 32 37/ >>>> >>>> CETA-Ciemat logo <http://www.ceta-ciemat.es/> >>>> >>>> ---------------------------- Confidencialidad: Este mensaje y sus >>>> ficheros adjuntos se dirige exclusivamente a su destinatario y puede >>>> contener información privilegiada o confidencial. Si no es vd. el >>>> destinatario indicado, queda notificado de que la utilización, >>>> divulgación y/o copia sin autorización está prohibida en virtud de >>>> la legislación vigente. Si ha recibido este mensaje por error, le >>>> rogamos que nos lo comunique inmediatamente respondiendo al mensaje >>>> y proceda a su destrucción. Disclaimer: This message and its >>>> attached files is intended exclusively for its recipients and may >>>> contain confidential information. If you received this e-mail in >>>> error you are hereby notified that any dissemination, copy or >>>> disclosure of this communication is strictly prohibited and may be >>>> unlawful. In this case, please notify us by a reply and delete this >>>> email and its contents immediately. ---------------------------- >>>> >>>> ______________________________**_________________ >>>> Lustre-discuss mailing list >>>> Lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org <Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org><mailto: >>>> Lustre-discuss@lists.**lustre.org <Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org>> >>>> http://lists.lustre.org/**mailman/listinfo/lustre-**discuss<http://lists.lustre.org/mailman/listinfo/lustre-discuss> >>>> >>>> >>> >>> -- >>> Brian O''Connor >>> ------------------------------**------------------------------**- >>> SGI Consulting >>> Email: briano-sJ/iWh9BUns@public.gmane.org, Mobile +61 417 746 452 <%2B61%20417%20746%20452> >>> Phone: +61 3 9963 1900, Fax: +61 3 9963 1902 <%2B61%203%209963%201902> >>> 691 Burke Road, Camberwell, Victoria, 3124 >>> AUSTRALIA >>> http://www.sgi.com/support/**services<http://www.sgi.com/support/services> >>> ------------------------------**------------------------------**- >>> >> >> >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org >> http://lists.lustre.org/mailman/listinfo/lustre-discuss >> >> >_______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
I am not able to picture your network architecture. Would it possible for you to post a network diagram? LACP is generally used between switches so that there is no looping issues when you have two switches cascaded using multiple cables. In such a case, only 1 cable is active and the second one takes over only when the first CABLE FAILS and not the switch it self. Hope you haven''t connected *just these 2 switches* using LACP. If so, you will have to change it as it won''t give you desired effect. LACP is used for UPLINK cables and not for peer-to-peer connectivity. With just 2 Switches, the right thing to do is ''stacking''. And one shouldn''t use LACP on a Server-Switch connectivity. Normal non-LACP bonding is the correct way to do it. Will be to tell you more if you could post a diagram. Even a rough one would do. Indivar Nair On Thu, Jun 27, 2013 at 12:28 PM, Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>wrote:> I would have high-availability if I have a bonding mode 0,1 or 2. But > I have LACP bonding attached to the same switch, if I loose a switch, my > OSS will be down. > > *From:* Indivar Nair <indivar.nair-OZBC9qKnwSOacSAJ64KLvA@public.gmane.org> > *Sent:* Thursday, June 27, 2013 8:30 AM > *To:* Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org> > *Cc:* Michael Shuey <shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> ; lustre-discuss<lustre-discuss@lists.lustre.org> > *Subject:* Re: [Lustre-discuss] Lustre over two TCP interfaces > > Then follow the instructions in my earlier mail. > No need to have bond0 and bond1. > You will achieve high-availability even with one bonded interface. > > Cheers, > > > Indivar Nair > > > On Thu, Jun 27, 2013 at 11:55 AM, Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>wrote: > >> Yes I have two swtiches, one to the bond0 interface and other switch >> to the second bond1 interface. >> >> >> >> *From:* Indivar Nair <indivar.nair-OZBC9qKnwSOacSAJ64KLvA@public.gmane.org> >> *Sent:* Wednesday, June 26, 2013 8:05 PM >> *To:* Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org> >> *Cc:* Michael Shuey <shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> ; WC-Discuss<WC-Discuss.Migration@intel.com>; >> lustre-discuss <lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> >> *Subject:* Re: [Lustre-discuss] Lustre over two TCP interfaces >> >> Hi Alfonso, >> >> I guess, you have two switches, with 2 interfaces (bond0) connected to >> one switch and the other 2 interfaces (bond1) to the second switch. >> >> --- >> >> What you need to do is merge the switches using a ''stacking'' cable (if >> the switches are stackable) and create a single trunk using 2 ports from >> each switch. >> Then create a single bond on the Linux side using all the 4 Interfaces >> (and have just 1 IP). >> >> Use bonding mode balance-rr or 0 without LACP to get load balancing >> across all the 4 NICs. >> >> If the switches aren''t stackable and a single trunk cannot be created on >> the switch side, then use bonding mode balance-alb or 6 on the Linux side. >> >> No changes need to be done to the cabling in either case. >> >> --- >> >> This way you get Load Balancing and H/A across NICs. >> >> >> >> Indivar Nair >> >> >> >> >> On Wed, Jun 26, 2013 at 5:47 PM, Michael Shuey <shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> wrote: >> >>> That will probably be slow - the machine you use to proxy the IPVS >>> address would be a bottleneck. Out of curiosity, what problem are you >>> trying to solve here? Do you anticipate whole-subnet outages to be an >>> issue (and if so, why)? >>> >>> -- >>> Mike Shuey >>> >>> >>> On Wed, Jun 26, 2013 at 4:53 AM, Alfonso Pardo <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>wrote: >>> >>>> oooh! >>>> >>>> >>>> Thanks for you reply! May be another way is a floating IP between two >>>> interfaces with IPVS (corosync). >>>> >>>> -----Mensaje original----- From: Brian O''Connor >>>> Sent: Wednesday, June 26, 2013 10:15 AM >>>> To: Alfonso Pardo >>>> Cc: ''Michael Shuey'' ; ''WC-Discuss'' ; lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org<lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org> >>>> Subject: Re: [Lustre-discuss] Lustre over two TCP interfaces >>>> >>>> >>>> >>>> >>>> >>>> On 06/26/2013 04:16 PM, Alfonso Pardo wrote: >>>> >>>>> But.... if I configure the OST assigning to the first interface of the >>>>> OSS (bond0) and as failover OSS the second inteface of the OSS. If the >>>>> bond0 network down, the client will try to connect to the failover, >>>>> that >>>>> is the second interface of the OSS. >>>>> is it possible? >>>>> >>>> >>>> >>>> I stand to be corrected, but no, I don''t think so. As I understand it >>>> the failover code looks for a different server instance, rather than a >>>> different nid. >>>> >>>> See >>>> >>>> http://lists.opensfs.org/**pipermail/lustre-devel-** >>>> opensfs.org/2012-August/**000028.html<http://lists.opensfs.org/pipermail/lustre-devel-opensfs.org/2012-August/000028.html> >>>> >>>> >>>> *From:* Brian O''Connor <mailto:briano-sJ/iWh9BUns@public.gmane.org> >>>>> *Sent:* Wednesday, June 26, 2013 1:09 AM >>>>> *To:* ''Alfonso Pardo'' <mailto:alfonso.pardo@ciemat.**es<alfonso.pardo@ciemat.es>> >>>>> ; ''Michael Shuey'' >>>>> <mailto:shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> >>>>> *Cc:* ''WC-Discuss'' <mailto:WC-Discuss.Migration@**intel.com<WC-Discuss.Migration-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>> >>>>> ; >>>>> mailto:lustre-discuss@lists.**lustre.org<lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.orgorg> >>>>> *Subject:* RE: [Lustre-discuss] Lustre over two TCP interfaces >>>>> Unless something has changed in the new versions of lustre, I don''t >>>>> think lustre can do failover between nids on the same machine. >>>>> >>>>> It can choose the available nid at mount time, but if an active nid >>>>> goes >>>>> away after you are mounted then the client chooses the failover nid, >>>>> and >>>>> this must be on a different server. >>>>> >>>>> Check the archives for more discussion in this topic :) >>>>> >>>>> >>>>> >>>>> -----Original Message----- >>>>> *From: *Alfonso Pardo [alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org >>>>> <mailto:alfonso.pardo@ciemat.**es <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>>] >>>>> *Sent: *Tuesday, June 25, 2013 07:23 AM Central Standard Time >>>>> *To: *Michael Shuey >>>>> *Cc: *WC-Discuss; lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org<lustre-discuss@lists.lustre.org> >>>>> *Subject: *Re: [Lustre-discuss] Lustre over two TCP interfaces >>>>> >>>>> thank Michael, >>>>> This is my second step, I will change the lnet with “options lnet >>>>> networks=tcp0(bond0,bond1)” because my machines has 4 nics. I have a >>>>> bond0 and bond1 with LACP. I need to comunicate the clients with two >>>>> network for HA network. >>>>> If the bond0 network is down, the clients can reach the OSS by the >>>>> second network bond1. >>>>> If I change the modprobe with “options lnet >>>>> networks=tcp0(bond0),tcp1(**bond1)”, how the clients mount the >>>>> filesystem >>>>> to reach the OSS by two network? >>>>> *From:* Michael Shuey <mailto:shuey-olO2ZdjDehc3uPMLIKxrzw@public.gmane.org> >>>>> *Sent:* Tuesday, June 25, 2013 2:14 PM >>>>> *To:* Alfonso Pardo <mailto:alfonso.pardo@ciemat.**es<alfonso.pardo@ciemat.es> >>>>> > >>>>> *Cc:* lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org<lustre-discuss-aLEFhgZF4x42wxj9z9LmtA@public.gmane.orgrg> >>>>> <mailto:lustre-discuss@lists.**lustre.org<lustre-discuss-aLEFhgZF4x53lozW1E3wSQ@public.gmane.org.org>> >>>>> ; WC-Discuss >>>>> <mailto:WC-Discuss.Migration@**intel.com<WC-Discuss.Migration@intel.com> >>>>> > >>>>> *Subject:* Re: [Lustre-discuss] Lustre over two TCP interfaces >>>>> Different interfaces need to be declared with different LNET networks - >>>>> something like "networks=tcp0(eth0),tcp1(**eth1)". Of course, that >>>>> assumes your clients are configured to use a mix of tcp0 and tcp1 for >>>>> connections (with each client only using one of the two). This is >>>>> really only useful in corner cases, when you''re doing something >>>>> strange; >>>>> if eth0 and eth1 are in the same subnet (as in your example), this is >>>>> almost certainly not productive. >>>>> A better bet might be to use a single LNET, and bond the two interfaces >>>>> together - either as an active/passive pair, or active/active (e.g., >>>>> LACP). Then you''d declare networks=tcp0(bond0), give the bond a single >>>>> IP address, and client traffic would be split across the two members in >>>>> the bond more like you probably expect (given the limits of the bond >>>>> protocol you''re using). >>>>> -- >>>>> Mike Shuey >>>>> >>>>> >>>>> On Tue, Jun 25, 2013 at 8:06 AM, Alfonso Pardo < >>>>> alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org >>>>> <mailto:alfonso.pardo@ciemat.**es <alfonso.pardo-8goy4ODAZcv1P9xLtpHBDw@public.gmane.org>>> wrote: >>>>> >>>>> hello friends, >>>>> I need to comunicate my OSS by two ethernet TCP interfaces: eth0 >>>>> and >>>>> eth1. >>>>> I have configured this feature in my modprobe.d with: >>>>> “options lnet networks=tcp0(eth0,eth1)” >>>>> And I can see two interfaces with: >>>>> lctl --net tcp interface_list >>>>> sa-d4-01.ceta-ciemat.es <http://sa-d4-01.ceta-ciemat.**es<http://sa-d4-01.ceta-ciemat.es> >>>>> >: >>>>> (192.168.11.15/255.255.255.0 <http://192.168.11.15/255.255.**255.0<http://192.168.11.15/255.255.255.0> >>>>> >) >>>>> npeer 0 nroute 2 >>>>> sa-d4-01.ceta-ciemat.es <http://sa-d4-01.ceta-ciemat.**es<http://sa-d4-01.ceta-ciemat.es> >>>>> >: >>>>> (192.168.11.35/255.255.255.0 <http://192.168.11.35/255.255.**255.0<http://192.168.11.35/255.255.255.0> >>>>> >) >>>>> npeer 0 nroute 0 >>>>> But, the clients only can communicate with the first interface: >>>>> lctl ping 192.168.11.15 >>>>> 12345-0@lo >>>>> 12345-192.168.11.15@tcp >>>>> lctl ping 192.168.11.35 >>>>> failed to ping 192.168.11.35@tcp: Input/output error >>>>> Any suggestions how to “enable” the second interface? >>>>> thank in advance >>>>> >>>>> /Alfonso Pardo Diaz/ >>>>> /*System Administrator / Researcher*/ >>>>> /c/ Sola nº 1; 10200 TRUJILLO, SPAIN/ >>>>> /Tel: +34 927 65 93 17 <tel:%2B34%20927%2065%2093%**2017> Fax: >>>>> +34 927 >>>>> 32 32 37/ >>>>> >>>>> CETA-Ciemat logo <http://www.ceta-ciemat.es/> >>>>> >>>>> ---------------------------- Confidencialidad: Este mensaje y sus >>>>> ficheros adjuntos se dirige exclusivamente a su destinatario y >>>>> puede >>>>> contener información privilegiada o confidencial. Si no es vd. el >>>>> destinatario indicado, queda notificado de que la utilización, >>>>> divulgación y/o copia sin autorización está prohibida en virtud de >>>>> la legislación vigente. Si ha recibido este mensaje por error, le >>>>> rogamos que nos lo comunique inmediatamente respondiendo al mensaje >>>>> y proceda a su destrucción. Disclaimer: This message and its >>>>> attached files is intended exclusively for its recipients and may >>>>> contain confidential information. If you received this e-mail in >>>>> error you are hereby notified that any dissemination, copy or >>>>> disclosure of this communication is strictly prohibited and may be >>>>> unlawful. In this case, please notify us by a reply and delete this >>>>> email and its contents immediately. ---------------------------- >>>>> >>>>> ______________________________**_________________ >>>>> Lustre-discuss mailing list >>>>> Lustre-discuss-aLEFhgZF4x704iTE6pzA/Q@public.gmane.org**org<Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org><mailto: >>>>> Lustre-discuss@lists.**lustre.org <Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org>> >>>>> http://lists.lustre.org/**mailman/listinfo/lustre-**discuss<http://lists.lustre.org/mailman/listinfo/lustre-discuss> >>>>> >>>>> >>>> >>>> -- >>>> Brian O''Connor >>>> ------------------------------**------------------------------**- >>>> SGI Consulting >>>> Email: briano-sJ/iWh9BUns@public.gmane.org, Mobile +61 417 746 452 <%2B61%20417%20746%20452> >>>> Phone: +61 3 9963 1900, Fax: +61 3 9963 1902 <%2B61%203%209963%201902> >>>> 691 Burke Road, Camberwell, Victoria, 3124 >>>> AUSTRALIA >>>> http://www.sgi.com/support/**services<http://www.sgi.com/support/services> >>>> ------------------------------**------------------------------**- >>>> >>> >>> >>> _______________________________________________ >>> Lustre-discuss mailing list >>> Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org >>> http://lists.lustre.org/mailman/listinfo/lustre-discuss >>> >>> >> > >_______________________________________________ Lustre-discuss mailing list Lustre-discuss-aLEFhgZF4x6X6Mz3xDxJMA@public.gmane.org http://lists.lustre.org/mailman/listinfo/lustre-discuss