Lustre discuss - Oct 2010 - OSS1 write performance impacted by OSS3 writes?!

Hi,

I have 4 OSSs with 4 OSTs each still running 1.6.7.2 on RHEL 4.  I
stripped files across OSTs specific to each OSS to test network bandwidth.
 Then wrote to each file from separate clients.  The network
is currently single GigE link to each OSS/client via Cisco 4503.
 Bonding 4 links to each OSS after resolving this
current issue I hope.

Here''s the problem.  When I start writing to OSS3 it cuts the
performance of OSS1 in half and only achieves half of it''s network
bandwidth.  Stop writing to OSS1 and 3 gets full bandwidth or vice
versa.  If I write to OSS1,2 and 4 or OSS2,3,4 all get full bandwidth.
 I''ve checked what OSTs the files were stripped against and
looked at the network config.  Further more collectl shows no cross
traffic (if I had messed up file stripes).  No ideas...

What could cause this?

Thanks,

Dan

-- Sent from my Palm Pre

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20101028/9911e348/attachment.html

Dan wrote:
> Hi,
>
> I have 4 OSSs with 4 OSTs each still running 1.6.7.2 on RHEL 4.  I 
> stripped files across OSTs specific to each OSS to test network 
> bandwidth.  Then wrote to each file from separate clients.  The 
> network is currently single GigE link to each OSS/client via Cisco 
> 4503.  Bonding 4 links to each OSS after resolving this current issue 
> I hope.
>
> Here''s the problem.  When I start writing to OSS3 it cuts the 
> performance of OSS1 in half and only achieves half of it''s network
> bandwidth.  Stop writing to OSS1 and 3 gets full bandwidth or vice 
> versa.  If I write to OSS1,2 and 4 or OSS2,3,4 all get full bandwidth. 
>  I''ve checked what OSTs the files were stripped against and looked
at
> the network config.  Further more collectl shows no cross traffic (if 
> I had messed up file stripes).  No ideas...
>
> What could cause this?
Look at your Ethernet switch.
> Thanks,
>
> Dan
>
> -- Sent from my Palm Pre
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>

Hi James,

I was curious about the switch blocking too.  It''s a 4503, not
the new "-E" version.  I have two 48 port GigE cards
installed.  I spread the OSSs across the two line cards to try and
avoid overloading a block of ports and that helped - originally writing to OSS3
slowed down OSS1 and 2.

Seems that I exceed 6Gb/s with 4 clients and 4 OSSs writing.  I
didn''t realize 4503s had so little bandwidth, wow.  Sounds
like that''s that.

Is the 6Gb/s per card or total? Cisco''s documentation indicates
it''s per slot (card?) but I''m not getting 12 Gb/s total.


Thanks,

Dan

-- Sent from my Palm Pre

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20101028/a3b39901/attachment.html

> Is the 6Gb/s per card or total? Cisco''s documentation indicates
it''s per
> slot (card?) but I''m not getting 12 Gb/s total.
  It''s 6gbit per line card.  Typically the 48 port line cards group
ports
into 6 groups of 8 ports.  Each group of 8 ports shares a 1gbit channel
to the backplane.

 Try putting two OSSes on one line card and two on another and seperate the
OSSes such that one is in first 8 ports and one is in the last 8.  Since
it''s full duplex there shouldn''t be too much problem putting
clients in the
same group as the OSSes unless you do reads and writes in parallel at
which point they can interfere with each other.

  Ultimately that''s not a good switch for performance purposes due to
the
backplane limitations.
> Thanks,
>
> Dan