LARTC - Apr 2003 - http://lartc.org/howto/lartc.cookbook.mtu-mss.html

current content below is annotated by some suggestions of things to
add along with questions for those who know more than I do [in brackets]

===============
  # iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS
  --clamp-mss-to-pmtu

  This calculates the proper MSS for your link.

[If I understand the code correctly ... - expert intervention invited]
More precisely, this sets the MSS option to the MTU of the outgoing
link for this packet (so it must have already been routed) minus the
minimal size of a TCP and IP header, which is typically but not
necessarily the "right" value.
[In particular, if tcp options such as timestamp might be used then
you really want to lower MSS a bit more to make room for these,
right?  So to be on the safe side, if you''re lowering MSS due to
a link with MTU<1500 then you should probably use something like
MTU - 60, or perhaps to be really safe, leave room for maximal IP 
and TCP headers with MTU - 120 ]

  If you are feeling
  brave, or think that you know best, you can also do something
  like this:

  # iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS
  --set-mss 128

  This sets the MSS of passing SYN packets to 128. Use this if you have
  VoIP with tiny packets, and huge http packets which are
  causing chopping in your voice calls.

More likely you want something like this:
iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN
 -m tcpmss --mss 1441:0xffff
 -j TCPMSS --set-mss 1440

This example is for the case where you know that somewhere upstream is
a link with MTU 1480.  Or, perhaps more realistically, you''re using
some
sort of tunnel that uses up 20 bytes per packet.

In general we don''t want to raise the MSS that someone has specified,
since he might have a good reason for lowering it (as in the VoIP
example above).  We just want to lower it, in this case to 1440
[or, following arguments above, somewhat lower than that], if
otherwise it would have been greater.  The -m tcpmss line checks
whether the current MSS is 1441 or more, and only in that case does
the -j TCPMSS line lower it to 1440.
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Further research on this:

 > [If I understand the code correctly ... - expert intervention invited]
 > More precisely, this sets the MSS option to the MTU of the outgoing
 > link for this packet (so it must have already been routed) minus the
 > minimal size of a TCP and IP header, which is typically but not
 > necessarily the "right" value.
 > [In particular, if tcp options such as timestamp might be used then
 > you really want to lower MSS a bit more to make room for these,
 > right?  So to be on the safe side, if you''re lowering MSS due to
 > a link with MTU<1500 then you should probably use something like
 > MTU - 60, or perhaps to be really safe, leave room for maximal IP 
 > and TCP headers with MTU - 120 ]

rfc879
   To resolve the ambiguity in the TCP Maximum Segment Size option
   definition the following rule is established:

      THE TCP MAXIMUM SEGMENT SIZE IS THE IP MAXIMUM DATAGRAM SIZE MINUS
      FORTY.

It goes on to say that when tcp or ip options are used then the 
MSS has to be effectively reduced.  And in fact, this is what I
see going on in limited testing so far.

http://www.rfc-editor.org/cgi-bin/rfcsearch.pl lists the status of
rfc879 as UNKNOWN, but rfc1191 is listed as status DRAFT STANDARD and
it refers to above.

So, new wording for above should be
 More precisely, this sets the MSS option to the MTU of the outgoing
 link for this packet (so it must have already been routed) minus 40
 which is supposed to tell the sender to limit packet size (including
 TCP and IP headers, including any options they contain) to MSS+40.
 For more details see RFC 879 and 1191.
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/