Tmail talk - Mar 2008 - MIME boundaries changing underfoot? (or: gosh, I hate RFC 3156)

Hi, everyone.  I''m struggling to cryptographically sign an email-with- 
attachments.

I made a little visual aid:  http://web.mac.com/videophile/signed.png

RFC 3156 (PGP/MIME) says that when you PGP-sign a message, you sign  
everything, attachments too.  In my visual aid, the blue shaded area  
(yes, that whole area) is the "signed" portion.  Any alteration of  
that area renders the signature invalid.

It''s possible to prepare and assemble the "blue section" as a
string,
and then generate the PGP signature and tack it on with its own MIME  
headers.  But when I do "mail=TMail::Mail.parse(string_version)",  
TMail alters all my MIME boundaries to its own internally-generated  
ones.  This poses something of a problem, as this invalidates the PGP  
signature; the bytes have been altered.

The challenge boils down to something very, very simple: is there a  
way to get TMail to just ''trust me'' and accept my fully-formed
body-
data and headers without altering them at parse-time?  That way I  
could just hand-assemble an exact string form of the message, then  
TMail-ify it and send.

I''m grateful for any thoughts...I sure can''t find any
reference of
prior art on this one.

Regards,
Chris P.
--
Chris Powell                        "Exit, pursued by a bear."
chris at grubbybaby.com                 - Shakespeare, "The
Winter''s Tale"




-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/tmail-talk/attachments/20080301/05a68492/attachment.html

On Sun, Mar 2, 2008 at 12:32 PM, Chris Powell <chris at grubbybaby.com>
wrote:
> The challenge boils down to something very, very simple: is there a way to
> get TMail to just ''trust me'' and accept my fully-formed
body-data and
> headers without altering them at parse-time?  That way I could just
> hand-assemble an exact string form of the message, then TMail-ify it and
> send.
TMail definitely clobbers every mimepart boundary, not quite sure if
this is a feature or error ;)

We could make it only create a new boundary if the existing boundary
does not exist... something like a switch on:

mail.rb:line 190 or so

unless
@header[''content-type''].params[''boundary'']
 bound = ::TMail.new_boundary
end

The above is really pseudo code as that would break, but that would be
the gist of it.

Any of the email experts on the list see any problem with that?

I can only think if you are replying to an email that is multipart,
that might break things as you could end up with the same mime
boundary, potentially... or forwarding an email.... you might end up
with two boundaries in the email, one set of real ones, and ones that
are quoted, but as these wouldn''t match /^--.*$/ it shouldn''t
not be a
problem for the parser.

Ideas list?

Mikel

I understand the message is composed in two phases, first the  
original content, here a multipart/mixed, is created. Then the  
original message is wrapped in a multipart/signed.

In the second phase, the original content should be added to the  
final message as-is, i.e. transparent, i.e. without modification. Not  
even the mime headers of the wrapped multipart/mixed are allowed to  
change. I think this will be difficult, since TMail will at least  
want to interpret the mime headers and even if the boundary is not  
changed, it may change little things like header line wrapping or  
indentation characters.

Did you also consider S/MIME for signing emails? I think S/MIME is  
more widely accepted than PGP.

Maarten Oelering

On 2-mrt-2008, at 4:47, Mikel Lindsaar wrote:
> On Sun, Mar 2, 2008 at 12:32 PM, Chris Powell  
> <chris at grubbybaby.com> wrote:
>> The challenge boils down to something very, very simple: is there  
>> a way to
>> get TMail to just ''trust me'' and accept my
fully-formed body-data and
>> headers without altering them at parse-time?  That way I could just
>> hand-assemble an exact string form of the message, then TMail-ify  
>> it and
>> send.
>
> TMail definitely clobbers every mimepart boundary, not quite sure if
> this is a feature or error ;)
>
> We could make it only create a new boundary if the existing boundary
> does not exist... something like a switch on:
>
> mail.rb:line 190 or so
>
> unless
@header[''content-type''].params[''boundary'']
>  bound = ::TMail.new_boundary
> end
>
> The above is really pseudo code as that would break, but that would be
> the gist of it.
>
> Any of the email experts on the list see any problem with that?
>
> I can only think if you are replying to an email that is multipart,
> that might break things as you could end up with the same mime
> boundary, potentially... or forwarding an email.... you might end up
> with two boundaries in the email, one set of real ones, and ones that
> are quoted, but as these wouldn''t match /^--.*$/ it
shouldn''t not be a
> problem for the parser.
>
> Ideas list?
>
> Mikel
> _______________________________________________
> Tmail-talk mailing list
> Tmail-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/tmail-talk

It may be heresy to mention this on this particular list, but I had the
issue with the mime headers being regenerated when I implemented DKIM
signing, and I just ended up using rmail to do it, though I use tmail for
the rest of the app''s mail handling.  Not an ideal solution but easy to
implement if you don''t mind using two mail libraries... :)

On Mon, Mar 3, 2008 at 12:51 AM, Maarten Oelering <maarten at
solovirtuoso.com>
wrote:
> I understand the message is composed in two phases, first the
> original content, here a multipart/mixed, is created. Then the
> original message is wrapped in a multipart/signed.
>
> In the second phase, the original content should be added to the
> final message as-is, i.e. transparent, i.e. without modification. Not
> even the mime headers of the wrapped multipart/mixed are allowed to
> change. I think this will be difficult, since TMail will at least
> want to interpret the mime headers and even if the boundary is not
> changed, it may change little things like header line wrapping or
> indentation characters.
>
> Did you also consider S/MIME for signing emails? I think S/MIME is
> more widely accepted than PGP.
>
> Maarten Oelering
>
> On 2-mrt-2008, at 4:47, Mikel Lindsaar wrote:
>
> > On Sun, Mar 2, 2008 at 12:32 PM, Chris Powell
> > <chris at grubbybaby.com> wrote:
> >> The challenge boils down to something very, very simple: is there
> >> a way to
> >> get TMail to just ''trust me'' and accept my
fully-formed body-data and
> >> headers without altering them at parse-time?  That way I could
just
> >> hand-assemble an exact string form of the message, then TMail-ify
> >> it and
> >> send.
> >
> > TMail definitely clobbers every mimepart boundary, not quite sure if
> > this is a feature or error ;)
> >
> > We could make it only create a new boundary if the existing boundary
> > does not exist... something like a switch on:
> >
> > mail.rb:line 190 or so
> >
> > unless
@header[''content-type''].params[''boundary'']
> >  bound = ::TMail.new_boundary
> > end
> >
> > The above is really pseudo code as that would break, but that would be
> > the gist of it.
> >
> > Any of the email experts on the list see any problem with that?
> >
> > I can only think if you are replying to an email that is multipart,
> > that might break things as you could end up with the same mime
> > boundary, potentially... or forwarding an email.... you might end up
> > with two boundaries in the email, one set of real ones, and ones that
> > are quoted, but as these wouldn''t match /^--.*$/ it
shouldn''t not be a
> > problem for the parser.
> >
> > Ideas list?
> >
> > Mikel
> > _______________________________________________
> > Tmail-talk mailing list
> > Tmail-talk at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/tmail-talk
>
> _______________________________________________
> Tmail-talk mailing list
> Tmail-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/tmail-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/tmail-talk/attachments/20080303/48f8ba57/attachment.html

On Tue, Mar 4, 2008 at 5:00 AM, Matt Murphy <mmmurf at gmail.com>
wrote:
> It may be heresy to mention this on this particular list, ...
> implement if you don''t mind using two mail libraries... :)
Nyah, whatever works is good :)

Though I am thinking that figuring out how to encapsulate something
without touching the boundaries would be a good idea.

Can anyone think of a problem that would come up if I made TMail not
generate a new boundary if one already exists?

That way if you did a mail.boundary = blahstring then it would use
that instead of re-generating a random string for you.

Doesn''t look that hard to implement and I can''t think of a
case that
would break because of it.

Mikel

> I''m grateful for any thoughts...I sure can''t find any
reference of prior art
> on this one.
I was just playing around and I think this should handle it:

Try the following:

irb
> require ''tmail''
> mail = TMail::Mail.new
> mail.body = your_body_text_string_with_custom_boundaries
> mail[''Mime-Version''] = "1.0"
> mail[''content-type''] = ''multipart/mixed;
boundary=--your-secret-boundary''
> puts mail.encoded
Does that solve it for you?

Mikel

Hi Matt,

Glad you wanted to share that with us. I am very interested in your  
ruby based DKIM solution. I also want to implement DomainKeys and  
DKIM in Ruby. Is you solution proprietary or open source?

I also started off with RubyMail, since it was more flexible and  
simpler than Tmail. However, it also has a number of drawbacks and  
shortcomings. I even considered writing a my own email library, which  
would be flexible and transparent suiting more advanced email  
applications. I have done this before in Java, before JavaMail was  
good enough to be used in serious email applications. Because I have  
little time to spare I decided to go for TMail and help it where and  
when I can. However I must say that most of the code is acacadabra to  
me.

Maarten Oelering

On 3-mrt-2008, at 19:00, Matt Murphy wrote:
> It may be heresy to mention this on this particular list, but I had  
> the issue with the mime headers being regenerated when I  
> implemented DKIM signing, and I just ended up using rmail to do it,  
> though I use tmail for the rest of the app''s mail handling.  Not
an
> ideal solution but easy to implement if you don''t mind using two  
> mail libraries... :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/tmail-talk/attachments/20080304/664dca64/attachment-0001.html

On Tue, Mar 4, 2008 at 6:21 PM, Maarten Oelering
<maarten at solovirtuoso.com> wrote:
> applications. Because I have little time to spare I decided to go for TMail
> and help it where and when I can. However I must say that most of the code
> is acacadabra to me.
Part of the problem is the total lack of comments and the parse engine.

I have been working on improving this, but I think that would be
almost the most valuable thing right now is to get this finished off.
Adding some basic docs to it would help it a lot.

I look through the code and it looks like Minero created it with a
grand master plan that didn''t get finished, some of the stuff is a bit
overly complex and could do with simplification.

Maarten, did you write your own parser engine on the Java version?

Mikel

Hi Mikel,

You are right. And you deserve all the credits, doing the dirty work  
of cleaning up and documenting the code.

Yes, I did write my own parser which could also do the reverse,  
generating emails, but it was part of commercial software.

For me the yaml parser and the "strategy" stuff (i don''t know
what it
does) is what makes the tmail code so obscure. You approach of first  
understanding, documenting  and cleaning up is I think a very good  
approach. Often programmers start refactoring before they completely  
understand the code, breaking things that previously worked, and  
underestimating the effort it takes to implement 100% working code.

Maarten

On 4-mrt-2008, at 8:38, Mikel Lindsaar wrote:
>
> Part of the problem is the total lack of comments and the parse  
> engine.
>
> I have been working on improving this, but I think that would be
> almost the most valuable thing right now is to get this finished off.
> Adding some basic docs to it would help it a lot.
>
> I look through the code and it looks like Minero created it with a
> grand master plan that didn''t get finished, some of the stuff is a
bit
> overly complex and could do with simplification.
>
> Maarten, did you write your own parser engine on the Java version?
>

On Tue, Mar 4, 2008 at 11:06 PM, Maarten Oelering
<maarten at solovirtuoso.com> wrote:
>  You are right. And you deserve all the credits, doing the dirty work
>  of cleaning up and documenting the code.
Thanks :)
>  Yes, I did write my own parser which could also do the reverse,
>  generating emails, but it was part of commercial software.
Dang :)
>  For me the yaml parser and the "strategy" stuff (i
don''t know what it
>  does) is what makes the tmail code so obscure. You approach of first
Same here... that RACC parser is enough to drive you nuts. As for the
strategy interface, I _really_ don''t get it either.  I can follow the
code path now that I have stepped through it enough times and
understand _how_ it works, but not _why_ he chose _that_ method of
doing it...

I see the new Ruby Parse Tree stuff... it has some nice documentation
and seems cleaner than RACC, I am thinking of going down that path,
but I know effectively zero about that sort of thing (I can read it
and see how it works, but never played seriously with it).  That might
be an option.

Anyway, idle thoughts...

I think the first thing I want to do after the documentation is to
move all the publically accessible class and instance methods into
their own file, right now they are sprinkled through the code,
TMail::Mail class for example is in about 8 files in total.

There is something to be said for the Java abstraction of the interface :)

Then, once complete, go back through and hack into the parser
engine... would like your help on that one Maarten :)

Mikel

Thank you all for the enthusiastic response and several excellent  
suggestions.  I will have to go back and experiment with them.

To solve my immediate problem I ended up taking the simplest approach  
possible:  I retained ActionMailer/TMail for the bulk of my mail  
duties, but I dropped down into Net::SMTP for this one particular  
task.  I carefully assembled a header/body combo exactly the way I  
wanted it, and then pushed it off to the SMTP server.

Regards,
Chris
--
Chris Powell                        "Exit, pursued by a bear."
chris at grubbybaby.com                 - Shakespeare, "The
Winter''s Tale"




On Mar 3, 2008, at 6:09 PM, Mikel Lindsaar wrote:
>> I''m grateful for any thoughts...I sure can''t find any
reference of
>> prior art
>> on this one.
>
> I was just playing around and I think this should handle it:
>
> Try the following:
>
> irb
>> require ''tmail''
>> mail = TMail::Mail.new
>> mail.body = your_body_text_string_with_custom_boundaries
>> mail[''Mime-Version''] = "1.0"
>> mail[''content-type''] = ''multipart/mixed;
boundary=--your-secret-
>> boundary''
>> puts mail.encoded
>
> Does that solve it for you?
>
> Mikel
> _______________________________________________
> Tmail-talk mailing list
> Tmail-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/tmail-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/tmail-talk/attachments/20080304/0ae2ec6a/attachment.html