sup talk - Feb 2010 - PGP INLINE - can't make it work

First of wall I want to say I absolutely love sup! I used all the major mail 
clients like mutt, claws, thunderbird, kmail. All mail clients just fail (mutt 
was the closest though...). Sup so far has the potential to be the best of 
them. If only I can get it to read the inline gpg that enigmail makes...

Example email looks like this in sup: 

 -----BEGIN PGP MESSAGE-----
  Charset: UTF-8
  Version: GnuPG v1.4.9 (MingW32)
  Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

  hQIOA3WKxDKSmLd7EAf/cKH72GNHWgkY46sjHuYBlV5Q9udlmjB1ucTqtVfSUysH
[cut]
 NYqMmmJ1HHG58B9bPqGFPNe/jEPeiiqj9Ez58kYzS32VeelJFwV+O8K9/TAvG4HN
  fEIgtYMGj0Uc5v0eZ7yVqFfluJrEX5V7mN2InahTCD+uFEfguvld8rZagEtlyxU5
  =nAMm
  -----END PGP MESSAGE-----

And it gets displayed like this, no error messages or anything. I have 
configured my gpg in sup as per this wiki page:
http://sup.rubyforge.org/wiki/wiki.pl?GPG
(with the numeric ID and such). 

And I enabled verbose debug logging, here''s the log:
(cutting leading timestamp for better reading)

using character set encoding "UTF-8"
No ''chronic'' gem detected. Install it for date/time query
restrictions.
No ''ncursesw'' gem detected. Install it for wide character
support.
dynamically loading setlocale() from libc.so.6
setting locale...
using xapian index
locking /home/piotr/.sup/lock...
crypto: detected gpg binary in /usr/bin/gpg
starting curses
initializing log buffer
Welcome to Sup! Log level is set to debug.
initializing inbox buffer
ready for interaction!
colormap: for color twiddle_color, using id 1 -> 4, -1
colormap: for color status_color, using id 2 -> 7, 4
colormap: for color tagged_color_highlight, using id 3 -> 3, 6
colormap: for color date_color_highlight, using id 4 -> 0, 6
colormap: for color to_me_color_highlight, using id 5 -> 2, 6
colormap: for color tagged_color, using id 6 -> 3, -1
colormap: for color date_color, using id 7 -> 7, -1
colormap: for color to_me_color, using id 8 -> 2, -1
colormap: for color snippet_color, using id 9 -> 6, -1
scanning maildir /home/piotr/Mail...
done scanning maildir
crypto: running: /usr/bin/gpg --quiet --batch --no-verbose --logger-fd 1 --
use-agent --output /tmp/32042-0-redwood.outp
scanning maildir /home/piotr/Mail...
done scanning maildir
crypto: running: /usr/bin/gpg --quiet --batch --no-verbose --logger-fd 1 --
use-agent --output /tmp/32042-0-redwood.outp
crypto: running: /usr/bin/gpg --quiet --batch --no-verbose --logger-fd 1 --
use-agent --output /tmp/32042-2-redwood.outp
crypto: running: /usr/bin/gpg --quiet --batch --no-verbose --logger-fd 1 --
use-agent --output /tmp/32042-0-redwood.outp
colormap: for color alternate_patina_color, using id 10 -> 0, 4
colormap: for color message_patina_color, using id 11 -> 0, 2
colormap: for color system_buf_color_highlight, using id 12 -> 7, 6

(no errors there as far as I can see... And looks like it''s running the
gpg
binary succesfully... I just changed the maildir path for privacy in this log)

GPG is configured to use gpg-agent, the gpg-agent is running, pinentry program 
shows up when running sup and I enter the correct password. I checked the 
password many times already. Also, the environment variable $GPG_AGENT_INFO is 
present and visible in sup environment (it''s global and visible
anywhere).
I already tried it without "use gpg-agent" and the effect was the
same.

In mutt this worked, I mean the inline gpg messages were decrypted using this 
same setup of gpg-agent and gpg. It also works in kmail (although not 
perfectly, but decodes the messages).

Now, this may be a clue: when normal gpg message is used (mime) it sometimes 
geta decrypted and sometimes doesn''t. If it doesn''t, sup only
says this in the
message: 
    x This message has been decrypted for display
    x Attachment: sup-attachment-1267203299-4513. (multipart/mixed; 71b)

When I''m trying to display this attachment it says: 
For some bizarre reason, RubyMail was unable to parse this attachment.
and in the status line:
Couldn''t execute view command, viewing as text.

(this may be because I have no run-mailcap file, but the message was in plain 
text, no html, so it shouldn''t be needed now should it? I''m
planning on
getting some run-mailcap script from some debian system when I have some time, 
but for now this inline GPG is my main priority)

I''m still not sure why they sometimes decrypt correctly and sometimes
not, but
I can work with this, at least it works sometimes. But even when mime gpg 
works, the inline stuff doesn''t. 

Verisions:
I''m using sup version 0.10.2 from Archlinux AUR. 
ruby version is 1.9.1
xapian-core is 1.0.18

Is there anything I can do to get this working? This is a crucial feature for 
me as almost all my work colleagues use thunderbird and enigmail to send their 
email and that uses inline pgp crap as default (which I know is not conforming 
to any standards etc.). And even if I can convince verybody to switch to mime, 
I still have a couple hundred megs of archives in this damn format. 

Moreover, I searched the list archives and it looks like some people have got 
it working as there have been patches to make it work a couple of months ago. 
And my version is recent (January) so those patches should be in it. It just 
doesn''t for me for some reason :(

I know I should read crypto.rb and figure it out from there. I tried, I just 
can''t understand it, as I''m not experienced in ruby, or email
clients code for
that matter. 

Please help!
Thank you
Piotr

Excerpts from kosmici-atakuja''s message of Fri Feb 26 18:08:36 +0100
2010:
> First of wall I want to say I absolutely love sup! I used all the major
mail
> clients like mutt, claws, thunderbird, kmail. All mail clients just fail
(mutt
> was the closest though...). Sup so far has the potential to be the best of 
> them. If only I can get it to read the inline gpg that enigmail makes...
> 
+1
> Is there anything I can do to get this working? This is a crucial feature
for
> me...
> 
+1
> Moreover, I searched the list archives and it looks like some people have
got
> it working as there have been patches to make it work a couple of months
ago.
> And my version is recent (January) so those patches should be in it. It
just
> doesn''t for me for some reason :(
> 
I think there was a (preliminary) patch from Michael Stapelberg
floating around, but somehow that didn''t make it into next?
> I know I should read crypto.rb and figure it out from there. I tried, I
just
> can''t understand it, as I''m not experienced in ruby, or
email clients code for
> that matter. 
> 
+1
> Please help!
+1
> Thank you
> Piotr
Ciao,
  Sven
-- 
Sven Schober, sven.schober at uni-ulm.de                    |UNI ULM
http://www-vs.informatik.uni-ulm.de/dept/staff/schober/  |DISTRIBUTED
Room O27-346, Phone: +49-731-5024146 [+49-179-5060182]   |SYSTEMS LAB
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL:
<http://rubyforge.org/pipermail/sup-talk/attachments/20100226/65d63480/attachment.bin>

I digged a bit further, trying to decode the emails from commandline with
"gpg
--decode". For some of the inline GPG emails the commandline version fails,
but for some other it works. Even for those that it works, sup just shows 
undecoded version which leads me to believe it doesn''t even try to
decode,
which means it doesn''t recognize the email as encrypted at all. 

As far as I can deduct, sup only recognizes encrypted emails if they have 
something like this inside:
Content-Type: multipart/encrypted;
  boundary="nextPart15239034.i9UOPnLxZ9";
  protocol="application/pgp-encrypted"

Or similar. Which is a reasonable behaviour.

On the other hand, if the damned enigmail creates an email with
"Content-Type:
text/plain;" and then proceeds to include "-----BEGIN PGP
MESSAGE-----" in the
body, sup treats this as (no surprise) text/plain and doesn''t decode.
If it
tried, it would succeed since this message passed through gpg --decode is 
decoded okay.

So, I guess if this functionality is to be included in sup (and I think it 
doesn''t have to be since enigmail just makes broken messages and
there''s no
reason why anybody should put up with that:) it should not be in crypto.rb but 
somewhere else, where the decision is made to even use functions from 
crypto.rb (like decrypt) on a message. I''m no programmer but I guess it
should
be something like this:
if payload contains "BEGIN PGP MESSAGE" then run the decrypt function
from
crypto.rb on it. 

For now - does anybody know a way to mass decrypt all messages in my mailbox 
so I can keep them in plain text and use sup anyway? That would be ideal 
solution but I just can''t find a program to do that. And gpg itself
isn''t
helpful since it just prints decoded payload on --output, without the headers, 
so it''s no use in batch. I can try to code this in bash but maybe
there''s
something out there already?

Thank you
Piotr

On Friday 26 February 2010 18:43:04 Sven Schober wrote:
> Excerpts from kosmici-atakuja''s message of Fri Feb 26 18:08:36
+0100 2010:
> > First of wall I want to say I absolutely love sup! I used all the
major
mail 
> > clients like mutt, claws, thunderbird, kmail. All mail clients just
fail
(mutt 
> > was the closest though...). Sup so far has the potential to be the
best of
> > them. If only I can get it to read the inline gpg that enigmail
makes...
> > 
> +1
> 
> > Is there anything I can do to get this working? This is a crucial
feature
for 
> > me...
> > 
> +1
> > Moreover, I searched the list archives and it looks like some people
have
got 
> > it working as there have been patches to make it work a couple of
months
ago. 
> > And my version is recent (January) so those patches should be in it.
It
just 
> > doesn''t for me for some reason :(
> > 
> I think there was a (preliminary) patch from Michael Stapelberg
> floating around, but somehow that didn''t make it into next?
> 
> > I know I should read crypto.rb and figure it out from there. I tried,
I
just 
> > can''t understand it, as I''m not experienced in ruby,
or email clients code
for 
> > that matter. 
> > 
> +1
> 
> > Please help!
> +1
> 
> > Thank you
> > Piotr
> 
> Ciao,
>   Sven
> -- 
> Sven Schober, sven.schober at uni-ulm.de                    |UNI ULM
> http://www-vs.informatik.uni-ulm.de/dept/staff/schober/  |DISTRIBUTED
> Room O27-346, Phone: +49-731-5024146 [+49-179-5060182]   |SYSTEMS LAB
>

Hi Piotr,

Excerpts from piotr.kempa''s message of Di M?r 02 11:49:59 +0100
2010:
> So, I guess if this functionality is to be included in sup (and I think it 
> doesn''t have to be since enigmail just makes broken messages and
there''s no
> reason why anybody should put up with that:) it should not be in crypto.rb
but
> somewhere else, where the decision is made to even use functions from 
> crypto.rb (like decrypt) on a message. I''m no programmer but I
guess it should
As mentioned, I wrote a patch to do this. It is currently in review and will
hopefully be included soon.

Best regards,
Michael

On Tue, Mar 02, 2010 at 11:49 (+0100), piotr.kempa at wp.eu
wrote:
> For now - does anybody know a way to mass decrypt all messages in my
> mailbox so I can keep them in plain text and use sup anyway?
Using mutt with tagging and decrypt-save should work (didn''t test that,
though). Google for decrypt-save, it''s not documented in the manual.


Best wishes,
  /Sam

Sounds cool, I can''t wait then. Let''s hope the patch is
included in 0.11.
Thanks Michael!
Piotr

On Tuesday 02 March 2010 14:59:19 Michael Stapelberg
wrote:
> Hi Piotr,
> 
> Excerpts from piotr.kempa''s message of Di M?r 02 11:49:59 +0100
2010:
> > So, I guess if this functionality is to be included in sup (and I
think it
> > doesn''t have to be since enigmail just makes broken messages
and there''s
no 
> > reason why anybody should put up with that:) it should not be in
crypto.rb
but 
> > somewhere else, where the decision is made to even use functions from 
> > crypto.rb (like decrypt) on a message. I''m no programmer but
I guess it
should 
> As mentioned, I wrote a patch to do this. It is currently in review and
will
> hopefully be included soon.
> 
> Best regards,
> Michael
> _______________________________________________
> sup-talk mailing list
> sup-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-talk
>

On Tuesday 02 March 2010 15:19:23 piotr.kempa at wp.eu
wrote:
> Sounds cool, I can''t wait then. Let''s hope the patch is
included in 0.11.
> Thanks Michael!
> Piotr
> 
> On Tuesday 02 March 2010 14:59:19 Michael Stapelberg wrote:
> > Hi Piotr,
> > 
> > Excerpts from piotr.kempa''s message of Di M?r 02 11:49:59
+0100 2010:
> > > So, I guess if this functionality is to be included in sup (and I
think
it 
> > > doesn''t have to be since enigmail just makes broken
messages and there''s
> no 
> > > reason why anybody should put up with that:) it should not be in 
crypto.rb 
> but 
> > > somewhere else, where the decision is made to even use functions
from
> > > crypto.rb (like decrypt) on a message. I''m no programmer
but I guess it
> should 
> > As mentioned, I wrote a patch to do this. It is currently in review
and
will
> > hopefully be included soon.
> > 
Can anyone confirm if this patch made it to 0.11? I just installed the new 
version and it still doesn''t work but if it''s there then maybe
I''m doing
something wrong. If it didn''t, is it posted anywhere? I''ll be
happy to patch
0.11 myself instead of waiting for the next version. I really, really want to 
start using sup :)

Thank you, 
Piotr

On Monday 08 March 2010 11:18:55 piotr.kempa at wp.eu
wrote:
> On Tuesday 02 March 2010 15:19:23 piotr.kempa at wp.eu wrote:
> > Sounds cool, I can''t wait then. Let''s hope the patch
is included in 0.11.
> > Thanks Michael!
> > Piotr
> > 
> > On Tuesday 02 March 2010 14:59:19 Michael Stapelberg wrote:
> > > Hi Piotr,
> > > 
> > > Excerpts from piotr.kempa''s message of Di M?r 02
11:49:59 +0100 2010:
> > > > So, I guess if this functionality is to be included in sup
(and I
think 
> it 
> > > > doesn''t have to be since enigmail just makes broken
messages and
there''s 
> > no 
> > > > reason why anybody should put up with that:) it should not
be in
> crypto.rb 
> > but 
> > > > somewhere else, where the decision is made to even use
functions from
> > > > crypto.rb (like decrypt) on a message. I''m no
programmer but I guess
it 
> > should 
> > > As mentioned, I wrote a patch to do this. It is currently in
review and
> will
> > > hopefully be included soon.
> > > 
> 
> Can anyone confirm if this patch made it to 0.11? I just installed the new 
> version and it still doesn''t work but if it''s there then
maybe I''m doing
> something wrong. If it didn''t, is it posted anywhere?
I''ll be happy to patch
> 0.11 myself instead of waiting for the next version. I really, really want 
to 
> start using sup :)
> 
> Thank you, 
> Piotr
> 
Okay, I found the patch here: http://www.mail-archive.com/sup-
talk at rubyforge.org/msg02770.html

And it is included in 0.11, I just checked in the code. 

However, this still doesn''t work because the header doesn''t
contain Content
Type = application/pgp, it is text/plain, which is absolutely crazy and I have 
no idea what the enigmail developers are thinking... Anyway, here''s how
it
looks like (only the important part):

X-Enigmail-Version: 0.95.7
Content-Type: text/plain;
  charset=UTF-8
Content-Transfer-Encoding: quoted-printable

-----BEGIN PGP MESSAGE-----
Charset: UTF-8
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
=20
hQIOA3WKxDKSmLd7EAf9GP5JLA77IcGOAUrOZ8fGCEtK8Pm838Me+pMD94ulHlDS
LcidQKPSVGXaj4lLU8RqxoMKTpxuaZAm1u286C+SL9eQc51N3gH/X57loKhjCRb+
[...]

So in order to decide about encrypted content you actually have to look inside 
the body of the message, not just at the header. At least I couldn''t
find any
indication in the headers :(

All in all, I wouldn''t be surprised if you decided not to support such
broken
emails so I''ll just go about learning enough ruby to patch it myself :)

Piotr

Hi Piotr,

Excerpts from piotr.kempa''s message of Mo M?r 08 11:29:47 +0100
2010:
> Okay, I found the patch here: http://www.mail-archive.com/sup-
> talk at rubyforge.org/msg02770.html
> 
> And it is included in 0.11, I just checked in the code. 
You are mistaken, this is a different patch (not for inline gpg).
> So in order to decide about encrypted content you actually have to look
inside
> the body of the message, not just at the header. At least I
couldn''t find any
> indication in the headers :(
This is actually the very intention of inline gpg -- to work with existing
standards/mailclients which do not support different headers.

Unfortunately I did not yet have time to rework the patch, but I hope to do
so in the next few days.

Best regards,
Michael

> However, this still doesn''t work because the header
doesn''t contain Content
> Type = application/pgp, it is text/plain, which is absolutely crazy and I
have
> no idea what the enigmail developers are thinking...
In fact I think this might be a design decision on the part of PGP, where the
original thinking went something like this: It should be possible to use it
with ANY mailer, even those that do not know about it. Thus the support for
"inline PGP", i.e. a PGP message embedded within the text of a regular
email.
>From this standpoint it makes sense to keep the Content-Type as text/plain.
Imagine your recipient''s MUA has no idea how to handle application/pgp.
With
text/plain, she''s still able to copy/paste into a separate PGP
application.

The merit of this design decision is debatable, but I think that''s the
way it
came to be.

So, I''d say it''s unlikely Enigmail is going to change the
header. I''d also
guess that everyone else is doing it the same way.

So, actually inspecting the body of the message is the way to go...

HTH,
pesco

On Monday 08 March 2010 11:50:56 Sven Moritz Hallberg
wrote:
> > However, this still doesn''t work because the header
doesn''t contain
Content 
> > Type = application/pgp, it is text/plain, which is absolutely crazy
and I
have 
> > no idea what the enigmail developers are thinking...
> 
> In fact I think this might be a design decision on the part of PGP, where 
the
> original thinking went something like this: It should be possible to use it
> with ANY mailer, even those that do not know about it. Thus the support for
> "inline PGP", i.e. a PGP message embedded within the text of a
regular
email.
> >From this standpoint it makes sense to keep the Content-Type as
text/plain.
> Imagine your recipient''s MUA has no idea how to handle
application/pgp. With
> text/plain, she''s still able to copy/paste into a separate PGP
application.
> 
> The merit of this design decision is debatable, but I think that''s
the way
it
> came to be.
> 
> So, I''d say it''s unlikely Enigmail is going to change the
header. I''d also
> guess that everyone else is doing it the same way.
> 
> So, actually inspecting the body of the message is the way to go...
> 
> HTH,
> pesco
> _______________________________________________
> sup-talk mailing list
> sup-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-talk
> 
> 
Thank you and Michael for the explanation. Now it all makes sense. I''ll
just
keep watching the git repo in case a patch makes it there. Or  this list. If 
anybody needs help testing such patch - I''m at your service. Coding it
myself
could be way over my head though...

Thank you, 
Piotr

Hi Piotr,

Excerpts from piotr.kempa''s message of Mo M?r 08 12:20:02 +0100
2010:
> Thank you and Michael for the explanation. Now it all makes sense.
I''ll just
> keep watching the git repo in case a patch makes it there. Or  this list.
If
> anybody needs help testing such patch - I''m at your service.
Coding it myself
> could be way over my head though...
I just sent an updated patch to sup-devel, you can find it at
http://rubyforge.org/pipermail/sup-devel/2010-March/000463.html

Testing it would be appreciated.

Best regards,
Michael

Excerpts from Michael Stapelberg''s message of 2010-03-09 18:06:35
+0100:
> Hi Piotr,
> 
> Excerpts from piotr.kempa''s message of Mo M?r 08 12:20:02 +0100
2010:
> > Thank you and Michael for the explanation. Now it all makes sense.
I''ll just
> > keep watching the git repo in case a patch makes it there. Or  this
list. If
> > anybody needs help testing such patch - I''m at your service.
Coding it myself
> > could be way over my head though...
> I just sent an updated patch to sup-devel, you can find it at
> http://rubyforge.org/pipermail/sup-devel/2010-March/000463.html
> 
> Testing it would be appreciated.
> 
> Best regards,
> Michael
Hello Michael, 

Thank you. I just tried the patch and unfortunately sup crashes on me
now. I patched the 0.11 version and checked that the patches were
applied successfully (as far as I could at least). 

I''m attaching the crash log. The crash happens like this: I add the
new source maildir with encrypted emails, run sup, it executes, sees
the new source and starts indexing emails and then crashes. 

Hope this helps somehow. 
 
Piotr
-- 
Piotr Kempa
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: exception-log.txt
URL:
<http://rubyforge.org/pipermail/sup-talk/attachments/20100310/bcbd243e/attachment-0001.txt>

Hi Piotr,

Excerpts from Piotr Kempa''s message of Mi M?r 10 11:53:00 +0100
2010:
> Thank you. I just tried the patch and unfortunately sup crashes on me
> now. I patched the 0.11 version and checked that the patches were
> applied successfully (as far as I could at least). 
Thanks for testing. Can you please see if the updated version works
for you (attached to this mail)?

Best regards,
Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Implement-inline-GPG.patch
Type: application/octet-stream
Size: 7925 bytes
Desc: not available
URL:
<http://rubyforge.org/pipermail/sup-talk/attachments/20100310/d99ca5a5/attachment.obj>

Excerpts from Michael Stapelberg''s message of 2010-03-10 12:14:04
+0100:
> Hi Piotr,
> 
> Excerpts from Piotr Kempa''s message of Mi M?r 10 11:53:00 +0100
2010:
> > Thank you. I just tried the patch and unfortunately sup crashes on me
> > now. I patched the 0.11 version and checked that the patches were
> > applied successfully (as far as I could at least). 
> Thanks for testing. Can you please see if the updated version works
> for you (attached to this mail)?
> 
> Best regards,
> Michael
Wow, this works!! Thank you so much Michael! This will make my job soooo much
easier, faster and more pleasant :) 

I have just two messages that give errors to sup-sync, but the remaining ones
(a hundred or more) just work beautifully.  The error is like this: 

   [2010-03-10 15:25:28 +0100] Error while running gpg: gpg: invalid armor
header:
	    gpg: malformed CRC
   gpg: encrypted_mdc packet with unknown version 255
   gpg: quoted printable character in armor - probably a buggy MTA has been used
   
   [2010-03-10 15:25:33 +0100] Error while running gpg: gpg: invalid armor
      header: gpg: CRC error; 9C0326 - DC39C0
   gpg: encrypted_mdc packet with unknown version 255
   gpg: quoted printable character in armor - probably a buggy MTA has been used

(two long lines with the timestamp broken by me)

But I guess this is what it says it is - an invalid message with CRC errors in
the GPG payload, so this should be nothing to worry about. 

Anyway, thanks a million Michael!
-- 
Piotr Kempa

Hi Piotr,

Excerpts from Piotr Kempa''s message of Mi M?r 10 15:41:35 +0100
2010:
> Wow, this works!! Thank you so much Michael! This will make my job soooo
much
> easier, faster and more pleasant :) 
You?re welcome. Thanks for testing. I will now post the updated patch to
sup-devel and hope that rlane will merge it soon.

Best regards,
Michael