sup talk - Jun 2009 - Character encoding when displaying quoted-printable messages

Hi, I''ve been looking at sup for only a day, as I''m trying to
find a
decent email client. I migrated from mutt to gmail, but now it''s time
to move away from gmail because their client is not smart enough.

Sup has passed my number 1 feature test; I can easily read messages
while keeping multiple unsent messages available. Mutt does that, but
it''s not easy to see what messages you are currently working on.

However, I''m coming across some character decoding issues with
quoted-printable that are making some messages hard to read. Any
advice on how to improve things would be welcome ... In the example
below, a NBSP entity is showing up as the characters "M-BM-"

I''m on Ubuntu 9.04, with the current master from git. Sup is running
in GNOME Terminal 2.26.0, with UTF-8 encoding, although unsurprisingly
switching to ISO-8859-1 makes no difference. $TERM=xterm (or
''screen''.
vt100 and vt220 cause sup to halt at startup).  This is the text that
gets displayed for a sample message sent from Yahoo! mail ...

Visible in sup thread-view-mode:
videos for specific hints regarding the gestures.M-BM-  Do you have a decent
recording device?M-BM-  Have you seen what''s available?M-BM-  Do you
have
editing software?

Message source:
X-Mailer: YahooMailClassic/5.4.12 YahooMailWebService/0.7.289.15
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="0-1465842634-1245251490=:14771"
...
--0-1465842634-1245251490=:14771
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
...
more videos for specific hints regarding the gestures.=A0 Do you have a decent
video recording device?=A0 Have you seen what''s available?=A0 Do you
have any video editing software?

Displayed in Gmail client:
I was thinking about cleaning up the videos on YouTube, and perhaps
adding more videos for specific hints regarding the gestures.  Do you
have a decent video recording device?  Have you seen what''s available?
 Do you have any video editing software?

HTML source when viewing the MIME attached message locally:
I was thinking about cleaning up the videos on YouTube, and perhaps
adding more videos for specific hints regarding the gestures.  Do
you have a decent video recording device?  Have you seen
what''s
available?  Do you have any video editing
software?<br><br>[1]

On Thu, Jun 18, 2009 at 9:39 AM, Jim Cheetham<jim at gonzul.net>
wrote:
> However, I''m coming across some character decoding issues with
> quoted-printable that are making some messages hard to read. Any
> advice on how to improve things would be welcome ... In the example
> below, a NBSP entity is showing up as the characters "M-BM-"
Nicely fixed by using the hack ncursesw branch, by the way.
http://sup.rubyforge.org/wiki/wiki.pl?UTF8

-jim

Excerpts from Jim Cheetham''s message of Wed Jul 01 07:20:10 +0200
2009:
> Nicely fixed by using the hack ncursesw branch, by the way.
> http://sup.rubyforge.org/wiki/wiki.pl?UTF8
Has anyone noticed that newer versions of ncurses-ruby plus that
ncursesw hack [1] (to link to ncurses with widechar support) breaks search?

When I search for ''foo'', sup launches a search for a bunch of
weird
characters, not for ''foo'', which obviously returns nothing
useful.

I can trigger this with ncurses-ruby 1.2.2, 1.2.3. Version 1.1 seems to
be unaffected.

[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=477366#52
-- 
Exherbo KDE, X.org maintainer

Reformatted excerpts from Ingmar Vanhassel''s message of
2009-07-01:
> [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=477366#52
Interesting thread. I disagree with the conclusion that a separate
libncursesw-ruby package should be created. Maybe I''ll respond.

The analysis of Sup is wrong. He says: "Looking at the source code of
the mailer I''d say that it is not really suited for UTF-8 encoded
strings yet, as it still assumes that the length of a string in bytes is
equal to the number of characters in the string." This is not really
true. What Sup assumes is that String#length gives the length of the
string in characters. This is obviously false in Ruby 1.8 (without
monkeypatching), but it is true in Ruby 1.9.

What *does* need to happen for Sup to really, actually, correctly
display non-ASCII characters, besides having a ncurses library that''s
linked against ncursesw, is to be able to call the ''wcwidth''
and
''wcswidth'' functions. The current #display_length function is
a hack
that''s broken for e.g. Chinese characters.

One way to do this would be to use dl/import like we do for setlocale().
I''ve played around with this but haven''t really gotten it to
work. If
someone can get any further than the below, please let me know:

  require ''dl/import''
  module LibC
    extend DL.const_defined?(:Importer) ? DL::Importer : DL::Importable
    dlload "libc.so.6"
    extern "void setlocale(int, const char *)"
    extern "int wcwidth(int)"
    extern "int wcswidth(const int*, int)"
  end

That "works", as in doesn''t throw any exceptions, but then
calling
LibC.wcswidth doesn''t return the right thing, presumably because the
Ruby string isn''t being converted into an array of ints, or something.
I
don''t really know how this works.
> Has anyone noticed that newer versions of ncurses-ruby plus that
> ncursesw hack [1] (to link to ncurses with widechar support) breaks
> search?
I don''t have these newer versions available on my system yet (Ubuntu).
> When I search for ''foo'', sup launches a search for a
bunch of weird
> characters, not for ''foo'', which obviously returns
nothing useful.
Disturbing.
-- 
William <wmorgan-sup at masanjin.net>

Ingmar Vanhassel, Wed Jul 01 10:38:36 +0200 2009:
 
> Has anyone noticed that newer versions of ncurses-ruby plus that
> ncursesw hack [1] (to link to ncurses with widechar support) breaks search?
I can confirm the behaviour.
> When I search for ''foo'', sup launches a search for a
bunch of weird
> characters, not for ''foo'', which obviously returns
nothing useful.
True. However, with the following workaround, you can get proper search results:
- search for foo -> garbled search, no proper results
- hit ''x'' to destroy the result buffer
- hit ''\'' to search again.
- hit uparrow to go to previous search. backspace all the garbled characters
there.
- repeat the uparrow + remove until it shows your last working search (or until
there''s no more previous search to go to)
- type foo again, hit enter: works.

Not that this is the type of thing you''d want to do for a simple
search. Guess we''ll have to wait for ruby 1.9?

cheers,
- J?rg-Hendrik

> 
> I can trigger this with ncurses-ruby 1.2.2, 1.2.3. Version 1.1 seems to
> be unaffected.
> 
> [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=477366#52