thr3ads.net - sup talk - [sup-talk] [bug] NoMemoryError on huge mbox [Jun 2008]

If this information is useful, please help other people find it:
Share via:

Gabriel Landais

2008-Jun-01 17:26 UTC

[sup-talk] [bug] NoMemoryError on huge mbox

Hi,
 I''m not able to load my Thunderbird mbox into sup. Added my Inbox
mbox with sup-add/sup-config :

-------------------------------------------------
[glandais at
server:~/.mozilla-thunderbird/4v86e51w.default/Mail/pops.codelutin.com]
$ /var/lib/gems/1.8/gems/sup-0.5/bin/sup-config
[Sun Jun 01 19:14:24 +0200 2008] using character set encoding "UTF-8"
[Sun Jun 01 19:14:24 +0200 2008] optional ''chronic'' library
not found
(run ''gem install chronic'' to install)
/var/lib/gems/1.8/gems/highline-1.4.0/lib/highline/system_extensions.rb:125:
warning: Insecure world writable dir /opt/java in PATH, mode 040777
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/crypto.rb:18: warning: Insecure
world writable dir /opt/java in PATH, mode 040777
[Sun Jun 01 19:14:25 +0200 2008] crypto: detected gpg binary in /usr/bin/gpg
Howdy neighbor! This here''s sup-config, ready to help you jack in to
the next generation of digital cyberspace: the text-based email
program. Get ready to be the envy of everyone in your internets
with your amazing keyboarding skills! Jump from email to email with
nary a click of the mouse!

Just answer these simple questions and you''ll be on your way.

What''s your name? (enter for "Gabriel Landais"):
What''s your (primary) email address? (enter for "glandais at
codelutin.com"):
Ok, your header will look like this:
  From: Gabriel Landais <glandais at codelutin.com>

Do you have any alternate email addresses that also receive email?
If so, enter them now, separated by spaces.
Alternate email addresses:
What file contains your signature? (enter for
"/home/glandais/.signature"):
What editor would you like to use? (enter for "nano"):
Ok, I''ve saved you up a nice lil''
/home/glandais/.sup/config.yaml.

Now, we''ll tell Sup where to find all your email.
Current sources:
*
mbox:/home/glandais/.mozilla-thunderbird/4v86e51w.default/Mail/pops.codelutin.com/Inbox
* sup://drafts
* sup://sent

1. Add a new source.
2. Done adding sources!
Your wish? 2

Ok. The final step is to import all your messages into the Sup index.
Depending on how many messages are in the sources, this could take
quite a while.

Run sup-sync to import all messages now? (enter for "n"): y
Ok, trying to run "/var/lib/gems/1.8/gems/sup-0.5/bin/sup-sync
--all-sources"...
/var/lib/gems/1.8/gems/sup-0.5/bin/sup-config:225: warning: Insecure
world writable dir /opt/java in PATH, mode 040777
[Sun Jun 01 19:14:37 +0200 2008] using character set encoding "UTF-8"
[Sun Jun 01 19:14:37 +0200 2008] optional ''chronic'' library
not found
(run ''gem install chronic'' to install)
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/crypto.rb:18: warning: Insecure
world writable dir /opt/java in PATH, mode 040777
[Sun Jun 01 19:14:37 +0200 2008] crypto: detected gpg binary in /usr/bin/gpg
[Sun Jun 01 19:14:37 +0200 2008] locking /home/glandais/.sup/lock...
[Sun Jun 01 19:14:37 +0200 2008] loading index...
[Sun Jun 01 19:14:37 +0200 2008] loaded index of 0 messages
Scanning
mbox:/home/glandais/.mozilla-thunderbird/4v86e51w.default/Mail/pops.codelutin.com/Inbox...
[Sun Jun 01 19:17:40 +0200 2008] unlocking /home/glandais/.sup/lock...
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/mbox/loader.rb:60:in
`load_header'': failed to allocate memory (NoMemoryError)
        from /var/lib/gems/1.8/gems/sup-0.5/lib/sup/mbox/loader.rb:56:in
`synchronize''
        from /var/lib/gems/1.8/gems/sup-0.5/lib/sup/mbox/loader.rb:56:in
`load_header''
        from /var/lib/gems/1.8/gems/sup-0.5/lib/sup/util.rb:534:in
`send''
        from /var/lib/gems/1.8/gems/sup-0.5/lib/sup/util.rb:534:in
`__pass''
        from /var/lib/gems/1.8/gems/sup-0.5/lib/sup/util.rb:523:in
`method_missing''
        from /var/lib/gems/1.8/gems/sup-0.5/lib/sup/message.rb:62:in
`initialize''
        from /var/lib/gems/1.8/gems/sup-0.5/lib/sup/poll.rb:151:in
`new''
        from /var/lib/gems/1.8/gems/sup-0.5/lib/sup/poll.rb:151:in
`add_messages_from''
         ... 6 levels...
        from /var/lib/gems/1.8/gems/sup-0.5/lib/sup/util.rb:497:in
`method_missing''
        from /var/lib/gems/1.8/gems/sup-0.5/bin/sup-sync:136
        from /var/lib/gems/1.8/gems/sup-0.5/bin/sup-sync:131:in `each''
        from /var/lib/gems/1.8/gems/sup-0.5/bin/sup-sync:131
Rats, that failed. You may have to do it manually.
Try again? (enter for "n"): n
[Sun Jun 01 19:17:47 +0200 2008] loading index...
[Sun Jun 01 19:17:47 +0200 2008] loaded index of 0 messages

Okee doke, you''ve got yourself an index of 0 messages. Looks
like you''re ready to jack in to cyberspace there, cowboy.

Just one last command:

  /var/lib/gems/1.8/gems/sup-0.5/bin/sup

Have fun!
-------------------------------------------------

It uses all my RAM then crash. Same when sup is run :

-------------------------------------------------
[glandais at
server:~/.mozilla-thunderbird/4v86e51w.default/Mail/pops.codelutin.com]
$ /var/lib/gems/1.8/gems/sup-0.5/bin/sup
[Sun Jun 01 19:18:37 +0200 2008] using character set encoding "UTF-8"
[Sun Jun 01 19:18:38 +0200 2008] optional ''chronic'' library
not found
(run ''gem install chronic'' to install)
[Sun Jun 01 19:18:38 +0200 2008] locking /home/glandais/.sup/lock...
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/crypto.rb:18: warning: Insecure
world writable dir /opt/java in PATH, mode 040777
[Sun Jun 01 19:18:38 +0200 2008] crypto: detected gpg binary in /usr/bin/gpg
[Sun Jun 01 19:18:38 +0200 2008] loading index...
[Sun Jun 01 19:18:38 +0200 2008] loaded index of 0 messages
[Sun Jun 01 19:18:38 +0200 2008] starting curses
[Sun Jun 01 19:21:47 +0200 2008] stopped cursing
[Sun Jun 01 19:21:47 +0200 2008] oh crap, an exception
[Sun Jun 01 19:21:47 +0200 2008] unlocking /home/glandais/.sup/lock...
----------------------------------------------------------------
I''m very sorry. It seems that an error occurred in Sup. Please
accept my sincere apologies. If you don''t mind, please send the
contents of sup-exception-log.txt and a brief report of the
circumstances to sup-talk at rubyforge dot orgs so that I might
address this problem. Thank you!

Sincerely,
William
----------------------------------------------------------------
--- NoMemoryError from thread: poll after loading inbox
failed to allocate memory
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/mbox/loader.rb:60:in
`load_header''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/mbox/loader.rb:56:in
`synchronize''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/mbox/loader.rb:56:in
`load_header''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/util.rb:534:in `send''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/util.rb:534:in `__pass''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/util.rb:523:in `method_missing''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/message.rb:62:in `initialize''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/poll.rb:151:in `new''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/poll.rb:151:in
`add_messages_from''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/source.rb:98:in `each''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/util.rb:534:in `send''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/util.rb:534:in `__pass''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/util.rb:523:in `method_missing''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/poll.rb:141:in
`add_messages_from''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/poll.rb:98:in `do_poll''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/poll.rb:86:in `each''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/poll.rb:86:in `do_poll''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/poll.rb:85:in `synchronize''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/poll.rb:85:in `do_poll''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/util.rb:497:in `send''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/util.rb:497:in `method_missing''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/modes/poll-mode.rb:17:in `poll''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/poll.rb:53:in `poll''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/util.rb:497:in `send''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/util.rb:497:in `method_missing''
/var/lib/gems/1.8/gems/sup-0.5/bin/sup:210
/var/lib/gems/1.8/gems/sup-0.5/lib/sup.rb:60:in `reporting_thread''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup.rb:58:in `initialize''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup.rb:58:in `new''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup.rb:58:in `reporting_thread''
/var/lib/gems/1.8/gems/sup-0.5/bin/sup:210
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/modes/thread-index-mode.rb:536:in
`call''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/modes/thread-index-mode.rb:536:in
`__unprotected_load_threads''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/modes/thread-index-mode.rb:478:in
`call''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/modes/thread-index-mode.rb:478:in
`load_n_threads_background''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup.rb:60:in `reporting_thread''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup.rb:58:in `initialize''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup.rb:58:in `new''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup.rb:58:in `reporting_thread''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/modes/thread-index-mode.rb:476:in
`load_n_threads_background''
/var/lib/gems/1.8/gems/sup-0.5/lib/sup/modes/thread-index-mode.rb:546:in
`__unprotected_load_threads''
(eval):12:in `load_threads''
/var/lib/gems/1.8/gems/sup-0.5/bin/sup:210
--- SystemExit from thread: main
failed to allocate memory
/var/lib/gems/1.8/gems/sup-0.5/lib/sup.rb:64:in `method_missing''
/var/lib/gems/1.8/gems/sup-0.5/bin/sup:226
-------------------------------------------------

Thanks

Gabriel

William Morgan

2008-Jun-03 02:34 UTC

head link

[sup-talk] [bug] NoMemoryError on huge mbox

Reformatted excerpts from Gabriel Landais''s message of
2008-06-01:>  I''m not able to load my Thunderbird mbox into sup. Added my Inbox
> mbox with sup-add/sup-config :
Interesting. This error seems to occur during a string construction.
Others have reported weird problems with Thunderbird mbox files before,
and I now suspect there''s something screwy about their mbox format.
Like, they use \r instead of \n for linebreaks. Which wouldn''t make any
fucking sense at all, but who knows.

Can you post the output of a 

  ruby -e ''p File.open("Inbox").read(1000)''

so we can take a look at what these guys are doing? (Or whatever
Thunderbird file you want instead of "Inbox".)

Thanks!
-- 
William <wmorgan-sup at masanjin.net>

Gabriel Landais

2008-Jun-03 08:09 UTC

head link

[sup-talk] [bug] NoMemoryError on huge mbox

On Tue, Jun 3, 2008 at 4:34 AM, William Morgan <wmorgan-sup at
masanjin.net> wrote:> Reformatted excerpts from Gabriel Landais''s message of 2008-06-01:
>>  I''m not able to load my Thunderbird mbox into sup. Added my
Inbox
>> mbox with sup-add/sup-config :
>
> Interesting. This error seems to occur during a string construction.
> Others have reported weird problems with Thunderbird mbox files before,
> and I now suspect there''s something screwy about their mbox
format.
> Like, they use \r instead of \n for linebreaks. Which wouldn''t
make any
> fucking sense at all, but who knows.
>
> Can you post the output of a
>
>  ruby -e ''p File.open("Inbox").read(1000)''
>
> so we can take a look at what these guys are doing? (Or whatever
> Thunderbird file you want instead of "Inbox".)
>
Here it is (for 10, same thing for 100 000!) :

[glandais at
server:~/.mozilla-thunderbird/4v86e51w.default/Mail/pops.codelutin.com]
$ ruby -e ''p File.open("Inbox").read(10)''
"\000\000\000\000\000\000\000\000\000\000"

Perhaps I just shouldn''t use that mbox. I never understood how
messages are saved in these directories. My folder looks like that :

[glandais at server:~/.mozilla-thunderbird/4v86e51w.default/Mail] $ durep
[ /home/glandais/.mozilla-thunderbird/4v86e51w.default/Mail/  520.3M ]
 276.2M [%%%%%%%%%%%%%%%               ]  1  53.08% Local Folders/
     252.9M [%%%%%%%%%%%%%%%%%%%%%%%%%%%   ]  5  91.59% Inbox.sbd/
          207.8M [%%%%%%%%%%%%%%%%%%%%%%%%      ]  5  82.16% Listes.sbd/
               202.2M [%%%%%%%%%%%%%%%%%%%%%%%%%%%%% ]  0  97.30% cl.sbd/
                 5.5M [                              ]  0   2.63%
decathlon.sbd/
                60.0K [                              ]  0   0.03% le.sbd/
                36.0K [                              ] 14   0.02% [FILES]
                28.0K [                              ]  1   0.01% bricks.sbd/
                     12.0K [%%%%%%%%%%%%                  ]  0
42.86% tapestry.sbd/
                      8.0K [%%%%%%%%                      ]  4  28.57% [FILES]
                12.0K [                              ]  0   0.01% lucene.sbd/
           31.3M [%%%                           ] 16  12.36% [FILES]
            7.1M [                              ]  0   2.81% Projets.sbd/
            4.5M [                              ]  0   1.77% formations.sbd/
            2.1M [                              ]  0   0.84%
simexplorer-si.sbd/
          124.0K [                              ]  0   0.05% sandra.sbd/
      23.2M [%%                            ] 15   8.41% [FILES]
 244.1M [%%%%%%%%%%%%%%                ]  0  46.92% pops.codelutin.com/

It was just to give a try to Sup on Thunderbird, I''ll try another mbox.

Thanks

Gabriel

William Morgan

2008-Jun-04 02:42 UTC

head link

[sup-talk] [bug] NoMemoryError on huge mbox

Reformatted excerpts from Gabriel Landais''s message of
2008-06-03:> Here it is (for 10, same thing for 100 000!) :
> 
> [glandais at
server:~/.mozilla-thunderbird/4v86e51w.default/Mail/pops.codelutin.com
> ]
> $ ruby -e ''p File.open("Inbox").read(10)''
> "\000\000\000\000\000\000\000\000\000\000"
Ok, well, you can''t really fault Sup for that one! It''s just
reading
until it gets a newline. :)

Still, why would Thunderbird produce a file full of zeros?
> Perhaps I just shouldn''t use that mbox. I never understood how
> messages are saved in these directories.
Here''s what the Thunderbird FAQ claims:

  Your mail files are inside your profile (see the Profile Folder), in
  the Mail and (if you use IMAP) ImapMail folders. Each mail folder
  (Inbox, Sent, etc.) is stored as two files ? one with no extension
  (e.g. INBOX), which is the mail file itself (in "mbox" format), and
  one with an .msf extension (e.g. INBOX.msf), which is the index (Mail
  Summary File) to the mail file. Tell the other program to import mail
  from the file with no extension.

  http://www.mozilla.org/support/thunderbird/faq

Which implies to me that Inbox was the correct file. But apparently not.
Are there any other files that look like normal mbox files?
-- 
William <wmorgan-sup at masanjin.net>

sup talk - Jun 2008 - [bug] NoMemoryError on huge mbox

[sup-talk] [bug] NoMemoryError on huge mbox

[sup-talk] [bug] NoMemoryError on huge mbox

[sup-talk] [bug] NoMemoryError on huge mbox

[sup-talk] [bug] NoMemoryError on huge mbox