Recently I''ve been seeing this crash[1] pretty consistently on the next branch with the Xapian backend. All I need to do to reproduce the crash is move to the last thread in the thread-index-mode and attempt to move to the next thread. I can also produce a very similar crash[2] by attempting to load all threads. Thanks, - Ben [1] Crash while scrolling: --- RuntimeError from thread: load threads for thread-index-mode wrong id called on nil /opt/exp/sup/lib/sup.rb:17:in `id'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'' /usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `sort_by'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `each'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `sort_by'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `synchronize'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `update'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:637:in `__unprotected_load_n_threads'' (eval):12:in `load_n_threads'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:609:in `load_n_threads_background'' /opt/exp/sup/lib/sup.rb:77:in `reporting_thread'' /opt/exp/sup/lib/sup.rb:75:in `initialize'' /opt/exp/sup/lib/sup.rb:75:in `new'' /opt/exp/sup/lib/sup.rb:75:in `reporting_thread'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:608:in `load_n_threads_background'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:679:in `__unprotected_load_threads'' (eval):12:in `load_threads'' /opt/exp/sup/lib/sup/mode.rb:51:in `send'' /opt/exp/sup/lib/sup/mode.rb:51:in `handle_input'' /opt/exp/sup/lib/sup/buffer.rb:267:in `handle_input'' /usr/bin/sup:238 [2] Crash after attempting to load all threads --- RuntimeError from thread: load threads for thread-index-mode wrong id called on nil /opt/exp/sup/lib/sup.rb:17:in `id'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'' /usr/lib/ruby/1.8/rubygems/custom_require.rb:31:in `sort_by'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `each'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `sort_by'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:225:in `update'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `synchronize'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:223:in `update'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:628:in `__unprotected_load_n_threads'' /opt/exp/sup/lib/sup/thread.rb:334:in `load_n_threads'' /opt/exp/sup/lib/sup/xapian_index.rb:151:in `each_id_by_date'' /opt/exp/sup/lib/sup/xapian_index.rb:144:in `each_id'' /opt/exp/sup/lib/sup/xapian_index.rb:144:in `each'' /opt/exp/sup/lib/sup/xapian_index.rb:144:in `each_id'' /opt/exp/sup/lib/sup/xapian_index.rb:151:in `each_id_by_date'' /opt/exp/sup/lib/sup/thread.rb:328:in `load_n_threads'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:625:in `__unprotected_load_n_threads'' (eval):12:in `load_n_threads'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:609:in `load_n_threads_background'' /opt/exp/sup/lib/sup.rb:77:in `reporting_thread'' /opt/exp/sup/lib/sup.rb:75:in `initialize'' /opt/exp/sup/lib/sup.rb:75:in `new'' /opt/exp/sup/lib/sup.rb:75:in `reporting_thread'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:608:in `load_n_threads_background'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:679:in `__unprotected_load_threads'' (eval):12:in `load_threads'' /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:658:in `load_all_threads'' /opt/exp/sup/lib/sup/mode.rb:51:in `send'' /opt/exp/sup/lib/sup/mode.rb:51:in `handle_input'' /opt/exp/sup/lib/sup/buffer.rb:267:in `handle_input'' /usr/bin/sup:238
Reformatted excerpts from Ben Gamari''s message of 2009-09-11:> Recently I''ve been seeing this crash[1] pretty consistently on the next > branch with the Xapian backend.Is your Xapian index somewhat old? There have been index format changes that have caused this type of thing recently. You might try rebuilding it. -- William <wmorgan-sup at masanjin.net>
On Sat, Sep 12, 2009 at 09:35:21AM -0700, William Morgan wrote:> Reformatted excerpts from Ben Gamari''s message of 2009-09-11: > > Recently I''ve been seeing this crash[1] pretty consistently on the next > > branch with the Xapian backend. > > Is your Xapian index somewhat old? There have been index format changes > that have caused this type of thing recently. You might try rebuilding > it.I wish I could say it was unfortunately I just rebuilt it two days ago, suspecting that might be part of the issue. I''ll try again, just to make sure but I''m fairly certain the index is up-to-date. - Ben
On Sat, Sep 12, 2009 at 09:35:21AM -0700, William Morgan wrote:> Reformatted excerpts from Ben Gamari''s message of 2009-09-11: > > Recently I''ve been seeing this crash[1] pretty consistently on the next > > branch with the Xapian backend. > > Is your Xapian index somewhat old? There have been index format changes > that have caused this type of thing recently. You might try rebuilding > it.Well, after several attempts at rebuilding my index, I finally got lucky and sup-sync ran to completion. Unfortunately, sup now fails to even start, failing out with the following exception while loading threads for inbox-mode, $ sup --- SystemExit from thread: main Thread#join: deadlock 0x7f592eec4d78 - mutual join(0x7f592f01d6e8) /opt/exp/sup/lib/sup/message.rb:240:in `select'' /opt/exp/sup/lib/sup/buffer.rb:35:in `nonblocking_getch'' /usr/bin/sup:213 However, at this point during debugging I happened to pipe stderr to a file and accidentally found that most of the backtrace was actually being hidden by curses. Examining the full output on stderr reveals, /usr/bin/sup:213(eval):3:in `synchronize'': Thread#join: deadlock 0x7f4d5da565c0 - mutual join(0x7f4d5daf8be0) (ThreadError) from (eval):3:in `raw_header'' from /opt/exp/sup/lib/sup/imap.rb:98:in `load_header'' from /opt/exp/sup/lib/sup/util.rb:560:in `send'' from /opt/exp/sup/lib/sup/util.rb:560:in `__pass'' from /opt/exp/sup/lib/sup/util.rb:547:in `method_missing'' from /opt/exp/sup/lib/sup/message.rb:238:in `load_from_source!'' from /opt/exp/sup/lib/sup/message.rb:219:in `chunks'' from /opt/exp/sup/lib/sup/message.rb:164:in `snippet'' from /opt/exp/sup/lib/sup/thread.rb:91:in `snippet'' from /opt/exp/sup/lib/sup/imap.rb:259:in `select'' from /opt/exp/sup/lib/sup/thread.rb:68:in `each'' from /opt/exp/sup/lib/sup/thread.rb:168:in `each_with_stuff'' from /opt/exp/sup/lib/sup/thread.rb:67:in `each'' from /opt/exp/sup/lib/sup/thread.rb:91:in `select'' from /opt/exp/sup/lib/sup/thread.rb:91:in `snippet'' from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:840:in `text_for_thread_at'' from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:758:in `regen_text'' from /opt/exp/sup/lib/sup/util.rb:364:in `map_with_index'' from /opt/exp/sup/lib/sup/imap.rb:259:in `each_with_index'' from /opt/exp/sup/lib/sup/util.rb:364:in `each'' from /opt/exp/sup/lib/sup/util.rb:364:in `each_with_index'' from /opt/exp/sup/lib/sup/util.rb:364:in `map_with_index'' from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:758:in `regen_text'' from /opt/exp/sup/lib/sup/modes/thread-index-mode.rb:687:in `resize'' from /opt/exp/sup/lib/sup/buffer.rb:88:in `resize'' from /opt/exp/sup/lib/sup/buffer.rb:329:in `draw_screen'' from /opt/exp/sup/lib/sup/buffer.rb:745:in `clear'' from /opt/exp/sup/lib/sup/util.rb:520:in `send'' from /opt/exp/sup/lib/sup/util.rb:520:in `method_missing'' from /opt/exp/sup/lib/sup/imap.rb:283:in `shutup'' from /opt/exp/sup/lib/sup/imap.rb:270:in `unsafe_connect'' from /opt/exp/sup/lib/sup/imap.rb:244:in `initialize'' from /opt/exp/sup/lib/sup/imap.rb:244:in `new'' from /opt/exp/sup/lib/sup/imap.rb:244:in `unsafe_connect'' from /opt/exp/sup/lib/sup/imap.rb:331:in `safely'' from /opt/exp/sup/lib/sup/imap.rb:148:in `unsynchronized_connect'' from (eval):3:in `connect'' from (eval):3:in `synchronize'' from (eval):3:in `connect'' from /opt/exp/sup/lib/sup/util.rb:560:in `send'' from /opt/exp/sup/lib/sup/util.rb:560:in `__pass'' from /opt/exp/sup/lib/sup/util.rb:547:in `method_missing'' from /usr/bin/sup:189 from /opt/exp/sup/lib/sup.rb:77:in `reporting_thread'' from /opt/exp/sup/lib/sup.rb:75:in `initialize'' from /opt/exp/sup/lib/sup.rb:75:in `new'' from /opt/exp/sup/lib/sup.rb:75:in `reporting_thread'' from /usr/bin/sup:187 from /usr/bin/sup:185:in `each'' from /usr/bin/sup:185 [Wed Sep 16 13:01:11 -0400 2009] ERROR: oh crap, an exception ---------------------------------------------------------------- I''m very sorry. It seems that an error occurred in Sup. Please accept my sincere apologies. If you don''t mind, please send the contents of ~/.sup/exception-log.txt and a brief report of the circumstances to sup-talk at rubyforge dot orgs so that I might address this problem. Thank you! Sincerely, William ---------------------------------------------------------------- with the first stacktrace following this on stdout. With this information it looks like this bug should become much more debuggable. Being in the imap module, it seems likely that it is my gmail source that is causing the failure. Unfortunately I have no way to verify that the problem is limited to the gmail source as sup raises as exception when it encounters an unknown source, precluding the option of simply commenting out the source in sources.yaml. Anyways, this is the current status of things on my machine. William, do you have any ideas what might cause such a backtrace. At this point classes have started and I really have no more time to devote to getting sup working. If there isn''t a fairly simple solution here I guess I''ll just need to stick with mutt (ugh). Anyways, thanks for your work. Cheers, - Ben
Sorry it''s taken me so long to get back to this. Reformatted excerpts from Ben Gamari''s message of 2009-09-16:> Thread#join: deadlock 0x7f592eec4d78 - mutual join(0x7f592f01d6e8) > /opt/exp/sup/lib/sup/message.rb:240:in `select'' > /opt/exp/sup/lib/sup/buffer.rb:35:in `nonblocking_getch'' > /usr/bin/sup:213 > > However, at this point during debugging I happened to pipe stderr to a > file and accidentally found that most of the backtrace was actually > being hidden by curses. Examining the full output on stderr reveals, > > /usr/bin/sup:213(eval):3:in `synchronize'': Thread#join: deadlock > 0x7f4d5da565c0 - mutual join(0x7f4d5daf8be0) (ThreadError) > from (eval):3:in `raw_header'' > from /opt/exp/sup/lib/sup/imap.rb:98:in `load_header''I definitely am not sure why there''s a deadlock happening, but I am guessing, based on the line numbers in that first exception, that it''s being triggered by an underlying problem with the source. If you run sup with -n, does that change anything? (Or produce a better exception?)> Anyways, this is the current status of things on my machine. William, do > you have any ideas what might cause such a backtrace. At this point > classes have started and I really have no more time to devote to > getting sup working. If there isn''t a fairly simple solution here I guess > I''ll just need to stick with mutt (ugh). Anyways, thanks for your work.Well, there''s a workaround, which is to switch over to an offlineimap setup where your Gmail is mirrored locally and Sup reads the mirror (which is what you want anyways if you''re serious about using Sup with an IMAP client). I don''t know what''s causing the deadlock, but I will try and reproduce it on my end. -- William <wmorgan-sup at masanjin.net>
Excerpts from William Morgan''s message of Sat Sep 26 10:31:56 -0400 2009:> Sorry it''s taken me so long to get back to this.Quite alright. I know we''re all busy.> > I definitely am not sure why there''s a deadlock happening, but I am > guessing, based on the line numbers in that first exception, that it''s > being triggered by an underlying problem with the source. If you run > sup with -n, does that change anything? (Or produce a better exception?)Well, things definitely failed a bit differently. I tried this a few times and after seeing no real improvement, I moved on to your suggestion below.> > > Anyways, this is the current status of things on my machine. William, do > > you have any ideas what might cause such a backtrace. At this point > > classes have started and I really have no more time to devote to > > getting sup working. If there isn''t a fairly simple solution here I guess > > I''ll just need to stick with mutt (ugh). Anyways, thanks for your work. > > Well, there''s a workaround, which is to switch over to an offlineimap > setup where your Gmail is mirrored locally and Sup reads the mirror > (which is what you want anyways if you''re serious about using Sup with > an IMAP client). I don''t know what''s causing the deadlock, but I will > try and reproduce it on my end.This is definitely what I should have done from the beginning. Given that sup seems to work fine after removing my imap sources, it seems that that might be the source of the above deadlock. I think I''ll just stick with offlineimap for now. This does raise one important question, however. It seems that offlineimap will delete messages if they are found to have been deleted in the remote respository. Is there any way that you know of to disable this behavior, such that it will only drop new messages into the local repository, thus making it somewhat compatible with sup''s add-only source requirement? It is awfully nice to be able to use GMail''s web interface when I''m away from my computer, but needing to re-sync sup afterwards becomes quite tiresome. This actually brings up a larger question. How difficult would it be to relax sup''s assumption that sources are add-only? This seems like a horribly restriction to have in an email program. I understand that in the case of sources such as imap where there is no globally unique message identifier trying to track message changes could be quite difficult, but for local sources (especially maildirs) it seems like this should be quite possible. Anyways, thanks a ton for your work. Now since I finally have sup up and running reliably, it''s been a joy to use. Leaps and bounds above my former mutt-based system. - Ben
Reformatted excerpts from Ben Gamari''s message of 2009-09-16:> Well, after several attempts at rebuilding my index, I finally got > lucky and sup-sync ran to completion. Unfortunately, sup now fails to > even start, failing out with the following exception while loading > threads for inbox-mode,Hey, I just started seeing this too, Xapian only. I think I''ve fixed in master. -- William <wmorgan-sup at masanjin.net>
Reformatted excerpts from Ben Gamari''s message of 2009-09-28:> This does raise one important question, however. It seems that > offlineimap will delete messages if they are found to have been > deleted in the remote respository. Is there any way that you know of > to disable this behavior, such that it will only drop new messages > into the local repository, thus making it somewhat compatible with > sup''s add-only source requirement?Maybe someone who uses offlineimap can comment on this. In the worst case, I''m sure the patch wouldn''t be too hard.> This actually brings up a larger question. How difficult would it be > to relax sup''s assumption that sources are add-only?It''s not difficult per se, it just requires scanning over the entire source, which is slow. Removing this assumption would be tantamount to running sup-sync -c every time you start up sup. Here''s the idea: scanning over a mailstore is slow. Much of this slowness is due to Ruby. So let''s rewrite this code in C. Then we would have something as fast as, say, Mutt. But Mutt bogs down on my mbox file because it''s too big. So my *only* reasonable choice with a large mailstore is Sup and the assumption that the source is add only. -- William <wmorgan-sup at masanjin.net>
Excerpts from William Morgan''s message of Thu Oct 01 09:43:48 -0400 2009:> Reformatted excerpts from Ben Gamari''s message of 2009-09-28: > > This actually brings up a larger question. How difficult would it be > > to relax sup''s assumption that sources are add-only? > > It''s not difficult per se, it just requires scanning over the entire > source, which is slow. Removing this assumption would be tantamount to > running sup-sync -c every time you start up sup. > > Here''s the idea: scanning over a mailstore is slow. Much of this > slowness is due to Ruby. So let''s rewrite this code in C. Then we would > have something as fast as, say, Mutt. But Mutt bogs down on my mbox file > because it''s too big. So my *only* reasonable choice with a large > mailstore is Sup and the assumption that the source is add only.It seems that C would definitely be a good start (or perhaps C++ would be a better idea as that is the language in which Xapian is written). However, I think one of the real issues is the exclusive nature of index access. In fact, this is one of my primary gripes with the sup workflow. After processing a large number of messages, the write-out time can be quite substantial upon killing the buffer. This can be a noticeable interruption to workflow. It seems to me that index access should be asynchronous at least. If this were the case, then we could get support for mutable sources for free, as we could synchronize against sources without interrupting workflow (although keeping the view in sync with the backend would be a bit tricky). As an aside, it would be quite nice if one could run multiple simultaneous instances of sup. It seems that if one only held write access to the index during writes (is this the case presently?), there should be nothing preventing this from being possible. Correct me if I''m wrong in any part of my above assessment. Hope things are going well, - Ben
Reformatted excerpts from Ben Gamari''s message of 2009-10-01:> It seems that C would definitely be a good start (or perhaps C++ would > be a better idea as that is the language in which Xapian is written).I was proposing that as a strawman argument. C does not solve my problem.> However, I think one of the real issues is the exclusive nature of > index access. In fact, this is one of my primary gripes with the sup > workflow. After processing a large number of messages, the write-out > time can be quite substantial upon killing the buffer.It is possible to address this in a variety of ways, many of which have been discussed over the years, but yes, I agree that a delay is nonideal.> If this were the case, then we could get support for mutable sources > for free, as we could synchronize against sources without interrupting > workflow (although keeping the view in sync with the backend would be > a bit tricky).Making message state saving fast, or backgrounded, is a different beast from scanning over a mailstore. I have been working on a Sup server for quite some time that would address many of these problems, but progress is slow.> As an aside, it would be quite nice if one could run multiple > simultaneous instances of sup. It seems that if one only held write > access to the index during writes (is this the case presently?), there > should be nothing preventing this from being possible.It might be possible to do this with Xapian, especially if there were no expectation of updates being transmitted across processes. With Ferret, if it is possible, it''s only with a tremendous amount of effort. -- William <wmorgan-sup at masanjin.net>
Excerpts from Ben Gamari''s message of Thu Oct 01 10:44:19 -0700 2009:> Excerpts from William Morgan''s message of Thu Oct 01 09:43:48 -0400 2009: > > Here''s the idea: scanning over a mailstore is slow. Much of this > > slowness is due to Ruby. So let''s rewrite this code in C. Then we would > > have something as fast as, say, Mutt. But Mutt bogs down on my mbox file > > because it''s too big. So my *only* reasonable choice with a large > > mailstore is Sup and the assumption that the source is add only. > > It seems that C would definitely be a good start (or perhaps C++ would > be a better idea as that is the language in which Xapian is written).I''ve started some experiments along these lines, (basically just writing C code (with C++ where necessary) using the Xapian tutorial and little peeks into sup/xapian-index.rb to get the right prefix values, etc.).> However, I think one of the real issues is the exclusive nature of index > access. In fact, this is one of my primary gripes with the sup > workflow. After processing a large number of messages, the write-out > time can be quite substantial upon killing the buffer. This can be a > noticeable interruption to workflow. It seems to me that index access > should be asynchronous at least.What I''m hoping to end up with is a C library that provides asynchronous access to a sup-compatible index. I''ll keep you posted if I make any actual progress on this front. -Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: not available URL: <http://rubyforge.org/pipermail/sup-talk/attachments/20091001/89e0a628/attachment.bin>
Excerpts from William Morgan''s message of Thu Oct 01 14:31:20 -0400 2009:> Reformatted excerpts from Ben Gamari''s message of 2009-10-01: > > It seems that C would definitely be a good start (or perhaps C++ would > > be a better idea as that is the language in which Xapian is written). > > I was proposing that as a strawman argument. C does not solve my > problem.Certainly, you will never be able to write a message indexer fast enough to index a source instantly. That is why we have an index to begin with. That being said, I think we can do substantially better than we are currently doing in a lower-level language. With mutt, I can come close to saturating my drive bandwidth while loading a folder. While synchronizing in sup, I am lucky to get a few hundred kilobytes/second. Certainly a large amount of that difference has to do with the amount of processing done by each, but even adjusting for this it seems to me that we should still be I/O bound (or at least close to it).> > > However, I think one of the real issues is the exclusive nature of > > index access. In fact, this is one of my primary gripes with the sup > > workflow. After processing a large number of messages, the write-out > > time can be quite substantial upon killing the buffer. > > It is possible to address this in a variety of ways, many of which have > been discussed over the years, but yes, I agree that a delay is > nonideal.Glad we agree.> > Making message state saving fast, or backgrounded, is a different beast > from scanning over a mailstore.If we are unable to update the index while the client is active, rescanning sources would destroy usability. I would argue that asynchronous writeout is very much a prerequisite for mutable sources.> > I have been working on a Sup server for quite some time that would > address many of these problems, but progress is slow.This was largely what I had in mind. It seems like moving index manipulation out-of-process might be best, ultimately.> > > As an aside, it would be quite nice if one could run multiple > > simultaneous instances of sup. It seems that if one only held write > > access to the index during writes (is this the case presently?), there > > should be nothing preventing this from being possible. > > It might be possible to do this with Xapian, especially if there were no > expectation of updates being transmitted across processes.I think initially no updates between processes would be fine.> With Ferret, > if it is possible, it''s only with a tremendous amount of effort.Is ferret even going to be supported after the Xapian backend stabilizes? Thanks for the comments and sup. - Ben
On Fri, Sep 11, 2009 at 6:58 PM, Ben Gamari <bgamari.foss at gmail.com> wrote:> I can also produce a very similar crash[2] by attempting to load all > threads. Thanks,I get the same thing when loading all the threads. Index has just been rebuilt, and I''m using an mbox file. -- Guillaume
Reformatted excerpts from Ben Gamari''s message of 2009-10-01:> Is ferret even going to be supported after the Xapian backend > stabilizes?No, I''m going to drop it at some point. I barely have enough time or energy for Sup''s current set of problems as is. :) -- William <wmorgan-sup at masanjin.net>