Re: [Evolution-hackers] Loading really large E-mails on devices with not enough Vm

[Philip Van Hoof <spam pvanhoof be>]

Hey Jeffrey,

I did some experimenting and after this it seems to work:

http://tinymail.org/trac/tinymail/changeset/3462

I had to get the value of folder_tell at the exact location where the
state is at CAMEL_MIME_PARSER_STATE_HEADER.

Then it worked.

I tested this against for example a testing E-mail that is floating
around the Lemonade test servers of 32 MB, various simpler E-mails and
one of 7 MB. Both big E-mails mostly had image attachments.

Tinymail has a mimepart viewer that uses a pixbuf loader, and it
succeeded just fine in loading the images.

I have a message open that is "40246320" bytes in size, this is my VmRSS
for Tinymail's demoui. 

VmRSS:     16428 kB

Those 16Mb is probably data in the GtkPixbuf and the summary.

lemonade.andrew.cmu.edu:143, testuser1, pass1 and pick the largest mail
on that server (~40 MB).


On Tue, 2008-03-04 at 18:40 +0100, Philip Van Hoof wrote:
> On Sat, 2008-01-26 at 23:22 -0500, Jeffrey Stedfast wrote:
> > Something like the attached patch might work, tho it is untested.
> 
> I had to change 
> 
> 	else if (!CAMEL_IS_SEEKABLE_SUBSTREAM (stream))
> 
> into
> 
> 	else if (!CAMEL_IS_SEEKABLE_STREAM (stream))
> 
> I don't know why you where testing for substream, as substream provides
> no extra functionality that seems to be related here ...
> 
> > So my guess is that this will break the parser :(
> > 
> > It might break in the stream case as well, you'd have to follow the code
> > paths a bit to know for sure. For instance, even if creating the
> > seekable substream doesn't perform an underlying seek on the original
> > stream, setting it in a data wrapper might call camel_stream_reset()
> > which /might/ do an lseek() on the source fs stream.
> 
> The problem with the patch is that it makes each MIME part's data start
> at the headers, in stead of at the actual content.
> 
> I tried determining the "start" right after the first call to
> camel_mime_parser_step but that just resulted in start == end.
> 
> 
> > Not an insurmountable problem to solve, but it does make things a little
> > more difficult and possibly touchy.
> 
> > 
> > 
> > On Sat, 2008-01-26 at 22:48 -0500, Jeffrey Stedfast wrote:
> > > On Sat, 2008-01-26 at 22:12 -0500, Jeffrey Stedfast wrote:
> > > > On Sat, 2008-01-26 at 13:44 +0100, Philip Van Hoof wrote:
> > > > > This is what happens if you try to open a truly large E-mail on a device
> > > > > that has not as much memory available:
> > > > > 
> > > > > Is there something we can do about this? Can we change the MIME parsing
> > > > > algorithm to be less memory demanding for example?
> > > > > 
> > > > > Note that GArray is not really very sparse with memory once you start
> > > > > having a really large array. Perhaps we can in stead change this to a
> > > > > normal pointer array of a fixed size (do we know the size before we
> > > > > start parsing, so that we can allocate an exact size in stead, perhaps?)
> > > > 
> > > > eh, why would you change it to a GPtrArray? It doesn't hold pointers, it
> > > > holds message part content.
> > > > 
> > > > Unfortunately we don't know the size ahead of time.
> > > > 
> > > > I suppose you could use a custom byte array allocator so that you can
> > > > force it to grow by larger chunks or something, dunno.
> > > >
> > > >
> > > > The way GMime handles this is by not loading content into RAM, but 
> > > > that may be harder to do with Camel, especially in the mbox case.
> > > 
> > > er, I should probably explain this:
> > > 
> > > - writing the code should be relatively easy to do, but in the mbox
> > > case, the mbox may end up getting expunged or rewritten for some other
> > > reason which may cause problems, not sure how that would work.
> > > 
> > > I think in Maildir, as long as the fd remains open, the file won't
> > > actually disappear after an unlink() until the fd gets closed, so that
> > > might work out ok assuming you can spare the fd (which might be the
> > > other problem with Evolution?).
> > > 
> > > Jeff
> > > 
> > > 
> > > _______________________________________________
> > > Evolution-hackers mailing list
> > > Evolution-hackers gnome org
> > > http://mail.gnome.org/mailman/listinfo/evolution-hackers
> > _______________________________________________
> > Evolution-hackers mailing list
> > Evolution-hackers gnome org
> > http://mail.gnome.org/mailman/listinfo/evolution-hackers
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be