RE: Upcoming GMime 3.0 changes

[Jeffrey Stedfast <jestedfa microsoft com>]

Hey Albrecht, thanks for replying!

Answers below...

> -----Original Message-----
> From: Albrecht Dreß [mailto:albrecht dress arcor de]
> Sent: Tuesday, March 14, 2017 2:14 PM
> To: Jeffrey Stedfast <jestedfa microsoft com>
> Cc: balsa-list gnome org
> Subject: Re: Upcoming GMime 3.0 changes
>
> Hi Jeff:
>
> Thank you so much for this information!

You're welcome!

> I believe your new lib will simplify many parts in Balsa (but we probably still
> need to support the old GMime 2.6 based stuff until all LTS distos switched to
> the new one).
>
> Some comments below...
>
> Am 13.03.17 23:38 schrieb(en) Jeffrey Stedfast via balsa-list:
> > 2. Next up is the replacement of the old custom GnuPG back-end with
> GpgMe. Also included with this change is full support for S/MIME via both
> multipart/signed and application/pkcs7-mime content-types using gpgsm (via
> GpgMe).
>
> That's cool!  You may recall that I never used your GMime GnuPG
> implementation in Balsa, but implemented my own solution on top of
> GpgME more than 10 years (IIRC) ago, which is working just fine afaict... ;-)

Yea, I do remember this... I think it's one of the reasons that I made the GMime API's return more complete 
info about signers rather than just good/bad/etc for 2.6.

In the end, it just makes sense to use gpgme now due to the fact that it is now installed pretty much 
everywhere, even my Mac (via homebrew)!

Hopefully you'll be able to reduce your usage of gpgme directly now that GMime uses gpgme 😊

> > During this change, I also took the liberty of simplifying the crypto API's a bit
> and so I was able to make it such that g_mime_multipart_signed_verify(), for
> example, no longer requires you to pass it a GMimeCryptoContext. Instead,
> GMime parses the Content-Type's protocol parameter and uses that to
> instantiate the correct crypto context (GMimeGpgContext for PGP and
> GMimePkcs7Context for S/MIME).
>
> Also very nice!
>
> > Many of the various state properties have been replaced with bitflags that
> can be passed to encrypt() and decrypt(). The sign() method now also takes a
> detach argument (might make this into a bitflag instead?) in order to support
> encapsulated signing.
>
> Do you have some more documentation about this?  It would be interesting
> to know how this approach could fit with the current GpgME-based
> implementation in Balsa.

When I was moving to gpgme, I noticed that the always_trust flag was one of the gpgme_encrypt_flags_t, so it 
made sense to use a similar mapping (GMimeEncryptFlags). Of course, ALWAYS_TRUST is currently the only flag 
for encrypting while EXPORT_SESSION_KEY is the only GMimeDecryptFlags flag. Currently GMimeVerifyFlags has no 
enum values available (the only one that would have made sense from the old GMime API would be 
auto-key-retrieve but gpgme doesn't seem to have a mapping for that except, perhaps, if I use the gpgme API 
for manipulating the configuration, but even then I'm not sure it'd work).

> > 4. New in GMime 3.0 is the GMimeParserOptions struct which can be
> passed to GMimeParser and other parser functions exposed in the lower-
> level API's. This structure helps define how strict/forgiving the various
> parsing routines should be with the input. This replaces the need for
> g_mime_init()'s flags so you can change these settings on the fly now.
>
> It would be *really* cool if the parser could optionally collect and provide
> information about violations of the various rfc's encountered during the
> parsing process.  Maybe something like a linked list containing the stream
> offset, a unique code and a comment, or something similar, with an upper
> limit of items collected.  This might provide valuable information for spam
> and/or malware checking (and would be *a lot* more performant than the
> perl/python based parsers typically used, I guess).  Not related to Balsa, but
> to an other project I'm working on...

Yea, I've thought about this off and on for at least the past decade... I just haven't done it because it'd 
make the API's more awkward with another argument. Plus GMime is mostly focused on mail clients that need to 
do their best to interoperate with everything as opposed to spam traps which might want to use that info for 
holding mail back.

The other problem is that GMime's GMimeParser really only parses the stream into field/value header pairs and 
individual mime-parts. Parsing of the header values doesn't occur until later, inside of the GMimeObject 
implementation and so there's no way to pass that info back to the GMimeParser to give back to its caller.

I sort of did this with MimeKit (my C# re-implementation fop GMime) using exceptions. E.g. if you set the 
ParserOptions to use strict(er) rules, it would throw an exception giving you the offset of the beginning of 
the token, the offset of the error byte, and a description of the error, but C doesn't have exceptions...

> > 6. Brand new rfc822 address parser which is more tolerant than the
> previous generation parser. What's not to love?
>
> Same as above - please provide more information about the rfc violations...

The main thing I did was to try and work around more common brokenness, such as having addresses like:

To: Warren Worthington, Jr. <warren worthington com>

(notice the unquoted , and .)

Also worked around are things like the ones mentioned in https://tools.ietf.org/html/rfc7103#section-7

I don't think I managed to support *all* of the broken address forms listed in section 7, but I managed to 
add support for a number of them.

> > 7. And finally we get to a nifty feature that I just hacked up while waiting
> for some other code to compile (hey, it takes an hour to compile... I needed
> something to do!) which is that GMimeParser now scans for -----BEGIN PGP
> MESSAGE-----/-----END PGP MESSAGE----- and -----BEGIN PGP SIGNED
> MESSAGE-----/-----END PGP SIGNED MESSAGE----- markers while looking for
> MIME boundaries and sets some state on the corresponding GMimePart that
> you can use to quickly decide if the part contains encapsulated OpenPGP
> data
>
> That's also a nice feature (and again something I implemented in Balsa more
> than a decade ago).  In case of a cleartext signed message, the header lines
> to look for are (in this order) "-----BEGIN PGP SIGNED MESSAGE-----", "-----
> BEGIN PGP SIGNATURE-----" and "-----END PGP SIGNATURE-----", btw.

Ah! Thanks for the catch! I'll fix that up...

> (see
> RFC 4880, sect. 7).  You might want to limit the check to text parts; malware
> spammers recently added fake OpenPGP blocks to M$ office documents
> containing malicious macros.

The GMimeParser just adds those lines to the list of boundaries it checks for since it's already checking for 
strings starting with -- 😊

There's no harm in checking for them - it doesn't actually do anything with them.

Jeff