Re: Re: Outgoing message truncated at dot wrapped on single line

[Ladar Levison <ladar lavabit com>]

Hi,

I built the Lavabit mail platform, so if there is a bug in the SMTP 
implementation its definitely my fault and that means I'll hang my head 
in shame while I fix it.

On 1:59 PM, Brian Stafford wrote:
> > If any of you seriously wants to try find the exact source of the
> > >  misbehaviour and maybe make Balsa compatible with Lavabit’s SMTP
> > >  server, I am willing to run some more tests (with wireshark or
> > >  whatever) and strip the results of any personal information, so that
> > >  I
> > >  can share them. The same goes for the logs of my previous tests.
> If there is a genuine bug in the client, I'd be interested in finding
> it, however there is only so far one can go with this sort of thing,
> beyond which the server needs fixing.
>
> My policy on libesmtp is that I will only fix issues regarding protocol
> if it can be demonstrated that it violates RFC normative requirements,
> i.e. those stated with MUST or SHOULD.  I will certainly never make a
> change that knowingly violates MUST or SHOULD.

Its been several years since I studied the SMTP RFCs, but as I recall 
the DATA command is terminated supposed to be terminated by a 
<CRLF>.<CRLF> which is basically a line that only contains a period. To 
keep the logic simple, messages lines that start with a period are 
supposed to be dotstuffed; <CRLF>. is converted to <CRLF>.. and the 
server condenses accordingly. See RFC 2821 section 4.5.2.

Where things get tricky is when a client breaks the rules. When I was 
first pointed at this thread that's what I thought was happening. 
Examples include sending <LF>.<LF> or <CRLF>.<LF> or sending <CRLF>.  
any text followed by <CRLF>, etc. Basically any scenario where a line 
starts with a period, but does not start with two periods and the 
complete line is not a single period. In these situations our server 
does not terminate the DATA command. Instead we keep reading the input 
stream and looking for a line with a single period.

If this issue hasn't surfaced before now it might be that lines with a 
single period were being sent as part of a larger TCP packet. In that 
situation I know some mail servers will continue accepting the input 
data after the terminator. These systems want the single period 
termination line to be at the end of a transmission or sent as a 
separate packet. I believe this behavior is relied on SMTP clients 
sending a message terminator and properly waiting for a server response. 
If you submit the test case via netcat to one of these servers you can 
see this scenario in action. Just be sure the command sequence stops 
with the message terminator and not a QUIT command. I believe this 
behavior was never formalized.

The other strategy I've seen is for SMTP servers to simply skip any data 
that remained in an input buffer following the line termination 
sequence. This pattern would appear to behave the same. The key 
difference would be that instead of the message data triggering protocol 
errors, it would be silently ignored.

I think the ESMTP PIPELINING RFC implicitly destroyed these practices by 
standardizing how a client is supposed to send multiple commands and/or 
data using a single TCP transmission. Clients are supposed to wait for 
server responses following DATA commands. What I don't remember are the 
RFC details what should happen when clients disobey the grouping rules.

I should also admit there are a number of RFCs after 2821 that could 
alter aspects of SMTP and the MIME format. Its possible that someone on 
this list will make me look/feel stupid...

> > >  Otherwise, I will probably just make a blog post about it and hope
> > >  that someone on the internet knows how to pass it on to the Lavabit
> > >  developer team.
> My instinct is that there is a serious bug in the Lavabit SMTP
> server, workarounds involving tabs on the end of lines just reinforces
> this impression.

In writing this email I discovered my code does mishandle lines starting 
with a single period that contain additional characters before the 
newline. The RFC says a serer should continue accepting data (which I 
do) but strip the single starting period from the final message (which I 
don't). Either way, this is behavior is what makes the tab hack to work.

And yes a user did email us about the blog posting. Over the years 
issues like this have come up and I've always tried to investigate the 
reports accompanied by enough details to reproduce the behavior.

> Perhaps a better workaround is to submit messages via, say, Gmail or
> Yahoo.

I checked out the Postfix implementation to see if my code behaved the 
same way. I've never met Wietse in person, but my virtual encounters on 
mailing lists and having seen some his code pasted over the urinal at my 
favorite bar, I think he's done a good job making Postfix conform to the 
RFCs. If I run the test case from earlier in this thread against Postfix 
v2.2.10 (the RHEL mutation of it) using telnet and typing by hand, I see:

[ladar@jenna ~]# telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
220 jenna.jameson.lavabit.com ESMTP Postfix
EHLO localhost
250-jenna.jameson.lavabit.com
250-PIPELINING
250-SIZE 1024000000
250-VRFY
250-ETRN
250 8BITMIME
MAIL FROM:<>
250 Ok
RCPT TO:<ladar lavabit com>
250 Ok
DATA
354 End data with<CR><LF>.<CR><LF>
From:<>

Hello world! Please touch me tenderly!
.