Fork me on GitHub
Subscribe 3

Ticket #949 (fixed bug)

Use \r\n for SMTP, FORUM_EOL for others

  • Created: 2014-02-06 10:08:40
  • Reported by: Franz
  • Assigned to: Franz
  • Milestone: 1.5.8
  • Component: email
  • Priority: normal

History

seven 2014-02-06 10:56:52

There is still a (minor) problem with the message: due to this snippet of code

// Make sure all linebreaks are LF in message (and strip out any NULL bytes)
$message = str_replace("\0", '', pun_linebreaks($message));

..from include/functions.php:

//
// Convert \r\n and \r to \n
//
function pun_linebreaks($str)
{
    return str_replace("\r", "\n", str_replace("\r\n", "\n", $str));
}

This means that the message, on Windows platforms, is passed to the local MTA using '\n' line endings rather than '\r\n'.

This is not good, because on Windows servers PHP's mail() is implemented by talking to a remote SMTP server rather than the local MTA: http://www.php.net/manual/en/mail.configuration.php

And things break again if the remote server is qmail, requiring the end of the message being '\r\n'.

This is the quick'n'dirty solution, but I think we should get rid of FORUM_EOL and use PHP_EOL.

	// Make sure all linebreaks are LF in message (and strip out any NULL bytes)
	$message = str_replace("\0", '', pun_linebreaks($message));

	if ($pun_config['o_smtp_host'] != '')
	{
		// Newlines must be \r\n
		if(FORUM_EOL != "\r\n")
		{
			$headers = str_replace(FORUM_EOL, "\r\n", $headers);
		}
		$message .= "\r\n";
		smtp_mail($to, $subject, $message, $headers);
	}
	else
	{
		// Newlines should have system's default.
		$message .= FORUM_EOL;
		mail($to, $subject, $message, $headers);
	}
}

Franz 2014-02-13 13:08:11

  • Owner set to Franz.

Franz 2014-02-13 13:14:07

Commit c9f5924 to fluxbb master

#949: Use \r\n for SMTP emails, PHP_EOL for local.

Franz 2014-02-13 13:15:40

Commit 0d2e568 to fluxbb master

#949: Remove FORUM_EOL constant.

Franz 2014-02-13 13:24:23

For SMTP, does the actual message need to have CRLF line endings? Or is one CRLF at the very end enough?

seven 2014-02-13 14:07:11

Hmmm... From what I read here http://cr.yp.to/docs/smtplf.html

Every line in an Internet mail message is required to end with CR LF. The entire message ends with CR LF dot CR LF. 822bis specifically prohibits other uses of LF.

The mail clients discussed above are incorrectly ending lines with LF and, in most cases, ending the entire message with LF dot LF. That's not CR LF dot CR LF, so a server such as msn.com will sit there waiting for the rest of the message. After a while it'll give up and drop the connection. Your mail doesn't get through.

And the mentioned RFC... ftp://ftp.rfc-editor.org/in-notes/rfc2822.txt

2.3. Body

   The body of a message is simply lines of US-ASCII characters.  The
   only two limitations on the body are as follows:

   - CR and LF MUST only occur together as CRLF; they MUST NOT appear
     independently in the body.

   - Lines of characters in the body MUST be limited to 998 characters,
     and SHOULD be limited to 78 characters, excluding the CRLF.

However, there is someone who doesn't quite agree with qmail's behaviour:
http://hewgill.com/journal/entries/30-q … your-email

The qmail document refers to a document it calls "822bis", which apparently was an older name for RFC 2822. Section 2.3 of this document does indeed prohibit the use of bare LF characters within the message body. However, this document is not considered a standard; to find a standard relating to email transfer we must look at RFC 822, also known as STD 11. RFC 822 does not specifically prohibit the use of bare LF characters in message bodies.

To achieve maximum compatibility I believe it's better to convert every bare LF to CRLF and terminate the message with '\r\n.\r\n' to end it (note the dot in the middle) before issuing the QUIT command.

Franz 2014-02-13 14:16:30

Why the dot in the middle?

The problem with replacing "\n" with "\r\n" is that we might replace already existing "\r\n" with "\r\r\n" - I believe that's how all of these issues got started in the beginning. So, some regex magic is necessary, I guess. If we have to, but you seem to be sure we do.

seven 2014-02-13 14:50:37

The dot in the middle tells the SMTP server that we're done sending the message. It may not be an immediate thing, but it dates back when people were writing their mails in the terminal talking with the smtp server over telnet.

Try it yourself, you'll be left there wondering how to tell the server that you're done if you don't close the message with a final dot. smile

pazzo@bender:~$ telnet alt1.gmail-smtp-in.l.google.com smtp
Trying 173.194.71.26...
Connected to alt1.gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP n3si3219651lae.94 - gsmtp
HELO bender.gamezoo.it
250 mx.google.com at your service
MAIL from: <hey-this-is-mine@gmail.com>
250 2.1.0 OK n3si3219651lae.94 - gsmtp
RCPT to: <this-one-is-mine-too@gamezoo.it>
250 2.1.5 OK n3si3219651lae.94 - gsmtp
DATA
354  Go ahead n3si3219651lae.94 - gsmtp
From: hey-this-is-mine@gmail.com
To: this-one-is-mine-too@gamezoo.it
Subject: Test mail

Questo è un test. Here I can't close the message.
No matter what.


Still not closing. HEY!!!
Ok, let's go with the dot. Next line...
.
250 2.0.0 OK 1392302649 n3si3219651lae.94 - gsmtp
QUIT
221 2.0.0 closing connection n3si3219651lae.94 - gsmtp
Connection closed by foreign host.

seven 2014-02-14 09:11:24

Forget about the final "\r\n.\r\n" sequence: smtp_mail() already does it. My bad for not looking at the code before writing.

pun_linebreaks could be edited like this:

//
// Convert \r\n and \r to \n
//
function pun_linebreaks($str)
{
    return str_replace(array("\r\n", "\r"), array("\n", "\n"), $str)
}

Looking at the current master email.php there's only one thing missing: the local mailer (that, on Windows systems, is PHP talking to a remote SMTP server) now gets always "\n" instead of PHP_EOL. Proposed fix:

if ($smtp)
{
    // Headers and Message should be \r\n
    $message = str_replace("\n", "\r\n", $message);
    smtp_mail($to, $subject, $message, $headers);
}
else
{
    // Headers and Message should be PHP_EOL
    if(PHP_EOL != "\n")
    {
        $message = str_replace("\n", PHP_EOL, $message);
    }
    mail($to, $subject, $message, $headers);
}

Franz 2014-02-15 01:31:58

Commit 1cd7114 to fluxbb master

#949: Improve pun_linebreaks() method.

Franz 2014-02-15 23:35:31

Commit 6ec7809 to fluxbb master

#949: Replace mail line endings correctly, shorten code.

Franz 2014-02-15 23:59:07

Commit fafa7cf to fluxbb master

#949: Reintroduce FORUM_EOL for weird mail programs.

Franz 2014-02-16 00:00:49

Okay, this should do it. I added FORUM_EOL because we would probably be back to previous issues with Postfix, qmail and whatnot if we wouldn't do that. For anybody with problems, defining FORUM_EOL as "\n" in their config should then do the trick.

Please review so I can close this ticket.

If this isn't the last issue related to these stupid line endings in emails, I'll probably go crazy. Sigh.

seven 2014-02-16 10:39:34

This code is compliant with the algorithm:

Linux server, mail() -> "\n" (FORUM_EOL)
Linux server, smtp_mail() -> "\r\n" (hardcoded)
Windows server, mail() -> "\r\n" (FORUM_EOL)
Windows server, smtp_mail -> "\r\n" (hardcoded)

Since most of us are on Linux servers, I'd add an if() on line #252 to avoid doing an unnecessary str_replace:

if($EOL != "\n") $message = str_replace("\n", $EOL, $message);

However, this looks OK and there should be no further issues. I'll pay you a beer if someone comes up with a problem related to this code. smile

Comment edited 1 times (Diff)

quy 2014-02-16 17:59:25

Looks good to me. Tested fine with mail/smtp on Linux server. I am not on a Window server, however, tested fine with smtp on my local computer. I say go for it.

Franz 2014-02-16 18:21:12

  • Status changed from open to fixed.

Wonderful, thanks guys!

I'll leave the code as is, it's more readable that way. I count on PHP to be smart when both parameters are equal. smile

Franz 2014-10-20 11:51:10

  • Milestone changed from 1.5.7 to 1.5.8.