Development

I think that the problem is MySQL and utf8 that accept utf-8 three bytes characters and does not accept utf-8 four bytes characters.
http://dev.mysql.com/doc/refman/5.5/en/ … f8mb4.html
To solve this problem, I think that the better way is to replace any four bytes characters by three bytes unknown character.

Does that mean we would only have to truncate 4-byte characters for MySQL?

You have to use functions in include/utf8/utils/bad.php like utf8_bad_replace before inserting strings into BDD but it seems that this function does not work correctly and does not replace 4-byte characters.

WordPress has the same problem and somebody wrote a patch for escaping those characters before storing them in the database and unescaping on retrieval.

That solution is a little big for such a small problem, I think we should just replace it.

Nontheless, this WP ticket (along with the attached patch) might help us find a good solution.

I also think that the best solution is to replace utf8 four bytes characters by a ? or a space.
It is possible to accept utf8mb4 but only with MySQL version 5.5.3 or more.
I wrote an article for help of french fluxbb (Sorry, only in French) here : http://fluxbb.fr/aide/doku.php?id=mysql … tre_octets

Frank (FSX) told me he's working on it, let's see if we can port it to FluxBB's (relatively) old version of the utf8 library.

In fact, if somebody can tell me how I can recognize four-byte characters in PHP, I would happily fix our implementation myself.

Hi,

For FluxBB 1.4.7, in post.php, just after :

   $now = time();

add :

//modif oto - Replace bad character by '?' including utf8 four bytes
$message = preg_replace('/[x00-x08x10x0Bx0Cx0E-x19x7F]'.
'|[x00-x7F][x80-xBF]+'.
'|([xC0xC1]|[xF0-xFF])[x80-xBF]*'.
'|[xC2-xDF]((?![x80-xBF])|[x80-xBF]{2,})'.
'|[xE0-xEF](([x80-xBF](?![x80-xBF]))|(?![x80-xBF]{2})|[x80-xBF]{3,})/S',
'?', $message );
$message = preg_replace('/xE0[x80-x9F][x80-xBF]'.
'|xED[xA0-xBF][x80-xBF]/S','?', $message );
//End modif oto

Any way to combine that with the utf8_bad_replace() function?

What about applying this function?

//
// Replace four-byte characters with a question mark
//
// As MySQL cannot properly handle four-byte characters with the default utf-8
// charset up until version 5.5.3 (where a special charset has to be used), they
// need to be replaced, by question marks in this case. 
//
function strip_bad_multibyte_chars($str)
{
	$result = '';
	$length = strlen($str);
	
	for ($i = 0; $i < $length; $i++)
	{
		// Replace four-byte characters (11110www 10zzzzzz 10yyyyyy 10xxxxxx)
		if ((ord($str[$i]) & 0xF8) == 0xF0)
		{
			$result .= '?';
			$i += 3;
		}
		else
		{
			$result .= $str[$i];
		}
	}
	
	return $return;
}

And a version that is a little easier to read, inspired by this excellent article:

//
// Replace four-byte characters with a question mark
//
// As MySQL cannot properly handle four-byte characters with the default utf-8
// charset up until version 5.5.3 (where a special charset has to be used), they
// need to be replaced, by question marks in this case. 
//
function strip_bad_multibyte_chars($str)
{
	$result = '';
	$length = strlen($str);
	
	for ($i = 0; $i < $length; $i++)
	{
		// Replace four-byte characters (11110www 10zzzzzz 10yyyyyy 10xxxxxx)
		$ord = ord($str[$i]);
		if ($ord >= 240 && $ord <= 244)
		{
			$result .= '?';
			$i += 3;
		}
		else
		{
			$result .= $str[$i];
		}
	}
	
	return $result;
}

@Otomatic: could you test my version?

Also, we probably need to apply this to more inputs than just the post contents.

Hi,

I test the second version in post.php and the four bytes utf8 characters are replaced by '?'.
It works!

Should be fixed now.

I guess we don't necessarily need to do this in other places. It is just very annoying when posts get truncated, I can live with the rest.