Forums

Unfortunately no one can be told what FluxBB is - you have to see it for yourself.

You are not logged in.

#1 2011-01-06 23:55:10

Franz
Lead developer
From: Germany
Registered: 2008-05-13
Posts: 6,545
Website

2.0: Language packs etc. (RFC)

If we really proceed as planned, the template system will be one of the first things to be implemented for v2.0. I think if we change the langization-related code at all, this would also be the perfect time.

I've thought about this for a while, and I came up with some ideas, which I'd like to present here:


1. Localized strings

I'm all for a class-based architecture in this matter. Printing strings in language files would look like this:

echo FluxLang::t('You are not allowed to do this, user with username %s', $user->id);

As you can see, the entire string would be passed to the function as an argument, not a key like we currently have in the $lang_* arrays. I think this is how it is usually done with gettext etc. I don't really care whether we'll use that as our backends or not, but that should be a simple implementation decision. I am more interested in interfaces here.

The benefit is also that we would not really have to create a language pack for English (also, corrupt English language packs would not cause a problem), since the complete string is already passed to the function, which also means that it could always be used as a fallback. Not to forget the better readability when actually seeing the texts in the templates.

Also, the t() function interface would work just like PHP's built-in functions printf() and sprintf(), so that variables can be easily substituted. (A shortcut function also seems reasonable.)


2. Loading language files

With an object-oriented interface, this seems simple:

FluxLang::load('admin_common');
FluxLang::load('admin_options');

This would call a function like this:

static public function load($res)
{
    $file = 'langdir/'.self::$language.'/'.$res.'.php';

    $array = include $file; // The file only consists of return array( ... );

    self::$strings = array_merge(self::$strings, $array);
}

As you might have noticed with the code from section 1, we don't need to specify the language file name (like $lang_admin_common or $lang_admin_options) when accessing language strings anymore, since they're all loaded to the same array. This should not be a problem anyway, since we're passing the complete string to the function, so if these are the same in two different places, the translations should be the same anyway (regardless of sections).


3. Summary / leftovers

I believe this would be a much nicer way to handle translation. Much easier to use than the often somewhat long variable names we currently have. Everything stored in one place. Easily adding new strings (without having to take care of language files). And no undefined index errors anymore.

What I really am unsure about is the performance. The classes wouldn't really be complex, so if PHP works at all like I'd expect it to, this itself shouldn't cause too much overhead. Is that true? And is there any reason to be concerned about memory when loading all these strings (I think I once read something like this about issues with WordPress)? And what about long texts like form intros?

P.S.: I believe that we can forget about gettext/PO now, as I'm about to finish the translation center for the site, which will easily allow for handling different formats, versions and projects.

Last edited by Franz (2011-01-06 23:56:59)


fluxbb.de | develoPHP

"As code is more often read than written it's really important to write clean code."

Offline

#2 2011-01-07 00:14:33

Mpok
Member
From: France
Registered: 2008-05-12
Posts: 389

Re: 2.0: Language packs etc. (RFC)

Just a briefly comment on :
1. Localized strings
Seems AWFUL !!! (imo)
This is a regression from the actual code.
And the only 'in' is for eventual fallback !

Please.
Ur on the way to SEPARATE things (code and html, ie). So, let's separated things that are ALREADY separated (language files).
Including english text in the code is a MAJOR drawback (we, foreign communities, have fighting for years to have complete code without these, we have it now, don't screw this up).

Offline

#3 2011-01-07 00:21:07

Mpok
Member
From: France
Registered: 2008-05-12
Posts: 389

Re: 2.0: Language packs etc. (RFC)

On the translator POV : how can i translate EASILY such 'messages inline' ?
(sry FranZ, but it is REALLY a bad idea…)

Offline

#4 2011-01-07 00:42:41

Reines
Administrator
From: Scotland
Registered: 2008-05-11
Posts: 3,197
Website

Re: 2.0: Language packs etc. (RFC)

Franz wrote:

I believe that we can forget about gettext/PO now, as I'm about to finish the translation center for the site, which will easily allow for handling different formats, versions and projects.

I still feel that PHP files aren't the way to go for storing translated strings. Translations are data, PHP is code.

I also (partially) agree with Mpok, if there isn't an English language pack to work from it will be harder for people to create a new pack. The idea of having them inline for fallback is good, except since the English pack is shipped as part of FluxBB I see no reason to need another fallback, it should never become out of date. There is also the problem that then an update to the English translation would break all other translations, which is not good design.

Offline

#5 2011-01-07 01:44:23

Mpok
Member
From: France
Registered: 2008-05-12
Posts: 389

Re: 2.0: Language packs etc. (RFC)

Was a little angry on my previous posts, i apologize…

Apart from the translation process being a pain with that method (as confirmed by Reines), the main question is : WHY ?
(the 'fallback' argue isn't convincing)

The 'language files' way is working well, it permits all translations easily, and it's the way we (foreign communities, at least mine) IMPOSE nowadays to our members for writting mods (making English files PRIOR to French ones, writting readme and description in english).
How can we justify this pain for us, if fluxbb.org just decide to write english messages text IN THE CODE itself ?

As u see, it is not only a 'code' decision, it's a 'political' decision.
Doing so would be a BAD message sent to us (foreign communities).

By the last year, we FOLLOWED u, with enthusiasm : the new version, the repository… Seems good. Did I mention the language files in admin part ? big_smile
All that (and ur announced urself some time ago), was ON THE WAY to better internationalization.

So, u cannot anounce now that u will write english messages in the code. That's just not logical…

Offline

#6 2011-01-07 06:33:42

François
Member
From: Lyon (France)
Registered: 2009-09-01
Posts: 197
Website

Re: 2.0: Language packs etc. (RFC)

Reines wrote:
Franz wrote:

I believe that we can forget about gettext/PO now, as I'm about to finish the translation center for the site, which will easily allow for handling different formats, versions and projects.

I still feel that PHP files aren't the way to go for storing translated strings. Translations are data, PHP is code.

Why not use XML files, they are did for that, encapsulate data in a normalized format ?


Sorry for my English, I'm French

Offline

#7 2011-01-07 10:26:58

Franz
Lead developer
From: Germany
Registered: 2008-05-13
Posts: 6,545
Website

Re: 2.0: Language packs etc. (RFC)

Uh, nice little storm I caused there.

You have some good points. Having an English language pack is all fine with me. We can just have that.

What I do not see is why having the complete strings in the template files would make them less readable. I find it much easier to read than some random short key that is not always self-explanatory, but that might just be opinions at this point.
Would it help if you actually didn't need the English language pack to create another one? Because that's how it will be done with the translation center. We will somehow read the English strings and, when provided with the strings, will automatically generate foreign language packs.

Anyway, the original idea I had (and that I'd still recommend) was using language packs just like we do now, but always including both the English and the foreign language version, so that we do have the English fallback in all cases:

static public function load($res)
{
    $file = 'langdir/'.self::$language.'/'.$res.'.php';
    $en_file = 'langdir/en/'.$res.'.php';

    $array = include $file; // The file only consists of return array( ... );
    $en_array = include $en_file;

    // Merge both languages so that we always have the English fallback
    $array = array_merge($en_array, $array);

    self::$strings = array_merge(self::$strings, $array);
}

Putting the English strings in the function calls directly was what I thought of an enhancement, mainly caused by me not knowing about possible overhead with including all arrays twice.

As I said, I don't really care about gettext or not. The implementation doesn't matter, as long as I rightly suppose that the files will be cached in PHP files in the end no matter what implementation we use.


fluxbb.de | develoPHP

"As code is more often read than written it's really important to write clean code."

Offline

#8 2011-01-07 10:35:00

Jérémie
Member
From: France
Registered: 2008-04-30
Posts: 629
Website

Re: 2.0: Language packs etc. (RFC)

Is Flux that light you need non-necessary ideas to increase its weight?

If you find Flux to be eating too few cpu cycles, memory and I/O, there's hundreds of things that will increase the user experience and ease. Check back the boards.

Offline

#9 2011-01-07 10:36:30

Franz
Lead developer
From: Germany
Registered: 2008-05-13
Posts: 6,545
Website

Re: 2.0: Language packs etc. (RFC)

Why would you think that this increases the weight? It's just another architecture.

Would you say we wouldn't need the template system either?


fluxbb.de | develoPHP

"As code is more often read than written it's really important to write clean code."

Offline

#10 2011-01-07 10:58:52

Reines
Administrator
From: Scotland
Registered: 2008-05-11
Posts: 3,197
Website

Re: 2.0: Language packs etc. (RFC)

I imagine its the idea of loading 2 whole language packs just to perform (assuming the language pack is valid, unrequired) fallback. It would be nicer to validate the language pack when it is installed (or the board updated), then we know its valid and there is no need to load double the amount of data and check every single string before using it.

As Rasmus put it in some talk he did on PHP - a lot of web applications do things in a weird way compared to "normal" applications. His example was along the lines of:

"Am I using MySQL? Yes? Okay - do this..."
"Am I still using MySQL? Yes? do this..."
"Am I still using MySQL? Yes? okay..."

His point was this kind of runtime check should try to be avoided since it will not change after the application is installed and does nothing but add overhead.

Offline

#11 2011-01-07 12:29:31

FSX
Former Developer
From: NL
Registered: 2008-05-09
Posts: 818
Website

Re: 2.0: Language packs etc. (RFC)

It looks a bit like Gettext. I've been working with Gettext for a Python application and it works pretty good, but it's annoying to have the full English string in the code, but there's no other way with Gettext since the files have to be recompiled with each change.

Calling functions to load strings is fast, but you're probably already internally storing the strings in an array. Why not use that array to call the strings?

echo TransLateClass::$s['stuff'];

Doesn't look very beautiful, but it's much faster than calling a function which also does some checks. You'll lose the functionality of using incomplete language packs, but I don't think that's a very important feature.

About not storing language files in PHP. If you're using that ugly XML thing be sure to adds caching. A simple text file would be easier an faster. Example:

String 1 = Tekst 1
String 2 = Tekst 2
String 3 = Tekst 3

Offline

#12 2011-01-07 12:30:02

François
Member
From: Lyon (France)
Registered: 2009-09-01
Posts: 197
Website

Re: 2.0: Language packs etc. (RFC)

The question is : how to propose translation files if the language string are in the template ? The English text will be used as "key" ?

Last edited by François (2011-01-07 12:30:14)


Sorry for my English, I'm French

Offline

#13 2011-01-07 12:54:56

FSX
Former Developer
From: NL
Registered: 2008-05-09
Posts: 818
Website

Re: 2.0: Language packs etc. (RFC)

François wrote:

The question is : how to propose translation files if the language string are in the template ? The English text will be used as "key" ?

Yes, the English text will be used as a key, but when there's no language entry that has that key it just uses the English text. That's what Franz means.

Offline

#14 2011-01-07 13:05:08

Reines
Administrator
From: Scotland
Registered: 2008-05-11
Posts: 3,197
Website

Re: 2.0: Language packs etc. (RFC)

FSX wrote:
François wrote:

The question is : how to propose translation files if the language string are in the template ? The English text will be used as "key" ?

Yes, the English text will be used as a key, but when there's no language entry that has that key it just uses the English text. That's what Franz means.

The problem with that is if the English language pack changes (typo, spelling mistake, bad grammar, whatever) it breaks every other language pack. When the language packs are developed by the same people as the software that isn't so much of a problem, but for our situation it's unacceptable.

Offline

#15 2011-01-07 14:29:12

Reines
Administrator
From: Scotland
Registered: 2008-05-11
Posts: 3,197
Website

Re: 2.0: Language packs etc. (RFC)

Something I threw together quickly to show what I was thinking. This would easily allow plugging in different formats (PHP-based, XML, INI or whatever) if needed (though I don't necessarily know if supporting different formats would be best).

<?php

define('FLUX_ROOT', dirname(__FILE__));

abstract class Language
{
    protected $code;
    protected $strings;
    protected $cache;

    public function __construct($code, $cache)
    {
        $this->code = $code;
        $this->cache = $cache;
        $this->strings = array();

        $this->import('common');
    }

    public function import($resource)
    {
        $strings = $this->cache->get('lang.'.$this->code.'.'.$resource);
        if ($strings === Cache::NOT_FOUND)
        {
            $strings = $this->_import($resource);
            $this->cache->set('lang.'.$this->code.'.'.$resource, $strings);
        }

        $this->strings = array_merge($this->strings, $strings);
    }

    public abstract function _import($resource);

    public function translate()
    {
        $args = func_get_args();
        $key = array_shift($args);

        // TODO: Maybe a custom version of vsprintf that runs htmlentities on %s and so on?
        // Or should the template engine handle that and we assume language entries aren't allowed markup in them.

        return vsprintf($this->strings[$key], $args);
    }
}

// XML based language files
class Language_XML extends Language
{
    public function _import($resource)
    {
        $xml = @simplexml_load_file(FLUX_ROOT.'lang/'.$this->code.'/'.$resource.'.xml');
        if ($xml === false)
            throw new Exception('Unable to find resource: '.$resource);

        $strings = array();

        foreach ($xml->lang as $lang)
        {
            $attributes = $lang->attributes();

            $key = (string) $attributes['key'];
            $value = (string) $lang;

            $strings[$key] = $value;
        }

        return $strings;
    }
}

// Example below: assumes $cache is an instance of php-cache (https://github.com/reines/php-cache)

$lang = new Language_XML('en', $cache);
$lang->import('topic');

var_dump($lang->translate('hello', $user->username));

Offline

#16 2011-01-07 16:09:26

Franz
Lead developer
From: Germany
Registered: 2008-05-13
Posts: 6,545
Website

Re: 2.0: Language packs etc. (RFC)

Ignore the fallback suggestion, by the way. We can easily take care of that when generating the language packs here on the site. smile


fluxbb.de | develoPHP

"As code is more often read than written it's really important to write clean code."

Offline

#17 2011-01-07 23:55:08

eureka
Member
From: France
Registered: 2008-08-29
Posts: 46

Re: 2.0: Language packs etc. (RFC)

I totally agree with the Franz solution:
1) the code is more readable
2) it is never necessary to load the English file which increases performance
3) The English files can be generated automatically (without errors) for translation


French language packs for FluxBB 1.3-Legacy and FluxBB 1.4.2

Offline

#18 2011-01-08 05:59:59

Mpok
Member
From: France
Registered: 2008-05-12
Posts: 389

Re: 2.0: Language packs etc. (RFC)

Reines wrote:

The problem with that is if the English language pack changes (typo, spelling mistake, bad grammar, whatever) it breaks every other language pack. When the language packs are developed by the same people as the software that isn't so much of a problem, but for our situation it's unacceptable.

+1
Whatever the solution u finally choose, seeing english messages text in the PHP code is BAD (except for error codes). Point. IMO.
And i'm still not understand why u want to change the way language packs work nowadays ('key' => 'localized string'). The 'key' is something GLOBAL, not the actual real english message. I don't mind 'key' is some english words, but NOT the whole displayed string.
I realize that it's maybe 'peanuts', but once more, u didn't prove ur way is better.

Offline

#19 2011-01-08 09:31:06

Otomatic
FluxBB Donor
From: Paris - France
Registered: 2010-01-26
Posts: 547
Website

Re: 2.0: Language packs etc. (RFC)

Hello

It is difficult to properly express his thoughts in a language not their mother tongue.
I apologize if some words may shock you, this is not my intention in writing in French.
I am not sufficiently knowledgeable of the intricacies of the English language to be sure that the words chosen properly express my thoughts.

In my view, any modifications to the software must be either to correct a defect, either to improve utilization.
In any case, it must be the comfort of the user who must prevail, not what would the developer.

For over thirty years, sat on my desk a quotation from Henry de Montherlant from the book "Va jour avec cette poussière (Go play with the dust)":
"Today the ideal of progress is replaced by the ideal of innovation: it's not better either, it's just that this is new, even if it's worse than before, and it obviously."
(See http://aviatechno.free.fr/bib/images.php?id=122 - Sorry, only in french)

Are you sure this "new" way of transcribing the text is real progress?

Are you sure that this will facilitate and simplify the user experience?

Would it not rather a "whim" programmer?

Of course, FluxBB is, somehow, your "baby " and raise you the way you want. Nevertheless, it is a "baby" which was adopted by many users around the world and is no longer "your baby" but "our baby".
All changes must simplify and assist the work of users. It is for the users you must direct your work, not for developers.

Thank you for your attention.


Ce n'est pas parce que l'erreur se propage qu'elle devient vérité. Ghandi
An error does not become truth by reason of multiplied propagation. Ghandi

Offline

#20 2011-01-08 11:23:55

Gil
Member
From: France
Registered: 2008-05-10
Posts: 175

Re: 2.0: Language packs etc. (RFC)

Mpok wrote:

Whatever the solution u finally choose, seeing english messages text in the PHP code is BAD (except for error codes). Point. IMO.

It's not bad because it's bad wink Be able to have a clear view of the exact text is a good aim; It's bad because of reason provided by Reines, which is an ample reason. The solution is to try have a clear label for each text (or errors, imo).

Otomatic wrote:

All changes must simplify and assist the work of users. It is for the users you must direct your work, not for developers.

Agree with ur message, but not totally agree with this particular point; Yes, most changes must be for the users; but if you can find a change to simplify the code or the developers' work, you must do it!

Reines wrote:

...different formats (PHP-based, XML, INI or whatever) if needed (though I don't necessarily know if supporting different formats would be best).

Could be great for a framework, but not useful imo for a forum.

Offline

#21 2011-01-08 12:25:10

Franz
Lead developer
From: Germany
Registered: 2008-05-13
Posts: 6,545
Website

Re: 2.0: Language packs etc. (RFC)

Don't worry, guys. Obviously I didn't think this through as much as I should have. So here's my updated proposal:

1. Translation

...itself will take part in the Translation Center here on the site. Here you will be provided with simple-to-use tools to contribute to the translation of your language.

The language packs will be generated automatically from the strings in the database. If they are not completed, missing strings will be filled with the English strings. This is done to make sure that you don't get a lot of undefined index error messages when using an incomplete language pack. English as a fallback also ensure a somewhat usable language pack in all cases (plus we don't have to do any existance checks when accessing language strings).

(Importing of currently existing language packs will, of course, be taken care of, too.)

2. Language pack format

Really doesn't matter either way (at least to me). Since we'll create cache files in PHP format if we use another format, I don't see any point in not using PHP right away - especially since we will generate the language packs on this site in whatever format we'll use.

Let me remind you here that the original idea behind using gettext/PO was that tools exist to keep language files up-to-date more easily that only supported that format.

3. Loading language files

Using the class-based approach, we'll get rid of multiple long variable names and instead can just do this:

Flux_Lang::import('common');
Flux_Lang::import('viewtopic');

This will load all the requested strings into one array, so that language strings can be fetched from one single point.

4. Language strings

Using keys is fine with me. You convinced me. That also means we'll have an English language pack in the releases.

Accessing the strings will be done with their keys, so maybe something like this:

Flux_Lang::t('Error message too long', $max_characters);

Again, we'll be able to simply pass variables as parameters, just like it's known from sprintf() etc.



So, sorry for all the mess. I hope this proposal suits everybody and I didn't forget anything.

@Otomatic: While I agree with your quote, one of our aims always was to have easy-to-read and -understand code that is easy to modify. So, partially, we are developing for developers, too.

Last edited by Franz (2011-01-08 12:28:47)


fluxbb.de | develoPHP

"As code is more often read than written it's really important to write clean code."

Offline

#22 2011-01-08 12:29:34

Franz
Lead developer
From: Germany
Registered: 2008-05-13
Posts: 6,545
Website

Re: 2.0: Language packs etc. (RFC)

Oh, now I remember another reason why I preferred the Flux_Lang::import() way: extensions.

If you've been wondering, too, about how to include system "resources" like language files, authentication adapters etc. in extensions while still keeping them inside the extension directory, here's what I thought of.

Flux_Lang::load('extensionname/languagefile1');
// or some other separator...

This way, extensions can simply "plug in" the existing langization system in the easiest possible way, while still keeping the directory structure of the core clean.
This can not only be done with language files, but with many other things, too.


fluxbb.de | develoPHP

"As code is more often read than written it's really important to write clean code."

Offline

#23 2011-01-08 12:38:56

Reines
Administrator
From: Scotland
Registered: 2008-05-11
Posts: 3,197
Website

Re: 2.0: Language packs etc. (RFC)

That would require some string parsing, a more efficient approach might be to define a second parameter that defaults to null - when it's null language files are loaded normally, otherwise they are taken from that extension.

Flux_Lang::load('langagefile1', 'extensionname');

PS. Why use a static class rather than a "proper" class? From a Java perspective that would be really bad design. In your example, how does it know what language to use?

Offline

#24 2011-01-08 12:52:06

Reines
Administrator
From: Scotland
Registered: 2008-05-11
Posts: 3,197
Website

Re: 2.0: Language packs etc. (RFC)

Franz wrote:
2. Language pack format

Really doesn't matter either way (at least to me). Since we'll create cache files in PHP format if we use another format, I don't see any point in not using PHP right away - especially since we will generate the language packs on this site in whatever format we'll use.

Let me remind you here that the original idea behind using gettext/PO was that tools exist to keep language files up-to-date more easily that only supported that format.

If we implement a proper cache layer (which we will) then the language files wont necessarily be cached in PHP format. The only reason I can see for keeping them in PHP format is simply because that's how it's always been.

Another, although not something I've come across, argument against using PHP files is security. A language pack can easily include executable code, or even a syntax error which would cause FluxBB to be unusable. There isn't any good way to protect against this if using include/require/eval - since they are specifically designed for including executable code. When using a data format such as gettext or XML there is no way to accidentally include executable code, and a syntax error would be easily detectable during the creation of the cache - where we could decide how to best handle it (fall back to English, for example).

Offline

#25 2011-01-08 13:31:41

Smartys
Former Developer
Registered: 2008-04-27
Posts: 3,139
Website

Re: 2.0: Language packs etc. (RFC)

My two cents: I'm seeing a lot of kneejerk reactions here. People are rejecting the idea because it changes something they're familiar with, not necessarily because of any real problems.

As Reines points out, gettext is a standard used in many applications. You can read more about it here. There are a lot of pre-built tools for it which should improve your ability to translate. It is possible to use neutral strings as keys, the same way we do now, although the strings themselves are often used as the keys. There is also the added benefit of security, as Reines correctly points out.

The developers are looking out for your interests: they want to make things simpler and easier for you. So, maybe trust them a little bit?

Offline

Board footer

Powered by FluxBB