Project

General

Profile

Feature #4467

Add gettext support

Added by Julio Montoya over 7 years ago. Updated over 4 years ago.

Status:
Needs more info
Priority:
Normal
Category:
Global / Others / Misc
Target version:
Start date:
11/03/2012
Due date:
% Done:

70%

Estimated time:
Spent time:
Complexity:
Normal
SCRUM pts - complexity:
?

Description

So we can use launchpad as a language repository instead of CTA


Files

translate_array_to_po.php (2.07 KB) translate_array_to_po.php Julio Montoya, 21/03/2012 12:45
spanish.tar.gz (228 KB) spanish.tar.gz Julio Montoya, 13/06/2013 17:06

Related issues

Related to Chamilo LMS - Bug #4511: get_lang() doesn't manage well empty translationsBug resolved23/03/2012

Actions
Related to Chamilo LMS - Feature #4557: Add languages forlder and files for pluginFeature implemented27/03/2012

Actions
Related to Chamilo LMS - Feature #5523: Replace custom autoloader with ComposerFeature implemented21/09/2012

Actions
Blocked by Chamilo LMS - Feature #5349: Fix typo in old database settings variablesAssigned26/08/2012

Actions

Associated revisions

Revision c4c328bc (diff)
Added by Julio Montoya over 7 years ago

Adding commented code in order to use gettext see #4467

Revision 2e89bc4b (diff)
Added by Julio Montoya over 6 years ago

Adding support to $app container in all files see #4467 and #5523

Revision 3a2cd26a (diff)
Added by Julio Montoya over 6 years ago

Adding export language to po files command see #4467

Revision 5ee28817 (diff)
Added by Julio Montoya over 6 years ago

Adding locale gettext translations demo translations (fr, en, es, de, pt-br) see #4467

Revision 29b543aa (diff)
Added by Yannick Warnier almost 6 years ago

Export to gettext command: avoided CRLF creating parse errors in resulting gettext files - refs #4467

Revision 3be68d97 (diff)
Added by Yannick Warnier almost 6 years ago

Export to gettext command: more fixes to CRLF and added commented section to get all languages at once - refs #4467

Revision 4b78dc64 (diff)
Added by Yannick Warnier almost 6 years ago

Fixed default English variables in conversion to gettext - refs #4467

Revision 0fd303d7 (diff)
Added by Yannick Warnier over 4 years ago

Remove chat.inc.php language file in all languages upon upgrade (variables have been moved to trad4all.inc.php) - refs #4467

Revision e127612c (diff)
Added by Yannick Warnier over 4 years ago

Remove slideshow.inc.php language file in all languages upon upgrade (variables have been moved to trad4all.inc.php) - refs #4467

Revision f244283e (diff)
Added by Yannick Warnier over 4 years ago

Remove course_description, external_module, myagenda and scormbuilder language file in all languages upon upgrade (variables have been moved to trad4all.inc.php) - refs #4467

Revision 0bf126df (diff)
Added by Yannick Warnier over 4 years ago

Remove all get_lang('lang...') - refs #4467

Revision 92c66317 (diff)
Added by Yannick Warnier over 4 years ago

Remove language files with little use and clean deprecated language variables - refs #4467

Revision 96a8d509 (diff)
Added by Yannick Warnier over 4 years ago

Update upgrade procedure to remove old language files - refs #4467

Revision cd0dc1ee (diff)
Added by Yannick Warnier over 4 years ago

Remove language files with little use and clean deprecated language variables (part 2) - refs #4467

Revision 422fa36e (diff)
Added by Yannick Warnier over 4 years ago

Remove link language file - refs #4467

Revision fcbf64bd (diff)
Added by Yannick Warnier over 4 years ago

Remove link language file - refs #4467

Revision 99689646 (diff)
Added by Yannick Warnier over 4 years ago

Remove md_link and md_scorm language files - refs #4467

Revision 6d4f1bcd (diff)
Added by Yannick Warnier over 4 years ago

Move coursebackup language file to trad4all - refs #4467

Revision d484a132 (diff)
Added by Yannick Warnier over 4 years ago

Move courses language file to trad4all - refs #4467

Revision 75bc0af5 (diff)
Added by Yannick Warnier over 4 years ago

Move courses language file to trad4all - refs #4467

Revision 03690fed (diff)
Added by Yannick Warnier over 4 years ago

Move md_document language file to trad4all - refs #4467

Revision ffc5f31f (diff)
Added by Yannick Warnier over 4 years ago

Move blog language file to trad4all - refs #4467

Revision f47af4ae (diff)
Added by Yannick Warnier over 4 years ago

Move videoconf language file to trad4all - refs #4467

Revision 652c6ecf (diff)
Added by Yannick Warnier over 4 years ago

Move index language file to trad4all - refs #4467

Revision 3b96011d (diff)
Added by Yannick Warnier over 4 years ago

Move work language file to trad4all - refs #4467

Revision 393af5fe (diff)
Added by Yannick Warnier over 4 years ago

Remove previous language files that were moved to trad4all - refs #4467

Revision 926c095e (diff)
Added by Yannick Warnier over 4 years ago

Remove group and forum language files - refs #4467

Revision 8265d4be (diff)
Added by Yannick Warnier over 4 years ago

Move group and forum language file to trad4all - refs #4467

Revision 5c7ad62b (diff)
Added by Yannick Warnier over 4 years ago

Remove dropbox, course_info and userInfo language files - refs #4467

Revision b87882c5 (diff)
Added by Yannick Warnier over 4 years ago

Move dropbox, userInfo and course_info language file to trad4all - refs #4467

Revision cd83504a (diff)
Added by Yannick Warnier over 4 years ago

Remove create_course and registration language files - refs #4467

Revision 5ff09937 (diff)
Added by Yannick Warnier over 4 years ago

Move registration and create_course language files to trad4all - refs #4467

Revision b3f0569c (diff)
Added by Yannick Warnier over 4 years ago

Forgotten change about moving create_course language file to trad4all - refs #4467

Revision 4f761602 (diff)
Added by Yannick Warnier over 4 years ago

Remove reservation language file as the tool has been removed previously - refs #4467

Revision d575fa47 (diff)
Added by Yannick Warnier over 4 years ago

Move survey, gradebook, wiki, learnpath, tracking, document, exercice, help and admin language files to trad4all - refs #4467

Revision 106040ab (diff)
Added by Yannick Warnier over 4 years ago

Update a few language variables' case to match new language files - refs #4467

Revision 9879b14a (diff)
Added by Yannick Warnier over 4 years ago

Remove deprecated language files during upgrade: update list - refs #4467

Revision 874a2744 (diff)
Added by Yannick Warnier over 4 years ago

Fix issue in installer with admin language file removal - refs #4467

Revision d75f753f (diff)
Added by Yannick Warnier over 4 years ago

Remove install language file - refs #4467

Revision 55243536 (diff)
Added by Yannick Warnier over 4 years ago

Added new trad4all that includes install.inc.php - refs #4467

Revision f8ec8fca (diff)
Added by Yannick Warnier over 4 years ago

Integrate install.inc.php to trad4all.inc.php - refs #4467

Revision bf87e869 (diff)
Added by Yannick Warnier over 4 years ago

Fix lang variable Firstname -> FirstName - refs #4467

Revision 3b3323ef (diff)
Added by Yannick Warnier over 4 years ago

Alter language vars to match conventions - refs #4467

Revision 99cb9145 (diff)
Added by Yannick Warnier over 4 years ago

Alter language vars to match conventions (bis) - refs #4467

Revision 06dc7d63 (diff)
Added by Yannick Warnier over 4 years ago

Remove shibboleth and reportlib language files from language dirs and integrate translation strings into trad4all.inc.php - refs #4467

Revision 68dd30ea (diff)
Added by Yannick Warnier over 4 years ago

Remove references to reportlib and shibboleth language files - refs #4467

History

#1

Updated by Julio Montoya over 7 years ago

  • Tracker changed from Bug to Feature
#2

Updated by Julio Montoya over 7 years ago

I created a script in order to convert the main/lang/* files into po files. (see attached file, only works with english/spanish)

The problem that I'm facing right now is to replace ALL the get_lang("PlatformSetting") in the chamilo code with something like gettext("Platform Settings") or _("Platform Settings") in ALL PHP files and with all the variables in maing/lang/english. This will take a lot of time in order to develop/test a script that do that kind of replacement ...

#3

Updated by Julio Montoya over 7 years ago

well I have a new idea here:

We don't replace all get_lang variables everywhere in the platform we leave it like that.
So In my code I will have:

$var = get_lang("WelcomeUser");

but get_lang will changed:

so we are going to do this;

function get_lang($var) {
    // look for the english variable in main/lang/english/trad4all.inc.php for example
    $var = get_lang_from_lang_folder($var); // this variable will have this value: "Welcome user";
    // With this variable I can use gettext()!
    $translated_variable = gettext($var);
}

for new variables get_lang variables we will use directly gettext or "_()"

I'm going to do a fast test ...

#4

Updated by Julio Montoya over 7 years ago

  • Status changed from New to Needs more info
  • Assignee set to Yannick Warnier

I did some test and if we use gettext some things are going to change:

1. we can have multiple po files (admin.po, announcements.po, etc) but we are going to merge (using msgmerge, msgcat) all files in a single file called:
default.po or all.po.

2. gettext doesn't like repeated values so we have to have only one value everywhere. (using msguniq)

3. we are going to use both utilities, the chamilo get_lang and gettext

4. total revision of all main/lang/english/*.php files in order to have a clean english translation according to gettext

5. for new translations get_lang will received the variable ("My new variable in my new tool") if it doesn't found it he's going to send it directly to gettext if gettext found something in locale/es_ES/LC_MESSAGES/all.po it's going to answer with the translated variable, otherwise it's going to print simply "My new variable in my new tool"

All this tricks are needed unless someone wants to change every "get_lang" call in all the platform.

more info here: http://www.gnu.org/software/gettext/manual/html_node/Manipulating.html#Manipulating

#5

Updated by Julio Montoya over 7 years ago

Notes:

to view duplicates: msguniq LC_MESSAGES/announcements.po -d

to merge po files msgcat LC_MESSAGES/*.po > all.po

#6

Updated by Yannick Warnier over 7 years ago

I'll have to have a chat with you to review the details. In particular, the fact that there would be only one single language file seems a bit dangerous to me in terms of load on the application (some large files are only loaded when required, against a complete load for every page, so if we implement a single file we might need caching of some sort).

#7

Updated by Julio Montoya over 7 years ago

well, po files (human readable) generate mo files some kind of binary mode (see http://www.gnu.org/software/gettext/manual/gettext.html#Files)

There are also some server restrictions:

http://www.php.net/manual/en/gettext.requirements.php:

The GNU gettext library works on a per-process, not per-thread basis

http://www.php.net/manual/en/gettext.installation.php

Also we have to add gettext in the server configurations ...

#8

Updated by Julio Montoya over 7 years ago

Another solution is to use the translation Symfony2 component that actually right now reads arrays like us, see:

http://symfony.com/doc/current/book/translation.html

// messages.fr.php
return array(
    'Symfony2 is great' => 'J\'aime Symfony2',
);

I also heard they're going to provide gettext support in symfony version 2.1 or someday so we can rely on them, we can focus in other chamilo tasks instead of rewriting our core??

This is the ticket:
https://github.com/symfony/symfony/pull/634

#9

Updated by Yannick Warnier over 7 years ago

  • Target version set to 1.9.2
#10

Updated by Laurent Opprecht over 7 years ago

I noticed there are a lot of global variables - I mean really a lot. Looks like the translations variables are being made globals. Could we try to get rid of that for the next release? This is killing me when using the debugger. Not speaking of globals being a source of troubles.

#11

Updated by Julio Montoya over 7 years ago

yeah, it will be great if we use a symfony component to manage that, task moved to 1.9.X

#12

Updated by Laurent Opprecht over 7 years ago

I happened to notice that get_lang actually include language files several times. That is caches only the translation it found. So if you call

get_lang('text1')
get_lang('text2')

and that text1, text2 are included in the same file - let's say admin - it will be included twice, etc. Which happens to be slow.

Looking forward to move that to symphony.

#13

Updated by Yannick Warnier about 7 years ago

  • Target version changed from 1.9.2 to 2.0
#14

Updated by Julio Montoya over 6 years ago

I added a new command that export our lang to po files:

php console.php translation:export_to_gettext

Only works from english to spanish now.

The current format that launchpad accept is something like that:

Sample directory layout
We recommend the standard GNU Gettext directory and file layout:
  template1/template1.pot
  template1/es.po
  template1/de.po
  ...
  template2/template2.pot
  template2/es.po
  template2/de.po

https://help.launchpad.net/Translations/YourProject/ImportPolicy?action=show&redirect=Translations%2FImportPolicy

So I'm generating files like this for now:

temp/lang/accessibility/es.po
temp/lang/blog/es.po
temp/lang/trad4all/es.po

etc...

I did not investigate yet if that works.
I copy/paste a code I have and then transformed in a command.

See :
https://github.com/chamilo/chamilo-lms/commit/3a2cd26a40ecf4c99668b156d138d1f16c8c3b68

#15

Updated by Julio Montoya over 6 years ago

I found some notices because some languages contains arrays which is bad.

PHP Notice:  Undefined variable: langNameOfLang['ukrainian'] in /var/www/chamilogits/src/ChamiloLMS/Command/Translation/ExportLanguagesCommand.php on line 83
#16

Updated by Julio Montoya over 6 years ago

An example ...

#17

Updated by Yannick Warnier over 6 years ago

  • Status changed from Needs more info to Assigned
  • Assignee changed from Yannick Warnier to Julio Montoya

Any idea how to fix the array issue (there are not so many terms that use array, as far as I know that's just days of the week, months of the year and languages)?

I would be happy to put the first two cases (days and months) as different variables, but the languages list is much more... variable (although I'm not sure we actually use it - I think we get it from the name of the language itself in the language table, and we don't translate it).

#18

Updated by Julio Montoya over 6 years ago

I cant't configured correctly a project in launchpad.net. I create a project and try to test the import to "po files" then after some hours they deleted my project. I'm probably doing something wrong ...

#19

Updated by Julio Montoya over 6 years ago

The po files are build this way:

1. Chamilo

msgid "AllEvents" 
msgstr "Todos los eventos" 

2. Default way

msgid "All events" 
msgstr "Todos los eventos" 

But because we have this type of calls get_lang('AllEvents') we are forced to use the first method.

#20

Updated by Julio Montoya over 6 years ago

After uploading a file that contains some translations I have this message:

Thank you for your upload. 49 files from the tarball will be automatically reviewed in the next few hours. If that is not enough to determine whether and where your files should be imported, they will be reviewed manually by an administrator in the coming few days. You can track your upload's status in the Translation Import Queue

But I don't have a direct feedback if my translations matches the launchpad policy. I don't think they are going to accept translations like this:

msgid "AllEvents" 
msgstr "Todos los eventos" 

my demo project
https://translations.launchpad.net/testlms3/trunk/+imports

#21

Updated by Julio Montoya over 6 years ago

Yannick Warnier wrote:

Any idea how to fix the array issue (there are not so many terms that use array, as far as I know that's just days of the week, months of the year and languages)?

I would be happy to put the first two cases (days and months) as different variables, but the languages list is much more... variable (although I'm not sure we actually use it - I think we get it from the name of the language itself in the language table, and we don't translate it).

We don't need to translate those strings. We can use the Locale symfony component.
They already resolve that problem:

$formatter = new \IntlDateFormatter(\Locale::getDefault(), \IntlDateFormatter::NONE, \IntlDateFormatter::NONE);        
$formatter->setPattern("EEEE d MMMM Y");
echo $formatter->format(time());

// the result is:

viernes 14 junio 2013 ­

Because I set up my locale value as "es" in the translation service provider:


// Implements Symfony2 translator (needed when using forms in Twig)
$app->register(new Silex\Provider\TranslationServiceProvider(), array(
    'locale' => 'es',
    'locale_fallback' => 'es'
));

They use this format patterns

http://userguide.icu-project.org/formatparse/datetime for date formats

#22

Updated by Yannick Warnier over 6 years ago

Julio Montoya wrote:

The po files are build this way:

1. Chamilo

[...]

2. Default way

[...]

But because we have this type of calls get_lang('AllEvents') we are forced to use the first method.

The idea is that gettext originally uses strings in en_US to identify the translations. Considering this, a conversion script (to import them in the first place) should use the corresponding English translation, and not the variable name.
We can still put an intermediate script (for now) between the get_lang() and gettext(), to take the English string and then ask for the translation. I don't think there will be many overrides (if two strings with two different language variables have the same English translation, there will, but I don't think there will be so many).

#23

Updated by Julio Montoya over 6 years ago

  • % Done changed from 30 to 50

I sent some sample code because the old translations files (lang/english/trad4all.php, etc) didn't work with the new structure.

#24

Updated by Julio Montoya over 6 years ago

  • % Done changed from 50 to 70
#25

Updated by Julio Montoya about 6 years ago

The current state in 1.10 is:

Po files are generated in the main/locale folder with 1 file per language.

Example:

   main/locale/es.po
   main/locale/en.po
   main/locale/fr.po

The format is "CourseHome" => "Course home".
The idea to use "Course home" => "Course home" is nice but it needs to read all variables first from the old system (main/lang) in order to detect the correct "key" ("CourseHome" to "Course home").

In order to generate those po files:

1. Update the main/lang folder with the zip file generated by in https://translate.chamilo.org
2. Execute a command

   php console.php translation:export_to_gettext

3. The main/locale/* files are going to be updated. The command is hardcoded in order to update only spanish, english and french.

#26

Updated by Yannick Warnier almost 6 years ago

I'm currently uploading all translations (transformed into .po) to transdev.chamilo.org (a temporary install).

#27

Updated by Yannick Warnier almost 6 years ago

There was apparently a problem with the upload and some languages appear with all variables in English

#28

Updated by Yannick Warnier almost 6 years ago

  • Category set to Global / Others / Misc
#29

Updated by Yannick Warnier over 4 years ago

I removed a series of language files again, moving all variables that were not already defined to trad4all.inc.php.
Apparently this already freed around 4MB in language terms... (and I'm only about 20% into the process)

#30

Updated by Yannick Warnier over 4 years ago

My technique:

locally:
cd main/lang/english/
ls -lahS
find the smallest file
on the server:
mysql> select * from cda_language_pack; (identify the smallest file's ID)
mysql> select id, variable from cda_variable where language_pack_id = 64 and variable in (select variable from cda_variable where language_pack_id = 77);
* I remove or rename the duplicates in web (https://translate.chamilo.org/run.php?go=admin_browse_language_packs&application=cda)
mysql> update cda_variable set language_pack_id = 64 where language_pack_id = 77;
locally:
search for all references to the file and remove them:
rgrep -n "\$language_file" | grep 'thefilewithout.inc.php'
also check main/cron/lang (two langstats files there declare the use of the language files)
add the removed files to update-files-1.9.0-1.10.0.php
remove the files from git:
cd main/lang/
git rm */thefile.inc.php
git commit

#31

Updated by Julio Montoya over 4 years ago

  • Status changed from Assigned to Needs more info
  • Assignee changed from Julio Montoya to Yannick Warnier

All lang vars should be moved to trad4all isn't it?

#32

Updated by Yannick Warnier over 4 years ago

Yes. This is a long process because we cannot remove anything else than the duplicated strings, and we have to make sure these are really duplicates (sometimes they have the same variable name but the meaning is different).

If it was easier (no double-meaning duplicates), I could just do it in one single query :-)

#33

Updated by Yannick Warnier over 4 years ago

  • Status changed from Needs more info to Assigned

Already passed from 24M to 18M only cleaning a few variables, removing duplicates, and moving the rest to trad4all.inc.php.

#34

Updated by Yannick Warnier over 4 years ago

I removed about 14 more files now. There are only 13 left, but it took me about 6h for these 14 files and it's bound to take more for the last ones.

Current lang folder size: 16736KB

Solo quedan:

+----+-------------+--------+------+
| id | name        | branch | type |
+----+-------------+--------+------+
| 45 | reservation |      1 |    2 |
| 52 | install     |      1 |    1 |
| 57 | admin       |      1 |    2 |
| 61 | learnpath   |      1 |    2 |
| 63 | exercice    |      1 |    2 |
| 64 | trad4all    |      1 |    1 |
| 65 | gradebook   |      1 |    2 |
| 71 | hotspot     |      1 |    2 |
| 72 | wiki        |      1 |    2 |
| 73 | tracking    |      1 |    2 |
| 75 | document    |      1 |    2 |
| 79 | help        |      1 |    2 |
| 81 | survey      |      1 |    2 |

As mentionned by Pestr on IRC today, this actually breaks (in a way) possible updates of language files in 1.9.x. I told him this was an intermediate stage before moving to gettext. Apparently just overwriting the trad4all.inc.php (and leaving all the other files there) gives acceptable results in this case.

#35

Updated by Julio Montoya over 4 years ago

the "reservation" tool was removed.

#36

Updated by Yannick Warnier over 4 years ago

  • Status changed from Assigned to Needs more info
  • Assignee changed from Yannick Warnier to Julio Montoya

I've done all files that I could without too many doubts.

There are just three files left now:
  • trad4all
  • install (I'm not sure I can safely remove that one as it is included directly from the installer)
  • hotspot (we need to check how this file is included from the Flash code of hotspot)
After that, there are a few files left in main/lang/english/ that do not correspond to any file in translate.chamilo.org:
  • reportlib.inc.php
  • shibboleth.inc.php

Not sure if we can also remove these ones... any idea?

#37

Updated by Yannick Warnier over 4 years ago

For hostpot, I am removing the file from translate.chamilo.org but not from Git. This way, we'll maintain the translations we have there until we can switch to a non-flash-based plugin for image zones, but the file is not in our way anymore.

#38

Updated by Yannick Warnier over 4 years ago

OK, the final size for the lang dir is 15M (so we gained about 9M from 1.9.x, and that's taking into account the new variables that have been added since then).

Now we're ready to make a greater move and leave this task hanging until we release 1.10.x (but we could already start making the languages files into an array format if necessary).

#39

Updated by Julio Montoya over 4 years ago

  • Assignee changed from Julio Montoya to Yannick Warnier

This is great. :)

Next steps might be:

- Changing the php files to po files (I remember that I wrote a script for that).
- Rename files from main/lang/english/trad4all.inc.php to main/lang/all.en.po
- Move files inside the new structure "src/Chamilo/CoreBundle/Resources/translations"
- Implement loading translations in Chamilo using the "Translation symfony2 component"
- Fix sub languages.
- Upload po files https://crowdin.com/ and deprecate translate.chamilo.org
- (Optional, depends if we take this risk) rename variables from "SearchAllVariable" to "Search all variables".

#40

Updated by Yannick Warnier over 4 years ago

Reordering the steps a little:

  • Changing the php files to po files (I remember that I wrote a script for that).
  • Rename files from main/lang/english/trad4all.inc.php to main/lang/all.en.po
  • Move files inside the new structure "src/Chamilo/CoreBundle/Resources/translations"
  • Rename variables from "SearchAllVariable" to "Search all variables" (both in Chamilo code and in .po)
  • Implement loading translations in Chamilo using the "Translation symfony2 component"
  • Fix sub languages.
  • Upload po files https://crowdin.com/ and deprecate translate.chamilo.org
  • Communicate to everyone about the change and design a small guide (or record a video) of how to import the files into their Chamilo installation

Also available in: Atom PDF