47 issues until version 2

I managed to do some cleanup in the issue tracker and scheduled the issues for version 2, 2.1 and 3. There are 47 issues for version 2, I have to fix all those before I release version 2 to the public.

More details here: http://code.google.com/p/narro/issues/list?can=2&q=label%3AVersion-2

Vagrant file for a development version of Narro

I wrote my first Vagrantfile. If you don't know what vagrant is, you might want to read vagrantup.com first.

The idea is to get a vanilla Narro running in a virtual machine so you can easily test it. It was a good exercise to find Narro dependencies.

It's really easy to write the Vagrantfile and manifests, but it was a pain to get the image running until I read on the issue tracker that there's a problem with the network. So I choose to boot with the GUI on and if needed I login with vagrant/vagrant and do a sudo /etc/init.d/./networking restart.

The archive with the Vagrantfile, manifests and instructions is here.

I'll keep updating if needed.

Next step is to trim down the list of reported issues, then a 2.0 release.

Support for command keys added and wrapping up for a release

I've just added support for command keys in Narro. This will save you from translating about ~100 one letter texs.

I also reduced the number of filters found on top of the translate page and replace them with an advanced search button that makes them visible.

2.0 is getting pretty stable, so next I was working on installing Narro using vagrant. Not that easy, sometimes vagrant won't up, so I found it easier for now to just boot the virtual machine with VirtualBox and work from there.

Here are the steps that you need to do to get Narro running on the lucid32 box:

apt-get update

apt-get install php5 php5-cli php5-pspell php5-gd php5-mysql apache2 mercurial mysql-server

mysql -u root

CREATE DATABASE `narro` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
CREATE USER 'narro'@'localhost';
GRANT USAGE ON * . * TO 'narro'@'localhost' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0 ;
GRANT ALL PRIVILEGES ON `narro` . * TO 'narro'@'localhost' WITH GRANT OPTION;

hg clone http://code.google.com/p/narro /tmp/narro
mv /tmp/narro/narro /var/www/narro
rm -rf /tmp/narro
mkdir -m 777 /var/www/narro/data
chmod 777 /var/www/narro/locale

mysql -u root -D narro < /var/www/narro/narro.sql

In the default Vagrant file uncomment:

config.vm.network "33.33.33.10"
config.vm.forward_port "http", 80, 8080

After this you should be able to access Narro at http://33.33.33.1:8080/narro
Now I just need to create a Vagrant file for all these operations.

Suggestions or help is welcome.

Followup on Mozilla access keys

So I've managed to somehow reduce the access key - label link to these cases for Firefox:


                '/^(.*)\.accesskey$/' => array('.label', '.message', '.title', '.button', 'placeholder', ''),
                '/^(.*)Access[kK]ey$/' => array('Label', 'Text', ''),
                '/^(.*)\.accessKey$/' => array('.label', '.message', '.title', ''),
                '/^(.*)\_accesskey$/' => array(''),
                '/^(.*)Accesskey$/' => array(''),
                '/^(.*)\.access$/' => array('', 'Button'),
                '/^accesskey\-(.*)$/' => array('button-')

There still are 79 command keys and 98 other one letter texts; some access keys some settings.


Translating 177 one letter texts is no fun, so there's still some work to do to on these.

Access keys revisited

Today I'm working on developing the access key detection to make sure I'm catching almost every access key if not all.

So Mozilla uses a separate entity to assign an access key to a text:

file.label=File
file.accesskey=F

Where in gettext you have &File or _File

The problem is that there are inconsistencies, so you have to write code to cope with every situation.

There still is the problem of command keys which are something different, e.g.:

Open Ctrl+F

Here O is an access key, F is a command key.

You can edit the access key in Narro as you wish, see the screenshot below. However, to make it easier, there are some automated processes to select that so you don't have to worry about that.

Not sure if I should add a box like that for the command key, will think about it.

Where does performance come from

First of all, the machine has to be good. You may be dealing with millions of rows in a table with indexes. Don't install it on your laptop.

Here's a comparison between two machines with the same database (+4 million rows in table):

A virtual machine: https://l10n.mozilla.org/narro/translate.php?l=vi&p=&f=&t=1&s=&o=&h=1&m=10&i=0#i

First page load: 9 seconds
Average load when scrolling down: 10 seconds

Cheap shared hosting: http://narro.alexxed.com/lmo2/translate.php?l=vi&p=28&f=&t=1&s=&o=&h=1&m=10&i=0

First page load: 3 seconds
Average load when scrolling down: 2 seconds

If you're thinking of having multiple active languages and huge translation projects, it might be best to have separate installations per language.

What I can still try and will try to make it better is:

• have a table for each language rather than all languages in a table
• play with indexes and see what works best
  
• try PostgreSQL

Query ran for the tests above:

SELECT
*
FROM `narro_context_info` AS `t0`
LEFT JOIN `narro_context` AS `t1` ON `t0`.`context_id` = `t1`.`context_id` 
LEFT JOIN `narro_file` AS `t2` ON `t1`.`file_id` = `t2`.`file_id` 
LEFT JOIN `narro_text` AS `t3` ON `t1`.`text_id` = `t3`.`text_id` 
LEFT JOIN `narro_project` AS `t4` ON `t1`.`project_id` = `t4`.`project_id` 
LEFT JOIN `narro_suggestion` AS `t5` ON `t0`.`valid_suggestion_id` = `t5`.`suggestion_id`
WHERE (
   (
   `t0`.`language_id` = 60 AND 
   `t1`.`active` != 0 AND
    `t2`.`active` != 0
    ) AND 
    `t1`.`project_id` = 28 AND 
    `t0`.`has_suggestions` = 0
) 
LIMIT 20

Been there, done that ? Your comments would help.

How did I (almost) automate Mozilla's localization process using Narro

Initially, I do these manually:

• hg clone [mozilla-repo] mozilla-repo
  
• setup mozilla-repo/.mozconfig file and create mozilla-repo/../l10n and mozilla-repo/../build/
• depending on the server I might also need to adjust some environment variables

After this is done, luckily with aurora and beta we have stable branches so I just need to pull and update. There's a scheduled process that runs

• hg pull
• hg update -C
• python client.py checkout (for thunderbird / seamonkey )
  
• make -sf client.mk configure // not launching the whole build process, just getting the files needed to build a language pack

After this process runs, there's a Narro import process that processes the en-US files and imports the files that are changed since the last import for all projects, all languages. This basically takes formats like dtd, ini, properties, html, po and parses them to allow a Narro user to translate them by just entering the translation in a textarea.

Each time a user exports a project, an attempt to build a language pack is made. To do this, I do:

• cd mozilla-repo/../build/[component_name]/locales
• make -s langpack-xx

It would be a lot easier if I didn't have to pull the entire repository to build a simple zip file. No easy way around it, too many variables. This works and helps with testing.

Now if the Narro user is satisfied, can choose to commit to Mercurial. I managed to include that in Narro, although it's quite difficult since you have to commit with your SSH key. So you can do that in Narro too.

After this, the next step is the Mozilla dashboard and a nightly build. The Mozilla dashboard uses a python script, compare-locales that does some general quality checks that I wouldn't want to reproduce in PHP. So I can run compare-locales myself and not wait for the dashboard. This is quite complicated because compare-locales wants as parameters the directories that need to be translated, to have those automatically I would need to search for l10n.ini files and parse them. Right now I hardcoded them for the major projects (Firefox, Thunderbird) and it works fine.

Now I have to wait for the next nightly build, and I'll have that as a link on the project list thanks to a Narro plugin.

Once the nightly was tested, I can go to the Mozilla Dashboard and sign off with my latest Mercurial commit.

So the only thing that is not automated is the sign off; that's doable too, but not a priority.

What this complicated process gets me is a Narro instance that automatically picks up new texts and changes, I or whoever can translate them, export them and commit them without the need of having anything else than a browser.

Just having the possibility to translate, export and get a language pack that easy is gold to any newcomer.

Now I'm trying to remove the manual part from this process so you can add a new project from Narro without having to go into the console. Even if it's a one time process and it doesn't change that often.

Oh, don't think that other localization projects are different just because they use gettext. You'll always have at least the problem of getting used to SVN, CVS, Mercurial, bugtracker or whatever before you commit your translations. I can't change each project. But building a tool that can deal with these situations is something that I need and use.