TMX export

Just added a new column that shows the number of translations for each language, and next to that number, there's a TMX button that allows exporting all the translations in a TMX file. See it here.

The Mozilla build system inside a Narro plugin

I just finished a Narro plugin that sets up the Mozilla build system.

What it actually does on import is:

1. clones or updates the appropiate Mercurial repository
2. parses l10n.ini files to build the en-US directory with files to localize

and on export:

1. if the project is not Firefox, copies the required translation files from the associated Firefox project
2. builds a languge pack
3. runs compare locales and interprets the output to build Narro links to the contexts with errors or warnings

Let's get into details. Here's how the project edit page would look in Narro for a Mozilla project with the Mozilla Build plugin activated:

You need the mercurial path to the proper repository. Common options are comm-central, comm-aurora, comm-beta, comm-release and mozilla-central, mozilla-aurora, mozilla-beta, mozilla-release.

You need to specify the internal application type, browser (Firefox), suite (Seamonkey), mobile (Fennec/Firefox Mobile), mail (Thunderbird), calendar (Sunbird).

Also need the release name, this is empty for central, aurora, beta or release. This is used to find specific l10n.ini files, e.g. l10n-aurora.ini.

If the project is Seamonkey, Sunbird, Thunderbird or Fennec/Firefox Mobile, then you need some folders from Firefox, that's why you need to do this association here.

It makes no sense translating dom and toolkit folders again if you already did that in Firefox.

Now here's how the export page looks after an export, if the Mozilla Build plugin is activated:

You can see at the bottom the output of compare locales, with a link to the text to correct.
Also, you can see the xpi language pack as well.

What's happening in the background is quite complex and very platform dependent (Fedora/Centos/Debian/Ubuntu only supported).

You need to have make, autoconf-2.13, python, easy_install installed on the server.

But the result is quite amazing, since you can get an xpi language pack directly from the browser, no Mercurial, no build commands, no terminal, just the browser.

I'm currently converting all l10n.mozilla.org to use this plugin, so expect possible problems today.

Goodbye ZendFramework!

I was using ZendFramework for OpenID, Caching and Session handling. I had to keep ~700 files for this, which was too much, so I just removed this dependency and added a lightweight library for OpenID support, used QCubed file caching and standard session handling. Eventually this brought more speed and less files, so it's a good thing.

Also, as a bonus, you can now authenticate with your Google account as well. Authenticating with OpenID or Google leads to a pretty ugly user name, so that takes me to adding a real name property to users.

Revisited form states since I've seen many errors reported on that. If you leave the page open with a formstate for a while and come back and your session expired, then the form state is invalid. Previously this lead to an exception and a error message to the user. This was replaced by a javascript alert message and a redirection to the project list, so that the user can start again with minimum disruption.

Oh yes, and if you're wondering how Narro looks, just check this blog's background ;^)

Last but not least, I really appreciate the patience that people have in helping me to fix bugs and am glad that so many people find my work useful.

47 issues until version 2

I managed to do some cleanup in the issue tracker and scheduled the issues for version 2, 2.1 and 3. There are 47 issues for version 2, I have to fix all those before I release version 2 to the public.

More details here: http://code.google.com/p/narro/issues/list?can=2&q=label%3AVersion-2

Vagrant file for a development version of Narro

I wrote my first Vagrantfile. If you don't know what vagrant is, you might want to read vagrantup.com first.

The idea is to get a vanilla Narro running in a virtual machine so you can easily test it. It was a good exercise to find Narro dependencies.

It's really easy to write the Vagrantfile and manifests, but it was a pain to get the image running until I read on the issue tracker that there's a problem with the network. So I choose to boot with the GUI on and if needed I login with vagrant/vagrant and do a sudo /etc/init.d/./networking restart.

The archive with the Vagrantfile, manifests and instructions is here.

I'll keep updating if needed.

Next step is to trim down the list of reported issues, then a 2.0 release.

Support for command keys added and wrapping up for a release

I've just added support for command keys in Narro. This will save you from translating about ~100 one letter texs.

I also reduced the number of filters found on top of the translate page and replace them with an advanced search button that makes them visible.

2.0 is getting pretty stable, so next I was working on installing Narro using vagrant. Not that easy, sometimes vagrant won't up, so I found it easier for now to just boot the virtual machine with VirtualBox and work from there.

Here are the steps that you need to do to get Narro running on the lucid32 box:

apt-get update

apt-get install php5 php5-cli php5-pspell php5-gd php5-mysql apache2 mercurial mysql-server

mysql -u root

CREATE DATABASE `narro` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
CREATE USER 'narro'@'localhost';
GRANT USAGE ON * . * TO 'narro'@'localhost' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0 ;
GRANT ALL PRIVILEGES ON `narro` . * TO 'narro'@'localhost' WITH GRANT OPTION;

hg clone http://code.google.com/p/narro /tmp/narro
mv /tmp/narro/narro /var/www/narro
rm -rf /tmp/narro
mkdir -m 777 /var/www/narro/data
chmod 777 /var/www/narro/locale

mysql -u root -D narro < /var/www/narro/narro.sql

In the default Vagrant file uncomment:

config.vm.network "33.33.33.10"
config.vm.forward_port "http", 80, 8080

After this you should be able to access Narro at http://33.33.33.1:8080/narro
Now I just need to create a Vagrant file for all these operations.

Suggestions or help is welcome.

Followup on Mozilla access keys

So I've managed to somehow reduce the access key - label link to these cases for Firefox:


                '/^(.*)\.accesskey$/' => array('.label', '.message', '.title', '.button', 'placeholder', ''),
                '/^(.*)Access[kK]ey$/' => array('Label', 'Text', ''),
                '/^(.*)\.accessKey$/' => array('.label', '.message', '.title', ''),
                '/^(.*)\_accesskey$/' => array(''),
                '/^(.*)Accesskey$/' => array(''),
                '/^(.*)\.access$/' => array('', 'Button'),
                '/^accesskey\-(.*)$/' => array('button-')

There still are 79 command keys and 98 other one letter texts; some access keys some settings.


Translating 177 one letter texts is no fun, so there's still some work to do to on these.

Access keys revisited

Today I'm working on developing the access key detection to make sure I'm catching almost every access key if not all.

So Mozilla uses a separate entity to assign an access key to a text:

file.label=File
file.accesskey=F

Where in gettext you have &File or _File

The problem is that there are inconsistencies, so you have to write code to cope with every situation.

There still is the problem of command keys which are something different, e.g.:

Open Ctrl+F

Here O is an access key, F is a command key.

You can edit the access key in Narro as you wish, see the screenshot below. However, to make it easier, there are some automated processes to select that so you don't have to worry about that.

Not sure if I should add a box like that for the command key, will think about it.

Where does performance come from

First of all, the machine has to be good. You may be dealing with millions of rows in a table with indexes. Don't install it on your laptop.

Here's a comparison between two machines with the same database (+4 million rows in table):

A virtual machine: https://l10n.mozilla.org/narro/translate.php?l=vi&p=&f=&t=1&s=&o=&h=1&m=10&i=0#i

First page load: 9 seconds
Average load when scrolling down: 10 seconds

Cheap shared hosting: http://narro.alexxed.com/lmo2/translate.php?l=vi&p=28&f=&t=1&s=&o=&h=1&m=10&i=0

First page load: 3 seconds
Average load when scrolling down: 2 seconds

If you're thinking of having multiple active languages and huge translation projects, it might be best to have separate installations per language.

What I can still try and will try to make it better is:

• have a table for each language rather than all languages in a table
• play with indexes and see what works best
  
• try PostgreSQL

Query ran for the tests above:

SELECT
*
FROM `narro_context_info` AS `t0`
LEFT JOIN `narro_context` AS `t1` ON `t0`.`context_id` = `t1`.`context_id` 
LEFT JOIN `narro_file` AS `t2` ON `t1`.`file_id` = `t2`.`file_id` 
LEFT JOIN `narro_text` AS `t3` ON `t1`.`text_id` = `t3`.`text_id` 
LEFT JOIN `narro_project` AS `t4` ON `t1`.`project_id` = `t4`.`project_id` 
LEFT JOIN `narro_suggestion` AS `t5` ON `t0`.`valid_suggestion_id` = `t5`.`suggestion_id`
WHERE (
   (
   `t0`.`language_id` = 60 AND 
   `t1`.`active` != 0 AND
    `t2`.`active` != 0
    ) AND 
    `t1`.`project_id` = 28 AND 
    `t0`.`has_suggestions` = 0
) 
LIMIT 20

Been there, done that ? Your comments would help.