Closing Narro

7 years ago I started this project that didn't have a name for a year. Mozilla asked me if I can make it open source, which I did 6 years ago, and for which I thank Mozilla for pushing me to create my first open source project and supporting me along the way.

I started this project because I couldn't bare to translate monolingual xml files side by side and keep them in sync with a diff viewer.

I continued this project because I had a dream: open translations.

Anyone can contribute, anyone can fork a translation; much like open source software. What if you wanted Firefox in Klingonian or you wanted to translate a particular word differently just because you liked it that way ?

The dream is still a dream, but for the last years, my job (director of software development at a translation agency) and my personal life (future wife and kids) have and will take most of my time, so I have to admit that I won't be developing Narro anymore, but would still want to see this dream come true at some point with or without my help.

Narro's code will remain open as is, with no support.

There were two instances of Narro maintained by me, http://tradu.softwareliber.ro and http://a.maimult.ro/lmo . I will take those down because they aren't used anymore. I exported the valuable data (translations) in tmx files, which is standard for translation memories. You can download them from here: http://alexxed.fedorapeople.org/narro/tmx and import them in Pootle.

If you're looking for alternatives to Narro, I suggest Transifex or Pootle. Both have right now most of the functionality of Narro and more.

If you ever used Narro, thank you for that, you made me feel awesome!

New look based on jQuery themes

Since I'm not that fond of designing, I'd rather use someone else's work if possible, so I changed most of Narro's UI to use native jQuery UI elements so that you can apply any jQuery UI theme to change Narro's look.

What you see above is the redmond theme, currently users can't change the theme, but that would be a nice feature to have.

Have a look at the other jQuery themes if you don't like the above ones.

What's been done so far

Performance improvements

Under the hood, Narro's core is actually the translation memory concept.
If you're not into programming, imagine yourself listening to hundreds of users talking in 80 different languages and trying to relate everything to English.
If you're in programming, imagine 80 texts longer than 255 characters for each English text stored in a database.
It's a continuous challenge.

Feedback

Getting feedback is important, but gathering feedback is more important. Sure you can have everyone post their worries and wishes on a list, but building on that is a separate task that you need to do before you can act on it.
Arki did a fantastic job getting that done and that helped a lot.

Documentation

I'm not a fan of reading documentation in general, I'd rather just browse it. Jeff wrote the Narro wiki page on Mdn.

UI improvements

Translate and reviewing (proofreading) are separate actions with separate UIs
Everything needed to work with a Mozilla repository is bundled in a plugin.
UI improvements
Work with several texts as once, scroll as much as you need to have a clear picture of what you're working with.

Core improvements

HTML support.
SVN and Mercurial commit support.

Server

The current server is slow. So slow that sometimes I can't connect through SSH. There's a plan to move Narro to another server, waiting for a bug to review security.

Facebook's translation app

Just had a look at Facebook's translation app. It has some good ideas that should be implemented in Narro:

• big numbers about your progress, e.g. added translations and votes; definitely stimulating
• click on the other translation suggestions to copy them in the translation box
• good way of using placeholders; really descriptive texts inside them rather than %S or %$1S; but this is determined by the developers that create them
• glossary right under the translation box
• definitely appreciate the fact that they manage to squeeze the translation app in the middle of the page; good design

Not so great ideas imho:

• the text comment is placed under the text to translate; although greyed out, I think it may lead to confusion on what to translate, the text or the comment
• vote / translate switch; clicking on translate hides the suggestion list and shows the translation box only

TMX export

Just added a new column that shows the number of translations for each language, and next to that number, there's a TMX button that allows exporting all the translations in a TMX file. See it here.

The Mozilla build system inside a Narro plugin

I just finished a Narro plugin that sets up the Mozilla build system.

What it actually does on import is:

1. clones or updates the appropiate Mercurial repository
2. parses l10n.ini files to build the en-US directory with files to localize

and on export:

1. if the project is not Firefox, copies the required translation files from the associated Firefox project
2. builds a languge pack
3. runs compare locales and interprets the output to build Narro links to the contexts with errors or warnings

Let's get into details. Here's how the project edit page would look in Narro for a Mozilla project with the Mozilla Build plugin activated:

You need the mercurial path to the proper repository. Common options are comm-central, comm-aurora, comm-beta, comm-release and mozilla-central, mozilla-aurora, mozilla-beta, mozilla-release.

You need to specify the internal application type, browser (Firefox), suite (Seamonkey), mobile (Fennec/Firefox Mobile), mail (Thunderbird), calendar (Sunbird).

Also need the release name, this is empty for central, aurora, beta or release. This is used to find specific l10n.ini files, e.g. l10n-aurora.ini.

If the project is Seamonkey, Sunbird, Thunderbird or Fennec/Firefox Mobile, then you need some folders from Firefox, that's why you need to do this association here.

It makes no sense translating dom and toolkit folders again if you already did that in Firefox.

Now here's how the export page looks after an export, if the Mozilla Build plugin is activated:

You can see at the bottom the output of compare locales, with a link to the text to correct.
Also, you can see the xpi language pack as well.

What's happening in the background is quite complex and very platform dependent (Fedora/Centos/Debian/Ubuntu only supported).

You need to have make, autoconf-2.13, python, easy_install installed on the server.

But the result is quite amazing, since you can get an xpi language pack directly from the browser, no Mercurial, no build commands, no terminal, just the browser.

I'm currently converting all l10n.mozilla.org to use this plugin, so expect possible problems today.

Goodbye ZendFramework!

I was using ZendFramework for OpenID, Caching and Session handling. I had to keep ~700 files for this, which was too much, so I just removed this dependency and added a lightweight library for OpenID support, used QCubed file caching and standard session handling. Eventually this brought more speed and less files, so it's a good thing.

Also, as a bonus, you can now authenticate with your Google account as well. Authenticating with OpenID or Google leads to a pretty ugly user name, so that takes me to adding a real name property to users.

Revisited form states since I've seen many errors reported on that. If you leave the page open with a formstate for a while and come back and your session expired, then the form state is invalid. Previously this lead to an exception and a error message to the user. This was replaced by a javascript alert message and a redirection to the project list, so that the user can start again with minimum disruption.

Oh yes, and if you're wondering how Narro looks, just check this blog's background ;^)

Last but not least, I really appreciate the patience that people have in helping me to fix bugs and am glad that so many people find my work useful.

47 issues until version 2

I managed to do some cleanup in the issue tracker and scheduled the issues for version 2, 2.1 and 3. There are 47 issues for version 2, I have to fix all those before I release version 2 to the public.

More details here: http://code.google.com/p/narro/issues/list?can=2&q=label%3AVersion-2

Vagrant file for a development version of Narro

I wrote my first Vagrantfile. If you don't know what vagrant is, you might want to read vagrantup.com first.

The idea is to get a vanilla Narro running in a virtual machine so you can easily test it. It was a good exercise to find Narro dependencies.

It's really easy to write the Vagrantfile and manifests, but it was a pain to get the image running until I read on the issue tracker that there's a problem with the network. So I choose to boot with the GUI on and if needed I login with vagrant/vagrant and do a sudo /etc/init.d/./networking restart.

The archive with the Vagrantfile, manifests and instructions is here.

I'll keep updating if needed.

Next step is to trim down the list of reported issues, then a 2.0 release.

Support for command keys added and wrapping up for a release

I've just added support for command keys in Narro. This will save you from translating about ~100 one letter texs.

I also reduced the number of filters found on top of the translate page and replace them with an advanced search button that makes them visible.

2.0 is getting pretty stable, so next I was working on installing Narro using vagrant. Not that easy, sometimes vagrant won't up, so I found it easier for now to just boot the virtual machine with VirtualBox and work from there.

Here are the steps that you need to do to get Narro running on the lucid32 box:

apt-get update

apt-get install php5 php5-cli php5-pspell php5-gd php5-mysql apache2 mercurial mysql-server

mysql -u root

CREATE DATABASE `narro` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
CREATE USER 'narro'@'localhost';
GRANT USAGE ON * . * TO 'narro'@'localhost' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0 ;
GRANT ALL PRIVILEGES ON `narro` . * TO 'narro'@'localhost' WITH GRANT OPTION;

hg clone http://code.google.com/p/narro /tmp/narro
mv /tmp/narro/narro /var/www/narro
rm -rf /tmp/narro
mkdir -m 777 /var/www/narro/data
chmod 777 /var/www/narro/locale

mysql -u root -D narro < /var/www/narro/narro.sql

In the default Vagrant file uncomment:

config.vm.network "33.33.33.10"
config.vm.forward_port "http", 80, 8080

After this you should be able to access Narro at http://33.33.33.1:8080/narro
Now I just need to create a Vagrant file for all these operations.

Suggestions or help is welcome.

Followup on Mozilla access keys

So I've managed to somehow reduce the access key - label link to these cases for Firefox:


                '/^(.*)\.accesskey$/' => array('.label', '.message', '.title', '.button', 'placeholder', ''),
                '/^(.*)Access[kK]ey$/' => array('Label', 'Text', ''),
                '/^(.*)\.accessKey$/' => array('.label', '.message', '.title', ''),
                '/^(.*)\_accesskey$/' => array(''),
                '/^(.*)Accesskey$/' => array(''),
                '/^(.*)\.access$/' => array('', 'Button'),
                '/^accesskey\-(.*)$/' => array('button-')

There still are 79 command keys and 98 other one letter texts; some access keys some settings.


Translating 177 one letter texts is no fun, so there's still some work to do to on these.

Access keys revisited

Today I'm working on developing the access key detection to make sure I'm catching almost every access key if not all.

So Mozilla uses a separate entity to assign an access key to a text:

file.label=File
file.accesskey=F

Where in gettext you have &File or _File

The problem is that there are inconsistencies, so you have to write code to cope with every situation.

There still is the problem of command keys which are something different, e.g.:

Open Ctrl+F

Here O is an access key, F is a command key.

You can edit the access key in Narro as you wish, see the screenshot below. However, to make it easier, there are some automated processes to select that so you don't have to worry about that.

Not sure if I should add a box like that for the command key, will think about it.

Where does performance come from

First of all, the machine has to be good. You may be dealing with millions of rows in a table with indexes. Don't install it on your laptop.

Here's a comparison between two machines with the same database (+4 million rows in table):

A virtual machine: https://l10n.mozilla.org/narro/translate.php?l=vi&p=&f=&t=1&s=&o=&h=1&m=10&i=0#i

First page load: 9 seconds
Average load when scrolling down: 10 seconds

Cheap shared hosting: http://narro.alexxed.com/lmo2/translate.php?l=vi&p=28&f=&t=1&s=&o=&h=1&m=10&i=0

First page load: 3 seconds
Average load when scrolling down: 2 seconds

If you're thinking of having multiple active languages and huge translation projects, it might be best to have separate installations per language.

What I can still try and will try to make it better is:

• have a table for each language rather than all languages in a table
• play with indexes and see what works best
  
• try PostgreSQL

Query ran for the tests above:

SELECT
*
FROM `narro_context_info` AS `t0`
LEFT JOIN `narro_context` AS `t1` ON `t0`.`context_id` = `t1`.`context_id` 
LEFT JOIN `narro_file` AS `t2` ON `t1`.`file_id` = `t2`.`file_id` 
LEFT JOIN `narro_text` AS `t3` ON `t1`.`text_id` = `t3`.`text_id` 
LEFT JOIN `narro_project` AS `t4` ON `t1`.`project_id` = `t4`.`project_id` 
LEFT JOIN `narro_suggestion` AS `t5` ON `t0`.`valid_suggestion_id` = `t5`.`suggestion_id`
WHERE (
   (
   `t0`.`language_id` = 60 AND 
   `t1`.`active` != 0 AND
    `t2`.`active` != 0
    ) AND 
    `t1`.`project_id` = 28 AND 
    `t0`.`has_suggestions` = 0
) 
LIMIT 20

Been there, done that ? Your comments would help.

How did I (almost) automate Mozilla's localization process using Narro

Initially, I do these manually:

• hg clone [mozilla-repo] mozilla-repo
  
• setup mozilla-repo/.mozconfig file and create mozilla-repo/../l10n and mozilla-repo/../build/
• depending on the server I might also need to adjust some environment variables

After this is done, luckily with aurora and beta we have stable branches so I just need to pull and update. There's a scheduled process that runs

• hg pull
• hg update -C
• python client.py checkout (for thunderbird / seamonkey )
  
• make -sf client.mk configure // not launching the whole build process, just getting the files needed to build a language pack

After this process runs, there's a Narro import process that processes the en-US files and imports the files that are changed since the last import for all projects, all languages. This basically takes formats like dtd, ini, properties, html, po and parses them to allow a Narro user to translate them by just entering the translation in a textarea.

Each time a user exports a project, an attempt to build a language pack is made. To do this, I do:

• cd mozilla-repo/../build/[component_name]/locales
• make -s langpack-xx

It would be a lot easier if I didn't have to pull the entire repository to build a simple zip file. No easy way around it, too many variables. This works and helps with testing.

Now if the Narro user is satisfied, can choose to commit to Mercurial. I managed to include that in Narro, although it's quite difficult since you have to commit with your SSH key. So you can do that in Narro too.

After this, the next step is the Mozilla dashboard and a nightly build. The Mozilla dashboard uses a python script, compare-locales that does some general quality checks that I wouldn't want to reproduce in PHP. So I can run compare-locales myself and not wait for the dashboard. This is quite complicated because compare-locales wants as parameters the directories that need to be translated, to have those automatically I would need to search for l10n.ini files and parse them. Right now I hardcoded them for the major projects (Firefox, Thunderbird) and it works fine.

Now I have to wait for the next nightly build, and I'll have that as a link on the project list thanks to a Narro plugin.

Once the nightly was tested, I can go to the Mozilla Dashboard and sign off with my latest Mercurial commit.

So the only thing that is not automated is the sign off; that's doable too, but not a priority.

What this complicated process gets me is a Narro instance that automatically picks up new texts and changes, I or whoever can translate them, export them and commit them without the need of having anything else than a browser.

Just having the possibility to translate, export and get a language pack that easy is gold to any newcomer.

Now I'm trying to remove the manual part from this process so you can add a new project from Narro without having to go into the console. Even if it's a one time process and it doesn't change that often.

Oh, don't think that other localization projects are different just because they use gettext. You'll always have at least the problem of getting used to SVN, CVS, Mercurial, bugtracker or whatever before you commit your translations. I can't change each project. But building a tool that can deal with these situations is something that I need and use.

A preview of the next version

There's been some heavy rewriting going on.

After you log in, you'll have all the permissions, so feel free to experiment, but with caution and expect failures.

You can pick your favourite Firefox extension, add a project from the project list and and use the xpi, as a source for texts and for translation (web link, or upload, whatever).

The speed is really improved and the translation process is really easier now. You just open the translation page, start translating and the translation is saved automatically as soon as you move to the next text. You can use the tab key to move forward.

Use scroll to load more texts. Or press the more button.

http://narro.alexxed.com/narro/translate.php?l=ro&p=10&f=&t=1&s=&o=&h=1&m=10&i=0#i

It's too complicated

I'm currently experimenting with important UI changes meant to simplify things. The ideas so far are:

- reduce the number of pages to the barely minimum; only one page for translating
- drop the pagination and load new content when scrolling down; it feels more natural to the user
- show only the original text and a text input for translating it; everything else only on demand

Any ideas or complaints are warmly welcomed!

Going to Mozilla Balkans Community Meet-up in Sofia

Thanks to Mozilla, I'll be in Sofia between 3 and 6 of June attending the Mozilla Balkans Community Meet-up.

Why do you have to go and make things so complicated?

Remember, localizers are usually volunteers. When I started Narro, my idea was to make their life easier by allowing them to receive translations from anyone and to pull/push translations without knowing SVN, Mercurial, or any other technical stuff.

The idea setup would be:

1. The localizer gets the translation files through Narro import
   
2. Translation begins (offline or online)
   
3. The localizer pushes the translation files through Narro export
   

And no, it's not that I'd like to introduce a new tool instead of running svn commit, it's just that I'd like to spare people of the nightmares of merging and maintaining branches when all they want to do is translate.

The current Mozilla setup is:

1. Pull the entire Mozilla repository (> 500MB)
2. Scan its directory for localization files and build through symlinks a folder structure that resembles the one that localizers need to push in their repository
3. Run an import from that folder
4. Translation begins
5. Translation is exported in XPI format for testing purposes
6. There's a script made by Axel Hecht, compare-locales that does some checks for validity, because translations can really break the product. To help, I'm running that after every export and post a link to a diff file that contains links to the texts that need fixing
7. Translation is exported in the structure that should be comitted to Mercurial
8. The localizer needs to have Mercurial through SSH with key authentication set up
9. The localizer commits the structure exported from Narro (hg pull .../xx-XX.zip && unzip xx-XX.zip && hg commit && hg push)
10. The localizer checks the tinderbox for any build failures
11. The localizer checks the dashboard for any problems detected by compare-locales
12. The localizer does a signoff (pushes a button for the revision he wants to release) in the dashboard

I may have forgotten some steps, but obviously this is not the perfect setup. I'm keen on helping making things easier as I'm a localizer myself but there are some blockers here:

1. I can commit via Narro, but that causes problems because several localizers commit with the same account. I can't let localizers commit with their credentials because that would mean uploading their private SSH key and that defeats the purpose of a SSH key. From my experience, getting rid of that is a real advantage as you often want to fix only a few strings while you're on the road and don't feel like hg pull, hg commit, hg push or don't have Mercurial installed or your SSH key set up.
2. I don't have access only to the localization files. I have to pull the entire repository.
3. To build a language pack I need the whole repository.

Oh, there are the web pages as well. Fortunately I'm close to handling those in Narro as well.

Now don't think that Mozilla is the only software that has this complicated setup. But this localizer tool should be able to cope even with this situation, because it's not a perfect world we live in. It's a challenge.

Scale up

Narro was designed with one pair of languages in mind. And at that time, running an import on a big project on a modern computer for half an hour was acceptable. The only performance goal was to have a maximum page loading time of 1 second. There was no goal set for the import process.

Back then: 1 target language * 1 big project * 1/2 hours

Times are changing and the instance hosted on Mozilla grew to 61 languages. For each language I run an import on 4 big projects.

Right now: 61 target languages * 4 big projects * 1/2 hours

As you see, running the import process in these conditions is almost impossible, so the much postponed step of optimizing the code is absolutely necessary right now.

Note that the import process is running in background and eating up all the CPU power available on the server.

Fortunately, the code is ready for such optimizations which are:

• cache database results whenever possible
• do stuff only when necessary
• use regular expressions only when really needed

So that's what I'll be working on before doing any release.

Is there anyone outhere ?

I know it's been over an year, but development is slowly but surely moving forward.
There's still a hurdle to pass, and that is building an upgrade procedure from 0.9.4, last stable to the upcoming 1.0.

Among the goodies added on the frontend is translations in lists aka mass translation and the ability to import and export from various sources, like another project, a directory, upload, Mercurial, SVN, and so on.

To quote someone famous, version 1.0 will be ready when it's ready, but this post is here to say that development will be accelerated in the next weeks.

---

---