Narro @ MozCamp

If you're attending this year's Mozilla Camp Europe on October 25/26 you can meet me there to talk, praise, criticise or make suggestions to improve Narro.

Lately I've been hearing about people discovering Narro by themselves and installing it successfully, but more than that, actually using it. This is great considering that I haven't made any changes recently. I've been using Narro myself for all the translations that I'm coordinating and some that others wanted to coordinate in Narro.

If I'd get some help from some php developers, that would be great, but the chance to find a php developer that is into translation seems quite low, but I'm still hoping.

See you in Barcelona!

Still here

Yes, I'm still here although it has been rather quiet. I've been busy translating and meanwhile I discovered that it's a bless to have an application that just works although I haven't worked on it for a month. Yes, I'm talking about Narro.

I've just returned to the code. I've started with several small fixes, like an issue on the login page. The big things that need to be done are still the improvement of the import process as people have been complaining that it doesn't work on new installations and there is no specific error and adding roles, since managing users is rather hard now with all these permissions. Yes, committing and retreiving from versioning systems is another big thing as well. I'd say it's the next big thing.

Upcoming: User roles, dashboard page and notifications

Thanks to a few people that gave valuable feedback and myself, I have many improvements planned for future versions. Among them are user roles, since the permissions are growing and it's getting hard to manage permissions for each user, a dashboard page that will replace the project list as a start page and improvements in notifications that should allow you to recieve just a digest message via RSS (or possibly email in the future) and see the whole bunch of modifications, no matter how large in Narro, nicely paginated.

For those of you using l10n.mozilla.org installation, note that lately the machine is slower than it used to be, other installations are just fine. I recommend that you find a place where you can keep your own Narro instance if possible.

Regarding integration with versioning systems, it will have to wait a bit, at least one version, because I need the features described above for the projects I'm translating. Remember, I actually translate, that's why I'm writing this application.

Version 0.9.2 is out

Here goes another Narro version. Several issues were fixed, many things were added. Among the biggest additions there is OpenId login, the possibility to debate a text and the fact that you can change the current translation language on any page.

If you're updating a database from version 0.9.1, use the database update script distributed separately.

I'm getting close to version 1.0.0 which would acomplish my dream of pushing the translations to the versioning systems through a button. I'll be able to focus just on translation, everything will be on the web, in the open, for everyone to see and anyone can help me translate. To achieve this, I'll considerably grow the import/export sections allowing more and more types of sources and destinations.

Among other cool things planned for version 1.0.0 there are glossaries per project and a page where you will see texts and associated suggestions for all projects. On this page you will be able to see where a text is used in all projects, or just in one, and how it is translated. So it's more or less a check for consistency.

Other low priority big additions are showing translations for similar texts and mass operations, like translating in the text list.

Discussions where they belong, in context

I've seen many translation communities using mailing lists to discuss terms. This leads to an email account pumped with messages that you will or not delete and a web archive somewhere that most of the people I met don't use. So valuable information is often lost.

I'm very happy to have the possibility to discuss texts right in Narro. These discussions are tied to the texts and every time a new project uses a text that was previously discussed, the discussion will be shown on the translation page.

You can follow these discussions with RSS. If you're not using RSS, you better start to. It's better to read something when you want instead of having it delivered in your email when somebody else wants. If you're worried about offline access, don't be. Every major reader caches the articles for offline use. I recommend Google Reader if you're already using Google products. There are other interesting feeds also, so if you're using Narro, check these out. Just click on the feed icon in the location bar.

In other news, Narro supports OpenId authentication and version 0.9.2 is on its way.

Many languages in many tabs

Some people expressed the need to work with different languages in different tabs. This was not possible, because the language was stored in session and if you tried this, you'd get some weird behavior.

Just back from the Firefox Summit, I spent some time to change this, so now you can select which language you are translating in at anytime and you can work on different languages in different tabs.

The one thing that's still stored in session is the user data, so you can't be logged in with two users at once in the same browser.

Another thing added is that now you can choose Narro's language independent of the language you are translating in.

There are lots of things that will be added, and a 0.9.2 release will be out soon.

The basics: users and texts

I'm beginning a series of posts that will explain how Narro works. Let's look at the Narro users.

Any registered user can translate. You could even let anonymous users translate, but it's better to get people committed. Their suggestion needs to be reviewed by someone with experience. So some people can validate translations. Right now, the anyone user is missing from existing translation processes. Usually there is one person translating and validating at the same time. So Narro lets anyone translate and someone with experience spends considerably less time just correcting or validating. This process ensures that any translation was reviewed at least once.


Narro stores a text that needs to be translated only once. One text can have many translation suggestions, not just one as usual so far. There's always room for improvement and some texts may be translated in various ways depending on the circumstances. One text may be used by many projects. So the same text present in many projects will have the same translation suggestions available to choose from.

Think of Narro's inside architecture like a language dictionary that's constantly growing. Or a translation memory if you prefer. It has the original text and several possible translations. You choose which translation to use in a particular context.

And we're off. Narro version 0.9.1

It's about time. To launch a version that everybody can easily use. This is no beta. This is Narro 0.9.1. I gave up on beta 2 because I really don't have a QA team to justify beta releases. So I'd rather release often.

Finally, Narro supports a different number of plural forms than the source language (usually english). Better yet, on import, based on the language's known plural forms, the correct number of plural forms are created even if they don't exist in the translations to import. This gives you extra quality. The PO header is filled with the last revision date set as the export date from Narro, the translator's name and email address from Narro, the correct plural form declaration, the language name and the project id if it's not defined. So you'll get a beautiful PO on export.

There are numerous other things that were added and fixed. Feel free to browse the checkin log.

I was thinking of adding some screencasts, but I'd have to upload some large videos because I'd need at least 800x600 so that you could notice something. So please visit https://l10n.mozilla.org/narro to see Narro in action.

To install Narro, download it, unpack it in your webroot directory and follow the instructions from the INSTALL file. If you succeed, please come back and post thoughts of joy. For frustrations and other issues, please use the narro-project google group.

Thanks to NGUYEN Manh Hung, Ehsan Akhgari, Huda Sarfraz, Adi Roiban and Jani Monoses for their help.

Beta 2 is getting closer

Things are stable so far. Beta 1 went out with some installation problems and import/export problems. All those that were reported were fixed.

I've spent most of my time adding minor things here and there, correcting bugs, so I'll release a Beta 2 at the end of this week. If it won't have serious bugs, version 1.0.0 stable will be next. Otherwise, Beta 3.

I know I really have to write a manual of some sort but I'm really busy coding, and people have understood so far the application without needing a manual.

On the left, you can see what projects are in the Narro installation used for Romanian language so far. You can see native OpenOffice format, Gettext, Dumbgettext (the one used by some mozilla sites, it's not a correct gettext, but hey, Narro deals with it), you can see even SVG and phpMyAdmin format there.

So you get the idea. Narro is meant to translate almost anything that is related to user interface.

First Beta 1 results

Beta 1 got some brave users who tried to install Narro on their own machines. Luckily they gave feedback too, so a first set of problems are set to be fixed in Beta 2.

The most obvious was the lack of support for system calls through php, so in Beta 2, the exec function will be replaced everywhere with php code.

If the proc_open function is disabled as well, instead of having the project import in the background with a nice progress bar, you'll have to wait for a page to finish loading, as usual on the web. Or import through the command line.

Some other bugs were fixed, I'm really really trying not to add more features to speed up a 1.0 stable release.

I had a few minutes to try my drawing skills and I have now a Narro logo, slogan and a cute favicon.

So, people hold on. For Beta 2.

Beta 1 seen in the wild

The day has come. You can take Narro for a spin on your own server. Download the archive and follow instructions from the INSTALL file inside it. Visit the Known Issues about Narro page. Report any other issues. Cross your fingers.

Remember that this is a beta version. Proceed with caution and have patience. Ask in the Narro google group for support because there's no help written yet as the application is growing. Expect some screencasts this week.

If you're a PHP developer and want to help, please say so in the group.

If you want to maintain your Mozilla translations on l10n.mozilla.org please contact me so I can give you necessary rights to do that. I'll have another post on this topic tomorrow.

Looking forward to the alpha release

I had a presentation on Narro on Friday and I received a first offer of help last week from Lourenzo Ferreira. This should make me more committed to update SVN more frequently and keep the progress visible there by keeping a ToDo page in the wiki updated. I'm planning a alpha release no matter what on July 1st. I'll post an article here and a tarball with some instructions on Narro's google code page.

I really hope this brings more help on the development part, since Narro needs more than my spare time because it's constantly growing.

Is it ready yet ?

I have now a system that is productive. I'm using it to translate OpenOffice, Firefox, Fedora and other projects.

I want a system that will help translation communities 100%. Usually people involved in software translation deal with the pain of multiple file formats to translate, the pain of trying to follow a glossary within a translation and the pain of discussing terms on a discussion list.

So there's still a glossary to add and the possibility of discussing texts, contexts and suggestions. Just think how useful it would be to see the discussion over a term right where you translate it.

Keeping translation quality high is of most importance to me. So there has to be a way of letting many people add suggestions. There is. But someone has to validate them through some means, so the person who validates needs a way to be notified that there are suggestions to validate or suggestions added to texts that were already validated, possible improvements. RSS is the simplest way, so that is one more thing that needs to be done.

There are very few dedicated people who translate. Most people come and go. There has to be a way of getting them back. So there has to be some competition involved. Keeping track of a user typing speed, number of characters typed, number of suggestions added, showing charts, giving prizes, giving boxes to integrate them into Facebook, personal blog, etc,; these are things that need to be done. Let's face it, no matter how easy the tool used to translate is, you will never be able to keep up with en_US with a few people.

Will all this change one day ? Will we all use XLIFF or something that would make translation easier? I doubt it. People like being different. I'm not counting on that. What I'm thinking is simple. I want a tool to feed it the english text to chew and get the translated text as quick as possible. I'll have it when it's ready.

RSS Feeds

At last, hundreds of texts and translations are ready to invade your feed reader !

For starters, there are two feeds, one with new texts that are added and one with new translation suggestions.

It's simple, there's just a link to the Narro installation.

Now I need to create a new page in Narro that allows you to see all original texts, no matter what project, a button to show all the suggestions, and a button to show all the projects that use a text.

After this, I can tie the RSS articles to this page.

And of course, add more and more feeds; who needs email anymore ?!

Until you wait, you can still subscribe to the current Mozilla translation feeds to see what new texts are added, no matter if you're using Narro or not for translation.

And you can subscribe to receive the translation suggestions.

Try it on the installation from l10n.mozilla.org.

Time stamps on actions

Adding time stamps in Narro kept me busy lately. Some users wanted to see the latest added suggestions. I wanted to move forward to provide RSS feeds with this kind of data. Like a feed with new untranslated texts. Or even a feed with new suggestions added.

So from now on, every type of user data that is added to the database is time stamped. There's a created field and a modified field. There may be a need of history in the future, but that's another table's job.

So what's next?

I'll have to add options to filter texts/contexts/suggestions by date.
The possibility to add comments to suggestions, contexts and texts.
Glossary maintenance and linking terms to texts.

Why isn't there a version you can install and use yet?

I wouldn't be able to handle all the feedback that this would generate. I'd rather lay low and let the small group of Romanian translators handle the testing and in time I'll have a stable product. Because you don't want another Pootle, do you ? You have that. But you want a complex application that allows you to manage your translation community better and faster. You're reading something I'm writing after 3 years of translating open source software. Be sure I know what's there.

The pain of gettext

Who would have thought ? I've done .dtd, .ini, .inc, .sdf and the popular gettext should have brought no problems.

So Narro likes to guess access keys. Because it's annoying to type _, &, ~ or something else in the middle of the word. So Narro deals with this by removing the annoyance and leaving the job of access key selection to the reviewer, the person who validates one of the suggestions for a text to translate.

There's even more, Narro looks for the access key in the original text and if it's found also in the translated texts, it automatically sets it. If it's not, it sets the first letter automatically.

In gettext access keys are usually preceded by & or _ . But I keep running into situations where there is a _ or & and it's not an access key like "HELP_TEXT here" or "Starting $SERVER_NAME". If you're reading this and you eat gettext for breakfast please help me find a secure way to determine access keys.

Sucess story

Firefox is closing up for RC1 and there's a lot of testing going on. I received a report that stated that on MacOS, the mac + Q combination didn't close the browser as expected.

So I immediately figured out this is about a command key, so I went off and looked in Narro 'Q' (the simple quotes do an exact search) in the original texts. Found it. It was translated as 'E'. Fixed it. Great, but I thought there might be more mistakes like this. So I looked up every context that has 'cmd' in it. Got 7 pages of command keys, access keys and texts. All I had to do is check each original text that has only one letter, remember I was looking for command keys. In the 7 pages I found about 20 command keys, checked them all. Now imagine how long would it take if I didn't have Narro.

With Narro it took 1 minute.

Project management

It's been two weeks. I had some work besides Narro, so the weeks went a little bit slower for Narro.

Still I got to do a lot of stuff:

• finally there's a project management page with export and import with nice progress bars
• there's a progress bar in the translation page too so you can see how you're going
• there are now three plugins: entity checker, punctuation checker and spell checker
• there's a new preference, where you can paste some characters that aren't on your keyboard; you'll see them when you translate and you can use your mouse to add them
• many thanks to Huda Sarfraz for his suggestions; I added some file icons to help you while browsing
• the comments from the dtd/ini/inc files are now visible together with the context (see an example)

I know I promised to lay low with the new features, but I just can't resist. But I'll leave the database structure alone until release and having that in mind, the current Narro installation for Mozilla projects will be stable, so have fun with it.

I'll run an import on Firefox CVS and Thunderbird CVS every night, so if you feel like giving it a try, just export the project in your language and compare it with what you got. Note that Narro will check if the access keys exist in the translated labels.

If you have comments, ideas or suggestions, don't be shy to share them.

I'm pushing hard for a first release in the begining of May. Until then I hope to add single file export/import, user management, user profile page with statistics and RSS notifications when new untranslated texts are imported.

Don't forget to poke the Narro installation to see what I'm talking about here. If you are experiencing long timings, it's because the server is pretty low powered and because that installation has over 40 languages in it, although Narro is meant to be installed for each localization team and you should have only the languages that you translate in or poke for ideas. Or just get a powerful server and put them all in.

XPI or CVS ?

I just discovered that there are some localization teams that have a language pack translated but don't have a CVS structure translated.

I remembered that before I got Romanian firefox released, I created a script that made the conversion from the CVS structure to the language pack and it was a nightmare.

Anyway, I think in Narro you should have the CVS structure. You could have the language pack structure and export that, but you do fight to get in CVS right ? So I assume that having in Narro a converter from CVS to XPI and XPI to CVS is a must, so I'll start working on it.

Upcoming release

I've decided to stop adding new features and start setting things up for a first public release. This upcoming release will have:

- only support for mozilla projects and narro itself
- user and permissions management
- voting for suggestions, editing, deleting, validating
- file browsing
- searching
- import only through command line
- export via web and command line
- entities checking, punctuation checking
- probably some other things I've forgotten

There are several functionalities that are there but I just don't have time to test them for this release:

- gettext po support
- native sdf open office support
- spell checking
- single file export/import
- import via web

How about the future ? Here are some features from the future:

- languages list page with translation progress
- rss/email notifications when new untranslated texts are ready to be munched
- an api for interacting with other tools, such as transifex
- translation statistics for users and profile (user's quality, speed, volume, etc)

So stay tuned for the release announcement.

What goes in, must come out

Import and export for the masses is now there. Just login or register and then login and you'll see those buttons.

Since I didn't receive feedback from all languages I've imported, I've decided to let you do the first import for your language. So go ahead and do it.

Things are pretty stable now and the next step is to pull a fresh cvs copy daily and let people use Narro as an alternative.

Until then, please take into consideration that nothing stays in Narro, so if you do use it, please export your work.

Happy testing!

Access Keys

So I initially started on to work on the export/import operations. Although they were working great, the code was commentless and needed beautifying. During that time I went on and made some modifications to the database. I think I have now a solid database structure, so the first release will be soon.

Another thing that I've discovered is that people who want to help translating do not know what access keys are. Or if they know, they don't look in the menu to see if there's a conflict. So I decided to adopt a new approach on this topic. The access key for the original text and the access key for the translated text are now tied to the context. Selecting the access key is up to the person who validates a suggestion for a context. Regular users will not have to type in "M&ove" but just "Move".



Also, I've changed the voting system. Users now vote for a suggestion only for that context. It made no sense seeing that a suggestion had 10 votes but still someone validated the one with no votes because in that context it was more appropriate.

Until now I didn't care about showing the dtd comments or ini comments in Narro. But since in Mozilla, the context is quite small, I've decided to import the comments too as context comments. You can opt in to see them by pressing a button.

I haven't posted yet any code on svn as I'm still cleaning up and don't want to put a mess in there.

I'll have something there by the end of this week.

Endurance test

Although I designed Narro to work with only a pair of languages, I received some feedback that some languages, like spanish for example have various forms that would like to be used on the same translation site.

On the other hand, the romanian translators want to see how was the text translated in a few key languages, like french, spanish, italian.

So now Narro supports multiple languages. It took me some time to change all the code, mostly carefully search and replace.

Even so I would not recommend keeping all the languages in one instance, I just don't see the point in that, this is a tool that is supposed to help people who translate from one language to another, so displaying 100 languages isn't productive.

Mozilla is kind enough to offer me a playground, so feel free to test the progress.
I've imported Firefox 3 into 57 languages as an endurance test and the speed is still good. Of course if you install only a few languages that you need you'll have better performance.

You might want to register first. Since it's a playground, by registering you'll receive super powers like delete, edit and validation rights.

Don't take anything for granted, everything you put in there is not going to stay. An official installation will be available soon in case you want to use Narro to translate software in your language.

Stay tuned until I add the export button so you can see how easy it is to manage your locale.

Code like crazy

That's what I do these days. Code like crazy. Mostly on Narro.

The first public release (a thing that is installable by anyone) should come in a month.

Now let me make this perfectly clear. This application is meant to help localization communities to do a better job, by allowing more people to translate, not for the developers to manage translations. That's the reason why it works only with one pair of languages. Original and translation. If you're already thinking at de-CH and de-DE, there are no worries. You can just take the suggestions from de-CH and import them into de-DE. Or the other way around.

When I started to write the application, I used Drupal's user management system because I just didn't need one of my own. I was thinking of an application that just did the job for me and only me. Since people got interested and wanted to use it, I thought Drupal was way too much overhead. I've got qcodo in there, I'll most probably use some ZendFramework as well so Drupal had to go.

I've wrote a simple user management system, login procedure, registration, permission and preferences. I still have to do a forgotten password thingie.

I've wrote a plugin handler and a first plugin. I'm still not sure about the need of these plugins, but hey, I need a special thing for Romanian, and I cannot distribute that for all of you. I have in the database s and t with comma under and I want people to be able to use my application with s and t with cedilla as well. Because we have people typing both of these caracters. Yes, it's madness, but I got that now in a plugin. Just delete it if you don't need it. Or maybe I'll add an active flag so it can be deactivated.

The application still works like the only working copy available online. Yes, that's the one with Drupal in it. Get over there and try it if you're curious. But I'm really using that one for the translation of those projects, so don't add thousands of suggestions, please. You can add suggestions without an account in the project named Firefox 3 (for Narro tests).

I am now working on the import/export procedures. I used to do this in the command line and I had cvs/svn commands there so I'll have to change this a bit. I need to build a page for import/export. Let me describe how would you import and export Firefox using Narro. I have a basic bash script to execute all these commands.

cvs -z3 -d:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co -r MOZILLA_1_8_BRANCH mozilla/client.mk
cd mozilla

Here I create a file .mozconfig with this in it:

mk_add_options MOZ_CO_PROJECT=browser
mk_add_options MOZ_CO_LOCALES=ro
mk_add_options LOCALES_CVSROOT=alexxed%gmail.com@cvs.mozilla.org:/l10n

I need to do the above only once, for my first checkout. Then, each time I want to import the changes from CVS in narro I execute:

make -f client.mk l10n-checkout
rm -rf ../l10n/en-US
make -f tools/l10n/l10n.mk create-en-US
make -f tools/l10n/l10n.mk check-l10n
cd ../l10n
rm -f mozilla.tar.bz2
tar cjf mozilla.tar.bz2 *

The next step is to run the php script to import mozilla.tar.bz2.

php importer.php --import-mozilla --project_id 2 --archive mozilla.tar.bz2

That's it. On my copy, I execute all the commands inside importer.php, but I cannot assume that you'll have cvs, make on the server where you're installing Narro. So you'll just do the cvs thing and produce the archive, then upload it in Narro. If you do have a server, well you'll just add a cron to do it all for you.

If I had to start a new translation, I'd just have to make an archive only with en-US in it.

The export operation is now done this way:

php importer.php --export-mozilla --project_id 2 --archive mozilla.tar.bz2

Just untar the archive over the l10n directory that you have. I am doing it automatically, the only thing I do is review the translation with a simple cvs diff command. It's not necessary, but I just like to check before I commit.

Now if all this sounds complicated, I understand. Mozilla promised me a server so I could install Narro there and you won't have to know nothing about cvs at all. You'll just push export and import buttons and that's it.

If you're a cvs freak and this sounds scary to you, I can give you more details in the comments upon request, as this is not the subject here.

Ok, that's it for today's briefing, see you in a few days with some pictures and demos and further development news.

Narro is now open

Here starts the adventure of my first open source project, Narro
If you're interested, please subscribe to this blog.