Sven Fuchs

Organizing translations with I18n::Cascade and I18n::MissingTranslations

2011-02-11T00:00:00Z

When it comes to I18n one of the questions I get asked most often is how to organize translation keys. So I thought I’d write down how we’re doing it at work, and thus adva-cms2.

DRY does not apply to L10n

Before I get into this let me (once again) explain why DRY doesn’t apply to L10n though because that’s the reason why we allow for those deeply nested keys and namespaces in I18n.

Here’s a slide from my Anatomy of Ruby I18n talk at Euruko 2010:

class Internationalization < Abstraction
  def perform
    @developer.work!
  end
end

class Localization < Concretion
  def perform
    @translator.work!
  end
end

Internationalization refers to the work we do as developers. E.g. we extract strings from our code and make them available in translation files. Obviously the principle of DRY applies to this work in the sense that we don’t want to reimplement those portions of our code that actually looks up translations and stuff like that.

But DRY does not apply to Localization and therefor also doesn’t apply to our translation keys which can be seen as our interface to or contract with our translators. Instead, as developers our job is to pass control and enable translators to define different translations for the (seemingly) same key in different contexts.

For example, even if the string “edit” works as a translation in almost any context in English, that might not be true for other languages that have richer semantics and might have different translations for “editing” different things. As a developer we won’t ever be able to know these things in advance and so we just can’t predict which keys can be joined (“DRYed up”) and which can’t. :'post.edit' may or may not have the same translation in every target language as :'user.edit'.

Using I18n::Cascade

That’s one reason why the I18n API supports defaults. Using defaults we might express “either use an existing flash message for this particular model or use the common default message” like this:

I18n.t(:'flash.post.update.success', :default => [:'flash.update.success'])

This way translators can decide whether to use the default translation or specify a different one for this particular model. But obviously we don’t want to type that all the time.

Also, Rails’ translate view helper supports a great convention of automatically scoping translation keys to the current controller name and the current view/template name. E.g. the following lines will both look up the same translation key :'posts.show.edit':

# in posts/show.html.erb
t(:'.edit')
t(:'posts.show.edit')

Now, why not extend this convention so that the translate view helper always adds defaults so it effectively behaves like this call:

t('posts.show.edit', :defaults => [:'posts.edit', :edit])

Enter I18n::Cascade.

This helper module one of the lesser known modules which are shipped with the I18n gem. It can be included into compliant I18n backends and can be configured by passing a :cascade option per request.

Starting from version 0.5.0 the behaviour above can be achieved like this:

I18n::Backend::Simple.send(:include, I18n::Backend::Cascade)

# should really be a module, but i have no idea how/where to include it :/
ActionView::Base.class_eval do
  def translate(key, options = {})
    super(key, options.merge(:cascade => { :offset => 2, :skip_root => false }))
  end
  alias t translate
end

Now either of these keys could be translated and will be found … obviously starting with the most specific key :'posts.show.edit' and cascading down to the least specific one :edit:

# en.yml
en:
  edit: Edit
  posts:
    edit: Edit
    show:
      edit: Edit

I generally recommend using Rails’ controller/view scoping convention. Together with I18n::Cascade it is quite easy to provide some of the more common translations (like “new”, “edit”, “delete”) on common scopes but still allow translators to provide translations for particular view specific keys.

I18n::MissingTranslations

The i18n-missing_translations gem hooks into the I18n::ExceptionHandler class and logs I18n::MissingTranslationData exceptions. It includes:

an in-memory logger that simply holds missing translations during a request or test run and
a middleware that can be used to dump the contents of the logger to a file after each request

Pretty handy.

Using the in-memory logger makes sense, e.g., in the test environment. E.g. we could add this to the test_helper or Cucumber env:

at_exit { I18n.missing_translations.dump }

This simply outputs a YAML snippet for all translations that were missing during the test run which can be copied over to the translations file.

In development mode one rather will want to log missing translations to an actual translations file. The provided middleware does that by logging to a file missing_translations.yml in your locales dir (which is config/locales if present or the current directory otherwise). You can also pass the filename as an optional argument:

config.app_middleware.use(I18n::MissingTranslations) if Rails.env.development?

The middleware reads and writes per request. That means that on subsequent requests missing translations are added to the missing_translations.yml file. So if you go ahead and copy translations from the missing_translations.yml to your actual locale files you will also want to clear or delete missing_translations.yml.

Also note that Rails does not pick up new locale files between requests (I’d consider that a bug, in development mode it should pick them up). That effectively means that manual changes to a new missing_translations.yml file might be overwritten unless you restart the server. Thus your workflow for finding and moving missing translation keys might look something like this:

start the server
click around/work on stuff
check config/locales/missing_translations.yml
copy any missing translation keys to your actual locale files and correct the translations
delete or clear config/locales/missing_translations.yml
restart the server

By the way, starting from Rails 3.1 the translate view helper will use the :rescue_format facility from I18n 0.5.0 exception handling. This means that missing translations will be returned as essentially: keys.last.to_s.gsub('_', ' ').titleize wrapped into a span tag with a translation_missing class set. I.e. a missing translation for :'post.show.edit' will return “Edit”.

Hopefully these hints help working with translations a little bit.

Let me know what you think!

DatabaseRecorder - recording and replaying the database connection

2011-02-09T00:00:00Z

I’ve had this idea to record calls to the database connection for replaying “real” results later on a while ago. I’ve now found a few hours of time on the train and tried it out.

tl’dr: It doesn’t seem worth the effort. There’s no performance gain so far.

Here’s what I found:

Initial implementation

I’ve started out setting up a simple library database_recorder which wraps the execute method on the ActiveRecord::Base.connection into a capture method. The capture method will either record or replay.

If it records it will simply push the result of the current call to connection.execute (i.e. the results from the database) into an array. An at_exit hook will then marshal this array and store it on the disk.
If it replays it will (assuming the array has been unmarshalled and loaded before) shift a result set from the array and directly return it – not calling the database this time (this approach probably won’t work well with an async database driver).

There’s a simle convenience method that makes it easy to hook this up in a Cucumber env.rb:

# features/env.rb
DatabaseRecorder.record_or_replay!(ENV['REPLAY'])

Now one can control the record vs replay mode by setting the REPLAY env variable like this:

$ cucumber              ; record the db connection
$ REPLAY=true cucumber  ; replay it

After throwing up a first simple implementation I’ve tried this with the Cucumber feature suite of adva-cms2 which contains 92 scenarios with 869 steps and uses an SQlite3 database for the tests. I was surprised to see that quite some features actually passed already.

Tweaking it

A few more tweaks were necessary though:

Recording and replaying execute isn’t enough, one also has to take care of such methods as last_insert_row_id which sits on the native database connection object which only is accessible as an instance variable. The implementation of the ActiveRecord SQlite3 adapter makes it so that there’s no single method and no single collection of methods that could be recorded/replayed like this. Instead I had to wrap the execute method on the ActiveRecord::Base.connection and last_insert_row_id and changes on the native SQlite3 database connection object it holds. A little bit ugly, but it works (and once again proves that stuff like LoD is totally valid for Ruby/Rails code as well and one should care about it for widely used libraries).

Once I’ve figured this out almost all scenarios passed. The remaining ones were related to the users signup process.

It now occured to me that my initial assumption (of course) wasn’t accurate: an application won’t always pass exactly the same data down to the database even with tests always running in the same order and test data not being generated randomly but explicitely defined. There’s stuff like random tokens! Wow.

Ok, so I added a line Devise.stubs(:friendly_token).returns('12345') … which made all tests pass. Basically the revelation here is that in order to make this record/replay approach work one needs to make absolutely sure that data is passed to the database in a completely deterministic manner. This may or may not an issue depending on the app, testing philosophy/assumptions and other things.

Now all 92 scenarios were green!

Results

The db_session file that contains the recorded, marshalled database connection results is ~1.4 MB big … just saying.

When I now looked at the run times of both test suite runs I was shocked to notice that both took almost exactly the same time. In fact the test run which was recording the database connection was slightly faster than the one which replayed it. So far I have no clue about the reasons for that. I would think the code that actually “replays” results is simple enough (just doing Array.shift) to outrule the actual database connection. I might overlook something else (garbage collection? … was running on MRI Ruby 1.8.7) but so far it seems that recording test runs are consistently (very) slightly faster than replaying ones.

How disappointing, this would have been nice.

Maybe someone else will pick this experiment up and find something I’ve missed!

Otherwise, I guess, at least I’ve tried it and proved myself wrong ;)

Travis - a distributed build server tool for the Ruby community

2011-02-05T00:00:00Z

So, I’ve started playing with the idea of a distributed build server tool once again in last year’s December … and Travis is what I came up with.

Status quo

Travis is currently running on http://travis.heroku.com and already notifies me about builds that pass or fail when someone pushes to those repositories I’ve registered for testing purposes.
The builds run on a virtual server that Julian Fischer has very kindly granted for this experiment.
The frontend is implemented using Backbone.js.
Build output and status information is “tailed” from the workers to the frontend (i.e. browsers) using websockets via Pusher. It is also pushed back to application (running on Heroku) using a standard REST API.

All of these building blocks might change in the future, but here’s an overview of how they currently work together. And here’s a short screencast (the UI has changed a bit in the meanwhile but you’ll get the idea): 1:20 quick demo screencast

The vision

Maybe Jenkins CI (formerly Hudson) is the best open-source build server tool for Ruby projects out there today and everybody seems to be using it. Or maybe that’s just my impression. In any case my feeling is that it just isn’t a particularly good fit to the Ruby community for a couple reasons.

Instead, imagine a simple and slim build server tool that is maintained by the Ruby community itself (just like Gemcutter is, or many other infrastructure/tool-level projects are) in order to support all the open-source Ruby projects/gems we’re using every day.

Why not have a very slick application that loads off all the heavy work to workers – and have workers started on boxes contributed by the community itself. A lot of people have underused boxes idling at their offices or somewhere else. Why not make it easy for them to contribute run-time on these boxes for building open-source Ruby projects they use every day. I am sure many people would love this kind of opportunity of giving something back. On top of that one could even post a tiny message alongside with builds that ran on their boxes (a la “this build was kindly sponsored by [link to their website]”) to make this even more appealing.

A system that uses a client-side JS frontend, websocket messages and a tiny JSON API will make it easy for people to come up with alternate UI implementations. The current implementation of Travis should already allow building a custom dashboard or mobile app pretty easily.

I have no idea whether this will actually happen. But the vision of building an open-source build server infrastructure that is as much community-driven as Gemcutter is, allowing the community to contribute the required computing resources in order to build projects they love and use … really fascinates me.

Please join and help

My first goal with this experiment was to build a simple system that already makes sense as a stand-alone, privately maintained tool for running private builds in a closed environment and as such it seems to work quite fine so far.

Now I think the next step should be to get you involved :)

Please get on board and help pushing this project. E.g.:

Register to the Google group and share your opinion
Pop in to the IRC channel irc://irc.freenode.net#travis and say hi
Try out the application on http://travis.heroku.com, sign in and set up a service hook for some of your open Github repositories.
Review the code and tell how to improve, e.g., the client-side JS application, the EM-based worker code, the architecture in general etc.
Have a look at tickets filed on Github, discuss and help resolving them.
Share ideas about how to improve the UI design (I think the current minimalistic design works for now, but certainly isn’t good. I don’t consider myself a design person, so any input is highly appreciated)
Come up with ideas about how to implement a dynamic spin-up of workers on EC2 instances, VMs running workers securely on contributed boxes.
Build an alternate websocket service in case we can’t use Pusher.app in future (Travis currently uses a free dev account on Pusher.app and paid plans seem quite expensive).
Whatever else you’re interested in … :)

If you have any questions just post to the mailing list or drop me a note via Twitter, Github or [email protected]">email.

Some of the questions that I definitely need some input for are:

This is the first single-page Backbone.js app I’ve done. Does my interpretation of JS MVC make sense? Should I change the way views are instantiated and/or bound to events?
How to dynamically spin up workers in VMs on EC2 instances or other boxes?
How to improve the websocket message flow from workers via Pusher to the browser? E.g. sometimes messages arrive at the browser in the wrong order because they’re not synchronized or ordered in any way.

In any case your input is highly appreciated!

Why exactly don't we stub that database?

2010-11-24T00:00:00Z

Ok, when it came to unit tests I’ve been in the heavy-mocking-and-stubbing camp a few years ago. But then I’ve then seen our team (including myself) crash into the same issue over and over again: the fact that with mocks/stubs one does not test the real thing. One tests mocks and stubs which can easily get out of sync with the “real thing”, i.e. ActiveRecord models in our case.

So I’ve completely turned away from my previous position. Our tests became somewhat slower, hitting the database all the time, but they also were more robust, easier to read and easier to change. We could be pretty sure that our tests test the real application and things that were broken would actually be caught by the test suite.

Now while I’ve been watching Mike Perham’s talk about Scalable Ruby Processing with EventMachine something rang a bell for me. At some point Mike mentions that in order to write a non-blocking Postgres adapter he only needed to change a single method: execute on the Postgres adapter. For some reason the question popped into my mind if that couldn’t apply to mocks/stubs for unit tests as well.

Actually when we say that we “mock out the database” that’s not even close to what we’re really doing. In fact we are using an arbitrary object that pretends to behave the same way as the “real” one. That’s something completely different from really mocking the database if you think about it – and to me it seems this difference is the same difference that I’ve been experiencing when mocks became out of sync with the actual application: some logic in the model had changed but an according change to the mocks was missing. Because my “mocks” didn’t actually mock the database but the whole model.

So what if we’d use a record-replay approach instead that wraps around the adapter’s execute method and records responses from the database when our tests run for the first time. On subsequent requests it would not hit the database any more but actually mock it by replaying the results from the first run.

[Update:] VCR is a test helper library that does something similar for HTTP. The README says: “Record your test suite’s HTTP interactions and replay them during future test runs for fast, deterministic, accurate tests.”

Let’s assume we have this totally made up test. When we run it for the first time the following happens:

User.create!(:name => 'David')     # => pass the CREATE query to the db and record that it's in state abcd (hash from the query)
david = User.find_by_name('David') # => pass the FIND query to the db and record the result for state abcd
david.name = 'Josh'
david.save                         # => pass the UPDATE query to the db and record that it's in state 1234 (hash from the query)
david.reload                       # => pass the FIND query to the db and record the result for state 1234
assert_equal 'Josh', david.name    # => GREEN

Running the same test again would allow the execute mock to kick in, preventing the test from hitting the database:

User.create!(:name => 'David')     # => take a note that the db is in state abdc (hash from the CREATE query)
david = User.find_by_name('David') # => return the result previously recorded for this FIND query executed on state abdc
david.name = 'Josh'
david.save                         # => take a note that the db is in state 1234 (hash from the UPDATE query)
david.reload                       # => return the result previously recorded for this FIND query executed on state 1234
assert_equal 'Josh', david.name    # => GREEN

Obviously when we now make a change to either the application or test code that changes the interaction with the database then our recorded responses would become stale. In that case a query would get executed that has not been recorded before so we can stop the test suite from executing and ask the developer to re-record the results from the actual database first. We could even consider stopping the test suite, wiping out any results and re-run the test suite in recording mode automatically.

This way we would actually stub the database, not the model. Our tests would still go through all the low level ActiveRecord code that turns database result sets into model instances, which will be slower than using fake models but also will be much more robust and make our tests more useful.

The other question obviously is, how much run time could we spare this way? Obviously we’d need some kind of temporary place (maybe Redis?) to store those db states between test runs. So in the end we’d use one database to stub another one. Hu?

Let me know what you think.

Travis - an experimental, distributed CI server on Heroku

2010-06-16T00:00:00Z

Please note that this information is completely outdated.

If you've got here via Google searching for a distributed build server tool that runs on Heroku, then please refer to this article.

We had the I18n gem tested on runcoderun, a great service for open source projects to run continous integration on the web, and this was working very well for us. Unfortunately runcoderun was taken down.

So I figured it should be easy to setup a few apps on Heroku and have them run the I18n test suite on different stacks (i.e. Ruby configurations on Heroku). It turns out it is not, apparently. Or I'm just too stupid to do it. Anyway I though I'd write things down so maybe someone else can pick this experiment up and push it a few steps farther.

The current state is that Travis can automatically set up a single ci server and 3 ci runners on Heroku. The server app takes the ping from github and pings the 3 runners. Each of the runners forks and immediately returns from the parent (forking) process while the child process runs the build command and posts the result back to the server. The server stores the result and displays it on the builds index page.

Everything's cool except that the runner does not work properly on neither aspen-mri-186 (i.e. Ruby 1.8.6) nor bamboo-mri-191 (i.e. Ruby 1.9.1). It seems to work fine on the bamboo-ree-187 stack though and I will leave it running for a while so we can see how it works out over a few commits to the I18n repository.

Setup

Even though I've published a few versions of Travis as a gem it's probably easier for you to just clone or fork the repository and mess with it directly.

Take a look at the files in the ci/ folder. You can configure which repository you want to test, which stacks you want (possible values are 186, 187, 191) and which command you want to be run.

Once you've tweaked these values according to your library you can run

$ ./bin/travis install

This will take a while as it creates four applications on Heroku (if you choose to use all three Heroku stacks), sets up a temporary Git branch containing only the relevant files and pushes this branch to each of the Heroku apps. (You obviously need Heroku to be set up locally for this to work.)

The runner uses Bob to check out your repository to the tmp/ directory and run the test command that you've set. (Hence the name Travis. I found it's the most suitable cool name from the Bob the Builder series that fits a CI server library.) The server uses Datamapper for storing the build results.

You can destroy this setup by running

$ ./bin/travis destroy

The applications will be named like this:

http://ci-[your_repo_name].heroku.com
http://ci-[your_repo_name]-runner-[stack].heroku.com

E.g. for the I18n gem this would be:

http://ci-i18n.heroku.com
http://ci-i18n-runner-186.heroku.com
http://ci-i18n-runner-187.heroku.com
http://ci-i18n-runner-191.heroku.com

Testing

You can then go to the admin page of your Github repository and setup a post-receive hook by simply adding the ci server url to the "Post-Receive URLs" section.

Then you can test the whole thing by clicking on the "Test Hook" button. This will post a payload containing the latest commit information to your Travis ci server.

Afterwards the server app should display the test run on at least the 1.8.7 stack, something like this: http://ci-i18n.heroku.com

You can also take a look at the applications' log files by running

$ heroku logs --app ci-i18n (for the server)
$ heroku logs --app ci-i18n-runner-187 (for the 1.8.7 runner)

Problems on 1.8.6 and 1.9.1

I have no idea why this setup fails on aspen-mri-186 and bamboo-mri-191. Both work fine when I run the whole thing locally using Thin as a server and RVM for switching Ruby stacks.

But for some reason on the bamboo-mri-191 stack the only log output I get on the runner is the output from Bob (about fetching the master branch from the repository etc.) while on aspen-mri-186 there seems to be a segfault happening in eventmachine.

Most probably both failures have something to do with the fakt that the Travis runner process forks itself in order to immediately return from the POST request from the server then continue running the test suite in the child process. Maybe I'm doing it wrong? Or maybe it helps to use a Thread for this purpose?

The reasoning behind this asynchronous setup is that a synchronous setup would not work when any of the test suite builds takes longer than 30 seconds because Heroku seems to reap processes running longer than that. I've also already had trouble with the I18n test suite which takes ~ 10 seconds because these add up to > 30 seconds on three runners.

Help needed

I probably won't have the time to investigate this much further soon ... and I feel I might not be the right person either because I have no idea what might be going on on Heroku with the process forking.

Maybe someone with a stronger knowledge about low-level unixy things and the differences in various Ruby versions could help.

Anyway I'm putting this here for you to pick up. If we could get this thing working on at least the 1.9.1 stack this could be a great contribution to the Ruby community because currently there's no easy or cheap way to set up continous integration for open source projects on various Ruby stacks.

Feel free to contact me for any questions :)

Release your Gems with ease

2010-04-05T00:00:00Z

After the recent, somewhat heated discussion about whether or not to check in gemspecs to a repository, manually crafting versus autogenerating them, using a dynamic piece of code to collect relevant files versus maintaining a static files list in the gemspec file ... I've felt motivated to polish my own gem plugin that I've been using to publish my gems and finally add some of the functionality that I've been missing myself.

In short, I'm a happy resident in the camp of manually maintained gemspecs - as long as I have a dynamic snippet of Ruby code collecting my files for me. Just check in a well done gemspec and you're good to go. (If you're interested in more explanation then see below.)

So, here's my rubygems gem-release plugin.

It's pretty simple, could as easily be done in Rake (I don't like Rake though) or Thor and maybe only suits my own needs well. In fact, if you don't publish any gems that often or aren't familiar with the process I'd probably recommend doing all of this manually. Also, obviously there are other, mostly Rake based, tools like Jeweler, rakegem or even Hoe. And if your happy with Textmate snippets, then Mislav has one for you that generates a gemspec.

The gem-release plugin adds four commands to the gem command:

$ gem bootstrap       # bootstraps a new gem repository
$ gem gemspec         # generates a gemspec file with sane defaults
$ gem tag             # creates a git tag and pushes it to the origin repository
$ gem release         # builds the gem and pushes it to rubygems.org

For more a detailled explanation of these commands and their options see the README.

The bootstrap command can optionally invoke the gemspec command. And the release command can optionally invoke the tag command. So this is what my workflow looks like:

$ mkdir foo-bar; cd foo-bar
$ gem bootstrap --scaffold --github
# code away
# check the generated gemspec file and add/edit stuff
$ git add/commit/push
$ gem release --tag

# come back later ...
# code away, manually bump version in lib/foo_bar/version.rb
$ git add/commit/push
$ gem release --tag

The tool assumes that you're following the good practice of having a lib/[name]/version.rb that defines [Name]::VERSION (and it will add that file for you if you use the --scaffold option on gem bootstrap).

It also assumes that you want a git-based strategy for dynamically collecting files in your gemspec (i.e. `git ls-files {app,lib}`.split("\n")). If you pass the option --strategy glob then you get a glob instead (i.e. Dir['{lib/**/*,[A-Z]*}']). Obviously you're free to change that in any way you want (and I'll happily accept pull requests for different strategies).

Also, the tool currently assumes that you have a bunch of config variables defined in your global Git setup. I.e. it will populate:

authors  from `git config --get user.name`
email    from `git config --get user.email`
homepage from `git config --get github.user` (http://github.com/[github.user]/[gem_name])

And obviously it uses `git config --get github.user` and `git config --get github.token` to create a repository on Github when you pass the --github option to gem bootstrap.

Enjoy :)

PS:

In case your interested in this discussion at all here's my point of view.

I don't want to manually maintain a list of files. I'm a slacker, so that's just to cumbersome and errorprone to me. But I don't really want to use an external tool for maintenance (like Jeweler, which I've been using a lot before) as I tend to forget updating the gemspec before I push.

Also, given the assumption that it is ok to have dynamic code in the gemspec I don't think it makes a lot of sense to keep gem meta information in a Rake file instead of the gemspec (like authors, email, homepage ...). And I'm definitely not concerned with any left over swap files, .svn directories or other rubbish making their way into my gems because I just don't have them in the relevant directories.

So this is one of the rare cases where I think code generation is a good thing. I'll just have a tool to generate my gemspec once and then I can manually maintain it in case I have to add a dependency or something.

I can't see why I shouldn't be checking the file in to my repository this way either. It's not autogenerated in the sense that I'd use some external tool to rebuild it from a git pre-commit hook or something. Instead it's scaffolded just as any Rails app is.

One thing I find nice about this approach is that my stuff works without any dependencies. People who check out the repository will be able to just do gem build; gem install and they're done. The fact that I'm using a dynamic piece of code to collect the files means that I don't need to worry about the file being in sync with recent changes to the codebase.

Aligning rubygems + bundler through rubylib environment variable

2010-03-31T00:00:00Z

Put this into your ~/.zprofile (or ~/.profile or whatever makes sense for your shell):

export RUBYLIB=$RUBYLIB:~/.ruby

Source the file:

$ source ~/.zprofile

Put this to ~/.ruby/b.rb

begin
  # should probably check parent directories, too?
  require File.expand_path('../.bundle/environment')
rescue LoadError
  require 'rubygems'
  require 'bundler'
  Bundler.setup
end

Done.

You can now:

$ ruby -rb test/all.rb        # run test/all.rb in the context of your bundle
$ ruby -rubygems test/all.rb  # run test/all.rb in the context of rubygems

Experimental Ruby I18n extensions: Pluralization, Fallbacks, Gettext, Cache and Chained backend

2009-07-19T00:00:00Z

Backend::Simple < Backend::Base

In Ruby what is the most obvious, elegant and maintainable pattern to extend an existing class' or object's functionality? No, the answer to that is definitely *not* in using alias_method_chain. It's simply including a module to that class. You probably knew that already ;) We've done this with the I18n Simple backend before but one needs to extend the Simple backend first in order to then inject additional modules to the inheritance chain so that these modules' methods would be able to call super and find the original implementation. To make this a bit easier I've moved the original Simple backend implementation to a new Base backend class and simply extend the (otherwise empty) Simple backend class from it (see here and here). This way you now do not need to extend the Simple backend class yourself but you can directly include your modules into it:

module I18n::Backend::Transformers
  def translate(*args)
    transform(super)
  end

  def transform(entry)
    # your transformer's logic
  end
end

I18n::Backend::Simple.send(:include, I18n::Backend::Transformers)

I have no clue what your Transformers module could do exactly but that's the point about extensible libraries, isn't it? In any case this is simply the pattern that the new, experimental Pluralization, Fallbacks, Gettext and Cache modules use that I wanted to talk about :)

Pluralization

Out-of-the-box pluralization for locales other than :en has been recurring requests for the I18n gem. Even though it was easy to extend the Simple backend to plug in custom pluralization logic and there are working backends doing that (e.g. in Globalize2) there does not seem to be a point in still rejecting a basic feature like this from being included to I18n. I've thus added a Pluralization module that was largely inspired by Yaroslav's work. It can be included to the Simple backend (or other compatible backend implementations) and will do the following things: * overwrite the existing pluralize method * try to find a pluralizer shipped with your translation data and if so use it * call super otherwise and use the default behaviour One can ship pluralizers (i.e. lambdas that implement locale specific pluralization algorithms) as part of any Ruby translation file anywhere in the I18n.load_path. The implementation expects to find them with the key :pluralize in a (newly invented) translation metadata namespace :i18n. E.g. you could store an Farsi (Persian) pluralizer like this:

# in locales/fa.rb
{ :fa  => { :i18n => { :pluralize => lambda { |n| :other } } } }

And include the Pluralization module like this:

require "i18n/backend/pluralization"
I18n::Backend::Simple.send(:include, I18n::Backend::Pluralization)

We still have to figure out how to actually ship a bunch of default pluralizers best, but you can find a complete list of CLDR's language plural rules compiled to Ruby here (part of ours test suite).

Locale Fallbacks

Another feature that was requested quite often, too, is Locale fallbacks. Simple backend just returns a "translation missing" error message or raises an exception if you tell it so. It won't check any other locales if it can't find a translation for the current or given locale though. There were proposals for a minimal fallback functionality that just checks the default locale's translations if a translation is not available for the current locale. Globalize2 on the other hand ships with a quite powerful Locale fallbacks implementation that also enforces RFC 4646/47 standard compliant locale (language) tags. I've discussed this with Joshua and we've decided to extract a simplified version from Globalize2's fallbacks that makes the RFC 4646 standard compliance an optinal feature but still allows enough flexibility to define arbitrary fallback rules if you need them. If you don't define anything it will just use the default locale as a single fallback locale. Again enabling Locale fallbacks is just a matter of including the module to any compatible backend:

require "i18n/backend/fallbacks"
I18n::Backend::Simple.send(:include, I18n::Backend::Fallbacks)

This overwrites the Base backend's translate method so that it will try each locale given by I18n.fallbacks for the given locale. E.g. for the locale :"de-DE" it will try the locales :"de-DE", :de and :en until it finds a result with the given options. If it does not find any result for any of the locales it will then raise a MissingTranslationData exception as usual. The :default option takes precedence over fallback locales, i.e. it will first evaluate a given default option before falling back to another locale. You can add custom fallback rules to the I18n.fallbacks instance like this:

# use Spanish translations if Catalan translations are missing:
I18n.fallbacks.map(:ca => :"es-ES")
I18n.fallbacks[:ca] # => [:ca, :"es-ES", :es, :en]

If you do not add any custom fallback rules it will just use the default locale and the default locales fallbacks:

# using :"en-US" as a default locale:
I18n.default_locale = :"en-US"
I18n.fallbacks[:ca] # => [:ca, :"en-US", :en]

If you want RFC 4646 standard compliance to be enforced for your locales you can use the Rfc4646 Tag class:

I18n::Locale::Tag.implementation = I18n::Locale::Tag::Rfc4646

This will make a locale "de-Latn-DE-1996-a-ext-x-phonebk-i-klingon" fall back to the following locales in this order:

de-Latn-DE-1996-a-ext-x-phonebk-i-klingon
de-Latn-DE-1996-a-ext-x-phonebk
de-Latn-DE-1996-a-ext
de-Latn-DE-1996
de-Latn-DE
de-Latn
de

Most of the time you probably won't need anything like this. Thus we've used the (much cheaper) I18n::Locale::Tag::Simple class as the default implementation. It simply splits locales at dashes and thus can do fallbacks like this:

de-Latn-DE-1996
de-Latn-DE
de-Latn
de

Should be good enough in most cases, right :)

Gettext

The difference between the Gettext support and all the other extensions discussed here is that I haven't had a real use for it myself - so far, so please consider this stuff highly experimental. It shares the fact that people requested it in one way or the other though, so I'd appreciate feedback about it. Gettext support comes with three parts, only two of them being relevant to the user: * classical Gettext-style accessor helpers * a Gettext PO file compatible backend storage (currently read-only) * a few internal helpers To include the accessor helpers to your application you can simply include the module whereever you need them (e.g. in your views):

require "i18n/helpers/gettext"
include I18n::Helpers::Gettext

The backend extension is, again, a matter of including the module to a compatible backend (e.g. Simple):

require "i18n/backend/gettext"
I18n::Backend::Simple.send(:include, I18n::Backend::Gettext)

Now you should be able to include your Gettext translation (\*.po) files to the I18n.load_path so they're loaded to the backend and you can use them as usual:

I18n.load_path += Dir["path/to/locales/*.po"]

Please note that following the Gettext convention this implementation expects that your translation files are named by their locales. E.g. the file en.po would contain the translations for the English locale.

Translate Cache

Right from the beginning people benchmarked the I18n gem implementation and compared their numbers against native Hash lookups or other, less rich implementations of similar APIs. Also, thedarkone has experimented with a Fast backend that implements some significant optimizations. For the I18n gem itself we've refrained from overly extensive "early optimizations" and only applied some tweeks that made the implementation cheaper in obvious ways. This rather conservative approach actually paid out: the clean and readable implementation made the Simple backend refactorings and extensions (like those discussed here) almost trivial. On the other hand, even though calls to I18n don't actually add that much load to an application sparing resources obviously is an important concern. The probably both most effective and least intrusive way of doing that is simply caching all calls to the backend. Again you can include the Cache layer by simply including the module:

require "i18n/backend/cache"
I18n::Backend::Simple.send(:include, I18n::Backend::Cache)

As we do not provide any particular cache implementation though you also have to set your cache to the I18n backend. The cache layer assumes that you use a cache implementation compatible to ActiveSupport::Cache. If you're using this in the context of Rails this is a matter of one line:

I18n.cache_store = ActiveSupport::Cache.lookup_store(:memory_store)

Obviously this pluggable approach again allows you to pick whatever cache is most appropriate for your setup. For example ActiveSupport out of the box ships with compatible implementations for plain memory, drb demon and compressed, synchronized and plain memcached storage - wow. These options should be sufficient for 99% of all Rails apps. Whatever cache implementation you use the I18n backend cache layer will simply cache results from calls to translate (and will do the right thing with MissingTranslationData exceptions raised in your backend). For that it relies on the assumption that calls to the backend are idempotent: "A unary operation is called idempotent if, whenever it is applied twice to any value, it gives the same result as if it were applied once.". I18n's library design does not garantuee that by itself but in all practical cases will behave this way unless your doing really weird things with it. Basically, make sure you only pass objects to the translate method that respond to the hash method correctly. If you use custom lambda translation data make sure they always return the same values when passed the same arguments.

Chained backend

The Chain backend is another feature ported from Globalize2. It can be used to chain multiple backends together and will check each backend for a given translation key. This is useful when you want to use standard translations with a Simple backend but store custom application translations in another backends. E.g. you might want to use the Simple backend for managing Rails' internal translations (like ActiveRecord error messages) but use a database backend for your application's translations. To use the Chain backend you can instantiate it and set it to the I18n module. You can then add chained backends through the initializer or backends accessor:

# preserves an existing Simple backend set to I18n.backend
I18n.backend = I18n::Backend::Chain.new(I18n::Backend::ActiveRecord.new, I18n.backend)

The implementation assumes that all backends added to the Chain implement a lookup method with the same API as Simple backend does.

Cool, what's next?

If you're interested in any of these features, please try these things out and provide some feedback on the rails-i18n mailinglist. They might make it into the next I18n gem release or not depending on the amount of "real world feedback" we've gotten until then. Also, by now there's ton of good stuff that uses or extends I18n to do useful things with it on Github. I've collected a few things that I found particular suited for being evaluated, maybe merged and potentially included to I18n here: Interesting I18n repositories. I've also done some work on my i18n-tools repository recently, so you hopefully you can expect some news from that front, too.

Shameless plug

In case you find this stuff useful I'm always happy to receive another recommendation on working-with-rails :)

Ruby I18n Gem hits 0.2.0

2009-07-12T00:00:00Z

Here's the release announcement on the rails-i18n mailinglist:


I've bumped the gem version to 0.2.0 and tagged v0.2.0 today.

Most importantly it includes:

* Lambda support (aka Yaroslav-prevails edition)

* Custom separators (aka Gettext-p0wned edition)

* Ruby 1.9 interpolation syntax (aka Masao-Mutoh-rocks edition)

For more details check out the changelog.

Thanks to everybody involved!

We still haven't managed to get the inclusion of Rails' vendorized gem right. 
Thus it's currently not possible to use 0.2.0 in Rails easily.

As we've moved from 0.1.x to 0.2.x the pessimistic gem version operator 
"~> 0.1.3" used in Rails won't load a 0.2.0 version of the gem. I've 
added a Rails patch that relaxes this to use the optimistic operator 
">= 0.1.3" instead.

With that patch being applied we'll still have to sort out the Rubyforge 
release process. Josè Valim has volunteered for this and I've asked Matt to
add him to his Rubyforge project.

Meanwhile, with above patch applied, we could always clone the Github repo 
and build the gem manually of course. This should also work fine for playing 
with experimental builds in the future.

Lambda support

Changesets: e27771, c90e62, 9d390a

Lambda support has been considered a potential feature from the beginning because it adds the remaining ton of flexibility you need to support wicked things like Russian's irregular date formatting.

The shipped simple backend can store lambdas as translations when used with Ruby files. Other backends might implement different mechanisms.

E.g. you might store the translation:

:salutation => lambda { |gender| 
    gender == "m" ? "Mr. {{name}}" : "Mrs. {{name}}" 
}

This will, of course, resolve to either "Mr." or "Mrs." depending on the gender argument passed.

This is certainly a power feature and you need to use it wisely. As a rule of thumb make sure your lambdas are cheap and always return the same stuff when passed the same arguments.

Custom scope separators

Changeset: 5b75bf

This feature was a result of a discussion on the Rails core mailinglist about how easy it was to replicate Gettext's behaviour using the I18n gem. The general answer was: It's easy. But the nitty-gritty details are that I18n uses a period/dot as a scope separator so you can not use a full sentence as both a key and default value – as Gettext uses to do that.

So we've added support for customizing the scope separator both globally and on a per-request basis. You can:


# set a different scope separator globally:
I18n.default_separator ="\001" 
I18n.t("Foo. And bar, too.")

# pass it as an option to #translate: 
I18n.t("Foo. And bar, too.", :separator => "\001")

Yes. Of course the I18n API totally allows you to use Strings as keys. It will make more sense if you add a Gettext-like helper to access the backend though.

Symlinking translations

Changeset: 8c4ce3

After we've implemented the mentioned lambda support we've refactored some portions of Simple backend and and cleaned up the implementation. When we were done we accidentally noticed that we've also implemented support for a previously requested feature that we refer to as "symlinking translations".

So far we're not completely convinced that this should be made an official feature so we did not document it in the I18n module docs, yet. On the other hand people have asked for it and there's no reason to officially "disallow" it either. So we'll just mention it here and let people decide :)

You can now symlink translations by returning a Ruby Symbol from either your literal translation data or computed lambdas. So, for example:


	# yaml translation data
	actions:
	  edit: Edit
	articles:
	  actions:
	    edit: :"actions.edit"

	I18n.t(:"articles.actions.edit") # => "Edit"

Ripper2Ruby: modify and recompile your Ruby code

2009-07-05T00:00:00Z

So, the combination Ripper/Ripper2Ruby lets you do similar things as you can do with ParseTree or RubyParser and Ruby2Ruby. The differences are:

Ripper requires Ruby 1.9 (I was told it possibly could be compiled to work with Ruby 1.8.x but I don’t know anything further. Please drop me a note if you know how to do this.)
Ripper2Ruby builds a full object-oriented representation of Ruby code. That means you can modify the representation much more easily compared to the rough sexp tree that you get from the parsers. It also provides complete information about the node’s original source position, whitespace, comments etc.
Therefor with Ripper2Ruby you can recompile the exact copy of the original source code, character by character (that’s not possible with Ruby2Ruby). Ripper2Ruby has been tested with 225 Ruby libraries including Rails, Merb, Ruby Stdlib etc.
Ripper2Ruby does more but it’s slower, too.

For example:


src = "I18n.t(:foo)"
code = Ripper::RubyBuilder.build(src)
code.to_ruby # => "I18n.t(:foo)"

foo = code.select(Ruby::Symbol).first
foo.identifier.token = 'bar'
code.to_ruby # => "I18n.t(:bar)"

Ripper2Ruby was build to make it easier to create refactoring tools for Ruby/Rails I18n support (see i18n-tools). Huge thanks go (again) to Torsten Becker, Bestgroup Software & Consulting for making this possible.

Using Ruby 1.9 Ripper

2009-07-05T00:00:00Z

While Ripper parses your code it continously fires events (or “calls callbacks”) when it finds something interesting. There are two types of events: scanner (lexer) and parser events.

The scanner basically goes through the code from the left to the right character by character. When it finds known things (such as a keyword, whitespace or a semicolon) it fires a corresponding even that you can react to. The parser works on a higher level and watches for known Ruby constructs (such as a symbol, a method call or a class definition) and also fires events.

You can check the available events by outputting Ripper::SCANNER_EVENTS and Ripper::PARSER_EVENTS.

You can respond to these events by simply defining methods named :"on_#{event_name}" (omitting the @ character for scanner events). As long as you do not mess this up (which you might want to do) the parser always passes the results from the last inner parser events to the current parser event. E.g.:

require 'ripper'

class DemoBuilder < Ripper::SexpBuilder
  def on_int(token) # scanner event
    super.tap { |result| p result }
  end

  def on_binary(left, operator, right) # parser event
    super.tap { |result| p result }
  end
end

src = "1 + 1"
DemoBuilder.new(src).parse

This outputs:

[:@int, "1", [1, 0]]
[:@int, "1", [1, 4]]
[:binary, [:@int, "1", [1, 0]], :+, [:@int, "1", [1, 4]]]

When a scanner event is fired you can check the current position (it is passed to the event but you can also always call self.position) which allows for tracking detailled positioning information. Positions are given as [row, column] with the row being 1-based. On parser level events the current position is not very useful (and not passed to your event callbacks) because parser events are fired when the parser recognizes a known ruby construct as completed - i.e. at the end of the construct.

Scanner events are fired “just so”, i.e. the scanner finds something and calls your callback method. The return values might or might not be passed to parser events. Parser events otoh build a meaningful tree and their return values are always passed to the next (outer) event. You can generally think of events being fired “from the inside out”, starting with lowlevel scanner events.

You can examine the hierarchie of these events by doing:

require "pp"
src = "1 + 1"
pp Ripper::SexpBuilder.new(src).parse

will output:


[:program,
 [:stmts_add,
  [:stmts_new],
  [:binary, [:@int, "1", [1, 0]], :+, [:@int, "1", [1, 4]]]]]

You think of this as a nested method call where the first element of each array is the method name and the rest are the arguments. In the example above there would be 5 method calls. The first :@int call would receive the arguments "1" and [1, 0], the :binary would receive ["1", [1, 0]], :+, ["1", [1, 4]]. The other calls, like :program would not receive any arguments.

When executed the (theoretical) interpreter would first evaluate the innermost arguments, right? That’s exactly what Ripper does, too. It will first fire the first @int event, then the second one and then pass the return values of these two events (together with the :+ operator token) to the next outer method, which is the :binary event in this case.

(“Theoretical” of course refers to these particular s-expressions. There are languages that are very much based on exactly this concept, like e.g. Lisp.)

As you can see even though the scanner fires events on whitespace there aren’t any whitespace characters passed to any of the callbacks. I don’t know if there’s anything else happening to these but of course you can define callbacks for the different kinds of whitespace and do something useful with it. The same is true for comments and quite some stuff that doesn’t make a semantical difference in Ruby (such as parentheses for method calls etc.).

To examine all events in the order they are actually fired you can use the event log that ships with Ripper2Ruby:


src = "1 + 1"
Ripper::EventLog.out(src)

will output:


@int                1
@sp                 " "
@op                 +
@sp                 " "
@int                1
binary
stmts_new
stmts_add
program

I’m not an expert here but Ripper’s s-expressions and events seemed to make more sense to me than ParseTree’s stuff. Ripper still doesn’t seem to be completely consistent though.

E.g. for word lists (i.e. Arrays that are defined using %w() syntax) there are different events fired depending whether you have %w() or %W().

src = '%W(foo bar)'
pp Ripper::SexpBuilder.new(src).parse

outputs:


[:program,
 [:stmts_add,
  [:stmts_new],
  [:words_add,
   [:words_add,
    [:words_new],
    [:word_add, [:word_new], [:@tstring_content, "foo", [1, 3]]]],
   [:word_add, [:word_new], [:@tstring_content, "bar", [1, 7]]]]]]

But on the other hand:


src = '%w(foo bar)'
pp Ripper::SexpBuilder.new(src).parse

outputs:


[:program,
 [:stmts_add,
  [:stmts_new],
  [:qwords_add,
   [:qwords_add, [:qwords_new], [:@tstring_content, "foo", [1, 3]]],
   [:@tstring_content, "bar", [1, 7]]]]]

As you can see for qwords (i.e. the non-interpolating version) there seems to be a :qwords_add and :qwords_new event missing. I can’t see any good reason for this.

Also, Ripper seems to get the method call operator wrong when you use "::"


src = "A::b()"
pp Ripper::SexpBuilder.new(src).parse

outputs:


[:program,
 [:stmts_add,
  [:stmts_new],
  [:method_add_arg,
   [:call,
    [:var_ref, [:@const, "A", [1, 0]]],
    :".",
    [:@ident, "b", [1, 3]]],
   [:arg_paren, nil]]]]

Watch the period which should be a :"::" symbol.

In quite some situations I’ve found the events ambigous or not explicit. E.g. for the closing parentheses in a words list like %w(foo bar) Ripper fires a :@tstring_end event - which is the same event as it fires for closing parentheses in Strings as in %(foobar).

It gets really weird when you try to build something from the events that Ripper fires for Heredocs or even stacked Heredocs combined with method calls on the Heredoc opener token - maybe the most weird Ruby construct anyway. In general though this stuff is fun to work with and quite obvious once you got the idea :)

Rails I18n revs up: Globalize2 preview released!

2008-09-19T00:00:00Z

Please note: the following explanations assume that you're familiar with the new I18n API in Rails and might leave some unanswered questions otherwise :-). Also note that this is a preview release targeted at Rails I18n developers. We'll do at least one more release and provide more complete documentation about how Globalize2 can be used by end users then.

Globalize2 preview

The first preview release of Globalize2 includes the following features and tools. Most of them can be used independent of each other so you can pick whatever tools you need and combine them with other libraries or plugins.

Model translations – transparently translate ActiveRecord data
Static backend – swap the Simple backend for this more powerful backend, enabling custom pluralization logic, locale fallbacks and translation value objects
Locale LoadPath – easily load translation data from standard locations enforcing conventions that suite your needs
Locale Fallbacks – make sure your translation lookups fall back transparently through a path of alternative locales that make sense for any given locale in your application
Translation value objects – access useful meta data information on the translations returned from your backend and/or translated models

Also, we've put together a small and simple demo application for demonstrating Globalize2's feature set. You can find globalize2-demo on GitHub. Instructions for installation are included in the readme at the bottom of that page.

The implementation of Globalize2 has been sponsored by the company BESTGroup consulting and software, Berlin.

Installation

To install Globalize2 with its default setup just use:


script/plugin install -r 'tag 0.1.0_PR1' git://github.com/joshmh/globalize2.git

This will:

activate model translations
set I18n.load_path to an instance of Globalize::LoadPath
set I18n.backend to an instance of Globalize::Backend::Static

Configuration

You might want to add additional configuration to an initializer, e.g. config/initializers/globalize.rb

Model translations

Model translations (or content translations) allow you to translate your models’ attribute values. E.g.


class Post < ActiveRecord::Base
  translates :title, :text
end

Allows you to translate values for the attributes :title and :text per locale:


I18n.locale = :en
post.title # Globalize2 rocks!
I18n.locale = :he
post.title # ?????????2 ????!

In order to make this work you currently need to take care of creating the appropriate database migrations manually. Globalize2 will provide a handy helper method for doing this in future.

The migration for the above Post model could look like this:


class CreatePosts < ActiveRecord::Migration
  def self.up
    create_table :posts do |t|
      t.timestamps
    end
    create_table :post_translations do |t|
      t.string     :locale
      t.references :post
      t.string     :title
      t.text       :text
      t.timestamps
    end
  end
  def self.down
    drop_table :posts
    drop_table :post_translations
  end
end

Globalize::Backend::Static

Globalize2 ships with a Static backend that builds on the Simple backend from the I18n library (which is shipped with Rails) and adds the following features:

It uses locale fallbacks when looking up translation data.
It returns an instance of Globalize::Translation::Static instead of a plain Ruby String as a translation.
It allows to hook in custom pluralization logic as lambdas.

Custom pluralization logic

The Simple backend has its pluralization algorithm baked in hardcoded. This algorithm is only suitable for English and other languages that have the same pluralization rules. It is not suitable for, e.g., Czech though.

To add custom pluralization logic to Globalize’ Static backend you can do something like this:


@backend.add_pluralizer :cz, lambda{|c| 
  c == 1 ? :one : (2..4).include?(c) ? :few : :other 
}

Locale Fallbacks

Globalize2 ships with a Locale fallback tool which extends the I18n module to hold a fallbacks instance which is set to an instance of Globalize::Locale::Fallbacks by default but can be swapped with a different implementation.

Globalize2 fallbacks will compute a number of other locales for a given locale. For example:


I18n.fallbacks[:"es-MX"] # => [:"es-MX", :es, :"en-US", :en]

Globalize2 fallbacks always fall back to

all parents of a given locale (e.g. :es for :"es-MX"),
then to the fallbacks’ default locales and all of their parents and
finally to the :root locale.

The default locales are set to [:"en-US"] by default but can be set to something else. The root locale is a concept borrowed from CLDR and makes sense for storing common locale data which works as a last default fallback (e.g. "ltr" for bidi directions).

One can additionally add any number of additional fallback locales manually. These will be added before the default locales to the fallback chain. For example:


fb = I18n.fallbacks
fb.map :ca => :"es-ES" 
fb[:ca] # => [:ca, :"es-ES", :es, :"en-US", :en]
fb.map :"ar-PS" => :"he-IL" 
fb[:"ar-PS"] # => [:"ar-PS", :ar, :"he-IL", :he, :"en-US", :en]
fb[:"ar-EG"] # => [:"ar-EG", :ar, :"en-US", :en]
fb.map :sms => [:"se-FI", :"fi-FI"]
fb[:sms] # => [:sms, :"se-FI", :se, :"fi-FI", :fi, :"en-US", :en]

Globalize::LoadPath

Globalize2 replaces the plain Ruby array that is set to I18n.load_path by default through an instance of Globalize::LoadPath.

This object can be populated with both paths to files and directories. If a path to a directory is added to it it will look up all locale data files present in that directory enforcing the following convention:


I18n.load_path << "#{RAILS_ROOT}/lib/locales" 
# will load all the following files if present:
lib/locales/all.yml
lib/locales/fr.yml
lib/locales/fr/*.yaml
lib/locales/ru.yml
lib/locales/ru/*.yaml
...

One can also specify which locales are used. By default this is set to "*" meaning that files for all locales are added. To define that only files for the locale :es are added one can specify:


I18n.load_path.locales = [:es]

One can also specify which file extensions are used. By default this is set to ["rb", "yml"] so plain Ruby and YAML files are added if found. To define that only *.sql files are added one can specify:


I18n.load_path.extensions = ['sql']

Note that Globalize::LoadPath “expands” a directory to its contained file paths immediately when you add it to the load_path. Thus, if you change the locales or extensions settings in the middle of your application the change won’t be applied to already added file paths.

Globalize::Translation classes

Globalize2’s Static backend as well as Globalize2 model translations return instances of Globalize::Translation classes (instead of plain Ruby Strings). These are simple and lightweight value objects that carry some additional meta data about the translation and how it was looked up.

Model translations return instances of Globalize::Translation::Attribute, the Static backend returns instances of Globalize::Translation::Static.

For example:


I18n.locale = :de

# Translation::Attribute
title = Post.first.title  # assuming that no translation can be found:
title.locale              # => :en
title.requested_locale    # => :de 
title.fallback?           # => true

# Translation::Static
rails = I18n.t :rails     # assuming that no translation can be found:
rails.locale              # => :en
rails.requested_locale    # => :de 
rails.fallback?           # => true
rails.options             # returns the options passed to #t
rails.plural_key          # returns the plural_key (e.g. :one, :other)
rails.original            # returns the original translation with no values 
                          # interpolated to it (e.g. "Hi {{name}}!")

Other notes

Please note that the Globalize2 Static backend (just like the Simple backend) does not support reloading translation data.

The Future of I18n in Ruby on Rails - RailsConf Europe 2008

2008-09-06T00:00:00Z

Welcome everybody!

I'm very happy to talk about "The future of Internationalization in Ruby on Rails" today.

So, who’s me?

This never occurred to me before but in a recent RailsEnvy podcast I’ve learned that my name could be pronounced like this ...

If you want to google me ...

... you probably have better luck using this.

I'm currently living in Berlin and I love it :)

I started programming in 1984

I've been in business as a developer, mostly as a web developer for some years and switched to working with Ruby and then Ruby on Rails a couple of years ago.

Some of you might know me because I've led the Rails I18n group for the last couple of months and did the final touches on the implementation of the new API in Rails recently.

Some people know my blog because of some tutorials I've written about Globalize.

And I’m currently working on a CMS application platform called adva_cms.

Of course, as a German developer I've always been interested in support for Internationalization so I'm really happy that I’ve able to contribute something in this area to Ruby on Rails.

If there are any questions please try to make a mental note and keep it for the end. I've set some time aside for Q&A at the end of the talk.

Ok, by the end of this talk I hope you’ll have a clear picture of what the Rails Internationalization project is, what’s included in the new Rails Internationalization API and what’s not and how you can use it.

I will give a very brief demo. Of course this is a huge topic so I’ll point you to where you can find more resources.

Then we have a short and sweet Interview video prepared with Joshua Harvey the original author of Globalize

And finally I’ll give you some hints about what we’re going to do in the future and how you can contribute to that.

So, to better understand the future of Internationalization in Rails I want to contrast that a bit with the history that we all suffered through. As you know the history of Internationalization in Rails is really, really dark.

These were times of chaos.

They were unclear, unpredictable, unstable and highly adventurous.

You'll see the reason for that in a minute. Let me first let you show how many different solutions we've already seen and how wide their variety is.

We have seen things like ...

Ruby Gettext - the dinosaur and grandfather of Internationalization

David’s original Localization plugin as well as ...

Thomas Fuchs’ Localization plugin - both very simple

GLoc - one of the first bigger implementations

Globalize - the only solution with solid Model translations

Globalite - a very slick solution by Matt Aimonetti

Simple Localization - also a very interesting solution by Stephan Soller

Both have contributed a lot to the Rails I18n project by the way.

Gibberish - maybe the most recent full solution, came along as the cool kid on the block.

... and many more.

There are solutions for special needs like ...

Localizing Rails to like Swedish, Brazilian, Korean and so on.

There are plugins that add functionality to other plugins. For example there is Jibberish that adds a Javascript Layer on top of Gibberish. And things that just adds separate, useful tools.

Stuff like that ... of course there’s much more. I just wanted to give you an impression of these things.

So when you take into account how complex our languages are and how many funny problems there are to solve and we have lots of lonely programmers working on this and everybody focusses on the features he actually needs himself ...

What do you think will you get?

What you get is a huge mess.

We have lots of isolated, incompatible solutions.

Things like support and documentation just don’t get better with a lonely developer solving their own needs and then - of course - often walking away and focussing on something completely different.

Then, all of our solutions basically were huge Monkeypatches to Rails with the usual problems resulting from that.

And all of us re-implemented the wheel in one way or the other. So, in a way, we wasted lots of resources.

But - on the other hand - this situation was actually necessary for us to experiment with solutions and learn how Rails Internationalization could be done.

Because of this situation last year in September a group of developers from several Rails Internationalization plugins gathered for a chat session. We agreed that we wanted to implemented a Rails core patch which should solve the problems we had with this stuff.

We invited developers from all existing solutions that we knew and started meeting regulary, collected requirements, discussed our ideas ...

Actually there was really a lot of discussion and initially it was hard to agree on anything.

You know that sort of situation. Everybody brings their own favorite little features to the table, everybody has different beliefs of what's good and what's necessary. So this discussion went on endlessly and in the beginning of 2008 our work slowed down. Apparently we really needed a creative break and some time to rethink everything.

Then, in May 2008 we got back to the drawing board with a fresh approach ...

... and now we were able to agree on a very slim and lightweight implementation that covered all of our requirements

It works as a Ruby gem which means it is not only suited for use in Rails but also other frameworks.

We wanted only minimal changes in Rails, we wanted it to add as less load to Rails as possible and of course we still wanted it to implement a common API where everybody could build on.

So, at this point Jeremy Kemper didn’t really know about our work but in a RailsConf Portland RailsEnvy interview he predicted for Rails 2.2 that ...

... Internationalization will be solved.

When we asked Jeremy about this he said it was a joke

So we think Jeremy should definitely crack more jokes about things like this.

Because in the end this joke actually got us going to finally propose our work to the core team.

In hindsight there were a few key decisions. Before we managed to agree on these we had not been able to get anything really done. These key decisions were:

We agreed to make the actual implementation of all the logic exchangeable, so you can always plug in something more powerful or more flexible if you need to.

We radically sticked to the principle of “doing the simplest thing that ever could work” for Rails.

And maybe most importantly we just stopped aiming for a solution that did anything else than the bare minimum that’s needed for Rails.

So, what we now ship with Rails is a common API for Internationalization which is basically just a bunch of definitions how things are supposed to work. So this is just a Ruby module with some public methods on that.

These methods delegate to a backend that actually implements the API. The backend that is shipped with Rails is intentionally named the Simple backend because the only requirement for the Implementation in the Simple backend is that it works for American English because that’s what we need for Rails to keep it working like it worked before.

That means: the most important parts that previously were hardcoded right into Rails are now abstracted out of the code and moved to a simplistic backend, accessible through just a few public methods.

So this is true

While this ... might work. Maybe you can localize Rails to your favorite language if it is very similar to English. There are quite some resources out there that show how to do this. For everything else we still need to add plugins or other libraries.

For Rails itself we settled with shipping the translations as YAML files and with stock Rails you can also use Ruby files, so you can have lambdas in your translation data.

I’ll run through the main features that this API provides.

As I said the API is provided as a Ruby module with two main methods on it.

These are:

I18n.translate which you can use to look up translations from the backend and for convenience it has an alias #t and you need to pass it at least one parameter which is the key.

So this would look up the translation that’s stored for the key :rails in the current locale, for example, by default American English.

The other method that you’ll use regularly is localize.

This one basically allows you to format Date and Time objects for a certain locale.

Again, for convenience there’s an alias ... of course that’s #l and you pass it the object that you want to localize

So the #translate gives you very powerful means to look up translations.

You can look up your translations using a Symbol as a key and this is what we’d generally encourage you to do. But you can also use a String.

Also, you have scopes, so you can namespace your translations and structure them in a reasonable way.

For example in the ActiveRecord translations YAML file there’s a scope activerecord and then there’s a scope errors, a scope messages and finally there’s finally the translation for the key :invalid.

So scopes are basically groups or namespaces of translations. You can think of them as nested Hashes

Consequently there’s a way to look up whole namespaces - or groups - of translations. For example this will look up all error messages in ActiveRecord and return a Hash with all those messages.

Then there’s bulk lookup. That just means that you can pass an Array of keys you want translated and it will return you a corresponding array of translations.

And finally, of course, there’s defaults so you can specify what happens when a translation can not be found.

For example when you give it a String as a default and the translation can not be found it just returns the String.

But when you give it a Symbol it will translate this Symbol and give you that translation as a default.

Finally you can specify an array of Symbols and/or Strings and it will just check everything in that order and return the first value that’s not nil.

Interpolation is another feature.

That means you want to abstract your translations in certain situations. For example you have a translation for the key :message and there’s variable :name in that. Then you can pass the value of that variable to #translate and it will parse the value into the translation.

This looks trivial but it’s actually a very important feature because concatenating translations is kind of an anti-pattern in many situations.

Also, the API provides means to define Singular and Plural translations of course.

For example you could have a Singular and a Plural translations like this. Then you can pass a count option to #translate. So in this case it would pick the Singular translation because count is 1. And when you pass it 2 it would pick the Plural translation.

So that’s it already - all very basic stuff of course. No worries, you can look up everything online :-)

Like I said, the I18n API ships with a Simple backend ...

... which implements all of these features for American English only. So for example you can’t use it to do pluralization for Czech because they have different rules for pluralization.

As I just showed you your English translations include something like this.

But in Czech you have two plural forms - one of them is called Paucal. They use the singular just like one does in English. Then they use the Paucal, the second form, for everything between 2 and 4. And the actual plural for everything else.

So the Simple backend can not do that but we can easily swap it and the Internationalization API actually encourages you to do so.

You would just create a class for your custom backend and maybe extend the Simple Backend, implement your custom logic and tell the I18n module to use it.

That’s it.

Basically this is what future plugins will do a lot. For example there definitely will be a backend that stores translations to the database and I’m sure there’s one soon to read and store them from and to Gettext files.

So let’s have a really brief look how you can use this in your application.

So, if you’re actually working on an application today, what can use use?

Of course this really depends on your requirements. So let’s have a look at some scenarios ...

Let’s say you just need a few static translations and maybe some date and number formats. Then you can easily reuse the Simple backend that Rails ships with. By now there are a couple of demo applications on GitHub that show how to do that.

Let’s say you have a language that requires different pluralization rules. Then there’s an experimental Pluralizing backend for that where you can hook in pluralization logic as lambdas.

Let’s say you need to translate your application to Russian, or Czech or another language that has some more complex date formats.

For that there’s an experimental Chain backend where you could hook in a custom backend tailored for just these languages’ date formats.

Also there’s already a full plugin solution by Yaroslav Markin from Russia that implements things like these and I think it’s easy to adapt it for other languages.

If you need model translations there’ll be Globalize2. Globalize2 will look completely different to Globalize. It will be pretty lightweight and unobtrusive. Joshua will tell you more about that in the interview.

Also there’s an alternative approach, that’s linked on rails-i18n.org

So, if we still have to rely on external code, like plugins, to localize our applications.

What do we win?

The short answer to that clearly is: we win a lot!

First of all, we don’t need no monkeypatching any more

Well, of course we still will monkeypatch Rails to experiment with new features.

But we definitely won't need to do probably 99% of what we’ve done before like translating ActiveRecord error messages, translating all the helpers that Rails provides or anything like that.

Also, because we now can build on a common API we expect things to become much better exchangeable.

We will aggregate common locale data over time. So we will build up a Rails specific pool of Localization knowledge.

Also we hope that plugins will be much more focussed on implementing certain atomic features.

This is actually something we encourage everybody to do. You know, so far, you could not possibly combine three, four, five different plugins and expect anything to work. We hope we’re now able to do a better job here in future.

So, of course, with all of this stuff released our next development cycle has started ... right now.

Of course we’re all going to implement our stuff next. In fact by now there are already quite some useful things out there. During this time we will try to stay in touch with each other and then, again, collect everything, review it, again, discuss it and finally aim for another core patch.

The reason why we want to first collect everything and look at it as a whole and then move it to Rails as a rather complete package is that many of these things make most sense in context with each other.

We hope this will then result in a landscape of Rails Internationalization solutions where we have a distributed toolbox of focussed plugins rather than those huge solutions trying to solve everything under the sun at once that we had before.

We’ll have more compatible solutions and maybe the most exciting thing is that we will get to have Rails specific, common locale data.

So, let’s see what Joshua Harvey has to say about Globalize 2.

By the way the guitar shop that you can see in the video is actually the shop rashgash and the website at rashgash.com is the reason why Joshua initially wrote Globalize as he mentions in the interview.

So this is a brief overview of features that Globalize 2 will provide.

Of course there’s model translations - kind of Globalize’s USP in the past. For View translations we’ll basically use the Rails I18n API. There will be some useful tools like standards compliance tool for locales which we’ll need for locale fallbacks and stuff like that. A locale load_path tool that helps loading translation data more easily.

Actually the implementation of Globalize 2 is sponsored by the company BESTGroup Consulting & Software from Berlin. So, thank you, BESTGroup for making this possible.

Also, Metaversum, a company also from Berlin will start sponsoring us right after this conference. So, thank you, Metaversum in advance.

And actually we are looking for more sponsors not only for the Globalize project but also for the Rails I18n project because we really want to put this project on a really solid foundation.

For example we want to have somebody working on stuff like documentation on a regular basis and we want to provide better support for newbies on the mailing list and other stuff that is all usually much easier to do when you have a small budget to spend.

So if you’re interested in sponsoring a really cutting edge and high-profile Ruby on Rails Open Source project that already gets tons of attention and definitely will get even more attention in the next couple of months ... just catch me afterwards and we can talk about that.

So, how can everybody in this room get involved?

Of course everybody can just start playing with all of this, register to our mailinglist and give feedback. We need tons of that.

You can share your translations. We already have a repository on GitHub for that, so if you’ve done translations for Rails please submit them.

You can build plugins focussed on certain features that are still missing. And please don’t forget to pitch them on the mailinglist and our wiki.

You can help improving the Rails integration. You know Rails helpers are there and they are working. But that doesn’t mean they’re already perfect regarding Internationalization. So you can help to improve that.

And of course, you can sponsor our work so we can invest some money in things that are hard to do otherwise.

So, basically this was my talk. We’ve collected pretty much everything I mentioned in this talk as well as lots of other resources on http://rails-i18n.org. So, look it up.

Thanks to everybody who contributed!

Ok, so what are all those questions you want to ask?

Ruby on Rails I18n, RailsConf Europe and Globalize2

2008-08-28T00:00:00Z

RailsConf Europe talk: "The future of I18n in Ruby on Rails"

I'll be giving a talk about "The future of I18n in Ruby on Rails" on RailsConf Europe next week together with Marko Seppae.

Am I excited about it? You bet! Will that be an interesting talk for everyone involved into Rails I18n? Of course, I'm sure :)

Also, there are some plans to also work on I18n/Rails during the Code Community Drive but I'm not sure what the status of this is right now.

I'm personally planning to take care of another project in the same context already: adva-cms. So I might not be able to put a great amount of effort into preparing a I18n/Rails workshop here. I'd be absolutely happy to help with it though, of course, if somebody wanted to jump at it.

So, if you're going to go to RailsConf Eu next week and are interested in these things be sure to drop me a note and/or catch me at Bratwurst on Rails (that's the awesome socializing event the evening before the actual RailsConf.)

Changes to the I18n gem library

Since the I18n API and the integration to Rails got merged back to Rails edge we've receive quite a lot of feedback from people trying the I18n API and Simple backend. This resulted in that a few things have been changed. E.g.:

The Simple backend is now a class. This makes it more easy to reuse its features and only overwrite a certain method (like, e.g., for more flexible pluralization).
Pluralization data is now expected to be provided as a Hash using keys such as :one, :few, :many, :zero, :other like defined by CLDR.
A method #load_translations has been added that takes a source for translation data (such as yml and rb files) and loads translations from there.
The scopes for translations provided by Rails have been cleaned up to be more consistent.
The code should work with Ruby 1.9 now.

Globalize2 under heavy development

A couple of weeks ago Joshua Harvey, Marko Seppae and I have started implementing Globalize2 which turns out to be an extremely interesting project because with the new I18n foundation it now looks completely different, very slick and nice.

As far as we can tell right now Globalize2 will be much more of a toolbox of small tools where you can pick what you need. ActiveRecord translations will be solved unobtrusively. We'll support a good part of RFC4646/47 compliance for Locales and use localization data for formats etc. from CLDR which is quite a big thing, in my opinion.

We now have a website at rails-i18n.org

You probably already found it because I've been throwing the link around everywhere lately but it might still be worth mentioning that we now have a website at http://rails-i18n.org which is, obviously, dedicated to I18n on and for Ruby on Rails. Right now we're basically collecting resources on the Wiki and I hope to post some news to the blog every once in a while.

If you're interested to publish any I18n-related blog posts over there - just let me know.

Btw http://rails-i18n.org is, of course, also driven by adva-cms which is the CMS project I've been working on for the last couple of months. Check it out!

Finally. Ruby on Rails gets internationalized

2008-07-18T00:00:00Z

In hindsight we've initially tried to accomplish way to much. Everybody brought their experience and thinking about "good I18n practices" to the table - which proved extremely valuable because it forced everbody to push their own horizon. But it also resulted in something that would have been "just another Rails I18n solution" ... build right into Rails. As such it would not have fully satisfied every one of us. Too heavyweight, too complicated, just too much of everything.

So with the beginning of 2008 our work slowed down, good new ideas kept popping up but eventually the project completely stalled and people focussed on other business. Apparently we needed a creative break.

Then in May we went back to the drawing board and came up with a fresh approach based on our previous experience. Now all of a sudden we were able to agree on a very slim and minimal implementation of our concepts that covers all of our requirements: it works as a Ruby gem and is suited not only for use in Rails, it adds only minimal load to Rails core and it still implements the common api that will allow I18n solutions to build on.

To me this project already proved extremely interesting and educating. Almost all of the ideas that are now implemented have been already present in our discussions last year. But even though we've went through lots of lots of discussions we initially just haven't been able to shave the whole yak fur down to this minimal level.

I guess all of us learned a great deal about I18n. In fact some of us (including me) completely changed their minds about what's useful, good practice and common need ... and more importantly: what really can be omitted.

So, what's in the box? And what's not?

First of all, there still won't be a fullblown one-fits-all I18n/L10n solution in Rails. Please, understand that. Rails still won't be able to translate your application to your language as long it's not very similar to US English.

Instead, Rails continues to be what it always was: a framework localized to en-US. But this time with a twist: it is also internationalized.

That means: all hardcoded message strings and logic that we've previously been monkey patching in all of our plugins are now abstracted out of the Rails core code (which we mean by the process of "internationalization") and stored in a central place, accessible through a common interface.

We worked hard to stick to the principle of doing "the simplest thing that ever could work" with this. Rails itself won't provide means to localize an application to anything else than en-US. Instead it ships with a solution that gets the job done and walks away.

Rails will ship with our I18n Ruby gem which consists of two parts:

The first part is the API itself which is just a Ruby module with a bunch of methods that will be used by Rails and delegate all requests to a backend.
The second part is the Simple backend which implements whatever is necessary to re-localize Rails back to en-US.

The Simple backend acutally might also work for simple localization tasks such as translating an error message to another language, but, really, that's a side effect.

So what do we win?

The whole point of the exercise is: the Simple backend can be swapped with a different implementation that supports the same API. Future Rails I18n plugins will do exactly that.

E.g. there certainly will be backends that provide different pluralization rules, more powerful means to localize dates and numbers, persist translations in other formats (e.g. Gettext files or in the database). There might be backends that themselves provide some kind of a framework to support pluggable extensions and custom solutions for certain locales etc.

So if we still need to rely on plugins for our fullblown L10n support, what do we win then?

The short answer: A lot!

In the past I18n developers implemented their solutions for the same problems again and again. Our repeatedly reinvented wheels came with different flavours of syntax sugar and different bikeshed colors of monkey patches. The latter repeatedly broke when Rails moved by a millimeter and plugin developers had to fix their stuff.

I'm not saying that this situation was all bad. Actually I really believe that it was necessary for Rails' ecosystem to experiment with all of these solutions. But we've arrived at a point where we can move on.

So with this patch applied Rails I18n developers can now build on what was extracted from former experience. Instead of reimplementing things over and over again they can focus on some of the more challenging features like localized inflections, special formattings for strings, dates, numbers etc.

We also hope that future solutions will be more exchangeable and pluggable. There hopefully won't be the need to stick to a certain solution anymore just because it's the only one supporting inflections for a certain locale. Or the only one with a strong Gettext backend. Or whatever special feature comes to mind.

Of course that's still a long road ahead. But we believe that this step, the merge of the work of the "Rails I18n group", will help a lot on the journey.

Get involved!

If you'd like to join us working on Ruby on Rails's future I18n support, provide feedback or ask questions please do so! You can find our Google Group over at http://groups.google.com/group/rails-i18n.

This solution was developed by the Rails I18n group consisting of Matt Aimonetti, Joshua Harvey, Saimon Moore, Stephan Soller and myself, with the additional work, help and feedback of many other Rails I18n developers including Tore Darell, Chris Eppstein, Luca Guidi Samuel Lown Yaroslav Markin, Joshua Sierles, Julian 'Julik' Tarkhanov and others.