Rails core - Nov 2005 - Localization ideas/proposal

So far all I''ve seen from the Rails community, with respect to
internationalization or localization have been gettext ports.

This doesn''t help me much, as I have two problems:
1. All of my text is in the database, and I prefer to keep it there.
2. I need different view code for different languages.

Below is the solution I am proposing.  I am willing to do most/all of
the legwork in the near future (next few weeks).  I''m not trying to
sneak this into 1.0 or something; I''m just looking for someone in the
core team to review this and give it his/her blessing, or possibly,
say that its an overworked pile of crap.  I know localization was a
point of empahasis for 1.0, but I haven''t seen any indication that it
ever was finished.

This solution has multiple parts.
0. General
1. Gettext
2. ActionPack changes
3. ActiveRecord changes

-- General --
def ActionController.set_locale(locale_code) ..
def ActionController.get_locale ..
Also, there would be a global variable like $L to retrieve the locale
code from.  The locales would be stored in a AR model called Locale. 
It would have a singleton method "current."
AC.get_locale == $L == Locale.current.code

It''s possible that Locale would use a yaml/csv fixture as the data
store, rather than a database table

-- Gettext --
Someone else''s gettext port will be adapted to fit this system, and
work where other methods are not preferred.

-- ActionPack changes --
Modify the render method so that if $L != nil, it will look for
template_name_#{$L}.rhtml, then if not found, template_name.rhtml,
then if not found, error.  It would also look for localized layouts
and partials.

-- ActiveRecord changes --
A model can have a magic "locale" field (similar to created_on), that
will (possibly with a acts_as_localized declaration) set up a Locale
has_many :models (and Model belongs_to :locale) relationship. 
Model.find will use Locale.current.models.find, and Model.new will
inject the current locale automatically.

Thanks a lot,
Kyle Maxwell

I think you hit the nail on the head in the first few paragraphs of
your mail where you list your specific requirements which are unlike
the way other people solved the problem  which presumably solved their
problems fine.

Localization is application and requirement specific. I believe that a
plugin would be the right place for such code.
There could be a rails endorsed Localization plugin if there will be a
generally agreed upon golden way to do it of course...

On 11/7/05, Kyle Maxwell <kyle@kylemaxwell.com>
wrote:
> So far all I''ve seen from the Rails community, with respect to
> internationalization or localization have been gettext ports.
>
> This doesn''t help me much, as I have two problems:
> 1. All of my text is in the database, and I prefer to keep it there.
> 2. I need different view code for different languages.
>
> Below is the solution I am proposing.  I am willing to do most/all of
> the legwork in the near future (next few weeks).  I''m not trying
to
> sneak this into 1.0 or something; I''m just looking for someone in
the
> core team to review this and give it his/her blessing, or possibly,
> say that its an overworked pile of crap.  I know localization was a
> point of empahasis for 1.0, but I haven''t seen any indication that
it
> ever was finished.
>
> This solution has multiple parts.
> 0. General
> 1. Gettext
> 2. ActionPack changes
> 3. ActiveRecord changes
>
> -- General --
> def ActionController.set_locale(locale_code) ..
> def ActionController.get_locale ..
> Also, there would be a global variable like $L to retrieve the locale
> code from.  The locales would be stored in a AR model called Locale.
> It would have a singleton method "current."
> AC.get_locale == $L == Locale.current.code
>
> It''s possible that Locale would use a yaml/csv fixture as the data
> store, rather than a database table
>
> -- Gettext --
> Someone else''s gettext port will be adapted to fit this system,
and
> work where other methods are not preferred.
>
> -- ActionPack changes --
> Modify the render method so that if $L != nil, it will look for
> template_name_#{$L}.rhtml, then if not found, template_name.rhtml,
> then if not found, error.  It would also look for localized layouts
> and partials.
>
> -- ActiveRecord changes --
> A model can have a magic "locale" field (similar to created_on),
that
> will (possibly with a acts_as_localized declaration) set up a Locale
> has_many :models (and Model belongs_to :locale) relationship.
> Model.find will use Locale.current.models.find, and Model.new will
> inject the current locale automatically.
>
> Thanks a lot,
> Kyle Maxwell
> _______________________________________________
> Rails-core mailing list
> Rails-core@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails-core
>

--
Tobi
http://jadedpixel.com    - modern e-commerce software
http://typo.leetsoft.com - Open source weblog engine
http://blog.leetsoft.com - Technical weblog

Hi Tobias,

couldn''t restrain myself. I was only going to lurk here, but
I''d like to
comment on that.
> Localization is application and requirement specific. I believe that a
> plugin would be the right place for such code.
> There could be a rails endorsed Localization plugin if there will be a
> generally agreed upon golden way to do it of course...
Access to databases is application and requirement specific, too. I think 
that both solutions have to be considered. Also, when someone comes and 
says this one solution is good and the other one chimes in with no I think 
*this* is better, usually a combination of both can be most beneficial :)

As I see it, there are some conventions which have to be adapted and 
excercised by the core developers. Like when having methods spitting out 
dates one has to think about the rest of the world. Although mainly 
translations of error messages and other potential strings in core Rails.

This would be _really_ easy, as Rails has only a select few places to 
change. Dummy functions would have to be created. All in all, this is maybe 
an hour worth of work.

Any other way the package maintainer will have to stay in sync, which just 
cries out for DRY.

Also, I think that some other stuff should be near to the core. Preferably 
all time and currency related stuff. Although, for this another Module 
might be fine. ActiveLocalization, ActionI18n, ActiveLanguages anyone? 
Other stuff, like harvesters, file format definitions etc. for example 
could be put into plugins. Although, I must say, when I look into what 
James Adams does with Rails Engines[1] it will probably be possible to do 
most of it as a plugin.

Sascha Ebach

PS: Tobias, in case you were wondering about my integration of Ri18n into 
Typo. Since I never was able to get in contact with the maintainer I 
finally gave up after 3 weeks of working on it. Ri18n is GPL and as such 
not viable for integration into Typo. Since all attempts of contacting the 
author have failed I finally gave up. Now I have to use Wordpress for my 
projects :(what a mess...) I''d much rather not start again with the
whole
process, but try to actually help with implementing this into Rails core.

http://svn.berlios.de/viewcvs/ri18n/trunk/
Sadly looks like the repo hasn''t been touched in 6 months.

[1] http://rails-engines.rubyforge.org/

On 11/8/05, Sascha Ebach <se@digitale-wertschoepfung.de>
wrote:
> Hi Tobias,
>
> couldn''t restrain myself. I was only going to lurk here, but
I''d like to
> comment on that.
Here''s koz being all java-esque again.

Java is widely regarded as a good example of how to do i18n and l10n
''right''.   None of the Java frameworks provide any support for
localisation in the models themselves.   They really provide nothing
beyond supporting ''current locale'' and ''get me the
string for this
key''.

I think that rails 1.x could probably provide support for the
determining the locale and storing  / retrieving keys.   But model
level i18n support is:

* Hard, and
* never ever ever going to satisfy everyone.

--
Cheers

Koz

Thanks for so much feedback so quickly!

This comes out of my need to get a Spanish version of a current
English website up in the next week so so.  I chose the quickest and
simplest idea that I could think of, that would allow me to contribute
back to the community and reuse code in the future.

So, I asked myself:
Q. Does it add value to Rails?
A. Yes, it solves my problem (And presumably, I''m not that unique. 
I''m sure others want to store multilingual data in a database.  The
custom views might be a little more of an edge case).

Q. Does it screw other stuff up?
A. No.  It''s fully backwards-compatible, and you can still use gettext
where preferred.

Is it the best-possible solution?  Thats doubtful.  But this works and
is really simple.

I find Kyle''s idea great! That''s how .NET does it (except the
model part)
and it seems the most natural way to go. The ActiveRecord localization
sounds a bit strange, but it wouldn''t mind if implemented as an
"acts_as".
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails-core/attachments/20051108/e1ec623e/attachment.html

I''m definitely open to other ideas on ActiveRecord localization. 
I''m
not sure what you find strange about the suggested idea.  Perhaps it
is that you are "locked" into your current locale?

I think that if we are going to do database localization in Rails,
then we will have a LocalizedModel belongs_to :locale situation.  That
seems obvious to me (please tell me if you think I''m wrong).  The
locale table can be either an actual table, or a fixture (I think I
prefer the latter).

Now then, the question is "what methods do we provide to interface
with this data model?"  Whatever methods we choose should (IMHO)
accommodate both a complete view of the database ("normal mode"), and
a view where one can only interact with data in a specific locale
("localized mode").  It is beneficial to make both paradigms as easy
and straightforward as possible.

In thinking about it over the last several hours, it is perhaps not
ideal to override Model.find with Locale.current.models.find, as that
diminishes the normal mode capabilities.  On the other hand, prefixing
Locale.current. to every find may be tedious, and I anticipate that
localized mode will be used in the (vast?) majority of circumstances.

Idea: Use _Model to represent a localized model (i.e.
_Model.find(...)).  This has the benefit of looking something like
gettext, and it doesn''t change the default functionality.

I haven''t quite reached a conclusion yet as to the interface to the
localized data model.  Please keep the feedback coming!

Kyle Maxwell

On 11/7/05, Ioan Bizau <ionut.bizau@gmail.com>
wrote:
>
>  I find Kyle''s idea great! That''s how .NET does it
(except the model part)
> and it seems the most natural way to go. The ActiveRecord localization
> sounds a bit strange, but it wouldn''t mind if implemented as an
"acts_as".
>
> _______________________________________________
> Rails-core mailing list
> Rails-core@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails-core
>
>
>

I''m about to release a plugin with Josh Sierles that addresses a lot of
these
issues. Here''s what we did.
 
To translate certain fields of a model (the
''model level'' ActiveRecord translations):

1. Don''t include those fields
in the model table.
2. Use the "translates" class method:
 
class Product
< ActiveRecord::Base
  translates :name, :description, :specs
end
 
3.
Do Local.set_locale("en_US").
 
4. Now you can do product.name, and it will
return the name in the language currently set. Almost all AR functionality
works as expected: assignment, new, create, associations.
 
Behind the scenes,
the plugin uses a table called "translations", and does outer joins to
pull
the
translated fields into the model.

The rest of the plugin is based
on the Multilingual Rails package from Per Wigren. Here is a rundown of its
features:
 
- String translation using _("str"), str.t, :str.t, :str %
[variable_for_interpolation]


- Translated templates and partials: app/views/shop/ice_creams.sv.rhtml
loads before ice_creams.rhtml

- Overrides for common local-related methods:
distance_of_time_in_words, ActiveRecord errors, Date constants

- Logging
of untranslated strings in logs/translation-misses

- Pluralization in various
languages: :some_books % [book_count]

- Charset conversion with String
extensions:  "This string is now ISO
8859-1".iconv_to(''iso-8859-1'')

- Pluggable
translators, default being the ruby file translations

- ISO-based lists
of locale codes, country names and languages, used by a new country_select
helper

- Optional unicode string normalization for upcase, downcase, etc.

 
Example default translation:
 
  s "I want ice cream!" do
    t :sv,
"Jag vill ha glass!"
    t :no, "Jeg vil ha is!"
  end

Since Per seems
to be out for the count, we thought it would be nice to extend his package.
MLR offers a Ruby way of doing things, and opts for clean code. It stores
all of its translations in Ruby files.  Details about this package here
http://www.tuxsoft.se/sv/oss/rails/multilingual.
It no longer requires the unicode gem, but will use unicode normalization
routines should the gem be installed.

Lingering doubts:

1. Should the
plugin ennforce utf-8 encoding for storage as MLR does currently?

2. Should
controllers and views be offered either in another package or as an
''engine''
for editing translations?

Hi jm,

snipped..

this is really good news! Looking forward to seeing what you have done. Why 
is Per out of this one? Other projects at the moment?
> 1. Should the
> plugin ennforce utf-8 encoding for storage as MLR does currently?
UTF-8 is still the lowest common (and compatible) denominator. It is not 
perfect, but at least until Ruby 2.0 comes out it will probably work for 
most ppl. As far as I understand Ruby 2.0 will have it''s own encoding, 
which could then be used. Maybe if we are lucky Christmas 2006 or 2007. 
Just guessing from Matz''s releasing habits in the past :)
> 2. Should
> controllers and views be offered either in another package or as an
''engine''
> for editing translations?
Hm, I don''t quite understand this. Do you have an example?

Sascha Ebach

To take full advantage of the AR translations, it would be nice to
offer a simple cotroller/view pair to get started with entering actual
translation data into the translations table, perhaps based on the
translated fields defined by the user in his or her models. This could
be a generator, or something like the recent plugin engines. Right
now, you have to create the translations yourself through the added
model methods.

Joshua Sierles

> > 2. Should
> > controllers and views be offered either in another package or as an
''engine''
> > for editing translations?
>
> Hm, I don''t quite understand this. Do you have an example?
>
> Sascha Ebach
>

I agree, Sascha -- utf-8 is best for now. The svn repo for the plugin
is at: svn://rashgash.co.il/rails-plugins/multilingual/trunk. The
tests should all pass (although I''ve only tested on my system thus
far), but we still have some work to do on the docs and packaging.
Comments and help are welcomed.

Josh Harvey

Joshua Sierles wrote:
> To take full advantage of the AR translations, it would be nice to
> offer a simple cotroller/view pair to get started with entering actual
> translation data into the translations table, perhaps based on the
> translated fields defined by the user in his or her models. This could
> be a generator, or something like the recent plugin engines. Right
> now, you have to create the translations yourself through the added
> model methods.
Now I get it. Was a little late last night. I think a generator (like the 
Login generator) would be the easiest and still the most common solution. 
But, what is actually the best, can probably only be found out testing all 
the different solutions in the wild.

Sascha Ebach

Joshua Harvey wrote:
> I agree, Sascha -- utf-8 is best for now. The svn repo for the plugin
> is at: svn://rashgash.co.il/rails-plugins/multilingual/trunk. The
> tests should all pass (although I''ve only tested on my system thus
> far), but we still have some work to do on the docs and packaging.
> Comments and help are welcomed.
Great. I''ll check it out later today.

Sascha Ebach

Hi Joshua,

Looking forward to what you come up with. A suggestion:

On 8 Nov 2005 19:16:49 -0000, jmharvey.19309139@bloglines.com <
jmharvey.19309139@bloglines.com> wrote:
>
> To translate certain fields of a model (the
> ''model level'' ActiveRecord translations):
>
> 1. Don''t include those fields
> in the model table.

I think it would be better to include the fields that require localization
in the original table. The advantage of this is that the values in these
fields could become the ''fallback'' values if a localization
for a value
doesn''t exist for the specified locale or if a locale isn''t
specified.

Another advantage would be that it''ll be possible to drop your plugin
into
an existing app and get the localization features without changing anything
(dropping columns).
Hammed
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails-core/attachments/20051109/6925b9c8/attachment.html

Joshua Harvey wrote:
> I agree, Sascha -- utf-8 is best for now. The svn repo for the plugin
> is at: svn://rashgash.co.il/rails-plugins/multilingual/trunk. The
> tests should all pass (although I''ve only tested on my system thus
> far), but we still have some work to do on the docs and packaging.
> Comments and help are welcomed.
OMG, amazing stuff. Although I only had time to skim around. I could see 
this as a definate way to do l10n in Rails. The schema.rb file alone is 
worth the download. I am really looking forward to your release. I think I 
would like to write a manual on how to implement (or if it is implemented 
by default in core Rails) use this. But the earliest I can do is by around 
christmas.

Sascha Ebach

Right on, Hammed. That''s a really good idea -- I''m going to
try to
work that into the code. The plugin already supports falling back to a
base language, but this would make it jive better with traditional
apps.

Hammed wrote:
I think it would be better to include the fields that require
localization in the original table. The advantage of this is that the
values in these fields could become the ''fallback'' values if a
localization for a value doesn''t exist for the specified locale or if
a locale isn''t specified.

Another advantage would be that it''ll be possible to drop your plugin
into an existing app and get the localization features without
changing anything (dropping columns).

Josh

I''ll be using your manual as a guide to write one for implementation
once we get the specifics sorted out. Thanks for keeping the topic
alive...

Joshua Sierles

On 11/9/05, Sascha Ebach <se@digitale-wertschoepfung.de>
wrote:
> Joshua Harvey wrote:
> > I agree, Sascha -- utf-8 is best for now. The svn repo for the plugin
> > is at: svn://rashgash.co.il/rails-plugins/multilingual/trunk. The
> > tests should all pass (although I''ve only tested on my system
thus
> > far), but we still have some work to do on the docs and packaging.
> > Comments and help are welcomed.
>
> OMG, amazing stuff. Although I only had time to skim around. I could see
> this as a definate way to do l10n in Rails. The schema.rb file alone is
> worth the download. I am really looking forward to your release. I think I
> would like to write a manual on how to implement (or if it is implemented
> by default in core Rails) use this. But the earliest I can do is by around
> christmas.
>
> Sascha Ebach
> _______________________________________________
> Rails-core mailing list
> Rails-core@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails-core
>

The ruby unicode gem doesn''t compile on windows.  Any ideas?

On 11/9/05, Joshua Sierles <jsierles@gmail.com>
wrote:
> I''ll be using your manual as a guide to write one for
implementation
> once we get the specifics sorted out. Thanks for keeping the topic
> alive...
>
> Joshua Sierles
>
> On 11/9/05, Sascha Ebach <se@digitale-wertschoepfung.de> wrote:
> > Joshua Harvey wrote:
> > > I agree, Sascha -- utf-8 is best for now. The svn repo for the
plugin
> > > is at: svn://rashgash.co.il/rails-plugins/multilingual/trunk. The
> > > tests should all pass (although I''ve only tested on my
system thus
> > > far), but we still have some work to do on the docs and
packaging.
> > > Comments and help are welcomed.
> >
> > OMG, amazing stuff. Although I only had time to skim around. I could
see
> > this as a definate way to do l10n in Rails. The schema.rb file alone
is
> > worth the download. I am really looking forward to your release. I
think I
> > would like to write a manual on how to implement (or if it is
implemented
> > by default in core Rails) use this. But the earliest I can do is by
around
> > christmas.
> >
> > Sascha Ebach
> > _______________________________________________
> > Rails-core mailing list
> > Rails-core@lists.rubyonrails.org
> > http://lists.rubyonrails.org/mailman/listinfo/rails-core
> >
> _______________________________________________
> Rails-core mailing list
> Rails-core@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails-core
>

The schema.rb file only loads the first 184 language entries for me. 
I would think it timed out.  Just thought you should be aware.   I
just quick fixed by deleting all languages except english and spanish.

Kyle

On 11/9/05, Joshua Sierles <jsierles@gmail.com>
wrote:
> I''ll be using your manual as a guide to write one for
implementation
> once we get the specifics sorted out. Thanks for keeping the topic
> alive...
>
> Joshua Sierles
>
> On 11/9/05, Sascha Ebach <se@digitale-wertschoepfung.de> wrote:
> > Joshua Harvey wrote:
> > > I agree, Sascha -- utf-8 is best for now. The svn repo for the
plugin
> > > is at: svn://rashgash.co.il/rails-plugins/multilingual/trunk. The
> > > tests should all pass (although I''ve only tested on my
system thus
> > > far), but we still have some work to do on the docs and
packaging.
> > > Comments and help are welcomed.
> >
> > OMG, amazing stuff. Although I only had time to skim around. I could
see
> > this as a definate way to do l10n in Rails. The schema.rb file alone
is
> > worth the download. I am really looking forward to your release. I
think I
> > would like to write a manual on how to implement (or if it is
implemented
> > by default in core Rails) use this. But the earliest I can do is by
around
> > christmas.
> >
> > Sascha Ebach
> > _______________________________________________
> > Rails-core mailing list
> > Rails-core@lists.rubyonrails.org
> > http://lists.rubyonrails.org/mailman/listinfo/rails-core
> >
> _______________________________________________
> Rails-core mailing list
> Rails-core@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails-core
>

Oops, didn''t even notice that. I just fixed it in svn, but now the
schema.rb for over 7k rows takes like 10 minutes to load. I included a
file called languages.sql in the db folder that contains all the SQL
inserts, and that should only take a few seconds to load.

Spanish actually was in the original table, but this fix will be a big
relief to all the Ipulo-speaking rails coders out there.

Josh

Kyle wrote:

The schema.rb file only loads the first 184 language entries for me.
I would think it timed out.  Just thought you should be aware.   I
just quick fixed by deleting all languages except english and spanish.

Kyle