Rails - Apr 2005 - Cannot force WeBRICK into submission (UTF-8)

Hello everyone!

Pushing my new rails project I stumbled upon some weird problem.
I cannot force WeBRICK to show me my MySQL records in UTF-8. All I 
recieve into the object (instead of russian letters) are question marks 
(a la ?????? style). I am running MySQL 4.1 (the one with switchable 
charsets) and a binary Ruby extension for it, all on OSX.

To me it looks like MySQL never recieves what I send to it from 
environment.rb when running under WeBRICK, but gets it when I am 
running FCGI or CGI with Apache (or the connection gets closed and then 
reopened without honoring what I have set up).

In my environment.rb I have put the following at the end:

ActiveRecord::Base.connection().execute(''SET NAMES UTF8'')
#when I
remove this line I get ???s into FCGI too
$KCODE = ''u''
require ''jcode''

but WeBRICK doesn''t comply (FCGI and CGI do, though).

The perspective of developing by restarting 10 FCGI processes and 
Apache every time doesn''t seem too pretty at the moment. Any ideas?


P.S. Is it only me or would an ActionCharset (or just some proper 
singleton charset manager) a good addition to Rails?
ActiveRecord adapters for instance don''t have methods for setting the 
client charset (which should logically be DB-agnostic)
-- 
Julian "Julik" Tarkhanov

> Pushing my new rails project I stumbled upon some weird problem.
> I cannot force WeBRICK to show me my MySQL records in UTF-8. All I
> recieve into the object (instead of russian letters) are question marks
> (a la ?????? style). I am running MySQL 4.1 (the one with switchable
> charsets) and a binary Ruby extension for it, all on OSX.
You need to specify the proper Content-Type header.

@response.headers["Content-Type"] = "text/html;
charset=UTF-8";

You can do this on a controller specific basis, or just stick it in your
application controller as an after_filter.

Steve

On 26-apr-05, at 3:41, Steve V wrote:
>
> You need to specify the proper Content-Type header.
>
I am doing it already and my problem has nothing to do with this.
I recieve Unicode strings _already_ truncated to question marks from 
the database - the Unicode text in the templates comes out fine.

-- 
Julian "Julik" Tarkhanov

When using FastCGI only one connection by fcgi process is opened once
and for all. This connection is the one that is opened in the
environment.rb file. So all settings you perform on it in environment.rb
will apply for the life of the process.
Under WebRick, a new DB connection is opened with each client request.
DB connections are not reused. This means that when you do SET NAMES
UTF8 you do it on a connection that will never be used. All the new
connection won''t have UTF8 set on them. To solve your problem, you
could
use a filter that would do ActiveRecord::Base.connection().execute(''SET
NAMES UTF8'') for each incoming request.


-----Original Message-----
From: rails-bounces-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
[mailto:rails-bounces-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org] On
Behalf Of Julian ''Julik''
Tarkhanov
Sent: Tuesday, April 26, 2005 12:35 PM
To: rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
Subject: Re: [Rails] Cannot force WeBRICK into submission (UTF-8)



On 26-apr-05, at 3:41, Steve V wrote:
>
> You need to specify the proper Content-Type header.
>
I am doing it already and my problem has nothing to do with this. I
recieve Unicode strings _already_ truncated to question marks from 
the database - the Unicode text in the templates comes out fine.

-- 
Julian "Julik" Tarkhanov

_______________________________________________
Rails mailing list
Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
http://lists.rubyonrails.org/mailman/listinfo/rails

On 26-apr-05, at 13:23, Faissolle, Julien wrote:
> Under WebRick, a new DB connection is opened with each client request.
> DB connections are not reused. This means that when you do SET NAMES
> UTF8 you do it on a connection that will never be used. All the new
> connection won''t have UTF8 set on them. To solve your problem, you
> could
> use a filter that would do
ActiveRecord::Base.connection().execute(''SET
> NAMES UTF8'') for each incoming request.
>
Exactly what I thought.
>> ...or the connection gets closed and then reopened without honoring 
>> what I have set up).
But the solution doesn''t fit quite nicely - can I somehow make this 
persistent (so that my ActiveRecords will work in unit tests, WeBRICK, 
whatever instead of the controller only?

Maybe it''s time to pull some dirty Ruby tricks out of my sleeve and do 
some aliasing on ActiveRecord.connection in my Environment setup? :-) 
But essentially I would expect some generic way to set up DB 
environment (you have to also do SET ENCODING UNICODE for PostgreSQL 
for instance, and I am sure you have something specific for Oracle).

I would expect something like

connection.add_init_procedure (&initProc)
which would get called after the connection is open.
-- 
Julian "Julik" Tarkhanov

Julian ''Julik'' Tarkhanov wrote:
> But essentially I would expect some generic way to set up DB environment
> (you have to also do SET ENCODING UNICODE for PostgreSQL for instance,
> and I am sure you have something specific for Oracle).
In PostgreSQL you can create the database with a default encoding rather
than for each new connection:
  createdb --encoding=UTF-8 mydb

This is the default encoding for PostgreSQL 8 so you may omit it entirely.

If you want to do some setup queries with the connection is made, please
do.  In config/environment.rb:
  ActiveRecord::Base.establish_connection
  ActiveRecord::Base.connection.execute "query me harder"

jeremy

Jeremy, please read the thread from start. The problem here is that what
you do on connections in environment.rb when using WebRick is not taken
into account.


-----Original Message-----
From: rails-bounces-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
[mailto:rails-bounces-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org] On
Behalf Of Jeremy Kemper
Sent: Tuesday, April 26, 2005 5:49 PM
To: rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
Subject: Re: [Rails] Cannot force WeBRICK into submission (UTF-8)


Julian ''Julik'' Tarkhanov wrote:
> But essentially I would expect some generic way to set up DB 
> environment (you have to also do SET ENCODING UNICODE for PostgreSQL 
> for instance, and I am sure you have something specific for Oracle).
In PostgreSQL you can create the database with a default encoding rather
than for each new connection:
  createdb --encoding=UTF-8 mydb

This is the default encoding for PostgreSQL 8 so you may omit it
entirely.

If you want to do some setup queries with the connection is made, please
do.  In config/environment.rb:
  ActiveRecord::Base.establish_connection
  ActiveRecord::Base.connection.execute "query me harder"

jeremy
_______________________________________________
Rails mailing list
Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
http://lists.rubyonrails.org/mailman/listinfo/rails

A callback (connection_established) might be a nice option to make
sure a connection doesn''t sneak through without the developer knowing
about it.

On 4/26/05, Jeremy Kemper <jeremy-w7CzD/W5Ocjk1uMJSBkQmQ@public.gmane.org>
wrote:
> Julian ''Julik'' Tarkhanov wrote:
> > But essentially I would expect some generic way to set up DB
environment
> > (you have to also do SET ENCODING UNICODE for PostgreSQL for instance,
> > and I am sure you have something specific for Oracle).
> 
> In PostgreSQL you can create the database with a default encoding rather
> than for each new connection:
>   createdb --encoding=UTF-8 mydb
> 
> This is the default encoding for PostgreSQL 8 so you may omit it entirely.
> 
> If you want to do some setup queries with the connection is made, please
> do.  In config/environment.rb:
>   ActiveRecord::Base.establish_connection
>   ActiveRecord::Base.connection.execute "query me harder"
> 
> jeremy
> _______________________________________________
> Rails mailing list
> Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
> 

-- 
John W Higgins
wishdev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

Faissolle, Julien wrote:
> Jeremy, please read the thread from start. The problem here is that what
> you do on connections in environment.rb when using WebRick is not taken
> into account.
Thank you Julien, I have.  You may notice that my suggestion solves the
webrick problem for PostgreSQL users.  Likewise, you can specify the
MySQL charset per-table and per-column, and even provide a connection
default for mysqld:
  init-connect="set names utf-8"

Nevertheless, webrick should not open connections per-request.  It''s a
bug.  Try wrapping your connection setup in config/environment.rb with
  unless ActiveRecord::Base.connected?
    ActiveRecord::Base.establish_connection
    ActiveRecord::Base.connection.execute(''set names utf-8'')
  end

Hope this helps.
jeremy

On 26-apr-05, at 18:33, Jeremy Kemper wrote:
>  you can specify the
> MySQL charset per-table and per-column, and even provide a connection
> default for mysqld:
>   init-connect="set names utf-8"
Looks like this is what I''m gonna do (considering that I would never 
deploy anything on WeBRICK anyway).
>
> Nevertheless, webrick should not open connections per-request. 
It''s a
> bug.  Try wrapping your connection setup in config/environment.rb with
>   unless ActiveRecord::Base.connected?
>     ActiveRecord::Base.establish_connection
>     ActiveRecord::Base.connection.execute(''set names
utf-8'')
>   end
Nope, didn''t help.
-- 
Julian "Julik" Tarkhanov

On 26-apr-05, at 21:19, Julian ''Julik'' Tarkhanov wrote:
>
> On 26-apr-05, at 18:33, Jeremy Kemper wrote:
>>  you can specify the
>> MySQL charset per-table and per-column, and even provide a connection
>> default for mysqld:
>>   init-connect="set names utf-8"
>
Ok, very odd. I''ve put the aforementioned into my /etc/my.cnf but it 
doesn''t seem to have any effect. Putting it into ~/.my.cnf did nothing 
as well.
Currently they both look like:

[mysqld]
init-connect="SET NAMES UTF8"

[mysql client]
default-character-set=utf8

The version is 4.1.10a, so it probably should support this.

On the bright side the fix in environment.rb (both my version and 
Jeremy''s) seems to let me edit ARs in the console (which is very, very 
lovely).

-- 
Julian "Julik" Tarkhanov