Hello all,
I''m trying to import some csv data (that has latin characters) using
faster csv but I keep getting a MalformedCSVError:
"Unclosed quoted field on line 1."
"/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.2.3/lib/faster_csv.rb:1592:in
`shift''"
I''ve tested the same code outside of the Rails application an it works
fine.
Can anyone help me?
Here is the code:
csv_options = {
:headers => false,
:return_headers => false,
:skip_blanks => false
}
csv_file_name = File.dirname(__FILE__) + "/../../Files.TXT"
data = ""
FasterCSV.foreach(csv_file_name, csv_options) do |row|
data << row.to_s + "<br>"
end
And the first two lines in the file:
"20.1","Maria José",,"9"
"10.2","José Maria",,"10"
I''m using fastercsv 1.2.3 and rails 2.1.2
Note: I''ve tried adding $KCODE = "utf8" but it still does not
work.
Thanks.
Best regards,
Migrate
--
Posted via http://www.ruby-forum.com/.
it can be issue with your rails version for current application. -- Posted via http://www.ruby-forum.com/.
Hi Migrate, On Thu, 2009-06-18 at 00:51 +0200, Hu Ma wrote:> Hello all, > I''m trying to import some csv data (that has latin characters) using > faster csv but I keep getting a MalformedCSVError: > "Unclosed quoted field on line 1." > "/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.2.3/lib/faster_csv.rb:1592:in > `shift''">From the error message, it looks like shift has a backtick as theopening quote and a single quote as the closing quote. HTH, Bill
Hello Bill, I''m not sure I understood you. Can you explain further? The file content has only double quotes. Best regards, Migrate bill walton wrote:> Hi Migrate, > > On Thu, 2009-06-18 at 00:51 +0200, Hu Ma wrote: >> Hello all, >> I''m trying to import some csv data (that has latin characters) using >> faster csv but I keep getting a MalformedCSVError: >> "Unclosed quoted field on line 1." >> "/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.2.3/lib/faster_csv.rb:1592:in >> `shift''" > >>From the error message, it looks like shift has a backtick as the > opening quote and a single quote as the closing quote. > > HTH, > Bill-- Posted via http://www.ruby-forum.com/.
Dharmdip Rathod wrote:> it can be issue with your rails version for current application.Hello Dharmdip, Do you know any issues between faster csv and rails? Best regards, Migrate -- Posted via http://www.ruby-forum.com/.
Yes i had it , my rails version was issue . FasterCSV was running smooth on my local but i deployed my application to server FasterCSV was not running so please check server operating system , and install gem related to OS. If your problem is not solved feel free to ask ! ba bye.... -- Posted via http://www.ruby-forum.com/.
First off, a quick note on $KCODE - the values it takes are a little weird. ''utf8'' is not valid; you''re looking for one of these: None (n or N) EUC (e or E) Shift_JIS (s or S) UTF-8 (u or U) (taken from http://blog.grayproductions.net/articles/bytes_and_characters_in_ruby_18 ) You can also pass an :encoding option (same choice of values) to FasterCSV. I''d also make sure that the file is really UTF8, as many CSVs produced by Excel will instead be in ISO8859-1 (technically, an incompatible MS variant of that standard [CP1252]). You may have to crunch the data somewhat to get it to behave, as the MS Office products are also notorious for stuffing "smart quotes" and other non-standard characters into files... --Matt Jones On Jun 17, 6:51 pm, Hu Ma <rails-mailing-l...-ARtvInVfO7ksV2N9l4h3zg@public.gmane.org> wrote:> Hello all, > I''m trying to import some csv data (that has latin characters) using > faster csv but I keep getting a MalformedCSVError: > "Unclosed quoted field on line 1." > "/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.2.3/lib/faster_csv.rb:1592:in > `shift''" > > I''ve tested the same code outside of the Rails application an it works > fine. > Can anyone help me? > > Here is the code: > > csv_options = { > :headers => false, > :return_headers => false, > :skip_blanks => false > > } > > csv_file_name = File.dirname(__FILE__) + "/../../Files.TXT" > data = "" > FasterCSV.foreach(csv_file_name, csv_options) do |row| > data << row.to_s + "<br>" > end > > And the first two lines in the file: > "20.1","Maria José",,"9" > "10.2","José Maria",,"10" > > I''m using fastercsv 1.2.3 and rails 2.1.2 > > Note: I''ve tried adding $KCODE = "utf8" but it still does not work. > > Thanks. > Best regards, > Migrate > -- > Posted viahttp://www.ruby-forum.com/.
Hello all, I''ve update to Rails 2.3.2 and the problem remains. I''ve also created a new application, and added just one controller and FasterCSV to environment.rb (no other plugins are present in the application) but still it does not work. I already said here that the error only happens in rails. I also found out the following weird behavior: this error only happens if the latin character is in the last two positions in the string. Example 1 (Not working): "1","Maria José",,"9" Example 2 (Not working): "1","Helder Serrão",,"9" Example 3 (working): "2","José Maria",,"10" Also I checked that it works if the latin caracters exist only in the last column: Example 4 (working): "1","9",,"Maria José" Note: I''ve removed the KCODE part from the code. -- Posted via http://www.ruby-forum.com/.
This confirms my original suspicion; your file is almost certainly in either ISO8859-1 or Windows-1252. The accented characters are being misread, as Ruby''s default value (or maybe Rails?) is $KCODE = ''u''. So the accented letters are interpreted as the first character of a 3- byte UTF-8 sequence. You might want to try passing the :encoding => ''n'' option to FasterCSV and see what that does; getting Windows-1252 characters to display correctly on other platforms is left as an exercise.. :) --Matt Jones On Jun 21, 7:52 pm, Hu Ma <rails-mailing-l...-ARtvInVfO7ksV2N9l4h3zg@public.gmane.org> wrote:> Hello all, > > I''ve update to Rails 2.3.2 and the problem remains. > I''ve also created a new application, and added just one controller > and FasterCSV to environment.rb (no other plugins are present in the > application) but still it does not work. > > I already said here that the error only happens in rails. I also found > out the following weird behavior: this error only happens if the latin > character is in the last two positions in the string. > Example 1 (Not working): > "1","Maria José",,"9" > Example 2 (Not working): > "1","Helder Serrão",,"9" > Example 3 (working): > "2","José Maria",,"10" > > Also I checked that it works if the latin caracters exist only in the > last column: > Example 4 (working): > "1","9",,"Maria José" > > Note: I''ve removed the KCODE part from the code. > -- > Posted viahttp://www.ruby-forum.com/.
On Jun 22, 3:23 pm, Matt Jones <al2o...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> You might want to try passing the :encoding => ''n'' option to FasterCSV > and see what that does; getting Windows-1252 characters to display > correctly on other platforms is left as an exercise.. :)If it is indeed the case that your accented characters are coming out in ISO8859-1, consider converting the character set in your input first: csv_data = Iconv.iconv(''utf-8'', ''ISO_8859-1'', file.read).to_s, Frankly the whole thing seems rather like black magic to me, but the above seems to solve the problem I was having with Excel files in a CSV importer that I''ve recently created. -Matt
Hello all, Matt thanks for the advice but I''m using faster csv 1.2.3 and only version 1.4 has the option :encoding => ''n''. I cannot upgrade to 1.4 because I''m using ruport and ruport only accepts 1.2.3. Matthew I''ve done what you suggested and it worked. Thanks! Best regards, Migrate -- Posted via http://www.ruby-forum.com/.
On Jun 17, 6:51 pm, Hu Ma <rails-mailing-l...-ARtvInVfO7ksV2N9l4h3zg@public.gmane.org> wrote:> Hello all, > I''m trying to import some csv data (that has latin characters) using > faster csv but I keep getting a MalformedCSVError: > "Unclosed quoted field on line 1." > "/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.2.3/lib/faster_csv.rb:1592:in > `shift''"[snip]> I''m using fastercsv 1.2.3 and rails 2.1.2You may want to try fastercsv 1.5. It was released just a few days ago. I updated our project to use trunk back in May because of a similar problem. Jonathan