Vpim talk - Apr 2006 - Field#to_text

We''re using Vpim to parse iCal and vCard files, and so far
it''s been
great. However, when we were parsing some iCal files we noticed that
all the endlines in the description section were disappearing. I start
peering through the source code and find (in the Field#to_text method
that is being called):

# The value as text. Text can have escaped newlines, commas, and escape
# characters, this method will strip them, if present.

Is there some reason that this is done? I''d much prefer to keep my
commas and newlines...

Thanks,
Emmett

Quoting emmett.shear at gmail.com, on Sun, Apr 02, 2006 at 06:47:12PM
-0400:
> We''re using Vpim to parse iCal and vCard files, and so far
it''s been
> great. However, when we were parsing some iCal files we noticed that
> all the endlines in the description section were disappearing. I start
> peering through the source code and find (in the Field#to_text method
> that is being called):
> 
> # The value as text. Text can have escaped newlines, commas, and escape
> # characters, this method will strip them, if present.
> 
> Is there some reason that this is done? I''d much prefer to keep my
> commas and newlines...
The comment isn''t well written, its stripping the ESCAPING, which
should get you back the newlines. See the definition of
Vpim::decode_text.

What version are you using, and can I have some sample input? Maybe
there is a bug.

But since it works correctly, to my knowledge:

  % ruby -I lib -r vpim/field -e ''puts
Vpim::DirectoryInfo::Field.decode("F:a\\, b\\, c\\ndnext line\\nnext
line\\n").to_text''
  a, b, c
  dnext line
  next line


I''m wondering if you might not have something like this:

----
NOTE:some text
 some more text
 another bunch of text
----

Because that note doesn''t have any newlines in it. A 3 line note would
look like:

----
NOTE:line 1\nline 2\nline 3
----

And would have the identical value if wrapped, where wrapping can
occur anywhere:

----
NOTE:lin
 e 1\nl
 in
 e 2\nl
 ine
  3
----

The above still only has two newlines in it.

I didn''t have an encode_text method until recently, so maybe this has
caused you trouble.

You can add a note to a vCard using Vcard#make and
Vcard::Maker#add_note, see the online docs for the last release.

Unfortunately, I''m about half-way through adding similar APIs for
Vevent. Maybe I''ll post what I''ve done this week, unstable
though it may
be, it will do description, summary, and the other text fields. Anyhow,
see Vpim::encode_text if you want to put text containing newlines, and
commas, and other special characters, into a field value.

Cheers,
Sam

I''m parsing the iCal feed from webcal://upcoming.org/calendar/metro/45
(the webcal:// protocol is exactly like http, so in ruby you can get
the contents using Net::HTTP.get if you replace "webcal" with
"http").
The description fields of the events look like "text text\n\r    \n\r 
  text text" in their raw form, and then "text text      text
text" in
their decoded form. Do I need to re-escape the endlines before
processing?

Thanks,
Emmett


On 4/2/06, Sam Roberts <sroberts at uniserve.com>
wrote:
> Quoting emmett.shear at gmail.com, on Sun, Apr 02, 2006 at 06:47:12PM
-0400:
> > We''re using Vpim to parse iCal and vCard files, and so far
it''s been
> > great. However, when we were parsing some iCal files we noticed that
> > all the endlines in the description section were disappearing. I start
> > peering through the source code and find (in the Field#to_text method
> > that is being called):
> >
> > # The value as text. Text can have escaped newlines, commas, and
escape
> > # characters, this method will strip them, if present.
> >
> > Is there some reason that this is done? I''d much prefer to
keep my
> > commas and newlines...
>
> The comment isn''t well written, its stripping the ESCAPING, which
> should get you back the newlines. See the definition of
> Vpim::decode_text.
>
> What version are you using, and can I have some sample input? Maybe
> there is a bug.
>
> But since it works correctly, to my knowledge:
>
>   % ruby -I lib -r vpim/field -e ''puts
Vpim::DirectoryInfo::Field.decode("F:a\\, b\\, c\\ndnext line\\nnext
line\\n").to_text''
>   a, b, c
>   dnext line
>   next line
>
>
> I''m wondering if you might not have something like this:
>
> ----
> NOTE:some text
>  some more text
>  another bunch of text
> ----
>
> Because that note doesn''t have any newlines in it. A 3 line note
would
> look like:
>
> ----
> NOTE:line 1\nline 2\nline 3
> ----
>
> And would have the identical value if wrapped, where wrapping can
> occur anywhere:
>
> ----
> NOTE:lin
>  e 1\nl
>  in
>  e 2\nl
>  ine
>   3
> ----
>
> The above still only has two newlines in it.
>
> I didn''t have an encode_text method until recently, so maybe this
has
> caused you trouble.
>
> You can add a note to a vCard using Vcard#make and
> Vcard::Maker#add_note, see the online docs for the last release.
>
> Unfortunately, I''m about half-way through adding similar APIs for
> Vevent. Maybe I''ll post what I''ve done this week,
unstable though it may
> be, it will do description, summary, and the other text fields. Anyhow,
> see Vpim::encode_text if you want to put text containing newlines, and
> commas, and other special characters, into a field value.
>
> Cheers,
> Sam
>
>

Quoting emmett.shear at gmail.com, on Sun, Apr 02, 2006 at 11:46:17PM
-0400:
> I''m parsing the iCal feed from
webcal://upcoming.org/calendar/metro/45
> (the webcal:// protocol is exactly like http, so in ruby you can get
> the contents using Net::HTTP.get if you replace "webcal" with
"http").
> The description fields of the events look like "text text\n\r    \n\r 
>   text text" in their raw form,
Not in the file I downloaded, they don''t. I grepped the entire file,
and
didn''t find a single occurence of a backslash followed by a n.

What it is is this:

BEGIN:VEVENT
DTSTART;TZID=US-Pacific:20060402
DTEND;TZID=US-Pacific:20060403
SUMMARY:Daniel Bell + John Tejada + the Sunset Party comes to boca
DESCRIPTION:Sunday\, April 2nd ^M 
      NOON - 2am^M
      ^M
      it&#39\;s time to get...^M
      OUT OF^M 
      [KONTROL]^M
      ^M
      ^M
      with 2 legends playing back to back!^M
      <b>DANIEL BELL</b> (7th City\, Accelerate\, Tresor)^M
      ^M
      +^M
      <b>JOHN TEJADA</b> ^M
      (Palette\, Poker Flat\, Playhouse)^M

The description text appears to be html, which may not be illegal, but sure
isn''t standard. It will look pretty bad in most calendar viewers,
I''d think,
especially since its one long line with random whitespace. It is pretty much
unreadable in iCal.

They are making some small attempts at escaping (you can see
"&#39\;", the ";''
was escaped), but they are putting newlines in as raw carriage returns, instead
of the two character sequence 0x5C 0x62 (backslash, n). Weird, I''d
expect them
to either not try to escape at all, or  to get it right, but they are half-way.
> and then "text text      text text" in
> their decoded form. Do I need to re-escape the endlines before
> processing?
Not sure what you mean by rescape, or by processing. What are you trying
to do?

If you tell me that, I can probably give some suggestions on how to deal with
this. Off the top of my head, the only place a raw CR shows up, I think, is
where they want a newline in a TEXT property value, so you could preprocess the
calendar, doing basically

  calstr = calstr.gsub("\r", "\\n")

Its late at night, so thats all for now. 

Cheers,
Sam

Quoting sroberts at uniserve.com, on Sun, Apr 02, 2006 at 11:57:29PM
-0700:
> Quoting emmett.shear at gmail.com, on Sun, Apr 02, 2006 at 11:46:17PM
-0400:
> > I''m parsing the iCal feed from
webcal://upcoming.org/calendar/metro/45
> > (the webcal:// protocol is exactly like http, so in ruby you can get
>   calstr = calstr.gsub("\r", "\\n")
Ok, the above approach runs a good chance of being killingly memory
intensive for large calendars, like the one you are looking at.

Here''s a better approach:

The object passed to Icalendar.decode needs to support #each, where each
line is yielded one by one. Happens to work well for String and IO, but
you can provide any kind of object you want, like this:

  require ''vpim/icalendar''
  require ''open-uri''

  class Convert
    def initialize(io)
      @io = io
    end

    def each
      @io.each do |line|
        line.gsub!("\r", "\\n")
        yield line
      end
    end
  end

  io = open("http://upcoming.org/calendar/metro/45")
  cvt = Convert.new(io)
  cal = Vpim::Icalendar.decode(cvt).first

  cal.components do |c|
    puts "-------------------------------------------------"
    puts c.description
  end


Output has the newlines where you most likely want them. Some days I
really, really like duck-typing.

Cheers,
Sam

Btw, here''s the first few dozen lines of output of the above, with a
line feed where I guess they were intended:

-------------------------------------------------
http://www.mobilemonday.com/
-------------------------------------------------
GWEN CHAN, Interim Superintendent, San Francisco Unified School District
     DENNIS CONAGHAN, Executive Director, San Francisco Center for Economic
Development
     AMIT GHOSH, Chief of Comprehensive Planning, San Francisco Planning
Department
     JOHN KILLACKY, Arts and Culture Program Officer, San Francisco Foundation
     SCOTT SHAFER, Host, "The California Report," KQED ??
Moderator
     
     FREE FOR MEMBERS (More)
     
     WHERE HAVE ALL THE YOUNG PEOPLE GONE? WHY SAN FRANCISCO IS LOSING ITS YOUNG
WORKERS AND FAMILIES
     
     
     
     
     According to a recent survey from the Mayor's Office, nearly half
the families in San Francisco with preschool children stated their intent to
move out in the next three years, citing the lack of affordable housing and the
state of the public schools as primary motivators. The findings confirm what
many have known for a long time - with the highest median home prices of any
major U.S. city and the smallest share of children under age 18, San Francisco
has a hard time attracting and retaining young workers and families. What is the
city doing to improve affordable housing, job growth and the quality of public
schools? Our panel will discuss what is needed to allow young residents and
families to stay in San Francisco for the long term.
     
     6:00 p.m., Check-in | 6:30 p.m., Program | 7:30 p.m., Wine and cheese
reception
      Free for Members, $15 for Non-members
     
     http://www.commonwealthclub.org/featured.html#sfaffordability
-------------------------------------------------
http://www.highlandteam.com/news/dev_series.html 
      
     Alumni Career Services at the Stanford Graduate School of Business,
together with Highland Team, a marketing strategy consulting firm, and Kathryn
Ullrich Associates, Inc., experts in executive recruiting, present Getting to
The Top: Getting Unstuck.
     
     Are you at a place in your career where you're asking questions
such as:
     **How do I redefine myself and move in to a new career? 
     **What can I do about a difficult boss? 
     **How do I add more fun or creativity to my job? 
     **What can I do to get beyond where I am today?
     
     In the upcoming program in the career development series, Getting to the
Top, executive coaches share tools and tips for getting unstuck in your career.
     
     Featured executive coaches: 
     **Susan Bernstein, Work from Within 
     **Deb Colden, Executive Coach and Consultant