Eventmachine talk - May 2007 - Multi-char support in BufferedTokenizer

Hey all,

This seemed like the approriate place to post this as I noticed that
Tony Arcieri''s BufferedTokenizer has been included into EM svn.

The protocol I am designing was in need of multi-character token
support.  I have attached a small patch to buftok.rb that does just
that.  I''ve run a lot of tests on it and it seems to work fine but
I''d
be interested in anyone''s feedback to be sure.  Note that this patch
was run against the buftok.rb that is found in EM''s svn not the gem.

One thing to be aware of, the test code:

#!/usr/bin/ruby

require ''./buftok_multichar.rb''

$i = 0
$buf = BufferedTokenizer.new("XXX")

def recv(str)
  puts "Recv #{$i+=1}:"
  $buf.extract(str).each do |e|
    puts e
  end
end

recv("aaaXXXbbbXXXcccXXXdddXXX")
recv("eeeXXXfffXXXgggXXXhhhXX")
recv("XiiiXXXjjjXXXkkkXXXlllX")
recv("XXmmmXXXnnnXXXoooXXXppp")
recv("XXX")

will produce the following output:
Recv 1:
aaa
bbb
ccc
ddd
Recv 2:
eee
fff
ggg
Recv 3:
hhh
iii
jjj
kkk
Recv 4:
lll
mmm
nnn
ooo
Recv 5:
ppp

It splits everything properly, however, notice that ''lll'' was
passed
in on the third call to recv() but not printed until the fourth call.
Same thing with ''hhh'' and ''ppp''. 
I''m not sure if this is an issue or
not, just something to be aware of.

Francis, thanks a lot for EM, it rules.  And Tony, thanks for
BufferedTokenizer, it made my life easier.

Marc

All that and I forgot to attach the patch...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: buftok-multichar.patch
Type: text/x-patch
Size: 933 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070506/4430ad79/attachment-0001.bin

On 5/6/07, Marc Soda <marcantoniosr at gmail.com>
wrote:
>
> All that and I forgot to attach the patch...


Thanks for this. Tony, should this be added to the distro?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070506/d056ad90/attachment.html

Sounds good to me.

- Tony

On 5/6/07, Francis Cianfrocca <garbagecat10 at gmail.com>
wrote:
>
> On 5/6/07, Marc Soda <marcantoniosr at gmail.com> wrote:
> >
> > All that and I forgot to attach the patch...
>
>
>
> Thanks for this. Tony, should this be added to the distro?
>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
(970) 232-4208
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070506/37652184/attachment.html

If the patch looks good to you, Tony, go ahead and check it in. Mark, can
you provide some tests that exercise the patch?

On 5/6/07, Tony Arcieri <tony at clickcaster.com>
wrote:
>
> Sounds good to me.
>
> - Tony
>
> On 5/6/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:
>
> > On 5/6/07, Marc Soda <marcantoniosr at gmail.com> wrote:
> > >
> > > All that and I forgot to attach the patch...
> >
> >
> >
> > Thanks for this. Tony, should this be added to the distro?
> >
> >
> > _______________________________________________
> > Eventmachine-talk mailing list
> > Eventmachine-talk at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> >
>
>
>
> --
> Tony Arcieri
> ClickCaster, Inc.
> tony at clickcaster.com
> (970) 232-4208
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070507/4524b1e3/attachment.html

On 5/7/07, Francis Cianfrocca <garbagecat10 at gmail.com>
wrote:
> If the patch looks good to you, Tony, go ahead and check it in. Mark, can
> you provide some tests that exercise the patch?
>
Here is the test script I was using.  I''m sure it doesn''t
cover every
scenario possible, but I can''t think of anything else at the moment.

#!/usr/bin/ruby -w

require ''eventmachine''
require ''base64''
require ''./buftok-multichar''

TOKEN = Base64.encode64(rand.to_s).chomp

$client_data = []

# Quick and dirty test for multichar tokenized buffers.
module TokenTestServer
  def post_init
    @recv_buffer = BufferedTokenizer.new(TOKEN)
    @server_data = []
  end

  def receive_data(stream)
    @recv_buffer.extract(stream).each do |m|
      @server_data << m
    end
  end

  def unbind
    if (($client_data <=> @server_data) == 0)
      puts "Good"
    else
      puts "Mismatch"
    end
    exit
  end
end

module TokenTestClient
  def post_init
    $client_data = []
    # Lot''s of data so we use multiple syscalls on both ends.
    5000.times do
      $client_data << rand.to_s
    end

    # I know this doesn''t guarentee these won''t be sent all in
    # one packet, but with with enough data we should get something.
    stream = $client_data.join(TOKEN) + TOKEN
    sent_bytes = 0
    stream_size = stream.size
    while (sent_bytes <= stream_size)
      bytes = rand(10)+1
      bytes = stream.size if (stream_size < bytes)
      send_data stream.slice!(0..bytes)
      sent_bytes += bytes
    end
    close_connection_after_writing
  end
end

EventMachine::run do
  EventMachine::start_server(''127.0.0.1'', 7777,
TokenTestServer)
  EventMachine::connect(''127.0.0.1'', 7777, TokenTestClient)
end

Marc

It''s checked in.

On 5/7/07, Francis Cianfrocca <garbagecat10 at gmail.com>
wrote:
>
> If the patch looks good to you, Tony, go ahead and check it in. Mark, can
> you provide some tests that exercise the patch?
>
> On 5/6/07, Tony Arcieri < tony at clickcaster.com> wrote:
> >
> > Sounds good to me.
> >
> > - Tony
> >
> > On 5/6/07, Francis Cianfrocca < garbagecat10 at gmail.com>
wrote:
> >
> > > On 5/6/07, Marc Soda <marcantoniosr at gmail.com> wrote:
> > > >
> > > > All that and I forgot to attach the patch...
> > >
> > >
> > >
> > > Thanks for this. Tony, should this be added to the distro?
> > >
> > >
> > > _______________________________________________
> > > Eventmachine-talk mailing list
> > > Eventmachine-talk at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > >
> >
> >
> >
> > --
> > Tony Arcieri
> > ClickCaster, Inc.
> > tony at clickcaster.com
> > (970) 232-4208
> > _______________________________________________
> > Eventmachine-talk mailing list
> > Eventmachine-talk at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> >
>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
(970) 232-4208
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070507/ff06ef50/attachment.html

Thanks guys.

On 5/7/07, Tony Arcieri <tony at clickcaster.com>
wrote:
>
> It''s checked in.
>
> On 5/7/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:
> >
> > If the patch looks good to you, Tony, go ahead and check it in. Mark,
> > can you provide some tests that exercise the patch?
> >
> > On 5/6/07, Tony Arcieri < tony at clickcaster.com> wrote:
> > >
> > > Sounds good to me.
> > >
> > > - Tony
> > >
> > > On 5/6/07, Francis Cianfrocca < garbagecat10 at gmail.com>
wrote:
> > >
> > > > On 5/6/07, Marc Soda <marcantoniosr at gmail.com>
wrote:
> > > > >
> > > > > All that and I forgot to attach the patch...
> > > >
> > > >
> > > >
> > > > Thanks for this. Tony, should this be added to the distro?
> > > >
> > > >
> > > > _______________________________________________
> > > > Eventmachine-talk mailing list
> > > > Eventmachine-talk at rubyforge.org
> > > > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > > >
> > >
> > >
> > >
> > > --
> > > Tony Arcieri
> > > ClickCaster, Inc.
> > > tony at clickcaster.com
> > > (970) 232-4208
> > > _______________________________________________
> > > Eventmachine-talk mailing list
> > > Eventmachine-talk at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > >
> >
> >
> > _______________________________________________
> > Eventmachine-talk mailing list
> > Eventmachine-talk at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> >
>
>
>
> --
> Tony Arcieri
> ClickCaster, Inc.
> tony at clickcaster.com
> (970) 232-4208
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070507/3f670aad/attachment-0001.html