asterisk users - Jan 2005 - Changes to manager outputs - A discussion

I am currently writing a prototype agent monitoring system, which (as 
most others in question) simply monitors the output from the event 
system, and displays the relevant information. I would hope to donate 
this back to the community once it works properly :)

However, I feel that it would be more useful for the manager output to 
be tab delimited, one record per line:

i.e. instead of

Event: Agents
Agent: 6038
Name: A User
Status: AGENT_LOGGEDOFF
LoggedInChan: n/a
LoggedInTime: 0
TalkingTo: n/a

Event: Agents
Agent: 6039
Name: An Other
Status: AGENT_LOGGEDOFF
LoggedInChan: n/a
LoggedInTime: 0
TalkingTo: n/a

I would prefer to see the following format:

Event~tAgents~t1~t6038~tA User~tAGENT_LOGGEDOFF~tn/a~t0~tn/a
Event~tAgents~t1~t6039~tAn Other~tAGENT_LOGGEDOFF~tn/a~t0~tn/a

Where ~t is a tab character, and the layout starts
Type~tName~tVersion~tData

Version is the version number of the data layout for the Agents Event. 
For version 1 we have

AgentNum/Name/Status/LoggedinChan/LoggedinTime/Talkingto

This allows us to add further fields to the record as and when required 
and not break any existing code in the field.

In my mind, (yes, a small one compared to the giants walking around 
here) There are several advantages in this method:

a) Parsing one line of data per record is in order of magnitude easier 
to code.

b) As mentioned, further fields can be added at any time without 
breaking code

c) output can be exported directly into spreadsheets

d) It is very easy to skip records that you are not interested in - If 
field 2 is not "Agents" then ignore

In order not to break existing code, we could provide a flag or setting 
to determine which output format to use: further more we could slowly 
update all of the manager events until they are all available in old and 
new formats.

I know that perhaps I've talked a load of BS - I would appreciate it if 
people could comment on this before I head up a blind alley. I feel that 
it would be more useful and easier for us as developers if there were a 
common event manager layout, rather than a fixed number of lines per 
action type / event type, and one that follows a more common data / 
record layout.

Please feel free to comment / flame whatever.

exten => 999,n,GetFlameSuit(MaxStrength)
exten => 999,n,WearFlameSuit(MaxStrength)
exten => 999,n,HoldBreathAndWaitToBeFlamedBigTime()

Julian.

On Tue, 11 Jan 2005, Asterisk wrote:
> In my mind, (yes, a small one compared to the giants walking around 
> here) There are several advantages in this method:
> 
> a) Parsing one line of data per record is in order of magnitude easier 
> to code.
Not really, unless you have to invent string handling first. It is very 
easy in C++/perl/java etc. It is not even hard in C with no libraries at 
all.
> b) As mentioned, further fields can be added at any time without 
> breaking code
The current format is tagged. How much easier can it get to add more 
fields? At the moment the order is not fixed either which is nice from a 
flexibility point. You can have optional fields that ar only output if 
they make sense.
> c) output can be exported directly into spreadsheets
True, but a trivial amount of script magic can transform the current 
tagged format into whatever you want. For more major post processing 
converting it to xml first may be useful/flexible.
> I know that perhaps I've talked a load of BS - I would appreciate it if
> people could comment on this before I head up a blind alley. I feel that 
> it would be more useful and easier for us as developers if there were a 
> common event manager layout, rather than a fixed number of lines per 
> action type / event type, and one that follows a more common data / 
> record layout.
It is not fixed, I think thats is why you are thinking along these lines. 
It is a tagged format. Given the nature of the data that is sent (what 
fields are valid may vary) a tagged format is probably the only sane way 
of representing the data.

Peter

Comments inline...

Asterisk wrote:
> I am currently writing a prototype agent monitoring system, which (as 
> most others in question) simply monitors the output from the event 
> system, and displays the relevant information. I would hope to donate 
> this back to the community once it works properly :)
> 
Great :)
> However, I feel that it would be more useful for the manager output to 
> be tab delimited, one record per line:
<snip>

Why?  How?
> 
> In my mind, (yes, a small one compared to the giants walking around 
> here) There are several advantages in this method:
> 
Don't belittle yourself, everyone's brain starts out as ONE cell...
But grows exponentially :)
> a) Parsing one line of data per record is in order of magnitude easier 
> to code.
> 
This is language and implementation specific.  I can parse a million 
line file or a one line file with the same piece of code, just depends 
on how you choose to do it.
> b) As mentioned, further fields can be added at any time without 
> breaking code
> 
Same with the current format, even more so that your tab delimited idea.
> c) output can be exported directly into spreadsheets
> 
Ok, on this one you win.
> d) It is very easy to skip records that you are not interested in - If 
> field 2 is not "Agents" then ignore
> 
Again, implementation specific.  For instance, you've got to read the 
whole pipe to clear way for the next message regardless of your method 
or the current, and with my statement for your point 'a', again - no 
real gain.
> In order not to break existing code, we could provide a flag or setting 
> to determine which output format to use: further more we could slowly 
> update all of the manager events until they are all available in old and 
> new formats.
> 
This already exists.  Need a new key/value pair in the event, just go 
add it.  You won't break anyone else's parser unless they did exactly 
what you say in the next paragraph and assume the messages are fixed in 
all respects - which they aren't.
> I know that perhaps I've talked a load of BS - I would appreciate it if
> people could comment on this before I head up a blind alley. I feel that 
> it would be more useful and easier for us as developers if there were a 
> common event manager layout, rather than a fixed number of lines per 
> action type / event type, and one that follows a more common data / 
> record layout.
> 
etc...  Again, your format is even more 'fixed'.  With your layout you 
have to specify the version you want * to give you, otherwise * has to 
spit out all versions to retain compatibility.  Both of these involve 
more work than the current model - for the programmer and for the 
running * process.
> Please feel free to comment / flame whatever.
> 
> exten => 999,n,GetFlameSuit(MaxStrength)
> exten => 999,n,WearFlameSuit(MaxStrength)
> exten => 999,n,HoldBreathAndWaitToBeFlamedBigTime()
Dial(flaming/999)

-Chris

-- 
Christopher L. Wade                     Unistar-Sparco Computers, Inc.
Senior Systems Administrator                            dba Sparco.com
Email: clwade@sparco.com                             7089 Ryburn Drive
Phone: (901) 872 2272 / (800) 840 8400            Millington, TN 38053
Fax:   (901) 872 8482                                              USA

Sometimes things are so obvious that you miss them. "just view a single LF
as the field separator and a double LF as the record separator" is, of
course, the point that makes me look soooooo stupid.

Seems as if the flamesuit was not good enough. Consider me crashed and
burned.

Can I pretend I never sent this email ?

:)
Julian

-----Original Message-----
From: asterisk-users-bounces@lists.digium.com
[mailto:asterisk-users-bounces@lists.digium.com] On Behalf Of Peter Svensson
Sent: 12 January 2005 08:24
To: Asterisk Users Mailing List - Non-Commercial Discussion
Subject: Re: [Asterisk-Users] Changes to manager outputs - A discussion

On Wed, 12 Jan 2005, Asterisk wrote:
> Peter Svensson wrote:
> > On Tue, 11 Jan 2005, Asterisk wrote:
> > 
> >>b) As mentioned, further fields can be added at any time without 
> >>breaking code
> > 
> > 
> > The current format is tagged. How much easier can it get to add more 
> > fields? At the moment the order is not fixed either which is nice 
> > from a flexibility point. You can have optional fields that ar only 
> > output if they make sense.
> 
> This was part of my problem - there was a varying number of lines per 
> "record" which made my processing easier if it were one record
per line.
> Again, that could be my blindness die to the language I use.
You just view a single LF as the field separator and a double LF as the
record separator. In other workds, instead of assembling the record until
the string "\n" is found you assemble until "\n\n" is found.
> >>c) output can be exported directly into spreadsheets
> > 
> > True, but a trivial amount of script magic can transform the current 
> > tagged format into whatever you want.
> 
> Why have all the various "listners" have to process the output,
rather
> than have it presented ?
Because it is _much_ simpler to parse and update a tagged format than a tab
separated one? Remember, the events and command responses are _not_ lines
from a database with exactly the same fields each time. Think of them as
overloaded RPC perhaps.
> > It is not fixed, I think thats is why you are thinking along these
lines. 
> > It is a tagged format. Given the nature of the data that is sent 
> > (what fields are valid may vary) a tagged format is probably the 
> > only sane way of representing the data.
> 
> If the data had a "start/end" tag, I would agree. However, it is 
> difficult to find the end of the data because of the "optional"
fields.
Not at all hard. If you insist on using line buffered reads then you read
until you find an empty line. If not, you just read until the token
"\n\n"
is found.

Peter

_______________________________________________
Asterisk-Users mailing list
Asterisk-Users@lists.digium.com
http://lists.digium.com/mailman/listinfo/asterisk-users
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users

On Wed, 12 Jan 2005, Asterisk wrote:
> Sometimes things are so obvious that you miss them. "just view a
single LF
> as the field separator and a double LF as the record separator" is, of
> course, the point that makes me look soooooo stupid.
Note that the order of the elements is only defined by their tags. You can 
not rely on a specific order and just ignore the tags. Reordering the tags 
once the records are isolated should not be too hard, if you wish to 
import to a spreadsheet.

Peter