Icecast dev - Aug 2004 - icecast2 ogg vorbis client request headers

At 10:33 PM 4/4/2004, you wrote:
>I'm not entirely sure of the performance implications on win32 - I'm
told
>it'll be substantially slower than on unix systems, but whether this
matters
>or not I'm not sure. Certainly, on a typical unix system, this approach
(fork
>and exec some program for each and every listener connection) will
>drastically out-perform an HTTP request for simple cases (i.e. if, in both
>cases, the actual authentication logic is simple/quick). The simple/naive
>implementation will definately scale acceptably well on unix systems.
quick and dirty I ran the following 2 programs :
/*****************************************************************************/
#include <stdio.h>
#include <curl/curl.h>

int main(void)
{
         CURL *curl;
         CURLcode res;
         int i = 0;
         curl = curl_easy_init();
         if(curl) {
                 curl_easy_setopt(curl, CURLOPT_URL, 
"http://localhost:8500/test.html");
                 for (i=0;i<1000;i++) {
                         res = curl_easy_perform(curl);
                         if (res != 0) {
                                 fprintf(stderr, "Curl failed: %d\n",
res);
                         }
                 }
                 /* always cleanup */
                 curl_easy_cleanup(curl);
         }
         return 0;
}
/*****************************************************************************/

/*****************************************************************************/
#include <stdio.h>

int main() {
         int i = 0;
         for (i=0;i<1000;i++) {
                 system("ls > /dev/null");
         }
}
/*****************************************************************************/

and got the following results :

/usr/home/oddsock/tmp-> time forkit > /dev/null
     5.13s real     1.82s user     3.27s system
/usr/home/oddsock/tmp-> time curlit > /dev/null
     0.76s real     0.14s user     0.18s system

o as far as performance goes, the HTTP request seemed to outperform in the 
simple/quick case.

from what I know, process creation is extremely expensive, much more so 
that thread creation.  Which is why I thought properly tuned apache 
installs are not configured to fork for each request.  Now, I agree a more 
proper implementation would require a fork once, and communicate multiple 
authentications over a pre-connected pipe, but still, you have to deal with 
possibly saturating that pipe, and also you've not make it a bit more 
complex to write a script that is pluggable into this authentication scheme.
>For this sort of thing, most implementations probably won't be
simple/quick,
>so the difference will likely end up in the noise (if the HTTP request is on
>a local network, at least). Persistent HTTP connections (for which we
don't
>have the infrastructure currently) help this a lot - but similarly, given
>more effort, the script approach could use a persistent "authentication
>daemon". You can get sufficient performance either way if you really
need it.
ince we would use libcurl for making HTTP requests, it supports http 
keep-alive requests...and so yes, persistent HTTP connections ARE possible.
>This approach also lets you get the http-request approach 'for free'
- you
>just provide a script that does an HTTP request to some URL and returns the
>appropriate value.
hardly 'for free' and this would be a horribly inefficient method of
doing it.

<p><p><p>> >
> > But obviously I would not be against implementing a script-based
> > authenticator, although this is pretty useless for most Win32 users
(yes
> > there are a bunch of those out there) as most Win32 users don't
generally
> > script things...
>
>The only reason this would be 'useless for most win32 users' is if
the
>overhead of running an executable for every request is too high on that OS.
I
>don't know whether it is. Remember, even if we call it a "script
>authentication module", there's nothing to force it to be in any
sort of
>actual "scripting language".
right, wether it's a script or a C program, it's still a fact that most
(if
not all) win32 users would not know what to do with a "script 
authentication module"...And again, I'm not saying that it
shouldn't be
implemented, but rather that I think it would be in the best interest of 
everyone if both methods (script and HTTP request) are implemented.

oddsock 

<p>--- >8 ----
List archives:  http://www.xiph.org/archives/
icecast project homepage: http://www.icecast.org/
To unsubscribe from this list, send a message to
'icecast-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

>but rather that I think it would be in the best interest of
>everyone if both methods (script and HTTP request) are implemented.
I completely agree on this, as it gives icecast2 much more flexibility and
options to choose from.

<p><p>Dave St John
Mediacast1 Administration

Need Support ?
http://mediacast1.com/helpdesk
----- Original Message ----- 
From: "oddsock" <oddsock@oddsock.org>
To: <icecast-dev@xiph.org>
Sent: Sunday, April 04, 2004 10:07 PM
Subject: Re: [icecast-dev] icecast2 ogg vorbis client request headers

<p>> At 10:33 PM 4/4/2004, you wrote:
>
> >I'm not entirely sure of the performance implications on win32 -
I'm told
> >it'll be substantially slower than on unix systems, but whether
this
matters
> >or not I'm not sure. Certainly, on a typical unix system, this
approach
(fork
> >and exec some program for each and every listener connection) will
> >drastically out-perform an HTTP request for simple cases (i.e. if, in
both
> >cases, the actual authentication logic is simple/quick). The
simple/naive
> >implementation will definately scale acceptably well on unix systems.
>
> quick and dirty I ran the following 2 programs :
>
/***************************************************************************
**/
> #include <stdio.h>
> #include <curl/curl.h>
>
> int main(void)
> {
>          CURL *curl;
>          CURLcode res;
>          int i = 0;
>          curl = curl_easy_init();
>          if(curl) {
>                  curl_easy_setopt(curl, CURLOPT_URL,
> "http://localhost:8500/test.html");
>                  for (i=0;i<1000;i++) {
>                          res = curl_easy_perform(curl);
>                          if (res != 0) {
>                                  fprintf(stderr, "Curl failed:
%d\n",
res);
>                          }
>                  }
>                  /* always cleanup */
>                  curl_easy_cleanup(curl);
>          }
>          return 0;
> }
> /*************************************************************************
****/
>
>
/***************************************************************************
**/
> #include <stdio.h>
>
> int main() {
>          int i = 0;
>          for (i=0;i<1000;i++) {
>                  system("ls > /dev/null");
>          }
> }
>
/***************************************************************************
**/
>
> and got the following results :
>
> /usr/home/oddsock/tmp-> time forkit > /dev/null
>      5.13s real     1.82s user     3.27s system
> /usr/home/oddsock/tmp-> time curlit > /dev/null
>      0.76s real     0.14s user     0.18s system
>
> so as far as performance goes, the HTTP request seemed to outperform in
the
> simple/quick case.
>
> from what I know, process creation is extremely expensive, much more so
> that thread creation.  Which is why I thought properly tuned apache
> installs are not configured to fork for each request.  Now, I agree a more
> proper implementation would require a fork once, and communicate multiple
> authentications over a pre-connected pipe, but still, you have to deal
with
> possibly saturating that pipe, and also you've not make it a bit more
> complex to write a script that is pluggable into this authentication
scheme.
>
> >For this sort of thing, most implementations probably won't be
simple/quick,
> >so the difference will likely end up in the noise (if the HTTP request
is
on
> >a local network, at least). Persistent HTTP connections (for which we
don't
> >have the infrastructure currently) help this a lot - but similarly,
given
> >more effort, the script approach could use a persistent
"authentication
> >daemon". You can get sufficient performance either way if you
really need
it.
>
> since we would use libcurl for making HTTP requests, it supports http
> keep-alive requests...and so yes, persistent HTTP connections ARE
possible.
>
> >This approach also lets you get the http-request approach 'for
free' -
you
> >just provide a script that does an HTTP request to some URL and returns
the
> >appropriate value.
>
> hardly 'for free' and this would be a horribly inefficient method
of doing
it.
>
>
>
>
> > >
> > > But obviously I would not be against implementing a script-based
> > > authenticator, although this is pretty useless for most Win32
users
(yes
> > > there are a bunch of those out there) as most Win32 users
don't
generally
> > > script things...
> >
> >The only reason this would be 'useless for most win32 users' is
if the
> >overhead of running an executable for every request is too high on that
OS. I
> >don't know whether it is. Remember, even if we call it a
"script
> >authentication module", there's nothing to force it to be in
any sort of
> >actual "scripting language".
>
> right, wether it's a script or a C program, it's still a fact that
most
(if
> not all) win32 users would not know what to do with a "script
> authentication module"...And again, I'm not saying that it
shouldn't be
> implemented, but rather that I think it would be in the best interest of
> everyone if both methods (script and HTTP request) are implemented.
>
> oddsock
>
>
> --- >8 ----
> List archives:  http://www.xiph.org/archives/
> icecast project homepage: http://www.icecast.org/
> To unsubscribe from this list, send a message to
'icecast-dev-request@xiph.org'
> containing only the word 'unsubscribe' in the body.  No subject is
needed.
> Unsubscribe messages sent to the list will be ignored/filtered.
>
<p>--- >8 ----
List archives:  http://www.xiph.org/archives/
icecast project homepage: http://www.icecast.org/
To unsubscribe from this list, send a message to
'icecast-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

On Sun, 4 Apr 2004, oddsock wrote:
> quick and dirty I ran the following 2 programs :
> and got the following results :
> 
> /usr/home/oddsock/tmp-> time forkit > /dev/null
>     5.13s real     1.82s user     3.27s system
> /usr/home/oddsock/tmp-> time curlit > /dev/null
>     0.76s real     0.14s user     0.18s system
> 
> so as far as performance goes, the HTTP request seemed to outperform in the
> simple/quick case.
That is on a Unix system?
> from what I know, process creation is extremely expensive, much more so
that
> thread creation.
It depends on what kind of thread you're talking about, and on what
system. On Windows, this is generally true. On Linux, linux threads that
use the clone system call are just as expensive as processes (they show up
as processes too). User space threads that use setjmp/ucontext are cheap
(they're "fibers" in Windows call, I believe).
> Which is why I thought properly tuned apache installs are
> not configured to fork for each request.
Well, if it's better to avoid thread creation algether, what about a
single threaded, multiplexed I/O web server?
> since we would use libcurl for making HTTP requests, it supports http
> keep-alive requests...and so yes, persistent HTTP connections ARE possible.
> 
> > This approach also lets you get the http-request approach 'for
free' - you
> > just provide a script that does an HTTP request to some URL and
returns the
> > appropriate value.
> 
> hardly 'for free' and this would be a horribly inefficient method
of doing it.
I think the hope is that one could amortize the cost because (1) 
connections to a media streaming server aren't supposed to be made 
frequently, (2) the authentication process (parsing a database file, etc) 
would take a while, or (3) authentication over the network is subject to 
network latency.

I was going to suggest if efficiency is a very important issue, one should
use shared library for authentication, but then you probably don't want to
deal with all the different kinds of shared libraries on systems, DLL on
windows, *.so for sane Unix, and *.dylib versus bundles on Darwin/Mac OS 
X. And I hate libtools...

It would possibly be useless for most users too, but it does take all the 
authentication logic outside of icecast, and it's efficient enough.

liulk
--- >8 ----
List archives:  http://www.xiph.org/archives/
icecast project homepage: http://www.icecast.org/
To unsubscribe from this list, send a message to
'icecast-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

On Monday 05 April 2004 13:04, oddsock wrote:
> At 07:41 PM 4/4/2004, you wrote:
> >That was your plan.
> >
> >My plan is to provide what currently exists (htpasswd-like) and a
'script'
> >authenticator which just calls an external program, as Geoff described.
> >
> >Both are, of course, possible. Your "URL call" (this is a
very strange way
> > to describe it, by the way - I assume you mean "HTTP
connection")
> > approach is much more complex to implement well, though, without
> > obviously being more powerful.
>
> invoking a URL is what I mean, I call it "URL Call" you call it
HTTP
> connection, it's the same thing...as you well know.
Yes. Your meaning was clear, but it's very unconventional (and inaccurate, 
technically - which matters later on when we document it, though not so much 
now) teminology. 
>
> and considering the way many broadcasters have implemented authentication
> (and I know more than a few that have, and they ALL have done it via a
"URL
> call" type method), I would say this method is more conventional than
the
> "invoke a script" type method.  And if you ask me, having icecast
fork and
> exec a script to do listener authentication is not going to scale too well
> (I hope you don't plan to fork/exec for every listener connection). 
AND we
> already have the infrastructure for making URL calls (the YP code has to do
> it).
I'm not entirely sure of the performance implications on win32 - I'm
told
it'll be substantially slower than on unix systems, but whether this matters
or not I'm not sure. Certainly, on a typical unix system, this approach
(fork
and exec some program for each and every listener connection) will 
drastically out-perform an HTTP request for simple cases (i.e. if, in both 
cases, the actual authentication logic is simple/quick). The simple/naive 
implementation will definately scale acceptably well on unix systems.

For this sort of thing, most implementations probably won't be simple/quick,
so the difference will likely end up in the noise (if the HTTP request is on 
a local network, at least). Persistent HTTP connections (for which we don't 
have the infrastructure currently) help this a lot - but similarly, given 
more effort, the script approach could use a persistent "authentication 
daemon". You can get sufficient performance either way if you really need
it.

This approach also lets you get the http-request approach 'for free' -
you
just provide a script that does an HTTP request to some URL and returns the 
appropriate value. 

<p><p><p>>
> But obviously I would not be against implementing a script-based
> authenticator, although this is pretty useless for most Win32 users (yes
> there are a bunch of those out there) as most Win32 users don't
generally
> script things...
The only reason this would be 'useless for most win32 users' is if the 
overhead of running an executable for every request is too high on that OS. I 
don't know whether it is. Remember, even if we call it a "script 
authentication module", there's nothing to force it to be in any sort
of
actual "scripting language". 

Mike

--- >8 ----
List archives:  http://www.xiph.org/archives/
icecast project homepage: http://www.icecast.org/
To unsubscribe from this list, send a message to
'icecast-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.