Vorbis - Oct 2004 - Help please streaming oggs as they are being created

Hi all,

I often record radio shows for posterity, and sometimes I have friends
who would like to listen to them live over the net.  When I am
recording for my own purposes, I use a command of the form: "brec |
oggenc", (options omitted for clarity) and I send the output to a
file, call it radio.ogg.

Locally, I use Debian stable.  If I want to listen to the show as I
record it, I can use xmms, the command "xmms radio.ogg" does the
trick.  xmms pops up with the current length of the recording - if it
is 20 minutes into the show, xmms reports a file length of 20 minutes.
If I want, I can "seek" into the file, by dragging a control, which
takes me into the file.  When I reach the 20 minute mark, the file
continues playing, even though the file length continues to be
indicated as 20 minutes.

I would like to set things up so someone listening to the same file
over the web can get the same results.  On a Linux machine, this would
probably be using xmms, on a Windows machine it seems WinAmp is one
common player.  I am new to both Apache and html.  I have tried to
google on this, and specifically google the xiph.org archives.  I have
tried three ways: linking to the file like any other in Apache, via
the mod_mp3 module for Apache, and via a python script web-server
called Edna.

Nothing does what I want.  Edna was easy to set up and use.  When I go
to my own edna web page, xmms opens and streams radio.ogg just fine,
but there is no seek control displayed, so I can't move forward in the
file to the current time.  However, the file does play through to the
end.  So if I start listening 20 minutes after a recording starts, I
can listen to the whole recording, but 20 minutes delayed.  I haven't
tested edna from Windows.

Using straight Apache, the entire file downloads before it starts to
play (no streaming) and there IS a seek control, so I can move forward
to the current time.  Which is good, except xmms stops playing when it
reaches the end of the file, even if the file has gotten larger on the
server.  (So if I start listening 20 minutes into the recording, I can
only hear 20 minutes of the recording.)  I have either not gotten
mod_mp3 to work properly, or it exhibits the same behavior.

Reading google and the xiph.org archives there seems to be a lot of
ideas on how to make this work, but nothing definitive.  Things I have
seen mentioned include:

  -using ices/icecast
  -setting the content-length field to 0
  -omitting the content-length field
  -changing the Cache-Control field
  -changing the Expires field
  -changing the Content-Type field
  -changing to MP3, which is better suited to seeking

All of the above I am sure I *could* do.  But I have little experience
with the needed tools.  I also can not easily test windows apps, as I
don't have access to a windows machine without special arrangements.
I am hoping people on this list might have definitive experience with
what works, and / or what does not work.  This would save me a lot of
time trying things in a somewhat random order.

Thank you,

-Ben Blout

On Sun, Oct 24, 2004 at 12:30:53PM -0400, ben-extra@MIT.EDU wrote:
> Nothing does what I want.  Edna was easy to set up and use.  When I go
> to my own edna web page, xmms opens and streams radio.ogg just fine,
> but there is no seek control displayed, so I can't move forward in the
> file to the current time.  However, the file does play through to the
> end.  So if I start listening 20 minutes after a recording starts, I
> can listen to the whole recording, but 20 minutes delayed.  I haven't
> tested edna from Windows.
> 
> Using straight Apache, the entire file downloads before it starts to
> play (no streaming) and there IS a seek control, so I can move forward
> to the current time.  Which is good, except xmms stops playing when it
> reaches the end of the file, even if the file has gotten larger on the
> server.  (So if I start listening 20 minutes into the recording, I can
> only hear 20 minutes of the recording.)
Yes. The issue here is seekability over the network. You can do this over
http, but afaik alsaplayer is the only one that supports it. You might
try that. I'm not sure if you'll need to set some of the no-cache
headers
for apache to keep serving the growing file or not.

Seeking over http is slow because it has to do a binary search to find
the seek point. An obvious optimization is to do the seek on the server
side. It's possible to write an apache extension to implement time-based
units for the Range: headers as well as bytes (if you want a fun little
project :). There's also an internet draft for addressing temporal
subranges through uri fragments. You might take a look at annodex.net; 
they have a mod_annodex that can return pre-seeked bits, but I don't know
if it works on generic ogg files or not.
> Reading google and the xiph.org archives there seems to be a lot of
> ideas on how to make this work, but nothing definitive.  Things I have
> seen mentioned include:
> 
>   -using ices/icecast
This can replicate what you're hearing for others over the net, but 
doesn't help anyone else do independent seeking.
>   -changing to MP3, which is better suited to seeking
Ha! This is not true. Except for the way metadata is conventionally
done it's not much worse either, so you might as well use Vorbis.

Hope that helps,
 -r

Hi:

The other problem you're going to have with Apache, assuming it can cope 
with growing files, is the rate at which data is streamed.  Your player 
will buffer just fine, but Apache will presumably push the data as fast as 
its able.  This will probably be faster than the file's bitrate, so Apache 
will run out of data to send when it catches up to the end of the file.

Geoff.


-- 
Geoff Shang <geoff@hitsandpieces.net>
Phone: +61-418-96-5590
MSN: geoff@acbradio.org

Make sure your E-mail can be read by everyone!
http://www.betips.net/etc/evilmail.html

Please avoid sending me Word or PowerPoint attachments.
See http://www.fsf.org/philosophy/no-word-attachments.html

Touche.


--- Ralph Giles <giles@xiph.org> wrote:
> On Mon, Oct 25, 2004 at 03:49:19AM -0700, HJ wrote:
> 
> > I thought that when streaming, upstream bandwidth is *almost* always
more
> of
> > a consideration than latency.  Though, in this case, yes, latency will
> > probably be noticibly higher than, say, fetching the stream from
icecast.
> 
> Upstream bandwidth is the limiting factor for how many simultaneous 
> streaming clients one may support. I was just talking about seek
> performance over http. Because the client has to do multiple seeks
> to converge on the byte offset of the designed temporal location,
> round-trip latency is the dominant factor for seeking.
> 
>  -r
>

On Sun, 24 Oct 2004 12:30:53 -0400, ben-extra wrote:
> Hi all,
> 
> I often record radio shows for posterity, and sometimes I have friends
> who would like to listen to them live over the net.  When I am
> recording for my own purposes, I use a command of the form: "brec |
> oggenc", (options omitted for clarity) and I send the output to a
> file, call it radio.ogg.
> 
> Locally, I use Debian stable.  If I want to listen to the show as I
> record it, I can use xmms, the command "xmms radio.ogg" does the
> trick.  xmms pops up with the current length of the recording - if it
> is 20 minutes into the show, xmms reports a file length of 20 minutes.
> If I want, I can "seek" into the file, by dragging a control,
which
> takes me into the file.  When I reach the 20 minute mark, the file
> continues playing, even though the file length continues to be
> indicated as 20 minutes.
> 

If all you want is to be able to record to disk and listen to the stream
live at the same time (i.e. the seeking is just to get to the currently
recording part of the file), you could use ices to stream to an icecast
server, and use its file dump feature to save a local copy at the same
time. see www.icecast.org.

This is an ices.xml config file I've been using to stream and record at
two bitrates using the ices-kh branch of ices, which can take its input
from the jack audio daemon. The save file name is filled in with the
current date and time as in the strftime function (man 3 strftime), and a
new file is started whenever the stream metadata is updated (by sending
sig USR1 to ices)

<?xml version="1.0"?>
<ices>

        <background>0</background> <!-- run in background?
(unimplemented) -->
        <logpath>/local/radio/log</logpath> <!-- where logs, etc
go. -->
        <logfile>ices.log</logfile>
        <loglevel>4</loglevel> <!-- 1=error,2=warn,3=info,4=debug
-->
    <logsize>2048</logsize> <!-- the size the log file must be
before rotation -->
    <consolelog>0</consolelog> <!-- set this to 1 to log to the
console instead
                                    of to the file above -->
    <pidfile>/local/radio/run/ices-ogg.pid</pidfile> <!-- file to
write process id to -->

        <stream>

                <!-- input module -->
                <input>
                        <module>jack</module>
                        <param name="channels">2</param>
                        <param
name="clientname">ices-ogg</param>
                        <param
name="metadatafilename">/local/radio/tmp/ices-metadata</param>
                        <param name="ignorexrun">1</param>
                        <param
name="connect">ecastream:out_1,ecastream:out_2</param>
                </input>

        <runner>
            <!-- Stream instance -->
            <instance>
                <shout>
                    <!-- Server details: -->
                    <hostname>localhost</hostname>
                    <port>8000</port>
                    <password>quarmoor</password>
                    <mount>/mono.ogg</mount>
                    <reconnectdelay>2</reconnectdelay>
                    <reconnectattempts>5</reconnectattempts>
                </shout>
                <downmix>1</downmix>
                <savefile>
                   
<filename>/local/radio/record/icecast-32-m_%d-%b-%Y-%H:%M:%S_.ogg</filename>
                    <duration>28800</duration> <!-- max 8 hours
-->
                    <on-metadata>1</on-metadata>
                    </savefile>
                <resample>
                    <out-rate>22050</out-rate>
                    </resample>
                <encode>
                    <!-- VBR mode can be selected via quality or
nominal-bitrate -->
                    <!-- selecting enabling managed has to be done separately
as -->
                    <!-- it's slower, but can be used to limit a high
threshold -->
<!--                <managed>1</managed>
                    <maximum-bitrate>24000</maximum-bitrate> -->
                    <nominal-bitrate>19456</nominal-bitrate>
                </encode>
            </instance>
            <instance>
                <shout>
                    <!-- Server details: -->
                    <hostname>localhost</hostname>
                    <port>8000</port>
                    <password>quarmoor</password>
                    <mount>/stereo.ogg</mount>
                    <reconnectdelay>2</reconnectdelay>
                    <reconnectattempts>5</reconnectattempts>
                </shout>
                <savefile>
                   
<filename>/local/radio/record/icecast-64-s_%d-%b-%Y-%H:%M:%S_.ogg</filename>
                    <duration>28800</duration> <!-- max 8 hours
-->
                    <on-metadata>1</on-metadata>
                    </savefile>
                <resample>
                    <out-rate>22050</out-rate>
                    </resample>
                <encode>
                    <!-- VBR mode can be selected via quality or
nominal-bitrate -->
                    <!-- selecting enabling managed has to be done separately
as -->
                    <!-- it's slower, but can be used to limit a high
threshold -->
<!--                <managed>1</managed>
                    <maximum-bitrate>80000</maximum-bitrate>-->
                    <nominal-bitrate>65536</nominal-bitrate>
                </encode>
            </instance>
        </runner>
        </stream>
</ices>