theora dev - May 2011 - [Cortado] How to support seeking in on-the-fly generated Theora stream?

Hello all!

i want to use Theora in a little video portal, because its free and open 
source and i want to contribute some code to get the Java Theora player 
(Cortado) more feature-rich.
But now I'm stuck and hope that someone can point me into the right 
direction.

I have videos stored in several formats (mostly H.264 or MPEG4) and use 
ffmpeg2theora to recode them on the fly. The recoded output is send via 
HTTP stream directly to the Cotrado player which stores the data and 
begins playing if enough data is buffered.
In practice this looks like this is a PHP file:
passthru("ffmpeg2theora -o - ".$source." 2> /dev/null");

I recoded the Queue.java (Queue class) that the queue array is not a dump 
FIFO anymore and called it QueueSeek.java.
Now the Buffer array is filled at the end and a variable points to the 
current play position which gets increased on every loop and sends the 
data to the next sink pad (at OggDemux).

QueueSeek Buffer:

           +--# data added at the end
+0-1-2-3-4-v-------
|#|#|#|#|#| | | | -> unlimited capacity
+----|-------------
     v
current play position
data will be send to demuxer
but not removed from the FIFO

Now i managed that the fill level of the buffer is the difference between 
the size of the data in the FIFO and the current play position. So the 
play begins if the data is enough and stops if the "watermark" gets
below
the lowest level.

     + current play position
+0-1-V-3-4-5-6-----
|#|#|#|#|#|#|#| | 
+----|-------|-----
     |buffer-|
      level=4

Now i want to seek. And thats where the problems start to begin. I 
understood how the demuxer works. It gets the data from the Queue, stores 
it in its own buffer and seeks for a "OggS" pattern which marks the
start
of a new Theora page. Then is checks if that whole page is present in the 
buffer and calculates a checksum. If all went well the page is processed 
in a way i did not look about any further.
But if i seek in my new Queue - which means just setting the play position 
to any value - then the demuxer stumbles, because the checksum and the 
length of the page  are not correct.
To get around this i did a little trick:
On my new FIFO i seek though the incoming data and look for the OggS 
pattern. If it is present the data will be split into two segments and 
both will be inserted info the FIFO. Now every page starts at the 
beginning of a FIFO buffer:

+------0-------------1-------------2------------3-----
| OggS...data | OggS...data.|....data    | OggS...
+-----------------------------------------------------

As you can see, sometimes the data is lager than the 4096 byte of one 
buffer, so it takes more than one buffer for a page. I've taken care of 
that in the seeking method by rewinding the play position until the buffer 
starts with an OggS pattern.
But now a second problem exists: The seeking should start if one page is 
completely submittet. And thats where i'm stucking now.

The pads of the elements are separate threads which work independently 
from each other. 

----+        +-----------[QueueSeek element]---------+         +----
prev|        |                                       |   to    | sink pad
src O--push->O sink pad --> [FIFO magic] --> src pad O--next-->O of
the
pad |        |                                       | element | demuxer
----+        +---------------------------------------+         +----

Thats why many code blocks are synchronized on the queue object, that only 
one thread can access the queue at one time. Now, i want so synchronize 
the sink- and sourcepad threads of the QueueSeek class on another object, 
which is called "permissionToSeek". The sinkpad should wait until it
gets
a permission to seek.
I tried to achieve this with wait() and notifyAll().
the sinkpad has a method which gets called if a seek request arrives. It 
synchronizes on the permission object and waits for the srcpad to grant 
it:

private void doSeek(Event event) {
    // wait until we get the permission to seek
    synchronized (permissionToSeek) {
    try {
        Debug.info("[BUFFER] sink -> doSeek() waiting for 
permissionToSeek");
        // wait until we are allowed to seek
        permissionToSeek.wait();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
    /* ... code to seek ... */
}

The sinkpad checks if the next buffer begins with OggS pattern and calls 
the notifyAll() method:

protected void taskFunc() {
    boolean runSeekIfWanted = false;
    synchronized (queue) {
        /* ... */
        if (playPosition+1 < queue.size()) {
            obj = queue.elementAt(playPosition+1);
            if (obj instanceof Buffer) {
                Buffer bb = (Buffer) obj;
                if (bb.data[0] == 'O' &&
                    bb.data[1] == 'g' &&
                    bb.data[2] == 'g' &&
                    bb.data[3] == 'S') {
                    runSeekIfWanted = true;
                    Debug.info("[BUFFER] src setting
permissionToSeek");
                }
            }
        }
        /* ... */
    }
    /* ... main method code ... */
    synchronized (permissionToSeek) {
        if (runSeekIfWanted) {
            Debug.info("[BUFFER] src granting permissionToSeek");
            permissionToSeek.notifyAll();
        }
    }
}

The problem is that the applet stops in the doSeek() method at the wait() 
and waits forever. The whole applet does not respond anymore:

[INFO] [BUFFER] playPosition = 324, queue.size = 669 Elements, size = 
1359872 bytePlayPosition = 678325 -> 64% full
[INFO] [BUFFER] src granting permissionToSeek
[INFO] [BUFFER] playPosition = 325, queue.size = 669 Elements, size = 
1359872 bytePlayPosition = 679936 -> 64% full
[INFO] [BUFFER] playPosition = 326, queue.size = 669 Elements, size = 
1359872 bytePlayPosition = 679962 -> 64% full
[INFO] [BUFFER] src setting permissionToSeek
[INFO] doSeek(): aPos=0.05273069679849341
[INFO] doSeek(): Element: [pipeline].sendEvent [Event] type: SEEK, format: 
5, position: 52730
[INFO] Element: [pipeline] sendEvent(): [Event] type: SEEK, format: 5, 
position: 52730
[INFO] Element: [pipeline] doSeek(): [Event] type: SEEK, format: 5, 
position: 52730
[INFO] Element: [pipeline] doSendEvent(): [Event] type: SEEK, format: 5, 
position: 52730 to Pad: httpsrc:src
[INFO] [BUFFER] sink -> doSeek() waiting for permissionToSeek

If i do a wait for 5 seconds to get around this issue, i get discontinue 
messages in the console, and i don't know why:

[INFO] theora: got discont
[INFO] theora: got discont
[INFO] theora: got discont

So my first question is:
Is my whole thinking right with providing full pages to the demuxer? Is 
that the right way for seeking in a Theora video stream?
Why does my applet wait forever? I suppose the srcpad does not fire the 
notifyAll() method but i have no idea why or where the srcpad gets stopped 
or interrupted.

I will provide the whole code project in a .tar.bz here, so everyone can 
use and experiment with my code changes:
http://www.wapkamera.de/cortado.tar.bz

Hopefully someone is able to help me, because i just started in coding 
Java and i don't know the Theora video stream structure in detail.

Best regards to everyone.

David

I think byte-range serving[1] would be easy to do
or use Chunked transfer encoding[2] 

[1] http://en.wikipedia.org/wiki/Byte_serving
[2] http://en.wikipedia.org/wiki/Chunked_transfer_encoding
--
tom_a_sparks "It's a nerdy thing I like to do"
Please use ISO approved file formats excluding Office Open XML -
http://www.gnu.org/philosophy/no-word-attachments.html
3 x (x)Ubuntu 10.04, Amiga A1200 WB 3.1, UAE AF 2006 WB 3.X, Sam440 AOS 4.1


--- On Mon, 23/5/11, David Schueler <david.schueler at wapkamera.de>
wrote:
> From: David Schueler <david.schueler at wapkamera.de>
> Subject: [theora-dev] [Cortado] How to support seeking in on-the-fly
generated Theora stream?
> To: theora-dev at xiph.org
> Received: Monday, 23 May, 2011, 8:02 PM
> Hello all!
> 
> i want to use Theora in a little video portal, because its
> free and open 
> source and i want to contribute some code to get the Java
> Theora player 
> (Cortado) more feature-rich.
> But now I'm stuck and hope that someone can point me into
> the right 
> direction.
> 
> I have videos stored in several formats (mostly H.264 or
> MPEG4) and use 
> ffmpeg2theora to recode them on the fly. The recoded output
> is send via 
> HTTP stream directly to the Cotrado player which stores the
> data and 
> begins playing if enough data is buffered.
> In practice this looks like this is a PHP file:
> passthru("ffmpeg2theora -o - ".$source." 2>
> /dev/null");
> 
> I recoded the Queue.java (Queue class) that the queue array
> is not a dump 
> FIFO anymore and called it QueueSeek.java.
> Now the Buffer array is filled at the end and a variable
> points to the 
> current play position which gets increased on every loop
> and sends the 
> data to the next sink pad (at OggDemux).
> 
> QueueSeek Buffer:
> 
> ? ? ? ? ???+--# data
> added at the end
> +0-1-2-3-4-v-------
> |#|#|#|#|#| | | | -> unlimited capacity
> +----|-------------
> ? ???v
> current play position
> data will be send to demuxer
> but not removed from the FIFO
> 
> Now i managed that the fill level of the buffer is the
> difference between 
> the size of the data in the FIFO and the current play
> position. So the 
> play begins if the data is enough and stops if the
> "watermark" gets below 
> the lowest level.
> 
> ? ???+ current play position
> +0-1-V-3-4-5-6-----
> |#|#|#|#|#|#|#| | 
> +----|-------|-----
> ? ???|buffer-|
> ? ? ? level=4
> 
> Now i want to seek. And thats where the problems start to
> begin. I 
> understood how the demuxer works. It gets the data from the
> Queue, stores 
> it in its own buffer and seeks for a "OggS" pattern which
> marks the start 
> of a new Theora page. Then is checks if that whole page is
> present in the 
> buffer and calculates a checksum. If all went well the page
> is processed 
> in a way i did not look about any further.
> But if i seek in my new Queue - which means just setting
> the play position 
> to any value - then the demuxer stumbles, because the
> checksum and the 
> length of the page? are not correct.
> To get around this i did a little trick:
> On my new FIFO i seek though the incoming data and look for
> the OggS 
> pattern. If it is present the data will be split into two
> segments and 
> both will be inserted info the FIFO. Now every page starts
> at the 
> beginning of a FIFO buffer:
> 
> +------0-------------1-------------2------------3-----
> | OggS...data | OggS...data.|....data? ? |
> OggS...
> +-----------------------------------------------------
> 
> As you can see, sometimes the data is lager than the 4096
> byte of one 
> buffer, so it takes more than one buffer for a page. I've
> taken care of 
> that in the seeking method by rewinding the play position
> until the buffer 
> starts with an OggS pattern.
> But now a second problem exists: The seeking should start
> if one page is 
> completely submittet. And thats where i'm stucking now.
> 
> The pads of the elements are separate threads which work
> independently 
> from each other. 
> 
> ----+? ? ? ? +-----------[QueueSeek
> element]---------+? ? ?
> ???+----
> prev|? ? ? ? |? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ?
> ???|???to? ? | sink
> pad
> src O--push->O sink pad --> [FIFO magic] --> src
> pad O--next-->O of the
> pad |? ? ? ? |? ? ?
> ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ?
> ???| element | demuxer
> ----+? ? ? ?
> +---------------------------------------+? ?
> ? ???+----
> 
> Thats why many code blocks are synchronized on the queue
> object, that only 
> one thread can access the queue at one time. Now, i want so
> synchronize 
> the sink- and sourcepad threads of the QueueSeek class on
> another object, 
> which is called "permissionToSeek". The sinkpad should wait
> until it gets 
> a permission to seek.
> I tried to achieve this with wait() and notifyAll().
> the sinkpad has a method which gets called if a seek
> request arrives. It 
> synchronizes on the permission object and waits for the
> srcpad to grant 
> it:
> 
> private void doSeek(Event event) {
> ? ? // wait until we get the permission to seek
> ? ? synchronized (permissionToSeek) {
> ? ? try {
> ? ? ? ? Debug.info("[BUFFER] sink ->
> doSeek() waiting for 
> permissionToSeek");
> ? ? ? ? // wait until we are allowed to
> seek
> ? ? ? ? permissionToSeek.wait();
> ? ? } catch (InterruptedException e) {
> ? ? ? ? e.printStackTrace();
> ? ? }
> ? ? /* ... code to seek ... */
> }
> 
> The sinkpad checks if the next buffer begins with OggS
> pattern and calls 
> the notifyAll() method:
> 
> protected void taskFunc() {
> ? ? boolean runSeekIfWanted = false;
> ? ? synchronized (queue) {
> ? ? ? ? /* ... */
> ? ? ? ? if (playPosition+1 <
> queue.size()) {
> ? ? ? ? ? ? obj > queue.elementAt(playPosition+1);
> ? ? ? ? ? ? if (obj
> instanceof Buffer) {
> ? ? ? ? ? ? ? ?
> Buffer bb = (Buffer) obj;
> ? ? ? ? ? ? ? ? if
> (bb.data[0] == 'O' &&
> ? ? ? ? ? ? ? ?
> ? ? bb.data[1] == 'g' &&
> ? ? ? ? ? ? ? ?
> ? ? bb.data[2] == 'g' &&
> ? ? ? ? ? ? ? ?
> ? ? bb.data[3] == 'S') {
> ? ? ? ? ? ? ? ?
> ? ? runSeekIfWanted = true;
> ? ? ? ? ? ? ? ?
> ? ? Debug.info("[BUFFER] src setting
> permissionToSeek");
> ? ? ? ? ? ? ? ? }
> ? ? ? ? ? ? }
> ? ? ? ? }
> ? ? ? ? /* ... */
> ? ? }
> ? ? /* ... main method code ... */
> ? ? synchronized (permissionToSeek) {
> ? ? ? ? if (runSeekIfWanted) {
> ? ? ? ? ? ?
> Debug.info("[BUFFER] src granting permissionToSeek");
> ? ? ? ? ? ?
> permissionToSeek.notifyAll();
> ? ? ? ? }
> ? ? }
> }
> 
> The problem is that the applet stops in the doSeek() method
> at the wait() 
> and waits forever. The whole applet does not respond
> anymore:
> 
> [INFO] [BUFFER] playPosition = 324, queue.size = 669
> Elements, size = 
> 1359872 bytePlayPosition = 678325 -> 64% full
> [INFO] [BUFFER] src granting permissionToSeek
> [INFO] [BUFFER] playPosition = 325, queue.size = 669
> Elements, size = 
> 1359872 bytePlayPosition = 679936 -> 64% full
> [INFO] [BUFFER] playPosition = 326, queue.size = 669
> Elements, size = 
> 1359872 bytePlayPosition = 679962 -> 64% full
> [INFO] [BUFFER] src setting permissionToSeek
> [INFO] doSeek(): aPos=0.05273069679849341
> [INFO] doSeek(): Element: [pipeline].sendEvent [Event]
> type: SEEK, format: 
> 5, position: 52730
> [INFO] Element: [pipeline] sendEvent(): [Event] type: SEEK,
> format: 5, 
> position: 52730
> [INFO] Element: [pipeline] doSeek(): [Event] type: SEEK,
> format: 5, 
> position: 52730
> [INFO] Element: [pipeline] doSendEvent(): [Event] type:
> SEEK, format: 5, 
> position: 52730 to Pad: httpsrc:src
> [INFO] [BUFFER] sink -> doSeek() waiting for
> permissionToSeek
> 
> If i do a wait for 5 seconds to get around this issue, i
> get discontinue 
> messages in the console, and i don't know why:
> 
> [INFO] theora: got discont
> [INFO] theora: got discont
> [INFO] theora: got discont
> 
> So my first question is:
> Is my whole thinking right with providing full pages to the
> demuxer? Is 
> that the right way for seeking in a Theora video stream?
> Why does my applet wait forever? I suppose the srcpad does
> not fire the 
> notifyAll() method but i have no idea why or where the
> srcpad gets stopped 
> or interrupted.
> 
> I will provide the whole code project in a .tar.bz here, so
> everyone can 
> use and experiment with my code changes:
> http://www.wapkamera.de/cortado.tar.bz
> 
> Hopefully someone is able to help me, because i just
> started in coding 
> Java and i don't know the Theora video stream structure in
> detail.
> 
> Best regards to everyone.
> 
> David
> _______________________________________________
> theora-dev mailing list
> theora-dev at xiph.org
> http://lists.xiph.org/mailman/listinfo/theora-dev
>

Tom Sparks <tom_a_sparks at yahoo.com.au> wrote on 05.23.2011 02:16:11 PM:
> I think byte-range serving[1] would be easy to do
> or use Chunked transfer encoding[2] 
> 
> [1] http://en.wikipedia.org/wiki/Byte_serving
> [2] http://en.wikipedia.org/wiki/Chunked_transfer_encoding
Hello Tom,

thanks for your reply.

The problem is that the data is generated on the fly. So there is no 
buffering possible on the server. I want to keep the traffix on the server 
low, so the best practice is to transfer all data to the client, which can 
seek in its buffer. For example Flash does the same.

So i think doing HTTP Content-Range requests and holding the data buffered 
on the server is no option for me.