Icecast dev - May 2019 - Webm files written without duration in header

Dear all

I am streaming live with webm with ffmpeg to icecast 2.4.2. After the
stream ends, I am unable to determine the duration of the file using
ffprobe or mediainfo. Not sure but it seems that this has to do with
headers?

Should icecast be writing the duration into the header or should this
somehow be passed from ffmpeg?

The requirement is really to determine the duration of the streamed event
(i.e. not radio).

Would appreciate some advice here.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xiph.org/pipermail/icecast/attachments/20190501/9526c7bb/attachment.htm>

Hi,

On 5/1/19 9:58 AM, Sytze Visser wrote:
> I am streaming live with webm with ffmpeg to icecast 2.4.2. After the
> stream ends, I am unable to determine the duration of the file using
> ffprobe or mediainfo. Not sure but it seems that this has to do with
> headers?

It's a fundamental limitation of this type of streaming that the
duration can not be determined beforehand and there is no way for a
simple client to append necessary information.


> Should icecast be writing the duration into the header or should this
> somehow be passed from ffmpeg?

Does this mean you are using the dump feature to write the complete
stream to a file on the machine running Icecast?


> The requirement is really to determine the duration of the streamed
> event (i.e. not radio).

I'm not sure how to interpret this, there seem to be some unstated
assumptions, as e.g. radio broadcast has the same fundamental issues.

Nevertheless there is a trivial solution to address a file that is
missing such information: remux it.
This does not change the payload in any way, the Video and Audio (and
any other) bit-streams will not be changed. Only the WebM container
information gets rewritten. If there are malformed portions of the
stream (caused by e.g. CPU starved encoder or network stall), then those
might get removed though as a player wouldn't play them properly either.

There are many dedicated tools for remuxing MKV and or WebM.
A somewhat brute-force quick fix is of course: `ffmpeg -i foo.webm -c
copy bar.webm`


Cheers,

TBR

Hi there

Thanks Thomas. I was hoping to avoid remuxing as my objective is to run an
extremely lightweight server. Just tested and the CPU runs between 9% and
23% for a lecture of 90 minutes - took 18sec. Yes it's a lightweight
server, but like I said, that's the objective, and quite a nice challenge
:-) CPU = cost and if you have to remux for 1000 clients in a day,
integrity of live streams currently running are at risk. Going to put some
more thinking into it.

*You mentioned:*
"Does this mean you are using the dump feature to write the complete
stream to a file on the machine running Icecast?"

*Yes*. Curious as to why you ask. Can one perhaps point it to a named pipe
and pass to ffmpeg to remux and be a little more efficient on CPU?
Alternatively, can you do <dump-file>ffmpeg -c copy
.....</dump-file> ? Is
it supported? Have been on Linux for less than a year so not sure if at all
possible? I have no idea yet what a 1000 fifo's will do.

*On another note:*
The other BIG issue is to support apple users. iOS/Safari seems to be 11%
of the market (https://caniuse.com/#search=HLS) and I cant ignore them.
Will have to use HLS to keep them happy but again the CPU overhead is
imminent as vorbis/VP8 is miles away from AAC/H.264. Icecast is very CPU
efficient ingesting, streaming and writing out a webm file, CPU 1-2%. Same
efficiency situation with nginx and RTMP streaming in and serving HLS to
iOS users. I read somewhere that MP4 is not supported on icecast, which is
a pity. I have invested a lot in an icecast based audio solution and want
expand it with video.

*So my question is now: *
MP4 (H.264) is a close relative of HLS (can also be H.264) which brings me
very close to a solution. If webm/ogv is supported but mp4 not, what is the
technical differences in how icecast handles the one and not the other?
What does icecast manage/do for supported formats as opposed to unsupported
ones?

Best regards


On Wed, May 1, 2019 at 4:39 PM Thomas B. Rücker <thomas at ruecker.fi>
wrote:
> Hi,
>
> On 5/1/19 9:58 AM, Sytze Visser wrote:
> > I am streaming live with webm with ffmpeg to icecast 2.4.2. After the
> > stream ends, I am unable to determine the duration of the file using
> > ffprobe or mediainfo. Not sure but it seems that this has to do with
> > headers?
>
>
> It's a fundamental limitation of this type of streaming that the
> duration can not be determined beforehand and there is no way for a
> simple client to append necessary information.
>
>
>
> > Should icecast be writing the duration into the header or should this
> > somehow be passed from ffmpeg?
>
>
> Does this mean you are using the dump feature to write the complete
> stream to a file on the machine running Icecast?
>
>
>
> > The requirement is really to determine the duration of the streamed
> > event (i.e. not radio).
>
>
> I'm not sure how to interpret this, there seem to be some unstated
> assumptions, as e.g. radio broadcast has the same fundamental issues.
>
> Nevertheless there is a trivial solution to address a file that is
> missing such information: remux it.
> This does not change the payload in any way, the Video and Audio (and
> any other) bit-streams will not be changed. Only the WebM container
> information gets rewritten. If there are malformed portions of the
> stream (caused by e.g. CPU starved encoder or network stall), then those
> might get removed though as a player wouldn't play them properly
either.
>
> There are many dedicated tools for remuxing MKV and or WebM.
> A somewhat brute-force quick fix is of course: `ffmpeg -i foo.webm -c
> copy bar.webm`
>
>
> Cheers,
>
> TBR
>
>
>
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xiph.org/pipermail/icecast/attachments/20190501/dff49881/attachment.html>

On 2019-05-01 11:58, Sytze Visser wrote:> I am streaming live with webm 
with ffmpeg to icecast 2.4.2. After the
 > stream ends, I am unable to determine the duration of the file using
 > ffprobe or mediainfo. Not sure but it seems that this has to do with
 > headers?
 >
 > Should icecast be writing the duration into the header or should this
 > somehow be passed from ffmpeg?
Do you mean the stream you sent to icecast or the stream listeners get 
from icecast?

If it's the user stream then no, as a listener may drop packets and any 
length would be incorrect, in this case it is the listeners player that 
should track duration played (received) and add this to the file header 
if the player allows archiving of live streams. For livestreams I think 
Icecast sets a length of -1 (but it could be 0, or no length at all).

If you use a stream capture tool then I'm sure there exists one that 
will write the duration header to a file. Have you tried using ffmpeg to 
capture the stream? Look at the manual, you should be able to make it 
write a duration to the header (if possible/supported by the file format)

 > The requirement is really to determine the duration of the streamed
 > event (i.e. not radio).
If you are using ffmpeg as the stream encoder then you should log the 
duration either prior to ffmpeg getting the audio or after ffmpeg 
encoding. There is no way for ffmpeg to magically add a length 
(duration) to the header (which may have been streamed an hour ago).

If you only want to track duration before streaming it "out", then 
encode and store the stream locally, then when it ends run a script to 
add/fix the header to have a duration then stream the file to icecast, 
this would mean the liveness of the stream would be delayed by the 
duration of it.

But I doubt you want to do this (why stream live otherwise right?)

I can't recall if you can get ffmpeg to trigger a script or executable 
when it quits or not, but you could (be it Windows or Linux) easily make 
a script that you call instead of ffmpeg directly, in the script you log 
the time before calling ffmepg and again after ffmpeg quits.
If you log as a datestamp you can simply subtract the start value from 
the end value to get duration. Do note that this may not match the 
actual duration of the archived stream (even if you archive it straight 
from ffmpeg), it could be off by a second for example.

You should also consider how to handle things in case ffmpeg crashes or 
looses the connection to icecast (something sure to happen if using 
public networks instead of a local LAN).

If a script you can "easily" restart ffmpeg and you can log the start 
and end time (you could maybe even detect a ffmpeg crash or quit and log 
a message).

I note you said "streamed event", are you using some form of broadcast
software? In that case that should have it's own log for events, or you 
can set it to trigger a script at the start and end of events.


Regards,
Roger.

-- 
Unless specified otherwise, anything I write publicly is considered 
Public Domain (CC0). My opinions are my own unless specified otherwise.
Roger Hågensen,
Freelancer, Norway.

Hey Roger and everyone else commenting – I appreciate all the advice and 
questions.

I now have more work than before 😊!

Greetings from sunny South Africa!

Sent from Mail for Windows 10

From: Roger Hågensen
Sent: Thursday, May 2, 2019 12:12 PM
To: icecast-dev at xiph.org
Subject: Re: [Icecast-dev] Webm files written without duration in header

On 2019-05-01 11:58, Sytze Visser wrote:> I am streaming live with webm 
with ffmpeg to icecast 2.4.2. After the
 > stream ends, I am unable to determine the duration of the file using
 > ffprobe or mediainfo. Not sure but it seems that this has to do with
 > headers?
 >
 > Should icecast be writing the duration into the header or should this
 > somehow be passed from ffmpeg?
Do you mean the stream you sent to icecast or the stream listeners get 
from icecast?

If it's the user stream then no, as a listener may drop packets and any 
length would be incorrect, in this case it is the listeners player that 
should track duration played (received) and add this to the file header 
if the player allows archiving of live streams. For livestreams I think 
Icecast sets a length of -1 (but it could be 0, or no length at all).

If you use a stream capture tool then I'm sure there exists one that 
will write the duration header to a file. Have you tried using ffmpeg to 
capture the stream? Look at the manual, you should be able to make it 
write a duration to the header (if possible/supported by the file format)

 > The requirement is really to determine the duration of the streamed
 > event (i.e. not radio).
If you are using ffmpeg as the stream encoder then you should log the 
duration either prior to ffmpeg getting the audio or after ffmpeg 
encoding. There is no way for ffmpeg to magically add a length 
(duration) to the header (which may have been streamed an hour ago).

If you only want to track duration before streaming it "out", then 
encode and store the stream locally, then when it ends run a script to 
add/fix the header to have a duration then stream the file to icecast, 
this would mean the liveness of the stream would be delayed by the 
duration of it.

But I doubt you want to do this (why stream live otherwise right?)

I can't recall if you can get ffmpeg to trigger a script or executable 
when it quits or not, but you could (be it Windows or Linux) easily make 
a script that you call instead of ffmpeg directly, in the script you log 
the time before calling ffmepg and again after ffmpeg quits.
If you log as a datestamp you can simply subtract the start value from 
the end value to get duration. Do note that this may not match the 
actual duration of the archived stream (even if you archive it straight 
from ffmpeg), it could be off by a second for example.

You should also consider how to handle things in case ffmpeg crashes or 
looses the connection to icecast (something sure to happen if using 
public networks instead of a local LAN).

If a script you can "easily" restart ffmpeg and you can log the start 
and end time (you could maybe even detect a ffmpeg crash or quit and log 
a message).

I note you said "streamed event", are you using some form of broadcast
software? In that case that should have it's own log for events, or you 
can set it to trigger a script at the start and end of events.


Regards,
Roger.

-- 
Unless specified otherwise, anything I write publicly is considered 
Public Domain (CC0). My opinions are my own unless specified otherwise.
Roger Hågensen,
Freelancer, Norway.
_______________________________________________
Icecast-dev mailing list
Icecast-dev at xiph.org
http://lists.xiph.org/mailman/listinfo/icecast-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xiph.org/pipermail/icecast-dev/attachments/20190502/44299b03/attachment.html>