Icecast - May 2019 - Webm files written without duration in header

Hi there

Thanks Thomas. I was hoping to avoid remuxing as my objective is to run an
extremely lightweight server. Just tested and the CPU runs between 9% and
23% for a lecture of 90 minutes - took 18sec. Yes it's a lightweight
server, but like I said, that's the objective, and quite a nice challenge
:-) CPU = cost and if you have to remux for 1000 clients in a day,
integrity of live streams currently running are at risk. Going to put some
more thinking into it.

*You mentioned:*
"Does this mean you are using the dump feature to write the complete
stream to a file on the machine running Icecast?"

*Yes*. Curious as to why you ask. Can one perhaps point it to a named pipe
and pass to ffmpeg to remux and be a little more efficient on CPU?
Alternatively, can you do <dump-file>ffmpeg -c copy
.....</dump-file> ? Is
it supported? Have been on Linux for less than a year so not sure if at all
possible? I have no idea yet what a 1000 fifo's will do.

*On another note:*
The other BIG issue is to support apple users. iOS/Safari seems to be 11%
of the market (https://caniuse.com/#search=HLS) and I cant ignore them.
Will have to use HLS to keep them happy but again the CPU overhead is
imminent as vorbis/VP8 is miles away from AAC/H.264. Icecast is very CPU
efficient ingesting, streaming and writing out a webm file, CPU 1-2%. Same
efficiency situation with nginx and RTMP streaming in and serving HLS to
iOS users. I read somewhere that MP4 is not supported on icecast, which is
a pity. I have invested a lot in an icecast based audio solution and want
expand it with video.

*So my question is now: *
MP4 (H.264) is a close relative of HLS (can also be H.264) which brings me
very close to a solution. If webm/ogv is supported but mp4 not, what is the
technical differences in how icecast handles the one and not the other?
What does icecast manage/do for supported formats as opposed to unsupported
ones?

Best regards


On Wed, May 1, 2019 at 4:39 PM Thomas B. Rücker <thomas at ruecker.fi>
wrote:
> Hi,
>
> On 5/1/19 9:58 AM, Sytze Visser wrote:
> > I am streaming live with webm with ffmpeg to icecast 2.4.2. After the
> > stream ends, I am unable to determine the duration of the file using
> > ffprobe or mediainfo. Not sure but it seems that this has to do with
> > headers?
>
>
> It's a fundamental limitation of this type of streaming that the
> duration can not be determined beforehand and there is no way for a
> simple client to append necessary information.
>
>
>
> > Should icecast be writing the duration into the header or should this
> > somehow be passed from ffmpeg?
>
>
> Does this mean you are using the dump feature to write the complete
> stream to a file on the machine running Icecast?
>
>
>
> > The requirement is really to determine the duration of the streamed
> > event (i.e. not radio).
>
>
> I'm not sure how to interpret this, there seem to be some unstated
> assumptions, as e.g. radio broadcast has the same fundamental issues.
>
> Nevertheless there is a trivial solution to address a file that is
> missing such information: remux it.
> This does not change the payload in any way, the Video and Audio (and
> any other) bit-streams will not be changed. Only the WebM container
> information gets rewritten. If there are malformed portions of the
> stream (caused by e.g. CPU starved encoder or network stall), then those
> might get removed though as a player wouldn't play them properly
either.
>
> There are many dedicated tools for remuxing MKV and or WebM.
> A somewhat brute-force quick fix is of course: `ffmpeg -i foo.webm -c
> copy bar.webm`
>
>
> Cheers,
>
> TBR
>
>
>
> _______________________________________________
> Icecast mailing list
> Icecast at xiph.org
> http://lists.xiph.org/mailman/listinfo/icecast
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xiph.org/pipermail/icecast/attachments/20190501/dff49881/attachment.html>

On Wed, 2019-05-01 at 19:56 +0200, Sytze Visser wrote:
> I read somewhere that MP4 is not supported on icecast, which is a
> pity. I have invested a lot in an icecast based audio solution and
> want expand it with video. 
It all depends on what one means by 'MP4' (which is more a marketing
term than a technical one). AAC+ (aka 'high efficiency' AAC) works very
well on IceCast, albeit that only covers the audio side on the
equation.

> So my question is now:  
> MP4 (H.264) is a close relative of HLS (can also be H.264) which
> brings me very close to a solution.
You're mixing apples and oranges here (pun accidental). H.264 is a
content encoding, like MPEG Layer III (so-called 'MP3') or OggVorbis.
HLS OTOH is a streaming architecture, and as such is largely orthogonal
to the content encoding used. It works by segmenting the media stream
into discrete files and then distributing them via HTTP(S). The big
advantage of this approach is that standard 'garden variety' web
servers (think Apache) can be used to serve the streams, which makes it
play particularly well with CDNs, at the cost of significantly greater
complexity in the encoder and player components. No specialized
'streaming server' (such as Icecast) is required in the HLS ecosystem
at all.

Cheers!


|---------------------------------------------------------------------|
| Frederick F. Gleason, Jr. |             Chief Developer             |
|                           |             Paravel Systems             |
|---------------------------------------------------------------------|
|  An expert is a person who avoids the small errors while he sweeps  |
|  on to the grand fallacy.                                           |
|                                                                     |
|                                              -- Benjamin Stolberg   |
|---------------------------------------------------------------------|

Hi Fred. 

Appreciate your response. 

Maybe in my explanation I have some red and green apples, but I can 
agree that my understanding is as you explained it. 😊

The point is that if I can successfully stream mp4 with H.264 and AAC 
encoding without any issues to icecast, I can then use ffmpeg to 
turn it into HLS which then solves my iOS support issue. The CPU 
cost of repackaging MP4 into HLS architecture should be minimal
because at the core it’s all H.264. No transcoding! Low CPU objective 
achieved!
Additional bonus is that I already use Apach2 with icecast and 
integrating the whole lot into my existing infrastructure is easy.

My question remains
If webm/ogv is supported but mp4 not, what are the technical differences 
in how icecast handles the one and not the other? What does icecast 
manage/do for supported formats as opposed to unsupported ones?

Kind regards

On Wed, 2019-05-01 at 19:56 +0200, Sytze Visser wrote:
> I read somewhere that MP4 is not supported on icecast, which is a
> pity. I have invested a lot in an icecast based audio solution and
> want expand it with video. 
It all depends on what one means by 'MP4' (which is more a marketing
term than a technical one). AAC+ (aka 'high efficiency' AAC) works very
well on IceCast, albeit that only covers the audio side on the
equation.
> So my question is now:  
> MP4 (H.264) is a close relative of HLS (can also be H.264) which
> brings me very close to a solution.
You're mixing apples and oranges here (pun accidental). H.264 is a
content encoding, like MPEG Layer III (so-called 'MP3') or OggVorbis.
HLS OTOH is a streaming architecture, and as such is largely orthogonal
to the content encoding used. It works by segmenting the media stream
into discrete files and then distributing them via HTTP(S). The big
advantage of this approach is that standard 'garden variety' web
servers (think Apache) can be used to serve the streams, which makes it
play particularly well with CDNs, at the cost of significantly greater
complexity in the encoder and player components. No specialized
'streaming server' (such as Icecast) is required in the HLS ecosystem
at all.

Cheers!


|---------------------------------------------------------------------|
| Frederick F. Gleason, Jr. |             Chief Developer             |
|                           |             Paravel Systems             |
|---------------------------------------------------------------------|
|  An expert is a person who avoids the small errors while he sweeps  |
|  on to the grand fallacy.                                           |
|                                                                     |
|                                              -- Benjamin Stolberg   |
|---------------------------------------------------------------------|

_______________________________________________
Icecast mailing list
Icecast at xiph.org
http://lists.xiph.org/mailman/listinfo/icecast

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xiph.org/pipermail/icecast/attachments/20190501/b1b66c92/attachment.html>

Hi,

On 5/1/19 5:56 PM, Sytze Visser wrote:
> Thanks Thomas. I was hoping to avoid remuxing as my objective is to
> run an extremely lightweight server. Just tested and the CPU runs
> between 9% and 23% for a lecture of 90 minutes - took 18sec. Yes it's
> a lightweight server, but like I said, that's the objective, and quite
> a nice challenge :-) CPU = cost and if you have to remux for 1000
> clients in a day, integrity of live streams currently running are at
> risk. Going to put some more thinking into it.

There should be no need to remux each file more than once, so unless you
produce 1000 files per day, I dont't understand your point.
Even in that case all those files won't be created at the exact same
point in time and even if it is really easy to devise a funnel/queue
that does the remuxing at low priority.

Please note again that remuxing is NOT reencoding, the encoded
information is not touched and it's very quick and efficient because of
that.

> *You mentioned:*
> "Does this mean you are using the dump feature to write the complete
> stream to a file on the machine running Icecast?"
>
> */Yes/*. Curious as to why you ask. Can one perhaps point it to a
> named pipe and pass to ffmpeg to remux and be a little more efficient
> on CPU? Alternatively, can you do <dump-file>ffmpeg -c copy
> .....</dump-file> ? Is it supported? Have been on Linux for less than
> a year so not sure if at all possible? I have no idea yet what a 1000
> fifo's will do.

Just needed to make sure how the files are generated. There may be
slight differences between dumping a stream via e.g. curl or wget vs
using the dump file. Nothing that matters to your use case though as
before presenting any of these you should very much remux them to ensure
file integrity as static files.

Also *please* avoid sending HTML emails to the mailing list. Thanks.

Cheers,

TBR