Philipp, Thank you for your detailed and enlightening answers. Just to make sure I understand how 'bitrate' works here. Assuming we are just working with audio to keep it simple. Let's also ignore metadata, as my use case is that I am generating an electronic music stream from ffmpeg and broadcasting that using Icecast, so I can control/calculate that. I presume that if we specify `audio_bitrate` for a particular mount, that is a 'target' rate that Icecast will attempt to transmit audio data at. You mentioned that the actual rate can vary. Am I right to assume this is because it is dependent on the source (ffmpeg) rate for filling the 'queue', which in turn is used to fill each buffer of each connected client? If so, wouldn't it generally be best for the mount `audio_bitrate` to be set to the same bitrate that the source is generating/sending at? I have the option of generating and compressing my mp3 stream at either a fixed bitrate or to use a LAME VBR setting, and from what I am now understanding it seems that using a fixed bitrate could be better to avoid leading or lagging the queue buffer. Thanks for educating all of us! On Thu, Dec 16, 2021 at 9:28 AM Philipp Schafft < phschafft at de.loewenfelsen.net> wrote:> Good afternoon, > > On Wed, 2021-12-15 at 13:40 -0800, Milton Huang wrote: > > Hi all, > > I just want to make sure I understand what the `queue-size` setting > > does in icecast.xml. My understanding is that for each mountpoint, a > > buffer of that size (default 0.5 MB) is maintained for serving to all > > connected clients. Each client is fed from that buffer, and if their > > connection lags so they can't keep up with the queue contents, they > > get kicked with a 'client has fallen too far behind' message in the > > log. > > That is basically right. However I would like to add that the default > queue size is fine for audio streaming and may need to be adjusted for > video streaming. If you see a lot of said messages in logs and you are > using the default the problem is likely not the value but something > else (e.g. the network being saturated). Or: If you think changing this > value will help you, rethink as it likely is not. > > > > I assume if we divide the queue-size by the mount's bitrate we would > > get the duration of how slow a connection can be which is a limit on > > possible latency of clients. > > This is wrong. Sadly a common myth. > > The 'bitrate' of a stream is generally 0) not constant, 1) only applies > to the audio data, not the stream. > > as for 0) it can easily jump between 0.1% and 200% of the nominal value > for many codecs. > > as for 1) the stream consists of more than just the audio data. E.g. it > contains of framing, setup headers, and metadata. Metadata can easily > account for the same amount of data than several seconds, sometimes > minutes of audio data. > > Keeping this in mind a useful value of a bitrate for a stream needs to > be calculated over at least an entire segment (track/song/title) > including all of it's metadata and format overhead. Also as metadata > may be very different for different tracks the value may still be > 'jumping around' a lot. > > > All of that said it is surely possible to create a stream with a more > constant or foreseeable bitrate. But this is NOT the general case. > > > So you calculation above may at best give a very rough estimation. > > > > On the client side, there is an input buffer to help with poor > > connections. > > The input buffer needs to be there independent on the connection > quality, but what a good size for it is, depends mostly on the quality > of the connection. > > > > This will be filled at the initial connection with a burst-on-connect > > if enabled in the icecast settings. There will be an initial delay in > > play while the buffer is filled, which the burst should help reduce. > > This is perfectly correct. > > > > Obviously, the burst-size has to be smaller than the queue size, or > > you will defeat the purpose of the burst. > > This is not really true. The burst-size is a request to Icecast. > However Icecast will not send exactly this amount of bytes as it needs > to adhere external constrains. One of them is how much data it has. So > setting this to an overly large value will just let Icecast send as > much as possible. > > Other such constrains are format specific aspects such as setup headers > and metadata, but also framing/syncing. > > > As a small addition: > Similar holds true for the queue-size. E.g. if Icecast has not yet got > a full queue-size of data from a source the buffer is smaller. > Depending on how the data is chunked (this depends on the version of > Icecast, the format, the operating systems, the sources, the network, > ...) the size might be slightly smaller or larger. > > Generally those values should be considered as requests and Icecast > tries to work with them as good as possible. > > > > Do I have this all right? Any comments or clarifications? > > Please see my comments inline. :) > > > With best regards, > > -- > Philipp Schafft (CEO/Gesch?ftsf?hrer) > Telephon: +49.3535 490 17 92 > > L?wenfelsen UG (haftungsbeschr?nkt) Registration number: > Bickinger Stra?e 21 HRB 12308 CB > 04916 Herzberg (Elster) VATIN/USt-ID: > Germany DE305133015 > _______________________________________________ > Icecast mailing list > Icecast at xiph.org > http://lists.xiph.org/mailman/listinfo/icecast >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.xiph.org/pipermail/icecast/attachments/20211221/7ae1594b/attachment.htm>
Good evening, On Tue, 2021-12-21 at 14:33 -0800, Milton Huang wrote:> Philipp, > > Thank you for your detailed and enlightening answers.you're very welcome.> Just to make sure I understand how 'bitrate' works here. Assuming we > are just working with audio to keep it simple. Let's also ignore > metadata,Ok. Just a word of warning to the general audience: This is a very limited view on the problem and might not apply to real deployments or fail to be valid at random.> as my use case is that I am generating an electronic music stream > from ffmpeg and broadcasting that using Icecast, so I can > control/calculate that.> I presume that if we specify `audio_bitrate` for a particular mount, > that is a 'target' rate that Icecast will attempt to transmit audio > data at.No, see below.> You mentioned that the actual rate can vary. Am I right to assume > this is because it is dependent on the source (ffmpeg) rate for > filling the 'queue', which in turn is used to fill each buffer of > each connected client?No. The bitrate depends on the codec, the encoder parameters, and the audio. E.g. bitrate can drop to virtually (or actually) zero for moments of silence as there just is no entropy to encode.> If so, wouldn't it generally be best for the mount `audio_bitrate` to > be set to the same bitrate that the source is generating/sending at?It is generally recommended to set as few options on the Icecast side as possible. Also see below.> I have the option of generating and compressing my mp3 stream at > either a fixed bitrate or to use a LAME VBR setting, and from what I > am now understanding it seems that using a fixed bitrate could be > better to avoid leading or lagging the queue buffer.Generally encoding with fixed bitrate tends to harm the quality of the signal. How much error is introduced depends again on the codec, the encoder parameters, and the signal. A quality based encoding is normally best as this results in the encoder trying to keep the signal on the same quality level. Bitrate is approximately a function of entropy*quality. The more constant you try to keep it the more quality of the signal will change with changes in entropy. Change of the amount of entropy per time in a signal is a very common event. E.g. you have a radical change between voice and music, but also between different kinds of music/compositions (fade- in/intro/fade-out/outro, instrumental-only, instruments+voice, simpler/more complex parts of a track, ...). With flexible transports such as IP based networks I see little use of constant or semi-constant bitrate modes. The main applications that comes to mind are fixed bitrate transports channels such as radio channels (e.g. GSM slots). Back to Icecast: Ignoring bursts and format specific headers, as well as syncing for a moment (which is exactly what happens once a listener is fully connected) Icecast sends data out as soon as it receives it. Icecast does not implement any bitrate control. There are no delays (which would be the only way for Icecast to do this, as it can not send data before it received it). All bitrate related options are cosmetic only (e.g. for user friendly display in directory services). Please note that this holds true for all official Icecast versions. Forks or custom versions may differ here.> Thanks for educating all of us!Again, just happy if this helps anyone. :) With best regards,> On Thu, Dec 16, 2021 at 9:28 AM Philipp Schafft < > phschafft at de.loewenfelsen.net> wrote: > > Good afternoon, > > > > On Wed, 2021-12-15 at 13:40 -0800, Milton Huang wrote: > > > Hi all, > > > I just want to make sure I understand what the `queue-size` > > setting > > > does in icecast.xml. My understanding is that for each > > mountpoint, a > > > buffer of that size (default 0.5 MB) is maintained for serving to > > all > > > connected clients. Each client is fed from that buffer, and if > > their > > > connection lags so they can't keep up with the queue contents, > > they > > > get kicked with a 'client has fallen too far behind' message in > > the > > > log. > > > > That is basically right. However I would like to add that the > > default > > queue size is fine for audio streaming and may need to be adjusted > > for > > video streaming. If you see a lot of said messages in logs and you > > are > > using the default the problem is likely not the value but something > > else (e.g. the network being saturated). Or: If you think changing > > this > > value will help you, rethink as it likely is not. > > > > > > > I assume if we divide the queue-size by the mount's bitrate we > > would > > > get the duration of how slow a connection can be which is a limit > > on > > > possible latency of clients. > > > > This is wrong. Sadly a common myth. > > > > The 'bitrate' of a stream is generally 0) not constant, 1) only > > applies > > to the audio data, not the stream. > > > > as for 0) it can easily jump between 0.1% and 200% of the nominal > > value > > for many codecs. > > > > as for 1) the stream consists of more than just the audio data. > > E.g. it > > contains of framing, setup headers, and metadata. Metadata can > > easily > > account for the same amount of data than several seconds, sometimes > > minutes of audio data. > > > > Keeping this in mind a useful value of a bitrate for a stream needs > > to > > be calculated over at least an entire segment (track/song/title) > > including all of it's metadata and format overhead. Also as > > metadata > > may be very different for different tracks the value may still be > > 'jumping around' a lot. > > > > > > All of that said it is surely possible to create a stream with a > > more > > constant or foreseeable bitrate. But this is NOT the general case. > > > > > > So you calculation above may at best give a very rough estimation. > > > > > > > On the client side, there is an input buffer to help with poor > > > connections. > > > > The input buffer needs to be there independent on the connection > > quality, but what a good size for it is, depends mostly on the > > quality > > of the connection. > > > > > > > This will be filled at the initial connection with a burst-on- > > connect > > > if enabled in the icecast settings. There will be an initial > > delay in > > > play while the buffer is filled, which the burst should help > > reduce. > > > > This is perfectly correct. > > > > > > > Obviously, the burst-size has to be smaller than the queue size, > > or > > > you will defeat the purpose of the burst. > > > > This is not really true. The burst-size is a request to Icecast. > > However Icecast will not send exactly this amount of bytes as it > > needs > > to adhere external constrains. One of them is how much data it has. > > So > > setting this to an overly large value will just let Icecast send as > > much as possible. > > > > Other such constrains are format specific aspects such as setup > > headers > > and metadata, but also framing/syncing. > > > > > > As a small addition: > > Similar holds true for the queue-size. E.g. if Icecast has not yet > > got > > a full queue-size of data from a source the buffer is smaller. > > Depending on how the data is chunked (this depends on the version > > of > > Icecast, the format, the operating systems, the sources, the > > network, > > ...) the size might be slightly smaller or larger. > > > > Generally those values should be considered as requests and Icecast > > tries to work with them as good as possible. > > > > > > > Do I have this all right? Any comments or clarifications? > > > > Please see my comments inline. :)-- Philipp Schafft (CEO/Gesch?ftsf?hrer) Telephon: +49.3535 490 17 92 L?wenfelsen UG (haftungsbeschr?nkt) Registration number: Bickinger Stra?e 21 HRB 12308 CB 04916 Herzberg (Elster) VATIN/USt-ID: Germany DE305133015 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 228 bytes Desc: This is a digitally signed message part URL: <http://lists.xiph.org/pipermail/icecast/attachments/20211222/c74e0f27/attachment.sig>
I have the queue size customised. I run 320Kbps bitrates by default (we're mostly all fibre over here) along side the odd 64Kbps for monitoring, and while using burst, and a customised queue I know roughly that a client sits at the estimated 8 seconds ahead of the queue (bitrate? / 8? x 8). With the default queue, if there are any issues with burst the clients get dropped to much where latency is increased (ie: international). The second reason is when I play with low bitrate streams, customising the queue means I can have the delay to the client at roughly the same point in time regardless of bitrate being used. But for 320Kbps, I found the default queue not good enough. There's the opposite problem with this though, if you don't set a queue for low bitrates for each mount point, those low bitrate queues become reallllllllyyyyy delayed. On 22/12/2021 11:33 am, Milton Huang wrote:> Philipp, > > Thank you for your detailed and enlightening answers. Just to make > sure I understand how 'bitrate' works here. Assuming we are just > working with audio to keep it simple. Let's also ignore metadata, as > my use case is that I am generating an electronic music stream from > ffmpeg and broadcasting that using Icecast, so I can control/calculate > that.? I presume that if we specify `audio_bitrate` for a particular > mount, that is a 'target' rate that Icecast will attempt to transmit > audio data at. You mentioned that the actual rate can vary. Am I right > to assume this is because it is dependent on the source (ffmpeg) rate > for filling the 'queue', which in turn is used to fill each buffer of > each connected client? > > If so, wouldn't it generally be best for the mount `audio_bitrate` to > be set to the same bitrate that the source is generating/sending at?? > I have the option of generating and compressing my mp3 stream at > either a fixed bitrate or to use a LAME VBR setting, and from what I > am now understanding it seems that using a fixed bitrate could be > better to avoid leading or lagging the queue buffer. > > Thanks for educating all of us! > > On Thu, Dec 16, 2021 at 9:28 AM Philipp Schafft > <phschafft at de.loewenfelsen.net <mailto:phschafft at de.loewenfelsen.net>> > wrote: > > Good afternoon, > > On Wed, 2021-12-15 at 13:40 -0800, Milton Huang wrote: > > Hi all, > > I just want to make sure I understand what the `queue-size` setting > > does in icecast.xml. My understanding is that for each mountpoint, a > > buffer of that size (default 0.5 MB) is maintained for serving > to all > > connected clients. Each client is fed from that buffer, and if their > > connection lags so they can't keep up with the queue contents, they > > get kicked with a 'client has fallen too far behind' message in the > > log. > > That is basically right. However I would like to add that the default > queue size is fine for audio streaming and may need to be adjusted for > video streaming. If you see a lot of said messages in logs and you are > using the default the problem is likely not the value but something > else (e.g. the network being saturated). Or: If you think changing > this > value will help you, rethink as it likely is not. > > > > I assume if we divide the queue-size by the mount's bitrate we would > > get the duration of how slow a connection can be which is a limit on > > possible latency of clients. > > This is wrong. Sadly a common myth. > > The 'bitrate' of a stream is generally 0) not constant, 1) only > applies > to the audio data, not the stream. > > as for 0) it can easily jump between 0.1% and 200% of the nominal > value > for many codecs. > > as for 1) the stream consists of more than just the audio data. > E.g. it > contains of framing, setup headers, and metadata. Metadata can easily > account for the same amount of data than several seconds, sometimes > minutes of audio data. > > Keeping this in mind a useful value of a bitrate for a stream needs to > be calculated over at least an entire segment (track/song/title) > including all of it's metadata and format overhead. Also as metadata > may be very different for different tracks the value may still be > 'jumping around' a lot. > > > All of that said it is surely possible to create a stream with a more > constant or foreseeable bitrate. But this is NOT the general case. > > > So you calculation above may at best give a very rough estimation. > > > > On the client side, there is an input buffer to help with poor > > connections. > > The input buffer needs to be there independent on the connection > quality, but what a good size for it is, depends mostly on the quality > of the connection. > > > > This will be filled at the initial connection with a > burst-on-connect > > if enabled in the icecast settings. There will be an initial > delay in > > play while the buffer is filled, which the burst should help > reduce. > > This is perfectly correct. > > > > Obviously, the burst-size has to be smaller than the queue size, or > > you will defeat the purpose of the burst. > > This is not really true. The burst-size is a request to Icecast. > However Icecast will not send exactly this amount of bytes as it needs > to adhere external constrains. One of them is how much data it has. So > setting this to an overly large value will just let Icecast send as > much as possible. > > Other such constrains are format specific aspects such as setup > headers > and metadata, but also framing/syncing. > > > As a small addition: > Similar holds true for the queue-size. E.g. if Icecast has not yet got > a full queue-size of data from a source the buffer is smaller. > Depending on how the data is chunked (this depends on the version of > Icecast, the format, the operating systems, the sources, the network, > ...) the size might be slightly smaller or larger. > > Generally those values should be considered as requests and Icecast > tries to work with them as good as possible. > > > > Do I have this all right? Any comments or clarifications? > > Please see my comments inline. :) > > > With best regards, > > -- > Philipp Schafft (CEO/Gesch?ftsf?hrer) > Telephon: +49.3535 490 17 92 > > L?wenfelsen UG (haftungsbeschr?nkt)? ? ?Registration number: > Bickinger Stra?e 21? ? ? ? ? ? ? ? ? ? ?HRB 12308 CB > 04916 Herzberg (Elster)? ? ? ? ? ? ? ? ?VATIN/USt-ID: > Germany? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?DE305133015 > _______________________________________________ > Icecast mailing list > Icecast at xiph.org <mailto:Icecast at xiph.org> > http://lists.xiph.org/mailman/listinfo/icecast > <http://lists.xiph.org/mailman/listinfo/icecast> > > > _______________________________________________ > Icecast mailing list > Icecast at xiph.org > http://lists.xiph.org/mailman/listinfo/icecast-- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.xiph.org/pipermail/icecast/attachments/20211222/54433b60/attachment.htm>