Ian Andrew Bell
2004-Nov-22 16:20 UTC
[Icecast] Re-Assembling SongData in Icecast Streams..
Like any other major radio station we output our music from automation software via analog audio through a mix board in a studio, where we insert other stuff like live DeeJays, etc. only to have that stream re-encoded by hardware MP3 encoders for distribution to our network of IceCast servers. The chain of song information gets broken as soon as we output from the Automation software, although nowadays most of the software such as RCS and Dalet can output raw text or XML. They had to do this for guys like XM and Sirius. While we re-unite song data with the stream via our Flash-based player at www.pulverradio.com but you can always tell such radio stations on Live365 or Shoutcast.com because they don't pump out song data with their stream, which makes listening to the streams on dedicated hardware or desktop clients less interesting. Can there be a way for Icecast to somehow re-include Song Data? Perhaps by ripping it from a web-available XML file and grafting it back into the stream there'd be some interesting possibilities for more big-budget broadcasters who actually move their audio through hundred-thousand-dollar recording studios. That way they'd be as cool as the guys who are dumping from their old PCs in their basement. :) Any interest in putting something like this together? I don't have it all sewn up in my head just yet but it might make an interesting discussion. -Ian. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2363 bytes Desc: not available Url : http://lists.xiph.org/pipermail/icecast/attachments/20041122/9812a20f/smime.bin
At 06:19 PM 11/22/2004, you wrote:>Like any other major radio station we output our music from automation >software via analog audio through a mix board in a studio, where we insert >other stuff like live DeeJays, etc. only to have that stream re-encoded by >hardware MP3 encoders for distribution to our network of IceCast servers. > >The chain of song information gets broken as soon as we output from the >Automation software, although nowadays most of the software such as RCS >and Dalet can output raw text or XML. They had to do this for guys like >XM and Sirius. > >While we re-unite song data with the stream via our Flash-based player at >www.pulverradio.com but you can always tell such radio stations on Live365 >or Shoutcast.com because they don't pump out song data with their stream, >which makes listening to the streams on dedicated hardware or desktop >clients less interesting.First off, from your description is seems that you might be a bit confused as to how metadata (song titles, etc) is handled and managed so I'll provide a little background. I am going to assume that you are dealing only with MP3, as metadata is handled differently (for instance) with vorbis. For MP3 streams, all metadata is updated in the streaming server via various administrative interfaces. Shoutcast has one (admin.cgi?mode=updinfo&pass=password&song=Booga) and Icecast has a different one (it also supports the Shoutcast style now too). Either way, the metadata is always sent (normally by the source client - i.e. the application sending the stream to the server - but not necessarily) to the streaming server. A common misconception is that the streaming servers somehow pull out ID3 tags embedded in the stream, they do not. For mp3 streams they all take in metadata via the various admin interfaces I described above. For delivering the metadata to listening clients, there is really only one supported mechanism, and that is the "shoutcast-metadata-protocol". A really great explanation of this is here http://www.smackfu.com/stuff/programming/shoutcast.html. Most mp3 clients support this protocol but not all do (Windows Media Player does NOT, and the various Flash players that you are using do NOT). Most other media players that people tend to use (Winamp, Foobar, XMMS) all do. So, enough background, to your specific problem. If you are looking to get metadata into WMP, or Flash (or other clients that do not support the shoutcast-metadata-protocol) you are pretty much out of luck. Outside of developing your own custom version (as you did with your flash player). For all other clients, you should be able to support metadata in the way you want, but you are just missing the piece that extracts the metadata from your repository (via XML or flatfile) and updates the streaming server via the well-defined admin interface. This should not really be the job of icecast or shoutcast (it's job is to primarily stream to clients), so you are probably looking at a fairly straightforward python script that parses an XML file, pulls out the metadata and then sends it to icecast/shoutcast. sorry for rambling on about this, but this question seems to come up quite a bit so I thought I would try to expound upon the topic a bit. Anyway, Vorbis streaming is quite different than MP3 (in many ways). Many streaming codecs (definitely all the ones supported by Shoutcast, which includes AAC, MP3, and NSV (which is technically a container)) are streamed in "passthru" fashion, in that the streaming server really need not know what is being streamed (outside of a content-type). What this means is that a listener can pretty much start at any point in the stream and begin listening, think of it as a big pipe with water flowing out of it, if you are thirsty, you can pretty much take a drink of water any time you want. As an aside, in the trunk of icecast we are now supporting this "passthru" mode, and as such now support the streaming of AAC/AACPlus as well as NSV. The vorbis spec has a small oddity that causes a bit of trouble at first, and thats the fact that each streaming server MUST cache the vorbis header packets that are sent at the beginning of the vorbis stream. This requirement means that the streaming server MUST know that the data coming in is a vorbis stream, and must somewhat intelligently interpret it (in order to pull out the vorbis header packets). It must save off these header packets because the spec says that the header packets must be the *first* thing sent to connecting listeners. So, in vorbis' case, you can just take a drink whenever you want, you have to first pick up the header packets before taking a sip of the water coming from the pipe (bad analogy I know, but lets go with it)...So what does this have to do with metadata ? Well, in these header packets is the metadata (stored as vorbis comments, which are name-value pairs of string. By convention, TITLE=X and ARTIST=Y are used by most media players to display "what's playing" in the player. So, in this case, it is still the responsibility of the source client to set the metadata, but it does it by embedding it into the actual vorbis stream (and it does it in a much better, more efficient, and more elegant way). So how do you update metadata in a vorbis stream ? Well, via some vorbis magic calls, you tear down the logical vorbis stream (not socket!) and rebuild a new vorbis stream with new header packets - which the streaming server must detect and cache (if you were following me above). The downside to this is that it falls apart when your metadata is not accessible by your source client (such is your specific case). If anyone ever wondered why you can't update the metadata in a vorbis stream via the admin interface, it's because in that case icecast would need to take this update, and rebuild it's cached header packets, which can be a bit tricky in all cases. Karl's branch does it, and I'm trying to get it integrated into the trunk right now. ok, I think I've gone on enough about how metadata is handled in a few different places...The only reason I'm being so in depth is that I don't think it's terribly common knowledge, and I tend to answer questions on it all the time... oddsock
Ian Andrew Bell
2004-Nov-24 02:00 UTC
[Icecast] Re-Assembling SongData in Icecast Streams..
Right... I understand how the stream works and we have no confusion at all then. So I need to create a script to re-inject MP3 Metadata from my handy-dandy XML file (when it changes) which is easily done if, as you say, there're a few variables that the admin.cgi can listen for (of course Icecast has no admin.cgi) The info you've provided for Shoutcast is useful and you allude to some other icecast methods ... need documentation for this function of icecast and googling produces no joy, nor do the official or unofficial guides for icecast. The Smackfu link is OK as a cursory introduction to how to pull attributes from a stream but not so instructive in injecting them in and it lists one of the variables. What are the variables used in the shoutcast-metadata-protocol and how do they work? I'm guessing through some extensive googling and have found the following... but a definitive answer and list of variables would be very, very helpful: streamtitle streamname streamurl streamgenre title artist album .... By inference I'm guessing that the shoutcast method is therefore: http://server:port/admin.cgi? mode=updinfo&pass=password&song=Booga&streamtitle=Fubar .... ad nauseum. ...and the password that is relevant is the admin password. Of course, I'm running Icecast 2.1 now. What's the icecast method and what variable names are supported... and are there any that AREN'T used in the shoutcast-metadata-protocol? Is it as simple as: http://server:port/admin/stats.xsl? mode=updinfo&pass=password&song=Booga&streamtitle=Fubar ??? -Ian. On 22-Nov-04, at 7:15 PM, oddsock wrote:> At 06:19 PM 11/22/2004, you wrote: >> Like any other major radio station we output our music from >> automation software via analog audio through a mix board in a studio, >> where we insert other stuff like live DeeJays, etc. only to have that >> stream re-encoded by hardware MP3 encoders for distribution to our >> network of IceCast servers. >> >> The chain of song information gets broken as soon as we output from >> the Automation software, although nowadays most of the software such >> as RCS and Dalet can output raw text or XML. They had to do this for >> guys like XM and Sirius. >> >> While we re-unite song data with the stream via our Flash-based >> player at www.pulverradio.com but you can always tell such radio >> stations on Live365 or Shoutcast.com because they don't pump out song >> data with their stream, which makes listening to the streams on >> dedicated hardware or desktop clients less interesting. > First off, from your description is seems that you might be a bit > confused as to how metadata (song titles, etc) is handled and managed > so I'll provide a little background. > > I am going to assume that you are dealing only with MP3, as metadata > is handled differently (for instance) with vorbis. For MP3 streams, > all metadata is updated in the streaming server via various > administrative interfaces. Shoutcast has one > (admin.cgi?mode=updinfo&pass=password&song=Booga) and Icecast has a > different one (it also supports the Shoutcast style now too). Either > way, the metadata is always sent (normally by the source client - i.e. > the application sending the stream to the server - but not > necessarily) to the streaming server. A common misconception is that > the streaming servers somehow pull out ID3 tags embedded in the > stream, they do not. For mp3 streams they all take in metadata via > the various admin interfaces I described above. For delivering the > metadata to listening clients, there is really only one supported > mechanism, and that is the "shoutcast-metadata-protocol". A really > great explanation of this is here > http://www.smackfu.com/stuff/programming/shoutcast.html. Most mp3 > clients support this protocol but not all do (Windows Media Player > does NOT, and the various Flash players that you are using do NOT). > Most other media players that people tend to use (Winamp, Foobar, > XMMS) all do. > > So, enough background, to your specific problem. If you are looking > to get metadata into WMP, or Flash (or other clients that do not > support the shoutcast-metadata-protocol) you are pretty much out of > luck. Outside of developing your own custom version (as you did with > your flash player). For all other clients, you should be able to > support metadata in the way you want, but you are just missing the > piece that extracts the metadata from your repository (via XML or > flatfile) and updates the streaming server via the well-defined admin > interface. This should not really be the job of icecast or shoutcast > (it's job is to primarily stream to clients), so you are probably > looking at a fairly straightforward python script that parses an XML > file, pulls out the metadata and then sends it to icecast/shoutcast. > > sorry for rambling on about this, but this question seems to come up > quite a bit so I thought I would try to expound upon the topic a bit. > Anyway, Vorbis streaming is quite different than MP3 (in many ways). > Many streaming codecs (definitely all the ones supported by > Shoutcast, which includes AAC, MP3, and NSV (which is technically a > container)) are streamed in "passthru" fashion, in that the streaming > server really need not know what is being streamed (outside of a > content-type). What this means is that a listener can pretty much > start at any point in the stream and begin listening, think of it as a > big pipe with water flowing out of it, if you are thirsty, you can > pretty much take a drink of water any time you want. As an aside, in > the trunk of icecast we are now supporting this "passthru" mode, and > as such now support the streaming of AAC/AACPlus as well as NSV. The > vorbis spec has a small oddity that causes a bit of trouble at first, > and thats the fact that each streaming server MUST cache the vorbis > header packets that are sent at the beginning of the vorbis stream. > This requirement means that the streaming server MUST know that the > data coming in is a vorbis stream, and must somewhat intelligently > interpret it (in order to pull out the vorbis header packets). It > must save off these header packets because the spec says that the > header packets must be the *first* thing sent to connecting listeners. > So, in vorbis' case, you can just take a drink whenever you want, you > have to first pick up the header packets before taking a sip of the > water coming from the pipe (bad analogy I know, but lets go with > it)...So what does this have to do with metadata ? Well, in these > header packets is the metadata (stored as vorbis comments, which are > name-value pairs of string. By convention, TITLE=X and ARTIST=Y are > used by most media players to display "what's playing" in the player. > So, in this case, it is still the responsibility of the source client > to set the metadata, but it does it by embedding it into the actual > vorbis stream (and it does it in a much better, more efficient, and > more elegant way). So how do you update metadata in a vorbis stream ? > Well, via some vorbis magic calls, you tear down the logical vorbis > stream (not socket!) and rebuild a new vorbis stream with new header > packets - which the streaming server must detect and cache (if you > were following me above). The downside to this is that it falls apart > when your metadata is not accessible by your source client (such is > your specific case). If anyone ever wondered why you can't update the > metadata in a vorbis stream via the admin interface, it's because in > that case icecast would need to take this update, and rebuild it's > cached header packets, which can be a bit tricky in all cases. > Karl's branch does it, and I'm trying to get it integrated into the > trunk right now. > > ok, I think I've gone on enough about how metadata is handled in a few > different places...The only reason I'm being so in depth is that I > don't think it's terribly common knowledge, and I tend to answer > questions on it all the time... > > oddsock > > _______________________________________________ > Icecast mailing list > Icecast@xiph.org > http://lists.xiph.org/mailman/listinfo/icecast >-------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2363 bytes Desc: not available Url : http://lists.xiph.org/pipermail/icecast/attachments/20041124/875dd3a5/smime.bin