Icecast - Aug 2004 - OOPS! (was BUG: page-aligned data in libshout)

>
> The bug has nothing to do with libshout I think, maybe, but it's more
> probobal with libvorbis.  The reason it was not crashing with strait up
> data but was segfaulting with page-aligned data was not because it was
> page-aligned but because I was also changing the serial number.. with
> the first stream being # 0.
>
> Apparently, if I bump the initial serialno to 1 it works great.
Committed for a fix for this to libshout cvs. Thanks for the bug report.

Mike

--- >8 ----
List archives:  http://www.xiph.org/archives/
icecast project homepage: http://www.icecast.org/
To unsubscribe from this list, send a message to
'icecast-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

>
> The bug has nothing to do with libshout I think, maybe, but it's more
> probobal with libvorbis.  The reason it was not crashing with strait up
> data but was segfaulting with page-aligned data was not because it was
> page-aligned but because I was also changing the serial number.. with
> the first stream being # 0.
>
> Apparently, if I bump the initial serialno to 1 it works great.
Hrmm.. Yep, looks like a libshout bug/limitation (it's not a libvorbis bug).
By the way, serialno is a 32 bit field, not 16 bit, so you should be 
resetting it after it gets to 2^32, not 2^16.

Mike

--- >8 ----
List archives:  http://www.xiph.org/archives/
icecast project homepage: http://www.icecast.org/
To unsubscribe from this list, send a message to
'icecast-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

Sorry everyone, one major flaw in my proof:

On Sat, May 17, 2003 at 02:07:36AM -0400, Arc wrote:
> 
> The last test, and the way the code currently stands, proves that the
> bug is in sending libshout page-aligned data:
> 
> [...]
>       self.shout.open()
>       self.shout.send('OggS')
>    def __call__(self, data) :
> [...]
>          out=self.page.header[3:]+self.page.body+'OggS'
>          print 'Sending Page %d' % self.page.pageno
>          self.shout.sync()
>          self.shout.send(data)
>          self.page=newpage
>       return
I was still sending it the raw data, but in a loop so it was doing the
whole broken record thing. that tipped me off, looked back, figured it
out..

<p>The bug has nothing to do with libshout I think, maybe, but it's
more
probobal with libvorbis.  The reason it was not crashing with strait up
data but was segfaulting with page-aligned data was not because it was
page-aligned but because I was also changing the serial number.. with
the first stream being # 0.

Apparently, if I bump the initial serialno to 1 it works great.

<p>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: part
Type: application/pgp-signature
Size: 188 bytes
Desc: not available
Url :
http://lists.xiph.org/pipermail/icecast-dev/attachments/20030517/85182a13/part.pgp

I've spent three days tracking this bug down.. I have yet to find how to
fix it in the code, tho I do have a workaround which proves the nature
of the bug.  Sorry for the somewhat-overkill nature of explaining this
bug, but some people have been doubting it so I'm going to give the
evidence nessesary to prove the nature of it.

If you send page-aligned data to libshout (via shout_send) it segfaults
on the first page containing vorbis data within the vorbis_packet_blocksize
function.  Gdb traceback as follows:

#0  0x282da38a in vorbis_packet_blocksize () from /usr/local/lib/libvorbis.so.2
#1  0x28419827 in blocksize (vd=0x8234400, p=0xbfbff54c) at vorbis.c:61
#2  0x28419aa4 in send_vorbis (self=0x81c6a80, data=0x827d014 "OggS",
len=8153) at vorbis.c:124
#3  0x2841732d in shout_send (self=0x81c6a80, data=0x827d014 "OggS",
len=8153) at shout.c:173
#4  0x28411486 in pshoutobj_send () from
/usr/local/lib/python2.2/site-packages/shout.so
#5  0x080b0c53 in PyCFunction_Call ()

I waded through libshout for more than a few hours trying to figure out
what could cause this, then went to look in libvorbis (and quickly gave
up seeing the complexity of it).

It was very predictable, segfaulting while sending pageno 2 no matter
what bitstream it tried to send.  I verified the data by having it save
to disk vs send through libshout, the page spliting function is working
great.  Then I tried to duplicate it on another machine..

Same OS I compiled identical versions of libogg, libvorbis, libshout,
py-ogg (my extended version), py-vorbis, py-shout (by brendan), and
python just to be on the safe side.  Then I ran brendan's example.py (vs
attempting to run the exact same python program), worked fine.  Then I
tried his example.py on the first machine, it also worked fine.

I then went back to the buggy code, still segfaulting, this is the block
of code that was generating the segfault.. note this does more than
split the pages, it's purpose is to "clean up" a physical
bitstream
where logical chained bitstreams may be unterminated (no EOS page) or
where duplicate logical bitstreams are chained sequentially which
confuses players because they have the same serialno... :

class shoutogg :
   def __init__(self) :
      self.ogg = OggSyncState()
      self.page = 0
      self.serial = 0
      self.shout = Shout()
      self.shout.host = stream.host
      self.shout.port = stream.port
      self.shout.user = stream.user
      self.shout.password = stream.password
      self.shout.mount = stream.mount
      self.shout.format = 'vorbis'
      self.shout.protocol = 'http'
      self.shout.open()
   def __call__(self, data) :
      self.ogg.bytesin(data)
      while 1:
         newpage = self.ogg.pageseek()
         if newpage == None :
            break
         if self.page == 0 :
            self.page = newpage
            continue
         if self.page.serialno != newpage.serialno : self.page.eos = 1
         if newpage.pageno==0 : self.page.eos = 1
         self.page.serialno=self.serial
         if self.page.eos == 1 :
            if self.serial<65536 : self.serial = self.serial + 1
            else : self.serial = 0
         print 'syncing'
         self.shout.sync()
         out=self.page.header+self.page.body
         print 'sending page %d' % self.page.pageno
         self.shout.send(out)
         print 'sent'
         self.page=newpage
      return

So I changed the end of it to this:

   def __call__(self, data) :
      self.ogg.bytesin(data)
      buffer=''
      while 1:
         newpage = self.ogg.pageseek()
         if newpage == None :
            break
         if self.page == 0 :
            self.page = newpage
            continue
         if self.page.serialno != newpage.serialno : self.page.eos = 1
         if newpage.pageno==0 : self.page.eos = 1
         self.page.serialno=self.serial
         if self.page.eos == 1 :
            if self.serial<65536 : self.serial = self.serial + 1
            else : self.serial = 0
         out=self.page.header+self.page.body
         print 'buffering page %d' % self.page.pageno
         buffer=buffer+out
         self.page=newpage
      if buffer!='' :
         print 'sending %d bytes' % len(buffer)
         self.shout.sync()
         self.shout.send(buffer)
      return

It still crashed.  I increased the size of the data being sent to the
function to 20k, and it continued to segfault in the same way even tho
it was sending multiple pages (vorbis headers plus data in first piece)
at once, it was still page-aligned. then I changed it like this:

   def __call__(self, data) :
      self.ogg.bytesin(data)
      while 1:
         newpage = self.ogg.pageseek()
         if newpage == None :
            break
         if self.page == 0 :
            self.page = newpage
            continue
         if self.page.serialno != newpage.serialno : self.page.eos = 1
         if newpage.pageno==0 : self.page.eos = 1
         self.page.serialno=self.serial
         if self.page.eos == 1 :
            if self.serial<65536 : self.serial = self.serial + 1
            else : self.serial = 0
         out=self.page.header+self.page.body
         print 'looked at page %d (%d bytes)' %
(self.page.pageno,len(out))
         self.page=newpage
      self.shout.sync()
      self.shout.send(data)
      return

So it still processed the pages the same as it always was, but ignored
them and sent the non-page aligned input data anyways.  Guess what?
  - It worked -

The last test, and the way the code currently stands, proves that the
bug is in sending libshout page-aligned data:

[...]
      self.shout.open()
      self.shout.send('OggS')
   def __call__(self, data) :
[...]
         out=self.page.header[3:]+self.page.body+'OggS'
         print 'Sending Page %d' % self.page.pageno
         self.shout.sync()
         self.shout.send(data)
         self.page=newpage
      return

As you can see it de-page-aligns the data by an offset of four, and this
code works beautifully, both fixing the ogg stream before sending it to
libshout and also doing so in a way that doesnt cause it to die. :-)

I assume the bug causes problems with normal streams, likely one out of
4k pages result in a segfault?  I can only guess.  Anyways I hope I have
provided enough information to track this down, it's too complex for me.


--
------------------------------------------------------------------------

Use GNU Privacy Guard to protect your email, see [http://www.gnupg.org/]

The attachement to this and every email I send is my GPG signature which
is used to verify that I am the sender and it is unmodified by any third
party.  You need GNU Privacy Guard installed to verify my GPG signature.

This was important even before our government's war on civil liberties,
but with the USA PAT RIOT Act signed into law it's especially important
to secure our right to communicate freely without federal surveillance.

If you have GNU Privacy Guard, please use it to encrypt and sign any and
all mail you send me.  I can help you in setting up and using encryption
to protect your email as well as host workshops for teaching your group.

------------------------------------------------------------------------


-------------- next part --------------
A non-text attachment was scrubbed...
Name: part
Type: application/pgp-signature
Size: 188 bytes
Desc: not available
Url :
http://lists.xiph.org/pipermail/icecast-dev/attachments/20030517/336c099d/part.pgp