Vorbis dev - Sep 2004 - copying an ogg stream

dear list,

i am trying to write a small program which reads an ogg file and writes 
it to another ogg file (and changes serial number, granulepos etc on the 
fly).

reading the ogg file is ok (ogg_sync_pageout, ogg_stream_pagein, 
ogg_stream_packetout). but writing the file doesn't work - the 
granulepos and page structures don't match with the original file.

here's what i am doing. (python-like pseudo-code). First i create copies 
of all packets in a page, store them in 'list'; i also save the 
granule-position of this page:

foreach page in oggfile:
   g=get_granulepos(page)
   list=[]
   foreach packet in page:
     list.append(dup_packet(packet))  # creates a deep copy of packet

then i write all pages but the last one to the output file (because, as 
i understand it, only the last packet has the correct granulepos; all 
other packets have a granulepos of -1).

   foreach packet in list[:-1]:   # does NOT include the last packet!
     packet.granulepos=-1
     packet.packetno=global_packetno++
     write_packet(packet)

now i write the last packet (this is also the place where i would set 
e_o_f...):

   packet=list[len(list)-1]
   packet.granulepos=g
   packet.packetno=global_packetno++
   write_packet(packet)


the write_packet function looks like this:

def write_packet(packet):
   ogg_stream_packetin(os, packet)

   ogg_page page;

   while (ogg_stream_pageout(os, page)):
     fwrite(page.header, page.header_len, f)
     fwrite(page.body, page.body_len, f)

(if the e_o_s flag is set, then i use ogg_stream_flush() instead of 
ogg_stream_pageout())

my problem is: when i look at the original file, it differs from the 
copied file. this is the output of the first few lines with oggzdump:

original file:

0000003a: serialno 1295053610, granulepos 0, packetno 0 *** bos:
00000fc7: serialno 1295053610, granulepos -1, packetno 1:
00000fc7: serialno 1295053610, granulepos 0, packetno 2:
00002030: serialno 1295053610, granulepos -1, packetno 3:
00002030: serialno 1295053610, granulepos -1, packetno 4:
00002030: serialno 1295053610, granulepos -1, packetno 5:
...
00002030: serialno 1295053610, granulepos 23296, packetno 38:

the new file:

0000003a: serialno 0000012345, granulepos 0, packetno 0 *** bos:
00001142: serialno 0000012345, granulepos -1, packetno 1:
00001142: serialno 0000012345, granulepos -1, packetno 2:
00001142: serialno 0000012345, granulepos -1, packetno 3:
00001142: serialno 0000012345, granulepos -1, packetno 4:
00002231: serialno 0000012345, granulepos -1, packetno 5:
...
00007527: serialno 0000012345, granulepos 175744, packetno 168:


first of all the granule positions have changed, and also in the 
original file packet no 38 is the last packet of the first page. in the 
new file, it's packet 168! but when stepping through the debugger, i'm 
pretty sure i have submitted the correct packets...

also, ogginfo complains a lot about "Negative granulepos on vorbis 
stream outside of headers. This file was created by a buggy encoder". 
Playback with ogg123 is ok, though.

i hope i have made my case clear and the pseudo code is not too 
overwhelming :) i can also send the "real" C/C++ code, if somebody
wants
to see it.

thanks for every idea!
Chris

Why is it that you break the packets out of the page ?

Why not just keep the pages intact and change their serial no (i assume the 
granule pos change you is to add a constant delta of some kind) ?

Zen.

----- Original Message ----- 
From: "Christoph Rupp" <crupp@umc-web.de>
To: <vorbis-dev@xiph.org>
Sent: Wednesday, September 22, 2004 11:41 PM
Subject: [Vorbis-dev] copying an ogg stream

> dear list,
>
> i am trying to write a small program which reads an ogg file and writes it 
> to another ogg file (and changes serial number, granulepos etc on the 
> fly).
>
> reading the ogg file is ok (ogg_sync_pageout, ogg_stream_pagein, 
> ogg_stream_packetout). but writing the file doesn't work - the
granulepos
> and page structures don't match with the original file.
>
> here's what i am doing. (python-like pseudo-code). First i create
copies
> of all packets in a page, store them in 'list'; i also save the 
> granule-position of this page:
>
> foreach page in oggfile:
>   g=get_granulepos(page)
>   list=[]
>   foreach packet in page:
>     list.append(dup_packet(packet))  # creates a deep copy of packet
>
> then i write all pages but the last one to the output file (because, as i 
> understand it, only the last packet has the correct granulepos; all other 
> packets have a granulepos of -1).
>
>   foreach packet in list[:-1]:   # does NOT include the last packet!
>     packet.granulepos=-1
>     packet.packetno=global_packetno++
>     write_packet(packet)
>
> now i write the last packet (this is also the place where i would set 
> e_o_f...):
>
>   packet=list[len(list)-1]
>   packet.granulepos=g
>   packet.packetno=global_packetno++
>   write_packet(packet)
>
>
> the write_packet function looks like this:
>
> def write_packet(packet):
>   ogg_stream_packetin(os, packet)
>
>   ogg_page page;
>
>   while (ogg_stream_pageout(os, page)):
>     fwrite(page.header, page.header_len, f)
>     fwrite(page.body, page.body_len, f)
>
> (if the e_o_s flag is set, then i use ogg_stream_flush() instead of 
> ogg_stream_pageout())
>
> my problem is: when i look at the original file, it differs from the 
> copied file. this is the output of the first few lines with oggzdump:
>
> original file:
>
> 0000003a: serialno 1295053610, granulepos 0, packetno 0 *** bos:
> 00000fc7: serialno 1295053610, granulepos -1, packetno 1:
> 00000fc7: serialno 1295053610, granulepos 0, packetno 2:
> 00002030: serialno 1295053610, granulepos -1, packetno 3:
> 00002030: serialno 1295053610, granulepos -1, packetno 4:
> 00002030: serialno 1295053610, granulepos -1, packetno 5:
> ...
> 00002030: serialno 1295053610, granulepos 23296, packetno 38:
>
> the new file:
>
> 0000003a: serialno 0000012345, granulepos 0, packetno 0 *** bos:
> 00001142: serialno 0000012345, granulepos -1, packetno 1:
> 00001142: serialno 0000012345, granulepos -1, packetno 2:
> 00001142: serialno 0000012345, granulepos -1, packetno 3:
> 00001142: serialno 0000012345, granulepos -1, packetno 4:
> 00002231: serialno 0000012345, granulepos -1, packetno 5:
> ...
> 00007527: serialno 0000012345, granulepos 175744, packetno 168:
>
>
> first of all the granule positions have changed, and also in the original 
> file packet no 38 is the last packet of the first page. in the new file, 
> it's packet 168! but when stepping through the debugger, i'm pretty
sure i
> have submitted the correct packets...
>
> also, ogginfo complains a lot about "Negative granulepos on vorbis
stream
> outside of headers. This file was created by a buggy encoder".
Playback
> with ogg123 is ok, though.
>
> i hope i have made my case clear and the pseudo code is not too 
> overwhelming :) i can also send the "real" C/C++ code, if
somebody wants
> to see it.
>
> thanks for every idea!
> Chris
> _______________________________________________
> Vorbis-dev mailing list
> Vorbis-dev@xiph.org
> http://lists.xiph.org/mailman/listinfo/vorbis-dev
>
>

Hi Zen,

illiminable wrote:
> Why is it that you break the packets out of the page ?
to set the e_o_s flag. i want to be able to cut off the file in the 
middle...
> Why not just keep the pages intact and change their serial no (i assume 
> the granule pos change you is to add a constant delta of some kind) ?
> 
> Zen.

Chris

On Wed, Sep 22, 2004 at 05:41:56PM +0200, Christoph Rupp wrote:
> reading the ogg file is ok (ogg_sync_pageout, ogg_stream_pagein, 
> ogg_stream_packetout). but writing the file doesn't work - the 
> granulepos and page structures don't match with the original file.
That's because you're changing them. :)
> then i write all pages but the last one to the output file (because, as 
> i understand it, only the last packet has the correct granulepos; all 
> other packets have a granulepos of -1).
This is incorrect. Only the header packets have a granulepos of -1, and 
pages that no packet ends on. That's at least part of the problem. I'm 
not sure what's up with the ordering issues.
>   foreach packet in list[:-1]:   # does NOT include the last packet!
>     packet.granulepos=-1
>     packet.packetno=global_packetno++
>     write_packet(packet)
Note that do to this correctly, you need to infer the granulepos of each 
packet and set that, so repagination is still accurate. The Ogg stream 
only stores the granulepos once per page, but you can always count up 
from the previous page by parsing enough of the packets to get their 
decoded length. This is the only way to accurately adjust the 
granulepos.

Note that Arc wrote a serial number changer in python. I expect it's in 
svn somewhere; check google for references. That only does half of want 
you want though, but it's much simpler since you can just copy pages.

Hope that helps some,
 -r