Vorbis - May 2004 - Various Ogg Vorbis largefile notes and/or patches

Greetings one and all;

I'm not subscribed to this list so I'm first sending this message
to verify that mails from me make it through, and then later I'll
send the juicy messages with patches.  Also, the address I'm using
is IPv6-only and doesn't often work, so drop me from any replies
and I'll catch the archives, or drop only the hostname part to get
an IPv4 address that doesn't always work either.

Anyway, I've been working with ogg123/oggenc and large files as
both source and destination.  (Large files being those several GB
in size, for which 64-bits are needed.)  This has exposed some
weaknesses that I've tried to hack around.  I'll explain in messages
to follow, each problem that I'm addressing along with brutish hacks
and/or patches that I've concocted to solve the observed problem.

The platform I've used is FreeBSD which is generally clean for large
files; others who are familiar with other platforms will need to fit
my hacks into a more general case.  My source code is a couple months
old by now since I've been offline and unable to update, so perhaps
some of my hacks are no longer relevant.

The problems I've noted, to be addressed in following messages (if
this one makes it through), include:
*)  WAV files (that store the alleged file size in a 32-bit header
    field) can easily be many gigs in size, but oggenc believes the
    header value rather than the actual file size;
*)  An Ogg Vorbis file greater than 4GB in size required a handful
    of hacks to the ogg123 source before it would partially play --
    still I haven't gotten playback to be perfect;
*)  After some time, playback of a long file would degenerate into
    static.

There may be others; I don't have my messages/hacks at hand right now.

Anyway, in some days or weeks or whenever I manage to get online
again, I'll send the messages describing the above problems in more
detail, as well as my alleged solutions for them, to help oggenc and
ogg123 handle files with days worth of audio.

thanks,
barry bouwsma

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

On Monday 24 May 2004 16:41, Barry Bouwsma wrote:
> The problems I've noted, to be addressed in following messages (if
> this one makes it through), include:
> *)  WAV files (that store the alleged file size in a 32-bit header
>     field) can easily be many gigs in size, but oggenc believes the
>     header value rather than the actual file size;
It does that for good reasons, though... When it didn't (in an older
version)
we got bug reports because it was incorrectly using data that wasn't part of
the audio. So we can't just blindly read to the end of the file. You can 
always use raw files and pass the rest of the info on the command line.

Is there actually some way to get the real data size for 'big' WAV
files? Or
is there a magic constant for the data size that is used to mean 'ignore 
this, just read to the end of the file'? If there is, I'd be happy to
look at
a patch for that. 

I'd also be unsurprised if there are places in oggenc where datatypes need 
changing anyway to work properly with large files - patches for those are 
very welcome regardless.

Mike

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

[this address is IPv6-only, and usually doesn't work, so don't
reply directly to me.  I'll catch the archives next time online]

I drooled the following all over my keyboard:
> Anyway, I've been working with ogg123/oggenc and large files as
> both source and destination.  (Large files being those several GB
> in size, for which 64-bits are needed.)  This has exposed some
> weaknesses that I've tried to hack around.  I'll explain in
messages
> to follow, each problem that I'm addressing along with brutish hacks

Hello.  For my next trick, I'll try to submit an alleged patch (more
like a hack) for a problem I've noticed when playing long-running ogg
vorbis files.

It appears that when playing via ogg123 a lengthy audio file (say, ten
hours at CD-sampling-rate stereo), with FreeBSD I get choppy static
mixed in the audio after several hours.  I don't know if other OSen
suffer the same.  This static is not in the encoded file, as I can
successfully seek within the file to the position where I'm hearing
the static and the playback is clean -- for a few hours.

My guess is that something is counting the amount of raw data output
(the static starts after what would be raw PCM data of 2 or 4GB, can't
remember precisely) and overflows.  Or something.  Nevertheless, I've
added a hack to my ogg123 source, and for more than the past 14 hours
I've been playing an encoded file and so far I've not encountered this
static, which usually should have happened long before now.  (Update:
I ran the playback for more than two days at which point I tripped
over the power cord, but if it could get that far without experiencing
the problems, I suspect they aren't likely to recur.  Sometimes I hate
my clumsiness.)

I believe the hack is in this comparison of 3 or 4 values, some of
which are ints, some long, and some 64-bit values.  When the value of
the PCM data size exceeds that which can be stored in a long, this
comparison probably bombs, explaining the static I used to hear.
I'm no programmer, so if someone wants to do this properly, I won't
be offended.



--- /stand/work/vorbis/vorbis-tools/ogg123/buffer.c-DIST	Thu Sep  4 03:53:02
2003
+++ /stand/work/vorbis/vorbis-tools/ogg123/buffer.c	Fri May  7 00:41:15 2004
@@ -165,7 +165,7 @@

int compute_dequeue_size (buf_t *buf, int request_size)
{
-  int next_action_pos;
+  ogg_int64_t next_action_pos;

/*
For simplicity, the number of bytes played must satisfy the following
@@ -180,10 +180,11 @@

next_action_pos = buf->actions->position;

-    return MIN4(buf->curfill, request_size,
-		buf->size - buf->start, next_action_pos - buf->position);
+    return MIN4((ogg_int64_t)buf->curfill, (ogg_int64_t)request_size,
+		(ogg_int64_t)(buf->size - buf->start),
+		next_action_pos - buf->position);
} else
-    return MIN3(buf->curfill, request_size, buf->size - buf->start);
+    return MIN3(buf->curfill, (long)request_size, buf->size -
buf->start);

}



(apologies, I'm not sure if this is a complete patch.  my source
tree is a mess.  by the time I post everything, I hope it will
all apply cleanly and that I exterminate my pointless debuggery.)

thanks
barry bouwsma

[this address is IPv6-only, and usually doesn't work, so don't
reply directly to me.  I'll catch the archives next time online]

I slobbered the following all into my keyboard:
> Anyway, I've been working with ogg123/oggenc and large files as
> both source and destination.  (Large files being those several GB
> in size, for which 64-bits are needed.)  This has exposed some
> weaknesses that I've tried to hack around.  I'll explain in
messages
> to follow, each problem that I'm addressing along with brutish hacks

Howdy.  My next problem was encountered when trying to play an ogg
vorbis file I had encoded, whose size was more than 4GB (nearly 100
hours worth of continuous .. um, music, I guess).

There seem to be a heap of fseek() and ftell() calls that use longs
and can't handle 64-bit-sized files.  FreeBSD (on which I've done these
hacks) has an off_t-capable version of these calls (ftello() and its
ilk), while other OSen may have something like fseek64().  Thus, I
suspect a bit of configure magic as well as a much more comprehensive
hack than my BSD-specific hack is needed in order to handle >4GB size
ogg vorbis files.

First of all, in order to do any sort of seeking to the end of the
file, I needed to replace several fseek() and ftell() with the off_t
versions.  (Corresponding hacks were needed in the http transport to
match.)  Without this, I'd get into a loop lseek()ing in the file.

It seems that part of the vorbisfile library is 64-bit ready; only
the OS-specific calls need to be hacked.  As always, these hacks are
not necessarily applicable to a non-BSD system -- please review!
But to make things easier, I'm waiting with these hacks until the
end of the message, since there's a few sets, then I'll hit you
with them all at once.


Next, giving a time-seek parameter in the file seemed to degenerate
quickly past a certain point.  This seems to be a problem in the
algorithm used to find the value of bisect.

This algorithm uses 64-bit math already; however, when one is
more-or-less squaring a value that doesn't fit into 32 bits, the
64-bit math overflows.  I converted to a float/double, and the
algorithm quickly found a suitable bisect value.  Yay.

My question, though, is: does it matter that the calculated value
of bisect is no longer a clean 64-bit integer?  Someone with more
Clue than I may wish to see if that is a problem.


--- /stand/work/vorbis/vorbis/lib/vorbisfile.c-DIST	Wed Sep 17 04:17:35 2003
+++ /stand/work/vorbis/vorbis/lib/vorbisfile.c	Fri May  7 00:59:52 2004
@@ -1123,18 +1133,24 @@
ogg_int64_t best=begin;

ogg_page og;
+/* XXX */
+/* XXX printf("end = %qd, begin = %qd, begintime = %qd,\n  endtime = %qd,
target = %qd, best = %qd\n", end, begin, begintime, endtime, target, best);
*/
+
while(begin<end){
-      ogg_int64_t bisect;
+      /* XXX ogg_int64_t bisect; */
+      double bisect;

if(end-begin<CHUNKSIZE){
bisect=begin;
}else{
/* take a (pretty decent) guess. */
bisect=begin +
-	  (target-begintime)*(end-begin)/(endtime-begintime) - CHUNKSIZE;
+	  (double)(target-begintime)*(double)(end-begin)/(double)(endtime-begintime) -
CHUNKSIZE;
if(bisect<=begin)
bisect=begin+1;
}
+/* XXX printf("bisect = %f\n", bisect); */
+/* XXX sleep(1); */

_seek_helper(vf,bisect);



With this hack, I can apparently seek at will within the first
2GB or so of the ogg vorbis file.  However, seeking to a time
value past this point results in only a fraction of a second of
audio playback before EOS is reached.  Likewise, seeking to
just before this point results in several seconds of playback
but then EOS is reached at the point where normal playback
ends.

I'm looking into this now.  Or rather, I had been thinking about
looking into it a while ago -- while oggenc had no problem blatting
encoded data onto the file, I wonder if perhaps there's something
being written in the ogg vorbis data including a 32-bit representation
of the file position, or if it's purely something to do with the
playback application (ogg123), before I dig around in the source
code and/or description of the format.





Here are the various hacks I have so far for 64-bit files, as
I threatened above.  As you have certainly noted, there is a
lot of debugging crap and stuff left in these diffs, so if I'm
doing the right thing, you'll want to redo this from scratch.
I think this is complete, but I could be overlooking something.



--- /stand/work/vorbis/vorbis/include/vorbis/vorbisfile.h-DIST	Thu Sep  4
03:50:02 2003
+++ /stand/work/vorbis/vorbis/include/vorbis/vorbisfile.h	Thu May  6 22:56:42
2004
@@ -40,7 +40,7 @@
size_t (*read_func)  (void *ptr, size_t size, size_t nmemb, void *datasource);
int    (*seek_func)  (void *datasource, ogg_int64_t offset, int whence);
int    (*close_func) (void *datasource);
-  long   (*tell_func)  (void *datasource);
+  off_t   (*tell_func)  (void *datasource);
} ov_callbacks;

#define  NOTOPEN   0



--- /stand/work/vorbis/vorbis/lib/vorbisfile.c-DIST	Wed Sep 17 04:17:35 2003
+++ /stand/work/vorbis/vorbis/lib/vorbisfile.c	Fri May  7 00:59:52 2004
@@ -738,7 +746,7 @@
(size_t (*)(void *, size_t, size_t, void *))  fread,
(int (*)(void *, ogg_int64_t, int))              _fseek64_wrap,
(int (*)(void *))                             fclose,
-    (long (*)(void *))                            ftell
+    (off_t (*)(void *))                            ftell
};

return ov_open_callbacks((void *)f, vf, initial, ibytes, callbacks);
@@ -788,7 +796,7 @@
(size_t (*)(void *, size_t, size_t, void *))  fread,
(int (*)(void *, ogg_int64_t, int))              _fseek64_wrap,
(int (*)(void *))                             fclose,
-    (long (*)(void *))                            ftell
+    (off_t (*)(void *))                            ftell
};

return ov_test_callbacks((void *)f, vf, initial, ibytes, callbacks);




--- /stand/work/vorbis/vorbis-tools/ogg123/transport.h-DIST	Wed Dec 19 03:52:54
2001
+++ /stand/work/vorbis/vorbis-tools/ogg123/transport.h	Thu May  6 19:50:27 2004
@@ -48,9 +48,11 @@
data_source_t* (* open) (char *source_string, ogg123_options_t *ogg123_opts);
int (* peek) (data_source_t *source, void *ptr, size_t size, size_t nmemb);
int (* read) (data_source_t *source, void *ptr, size_t size, size_t nmemb);
-  int (* seek) (data_source_t *source, long offset, int whence);
+/* XXX  int (* seek) (data_source_t *source, long offset, int whence); */
+  int (* seek) (data_source_t *source, off_t offset, int whence);
data_source_stats_t * (* statistics) (data_source_t *source);
-  long (* tell) (data_source_t *source);
+/* XXX  long (* tell) (data_source_t *source); */
+  off_t (* tell) (data_source_t *source);
void (* close) (data_source_t *source);
} transport_t;



--- /stand/work/vorbis/vorbis-tools/ogg123/file_transport.c-DIST	Sat Jan 26
12:06:37 2002
+++ /stand/work/vorbis/vorbis-tools/ogg123/file_transport.c	Thu May  6 23:03:28
2004
@@ -77,20 +77,22 @@
}


+/* XXX HACK off_t ... */
int file_peek (data_source_t *source, void *ptr, size_t size, size_t nmemb)
{
file_private_t *private = source->private;
FILE *fp = private->fp;
int items;
-  long start;
+  off_t start;

+/* XXX     fprintf(stderr, _("Called file_peek.....\n")); */
/* Record where we are */
-  start = ftell(fp);
+  start = ftello(fp);

items = fread(ptr, size, nmemb, fp);

/* Now go back so we maintain the peek() semantics */
-  if (fseek(fp, start, SEEK_SET) != 0)
+  if (fseeko(fp, start, SEEK_SET) != 0)
items = 0; /* Flag error condition since we couldn't seek back to
the beginning */

@@ -113,12 +115,14 @@
}


-int file_seek (data_source_t *source, long offset, int whence)
+/* XXX HACK NEEDS TO BE MADE 64-bit-safe off_t ... */
+int file_seek (data_source_t *source, off_t offset, int whence)
{
file_private_t *private = source->private;
FILE *fp = private->fp;

-  return fseek(fp, offset, whence);
+/* XXX     fprintf(stderr, _("Called UNHACKED file_seek offset %qd whence
%d.....\n"), offset, whence); */
+  return fseeko(fp, offset, whence);
}


@@ -130,12 +134,17 @@
}


-long file_tell (data_source_t *source)
+/* XXX HACK NEEDS TO BE MADE 64-bit-safe off_t ... */
+off_t file_tell (data_source_t *source)
{
file_private_t *private = source->private;
FILE *fp = private->fp;
+  ogg_int64_t pos;

-  return ftell(fp);
+  /* XXX return ftello(fp); */
+  pos = ftello(fp);
+/* XXX     fprintf(stderr, _("Called UNHACKED file_tell position
%qd.....\n"), pos); */
+    return pos;
}




--- /stand/work/vorbis/vorbis-tools/ogg123/http_transport.c-DIST	Thu Sep  4
03:53:13 2003
+++ /stand/work/vorbis/vorbis-tools/ogg123/http_transport.c	Thu May  6 19:52:56
2004
@@ -285,7 +285,7 @@
}


-int http_seek (data_source_t *source, long offset, int whence)
+int http_seek (data_source_t *source, off_t offset, int whence)
{
return -1;
}
@@ -309,7 +309,7 @@
}


-long http_tell (data_source_t *source)
+off_t http_tell (data_source_t *source)
{
return 0;
}




--- /stand/work/vorbis/vorbis-tools/ogg123/oggvorbis_format.c-DIST	Thu Sep  4
03:53:14 2003
+++ /stand/work/vorbis/vorbis-tools/ogg123/oggvorbis_format.c	Fri May  7
00:47:21 2004
@@ -271,11 +277,15 @@
return 1; /* Ignore close request so transport can be closed later */
}

-long vorbisfile_cb_tell (void *arg)
+off_t vorbisfile_cb_tell (void *arg)
{
decoder_t *decoder = arg;
+    ogg_int64_t pos;

-  return decoder->source->transport->tell(decoder->source);
+  /* return decoder->source->transport->tell(decoder->source); */
+  pos = decoder->source->transport->tell(decoder->source);
+/* XXX printf("vorbisfile_cb_tell returned %qd\n", pos); */
+    return pos;
}




thanks
barry bouwsma

<200406081133.i58BXFT04661@Mail.NOSPAM.DynDNS.dK>
Message-ID: <20040615121753.GA9687@nowster.zetnet.co.uk>

On Tue, Jun 08, 2004 at 01:33:15PM +0200, Barry Bouwsma wrote:
> Hello.  For my next trick, I'll try to submit an alleged patch (more
> like a hack) for a problem I've noticed when playing long-running ogg
> vorbis files.
Lots of people have been reporting this bug. (It happens mainly when
playing a stream.) Looks like you've found the cause.

--
Paul Martin <pm@zetnet.net> (work)
<pm@nowster.zetnet.co.uk> (home)